User blog comment:B214/Chapter 499: Gray & Juvia/@comment-7418318-20160829143550/@comment-5303512-20160829144226

Couple things

1. Magic takes extraordinary mental concentration in order for it to work.

2. Think of those situations like two CoA Haki Logia users fighting, they would end you hitting each other or touching their physical bodies anyway with their energy.