User blog comment:SDPanthera/Fairy Tail Chapter 415 Review & Other Thoughts/@comment-1776824-20150111134714/@comment-7418318-20150112202446

Not everyone who has the potential to become a mage actually becomes one. Maybe their parents came from very poor circumstance and never learned of the power in their bodies.

A rather unlikely possibility would be some kind of Dragon cult that voluntarily and happily gave their children to the Dragons and felt honoured about it.

Or it could have been an incitend similar to what happened on Tenrou Island in Fairy Tail Zero. Maybe they grew up in a mage guild, but that guild was attacked and destroyed by rivals. We have barely and idea about what happened all over Fiore before X777 and know absolutely nothing about other countries after all, so such an event could have happened without us knowing.

Or maybe the process that turns Humans into Dragon Slayers turns even those into Mage who otherwise couldn't have become Mages. If this process can alter the bodies to such a degree that it sharpens senses and all that, turning the child into a Mage doesn't seem impossible either.