User blog:Awareness Bringer/Who built the Tower of Heaven?

Hey, everyone. I'm kind of a new guy to Fairy Tail, most of my interest comes from my reading the stories of Fairy Tail on fanfiction.net, but I'll get my point across one way or another. What I want to ask is who built the Tower of Heaven? I know it was designed by Zeref Dragneel and all that, but who were the cultists who used slave labor to build it exactly? What happened to them? How was Ultear Milkovich involved? My first instinct tells me Grimoire Heart were the true masterminds and they sold Brain's future partners in crime to him for whatever reason, but I just want to be sure. I'm kind of a hard questions kind of guy. Thoughts?