User blog comment:FairyRaven/Fairy Tail Chapter 425: Sabertooth X792, Review and Commentary/@comment-121.54.54.60-20150323130134/@comment-7418318-20150323174827

There were plenty of other people who worshipped Zeref as their god without being Demons themselves.

Remember that the people who build the Tower of Heaven before Jellal killed their leaders and took over were the remnants of a cult big enough to build several of those towers. That speaks of some considerable manpower.