User blog comment:Miskos3/Fairy Tail Chapter 374: Revolution, Review/@comment-5268982-20140307203101/@comment-5268982-20140307212531

Although you do have a point, Gray has met his father in the past, whereas Cana and Gildarts were practically strangers-he smelled as much of Cana on Gildarts as any other guild member. If Gray is the only familiar person that Silver has interacted with, though, it makes sense why Natsu recognized only that smell.

Or Silver could keep a memento or something that belongs to Gray.