Answer:  From the buff-colored coats once worn by New York City volunteer firemen who became known as fire buffs.

 

Source:  The 365 amazing trivia facts calendar for 2007