Answer: From the buff-colored coats once worn by New York City volunteer firemen who became known as fire buffs.
Source: The 365 amazing trivia facts calendar for 2007