The building is on Grant Street and across the street from the main Boys and Girls Club building.
The office building has been declared a total loss, including old pictures and records that were kept inside.
"The items that can't be replaced, 73 years worth of history gone in a couple hours is pretty devestating. On top of that, the program supplies and things we had in the building for the kids," Sheila Ryba, of the John Avery Boys and Girls Club said.
Ryba said for now, they will have to try to get new sports and music equipment for the kids, which could be difficult in a time when money for nonprofits is scarce.
"The bright spot is it's not where our kids, they do not use it on a daily basis," Ryba said.
The fire chief told ABC11 no one was injured during the blaze.
The cause of the fire is under investigation.