I'm confident that some buildings were commonly referred to by name same as address (e.g. 70 South Williams, 284 East Avenue, and a few others).

Some others I mostly heard referred to by function or what was in them (e.g. before it was renamed I heard 438 College Street referred to as "the arts and sciences building", similar for 34 South Williams).

I never figured out what was up with the 475-479 thing. IIRC, my instructors who worked there avoided trying to name the building and just said "come to the Asian Languages department". I'd be happy copying the official map and using the name 479 Main Street with proper address tagging for both.

My overall impression is that people on campus get used to buildings having names and just start acting like they all do. This applies both to people talking and to official sources like the campus map and internal facilities data I worked with that always had a name field filled in.

Of course, all my knowledge is out of date and also UVM specific, I don't know how things work at Champlain.