Thanks for the feedback.  I wasn't sure about doing this edit, thinking something like that might be the case.  The reason I ultimately did go ahead with it is because I noticed a signage difference while driving by these buildings.  Named building signs have the name in all caps and the address in proper case on the line below like this:

POMEROY HALL
489 Main Street

Signs for address only buildings use the same proper case as other addresses, and they don't duplicate the address in the all caps name style.  For example like this:

481 Main Street

Not like this:

481 MAIN STREET
481 Main Street

These two examples can both be seen here on Google street view:  https://maps.app.goo.gl/mRSiDEra4z3zQTAJA

To me, this seems to indicate that some signs show a name and an address, while others showed just an address.  So OSM tagging could match that.  However, your experience contradicts that with these addresses seeming to be treated as names among the UVM community.  I wonder if this really makes the address the name of such a building, or if people just use the address in lieu of a name?    On the other hand I am aware of cases where an address is definitively used as a name.  For example this restaurant in Jeffersonville:  https://158main.com/

If you're confident that these buildings really are named by their address (and not just referred to  as such for lack of a name), I'm happy to restore the name tags.  