That's it though, if anything they act like a unit not addr:housenumber.

How you have it currently means the house at 30 Wicks Road https://www.openstreetmap.org/node/6395430523 is the same address as https://www.openstreetmap.org/way/332096380 which you've also tagged as addr:housenumber=30 which is then inside the campus which has addr:street=Wicks Road giving the hospital building 30 Wicks Road, which is wrong.

addr:unit is usually for a townhouse or apartment within an address, not usually for building numbers.

I'd still tag these as ref=* since they are reference numbers for the buildings within the campus.

Then separately to ref=* I'd also consider if there's a way to incorporate them into addr:*=* tags, likely just using addr:full for now.