No popular tools (maps, routers) would want to use that information, so it all depends on how you want to use it in your particular tool.
You could also move it to something like "building:number" just so that information would remain in the database.