If the buildings aren't egregiously wrong I would certainly not remove them from the map even if they may not be as high quality as anyone would like. 

Obviously at a certain point data could become bad enough that you might consider it vandalism, at which point removing them would be justified. But I that obviously isn't the case here, and it would be a good idea to discuss this with other mappers before making a deletion.

If you don't feel existing building outlines are good enough, you should improve them yourself, rather than deleting them. It preserves the data and also importantly the history of the object.