They're not perfect, but they're still accurate in that there's a house in that location. Even people can't map houses perfectly, I've corrected plenty that have been mapped manually.

Deleting those objects means that instead of knowing there's a house there, we now don't know that. That's a significant loss when you delete several streets worth.

If you're not happy with the minute level of inaccuracy from these buildings, either fix them up, or raise the issue with the people importing them. 