Are you removing because of poor quality, or because of licensing issues?  Google Buildings are explicitly allowed to be used in OSM, see https://sites.research.google/open-buildings/#open-buildings-faq

If you are doing this for quality reasons you might want to redraw by hand at the same time you are deleting.