Downvotes are complex. And in real world downvotes has different faces. In real world, society ignores specific set of the bad in order to uplift the good. This trick has been used by Woke culture on Netflix so much so they literally curate their own specific interest and ignore everything else that the audience needs. This helps them and frees them from the curation of the content they don't want. This may sound good in theory but it brings up the "freedom of speech" issue. 

If you use same logic on downvotes they can be harvested and opinion of specific can be kept down in the grave. At the end what people can do is they continue to uplift the good content they want. So that way the other opinion is neither silenced nor downvoted. So on second level what happens with bad content, spam is another level to worry about. But I don't think people who want to uplift the good should be dealing with the spam or downvoted content the same way. Because you don't see that happening in real life either. Different people deal with creation, different people deal with maintaining, different set of people for recycling, cleaning etc. Why the same logic can't be used?

My 2 cents.