It is a personal choice what you do, but there is so much blatant abuse, e.g. buying votes on junk comments and posts. This platform is too precious to let people wreck it like that. They do not deserve to profit and they reduce what others can make. Of course there are some I would be nervous to take on, but I am not relying on my Steem income and so will not be too upset if I lose some. You cannot stop someone making use of their SP, but some previous abusers have been persuaded to change their ways by campaigns of downvotes as well as actually talking to them. Some do not even seem to realise that what they were doing was considered wrong, but I find that hard to believe really.

I tend to shy away from downvoting too. Unless it's blatant spam, or abuse of the system I generally look the other way. I've never downvoted based on a difference of opinion. The voting system definitely has its pros and cons.