I tend to shy away from downvoting too. Unless it's blatant spam, or abuse of the system I generally look the other way. I've never downvoted based on a difference of opinion. The voting system definitely has its pros and cons.