You are right that there could be a little negative side effect for self upvotes. 

I'm afraid there are many more issues that need to be solved. Like people having more accounts, paid upvote bots, long sign up process, etc.

To give an example, I think that among my latest 50 new followers, there were 48 bots.
So at this moment I seem to have 525 followers, but in reality I estimate that I have 25 followers.