What if we limit the reputation to 5 per person, per day?
What I mean by this is Player A can only downvote or upvote 5 of Player B’s things. Player A can then go and upvote or downvote Player C’s things, until Player A reaches the limit with that person.
I don’t think removing it entirely would be the best idea but we do need to cut down on mass downvoting (ahem, Noctis :P)