I’ve seen downvotes on cat photos. Some people just downvote.
Of course, sometimes it’s because swiping might not always work correctly: I found on an earlier version of Lemmy, before actual arrows, that accidentally downvoting sort of locked the swipe in place, such that I couldn’t switch to an upvote without really fiddling with it, and some might not notice or go back and correct it.
It might be growing pains, it might be trolls, it might be earnest but sincerely pigheaded people: who knows? It’s a small userbase, too, so 3-5 baddies makes up a much larger portion of active users than on even a small subreddit, so I wouldn’t worry too much about it. You can’t please everybody, so just focus on having good conversations with people: the ones who are worth it are listening to you first before worrying about votes.