I think that there is a combination of all those things. I believe there are Village agents, "wolves among the sheep", who influence villager actions. I also believe that many of the villagers have long been broken via drugs and/or brainwashing. This makes them pliable to anything that the agents embedded in the community wants them to do. I think that the Village probably doesn't take on more than a handful of people like #6 at any one time for fear that they might team up and influence an easily influenced group of villagers by emulating what the embedded agents do. TLDR, I think most of the villagers are more or less brainwashed lemmings and the ones that aren't are agents.
The Prisoner
Questions are a burden to others; answers, a prison for oneself.
Watch all episodes online for free at https://www.shoutfactorytv.com/series/the-prisoner
And when it comes to The Prisoner, we need to think allegorically. This probably makes your answer right, since in the real world, who are the guards? They're much more than just police, and indeed, "enforcement" is relatively rare in the Village (and usually executed by weather balloon). The guards are, I think, a sort of generalized state apparatus, including (of course) the media. Which is brilliantly demonstrated in "Free for All" when the Tally Ho has an article about Number Six prewritten.
Hence in the real world, for the average middle class white, strict enforcement is pretty rare; we hardly ever have run ins with the fuzz. We respond to softer kinds of power, from propaganda to economic threats.
In the real world, the "drugs and/or brainwashing" can be replaced with poor education and fixation on cults of personality and outrage culture pushed into their brains and eyeballs through 24/7 media coverage. The wolves are the media, the billionaires, and others in power who need the sheep to move and vote in the ways that is often against their own best interest. Sorry to get a little political, but those factors are in play no matter what the ideology is.