You are correct about how our skin has these natural defenses that are much more sophisticated than most people are aware of, but semi-regular hygiene is still important to clear off the debris left behind by this ecosystem, as has been known for some thousands of years. Once every couple days to a week is ideal for most people depending on factors such as humidity and personal biology.
Exposure to "everyday contagions" can greatly strengthen the immune system without making us sick as well, but not always, and the behavior of primates and young children eating their little green nose warriors is actually believed to have evolved for this benefit. Exposure has become more of an issue as civilization has grown in size though, and contagious diseases are being spread simultaneously from multiple points in the global population with more opportunities to find their way into that circle.
If you have a lifestyle that involves regular contact with other people, you may want to be washing your hands frequently.
We are starting a homestead and getting away from all the fuss.