My skin is mostly blemish free aside from a few moles, some scars from old piercings, and the occasional spot of hyperpigmentation from old wounds.

Why would you think otherwise?

I use my own homemade true soap (I'm convinced modern detergents are much harsher than they need to be and that a superfat of olive and jojoba oil is good for my skin) if a simple wash with water and a towel won't clean me off.

Skin will only get angry if you strip away protective oils or clog up the pores with who knows what. I don't scent my soap either, perhaps that makes a difference?