Schools of Skepticism seem to fall into two camps: Soft Skepticism, which acknowledges that some knowledge might exist and is attainable for humans, and Hard Skepticism, which either negates the existence of true knowledge or the human capacity to grasp true knowledge. Thus Soft Skepticism would by default allow for at least this one piece of knowledge, namely that we know that we know nothing. Hard Skepticism would even doubt this and state instead, that we know nothing, not even that we know nothing.
I've had similar experiences as the ones that you described, when bringing up epistemological nihilism/skepticism in an argument. The common response being to attack skeptic arguments by applying them to themselves - "If you can not know anything, then you can not know that you know nothing. Therefore your claim refutes itself. Reductio ad absurdum".
The reductio ad absurdum attack however reveals the circular reasoning of its speaker, which you already pointed out as well. The beliefs that a "non-epistemological-skeptic" has are based on assumption whose trueness is in turn in some way or another derived from those very beliefs - e.g. The Bible is true, because it is God's word, and God exists, because the Bible says so. This is what I would refer to as (positive) circular reasoning, because the arguments posit each other. The ad absurdum fallacy against skepticism seems to be part of a type of (negative) circular reasoning, because the arguments negate each other. This is the same problem with a lot of paradoxes and it seems to be caused by self-reference. Self-reference thus seems to be an area where our logic breaks, which gives some credence to the idea, that our theory of logic is not yet complete or fundamentally flawed. Thus even the one area of human wisdom pointed to as a counter argument for hard skepticism seems to fail.
The Münchhausen trilemma or Agrippa's trilemma addresses this issue by positing that every proof falls into this or two other categories:
- Circular reasoning - as explained above
- Infinite regress - continuously and infinitely expanding dependence of arguments
- Dogmatic assumption - reduction to an unjustifiable base assumption
... and thus, that true (objective/un-assumptive) knowledge is unattainable.
Even if we assume that knowledge might be attainable, there may be limits to what we can know. This is called Cognitive closure and it posits that biological or physical limitations exist regarding what thoughts we can conceive and what we are capable of knowing. We are for example incapable of visually imagining four or higher dimensional space - Although the case could be made, that we do not even perceive three dimensional in the first place, but that our perception is entirely two dimensional.
You already mentioned a similar limitation: Linguistic relativity. Our language does to some extent influence how we perceive the world and what thoughts we can formulate. Studies have however shown that this extent is not near as strong as Whorf originally claimed - A language missing temporal conjugation does not mean that a speaker of that language has no concept of time. There are however strong indicators that language shapes how we perceive/think about color, spatial relations, social relations... like a cognitive framework for processing our perceptual input. This does not however mean that language is an actual limitation, since language gives us the ability to conceptualize abstract thought in the first place. It is more akin to an enabler with some languages being better tools for certain cognitive processes than others. I would even argue, that all languages are perfect to process the environments of their corresponding speakers, shaping their perceptions and in turn being shaped by their cultural interactions based on their perceptions.
In summary: No knowledge can be inferred without an assumption. Skepticism does not refute itself, but the assumption that it would seems to point to a flaw in our theory of logic. There seem to exist biological and physical limits on what humans can think and our perception of our world is shaped to some extent by our language.
I hope this answers some of your questions and provides you with a good basis for further research - even though nothing can truly be known ;) - and I would love to hear your further thoughts and ideas on the matter!