That's a thought exercise I first explored when I was tripping on LSD one time. Once you accept that colors could look entirely different, what's to stop us from imagining that the entire makeup of our perception is adjusted to the individual, but the chaos of the universe keeps all of our unique perceptions in enough alignment that we can interact seamlessly with one another. The concept of an "alien green man" could be what someone actually sees the whole human species as, and the words "alien green man" just fit a different description to that individual, so when refrenced, that person envisions an alien being as foreign as the "alien green man" is to us, and can communicate with others on that concept.

Perception defines your experience