Yeah, it definitely wasn't a given for the founders. There was a whole lot of talk about only letting landowners vote at one point.
Thomas Jefferson even invented a term for a specific type of political elite that he thought should be in power.
Look at the treatment of black people. Article one, section two of the Constitution counted enslaved people as three-fifths of a person when determining congressional representation. Rights were definitely different based on who you are.
Heck, they still are... There are plenty of examples of modern subcultures with different rights. Gay marriage, etc.
"Yeah his mindset didn't include women or people of color but it doesn't make the core idea invalid"
I'm not sure how you could think that the founding fathers could exclude people and still mean everyone. "Everyone" specifically means not excluding anyone.