They weren't as far off as one might think. If everyone had a standard of living growing up that scaled with trends at the time, women would probably be taller than they actually are.

When I was 5' 11" and starting college as an honors student, I was shocked that most women in the program were around my height or taller, while the men were all taller than me. I think it was mostly because the other honors students came from well off families that pushed them to have high GPAs, made them eat healthy as they grew, and gave them every other possible advantage. Honestly, they didn't seem much smarter than many other students I later met. They mostly had privileged upbringings.