Close relatives, yes, but not so much with cousins. And it wasn't until Darwin that they truly started to grasp what was going on.

It's easy to spot the recurring issue of diseased/sickly/malformed offspring with incest between siblings, or between parents and children, because the rate of birth defects is much higher. Fourfold, in fact.

Cousins didn't produce them nearly as much, so it wasn't an obvious enough trend. It's perfectly possible for first cousins to have healthy offspring more than half the time. In fact some studies have found the risk of genetic defeats is not much higher than a regular couple where the women is over the age of 40. Which is to say, it's low, but not low enough to ignore. There's also a lot of other factors involved especially when it comes to closed ethnic groups that tend to only reproduce amongst themselves.