I don't think you understand how hard it is to plow a field without draught animals. They didn't have domestic horses or oxen when farming began. It was incredibly hard work vs. just cutting down wild plants and shooting animals with animals or hitting them with spears. And, of course, processing grains by hand before milling was invented was also very hard work. You can't just eat wheat as-is. You have to turn it into flour and cook with it.

The "lazy" people would be the ones who didn't want to do all of that and instead just walk around the woods until they saw a deer and then shoot it.

The biggest advantage of agriculture over hunter-gathering is storage during cold or dry seasons when foraged food could be harder to come by, but it is not clear that this was an advantage of farming or the reason for it.

This isn't even something we have to infer from ancient peoples. There have been studies of modern peoples that show that hunter-gatherers do not work as hard as farmers, and that is with draught animals and other techniques that were developed after the development of agriculture: https://www.cam.ac.uk/research/news/farmers-have-less-leisure-time-than-hunter-gatherers-study-suggests