According to Jared Diamond, it's agriculture!
"archaeology is demolishing another sacred belief: that human history over the past million years has been a long tale of progress. In particular, recent discoveries suggest that the adoption of agriculture, supposedly our most decisive step toward a better life, was in many ways a catastrophe from which we have never recovered."
here's one of his cases:
"One straightforward example of what paleopathologists have learned from
skeletons concerns historical changes in height. Skeletons from Greece and Turkey show
that the average height of hunter-gatherers toward the end of the ice ages was a
generous 5'9" for men, 5'5" for women. With the adoption of agriculture, height
crashed, and by 3000 B.C. had reached a low of 5'3" for men ,5' for women. By classical
times heights were very slowly on the rise again, but modern Greeks and Turks have still
not regained the average height of their distant ancestors."
While his cases are (as always) just so, the argument seems iffy. After all, agriculture accompanied a rise in the population which in most endogenous growth models, raises the stock of ideas and long run growth. Plus, if we'd stayed hunter-gatherers neither me or Lebron James would currently stalk this planet and that, people, would be a tragedy indeed.