The Feedback Loop
Here is what happens when AI models train on AI-generated data.
Version one: trained on human content. Accurate at the edges. Captures real variation in how people write, think, and argue. Includes the outlier takes, the minority positions, the genuinely weird ideas that turned out to be correct.
Version two: trained partly on outputs from version one. Slightly more homogeneous. The outliers are slightly less represented. Confident errors from version one are amplified rather than corrected. The nuance that came from actual human disagreement is smoothed away.
Version three: trained on outputs from version two, which was trained on outputs from version one. The compounding has begun.
Researchers studying this named it the Habsburg problem. The dynasty that kept marrying within the family to concentrate power eventually produced a king who could not chew his own food. The gene pool collapsed. The AI quality equivalent is already underway.
What It Actually Looks Like
Outputs that are fluent but slightly wrong in ways that are hard to pin down. Confident errors presented without uncertainty. A narrowing of the range of perspectives the model considers. The same framing applied to every problem regardless of whether that framing fits.
The Google AI Overview that told users to add Elmer's glue to pizza was model collapse made visible. A joke from Reddit, processed as factual, reproduced with full confidence. The model did not know it was at several generations of remove from original human thinking.
That is the specific danger. The degradation does not look like degradation. It looks like an answer.
Why Nobody Is Fixing It
The companies building AI models need training data at scale. Human-generated data is finite and increasingly expensive. AI-generated data is cheap, fast, and abundant.
Every model now training on internet content is training partly on outputs from previous AI models. The ratio is increasing. The internet fills faster with AI content than with human content. The training pipeline problem compounds with every new generation of models.
The fix requires deliberately preserving and weighting human-generated data. It requires resisting the pressure to use the cheap, fast alternative. It requires prioritizing quality of training signal over quantity of training data.
That is the opposite direction from where the economic incentives point.
Which is why this keeps happening, and why it will keep getting worse before anyone with the ability to fix it decides to fix it.