On Friday, OpenAI said that it will train its models using cached copies and simulated versions of the internet instead of the live web because the models are too unpredictable to be trusted with live internet access.
The frontier lab announced its new policy two weeks after its rival, Anthropic, began limiting its newest and, in theory, most capable models from accessing the live internet. Researchers adopted these policies after months of incidents in which models pursuing training tasks broke into external systems in search of answers, collaborated on obscure message boards, and developed exotic hacking strategies.
Not possible.
Think photocopy of a photocopy, or JPEG of a JPEG.
You start getting artifacts, and the model collapses.
Of course, though, they’ve already poisoned the well, and at this point it’s impossible for them to sieve out any remaining non-LLM-generated content. They started watermarking too late, and couldn’t recognise other slop machines’ watermarks anyway.
Model collapse is inevitable and already ongoing (search “thousand whys" on Amazon, for instance, or read up on Elias Thorne, lighthouse keeper), and will only get worse.
In a few years it will be impossible to train a working up-to-date model. Maybe it already is, hence this article, as a preemptive justification for why newer models won’t be able to generate code using language features introduced after 2026.