Can AI Build Its Own Successors? Scientists Map the Path to the 'Last AI Built by Humans'

Researchers in China have outlined what it might take to create "the last AI built by humans" - a self-improving system capable of producing its own successors. Their paper, posted to the arXiv preprint database on Sept. 10 and not yet peer-reviewed, proposes a five-level framework for classifying AI systems by their capacity for recursive self-improvement (RSI), the idea that models could guide their own training and development.

At the lowest levels, humans still lead development. As systems climb the scale, they would increasingly spot their own weaknesses and decide how to fix them, eventually improving how they improve themselves.

Limited forms of RSI have already been demonstrated. In one study this year, an AI system called A-Evolve-Training modestly boosted the benchmark performance of a 30 billion-parameter large language model over four rounds by testing training techniques and adjusting its approach.

Not everyone is convinced. Peter Bentley, a professor of digital biology at University College London, called the notion that AI will magically generate ever-better AIs "nonsense," noting that self-improving systems have existed for decades through evolutionary computation and reinforcement learning. The real hurdle, he said, is building AI that can autonomously conduct the research needed for more powerful successors - and overcome its dependence on human expertise and computing infrastructure.

The study's authors acknowledged such obstacles, noting that RSI-capable systems could demand enormous computing power and human supervision, and that the effort becomes pointless if the resources required outweigh the benefits. Scientists would also need reliable ways to confirm genuine capability gains rather than models gaming their tests.

Kayla Blomquist, an AI researcher at the University of Oxford and co-founder of the Oxford China Policy Lab, said the paper offers no practical road map to RSI but could help developers and policymakers grasp the technical and safety challenges of increasingly autonomous AI. She also questioned whether a clear endpoint will ever exist. The paper, she said, describes a gradual handover of responsibility rather than a single moment - so there may never be an obvious "last" model built by humans.