The Self-Improving AI Puzzle: Why Full Automation Remains Out of Reach

The concept of machines that upgrade themselves has captured the attention of the AI world. Two startups now carry the name Recursive. Dozens more list self-improvement as a core goal. The promise is tantalizing: a system that handles its own research, coding, and testing, leaving humans as spectators. But turning that vision into reality has proven stubbornly difficult.

Definitions vary widely. Some see it as an AI that can rewrite its own code without human help. Others imagine a closed loop where every step of advancement is automated. Richard Socher recently launched Recursive Superintelligence with hundreds of millions in backing, aiming to automate the entire research process. Andrej Karpathy at Anthropic experiments with agent swarms that refine smaller models, though gains have been modest.

Sara Hooker’s Adaption released AutoScientist, which automates parts of model training. Doris Xin’s team at Disarray trained an agent that won 28 medals in Kaggle competitions. Yet Xin notes the real barrier is reliability. “Given infinite compute and infinite time, we are already there,” she said. “This is just a lot of engineering work.”

Industry leaders remain measured. Google CEO Sundar Pichai described progress as a continuum. “We aren’t quite there yet,” he said. An Anthropic engineer estimated that nearly all code for a recent project came from the model itself. A survey found that many engineers believe an improved version could soon replace mid-level programmers. But the same report highlighted gaps: inability to manage ambiguous projects, grasp priorities, or maintain reliable verification.

Helen Toner of Georgetown’s Center for Security and Emerging Technology reviewed expert opinions on recursion. Assessments varied sharply. Some predicted rapid progress toward superintelligence. Others foresaw plateaus. Toner draws a clear distinction: “They’re just using AI for as much as they can. That is different from the classic definition of RSI, where no humans are needed.”

Ajeya Cotra of METR outlined three milestones. Adequacy means AI can continue research after humans step away, even if quality drops. Parity occurs when an all-AI team matches a human one. Supremacy follows when AI-only teams outperform mixed groups. Cotra believes adequacy may already be here or will arrive within two years. Once parity hits, she expects rapid movement to supremacy within about twelve months.

Public discussion carries risks. Overhype can distort policy. Underestimation can blindside regulators. The CSET expert group concluded that recursion demands careful scrutiny and greater transparency. Safety considerations cannot wait until the loop is closed.

So far, the evidence is mixed. Code generation reaches near-total coverage in narrow settings. Agent performance beats humans in structured competitions. But no system has removed the human from the research loop entirely. Karpathy’s public experiments show small models improving through automated iteration. Scale those methods, and the trajectory looks promising. But each leap introduces new failure modes. Verification grows harder. Reliability becomes the quiet killer.

Investors pour capital into the vision anyway. Valuations for self-improvement startups reach billions within months. Talent follows the funding. The pattern echoes the early AGI chase: definitions fluid, predictions varied, progress arriving but not always as expected.

The industry finds itself in familiar territory. Excitement runs high. Skepticism persists. Concrete demonstrations stay partial. Pichai’s continuum feels accurate. Progress accumulates. The next level of acceleration has not arrived. Whether it appears in two years or ten depends on solving problems that have so far resisted clean solutions. The loop remains open. For now.