The Self-Improving AI Mirage: Why Full Automation Remains Out of Reach

The idea of an AI that upgrades itself without human help has become the tech industry’s new obsession. Two startups now brand themselves around it. Dozens more promise recursive self-improvement in their roadmaps. The acronym RSI—recursive self-improvement—carries the same weight AGI once did: a vision of explosive progress that could leave humans behind.

But defining what RSI actually requires is proving difficult. Timelines vary. Definitions blur. And the gap between incremental automation and a closed loop where machines handle every step of their own advancement remains wide.

At its simplest, RSI describes an AI system that improves its own architecture, training methods, or code with minimal human input. If that loop closes, progress could accelerate dramatically. Compute becomes the only bottleneck. Human judgment becomes optional. That’s the theory.

Richard Socher launched Recursive Superintelligence earlier this month to chase exactly that. “Our main focus is to build truly recursive, self-improving superintelligence at scale,” he told TechCrunch. The former Salesforce chief scientist raised hundreds of millions at a multibillion-dollar valuation. Backers include GV and Greycroft.

Others are taking parallel paths. Andrej Karpathy, now at Anthropic, experiments with agent swarms that refine smaller language models. His Auto-Research project lives openly on GitHub. Milestones appear regularly on X. Gains remain modest. Karpathy himself noted in March that the work had not yet produced novel breakthroughs. Still, the approach inspires copycats.

Sara Hooker’s Adaption released AutoScientist, a system designed to automate parts of frontier model training. Agents propose incremental changes. The goal is one step removed from full autonomy: make it easier for humans to push models to new scales. Push those improvements far enough, however, and the line between assistance and replacement starts to blur.

Doris Xin’s work at Disarray produced perhaps the clearest early signal. A machine-learning agent she helped train secured 28 medals in a Kaggle competition, outperforming many human teams. Xin sees reliability as the central obstacle. “I would argue, given infinite compute and infinite time horizon, we are already there,” she said. “This is not a creative endeavor. It’s just a lot of meat-and-potatoes engineering.”

Industry leaders strike more cautious notes. Google CEO Sundar Pichai described current efforts as part of a continuum. “It’s a continuum, and we are all definitely making progress,” he said in a recent podcast. “But in the way people describe RSI, that would represent a next level of acceleration and would have a lot of implications, but we aren’t quite there yet.”

Evidence of partial automation abounds. One Anthropic engineer estimated that nearly all code for a recent Claude project came from the model itself. A survey tied to the company’s Mythos preview found several engineers believed an improved version could soon replace mid-level human programmers on complex tasks. Yet the same report listed persistent shortcomings: inability to manage ambiguous week-long projects, grasp organizational priorities, exercise taste, or maintain reliable verification.

Those gaps matter. Self-direction sits at the heart of meaningful RSI. Without it, tools remain powerful but subordinate.

Helen Toner directs Georgetown’s Center for Security and Emerging Technology. She reviewed expert opinions assembled last year on the topic. Assessments diverged sharply. Some foresaw rapid movement toward superintelligence. Others predicted plateaus. All noted that recursion makes forecasting especially uncertain. Toner draws a clear line. “They’re just using AI for as much as they can,” she told TechCrunch. “And I think that is different from the classic definition of RSI, which is really that there are no humans needed.”

Ajeya Cotra, affiliated with METR, outlined three milestones in a recent analysis. Adequacy arrives when AI systems can continue research after humans step away, even if output quality drops. Parity occurs when an all-AI research effort matches a human one. Supremacy follows when AI-only teams outperform mixed groups. Cotra believes adequacy may already sit behind us or will arrive within the next couple of years. Once parity hits, she expects rapid movement to supremacy within roughly twelve months. The pace of AI progress would then accelerate sharply.

Many in the field assume scaling laws will carry the day. Add more compute. Watch capabilities climb. Toner cautions against that view. History of computing shows humans gradually handing off lower-level tasks while retaining strategic control. Assembly languages replaced machine code. Compilers replaced assembly. The human remained in charge. True RSI would sever that final link. Engineering hurdles multiply. Alignment questions intensify. Finite resources impose hard limits.

Recent projects illustrate both momentum and friction. Ricursive Intelligence, founded by former Google DeepMind researchers behind AlphaChip, aims to close the loop between AI and hardware. The company plans to use AI to design better chips that in turn train stronger AI. Cofounder Azalia Mirhoseini expects design cycles to shrink from years to days. Human supervision remains part of early phases. A Bloomberg report from mid-May highlighted Recursive AI’s $4.65 billion valuation after a $650 million raise. The firm focuses on safe self-experimentation. Socher, involved in related efforts, reiterated the drive to automate knowledge discovery.

An IEEE Spectrum article published earlier this month surveyed similar activity across labs. Systems now write code, optimize chips, and refine experiments. Yet humans still direct the highest-level decisions. The intelligence explosion imagined by I.J. Good in 1965 stays theoretical.

Public discussion carries risks. Overhype can distort policy. Underestimation can blindside regulators. The CSET expert group concluded that recursion demands careful scrutiny, greater transparency, and broader input on potential outcomes. Safety considerations cannot wait until the loop snaps shut.

So far the data points conflict. Code generation reaches near-total coverage in narrow settings. Agent performance beats humans in structured competitions. Frontier training tools automate incremental gains. None have removed the human from the research loop entirely. The difference feels subtle until one considers the consequences.

Karpathy’s public experiments offer a window. Small models improve through automated iteration. Scale those methods. Apply them to larger systems. The trajectory looks promising. But each leap introduces new failure modes. Verification grows harder. Unintended behaviors compound. Reliability, as Xin noted, becomes the quiet killer.

Investors pour capital into the vision anyway. Valuations for self-improvement-focused startups reach billions within months of founding. Talent follows the funding. Roadmaps fill with recursive language. The pattern echoes the early AGI chase. Definitions stayed fluid. Predictions varied wildly. Progress arrived, but not always in the expected form.

Observers inside the field watch for concrete signals. When an AI system can identify a promising research direction, implement the experiment, evaluate results, and incorporate findings into its next version without human approval at any stage, the threshold will have been crossed. Current systems handle pieces. The full chain remains broken.

Cotra’s adequacy milestone may indeed sit close. Remove the humans and some research output continues. That marks real automation. It does not yet mark domination. Parity and supremacy would change the equation. Acceleration becomes self-reinforcing. Control questions turn urgent.

The industry finds itself in familiar territory. Excitement runs high. Skepticism persists. Concrete demonstrations stay partial. And the public wonders how worried to become. Pichai’s continuum feels accurate. Progress accumulates. The next level of acceleration has not arrived. Whether it appears in two years or ten depends on solving problems that have so far resisted clean solutions.

Engineers will keep pushing. Startups will keep launching. Experts will keep debating definitions. RSI has replaced AGI as the term that captures both ambition and uncertainty. The loop remains open. For now.