The word “recursion” is the latest buzzword in the AI world. Two separate startups have taken over the name, and many more are starting to mention recursive self-improvement (RSI) in their roadmaps. Like AGI before it, RSI has become a three-letter synonym for AI’s catastrophic takeoff. Even though there is still some disagreement about what exactly that means.
Essentially, RSI refers to an AI system that can continually upgrade itself. As AI systems become better able to manage upgrade cycles than humans, the process becomes closed-loop, limited only by the computational power they have access to, and humans are not only unnecessary, but useless.
Scary or not, this is a vision that many AI labs are passionately pursuing.
Earlier this month, renowned AI researcher Richard Socher launched the aptly named Recursive SuperIntelligence, with RSI as its explicit goal. “Our main focus is building a truly recursive, self-improving superintelligence at scale,” Sorcher told TechCrunch at the launch. “This means that the entire process of ideating, implementing, and validating research ideas will be automated.”
Many other prominent researchers are already pursuing the same goal, hoping for breakthroughs that will enable recursive self-improvement.
One of the most prominent is Tesla and OpenAI legend Andrej Karpathy. He uses a fleet of agents to train LLMs on simple tasks in a project he calls Auto-Research. Karpathy has been unusually open about the project, regularly tweeting about milestones and making the building blocks available through a public GitHub repository. So far, the research has been largely limited to making small improvements to the GPT-2 scale model, but as Karpathy said in March, “This is not (yet) a novel or groundbreaking ‘research,'” but it was enough to convince many other researchers to follow RSI’s dream. And with Karpathy currently working on pre-training at Anthropic, there will be plenty of opportunities to apply this idea on a larger scale.
Adaption, founded by Cohere and Google alumnus Sara Hooker, recently launched a similar tool called AutoScientist in an effort to automate frontier training. Similar to Karpathy’s Automated Researcher, this system trains agents to make incremental improvements. However, the goal of Adaption is to make it easier to train full-fledged frontier models. If the same researchers start pushing that frontier forward, this system could soon become much like RSI.
Disarray founder Doris Xin caught the attention of RSI more specifically when her self-trained machine learning agent won 28 medals in a recent Kaggle competition, beating out many human-trained agents. The big challenge, she says, is reliability.
“Given infinite computing and infinite time horizons, I would argue that we are already there,” Xin told me. “I would argue that this is not a creative endeavor. This is just meat-and-potatoes engineering.”
not there yet
There is also a lot of evidence that the AI industry is not that close to recursive systems in any meaningful sense. And it remains committed to telling a wary public about its progress. So Google CEO Sundar Pichai basically admitted it in a recent podcast interview.
“This is an ongoing thing and we’re all definitely making progress,” Pichai said. “But the way people describe RSI, it means the next level of acceleration, and it’s going to have a lot of impact. But we’re not there yet.”
But this continuum includes so many self-improving AI systems. In January, one of Anthropic’s lead programmers, Claude Code, estimated that “nearly 100%” of the team’s code was written by the tool. This is a frank admission that Claude Code was literally writing itself.
Just because engineers are using AI tools doesn’t mean the tools can replace them. But Anthropic also appears to be getting closer to replacing engineers. In a recent survey related to the Mythos preview, 5 out of 18 Anthropic engineers believe that with the harness improvements, this version of Mythos could soon replace L4 engineers, mid-level programmers who can work on complex projects without supervision.
Still, it had some of the same weaknesses you might expect.
“Claude’s main weaknesses reported compared to L4 include self-management of ambiguous tasks over the course of a week, understanding organizational priorities, preferences, validation, following instructions, and epistemology,” the report said.
In other words, its weakness is all about self-direction, which is the basis of RSI. But certainly, when it comes to other things, Claude is ready to intervene at a moment’s notice.
Similar to the previous term AGI, the AI industry also doesn’t know how far it is from showcasing meaningful recursive systems. When Georgetown’s Center for Security and Emerging Technologies assembled a group of experts last year to study RSI, the group found that there were wide differences in assessment. Some predicted an imminent “superintelligence”-type explosion, while others predicted slow progress and an eventual plateau. However, everyone agreed that recursion makes predicting the future particularly difficult.
Helen Toner, director of CSET and former board member of OpenAI, told TechCrunch that simply using an AI tool to conduct AI research is not enough to qualify as an RSI. “They’re just leveraging AI as much as possible,” Toner told TechCrunch. “And I think that’s different from the classic definition of RSI, which is that you don’t actually need humans.”
Toner points to a recent post by METR’s Aiya Kotla that distinguishes the various milestones leading up to the takeover of AI research. One step, which Kotla calls “adequacy,” would come when the system could perform research even after all humans were removed, even if the resulting research was less valuable or efficient. “Equivalence” occurs when an AI-only system performs just as well in research as a human-only system. The final stage, “superiority,” occurs when an AI-only system outperforms a collaborative human-AI system.
Ultimately, Cotra concludes, AI is very close to the sufficiency threshold of being able to produce some results on its own, similar to the incremental changes made by Karpathy’s automated research system. “I would not be shocked at all to be told that this milestone has already been passed. I expect it to be within the next few years,” Kotla wrote.
She says it’s less clear when equality will be achieved, but she thinks it would “significantly accelerate the pace of AI progress and bring AI research to an advantage within the next year.”
bumps in the road
Since many AIs are built on scaling laws, there is a strong tendency to think that RSI follows the same curve. Toner believes that many of the people who are advancing AI research and development through RSI “think of RSI as a very smooth ladder that they can continue to scale up.”
But even if AI researchers were able to make incremental improvements like Karpathy’s automated researchers, there would be even greater challenges in taking over the entire research process. Toner explains this in terms of the history of computing. This means that more and more processes have been delegated, with humans directing things from above.
“We’ve gone from machine language to assembly language to compiled languages. We’re getting further and further away from the essence of computers,” says Toner. “But in some intuitive sense, humans are still running the show.”
Moving beyond that paradigm will require significant challenges in both engineering and coordination. But even with large investments, computing is not available indefinitely, and the fundamental trade-off between human labor and machine intelligence will be difficult to overcome.
What about an apocalyptic vision of a fully recursive AI system? The only thing researchers basically agree on is that, like AGI, it’s not yet a reality.
If you buy through links in our articles, we may earn a small commission. This does not affect editorial independence.
