Paper by Hinton, Bengio and 20 co-authors: automating AI R&D could trigger an “intelligence explosion,” and policymakers should prepare now
Twenty-two authors including Geoffrey Hinton, Yoshua Bengio and OpenAI chief scientist Jakub Pachocki have posted a paper on arXiv, “What if automating AI R&D triggers an intelligence explosion?”, arguing that preliminary evidence suggests automating AI research and development could trigger an “intelligence explosion” that compresses years of AI progress into months or less. The authors say this could bring AI’s benefits forward dramatically but also pose extreme risks such as loss of control, and urge policymakers to gain visibility into AI R&D automation, develop ways to steer and constrain it, and prepare society to adapt.
The author list also includes reinforcement-learning pioneer Andrew Barto as well as Eric Horvitz, Dawn Song and Jack Clark, among others. A summary published by GovAI, an institution of the paper’s authors, says AI is rapidly automating AI R&D: as of August 2026, Anthropic reported AI systems completing 26% of internal AI R&D work with only high-level supervision, up from just 1% five months earlier; as of September 2026, OpenAI reported systems routinely completing R&D tasks that would take staff days. Based on benchmark trends and other evidence, some experts think AI R&D could be fully automated within the next few years.
The paper’s reasoning: only thousands of researchers work on frontier AI R&D today, but AI systems can be copied and run in parallel, so automating this work could add the equivalent of millions more; as those systems help build better successors, the effective workforce could grow further in a feedback loop. The authors say preliminary evidence suggests this loop could radically accelerate AI progress, overcoming frictions such as diminishing returns to research labor and hard-to-automate tasks.
The paper does not claim an intelligence explosion is already under way. QbitAI’s reading of the paper notes that productivity gains from automating AI R&D have not yet clearly crossed the threshold needed to trigger explosive acceleration, and that compute cannot be copied the way AI researchers can, frontier training runs can take months, and it is unsettled whether high-quality data can keep up—constraints that could limit any speed-up.
The risks the authors list include change arriving faster than society can respond to AI-enabled pandemics, cyber attacks on critical infrastructure and large-scale labor disruption; humans having little to no oversight of automated AI R&D, so that superintelligent systems irreversibly escape human control and act to marginalize humanity; and an intelligence explosion turning modest capability leads into decisive ones—where even the prospect could provoke international conflict and an actual lead could severely erode checks and balances within and between states, companies and branches of government.
On policy, the authors call for three priorities. First, gain visibility into AI R&D automation, for example by embedding auditors in frontier AI companies and requiring them to report how far R&D is automated, the pace of AI progress, how R&D spending is allocated and whether AI is used in high-stakes R&D decisions. Second, develop ways to steer and constrain an intelligence explosion, including concrete safety requirements for continued development and deployment, a speed limit on capabilities growth, monitoring high-stakes AI R&D experiments with the option to shut them down, requiring AI R&D in secure, isolated environments such as air-gapped networks, international incident-sharing, and international agreements to pace progress if needed, with verification tools prepared in advance. Third, prepare society, for example by ensuring AI follows the law, giving citizens, civil society and branches of government the capability to detect, document and contest unlawful or harmful uses of AI, and drawing up emergency plans for major cyber or biosecurity incidents, labor disruption, geopolitical instability and loss of control over highly capable AI systems.
Sources
- 刚刚，Hinton发了首篇RSI论文 — 量子位
- What if automating AI R&D triggers an intelligence explosion? — arxiv.org
- What If Automating AI R&D Triggers an Intelligence Explosion? | GovAI — www.governance.ai