The Selfish Moratorium

Why States Might Want a Pause on ASI Development

In the corridors of Silicon Valley and among policy makers in Washington and Beijing, a new mantra has taken hold: the race for Artificial Superintelligence (ASI) is the ultimate zero-sum game. The prevailing logic is simple. If we stop and they don’t, they win everything, and we lose our place in history.

The dominant policy narrative runs roughly as follows. The United States and China are locked in a competition that neither can exit unilaterally. If one pauses, the other races ahead. The American Enterprise Institute captured this logic precisely when it argued that a moratorium would “delay innovation, empower bureaucracies, and hand an enormous advantage to China” (Pethokoukis, 2025). The argument is clean, intuitive, and, on its surface, looks irrefutable. It is the logic that has driven arms competitions throughout modern history.

But is it correct when applied to superintelligence? In our recent paper Are we Doomed to an AI Race? Why Self-Interest Could Drive Countries Towards a Moratorium on Superintelligence (Roussel et al., 2026), we analyse the strategic structure of ASI development using the formal tools of game theory, revealing a more complicated and nuanced logic. Racing is not always the dominant strategy. Under specific, identifiable conditions, a moratorium is not a moral concession but in a states’ self-interest. Whether those conditions are currently met is a separate question. But the first step is to establish that they can be.

Join us for monthly philosophical reflections on technology and AI from the Chair Ethics and AI at KU Leuven.

1. The Prisoner’s Dilemma Critique

The critics of a moratorium make what is essentially a Prisoner’s Dilemma argument. Even if both states would be collectively better off with a mutual pause, each state individually has the incentive to defect and keep racing. Pausing while your rival races means falling behind in a competition for geopolitical supremacy. Racing therefore remains the dominant strategy, even when both parties acknowledge it could lead to catastrophic outcomes.

This argument is not wrong as a description of many strategic situations. It captures something real about how arms-and technology competitions operate. But it rests on a hidden assumption that deserves explicit scrutiny: that the cost of losing control over ASI remains lower than the expected benefit of technological supremacy. Once you make that assumption visible, you can ask whether it is actually warranted; and under what conditions it might break down.

Hence, we ask the following question: as the perceived cost of uncontrolled ASI rises, can the strategic calculus flip?

2. The Unique Logic of ASI: A Cost That Falls on Everyone

To understand why ASI introduces a structurally different strategic problem, one must be precise about a feature that distinguishes it, from the perspective of conventional arms races: the cost of losing control is a public cost.

A hypersonic missile, a cyberweapon, or even a nuclear warhead imposes its primary cost on the adversary. The risk to the developer lies in the adversary’s response. That is, nota bene, the crucial feature underlying the logic of deterrence. But an uncontrolled ASI, a system that substantially exceeds human cognitive performance and operates outside of human control, with no clear path to regain control, threatens the rival and the developer alike. If the developer loses control over its ASI, then they might have created a new, superior rival, rather than a tool for dominating theirs.

This transforms the strategic conclusions. In our model, we define the cost of uncontrolled ASI as the disutility arising from the deployment of an uncontrolled system, measured relative to the benefit of a safe win. Crucially, and unlike most game-theoretic treatments of arms races, we allow the disutility of loss of control to exceed the utility of winning. This reflects the possibility, taken seriously by a significant portion of the AI research experts (Grace et al., 2025; Müller & Bostrom, 2016), that the downside of misaligned superintelligence is not merely losing the geopolitical competition but something categorically worse.

When the cost of uncontrolled ASI is high enough, racing loses its appeal, not because states have become altruistic, but because the expected value of racing, weighted by the probability of catastrophe, turns negative. The moratorium ceases to be a sacrifice and becomes the utility-maximizing strategy.

The comparison to the race to build an atomic bomb in the second world war is instructive here. The Manhattan Project unfolded under a specific logic: Germany might be developing nuclear weapons, and the United States could not afford to arrive second. This is precisely the logic invoked today against an ASI moratorium. The analogy feels compelling because it draws on a historical case where racing was vindicated.

But the analogy breaks down at a structural level. The Manhattan Project scientists knew what they were building and understood the mechanism of the weapon with considerable precision (the atmospheric ignition hypothesis quickly proved impossible). The dangers were external: the bomb would devastate the target, not the developer. ASI is structurally different in one critical respect: the technology performing as designed is not a sufficient guarantee of safety. A system that successfully achieves superintelligence but pursues misaligned objectives out of human control poses risk to its creators. Racing ahead does not insulate you from the downside. Rather, it makes you first to encounter whether it is there.

There is a further disanalogy worth pressing. The nuclear race had a legible finish line: a weapon that detonated with a known yield, testable and deployable with reasonable confidence. The race toward ASI has no equivalent. The threshold between highly capable AI and genuine superintelligence is not precisely defined, the behaviour of a system that crosses it is not predictable from its behaviour before crossing, and the problem of control remains an open scientific question. Racing toward a goal whose properties you cannot fully specify, under time pressure that systematically incentivizes cutting safety corners, is not the Manhattan Project. It is something more like racing to deploy a device whose yield you cannot calculate, in a laboratory you cannot evacuate.

3. A Rising Perceived Cost

Decisions are not made based on reality but on the perception of it. The relevant question, then, is not only what the objective cost of uncontrolled ASI is—a matter of genuine scientific uncertainty—but in which direction the perceived cost is moving.

The data suggests the perceived cost of uncontrolled ASI increased over the past few years among experts, the public and policy makers. In early 2023, the Future of Life Institute’s open letter calling for a temporary pause attracted more than 30,000 signatories, including Turing Award laureates and laboratory founders (Future of Life Institute, 2023). Within months, the Center for AI Safety published a statement—signed by the CEOs of OpenAI, Anthropic, and DeepMind—declaring that AI extinction risk should be treated as a global priority alongside nuclear war (Center for AI Safety, 2023). A YouGov survey found 43% of respondents at least somewhat concerned about AI causing human extinction (Orth & Bialik, 2023). A Reuters/Ipsos poll found that 61% of Americans believed AI poses risks to humanity (Tong, 2023). This increased risk-awareness institutionalised in the past three years, with the 2026 International AI Safety Report (Bengio et al., 2025), written by over 100 independent researchers from over 30 countries, as its cumulation point.

Political institutions have responded in parallel. The Biden executive order on AI safety in October 2023, the Bletchley Declaration signed by twenty-eight nations in February 2025 (UK Government, 2025), the creation of AI Safety Institutes in the UK and the US (Pillay, 2024), and the subsequent network of eleven national safety institutes agreed at the Seoul summit represent a progression from expert alarm to institutional infrastructure that unfolded in under three years (Desmarais, 2024).

This trajectory matters for the rationality of an AI race. As the perceived cost of loss of control rises, driven by incidents of unpredictable AI behaviour, increasing institutional recognition of alignment risk, and public and political attention, the likelihood of entering a strategic world in which pausing becomes a rational strategy increases.

This does not mean we have crossed the threshold. The current investment levels and competitive rhetoric suggest the major powers still perceive the potential benefits of racing as outweighing the potential costs. But the direction of travel over the past few years is clear: the perceived cost of uncontrolled ASI increased.

4. Self-Interest, Not Idealism

Let us be direct about what this argument is and is not.

It is not a moral appeal. It does not rest on the claim that states ought to prioritize humanity’s long-term welfare over their immediate strategic position, or that AI companies have special ethical obligations. Those arguments have their place in the broader debate, but they have consistently failed to move the strategic needle.

The argument here operates on different ground. It holds that a state’s own rational self-interest, the baseline assumption of realist international relations, can, under specifiable conditions, point toward a moratorium rather than away from it. When the cost of losing control over superintelligent AI is perceived as sufficiently severe relative to the winner’s advantage, the capability gap, and technological uncertainty, racing is no longer the utility-maximizing strategy. The moratorium becomes stable not because states have become cooperative by disposition, but because the expected payoff of racing turns negative.

The model does not predict that a pause on the development of ASI will be imposed, or that states are currently positioned to coordinate on one. It demonstrates something more limited but important: that the standard dismissal of a moratorium as necessarily contrary to self-interest is logically incorrect. There exists a class of conditions under which it is not.

What the empirical evidence suggests is that we may be moving toward that class of conditions faster than the conventional policy debate assumes. The perceived cost of uncontrolled ASI is rising among researchers, institutions, and the public. Uncertainty about the path to superintelligence remains substantial.

None of this guarantees a stable moratorium. Even if the cost is sufficiently high compared to the benefit, practical problems of trust and verification might prevent a mutual moratorium to take into effect. But it does mean that the question is no longer whether a moratorium is conceivable as a matter of rational strategy. It is whether the conditions that make it rational are upon us, and whether states will recognize them in time to act.

Liked this post? Join us for monthly philosophical reflections on technology and AI from the Chair Ethics and AI at KU Leuven.

References

• Bengio, Y., Mindermann, S., Privitera, D., Besiroglu, T., Bommasani, R., Casper, S., Choi, Y., Fox, P., Garfinkel, B., Goldfarb, D., Heidari, H., Ho, A., Kapoor, S., Khalatbari, L., Longpre, S., Manning, S., Mavroudis, V., Mazeika, M., Michael, J., … Zeng, Y. (2025). International AI Safety Report. Https://Internationalaisafetyreport.Org. http://arxiv.org/abs/2501.17805

• Center for AI Safety. (2023). Statement on AI Risk. https://aistatement.com

• Desmarais, A. (2024, May 22). World Leaders Agree to Launch Network of AI Safety Institutes. Euronews. https://www.euronews.com/next/2024/05/22/ai-seoul-summit-world-leaders-agree-to-launch-network-of-safety-institutes

• Future of Life Institute. (2023). Pause Giant AI Experiments: An Open Letter. https://futureoflife.org/open-letter/pause-giant-ai-experiments/

• Grace, K., Sandkühler, J. F., Stewart, H., Weinstein-Raun, B., Thomas, S., Stein-Perlman, Z., Salvatier, J., Brauner, J., & Korzekwa, R. C. (2025). Thousands of AI Authors on the Future of AI. Journal of Artificial Intelligence Research, 84. https://doi.org/10.1613/jair.1.19087

• Müller, V. C., & Bostrom, N. (2016). Future Progress in Artificial Intelligence: A Survey of Expert Opinion. In V. C. Müller (Ed.), Fundamental Issues of Artificial Intelligence (pp. 555–572). Springer International Publishing. https://doi.org/10.1007/978-3-319-26485-1_33

• Orth, T., & Bialik, C. (2023, April 14). AI Doomsday Worries many Americans. So does Apocalypse from Climate Change, Nukes, War, and More. YouGov. https://yougov.com/en-us/articles/45565-ai-nuclear-weapons-world-war-humanity-poll

• Pethokoukis, J. (2025, October 27). AI Ban Backers Risk Freezing Progress. American Enterprise Institute. https://www.aei.org/economics/ai-ban-backers-risk-freezing-progress/

• Pillay, T. (2024, November 21). Why the U.S. Launched an International Network of AI Safety Institutes. TIME. https://time.com/7178133/international-network-ai-safety-institutes-convening-gina-raimondo-national-security/

• Roussel, E., Lauwaert, L., Swoboda, T., Ramsey, G., Uuk, R., Dung, L., & Aguirre, A. (2026). Are we Doomed to an AI Race? Why Self-Interest Could Drive Countries Towards a Moratorium on Superintelligence (Version 1). arXiv. https://doi.org/10.48550/ARXIV.2605.01297

• Tong, A. (2023, May 17). AI Threatens Humanity’s Future, 61% of Americans say: Reuters/Ipsos poll. Reuters. https://www.reuters.com/technology/ai-threatens-humanitys-future-61-americans-say-reutersipsos-2023-05-17/

• UK Government. (2025, February 13). The Bletchley Declaration by Countries Attending the AI Safety Summit, 1-2 November 2023. https://www.gov.uk/government/publications/ai-safety-summit-2023-the-bletchley-declaration/the-bletchley-declaration-by-countries-attending-the-ai-safety-summit-1-2-november-2023

This essay is based on the authors’ paper “Are we Doomed to an AI Race? Why Self-Interest Could Drive Countries Towards a Moratorium on Superintelligence.” The essay is written with the help of an LLM.