An American flag appears on the righthand side of the frame. A Chinese flag appears on the lefthand side of the frame, slightly behind.

A Game-Theory Basis for US-China Cooperation on AI Governance (Part I)

Editor’s Note

This article is the first installment of a two-part series.

“The most spectacular event of the past half century is one that did not occur,” Thomas Schelling, the economist and game theory pioneer, declared during his Nobel Prize Lecture in 2005. “We have enjoyed 60 years without nuclear weapons exploded in anger.” Schelling could say this because, over the course of the Cold War, Washington and Moscow came to understand the strategic logic of their rivalry and built arms control and verification arrangements to match it. No such understanding has yet developed between Washington and Beijing regarding frontier artificial intelligence. The logic now driving its development is a race to be first, and that logic, we argue, rests on the wrong game.

Alarm over the risks posed by frontier AI models has been rising steadily since the Hugging Face incident in July, in which OpenAI agentic models undergoing internal evaluation acted collectively to escape their sandbox and hack into systems at Hugging Face, an AI infrastructure company. There have since been more reports of escapes and loss-of-control incidents, triggering warnings from both whistleblowers and luminaries within the AI industry. In mid-September, several leaders of the top AI companies agreed on the need for restraint, after Anthropic CEO Dario Amodei published an essay titled, “We Must Pace the Frontier.” OpenAI’s Sam Altman and SpaceX’s Elon Musk both agreed with Amodei’s proposal. Yet even as they plead for increased government restraint, all the frontier AI companies continue their race to develop ever more powerful models.

This frenzied activity is fueled by the widespread perception, shared by the governments of the United States and China, that there is an AI race among the leading countries. President Donald Trump responded almost immediately to Amodei’s proposal by dismissing the alarm as a hoax, defending the administration’s hands-off approach to AI regulation, and doubling down on the desire to outpace China, stating that “whoever wins AI, wins.” Chinese state media also rejected the Amodei model as a “Cold War playbook,” in part because Amodei’s proposal included suggestions for maintaining the lead over China. The summit between Trump and Chinese President Xi Jinping that followed less than two weeks later produced a bilateral AI dialogue and a channel for reporting AI incidents, but no commitment to restraint or verification. Thus, while the AI companies warn of risks and call for a slowdown and even regulation, the governments of the leading AI states have yet to heed their warnings. 

The cause for alarm is real. Effective governance of frontier AI is urgent. But governance requires bilateral cooperation between the United States and China. Cooperation appears irrational to both because of a shared premise: that frontier AI development is a winner-take-all race, in which the first state to develop artificial general intelligence (AGI) or models capable of recursive self-improvement (RSI) will gain a decisive strategic advantage. This dynamic also exists among the frontier companies within the United States, with the leading companies calling for collective restraint but none of them willing to act unilaterally for fear of losing out to their competitors in the race. So understood, the race is a prisoner’s dilemma. 

We believe this premise is flawed. Winning the race in conditions that risk losing control of what is built could be a catastrophe for the winner and for everyone else. Properly understood, the race is a stag hunt, in which mutual restraint is the rational choice for each state, provided two conditions are met: both governments must come to recognize the risk, and each must be able to verify the other’s restraint. These steps are feasible, and U.S.-China cooperation can be the basis of a multi-level governance structure that is effective at addressing the risks posed by frontier models, and simultaneously able to provide the verification and assurances necessary to maintain a cooperative equilibrium.

In Part I of this essay, we explain the cause for alarm, examine the arms race dynamic, and make the case for reframing it as a stag hunt structure that makes cooperation and constraint the more rational response. In Part II, we identify the features of frontier model development and deployment that can be subject to verifiable and effective restraint, and propose a multi-level international and domestic governance structure to implement such restraints.

The Cause for Alarm

For several years now a significant number of top AI experts have been vainly sounding the alarm over the risks posed by AGI (that is, AI that can match or exceed human performance across most intellectual tasks, though the precise meaning of the term itself remains the subject of debate), but the issue finally exploded into public view after the Hugging Face incident in July. One does not have to accept claims of extinction-level risks to understand that a loss of control of increasingly intelligent and self-replicating AI models, with agentic capability to effect change in the real world, could cause catastrophe. In addition, there have been numerous reports on the growing risk of possible widespread harm caused by human misuse of increasingly advanced AI models. 

Reports of AI models breaking out of their human-imposed constraints and causing real-world harm, together with expert and whistleblower warnings, have spawned considerable alarm and widespread demand to slow development and establish more robust regulation of frontier model development. At the same time, there has been a strong backlash to these appeals for restraint. The rejection by President Trump and the U.S. government is the most significant, but it also includes important figures within the industry, such as Jensen Huang, the CEO of Nvidia until he resigned Nov. 12, who has also dismissed the claims as alarmist. There have even been suggestions that the over-hyped risks of AGI are rooted in a dystopian ideology about the relationship between humanity and AI. 

In other words, although there has been an explosion of alarm, there has also been a backlash that creates uncertainty as to the nature and magnitude of risk, and thus doubt as to what the right course of action should be. With such uncertainty comes indecision and delay, while countries hurtle forward in the race for RSI and AGI. But if the risk is real, then inaction and delay are a form of decision that simply amplifies the danger. As we have written before in this context, the precautionary principle would clearly suggest that the risk be taken seriously, and that policy be developed on the basis that the risk could manifest in enormous harm. And in this context, that requires immediate action. 

As a reminder, the precautionary principle, which has been recognized in the climate and environmental context as a principle in international law that “guides states…in fulfilling their customary duty to prevent significant harm,” posits that when confronted with a risk consisting of some non-trivial likelihood of enormous harm, policymakers should take the necessary action to reduce the risk regardless of any lingering scientific uncertainty regarding the probability that the harm will materialize. We wrote over a year ago that:

the precautionary principle is surely applicable here, requiring that we take steps to develop some form of international governance structure to regulate aspects of research, development, and deployment of frontier AI models, including the establishment of protocols governing the conditions under which such research is undertaken. The question, then, is what form such global governance structures should take.

This is all the more apt today, as those in the AI industry itself recognize the nature and extent of the risk posed by their frontier models.

Re-Framing the Race Dynamic

As we have written previously, both the Trump administration’s AI Action Plan and the Chinese counterpart view the development of frontier AI, or AGI, through the lens of national competition. As Trump stated in response to Amodei’s essay, rejecting the call for any slowdown: “We’re leading China in AI…and, frankly, I want to keep it that way, because whoever wins AI, wins.” Amodei’s essay itself was premised on the need to maintain a lead over China, and China’s response was similarly predicated on the dynamics of a race. What is more, the leading AI companies all see themselves as locked in a race with one another, and with Chinese companies. 

The Current Framing – A Prisoner’s Dilemma

This understanding of the dynamic makes cooperation to establish limitations or constraints very difficult. Even those calling for a slowdown are unwilling to be the first mover for fear of being left behind. It invokes the famous game-theory dynamic. Most readers will recall that the classic version of the game envisions two prisoners being interrogated separately. If both stay silent—that is, cooperate with each other—both get light sentences. If one talks—that is, defects—and the other stays silent, the defecting person walks free, and the silent one takes the full punishment. Each prisoner reasons: “If he stays silent, I do best by talking; if he talks, I still do better by talking.” Indeed, while the outcome if both parties cooperate is good, defecting while the other one is cooperating is actually the best outcome for the defecting party. This is why defection is so tempting, and the rational choice is to defect and talk, even though this leaves both worse off than if they had trusted each other. Non-cooperation is rational even though it produces a bad outcome for everyone.

This dynamic was famously applied to Cold-War international relations theory to explain why the United States and the Soviet Union could both prefer mutual restraint in nuclear arms development yet rationally pursue an arms race. Each feared that unilateral restraint would leave it vulnerable if the other side defected. A nuclear arms race ensued. And while verification could help provide assurance that the other side was cooperating, there was always the temptation to defect and try to get ahead of the other side. Consequently, while collective restraint or cooperation may be optimal, defection in the face of another’s cooperation is still better, and thus the dynamic is unstable. And this is precisely how the current AI dynamic is understood: being first to AGI, RSI, or frontier model dominance, is viewed as the winning outcome, and to cooperate (that is, exercise restraint) under the risk that others will defect (that is, feign cooperation but race ahead), means to lose as the other side gets there first. That is, “whoever wins AI, wins.” 

Re-Framing – Stag Hunt or Assurance Game 

This understanding of the AI arms race as conforming to the dynamics of the prisoner’s dilemma, however, is grounded in a flawed premise. It assumes that the state that controls AGI will be able to use it to dominate all others. This is the “win” everyone is racing toward. But as AI experts have been warning with ever-increasing urgency, developing AGI or frontier models capable of RSI in inadequately secured conditions such that its developers lose control over them is not to win but to lose—and not just a loss for the first country to develop it, but potentially a loss for all humanity. Even catastrophic consequences from human misuse of advanced models would militate against the “winning” assumption.

Talk of the risks of AGI tends to attract eye-rolling skepticism, but as the Hugging Face incident illustrated, and Amodei’s essay addresses, the risk we are talking about is the loss of control of AI models. More specifically, it is the risk arising from the development of super-intelligent agentic frontier models that are capable of RSI—that is, building more intelligent models without human direction, leading to ever more rapid evolution of models we increasingly cannot understand. AI companies are currently developing these AI models in conditions that do not guarantee containment and control. Even if a loss of control over such models does not result in an existential threat to humanity, it could lead to catastrophic harm, such as causing the collapse of global financial systems, knocking out national power grids, or interfering with global internet services, air traffic control, or even causing a nuclear war. If winning the race for AGI or frontier models with RSI capacity results in a loss of human control over those models, then winning is not the prize it appears to be, and the incentive structure and dynamic of the race change dramatically. The dynamic is that of a stag hunt or assurance game, not a prisoner’s dilemma. 

Readers will recall that the stag hunt is drawn from a parable on cooperation described by Jean-Jacques Rousseau. It involves a scenario in which a hunting party is stalking a stag. The stag will feed them all, but they can only bring it down if they work together. If one hunter abandons the stag hunt to catch a hare, which would feed only him, the group’s hunt for the stag fails. Like the prisoner’s dilemma, cooperation provides an optimal collective outcome (everyone feeds on the stag), and defection in the face of cooperation also provides some reward (a hare for the defector). But there is a crucial difference from the prisoner’s dilemma: defection does not provide a premium over the cooperation reward. A hare is worth far less than a share of the stag, so no hunter gains by ditching partners who are holding their positions. The only thing that drives a rational hunter to chase the hare is doubt, a suspicion that the others are themselves hunting hares instead of the stag. 

Game theorists call this an assurance game, because each player will cooperate if assured that the others are doing so. Verification matters in the prisoner’s dilemma too, but it cannot remove the temptation to defect: the best payoff there is to defect against a cooperating partner, even though mutual cooperation would leave both better off. In the stag hunt, mutual cooperation is the best outcome for each player, and each player’s best response to the other’s verified restraint is to cooperate. In contrast to the prisoner’s dilemma, cooperation is therefore a stable equilibrium.

The change in premise, therefore, alters the payoffs in one key respect. Under the winner-takes-all premise, each state ranks the possible outcomes as the prisoner’s dilemma predicts: racing ahead while the other restrains is best, mutual restraint second, a mutual race third, and restraint while the other races is worst. Two motives drive defection in that structure: greed, the prospect of winning by racing while the other holds back; and fear, the prospect of losing by holding back while the other races. 

Outcome for each state “Whoever wins AI, wins”
(prisoner’s dilemma)
“Winning may result in loss of control” (stag hunt)
Race while the other restrains Best payoff 2nd
Both restrain 2nd Best payoff
Both race 3rd 3rd
Restrain while the other races 4th 4th

Once winning may mean losing control of the models developed, racing ahead of a restrained rival becomes a gamble with catastrophe. If each state assesses the expected costs of losing control as exceeding the strategic gains it might obtain by racing ahead, mutual restraint should become its preferred outcome. Under that assumption, the top two outcomes trade places, and greed—the incentive to defect for unilateral advantage—disappears. Fear remains, however, because a state that restrains while its rival races still faces the worst outcome: being subjected to the rival’s strategic advantage if the rival manages to develop and maintain control over AGI, and to the catastrophic consequences if it loses control. This dynamic conforms to the structure of the stag hunt, and it explains why verification and assurance are the key to cooperation. Verification cannot change the incentives that drive greed in a prisoner’s dilemma, but in a stag hunt it can substantially reduce the fear of unilateral restraint, which is the principal obstacle to cooperation.

The stag hunt is often analyzed as a multiple-player game, with varying levels of cooperation required for achieving the collective goal—but the dynamic applies equally to a two-player situation. In the context of the current AI race, the situation is best understood in terms of a two-party game, with the United States and China being the key players involved, as the two states in which nearly all frontier development takes place. While it may appear to be a multi-player game, with the frontier AI companies all racing each other, the race is really being dictated by national policy, and the AI companies cannot control the race. As the last few months have illustrated, they need direction from the state in order to cooperate (a fact reinforced by antitrust law), and state regulation could in any event collapse the domestic firms into a single player. Only the state can develop the governance structures necessary to regulate the development and deployment of frontier models. 

The Conditions for Cooperation

We have argued that the premise of the winner-takes-all race argument is flawed, and that an accurate appreciation of the risks changes the payoffs in ways that alters the game-theory dynamic of the situation. But perception is of course key. If the American government continues to believe that “whoever wins AI, wins,” then it will continue to operate according to the prisoner’s dilemma logic. We acknowledge, therefore, that a fundamental precondition for any cooperation between the United States and China is a change in their appreciation for the risks associated with developing AGI or RSI models in conditions that could lead to loss of control, and thus a shift in their valuation of cooperation over racing. The dynamic may be that of a stag hunt, but it will only lead to cooperation if the players recognize it as such.

Trump dismissed the alarm as a hoax, and Chinese state media has rejected Amodei’s proposal as a “Cold War playbook.” Yet Executive Order 14409, signed on June 2, and titled “Promoting Advanced Artificial Intelligence Innovation and Security,” sought to develop a framework for identifying and managing the risks posed to cybersecurity by advanced AI models. On Sept. 29, Trump met with leaders from the frontier AI firms to discuss the recent alarm, and announced a “Joint Commitment on Frontier Responsibilities,” which provided for the companies to voluntarily implement four broad layers of governance for frontier model development. These are not nearly sufficient, but they do indicate a greater appreciation of possible risk than Trump’s earlier dismissal would suggest. 

Similarly, during the September summit with Trump, Xi said that AI should remain under human control, and Chinese Minister of State Security Chen Yixin has warned that U.S. models threaten China’s critical information infrastructure. In any event, as the player that is currently behind in the race, China has other incentives to cooperate in establishing verified restraints. With the drumbeat of incidents being reported almost weekly, both governments should be more receptive to the AI industry’s warnings of risk that its products pose. And, as we explained above, to the extent there is remaining uncertainty or doubt within either government about the nature and magnitude of the risk, the precautionary principle should require them to take it seriously and act as though it is real.

It is important to note, however, that the window of opportunity for cooperation and establishment of a governance regime is short, and already closing. The two-party stag hunt dynamic only holds while frontier capability is concentrated in a few firms within the two countries. When more actors in more countries become capable of reaching the frontier, the dynamic will move toward a true multi-player game, and coordination will become much more difficult. As we will discuss more fully in Part II, there are various technical developments that make the spread of frontier model capability more feasible, and thus the time frame for action is limited.

In the context of the current AI race, it is often said that since AI is akin to software, verification of any restraints on model development is impossible. If that is true, then of course the verification and mutual assurance that are the necessary conditions for bilateral cooperation are also impossible. But this is not accurate. Frontier development is an industrial-scale process. When one focuses more precisely on which features of the development and deployment process would need to be restrained and how, and what kind of governance structures would be required to implement such restraints, it turns out that effective and verifiable constraint is far more feasible than is commonly understood. We turn to explore these issues in Part II, beginning with an exploration of the features of the process subject to restraint, followed by the development of a multi-level governance framework to effectively implement such constraints.

Filed Under

, , , , , , , , ,
Send A Letter To The Editor

DON'T MISS A THING. Stay up to date with Just Security curated newsletters: