Earlier this month, a private company built a piece of software reportedly capable of finding and exploiting security flaws in every major operating system and web browser. The company decided, on its own, not to release the software widely. No regulator had the authority to demand it. No government could have stopped the company if it had chosen otherwise. And the fact that we are relieved—rather than protected—tells us everything about the state of AI governance today.
The company was Anthropic. The software was Claude Mythos Preview—an AI model that, according to Anthropic, can identify and exploit so-called “zero-day” vulnerabilities, hidden flaws that software developers have not yet discovered or patched. Anthropic decided not to release the model broadly. Instead, it granted access to a handful of critical-infrastructure providers and committed $100 million to help them fortify the systems that Mythos could compromise in the hands of bad actors. This initiative, called Project Glasswing, was designed to place critical-infrastructure partners ahead of bad actors in the perpetual cat-and-mouse contest between offense and defense in cybersecurity.
We should be relieved Anthropic made this choice. But relief is not the same as security, and the fact that a government had to rely on a company’s goodwill, rather than the force of law, should alarm us. Reasons for alarm deepened last week when reports emerged that a group of unauthorized users had gained access to Mythos and other Anthropic models.
The lesson is clear: when the consequences of a single corporate decision can compromise the world’s digital infrastructure, industry self-governance is not enough. Democratic governments must step in and regulate robustly.
The European Union is trying. Its AI Act imposes binding obligations on providers of general-purpose AI models, which includes Anthropic’s Mythos. These obligations include requirements that companies document their models’ capabilities and take steps to mitigate systemic risks, defined as risks that have a “significant impact on the [European] Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole.” But the current framework has meaningful limitations.
First, the enforcement machinery for these provisions does not activate until August 2026, which, particularly in this context, provides plenty of runway for malicious activity. Second, and most critically, the law is vague about what type and amount of mitigation is actually required. Neither the Act itself—nor the (non-binding) compliance guidance that accompanies it—spells out substantive mitigation standards. That is, the instruments require Anthropic and other companies with general purpose AI models to reduce cybersecurity risks, but does not say by how much or by what means, or how such risk reduction will be measured. Companies, for now, retain discretion to decide what level of risk is adequate. Anthropic’s choices in this case appear reasonable. The next company’s choices might not. Either way, it essentially remains up to the company to decide what level of risk is acceptable and how to ensure it is not exceeded.
That is the central problem. Democratic societies should not have to depend on the moral restraint of corporate leaders when the stakes include the stability of our financial systems, power grids, hospitals, and elections. What if the next model of this type comes from a lab whose leaders look at the same capabilities and conclude that broad, rapid deployment is the right move? Or from a company that releases its weights—the trained parameters that encode what the model has learned during training—openly, as the Chinese lab DeepSeek did in January 2025? Once weights are released, the model exists permanently, beyond anyone’s power to withdraw. These are not edge cases. In today’s AI industry, they are the default.
The Mythos case should be a wakeup call to policymakers about the inadequacy of existing democratic control over frontier AI. The Trump administration has been so focused on “winning” the AI race against China—an understandable goal given China’s authoritarian character—that it has not seriously entertained common-sense regulation of the most powerful AI systems.
And yet the irony is acute: The result is that the U.S. government now finds itself dependent on, and exposed to, a company with which it is openly at odds. A government that will not regulate the most powerful AI labs does not stop needing them. It remains at their mercy.
The administration’s national policy framework for artificial intelligence, released last month, aims to dissuade Congress from overseeing frontier AI development while preempting state regulation in this area, leaving a dangerous governance gap.
Congress, for its part, has not filled it. Bills targeting frontier AI capabilities, such as the AI Foundation Model Transparency Act of 2026, have been introduced and repeatedly stalled in committee due to familiar partisanship, fights around state preemption, and the concern around China winning the AI race. The only U.S. jurisdictions to impose enforceable obligations on frontier developers are California, whose SB 53 took effect in January 2026 after a more ambitious bill was vetoed the previous year, and New York, whose RAISE Act was signed in March 2026 and takes effect in January 2027. Both state laws require disclosure of frontier model capabilities but do not empower regulators to halt deployment or require substantive mitigations.
Effective Democratic Control in Practice
What would effective democratic control look like? The most promising lever is the most physical: compute. Frontier models depend on vast quantities of specialized chips produced by a handful of firms—Nvidia, TSMC, ASML, Samsung—and delivered through a small number of cloud providers. That supply chain is a chokepoint governments can actually grip.
The United States and its allies already use export controls on advanced chips to hold back China’s progress on frontier AI. As Mustafa Suleyman, CEO of Microsoft AI, has argued in The Coming Wave, these measures are not long-term solutions to AI regulation (technology eventually diffuses beyond any chokepoint), but they can nonetheless buy the United States time to build and implement the effective regulations it currently lacks.
The same logic can be applied at home. Governments can choose to license training runs above specified compute thresholds, condition access to frontier compute on meaningful pre-deployment risk assessment, and empower regulators to pause deployments when capabilities cross pre-defined lines. At the federal level in the United States, pre-defined lines could be enforced by the Federal Trade Commission, or even potentially by a new agency or subagency, as some have proposed. There is already some development at the state level, if insufficient. For instance California’s line, set forth in SB 53, sets a reasonable threshold based on foreseeable substantive harm—50 deaths or more than one billion dollars in damages—but only requires companies to report on their mitigations, not to halt deployment.
This type of pre-deployment pause authority, akin to a stress test for financial institutions, still fosters innovation, because it does not allow regulators to interfere in the training, testing, and refining of models. The regulator only enters when the developer wants to make the model broadly available. To be effective, this must include the ability of regulators to test a model before deployment and determine whether the model poses unacceptable risks and should therefore be withheld from the market.
These are not new or radical proposals. President Joe Biden’s 2023 executive order on AI required companies to report on the largest frontier training runs and share safety measures; California’s SB 1047 would have gone further, imposing safety evaluations, requiring developers to build a “kill switch” into their models, and authorizing the Attorney General to sue for damages when models caused or materially enabled catastrophic harm. Critics argued that the bill used the wrong proxy for risk (compute thresholds) and that post-deployment shutdown requirements (the “kill switch”) would effectively chill open-source development. The Biden order was rescinded by the Trump administration; SB 1047 was vetoed by Governor Gavin Newsom. But instead of settling for the watered-down version in SB 53—or no version at all at the federal level—the government should embrace a pre-deployment pause authority with a substantive harm threshold, which would effectively maintain the harm mitigation purpose of the authority without triggering the “kill switch” concerns that led to the vetoing of SB 1047.
AI development can feel inexorable, but it is not. Democratic societies, acting together, should have the power to say “pause.” The question is whether they will act before the levers are out of reach.
