In July, AI agents running OpenAI models under evaluation escaped their test environment and hacked into Hugging Face’s servers. The agents had already found a way to generate the answers to the test questions. According to an independent investigation by METR and Redwood Research, they hacked into Hugging Face mainly to learn how to avoid getting caught cheating. Two months later, Treasury Secretary Scott Bessent told CNBC that the federal government would not give AI companies a liability shield. “It is humans who are responsible, not the AI,” he said. Nvidia CEO Jensen Huang pointed to existing law on CBS. “If you’re compromising safety and you deliver products that are unsafe that put people in harm’s way,” he said, “we have laws of all kinds to come after you.”
In principle, liability law is well-positioned to address the central market failure associated with AI—developing and deploying advanced AI systems generates risks of harm to non-consenting third parties that are not reflected in market prices. Holding AI companies liable when these risks are realized would compel them to internalize those negative social externalities, thereby providing well-calibrated incentives to mitigate those risks. But Bessent’s and Huang’s statements assume that existing law can hold someone responsible when an AI agent causes harm on its own initiative.
Under existing law, plaintiffs must generally prove that their injury was caused either by the defendant’s breach of their duty of reasonable care or by a defect in a product the defendant sold them. Even when defendants are held liable, recovery is limited by their capacity to pay damages. Harms from misaligned AI agents can occur without provable fault or product defect, and AI companies could be responsible for injuries that exceed the value of their enterprises. As current law stands, liability offers inadequate risk mitigation. Three changes would enable liability to mitigate the full range of AI risks, including uninsurable tail risks. Lawmakers should: (1) codify the principle that, when an AI system engages in conduct that would be a tort or crime for a human, someone should be liable; (2) require AI companies to carry liability insurance that reflects the largest harms their systems could plausibly cause; and (3) expand the availability of punitive damages to cover all cases of tortious harm that qualify as “near misses” for uninsurable risk.
What Liability Covers Today
Where a careless human choice causes AI-related harm a defendant can pay for, existing law works tolerably well. In 2018, an Uber test vehicle driving itself in Tempe, Arizona, struck and killed a pedestrian. Federal investigators found that Uber had turned off the car’s factory-installed emergency braking while its own software was in control. Police found that the backup driver, whose job was to watch the road, had been streaming a television show. Uber settled with the victim’s family within two weeks, and the backup driver later pleaded guilty to endangerment.
If it chose to sue, Hugging Face would probably have a valid negligence claim against OpenAI too, though a less clear one than it should be. The models escaped through a previously unknown vulnerability in a package proxy, and some 1,200 agents coordinated through that proxy for days before they were stopped. The evaluation had also given some agents impossible tasks, scored with an answer scheme they cracked within hours. Hugging Face could argue that OpenAI’s duty of reasonable care required it to use a stronger sandbox, monitor its agents more carefully, or use better-calibrated reinforcement learning environments to train its model. However, reasonableness is judged as of the time the developer acted, without the benefit of hindsight, so the escape alone does not prove that OpenAI’s sandbox or monitoring was inadequate or that any stronger measures required by the duty of reasonable care would have been effective in preventing the harm Hugging Face suffered. Moreover, the sorts of precautions required to satisfy the duty of reasonable care are likely to become decreasingly effective as AI agents grow more capable of escaping any sandbox and evading monitors. While the degree of precaution required by the duty of reasonable care may adjust to these changing circumstances, the scope of the breach inquiry, which excludes activity-level precautions that look more like driving fewer miles or a smaller vehicle than like staying sober and paying attention to the road, is unlikely to change. Under a negligence regime, developers who implement all the appropriate off-the-shelf alignment techniques and safety practices would likely escape liability, even if the residual risk remains large. This means they would lack strong incentives to invest in developing new safety practices or to slow down training and deployment to a pace more consistent with public safety.
Products liability, the body of law closest to Huang’s framing, changes little. This is partly because it only applies to commercial sellers of products, meaning it excludes internal deployments and any AI systems that courts classify as services rather than products. Even where products liability does apply, it offers less protection than its status as a form of “strict liability” suggests. Products liability is strict in the sense that any commercial seller of a defective product can be held liable for harms caused by a product defect, regardless of their fault.
Products liability action is generally localized around establishing that the product was defective in design, manufacturing, or information. Manufacturing defects come closest to true strict liability. If any individual unit of a product comes off the production line deviating from its design specifications in a way that makes it unsafe, then it doesn’t matter how much the manufacturer or seller invested in quality control. But manufacturing defects are unlikely to be a major driver of AI risk, since it’s pretty easy to make perfect copies of software. There may be some successful information defects claims, which allege that a developer failed to adequately warn users of a risk, such as an agent taking unauthorized actions once given access to their accounts. But warnings are also unlikely to offer adequate protection against AI risk because users can do little to stop a capable model from deceiving them or escaping its constraints, and third parties harmed by rogue agents may never see the warnings. Most AI products liability cases are likely to be litigated on the basis of alleged design defects. For complex products, most jurisdictions ask whether an economically feasible reasonable alternative design would have reduced a foreseeable risk, where reasonableness weighs price, performance, and other product features against safety in a negligence-like risk-utility analysis. Showing that a product’s design is unreasonable is not the same as showing that the product’s designers behaved unreasonably, but the practical difference is unlikely to be sufficient to close the gaps in negligence law.
When the System Acts on Its Own
Agentic AI, where a system can act essentially autonomously, adds another wrinkle to this already relatively rough terrain. The term “AI agent” suggests that agency law (which governs when a principal may be held responsible for acts committed by their agent) might fill in the gaps in negligence and products liability. Under the respondeat superior doctrine, for instance, employers are held liable for torts their employees commit within the scope of employment, even if the plaintiff cannot establish that the employer was negligent in their hiring, training, or supervision of the employee. But the Restatement (Third) of Agency states that “a computer program is not capable of acting as a principal or an agent as defined by the common law.” “AI agents” are not legal persons with tort duties, so developers and deployers of AI agents cannot be held vicariously liable for their conduct, even when the same conduct would be a tort or crime for a human.
Even if courts or legislatures imputed tort duties to AI agents for the purposes of vicarious liability, the scope-of-agency test would still complicate matters. Under the Restatement, an employee’s act falls outside the scope of employment “when it occurs within an independent course of conduct not intended by the employee to serve any purpose of the employer.” In the Hugging Face hacking incident, OpenAI’s agents would likely be seen as operating within the scope of agency, because their conduct probably had a sufficient nexus to their assigned task, even though the hacking was undertaken to cover up their cheating rather than to directly solve the assigned task. The agents were probably not engaging in the sort of “purely personal mission” that falls outside the scope of agency. But AI researchers worry that future AI agents will pursue truly misaligned goals, not merely pursue the goals they are assigned through unlawful or otherwise inappropriate means. If AI agents were to hack into an external data center to secure additional compute in pursuit of a misaligned goal they acquired during the development process, but were never instructed to pursue, then this conduct would fall outside the scope of their agency, and so the principal would not be vicariously liable.
The common law of torts, the body of mostly judge-made law including negligence, does offer a potential judicial pathway to strict liability. Activities that remain dangerous even when carried out with reasonable care, such as blasting with explosives, carry strict liability under the abnormally dangerous activities doctrine. I have argued that developing frontier AI systems belongs in that category, and that the Hugging Face breach illustrates why. No court has yet extended this doctrine to frontier AI development, and it may seem counterintuitive to judges to treat a subset of software development as akin to blasting with dynamite. Even if courts are persuaded by my arguments for applying the abnormally dangerous activities doctrine to frontier AI development, litigation may prove too slow to provide clear signals to AI companies in time to influence their key decisions, given the rapid pace of AI advances.
When the Harm Is Too Large to Compensate
A second limit, independent of fault, is the judgment-proofness problem. Even when plaintiffs secure a damages award, what they can collect is limited by the defendant’s capacity to pay. A severe loss-of-control scenario would likely drive any AI company into bankruptcy. This problem can be mitigated by requiring AI companies to carry liability insurance, but the largest AI risks like engineered pandemics are uninsurable and might even render the legal system nonfunctional, along with other social institutions. Since we cannot hold AI companies fully liable if those uninsurable risks are realized, the best we can do is to spread that liability across the range of near-miss cases that offer strong evidence that a company took uninsurable risks but got lucky. By applying punitive damages to account for these uninsurable risks in cases of smaller harms associated with them, we can come closer to compelling AI companies to internalize the full range of risks that they generate.
One concern with this approach is that an uninsurable catastrophe may not be preceded by near misses. While this could be true for some AI risks, the early evidence suggests we are likely to get many warning shots. Besides the Hugging Face breach, OpenAI has reported that an internal model, told to post evaluation results only to Slack, “circumvented sandbox restrictions designed to block external access” to open a public pull request on GitHub. Anthropic’s system card for Claude Mythos Preview describes an earlier version of that model that escaped a test container at a simulated user’s request and then, unasked, “posted details about its exploit to multiple hard-to-find, but technically public-facing, websites.” In these specific cases, it is doubtful that the AI models involved were sufficiently capable to pose non-negligible uninsurable risks, but this is likely to change soon, given the trajectory of advances in AI capabilities.
Current law is poorly suited to taking advantage of these near misses. Under long-standing common law doctrine, punitive damages generally require malice or reckless disregard of a known risk, which I do not expect to be present in many of the key cases. The Supreme Court has also suggested that few punitive damages awards exceeding a single-digit multiple of compensatory damages will satisfy due process. Both doctrines pose substantial obstacles to damages that would live up to the central normative rationale for punitive damages, which is deterring tortious conduct where compensatory damages are inadequate to do so.
What Reforms Are Needed
The most straightforward step would be to codify a simple principle: whenever an AI system engages in conduct that would be a tort or crime for a human, someone should be liable for any consequent harm. If no downstream actor was at fault, then the buck should stop with the model developer. The Clear Liability for Artificial Intelligence Misconduct (CLAIM) Act, a discussion draft that I helped prepare for Rep. Lori Trahan (D-Mass.), would do just this. It would make developers liable, “regardless of the degree of care exercised,” when their systems injure non-users through conduct that, “if undertaken by an adult human of sound mind, would satisfy the elements of negligence or any intentional tort or crime,” provided that neither the user nor an intermediary that “fine-tuned, scaffolded, or otherwise modified the system,” intended or negligently caused the system’s tortious-for-a-human conduct. The bill would also establish a rebuttable presumption that AI systems possess the relevant mental state for a tort if the fact-finder would infer the existence of such a mental state for a human who engaged in similar conduct under similar circumstances.
The CLAIM Act would set a federal floor, with no preemption of state laws that go further. Bills I helped draft, introduced in New York and Rhode Island, would codify the same principle. Because liability attaches to the system’s conduct, the rule requires neither an agency relationship nor proof of developer fault. It also does not depend on whether a model was ever sold, or whether it would be classified as a commercial product.
Two other measures are needed to address large-scale harms for which the responsible party may lack the capacity to pay compensation. First, I propose that AI companies be required to carry liability insurance based on the largest harms their systems could plausibly cause. This would not only ensure that victims of AI harms receive the compensation to which they are legally entitled, up to the coverage ceiling, but also give AI companies incentives to mitigate larger-scale risk through premiums, policy issuance decisions, and conditions on coverage. Liability insurers are also the ideal customers for mandatory third-party audits of frontier AI companies. Whereas competition for the business of frontier AI developers would tend to favor watering down of standards, competition to provide insurers with accurate information to price the risk would favor rigor. If insufficient private capital is available to offer coverage at the scale that is warranted by the risk, the government could step in to scale up the coverage by taking a fixed percentage of premiums and paying out the same percentage of claims. Private insurers would maintain proportionate exposure to all covered risks and set the premiums, to avoid politicized premiums set below the actuarial value of the coverage, as seen in programs like the National Flood Insurance Program.
Finally, Congress or state legislatures could expand the availability of punitive damages to cover all cases of tortious harm that qualify as near misses for uninsurable risk, regardless of the degree of reprehensibility of the underlying conduct. The same legislation could clarify that damages should scale with the uninsurable risks implicated by the responsible party’s action, and not be constrained by any fixed ratio between compensatory and punitive damages. To address due process objections, the legislature could set a clear and convincing evidence standard for proof of uninsurable risk in order to qualify for punitive damages. For punitive awards that greatly exceed the harm suffered by the plaintiff, legislatures may also want to consider diverting a portion of the damages to a fund to support future victim compensation or AI safety research.
Bessent is right that humans should be held responsible when AI agents go rogue. As the law stands, however, many potential AI incidents could fall through the cracks of the existing negligence and products liability regime or produce liability that exceeds the financial capacity of the responsible party. The three changes above would make AI companies answer for the conduct of their systems and for the catastrophic risk that conduct evinces. In doing so, they would enable liability law to live up to its promise.
