Public concern about how to tackle the serious safety risks posed by frontier AI has reached new heights in the last few weeks. In the absence of government regulation, independent evaluations have been proposed as a way forward. On September 12, Anthropic’s CEO Dario Amodei called for, among other things, “embedded third-party evaluators…to verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes.” OpenAI’s CEO Sam Altman agreed, saying “independent evaluators with employee-like access is a great idea, and we will do the same.” And SpaceX CEO Elon Musk gave a general concurrence.
On September 29, leaders from six companies (Google, Anthropic, OpenAI, Meta, xAI, and Nvidia) signed the White House Accord on Super Intelligence, a set of non-legally binding commitments, which includes a provision that the companies would “[p]artner with an independent external auditor or evaluator to carry out independent assessments of whether the controls, monitoring, and detection are operating as intended.”
Anthropic shared an announcement on October 15 with respect to its commitment to undergo independent evaluation. The company stated it had already started to afford “employee-like” access to METR. Anthropic noted that it was “also working with SecureBio and Accenture as evaluators,” and it was in talks with others, including Hugging Face. The SpaceX CEO replied to Anthropic’s announcement stating his company “will probably do the same.”
Anthropic’s October 15 announcement lists expertise, independence, sufficient access to information, and freedom to publish results as the key components for the external review process. As a first mover in this space, its actions may set the norm. It is worth taking a moment to contemplate whether Anthropic’s path will bring teeth and legitimacy to third-party evaluation of frontier AI or, instead, compromise the goal of meaningful oversight and risk assessment.
While it is commendable that Anthropic has publicly shared its view of how to operationalize third-party evaluation and has quickly moved forward, a number of questions emerge with respect to this approach to independent oversight. Given the Anthropic CEO’s stated commitment to using the company’s AI leadership position to create a “race to the top” (as well as the company’s statement that it is “learning actively about what kinds of partnership work”), the reflections below are intended to help improve this approach to external evaluation before it crystallizes for the company, becomes the industry norm, or is codified in any potential regulation.
Substantive Safety Standards? What Standards? Whose Standards?
Anthropic’s October 15 announcement skips any discussion of the substantive safety standards that will be used in the audits. Will the safety benchmarks be whatever each company agrees to with a particular third-party auditor? Shouldn’t outside experts (in government, the academic research community, and civil society) be involved in designing them? Shouldn’t the public at least have access to those standards before the audits take place?
This glaring omission is jarring, especially in light of Politico’s release of Anthropic’s lobbying effort regarding federal AI regulation. The outlet reported that Anthropic’s proposal would allow companies to develop their own safety frameworks and permit “the government to block a model’s release or use only if it poses a foreseeable, imminent risk of more than $1 billion in damages, or of killing or seriously injuring more than 50 people. The risk must stem from the creation or release of a biological weapon, the model evading its developer’s controls or a cyberattack.” To say the least, such safety thresholds are profoundly insufficient to meet the moment, if not altogether shocking to most people.
Will those be the types of safety thresholds the independent auditors are expected to use in their risk assessments? Will the evaluators themselves have any say in what the controls, monitoring, and detection systems should be to avert certain safety risks, or, as indicated in the Accord on Super Intelligence, will they only look at “whether the controls, monitoring, and detection are operating as intended” by the company? Determining solely whether the company is doing what it intended to do could set too low a threshold or even incentivize companies to lower their own standards in order to perform better in the audits.
In addition, will the evaluators be considering solely safety risks within the U.S. context or, as these are global products, the risks arising abroad as well? If so, do the evaluators have the expertise to consider impacts beyond the United States?
Surely there would be broad consensus that the basic safety standards need to be substantively appropriate and subject to public scrutiny. Without adequate government regulation (or even industry consensus on those substantive standards), we are left guessing what each bilateral contract between a frontier AI company and an evaluator will say. The lack of regulation means a variety of unknown substantive standards will be applied without public input. It will be challenging for the public to accept the legitimacy of the evaluations with such foundational aspects of the audits enveloped in obscurity at their launch.
Issues of Independence
Former SEC Chair Arthur Levitt once stated “The accounting profession must be like Caesar’s wife. To be suspected is almost as bad as to be convicted.” This observation referred to Caesar divorcing one of his wives (Pompeia) on the mere suspicion of misbehavior because, given his position, the appearance of impropriety was untenable. Given the high stakes for these frontier AI evaluations, similarly high standards will no doubt be used in considering independence issues, which is fundamental to the legitimacy of third-party auditors.
Anthropic has highlighted the thorny issues that are triggered for independence when it comes to funding these audits. The company has previously proposed that funding should be pooled or government-subsidized, but noted that (given neither option exists) it will engage with some evaluators who have their own funding and pay others directly.
Anthropic’s October 15 update briefly explains the principle of independence as evaluators “shouldn’t hold equity in the company they’re evaluating, or be financially or operationally dependent on it.” It is true that the actual embedded team members of the evaluator, those working on the audit within the evaluator’s offices (but not embedded), and the entire chain of command up to the head of the evaluation entity should not hold equity in the company under review, but what about members of those employees’ households? And are there other conflicts of interest that should be considered? What if the third-party evaluators are recent alumni of the company under evaluation? And what if there are other substantial links between the highest levels at the company doing the evaluating and the company being evaluated? Is transparency a sufficient mitigation in such instances or are more guardrails needed? While it is notable that the evaluators will come from legal entities distinct from Anthropic and for which Anthropic does not select the board of directors, executives, or staff, a variety of conflict-of-interest issues persist that merit addressing.
Indeed, in some sense, any entity engaging in such audits could risk being “financially or operationally dependent” on the company under review if it earns significant revenue from the process or its donated funding is affected by its performance in such audits. There can be problematic incentives to produce sufficiently pleasing audits to continue funding flows. Moreover, if the company under review has other business dealings with the entity conducting the evaluation, concerns can arise about the creation of questionable incentives influencing the audit in order to continue those other dealings. Even the appearance of such incentives could put auditors in the same boat as Caesar’s wife, i.e., at the very least creating the appearance of impropriety, which eventually undermines the public’s view of the legitimacy of the endeavor.
Some have argued that “randomized selection of qualified auditors combined with finite periods for observation can go a long way toward addressing [such] concerns.” Would such measures be helpful and sufficient in this context? Are rotations feasible given the level of needed expertise, the number of qualified evaluation firms, and the rapidity with which the technology evolves and issues arise in this space? Further thought should be given to how and whether a self-regulatory system of frontier AI audits can ensure meaningful independence.
Access to Information
Anthropic’s October 15 announcement states evaluators should have “[e]nough access to meaningfully evaluate risks. Beyond access to the latest models, evaluators also need visibility into how those models were trained, into how our safeguards function, and more.” Later it states: “Outside a small number of clearly defined categories (e.g., data about customers, attorney-client privileged information, or information about our model architectures), Anthropic employees are encouraged to share information with them just as they would with a fellow employee working on alignment risk.” Then a footnote explains “[i]nformation about architectures presents a high risk of inadvertent leaks, and its relevance to risk assessment is typically low. When it is highly relevant, we will generally seek to share it, though this will require explicit internal approvals.”
Meaningful transparency is also needed about the contractual arrangements between the company under evaluation and the entity evaluating it for the public to fully understand, among other things, the extent of the evaluators’ access to information instead of being given an illustrative listing of exempted information. For example, the phrase “model architecture” can be interpreted in a variety of ways. Perhaps it is defined with more clarity in the contractual relationship between the company and the evaluators? Will evaluator entities auditing the same company have different levels of access to information for different audits? And will the same evaluator entity be given different access to information across the frontier AI industry or will evaluator entities assert minimum baselines of access to information in order to participate in the process? It will be difficult for the public to accept the legitimacy of such audits without greater clarity on the material aspects of access to information in the potentially myriad bilateral legal arrangements that are coming down the pike.
Transparency of Outcomes
Anthropic’s October 15 announcement states the evaluators will have the “freedom to publish observations and assessments of a model or a process.” No doubt the importance of public findings and reason-giving in these reports is essential to the legitimacy and utility of any oversight of these companies. The announcement explains that, in the context of its particular contractual arrangement with METR, that means
“METR will publish its analysis of our risk levels, incidents, or practices. Anthropic can’t veto publication, but we can require them to redact information if it falls into predefined categories like the ones above [e.g., data about customers, attorney-client privileged information, or information about our model architectures]. (Anthropic ultimately decides what information does and does not fall into these categories, although METR reserves the right to make certain statements about the level of access and information it received.)”
Without access to the governing contractual documents and given the brevity of the post, we have only an illustrative list of topics that can justify Anthropic’s redactions and thus cannot assess the breadth of the redaction power. Nor is it clear if METR can publicly share general reflections about whether the redactions were justified. Reporting seems to have noted from a prior version of Anthropic’s October 15 announcement that METR “retains the right to publicly characterize any redactions and state whether it believes they were justified.” Per footnote 2 of the October 15 announcement, the language in question appears to have been updated to “the right to make certain statements about the level of access and information it received,” which covers a different topic than redactions. That said, language later in the post implies METR has “similar” (but not the same) publication rights as other evaluators who are afforded the right to comment on redactions. Overall, the announcement is unclear about whether METR has such rights, though all evaluators should be provided these rights.
Beyond Anthropic’s right to redact for unknown reasons, are there specific guardrails in place to ensure the company’s pre-publication review of the audit report (to exercise its redaction power) doesn’t otherwise result in improper influence over the content of the report? News reporting indicates the discussions between companies and evaluators on the topic of what gets released to the public has been quite contentious; it is likely to remain so.
More Inconvenient Questions
A key set of unanswered questions stems from the unknown process for what happens after an evaluation is completed. If the evaluators give failing grades to the companies, what happens next? Will the companies be bound under the contractual arrangement with the evaluators to change course? Or will the companies merely be subject to judgment in the court of public opinion? Without regulation in place, will the government launch any kind of investigation based on the evaluators’ reports?
What if the evaluator issues recommendations for future safety improvements? Must the frontier AI companies at least answer the recommendations publicly or will they just be buried in the flurry of daily frontier AI news? Will the evaluators be open to receiving inputs and concerns from outside stakeholders (such as researchers, NGOs, etc.) or will the evaluators’ in-house expertise be the limit of what is applied in the evaluation?
The questions go on and on. In the absence of a regulatory framework (or even rigorous and transparent scrutiny by academic researchers and civil society underpinning the oversight of frontier AI safety), many issues about the independence of evaluators, the substantive standards, and other key topics remain murky, complicated, and problematic (to put it mildly) at this critical juncture. Without discounting the effort that it has no doubt taken within Anthropic to get to the current stage in the development of independent evaluation, the need for substantial improvement is evident and time is short.
These audits should not be about doling out participation trophies to some of the wealthiest and most powerful companies on the planet. They must be about meeting the moment. More needs to be done to create a meaningful, viable, effective, and transparent evaluation system for frontier AI. Otherwise, the audits will end up at best like Caesar’s wife (sidelined and under suspicion) and at worst set up to fail in the critical mission of averting what the frontier AI CEOs themselves have warned are grave risks to humanity.
