AI anxiety has reached a fever pitch. Anthropic CEO Dario Amodei has called for an urgent slowdown at the frontier to buy time to better manage risks and vowed to erect a robust internal oversight mechanism within the company. OpenAI CEO Sam Altman has chimed in to back “pacing” and independent internal evaluation, and even buccaneer Elon Musk concurred with a terse “Dario is right.”
If they are serious about oversight to curtail the most serious risks of AI, tech leaders will need to endow outside evaluators with enough independence, information, money, and clout to force decisions company leaders would rather not make, reining themselves in ways that will feel counterintuitive and even excruciating. Short of that, this weekend’s announcements will be remembered as barely a speed bump en route to a future tech titans have warned us to fear.
Amodei’s latest essay acknowledges the obvious: neither Congress, the administration, nor international bodies have the expertise, cohesion, or will to effectively rein in AI. The sole viable path to robust AI safety in real time is for the companies to act themselves. Amodei has pledged to equip “embedded evaluators” with “employee-like access to verify safety practices and report incidents,” voluntarily adopting the sorts of constraints he has called on governments to force upon Anthropic and its peers. Amodei has decided that while, ideally, companies would act jointly with common standards, there is no time to waste and he must allow the policing of his own creation.
The idea of independent oversight for cutting-edge tech companies is not entirely new. In 2020, Meta, then Facebook, established a pioneering independent Oversight Board, a model that offers a dose of inspiration and a larger helping of caution. The Oversight Board, of which I am a member, was established by CEO Mark Zuckerberg after the company suffered a series of black eyes including the wholesale privacy breaches of the Cambridge Analytica scandal and accusations of complicity in mass killings of Rohingya Muslims in Myanmar. Opining that “I don’t think as a society we want private companies to be the final word” governing social media, Zuckerberg constituted a now 20-member body of independent journalists, scholars and human rights experts to navigate tensions between free expression and harms like privacy violations, incitement to violence, disinformation, harassment, and bigotry.
Today’s impulse from AI leaders is similar: wanting disinterested experts to draw boundaries and curb hazards. But, as Meta acknowledged in its recent $17 billion settlement of claims regarding harms to youth, the Oversight Board hardly rendered Facebook and Instagram risk-free. A narrow mandate has confined the Board to examining just a few hundred drops of Meta’s vast ocean of content, leaving major aspects of the company’s operations and impact beyond its reach. Moreover, as acute as the ills social media has posed for young people, mental health, and public discourse, the potential perils of AI, including the potential destruction of humanity, are far more dire. To meet the moment, independent oversight of AI will have to be empowered, equipped, and intrusive, treading where tech leaders will be predictably reluctant to let it go.
The Oversight Board has issued hundreds of thoughtful, credible, and sometimes consequential published rulings advancing users’ rights to privacy, control over their feeds, and free speech on Meta platforms. It is globally representative, and its members exercise informed judgment. Through its public comments and consultations, the Board offers a crucial point of entry for global civil society to influence Meta to uphold free expression, gender equality, and other human rights.
But the Board’s limited authority and resources are no match for the global reach of social media, much less AI. It cannot initiate its own investigations, probe algorithmic functions, adjudicate account suspensions or monetization policies, or shape other matters of vital concern to platform users. Despite some independence safeguards, every aspect of its operations hinges on cooperation from the company: the Board depends on Meta for access to information, implementation of its recommendations, and funding.
Amodei’s announcement avers that effective AI oversight will need to go further. In his recent statement, he promises “embedded” outside evaluators desks at headquarters, permissions “mostly comparable” to what internal risk assessors have, and the right to publish their findings, including noting any company-imposed redactions. Those are positive steps, but Anthropic will have to go much farther to achieve his stated goal of a “radical” new approach to safety.
An independent evaluation team will need empowered leaders with fixed terms and long-term funding commitments that do not depend on the company’s discretion. Without that, every “independent” decision on what to recommend and how far to push will be colored by evaluators’ interest and dedication to continuing their work, which will hinge on sustaining the company’s favor. Despite an independent “trust fund,” the Oversight Board has been continually dogged by doubts over Meta’s commitment, and the prospect of being shut down.
Operating at the pace of AI development, evaluators will need the power to initiate investigations of their own volition, requisition proprietary data, access the time and candid insights of top officials, and oversee the implementation of their recommendations, including adjudicating legal, technical, and commercial hurdles asserted by the company. They will need unfettered access to model evaluations, safety incident reports, interoperability records, red-team results, and internal disagreement logs. They should be able to publish findings without delay, with a formal company response required.
The complex and varied nature of AI risks demands independent overseers and support staff that bring a range of expertise, including technical virtuosity as well as legal, ethics, and human rights mastery and fluency across languages and cultures. Assembling and equipping such high-level teams – including senior-level internal company interlocutors charged with supporting their probes – will not come cheap. AI leaders will also have to force cultural change that does not come naturally to tech innovators whose brilliance lies in pushing boundaries. Meta’s executives and teams have been sharply divided about the Oversight Board, with some embracing the elevation of user rights and others resenting perceived prying. If company leaders fail to back evaluators in inevitable power struggles with valued corporate officials, their influence will evaporate.
AI leaders should enact evaluation with different expectations than those that motivated Zuckerberg, who deserves credit for being the sole tech leader to subject his company to any form of independent oversight to date. The Meta founder originally hoped high profile overseers would unburden him, saying the Board, “almost like a Supreme Court,” would make “the final judgment call” on acceptable speech. But the Board’s limited powers thwarted that hope, probably unavoidably. No independent body can subsume the role of a CEO and Board in rendering the most consequential decisions.
Amodei, Altman, and their peers should not expect oversight to free them from hard calls, or their reputational costs. There will be times when they regret opening themselves up to meddlesome criticism and disagree utterly with what independent experts advise. There will also be inescapable and substantial commercial trade-offs, when following evaluators’ advice will hurt profits, market share, and competitiveness with U.S. and foreign counterparts. There will always be arguments that precautions will not eliminate targeted risks, or may create new ones. Watertight controls are impossible. But if Amodei believes what he has written about the prospects for oversight, he needs to allow that potential to be realized.
A defining moment in this summer’s box office blockbuster, The Odyssey, comes when Odysseus orders himself tied to the mast to resist the temptation of Sirens seeking to lure his ship to ruin. The hero writhes in pain, begging to be released, but his oarsmen, their ears blocked with beeswax, do not hear him and follow his initial instruction, saving them all. As tech CEOs implement independent oversight, they need to prepare themselves for not just second thoughts but acute discomfort in the name of warding off catastrophic risk.
