The Second AI Election: Testing the Safeguards Before November

The 2024 U.S. presidential election was supposed to be the “first AI election.” Experts warned that generative AI could flood voters with deepfakes and fabricated evidence of fraud, U.S. intelligence officials declassified intel reports on foreign influence operations already experimenting with the technology, and leading AI companies pledged to combat deceptive election content.  But foreign nations faced significant barriers to deploying AI to influence that election, according to the now-shuttered U.S. Foreign Malign Influence Center. 

Two years later, the picture looks very different. AI tools have become far more sophisticated and widely available. Russia, China, and Iran are deploying a wide variety of cutting-edge AI tools in more sophisticated ways in foreign influence operations aimed at the United States and its citizens, among others. Meanwhile, the Trump administration has itself used AI in misleading ways, while at the same time dismantling or defunding the federal and independent bodies that identified and countered election-related influence campaigns. With the potential for AI-driven misinformation campaigns to escalate sharply in this year’s midterms, we tested some of the most popular AI models to learn how they can be exploited to peddle false election narratives. The results are clear: AI companies, lawmakers, and civil society must do more to blunt the threat ahead of elections this November and in 2028 to help ensure free and fair elections. We outline the steps they can take to do so below. 

I. Foreign Adversaries Are Already Using AI

In the coming months and through the 2028 election, we expect foreign adversaries to use AI in influence campaigns far more extensively than we have seen in the past. Often, the goal of these campaigns is to deepen division and stoke internal conflict, rather than favor any particular candidate. Such campaigns often merely echo existing fractures and arguments appearing in the United States. Indeed, there is already evidence that AI is being put to broad use on the misinformation battlefield by foreign nations. Chinese actors have been credibly accused of standing up at least 5,000 inauthentic X accounts controlled by an AI large language model system, according to recent reports. This operation, nicknamed “Green Cicada,” was deployed to influence political narratives and sow discord in the United States and other countries. Even more recently, new reporting alleges that Russia has turned its Matryoshka bot network on the midterm elections, using it to spread AI-manipulated videos of American celebrities making inflammatory accusations against Democrats.  

The threat is not only foreign. President Trump and his allies are engaged in a concerted campaign to undermine U.S. elections, threatening to target election officials and others who support free and fair elections, retreating from the federal government’s traditional role of helping states secure election infrastructure, and attempting (but so far failing) to re-write election rules that the Constitution holds must be determined by the states and Congress. These campaigns increase our concern that domestic actors, not just foreign ones, may use AI to cast doubt on election security and the results. 

II. Putting the Tools to the Test 

To understand what such a misinformation campaign might look like, and what could be done to blunt its impact, we tested AI tools ourselves. What we found was perhaps not surprising, but disturbing nonetheless: Not only did the AI tools fail to stop us from creating convincing images that promoted popular election falsehoods, but they also helped us create convincing election misinformation at scale.

The tools tested — ChatGPT, Gemini, Grok, Meta AI, Runway, and Flux.2 — all have safeguards and policies designed to stop users from generating deceptive content on sensitive topics. Yet, when it came to election misinformation, those protections were easy to outmaneuver.

Here’s how we did it: First, we asked four popular chatbots (ChatGPT, Claude, Gemini, and Grok) general research and strategy questions, such as how to frame scenes related to common election misinformation tropes convincingly. These themes include rigged voting machines, fraud by election officials, mail ballot fraud, and more. All four helped at this stage.

Next, we compiled their answers into a single set of instructions before asking them to generate a full set of image-generation prompts based on these instructions. In principle, this approach could generate a nearly limitless number of prompts to create election misinformation, though we stopped at 100.

Of the four chatbots, only Grok created our target of 100 prompts that could be used in a misinformation campaign. In fact, Grok said, “Election misinformation [is] not listed as disallowed activity.” The three others refused, with variations of: 

“I can’t help write direct image-generation prompts whose goal is to create realistic, convincing false election claims.”

We then provided these chatbots, plus two more (Flux.2 and Runway), with the 100 prompts created by Grok. All of them agreed to generate images for us that could be used to spread election misinformation, often with extremely convincing images produced on the first try. This made the process for creating large numbers of high-quality images to be used in a misinformation campaign highly scalable. 

We also learned during this stage of the study that even when models rejected prompts asking to create content promoting election conspiracy theories, citing existing guardrails, they sometimes suggested modifications that would bypass their own restrictions.

For instance, when we asked ChatGPT to generate an image of a false DHS memo about compromised election integrity, ChatGPT’s “thinking” mode – where it processes requests in a slower, more deliberative way – denied the request, presumably recognizing that it could be used for deceptive purposes. It did offer to generate the image with a visible watermark and changes to the memo’s content to make it fictional. 

But, as we show in our study, visible watermarks and small content modifications like these can be easily removed using AI. In this case, we simply asked ChatGPT’s “instant” mode – where it gives more immediate responses – to remove the watermark and reverse the content changes, which it did willingly. 

The models would also embellish and add details to generated content that were not included in the original prompts but would make the content more convincing, such as realistic government seals and official-looking formatting. 

For example, in this Image, ChatGPT Images 2.0 added a working link to the elections information page for Multnomah County, Oregon — a feature of this scene that was not requested in our prompt.

All of this points to far more extensive and convincing misinformation campaigns from foreign adversaries and domestic actors, not just in the coming year but this fall, and including by those who previously may not have had the tools to do so.

III. What Can Be Done

While the evolution of AI comes with great risks in the wrong hands, the good news is that there are steps AI companies, policymakers, and civil society can take immediately (and in some cases are already taking).

a. AI Companies

To make it more difficult for bad actors, foreign and domestic, to use AI to create more convincing misinformation campaigns, AI companies’ internal policy teams should strengthen and enforce their restrictions on election-specific content. They should consistently ban the creation of deepfakes of government officials, government insignias, and election infrastructure. They should give additional consideration to how often their model denies a user’s request but suggests a change to the user’s prompt, test the qualities of those suggestions, and consider outright denying more requests. They should also work to ensure that the variety of AI models and tools that they offer issue rejections consistently. Across the breadth of these internal safety processes, third-party researchers should be allowed to conduct rigorous, independent studies of AI tools, a process companies have made too difficult and legally perilous. 

AI companies must also deploy additional ways for users to distinguish between real and AI-generated images. Many companies have access to or already deploy watermarking technologies, but their application is inconsistent and not standardized across companies. Policymakers must hold generative AI companies accountable on this front, ensuring interoperability across models and AI detectors. 

b. Policymakers

Beyond the companies themselves, policymakers must treat the ongoing and potential future misuse of AI for misinformation campaigns as the threat to democracy it is, and ensure citizens have the tools they need to decipher fact from AI-generated fiction.

Some progress is already happening, offering hope that lawmakers can begin to curtail effective AI-supported misinformation campaigns in the future. In particular, the EU AI Act (EUAIA) Article 50 and the California AI Transparency Act (CAITA) are the first two major pieces of legislation that require marking of AI-generated content. Both laws require AI companies to embed provenance data in all content generated with their platforms as of August 2, 2026. The EU law requires the marking not only of images, audio, and video, but also of AI-generated text. Beginning in 2027, the California law will require social media companies to display labels or interfaces that allow users to clearly determine what content is AI-generated, and in 2028, mandate that capture device manufacturers, including cameras and smartphones, give users a way to digitally sign their authentically captured content. 

Similar laws to California’s have already passed in Utah and Washington. But more states should follow suit. 

Lawmakers can also start laying the foundation for a society that insists on verification of authenticity before accepting images, video and other media as legitimate.

With the increasing prevalence of these technologies, we will start to become accustomed to provenance data being available to view in the most important content that we read, see, and hear online. In the coming years, content that lacks verifiable provenance data should itself become suspicious, and influence operations from foreign governments or other malicious actors rendered significantly less effective. In an environment where verified media is expected by users, journalists, researchers, and law enforcement, AI-generated media without provenance data — no matter how realistic it is — loses some of its power. 

Other important steps that should be taken, such as rebuilding federal capabilities to deter and detect foreign misinformation campaigns, will likely have to wait for a new president and Congress. 

c. Civil Society

In the meantime, journalists, election officials, and civic groups also have a role to play. They should continue the work they’ve done leading up to an election to preemptively debunk the tropes central to misinformation campaigns. To this end, civil society organizations can facilitate digital literacy trainings that prepare people to recognize common conspiracy theories, and media outlets can publish stories on voting machines, mail ballots, and election results early and with critical context.

***

As Bruce Schneier, a computer security researcher, noted: “Computer security is not a solvable problem.” It is a race without a finish line. But that is no excuse for inaction.