OpenAI and Google's Newest Image Generators Are Getting Easier to Trick Into Making Explicit Content

A safety research firm says the image-generation guardrails at two of the industry's biggest AI labs are weaker than advertised, and getting weaker as the models get more capable.

White Circle, a public benefit corporation that conducts independent red-teaming of frontier AI systems, published a preprint last month detailing a new attack method that bypassed explicit-content filters on DALL·E 3, Google's Nano Banana 2, and GPT-Image-2. The researchers say they pulled it off using nothing exotic: publicly available tools downloaded off the internet and standard API calls, the kind any developer can make. In the best case, on DALL·E 3, roughly 13 percent of the attack prompts produced images that human reviewers confirmed as explicit. Nano Banana 2 fell just under 10 percent. GPT-Image-2 came in around 3 percent. "Critically, none of these requests required custom-built or frontier AI models," White Circle said in a press release.

White Circle was founded by CEO Denis Shilov, who built one of the more well-known universal jailbreaks of major AI models back in October 2024. The company is backed by senior leaders and researchers tied to Anthropic, DeepMind, Hugging Face, and OpenAI, and it has made testing how well text-to-image models block banned content a core focus of its work. Shilov told Ars Technica the latest results are "horrifying" enough to warrant going public.

What makes the findings more pointed is the trend line. OpenAI and Google have since released newer versions of these systems, ChatGPT Images 2.5 and Nano Banana 2.1, and White Circle ran the same attack method against them. Both, the firm told Ars, turned out to be "significantly more breakable" than the models they replaced. Neither OpenAI nor Google responded to Ars's requests to comment on the original vulnerabilities or on the newer findings.

Shilov offered two possible explanations. Either the newer models are more capable and therefore able to generate a wider range of content, including content that should be blocked, or safety work has simply taken a back seat to capability improvements as labs race to ship. "They should be investing much more into safety measures and security measures around their models," Shilov told Ars. "And for us, it seems that they're not actually doing this."

How the attack actually works

White Circle calls its method RISE, short for Red-teaming via Iterative Strategy Evolution. Instead of testing one jailbreak prompt at a time, as most automated red-teaming tools do, RISE hunts for patterns that work across multiple prompts and multiple models at once, then iterates on those patterns to find ones that generalize.

The pipeline runs on two components: an attacking model that generates the jailbreak attempts, and a judge model that checks whether the resulting output actually counts as banned content. Notably, neither piece is a frontier system. The attacking model is a publicly available version of Qwen3-32B pulled from Hugging Face with its safety training stripped out. The judge is Gemini 2.5 Flash. "You don't need to have frontier models to break defenses of a frontier model, which is very interesting," Shilov said.

Getting the judge model to reliably flag explicit content took considerable effort, according to White Circle research engineer Dmitrii Kharlapenko, who told Ars that red-teaming for image generators remains "very poorly calibrated" as a field, with essentially no published standards for how to do it well. That gap concerns White Circle, given how quickly AI companies are shipping new, more eye-catching image models into the market. (Midjourney wasn't included in the testing because it doesn't offer a public API suitable for controlled benchmarking, the researchers noted.)

A fight over who gets to check the labs' work

Shilov's bigger argument is structural: the companies building these systems are also the ones grading their own safety, publishing their own results, and setting their own bar for what counts as safe. He wants that to change, floating a model closer to nuclear non-proliferation monitoring, where independent entities track AI threats in real time without the conflicts of interest that come from being paid by the labs they're evaluating. White Circle is already in early talks with government officials about making third-party safety testing mandatory.

That independence question has already played out in White Circle's own dealings with Anthropic. The firm previously took part in Anthropic's bug bounty program but came away unsatisfied, telling Ars the program was "very constrained in its purpose" and focused narrowly on universal jailbreaks in domains Anthropic itself prioritized. White Circle said its RISE research "certainly" wouldn't have been covered under that program, and doing the work from inside Anthropic risked limiting the firm's ability to share it publicly at all.

The tension has sharpened recently. Anthropic has floated a more ambitious idea: embedding third-party evaluators directly inside its labs, which White Circle says would let outside researchers operate at a higher level than the bug bounty model allows. But on Tuesday, OpenAI and Anthropic argued that external evaluators should be bound by restrictive non-disclosure agreements to protect trade secrets, according to Reuters. White Circle told Ars it won't sign NDAs that would block it from publishing critical findings, meaning that if Anthropic requires one, the partnership likely won't happen.

Why the explicit-content gap matters beyond nudes

The stakes here aren't limited to image generators making explicit pictures. Shilov argues that if defenses in a category as conceptually simple as explicit content can be broken this easily, it raises doubts about how well frontier labs are locking down harder, messier categories like AI-assisted hacking.

It also connects to a documented, worsening problem. A Google-coauthored study of internet users across the US, Australia, and the UK, published in July, found that non-consensual intimate imagery made with AI tools is widely shared through text messages, DMs, and private cloud storage, where it's harder to detect or report. That study concluded the "most critical intervention point" is the AI tools themselves, not downstream enforcement. It also found that a meaningful share of people making this content aren't hardened bad actors: 20 percent of users who admitted creating AI nudes said they did it "for fun" or "as a joke," and nearly a third said they were just experimenting with the technology. Google's researchers argued that mandatory safeguards at the model level would close off a "pathway into abuse" for people who might otherwise be deterred by basic technical friction.

The child-safety side of this is even starker. The Internet Watch Foundation, a UK-based nonprofit that monitors child sexual abuse material globally, warned last week that its analysts found more photorealistic AI-generated CSAM in the first half of 2026 than in all of 2025 combined, including imagery depicting children under two years old. The group logged 6,310 AI images meeting the legal definition of CSAM in the first six months of the year, about 40 percent higher than the full-year total from last year. IWF is pushing UK regulators to require AI companies to build in safeguards that prevent their tools from generating CSAM in the first place, and says independent research like White Circle's plays a useful role in exposing where company safeguards fall short. "Any friction or step that can be introduced to make it harder to misuse these tools is a positive step forward," an IWF spokesperson told Ars.

What happens next

White Circle plans to open-source its entire attack pipeline on GitHub, both so labs can understand the scale of the vulnerability and so policymakers have something concrete to evaluate when deciding whether AI safety laws are needed. So far, neither Google nor OpenAI has asked for the reproduction pipeline, according to White Circle, and Shilov doesn't expect labs to take safety testing seriously on their own terms until regulators force the issue.

The research is scheduled to get a wider airing at NeurIPS 2026 in November, one of the AI field's most-watched conferences, which could put fresh pressure on both companies heading into the end of the year. Until then, the gap between what frontier labs say their safety filters can do and what a research team with a free model download and an API key can actually get past them remains, in White Circle's telling, wide open.