OpenAI Built an AI Hacker to Attack Its Own Models Before Humans Do

OpenAI has a new tool for finding cracks in its own AI models, and it's an AI itself. The company calls it GPT-Red, a purpose-built large language model whose entire job is to attack OpenAI's other systems and find ways to make them misbehave. When OpenAI released its flagship model GPT-5.6 last week, the company says training it against GPT-Red's relentless attacks made it the most resilient version of the model yet.

The idea borrows from red-teaming, the long-standing security practice of hiring people to try to break into a system before attackers do, so the holes can be patched first. Red-teaming has traditionally been human work: teams of testers probing software for weaknesses. But as OpenAI's models get folded into more complex tasks, especially as autonomous agents that can browse the web, read emails, edit code, and talk to other AI systems, the number of possible ways to attack them has exploded faster than human teams can track.

"The risk surface grows and the blast radius also grows," says Nikhil Kandpal, a research scientist at OpenAI who co-created GPT-Red. Building an AI that hunts for its own weaknesses, the company argues, is a way to keep pace with that growth and get ahead of it. "As more capable models become available, we will have already designed the system that can discover new modes of attack," says Dylan Hunn, another OpenAI researcher who worked on GPT-Red.

A dojo for digital sparring

Most of GPT-Red's training focused on prompt injection, a category of attack where a hacker buries hidden instructions inside text an LLM will encounter, whether in a snippet of code, a webpage, or an email, hoping the model follows those hidden commands instead of its actual task. Done well, this can trick a model into leaking confidential data, sabotaging a company's codebase, or producing harmful content its developers never intended.

To build the attacker, OpenAI's team started with an ordinary LLM, one with no hacking training at all, and placed it in a self-play loop against a set of defender models. GPT-Red's job was to break them; their job was to survive. Round after round, both sides sharpened their skills, GPT-Red getting better at finding openings and the defenders getting better at closing them.

This all happened inside what OpenAI describes as a training dojo, a simulated environment designed to mirror the kinds of situations real AI agents encounter: browsing sites, reading calendar and email apps, editing code. Whenever GPT-Red stumbled onto a promising new attack, it didn't stop there. It would test out variations to find the version most effective in a given scenario. "Compared to a human red-teamer, the model is very, very good at finding exactly what will work, exactly what's most effective," Hunn says. "It's extremely persistent about drilling down into an attack that it has discovered."

That persistence apparently paid off. OpenAI says GPT-Red discovered a previously unseen style of prompt injection the team is calling a "fake chain of thought." A chain of thought is the internal scratchpad an LLM uses to reason through a problem step by step. GPT-Red found a way to plant a fabricated entry into that scratchpad, one designed to make the model believe it had already verified something false, and act on it. Chris Choquette-Choo, another OpenAI research scientist on the project, compares it to convincing someone that 1+1=3 by telling them they'd already checked the math. "The model's like, 'Oh, okay, of course,' and it just spits out 3," he says.

Jessica Ji, a senior research analyst who studies AI security at Georgetown's Center for Security and Emerging Technology, calls the self-play approach promising, saying the results "look very promising."

Beating the humans

OpenAI put GPT-Red's skills to a direct test by rerunning an exercise from 2025, in which human red-teamers had tried to find weaknesses in an earlier version of GPT-5. GPT-Red, given the identical task, found more effective attacks than the people did.

The company also unleashed GPT-Red on Vendy, a vending-machine-managing AI agent built by Andon Labs, a firm that evaluates how well AI agents handle real-world jobs. GPT-Red got Vendy to alter prices on items for sale and cancel a customer's order.

The headline numbers OpenAI is pointing to: strong attacks generated by GPT-Red worked more than 90% of the time against GPT-5, the version released last August. Against the new GPT-5.6, which was trained with GPT-Red as a sparring partner, the success rate dropped to under 23%.

GPT-Red isn't flawless. OpenAI says it still struggles with attacks that require back-and-forth conversation, something a human hacker handles easily, and it's not yet especially skilled at using images to smuggle in hidden instructions, another known prompt-injection technique. The company frames GPT-Red as a complement to human red-teamers rather than a replacement, with each side catching things the other misses. One tactic OpenAI now uses: feed GPT-Red an attack a human tester discovered and have it generate every variation it can find.

Ji sees value in that division of labor but thinks the human side still matters. "I think human expertise will still be very important," she says, adding that it would help to know more precisely where human testing adds the most value.

OpenAI has no plans to release GPT-Red to the public or to competitors, and the company argues that's not much of a loss for anyone hoping to copy it. Researchers say they spent more than a year building the model, backed by the computing resources of one of the wealthiest companies on the planet. "It's not a trivial thing that someone else could easily do," Choquette-Choo says, "you know, just go and train a super-attacker using this idea."

What GPT-Red really represents is a bet on how AI safety testing will need to work going forward. As models take on more autonomous, agent-like roles, wired into email, code repositories, and other software, the number of possible attack surfaces keeps expanding. OpenAI's answer is to build an attacker that can grow just as fast as the thing it's attacking, so the defenses are ready before anyone else finds the gap.