San Francisco Tells Apple and Google: Pull the AI 'Nudify' Apps or Face Legal Action

San Francisco's city attorney has given Apple and Google a formal ultimatum: pull 13 AI "nudify" apps from their stores or risk legal consequences for helping sell nonconsensual explicit deepfakes.

City Attorney David Chiu sent cease-and-desist letters to both companies on Thursday, obtained by WIRED, demanding they remove eight apps from Apple's App Store and five from Google's Play Store. The apps let users upload photos of real people and generate AI images that strip away their clothing or place them in sexual scenarios, typically marketed under the friendlier label of "face-swapping" tools. Chiu's letters go further than just asking for takedowns. They call on Apple and Google to cut business ties with the developers entirely and stop taking a cut of the in-app payments these apps generate.

"Generating non-consensual intimate images is illegal, harmful, and completely unacceptable," Chiu told WIRED. He estimates the two companies have "made millions of dollars in fees" from apps built around this capability, and argues both should overhaul their app-review processes so nudify tools never make it into the stores in the first place. "These companies have responsibility to ensure that apps on their platforms do not facilitate sexual abuse," he said.

The legal notices lean on California law, which prohibits supporting services that produce deepfake pornography. Chiu's office has been down this road before: it previously took action against 16 deepfake websites offering similar services.

A problem both companies say they're already fighting

Apple and Google both maintain developer policies barring pornography, abuse, and harassment, and both have pulled batches of offending apps in the past after being flagged by journalists and researchers. Google spokesperson Dan Jackson told WIRED the company has already removed "hundreds" of nudifying apps for policy violations, including the five Android apps named in Chiu's letter, and has taken additional steps like restricting search terms such as "nudify" on the Play Store.

"Google Play does not allow apps that contain sexual content, and we continually take proactive steps to detect and remove apps with harmful content," Jackson said in a statement, adding that reported violations trigger investigation and "swift action." Apple did not provide a comment ahead of publication.

But the recurring nature of these takedowns is exactly what worries watchdogs. Katie Paul, director of the Tech Transparency Project, has tracked this issue across multiple reports this year. Her group identified roughly 100 apps capable of generating nonconsensual nudes across both app stores in January and April, along with ads promoting the technology. Those apps were collectively downloaded an estimated 480 million times and may have generated around $120 million in combined revenue, according to TTP's estimates. Google says it has since removed most of the apps TTP flagged.

"We didn't think after the first report that we would see this as a problem again—and it was just as bad, if not worse, after the second report," Paul said. "Apple and Google make a lot of promises in their marketing about how trusted and safe their app stores are. And that is just not what is playing out in reality."

The 'dual-use' loophole

Part of what makes this hard to police is that many of these apps don't advertise themselves as nudify tools at all. A preprint study published in May by researchers at Cornell University and Georgetown University examined 420 apps on Apple's and Google's stores that offered general face-swapping features. Testing 155 of them, the researchers found that 70 percent could be used to swap faces onto nude images, without any safety guardrails to stop it.

The researchers described these as "dual-use" apps: tools that present themselves as harmless and slip past content moderation, while quietly retaining the capability to produce abusive content. One app named in Chiu's letters, which has more than a million downloads, advertises over a dozen AI-generated image styles on its website, several with names like "bikini queen curvy" and "cinematic intimacy" paired with sexualized images of women. Another targeted app promotes itself as producing "free and uncensored" videos. WIRED chose not to name the specific apps, to avoid driving traffic toward them.

Real-world harm, growing scale

The broader nudify ecosystem has expanded rapidly over the past five years, most visibly this past January when xAI's Grok chatbot was used to generate millions of sexualized images. The pattern is consistent across these tools: overwhelmingly, men use them to create explicit images of women and girls, often from nothing more than a single reference photo and a few clicks, with results appearing in seconds thanks to improving generative AI models.

The consequences have shown up well beyond adult targets. Reporting by WIRED and Indicator Media has documented deepfake sexual abuse incidents involving minors at more than 90 schools. Chiu described the human toll bluntly: "These images are used to bully, humiliate, and threaten women and girls. This industry has a horrific impact on one's reputation, mental health, loss of autonomy. There have been victims who've been suicidal."

Chiu says his office intends to keep pressing the issue, calling himself "absolutely horrified" by the scale of the harm. "My hope is that Apple and Google will immediately remove these apps and strengthen their screening systems to make sure that apps like this never get onto their platforms in the future," he said. "It's our hope that these companies will do the right thing—but if they don't, we will have to consider all of our legal options."

For now, the burden sits with Apple and Google to decide how seriously they take that warning, and whether their next round of takedowns will actually stick this time.