California AG sends Musk’s xAI a cease-and-desist order over sexual deepfakes

Earlier this week, the California attorney general’s office announced that it was investigating xAI after reports that the company’s chatbot, Grok, was being used to generate nonconsensual sexual imagery involving women and minors. On Friday, the state escalated its response by issuing a cease-and-desist letter to the company, ordering it to immediately halt the creation and distribution of nonconsensual intimate images and child sexual abuse material (CSAM).

“Today, I sent xAI a cease-and-desist letter, demanding the company immediately stop the creation and distribution of deepfake, nonconsensual, intimate images and child sexual abuse material,” said Rob Bonta in a press release. “The creation of this material is illegal. I fully expect xAI to immediately comply. California has zero tolerance for CSAM.”

The attorney general’s office said it believes xAI may be “facilitating the large-scale production” of nonconsensual nude images that are being used to harass women and girls across the internet. The agency said it expects the company to demonstrate concrete steps to address the issue within five days.

Much of the controversy centres on Grok’s “spicy” mode, a feature designed to generate explicit content. Concerns about the tool have extended beyond California. Authorities in Japan, Canada, and the United Kingdom have launched investigations into Grok, while Malaysia and Indonesia have temporarily blocked access to the platform. Although xAI introduced some restrictions on its image-editing features late Wednesday, the California attorney general’s office proceeded with the cease-and-desist order.

An account associated with X previously condemned such misuse, stating that anyone who uses or prompts Grok to generate illegal content would face the same consequences as users who upload unlawful material. TechCrunch contacted xAI for comment and received an automated response reading “Legacy Media Lies.” The publication also reached out to the California attorney general’s office for additional context.

The rise of widely available generative AI tools has contributed to a growing volume of nonconsensual sexual content online, prompting scrutiny across the tech industry. The issue has also attracted attention from lawmakers. On Thursday, members of Congress sent a letter to executives at several companies — including X, Reddit, Snap, TikTok, Alphabet, and Meta — seeking details on how they plan to address the spread of sexualized deepfakes.