Skip to content
Theme
OnDevice Tools

AIChatbots & Assistants

OpenAI CSAM classifier and ChatGPT warnings target abuse images

OpenAI has released an image classifier that flags child sexual abuse material for selected partners and added warning screens to ChatGPT that can suspend an account for 24 hours.

Illustration of a ChatGPT warning screen beside an image classifier scanning for abusive content

Key takeaways

  • OpenAI released an image classifier that scores suspected CSAM, including AI-generated images, and gives selected companies free API access.
  • In early partner testing, OpenAI says the tool flagged 48% fewer images while finding the same number of abusive ones.
  • ChatGPT now shows a warning and suspends an account for 24 hours when a user tries prompts that sexualise children, with permanent removal possible for repeat offences.
  • OpenAI also published developer guidelines and is offering investigative tools and training to law enforcement.
In this article
  1. What OpenAI announced
  2. How the image classifier works
  3. Fewer false alarms in early tests
  4. Warnings and temporary blocks in ChatGPT
  5. Guidance for developers and tools for investigators
  6. Why the pressure is growing

What OpenAI announced

OpenAI announced a package of child safety measures on 16 November 2026, including a new image classifier for child sexual abuse material (CSAM) and warning screens inside ChatGPT, TechCrunch reports. The company says the classifier can detect abusive images, including pictures generated by AI, and that ChatGPT will now interrupt some prompts with an alert.

The announcement covers four areas: the classifier itself, deterrent messages in ChatGPT, written guidance for developers and a set of tools for investigators. OpenAI has not said how many organisations have been given access so far.

How the image classifier works

The tool is built to spot visual signs of abuse in images it has not seen before, rather than matching files against a known list. It then gives each image a score, which OpenAI says can give human reviewers an indication of how to handle a case. Access is being offered through OpenAI's API at no cost to a limited group of companies.

OpenAI developed the classifier with the National Center for Missing and Exploited Children (NCMEC), the US body that collects reports of abuse material. The company is also working with Tech Coalition and ROOST to widen access to the tool.

Fewer false alarms in early tests

In early testing with partners, OpenAI says the classifier flagged 48% fewer images while still finding the same number of abusive ones. Images it did flag were twice as likely to be abusive, according to the company. Both figures come from OpenAI's own account of the trial and have not been independently verified.

If the results hold, the practical effect would be less review work for the teams that check reports, because reviewers would spend less time on images that turn out to be harmless.

Warnings and temporary blocks in ChatGPT

ChatGPT already refuses CSAM content and requests for it. The new screens go further: when someone tries a prompt that sexualises children, the app shows an alert, points the person towards specialised support and suspends the account for 24 hours. OpenAI says repeated offences could lead to an account being removed permanently.

The company says the wording was shaped with outside organisations. These include the Lucy Faithfull Foundation and Protect Children, which helped form the guidance. Joel Scanlan, a researcher at the University of Tasmania, independently evaluated a trial of the deterrent messages.

Guidance for developers and tools for investigators

Illustration: code, OpenAI logo, building, AI and the labels ChatGPT, Child Safety, Image Classifier

OpenAI is publishing guidelines for developers who build on its tools, covering prevention, detection and ways for users to report material. In early testing, the company says the guidance led several developers to adopt child safety tooling and to register so they can report material to NCMEC.

For law enforcement, OpenAI is offering AI features aimed at child safety work, among them specialised plugins, tools for investigations and help with analysing complex data. Access is managed by programme administrators, and the company is providing training for investigators along with a field guide on using AI in these cases.

Why the pressure is growing

The announcement follows a run of lawsuits blaming OpenAI's technology for harming users, particularly children. The nonprofit Common Sense Media has labelled OpenAI's technology an "unacceptable risk" for teens. Other chatbots, notably Grok, have been used to create nonconsensual sexualised imagery, including of children.

Over the past 12 months OpenAI has published a safety blueprint, updated its content guidelines, retuned its models to limit harmful output and released a version of ChatGPT for teens. The new measures extend that work from the chatbot itself to the partners and investigators who handle reports.

For most users the visible change is the warning screen: a prompt that crosses the line now ends the session for a day. Developers who want the classifier have to be selected by OpenAI. Organisations that deal with reports should follow the published guidance and the reporting process to NCMEC that it describes.

Sources

  1. OpenAI releases a new image classifier and ChatGPT warnings to combat CSAMTechCrunch AI
  2. OpenAI releases a new image classifier and ChatGPT warnings to combat CSAMTechCrunch

Tags

Spotted a mistake? See our corrections policy. Read how News by OnDevice works in our editorial policy.