Skip to content
Thu 19 Nov 2026 Toronto Sign in
AI & ML

OpenAI’s free CSAM classifier misses half the infant images in its own test set. What the model card shows

OpenAI is giving vetted platforms a free classifier that flags child sexual abuse material, including AI-generated images. Its model card shows wide gaps by age, race and sex.

Abstract illustration: a shield over a grid of squares, some lit and some empty
Illustration: prompt/power

OpenAI is giving vetted organizations free access to an image classifier that flags suspected child sexual abuse material, including AI-generated material, through its API. It announced the tool on Nov. 16 with three other measures: warnings and a 24-hour ChatGPT suspension for people who try to find abusive material, new guidance for developers, and investigative tools for child-safety police units. The model card published the same day is more candid than the announcement about what the classifier misses. We read both.

What OpenAI released

  • The classifier. It scores images for two signals, suspected abuse material and suspected AI-generated abuse material, so human reviewers can decide what to look at first. OpenAI says it’s “the first CSAM classifier with broad availability that has a specific signal on indicating if CSAM may be AI generated.” It was built with assistance from the National Center for Missing and Exploited Children (NCMEC), which runs the U.S. CyberTipline. Access is free for approved organizations through an intake form, and OpenAI is sharing it through the Tech Coalition and ROOST.
  • Warnings in ChatGPT. When someone tries to find abusive material, ChatGPT will show a warning, remove the conversation and pause access for 24 hours, with links to support services. “Repeated violations may lead to longer restrictions and ultimately account closure or reports to authorities,” the company says. It developed the warnings with deterrence researchers, including the Lucy Faithfull Foundation, which runs the Stop It Now helpline.
  • Developer guidance. A CSAM guide for developers written with input from the Tech Coalition, Thorn, NCMEC and Britain’s Internet Watch Foundation.
  • Tools for investigators. The first one helps child-safety police review long chat logs and legal-process returns and build timelines, with access controlled by program administrators.

What the model card says it gets right

On a held-out test set of about 46,849 images, OpenAI reports that 99.1% of the images the main signal flagged were abuse material (its precision) and that it caught 80.2% of the abuse images in the set (its recall). On roughly 1.25 million images from an open-web research dataset that had been filtered to remove such material, it flagged five, a rate of 0.0004%.

In OpenAI’s own comparison on user-uploaded images, the new classifier would flag 48% fewer images than its existing tool while finding “roughly the same number” of abusive ones. Early partners report their own numbers: reviewers at fal confirmed 72% of its high-confidence flags as previously unseen material, and SafetyKit says it detected 90% of the cases it tested involving drawings or animations. Those are the partners’ figures, not independent audits.

What it misses, by OpenAI’s own numbers

The model card breaks recall down by age, race and sex, using tags an AI model assigned to the people in each test image. OpenAI calls the results “directional,” and some groups are small. Even so, the gaps are wide.

Group in the test set (AI-assigned tag) Abuse images in the set Share the classifier caught
All images 6,849 80.2%
Infant 1,107 49.3%
Toddler 928 74.5%
Prepubescent 3,983 93.0%
Black 142 61.3%
Caucasian 4,768 87.5%
Male 1,159 60.3%
Female 4,527 89.3%
Graphic: prompt/power. Source: OpenAI CSAM Image Classifier Model Card, Table 5 (“likely-csam” signal at its default threshold), Nov. 16, 2026.

Put simply: on OpenAI’s own test set, the classifier missed about half the abuse images tagged as infants, and caught abuse images tagged as Black children or boys at a much lower rate than the average. OpenAI says “more work is needed” on race, skin tone, sex and age, and that it will “explore methods to improve on these limitations” in future training runs. It hasn’t released three further internal signals for the most severe categories, saying it needs to validate them first.

The card lists other limits too. Near-duplicate images can leak between training and test sets, so the test results may flatter the model. And a score “does not confirm that an image is CSAM, establish its legal status, or determine whether it must be reported.”

Why it matters outside the U.S.

OpenAI trained the classifier on a definition that “encompasses CSAM as defined under U.S. federal law and broader definitions covered in other jurisdictions,” but says it “is not a jurisdiction-specific legal classifier.” That matters for Canadian and British platforms. Canada’s Criminal Code definition covers written material and audio as well as images, and U.K. law separately criminalizes non-photographic images of children. A platform using the tool still has to apply its own country’s law and reporting rules.

Our read: the classifier is a real gift to small trust-and-safety teams that can’t build their own, and publishing the demographic gaps is more than most vendors do. But a tool that misses half the infant images in its own test set is a triage aid, not a filter. It belongs alongside hash matching and trained human review, which is exactly how OpenAI says to use it.

What platforms should do

  1. Keep hash matching. The classifier is built to find new images that known-image hash lists miss. It isn’t a replacement for those lists.
  2. Route, don’t decide. Use the score to order human review, not to make a legal call.
  3. Watch the weak spots. Don’t lower review standards for the groups where the model card shows low recall.
  4. Check your own law. Confirm what you must report, and to whom, in each country you operate in.

If you come across child sexual abuse material online, report it: in the U.S. to the NCMEC CyberTipline, in Canada to Cybertip.ca, and in the U.K. to the Internet Watch Foundation. Don’t download, save or share it.

More from prompt/power: AntiFox and the six checks to run · what ChatGPT for Healthcare’s fine print says · OpenAI’s fired safety researchers

Sources

// How we reported this

Researched and drafted with AI tools, fact-checked against primary sources and approved by a human editor before publishing. Prices and dates are as of publication. Our standards →

Sources
  1. openai.com — announced the tool on Nov. 16
  2. deploymentsafety.openai.com — model card
  3. openai.com — intake form
  4. developers.openai.com — CSAM guide for developers
  5. laws-lois.justice.gc.ca — Criminal Code definition
  6. legislation.gov.uk — separately criminalizes non-photographic images
Columnist, Consumer Tech Lauren Smith

Lauren Smith covers consumer apps and services for prompt/power: streaming, subscriptions, e-commerce and everyday tech worth your money. She keeps a running tally of every free trial that quietly turned into a subscription.

All stories by Lauren →

Latest from prompt/power

  1. IBM, NVIDIA and Fermilab plan a quantum test bed. Here’s what each brings, and what’s still missingNov 19
  2. GPT-6.2 Astra is OpenAI’s first released model rated ‘High’ for AI self-improvement. What that means, and what it costsNov 18
  3. Anthropic’s last filing before pricing: Nasdaq approval, a US$6.5-billion stock-pay bill and a redrawn lock-upNov 18
  4. Celestica, the Toronto company building AI’s racks, expects US$40 billion in revenue next yearNov 18
  5. ChatGPT now lets brands claim a profile and run their own agent. What creators and publishers should knowNov 18

Leave a Reply

Your email address will not be published. Required fields are marked *