…বাংলা
Barlekha Today logo
All the big stories — in your language, updated every 15 minutes
Home › World › News
World

OpenAI launches new tool to detect AI-generated child abuse imagery, adds warnings and 24-hour suspensions on ChatGPT

Staff Reporter
OpenAI launches new tool to detect AI-generated child abuse imagery, adds warnings and 24-hour suspensions on ChatGPT
Photo: Collected

OpenAI is introducing a new artificial intelligence tool designed to detect suspected child sexual abuse material, including content created with AI. The company is also cracking down on attempts to find abusive material through ChatGPT, with new warnings and 24-hour suspensions.

Many existing detection systems use digital fingerprints, called hashes, to recognize previously identified abusive files. However, newly created AI-generated material may have no matching fingerprint. OpenAI's new classifier examines visual signs of potential child sexual abuse material, known as CSAM, allowing it to flag suspected abuse even when the content has never been identified before.

The system assigns scores to potentially abusive pictures so human reviewers can decide which cases need attention first. OpenAI says it developed and evaluated the technology with help from the National Center for Missing & Exploited Children (NCMEC). The company is offering the classifier at no cost to vetted organizations through its API, which could give smaller child protection teams access to detection technology they otherwise could not afford.

In early testing on pictures uploaded to its systems, OpenAI compared the new classifier with its existing protections. The tool flagged 48% fewer pictures for review while detecting roughly the same amount of abusive material, and pictures flagged by the new classifier were about twice as likely to contain abuse.

Outside organizations reported additional results. AI platform Fal said its reviewers confirmed 72% of high-confidence flags as previously unidentified CSAM, while SafetyKit reported that the classifier detected 90% of cases involving drawings or animations that sexualize children in its evaluation. These results come from OpenAI and participating organizations; independent testing will help determine how reliably the technology performs across different platforms.

On ChatGPT, new warnings will interrupt attempts to find child sexual abuse material and explain the harm caused by sexualizing children. The warnings specifically address AI-generated material, which can cause serious harm even when a picture has been digitally fabricated. Along with displaying a warning, OpenAI says it will remove the flagged conversation and pause access to ChatGPT for 24 hours, and the warning will direct the person toward specialist support designed to help prevent harmful behavior.

Repeated violations may lead to longer restrictions or permanent account closure, and could also result in reports to authorities. Under its existing policies, OpenAI says it bans accounts and reports attempts to generate or upload CSAM to NCMEC. The company developed the warnings with input from child protection organizations and researchers who study ways to interrupt harmful online behavior, though whether these interventions prevent future abuse will require continued evaluation.

OpenAI is also expanding its work with organizations investigating online child exploitation. It has begun participating in the Tech Coalition's Lantern program, which allows organizations to share information about suspected exploitation across platforms, and is introducing specialized AI tools and training for child safety law enforcement units. The first investigative tool will help authorized teams review lengthy chat logs and records obtained through legal processes, identify potential connections and organize timelines, with findings linked to source material. The company is also publishing guidance to help developers detect suspected abuse and establish review procedures.

The scale of the problem is alarming. According to NCMEC, its CyberTipline received more than 400,000 reports involving generative AI in 2025, covering several categories of AI-related child exploitation, although not all involved the creation or sharing of AI-generated abuse material. The new classifier works behind the scenes, so its impact will depend on how widely organizations adopt it and how effectively their safety teams respond to warnings.

Get the news firstNew stories in your inbox — subscribe free.

More in World