Key Takeaways for Platform Leaders |
|
CaseScan, built by Netspark, is an AI-powered CSAM detection API used by platforms including Wix and DoubleVerify and validated by more than 100 law enforcement units globally, including the FBI and HSI. This post uses CaseScan’s published benchmarks as the reference point for each evaluation criterion, because they’re the most detailed publicly available figures in the market. The criteria themselves apply to any vendor.
Every CSAM detection vendor claims high accuracy. Almost none of them publish the numbers that let you verify that.
If you’re a platform PM, trust and safety lead, or head of compliance currently evaluating detection APIs, you’ve already experienced this. Vendor calls where “industry-leading precision” features prominently without a benchmark attached. Whitepapers built around single headline metrics with no context on how they were measured. The evaluation problem in this space is real: the stakes are high in both directions, the technical criteria are genuine, and the vendors look similar until you push on the specifics.
The regulatory environment has sharpened the urgency. In March 2026, a New Mexico jury ordered Meta to pay $375 million in civil penalties for failing to protect children from sexual exploitation on its platforms, the first time a US state has prevailed at trial against a major tech company on child safety grounds. The REPORT Act, signed into law in 2024, expanded mandatory NCMEC reporting obligations to cover online enticement and child sex trafficking alongside CSAM. Regulators are no longer asking whether platforms have safety policies. They’re asking whether those policies work.
What is CSAM detection?
CSAM detection refers to the automated scanning of images and video to identify child sexual abuse material before or after it is uploaded to a platform. It covers two distinct technical approaches: hash matching, which compares content against databases of previously identified material, and AI classification, which uses machine learning to assess whether previously unseen content depicts abuse. A production-grade CSAM detection API combines both, and allows platforms to configure the balance between detection coverage and false positive rate based on their operational requirements.
1. Is there a genuine AI classifier, or just hash matching?
This is the foundational question, and it matters more now than it did a few years ago.
Hash matching (the technology underlying tools like Microsoft’s PhotoDNA and Meta’s PDQ) works by generating a fingerprint of previously identified CSAM and comparing new uploads against that database. It’s fast, reliable, and widely deployed. It is also, by design, blind to any image that hasn’t already been found, confirmed, and added to a hash list.
That limitation is increasingly consequential. According to NCMEC’s CyberTipline data, reports of AI-generated CSAM rose by more than 1,300% between 2023 and 2024, from roughly 4,700 to over 67,000. First-generation content, material produced by hands-on offenders and not yet reported anywhere, presents the same problem. A platform relying exclusively on hash matching has a systematic blind spot for the fastest-growing categories of offending material. For a deeper look at why hash matching falls short as a standalone approach, see our post Why Hash Matching Alone Can’t Protect Your Platform.
A genuine AI classifier works differently. It analyzes the visual content of an image directly, using pattern recognition trained on verified CSAM datasets, to assess whether previously unseen material is likely to be abusive. It can find content that has never been reported before.
The questions to ask any vendor: do you have a proprietary classifier, or are you licensing someone else’s technology? What datasets was it trained on? Has it been independently validated by law enforcement against real first-generation material, not just controlled lab datasets?
That last question has a specific answer in CaseScan’s case. The detection engine has been deployed and field-validated by more than 100 law enforcement units globally, including the FBI and Homeland Security Investigations. Wix, one of the world’s largest website-building and user-generated content platforms, uses CaseScan to scan content across its platform. DoubleVerify, the global media measurement and ad verification company, also uses CaseScan for content safety. The value of law enforcement field validation for a commercial platform buyer is that it provides independent corroboration that the classifier works against real-world offending material, not just controlled benchmarks. No lab test replicates that.
2. What is the recall rate, and how was it measured?
Recall (the true positive rate) is the percentage of actual CSAM the system correctly identifies. It’s the metric most platform teams want first, and it matters. But the number alone tells you very little without the methodology behind it.
Formal recall measurement happens in controlled lab conditions, because researchers are legally prohibited from possessing real CSAM for testing. Lab benchmarks are the industry standard, and any credible vendor should have them. The question is whether those lab-measured figures have been directionally corroborated in real field conditions, because lab datasets, however carefully curated, don’t fully capture the visual complexity and variety of real-world offending material.
The more important question for a platform buyer is not just “what’s your recall?” but “where on the precision-recall curve do you operate, and can I adjust that?”
Every detection system involves a tradeoff. Push recall up, catch more content, and false positives increase. Pull false positives down, and some genuine content gets missed at the image level. What separates a mature detection platform from a single-mode tool is whether it gives the customer meaningful control over that tradeoff, based on their risk tolerance and operational capacity.
CaseScan offers three operating profiles. The High Coverage profile achieves a 95.48% true positive rate with roughly 1 false alert per 76,000 images on general-image traffic, appropriate for platforms that want maximum detection coverage and have a moderation team to review flagged content. The Balanced profile delivers 85.93% recall with approximately 1 false alert per 1.9 million images. The Autonomous Maximum Precision profile achieves 69.34% recall with 1 false alert per 45 million images, a rate suited to fully automated enforcement without human review.
The choice between these profiles is a business decision, determined by your team’s capacity, your risk tolerance, and whether you’re running autonomous enforcement or a hybrid human-in-the-loop workflow. Ask whether those options exist before you get into contract discussions.
3. What does the false positive rate actually mean at your scale?
This is where the evaluation gets practical. A false positive rate of 0.001% sounds like a rounding error until you do the math at production volume.
A platform processing 10 million image uploads per day at a 0.001% false positive rate is generating 100 wrongly flagged pieces of content every single day. Those land in your moderation queue, trigger automated account actions, or both. For platforms with limited analyst capacity, that number can overwhelm a team faster than the actual CSAM problem does.
Run this calculation before any vendor call: take the stated false positive rate, multiply by your daily image volume, and ask whether your team can process the resulting review load. If the answer is no, you need a more precise operating profile or a different vendor.
The second dimension of this question is methodology. False positive rates can be measured in several ways that produce very different numbers. General-image traffic benchmarks, measured against production environments where the vast majority of content is legitimate, represent the most realistic predictor of real-world performance. Pre-flagged traffic benchmarks, where the dataset is deliberately concentrated with borderline and difficult content, will look significantly worse. Lab stress-test benchmarks using curated ambiguous imagery will look worse still.
Many vendors quote one number without specifying which environment it came from. CaseScan publishes all three. At the base detection threshold on general-image traffic, the false positive rate is approximately 1 in 680,000. Against pre-flagged high-risk traffic (queues already concentrated with difficult edge cases), it rises to roughly 1 in 4,500. Against lab stress-test conditions designed to be as difficult as possible, it’s 1 in 2,000. That layered transparency matters because it tells you what to expect across different content queues, not just a best-case headline.
The secondary validation layer in CaseScan’s architecture, called Context Analysis, adds a deep semantic understanding step that distinguishes genuine CSAM from complex benign scenarios: medical imagery, family photos, standard adult content. In its maximum precision mode, this layer reduces false alert volume by up to 98.5% compared to base classifier output alone. It’s what produces the 1 in 45 million figure for the Autonomous profile.
4. Does the vendor store the images it processes?
This question almost never comes up in a first vendor call. It should.
When a platform routes user-uploaded content through a third-party detection API, two meaningfully different architectures are possible. In one, the vendor processes the image entirely in memory and discards it immediately: no write to disk, no cache, no persistent copy anywhere on the vendor’s infrastructure. In the other, the vendor logs, temporarily stores, or retains flagged content as part of their processing pipeline.
The difference carries real legal and security weight. A vendor that writes processed images to disk creates a data retention footprint that may trigger obligations under GDPR, CCPA, or other applicable privacy regulations, increases the vendor’s attack surface, and creates a liability question your legal team will want to address before a contract is signed. The nature of the content being processed makes this more sensitive, not less.
CaseScan processes images entirely in volatile memory. Nothing is written to disk at any point. The stateless architecture was designed this way deliberately: it eliminates the storage attack surface, meets stringent data sovereignty requirements, and allows the system to scale horizontally without state management overhead.
Ask every vendor for a written, specific answer to this question: where does the image go during processing, and what is your data retention policy? If the answer is vague or comes with qualifications about “temporary” storage, ask what “temporary” means in hours and where those logs go.
5. What happens when traffic spikes?
Content moderation infrastructure tends to degrade at precisely the moment it’s needed most: during a viral event, a major product launch, or an unexpected surge. The architecture question is whether the vendor’s detection system is stateless and horizontally scalable, or whether it carries state that creates bottlenecks under load.
Latency is the user-facing dimension. If image scanning adds noticeable delay to an upload flow, users experience it as a performance problem, not a safety feature. Sub-second processing is the threshold a production deployment should expect.
The specifics to ask for: P95 and P99 latency, stated as end-to-end API response time including network ingestion. What daily volume has the system been tested against? How does scaling work during traffic spikes: is it pre-provisioned capacity, auto-scaling, or something else?
CaseScan’s API returns results in under one second at the 95th percentile and under two seconds at the 99th percentile. The stateless cloud-native architecture scales to billions of files per day, with horizontal scaling triggered automatically and no latency degradation during traffic spikes. These are verified performance specifications, not targets.
6. Per-image recall understates real-world detection: ask about account-level probability
All detection benchmarks are stated per image, because that’s the unit of technical measurement. But enforcement decisions at real platforms happen at the account level, and the statistical picture looks quite different there.
An account uploading CSAM to a platform typically does so across multiple files. If individual images are caught at a 70% probability, the likelihood of detecting at least one image across five offending uploads from the same account is approximately 99.8%, under a reasonable independence assumption. The per-image figure implies meaningful exposure. The account-level figure tells a different story.
This matters most for platforms evaluating whether fully autonomous enforcement is viable. The Autonomous Maximum Precision profile has a per-image recall of 69.34%, which sounds like it leaves meaningful content undetected. At the account level, across a typical pattern of multi-file offending behavior, the detection probability is near-total. That’s the operational reality the per-image figure doesn’t show.
Ask any vendor to explain this distinction. A vendor who hasn’t thought through account-level detection probability is missing a meaningful part of the operational picture. For autonomous enforcement decisions specifically, it’s the more relevant metric.
Six questions before the next vendor call
Go into any vendor conversation with these six questions written down. Push for specific numbers, not characterisations.
- Do you have a proprietary AI classifier trained on verified CSAM data, and has it been independently field-validated by law enforcement against real first-generation material?
- What is your recall rate, was it lab-measured or field-corroborated, and do customers get to choose where on the precision-recall curve to operate?
- What is your false positive rate, and against which traffic type was it measured? General-image production traffic, pre-flagged content queues, and lab stress-test datasets produce very different numbers. Ask for all three.
- Do you write any processed images to disk at any point? Get a written answer to this before the contract stage.
- What are your P95 and P99 API response times, stated as end-to-end including network ingestion? How does the system scale during unexpected traffic spikes?
- What is the account-level detection probability for a typical multi-file offender, not just the per-image recall rate?
A vendor who answers all six in writing, with specific figures attached, is worth a serious conversation. Vague answers to more than one of them is a signal to push harder before the evaluation goes further.
CaseScan is purpose-built for platforms that need detection they can stake their compliance posture on. The same engine trusted by the FBI, HSI, and more than 100 law enforcement units worldwide runs behind the platform API used by Wix, DoubleVerify, and others to protect their users and their brand at scale.
If you’d like to see how CaseScan answers each of these questions, with the actual documentation, book an intro call here.
Frequently asked questions
What is a CSAM detection API? A CSAM detection API is a programmable interface that allows platforms to automatically scan images and video for child sexual abuse material at the point of upload or in stored content. It processes media files and returns a classification result (along with a confidence score) that platforms use to trigger enforcement actions such as blocking, flagging for review, or reporting to NCMEC. Enterprise-grade APIs operate in real time at scale, with no storage of the content they scan.
What is the difference between hash matching and AI-based CSAM detection? Hash matching identifies content by comparing a file’s digital fingerprint against a database of previously confirmed CSAM. It’s highly accurate for known material but cannot detect first-generation or AI-generated content that hasn’t been reported before. AI-based classification analyzes the visual content of an image directly, allowing it to identify novel material regardless of whether it has ever been seen or reported. Production platforms should use both in combination.
How is CSAM detection recall measured? Recall (true positive rate) is formally measured in controlled lab conditions using curated datasets of confirmed CSAM, because legal restrictions prevent researchers from possessing real-world material for testing. Any credible vendor should publish lab-measured recall across their operating profiles. Additionally, field corroboration by law enforcement, where investigators use the tool on real cases and report results, provides directional validation that lab benchmarks translate to real-world performance.
What false positive rate should a platform expect from a CSAM detection API? It depends on the operating profile and traffic type. On general-image production traffic (the most realistic benchmark), a well-configured AI classifier with a secondary validation layer can achieve false positive rates as low as 1 in 45 million images. Platforms should ask vendors to specify which traffic environment their stated false positive rate was measured against, since pre-flagged content queues and lab stress-test conditions can make the same system appear far less precise.
Do CSAM detection APIs store the images they process? Not all of them. Some vendors retain flagged images or write processed content to disk as part of their pipeline, which creates data retention obligations and security exposure. CaseScan processes all content in volatile memory with a zero media retention policy, meaning nothing is written to disk at any point. Platforms should require a written, specific answer to this question before signing any vendor contract.
Are platforms legally required to scan for CSAM? Under US federal law (18 U.S.C. § 2258A), electronic service providers are required to report CSAM to NCMEC’s CyberTipline when they become aware of it on their platforms. The 2024 REPORT Act expanded these obligations to cover online enticement and child sex trafficking. Failure to report known CSAM carries fines of up to $150,000 for a first offense and $300,000 for subsequent violations. Beyond federal requirements, state attorneys general are increasingly pursuing platform-level accountability, as demonstrated by the $375 million verdict against Meta in New Mexico in March 2026.
What is account-level CSAM detection probability? Per-image recall rates describe the likelihood of detecting a single offending image. Account-level detection probability describes the likelihood of catching at least one image when an account uploads multiple offending files, which is the more operationally relevant figure for enforcement decisions. Under a reasonable independence assumption, a 70% per-image recall rate translates to approximately 99.8% account-level detection probability across five offending uploads from the same account.
How do CSAM detection APIs handle AI-generated CSAM? Hash matching cannot detect AI-generated CSAM because it has no previously confirmed fingerprint in any database. An AI classifier, trained on verified datasets and updated to account for synthetic content characteristics, can assess whether AI-generated imagery depicts abuse based on visual content analysis alone. NCMEC reported 67,000 CyberTipline filings related to generative AI in 2024, a 1,325% increase from 2023, which makes AI classifier capability a non-optional evaluation criterion for any platform operating at scale.
CaseScan is built by Netspark. All performance figures in this post are drawn from CaseScan’s internal benchmarking methodology, validated against multi-million-image production samples. False positive benchmarks are based on general-image traffic unless otherwise stated. Law enforcement usage cited with permission.
Ready to evaluate your platform’s CSAM detection capabilities? Learn more about CaseScan for UGC platforms or contact our team to schedule a demo.
