Design a spam / content moderation classifier

The problem

A platform receives half a billion posts, comments, images and videos a day. A small fraction is spam, scams, harassment, threats or other content its policies forbid. Leave it up and users get hurt and leave; take down legitimate posts and users feel silenced and leave too. Both errors are costly, and the cost differs by policy: a spam comment wrongly left up is an annoyance, a credible threat left up is a safety incident.

The second difficulty is that the "positives" fight back. Spammers test which words get blocked and change them within hours. Any fixed model decays.

Think of airport security. Every bag goes through the X-ray machine (fast, cheap, automatic). Bags the machine is unsure about go to a second screening with better equipment. The few that are still unclear are opened by a trained officer. And the rules are stricter for some items than others. Content moderation is the same cascade: a fast classifier on everything, a more expensive model on the uncertain, and people for what remains, with a different threshold for each policy.

Requirements

Functional

• Classify every new item (text, image, video, links, account behaviour) against each policy: spam, scams, harassment, hate, violent threats, adult content, and so on.
• For each policy decide one of: allow, remove, reduce distribution, add a warning, or send to human review.
• Give users an appeal path and feed appeal outcomes back as labels.
• Explain each action with the policy it violated, for the user and for audits.

Non-functional

• Latency: under 100 ms for the fast tier before an item is visible; the LLM tier and humans can act after posting (seconds and hours), with distribution held back while they decide.
• Precision targets per policy, for example 99 percent for automatic spam removal, and recall targets for severe harms, for example 95 percent for violent threats, with a human looking at everything in between.
• Cost: the LLM tier and human review are budgets, not afterthoughts.
• Adaptivity: new evasion patterns detected and handled within a day.

Out of scope

• Writing the policies themselves (a policy and legal decision, but the system must encode them precisely).
• Account-level fraud and payment abuse ( Design a fraud detection system ).
• Legal reporting obligations and law-enforcement workflows.

Estimates

Assumed numbers for a large platform, not any company's figures:

# Back-of-envelope sizing for a content moderation system. Every input is an assumption; change them and re-run.
posts_per_day = 500_000_000      # text posts, comments, images and short videos
prevalence = 0.005               # share that violates some policy
fast_ms = 5                      # fast classifier latency per item
to_llm = 0.03                    # share of items the fast tier sends to the LLM tier
llm_tokens = 800                 # prompt (policy + item) and answer tokens per LLM call
llm_dollars_per_m_tokens = 0.20  # assumed price of a small hosted model
to_human = 0.002                 # share of items that reach a human reviewer
reviews_per_hour = 60            # one decision a minute, including context
hours_per_shift = 6              # productive review hours per reviewer per day

print(f"items per second (avg)     {posts_per_day / 86_400:,.0f}")
print(f"violating items per day    {posts_per_day * prevalence / 1e6:,.1f} million")
print(f"fast tier CPU-seconds/day  {posts_per_day * fast_ms / 1000 / 1e6:,.1f} million (about {posts_per_day * fast_ms / 1000 / 86_400:,.0f} cores busy)")
llm_calls = posts_per_day * to_llm
print(f"LLM calls per day          {llm_calls / 1e6:,.0f} million, {llm_calls * llm_tokens / 1e9:,.0f} billion tokens,"
      f" about {llm_calls * llm_tokens / 1e6 * llm_dollars_per_m_tokens:,.0f} dollars a day")
print(f"LLM on every item instead  about {posts_per_day * llm_tokens / 1e6 * llm_dollars_per_m_tokens:,.0f} dollars a day")
reviews = posts_per_day * to_human
reviewers = reviews / (reviews_per_hour * hours_per_shift)
print(f"human reviews per day      {reviews / 1e6:,.1f} million, about {reviewers:,.0f} reviewers")
print(f"if humans saw 1% of items  about {posts_per_day * 0.01 / (reviews_per_hour * hours_per_shift):,.0f} reviewers")

items per second (avg)     5,787
violating items per day    2.5 million
fast tier CPU-seconds/day  2.5 million (about 29 cores busy)
LLM calls per day          15 million, 12 billion tokens, about 2,400 dollars a day
LLM on every item instead  about 80,000 dollars a day
human reviews per day      1.0 million, about 2,778 reviewers
if humans saw 1% of items  about 13,889 reviewers

• Volume: 500 million items a day is about 5,800 per second on average. At 0.5 percent prevalence, 2.5 million violate a policy every day.
• The fast tier is cheap: at 5 ms of CPU per item, about 29 cores are busy around the clock.
• The LLM tier is affordable only on a slice: at 800 tokens per call and an assumed 0.20 dollars per million tokens, running it on the 3 percent of items the fast tier is unsure about costs about 2,400 dollars a day; running it on everything would cost about 80,000.
• Humans are the scarcest resource: 0.2 percent of items is 1 million reviews a day, about 2,800 reviewers at 360 decisions each. Sending 1 percent would need nearly 14,000.

Data and labels

Labels come from three places, each with its own bias:

• User reports: plentiful but noisy, and biased towards what users see and dislike (popular posts get reported, and so do people with unpopular opinions).
• Reviewer decisions: the gold standard, but expensive and only available for items the system already chose to send for review. A model trained only on reviewed items learns the old model's blind spots.
• Random samples labelled by experts: the only unbiased estimate of prevalence and recall. Keep a small random stream going forever.

Active learningChoosing which unlabelled items to send for labelling so that each label teaches the model as much as possible, for example items the model is most uncertain about. picks what to label. OpenAI's moderation paper (Markov et al., 2023) describes a three-way mix: a random sample to stay true to production traffic, a sample of items scoring above a threshold to find likely violations, and items whose score is closest to 0.5, where the model is least sure. It reports that active learning captured up to 22 times more undesired samples for rare categories, and that rare categories are very rare: in sampled user prompts, 0.04 percent involved self-harm and 0.017 percent hateful content involving threats.

Labelling instructions are part of the model. The same paper found that imprecise instructions force annotators to rely on their own judgment and produce inconsistent labels, so it used regular calibration sessions with annotators. It also caught mislabels by training two models on two halves of the data, scoring the other half with each, and auditing disagreements; if more than 30 percent of a sample of flagged items were wrong, all flagged items were relabelled.

Features: multi-modal signals

A violation is rarely visible in one field. Strong systems combine:

• Text: a fine-tuned transformer embedding of the post, plus character n-grams that survive misspellings ("fr3e m0ney").
• Images and video: an image embedding, OCR of text inside images (a classic way to hide spam from text models), perceptual hashes matched against known violating media.
• Links: domain age and reputation, redirect chains, whether the landing page matches the text.
• Account and behaviour: account age, posting rate, how many near-identical posts the account or its network made in the last hour, previous strikes.
• Graph and context: who is replying to whom, whether the target is a frequent target of the same senders, which group or thread the item is in.

A multi-modal modelA model that takes several input types (text, image, metadata) and fuses their embeddings into one prediction, so evidence in one modality can be weighed with the others. fuses these, typically by concatenating per-modality embeddings with behavioural features before a small classifier head per policy. Behaviour features are especially valuable against spam: the text can be rewritten endlessly, but sending the same message to 5,000 people in an hour is hard to hide.

Model choice and training

One model per policy, or one model with many heads

Each policy needs its own threshold, metric and owner, but they share most of the signal. A common design is one shared multi-modal encoder with one output head per policy (multi-label, since a post can be both spam and a scam), trained jointly and calibrated per head.

Precision vs recall is a policy decision

For each policy the team chooses an operating point, using the costs of each error:

• Spam: high precision for automatic removal (removing a real message is worse than missing one spam), recall made up by rate limits and user controls.
• Violent threats: high recall, accepting more human review, because a miss is a safety incident.
• Borderline content: reduce distribution instead of removing, a softer action with a lower precision bar.

The expected cost of a threshold $t$ is $\text{FP}(t)\cdot c_{FP}+\text{FN}(t)\cdot c_{FN}$, and the right $t$ depends on the ratio $c_{FN}/c_{FP}$ for that policy. One model, several thresholds.

The LLM tier

An LLM classifierA large language model prompted or fine-tuned to read content together with a written policy and output a label, often used as an expensive second-stage classifier. reads the item together with the policy text and returns a label. It understands context, sarcasm and new phrasings far better than a small classifier, and it can follow a new or changed policy written in plain language without retraining. Meta's Llama Guard paper (Inan et al., 2023) fine-tuned a 7-billion-parameter model on 13,997 labelled prompts and responses; the policy categories are given in the input, the model outputs "safe" or "unsafe" plus the violated categories, and the classifier score is read off the probability of the first token. Because the taxonomy is part of the input, the same model could be adapted to different taxonomies by zero-shot or few-shot prompting.

The cost is latency and money, so the LLM sits behind the fast tier: it sees only the uncertain middle, and its own uncertain middle goes to humans. Treat it as a model like any other: version its prompt, evaluate it on a labelled set per policy, calibrate its scores, and monitor it.

High-level design

Deep dives

Deep dive 1: Thresholds and the cascade

The sample below simulates 400,000 items, 1 percent violating. The fast classifier's score separates the classes reasonably, the LLM tier better (and partly agrees with the fast tier, since both see the same text), and human reviewers are right 97 percent of the time.

# A three-tier moderation cascade on synthetic items: fast classifier, LLM classifier, human review.
import numpy as np

rng = np.random.default_rng(23)
n, prev = 400_000, 0.01
y = rng.random(n) < prev                                   # 1 percent of items violate the policy
fast = rng.normal(0, 1, n) + 2.6 * y                       # fast model: cheap, decent separation
llm = 0.4 * fast + rng.normal(0, 1, n) + 3.4 * y           # LLM tier: sees more context, partly agrees with fast
human = np.where(rng.random(n) < 0.97, y, ~y)              # reviewers are right 97 percent of the time

def cascade(f, l, lo, hi, l_lo, l_hi):
    act = f >= hi                                          # tier 1 acts alone on clear cases
    mid = (f > lo) & (f < hi)                              # the uncertain middle goes to the LLM
    act |= mid & (l >= l_hi)
    to_h = mid & (l > l_lo) & (l < l_hi)                   # the LLM's own uncertain middle goes to people
    act = (act & ~to_h) | (to_h & human)
    tp = (act & y).sum()
    return tp / max(act.sum(), 1), tp / y.sum(), mid.mean(), to_h.mean()

SPAM = (1.5, 4.0, 3.0, 4.5)                                # fast lo, fast hi, LLM lo, LLM hi
print(f"{n:,} items, {y.sum():,} violating")
for name, hi in [("fast tier alone, act at 3.0", 3.0), ("fast tier alone, act at 2.0", 2.0)]:
    act = fast >= hi
    print(f"  {name:32s} precision {(act & y).sum() / act.sum():.3f}  recall {(act & y).sum() / y.sum():.3f}"
          f"  removed {act.mean():.2%}")
for name, args in [("spam: precision first", SPAM), ("threats: recall first", (0.5, 4.0, 2.0, 4.5))]:
    p, r, m, h = cascade(fast, llm, *args)
    print(f"  {name:32s} precision {p:.3f}  recall {r:.3f}  to LLM {m:.2%}  to humans {h:.3%}")

# Adversaries adapt: 40 percent of violators obfuscate their text, which the fast model mostly misses.
obf = y & (rng.random(n) < 0.4)
fast2 = fast - 1.8 * obf
p, r, m, h = cascade(fast2, 0.4 * fast2 + (llm - 0.4 * fast), *SPAM)
print(f"after obfuscation                 precision {p:.3f}  recall {r:.3f}  (only {np.mean(fast2[obf] > SPAM[0]):.0%} of obfuscated items get past tier 1)")
fast3 = fast2 + 1.4 * obf                                  # retrained on reviewer labels of the new pattern
p, r, m, h = cascade(fast3, 0.4 * fast3 + (llm - 0.4 * fast), *SPAM)
print(f"after retraining on new labels    precision {p:.3f}  recall {r:.3f}")

400,000 items, 3,996 violating
  fast tier alone, act at 3.0      precision 0.724  recall 0.356  removed 0.49%
  fast tier alone, act at 2.0      precision 0.244  recall 0.725  removed 2.97%
  spam: precision first            precision 0.991  recall 0.790  to LLM 7.33%  to humans 0.421%
  threats: recall first            precision 0.936  recall 0.959  to LLM 31.42%  to humans 2.485%
after obfuscation                 precision 0.987  recall 0.558  (only 24% of obfuscated items get past tier 1)
after retraining on new labels    precision 0.990  recall 0.747

• One threshold cannot serve both goals. Acting on fast scores of 3.0 or more gives 72.4 percent precision and only 35.6 percent recall. Lowering it to 2.0 lifts recall to 72.5 percent but precision collapses to 24.4 percent: three out of four removals would be wrong, because 99 percent of items are fine.
• The cascade gets both. With the spam settings (allow below 1.5, remove at 4.0, LLM in between), precision is 99.1 percent and recall 79.0 percent, with 7.33 percent of items going to the LLM and 0.421 percent to people.
• A recall-first policy pays in compute. Opening the band down to 0.5 for threats reaches 95.9 percent recall at 93.6 percent precision, but 31.42 percent of items now go to the LLM and 2.485 percent to humans, about six times as many reviews.

Deep dive 2: Adversarial users

Spammers probe the system: post a variant, see whether it survives, repeat. They swap letters for digits, put text inside images, add invisible characters, split a message across posts, or create fresh accounts and domains. In the second half of the sample, 40 percent of violators obfuscate their text so the fast tier scores them much lower. Recall falls from 79.0 to 55.8 percent, because only 24 percent of obfuscated items now get past tier 1 to be looked at. After retraining on reviewer labels of the new pattern, recall recovers to 74.7 percent.

Defences that work in practice:

• Signals that are expensive to fake: account age, sending patterns, network structure, payment history. Text is cheap to change; behaviour is not.
• Fast loops from review to model: new patterns found by reviewers become training data within hours; rules or hash lists bridge the gap until the model catches up.
• Normalization before classification: undo common obfuscations (homoglyphs, zero-width characters, leetspeak), OCR images, follow redirects.
• Do not reveal the boundary: vary enforcement timing, avoid telling a suspected spammer exactly which word triggered the block, rate-limit probing.
• Random sampling: the only way to see what the classifier is missing is to label items it did not flag.

Deep dive 3: Human-in-the-loop review

Human review is a system to design, not a bin to throw leftovers in:

• Queue by expected harm, not arrival time: severity of the suspected policy times the reach of the item (views in the next hour).
• Hold, then decide: uncertain items get reduced distribution until reviewed, so slow review does not mean wide exposure.
• Measure reviewers like models: agreement with expert auditors, per policy. With rare violations, plain accuracy is misleading because almost everything is fine; OpenAI's paper audits a sample of items labelled undesired and a sample with high model scores, and computes F1 per category instead.
• Protect reviewers: exposure limits, blurred previews, rotation and support. Every item a model handles correctly is one a person does not have to see.

Try it

1. Set both thresholds to 3.0 so there is no middle band: the fast tier alone removes with 75.9 percent precision but catches only 34.9 percent of violations, leaving 639 of them up. Now set both to 2.0: recall rises to 71.8 percent, but 2,254 fine items are removed.
2. Go back to allow below 1.50 and remove at 4.00 with the LLM tier on: precision 99.3 percent, recall 81.6 percent, with 7.35 percent of items going to the LLM and 0.41 percent to humans, about 5,700 reviewers at full scale. Turn the LLM tier off: the same band now sends 7.35 percent of items to humans, over 100,000 reviewers, and precision drops to 81.8 percent.
3. Turn obfuscation on: recall falls to 58.1 percent, with almost no change in precision, so the precision dashboard would not notice. Lower the allow threshold to 0.50: recall climbs back to 75.2 percent, but now 31.08 percent of items go to the LLM. Widening the band buys time; retraining on the new pattern fixes the cause.

Going deeper

Evaluation

• Per policy, per language, per surface. A single overall metric hides a model that works in English and fails in a language with fewer labels.
• Prevalence: the share of views (not items) that violated a policy, estimated from a random sample labelled by experts. It is the closest thing to a north-star metric, because it measures what users actually saw.
• Precision of actions from audits of removed items, and appeal overturn rates.
• Recall only from random or stratified samples, never from reports alone.
• Time to action for severe policies, weighted by views before removal.

Monitoring

Watch the score distribution and action rate per policy every hour (a sudden drop in removals is as alarming as a spike), the LLM tier's agreement with human reviewers, queue sizes and age, appeal overturn rates, and prevalence estimates by language. Track new clusters of near-identical content that the model does not flag: they are often the next evasion pattern.

Bottlenecks and failure modes

Trade-offs we made

Extensions

• Account-level models that aggregate many items, catching coordinated campaigns that look harmless post by post.
• Proactive detection of near-duplicates of removed items across the platform, using embeddings and perceptual hashes.
• Policy-as-prompt: when a policy changes, update the LLM tier's prompt the same day and use its labels to retrain the fast tier.
• User-level controls: let users filter more or less of a category for themselves.

In the wild

Check yourself

Recap