Content review
Approve, flag or remove with a probability for every option.
The problem
Moderation needs a decision on every post, fast, against a policy that changes. Free-text model output has to be parsed and validated, and a single wrong parse lets something through.
What Needle replaces
The moderation prompt and the code that parses its answer.
What you do with the answer
- Remove automatically only above a high threshold; send the flagged middle to reviewers, ordered by probability.
- Split the decision: a Noul for scam likelihood and a Score for toxicity, so different teams own different thresholds.
- Declare your policy's options as the criteria, so the model can only return actions you actually take.
{
"model": "needle-1",
"state": "Forum post: Selling 2 concert tickets, DM me.
Payment by gift card only, no refunds!!",
"questions": {
"action": {"type": "choice",
"criteria": {"approve": null, "flag": null,
"remove": null}},
"scam": {"type": "noul",
"instructions": "Is this likely a scam?"},
"toxicity": {"type": "score",
"criteria": ["none", "mild", "severe"]}
}
}200 answers 1 forward pass
action choice → flag
approve 0.08
flag 0.63
remove 0.29
scam noul → 0.86
P(true) 0.86
toxicity score → 0.05
none 0.95
mild 0.05
severe 0.00
0 output tokens Illustrative response