Send us a result your model already returned. A decision model approves what it's sure of in about 100 ms. People check the rest. Their corrections train the machine, so next month fewer items need a person.
Onboarding teams in small batches, starting now.
Each item starts on the lowest rung that is calibrated for your task. It climbs one rung whenever confidence falls under your threshold.
Every correction comes back to you by webhook and becomes a golden-set candidate.
No new labeling tool, no model switch, no data migration. Validate works on output from any model.
from annotators import Annotators client = Annotators() # reads ANNOTATORS_API_KEY tags = my_model.tag(doc) # the call you already make client.validate( "doc-tagging", input={"title": doc.title, "content": doc.text}, output=tags.fields, 1 source={"model": "my-model", "confidence": tags.p}, mode="auto", 2 workforce="vetted", callback_url=WEBHOOK_URL, 3 ) # later, at your webhook { "type": "validation.completed", "data": { "id": "val_7Q2M4K", "verdict": "corrected", "corrected_output": {"communication_type": "Press Release"}, "agreement": 1.0, "resolved_by": "human" } }
The default. The machine checks every field; people review only what it doubts, and they see which fields those were.
Cheapest. Most items never reach a person.
Every item goes to people, with as many reviewers per item as you ask for. Disagreements add a reviewer, up to five.
For high-stakes output and building a first golden set.
A fixed share of a stream, 5% by default, goes to people and the rest passes through. You get a running accuracy number for any model, including yours.
Audit mode for production models.
Every call is an audit or a gate. By default your result has already shipped and a correction arrives later. Set hold and your call waits for the verdict; held items are reviewed first.
Sensitive work never goes to the open network. Your own staff can work the same queue on your private data.
| Workforce | Who reviews | Data exposure |
|---|---|---|
| open | Qualified annotators.org members | Non-sensitive data only |
| vetted | Members with a strong check record and a domain test: finance, legal, multilingual | Under our NDA |
| private | Your own staff, contractors or BPO, invited by email | Never leaves your team |
| dedicated Later | Named vetted annotators assigned only to you | Under your NDA |
The number to watch is cost per validation. It should fall every month you use us.
Two or more reviewers agree the model was wrong.
Unanimous corrections queue up as golden-set candidates.
Approved items become check items for reviewers and training data for your task.
More items resolve on the cheap rung. Fewer reach people.
Send a message, headline, ad or document and get responses from real people: sentiment scores, choices and verbatims, with speeders and AI answers filtered out. Run it next to your synthetic audience and see how close it really is.
Free tier, monthly plans, and pay as you go. We're onboarding teams in small batches, starting now.
Coming from Mechanical Turk, Ground Truth or A2I? Export your results before 30 Oct and tell us what you run. We'll help you move it.
Audit a sample of a production model, or gate what ships. Corrections come back by webhook.
A published pay floor, paid from the first item, and a score you can always see.