STEP 01
WHAT WE CODIFY
Define the rubric
The label schema, the guidance, and the examples are versioned. Every annotator works against the same definition.
→HUMAN DATA · ANNOTATION
Label the data. Sign the proof.
Every label leaves with the rubric it was graded against, the reviewer who cleared it, and a signed chain of who created it and under what rights. A dataset that survives an audit — and runs on infrastructure you keep.
RUBRIC
Versioned
The label schema is reviewable and the same on every batch.
REVIEW
Attached
The reviewer who cleared the label follows it out of the workspace.
DATASET
Signed
Who created it, who reviewed it, under what rights — sealed with the export.
HOW IT WORKS
Three steps. One signed dataset.
Define the rubric. Label with review. Sign the dataset before it leaves.
STEP 02
WHAT WE CHECK
Label with review
Annotation and review run in the same workspace. IAA, reopen rate, and reviewer notes stay attached to the batch.
→STEP 03
WHAT WE SIGN
Sign the dataset
Export only when the gate clears. Rubric version, reviewer record, and verdict travel with the dataset.
WHO DOES THE LABELING
Your experts, ours, or both.
Expert pay is quote-scoped because frontier work needs real domain experts. Run yours, ours, or a mix — every reviewer is identity-verified and calibration-tested before they touch a row.
YOUR EXPERTS
Bring your own bench
Your coders, clinicians, lawyers, and analysts work in the same governed workspace. Their reviews carry the same signed chain as everyone else's.
OUR SPECIALISTS
Vetted and routed
Identity-verified, credential-checked, calibration-tested specialists routed to the exact rows the rubric flagged. Frontier work needs experts, not generalist crowds.
BOTH
One review record
Mix your bench with ours on the same program. Every label — whoever made it — leaves with the reviewer who cleared it and the rights it was created under.
For real-work programs, experts recreate professional tasks from clean-room scenarios — no employer data, with a signed no-employer-data attestation. The trade-secret and PII exposure stays out of your dataset.
PROVENANCE & CONSENT
A dataset that survives an audit.
78% of teams cannot validate their training data and 77% cannot trace where it came from. The EU AI Act provenance provisions enforce in August 2026. Every label here carries the chain that answers them.
01 · ORIGIN
Who created every datapoint
Each label carries the identity-verified person who made it — not an anonymous contractor onboarded like a consumer. The chain holds under audit.
02 · REVIEW
Who reviewed it, and when
The reviewer who cleared the row, the time they cleared it, and the rubric version they cleared it against travel inside the dataset.
03 · RIGHTS
Under what rights it was created
Signed consent attached at the datapoint, not assumed at the contract. You can show what you are allowed to train on, row by row.
04 · RESIDENCE
Where it lives, and only there
Your data runs on your infrastructure, never pooled in one vendor's data center. A competitor lost four terabytes — including who its workers were.
INDEPENDENT
A neutral second source
Founder-led, and not a data vendor aligned with one of the labs it serves. The kind of supply you can defend under audit.
REGULATED & CLINICAL
Built for the reading room
PII and PHI handling, on-prem or VPC deployment, data residency controls, and audit logging on every action.
THE EXPORT GATE
Weak data does not leave.
IAA below the threshold, reopen rate above the ceiling, coverage or gold-set agreement below the floor — any one check holds the release, with the reason attached. A workspace, mid-program.
Release status: review-scoped. This is a checked-in metrics snapshot; the provider-backed workspace is scoped only after provider readiness and review evidence are accepted. Pricing remains quote-scoped.
IAA MEDIAN · 30D
0.84
Target 0.75 · trend up
REOPEN RATE · 30D
1.8%
Ceiling 2.0% · trend down
EXPORT GATE
26/30
3 blocked · 1 in review
QUEUE DEPTH
14
6 auto-escalated · 38m median wait
QUALITY READOUT · LAST 30 DAYS
MODALITIES
One workflow for every modality.
Image, video, audio, text, structured, 3D point cloud, biosignal — seven modalities, one quality path. Masks, timelines, waveforms, cuboids, and spans all attach to the same review record.
01 · IMAGE
Pixel-mask with SAM2 assist
Bounding boxes, polygons, and pixel-mask segmentation with SAM2 assist. Every stroke and every correction stays on the same review record.
02 · VIDEO
Action recognition tracks
Frame-by-frame tracking with timelines and action-recognition segments that reviewers can scrub together. Spans survive review.
03 · AUDIO & SPEECH
Threaded voice comments
Voice comments on segments, diarization, transcripts, and biosignal overlays kept together. The reviewer's voice note lives on the row that earned it.
04 · TEXT & NLP
Spans with the policy attached
Named entity recognition, sentiment, classification, span annotation, and multilingual prompts in one governed workflow.
05 · STRUCTURED
Bulk-edit + taxonomy editor
Hierarchical taxonomies, metadata validation, and bulk edits for tables and schema-driven labels. One source of truth for the schema.
06 · 3D POINT CLOUD
Calibration-aware cuboids
LiDAR and depth-sensor annotation with cuboids, measurement tools, and calibration evidence that travels with the sensor frame.
THE WORKSPACE
Four surfaces. One annotation workflow.
The Studio is where the work happens. Review is where it gets cleared. The Quality Hub is where it gets measured. Datasets are where it leaves.
STUDIO
Annotation Studio
Pull source data from your cloud buckets, pre-label with model assist, then work across image, video, text, audio, and LiDAR without relearning the workflow. Autosave keeps work intact when a connection drops.
REVIEW
Collaborative adjudication
Disagreements turn into clear decisions with shared context. Voice and video comments stay attached to the task, not scattered in side channels.
QUALITY
Quality Control Hub
IAA, annotator drift, gold-set gaps, and the export gate sit in one hub before data leaves the workflow. Threshold rules can escalate when metrics drift.
DATASETS
Projects, datasets, exports
Projects, datasets, queue routing, and exports stay tied to the same review workflow. Dataset management and export flows share one source of truth.
COVERAGE READOUT
The rubric, the review, the verdict.
Every batch shows where the rubric was hit, where review caught the disagreement, and where policy held the line. One readout, three readings.
BATCH READOUT · LAST 30 DAYS
WHAT COMES OUT
What your dataset leaves with.
Every batch leaves something the training team can act on — and something the next reviewer can read.
01
Rubric versions
The label schema and guidance the batch was annotated against, pinned and immutable.
↳ ARTIFACT
02
Labeled batches
Annotations tied to the rubric version, the annotator, and the time they were made.
↳ ARTIFACT
03
Review notes
The disagreement, the reasoning, and the resolution stay with the row that triggered them.
↳ ARTIFACT
04
Signed datasets
Export only after the gate clears. Rubric version and verdict travel inside the package.
↳ ARTIFACT
05
IAA reports
Inter-annotator agreement, reopen rate, and pass rate for every batch — readable, not buried.
↳ ARTIFACT
EXPORT
Exports leave in the format the training stack expects.
Seven export targets can be scoped during project setup. Every format carries the manifest with it — rubric version, reviewer coverage, gate state, and the checksum on the dataset.
COCO
Instance + keypoint + panoptic JSON for image and video workflows.
YOLO
Bounding-box TXT files per image, class-index manifest included.
YOLO-seg
Polygon + mask coordinates for YOLO segmentation training.
VOC
Pascal VOC XML per image for long-tail legacy pipelines.
LabelMe
LabelMe JSON with shapes, groups, and flags preserved.
JSONL
JSON Lines for streaming and incremental dataset updates.
Parquet
Columnar dataset with schema enforced by the taxonomy editor.
MANIFEST
Rubric version
the schema the batch was graded against
MANIFEST
Reviewer coverage
who cleared what, and the gate state
MANIFEST
Checksum
the training side verifies what it received
QUALITY NOTE · ON THE RECORD
“Adjudication moved from spreadsheets to one review record. IAA is visible before export, and the quality lead can reopen a case without losing context.”
DIFFERENTIATORS · DAY ONE
What teams notice on day one.
Six things change the first time the whole workflow runs. None of them are about labelling speed — they are about the record the labels leave behind.
01 · ADJUDICATION
Disputes leave email
Disputed labels go through adjudication instead of spreadsheets and side threads. Every disagreement turns into a row with a resolution.
02 · AGREEMENT
IAA, calculated continuously
Inter-annotator agreement is not a quarterly report. It runs against every batch, with reviewer-level and project-level views.
03 · REOPEN RATE
Reopens tracked per annotator
Reopen rate per annotator and per project with service-target alerts. A drift in either one can route to review before the dataset leaves.
04 · GATE
Export blocked when weak
Four checks can block a release when the dataset is weak. Gate state and the evidence stay on the job — not on a dashboard nobody reads.
05 · AUDIT
One trail per label
Complete audit trail for every label decision. Who labelled it, who reviewed it, what changed, and why the cleared dataset was allowed to leave.
06 · DURABILITY
Offline does not lose work
Work stays intact when the connection drops. Robotics sessions captured in the field arrive attached to the same clip record once the network returns.
OFFLINE · DURABILITY
The connection drops. The work does not.
The workspace keeps running when the network goes away. Mask edits, span boundaries, and reviewer notes queue locally and reconcile the moment you reconnect.
Robotics teams capturing in the field, clinical reviewers in a basement reading room, audio teams in a studio booth — none of them lose work. Two reviewers editing the same span merge cleanly, with the resolution kept.
REPLAY TIMELINE · EVERY EDIT RECONCILED
00:00
01
Connected
Autosave committed
00:17
02
Offline
29 mask edits queued locally
00:46
03
Conflict
Two reviewers edited the same span
01:04
04
Merged
Edits reconciled · resolution kept
QUESTIONS · BEFORE YOU SHIP
What teams ask before the dataset leaves.
Six questions the program lead asks before they sign the export. The same six come up across labs, programs, and regulated workflows.
01 · ON THE RUBRIC
What happens when the rubric changes mid-program?
Rubrics are versioned. A change creates a new version, and the program lead approves the move. Existing batches stay on the version they were labelled against — the export carries the version with it.
02 · ON ADJUDICATION
Who decides when annotators disagree?
Adjudication is a defined role with its own queue. Disagreements flow into a single record with both labels, both reasons, and the resolver's call. The decision lives with the row.
03 · ON IAA
How is inter-annotator agreement measured?
Per batch and rolling 30-day, per annotator and per project. The Quality Hub surfaces the trend, the drift, and the threshold against the gate. Reopens and IAA share one source of truth.
04 · ON THE GATE
What blocks an export?
Four checks. IAA below the threshold. Reopen rate above the ceiling. Coverage below the required percentage. Gold-set agreement below the floor. Any one of them blocks the release with the reason attached.
05 · ON DATA
Where does the training stack pick up the dataset?
Through the export manifest. Export formats are scoped during project setup. The manifest carries the rubric version, reviewer coverage, gate state, and the dataset checksum so the training side can verify what it received.
06 · ON MODALITIES
What about modalities we have not labelled before?
Image, video, audio, text, structured, 3D point cloud, and biosignal are all on the same workflow. New modalities are added behind the same review, quality, and export gates as the existing ones.
WHAT CHANGES
What changes when annotation runs on one record.
Annotation breaks when the label, the disagreement, the quality reading, and the export gate live in different systems. The labels arrive. The disputes get emailed. The quality report comes out a week later. And the dataset leaves before anyone has actually looked at it.
On one record, the rubric version is pinned to the batch. The disagreement is a row in adjudication, not a thread in chat. The IAA reads against the gate before the export is even staged. And the dataset only leaves when the gate has cleared — with the reason, the reviewer, and the version still attached.
WHERE IT FITS
In the loop, this is where you review.
Test the run. Review the hard cases. Recruit the right specialist. Remember the misses. Approve what's right. When Evaluation Studio hits a case a model can't score, this is where a human scores it.
01
Test
02
Review
● YOU ARE HERE
03
Recruit
04
Remember
05
Approve
RELATED MODULES
Next to this in the Human Data OS.
WORKFORCE
The right reviewer. On the right case.
Specialists routed to the rows the rubric flagged.
See the page →CLEO
An instrument, not a chatbot.
Reads the rubric, the review, and the dataset alongside you.
See the page →EVALUATION STUDIO
Review it before release.
The same rubric that grades a release grades the dataset.
See the page →ANNOTATION
Labels that don't end at delivery.
Bring the rubric your reviewers already trust. We'll keep it attached to every label, every review, and every dataset that leaves — on infrastructure you keep.
Pilots start at a single reviewed batch. Bring the modality, the rubric, and the rights — we'll sign the dataset.