Evaluation examples
Your tasks, graded against your criteria, with every score tied to the rubric version it used.
- Criterion-level labels and scores
- Side-by-side comparisons and scorecards
AuraOne / Human Data / Expert AI Quality
Expert judgment, with the reasoning and review attached.
Qualified specialists create or assess difficult examples; review follows the agreed criteria and records disagreements.
Qualified authors do the work, an independent reviewer checks it, and an adjudicator settles disagreements against criteria you control.
- Input
- Model outputs, test cases, criteria, and expertise requirements
- Output
- Reviewed judgments, rationales, scorecards, and evaluation data
TaskFix the slow product list. Do not change the architecture.
Model A
cache.set('products', qs)Model BPreferred
.select_related('category')
.prefetch_related('reviews')Reviewer · senior backend engineerGrading Model B against rubric v2.2
- Finds the root cause4/4
- Keeps the architecturePass
- Regression test2/4
“Right loading per relation. Count only the relationship queries.”
Second reviewer disagreed → adjudicated. Pass after revision.
Decision · rubric v2.2CORRECTION_ACCEPTED
Expert evaluation and RLHF data
Expert evaluation, RLHF and preference data.
Expert evaluation data is human judgment on model answers: graded scores, side-by-side comparisons, corrections, and the reason for each. The comparisons are preference data, the input to RLHF and DPO. Qualified specialists produce it here, and an independent reviewer checks the work.
RLHF and preference data and SFT data say what each one is and what a program delivers.
One reviewed correction
See what an expert catches.
The task: fix a slow product list without changing the architecture. A reviewer checked the fix against the criteria and wrote down why it holds.
What you receive
A score is not an answer.
You get the judgments, the reasons, and material you can use again. Your contract sets the exact scope.
Corrections and rationales
Where the model is wrong, the expert says what right looks like, and why.
- Expert corrections and written rationales
- Adjudicated disagreements and failure classifications
Review and regression material
Accepted judgments become datasets. Adjudicated failures become regression cases. Both are scoped per program.
- Scoped JSONL or CSV datasets, each score tied to its rubric version
- Reusable regression cases
Evaluation program
Grade it. Then check the grading.
Start from the decision you need to make. Then find the people qualified to make it.
Scoped expert gradingReviewers apply your criteria to each candidate's work under a defined protocol. Recorded: Task, rubric version, required credential, and expert rationale.Independent quality reviewA second reviewer approves the work, returns it, or rejects it. Recorded: Reviewer, disposition, and lifecycle history.Adjudicated disagreementWhen two experts disagree, a named adjudicator decides. No averaging it away. Recorded: Competing judgments, adjudication rationale, and final decision.Qualified reviewersEach task states the expertise it takes to judge. Qualified experts are assigned to it. Recorded: Required credential and reviewer assignment.
What the work lets you decide
Every batch ships with inter-annotator agreement metrics, reviewer qualification IDs, and criterion-level rationales.
| Outcome | Work | What you receive | Program fit |
|---|---|---|---|
| Compare model candidates | Run your criteria against the same task set and adjudicate material disagreements. | Per-criterion results, rationales, and the final scorecard. | The comparison applies to the scoped test set and rubric version. |
| Build a failure dataset | Keep each adjudicated failure as a regression case for later evaluation runs. | Failure labels, source cases, and the adjudication behind each. | Reuse depends on the customer's data rights and retention policy. |
| Prepare post-training data | Select accepted judgments and rationales that meet the program's delivery criteria. | Approved records, stated exclusions, and the delivery manifest. | Training, fine-tuning, and model-performance gains are separate scopes. |
Related capabilities
Continue into the program the work needs.
RLHF and preference dataExperts compare model answers side by side and write the reason.SFT dataExperts write and correct the answers you want a model to give.Human Data overviewThe data your model cannot find on the internet.Voice AISpeech recorded with consent. The permission stays with each recording.Physical AIPeople record everyday tasks, with consent, so robots learn how real work is done.