Preference data
A human choice between model answers. Each record holds the prompt, the answers compared, which one is better, and the reason.
AuraOne / Human Data / RLHF and preference data
RLHF and preference data from qualified experts.
Preference data records which of several model answers is better, and why. RLHF (reinforcement learning from human feedback) and DPO learn from it. AuraOne's experts produce it for AI teams that train or evaluate language models.
Qualified specialists compare answers side by side and write the reason. On expert programs, a second reviewer checks the work, and an adjudicator settles disagreements against criteria you control.
- Input
- Model answers, your criteria, and the expertise the task needs
- Output
- Reviewed judgments, rationales, scorecards, and evaluation data
The terms
What RLHF and preference data are
Four terms, in plain words. For the research behind them, read the InstructGPT paper, which trains with human demonstrations and preference judgments, and the DPO paper, which optimizes directly on preference data. The glossary defines the rest.
RLHF
Reinforcement learning from human feedback. People compare or rate model answers. Those judgments train a reward model, and the reward model steers the model toward the answers people prefer.
DPO
Direct preference optimization. It tunes a model on preference pairs directly, without a separate reward model.
Reward model
A model trained on preference data to score answers. Training uses its score as a stand-in for a human judge.
What you receive
What a preference-data program delivers
Your contract sets the exact scope.
Side-by-side comparisons
Your prompts and model answers, graded against your criteria, with every score tied to the rubric version it used.
- Criterion-level labels and scores
- Side-by-side comparisons and scorecards
Corrections and reasons
Where the model is wrong, the expert says what right looks like, and why.
- Expert corrections and written rationales
- Adjudicated disagreements and failure classifications
Datasets you can reuse
Accepted judgments become datasets. Adjudicated failures become regression cases. Both are scoped per program.
- Scoped JSONL or CSV datasets, each score tied to its rubric version
- Reusable regression cases
One record
What one preference record holds
Each judgment keeps its context, so a reviewer can trace it later.
Scope set by contract
- Prompt and answers
- The task and the answers compared, side by side.
- Preference
- Which answer is better, scored against your criteria.
- Reason
- The expert's written rationale, and the correction where the model was wrong.
- Criteria
- The rubric version the judgment used.
- Reviewer
- Who judged it, and the credential the task required.
- Review outcome
- The second reviewer's decision on expert programs, and the adjudication if experts disagreed.
How the work is checked
Each step leaves a record you can inspect.
01
Define the work
Tell us what needs judging and who is qualified to judge it.
Program brief and rights scope
02
Qualify the experts
On expert programs, work is assigned only after a passed test. The score stays on the expert's record.
Qualification record
03
Compare and write the reason
Experts compare the answers against your criteria and write down why one is better.
Task, rubric version, and rationale
04
Check the hard calls
On expert programs, a second reviewer checks the work. When two experts disagree, a named adjudicator decides. No averaging it away.
Reviewer, disposition, and adjudication
05
Deliver with the record
Accepted work is delivered with the review record and the agreed use terms for the program.
Delivery manifest, review history, and acceptance
Buying guide
What to ask any RLHF data provider
Six questions that show whether preference data can be trusted. They come from our guide to RLHF reviewer drift. The Expert Reviewer Certification Template turns the answers into a qualification record.
Who judges
- How are reviewers qualified for this task, and when does that qualification expire?
- Do reviewers calibrate on reference cases before they touch live work?
How the criteria are kept
- Is the rubric versioned, and what happens to work already produced when it changes?
- Do resolved disagreements become reference cases with an accountable owner?
What you can trace
- Is quality reviewed by reviewer and by task slice, or only as a pool-wide average?
- Can an accepted example be traced to its task version, source permissions, reviewer approval and adjudication?
Scope and limits
What this program does not do
Data, not training
AuraOne prepares the judgments and the evidence. Training, fine-tuning, and model-performance gains are separate scopes.
Rights come first
Reuse of accepted judgments depends on your data rights and retention policy. Use terms are agreed per program.
Pilot first
A program starts as a pilot with an agreed scope, a defined deliverable and an acceptance check.
Questions
Questions about RLHF and preference data
What is RLHF data?
RLHF data is the human feedback used in reinforcement learning from human feedback. In practice it is preference data: people compare model answers and say which is better, and why. A reward model, or a method such as DPO, learns from those choices.
What is the difference between preference data and SFT data?
Preference data compares several answers and says which one is better. SFT data shows the model one good answer for each prompt. Both come from expert judgment, and teams often use both.
Who compares the answers?
Domain specialists, including law, medicine and finance, compare the answers against your criteria and write the reason. On expert programs, a second reviewer checks the work, and an adjudicator settles disagreements.
How is the quality of the judgments checked?
On expert programs, work is assigned only after a passed test, and a second reviewer checks the work. When two experts disagree, a named adjudicator decides. Each score stays tied to the rubric version it used, and accepted work is delivered with its review record.
Does AuraOne train the model?
A preference-data program delivers the judgments and the evidence, not a trained model. Training, fine-tuning and model-performance gains are separate scopes, agreed separately.
How do I start, and what does it cost?
Tell us what your model needs to learn. We scope the specialists, the review and the rights with you, and every engagement gets a written quote before work starts. Pricing is scope-based, with no rate card and no tiers.
Guides
Related reading
RLHF Reviewer Drift: Calibration and Handoff ControlsDiagnose reviewer drift in RLHF and DPO workflows with calibration, rubric versioning, adjudication records and regression cases.Human Data Reviewers: From Resumes to Task-Specific ReputationAssess expert reviewers through calibration, adjudicated disagreements and task-specific quality records rather than resumes or a universal score.
Tell us what your model needs to learn.
We scope the specialists, the review, and the rights with you. Related pages cover SFT data, LLM evaluation and the Human Data overview.