- Published
- May 15, 2026
- Reviewed
- October 25, 2026
- Author
- AuraOne Open team
- Category
- AuraOne Open
- Reading
- 8 min
Rubric Studio Open is now at 0.2.0, a local-first macOS Apple Silicon release with an inspectable repository, the MIT license, and a checksummed artifact. The release notes report Developer ID signing and notarization. The npm validator companion ships separately as 0.2.1; it does not bundle the desktop or browser editor. These separate release lines were checked for this article update on October 25, 2026.
The short version: rubrics have become infrastructure, but the workflow for writing them still looks like office software and one-off scripts. We think the rubric deserves an IDE.
Rubric Studio Open is built for the researcher who wants to treat a rubric like a durable artifact. It should live in a folder on disk. It should validate. It should be reviewable in git. It should run against samples. It should calibrate against expert labels. It should show what changed when a criterion changed. It should export to the evaluation frameworks already in use. It should not require a hosted account to be useful.
That is the product.
Why this exists
The AI evaluation stack has changed. A few years ago, many teams could get away with scalar preferences, benchmark averages, or broad pass/fail judgments. That is no longer enough for the systems being trained and shipped now.
Modern frontier work is increasingly criterion-level. A good evaluation does not just ask whether an answer is better. It asks whether the answer followed a policy boundary, cited the right source, avoided an unsafe claim, preserved uncertainty, respected task constraints, handled a refusal correctly, or satisfied a domain-specific expert standard.
Our AI measurement guide explains why a rubric is one kind of evidence rather than a complete readiness score. The RLHF pipeline guide examines the separate problem of reviewer consistency.
Those criteria are not incidental. They become the object that researchers debate, improve, run, publish, and sometimes use as a reward or review signal. The rubric is the durable thing.
And yet, in many teams, the rubric is still scattered across a document, a spreadsheet, a prompt file, a notebook, a CSV, and a Slack thread. That fragmentation creates predictable failures:
- A criterion changes, but the scoring run does not say which samples flipped.
- A reviewer disagrees, but the agreement metric lives in a notebook only one person understands.
- A benchmark artifact ships, but the judge prompt and provenance are hard to reproduce.
- A team wants to move from local research to expert review at scale, but the handoff is a custom export every time.
- A new contributor opens the repo and cannot tell what the rubric is, what version it is, or how to run it.
Rubric Studio Open is our answer to that gap.
What it does
Rubric Studio Open turns a rubric into a project folder:
rubric-project/
rubric.toml
criteria/
judges/
samples/
calibration/
runs/
exports/
The desktop app gives that folder an authoring environment. The Agent Studio replay workflow provides a complementary pattern for inspecting agent traces; rubric authoring and trace replay are separate tasks. The CLI supports automation. The browser editor is a separate, narrower workflow and should not be read as complete feature parity.
The core workflow has five verbs.
Author
write criterion-level rubrics with IDs, weights, severity, evidence requirements, examples, theme tags, and sibling links. The inline validator catches missing examples, duplicate IDs, invalid weights, and adapter compatibility issues.
Test
load prompt and response samples, use local fixtures or a configured provider where supported, and inspect criterion-level results. A provider-backed result depends on user configuration and is not implied by opening the project.
Calibrate
import expert-scored gold sets and compute agreement metrics, including Cohen kappa, Fleiss kappa, Krippendorff alpha, weighted variants, and confidence intervals. The calibration tab ranks criteria that need better wording or better examples. The versioned expert-score calibration tutorial specifies the sample and gold-label inputs; agreement measures do not establish correctness by themselves.
Diff
compare rubric versions semantically, then see which held-out samples changed behavior. The goal is not just "what text changed?" The real question is "what broke or improved because the text changed?"
Export
generate rubric-spec, judge cards, eval-run manifests, conformance badges, and framework adapters for tools such as lm-eval-harness, Inspect, OpenAI Evals, Promptfoo, and Hugging Face Hub metadata. The lm-eval-harness export tutorial identifies generated files and adapter warnings; unsupported rubric features must remain in accompanying evidence rather than silently disappear. An AuraOne intake export is a local artifact after explicit preview, not an automatic upload or managed-program enrollment.
The trust model
Open-source evaluation tools need a clean trust contract.
Rubric Studio Open is local-first. Projects live on disk. Telemetry is off by default. Crash reporting is off by default. API keys live in the operating system keychain. The app has a documented no-network mode. The telemetry settings panel shows the exact event JSON that would be sent if telemetry is enabled.
AuraOne intake export is explicit. The app does not send your rubric to AuraOne in the background. When you create an intake export, the preview shows what is included, what has been redacted, and what is never sent.
That boundary matters. The people building evaluations that decide what ships are right to be skeptical of software that quietly phones home. We want the first reaction from a security-minded researcher to be "I can inspect this."
Why open and managed programs both exist
Rubric Studio Open is not a trial. It is not a limited free tier. It is the single-user, local-first authoring environment published under the MIT license, with its source revision and release artifact identified on the product page.
Teams that need multi-author review, adjudication, approval, and audit trails across many people work with AuraOne in a managed Rubric Program, with expert authoring and review at scale.
The boundary is simple:
- Open: author, test, calibrate, diff, and export locally.
- Programs: review, approve, and adjudicate as a team, with managed expert reviewers in the loop.
We want Open to be useful even if you never become a customer. We also want the handoff to a managed program to be clean when a researcher reaches the scale wall.
Built from AuraOne Open engines
Rubric Studio Open is the cockpit for the AuraOne Open evaluation libraries.
rubric-spec defines the portable schema. iaa-kit handles agreement metrics. judge-bench runs judge diagnostics. judge-card produces disclosure artifacts. eval-run-manifest preserves provenance. contamination-audit checks leakage risk. prompt-rubric-drift powers semantic review notes. eval-adapter handles framework exports. synthetic-disagreement supports calibration stress tests.
Each library remains useful on its own. The IDE makes them feel like one product.
That structure is important. We do not want a polished interface hiding a weak foundation. We want the interface to sit on public, inspectable, independently useful tools.
Who it is for
Rubric Studio Open is primarily for three groups.
Post-training and eval researchers: people maintaining behavior rubrics, reward-model evaluation criteria, safety criteria, or reviewer instructions that need to survive more than one experiment.
Safety and alignment teams: people who need reproducible artifacts, judge disclosures, contamination checks, and defensible methodology.
Academic eval researchers: people publishing rubric-heavy work who want reviewers and peers to run the artifact without a 30-page setup document.
It is also useful for community eval maintainers, policy teams, trust and safety teams, and anyone tired of treating rubrics as copy-paste text instead of structured artifacts.
What to try first
Start with the verified desktop release record:
macOS Apple Silicon DMG:
https://github.com/auraoneai/rubric-studio-open/releases/download/v0.2.0/Rubric.Studio.Open_0.2.0_aarch64.dmg
SHA-256:
7dcb7de67835947b421089eab5fc244bcd8f75d503ebc7e763921c229c68f23d
For a reproducible first project, use the versioned Helpful Response Evaluation example rather than an unspecified folder. It contains rubric.toml, criterion files, sample responses and a local-mock judge.
- Download the v0.2.0 source archive from the release page and open
examples/helpful-response-evaluationin the desktop app. - Inspect the specificity criterion, including its positive examples, negative examples and boundary for short factual answers.
- Preview the bundled responses with the local mock judge. Inspect the criterion-level evidence and change one criterion description in a copy of the project.
- Compare the versions and export a local artifact. Check that the criterion IDs, selected samples and judge configuration remain visible in the result.
The expected output is an inspectable local rubric/evidence artifact, not proof of a model’s quality. Mock scores are fixture evidence; provider-backed scoring requires an explicit configured provider. The versioned example README documents the first-run workflow.
The best first test is not whether the UI looks good. It is whether the artifact makes sense after you close the app. Can you read the criterion files? Can you understand the score run? Can you diff the change? Can a teammate reproduce it?
That is the bar.
What we want feedback on
We are especially interested in:
- Rubric schema edge cases.
- Export adapter gaps.
- Agreement metrics researchers actually use.
- Diff views that help during review.
- No-network mode expectations.
- Telemetry wording and event transparency.
- What would make a rubric artifact acceptable as paper supplementary material.
We also want examples of where the product should stay boring. A rubric IDE should not hide every hard decision behind an AI rewrite button. The point is to help experts make criteria precise, not to replace the expert judgment that gives a rubric its value. That is why the first release emphasizes validation, provenance, calibration, and diffs over generative polish. Autocomplete and draft assistance are useful only when they make the author more accountable to the artifact.
The same principle applies to exports. The winning workflow is not one universal format that forces every team to migrate. It is a source project that can produce the adapter a team needs today while keeping the richer rubric, calibration, and provenance material beside it. If an adapter loses information, the export should say so clearly.
If you maintain evals, write rubrics, review model behavior, or publish benchmark artifacts, we want the uncomfortable feedback.
Rubric Studio Open is our first flagship AuraOne Open product because this category needs a real authoring layer. The rubric is too important to live as scattered text.
It should have an IDE.
Start with the Rubric Studio Open v0.2.0 release or the Rubric Studio Open product page. A read-only example build of the editor runs at rubric-studio.auraone.ai to show the interface.
