Author, validate, preview, calibrate, diff, and export criterion-level AI evaluation rubrics.

Criterion definition, response samples, scoring evidence, calibration, and score-impact diff stay in one local-first decision workspace.

Output

Versioned rubric files, deterministic preview traces, calibration records, score-impact diffs, and export manifests.

Environment

Projects stay on the device unless the operator explicitly exports, shares, or connects a hosted service.

Connect agent systems, inspect tool traces, replay failures, compare candidates, and export regression evidence.

Connection identity, tool sequence, timing, cost, payload, replay, model comparison, and release evidence remain attached to the selected trace.

Output

Portable trace cards, deterministic replay fixtures, behavior comparisons, CI exports, and native keychain-backed signing.

Environment

Trace and replay files stay local by default; network movement follows only the model, tool, endpoint, or telemetry connection configured by the operator.

Synchronize robot episodes, inspect failures and recoveries, record review decisions, and export checksummed evidence.

A dominant deterministic sensor scene, transport timeline, data health, annotations, and release decision stay synchronized in one cockpit.

Output

Synthetic fixture provenance, episode annotations, quality findings, decision records, local export archives, and capture evidence.

Environment

Review is local-first; upload or hosted movement occurs only through an explicitly configured destination adapter.

Call hosted evaluation, analytics, training-export, and governance APIs from Python services, notebooks, and automation.

Typed sync and async clients share an explicit credential, base-URL, output, error, and retry contract.

Output

Typed client methods, deterministic CLI output modes, stable exit codes, clean-package tests, and registry provenance evidence.

Environment

Only explicit API requests are sent to the configured AuraOne base URL; credentials remain in the operator's environment.

Integrate hosted AuraOne evaluation workflows through typed REST and GraphQL clients.

The package makes runtime targets, exports, retries, polling, credential handling, and response types inspectable from one public contract.

Output

Generated declarations, package exports, clean-consumer tests, packed tarball inspection, and npm provenance evidence.

Environment

Requests go only to the configured AuraOne API; browser credentials must be scoped and protected by the host application.

Validate rubrics, score approved fixtures, audit reviewer agreement and leakage, and generate portable evaluation reports.

One local Python CLI keeps rubric checks, deterministic scoring, reviewer QA, leakage checks, and evidence reports in the same inspectable workflow.

Output

Validated rubric output, score records, agreement and leakage findings, plus self-contained HTML and Markdown reports.

Environment

Inputs and reports remain local or inside the operator's CI environment unless the operator uploads them elsewhere.

Validate, lint, diff, and convert portable AuraOne Rubric Schema v1 files.

One criterion-level schema preserves scoring intent across EvalKit and supported framework adapters.

Output

Schema validation, lint findings, semantic diffs, adapter output, and conformance fixtures.

Environment

Rubric files are processed locally; adapters do not call a model unless a consuming workflow explicitly does so.

Measure inter-annotator agreement with modern metrics, uncertainty, ordinal support, and missing-data handling.

Agreement estimates and bootstrap intervals are available from one pure-NumPy library with deterministic fixtures.

Output

Metric output, confidence intervals, missing-data diagnostics, synthetic examples, and test vectors.

Environment

Annotation arrays are processed locally and no network access is required.

Probe judge behavior for position, verbosity, self-preference, paraphrase, anchoring, and calibration failures.

Synthetic diagnostic cases isolate judge failure modes before they are mixed into production evaluation scores.

Output

JSON and Markdown findings, plots, probe fixtures, calibration summaries, and reproducible run metadata.

Environment

Fixtures and reports remain local; model calls occur only through the judge provider configured by the operator.

Validate and render a disclosure record for judge prompts, calibration, bias, limitations, and intended use.

The card keeps operational judge evidence beside the evaluation system instead of burying it in prose.

Output

Validated Judge Card JSON, rendered Markdown or HTML, calibration references, and known-limit records.

Environment

Card generation is local and does not invoke the documented judge.

Translate one rubric-spec rubric and run configuration into supported evaluation-framework inputs and exports.

The portable rubric remains authoritative while framework-specific files are generated and reviewable.

Output

Generated adapter files, normalized run configuration, fixtures, and round-trip or conformance checks.

Environment

Conversion is local; downstream framework network behavior is outside the adapter and must be reviewed separately.

Run executable rubric-spec v1 compatibility checks and emit reproducible conformance evidence.

Compatibility claims are tied to executable fixtures and a versioned badge rather than documentation alone.

Output

Conformance results, fixture-level failures, machine-readable output, and reproducible SVG badges.

Environment

Conformance runs locally against supplied adapter behavior and requires no hosted service.

Create and validate a portable envelope for evaluation provenance, environment, artifacts, result digests, and optional signatures.

The decision record links code, data, rubric, judge, contamination, and result evidence through one schema.

Output

Validated manifest JSON, artifact digests, reference checks, and optional signature verification.

Environment

Manifest generation and validation are local; referenced artifact storage remains under the operator's control.

Scan evaluation data for n-gram overlap, canaries, answer patterns, hashes, and optional embedding similarity.

Multiple local contamination signals are reported together with evidence instead of collapsed into one unsupported certainty score.

Output

Finding records, matched spans or hashes, severity, configuration, summary report, and deterministic fixtures.

Environment

Local checks require no upload; optional embedding behavior depends on the model or endpoint explicitly configured by the operator.

Generate controlled synthetic annotator disagreement before real reviewer data is collected.

Deterministic disagreement patterns let teams test metric, missingness, severity, and adjudication behavior against known conditions.

Output

Generated labels, seed and scenario metadata, expected disagreement properties, and reproducibility tests.

Environment

Generation is local and uses synthetic records only.

Run AuraOne evaluations for an exact commit and publish one lifecycle-aware Check Run plus one idempotent evidence summary.

Check state, evaluated commit, threshold, template evidence, remediation, permissions, and deployment identity remain attached to one native review path.

Output

GitHub Check Run output, bot-owned summary marker, webhook tests, package contents, and deployed-app permission evidence.

Environment

The self-hosted app reads configured repository context, calls AuraOne, and writes only the approved Checks and optional summary comment.

Run Agent Studio's trace and regression workflows in terminals and CI without the visual workbench.

CLI output and exported evidence follow the same trace, replay, and release contracts as the desktop application.

Output

Structured trace output, replay diffs, JUnit and regression artifacts, package tests, and clean-install evidence.

Environment

Local files remain local; network access occurs only through explicitly configured endpoints or providers.

Score repository responses against an AuraOne rubric and publish one consistent GitHub evidence surface.

Annotations, job summary, outputs, threshold decision, and optional idempotent pull-request comment agree on one result.

Output

Action outputs, annotations, job summary, bot-owned comment, self-tests, and immutable-tag evidence.

Environment

Repository files are processed inside GitHub Actions; any model or hosted API access depends on the supplied judge configuration and secrets.

Validate required Datasheet, Model Card, and Data Card sections and surface actionable review findings.

Blocking omissions and high-recall PII-like review signals are separated into one inspectable decision and one idempotent summary.

Output

Checked-file count, blocking failures, warnings, job summary, pull-request comment, and standalone validator report.

Environment

Markdown files are processed in CI; optional pull-request access uses the supplied GitHub token.

Diff and lint rubric changes, annotate affected lines, and publish one merge decision for the exact commit.

The bot updates one Check Run and one marker-owned summary instead of producing duplicate or detached review noise.

Output

Rubric diff, lint findings, annotations, Check Run conclusion, summary marker, and deployed-app permission evidence.

Environment

The app reads configured rubric files and pull-request context and writes only approved Checks, annotations, and optional comments.

Compare before and after prompt and rubric directories and generate deterministic drift review notes.

Scoring-boundary, criterion, weight, and prompt changes are surfaced as evidence-oriented review items.

Output

Deterministic Markdown report, changed-file evidence, job summary, and idempotent pull-request comment.

Environment

Repository files are compared inside the local or GitHub Actions environment and require no model call.

Turn one agent trace into portable Markdown, HTML, and JSON evidence cards.

Goal, outcome, tools, retries, data touched, checks, timing, and remediation remain attached to the same run identity.

Output

Rendered trace card, normalized JSON, referenced artifacts, checksums, and deterministic fixtures.

Environment

Trace files are processed locally; the renderer makes no model or telemetry call.

Normalize agent tool-call traces into deterministic replay artifacts and pytest cases.

Recorded tool inputs and outputs can be locked, redacted, replayed, diffed, and promoted into CI without rerunning the live tool.

Output

Replay bundle, redaction record, generated pytest, expected outputs, and diff report.

Environment

Replay is local by default; live tool access occurs only when the operator explicitly chooses a non-mocked adapter.

Convert OpenTelemetry or Phoenix GenAI trace exports into local evaluation regression cases and manifests.

Operational traces become portable evaluation inputs without requiring a proprietary trace store.

Output

Normalized cases, trace linkage, eval-run-manifest references, result summaries, and deterministic conversion tests.

Environment

Export files are processed locally; attaching results to a remote backend requires an explicitly configured endpoint.

Scan MCP manifests, package metadata, source, and docs for tool, auth, filesystem, shell, network, and readiness risks.

The linter reports inspectable repository evidence and configurable severity instead of claiming runtime security certification.

Output

Markdown, JSON, or SARIF findings, file references, severity threshold, job summary, and optional idempotent comment.

Environment

Scanning runs inside the local or CI repository boundary and requires no model call.

Validate agent cards, capabilities, structured payloads, errors, cancellation, and task lifecycle behavior.

Offline contract fixtures test the public protocol boundary without requiring a live agent deployment.

Output

Agent-card validation, transcript findings, lifecycle report, and deterministic contract fixtures.

Environment

The default suite runs locally against files; no agent endpoint is contacted unless a future adapter is explicitly configured.

Review event streams, failures, and decisions in one self-contained offline HTML artifact.

The canonical viewer exports review metadata without exporting observations, actions, sensor payloads, media, or training shards.

Output

Self-contained HTML, event and decision records, metadata-only export tests, and network-independence checks.

Environment

The viewer runs offline and exports review metadata only.

Browse, filter, validate, and reproduce public synthetic failure cases.

Each case keeps the symptom, reproduction, expected behavior, observed behavior, tags, and contribution provenance together.

Output

Validated case records, searchable static gallery, reproduction metadata, and deterministic generated output.

Environment

Only public or approved cases are displayed; private datasets are not uploaded to the gallery.

Validate and render structured cards for robot morphology, sensors, control, environment, data, and known limitations.

Embodiment-specific release facts stay machine-readable and render into a reviewer-facing card.

Output

Validated card JSON, rendered documentation, schema errors, dataset or model references, and checksums.

Environment

Card generation is local and does not upload robot data or invoke a policy.

Validate and summarize human intervention and recovery segments in robot episodes.

Recovery latency, success, recurrence, operator action, and evidence references stay attached to the source segment.

Output

Validated segment records, aggregate metrics, failure and recovery summaries, and deterministic fixtures.

Environment

Episode metadata is processed locally; media and sensor payloads remain referenced rather than uploaded.

Generate simulator-light perturbation diagnostics for vision-language-action policies.

Task-instruction and observation perturbations produce bounded, reviewable diagnostics without claiming full simulator coverage.

Output

Perturbation manifest, expected and observed behavior, robustness summary, seed, and reproducible fixtures.

Environment

Fixture generation is local; policy inference uses only the adapter and endpoint explicitly configured by the operator.

Index robotics datasets, generate thumbnails, run QA and clustering, and export reviewed evidence through a headless engine.

The CLI exposes the same dataset and evidence contracts used by Robotics Studio Open for automation and integration.

Output

SQLite index, adapter diagnostics, QA findings, clusters, thumbnails, export manifests, and engine tests.

Environment

Datasets are processed locally; remote storage, model, or hosted destinations require explicit adapters.

Provide the shared shell, contracts, release controls, keychain bridge, updater conventions, and evidence infrastructure for the three desktop Studios.

Shared runtime and release behavior is implemented once while each product keeps its own workflow and visual hierarchy.

Output

Platform contract tests, distribution manifests, security gates, release-evidence schemas, and clean package builds.

Environment

The platform is source infrastructure; product-specific data movement depends on the consuming Studio and its configured adapters.

Provide OSS-safe semantic tokens and evidence UI primitives without shipping AuraOne's private font binaries.

The package separates public token and component contracts from the private official-font delivery boundary.

Output

Token CSS, component tests, accessibility checks, public-asset verification, and package-content inspection.

Environment

No private font binary is included; consuming public apps use declared fallbacks unless an authorized same-origin font boundary is available.

Provide tested shell, navigation, telemetry, evidence, and workbench primitives for the Open Studio flagships.

The kit is optimized for repeated technical work and inherits the shared Proofline semantic contract instead of legacy glass styling.

Output

Component tests, zero-violation accessibility audits, generated declarations, canonical CSS, and clean build output.

Environment

The package renders UI only; runtime, storage, network, and secret behavior are supplied by the consuming application.

Define portable contracts for keychain access, intake signing, telemetry, updater evidence, and release boundaries.

Applications share typed behavior while native implementations remain replaceable and fail closed.

Output

Type declarations, contract tests, native integration tests, and release-evidence validation.

Environment

The package defines interfaces and normalization; no secret or telemetry payload is sent by the package itself.

Import rubric JSON and response JSONL, validate them, run deterministic scoring, and inspect exact evidence in the browser.

The primary workflow runs in-browser with a Pyodide-compatible local execution boundary and shareable state instead of requiring an account.

Output

Validation output, score evidence, permalink state, deterministic fixtures, and browser interaction tests.

Environment

Imported evaluation data remains in the browser unless the user explicitly exports or shares it.

Show what each tool does, who it is for, and exactly what state it ships in.

Source, registry, binary, capture, runtime, and support claims are shown as separate evidence classes.

Output

Canonical product pages, one selected flagship image per visual product, structured data, release records, and conversion events.

Environment

The website is a marketing and evidence surface; product runtime and data movement remain defined by each offering.

Check dataset structure, episodes, timestamps, media, actions, states, metadata, and release readiness.

Training-readiness findings remain explicit by severity and can render as Markdown, JSON, card, or badge output.

Output

Dataset findings, severity threshold, report artifact, deterministic fixtures, and Action summary.

Environment

The dataset path is scanned locally or in CI; no dataset upload is required.

Provide repo-ready MCP, A2A, OTEL, replay, compare, and CI examples.

Each example is intentionally small, synthetic, local-first, and linked to the exact evidence it produces.

Output

Runnable fixtures, expected output, generated artifacts, and example-specific verification commands.

Environment

Examples are synthetic by default; provider or endpoint access is explicitly opt in per example.

Turn open-source and AI-system evaluation into a concrete technical, security, data, support, and release review.

The toolkit separates inspectable evidence, open questions, owners, and blocked claims instead of presenting generic vendor guidance.

Output

Decision checklists, evidence-request templates, comparison records, and owner or next-action fields.

Environment

The resources are public guidance and do not collect product data unless a user explicitly follows a linked contact workflow.