Validate rubrics, score approved fixtures, audit reviewer agreement and leakage, and generate portable evaluation reports.

One local Python CLI keeps rubric checks, deterministic scoring, reviewer QA, leakage checks, and evidence reports in the same inspectable workflow.

Output

Validated rubric output, score records, agreement and leakage findings, plus self-contained HTML and Markdown reports.

Environment

Inputs and reports remain local or inside the operator's CI environment unless the operator uploads them elsewhere.

Run AuraOne evaluations for an exact commit and publish one lifecycle-aware Check Run plus one idempotent evidence summary.

Check state, evaluated commit, threshold, template evidence, remediation, permissions, and deployment identity remain attached to one native review path.

Output

GitHub Check Run output, bot-owned summary marker, webhook tests, package contents, and deployed-app permission evidence.

Environment

The self-hosted app reads configured repository context, calls AuraOne, and writes only the approved Checks and optional summary comment.

Run Agent Studio's trace and regression workflows in terminals and CI without the visual workbench.

CLI output and exported evidence follow the same trace, replay, and release contracts as the desktop application.

Output

Structured trace output, replay diffs, JUnit and regression artifacts, package tests, and clean-install evidence.

Environment

Local files remain local; network access occurs only through explicitly configured endpoints or providers.

Review event streams, failures, and decisions in one self-contained offline HTML artifact.

The canonical viewer exports review metadata without exporting observations, actions, sensor payloads, media, or training shards.

Output

Self-contained HTML, event and decision records, metadata-only export tests, and network-independence checks.

Environment

The viewer runs offline and exports review metadata only.

Browse, filter, validate, and reproduce public synthetic failure cases.

Each case keeps the symptom, reproduction, expected behavior, observed behavior, tags, and contribution provenance together.

Output

Validated case records, searchable static gallery, reproduction metadata, and deterministic generated output.

Environment

Only public or approved cases are displayed; private datasets are not uploaded to the gallery.

Score repository responses against an AuraOne rubric and publish one consistent GitHub evidence surface.

Annotations, job summary, outputs, threshold decision, and optional idempotent pull-request comment agree on one result.

Output

Action outputs, annotations, job summary, bot-owned comment, self-tests, and immutable-tag evidence.

Environment

Repository files are processed inside GitHub Actions; any model or hosted API access depends on the supplied judge configuration and secrets.

Validate required Datasheet, Model Card, and Data Card sections and surface actionable review findings.

Blocking omissions and high-recall PII-like review signals are separated into one inspectable decision and one idempotent summary.

Output

Checked-file count, blocking failures, warnings, job summary, pull-request comment, and standalone validator report.

Environment

Markdown files are processed in CI; optional pull-request access uses the supplied GitHub token.

Diff and lint rubric changes, annotate affected lines, and publish one merge decision for the exact commit.

The bot updates one Check Run and one marker-owned summary instead of producing duplicate or detached review noise.

Output

Rubric diff, lint findings, annotations, Check Run conclusion, summary marker, and deployed-app permission evidence.

Environment

The app reads configured rubric files and pull-request context and writes only approved Checks, annotations, and optional comments.

Validate, lint, diff, and convert portable AuraOne Rubric Schema v1 files.

One criterion-level schema preserves scoring intent across EvalKit and supported framework adapters.

Output

Schema validation, lint findings, semantic diffs, adapter output, and conformance fixtures.

Environment

Rubric files are processed locally; adapters do not call a model unless a consuming workflow explicitly does so.

Measure inter-annotator agreement with modern metrics, uncertainty, ordinal support, and missing-data handling.

Agreement estimates and bootstrap intervals are available from one pure-NumPy library with deterministic fixtures.

Output

Metric output, confidence intervals, missing-data diagnostics, synthetic examples, and test vectors.

Environment

Annotation arrays are processed locally and no network access is required.

Probe judge behavior for position, verbosity, self-preference, paraphrase, anchoring, and calibration failures.

Synthetic diagnostic cases isolate judge failure modes before they are mixed into production evaluation scores.

Output

JSON and Markdown findings, plots, probe fixtures, calibration summaries, and reproducible run metadata.

Environment

Fixtures and reports remain local; model calls occur only through the judge provider configured by the operator.

Validate and render a disclosure record for judge prompts, calibration, bias, limitations, and intended use.

The card keeps operational judge evidence beside the evaluation system instead of burying it in prose.

Output

Validated Judge Card JSON, rendered Markdown or HTML, calibration references, and known-limit records.

Environment

Card generation is local and does not invoke the documented judge.

Translate one rubric-spec rubric and run configuration into supported evaluation-framework inputs and exports.

The portable rubric remains authoritative while framework-specific files are generated and reviewable.

Output

Generated adapter files, normalized run configuration, fixtures, and round-trip or conformance checks.

Environment

Conversion is local; downstream framework network behavior is outside the adapter and must be reviewed separately.

Run executable rubric-spec v1 compatibility checks and emit reproducible conformance evidence.

Compatibility claims are tied to executable fixtures and a versioned badge rather than documentation alone.

Output

Conformance results, fixture-level failures, machine-readable output, and reproducible SVG badges.

Environment

Conformance runs locally against supplied adapter behavior and requires no hosted service.

Create and validate a portable envelope for evaluation provenance, environment, artifacts, result digests, and optional signatures.

The decision record links code, data, rubric, judge, contamination, and result evidence through one schema.

Output

Validated manifest JSON, artifact digests, reference checks, and optional signature verification.

Environment

Manifest generation and validation are local; referenced artifact storage remains under the operator's control.

Scan evaluation data for n-gram overlap, canaries, answer patterns, hashes, and optional embedding similarity.

Multiple local contamination signals are reported together with evidence instead of collapsed into one unsupported certainty score.

Output

Finding records, matched spans or hashes, severity, configuration, summary report, and deterministic fixtures.

Environment

Local checks require no upload; optional embedding behavior depends on the model or endpoint explicitly configured by the operator.

Compare before and after prompt and rubric directories and generate deterministic drift review notes.

Scoring-boundary, criterion, weight, and prompt changes are surfaced as evidence-oriented review items.

Output

Deterministic Markdown report, changed-file evidence, job summary, and idempotent pull-request comment.

Environment

Repository files are compared inside the local or GitHub Actions environment and require no model call.

Generate controlled synthetic annotator disagreement before real reviewer data is collected.

Deterministic disagreement patterns let teams test metric, missingness, severity, and adjudication behavior against known conditions.

Output

Generated labels, seed and scenario metadata, expected disagreement properties, and reproducibility tests.

Environment

Generation is local and uses synthetic records only.

Turn one agent trace into portable Markdown, HTML, and JSON evidence cards.

Goal, outcome, tools, retries, data touched, checks, timing, and remediation remain attached to the same run identity.

Output

Rendered trace card, normalized JSON, referenced artifacts, checksums, and deterministic fixtures.

Environment

Trace files are processed locally; the renderer makes no model or telemetry call.

Normalize agent tool-call traces into deterministic replay artifacts and pytest cases.

Recorded tool inputs and outputs can be locked, redacted, replayed, diffed, and promoted into CI without rerunning the live tool.

Output

Replay bundle, redaction record, generated pytest, expected outputs, and diff report.

Environment

Replay is local by default; live tool access occurs only when the operator explicitly chooses a non-mocked adapter.

Convert OpenTelemetry or Phoenix GenAI trace exports into local evaluation regression cases and manifests.

Operational traces become portable evaluation inputs without requiring a proprietary trace store.

Output

Normalized cases, trace linkage, eval-run-manifest references, result summaries, and deterministic conversion tests.

Environment

Export files are processed locally; attaching results to a remote backend requires an explicitly configured endpoint.

Scan MCP manifests, package metadata, source, and docs for tool, auth, filesystem, shell, network, and readiness risks.

The linter reports inspectable repository evidence and configurable severity instead of claiming runtime security certification.

Output

Markdown, JSON, or SARIF findings, file references, severity threshold, job summary, and optional idempotent comment.

Environment

Scanning runs inside the local or CI repository boundary and requires no model call.

Validate agent cards, capabilities, structured payloads, errors, cancellation, and task lifecycle behavior.

Offline contract fixtures test the public protocol boundary without requiring a live agent deployment.

Output

Agent-card validation, transcript findings, lifecycle report, and deterministic contract fixtures.

Environment

The default suite runs locally against files; no agent endpoint is contacted unless a future adapter is explicitly configured.

Validate and render structured cards for robot morphology, sensors, control, environment, data, and known limitations.

Embodiment-specific release facts stay machine-readable and render into a reviewer-facing card.

Output

Validated card JSON, rendered documentation, schema errors, dataset or model references, and checksums.

Environment

Card generation is local and does not upload robot data or invoke a policy.

Validate and summarize human intervention and recovery segments in robot episodes.

Recovery latency, success, recurrence, operator action, and evidence references stay attached to the source segment.

Output

Validated segment records, aggregate metrics, failure and recovery summaries, and deterministic fixtures.

Environment

Episode metadata is processed locally; media and sensor payloads remain referenced rather than uploaded.

Generate simulator-light perturbation diagnostics for vision-language-action policies.

Task-instruction and observation perturbations produce bounded, reviewable diagnostics without claiming full simulator coverage.

Output

Perturbation manifest, expected and observed behavior, robustness summary, seed, and reproducible fixtures.

Environment

Fixture generation is local; policy inference uses only the adapter and endpoint explicitly configured by the operator.

Index robotics datasets, generate thumbnails, run QA and clustering, and export reviewed evidence through a headless engine.

The CLI exposes the same dataset and evidence contracts used by Robotics Studio Open for automation and integration.

Output

SQLite index, adapter diagnostics, QA findings, clusters, thumbnails, export manifests, and engine tests.

Environment

Datasets are processed locally; remote storage, model, or hosted destinations require explicit adapters.