AuraOne Open / Evaluation CLI

Offline LLM evaluation from the command line and CI.

Validate rubrics, score outputs, check reviewer agreement and leakage, and write portable reports. Runs locally or inside GitHub Actions.

Input

  • Rubric JSONL with weighted criteria
  • Model outputs as JSONL
  • Criterion labels from reviewers or a configured judge

Install and run

pip install auraone-evalkit

The example inputs live in the repository. Get them first:

git clone --depth 1 https://github.com/auraoneai/open.git && cd open

Then run the example from the repository root:

evalkit score --rubric packages/evalkit/examples/tutorial/rubric.jsonl --responses packages/evalkit/examples/tutorial/model_outputs.jsonl --labels packages/evalkit/examples/tutorial/labels.jsonl --format csv
- uses: auraoneai/[email protected]
  with:
    rubric-path: evals/rubric.jsonl
    responses-path: evals/outputs.jsonl
    labels-path: evals/labels.jsonl
Real outputScores for three outputs from the bundled tutorial fixture. auraone-evalkit 0.3.0, captured 2026-11-20.
output_id,score,passed,applicable_weight,weighted_points,missing_criteria
out-001,0.8375,True,1.0,0.8375,
out-002,0.3,False,1.0,0.3,
out-003,0.8,True,1.0,0.8,

Limitations

  • EvalKit scores the labels you supply; it does not generate them.
  • Model-judge labeling depends on the provider and credentials you configure.
  • Reports are local files; sharing them is up to you.

Included packages

Packages that ship with AuraOne EvalKit.

  • rubric-specValidate, lint, diff, and convert portable AuraOne Rubric Schema v1 files.python -m pip install rubric-spec
  • eval-adapterTranslate one rubric-spec rubric and run configuration into supported evaluation-framework inputs and exports.python -m pip install eval-adapter
  • eval-conformance-suiteRun executable rubric-spec v1 compatibility checks and emit reproducible conformance evidence.python -m pip install eval-conformance-suite
  • eval-run-manifestCreate and validate a portable envelope for evaluation provenance, environment, artifacts, result digests, and optional signatures.python -m pip install eval-run-manifest
  • synthetic-disagreementGenerate controlled synthetic annotator disagreement before real reviewer data is collected.python -m pip install synthetic-disagreement

What you can do

Follow the work from input to a usable result.

Rubric validation

Lint, diff, and convert portable rubrics before they reach a release gate.

Deterministic scoring

Weighted criterion scores with pass/fail thresholds and missing-criterion reporting.

Pull request gate

EvalKit Action posts one consistent check on every pull request.

AuraOne EvalKit workflow

See how the product moves from setup through a useful output.

  1. 01

    Install

    Install auraone-evalkit 0.3.0 from the package registry.

    Package version

  2. 02

    Run on the bundled example

    Run the quick start on the synthetic fixture and compare it with the output above.

    Captured output

  3. 03

    Run on your data

    Point the same command at your own files, locally or in CI.

    Local report

Release specifications

What is included in this release

Review the current source, release, runtime, and support information.

Package
auraone-evalkit 0.3.0Discovery snapshot registry readback
Source tag
v0.3.0 (60ad0ab1ad7a)Public GitHub release tag
Sample
packages/evalkit/examples/tutorial/, captured 2026-11-20public/open/samples.json
Data handling
Inputs and reports remain local or inside the operator's CI environment unless the operator uploads them elsewhere.Project README

Availability

Current release and install details.

Review the current version, source, packages, downloads, supported platforms, and install options.

Verified

AuraOne EvalKit availability

PyPI 0.3.0 is available with a linked source tag v0.3.0.

When the work grows

Need expert reviewers to calibrate or score this evaluation?

Talk to AuraOne