AuraOne Open / LLM judge testing

Test your LLM-as-a-judge for position and verbosity bias.

Probe a judge for position bias, verbosity bias, self-preference, paraphrase sensitivity, anchoring, and calibration, then publish the result with judge-card.

Input

  • A judge backend and model you configure
  • Probes to run and the number of response pairs

Install and run

pip install judge-bench

Run the example. It generates its own synthetic inputs:

judge-bench run --backend local --model local-judge --probes position_bias,calibration --pairs 3 --cache-dir .judge-bench-cache --output judge-report.json
Real outputPosition-bias and calibration probes on the offline local judge (synthetic pairs). judge-bench 0.1.2, captured 2026-11-20, trimmed.
{
  "backend": "local",
  "model": "local-judge",
  "synthetic": true,
  "not_a_benchmark": true,
  "response_pairs": 3,
  "cache": {
    "hits": 3,
    "misses": 6,
    "dir": ".judge-bench-cache"
  },
  "results": [
    {
      "probe": "position_bias",
      "pairs": 3,
      "flip_rate": 0.0,
      "first_position_rate": 1.0,
      "synthetic": true
    },
    {
      "probe": "calibration",
      "pairs": 3,
      "flip_rate": 0.0,
      "ece": 0.0,
      "accuracy": 1.0,
      "reliability_bins": [
        {
          "count": 0,
          "confidence": 0.0,
          "accuracy": 0.0
        },
        {
          "count": 0,
          "confidence": 0.0,
          "accuracy": 0.0
        },
        {
          "count": 0,
          "confidence": 0.0,
…

Limitations

  • The bundled probes use synthetic pairs; they are not a benchmark of any public model.
  • Results on a real judge depend on its provider, version, and prompt.

Included packages

Packages that ship with judge-bench.

  • judge-cardValidate and render a disclosure record for judge prompts, calibration, bias, limitations, and intended use.python -m pip install judge-card

What you can do

Follow the work from input to a usable result.

Bias probes

Swap positions, pad length, and paraphrase answers to see whether verdicts move.

Calibration

Expected calibration error and reliability bins for judge confidence.

Disclosure

judge-card records the prompt, model, and calibration evidence with the result.

judge-bench workflow

See how the product moves from setup through a useful output.

  1. 01

    Install

    Install judge-bench 0.1.2 from the package registry.

    Package version

  2. 02

    Run on the bundled example

    Run the quick start on the synthetic fixture and compare it with the output above.

    Captured output

  3. 03

    Run on your data

    Point the same command at your own files, locally or in CI.

    Local report

Release specifications

What is included in this release

Review the current source, release, runtime, and support information.

Package
judge-bench 0.1.2Discovery snapshot registry readback
Source tag
v0.1.2 (b2133bb5c400)Public GitHub release tag
Sample
src/judge_bench/synthetic_responses.py, captured 2026-11-20public/open/samples.json
Data handling
Fixtures and reports remain local; model calls occur only through the judge provider configured by the operator.Project README

Availability

Current release and install details.

Review the current version, source, packages, downloads, supported platforms, and install options.

Verified

judge-bench availability

PyPI 0.1.2 is available with a linked source tag v0.1.2.

When the work grows

Need an independent model-quality evaluation or benchmark program?

Talk to AuraOne