Bias probes
Swap positions, pad length, and paraphrase answers to see whether verdicts move.
AuraOne Open / LLM judge testing
Probe a judge for position bias, verbosity bias, self-preference, paraphrase sensitivity, anchoring, and calibration, then publish the result with judge-card.
pip install judge-benchRun the example. It generates its own synthetic inputs:
judge-bench run --backend local --model local-judge --probes position_bias,calibration --pairs 3 --cache-dir .judge-bench-cache --output judge-report.json{
"backend": "local",
"model": "local-judge",
"synthetic": true,
"not_a_benchmark": true,
"response_pairs": 3,
"cache": {
"hits": 3,
"misses": 6,
"dir": ".judge-bench-cache"
},
"results": [
{
"probe": "position_bias",
"pairs": 3,
"flip_rate": 0.0,
"first_position_rate": 1.0,
"synthetic": true
},
{
"probe": "calibration",
"pairs": 3,
"flip_rate": 0.0,
"ece": 0.0,
"accuracy": 1.0,
"reliability_bins": [
{
"count": 0,
"confidence": 0.0,
"accuracy": 0.0
},
{
"count": 0,
"confidence": 0.0,
"accuracy": 0.0
},
{
"count": 0,
"confidence": 0.0,
…Included packages
python -m pip install judge-cardWhat you can do
Swap positions, pad length, and paraphrase answers to see whether verdicts move.
Expected calibration error and reliability bins for judge confidence.
judge-card records the prompt, model, and calibration evidence with the result.
See how the product moves from setup through a useful output.
01
Install judge-bench 0.1.2 from the package registry.
Package version
02
Run the quick start on the synthetic fixture and compare it with the output above.
Captured output
03
Point the same command at your own files, locally or in CI.
Local report
Release specifications
Review the current source, release, runtime, and support information.
Availability
Review the current version, source, packages, downloads, supported platforms, and install options.
PyPI 0.1.2 is available with a linked source tag v0.1.2.
When the work grows
Continue exploring
Each one lists its source, current version, and install path.