AuraOne Open / Benchmark contamination

Detect benchmark contamination in your evaluation data.

Scan evaluation items for n-gram overlap, canary strings, answer patterns, and hashes, with optional embedding similarity. Every finding names the item and the detector.

Input

  • Evaluation items as JSONL
  • Optional reference corpus and embedding endpoint

Install and run

pip install contamination-audit

The example inputs live in the repository. Get them first:

git clone --depth 1 https://github.com/auraoneai/contamination-audit.git && cd contamination-audit

Then run the example from the repository root:

contamination-audit run --eval-data examples/eval.jsonl --skip-embedding
Real outputItem-level findings for the bundled two-item example (embedding check skipped). contamination-audit 0.1.2, captured 2026-11-20.
{
  "item_count": 2,
  "finding_count": 1,
  "aggregate_risk_score": 0.1,
  "findings": [
    {
      "detector": "answer_pattern",
      "item_id": "1",
      "match": "The answer is (B)"
    }
  ],
  "synthetic": true,
  "reproducible": true
}

Limitations

  • Signals indicate risk; a finding is not proof of training-data exposure.
  • Embedding similarity runs only with a model or endpoint you configure.

What you can do

Follow the work from input to a usable result.

Lexical detectors

N-gram overlap, canaries, hashes, and answer-format patterns.

Item-level evidence

Each finding names the item, detector, and matched text.

Reproducible runs

Deterministic output for the same inputs and settings.

contamination-audit workflow

See how the product moves from setup through a useful output.

  1. 01

    Install

    Install contamination-audit 0.1.2 from the package registry.

    Package version

  2. 02

    Run on the bundled example

    Run the quick start on the synthetic fixture and compare it with the output above.

    Captured output

  3. 03

    Run on your data

    Point the same command at your own files, locally or in CI.

    Local report

Release specifications

What is included in this release

Review the current source, release, runtime, and support information.

Package
contamination-audit 0.1.2Discovery snapshot registry readback
Source tag
v0.1.2 (743f1774583c)Public GitHub release tag
Sample
examples/eval.jsonl, captured 2026-11-20public/open/samples.json
Data handling
Local checks require no upload; optional embedding behavior depends on the model or endpoint explicitly configured by the operator.Project README

Availability

Current release and install details.

Review the current version, source, packages, downloads, supported platforms, and install options.

Verified

contamination-audit availability

PyPI 0.1.2 is available with a linked source tag v0.1.2.

When the work grows

Need an independent model-quality evaluation or benchmark program?

Talk to AuraOne