AuraOne Open / Agreement metrics

Inter-annotator agreement with confidence intervals.

Compute Krippendorff's alpha, Fleiss' kappa, and related metrics with bootstrap confidence intervals, ordinal scales, and missing-data handling.

Input

  • A ratings matrix: items by raters, missing values allowed
  • Scale type: nominal, ordinal, interval, or ratio

Install and run

pip install iaa-kit

The example inputs live in the repository. Get them first:

git clone --depth 1 https://github.com/auraoneai/iaa-kit.git && cd iaa-kit

Then run the example from the repository root:

iaa-kit analyze --input examples/ratings_matrix.json --scale nominal --bootstrap-samples 200 --seed 7
Real outputAlpha and kappa for the bundled four-item, three-rater example, with a 95% bootstrap interval. iaa-kit 0.1.2, captured 2026-11-20.
{
  "item_count": 4,
  "metrics": {
    "fleiss_kappa": {
      "value": 0.3333333333
    },
    "krippendorff_alpha": {
      "bootstrap_samples": 200,
      "ci95_high": 1.0,
      "ci95_low": -0.2571428571,
      "seed": 7,
      "value": 0.3888888889
    }
  },
  "ok": true,
  "rater_count": 3,
  "scale": "nominal"
}

Limitations

  • Small item counts produce wide intervals; the output reports them rather than hiding them.
  • Agreement measures consistency, not correctness.

What you can do

Follow the work from input to a usable result.

Modern metrics

Krippendorff's alpha, Fleiss' kappa, Cohen's kappa, and Gwet's AC.

Uncertainty

Seeded bootstrap confidence intervals, so results are reproducible.

Synthetic disagreement

Pair with synthetic-disagreement in EvalKit to test thresholds before real reviews.

iaa-kit workflow

See how the product moves from setup through a useful output.

  1. 01

    Install

    Install iaa-kit 0.1.2 from the package registry.

    Package version

  2. 02

    Run on the bundled example

    Run the quick start on the synthetic fixture and compare it with the output above.

    Captured output

  3. 03

    Run on your data

    Point the same command at your own files, locally or in CI.

    Local report

Release specifications

What is included in this release

Review the current source, release, runtime, and support information.

Package
iaa-kit 0.1.2Discovery snapshot registry readback
Source tag
v0.1.2 (e35ecf2d68e7)Public GitHub release tag
Sample
examples/ratings_matrix.json, captured 2026-11-20public/open/samples.json
Data handling
Annotation arrays are processed locally and no network access is required.Project README

Availability

Current release and install details.

Review the current version, source, packages, downloads, supported platforms, and install options.

Verified

iaa-kit availability

PyPI 0.1.2 is available with a linked source tag v0.1.2.

When the work grows

Need expert reviewers to calibrate or score this evaluation?

Talk to AuraOne