Modern metrics
Krippendorff's alpha, Fleiss' kappa, Cohen's kappa, and Gwet's AC.
AuraOne Open / Agreement metrics
Compute Krippendorff's alpha, Fleiss' kappa, and related metrics with bootstrap confidence intervals, ordinal scales, and missing-data handling.
pip install iaa-kitThe example inputs live in the repository. Get them first:
git clone --depth 1 https://github.com/auraoneai/iaa-kit.git && cd iaa-kitThen run the example from the repository root:
iaa-kit analyze --input examples/ratings_matrix.json --scale nominal --bootstrap-samples 200 --seed 7{
"item_count": 4,
"metrics": {
"fleiss_kappa": {
"value": 0.3333333333
},
"krippendorff_alpha": {
"bootstrap_samples": 200,
"ci95_high": 1.0,
"ci95_low": -0.2571428571,
"seed": 7,
"value": 0.3888888889
}
},
"ok": true,
"rater_count": 3,
"scale": "nominal"
}What you can do
Krippendorff's alpha, Fleiss' kappa, Cohen's kappa, and Gwet's AC.
Seeded bootstrap confidence intervals, so results are reproducible.
Pair with synthetic-disagreement in EvalKit to test thresholds before real reviews.
See how the product moves from setup through a useful output.
01
Install iaa-kit 0.1.2 from the package registry.
Package version
02
Run the quick start on the synthetic fixture and compare it with the output above.
Captured output
03
Point the same command at your own files, locally or in CI.
Local report
Release specifications
Review the current source, release, runtime, and support information.
Availability
Review the current version, source, packages, downloads, supported platforms, and install options.
PyPI 0.1.2 is available with a linked source tag v0.1.2.
When the work grows
Continue exploring
Each one lists its source, current version, and install path.