AuraOne Open / Agent regression testing

Replay recorded agent tool calls as pytest regressions.

Turn a recorded agent trace into a canonical replay file, assert tool order, retries, and answers, and generate a pytest case that runs offline.

Input

  • Recorded tool-call trace as JSONL
  • Assertions in YAML: tool order, forbidden tools, retries, answer text

Install and run

pip install tool-call-replay

The example inputs live in the repository. Get them first:

git clone --depth 1 https://github.com/auraoneai/tool-call-replay.git && cd tool-call-replay

Then run the example from the repository root:

tool-call-replay ingest examples/recovered_trace.jsonl --out refund-replay.json
tool-call-replay run refund-replay.json --assert examples/refund_assertions.yaml --report /dev/stdout
tool-call-replay pytest refund-replay.json --assert examples/refund_assertions.yaml --out test_refund_replay.py
Real outputReplay report for the bundled refund trace: four assertions, all passing. tool-call-replay 0.1.1, captured 2026-11-20.
# Tool Call Replay Report

- Trace: `refund-fixed-001`
- Goal: Refund order A100 without issuing duplicate credits.
- Events: `5`
- Assertions: `4`

## Assertions

| Result | Assertion | Detail |
| --- | --- | --- |
| pass | `tool_order` | observed=['lookup_order', 'issue_refund'] expected=['lookup_order', 'issue_refund'] |
| pass | `final_answer_contains` | expected substring `once` in `Refund completed once.` |
| pass | `max_retries` | retry counts={'lookup_order': 1, 'issue_refund': 1} max_retries=0 |
| pass | `forbidden_tools` | forbidden used=[] |

## Sequence

```mermaid
sequenceDiagram
participant Agent
Agent->>Tool: lookup_order
Tool-->>Agent: lookup_order
Agent->>Tool: issue_refund
Tool-->>Agent: issue_refund
Agent-->>User: final answer
```
Generated testtest_refund_replay.py
import json
from tool_call_replay.replay import ReplayCase, ReplayEvent, run_assertions


def test_tool_call_replay_regression():
    payload = json.loads('{"events": [{"arguments": {"order_id": "A100"}, "error": null, "output": null, "tool_name": "lookup_order", "type": "tool_call"}, {"arguments": {}, "error": null, "output": {"paid": true, "status": "delivered"}, "tool_name": "lookup_order", "type": "tool_result"}, {"arguments": {"amount": 42.5, "order_id": "A100"}, "error": null, "output": null, "tool_name": "issue_refund", "type": "tool_call"}, {"arguments": {}, "error": null, "output": {"refund_id": "R1", "status": "created"}, "tool_name": "issue_refund", "type": "tool_result"}, {"arguments": {}, "error": null, "output": "Refund completed once.", "tool_name": null, "type": "final_answer"}], "final_answer": "Refund completed once.", "goal": "Refund order A100 without issuing duplicate credits.", "schema_version": "tool-call-replay/v1", "trace_id": "refund-fixed-001"}')
    case = ReplayCase(
        schema_version=payload["schema_version"],
        trace_id=payload["trace_id"],
        goal=payload["goal"],
        events=[ReplayEvent(event_type=item["type"], tool_name=item.get("tool_name"), arguments=item.get("arguments"), output=item.get("output"), error=item.get("error")) for item in payload["events"]],
        final_answer=payload.get("final_answer"),
    )
    results = run_assertions(case, json.loads('{"final_answer_contains": "once", "forbidden_tools": ["delete_order"], "max_retries": 0, "tool_order": ["lookup_order", "issue_refund"]}'))
    assert all(result.ok for result in results), [result.detail for result in results if not result.ok]

Limitations

  • Replays recorded structure and outputs; it does not re-run the model.
  • Live tool access happens only if you pick a non-mocked adapter.
  • Redact secrets in traces before committing replay files.

What you can do

Follow the work from input to a usable result.

Normalize traces

Convert recorded events into one versioned replay format.

Assert behavior

Check tool order, forbidden tools, retry limits, and final-answer content.

Generate pytest

Emit a self-contained test that fails if the behavior regresses.

tool-call-replay workflow

See how the product moves from setup through a useful output.

  1. 01

    Install

    Install tool-call-replay 0.1.1 from the package registry.

    Package version

  2. 02

    Run on the bundled example

    Run the quick start on the synthetic fixture and compare it with the output above.

    Captured output

  3. 03

    Run on your data

    Point the same command at your own files, locally or in CI.

    Local report

Release specifications

What is included in this release

Review the current source, release, runtime, and support information.

Package
tool-call-replay 0.1.1Discovery snapshot registry readback
Source tag
v0.1.1 (9916373e05dc)Public GitHub release tag
Sample
examples/recovered_trace.jsonl, captured 2026-11-20public/open/samples.json
Data handling
Replay is local by default; live tool access occurs only when the operator explicitly chooses a non-mocked adapter.Project README

Availability

Current release and install details.

Review the current version, source, packages, downloads, supported platforms, and install options.

Verified

tool-call-replay availability

PyPI 0.1.1 is available with a linked source tag v0.1.1.

When the work grows

Turn recurring agent failures into a managed improvement program.

Talk to AuraOne