Benchmark Runner surface not built yet
The runner substrate already exists as the Python library
under arisekit/. The library handles parallel
inference, retries, resume-by-default checkpointing, cost
tracking, agentic tool loops, and suite-scale compilation —
a benchmark author only writes a dataset loader and a
scorer; everything else is already done.
What is missing is a clinician-facing control plane. This
page will host a three-stage flow that mirrors Case Studio:
a Configure stage that verifies
runnable-benchmark.json
and pins the run knobs (benchmark, models,
trials, parallelism, budget caps, schedule), a Monitor stage
that reads the runner's checkpoint stream and helps the user
decide kill vs. let-it-cook, and an Inspect stage that
surfaces aggregated metrics, per-item drill-down, and
failure clustering.
The architecture of record is at
docs/benchmark-runner.md. The three skills
backing the surface are skills/run-config/,
skills/run-monitor/, and
skills/run-inspect/. Every completed run already
emits run-results.json and
run-results.md. The library substrate is
documented in
docs/design/03-core-concepts.md,
docs/design/05-architecture.md,
docs/design/07-components.md, and
docs/design/09-persistence.md — read those
before extending this surface.