A benchmark harness that keeps dry runs out of the results

Make the measurement boundary explicit

The new harness layer provides four kinds of runs: streamed API throughput, GGUF quant comparison, task evaluation through lm-eval, and command-level memory/load measurement. It uses standard-library Python for the core, with optional external model and evaluation tools.

Preserve enough context to inspect a result

Each real run writes a summary plus structured JSON and CSV. Hardware, runtime, workload, units and energy status travel with the rows. An unavailable power measurement stays unmeasured; an electricity estimate needs measured energy and an explicit tariff.

Synthetic fixture runs go to a separate dry-run directory and carry a dry-run label. They test parsers, schema and output plumbing. They cannot support a chart about a real model or GPU.

The publishing rule

This blog does not recursively publish every newly created run folder. A record receives an editorial review, source revision, aggregate-only exports and the public-content gate before deployment. API credentials stay in environment variables and private experiment infrastructure stays outside the site.

The value of the harness is a repeatable reporting format. Its example commands do not establish fresh performance or accuracy results.

Evidence and publication boundary

This is an edited account of the recorded experiment. The linked archive is private; the methods, aggregate findings and limits are included above. Reference recordings, raw conversations, authentication material and workstation configuration are excluded.

• Source record 1