A benchmark harness that keeps dry runs out of the results
The new harness layer provides four kinds of runs: streamed API throughput, GGUF quant comparison, task evaluation through lm-eval, and command-level memory/load measurement. It uses standard-library Python for the core, with optional external model and…
Key finding
The archive now includes standard-library harnesses for throughput, quant comparison, task evaluation and VRAM measurement.
Edited finding from the pinned experiment record. The report and figure metadata preserve the original workload and limits.
The boundary
This is tooling documentation. Synthetic fixtures and example configurations are not measured GPU results.
Write-ups
Report bench/README.md
Make the measurement boundary explicit
The new harness layer provides four kinds of runs: streamed API throughput, GGUF quant comparison, task evaluation through lm-eval, and command-level memory/load measurement. It uses standard-library Python for the core, with optional external model and evaluation tools.
Preserve enough context to inspect a result
Each real run writes a summary plus structured JSON and CSV. Hardware, runtime, workload, units and energy status travel with the rows. An unavailable power measurement stays unmeasured; an electricity estimate needs measured energy and an explicit tariff.
Synthetic fixture runs go to a separate dry-run directory and carry a dry-run label. They test parsers, schema and output plumbing. They cannot support a chart about a real model or GPU.
The publishing rule
This blog does not recursively publish every newly created run folder. A record receives an editorial review, source revision, aggregate-only exports and the public-content gate before deployment. API credentials stay in environment variables and private experiment infrastructure stays outside the site.
The value of the harness is a repeatable reporting format. Its example commands do not establish fresh performance or accuracy results.
Evidence and publication boundary
This is an edited account of the recorded experiment. The linked archive is private; the methods, aggregate findings and limits are included above. Reference recordings, raw conversations, authentication material and workstation configuration are excluded.