Skip to content
exExperimentos
← Lab Notes
Free8 Oct 2026 · 1 min read

An unfinished gauntlet cannot name a winner

This campaign compared Qwen UD-Q4_K_XL with MTP2 against Bonsai through coding harnesses. It stopped at the 70 GiB free-space guard, retaining the incomplete evidence rather than risking the workstation.

The comparison that actually finished

This campaign compared Qwen UD-Q4_K_XL with MTP2 against Bonsai through coding harnesses. It stopped at the 70 GiB free-space guard, retaining the incomplete evidence rather than risking the workstation.

Qwen finished 18 cells with nine passes. Bonsai finished 15 with six passes. The three extra Qwen passes were unpaired, so comparing 9/18 with 6/15 would tell the wrong story.

Matched evidence

On the 15 cells available for both, each model passed six and failed nine. Qwen made 564 API requests versus 823 and used 525,627 output tokens versus 768,296. Recorded API duration was 16,789.1 versus 19,805 seconds; effective output rates were 31.31 versus 38.79 tokens/s. Fewer tokens and requests can outweigh a lower decode rate on a task.

These are observed system outcomes under different runtime and sampling conditions. They do not isolate model weights, MTP or a single harness as the cause.

What remains

Resume only after restoring the storage budget, preserve the matched-cell protocol, and report the missing cells explicitly. There is no broad accuracy winner in this partial campaign.

Evidence and publication boundary

This is an edited account of the recorded experiment. The linked archive is private; the methods, aggregate findings and limits are included above. Reference recordings, raw conversations, authentication material and workstation configuration are excluded.

Source: qwen38/gauntlet-udq4xl-mtp2-vs-bonsai-2026-10-08/README.md

Free Lab Notes

Follow the next experiment

Get new findings in your feed reader. All published articles and measurement exports are free to read.

Follow via RSS