MOSS: the best similarity score did not mean the fewest word errors

Porting the evaluation method

Nine generator profiles were tested against BF16 plus the same Argentina LoRA. The English study retained ninety complete generations, controlled decoder runs and a timing recheck. A separate Spanish extension used one generation per phrase.

Two metrics, different answers

The directly-from-BF16 Q6/Q8 profile had 2/65 Spanish ASR edits and numerical weight RMS error 2.127%. Older Q4 had 9/65 edits but higher Spanish speaker cosine, 0.95181 versus 0.91631. A Q4/Q5/Q6 mix had a fast 8.167-second English warm median yet 47/65 Spanish edits.

All controlled decoder outputs matched because every generator profile reused the same codec and codes. That verifies the pipeline control, while saying nothing about which quantized generator preserves the best voice.

Preserve the uncomfortable evidence

Forty-two of ninety complete English clips approached the 400-token ceiling. The original reference encountered severe memory pressure and repeated words. Spanish BF16 generation and decoding had to run in separate processes after failed attempts, so that extension has no comparable full-pipeline latency claim.

The useful finding is a trade-off between storage, content and identity proxies. Larger frozen corpora and human listening should precede a general recommendation.

Evidence and publication boundary

This is an edited account of the recorded experiment. The linked archive is private; the methods, aggregate findings and limits are included above. Reference recordings, raw conversations, authentication material and workstation configuration are excluded.

• Source record 1