A quantization job goes silent
A full-spec quantization job stopped making progress after only six blocks. The incident record follows the monitor, idle GPU and checked failure explanations without inventing a confirmed cause.
Key finding
The full-spec job stopped after six of 36 blocks; the monitor detected the dead wrapper and idle GPU.
Edited finding from the pinned experiment record. The report and figure metadata preserve the original workload and limits.
The boundary
Reboot, recorded OOM and contemporaneous GPU faults were excluded. An external kill remains an unconfirmed explanation.
Write-ups
Report moss-tts-hq4/phase1-incident-2026-09/JOURNEY.md
The question
A full-spec quantization job stopped making progress after only six blocks. The incident record follows the monitor, idle GPU and checked failure explanations without inventing a confirmed cause.
From the original notebook
Timeline (NZST, machine clock verified via log correlation)
- 12:19 — Full-spec build launched on GPU 0 via
ops/hq4-full/retry-wrapper-hq4-full.sh(setsid-detached): 256 samples / seqlen 1024 / 200 iters / grad-accum 8 / symmetric W4A16 G128, targetartifacts/MOSS-TTS-v1.5-HQ4-256. Cron monitorMOSSHQ4FULL_HAIKU_MONITORinstalled after a passing manual test. - 12:19–12:47 — Healthy progress: 6/36 blocks, steady ~280–295 s/block, per-block loss converging (e.g. 115.2→9.4, 1.66→0.52), peak RAM 45.6 GB, peak VRAM 19.2 GB. Two monitor checks green, Telegram progress updates sent.
- 12:47:49 — Last log line (
layers.6: 6/36). Process goes silent mid-block-6. - ~14:01 — Scheduled monitor check finds: wrapper dead, quant pid dead,
GPU 0 idle, no
MONITOR_RESULT, log frozen 73 min. Sends Telegram needs-attention alert (case 4: wrapper died before exhausting retries), keeps cron per its instructions. A second attention check follows. - 14:05 — Human-owned investigation (this entry): rules out reboot (uptime continuous), kernel GPU faults (Xid-free window 12:30–14:05, journal readable), kernel + userspace OOM (no signatures). Remaining explanation: external kill from parallel session activity. Killer unidentified.
- 14:0x — Cron entry removed to stop 15-minute alert spam now that a human owns the situation (reversible one-liner, recorded here). No relaunch per explicit user instruction (“don’t quantize nothing else right now”).
Notes
- An earlier
NVRM: Xid 43at 11:50 is unrelated (predates the 12:19 launch; hit parity/envelope-era processes, which completed normally). work/hq4-full/contains onlybackbone_stub/— nothing to salvage or resume.- Open question for the user/peer: did anyone run
kill/pkill(or a cleanup matchingquantize) between 12:47 and 14:00? Answer would close the case.
Keep exploring
Original experiment record. Workstation paths have been generalized. Detailed measurements below retain their original workload and validation boundaries.
Supporting notebooks
More in Speech & codecs
MOSS on a 3090: Q2 failed, AWQ4 stayed closer
Every profile began with the pinned BF16 generator. The comparison reused fixed reference codes and an FP32 decoder for the main quality measurements, keeping codec changes separate from generator quantization.
AWQ4 stayed close to the BF16 reference on WavLM and character errors; the sensitivity Q2 candidate was rejected.
Drama3: BF16 decoded the same codes about three times faster
The experiment replayed saved Spanish and English codes representing 21.5481 and 15.8824 seconds of audio. It compared FP32, BF16, mixed BF16 and two INT8 decoder variants on one RTX 3090.
BF16 reduced median decoder time from 390.07 to 129.54 ms in Spanish and from 292.42 to 96.88 ms in English.
Keeping FP32 and moving Supertonic to Triton
Rather than reduce the CPU graph precision again, this follow-up used NVIDIA Triton’s ONNX backend with CUDA execution and the unchanged FP32 graphs. The paired comparison retained both speed and acoustic checks.
The paired GPU test cut median first-audio time to about 0.35 seconds while preserving the measured ASR results.