Moving speech decode onto the GPU
Moving the final audio decode onto the GPU shortened both that stage and total render time. The deployment A/B also records the extra GPU memory, so the speed improvement can be weighed against its cost.
Key finding
Decode dropped from 3.36 seconds to 0.24; total render wall time dropped from 35.1 to 24.4 seconds.
Edited finding from the pinned experiment record. The report and figure metadata preserve the original workload and limits.
The boundary
The decoder adds resident GPU memory. Reference encoding stays FP32 on CPU; the seed-matched A/B does not cover every voice.
Data
Reviewed aggregate results with explicit units, sources and qualification limits. Different cohorts remain separate.
GPU decode shortened the final stage
Seed-matched recorded deployment A/B · FP32 reference encoder on CPU
- CPU INT8 decode3.36
- GPU FP8 decode0.24
seconds
Report-table values. The archived before JSON is empty; these are not presented as recovered raw logs.
View data & source
| Configuration | Value |
|---|---|
| CPU INT8 decode | 3.36 |
| GPU FP8 decode | 0.24 |
Origin: reported table. Revision 6550ead3945b.
End-to-end render wall time
Seed-matched recorded deployment A/B · FP32 reference encoder on CPU
- Before35.10
- After24.40
seconds
Report-table values. The archived before JSON is empty; these are not presented as recovered raw logs.
View data & source
| Configuration | Value |
|---|---|
| Before | 35.10 |
| After | 24.40 |
Origin: reported table. Revision 6550ead3945b.
The faster decoder added GPU memory
Seed-matched recorded deployment A/B · FP32 reference encoder on CPU
- Before6.77
- After8.01
GiB
Report-table values. The archived before JSON is empty; these are not presented as recovered raw logs.
View data & source
| Configuration | Value |
|---|---|
| Before | 6.77 |
| After | 8.01 |
Origin: reported table. Revision 6550ead3945b.
Write-ups
Report moss-tts-awq/fp8-codec-deploy-2026-09-22/README.md
The question
Moving the final audio decode onto the GPU shortened both that stage and total render time. The deployment A/B also records the extra GPU memory, so the speed improvement can be weighed against its cost.
From the original notebook
Took the FP8 codec validated in codec-roundtrip-fp8-2026-09-21 and deployed it in the live production daemon on the .54 RTX 3060: the artifact dequantizes to BF16 at load, the encoder half is dropped (decode-only, ~1.6 GiB), and the decoder stays resident on cuda:0 next to the warm generator (6.37 GiB). Reference encoding stays on the FP32/CPU codec (the validated clone fix; the FP8 encoder remains untested for conditioning).
Production code: groxaxo/moss54 commit 40ac7f0
(ops/moss54d/moss54d.py, flag --codec-fp8).
Measured (A/B, seed 1234, same reference, .54 RTX 3060 12 GB)
| Metric | Before (CPU fp32 decode) | After (FP8 GPU decode) |
|---|---|---|
| decode_time_s (clone) | 3.36 | 0.24 (14×) |
| decode_time_s (default voice) | 2.76 | 0.23 |
| gen_time_s | 22.9 | 22.91–22.94 (unchanged) |
| wall_s per render (clone) | 35.1 | 24.4 |
| new_tokens / duration_s | 193 / 12.72 | identical |
| audio_health flags | all false | all false |
| daemon VRAM steady | 6.77→8.01 GiB torch-reserved (net +1.6) |
Verification per the golden protocol: identical sample counts and token/duration parity (PCM mean-abs diff 0.0018 — codec-precision level); ASR A/B (whisperx medium es) identical transcripts before/after in both voices (0.0625/0.0625, 0.0938/0.0938 — residual = transcriber mishears present in both). First post-restart render shows warmup-inflated gen_time; repeat runs are the reportable number.
Contents
docs/before_results.json,docs/after_results.json— raw A/B metrics.docs/wer_ab.json— whisperx transcripts + WER, all four files.samples/— the four A/B WAVs.scripts/render_ab.py— the exact harness (POST /v1/audio/speech).
Boundaries
Single seed, one reference, one text per voice; decode speedup scales with audio length. RTF gen unchanged (7–8 tok/s). The FP8 encoder path is deliberately NOT exercised. No blind ABX listening — parity rests on the round-trip shootout + identical ASR transcripts + unchanged speaker path.
Keep exploring
Original experiment record. Workstation paths have been generalized. Detailed measurements below retain their original workload and validation boundaries.
Supporting notebooks
More in Speech & codecs
MOSS on a 3090: Q2 failed, AWQ4 stayed closer
Every profile began with the pinned BF16 generator. The comparison reused fixed reference codes and an FP32 decoder for the main quality measurements, keeping codec changes separate from generator quantization.
AWQ4 stayed close to the BF16 reference on WavLM and character errors; the sensitivity Q2 candidate was rejected.
Drama3: BF16 decoded the same codes about three times faster
The experiment replayed saved Spanish and English codes representing 21.5481 and 15.8824 seconds of audio. It compared FP32, BF16, mixed BF16 and two INT8 decoder variants on one RTX 3090.
BF16 reduced median decoder time from 390.07 to 129.54 ms in Spanish and from 292.42 to 96.88 ms in English.
Keeping FP32 and moving Supertonic to Triton
Rather than reduce the CPU graph precision again, this follow-up used NVIDIA Triton’s ONNX backend with CUDA execution and the unchanged FP32 graphs. The paired comparison retained both speed and acoustic checks.
The paired GPU test cut median first-audio time to about 0.35 seconds while preserving the measured ASR results.