A speech service that sleeps between requests
A speech service need not hold all of its resources between requests. The lazy-server experiment checked wake-up, warm rendering, reap and re-wake, while preserving the known short-prompt silence behavior.
Key finding
The recorded service used about 10.3GB resident memory; first render took 14.8 seconds and warm render 5.7.
Edited finding from the pinned experiment record. The report and figure metadata preserve the original workload and limits.
The boundary
Cold and warm timings describe different service states. Short-prompt silence existed in both FP32 and INT8 paths.
Data
Reviewed aggregate results with explicit units, sources and qualification limits. Different cohorts remain separate.
First request and warm request are different states
HQ4 + INT8 codec lazy service · recorded request timings
- Cold first render14.8
- Warm render5.7
seconds
Cold start includes service wake-up. This is not two equivalent decoding workloads.
View data & source
| Configuration | Value |
|---|---|
| Cold first render | 14.8 |
| Warm render | 5.7 |
Origin: reported live-service timings. Revision 6550ead3945b.
Write-ups
Report moss-tts-hq4/int8-codec-lazy-server-2026-09-19/README.md
The question
A speech service need not hold all of its resources between requests. The lazy-server experiment checked wake-up, warm rendering, reap and re-wake, while preserving the known short-prompt silence behavior.
From the original notebook
Host: research workstation, GPU 2 (RTX 3090 24GB).
Replaces the MOSS-TTS v1.5 HQ4 + FP32-codec server with HQ4 + full INT8-RTN
codec behind a lazy gate. Public contract unchanged (moss-8b,
POST /v1/audio/speech → WAV on :9840).
What
- INT8-RTN codec master (
weight_int8I8 + per-channel F32 scale, 620 modules) staged at$WORKSTATION/MOSS-TTS/weights/MOSS-Audio-Tokenizer-INT8. - Server dequantizes inline at load: 554 linears → BF16, 66 weight-norm convs → exact F32 g/v re-split, codebooks stay F32. Strict key check (1600/1600 vs FP32 reference); no-op for FP32/BF16 dirs.
- Codec encode/decode run under BF16 autocast when INT8 is active (fixes
the
float vs BFloat16matmul mismatch; FP32 path bit-identical). - Backend moved to loopback
localhost, autostart off; lazy gate on:9840starts it on first request, reaps after 300s idle. - HQ4 weights reused in place: safetensors sha256-verified identical to the published artifact (Hub repo is private, no token available).
Measured
- Resident: ~10.3GB (was ~13.9GB FP32).
- INT8 renders healthy (male/short/female-clone); gate cold start 14.8s first render, 5.7s warm; reap + re-wake verified.
- Note: 7-char prompts sometimes render mostly silence on FP32 and INT8 alike (sampling variance, pre-existing).
Files
moss_server_cuda.py— server (deployed to$WORKSTATION/moss-server/).moss-tts-cuda.service— backend unit (loopback:17790).moss-tts-lazy-proxy.service— gate unit (:9840→:17790).dequant_codec_int8_to_bf16.py— provenance tool: builds a loadable BF16 runtime from the INT8 master (unused at runtime; server dequants inline because this host’s disk was full).verify_int8_rt.py— CPU check of the server’s exact codec load path.
Redeploy
cp moss_server_cuda.py $WORKSTATION/moss-server/moss_server_cuda.py
cp moss-tts-cuda.service moss-tts-lazy-proxy.service $WORKSTATION/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user disable moss-tts-cuda.service
systemctl --user enable --now moss-tts-lazy-proxy.serviceKeep exploring
Original experiment record. Workstation paths have been generalized. Detailed measurements below retain their original workload and validation boundaries.
Supporting notebooks
More in Speech & codecs
MOSS on a 3090: Q2 failed, AWQ4 stayed closer
Every profile began with the pinned BF16 generator. The comparison reused fixed reference codes and an FP32 decoder for the main quality measurements, keeping codec changes separate from generator quantization.
AWQ4 stayed close to the BF16 reference on WavLM and character errors; the sensitivity Q2 candidate was rejected.
Drama3: BF16 decoded the same codes about three times faster
The experiment replayed saved Spanish and English codes representing 21.5481 and 15.8824 seconds of audio. It compared FP32, BF16, mixed BF16 and two INT8 decoder variants on one RTX 3090.
BF16 reduced median decoder time from 390.07 to 129.54 ms in Spanish and from 292.42 to 96.88 ms in English.
Keeping FP32 and moving Supertonic to Triton
Rather than reduce the CPU graph precision again, this follow-up used NVIDIA Triton’s ONNX backend with CUDA execution and the unchanged FP32 graphs. The paired comparison retained both speed and acoustic checks.
The paired GPU test cut median first-audio time to about 0.35 seconds while preserving the measured ASR results.