A faster speech encoder with the same 50 transcripts
Could an RTX 3060 transcribe the same speech with less GPU memory by accelerating the encoder while retaining the existing decoder?
Key finding
TensorRT FP16 encoding raised measured throughput from 128.80× to 159.76× real time with identical transcripts.
Edited finding from the pinned experiment record. The report and figure metadata preserve the original workload and limits.
The boundary
50 clean English, single-speaker clips: 304.515 seconds and 737 reference words. Warm aggregate throughput is not streaming conversation latency.
Data
Reviewed aggregate results with explicit units, sources and qualification limits. Different cohorts remain separate.
A faster encoder with unchanged transcripts
50 clean English clips · 304.515 s · sum of per-clip warm medians
- CUDA FP32128.80
- TRT FP16 encoder + FP32 decoder159.76
times real time
Three timing repeats per clip. Aggregate offline speed is not first-word or conversation latency.
View data & source
| Configuration | Value |
|---|---|
| CUDA FP32 | 128.80 |
| TRT FP16 encoder + FP32 decoder | 159.76 |
Origin: reviewed report table. Revision 7e6fdbe3b1d1.
Less additional GPU memory
Same RTX 3060 transcription workload
- CUDA FP323.06
- TRT FP16 encoder + FP32 decoder1.93
additional GiB
Memory above baseline, not full driver or system memory. Both routes made the same 26/737 word edits.
View data & source
| Configuration | Value |
|---|---|
| CUDA FP32 | 3.06 |
| TRT FP16 encoder + FP32 decoder | 1.93 |
Origin: reviewed report table. Revision 7e6fdbe3b1d1.
Write-ups
Report speech-to-text/parakeet-tensorrt-rtx3060-2026-10-05/README.md
The question
Could an RTX 3060 transcribe the same speech with less GPU memory by accelerating the encoder while retaining the existing decoder?
The comparison used CUDA FP32 throughout against a TensorRT FP16 encoder plus CUDA FP32 decoder. Both produced exactly the same transcripts on all 50 clips, with 26 edits over 737 reference words: 3.53% WER.
Timing and memory
CUDA FP32 measured 128.80 times real time and 3.06 GiB of additional GPU memory. The TensorRT encoder route measured 159.76 times real time and 1.93 GiB. Three timing repetitions per clip were reduced to per-clip medians before totals were calculated. Profiling was separate from timing, and execution-provider evidence confirmed which graph actually ran.
A quiet-first-word boundary bug was found and repaired during integration. Failed earlier evidence remains in the archive. Twenty-four unit checks and sixteen deployed API checks passed for the qualified route.
What this establishes
The encoder swap improved this offline workload without changing its recognized words. Noisy microphones, overlapping speakers, other languages and real-time interruption handling still require their own tests.
Evidence and publication boundary
This is an edited account of the recorded experiment. The linked archive is private; the methods, aggregate findings and limits are included above. Reference recordings, raw conversations, authentication material and workstation configuration are excluded.
More in Speech & codecs
MOSS on a 3090: Q2 failed, AWQ4 stayed closer
Every profile began with the pinned BF16 generator. The comparison reused fixed reference codes and an FP32 decoder for the main quality measurements, keeping codec changes separate from generator quantization.
AWQ4 stayed close to the BF16 reference on WavLM and character errors; the sensitivity Q2 candidate was rejected.
Drama3: BF16 decoded the same codes about three times faster
The experiment replayed saved Spanish and English codes representing 21.5481 and 15.8824 seconds of audio. It compared FP32, BF16, mixed BF16 and two INT8 decoder variants on one RTX 3090.
BF16 reduced median decoder time from 390.07 to 129.54 ms in Spanish and from 292.42 to 96.88 ms in English.
Keeping FP32 and moving Supertonic to Triton
Rather than reduce the CPU graph precision again, this follow-up used NVIDIA Triton’s ONNX backend with CUDA execution and the unchanged FP32 graphs. The paired comparison retained both speed and acoustic checks.
The paired GPU test cut median first-audio time to about 0.35 seconds while preserving the measured ASR results.