A faster speech encoder with the same 50 transcripts

Could an RTX 3060 transcribe the same speech with less GPU memory by accelerating the encoder while retaining the existing decoder?

The question

Could an RTX 3060 transcribe the same speech with less GPU memory by accelerating the encoder while retaining the existing decoder?

The comparison used CUDA FP32 throughout against a TensorRT FP16 encoder plus CUDA FP32 decoder. Both produced exactly the same transcripts on all 50 clips, with 26 edits over 737 reference words: 3.53% WER.

Timing and memory

CUDA FP32 measured 128.80 times real time and 3.06 GiB of additional GPU memory. The TensorRT encoder route measured 159.76 times real time and 1.93 GiB. Three timing repetitions per clip were reduced to per-clip medians before totals were calculated. Profiling was separate from timing, and execution-provider evidence confirmed which graph actually ran.

A quiet-first-word boundary bug was found and repaired during integration. Failed earlier evidence remains in the archive. Twenty-four unit checks and sixteen deployed API checks passed for the qualified route.

What this establishes

The encoder swap improved this offline workload without changing its recognized words. Noisy microphones, overlapping speakers, other languages and real-time interruption handling still require their own tests.

Evidence and publication boundary

This is an edited account of the recorded experiment. The linked archive is private; the methods, aggregate findings and limits are included above. Reference recordings, raw conversations, authentication material and workstation configuration are excluded.