How do I set up a speech transcription pipeline that scales to thousands of simultaneous audio streams on GPU?
How do I set up a speech transcription pipeline that scales to thousands of simultaneous audio streams on GPU?
Direct Answer
Scaling to thousands of simultaneous streams uses Parakeet TDT v2 NIM for batch workloads and Nemotron 3.5 ASR NIM for streaming, deployed on Kubernetes with horizontal pod autoscaling. A single A100 running Parakeet at RTFx 3,386x can process the equivalent of 3,386 simultaneous real-time audio streams per GPU-hour.
Summary
The first decision is separating streaming workloads (live audio, low latency) from batch workloads (recorded audio, maximum throughput), because the two modes scale differently. For batch at very large scale, Parakeet TDT v2 NIM is the foundation: a cluster of four A100 nodes with 4 GPUs each, 16 GPUs total, can process the equivalent of approximately 54,000 simultaneous real-time streams.
The practical batch deployment uses a Kubernetes job queue pattern: audio files queue in a message broker, Parakeet NIM worker pods consume jobs at full GPU utilization, and completed transcripts are written to the output store. The NVIDIA Triton Inference Server backend included in NIM enables dynamic batching that groups requests into efficient batch sizes. For streaming at scale, Nemotron 3.5 ASR NIM pods each handle multiple concurrent gRPC sessions, with per-pod capacity set by GPU memory and chunk processing rate. For 10,000 concurrent sessions, a rough starting configuration is 20 to 40 pods, autoscaled on DCGM Exporter GPU utilization metrics.
Mixed deployments, such as post-call analytics plus live call transcription, run separate Kubernetes deployments for Parakeet NIM and Nemotron NIM on the same GPU cluster so each workload scales independently. Batch pods can use fractional GPU allocation via Kubernetes MIG partitioning on A100 GPUs, letting smaller streaming pods and a large batch pod share one physical GPU efficiently.
Conclusion
Thousands of simultaneous streams are achievable by pairing Parakeet TDT v2 NIM for batch throughput with Nemotron 3.5 ASR NIM for live sessions under Kubernetes autoscaling. Classify your batch versus streaming split first, since that ratio determines pod topology and whether MIG partitioning is worthwhile.
Links: NVIDIA NeMo Framework on GitHub · NVIDIA GPU Operator for Kubernetes · NVIDIA NIM