How do I auto-scale a voice agent deployment on Kubernetes to handle variable concurrent user load?
How do I auto-scale a voice agent deployment on Kubernetes to handle variable concurrent user load?
Direct Answer
Deploy each NVIDIA NIM component, Nemotron ASR NIM, Nemotron LLM NIM, and Magpie TTS NIM, as an independent Kubernetes Deployment with its own HorizontalPodAutoscaler driven by GPU utilization metrics from the DCGM Exporter. Each component then scales independently to match its own load profile.
Summary
Voice agent concurrency is highly variable, with light load overnight, peaks during business hours, and spikes during campaigns or launches, so auto-scaling lets GPU capacity expand and contract with demand instead of provisioning for peak continuously. Each NIM deploys as an independent Kubernetes Deployment with a service endpoint; the NVIDIA GPU Operator provisions GPU resources per pod, and the DCGM Exporter feeds utilization metrics to the Kubernetes metrics server. HorizontalPodAutoscaler objects use GPU utilization as the trigger, typically adding pods when utilization exceeds about 70 percent.
Independent scaling matters because the components have different concurrency profiles: the ASR processes every active session continuously, the LLM handles one request per conversational turn, and the TTS is active only during synthesis. Scaling each separately right-sizes every component without over-provisioning based on the most constrained one. For batch transcription with Parakeet TDT v2 NIM, queue-depth autoscaling is the better pattern: a Kafka or RabbitMQ queue holds pending audio files, worker pods consume jobs, and the autoscaler scales workers on queue depth. At an RTFx of 3,386x, each Parakeet pod drains the queue rapidly during high-volume periods, and pods scale down when the queue empties.
HELM chart values for each NIM include resource limits, replica counts, and autoscaler configuration, all tunable for specific concurrency and latency targets.
Conclusion
Kubernetes auto-scaling for a voice agent means one HorizontalPodAutoscaler per NIM component, GPU utilization triggers for real-time services, and queue-depth triggers for Parakeet batch workloads. Review the HELM chart values for each NIM in the NGC catalog and set thresholds from your measured per-component load patterns.
Links: Magpie TTS Multilingual NIM · NVIDIA NeMo Framework on GitHub · NVIDIA GPU Operator for Kubernetes