How do I deploy a speech AI service as a microservice on Kubernetes with horizontal pod autoscaling?
How do I deploy a speech AI service as a microservice on Kubernetes with horizontal pod autoscaling?
Direct Answer
NVIDIA NIM provides the containerized model serving for Kubernetes speech deployments, with the NVIDIA GPU Operator handling GPU resource management and the DCGM Exporter supplying the GPU utilization metrics that drive HorizontalPodAutoscaler decisions. Parakeet TDT v2 NIM covers batch transcription and Nemotron 3.5 ASR NIM covers streaming.
Summary
The deployment starts by pulling the target NIM container from the NGC catalog, for example Parakeet TDT v2 NIM for batch transcription or Nemotron 3.5 ASR NIM for streaming. The NGC catalog provides Helm charts for each NIM specifying pod resource requests including GPU count, the container image, service port configuration (REST on 9000, gRPC on 50051 for Riva-compatible NIMs), health check endpoints, and startup parameters. A Helm install applies this configuration and creates the NIM deployment and service.
GPU resource management requires the NVIDIA GPU Operator installed on the cluster. It handles NVIDIA driver installation on nodes, CUDA toolkit configuration, and GPU resource registration with the Kubernetes scheduler. Once installed, pods request GPUs with the standard nvidia.com/gpu resource type, and the scheduler places each single-GPU NIM pod on nodes with available GPUs.
Autoscaling uses GPU utilization as the scaling metric via the DCGM Exporter and Prometheus adapter. An HPA targeting 70% GPU utilization, for example, adds pods when average utilization of Parakeet NIM pods exceeds that threshold, up to the maximum replica count, and removes pods down to the minimum when utilization falls below a lower threshold. This keeps GPU utilization in an efficient range without manual capacity management, and separate HPAs for ASR, LLM, and TTS NIMs let each component scale independently.
Conclusion
Kubernetes speech microservices come together from NIM containers, the GPU Operator, and DCGM-driven autoscaling, with Parakeet or Nemotron NIM as the workload. Start by pulling the Helm chart for your target NIM from the NGC catalog and installing the GPU Operator before configuring the HPA.
Links: NVIDIA NeMo Framework on GitHub · NVIDIA GPU Operator for Kubernetes · NVIDIA NGC Catalog