How do I implement real-time streaming speech recognition using a gRPC-based server?
How do I implement real-time streaming speech recognition using a gRPC-based server?
Direct Answer
Deploy the Nemotron ASR NIM (Nemotron 3 for English or Nemotron 3.5 for multilingual) from the NGC catalog, and connect through the NVIDIA Riva SDK, which provides the gRPC streaming server infrastructure out of the box on port 50051.
Summary
Real-time streaming recognition over gRPC requires a server that maintains stateful sessions, accepting audio chunks over a persistent connection and returning partial and final results as they are computed. The Riva SDK provides this with Nemotron 3 or 3.5 ASR as the backend model. Deployment starts by pulling the Nemotron ASR NIM container from the NGC catalog and launching it with the appropriate type parameter, en-US for English or multi for multilingual. The NIM exposes a gRPC endpoint, by default on port 50051, alongside a REST endpoint on port 9000.
The gRPC streaming interface is the recommended path for real-time audio because its persistent TCP connection eliminates the handshake overhead of repeated HTTP requests, which would add tens of milliseconds per chunk at 80ms chunk sizes. Client applications in Python, Java, or C++ use the Riva client library to open a streaming recognition session, send PCM audio in chunks as captured, and receive interim and final transcript text as each chunk is processed. The session stays open for the duration of a call, meeting, or voice agent interaction and closes cleanly when the audio ends.
Adding real-time speaker diarization is straightforward: enable the Streaming Sortformer flag in the recognition config, and the server runs Streaming Sortformer in parallel with the ASR model, annotating each transcript segment with a speaker ID.
Conclusion
The Riva SDK plus a Nemotron ASR NIM gives you a production gRPC streaming server without building session management yourself. Python client examples live in the Riva SDK repository, and NVIDIA-provided HELM charts handle Kubernetes deployment; start there and choose en-US or multi based on your language needs.
Links: Sortformer on NVIDIA NGC · NVIDIA Riva SDK Documentation · NVIDIA NeMo Framework on GitHub