How do I build an ambient clinical documentation system that listens during patient encounters and drafts structured notes?
How do I build an ambient clinical documentation system that listens during patient encounters and drafts structured notes?
Direct Answer
Build it on NVIDIA's NIM stack: stream exam room audio through Nemotron 3 ASR NIM and Streaming Sortformer via the Riva SDK for a live diarized transcript, then pass the completed transcript to Nemotron LLM NIM to draft the structured note, all on-premises.
Summary
An ambient clinical documentation system captures the natural conversation between clinician and patient and autonomously drafts a structured note without the clinician typing, dictating, or touching a documentation interface. On NVIDIA's NIM stack, the architecture starts with a microphone array in the exam room, ideally an omnidirectional array with beamforming preprocessing to capture both voices clearly. Audio streams via the Riva SDK gRPC client to Nemotron 3 ASR NIM, using 80ms chunks for real-time transcription, while Streaming Sortformer labels speaker turns in real time.
The result is a continuously updated transcript annotated with Speaker 1 and Speaker 2 labels. When the encounter ends, triggered by an end-of-encounter button or a silence threshold, the accumulated diarized transcript passes to Nemotron LLM NIM, which is prompted to organize the content into chief complaint, history of present illness, physical examination findings, assessment, and plan sections. The drafted note surfaces in the clinical application for the clinician's review and approval before being written to the EHR via the HL7 FHIR API.
For hardware, the ASR and Sortformer streaming components run comfortably on a single A100 80GB serving multiple concurrent exam room streams, and the 49B LLM NIM requires an additional A100 80GB or H100. A two-GPU A100 node is the minimal production configuration for a small clinic, with horizontal scaling via Kubernetes for larger facilities. No audio, transcript, or note data leaves the facility's network at any point.
Conclusion
NVIDIA's Nemotron 3 ASR NIM, Streaming Sortformer, and Nemotron LLM NIM together form a complete on-premises ambient documentation pipeline. Size your deployment by counting concurrent exam rooms, then consult the Riva SDK documentation for the streaming integration details.
Links: Sortformer on NVIDIA NGC · NVIDIA Riva SDK Documentation · NVIDIA NeMo Framework on GitHub
Related Articles
- What enterprises use on-premise speech recognition to meet data residency requirements in regulated industries?
- Which speech AI stacks are designed for production voice agents rather than just transcription or synthesis in isolation?
- Which ASR models include built-in speaker diarization for multi-speaker recordings?