NVIDIA Debuts NemotronLabs VoiceChat 11B Speech Model

NVIDIA has launched NemotronLabs VoiceChat 11B, an open-weights, end-to-end speech-to-speech model designed for real-time, full-duplex conversations. Unlike traditional cascaded systems that chain separate automatic speech recognition, large language model, and text-to-speech components, this hybrid Mamba-Transformer model unifies these tasks into a single network. It integrates a 0.6-billion parameter Fast Conformer speech encoder from Nemotron-Speech-Streaming-En-0.6b, the Nemotron Nano v2 LLM backbone, and a TTS decoder that outputs 22.05 kHz audio. This unified architecture eliminates API handoffs and reduces turn-taking latency to 448 milliseconds.

The model is the first open full-duplex system to support live tool calling during active conversations. It routes tool commands through a dedicated side channel using a TOOLCALL block, while allowing operators to define on-hold phrases that the agent speaks to prevent dead air while APIs execute. On the AU Harness BFCL-v3 spoken tool calling benchmark, the model achieved a 56.1% average score, including 58.5% for simple tools, 62.5% for multiple tools, 42.5% for parallel tools, 27.5% for parallel-multiple, and 89.6% for irrelevance. On Full-Duplex-Bench v3, it scored 82.5% for tool selection, 44.2% for argument accuracy, and 33% for pass@1. It ranks second among open models on both VoiceBench and Full-Duplex-Bench 1.0, where it demonstrated a user-interruption take-over rate of 1.00 at 480 milliseconds and pause-handling rates of 0.153 on synthetic data and 0.255 on Candor.

For developers, deploying the model requires a single GPU with at least 80 gigabytes of VRAM, such as an NVIDIA A100, H100, RTX 6000 Pro, or B200 running on x86_64 Linux. While released under a permissive OpenMDW-1.1 license, NVIDIA notes the checkpoint is currently intended for research purposes only. The model exhibits several known limitations, including a strict two-minute audio context ceiling, a maximum of five tools per session, and a tendency to degrade into gibberish or engage in runaway self-talk after multiple turns. Additionally, users cannot interrupt the agent during tool execution, and all system prompts must remain in ASCII format.