Nvidia Nemotron 3.5 Transcribe: Open ASR for 40 Languages

NVIDIA Nemotron 3.5 Transcribe and Granary v2 represent a leap forward in intelligent, multilingual transcription. This open-weight framework allows users to not only convert speech to text but to prompt and adapt the model for specific needs.

🎙️ Beyond Simple Transcription

A standard transcript often fails to capture the "who" and "how" of a conversation. Traditional ASR just gives you a wall of text. Modern applications require nuance:

• Multilingual Calls: Need seamless translation.
• Fast-paced Meetings: Require precise speaker labels (diarization).
• Medical Dictation: Demand strict adherence to domain-specific terminology.

Nemotron 3.5 Transcribe solves this by integrating transcription, translation, speaker attribution, and audio Question & Answering (QA) into a single checkpoint.

Core Capabilities Checklist

• Multilingual Support: Transcribes 40 languages.
• Translation: 31 languages (to/from English).
• Speaker Attribution: 13 languages supported.
• Capacity: Handles audio up to 60 minutes.
• Diarization: Identifies up to 8 distinct speakers.

---

🛠️ Architecture and Efficiency

The model utilizes a sophisticated hybrid design to balance power and speed. It pairs a speech encoder with a Mamba–Transformer Mixture-of-Experts (MoE) language model.

While the model has a total of 30B parameters, it only activates approximately 4B parameters per token. This efficiency can be expressed as: $\text{Active Ratio} = \frac{4\text{B active parameters}}{30\text{B total parameters}} \approx 13.3\%$

This puts Nemotron 3.5 Transcribe on the Artificial Analysis Pareto frontier, offering proprietary-level accuracy with a fraction of the active computational cost.

---

📈 Leaderboard Dominance

Transcribe is currently a top performer across multiple industry benchmarks:

1. Hugging Face Open ASR (English): Ranks 1st among open-weight models with an average Word Error Rate (WER) of $5.33\%$.
2. Artificial Analysis (AA-WER): Ties for 1st among open-weight models at $2.8\%$ AA-WER.
   • On the AA-AgentTalk set, it achieved a superior $1.8\%$ WER.
3. Diarization:
   • Lowest error rate on David AI’s 21-language leaderboard.
   • Ranked 1st in VoiceArena’s 14-system comparison.

Note: The model's ability to tag speakers during generation—rather than as a post-processing step—is driven by Sortformer cues delivered alongside acoustic features.

---

✍️ Promptable Control & Adaptation

Developers can guide the model using natural language instructions. This allows one checkpoint to serve multiple purposes:

Prompt A: "Transcribe this recording and label each speaker." Prompt B: "Translate this audio into English." Prompt C: "What decisions were made in this meeting?"

Real-World Applications

• Entertainment: The model successfully captioned a scene from the 1940 film His Girl Friday, managing its notoriously rapid and overlapping dialogue.
• Healthcare: By using specific prompts (e.g., "Preserve standard medical spelling"), the model ensures professional accuracy.
• Education: Agora implemented word-level timestamps to create synchronized transcripts for classroom replays.

Granary v2 & Fine-Tuning

NVIDIA has released the Granary v2 data pipeline and NeMo Speech tools. This allows developers to move beyond prompting into full model adaptation.

Case Study: Medical Domain Fine-tuning the model using Granary v2 resulted in a massive improvement: $\text{WER Reduction} = 45.8\% \text{ (relative to the best prompt)}$

---

🚀 Deployment and Availability

The model is available on Hugging Face and ModelScope.

Hardware Compatibility:

• Local: RTX Spark, DGX Spark, and Jetson AGX Thor (Quantized versions arriving soon).
• Enterprise: Scalable deployment on Blackwell data-center GPUs.

Team and Contributors

Contributors: Maryam Motamedi, Piotr Żelasko, Kunal Dhawan, Nikolay Karpov, Nune Tadevosyan, Meline Mkrtchyan, Nithin Rao Koluguri, and Francesco fciannella.