One open mannequin for multilingual transcription, speaker attribution, translation, and audio Q&A, with prime leaderboard outcomes, plus the open Granary v2 information pipeline and fine-tuning recipes to adapt it.
A transcript can seize each spoken phrase and nonetheless depart customers or purposes questioning who mentioned what, methods to translate it, or methods to format domain-specific jargon. Multilingual calls, fast-paced conferences, and medical dictation every want completely different output: a translation, speaker labels, or accurately spelled medical phrases.
NVIDIA Nemotron 3.5 Transcribe is an open-weight, promptable mannequin that turns recorded speech into helpful textual content, now obtainable on Hugging Face and ModelScope. It brings transcription, speaker attribution (diarization), translation, and audio query answering into one checkpoint: it transcribes 40 languages, interprets 31 of them to or from English, and attributes audio system in 13. It handles recordings as much as an hour lengthy and labels as much as eight audio system. Builders can information output with pure language, select the place their audio is processed, and adapt recognition to area vocabulary and accents.
Transcribe is first amongst open-weight fashions on the Hugging Face Open ASR Leaderboard for English and sits on Synthetic Evaluation’s accuracy–active-parameter frontier. For speaker attribution, it posts the bottom error charge on David AI’s 21-language diarization leaderboard and ranks first in VoiceArena’s 14-system comparability. It runs on a single NVIDIA GPU, from DGX Spark to Blackwell data-center techniques.
Granary v2’s multilingual information (obtainable on Hugging Face and ModelScope) and reusable curation pipeline, along with NeMo Speech fine-tuning instruments, prolong prompt-level management to coaching: builders can outline the written kind they want and adapt the mannequin to the speech their purposes encounter. In a medical case research, fine-tuning minimize phrase error charge (WER) on held-out audio system by 45.8% relative to one of the best immediate.
The identical checkpoint can flip one assembly recording into completely different outputs, relying on the instruction:
- “Transcribe this recording and label every speaker.”
- “Translate this audio into English.”
- “What choices have been made on this assembly?”
Expertise the mannequin in motion within the video beneath, the place it tackles a basic scene from the 1940 screwball comedy His Woman Friday—a movie well-known for its rapid-fire, closely overlapping dialogue.
Nemotron 3.5 Transcribe captions a scene from His Woman Friday (1940) and labels audio system, together with overlapping dialogue.
For unfamiliar phrases, strive context earlier than coaching. In a healthcare setting, for instance:
Transcribe this medical dialogue within the authentic language. Doable phrases: HbA1c, metformin. Use these phrases solely when supported by the audio. Protect commonplace medical spelling and write numbers in typical written kind.
The mannequin card has really useful presets, together with reasoning for audio questions, plus the total immediate information and a task-by-language matrix: language help differs by job, and audio query answering is skilled in English. Phrase-level timestamps are additionally supported. As with all speech mannequin, spot-check names, speaker adjustments, and solutions in opposition to the recording.
From reside lessons to lasting classes. Agora, whose real-time voice and video platform powers reside conversations in apps, used word-level timestamps in a classroom demo: Nemotron 3.5 Transcribe turns a recorded class right into a transcript proven alongside the replay, so college students can see what was mentioned and when. Agora’s weblog walks by way of how they constructed it.
Amongst open-weight fashions, Transcribe leads on what was mentioned; on who mentioned it, it tops two speaker-attribution benchmarks outright.
Hugging Face Open ASR Leaderboard. The leaderboard measures transcription accuracy as phrase error charge (WER) on public take a look at units, in three tracks:
- English: first amongst open-weight fashions at 5.33% common WER (Determine 1).
- Multilingual: 4.62% imply WER throughout seven languages, the bottom amongst evaluated open-weight fashions within the mannequin card’s Open ASR comparability.
- Lengthy-form: 10.98% imply WER, additionally the bottom in that comparability.
Determine 1. Nemotron 3.5 Transcribe ranks first amongst open-weight fashions on the Hugging Face Open ASR Leaderboard (English) at 5.33% common WER, with API fashions excluded. Decrease WER means fewer transcription errors.
Synthetic Evaluation Speech-to-Textual content Leaderboard. Synthetic Evaluation independently ranks speech-to-text fashions by AA-WER, its phrase error charge index. Nemotron 3.5 Transcribe ties for first amongst open-weight fashions at 2.8% AA-WER, with about 4B lively parameters versus 24B for the mannequin it ties. On AA-AgentTalk, the held-out voice-agent speech set that makes up 50% of the AA-WER index, Transcribe scores 1.8% WER, versus 2.3% for the next-best open-weight mannequin. Transcribe pairs its speech encoder with a hybrid Mamba–Transformer mixture-of-experts language mannequin, so solely about 4B of its 30B parameters are lively per token. That places it on Synthetic Evaluation’s open-weight Pareto frontier of accuracy versus lively parameters. On AA’s all-model chart, it ranks forward of a number of broadly used proprietary speech APIs.
Determine 2. Main AA accuracy at about one-sixth the lively parameters: Nemotron 3.5 Transcribe achieves 2.80% AA-WER with roughly 4B lively parameters.
Speaker attribution throughout languages and overlap. Sortformer speaker cues attain the language mannequin with the acoustic options, so Transcribe can tag every flip throughout technology as an alternative of relying solely on a later step to connect audio system to phrases.
David AI DAI-ASR-I18N Leaderboard. On David AI’s DAI-ASR-I18N mixed transcription-and-speaker-attribution observe, Transcribe ranks first amongst evaluated open-weight fashions and second total. On the benchmark’s diarization observe, which scores who spoke when throughout 21 languages, it has the bottom diarization error charge of the 14 techniques examined, barely forward of NVIDIA’s devoted Nemotron 3 Diarization mannequin.
Determine 3. NVIDIA fashions maintain the highest two spots on David AI’s multilingual diarization leaderboard: Nemotron 3.5 Transcribe posts the bottom total diarization error charge (DER), 18.26% throughout 21 languages, adopted by Nemotron 3 Diarization at 18.38%. DER measures speaker-time errors; decrease is healthier.
VoiceArena Diarization-Bench. On VoiceArena’s English full-session benchmark, Transcribe posts the bottom concatenated minimum-permutation WER (cpWER) of 14 techniques and ranks first in each in-person and on-line settings. It additionally drops the fewest phrases: deletions are 8.32% of reference phrases, versus a median of about 31%.
Determine 4. NVIDIA fashions maintain the highest two spots in VoiceArena’s 14-system comparability: Nemotron 3.5 Transcribe leads at 32.4% cpWER, adopted by NVIDIA Multitalker Parakeet with Nemotron 3 Diarization at 34.2%. Diarization-Bench v1 covers English full periods, overlapping speech and a pair of–8 audio system. cpWER counts phrase errors after matching speaker labels; decrease is healthier.
Extra evaluations. Ocular AI’s Converse-STT analysis ranks Transcribe the highest open mannequin. Josh Talks’ Voice of India leaderboard reviews Transcribe’s outcomes throughout Indian languages.
Runs from edge to information middle. Transcribe runs on a single NVIDIA GPU, from Jetson AGX Thor and DGX Spark to Blackwell data-center GPUs. On one NVIDIA RTX PRO 6000 Blackwell Server Version GPU with NVFP4 precision, it processes 100 hours of audio in below three minutes, measured on 30-second clips at excessive concurrency.
Serve the checkpoint with vLLM as we speak or run it domestically with NeMo-Speech.cpp. Transcribe is constructed for recorded audio, not reside streaming.
Specialised recordings can nonetheless name for adaptation to area vocabulary and transcript conventions. Granary v2 and NeMo Speech fine-tuning deal with that, beginning with the information behind Transcribe.
A promptable speech mannequin requires coaching information with unambiguous formatting targets. Granary v2 is a part of the Nemotron speech assortment on Hugging Face and ModelScope. Constructed with NVIDIA NeMo Curator, its reusable pipeline ready multilingual speech utilized in Nemotron 3.5 Transcribe’s coaching, producing aligned transcript variants, job labels, and inspectable high quality metadata. The general public manifest covers 463,062 hours throughout 48 languages and 16 supply corpora, up from 25 languages in Granary v1. These figures describe the dataset launch, not Transcribe’s full coaching combination or its sampling weights.
Granary v2 generates three transcript types for every recording (Desk 1), the identical verbatim and clear kinds builders can request from Transcribe with a immediate. Past transcription, the pipeline produces quick audio captions, code-switching restoration of loanwords, speech-grounded Q&A pairs, contextual ASR biasing phrases, and translations with automated high quality scores.
| Coaching goal | Illustrative transcript |
|---|---|
| Textual content-normalized, verbatim | I paid, uh, twenty-five euros. |
| Inverse textual content normalization, with disfluencies | I paid, uh, €25. |
| Inverse textual content normalization, with out disfluencies | I paid €25. |
Desk 1. Granary v2 coaching targets for a similar utterance.
The pipeline is organized into 4 reusable phases:
- 1. Audio Preparation (Metadata): Segments uncooked or long-form audio, tracks audio system throughout the total recording, and tags acoustic occasions and language. Granary v2’s sources have been already segmented, so this launch skipped this stage.
- 2. Transcription (ASR): Routes every language to major and restoration ASR fashions, falling again when the first fails a hallucination examine. Human reference transcripts assist protect trusted spellings whereas including spoken types and disfluencies.
- 3. Activity Label Technology (Textual content): Derives transcript variants, then checks character charge, hallucination patterns, and mannequin or human-reference settlement. Solely accepted information obtain contextual-ASR phrases and translations.
- 4. Excessive Constancy (HiFi): Filters out segments which are noisy, clipped, or include a number of audio system, conserving a smaller set of high-quality recordings for makes use of past ASR. For the VoxPopuli dataset, it diminished 354k hours to 111k hours throughout 22 European languages.
Determine 6. Granary v2 information curation pipeline, following one pattern from uncooked audio by way of the Metadata, ASR, and Textual content phases to a coaching instance.
Builders can run the NeMo Curator pipeline with Ray and NVIDIA Dynamo backends on their very own recordings or licensed public corpora, or construct on the Granary v2 public manifests. The manifests embody the three transcript types as tn_raw, itn_raw, and itn_clean, the labels described above, entity hints, derived instruction pairs, provenance, and a per-record high quality flag. Obtain supply audio individually below every corpus’s authentic license, and comply with the discharge’s revealed phrases.
Begin with a development-set immediate that specifies language, script, and transcript format. Within the case research beneath, naming English and the Latin alphabet within the immediate minimize full-benchmark WER from 30.8% to 23.8%. For area errors that stay, the NeMo Speech fine-tuning workflow pairs consultant speech with express written targets.
The NeMo Speech medical ASR case research reveals this development on Eka Care’s open benchmark of recordings from Indian medical doctors and medical college students, which has 3,619 clips and no designated coaching cut up. The research builds adaptation information from generated medical textual content and public drugs catalogs, then combines roughly 153 hours of artificial speech, 29 hours of accented English from Frequent Voice, and 63 hours of medical recordings from MultiMed.
Determine 7. On the total take a look at set of Eka Care’s Indian medical speech benchmark, a greater immediate cuts WER from 30.8% to 23.8%, and fine-tuning brings it to 11.8%, about 62% fewer transcription errors than the mannequin out of the field. The complete set contains the 855 tuning clips used to decide on the immediate and checkpoints; the held-out outcomes beneath are the cleaner take a look at.
Artificial technology separates the written goal from its spoken rendering: “HbA1c” is the label, whereas “H B A one C” is shipped to the synthesizer. A number of TTS voices broaden the audio; actual recordings add accents and recording circumstances. Coaching textual content is filtered for precise matches and shared eight-word sequences with benchmark references.
As with Granary v2’s express targets, coaching examples range digits, unit spacing, company-name growth, and brand-name hyphens, with directions specifying the required conference. Some prompts embody time period lists with near-miss distractors, so the mannequin learns to make use of a spelling solely when the audio helps it. The ultimate system makes use of one fastened conference immediate and one mannequin move, with out retrieval or transcript post-processing.
The recipe trains the speech encoder and projection and makes use of LoRA to adapt the language mannequin. Two runs use completely different information mixtures; after adapter merging, chosen checkpoints are uniformly averaged into one remaining mannequin. Checkpoint choice makes use of decoded development-set outcomes, somewhat than coaching loss alone. The complete adaptation, from textual content technology by way of analysis, took about 14 GPU-hours on one NVIDIA GB300.
Area positive factors, measured on held-out audio system
The research splits the benchmark by speaker into 855 tuning clips, used to decide on the immediate and checkpoints, and a pair of,764 held-out clips, decoded as soon as per remaining system. Utilizing Eka Care’s KARMA scoring pipeline, fine-tuning cuts held-out WER from 22.44% for the best-prompt baseline to 12.17%, a forty five.8% relative discount and the clearest proof that the positive factors generalize to new audio system. Key phrase WER falls from 9.2% to six.5%, and semantic WER from 10.0% to 7.6%. On the identical held-out audio system and scorer, the tailored mannequin’s 12.17% WER beats an open medical-specialist mannequin (13.79%) and a broadly used general-purpose ASR mannequin (14.92%).
On normal English, LibriSpeech test-clean WER rises barely, from 1.54% to 1.69%. The medical specialist nonetheless leads on precise matches for brand-name medicine and diagnostics; the research factors to actual recordings of clinicians saying model names as the subsequent step. In your personal area, hold at the least 100 actual, labeled clips for checkpoint choice, and consider the entity sorts your utility is dependent upon alongside general-speech retention.
The case research contains instructions to breed every step. The fine-tuning tutorials cowl recipes, useful resource necessities, and instruments for coaching, export, and analysis, together with new-language and code-switching ASR. The accompanying agent ability lets a coding agent run the workflow, from baseline choice and information checks by way of coaching validation, adapter merging, and analysis.
ModelScope additionally permits builders to fine-tune Nemotron 3.5 Transcribe by way of its ms-swift framework for specialised multimodal purposes.
Attempt Nemotron 3.5 Transcribe within the Hugging Face demo, or on ModelScope Studios, then obtain the checkpoint from Hugging Face or ModelScope and serve it with the vLLM deployment information.
To skip self-hosting, use a hosted endpoint from Baseten, DeepInfra, or Simplismart.
To adapt it to your area, begin with the Granary v2 manifests on Hugging Face or ModelScope and the medical ASR case research, or let a coding agent drive the workflow with the fine-tuning agent ability.
Fashions & Datasets
Repositories & Tutorials: NVIDIA NeMo Speech GitHub | NeMo Curator Granary v2 Pipeline | vLLM Deployment Information | Speech Nice-Tuning Information | Nice-Tuning Agent Talent
*Accomplice Blogs: Agora | Inflection AI | VoiceArena | Simplismart *
Native AI: What’s new in native AI on NVIDIA RTX PCs, laptops, and DGX Spark
