Evaluating NVIDIA's Nemotron 3.5 Transcribe 30B-A4B on Converse-STT and Converse-Diarization

We evaluated NVIDIA's Nemotron 3.5 Transcribe 30B-A4B ahead of its release on Converse-STT and Converse-Diarization. On four two-person American English conversations it scores 2.95% word error rate, 371 errors on 12,555 reference words: fourth of 21 systems, two words behind Grok Voice Transcribe 2.0 (2.94%), four behind Reson8 (2.92%), 218 behind Scribe v2 (1.22%). It is the best open-weight transcriber on the board by 1.16 points, ahead of NVIDIA's own Nemotron 3 ASR Streaming (4.11%).

Its 138 deletions are the second-lowest on the board. Its insertions are not, and 57 of the 107 are a single token, hm, which the Whisper normalizer keeps where it strips every other spelling of the same backchannel. Spelled any other common way, the model scores 2.50% and ranks second.

On the same four conversations, scored as a diarizer on the speaker-activity segments it returns with the transcript, it scores 28.15% diarization error rate: second of 13 systems, 0.17 points behind Pyannote Community-1 (27.98%), 0.01 ahead of DiariZen (28.16%), and first on the high-overlap conversation at 29.42%. It holds up better than the rest of the board as overlap increases: the step from the medium-overlap conversations to the high one costs it 1.72 points of DER where every other system except Nemotron 3 Diarization loses 2.4 to 9.8. It returned the right speaker count on all four conversations.

We worked with NVIDIA's research team to set the model up ahead of release. It is a 30B-parameter mixture-of-experts with ~4B active parameters that reads a whole track and transcribes it in one pass; it was scored on the same audio, reference, normalization, and scoring as the other rows on each board, described in the two technical reports.^[1][2]

Result on Converse-STT

Converse-STT, Version 2

The leaders and every open-weight system.

System	Rank	Weights	WER	CER	S	D	I
Scribe v2	1	Proprietary	1.22%	0.55%	67	57	29
Reson8	2	Proprietary	2.92%	1.53%	139	168	60
Grok Voice Transcribe 2.0	3	Proprietary	2.94%	1.52%	149	178	42
Nemotron 3.5 Transcribe 30B-A4B	4	Open	2.95%	1.56%	126	138	107
Gemini 3.5 Transcribe	5	Proprietary	3.13%	1.76%	136	224	33
Nemotron 3 ASR Streaming (EN) 0.6B	11	Open	4.11%	2.26%	186	278	52
Parakeet TDT 0.6B v3	12	Open	4.54%	3.15%	173	175	222
Qwen3-ASR 1.7B	14	Open	5.03%	2.37%	136	209	287
Nemotron 3.5 ASR Streaming 0.6B	17	Open	5.87%	3.58%	213	488	36
Cohere Transcribe	20	Open	8.51%	6.18%	123	593	352

Pooled over 12,555 reference words. S, D, I: substitutions, deletions, insertions. Full board of 21 at /converse/stt.

Second through fourth are 367, 369, and 371 errors: a tie at this resolution, where 0.1 points is 13 words. Among open-weight systems there is no tie; the next is 145 words behind.

The deletion count is the signal. The corpus-wide most-deleted tokens on this board are i, a, and, it, you, know, yeah, right, like: fragments, repeated function words, backchannels. Nemotron deletes i 15 times, a 12, and 10, right 4, yeah 0. Universal-3.5 Pro deletes 105 you/know tokens; MAI-Transcribe 2 drops 48 isolated yeah/right; the two Nemotron streaming models drop 27 and 74. This model applies no disfluency cleanup, which is the reference convention.

Against NVIDIA's other systems

Converse-STT, Version 2

NVIDIA's four systems, at their board ranks.

WERCERlower is better

2.95%

1.56%

#4Nemotron 3.5 Transcribe 30B-A4BNVIDIA

4.11%

2.26%

#11Nemotron 3 ASR Streaming (EN) 0.6BNVIDIA

4.54%

3.15%

#12Parakeet TDT 0.6B v3NVIDIA

5.87%

3.58%

#17Nemotron 3.5 ASR Streaming 0.6BNVIDIA

WER and CER, pooled; ranks are by WER on the full board. The two streaming models run locally through Transformers at their 1.12 s lookahead (13 frames); Parakeet runs through OpenRouter. Lower lookaheads are scored as operating points on /converse/stt, not ranked.

The gap to the next NVIDIA system is 145 words: Nemotron 3 ASR Streaming (EN) 0.6B makes 516 errors (186 S, 278 D, 52 I) to this model's 371. Parakeet TDT 0.6B v3 makes 570, 222 of them insertions and 112 of those partner bleed on the gated track. Nemotron 3.5 ASR Streaming 0.6B makes 737, 488 of them deletions.

The streaming models trade accuracy for latency and the trade is visible at each operating point. Nemotron 3 ASR Streaming goes from 4.11% at 1.12 s lookahead to 4.61% at 80 ms; Nemotron 3.5 ASR Streaming from 5.87% to 6.18% at 320 ms and 6.99% at 80 ms. Nemotron 3.5 Transcribe has no such setting: it reads the whole track and decodes once, which is where the 138 deletions come from. The comparison across the four is between a batch 30B-A4B model and three sub-1B models built to run in a stream; the 1.16 points separating the two best are the cost of that constraint on this data.

Against open-weight systems

Converse-STT, Version 2

Every open-weight system on the board, at its board rank.

WERCERlower is better

2.95%

1.56%

#4Nemotron 3.5 Transcribe 30B-A4BNVIDIA

4.11%

2.26%

#11Nemotron 3 ASR Streaming (EN) 0.6BNVIDIA

4.54%

3.15%

#12Parakeet TDT 0.6B v3NVIDIA

5.03%

2.37%

#14Qwen3-ASR 1.7BAlibaba

5.87%

3.58%

#17Nemotron 3.5 ASR Streaming 0.6BNVIDIA

8.51%

6.18%

#20Cohere TranscribeCohere

WER and CER, pooled; ranks are by WER on the full board of 21. Open-weight means the model weights are published; the proprietary systems are hosted APIs.

Six of the 21 systems publish their weights, and this is the only one in the top ten. The next open-weight system is NVIDIA's own Nemotron 3 ASR Streaming at 4.11%, 1.16 points and 145 words behind; then Parakeet TDT 0.6B v3 at 4.54%, Qwen3-ASR 1.7B at 5.03%, Nemotron 3.5 ASR Streaming 0.6B at 5.87%, and Cohere Transcribe at 8.51%. On CER the order is the same at the top: 1.56% against 2.26% for the next.

The error profiles differ more than the ranks suggest. The open-weight group holds the three highest insertion counts on the board: Cohere Transcribe 352, Qwen3-ASR 287, Parakeet 222. Cohere's 1,068 errors also include 593 deletions, second only to GPT-4o Transcribe. Nemotron 3.5 Transcribe makes 371 with 138 deletions, and more than half of its 107 insertions are the hm spelling discussed below. Among the systems one can download and run, it is the first on this board to score inside the proprietary leaders' range: fifth-placed Gemini 3.5 Transcribe is 0.18 points behind it, and every other open-weight system is more than a point behind.

Where the 371 errors are

hm: 57 insertions from one spelling

The reference writes an agreeing backchannel as mm-hmm (124 occurrences). The Whisper English normalizer deletes mm-hmm, mhm, hmm, um, and uh, so Scribe's mm-hmm (61), Reson8's mhm (43), Grok's mhm (62), and Gemini's mhm (32) all vanish before alignment. Nemotron writes Hm. The two-letter form is not in the filler list, survives normalization, and aligns against nothing: 57 insertions, 43 of them on one track (2bb0cfe2/ch1), which is why that track is the model's worst at 5.08% and the conversation its worst at 4.09%. Eight more insertions are huh, from uh-huh, which the normalizer reduces to huh on both sides but which the model writes more often than the reference.

Backchannel spelling

Raw spelling by system, and what survives the normalizer.

System	Raw spelling	Normalized residue
Nemotron 3.5 Transcribe 30B-A4B	Hm 57, Mm-hmm 13	hm × 57
Scribe v2	mm-hmm 61, hmm 3	0
Reson8	mhm 43, mm-hmm 19, hmm 3	0
Grok Voice Transcribe 2.0	mhm 62, hm 1	hm × 1
Gemini 3.5 Transcribe	mhm 32, hmm 1	0

Counts over all eight raw transcripts. The reference contributes no hm tokens.

The board does not correct for this; the normalizer is identical for every system and the spelling is the model's. The sensitivity: with those 57 tokens written Hmm or Mm-hmm, the run is 314 errors, 2.50% WER, second on the board, 53 words ahead of Reson8. On 2bb0cfe2/ch1 alone, 5.08% becomes 2.13%.

Acronyms: ~25 errors

The reference spells acronyms as spoken letters (a i, g p s, n p r, u p s); the model writes AI, GPS, NPR, UPS. Each costs one substitution (a → ai ×6, g → gps ×3) plus one deletion per remaining letter (p 6, s 4, m 4, r 3, e 3). The remaining substitutions are ordinary: was → is ×4, in → and ×3, a → an ×3, haarp → harp ×2.

Per track

Track scores run from 1.45% (86c27418/ch1, fitness plans) to 5.08% (2bb0cfe2/ch1, the hm track); the other six sit between 1.94% and 3.88%. The gated channel, 0fcd38f2/ch1, whose partner talks 70% more than he does and whose bleed drives 112 of Parakeet's 130 insertions there and 44–45 for MAI-Transcribe 2 and AssemblyAI, costs this model nothing: 9 insertions and 2.63%, none of them the partner's speech. It transcribes the channel's speaker and ignores the bleed, as Scribe, Reson8, and Grok do and Parakeet does not.

Result on Converse-Diarization

Converse-Diarization scores the same four conversations as a who-spoke-when problem: the two tracks are mixed, the system returns speaker-labelled segments, and the metric is diarization error rate pooled over 3,807 seconds of reference speech, with overlapping speech kept and no collar. Nemotron 3.5 Transcribe returns speaker-activity segments alongside its <spk:N>-tagged transcript, and those segments are what is scored.

Converse-Diarization, Version 1

The board, pooled over four conversations.

DERDER, 0.25 s collarpooled over four conversations · lower is better

27.98%

18.66%

#1Pyannote Community-1pyannote

28.15%

21.35%

#2Nemotron 3.5 Transcribe 30B-A4BNVIDIA

28.16%

19.09%

#3DiariZen WavLM Large s80 MD v2BUT

28.21%

21.39%

#4Nemotron 3 DiarizationNVIDIA

30.34%

25.86%

#5Scribe v2ElevenLabs

31.22%

22.02%

#6Voxtral Mini Transcribe 2Mistral

31.33%

28.03%

#7Grok Voice Transcribe 2.0xAI

31.57%

20.38%

#8GPT-4o Transcribe DiarizeOpenAI

32.36%

27.45%

#9MAI Transcribe 2 (direct)Microsoft

32.50%

24.93%

#10Gemini 3.5 TranscribeGoogle

32.91%

25.41%

#11Qwen3-ASR 1.7B + FunASR speakersAlibaba

34.28%

28.23%

#12Grok STT 1.0xAI

35.84%

25.56%

#13Muse Voice Transcribe 1.0Meta

DER pooled over 3,807 s of reference speech, overlap included, no collar. The lighter bar is the same run scored with a 0.25 s collar around every reference boundary. Full board with the error breakdown at /converse/diarization.

It scores 28.15% DER, 1,072 seconds of error: second of 13 systems, 6.5 seconds behind Pyannote Community-1 (27.98%), 0.4 seconds ahead of DiariZen WavLM Large (28.16%), 2.3 ahead of Nemotron 3 Diarization (28.21%). The top four are within 0.23 points and all four are open-weight; the best proprietary system, Scribe v2, is fifth at 30.34%, 83 seconds behind. With the 0.25 s collar it scores 21.35%, fourth, behind Pyannote (18.66%), DiariZen (19.09%), and GPT-4o Transcribe Diarize (20.38%). On JER it is third at 26.59%. It returned two speakers on all four conversations, as did every system except DiariZen (zero of four) and Muse Voice Transcribe 1.0 (two of four).

Converse-Diarization, Version 1

The leaders, every open-weight system, and the error composition.

System	Rank	Weights	DER	Missed	False alarm	Confusion	Collar	High	Medium
Pyannote Community-1	1	Open	27.98%	15.01	11.01	1.96	18.66%	30.77%	26.99%
Nemotron 3.5 Transcribe 30B-A4B	2	Open	28.15%	18.42	8.49	1.25	21.35%	29.42%	27.70%
DiariZen WavLM Large s80 MD v2	3	Open	28.16%	16.01	10.44	1.71	19.09%	29.90%	27.55%
Nemotron 3 Diarization	4	Open	28.21%	18.43	8.52	1.26	21.39%	29.46%	27.76%
Scribe v2	5	Proprietary	30.34%	27.06	2.54	0.74	25.86%	33.11%	29.36%
Qwen3-ASR 1.7B + FunASR speakers	11	Open	32.91%	25.56	5.47	1.89	25.41%	36.58%	31.61%

DER and its three terms are pooled seconds as a share of reference speech; the terms sum to DER. Collar: DER with a 0.25 s collar. High: the one conversation with 20% or more of its speech overlapped. Medium: the three with 10–20%. Full board of 13 at /converse/diarization.

The error is almost all speech detection. Of the 28.15 points, 18.42 are missed speech, 8.49 are false alarms, and 1.25 are speaker confusion: when the model says someone is speaking it is nearly always right about who. With the collar the false alarms fall to 1.20 and confusion to 0.28, so most of both terms sit within a quarter-second of a reference boundary; the 19.87 points of missed speech that remain do not, and are the model's actual gap to the top. Pyannote's composition is the opposite trade: 3.4 points fewer missed, 2.5 more false alarms. The proprietary transcribers are more conservative still, Scribe v2 at 27.06 missed and 2.54 false alarm; they call less of the audio speech, and on a corpus where 14–24% of it is two people at once, that is where the 2-point gap between them and the four leaders comes from.

By overlap

Converse-Diarization, Version 1 · High overlap

The one conversation where a quarter of the speech is overlapped.

DERthe high-overlap conversation · lower is better

29.42%

#1Nemotron 3.5 Transcribe 30B-A4BNVIDIA

29.46%

#2Nemotron 3 DiarizationNVIDIA

29.90%

#3DiariZen WavLM Large s80 MD v2BUT

30.77%

#4Pyannote Community-1pyannote

33.11%

#5Scribe v2ElevenLabs

34.70%

#6Grok Voice Transcribe 2.0xAI

34.71%

#7MAI Transcribe 2 (direct)Microsoft

34.72%

#8Gemini 3.5 TranscribeGoogle

DER on 2bb0cfe2, 23.97% of its reference speech overlapped, 998 s of reference. Ranks are within this conversation; the top eight of 13 shown.

On the high-overlap conversation it is first at 29.42%, 0.04 points ahead of Nemotron 3 Diarization, 0.48 ahead of DiariZen, 1.35 ahead of Pyannote, 3.69 ahead of the best proprietary system (Scribe v2, 33.11%). On the three medium-overlap conversations it is third at 27.70%, 0.71 behind Pyannote and 0.15 behind DiariZen. The model does relatively better the more the speakers talk over each other: going from the medium conversations to the high one costs it 1.72 points, the smallest increase on the board after Nemotron 3 Diarization's 1.70, against 2.35 for DiariZen, 3.78 for Pyannote, 3.75 for Scribe v2, and 5 to 10 for the hosted transcribers (Voxtral 5.76, GPT-4o Transcribe Diarize 5.80, Muse Voice 9.83). Every system's DER rises with overlap; this one's rises least, which is why it moves from third to first. Pyannote's lead on the pooled board is built on one conversation: on ea217639 it scores 23.46% to Nemotron's 27.81%, a 4.35-point margin, and Nemotron is ahead on the other three (28.51 vs 29.59, 29.42 vs 30.77, 26.80 vs 28.19). Nemotron's four conversation scores span 2.6 points, 26.80% to 29.42%, the narrowest spread of the top four; Pyannote's span 7.3.

Against NVIDIA's other system

Converse-Diarization, Version 1

NVIDIA's two systems, at their board ranks.

DERDER, 0.25 s collarpooled over four conversations · lower is better

28.15%

21.35%

#2Nemotron 3.5 Transcribe 30B-A4BNVIDIA

28.21%

21.39%

#4Nemotron 3 DiarizationNVIDIA

DER pooled, and with the 0.25 s collar. Nemotron 3 Diarization is the standalone speaker model; Nemotron 3.5 Transcribe is scored on the speaker activity it returns with the transcript.

The two runs are indistinguishable at this resolution. 28.15% against 28.21% pooled; 18.42/8.49/1.25 against 18.43/8.52/1.26 on the three terms; 29.42% against 29.46% on the high-overlap conversation, 27.70% against 27.76% on the medium three; per conversation, never more than 0.16 points apart. Every term matches to within 0.03 points, which is consistent with the transcribe model's speaker activity coming from the same diarization stack as Nemotron 3 Diarization. The practical difference is that this model returns the transcript at the same time, 2.95% WER on the separate tracks, where Nemotron 3 Diarization returns speaker segments only.

Per conversation

The four conversations score 26.80% (86c27418, 16.5% overlap), 27.81% (ea217639, 17.6%), 28.51% (0fcd38f2, 14.7%), and 29.42% (2bb0cfe2, 24.0%). Missed speech runs 138 to 211 seconds per conversation and is highest on ea217639, the conversation that decides second from first; false alarms run 54 to 101 seconds and are lowest there. Confusion is 8 to 16 seconds per conversation, under 1.7% everywhere. The gated channel in 0fcd38f2, where the partner talks 70% more than the speaker and whose bleed costs Parakeet 112 insertions on the STT board, costs nothing here either: 14 seconds of confusion on 902 seconds of reference.

Limitations

Four conversations, eight speakers, 12,555 words, 3,807 seconds of speech; the four-word spread across second, third, and fourth on Converse-STT and the 0.23-point spread across the top four on Converse-Diarization are not rankings. Both scores are from a pre-release deployment; the shipped model could score differently. The diarization deployment could not take a 15-minute mix in one request, so each was scored in 8-minute windows with 30 seconds of overlap and speakers linked across the join; a deployment that takes the whole mix could score differently, most plausibly on confusion. The hm figure is a sensitivity analysis of the normalizer on this model's output, not a corrected score. The audio is close-talking studio American English; other languages, accents, and channels are a further test.