October 13, 2026Technical Report

Converse-Diarization Technical Report

The benchmark

Converse-Diarization asks who spoke when^[1], on one channel, in unscripted American English conversation between two people. Each person was recorded on their own microphone. The harness sums those tracks into the single mix a phone call or a meeting would present, then compares the model’s speaker timeline with the timeline implied by the published word timings.

The leaderboard ranks the eleven scored systems by diarization error rate with no collar^[2]. Lower is a closer match to the reference. The same board can be read as DER with a ±0.25 s collar^[3], as Jaccard error rate^[2], or as speaker-count correctness, and it can be filtered by overlap tier. Those views explain a system. They do not replace the ranking. Transcript text is stored and shown, and it is not scored. Word error rate is the sibling benchmark, Converse-STT, on the same recordings.

Conversations: 4; Two speakers each
Minutes: 60.4; Wall-clock audio
Speakers: 8; American English
Systems scored: 11; Ranked by DER

These four conversations are the public sample of a larger collection Ocular AI licenses, published as the Hugging Face dataset^[4]. The reference is built from each speaker’s own microphone: the reviewed words, the fillers, and the timed tags, closed into one region when the gap inside a speaker is at most 0.2 s. Overlap is kept, and it counts once per speaker^[5]. The system does not receive those channels, and it is not told how many speakers to find^[2]. It hears their mono sum. Diarization error rate is missed speech, false alarm, and speaker confusion, divided by that reference time^[5]. The ranked number uses no collar.

Scoring convention

A diarization error rate is only comparable with another when the collar, the overlap rule, and whether the system was told how many speakers to find are the same^[2]^[3]. Every number on the leaderboard and in the findings uses the convention below. The sections after it are how the reference timeline was built and how the percentage was computed.

One rule for every system

What the ranked number includes.

Rule	This benchmark
Ranked score	Pooled DER, no collar. Lower is better.
Overlapping speech	Scored. Each active reference speaker counts, and overlap is not dropped.
Speech activity	Left to the system. Missed speech and false alarms count.
Speaker count	Never given to the system. Printed beside the rank, and not used to order it.
Audio in	One mono mix of the two microphones. The separate channels build the reference. They are not the system input.
Also reported	DER with a ±0.25 s collar, JER, missed speech, false alarm, confusion, and speaker-count correctness.

Corpus DER pools error seconds and reference seconds across the four conversations, then divides once. A conversation with more speaker time weighs more. JER is the unweighted mean of the four conversations, so each conversation counts once.

Dataset collection

The four conversations were recorded on Ocular AI’s platform by American English experts hired through Workbolt^[4]. Each session is a topic-based, full-duplex conversation: two people talking freely, at the same time, on a shared topic. Topics in this sample include interpreting news, drones and their implications, customer engagement, and fitness decisions. A discussion prompt can appear in the transcript next to the talk that follows. The speech is unscripted. It is not a claim that every utterance was unprompted.

Each expert is on their own microphone. The two voices are never mixed at capture. One session becomes two mono files, each the whole conversation from one speaker’s mic, stored as 48 kHz 24-bit FLAC. That channel split is the speaker label. ch1 and ch2 are the two microphones, not a guess about who was talking. The filename carries a room-start epoch, the speaker id, then hifi-audio and the capture-start epoch. The later microphone in this sample starts 1.298 s, 1.321 s, 2.109 s, or 2.125 s after the earlier one. Those offsets put both channels on one clock. A little of the other voice can leak into a channel. The release was not processed to remove that bleed.

Demographics on the tracks are self-reported: accent, native language, gender, and, on four of the eight tracks, an age band. The public metadata keeps speaker_id, take_id, and datetime_utc. It drops the vendor QC object (passed, issues_flagged, notes), so the audit flags themselves are not in this sample. Participant consent covers public redistribution. The sample is CC BY 4.0.

Machine draft, then two human review steps

The transcript of each isolated channel starts as a machine transcript. An ASR system writes the first pass. People then correct and verify that draft before it is treated as the reference.

Transcript production

01
Machine draft. A first transcript is generated from that speaker’s own microphone, not from the mix.
02
Correction. The reviewer listens to the isolated channel and corrects the draft so the text matches what was said, including fillers, repetitions, false starts, and event tags.
03
Verification. A second pass checks the corrected transcript against the same channel. This is the verification step.
04
Ocular audit. After correction and verification, Ocular audits the transcript. The dataset card names this audit as part of producing the reviewed text.

The reviewed text is what gets timed. Word timestamps are forced-aligned to that reviewed transcript on the speaker’s own channel. They are not the ASR draft’s timestamps. Non-speech tags (breath, laugh, spn, unclear, clear-throat) are written by the reviewers. Their times are interpolated around the aligned lexical words. They are not measured breath or laughter onsets. The harness does not rerun ASR, review, or alignment. It scores the word intervals this process published.

Recordings

The published AudioFolder is one metadata row per speaker track, plus one FLAC. Those bytes are copied unchanged. They are not resampled, padded, or re-encoded. A SHA-256 of each FLAC is stored on the track, and a file is rejected when its checksum, duration (tolerance 0.05 s), or sample rate disagrees with the published record.

The harness orders tracks by capture time, then by speaker id, and names them ch1 and ch2. offset_s is how many seconds after the earlier microphone this track started. Word times in the manifest are already shifted by that offset onto one conversation clock. Conversation duration is the max of offset plus track duration.

All eight speakers self-report American English as their accent and English as their native language. Five report female and three report male. Age is present on four tracks (three in 45–55, one in 35–45) and absent on the other four. Locale is en-US on every conversation.

The English sample

Four conversations, timed onto one clock.

Conversation	Wall	Later-mic offset	Words	Overlap	Tier	Speaker time	Regions
ocular-0fcd38f2	15:04	2.125 s	3,095	14.68%	medium	15:02	386
ocular-2bb0cfe2	15:06	1.321 s	3,357	23.97%	high	16:38	363
ocular-86c27418	15:06	1.298 s	3,251	16.53%	medium	15:25	410
ocular-ea217639	15:08	2.109 s	3,216	17.57%	medium	16:22	239

Overlap is time where both speakers are active, divided by time where at least one is active, after the 0.2 s merge. Speaker time counts both speakers, so it exceeds wall-clock wherever they talk at once.

Transcripts and word timings

Each metadata row carries a verbatim transcript and a list of segments. The harness prefers the transcript string when it is non-empty, and otherwise joins segment texts in index order. That string is kept for display. Scoring never reads it.

The reference timeline is built only from segments[].words[]. Each word supplies word, start, and end on that microphone’s own clock. The harness drops a word whose text is empty, whose times are missing or non-finite, or whose end is not strictly after its start. A start slightly before zero is clamped to zero. Surviving times are shifted by offset_s and rounded to the millisecond. The two tracks together contain 12,919 scored words.

Those published times are the forced alignment of the reviewed text, run once upstream of the harness^[4]. The scoring code does not align again. A word the aligner placed across a pause stays that wide. Short fillers and unstressed function words are the cases the dataset card flags as less reliable^[4]. Two source segments have an empty words array, so their text is in the transcript string and they contribute no word intervals. One source segment ends about 69 ms past its track duration; the harness clips to the conversation duration at score time.

Non-lexical tags are timed speech

Tokens whose text is a tag still have a start, an end, and a duration, so they enter the reference. The tag inventory in the transcript strings is 616 breath, 122 laugh, 20 spn, 7 clear-throat, and 4 unclear. Merged on their own, those tag intervals cover 627.9 s. Merged together with the lexical words, reference speaker time is 3,807.1 s, against 3,256.8 s from lexical words alone. The difference is tag time that is not already inside a lexical region.

A breath can be 40 ms or several hundred milliseconds. A laugh on ocular-0fcd38f2, channel 1, runs 394.600–396.358 s (1.758 s) and another 397.501–399.600 s (2.099 s). Those intervals are reference speech for ch1. A model that stays silent through a laugh is charged a miss. A model that continues a turn through a tagged breath is not charged a false alarm for that breath.

Text is not normalized, because text is not the score
Punctuation stays attached to the token (uh,, Mm-hmm.). Tags stay in the token. Neither affects DER, because the scorer sees only a speaker label and an interval. The operations that change the score are temporal: which tokens count as speech, and which gaps are closed.

Temporal normalization

Same-speaker words separated by at most 0.2 s are one speech region. A longer gap stays silence. The threshold is MERGE_GAP_S = 0.2, with a 1 ns epsilon so a gap that floating point stores as just over 0.2 s still merges. Intervals of different speakers are never merged, so overlap survives. After the merge, regions are clipped to the conversation duration, and touching or overlapping intervals of one speaker are unioned. That canonical form is what DER, JER, speaker count, and the overlap ratio all read.

Inside a track, 10,391 gaps between successive scored words are positive. The median gap is 60 ms. 8,956 gaps are at most 0.2 s and collapse into the surrounding region. 1,435 gaps are longer and remain silence. A system that bridges one of those longer pauses is charged a false alarm for the bridged time. NIST’s meeting evaluations treated a pause shorter than 0.3 s as continuous speech^[3]. The threshold here is 0.2 s, and it is a choice of this reference, not that 0.3 s rule.

Hypothesis handling depends on what the provider labelled. granularity = "word" means the items are speaker-labelled words, and they receive the same 0.2 s merge as the reference. granularity = "segment" means the items are already turns: they are clipped and unioned per speaker, and the 0.2 s rule is not applied again. Word-merged systems on this board are Scribe v2, both Grok models, MAI Transcribe 2, and Gemini 3.5 Transcribe. Segment systems are pyannote Community-1, DiariZen, Voxtral Mini Transcribe 2, GPT-4o Transcribe Diarize, and Muse Voice Transcribe 1.0.

What the scorer changes

Reference and hypothesis, step by step.

Step	Reference	Hypothesis
Source clock	Per-microphone word start/end, then add offset_s	Provider timestamps, already on the mix
Drop	Empty text, non-finite times, end ≤ start	Provider must return a speaker on every scored item
Quantize	Round shifted times to 1 ms	Kept as returned; the scorer does not snap to a frame
Gaps ≤ 0.2 s	Merged	Merged only when granularity is word
Gaps > 0.2 s	Silence	Silence for words; a segment turn is kept whole
Clip	To [0, conversation duration]	Same
Per-speaker union	Touching or overlapping intervals become one	Same, gap 0
Text	Ignored after the empty-text check	Ignored. Extra keys such as text are dropped

The mix the model hears

Policy id mono-mix-v1. For each conversation the harness checks the FLAC checksum, decodes to float32 mono, and resamples 48 kHz to 16 kHz with scipy.signal.resample_poly^[8]. The ratio 48000:16000 reduces by 3, so the polyphase filter downsamples by 3. Each track is then delayed by round(offset_s × 16000) leading zeros and the shorter track is zero-padded at the end. The two arrays are added sample-wise. They are not averaged and not ducked, so the relative level of the two microphones survives. If the absolute peak exceeds 1.0, the whole mix is divided by that peak. Otherwise the gain is 1. The result is written as mono PCM16 WAV.

Each mix records the policy, the sample count, and a SHA-256 of the WAV bytes. Each run records the same policy and the dataset digest, so a score points at one mix. Mixes on this sample are about 904–908 s, about 29 MB of 16 kHz PCM16.

What each system actually received is not identical, because upload encoding is part of the provider. upload = "flac" is a passthrough of the WAV mix. upload = "mp3" is ffmpeg, mono, 16 kHz, 64 kbps. OpenRouter’s default is MP3, and so is the OpenAI-compatible default. Direct xAI and Azure default to passthrough. Local pyannote and DiariZen read the WAV. Meta re-encodes 16 kHz PCM WAV windows from the mix.

How a score is computed

There is no frame grid. Boundaries stay floats. A sweep line walks every start and end of a reference interval, a hypothesis interval, and any collar exclusion. Between two event times the set of active reference speakers and hypothesis speakers is constant. On an elementary interval the scorer adds the four quantities below^[6].

One elementary interval

T M F A K = d \cdot ∣ R ∣ = d \cdot max (0, ∣ R ∣ - ∣ H ∣) = d \cdot max (0, ∣ H ∣ - ∣ R ∣) = d \cdot min (∣ R ∣, ∣ H ∣)

$d$: Length of the interval, in seconds
$R$: Reference speakers active on it
$H$: Hypothesis speakers active on it
$T$: Reference time added. Overlap counts once per speaker
$M$: Missed speech added
$F A$: False alarm added
$K$: Time in which some speaker could have been named correctly

Speaker labels are not compared as strings. A one-to-one assignment, the Hungarian algorithm on the pair matrix^[7], maximizes total overlap^[6]. Every pair of a reference speaker and a hypothesis speaker active together adds d seconds to that matrix. The overlap credited by the assignment is correct speech. Confusion is the remainder of K.

Diarization error rate

DER = \frac{M + F A + C}{T}

$M$: Missed speech
$F A$: False alarm
$C$: Speaker confusion
$T$: Reference speaker time

Pooled across conversations: error seconds and reference seconds are summed, then divided once. A conversation with more speaker time weighs more. The same pooling is used inside an overlap tier. For pyannote Community-1 the three shares are 15.01, 11.01, and 1.96, which sum to 27.98.

Collar, JER, speaker count, overlap tier

The ranked number uses no collar^[2]. The secondary number places a no-score zone around every reference boundary^[3].

Collar

[b - c, b + c], c = 0.25 s

$b$: A reference boundary
$c$: Half-width of the no-score zone. The ranked DER uses none

Overlapping zones are unioned and skipped. They contribute neither errors nor reference time. On this sample the collar moves GPT-4o from 31.57% to 20.38% and pyannote from 27.98% to 18.66%.

Jaccard error rate asks, for each reference speaker, how much of the union with its paired hypothesis speaker was missed^[2]. The pairing minimizes the sum of those errors and can differ from the DER pairing.

Jaccard error rate

JER = 1 - \frac{∣ r \cap h ∣}{∣ r \cup h ∣}

$r$: Speech time of a reference speaker
$h$: Speech time of the hypothesis speaker paired with it

A reference speaker with no partner scores 1. Corpus JER is the unweighted mean of the four conversations, so each conversation counts once regardless of length.

Speaker count compares how many distinct labels still have speech after clipping. It is correct when those two integers match. The board prints the fraction, for example 4/4. It does not rank by that fraction. Every conversation here has two reference speakers.

Overlap ratio is measured on the canonical reference, after the 0.2 s merge.

Overlap ratio

O = \frac{t _{\geq 2}}{t _{\geq 1}}

$t_{\geq 2}$: Time both reference speakers are active
$t_{\geq 1}$: Time at least one reference speaker is active

Low is below 10%. Medium is from 10% up to but not including 20%. High is from 20%. This sample has three medium conversations and one high conversation. There is no low-overlap conversation.

Examples from the audio

These are short slices of the published recordings. The spectrogram is each speaker’s own microphone, 16 kHz, low frequency at the bottom. Switch the channel under the player to hear one microphone or the mix that was scored.

Inference configuration

Every scored system received the full mix and was not told how many speakers to find. Where a language hint is sent, it is en. Three systems only diarize. pyannote Community-1 follows the segmentation, embedding, and clustering pipeline described for pyannote.audio^[9]. DiariZen’s local stage is the WavLM end-to-end diarization of Han et al.^[10], and WavLM Large is the self-supervised model of Chen et al.^[11]. Nemotron 3 Diarization is NVIDIA’s Sortformer^[12], an end-to-end diarizer that orders speakers by arrival time, run through NeMo with the published speaker-cache profile of its streaming variant^[13]. The scored weights are the checkpoints in the table, not the checkpoints reported in those papers. The other eight systems also return words, and those words are used only to place the speaker timeline.

Eleven scored systems

What each system received.

Provider	System	Access	ASR	Diarization	Endpoint	Checkpoint	Request
pyannote	pyannote Community-1	Open	—	Yes	Self-hosted	pyannote/speaker-diarization-community-1	HF-gated pipeline. Device auto: CUDA, otherwise MPS, otherwise CPU. No speaker count. Overlapping turns kept. 16 kHz PCM16 WAV. Scored as segments.
BUT-FIT	DiariZen WavLM Large s80 MD v2	Open	—	Yes	Self-hosted	BUT-FIT/diarizen-wavlm-large-s80-md-v2	Published pipeline: 16 s chunks, VBx clustering, minimum 1 speaker, maximum 20. No language hint and no supplied speaker count. 16 kHz PCM16 WAV. Scored as segments. Weights CC BY-NC 4.0.
NVIDIA	Nemotron 3 Diarization	Open	—	Yes	Self-hosted	nvidia/Nemotron-3-Diarization	NeMo Sortformer, offline profile: speaker cache 264, FIFO 40, chunk 340, right context 40, cache update every 300 frames of 80 ms, a 30.4 s input buffer. No language hint and no supplied speaker count; up to 8 arrival-ordered speakers. CPU, FP32. 16 kHz PCM16 WAV. Scored as segments. Weights OpenMDW 1.1.
ElevenLabs	Scribe v2	Proprietary	Yes	Yes	openrouter.ai/api/v1/audio/transcriptions	elevenlabs/scribe-v2	ElevenLabs diarize flag on. Language hint en. Mono 16 kHz 64 kbps MP3. Speaker-labelled words, merged at 0.2 s.
Mistral	Voxtral Mini Transcribe 2	Proprietary	Yes	Yes	api.mistral.ai/v1/audio/transcriptions	voxtral-mini-2602	Diarize on, segment timestamps, verbose JSON. No language field is sent. Mono 16 kHz 64 kbps MP3. Scored as segments, with no second merge.
xAI	Grok Voice Transcribe 2.0	Proprietary	Yes	Yes	api.x.ai/v1/stt	grok-voice-transcribe-2.0	Diarize on, format off, filler words on, VAD threshold 0.5, language en. WAV mix sent unchanged. Words merged at 0.2 s.
OpenAI	GPT-4o Transcribe Diarize	Proprietary	Yes	Yes	api.openai.com/v1/audio/transcriptions	gpt-4o-transcribe-diarize	Diarized JSON, chunking strategy auto, language en. Mono 16 kHz 64 kbps MP3. Native segments scored as turns, with no second merge.
Microsoft	MAI Transcribe 2	Proprietary	Yes	Yes	Azure Speech fast transcription	mai-transcribe-2	API version 2025-10-15. Locale en, diarization on, enhanced mode, word timestamps, verbatim. Maximum speakers is not sent. The phrase speaker is copied onto each word. WAV mix sent unchanged. Words merged at 0.2 s.
Google	Gemini 3.5 Transcribe	Proprietary	Yes	Yes	openrouter.ai/api/v1/audio/transcriptions	google/gemini-3.5-transcribe	Google AI Studio diarization mode speaker. Language hint en. Mono 16 kHz 64 kbps MP3. Words merged at 0.2 s.
xAI	Grok STT 1.0	Proprietary	Yes	Yes	openrouter.ai/api/v1/audio/transcriptions	x-ai/grok-stt-1.0	xAI diarize flag on. Language hint en. Mono 16 kHz 64 kbps MP3. Words merged at 0.2 s. A different model from Grok Voice Transcribe 2.0.
Meta	Muse Voice Transcribe 1.0	Proprietary	Yes	Yes	api.meta.ai/v1/asr/transcribe	muse-voice-transcribe-1.0	Mode diarization, WAV, language bias English. Mixes are split into 180 s windows with 30 s of overlap. Speaker labels restart each window and are linked by overlap time. Scored as segments after that link. Nine-minute windows returned HTTP 504 and were not used.

OpenRouter requests share one shape: verbose JSON, segment and word timestamps, and a client timeout of 900 seconds. A mix that already has a hypothesis for the same configuration is not sent again. Three consecutive failures stop the run. The API key is redacted from the failure record.

Findings

The three systems that only diarize are closer than every system that labels speakers while transcribing. pyannote Community-1 is at 27.98% DER, DiariZen at 28.16%, and Nemotron 3 Diarization at 28.21%. Scribe v2, the next system, is at 30.34%. Muse Voice Transcribe 1.0 is at 35.84%. The 0.23 point spread across the first three is smaller than the spread inside any one system across the four conversations.

DER rank is not speaker-count rank. DiariZen is second on DER and reports three speakers on every conversation, so it is correct on 0 of 4. Muse is last on DER and correct on 2 of 4. The other nine match both reference speakers on all four conversations. Speaker count does not change the ranking.

Every system is worse on the one high-overlap conversation. That conversation, ocular-2bb0cfe2, has 23.97% overlap. Each system’s DER there is 1.70 to 9.83 points above its pooled DER on the three medium-overlap conversations. There is no conversation under 10% overlap, so a low-overlap rate is not reported.

Missed speech is the largest part of every system’s error, and speaker confusion is the smallest. Muse’s confusion, 9.4 points, is the only value above 4. A ±0.25 s collar changes the picture: GPT-4o Transcribe Diarize drops from 31.57% to 20.38%, the largest move, and pyannote drops from 27.98% to 18.66%. On JER the order changes. DiariZen is lowest at 26.16%, pyannote is 26.38%, and Nemotron is 26.64%.

Eleven systems

Lower is a closer match.

Pyannote Community-1 · error mix

MissedFalse alarmConfusion

Missed 15.0 · false alarm 11.0 · confusion 2.0 points of reference time. Speakers 4/4.

This system, per conversation

0fcd38f2 · 14.7%

29.59%

2bb0cfe2 · 24.0%

30.77%

86c27418 · 16.5%

28.19%

ea217639 · 17.6%

23.46%

Conversation DER, no collar.

Pooled DER, no collar, on 3,807.1 s of reference speaker time. Lower is better. Click a system to read its error mix.

The table below carries every number behind that chart in one place: the ranked DER, the collar DER, JER, the speaker count, and the pooled DER on the medium and high-overlap tiers. Rows are in ranked DER order. Clicking a row makes that system the selected one, so its error mix and per-conversation scores appear in the figure above and its predictions in the figures below.

Eleven scored systems

DER, collar, JER, and speaker count.

System	DER	DER ±0.25 s	JER	Speakers	Medium	High
Pyannote Community-1	27.98	18.66	26.38	4/4	26.99	30.77
DiariZen WavLM Large s80 MD v2	28.16	19.09	26.16	0/4	27.55	29.90
Nemotron 3 Diarization	28.21	21.39	26.64	4/4	27.76	29.46
Scribe v2	30.34	25.86	29.71	4/4	29.36	33.11
Voxtral Mini Transcribe 2	31.22	22.02	30.34	4/4	29.71	35.47
Grok Voice Transcribe 2.0	31.33	28.03	31.05	4/4	30.14	34.70
GPT-4o Transcribe Diarize	31.57	20.38	31.28	4/4	30.05	35.85
MAI Transcribe 2 (direct)	32.36	27.45	31.68	4/4	31.52	34.71
Gemini 3.5 Transcribe	32.50	24.93	32.78	4/4	31.71	34.72
Grok STT 1.0	34.28	28.23	33.91	4/4	32.89	38.18
Muse Voice Transcribe 1.0	35.84	25.56	34.16	2/4	33.26	43.09

Medium is the pool of the three conversations under 20% overlap. High is the single conversation at 23.97% overlap (ocular-2bb0cfe2). DER on one conversation moves by 2.5 to 15.3 points depending on the system. The 0.23 point spread across pyannote, DiariZen, and Nemotron is much smaller than that within-system spread. DiariZen returns 3 speakers on every conversation. Muse returns 3 on ocular-2bb0cfe2 and ocular-ea217639.

The same scores split by how much the two speakers talk over each other. On each of the three rate metrics, every system does worse on the high-overlap conversation than on the medium pool; for DER the gap runs from 1.70 points (Nemotron) to 9.83 points (Muse). Use the tabs to switch the chart between DER, collar DER, JER, and speaker count. The two panels under it show the selected system’s prediction on a 15-second window from a medium and a high-overlap conversation.

By overlap tier

Medium and high-overlap conversations, side by side.

DER (%) · lower is better

Medium overlap (10–20%)High overlap (≥20%)Overall

50250

Medium overlap

ocular-86c27418 · 436–451 s · 5.23 s of overlap in this slice

High overlap

ocular-2bb0cfe2 · 864–879 s · 6.83 s of overlap in this slice

Channel 1Channel 2OverlapUnassigned speaker

There is no conversation under 10% overlap, so a low-overlap panel is not shown. Medium pools three conversations. High is ocular-2bb0cfe2, the only conversation at 23.97%. Each timeline is 15 seconds. The hatch is time both reference channels are active. A gray bar is a hypothesized speaker with no reference partner. Click a system to change the prediction.

The last figure puts every system on one of those same two windows. Choose the medium or the high-overlap slice and press play: the reference and each system’s prediction scroll together under the playhead, so a missed backchannel or a bridged pause can be compared system by system at the same instant. The provider filter narrows the stack and the zoom widens the timeline.

One window, every system

Predictions on the clip that is playing.

The same 15 s mix, played once. Each row is one system’s prediction.

Conversation: ocular-2bb0cfe2
Speakers: 2
Overlap: 23.97% overlap on the conversation
Clip: 15 s of 15:06, from 864 s

0.00 / 15.00 sNow speaking Channel 2

Zoom1×

Reference

Speaker ASpeaker BOverlapping speechUnassigned speaker

Pyannote Community-1

miss 3.66 s · false alarm 1.44 s · confusion 0.38 s · 26.25% on this clip

Model
Prediction

DiariZen WavLM Large s80 MD v2

miss 3.10 s · false alarm 1.38 s · confusion 0.19 s · 22.39% on this clip

Model
Prediction

Nemotron 3 Diarization

miss 3.67 s · false alarm 1.45 s · confusion 0.19 s · 25.42% on this clip

Model
Prediction

Scribe v2

miss 8.27 s · false alarm 0.38 s · confusion 0.35 s · 43.09% on this clip

Model
Prediction

Voxtral Mini Transcribe 2

miss 9.78 s · false alarm 0.30 s · confusion 0.11 s · 48.77% on this clip

Model
Prediction

Grok Voice Transcribe 2.0

miss 7.67 s · false alarm 0.66 s · confusion 0.66 s · 43.05% on this clip

Model
Prediction

GPT-4o Transcribe Diarize

miss 7.56 s · false alarm 0.56 s · confusion 1.38 s · 45.49% on this clip

Model
Prediction

MAI Transcribe 2 (direct)

miss 7.68 s · false alarm 0.63 s · confusion 0.09 s · 40.25% on this clip

Model
Prediction

Gemini 3.5 Transcribe

miss 5.22 s · false alarm 0.94 s · confusion 0.29 s · 30.84% on this clip

Model
Prediction

Grok STT 1.0

miss 8.91 s · false alarm 0.23 s · confusion 0.56 s · 46.44% on this clip

Model
Prediction

Muse Voice Transcribe 1.0

miss 7.60 s · false alarm 0.92 s · confusion 0.63 s · 43.84% on this clip

Model
Prediction

Speaker A is channel 1 and Speaker B is channel 2. Each system is drawn on the reference speaker it was paired with, and a gray bar is a hypothesized speaker with no partner. The hatch is time both reference speakers are active. The clip percentages use the same no-collar accounting as the ranked score, on this 15 s window only. Rows stay in leaderboard order. A window this short can reorder systems, so the clip percentage is not the number on the leaderboard. Overlap inside the window is 6.83 s.

Reproducing a number

The harness is open source at github.com/OcularEngineering/converse-diarization. It downloads the pinned revision, re-checks the FLACs, and builds the mono-mix-v1 mix. Each mix is sent to the named model, scored, and written as one RTTM per conversation. The leaderboard is rendered from those scores. The harness is MIT. The conversation audio keeps the dataset card’s own terms.

Limits of this sample

The leaderboard is a ranking on this sample, under the convention above. It is not a result for every room, language, or microphone a diarizer will meet.

Scale

Four conversations, one language, and two speakers. There is no conversation under 10% overlap, so a low-overlap DER cannot be quoted. High overlap is one conversation, ocular-2bb0cfe2, at 23.97%. DER on a single conversation moves by 3.6 to 15.3 points depending on the system. The 0.18 point gap between pyannote and DiariZen is smaller than that within-system spread. An error bar on four conversations would be wider than most of the gaps in the table, so none is reported, and the board does not mark ties or separations with a significance test.

The channel

Separate close-talking microphones, summed without ducking, are one acoustic condition. A single distant microphone, a phone channel, a noisy street, or a meeting of more than two people is another. A little of the other voice leaks into each channel, and the release was not processed to remove that bleed. The reference therefore comes from a cleaner split than a person marking one mixed recording would have.

What counts as speech

Breaths, laughs, throat-clears, and unintelligible tags are reference speech. Their times are interpolated around the aligned lexical words. They are not measured breath or laughter onsets. Merged on their own, those tag intervals cover 627.9 s. A system tuned to lexical speech only pays for them in missed-speech time. Short fillers and unstressed function words are the timings the dataset card flags as less reliable^[4]. Two source segments have an empty word list, so their text is shown and they contribute no intervals. Age is present on four of the eight tracks and absent on the other four. The vendor audit flags are not in the public metadata.

What the runs do not equalize

Systems reached through OpenRouter, and GPT-4o and Voxtral, were scored on 64 kbps MP3. Direct xAI, Azure, Meta, pyannote, DiariZen, and Nemotron heard the PCM mix. Muse was cut into 180 s windows with 30 s of overlap because nine-minute windows returned HTTP 504, and speakers were linked across those windows by overlap time. The board does not correct for the encoding or the windowing. DiariZen’s third speaker is the published clustering range, not a count that was repaired after the fact. The public sample is the whole of this report. A larger licensed collection exists and was not scored here. Streaming output, word-level speaker error, and a collar other than ±0.25 s are not on this board.

What this ranking does not claim

01
Four conversations, one language, two speakers. No conversation is under 10% overlap, and the high-overlap tier is a single conversation. An error bar on four sessions would be wider than most gaps on the board, so none is reported.
02
Close-talking microphones, summed without ducking. A distant microphone, a phone channel, or a meeting of more than two people is a different problem. Channel bleed was left in the release.
03
Breaths, laughs, throat-clears, and unintelligible tags are reference speech. Their times are interpolated around the aligned words, not measured onsets. A system that ignores them is charged a miss.
04
OpenRouter, GPT-4o, and Voxtral heard 64 kbps MP3. Direct xAI, Azure, Meta, pyannote, DiariZen, and Nemotron heard the PCM mix. Muse was windowed after nine-minute requests returned HTTP 504. The board does not correct for that.

Conclusion

On four unscripted American English conversations, pooled DER with no collar puts the three dedicated diarizers ahead of every system that attributes speakers while transcribing. pyannote Community-1 is at 27.98%. DiariZen WavLM Large s80 MD v2 is at 28.16%. Nemotron 3 Diarization is at 28.21%. Scribe v2, the next system, is at 30.34%. Muse Voice Transcribe 1.0 is at 35.84%.

That order is not a speaker-count order, and a collar changes it. DiariZen is second on DER and correct on 0 of 4 speaker counts. Muse is last on DER and correct on 2 of 4. A ±0.25 s collar moves GPT-4o Transcribe Diarize from 31.57% to 20.38%, the largest drop, and pyannote from 27.98% to 18.66%. On JER, DiariZen is lowest at 26.16%, pyannote is 26.38%, and Nemotron is 26.64%.

Missed speech is the largest part of every system’s error, and speaker confusion is the smallest. Muse’s confusion, 9.4 points, is the only value above 4. Every system is worse on the one high-overlap conversation, by 1.70 to 9.83 points against its own medium-overlap DER. There is no low-overlap conversation, so that rate is not reported.

The number to quote from the leaderboard is the pooled DER with no collar: overlap scored, speech activity left to the system, and no speaker count supplied. The spread across the first three systems is smaller than the spread inside any one system across the four conversations. Read the ranking as a result on this sample, under that convention.

References

[1]
Park, T. J., Kanda, N., Dimitriadis, D., Han, K. J., Watanabe, S., & Narayanan, S. (2022). A review of speaker diarization: recent advances with deep learning. Computer Speech & Language, 72, 101317
Survey of the who-spoke-when task, including systems that attribute speakers while recognizing speech.
[2]
Ryant, N., Church, K., Cieri, C., Cristia, A., Du, J., Ganapathy, S., & Liberman, M. (2019). The Second DIHARD Diarization Challenge: dataset, task, and baselines. Interspeech 2019
Scores diarization error rate with no forgiveness collar and with overlap included, and introduces Jaccard error rate. Systems are not given a reference speaker count.
[3]
National Institute of Standards and Technology (2005). Spring 2005 (RT-05S) Rich Transcription Meeting Recognition Evaluation Plan. NIST
Sets the forgiveness collar at 0.25 s around each reference boundary, and treats a within-speaker pause shorter than 0.3 s as continuous speech.
[4]
Ocular AI (2026). American English transcribed Hi-Fi full-duplex two-speaker conversational dataset, sample. Hugging Face
Public sample scored in this report: separate close-talking tracks, verbatim transcripts, and word timings.
[5]
Fiscus, J. G., Ajot, J., Michel, M., & Garofolo, J. S. (2006). The Rich Transcription 2006 Spring Meeting Recognition Evaluation. Machine Learning for Multimodal Interaction, LNCS 4299
Diarization error rate is missed speech, false alarm, and speaker confusion, divided by reference speaker time. Overlap counts.
[6]
Bredin, H. (2017). pyannote.metrics: a toolkit for reproducible evaluation, diagnostic, and error analysis of speaker diarization systems. Interspeech 2017
Reference definitions for diarization error rate, including overlap and the optimal one-to-one speaker mapping.
[7]
Kuhn, H. W. (1955). The Hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(1–2), 83–97
The assignment algorithm that pairs each reference speaker with at most one hypothesis speaker.
[8]
Virtanen, P., Gommers, R., Oliphant, T. E., et al. (2020). SciPy 1.0: fundamental algorithms for scientific computing in Python. Nature Methods, 17, 261–272
Provides the polyphase resampler that takes each track from 48 kHz to 16 kHz.
[9]
Bredin, H. (2023). pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe. Interspeech 2023
Segmentation, embedding, and clustering pipeline that pyannote Community-1 descends from. The scored checkpoint is speaker-diarization-community-1, not the 2.1 model in this paper.
[10]
Han, J., Landini, F., Rohdin, J., Silnova, A., Diez, M., & Burget, L. (2025). Leveraging self-supervised learning for speaker diarization. ICASSP 2025
WavLM and Conformer local end-to-end diarization with clustering, released as DiariZen. The scored checkpoint is diarizen-wavlm-large-s80-md-v2.
[11]
Chen, S., Wang, C., Chen, Z., Wu, Y., et al. (2022). WavLM: large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6), 1505–1518
WavLM Large, the self-supervised model named in the DiariZen checkpoint.
[12]
Park, T. J., Medennikov, I., Dhawan, K., Wang, W., Huang, H., Koluguri, N. R., Puvvada, K. C., Balam, J., & Ginsburg, B. (2024). Sortformer: a novel approach for permutation-resolved speaker supervision in speech-to-text systems. arXiv:2409.06656
End-to-end diarization that orders speakers by arrival time. The architecture behind Nemotron 3 Diarization.
[13]
Medennikov, I., Park, T. J., Wang, W., Huang, H., Dhawan, K., Wang, J., Balam, J., & Ginsburg, B. (2025). Streaming Sortformer: speaker cache-based online speaker diarization with arrival-time ordering. arXiv:2507.18446
The speaker cache, FIFO, and chunk profile that the scored Nemotron 3 Diarization checkpoint runs with.