Converse-Diarization Benchmark

The Conversational Speaker Diarization Benchmark (Converse-Diarization) measures how frontier AI models attribute speech to the right speaker in real-world, two-person American English conversations.

Work with us Technical Report

Models

27.98%

Lowest DER

26.16%

Lowest JER

Conversations

Hour of audio

Metric

Speaker
Overlap

Models

Model

Score

01
Pyannote Community-1
27.98%
02
DiariZen WavLM Large s80 MD v2
28.16%
03
Nemotron 3 Diarization
28.21%
04
Scribe v2
30.34%
05
Voxtral Mini Transcribe 2
31.22%
06
Grok Voice Transcribe 2.0
31.33%
07
GPT-4o Transcribe Diarize
31.57%
08
MAI Transcribe 2 (direct)
32.36%
09
Gemini 3.5 Transcribe
32.50%
10
Qwen3-ASR 1.7B + FunASR speakers
32.91%
11
Grok STT 1.0
34.28%
12
Muse Voice Transcribe 1.0
35.84%

Audio

The Converse-Diarization leaderboard

Last Updated: Oct 14, 2026 · Version 1.0

Converse-Diarization measures whether a model can tell who is speaking, and when, in a real two-person conversation. The score is diarization error rate. A lower score is a closer match to who actually spoke.

Ocular AI supplied the conversations: 48 kHz American English, recorded on separate speaker tracks with time-aligned, verbatim transcripts. The set is four conversations of about 15 minutes each, roughly 60 minutes across eight tracks, delivered as 24-bit mono FLAC. The recordings keep the pauses, fillers, repetitions, false starts, and overlapping speech of ordinary talk.

The model hears the mixed conversation. The separate tracks are the reference: each span of speech already belongs to a speaker. Missed speech, false alarms, and speaker confusions all count.

Technical Report Dataset Harness Add your model

Category

Speaker Diarization

Languages

English

Pyannote Community-1(pyannote)

Loading the speaker timeline.

Every system on one clip

The same 15 seconds of conversation, with the reference timeline on top and every system’s prediction stacked under it. Press play and they scroll together under the playhead, so a missed backchannel, a bridged pause, or a turn given to the wrong speaker can be compared system by system at the same instant. Switch between a medium and a high-overlap conversation, filter the stack by provider, and zoom the timeline to read the boundaries.

The same 15 s mix, played once. Each row is one system’s prediction.

Conversation: ocular-2bb0cfe2
Speakers: 2
Overlap: 23.97% overlap on the conversation
Clip: 15 s of 15:06, from 864 s

0.00 / 15.00 sNow speaking Channel 2

Zoom1×

Reference

Speaker ASpeaker BOverlapping speechUnassigned speaker

Pyannote Community-1

miss 3.66 s · false alarm 1.44 s · confusion 0.38 s · 26.25% on this clip

Model
Prediction

DiariZen WavLM Large s80 MD v2

miss 3.10 s · false alarm 1.38 s · confusion 0.19 s · 22.39% on this clip

Model
Prediction

Nemotron 3 Diarization

miss 3.67 s · false alarm 1.45 s · confusion 0.19 s · 25.42% on this clip

Model
Prediction

Scribe v2

miss 8.27 s · false alarm 0.38 s · confusion 0.35 s · 43.09% on this clip

Model
Prediction

Voxtral Mini Transcribe 2

miss 9.78 s · false alarm 0.30 s · confusion 0.11 s · 48.77% on this clip

Model
Prediction

Grok Voice Transcribe 2.0

miss 7.67 s · false alarm 0.66 s · confusion 0.66 s · 43.05% on this clip

Model
Prediction

GPT-4o Transcribe Diarize

miss 7.56 s · false alarm 0.56 s · confusion 1.38 s · 45.49% on this clip

Model
Prediction

MAI Transcribe 2 (direct)

miss 7.68 s · false alarm 0.63 s · confusion 0.09 s · 40.25% on this clip

Model
Prediction

Gemini 3.5 Transcribe

miss 5.22 s · false alarm 0.94 s · confusion 0.29 s · 30.84% on this clip

Model
Prediction

Qwen3-ASR 1.7B + FunASR speakers

miss 8.06 s · false alarm 0.63 s · confusion 0.22 s · 42.67% on this clip

Model
Prediction

Grok STT 1.0

miss 8.91 s · false alarm 0.23 s · confusion 0.56 s · 46.44% on this clip

Model
Prediction

Muse Voice Transcribe 1.0

miss 7.60 s · false alarm 0.92 s · confusion 0.63 s · 43.84% on this clip

Model
Prediction

Speaker A is channel 1 and Speaker B is channel 2. Each system is drawn on the reference speaker it was paired with, and a gray bar is a hypothesized speaker with no partner. The hatch is time both reference speakers are active. The clip percentages use the same no-collar accounting as the ranked score, on this 15 s window only. Rows stay in leaderboard order. A window this short can reorder systems, so the clip percentage is not the number on the leaderboard. Overlap inside the window is 6.83 s.

Per-conversation breakdown

Every system on each of the 4 conversations, with the overlap share of each one under its name. The number beside a system is its board score; the cells are the same metric on one conversation. Switch between DER, DER with a ±0.25 s collar, and JER to see how much of a system’s error is boundary timing and whether it is worse where the two speakers talk over each other.

Metric

System · DER	0fcd38f2medium overlap · 14.7%15 min	2bb0cfe2high overlap · 24.0%15 min	86c27418medium overlap · 16.5%15 min	ea217639medium overlap · 17.6%15 min
Pyannote Community-127.98	29.59	30.77	28.19	23.46
DiariZen WavLM Large s80 MD v228.16	28.46	29.90	29.14	25.21
Nemotron 3 Diarization28.21	28.55	29.46	26.96	27.79
Scribe v230.34	28.54	33.11	28.82	30.62
Voxtral Mini Transcribe 231.22	31.79	35.47	30.84	26.74
Grok Voice Transcribe 2.031.33	30.84	34.70	28.74	30.81
GPT-4o Transcribe Diarize31.57	33.33	35.85	29.81	27.27
MAI Transcribe 2 (direct)32.36	31.11	34.71	31.91	31.54
Gemini 3.5 Transcribe32.50	29.57	34.72	36.98	28.70
Qwen3-ASR 1.7B + FunASR speakers32.91	33.38	36.58	31.28	30.29
Grok STT 1.034.28	34.02	38.18	31.65	33.02
Muse Voice Transcribe 1.035.84	30.25	43.09	27.77	41.21

Lower is better. Shade saturates at 50% DER and JER and 40% collar DER; cells at or above it are solid. Overlap is the share of reference time where both speakers are talking. Rows re-sort on the chosen metric.

Key Takeaways

01
Who spoke matters as much as what was said. A transcript can be accurate and still assign a turn, a name, or a correction to the wrong person.
02
The reference is a separate track for each speaker. Two people are recorded at the same time, each on their own microphone, so overlap can be attributed instead of inferred from a single mix.
03
The model hears the conversation mixed. Scoring uses the separate tracks and their time-aligned transcripts as the reference for who was speaking.
04
The score is diarization error rate. Missed speech, false alarms, and speech given to the wrong speaker are divided by the speech in the reference. A lower score is a more accurate account of who spoke when.
05
The recordings are real two-person conversation. Four American English conversations, about 60 minutes across eight speaker tracks, recorded at 48 kHz as 24-bit mono FLAC.
06
Overlap stays in the evaluation. Interruptions, backchannels, and speech that lands on top of the other person are part of the reference.

Cite this benchmark

@techreport{ocularai2026converse-diarization,
  title       = {Converse-Diarization: Benchmarking Speaker Diarization
                 on Real Two-Person American English Conversation},
  author      = {{Ocular AI}},
  institution = {Ocular AI},
  year        = {2026},
  month       = {10},
  url         = {https://www.useocular.com/converse/diarization},
}

Frequently Asked Questions

The Conversational Speaker Diarization Benchmark (Converse-Diarization) measures whether a frontier model can tell who is speaking, and when, in a real two-person American English conversation. The model hears the conversation mixed. The reference is a separate track for each speaker, with time-aligned verbatim transcripts. A lower diarization error rate is a closer match to who actually spoke.

The score is diarization error rate: missed speech, false alarms, and speech assigned to the wrong speaker, divided by the speech in the reference. Breath and laughter tags are not speech, so they are not part of the reference. Overlap still counts. A turn only scores as correct when it is given to the person who spoke.

Ocular AI supplied four American English conversations of about 15 minutes each, roughly 60 minutes across eight speaker tracks. They were recorded at 48 kHz on separate tracks, with time-aligned verbatim transcripts, and delivered as 24-bit mono FLAC. The recordings keep the pauses, fillers, repetitions, false starts, and overlapping speech of ordinary talk.

A separate track already knows who is speaking. Products usually hear one mix. The separate tracks are the answer key: they show who was speaking through interruptions, backchannels, and overlap. The model is scored on the mix, against that key.

A public sample of the conversations is on Hugging Face. It includes the separate speaker tracks and reference transcripts. It is a sample of a larger collection.

Yes. Add a model to the benchmark, or contact us to license Hi-Fi conversational speech.