Get Data

Ocular Converse

Converse-Diarization Benchmark

The Conversational Speaker Diarization Benchmark (Converse-Diarization) measures how frontier AI models attribute speech to the right speaker in real-world, two-person American English conversations.

Work with usTechnical Report
11

Models

27.98%

Lowest DER

26.16%

Lowest JER

4

Conversations

1

Hour of audio

Metric

Speaker
Overlap

Models

Model

Score

  1. 01

    Pyannote Community-1 (pyannote)

    27.98%
  2. 02

    DiariZen WavLM Large s80 MD v2 (BUT)

    28.16%
  3. 03

    Nemotron 3 Diarization (NVIDIA)

    28.21%
  4. 04

    Scribe v2 (ElevenLabs)

    30.34%
  5. 05

    Voxtral Mini Transcribe 2 (Mistral)

    31.22%
  6. 06

    Grok Voice Transcribe 2.0 (xAI)

    31.33%
  7. 07

    GPT-4o Transcribe Diarize (OpenAI)

    31.57%
  8. 08

    MAI Transcribe 2 (direct) (Microsoft)

    32.36%
  9. 09

    Gemini 3.5 Transcribe (Google)

    32.50%
  10. 10

    Grok STT 1.0 (xAI)

    34.28%
  11. 11

    Muse Voice Transcribe 1.0 (Meta)

    35.84%

Key Takeaways

  • 01

    Who spoke matters as much as what was said. A transcript can be accurate and still assign a turn, a name, or a correction to the wrong person.

  • 02

    The reference is a separate track for each speaker. Two people are recorded at the same time, each on their own microphone, so overlap can be attributed instead of inferred from a single mix.

  • 03

    The model hears the conversation mixed. Scoring uses the separate tracks and their time-aligned transcripts as the reference for who was speaking.

  • 04

    The score is diarization error rate. Missed speech, false alarms, and speech given to the wrong speaker are divided by the speech in the reference. A lower score is a more accurate account of who spoke when.

  • 05

    The recordings are real two-person conversation. Four American English conversations, about 60 minutes across eight speaker tracks, recorded at 48 kHz as 24-bit mono FLAC.

  • 06

    Overlap stays in the evaluation. Interruptions, backchannels, and speech that lands on top of the other person are part of the reference.

Frequently Asked Questions

The Conversational Speaker Diarization Benchmark (Converse-Diarization) measures whether a frontier model can tell who is speaking, and when, in a real two-person American English conversation. The model hears the conversation mixed. The reference is a separate track for each speaker, with time-aligned verbatim transcripts. A lower diarization error rate is a closer match to who actually spoke.

The score is diarization error rate: missed speech, false alarms, and speech assigned to the wrong speaker, divided by the speech in the reference. Breath and laughter tags are not speech, so they are not part of the reference. Overlap still counts. A turn only scores as correct when it is given to the person who spoke.

Ocular AI supplied four American English conversations of about 15 minutes each, roughly 60 minutes across eight speaker tracks. They were recorded at 48 kHz on separate tracks, with time-aligned verbatim transcripts, and delivered as 24-bit mono FLAC. The recordings keep the pauses, fillers, repetitions, false starts, and overlapping speech of ordinary talk.

A separate track already knows who is speaking. Products usually hear one mix. The separate tracks are the answer key: they show who was speaking through interruptions, backchannels, and overlap. The model is scored on the mix, against that key.

A public sample of the conversations is on Hugging Face. It includes the separate speaker tracks and reference transcripts. It is a sample of a larger collection.

Yes. Add a model to the benchmark, or contact us to license Hi-Fi conversational speech.

Ready to bring AI into the real world?
Get Data
Converse-Diarization Benchmark · Ocular Converse | Ocular AI