BenchmarkIntroducing The Converse-Diarization Benchmark

The Converse-STT Benchmark

The Conversational Speech-to-Text Benchmark measures how accurately frontier AI models transcribe real-world, two-person American English conversations. Version 2: 20 models, scored with Ocular AI's open-source harness.

View Leaderboard20 Technical Report Dataset

Metric

Model

Score

01
Scribe v2 (ElevenLabs)
1.22%
02
Reson8 (Reson8)
2.92%
03
Grok Voice Transcribe 2.0 (xAI)
2.94%
04
Gemini 3.5 Transcribe (Google)
3.13%
05
Muse Voice Transcribe 1.0 (Meta)
3.31%
06
Nova-3 (Deepgram)
3.42%
07
Omi General (Omi)
3.46%
08
Chirp 3 (Google)
3.72%
09
MAI-Transcribe 2 (Microsoft AI)
3.74%
10
Nemotron 3 ASR Streaming (EN) 0.6B (NVIDIA)
4.11%
11
Parakeet TDT 0.6B v3 (NVIDIA)
4.54%
12
GPT Transcribe (OpenAI)
4.58%
13
Qwen3-ASR 1.7B (Alibaba)
5.03%
14
GPT-4o mini Transcribe (OpenAI)
5.11%
15
Universal-3.5 Pro (AssemblyAI)
5.13%
16
Nemotron 3.5 ASR Streaming 0.6B (NVIDIA)
5.87%
17
Voxtral Mini Transcribe 2 (Mistral AI)
6.59%
18
GPT Realtime Whisper (OpenAI)
6.71%
19
Cohere Transcribe (Cohere)
8.51%
20
GPT-4o Transcribe (OpenAI)
13.82%

Encoding Human Expertise into Machines.

Human voice, vision, reasoning, & expertise — encoded into training data by our Expert Network and Data Foundry.

Get Data Research & Evals

The Converse-Diarization Benchmark

The Conversational Speaker Diarization Benchmark measures how accurately frontier AI models attribute speech to the right speaker in real-world, two-person American English conversations. 11 models, scored on diarization error rate.

View Leaderboard12 Technical Report Dataset

Metric

Model

Score

01
Pyannote Community-1 (pyannote)
27.98%
02
DiariZen WavLM Large s80 MD v2 (BUT)
28.16%
03
Nemotron 3 Diarization (NVIDIA)
28.21%
04
Scribe v2 (ElevenLabs)
30.34%
05
Voxtral Mini Transcribe 2 (Mistral)
31.22%
06
Grok Voice Transcribe 2.0 (xAI)
31.33%
07
GPT-4o Transcribe Diarize (OpenAI)
31.57%
08
MAI Transcribe 2 (direct) (Microsoft)
32.36%
09
Gemini 3.5 Transcribe (Google)
32.50%
10
Qwen3-ASR 1.7B + FunASR speakers (Alibaba)
32.91%
11
Grok STT 1.0 (xAI)
34.28%
12
Muse Voice Transcribe 1.0 (Meta)
35.84%

Fueling the World's Top Frontier AI Labs
and Fortune 100 Enterprises
with Frontier AI Data

Built By Engineers From

Backed By

Research & Evals

Converse Benchmarks

We believe voice will be the natural interface between people and AI, but human speech is uniquely hard to get right. Unlike code or math, there is rarely one right answer: a good conversation depends on the setting, and every utterance carries emotion, tone, pace, accent, and language on top of its words. Converse tests whether voice-native frontier models can understand, speak, and act on real-world speech the way people do.

View all

Speaker Diarization

Converse-Diarization

Evaluates frontier AI and speaker diarization models on Real World Conversational Data, measuring diarization error rate (DER): missed, false, and mis-attributed speech.

Diarization Error Rate · lower is better

Show 7 more

Speech-to-Text (STT)

Converse-STT

Evaluates frontier AI and speech-to-text models on Real World Conversational Data, measuring word error rate (WER) and character error rate (CER) with Ocular AI's open-source harness.

Word Error Rate · lower is better

Show 15 more

View all

The Problem

AIcansolveolympiadproblems.Itstilllackshumannuance.

We'reinaTechnologicalRenaissance.Modelshavememorizedtheinternet.Theycanwriteessays,passbarexams,andprovetheorems.Butaskonetonegotiateadeal,comfortagrievingpatient,orspeakwiththewarmthandtimingofarealhumanvoice—andtheillusionbreaks.

Humanexpertiseisstaggeringlycomplex.Itspanseverymodalityandeveryculture—howasurgeonseestheoneshadowonascanthatchangeseverything,howatraderhearsriskinapausebetweenwords,howatherapistreadsafacebeforeasinglesentenceisspoken,howmeaningshiftsbetweenlanguages,accents,anddialectsthatnomodelwastrainedtounderstand.

Noneofthiswaseverinthetrainingdata.Scalingcomputewon'tconjureit.Syntheticdatawon'tapproximateit.Thebottleneckwasneverintelligence—it'stherichnessoflivedhumanexperience.

Our Solution

Weencodehumanexpertiseintomodelsthatworkfortherealworld.

We'reanappliedresearchlabbuildingthedatainfrastructureandhumanexpertisenetworktoencodereal-worldknowledgeintofrontiermodels—acrosseverymodality,language,anddomain.

Wepartnerwitheliteprofessionalstocapturewhattheyactuallydo.Thereasoningbehindadiagnosis.Theinstinctinanegotiation.Thecadenceofanativespeaker.Theengineeringinsightinadesigndecision.Themicro-expressionsamachinehasneverbeentaughttosee.

ThisflowsthroughourDataFoundry—apurpose-builtenginethattransformsrawexpertiseintostructuredtrainingdata,alignmentsignals,andrigorousevaluationsatscale.FromPhDmathematiciansandvoiceactorstoconstitutionallawyersandlinguists,everydiscipline,accent,anddialectgetsitsownpipeline.