Converse-STT Benchmark

The Conversational Speech-to-Text Benchmark (Converse-STT) measures how frontier AI models can accurately transcribe real-world, two-person American English conversations.

Work with us Technical Report

Models

1.22%

Lowest WER

0.55%

Lowest CER

Conversations

12,555

Reference words

Metric

Models

Model

Score

01
Scribe v2
1.22%
02
Reson8
2.92%
03
Grok Voice Transcribe 2.0
2.94%
04
Gemini 3.5 Transcribe
3.13%
05
Muse Voice Transcribe 1.0
3.31%
06
Nova-3
3.42%
07
Omi General
3.46%
08
Chirp 3
3.72%
09
MAI-Transcribe 2
3.74%
10
Nemotron 3 ASR Streaming (EN) 0.6BStreaming
4.11%
11
Parakeet TDT 0.6B v3
4.54%
12
GPT Transcribe
4.58%
13
Qwen3-ASR 1.7B
5.03%
14
GPT-4o mini Transcribe
5.11%
15
Universal-3.5 Pro
5.13%
16
Nemotron 3.5 ASR Streaming 0.6BStreaming
5.87%
17
Voxtral Mini Transcribe 2
6.59%
18
GPT Realtime WhisperStreaming
6.71%
19
Cohere Transcribe
8.51%
20
GPT-4o Transcribe
13.82%

Per-track breakdown

Every system on each of the 8 speaker tracks, in conversation order. The number beside a system is its board score; the cells are the same metric on one track. Switch between word and character error rate to see where a system loses words, and whether the loss follows the speaker or the conversation.

Metric

System · WER	0fcd38f2/ch1M1,257 w	0fcd38f2/ch2F1,806 w	2bb0cfe2/ch1F, 45–551,458 w	2bb0cfe2/ch2M1,841 w	86c27418/ch1F, 45–551,655 w	86c27418/ch2F1,547 w	ea217639/ch1F, 35–451,447 w	ea217639/ch2M, 45–551,544 w
Scribe v21.22	0.88	1.94	1.30	1.74	0.48	1.10	1.59	0.52
Reson82.92	3.34	4.76	2.67	3.10	0.79	2.84	3.46	2.33
Grok Voice Transcribe 2.02.94	2.78	4.71	3.16	3.31	1.27	3.23	2.28	2.46
Gemini 3.5 Transcribe3.13	5.17	5.09	2.61	3.26	1.27	2.97	2.14	2.59
Muse Voice Transcribe 1.03.31	3.34	4.82	3.22	4.62	1.15	3.75	2.42	2.78
Nova-33.42	3.26	4.71	4.66	3.91	1.33	2.97	3.25	3.11
Omi General3.46	2.78	5.04	4.94	3.75	1.51	3.30	3.39	2.72
Chirp 33.72	4.38	5.65	3.84	4.29	1.33	2.97	3.25	3.89
MAI-Transcribe 23.74	7.32	3.54	2.26	3.04	2.78	5.43	3.11	3.24
Nemotron 3 ASR (EN)4.11	2.78	6.26	5.21	5.00	1.27	5.24	3.39	3.17
Parakeet TDT4.54	12.81	4.87	3.22	5.05	1.57	3.43	3.25	3.56
GPT Transcribe4.58	5.73	6.09	3.98	5.49	3.44	4.72	2.56	4.34
Qwen3-ASR 1.7B5.03	3.18	7.14	7.27	4.78	2.96	5.56	5.32	3.69
GPT-4o mini Transcribe5.11	6.84	7.14	4.60	5.87	2.42	4.98	2.97	5.96
Universal-3.5 Pro5.13	11.61	4.37	2.74	3.26	1.63	12.67	2.63	3.76
Nemotron 3.5 ASR5.87	7.40	7.20	6.58	6.52	2.24	5.88	4.84	6.48
Voxtral Mini Transcribe 26.59	7.64	6.26	5.01	6.74	3.56	5.62	3.18	14.90
GPT Realtime Whisper6.71	6.68	7.48	11.04	6.30	3.81	9.31	3.04	6.22
Cohere Transcribe8.51	7.08	7.25	8.71	6.79	5.56	6.59	22.11	5.31
GPT-4o Transcribe13.82	23.07	26.47	12.35	13.63	7.73	13.90	4.70	8.10

Lower is better. Shade saturates at 12% WER and 8% CER; cells at or above it are solid. The word count under each column is that track’s normalized reference length. Rows re-sort on the chosen metric.

Key Takeaways

01
Scribe v2 is alone at the top. ElevenLabs Scribe v2 scores a 1.22% word error rate, 1.7 points and 214 words ahead of second place, which is itself a tie: Reson8 at 2.92% and Grok Voice Transcribe 2.0 at 2.94% are two words apart.
02
The spread is wider than on short-clip benchmarks. Twenty models span 1.22% to 13.82%, a factor of 11 between best and worst (7 without GPT-4o Transcribe), where public short-clip boards span under a factor of two.
03
Deletions dominate, and what gets deleted is conversational. For 17 of 20 systems, missing words are the largest error category: fragments, repeated function words, and backchannels like “yeah” and “right”.
04
The hard track is the quiet one. A gated channel where the partner’s voice bleeds through is the worst or second-worst track for 10 of 20 systems and drives nearly all cross-talk insertions.
05
Newer is not always better within a vendor. OpenAI’s four rows run from GPT Transcribe at 4.58% to GPT-4o Transcribe at 13.82% on the same audio; Google’s Chirp 3 sits within 0.6 points of Gemini 3.5 Transcribe.
06
The newest open-weight model is not the best one. Cohere Transcribe, a 2B model released in March 2026, ranks 19th at 8.51%, behind the 0.6B Nemotron and Parakeet models; its substitution count is the second-lowest on the board, and three decoder loops on near-silent audio cost it 2.6 points.
07
Configuration decides whether a score is about the model. Voxtral Mini Transcribe 2 scored 107.7% with a plain request and 6.59% with segment timestamps; GPT-4o Transcribe 21.87% without chunking and 13.82% with it; Cohere Transcribe 11.36% without the noise gate its model card recommends and 8.51% with it.
08
Every number is reproducible. Scores come from Ocular AI’s open-source harness on the public Hugging Face sample, and every run keeps the raw and normalized transcripts behind its score. Open a system to read them.

Changelog

Version history

A new version is cut when the board is scored under a different methodology. Every version stays online as published, and scores are compared within a version, not across them.

Version 2
Current
Converse-STT 2.0 · Oct 2026
Scored independently by Ocular AI with the open-source ocular-evals harness. 20 systems ranked on word error rate with character error rate alongside: each vendor's latest model plus every Version 1 model the harness can reach. Version 1 scores are not carried over.
- Every score is produced by Ocular AI's open-source harness: the dataset manifest, model configurations, normalizer, scorer, and report are code, and every run keeps the raw and normalized transcripts behind its number. Open a system on the board to read the reference beside its transcript, word-aligned.
- 20 systems: each vendor's latest general-purpose transcription model, every Version 1 model the credentials can reach (Reson8, Chirp 3, GPT-4o Transcribe, GPT-4o mini Transcribe, GPT Realtime Whisper, re-run as independent harness runs), and three models run locally from open weights: NVIDIA's Nemotron 3 and 3.5 streaming models and Cohere Transcribe. 15 proprietary, 5 open-weight.
- Character error rate is reported next to word error rate, and the board can be filtered to open-weight or proprietary systems.
- Whisper English text normalization with edit counts pooled over 12,555 reference words. The Pipecat comparison from Version 1 is retired; the Hi-Fi conversations are the single reference.
- Version 1 scores stay on their own page and are not mixed into this board, since they came from a different pipeline. Version 1 systems the harness cannot reach yet (Cartesia Ink 2, Smallest Pulse, Chirp 2, Speechmatics Linden, Deepgram Flux, Gradium, Inworld STT-1) will be added as access becomes available.
You are viewing this version
Version 1
Superseded
Converse-STT 1.0 · Sep 2026
Scored in collaboration with Cekura. 15 speech-to-text models on the Hi-Fi conversations and on public Pipecat audio, with each model's rank shift between the two.
- First release of the Conversational Speech-to-Text Benchmark: 15 models scored on word error rate.
- Two datasets side by side, Ocular AI's two-person Hi-Fi conversations and public Pipecat audio. Twelve of 15 models scored worse on conversation and four more than doubled.
- Reson8 led the conversational recordings at 2.93%; AssemblyAI Universal 3.5 Pro led Pipecat at 1.93% and ranked ninth on conversation.
View Version 1 leaderboard

Cite this benchmark

@techreport{ocularai2026converse-stt,
  title       = {Converse-STT 2.0: Benchmarking Speech-to-Text on
                 Real Two-Person American English Conversation},
  author      = {{Ocular AI}},
  institution = {Ocular AI},
  year        = {2026},
  month       = {10},
  url         = {https://www.useocular.com/converse/stt},
}

Frequently Asked Questions

The Conversational Speech-to-Text Benchmark (Converse-STT) measures how frontier AI models transcribe real-world, two-person American English conversations. Each speaker track is sent to a model as recorded, the transcript is normalized the same way as the verbatim reference, and the score is word error rate. A lower score is a more accurate transcript. Version 2 is run by Ocular AI with its open-source harness, so every number can be reproduced.

ElevenLabs Scribe v2 leads at 1.22% word error rate, 1.7 points ahead of second place, which is itself a tie: Reson8 at 2.92% and xAI Grok Voice Transcribe 2.0 at 2.94% are two words apart. The 20 systems on the board span 1.22% to 13.82%. Differences under about 0.3 points between adjacent systems should be read as ties: one WER point is about 126 words on this data.

Four American English conversations of about 15 minutes each, roughly 60 minutes across eight speaker tracks and 12,555 reference words. They were recorded at 48 kHz on separate tracks, with time-aligned verbatim transcripts, and delivered as 24-bit mono FLAC. The recordings keep the pauses, fillers, repetitions, and false starts of ordinary talk.

Word error rate: substituted, deleted, and inserted words divided by the number of words in the reference, pooled over all eight tracks. Reference and transcript both pass through Whisper English normalization, which removes fillers like “um” and “uh” and the breath and laughter tags, so a model is not penalized for leaving them out. Character error rate is reported alongside and counts the same edits over characters.

Two sets, ranked together: each vendor’s latest general-purpose transcription model, and every Version 1 model the harness can reach, so the two boards can be read model for model. That gives Google two rows and OpenAI four; within a vendor each row is an independent run. 15 are proprietary APIs and 5 are open-weight models. Hosted models were called on the vendor’s endpoint or through OpenRouter with vendor defaults and language=en; NVIDIA’s Nemotron streaming models and Cohere Transcribe were run locally from open weights. The configuration of every run is recorded in the harness, and selecting a system on the board shows its per-track counts and the reference beside its transcript.

Version 1 was scored in September 2026 in collaboration with Cekura on the same conversations and on public Pipecat audio. It is kept online as published at Version 1. Its 15 published results are not mixed into the Version 2 board: they came from a different pipeline and normalizer, so the numbers are not comparable across versions. Seven of those models, including the Version 1 leader Reson8, have been re-run with the Version 2 harness and appear on this board as ordinary runs; the rest will be added as access to their APIs becomes available. The changelog above lists what changed.

A public sample of the Hi-Fi conversations is on Hugging Face. It includes recordings and reference transcripts and is the sample every Version 2 score was produced on. It is a sample of a larger collection that Ocular AI licenses.

Yes. Add a model to the benchmark, or contact us to license Hi-Fi conversational speech.