Speech-to-text benchmarks
Measured by us on public datasets, with a harness anyone can run and datasets anyone can download. Zephyr leads on diarization error rate on every subset scored, at both the strict and the lenient collar.
Full run 2026-07-12 · clean + SPGISpeech re-run 2026-07-13
Speaker diarization
DER measures missed or misassigned speaker turns. Lower DER means cleaner speaker labels for analytics, compliance, and conversation intelligence.
Datasets
- AMI-SDM
- NotSoFar
↓ Lower is better
- SpeechRevolutions ZephyrLowest DER9.70% · 10.9%
- AssemblyAI Universal 3.5 Pro25.3% · 23.3%
- ElevenLabs Scribe V233.7% · 24.4%
- Deepgram Nova-330.9% · 28.4%
- Soniox V536.6% · 32.2%
SpeechRevolutions Zephyr
AssemblyAI Universal 3.5 Pro
ElevenLabs Scribe V2
Deepgram Nova-3
Soniox V5
Why Zephyr leads
Zephyr delivers the industry's lowest speaker diarization error rate, ranking #1 across every evaluated dataset. Fewer missed speakersand cleaner speaker attribution make it the ideal choice for meetings, podcasts, contact centers, and conversation intelligence.
Measured by Speech Revolutions on public datasets. View methodology →
Diarization error rate
DER measures who spoke when: missed speech, false alarms and speaker confusion, as a percentage of speaking time. Lower is better. These figures are overlap-aware at a 0.25 s collar, which is the lenient, widely quoted condition.
| Dataset | Zephyr | Best other provider |
|---|---|---|
| AMI-SDM | 9.7% | 25.3% — AssemblyAI |
| NotSoFar | 10.9% | 23.3% — AssemblyAI |
| DiPCo | 26.7% | 28.9% — AssemblyAI |
| Pooled | 16.7% | 24.6% — AssemblyAI |
At the stricter collar-0 condition — the one the DiariZen and pyannote model cards report, and so the one to compare against a published model card — we lead too: 14.3% on AMI-SDM and 24.4% pooled, against 33.5% and 35.9% for the next best. The suite emits both collars on every run; neither is a default you have to infer.
What was tested
Every provider was called through its own public API with the features the metric needs. Speech recognition moves quickly, so the exact models matter as much as the numbers:
| Provider | Model |
|---|---|
| Speech Revolutions (Zephyr) | Whisper-large-v3-turbo (TensorRT-LLM) + DiariZen diarization |
| Deepgram | nova-3 |
| AssemblyAI | API default at the time of the run (no model pinned) |
| ElevenLabs | scribe_v2 |
| Soniox | stt-async-v5 |
| OpenAI | gpt-4o-transcribe |
Datasets are public: AMI-SDM, NotSoFar and DiPCo for diarization; LibriSpeech and SPGISpeech for word error rate. Word error rate uses the standard Whisper text normalizers. Where a provider offers keyword boosting, the same per-dataset glossary was supplied to each one that has the feature; providers without one are shown un-boosted, which if anything flatters them.