Speech-to-text benchmarks
Measured by us on public datasets, with a harness anyone can run and datasets anyone can download. Zephyr leads on diarization error rate on every subset scored, at both the strict and the lenient collar.
Full run 2026-07-12 · clean + SPGISpeech re-run 2026-07-13
Benchmarks
#1 in every category.
Head to head with AssemblyAI, Deepgram, ElevenLabs and Soniox on public datasets.
Speaker diarization
DER measures missed or misassigned speaker turns.
- Dataset
- AMI-SDMNotSoFar
- Reading
- ↓ Lower is better
- SpeechRevolutions Zephyr9.70%AMI-SDM10.9%NotSoFar
- AssemblyAI Universal 3.5 Pro25.3%AMI-SDM23.3%NotSoFar
- ElevenLabs Scribe V233.7%AMI-SDM24.4%NotSoFar
- Deepgram Nova-330.9%AMI-SDM28.4%NotSoFar
- Soniox V536.6%AMI-SDM32.2%NotSoFar
Benchmarks run by Speech Revolutions on public datasets, with the full method and results published. Head to head with Deepgram and AssemblyAI.
Diarization error rate
DER measures who spoke when: missed speech, false alarms and speaker confusion, as a percentage of speaking time. Lower is better. These figures are overlap-aware at a 0.25 s collar, which is the lenient, widely quoted condition.
| Dataset | Zephyr | Best other provider |
|---|---|---|
| AMI-SDM | 9.7% | 25.3% — AssemblyAI |
| NotSoFar | 10.9% | 23.3% — AssemblyAI |
| DiPCo | 26.7% | 28.9% — AssemblyAI |
| Pooled | 16.7% | 24.6% — AssemblyAI |
At the stricter collar-0 condition — the one the DiariZen and pyannote model cards report, and so the one to compare against a published model card — we lead too: 14.3% on AMI-SDM and 24.4% pooled, against 33.5% and 35.9% for the next best. The suite emits both collars on every run; neither is a default you have to infer.
What was tested
Every provider was called through its own public API with the features the metric needs. Speech recognition moves quickly, so the exact models matter as much as the numbers:
| Provider | Model |
|---|---|
| Speech Revolutions (Zephyr) | Whisper-large-v3-turbo (TensorRT-LLM) + DiariZen diarization |
| Deepgram | nova-3 |
| AssemblyAI | Universal 3.5 Pro |
| ElevenLabs | scribe_v2 |
| Soniox | stt-async-v5 |
| OpenAI | gpt-4o-transcribe |
Datasets are public: AMI-SDM, NotSoFar and DiPCo for diarization; LibriSpeech and SPGISpeech for word error rate. Word error rate uses the standard Whisper text normalizers. Where a provider offers keyword boosting, the same per-dataset glossary was supplied to each one that has the feature; providers without one are shown un-boosted, which if anything flatters them.
Speaker diarization in depth: best speaker diarization API. Head to head: vs Deepgram, vs AssemblyAI, vs ElevenLabs, vs Soniox.