← All posts

· 6 min read

Why two providers can both claim the lowest diarization error rate

Diarization error rate depends on a tolerance window called the collar, and a figure quoted without one is not comparable to anything. What the collar does, why 0.25 s and 0 give different answers, and how to read a DER claim.

If you have compared speech-to-text providers on speaker diarization, you have probably seen two vendors claim the best diarization error rate in the same week. Usually neither is lying. They are quoting different measurements under the same name.

What DER actually counts

Diarization error rate asks a narrow question: for each moment of audio, did the system attribute the speech to the right speaker? It adds up three failures as a fraction of total speaking time.

  • **Missed speech** — someone was talking and the system heard nobody.
  • **False alarm** — the system heard a speaker where there was none.
  • **Speaker confusion** — speech was found, and attributed to the wrong person.

Lower is better, and unlike word error rate it can exceed 100%, because false alarms are not bounded by how much speech there is.

The collar is the part nobody quotes

Speaker boundaries are fuzzy. When one person stops and another starts, the exact millisecond of the handover is a judgement call, and the human annotator who produced the reference made that call differently than your system will. Scoring those boundary moments strictly punishes a system for disagreeing about something the reference itself is not confident about.

So the standard scoring tools support a **collar**: a tolerance window around each boundary that is excluded from scoring. A 0.25 second collar ignores a quarter second either side of every speaker change. Collar 0 scores everything.

This is not a small adjustment. On the same run, on the same audio, our own system scores **9.7%** on AMI-SDM at a 0.25 s collar and **14.3%** at collar 0. Same system, same recording, same metric name — a 4.6 point difference that comes entirely from which convention was used.

A vendor quoting 9.7 and a vendor quoting 14.3 could be the same system.

Overlapping speech is the second switch

The other unstated choice is whether overlapped speech counts. Real meetings have people talking over each other, and a system producing one transcript stream cannot attribute two simultaneous speakers to both. Scoring that as an error is fair — it is a real limitation — but plenty of published figures quietly exclude overlap regions, which flatters every provider by removing the hardest part of the problem.

When you see a DER number, there are therefore at least three questions before it means anything:

  1. What collar?
  2. Is overlapping speech scored, or excluded?
  3. Which dataset, and how many files?

Why the dataset matters as much as the metric

Diarization difficulty varies enormously by recording conditions. Close-talk microphones in a quiet room are a different problem from a single far-field microphone in the middle of a table picking up four people, a projector fan and a corridor.

That is why the same system can post 9.7% on AMI-SDM and 26.7% on DiPCo. DiPCo is dinner-party audio: overlapping, far-field, and genuinely hard. Neither figure is the "real" one. They describe different conditions, and the one that matters is whichever is closest to your audio.

A provider quoting a single DER without naming the dataset has told you almost nothing.

How to read a claim, and what to ask for

When you are evaluating providers, the useful questions are boring ones:

  • **What collar, and is overlap scored?** If the answer is not immediate, the number was probably copied from somewhere without the condition attached.
  • **Which datasets, and how many files per dataset?** Twenty files is a demo. Two hundred is a measurement.
  • **When was it run, and against which model version?** Speech recognition models ship constantly. A figure from eighteen months ago describes a model nobody is serving now.
  • **Can you reproduce it?** Public datasets and a public harness turn a marketing claim into something you can check in an afternoon.

What we publish, and why both numbers

We report both collars, on every dataset, on every run. The strict collar-0 figure is the one comparable to published academic model cards, and the 0.25 s figure is the one comparable to most vendor marketing. Reporting only the flattering one would make our numbers incomparable to half the field, which defeats the point of measuring.

The harness is public and the datasets are public, so anyone who doubts a figure can run it themselves rather than argue about it. That is the only real answer to "how do we know", and it is worth more to us than any single number.