Whisper speaker diarization: three ways to get speaker labels

Whisper is very good at turning speech into words, and it says nothing about who spoke them. Ask it for a meeting transcript and you get a single block of text, with no turns and no labels. Adding speaker labels — diarization — is a separate step, and there are three practical ways to do it.

Why Whisper has no speakers

Whisper is trained to predict text from 30-second windows of audio. Nothing in that objective involves telling voices apart, and a 30-second window is too short to keep a speaker's identity consistent across an hour-long recording anyway. Diarization needs a different kind of model: one that turns stretches of audio into voice embeddings and clusters them across the whole file.

So every route below pairs a transcription model with a diarization model, then lines the two up word by word. The quality of that last step decides whether a sentence that starts with one speaker and is finished by another gets split correctly.

Option 1: WhisperX and pyannote, self-hosted

WhisperX is the open-source pipeline most people reach for. It runs a fast Whisper implementation for the words, a forced-alignment model for word-level timestamps, and pyannote for speaker diarization, then assigns each word to a speaker.

• Good: no per-minute cost, audio stays on your machines, and you control every model version.
• Setup: pyannote's pretrained models are gated on Hugging Face, so you accept their terms and pass an access token.
• Running it: you need GPUs, and at volume you also need the queueing, retries and monitoring around them. That is usually the real cost.
• Accuracy: depends on the pyannote version and on tuning, particularly the number of speakers and how short turns are handled.

It is the right choice when audio cannot leave your infrastructure, or when you already run a GPU fleet and have someone to own it.

Option 2: OpenAI's gpt-4o-transcribe-diarize

OpenAI's transcription API now has a separate model for this. Request gpt-4o-transcribe-diarize with response_format="diarized_json" and you get segments with a speaker, a start and an end.

• Good: one call if you already use the OpenAI API, and you can pass short reference clips for up to four known speakers to get their names back.
• Limits: uploads are capped at 25 MB, so long recordings have to be split first, and audio over 30 seconds needs a chunking_strategy.
• Missing: the diarized format has no word-level timestamps, so you can't cut clips or time captions to the word from it.

Option 3: a hosted API with diarization built in

Most dedicated speech-to-text APIs return speakers alongside the words: Speech Revolutions, AssemblyAI, Deepgram, ElevenLabs Scribe and others. You upload the file and get the transcript, word timestamps and speaker turns back together, with no models to host.

With Speech Revolutions, speaker labels are on by default:

from speechrevolutions import SpeechRevolutions

client = SpeechRevolutions()
result = client.transcribe("meeting.mp3")

for u in result.utterances:
    print(f"[{u.start:.1f}s] {u.speaker}: {u.text}")
# [0.5s] SPEAKER_1: Hi, thanks for joining.
# [3.2s] SPEAKER_2: Happy to be here.

Every word in result.words also carries its speaker and its start and end time, so the same response drives a transcript view, clip cutting and captions.

The APIs differ most on how well they label speakers, which is easy to measure on public datasets. On AMI-SDM, NotSoFar and DiPCo meeting audio, Speech Revolutions measured 16.7% pooled diarization error rate, against 24.6% to 36.2% for the four other APIs we ran. The tables and method are on best speaker diarization API.

Which to choose

• Audio can't leave your network: WhisperX and pyannote, and budget for running them.
• Plain text is enough, and you're already on OpenAI: gpt-4o-transcribe, and skip diarization.
• You need speakers and word timing together, on long files: a hosted API. Test two or three on your own recordings: speaker labelling is where they differ most.

Coming from OpenAI? The OpenAI Whisper migration guide maps each request and response field, and the alternatives page compares the options.