# Provider comparison: whole-file transcription

What we learned from running the same recording through each provider's whole-file service in ZumiScribe.
Last updated 18 Nov 2026 (the cloud results are from 18 Oct 2026, VibeVoice's from 18 Nov 2026; the other transcribers on your own device are in "On your own device" below).

## The test

- One clip, `DIALOGUE.ogg`: about 4 min 5 s of conversation between an interviewer (speaker 1) and a guest (speaker 2), with
  interruptions, short replies ("Gotcha.", "Yeah.", "Right.") and overlapping speech around 3:22.
- **A probable third voice.** Around 151 to 156 s the guest's website is opened and a greeting plays: "Hey, it's Rick Frischman
  and welcome... I'm so happy you made it here." The interviewer says "Let me check that out" in between. That is most likely a
  recording playing through the call, so a third speaker label there is probably **correct**. I judged this from the transcripts,
  not by listening, so please confirm. (An earlier version of this document called those lines a phantom speaker; that was wrong.)
- Each provider was run through **Transcribe the whole file** on the same audio, with default settings, and the result
  was saved as a ZumiScribe transcript (`zenity-conf/DIALOGUE-*.json`).
- This is **one clip** and there is no verified reference transcript. It shows how the providers differ; it is not a benchmark.
  Re-run on your own audio before choosing a default.

## Results at a glance

| Provider | Speaker 1 words | Speaker 2 words | Other speakers | Total words | Timing |
|---|---|---|---|---|---|
| **ElevenLabs** (`scribe_v1`; the app now uses `scribe_v2`, not yet re-run on this clip) | 590 | 101 | speaker 3: 19 words (the website greeting, plus "40 minutes on a holiday.") | 710 | tenths of a second, continuous |
| **Gemini** (`gemini-2.5-flash`) | 587 | 103 | none | 690 | whole seconds, model-estimated |
| **Speechmatics** (batch) | 570 | 93 | speaker 3: 8 words (the greeting) | 671 | tenths of a second, by sentence |
| **Soniox** (async) | 564 | 92 | speaker 3: 13 words (the greeting) | 669 | tenths of a second, by sentence |
| **OpenAI** (`gpt-4o-transcribe-diarize`) | 551 | 1 | speaker 3: 101 words (the real speaker 2) | 653 | milliseconds, by sentence |
| **Azure Speech** (fast transcription) | 527 | 92 | none (the website greeting is lost and the interviewer's reply is put with speaker 2) | 619 | hundredths of a second, by sentence |
| **Mistral Voxtral** (`voxtral-mini-latest`) | 539 | 100 | none (the website greeting is folded into speaker 2) | 639 | tenths of a second, by sentence |
| **Modulate** (live) | 545 | 95 | speaker 3: 7 words | 647 | live finals, batched late |
| **AssemblyAI** (files, default model) | 525 | 88 | none (missed the website greeting as a third voice) | 613 | milliseconds, one entry per turn (a 100-second turn in this run; see the note) |
| **Gladia** (pre-recorded) | 545 | 38 | speaker 3: 8 words (greeting); speaker 4: 7 words (wrong) | 598 | milliseconds, short fragments |
| **Modulate** (whole file) | 565 | 38 | none | 603 | milliseconds |
| **VibeVoice** (on this Mac; `mlx-community/VibeVoice-ASR-4bit`, names as hotwords) | 576 | 106 | none (missed the website greeting as a third voice) | 682 | hundredths of a second, by segment (long segments split into sentences) |

Word counts are rough (they include filler and punctuation tokens) and are not a word error rate.

## Ranking on this clip

1. **ElevenLabs** and **Soniox** (joint first). ElevenLabs is the most complete, the most verbatim and the only run to spell
   "rickfrischman.com" correctly; Soniox is the best punctuated and easier to read.
2. **Speechmatics** - almost identical accuracy to Soniox, verbatim, correct "riot act".
3. **Gemini** - the most complete text and clean two-speaker turns, but it misses the website greeting as a third voice and
   its timestamps are rough.
4. **AssemblyAI**, **Modulate (live)** and **OpenAI** - AssemblyAI has a complete second speaker and good names, but two turn boundaries fall mid-sentence and the greeting is missed.
4. **Azure Speech** - the first real run of the Azure client, which worked first time. Well punctuated and the best of the late group on names it knows ("Riot Act", "Shimoff", "frischman.com"), but it makes the worst turn mistakes among the mid-ranked providers (see its notes).
4. **Mistral Voxtral** - clean, well punctuated two-speaker turns and the first real run of its reply layout; no third voice, some misheard names and a few run-on stretches (see its notes). Sits level with Gemini and AssemblyAI.
4. **Modulate (live)** and **OpenAI** - see notes below.
5. **Gladia**
6. **Modulate (whole file)**

## Per-provider notes

### ElevenLabs
- **Most complete and most verbatim.** 710 words: keeps every "um", "uh" and repeated word ("what, what", "he, he's"), and
  catches the short overlapping replies others merge away ("Gotcha", "So-", "True", "Sure.", "Hey, Rick Frischman").
- **Best names.** "Harvey Mackay", "Harv Eker's", "rickfrischman.com" and "rickfrischmanblog.com" are all right; "Marcie Shimoff"
  is close. Every other provider got the web address wrong.
- **Finds the third voice** (the greeting at about 151 to 156 s) and keeps speaker 1 and speaker 2 clean elsewhere.
- **Weaker points:** 32 entries, many tiny, because it follows interruptions closely (for example "Oh, golly." / "So-" / "Oh, I'm
  sorry to hear that" / "... but, you know, stuff happens"). That may be an accurate picture of the crosstalk, but it reads
  choppily. Short segments often lack a final full stop ("Gotcha", "True", "Let me check that out"), and some segments begin
  with a stray "...". One line, "40 minutes on a holiday." at about 162 s (speaker 3), appears in no other run and may be
  a hallucination or faint audio from the website video - worth listening to.
- Times to a tenth of a second and continuous (no gaps).

### Soniox
- All of speaker 2's replies are present, including the short ones. The overlap at 3:22 lands on the right speakers.
- The best-punctuated output of the group (dashes for broken-off words, colons, quoted speech).
- Finds the third voice for the greeting. "Rickfrishman.com" came out as "rickfreshman.com".

### Speechmatics
- Second speaker complete (93 words). Verbatim, so fillers and repeats stay in; good if you want to edit the text yourself.
- Only run besides ElevenLabs to get "Marci Shimoff" and "riot act" right.
- Finds the third voice for the greeting, plus a stray "Yeah." at the end labelled speaker 3 (wrong).
- "Harv Eker's" became "Harv s"; the web address was cut to "just.com".

### Gemini
- Exactly two speakers, 24 entries alternating cleanly. Most complete text after ElevenLabs.
- **Misses the third voice:** the website greeting is attributed to the guest.
- **Timestamps are the weak point.** Whole seconds, estimated by the model, and short interjections get a start equal to the end
  (for example "Gotcha." at 79 and "Let me check that out." at 153). Not good enough for replay or seeking.
- As a language model it can paraphrase; our prompt asks for an exact transcription at temperature 0.
- 15 minutes of audio per file with these features (preview model); larger files need splitting.
- Whole files only. The live API has no speaker labels.

### AssemblyAI
- **Second speaker complete** (88 words, 8 turns), no stray speakers, and good names: "Marci Shimoff", "Harv Eker's", "riot act", and "rickfrischman.com" right the first time (later written "rickfishman.com" twice).
- **Clean text**: numbers as digits ("3 questions", "2 pizzas"), dashes for broken-off words like Soniox, fillers ("um", "uh") mostly removed, repeats kept ("the, the").
- **Speaker boundaries wrong in two places**: "Good, hanging in" (speaker 2) / "there. Did you guys have a nice holiday?" (speaker 1) splits a sentence between two people, and the website greeting is cut into "Hey, it's Rick" (speaker 2) and "Fishman. Let me check that out. I'm so happy you made it here." (speaker 1). The third voice is not found at all.
- **"Gotcha." swallowed**: the guest's one-word reply is inside speaker 1's first turn, which then runs for **100 seconds** as a single entry. The app now cuts a turn into sentences using its words' own times, so new runs give a bubble per sentence; this run (and its audio replay) was before that change.
- A word error: "My father, who was very old" (the others have "Her father").
- Times in milliseconds. Not run with the default settings changed (no key terms, no expected speaker count).

### Azure Speech
- **First real run (18 Oct 2026).** The client worked first time: the request was accepted, the reply was read, two speakers came out as 1 and 2, in 18 entries and 50 lines, with times to a hundredth of a second. Speaker 1 has 527 words and speaker 2 has 92, 619 in all (Voxtral 639, ElevenLabs 710).
- **Good where it knows the words.** "Marcy Shimoff read me the Riot Act" is right (Voxtral wrote "right act"), "frischman.com" is right in the interviewer's line, and the long first turns are cleanly punctuated.
- **Loses the website greeting.** "Hey, it's Rick Frischman and welcome ..." does not appear at all; only "I'm so happy you made it here" survives, and it is put with speaker 2.
- **Wrong speaker on a short reply.** The interviewer's "Let me check that out" (about 152 s) is joined to speaker 2's turn, so a guest speech reads "I also have rickfishmanblog.com. Let me check that out. I'm so happy you made it here."
- **A turn split in the wrong place.** Speaker 2's "What are you going to do?" ends up as "What are you going?" for speaker 2 and "To do okay so I'm going to ..." at the start of speaker 1's next turn (about 120 to 123 s).
- **Misheard words.** "Harv Ecker's" (Eker), "Harvey McKay" (Mackay), "cruddy holiday hurt. father" (her father), "we actually" (he actually), "rickfishman.com" and "rickfishmanblog.com" (Frischman). The first-line "Oh golly oh I'm sorry to hear that but." gets a stray "but".
- **Keeps some fillers.** "um" survives in one place, so it is closer to verbatim than Voxtral but not as far as ElevenLabs.
- **Phrase list.** The client sends attendee names and auto-corrections as a phrase list. Whether any were set on this run is not recorded; the roster in the saved file is empty, so probably none were, and a roster with "Frischman", "Mackay" and "Eker" might fix the name errors.
- Judged from the transcript, not by listening.

### Mistral Voxtral
- **First real run (18 Oct 2026, a paid key).** The reply layout was as assumed: `segments` with `speaker_id`, `start`, `end` and `text` all arrived and were read correctly, so the client needed no change. Two speakers numbered 1 and 2, 20 entries, 59 lines, times to a tenth of a second.
- **Clean and readable.** Punctuated sentences, no stray "..." fragments, and the same two-speaker turns as Gemini. Fillers such as "um" and "uh" are dropped, so it is less verbatim than ElevenLabs or Speechmatics (639 words against 710 and 671).
- **Misses the third voice.** The website greeting at about 151 to 156 s ("Hey, it's Rick Frischman ... I'm so happy you made it here") stays with speaker 2, like Gemini and AssemblyAI, where ElevenLabs, Soniox and Speechmatics found a third speaker.
- **Names are mixed.** "rickfrischman.com" is right in the interviewer's line, which most providers got wrong, but the guest's lines say "rickfishmanblog.com", "justrickfishman.com" and "Rick Fishman". "Harvey McKay" for Mackay, "Marcy Shymoff" for Shimoff and "right act" for "riot act" are wrong. The client sends names from the attendee list and auto-corrections as hints; whether any were set on this run is not recorded, so a roster might help.
- **A few long lines.** One segment around 180 to 191 s runs on for about 10 seconds with little punctuation ("of success is who you know and I you know I think that's important ..."), which suggests the model gave up punctuating while the speaker rambled.
- **Cost:** about $0.003 a minute by public listings, so this clip cost under two cents. The free Experiment plan was not tried.
- Judged from the transcript, not by listening.

### Modulate
- **Live:** second speaker found (95 words) and a 7-word third speaker. Live finals arrive in batches and late (see the
  live-log analysis in the roadmap); the English fast model was added to reduce this.
- **Whole file:** accurate times, but it lost most of speaker 2's words (38 words) and found no third voice.

### OpenAI
- Best timing of the group (milliseconds, sentence-sized segments), clean text, removes fillers.
- **Speaker labels were wrong here:** "Gotcha." became its own speaker 2, and the rest of the real speaker 2 became speaker 3.
- Dropped one reply ("Uh, good, hanging in there.") and merged "Yeah. Did you guys have a nice holiday?" into speaker 1's turn.
- Misheard names ("rick richmond.com", "Chris Van Dyck now", "write accent").
- 25 MB limit; ZumiScribe shrinks bigger files to mono 16 kHz WAV in the browser (about 13 minutes of audio), no chunking.
- Whole files only (live has no speaker labels).

### Gladia
- Second speaker truncated (38 words), one reply garbled ("Very old in a way."), another missing.
- Finds the greeting as speaker 3 but invents a speaker 4 for "If food affects your ability to..." and "Right.", which belong to speaker 2.
- Good millisecond timing, but many short fragments.

## Patterns across providers

- **The third voice is the hardest part.** Five of eight runs (ElevenLabs, Soniox, Speechmatics, Gladia, Modulate live) separated the
  website greeting; Gemini, OpenAI and Modulate whole file did not. A user-facing "extra speaker" is not always wrong, so a
  merge suggestion should only be offered, never applied automatically.
- **Wrong extra speakers do happen** (Gladia's speaker 4, OpenAI's split of speaker 2, Speechmatics' stray "Yeah."). A "merge
  speakers" suggestion (offer to fold a speaker with a handful of words into the surrounding voices) would still help.
- **Names and web addresses** ("Rickfrishman.com") were wrong in almost every run; only ElevenLabs got it right. A custom
  vocabulary / key terms list would help where the provider supports it (Soniox, Speechmatics, Deepgram, Gladia, AssemblyAI, ElevenLabs).
- **Accuracy of words is similar** across the top runs; the differences that matter are speaker separation, completeness of short
  replies, punctuation and timing.
- **Verbatim vs clean:** ElevenLabs, Speechmatics and Soniox keep fillers and false starts; OpenAI and Gladia clean them up.

## Suggested defaults

| Need | Use |
|---|---|
| Best overall for meetings | ElevenLabs, Soniox or Speechmatics |
| Readable punctuation | Soniox |
| Complete verbatim record, names right | ElevenLabs |
| Tidy two-person turns, timing not critical | Gemini |
| Precise timestamps | ElevenLabs, Soniox, Speechmatics, OpenAI |
| Live recording with speaker labels | Soniox, Modulate, Deepgram, Speechmatics |
| Private / offline | VibeVoice for recordings you load (the most accurate here; a Mac with 16 GB of memory or more), Granite (reports nothing with speaker labels off), Voz + Who, or Local (Parakeet) |
| Free, no key, in the browser | Granite or Voz + Who |

## On this Mac: VibeVoice

Added 18 Nov 2026: Microsoft's VibeVoice-ASR (MIT licence; Microsoft says it is for research and development and not
recommended for commercial use without further testing), the 4-bit MLX copy, run by the ZumiScribe Local app through
mlx-audio 0.6.6 (setup step 6). Files and Batch only, no live recording; it writes the speakers and times itself.

| On `DIALOGUE.ogg`, M5 Max | VibeVoice (4-bit) | Local (Parakeet + Sortformer) |
|---|---|---|
| Time for the 4 min 5 s clip | about 20 s (12x real time) | 81 s |
| Word difference against ElevenLabs / Speechmatics | 5.3% / 6.9% with the names as hotwords (6.3% / 8.1% without) | 9.5% / 12.9% |
| The five hard names (Frischman, Mackay, Eker, Shimoff, riot act) | all five, with hotwords | one |
| Speakers | two (576 and 106 words, like the cloud services); the short website greeting is missed | the greeting is missed too |
| Memory, download | about 9 GB of memory while it works (let go after ten quiet minutes); about 6 GB on disk | about 6 GB on disk |

The word difference is against another service's transcript, not a checked reference, so it shows closeness to the best
cloud results rather than a true error rate. Recordings over 59 minutes are done in pieces of up to 55 minutes, each with
its own speaker numbers. Microsoft's streaming model (`VibeVoice-ASR-Streaming-1.5B`) was also tried for live: fast
enough (0.48 s per 2.9 s step), 8.5% / 8.7%, but only 1 of the 5 names and 10 languages, so it is not wired up.

## On this Mac: Phonon + Who

Added 20 Nov 2026: Fermion Research's Phonon-2 and Phonon-2.5 (weights CC BY 4.0, code Apache 2.0; 4-bit copies of NVIDIA's
Parakeet TDT 0.6B v3, 164 and 156 MB) run by the `fermion-research` package, started with `local-engine/phonon.sh`. It writes
words and times and no speakers, so the page gives the result to Who. Tried on three NotebookLM podcasts (20, 22 and 40
minutes), against the Parakeet engine's own model (`mlx-community/parakeet-tdt-0.6b-v3`) on the same files, M5 Max:

| | Parakeet v3 (parakeet-mlx) | Phonon-2 | Phonon-2.5 |
|---|---|---|---|
| Time for a 20-minute file | about 11 s (107x real time) | 4 to 8 s (155 to 370x) | 4 to 8 s |
| Download | about 2.5 GB | 164 MB | 156 MB |
| Words that differ from Parakeet | none | 1.3 to 4.3% | 1.4 to 3.4% |

There is no checked reference text, so the last row shows how much Phonon changes the text, not which is right (Fermion's
own benchmark has Phonon-2 averaging 5.21% word error against 4.96% for full Parakeet v3). Names went wrong in the same
places as with Parakeet (Stiva as "Steva" or "Steve", Hyrum's law as "Hiram", Addy Osmani as "Adios Mani"). Giving Phonon the
names as hints (up to 25) fixed "Addy Osmani" in Phonon-2 and Phonon-2.5 and nothing else we tried. VibeVoice is still the one
that gets names right. Parakeet Unified and the real-time Parakeet model were not tried.

## On your own device: Voz + Who and Granite

Added 22 to 28 Oct 2026, after the clip above was run, so they are not in its table. Both run in this browser (WebGPU) or in the ZumiScribe Local app, need no key and send no audio anywhere.

| | Voz + Who | Granite |
|---|---|---|
| Maker, licence | Desert Ant Labs (Voz is NVIDIA Parakeet TDT 0.6B v3 in their runtime; Who labels speakers), source-available, free | IBM Granite Speech 5.0 (470M, CTC), Apache 2.0 |
| Download | about 450 MB once (Voz and Who) | about 320 MB once (Who too with speaker labels) |
| Languages | 25 European | 7 (English, French, German, Spanish, Portuguese, Italian, Japanese) |
| Speed, files | the words at about 200x real time on the GPU; Who much faster on the Mac than in the browser | about 280x real time (31 minutes in under 7 s) |
| Speakers | Who | Who, on by default; can be turned off |
| Live speakers | with the ZumiScribe Local app | in the browser too (Who runs beside the passes) or with the ZumiScribe Local app |
| Reports | a monthly usage count to Desert Ant Labs | nothing, unless speaker labels are on (then Who's usage count) |
| BBC broadcast, WER against Soniox | 27.7% file, 23.2% live | 22.8% file, 22.8% live (21.5% with 10 s pieces) |

Not yet run on `DIALOGUE.ogg`. Details in [PLAN-VOZ-WHO.md](PLAN-VOZ-WHO.md) and [PLAN-GRANITE.md](PLAN-GRANITE.md).

**What this changed.** Granite only writes words; Who adds the speakers. So criterion 1 can be met by pairing a words-only model with Who, which also changes the case for Whisper (see "Whisper and WhisperX" below).

## Entry criteria

A service is added only if it passes the first five. The sixth decides between services that do.

| # | Criterion | Why |
|---|---|---|
| 1 | **Speaker labels** | The app is built around who said what. Text-only output means the person labels everything by hand. |
| 2 | **Times on the words or lines** | Playing audio from a bubble, VTT export and follow-along all need start and end times. |
| 3 | **Callable from a web page with a pasted key** | One key in Settings, and a call the browser is allowed to make (it answers the cross-origin permission check). No signed requests, no cloud project, no storage bucket. |
| 4 | **No server of our own** | Anything that needs a hosted proxy or relay is out. A small helper that runs on the user's own computer is allowed (the AssemblyAI token helper is the model), but only where the service is otherwise worth it. |
| 5 | **A general transcription service** | It must transcribe audio you send. A service that only reads transcripts of its own meeting bot, or only works inside its own call platform, does not count. |
| 6 | **Adds something we lack** | A better language set, a lower price, live mode, a privacy stance or better accuracy than what is already in. A near-duplicate is not worth another settings section and another thing to test. |

### How the left-out services score

| Service | 1 Speakers | 2 Times | 3 Key from a page | 4 No server | 5 General | Failed on |
|---|---|---|---|---|---|---|
| AWS Transcribe | yes | yes | **no** (signed requests, S3 bucket) | no (helper possible) | yes | 3 |
| Google Cloud Speech | yes | yes | **no** (cloud project, OAuth) | no | yes | 3 |
| Rev AI | yes | yes | **no** (no cross-origin permission, rechecked 17 Oct 2026) | **no** (needs a proxy) | yes | 3, 4 |
| Groq (Whisper) | **no** | yes | yes | yes | yes | 1 |
| Fireworks (Whisper) | partial | yes | yes | yes | yes | 1 |
| Twilio | channels only | yes | no | no | **no** (Twilio calls only) | 1, 3, 5 |
| Fireflies | yes | yes | yes (permission check allows any origin, 17 Oct 2026) | yes | **no** (reads transcripts of its own meeting bot; uploads unconfirmed on the free plan, 50 requests a day) | 5; possible as an import source |
| MAI-Transcribe (Microsoft) | **no** at launch | partial | yes (through Azure Speech) | yes | yes | 1; look again if it adds diarization |
| Whisper (local) | **no** by itself | yes | n/a | yes | yes | 1 |
| WhisperX (local) | yes (with pyannote) | yes | n/a | yes | yes | passes; **not built, see below** |

### Other services considered and not included

Judged on the entry criteria from what is publicly known; none was tested with a key. "Browser" is criterion 3.

| Service | What it is | Failed on | Note |
|---|---|---|---|
| **Otter, Fathom, Read.ai, tl;dv** | Meeting-bot platforms | 5 | Transcripts of their own recorded meetings. Like Fireflies, at most an import source. |
| **Rev, Sonix, Trint, Happy Scribe, Verbit** | Transcription services with job-based APIs, some with human editing | 3 or 6 | Rev AI fails the browser check. The others are job-based and add little over what is in. |
| **IBM Watson Speech to Text** | Cloud speech service | 3, 6 | Token-based sign-in rather than a pasted key, and nothing the included providers lack. Its current status was not checked. |
| **Browser speech recognition** (Web Speech API, Chrome on-device) | Free, built into the browser | 1, 2 | No speaker labels or word times. Chrome's cloud-backed version also sends audio to Google, which cuts against the privacy position. |
| **Apple on-device speech** (SpeechAnalyzer and similar) | Native Apple framework | 1, 3 | Native apps only, no speaker labels. |
| **NVIDIA Riva and Nemotron speech** | Self-hosted server engines | 6 | Parakeet already covers the same ground in the local engine. |
| **Picovoice (Leopard, Falcon)** | On-device engine with diarization | 6 | Needs an access key and a paid licence, and is file-based. A possible private option, but it adds little over Parakeet. |
| **sherpa-onnx, pyannote (local)** | Open-source local engines | 6 for now | Same decision as WhisperX: worth it only if Parakeet and Sortformer prove too weak. |

### Whisper and WhisperX

- **Whisper by itself** transcribes but does not label speakers, so it fails criterion 1. We did run it first, as WhisperLiveKit, and then moved to Parakeet plus Sortformer, which gives live speakers. WhisperLiveKit is still in `local-engine/start.sh` as an earlier experiment, and the app does not use it.
- **WhisperX** is Whisper plus pyannote speaker labels and word times. It passes criteria 1, 2, 4 and 5, but it works on finished files, not live. The local engine already transcribes files with Parakeet and Sortformer, so on criterion 6 it adds only Whisper's wider language coverage and a possibly better diarizer. **Decision: not built.** It becomes worth it if someone needs an offline language Parakeet does not cover, or if Sortformer's labels prove worse than pyannote's on real recordings. The test would be to run both on the same DIALOGUE clip and compare, as was done for the cloud providers. Nothing has been tested yet.

- **Paired with Who (28 Oct 2026).** Granite showed that a words-only model plus Who meets criterion 1, in the browser and on the Mac. Whisper could be paired the same way. What it would add on criterion 6 is languages: about 99, against 25 for Voz and Parakeet and 7 for Granite, so Arabic, Hindi, Chinese or Korean on your own device. Against it: Whisper is larger and slower than Granite and is known to invent text during silence. Not tried; it would need a trial like Granite's Phase 0.

## Other providers to consider

Browser access was checked for each with a CORS preflight from `https://zumilabs.com`; all of the below allow it. None has
been tried with a real key or on the test clip.

| Provider | Whole file | Live | Speaker labels | Notes |
|---|---|---|---|---|
| **AssemblyAI** | **added** (upload, then a transcript job, polled) | **added, through a small local helper**: AssemblyAI's token address does not accept calls from web pages, so `local-engine/token-helper.sh` fetches the token; live has speaker labels (A, B ...) | whole file and live: yes | Not yet run on the test clip. With the helper running, finished file transcripts are deleted from the account; without it they stay there until you delete them. |
| **ElevenLabs Scribe** | **added** (one multipart POST; `scribe_v2` since 28 Oct 2026, as `scribe_v1` is deprecated) | `scribe_v2_realtime` exists, not wired up: a browser can use it with a single-use token, but its docs only say a word's `speaker_id` is filled "if available", so live speaker labels need a test | whole file: yes, with word times | Tested on the clip with `scribe_v1`: see above. `scribe_v2` adds key terms (a 20% surcharge), a clean-transcript option and files up to 10 hours. |
| **Mistral Voxtral** | **added** (OpenAI-style multipart POST to `/v1/audio/transcriptions`, model `voxtral-mini-latest`; up to 3 hours) | a separate realtime model exists, not wired up | whole file: yes by Mistral's docs (a `diarize` option; segment or word times; up to 100 context-bias terms; 13 languages) | Browser access confirmed on 17 Oct 2026. Tried on the test clip on 18 Oct 2026: the reply layout (`segments` with `speaker_id`, `start`, `end`) was as expected; two speakers, no third voice. A review notes overlapping speech comes out as one speaker. |
| **Azure AI Speech (Microsoft Foundry)** | **added, tried on the test clip on 18 Oct 2026** (fast transcription; diarization up to 35 speakers; phrase list; 500 MB / 5 hours; browser calls allowed) | not built: it needs Microsoft's own SDK and protocol (its token address allows browser calls, so it could be done later) | whole file: yes | Not yet run on the test clip. Needs an Azure Speech resource and its region. |
| **Rev AI** | yes (job-based) | yes | yes | Enterprise-oriented; preflight returned 204 without CORS headers in our check, so it needs a proxy before the browser can call it. |
| **Groq (Whisper)** | yes, very fast | no | **no** | Fast and cheap, no diarization; useful only as a text-only option. |
| **Fireworks (Whisper)** | yes | partial | optional diarization via a separate step | Similar to Groq. |
| **AWS Transcribe** | yes, but only through an S3 bucket (upload, start a job, poll) | yes, over a signed WebSocket in AWS's own binary format | yes (up to 30 speakers) | **Not included: see "Why AWS Transcribe is not included" below.** |
| **Google Cloud Speech (Chirp)** | yes | yes | yes | Cloud-account setup and signed or token-based requests rather than a simple key; not included for now, for the same kind of reasons as AWS. |
| **Twilio** | yes (Batch Transcription of a Twilio recording or a file address) | yes, but only for Twilio calls | channels, not speaker labels | Tied to Twilio's own call platform; list price $0.024 to $0.027 a minute; batch is not HIPAA-eligible. Not included. |
| **pyannote / WhisperX / NeMo Sortformer (local)** | yes | no | yes (best open diarization) | Would extend the local engine so speaker separation also works offline. |

### Why AWS Transcribe is not included

AWS Transcribe is a good service (speaker labels for up to 30 speakers, in files and live), and the browser is not strictly shut out: on 17 Oct 2026 a test request showed that Transcribe's API answers a browser's permission check for any origin. It is left out because making it work from a web page costs far more than the other providers, and the cheap ways to do it are the unsafe ones.

| Obstacle | Why it matters here |
|---|---|
| **Signed requests, not a key** | Every AWS call must be signed with an access key and secret (SigV4). A web page would have to hold those, and AWS's own advice is never to put long-lived keys in browser code. The safe forms (temporary credentials from a Cognito identity pool, or a server that signs) need an AWS account set up for it, not just a pasted key. |
| **Files go through S3** | Transcribe's batch service reads a file from an S3 bucket. The page would have to upload to the user's bucket first (the bucket needs a CORS setting that only the owner can add), start a job, poll it, and read the result from S3 again. The other whole-file providers take the file in one request. |
| **Live uses AWS's own message format** | Streaming runs over a signed WebSocket carrying a binary "event stream" framing, not plain audio frames. It is a sizeable piece of code to write and test, and the streaming address did not answer a browser permission check in our test (it returned 404). |
| **Setup burden for the user** | A key plus a region is what a person will paste. An IAM user or role, an S3 bucket, a CORS rule and a policy is a project. |
| **Already covered** | Soniox, Deepgram, Speechmatics, AssemblyAI and Modulate give live speaker labels with a pasted key; Azure Speech and the others give files. AWS adds little that is not already there for people without AWS. |

**If AWS is wanted** (for example in a company that is all on AWS and cannot send audio elsewhere): the way I would build it is a small local helper, like the AssemblyAI token helper, that uses the person's own AWS credentials (their profile, or an assumed role) on their computer, never in the page. It could sign the requests, put the file in a bucket and run the job, or hand the page a pre-signed live address. The page would then only ever talk to `127.0.0.1`. That is a real project (several days with tests, and it needs an AWS account to try it), so it is only worth doing for a specific user.

### Recommendation

1. **ElevenLabs Scribe** is now added (see above) and did very well. **AssemblyAI** is now added for files (live is blocked from the browser, see the table).
2. **Mistral Voxtral** is now added for files. Run the test clip through it with a key and send the JSON, so the reply layout can be checked and the result ranked.
3. Add a **merge-speakers suggestion** and **key terms** support before adding more providers; they improve every result.
4. Skip Groq, Fireworks, Rev AI, Twilio, Google Cloud Speech and AWS Transcribe for now (AWS, with the reasons and the way to build it if wanted, is explained above; Azure Speech turned out to be simple for files and is added).

## Caveats

- Judgements about who is speaking (especially the third voice) come from reading the transcripts, not from listening.
- One short two-person clip; results will vary with audio quality, accents, more speakers and length.
- The Gemini model (`gemini-2.5-flash`) and the OpenAI model were used at default settings; a different Gemini model or
  prompt could change the timing and speaker behaviour.
- Word counts are a rough proxy (they include filler and punctuation tokens) and do not measure word error rate. There is no
  verified reference transcript beyond the one in the Modulate review.
