A speech recognizer can post a respectable average score and still fail on the transmission that matters. Deleting not, changing fifteen to fifty, inventing words during squelch noise, or taking longer than the workflow allows are different failures with different consequences. Radio adds clipping, narrowband audio, intermittent noise, short turns, and repeated local vocabulary. The evaluation must be designed around those conditions before a model is chosen.
Start with WER, then open it up
Word error rate aligns a reference transcript with a hypothesis and counts substitutions, deletions, and insertions relative to the number of reference words. It remains useful because it is inspectable and widely comparable. But preprocessing decisions matter. Case, punctuation, filled pauses, compound words, abbreviations, and number formatting can change the score without changing the audio. A report should publish both the reference convention and the exact normalization used, and should preserve raw outputs for audit.[4]
One overall WER is not evidence of robustness. CEASR compared systems across public corpora and reported accuracy differences of up to a factor of ten for a single system across different corpora. Whisper's own paper likewise reports substantially different WER across test sets. Neither study is a radio benchmark, but both show why a clean-speech average cannot be transferred to a radio channel without direct testing.[2][4]
Count errors that change the work
WER assigns the same unit cost to every word edit. In practice, losing a filler and changing a quantity can have very different effects. Research on simulated patient-doctor conversations found that proposed error-severity measures captured characteristics that WER did not, while also discussing limits of those measures. The transfer to radio work is not automatic. It supports the method choice of supplementing WER with a task-defined review rather than claiming that any universal severity score already exists.[1]
For a radio evaluation, pre-register categories such as people or call signs, locations, asset identifiers, quantities, units, negations, and short acknowledgements. Report precision, recall, and error type for each category with its denominator. Keep the reference audio available to authorized reviewers, and adjudicate disagreements rather than forcing uncertain spans into a confident transcript. A model ranking that changes when critical entities replace all words as the target is a result, not an inconvenience.[1]
Test what happens when nobody says the words
Insertions are especially important when a model generates plausible text from silence, noise, or weak speech. A 2024 audit found hallucinations in a bounded set of Whisper transcriptions and reported that longer pauses were associated with more hallucinations for the tested speakers. It found no comparable hallucinations in the competing services tested on the 187 identified segments. That contradictory result matters: hallucination was not universal across systems in that experiment, and findings about one model should not become a claim about all ASR.[3]
A radio benchmark should include keyed silence, squelch tails, carrier noise, clipped starts, background machinery, empty captures, and speech from another channel where available lawfully. Report false words per speech-free minute, the proportion of speech-free clips with any output, and whether invented text contains a critical entity. Timeouts, decoder failures, and filtered clips stay in the denominator. A separate voice-activity detector may reduce false outputs, but then the benchmark is evaluating the full pipeline and should say so.[3]
Keep the test unknown to the system builders
Leakage occurs when evaluation material influences training, vocabulary selection, prompts, thresholds, normalization choices, or model selection. It can also occur when adjacent clips from one episode, repeated transmissions, the same speaker, or near-duplicate recordings cross the development and test boundary. Whisper's authors describe de-duplicating training data against evaluation sets, yet the underlying web-scale data are not fully inspectable. Public benchmark scores for broadly trained models therefore carry an uncertainty that a local evaluation should state.[4]
Split by the claim you want to make. If the claim is performance on unseen transmissions from known conditions, hold out sessions and episodes. If it is transfer to a new site, hold out the site. Freeze a sealed test before tuning and use a separate development set for vocabulary, prompting, thresholds, and error analysis. Report known overlap and do not repeatedly inspect the sealed test after each model change.[4]
Measure the system that must run
Accuracy without operating conditions is incomplete. Pin the model weights, quantization, decoder, beam settings, audio preparation, software versions, and hardware power mode. Measure end-of-transmission to final-output latency, real-time factor, peak memory, sustained throughput, thermal behavior, and failure rate. Report medians and tail values rather than only an average, because a busy channel and a short urgent call stress different parts of the system.[5]
MLPerf separates edge and datacenter systems and distinguishes a rules-compliant reference implementation from optimized implementations. The practical lesson is narrower than endorsement of any benchmark result: hardware, software, load pattern, and quality target belong in the result. A radio report should add its own streaming or batch condition, channel concurrency, warm-up policy, power measurement, and failed-request accounting. Only a run on the intended device can support a claim about that device.[5]
Evidence boundary
Limits & open questions
- The cited studies do not constitute a benchmark of Memo, a particular radio protocol, or a particular edge device.
- Critical-entity categories are a proposed task-specific metric and require a published annotation guide plus independent review.
- WER comparisons are not valid across reports unless reference conventions, normalization, datasets, and decoding conditions are aligned.
- A held-out local corpus can establish performance only for its sampled sources and conditions; broader site, language, speaker, and protocol transfer needs separate evidence.
How this article was researched
Proposed benchmark protocol synthesized from primary ASR evaluation studies, a model paper, a hallucination audit, and official inference benchmark documentation. Before execution, freeze human-referenced development and sealed evaluation sets by source, session, episode, speaker where appropriate, and site when claiming site transfer. Pre-register normalization, critical entities, speech-free negatives, decoding, hardware, quality gates, latency endpoints, exclusions, and statistical units. Preserve all failed runs. Publish paired model differences with uncertainty clustered at the sampling unit and report negative results.
Article type: methods. The diagrams are explanatory schematics. No new Memo field trial, corpus measurement or model benchmark is reported here.
Sources & further reading
Sources reviewed on 23 September 2026. “n.d.” indicates no fixed publication date for the linked version.
- Evaluating and Improving Automatic Speech Recognition using SeverityAssociation for Computational Linguistics · 2023
- CEASR: A Corpus for Evaluating Automatic Speech RecognitionEuropean Language Resources Association · 2020
- Careless Whisper: Speech-to-Text Hallucination HarmsACM Conference on Fairness, Accountability, and Transparency · 2024
- Robust Speech Recognition via Large-Scale Weak SupervisionOpenAI · 2022
- MLPerf Inference: Speech to Text using WhisperMLCommons · n.d.