Open source · Apache-2.0
On-device pronunciation assessment with cloud-level accuracy.
Pronunciada is an open-source Python library that grades a language learner's speech sound by sound, in ten languages.
Tested against Azure in ten languages
Most language-learning apps get their pronunciation scores from a cloud service, and the most widely used one is Microsoft Azure Pronunciation Assessment. Pronunciada was compared with it in every language it supports: English, Mandarin, Japanese, Spanish, French, German, Italian, Portuguese, Russian and Korean. There are 84 comparisons. Each is one measure on one public test set, with the same recordings and text given to both systems.
- English
- 中文
- 日本語
- Español
- Français
- Deutsch
- Italiano
- Português
- Русский
- 한국어
Ten languages on one shared speech model, with a small adapter for each.
84 comparisons on held-out public test sets. Significance is from paired tests at p < 0.05.
Pronunciada is significantly better in 40 comparisons and Azure in 7. The other 37 show no significant difference. Azure's seven are English intonation, how often a native speaker's words are marked down in English, Japanese, Spanish and Portuguese, and two tests on Common Voice recordings. Every comparison is listed in the scoreboard.
Closer to expert raters on learner speech
The most direct test uses recordings of learners that experts have scored by hand. Such recordings are public for English and Mandarin. On them, Pronunciada agrees with the experts more closely than Azure on individual sounds, on whole sentences and on Mandarin tones. Azure is closer on English intonation.
Higher is better. Agreement is the correlation with expert scores, and finding errors is the area under the ROC curve. Test sets: speechocean762, L2-ARCTIC and OMPAL. All five differences are statistically significant.
All figures come from held-out test recordings. Azure was used as provided, through its REST API in November 2026.
Runs on the device in about 50 ms
Scoring needs no network connection and no GPU, and the audio does not leave the machine. A sentence of 3.5 seconds is scored in about 50 ms on an Apple M3 Pro laptop.
The system was designed with phones in mind. The quantized model is a 67 MB file, runs on a single CPU thread and uses about 0.4 GB of memory. The full model is 248 MB and uses about 0.6 GB. Packaged iOS and Android builds are not available yet.
Lower is better. Median over English recordings. This is not a controlled comparison: the two sets of recordings differ, and the Azure time is the whole request from the same laptop, upload included.
Built on ZIPA, and far more accurate than ZIPA alone
No large model was trained for this project. The sounds in a recording are recognized by ZIPA (Zhu et al., ACL 2025), an open speech model trained on many languages, and Pronunciada adds the scoring around it. To measure what that adds, the same tests were run on ZIPA alone with the standard scoring method.
Higher is better. ZIPA alone is the unmodified model with the standard goodness-of-pronunciation score. All four differences are statistically significant.
Across 60 tests, Pronunciada is significantly better in 45 and significantly worse in none. The gap is widest for Mandarin tones, which ZIPA does not output. The full comparison is in the baseline write-up.
How it works
Pronunciada compares the sounds a sentence should contain with the sounds a speech model hears in the recording. The figure follows one word from the learner recording in the film above.
The speech model heard no “l” and a different vowel, so those two sounds score low and the word as a whole gets 0.10.
- The text is converted to the sequence of sounds a speaker should produce, using a pronunciation dictionary and rules for each language.
- ZIPA listens to the recording and reports which sounds it hears, moment by moment, without seeing the text. A small adapter per language corrects its systematic confusions.
- For every expected sound, the scorer weighs three explanations: it was said, it was replaced by another sound, or it was dropped. It sums over every way of lining the sounds up with the audio, so no single alignment decides the result.
- Calibration models fitted on expert-scored speech turn that evidence into the probability that a rater would accept each sound, each word and the sentence. One strictness setting moves all of them together.
- Separate small models handle what the sequence of sounds cannot show: tone, stress, length and a few consonant contrasts.
The models trained for this project add about 10 MB to the package. The design document has the details.
Limitations
- Accuracy on learner speech has been measured for English and Mandarin only. For the other eight languages the evidence comes from native speech.
- Azure agrees with expert raters more closely on English intonation (0.83 against 0.77).
- At the default setting, about 3% of a native speaker's words score below 0.6 in English, Japanese, Spanish and Portuguese, against about 2% for Azure.
- Italian double consonants and Portuguese nasal vowels are detected poorly, and Japanese pitch accent is not scored.
- The text being read is required, and there are no iOS, Android or browser builds yet.
The README has the full list.
Get started
pip install pronunciada
import pronunciada
result = pronunciada.score(
"clip.wav", "It was a beautiful day for a float", lang="en"
)
result.prob # 0.75, the sentence
result.words[7].prob # 0.10, the word "float"
result.words[7].phones # f 0.83, l 0.27, oʊ 0.35, t 0.56
result.coaching() # detected errors, most severe first
The values are for the learner recording in the film above, in which the “l” of “float” is dropped. The model is downloaded on first use. A command-line tool, a local HTTP server, output in Azure's JSON layout and an MCP server are documented in the README.
Project
- Maintainer
- Bill Sun
- License
- Apache-2.0 for the code. ZIPA and the sample recordings are licensed under CC BY 4.0.
- Built on
- ZIPA (Zhu et al., ACL 2025) and ONNX Runtime
- Test data
- speechocean762, L2-ARCTIC, EpaDB, OMPAL, FLEURS and Common Voice
- Citing
- CITATION.cff in the repository
- Contributing
- Issues and pull requests are welcome on GitHub. See CONTRIBUTING.md.
Apache-2.0, on GitHub
The code, every result table and the scripts that produce them are in the repository.