RASR release

This is a repository sharing pre-trained acoustic models and language models for our new paper Rethinking Evaluation in ASR: Are Our Models Robust Enough?.

Dependencies

• Flashlight
• Flashlight ASR app

Models

Acoustic Model

All the acoustic models are retrained using Flashlight with wav2letter++ consolidated. Tedlium is not used as training data here due to license issue. All the training data has more standardized sample rate 16kHz rather than 8kHz used in the paper.

Here, we are releasing models with different architecture and different sizes. Note that the models may not fully reproduce results in the paper because of both data and toolkit implementation discrepancies.

Language Model

Language models are trained on Common Crawl corpus as mentioned in paper. We are providing 4-gram LMs with different pruning here with 200k-top words. All the LMs are trained with KenLM toolkit.

The perplexities of the LMs on different development sets are listed below.

WER

Here we summarize the decoding WER for all releasing models. All the numbers in the table are in format viterbi WER -> beam search WER (small beam/large beam).

Decoding is done with lexicon-based beam-search decoder using 200k common crawl lexicon and small common crawl lm.

• tokens
• inference lexicon
• Decoding parameters (beamthreshold=100, beamsizetoken=30):

Tutorial

To simply serialize all the models and interact with them, please refer to the Flashlight ASR app tutorials.

Citation

@article{likhomanenko2020rethinking,
  title={Rethinking Evaluation in ASR: Are Our Models Robust Enough?},
  author={Likhomanenko, Tatiana and Xu, Qiantong and Pratap, Vineel and Tomasello, Paden and Kahn, Jacob and Avidov, Gilad and Collobert, Ronan and Synnaeve, Gabriel},
  journal={arXiv preprint arXiv:2010.11745},
  year={2020}
}