Self-Training for End-to-End Speech Recognition

Abstract

We revisit self-training in the context of end-to-end speech recognition. We demonstrate that training with pseudo-labels can substantially improve the accuracy of a baseline model. Key to our approach are a strong baseline acoustic and language model used to generate the pseudo-labels, filtering mechanisms tailored to common errors from sequence-to-sequence models, and a novel ensemble approach to increase pseudo-label diversity. Experiments on the LibriSpeech corpus show that with an ensemble of four models and label filtering, self-training yields a 33.9% relative improvement in WER compared with a baseline trained on 100 hours of labelled data in the noisy speech setting. In the clean speech setting, self-training recovers 59.3% of the gap between the baseline and an oracle model, which is at least 93.8% relatively higher than what previous approaches can achieve.

Reproduction

Acoustic model configuration files are provided for each dataset to reproduce results from the paper (training and decoding steps).

Pretrained convolutional language models used in the paper are also included, as well as steps to generate the language model corpus used to train the language models and steps to reproduce acoustic model training.

Training and decoding broadly follow the existing TDS seq2seq recipes.

Dependencies

All results from the paper can be reproduced exactly with the following project commits:

• flashlight - commit 77ad2f79249c6833875f57865712de4666617d00
• wav2letter - commit 57b4904c8c4a808d393f047a9352c2d5be57ae8f

Each commit contains versioned documentation for building and installing requisite dependencies.

Tokens and Lexicon Sets

Tokens and lexicon files generated in the $MODEL_DST/am/ and $MODEL_DST/decoder/ directories following the LibriSpeech recipe are the same as in the table.

Pre-Trained Models

Acoustic Models

Components of the baseline model trained only on LibriSpeech training sets are below.

Below are models trained on pseudo-labels. All train sets include the base train-clean-100 set in addition to generated pseudo-labels. Steps for generating pseudo-labels can be found here:

Language Models

The instructions in LibriSpeech contain steps to reproduce the language model training corpus. Below are components of the GCNN language model used for decoding:

Citation

@article{kahn2019selftraining,
    title={Self-Training for End-to-End Speech Recognition},
    author={Jacob Kahn and Ann Lee and Awni Hannun},
    year={2019},
    eprint={1909.09116},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}