Language modeling

Language modeling is the task of predicting the next word or character in a document.

* indicates models using dynamic evaluation; where, at test time, models may adapt to seen tokens in order to improve performance on following tokens. (Mikolov et al., (2010), Krause et al., (2017))

Word Level Models

Penn Treebank

A common evaluation dataset for language modeling is the Penn Treebank, as pre-processed by Mikolov et al., (2011). The dataset consists of 929k training words, 73k validation words, and 82k test words. As part of the pre-processing, words were lower-cased, numbers were replaced with N, newlines were replaced with <eos>, and all other punctuation was removed. The vocabulary is the most frequent 10k words with the rest of the tokens replaced by an <unk> token. Models are evaluated based on perplexity, which is the average per-word log-probability (lower is better).

Model	Validation perplexity	Test perplexity	Number of params	Paper / Source	Code
Mogrifier RLSTM + dynamic eval (Melis, 2022)	42.9	42.9	24M	Circling Back to Recurrent Models of Language	Official
Mogrifier LSTM + dynamic eval (Melis et al., 2019)	44.9	44.8	24M	Mogrifier LSTM	Official
AdvSoft + AWD-LSTM-MoS + dynamic eval (Wang et al., 2019)	46.63	46.01	22M	Improving Neural Language Modeling via Adversarial Training	Official
FRAGE + AWD-LSTM-MoS + dynamic eval (Gong et al., 2018)	47.38	46.54	22M	FRAGE: Frequency-Agnostic Word Representation	Official
AWD-LSTM-DOC x5 (Takase et al., 2018)	48.63	47.17	185M	Direct Output Connection for a High-Rank Language Model	Official
AWD-LSTM-MoS + dynamic eval (Yang et al., 2018)*	48.33	47.69	22M	Breaking the Softmax Bottleneck: A High-Rank RNN Language Model	Official
Mogrifier RLSTM (Melis, 2022)	48.9	47.9	24M	Circling Back to Recurrent Models of Language	Official
Mogrifier LSTM (Melis et al., 2019)	51.4	50.1	24M	Mogrifier LSTM	Official
AWD-LSTM + dynamic eval (Krause et al., 2017)*	51.6	51.1	24M	Dynamic Evaluation of Neural Sequence Models	Official
AWD-LSTM-DOC + Partial Shuffle (Press, 2019) preprint	53.79	52.00	23M	Partially Shuffling the Training Data to Improve Language Models	Official
AWD-LSTM-DOC (Takase et al., 2018)	54.12	52.38	23M	Direct Output Connection for a High-Rank Language Model	Official
AWD-LSTM + continuous cache pointer (Merity et al., 2017)*	53.9	52.8	24M	Regularizing and Optimizing LSTM Language Models	Official
Trellis Network (Bai et al., 2019)	-	54.19	34M	Trellis Networks for Sequence Modeling	Official
AWD-LSTM-MoS + ATOI (Kocher et al., 2019)	56.44	54.33	22M	Alleviating Sequence Information Loss with Data Overlapping and Prime Batch Sizes	Official
AWD-LSTM-MoS + finetune (Yang et al., 2018)	56.54	54.44	22M	Breaking the Softmax Bottleneck: A High-Rank RNN Language Model	Official
Transformer-XL (Dai et al., 2018) under review	56.72	54.52	24M	Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context	Official
AWD-LSTM-MoS (Yang et al., 2018)	58.08	55.97	22M	Breaking the Softmax Bottleneck: A High-Rank RNN Language Model	Official
AWD-LSTM 3-layer with Fraternal dropout (Zołna et al., 2018)	58.9	56.8	24M	Fraternal dropout	Official
AWD-LSTM (Merity et al., 2017)	60.0	57.3	24M	Regularizing and Optimizing LSTM Language Models	Official

WikiText-2

WikiText-2 has been proposed as a more realistic benchmark for language modeling than the pre-processed Penn Treebank. WikiText-2 consists of around 2 million words extracted from Wikipedia articles.

Model	Validation perplexity	Test perplexity	Number of params	Paper / Source	Code
Mogrifier RLSTM + dynamic eval (Melis, 2022)	39.3	38.0	24M	Circling Back to Recurrent Models of Language	Official
Mogrifier LSTM + dynamic eval (Melis et al., 2019)	40.2	38.6	35M	Mogrifier LSTM	Official
AdvSoft + AWD-LSTM-MoS + dynamic eval (Wang et al., 2019)	40.27	38.65	35M	Improving Neural Language Modeling via Adversarial Training	Official
FRAGE + AWD-LSTM-MoS + dynamic eval (Gong et al., 2018)	40.85	39.14	35M	FRAGE: Frequency-Agnostic Word Representation	Official
AWD-LSTM-MoS + dynamic eval (Yang et al., 2018)*	42.41	40.68	35M	Breaking the Softmax Bottleneck: A High-Rank RNN Language Model	Official
AWD-LSTM + dynamic eval (Krause et al., 2017)*	46.4	44.3	33M	Dynamic Evaluation of Neural Sequence Models	Official
AWD-LSTM + continuous cache pointer (Merity et al., 2017)*	53.8	52.0	33M	Regularizing and Optimizing LSTM Language Models	Official
AWD-LSTM-DOC x5 (Takase et al., 2018)	54.19	53.09	185M	Direct Output Connection for a High-Rank Language Model	Official
Mogrifier RLSTM (Melis, 2022)	56.7	55.0	24M	Circling Back to Recurrent Models of Language	Official
Mogrifier LSTM (Melis et al., 2019)	57.3	55.1	35M	Mogrifier LSTM	Official
AWD-LSTM-DOC + Partial Shuffle (Press, 2019) preprint	60.16	57.85	37M	Partially Shuffling the Training Data to Improve Language Models	Official
AWD-LSTM-DOC (Takase et al., 2018)	60.29	58.03	37M	Direct Output Connection for a High-Rank Language Model	Official
AWD-LSTM-MoS (Yang et al., 2018)	63.88	61.45	35M	Breaking the Softmax Bottleneck: A High-Rank RNN Language Model	Official
AWD-LSTM 3-layer with Fraternal dropout (Zołna et al., 2018)	66.8	64.1	34M	Fraternal dropout	Official
AWD-LSTM + ATOI (Kocher et al., 2019)	67.47	64.73	33M	Alleviating Sequence Information Loss with Data Overlapping and Prime Batch Sizes	Official
AWD-LSTM (Merity et al., 2017)	68.6	65.8	33M	Regularizing and Optimizing LSTM Language Models	Official

WikiText-103

WikiText-103 The WikiText-103 corpus contains 267,735 unique words and each word occurs at least three times in the training set.

Model	Validation perplexity	Test perplexity	Number of params	Paper / Source	Code
Routing Transformer (Roy et al., 2020)* arxiv preprint	-	15.8	-	Efficient Content-Based Sparse Attention with Routing Transformers	-
Transformer-XL + RMS dynamic eval (Krause et al., 2019)* arxiv preprint	15.8	16.4	257M	Dynamic Evaluation of Transformer Language Models	Official
Compressive Transformer (Rae et al., 2019)* arxiv preprint	16.0	17.1(16.1 with basic dynamic evaluation)	~257M	Compressive Transformers for Long-Range Sequence Modelling	-
SegaTransformer-XL (Bai et al., 2020)	-	17.1	257M	Segatron: Segment-Aware Transformer for Language Modeling and Understanding	Official
Transformer-XL Large (Dai et al., 2018) under review	17.7	18.3	257M	Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context	Official
Transformer with tied adaptive embeddings (Baevski and Auli, 2018)	19.8	20.5	247M	Adaptive Input Representations for Neural Language Modeling	Link
TaLK Convolutions (Lioutas et al., 2020)	-	23.3	240M	Time-aware Large Kernel Convolutions	Official
Transformer-XL Standard (Dai et al., 2018) under review	23.1	24.0	151M	Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context	Official
AdvSoft + 4 layer QRNN + dynamic eval (Wang et al., 2019)	27.2	28.0		Improving Neural Language Modeling via Adversarial Training	Official
LSTM + Hebbian + Cache + MbPA (Rae et al., 2018)	29.0	29.2		Fast Parametric Learning with Activation Memorization
Trellis Network (Bai et al., 2019)	-	30.35	180M	Trellis Networks for Sequence Modeling	Official
AWD-LSTM-MoS + ATOI (Kocher et al., 2019)	31.92	32.85		Alleviating Sequence Information Loss with Data Overlapping and Prime Batch Sizes	Official
LSTM + Hebbian (Rae et al., 2018)	34.1	34.3		Fast Parametric Learning with Activation Memorization
LSTM (Rae et al., 2018)	36.0	36.4		Fast Parametric Learning with Activation Memorization
Gated CNN (Dauphin et al., 2016)	-	37.2		Language modeling with gated convolutional networks
Neural cache model (size = 2,000) (Grave et al., 2017)	-	40.8		Improving Neural Language Models with a Continuous Cache	Link
Temporal CNN (Bai et al., 2018)	-	45.2		Convolutional sequence modeling revisited
LSTM (Grave et al., 2017)	-	48.7		Improving Neural Language Models with a Continuous Cache	Link

1B Words / Google Billion Word benchmark

The One-Billion Word benchmark is a large dataset derived from a news-commentary site. The dataset consists of 829,250,940 tokens over a vocabulary of 793,471 words. Importantly, sentences in this model are shuffled and hence context is limited.

Model	Test perplexity	Number of params	Paper / Source	Code
Transformer-XL Large (Dai et al., 2018) under review	21.8	0.8B	Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context	Official
Transformer-XL Base (Dai et al., 2018) under review	23.5	0.46B	Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context	Official
Transformer with shared adaptive embeddings - Very large (Baevski and Auli, 2018)	23.7	0.8B	Adaptive Input Representations for Neural Language Modeling	Link
10 LSTM+CNN inputs + SNM10-SKIP (Jozefowicz et al., 2016) ensemble	23.7	43B?	Exploring the Limits of Language Modeling	Official
Transformer with shared adaptive embeddings (Baevski and Auli, 2018)	24.1	0.46B	Adaptive Input Representations for Neural Language Modeling	Link
Big LSTM+CNN inputs (Jozefowicz et al., 2016)	30.0	1.04B	Exploring the Limits of Language Modeling
Gated CNN-14Bottleneck (Dauphin et al., 2017)	31.9	?	Language Modeling with Gated Convolutional Networks
BIGLSTM baseline (Kuchaiev and Ginsburg, 2018)	35.1	0.151B	Factorization tricks for LSTM networks	Official
BIG F-LSTM F512 (Kuchaiev and Ginsburg, 2018)	36.3	0.052B	Factorization tricks for LSTM networks	Official
BIG G-LSTM G-8 (Kuchaiev and Ginsburg, 2018)	39.4	0.035B	Factorization tricks for LSTM networks	Official

Character Level Models

Hutter Prize

The Hutter Prize Wikipedia dataset, also known as enwiki8, is a byte-level dataset consisting of the first 100 million bytes of a Wikipedia XML dump. For simplicity we shall refer to it as a character-level dataset. Within these 100 million bytes are 205 unique tokens.

Model	Bit per Character (BPC)	Number of params	Paper / Source	Code
Mogrifier RLSTM + dynamic eval (Melis, 2022)	0.935	96M	Circling Back to Recurrent Models of Language	Official
Transformer-XL + RMS dynamic eval (Krause et al., 2019)* arxiv preprint	0.94	277M	Dynamic Evaluation of Transformer Language Models	Official
Compressive Transformer (Rae et al., 2019) arxiv preprint	0.97	-	Compressive Transformers for Long-Range Sequence Modelling	-
Mogrifier LSTM + dynamic eval (Melis et al., 2019)	0.988	96M	Mogrifier LSTM	Official
24-layer Transformer-XL (Dai et al., 2018) under review	0.99	277M	Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context	Official
Longformer Large (Beltagy, Peters, and Cohan; 2020)	0.99	102M	Longformer: The Long-Document Transformer	Official
Longformer Small (Beltagy, Peters, and Cohan; 2020)	1.00	41M	Longformer: The Long-Document Transformer	Official
18-layer Transformer-XL (Dai et al., 2018) under review	1.03	88M	Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context	Official
Mogrifier RLSTM (Melis, 2022)	1.042	96M	Circling Back to Recurrent Models of Language	Official
12-layer Transformer-XL (Dai et al., 2018) under review	1.06	41M	Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context	Official
64-layer Character Transformer Model (Al-Rfou et al., 2018)	1.06	235M	Character-Level Language Modeling with Deeper Self-Attention
mLSTM + dynamic eval (Krause et al., 2017)*	1.08	46M	Dynamic Evaluation of Neural Sequence Models	Official
12-layer Character Transformer Model (Al-Rfou et al., 2018)	1.11	44M	Character-Level Language Modeling with Deeper Self-Attention
Mogrifier LSTM (Melis et al., 2019)	1.122	96M	Mogrifier LSTM	Official
3-layer AWD-LSTM (Merity et al., 2018)	1.232	47M	An Analysis of Neural Language Modeling at Multiple Scales	Official
Large mLSTM +emb +WN +VD (Krause et al., 2017)	1.24	46M	Multiplicative LSTM for sequence modelling	Official
Large FS-LSTM-4 (Mujika et al., 2017)	1.245	47M	Fast-Slow Recurrent Neural Networks	Official
Large RHN (Zilly et al., 2016)	1.27	46M	Recurrent Highway Networks	Official
FS-LSTM-4 (Mujika et al., 2017)	1.277	27M	Fast-Slow Recurrent Neural Networks	Official

Text8

The text8 dataset is also derived from Wikipedia text, but has all XML removed, and is lower cased to only have 26 characters of English text plus spaces.

Model	Bit per Character (BPC)	Number of params	Paper / Source	Code
Transformer-XL + RMS dynamic eval (Krause et al., 2019)* arxiv preprint	1.038	277M	Dynamic Evaluation of Transformer Language Models	Official
Mogrifier RLSTM + dynamic eval (Melis, 2022)	1.044	96M	Circling Back to Recurrent Models of Language	Official
Transformer-XL Large (Dai et al., 2018) under review	1.08	277M	Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context	Official
Mogrifier RLSTM (Melis, 2022)	1.096	96M	Circling Back to Recurrent Models of Language	Official
Longformer Small (Beltagy, Peters, and Cohan; 2020)	1.10	41M	Longformer: The Long-Document Transformer	Official
64-layer Character Transformer Model (Al-Rfou et al., 2018)	1.13	235M	Character-Level Language Modeling with Deeper Self-Attention
12-layer Character Transformer Model (Al-Rfou et al., 2018)	1.18	44M	Character-Level Language Modeling with Deeper Self-Attention
mLSTM + dynamic eval (Krause et al., 2017)*	1.19	45M	Dynamic Evaluation of Neural Sequence Models	Official
Large mLSTM +emb +WN +VD (Krause et al., 2016)	1.27	45M	Multiplicative LSTM for sequence modelling	Official
Large RHN (Zilly et al., 2016)	1.27	46M	Recurrent Highway Networks	Official
LayerNorm HM-LSTM (Chung et al., 2017)	1.29	35M	Hierarchical Multiscale Recurrent Neural Networks
BN LSTM (Cooijmans et al., 2016)	1.36	16M	Recurrent Batch Normalization	Official
Unregularised mLSTM (Krause et al., 2016)	1.40	45M	Multiplicative LSTM for sequence modelling	Official

Penn Treebank

The vocabulary of the words in the character-level dataset is limited to 10 000 - the same vocabulary as used in the word level dataset. This vastly simplifies the task of character-level language modeling as character transitions will be limited to those found within the limited word level vocabulary.

Model	Bit per Character (BPC)	Number of params	Paper / Source	Code
Mogrifier RLSTM + dynamic eval (Melis, 2022)	1.061	24M	Circling Back to Recurrent Models of Language	Official
Mogrifier LSTM + dynamic eval (Melis et al., 2019)	1.083	24M	Mogrifier LSTM	Official
Mogrifier RLSTM (Melis, 2022)	1.096	24M	Circling Back to Recurrent Models of Language	Official
Mogrifier LSTM (Melis et al., 2019)	1.120	24M	Mogrifier LSTM	Official
Trellis Network (Bai et al., 2019)	1.159	13.4M	Trellis Networks for Sequence Modeling	Official
3-layer AWD-LSTM (Merity et al., 2018)	1.175	13.8M	An Analysis of Neural Language Modeling at Multiple Scales	Official
6-layer QRNN (Merity et al., 2018)	1.187	13.8M	An Analysis of Neural Language Modeling at Multiple Scales	Official
FS-LSTM-4 (Mujika et al., 2017)	1.190	27M	Fast-Slow Recurrent Neural Networks	Official
FS-LSTM-2 (Mujika et al., 2017)	1.193	27M	Fast-Slow Recurrent Neural Networks	Official
NASCell (Zoph & Le, 2016)	1.214	16.3M	Neural Architecture Search with Reinforcement Learning
2-layer Norm HyperLSTM (Ha et al., 2016)	1.219	14.4M	HyperNetworks

Multilingual Wikipedia Corpus

The character-based MWC dataset is a collection of Wikipedia pages available in a number of languages. Markup and rare characters were removed, but otherwise no preprocessing was applied.

MWC English in the single text, large setting.

Model	Validation BPC	Test BPC	Number of params	Paper / Source	Code
Mogrifier LSTM + dynamic eval (Melis et al., 2019)	1.200	1.187	24M	Mogrifier LSTM	Official
Mogrifier LSTM (Melis et al., 2019)	1.312	1.298	24M	Mogrifier LSTM	Official
HCLM with Cache (Kawakami et al. 2017)	1.591	1.538	8M	Learning to Create and Reuse Words in Open-Vocabulary Neural Language Modeling
LSTM (Kawakami et al. 2017)	1.793	1.736	8M	Learning to Create and Reuse Words in Open-Vocabulary Neural Language Modeling

MWC Finnish in the single text, large setting.

Model	Validation BPC	Test BPC	Number of params	Paper / Source	Code
Mogrifier LSTM + dynamic eval (Melis et al., 2019)	1.202	1.191	24M	Mogrifier LSTM	Official
Mogrifier LSTM (Melis et al., 2019)	1.327	1.313	24M	Mogrifier LSTM	Official
HCLM with Cache (Kawakami et al. 2017)	1.754	1.711	8M	Learning to Create and Reuse Words in Open-Vocabulary Neural Language Modeling
LSTM (Kawakami et al. 2017)	1.943	1.913	8M	Learning to Create and Reuse Words in Open-Vocabulary Neural Language Modeling

Go back to the README