Experiment - Annotated Deep Learning Paper Implementations

Transformer XL

This is an experiment training Shakespeare dataset with a Transformer XL model.

Install the labml-nn package

!pip install labml-nn

Imports

from labml import experiment
from labml_nn.transformers.xl.experiment import Configs

Create an experiment

experiment.create(name="transformer_xl")

Initialize configurations

conf = Configs()

Set experiment configurations and assign a configurations dictionary to override configurations

experiment.configs(conf,
                   # A dictionary of configurations to override
                   {'tokenizer': 'character',
                    'text': 'tiny_shakespeare',
                    'optimizer.learning_rate': 1.,
                    'optimizer.optimizer': 'Noam',
                    'prompt': 'It is',
                    'prompt_separator': '',

                    'train_loader': 'sequential_train_loader',
                    'valid_loader': 'sequential_valid_loader',

                    'seq_len': 2,
                    'mem_len': 32,
                    'epochs': 128,
                    'batch_size': 32,
                    'inner_iterations': 25,
                    })

Set PyTorch models for loading and saving

experiment.add_pytorch_models({'model': conf.model})

Start the experiment and run the training loop.

# Start the experiment
with experiment.start():
    conf.run()