Experiment - Annotated Deep Learning Paper Implementations

Fast Weights Transformer

This is an experiment training Shakespeare dataset with a Compressive Transformer model.

Install the labml-nn package

!pip install labml-nn

Imports

from labml import experiment
from labml_nn.transformers.fast_weights.experiment import Configs

Create an experiment

experiment.create(name="fast_weights_transformer")

Initialize configurations

conf = Configs()

Set experiment configurations and assign a configurations dictionary to override configurations

experiment.configs(conf,
                   # A dictionary of configurations to override
                   {'tokenizer': 'character',
                    'text': 'tiny_shakespeare',
                    'optimizer.learning_rate': 1.0,
                    'optimizer.optimizer': 'Noam',
                    'prompt': 'It is',
                    'prompt_separator': '',

                    'train_loader': 'shuffled_train_loader',
                    'valid_loader': 'shuffled_valid_loader',

                    'seq_len': 128,
                    'epochs': 128,
                    'batch_size': 16,
                    'inner_iterations': 25})

Set PyTorch models for loading and saving

experiment.add_pytorch_models({'model': conf.model})

Start the experiment and run the training loop.

with experiment.start():
    conf.run()