labml_nn/activations/fta/experiment.ipynb
Here we train a transformer that uses Fuzzy Tiling Activation in the Feed-Forward Network. We use it for a language model and train it on Tiny Shakespeare dataset for demonstration. However, this is probably not the ideal task for FTA, and we believe FTA is more suitable for modeling data with continuous variables.
!pip install labml-nn --quiet
import torch
import torch.nn as nn
from labml import experiment
from labml.configs import option
from labml_nn.activations.fta.experiment import Configs
experiment.create(name="fta", writers={'screen'})
conf = Configs()
Set experiment configurations and assign a configurations dictionary to override configurations
experiment.configs(conf, {
'tokenizer': 'character',
'prompt_separator': '',
'prompt': 'It is ',
'text': 'tiny_shakespeare',
'seq_len': 256,
'epochs': 32,
'batch_size': 16,
'inner_iterations': 10,
'optimizer.optimizer': 'Adam',
'optimizer.learning_rate': 3e-4,
})
Set PyTorch models for loading and saving
experiment.add_pytorch_models({'model': conf.model})
# Start the experiment
with experiment.start():
conf.run()