Modules

The modules.py module implements common multi-layer blocks that appear across many modern deep networks. It includes:

• Bidirectional LSTMs (Schuster & Paliwal, 1997)
• ResNet-style "identity" (i.e., same-convolution) residual blocks (He et al., 2015)
• ResNet-style "convolutional" (i.e., parametric) residual blocks (He et al., 2015)
• WaveNet-style residual block with dilated causal convolutions (van den Oord et al., 2016)
• Transformer-style multi-headed dot-product attention (Vaswani et al., 2017)