Layers

The layers.py module implements common layers / layer-wise operations that can be composed to create larger neural networks. It includes:

• Fully-connected layers
• Sparse evolutionary layers (Mocanu et al., 2018)
• Dot-product attention layers (Luong, Pho, & Manning, 2015; Vaswani et al., 2017)
• 1D and 2D convolution (with stride, padding, and dilation) layers (van den Oord et al., 2016; Yu & Kolton, 2016)
• 2D "deconvolution" (with stride and padding) layers (Zeiler et al., 2010)
• Restricted Boltzmann machines (with CD-n training) (Smolensky, 1996; Carreira-Perpiñán & Hinton, 2005)
• Elementwise multiplication operation
• Summation operation
• Flattening operation
• Embedding layer
• Softmax layer
• Max & average pooling layer
• 1D and 2D batch normalization layers (Ioffe & Szegedy, 2015)
• 1D and 2D layer normalization layers (Ba, Kiros, & Hinton, 2016)
• Recurrent layers (Elman, 1990)
• Long short-term memory (LSTM) layers (Hochreiter & Schmidhuber, 1997)