docs/basic/layers_on_angel_en.md
Most of the algorithms in Angel are based on computational graphs, in which a node represents a layer. These layers can be separated into three classes basing on their topological structure:
There are two types of input layers in Angel:
As its name implies, SimpleInputLayer receives input data. The constructor of this class is as follows:
class SimpleInputLayer(name: String, outputDim: Int, transFunc: TransFunc, override val optimizer: Optimizer)(implicit graph: AngelGraph)
extends InputLayer(name, outputDim)(graph) with Trainable
SimpleInputLayer has following characteristics:
RowBasedMatrix, in which each line is a sparse vector, and the parameters are calculated using Angel's internal math libraryThe calculation formula is:
A typical Json expression is:
{
"name": "wide",
"type": "Simpleinputlayer",
"outputdim": 10,
"transfunc": "identity",
"optimizer": "adam"
},
Embedding is common to many deep learning algorithms. The constructor of Embedding class is as follows:
class Embedding(name: String, outputDim: Int, val numFactors: Int, override val optimizer: Optimizer)(implicit graph: AngelGraph)
extends InputLayer(name, outputDim)(graph) with Trainable
The two parameters except name and optimizer are:
Embedding can be abstractly regarded as a table that provides lookup methods (lookup/calOutput). Angel's Emgedding is special in that it allows some operations after looking up the table. Thus there are two steps in total:
Following is the embedding results of a sparse vector when value is 1 (result of one hot encoding, represented by dummy format) and when value is float (represented by libsvm format):
A typical expression in Json is:
{
"name": "embedding",
"type": "Embedding",
"numfactors": 8,
"outputdim": 104,
"optimizer": {
"type": "momentum",
"momentum": 0.9,
"reg2": 0.01
}
},
Linear layers are layers with only one input and one output, mainly consisting of fully connected layer (FCLayer) and a series of feature crossing layers.
FCLayer is the most common layer in DNN. Its calculation can be expressed by:
The constructor of FCLayer in Angel is:
class FCLayer(name: String, outputDim: Int, inputLayer: Layer, transFunc: TransFunc, override val optimizer: Optimizer
)(implicit graph: AngelGraph) extends LinearLayer(name, outputDim, inputLayer)(graph) with Trainable
As we can see from the constructor and formula, FCLayer is quite similar to DenseInputLayer/SparseInputLayer, except that the input of a FCLayer is a layer, while the input of the latter is data (not specifying the input layer in the constructor).
For the parameter storage, FCLayer, like DenseInputLayer, also uses a dense method and is calculated using BLAS.
Since FCLayers are usually used when being stacked together, Angel makes some simplifications in data configuration, that is, reduces the parameters of multiple stacked layers. Following is an example:
{
"name": "fclayer",
"type": "FCLayer",
"outputdims": [
100,
100,
1
],
"transfuncs": [
"relu",
"relu",
"identity"
],
"inputlayer": "embedding"
},
There are three FCLayers stacking together. The input layer is the input of the first layer, and the output should be that of the last layer. Each layer's outputDim and transfunc are represented by a column, that is:
Note: user can specify optimizer for stacked FCLayers. In this way, all layers share the same type of optimizer. The Json configuration will be as follows if FCLayers are specified separately:
{
"name": "fclayer_0",
"type": "FCLayer",
"outputdim": 100,
"transfuncs": "relu",
"inputlayer": "embedding"
},
{
"name": "fclayer_1",
"type": "FCLayer",
"outputdim": 100,
"transfuncs": "relu",
"inputlayer": "fclayer_0"
},
{
"name": "fclayer",
"type": "FCLayer",
"outputdim": 1,
"transfuncs": "identity",
"inputlayer": "fclayer_1"
},
Feature cross layer. The calculation formula is:
in which the is the output of Embedding layer. Specifically, this formula sums up the inner products of all possible pair combinations of elements in Embedding output. Therefore, BilnnerSumCross has no parameter and is untrainable. The output dimension is 1.
The constructor of BilnnerSumCross is:
class BiInnerSumCross(name: String, inputLayer: Layer)(
implicit graph: AngelGraph) extends LinearLayer(name, 1, inputLayer)(graph)
An Json example:
{
"name": "biinnersumcross",
"type": "BiInnerSumCross",
"inputlayer": "embedding",
"outputdim": 1
},
Feature cross layer. The calculation formula is:
in which the is the output of Embedding layer. Specifically, the Bilnner Cross computes the inner products of all possible pair combination of elements in the Embedding output, thus the output dimension is . Therefore, BiInnerCross also has no parameter and is untrainable, and the output dimension is .
The constructor of BilnnerCross class is:
class BiInnerCross(name: String, outputDim: Int, inputLayer: Layer)(
implicit graph: AngelGraph) extends LinearLayer(name, outputDim, inputLayer)(graph)
A Json example:
{
"name": "biInnerCross",
"type": "BiInnerCross",
"outputdim": 78,
"inputlayer": "embedding"
},
Feature cross layer. The calculation formula is:
in which the is the output of Embedding layer. Bi-interaction cross computes the element-wise products of all combinations of elements in Embedding vector, and then sum them up. Hence the output dimension is identical to the dimension of , and is independent of dimension of input data. Hence, BiInteactionCross also has no parameter and is untrainable.
The constructor of BiInteractionCross is:
class BiInteractionCross(name: String, outputDim: Int, inputLayer: Layer)(
implicit graph: AngelGraph) extends LinearLayer(name, outputDim, inputLayer)(graph)
A json example:
{
"name": "biinteractioncross",
"type": "BiInteractionCross",
"outputdim": 8,
"inputlayer": "embedding"
},
Join layers are those layers with multiple input and one output, mainly including:
Concat multiple input layers to output a Dense matrix. The constructor is:
class ConcatLayer(name: String, outputDim: Int, inputLayers: Array[Layer])(implicit graph: AngelGraph)
extends JoinLayer(name, outputDim, inputLayers)(graph)
An example of Json parameters:
{
"name": "concatlayer",
"type": "ConcatLayer",
"outputdim": 182,
"inputlayers": [
"embedding",
"biInnerCross"
]
},
Since there exist multiple input layers, they should be indicated by "inputlayers" in list format.
Sum up the input elements and output the result. The constructor is:
class SumPooling(name: String, outputDim: Int, inputLayers: Array[Layer])(implicit graph: AngelGraph)
extends JoinLayer(name, outputDim, inputLayers)(graph)
An example of Json parameters:
{
"name": "sumPooling",
"type": "SumPooling",
"outputdim": 1,
"inputlayers": [
"wide",
"fclayer"
]
},
Since there exist multiple input layers, they should be indicated by "inputlayers" in list format.
Multiply the input elements and output the result. The constructor is:
class MulPooling(name: String, outputDim: Int, inputLayers: Array[Layer])(implicit graph: AngelGraph)
extends JoinLayer(name, outputDim, inputLayers)(graph)
An example of Json parameters:
{
"name": "mulPooling",
"type": "MulPooling",
"outputdim": 1,
"inputlayers": [
"wide",
"fclayer"
]
},
Since there exist multiple input layers, they should be indicated by "inputlayers" in list format.
Multiply the corresponding elements then sum up by row, and output a matrix with n rows and one column. The constructor is:
class DotPooling(name: String, outputDim: Int, inputLayers: Array[Layer])(implicit graph: AngelGraph)
extends JoinLayer(name, outputDim, inputLayers)(graph)
An example of Json parameters:
{
"name": "dotPooling",
"type": "DotPooling",
"outputdim": 1,
"inputlayers": [
"wide",
"fclayer"
]
},
Since there exist multiple input layers, they should be indicated by "inputlayers" in list format.
A loss layer should be the top layer of a network with only input layer and no output layer. LossLayer is used to compute the loss. Please refer to Loss functions in Angel for loss functions.
The constructor of SimpleLossLayer is:
class SimpleLossLayer(name: String, inputLayer: Layer, lossFunc: LossFunc)(
implicit graph: AngelGraph) extends LinearLayer(name, 1, inputLayer)(graph) with LossLayer
An example of Json parameters:
{
"name": "simplelosslayer",
"type": "Simplelosslayer",
"lossfunc": "logloss",
"inputlayer": "sumPooling"
}