Linear Regression

Linear Regression is a regression model, which uses the least squares function to model the relationship between one or more independent variables and a dependent variable. It is a common predictiction model.

1. Introduction

Linear regression is a simple regression method. Given a data set of n statistical units, a linear regression model assumes that the relationship between the dependent variable and the of regressors X is linear. This relationship is modeled through a disturbance term or error variable ε — an unobserved random variable that adds "noise" to the linear relationship between the dependent variable and regressors. The model is expressed in the following form:

The objective function of linear regression is to minimize the sum of squares of residuals:

where is a group of samples.

2. Distributed Implementation on Angel

1. Model Storage

Linear regression algorithm can be abstracted as a 1×N PSModel, denoted by w, where , as shown in the following figure:

2. Algorithm Logic

Angel MLLib provides Linear regression algorithm trained with the mini-batch gradient descent method.

Worker:
In each iteration, worker pulls the up-to-date w from PS, updates the model parameters, △w, using the mini-batch gradient descent optimization method, and push △w back to PS.
PS:
In each iteration, PS receives △w from all workers, add their average to w，obtaining a new model.
- Flow:
- Algorithm:
Decaying learning rate
The learning rate decays along iterations as , where:
- α is the decay rate
- T is the epoch

3. Execution & Performance

Input Format

Data fromat is set in "ml.data.type", supporting "libsvm", "dense" and "dummy" types. For details, see Angel Data Format
Model size is set in "ml.model.size", for some sparse model, there are features that all samples are zero at those indices (invalidate indices), therefore ml.model.size = ml.feature.index.range - number of invalidate indices
Feature vector's dimension is set in "ml.feature.index.range"

Parameters

Algorithm Parameters
- ml.epoch.num: number of iterations
- ml.num.update.per.epoch: number update in each epoch
- ml.data.validate.ratio: proportion of data used for validation, no validation when set to 0
- ml.learn.rate: initial learning rate
- ml.learn.decay: decay rate of the learning rate
- ml.lr.reg.l1: coefficient of the L1 penalty
- ml.lr.reg.l2: coefficient of the L2 penalty
I/O Parameters
- ml.feature.num: number of features
- ml.data.type: Angel Data Format, supporting "dense" and "libsvm"
- angel.save.model.path: save path for trained modelangel.train.data.path: input path for train
- angel.predict.data.path: input path for predict
- angel.predict.out.path: output path for predict
- angel.log.path: save path for the log
Resource Parameters
- angel.workergroup.number: number of workers
- angel.worker.memory.mb: worker's memory requested in G
- angel.worker.task.number: number of tasks on each worker, default is 1
- angel.ps.number: number of PS
- angel.ps.memory.mb: PS's memory requested in G

Output

Submit Command

Training Job

java

./bin/angel-submit \
	--action.type=train \
	--angel.app.submit.class=com.tencent.angel.ml.core.graphsubmit.GraphRunner \
	--ml.model.class.name=com.tencent.angel.ml.regression.LinearRegression \
	--angel.train.data.path=$input_path \
	--angel.save.model.path=$model_path \
	--angel.log.path=$log_path \
	--ml.data.is.classification=false \
	--ml.model.is.classification=false \
	--ml.epoch.num=10 \
	--ml.feature.index.range=$featureNum+1 \
	--ml.data.validate.ratio=0.1 \
	--ml.learn.rate=0.1 \
	--ml.learn.decay=1 \
	--ml.reg.l2=0.001 \
	--ml.num.update.per.epoch=10 \
	--ml.worker.thread.num=4 \
	--ml.data.type=libsvm \
	--ml.model.type=T_FLOAT_DENSE \
	--angel.workergroup.number=2 \
	--angel.worker.memory.mb=5000 \
	--angel.worker.task.number=1 \
	--angel.ps.number=2 \
	--angel.ps.memory.mb=5000 \
	--angel.job.name=linearReg_network \
	--angel.output.path.deleteonexist=true \

IncTraining Job

java

	./bin/angel-submit \
		--action.type=inctrain \
		--angel.app.submit.class=com.tencent.angel.ml.core.graphsubmit.GraphRunner \
		--ml.model.class.name=com.tencent.angel.ml.regression.LinearRegression \
		--angel.train.data.path=$input_path \
		--angel.load.model.path=$model_path \
		--angel.save.model.path=$model_path \
		--angel.log.path=$log_path \
		--ml.model.is.classification=false \
		--ml.data.is.classification=false \
		--ml.epoch.num=10 \
		--ml.feature.index.range=$featureNum+1 \
		--ml.data.validate.ratio=0.1 \
		--ml.learn.rate=0.1 \
		--ml.learn.decay=1 \
		--ml.reg.l2=0.001 \
		--ml.num.update.per.epoch=10 \
		--ml.worker.thread.num=4 \
		--ml.data.type=libsvm \
		--ml.model.type=T_FLOAT_DENSE \
		--angel.workergroup.number=2 \
		--angel.worker.memory.mb=5000 \
		--angel.worker.task.number=1 \
		--angel.ps.number=2 \
		--angel.ps.memory.mb=5000 \
		--angel.job.name=linearReg_network \
		--angel.output.path.deleteonexist=true
	```

* **Prediction Job**

    ```java
	./bin/angel-submit \
		--action.type=predict \
		--angel.app.submit.class=com.tencent.angel.ml.core.graphsubmit.GraphRunner \
		--ml.model.class.name=com.tencent.angel.ml.regression.LinearRegression \
		--angel.predict.data.path=$input_path \
		--angel.save.model.path=$model_path \
		--angel.predict.out.path $predict_path \
		--angel.log.path=$log_path \
		--ml.feature.index.range=$featureNum+1 \
		--ml.data.type=libsvm \
		--ml.model.type=T_FLOAT_DENSE \
		--ml.worker.thread.num=4 \
		--angel.workergroup.number=2 \
		--angel.worker.memory.mb=5000 \
		--angel.worker.task.number=1 \
		--angel.ps.number=2 \
		--angel.ps.memory.mb=5000 \
		--angel.job.name=linearReg_network_predict \
		--angel.output.path.deleteonexist=true \
	```

### Performance
* Data: E2006-tfidf, 1.5×10^5 features, 1.6×10^4 samples
* Resources:
	* Angel: executor: 2, 5G memory, 1 task; ps: 2, 5G memory
* Time of 100 epochs:
	* Angel: 25min