Random Forests(TM) in XGBoost - Xgboost

############################# Random Forests(TM) in XGBoost #############################

XGBoost is normally used to train gradient-boosted decision trees and other gradient boosted models. Random Forests use the same model representation and inference, as gradient-boosted decision trees, but a different training algorithm. One can use XGBoost to train a standalone random forest or use random forest as a base model for gradient boosting. Here we focus on training standalone random forest.

We have native APIs for training random forests since the early days, and the Scikit-Learn estimators can train random forests as well by setting the appropriate parameters, as shown below. The dedicated XGBRFClassifier and XGBRFRegressor wrappers are deprecated since 3.4.0.

Standalone Random Forest With XGBoost API

The following parameters must be set to enable random forest training.

booster should be set to gbtree, as we are training forests. Note that as this is the default, this parameter needn't be set explicitly.
subsample must be set to a value less than 1 to enable random selection of training cases (rows).
One of colsample_by* parameters must be set to a value less than 1 to enable random selection of columns. Normally, colsample_bynode would be set to a value less than 1 to randomly sample columns at each tree split.
num_parallel_tree should be set to the size of the forest being trained.
num_boost_round should be set to 1 to prevent XGBoost from boosting multiple random forests. Note that this is a keyword argument to train(), and is not part of the parameter dictionary.
eta (alias: learning_rate) must be set to 1 when training random forest regression.
random_state can be used to seed the random number generator.

Other parameters should be set in a similar way they are set for gradient boosting. For instance, objective will typically be reg:squarederror for regression and binary:logistic for classification, lambda should be set according to a desired regularization weight, etc.

If both num_parallel_tree and num_boost_round are greater than 1, training will use a combination of random forest and gradient boosting strategy. It will perform num_boost_round rounds, boosting a random forest of num_parallel_tree trees at each round. If early stopping is not enabled, the final model will consist of num_parallel_tree * num_boost_round trees.

Here is a sample parameter dictionary for training a random forest on a GPU using xgboost::

params = { "colsample_bynode": 0.8, "learning_rate": 1, "max_depth": 5, "num_parallel_tree": 100, "objective": "binary:logistic", "subsample": 0.8, "tree_method": "hist", "device": "cuda", }

A random forest model can then be trained as follows::

bst = train(params, dmatrix, num_boost_round=1)

Standalone Random Forest With Scikit-Learn-Like API

.. deprecated:: 3.4.0

XGBRFClassifier, XGBRFRegressor, and their Dask counterparts are deprecated and will be removed in a future release. They are thin wrappers over the boosting interface rather than a conventional random forest implementation, and they do not support features like early stopping. Use the native API described above (set num_parallel_tree with num_boost_round=1), pass num_parallel_tree along with n_estimators=1 to :py:class:~xgboost.XGBClassifier or :py:class:~xgboost.XGBRegressor, or use a dedicated implementation such as :py:class:sklearn.ensemble.RandomForestClassifier.

:py:class:~xgboost.XGBClassifier and :py:class:~xgboost.XGBRegressor can train random forests by setting the same parameters as the native API described above. Since n_estimators specifies the number of boosting rounds for these estimators, set it to 1 and use num_parallel_tree for the size of the forest:

num_parallel_tree specifies the size of the forest to be trained
n_estimators should be set to 1 to prevent boosting multiple random forests
learning_rate should be set to 1
subsample and one of the colsample_by* parameters must be set to a value less than 1

For a simple example, you can train a random forest regressor with::

from sklearn.model_selection import KFold

# Your code ...

kf = KFold(n_splits=2)
for train_index, test_index in kf.split(X, y):
    xgb_model = xgb.XGBRegressor(
        n_estimators=1,
        num_parallel_tree=100,
        learning_rate=1,
        subsample=0.8,
        colsample_bynode=0.8,
        random_state=42,
    ).fit(X[train_index], y[train_index])

Caveats

XGBoost uses 2nd order approximation to the objective function. This can lead to results that differ from a random forest implementation that uses the exact value of the objective function.
XGBoost does not perform replacement when subsampling training cases. Each training case can occur in a subsampled set either 0 or 1 time.