5  Tree-based Algorithms

5.1 Classification and Regression Trees

Tree-based models in general use a series of if-then rules to generate predictions from one or more decision trees. In this lecture, we will explore regression and classification trees by the example of the airquality data set. There is one important hyperparameter for regression trees: “minsplit”.

  • It controls the depth of tree (see the help of rpart for a description).
  • It controls the complexity of the tree and can thus also be seen as a regularization parameter.

We first prepare and visualize the data and afterwards fit a decision tree.

library(rpart)
library(rpart.plot)

data = airquality[complete.cases(airquality),]

Fit and visualize one(!) regression tree:

rt = rpart(Ozone~., data = data, control = rpart.control(minsplit = 10))
rpart.plot(rt)

Visualize the predictions:

pred = predict(rt, data)
plot(data$Temp, data$Ozone)
lines(data$Temp[order(data$Temp)], pred[order(data$Temp)], col = "red")

The angular form of the prediction line is typical for regression trees and is a weakness of it.

5.2 Random Forest

To overcome this weakness, a random forest uses an ensemble of regression/classification trees. Thus, the random forest is in principle nothing else than a normal regression/classification tree, but it uses the idea of the “wisdom of the crowd” : By asking many people (regression/classification trees) one can make a more informed decision (prediction/classification). When you want to buy a new phone for example you also wouldn’t go directly into the shop, but search in the internet and ask your friends and family.

There are two randomization steps with the random forest that are responsible for their success:

  • Bootstrap samples for each tree (we will sample observations with replacement from the data set. For the phone this is like not everyone has experience about each phone).
  • At each split, we will sample a subset of predictors that is then considered as potential splitting criterion (for the phone this is like that not everyone has the same decision criteria). Annotation: While building a decision tree (random forests consist of many decision trees), one splits the data at some point according to their features. For example if you have females and males, big and small people in a crowd, you con split this crowd by gender and then by size or by size and then by gender to build a decision tree.

Applying the random forest follows the same principle as for the methods before: We visualize the data (we have already done this so often for the airquality data set, thus we skip it here), fit the algorithm and then plot the outcomes.

Fit a random forest and visualize the predictions:

library(randomForest)
set.seed(123)

data = airquality[complete.cases(airquality),]

rf = randomForest(Ozone~., data = data)
pred = predict(rf, data)
plot(Ozone~Temp, data = data)
lines(data$Temp[order(data$Temp)], pred[order(data$Temp)], col = "red")

One advantage of random forests is that we will get an importance of variables. At each split in each tree, the improvement in the split-criterion is the importance measure attributed to the splitting variable, and is accumulated over all the trees in the forest separately for each variable. Thus the variable importance shows us how important a variable is averaged over all trees.

rf$importance
        IncNodePurity
Solar.R      17969.59
Wind         31978.36
Temp         34176.71
Month        10753.73
Day          15436.47

There are several important hyperparameters in a random forest that we can tune to get better results:

Hyperparameter Explanation
mtry Subset of features randomly selected in each node (from which the algorithm can select the feature that will be used to split the data).
minimum node size Minimal number of observations allowed in a node (before the branching is canceled)
max depth Maximum number of tree depth

5.3 Boosted Regression Trees

Random forests fit hundreds of trees independent of each other. Here, the idea of a boosted regression tree comes in. Maybe we could learn from the errors the previous weak learners made and thus enhance the performance of the algorithm.

A boosted regression tree (BRT) starts with a simple regression tree (weak learner) and then sequentially fits additional trees to improve the results. There are two different strategies to do so:

  • AdaBoost: Wrong classified observations (by the previous tree) will get a higher weight and therefore the next trees will focus on difficult/missclassified observations.
  • Gradient boosting (state of the art): Each sequential model will be fit on the residual errors of the previous model (strongly simplified, the actual algorithm is very complex).

We can fit a boosted regression tree using xgboost, but before we have to transform the data into a xgb.Dmatrix (which is a xgboost specific data type, the package sadly doesn’t support R matrices or data.frames).

library(xgboost)
set.seed(123)

data = airquality[complete.cases(airquality),]
data_xg = xgb.DMatrix(data = as.matrix(scale(data[,-1])), label = data$Ozone)
brt = xgboost(data_xg, nrounds = 16L)
[1] train-rmse:39.724624 
[2] train-rmse:30.225761 
[3] train-rmse:23.134840 
[4] train-rmse:17.899179 
[5] train-rmse:14.097785 
[6] train-rmse:11.375457 
[7] train-rmse:9.391276 
[8] train-rmse:7.889690 
[9] train-rmse:6.646586 
[10]    train-rmse:5.804859 
[11]    train-rmse:5.128437 
[12]    train-rmse:4.456416 
[13]    train-rmse:4.069464 
[14]    train-rmse:3.674615 
[15]    train-rmse:3.424578 
[16]    train-rmse:3.191301

The parameter “nrounds” controls how many sequential trees we fit, in our example this was 16. When we predict on new data, we can limit the number of trees used to prevent overfitting (remember: each new tree tries to improve the predictions of the previous trees).

Let us visualize the predictions for different numbers of trees:

oldpar = par(mfrow = c(2, 2))
for(i in 1:4){
  pred = predict(brt, newdata = data_xg, ntreelimit = i)
  plot(data$Temp, data$Ozone, main = i)
  lines(data$Temp[order(data$Temp)], pred[order(data$Temp)], col = "red")
}
[15:45:43] WARNING: src/c_api/c_api.cc:935: `ntree_limit` is deprecated, use `iteration_range` instead.
[15:45:43] WARNING: src/c_api/c_api.cc:935: `ntree_limit` is deprecated, use `iteration_range` instead.
[15:45:43] WARNING: src/c_api/c_api.cc:935: `ntree_limit` is deprecated, use `iteration_range` instead.
[15:45:43] WARNING: src/c_api/c_api.cc:935: `ntree_limit` is deprecated, use `iteration_range` instead.

par(oldpar)

There are also other ways to control for complexity of the boosted regression tree algorithm:

  • max_depth: Maximum depth of each tree.
  • shrinkage (each tree will get a weight and the weight will decrease with the number of trees).

When having specified the final model, we can obtain the importance of the variables like for random forests:

xgboost::xgb.importance(model = brt)
   Feature        Gain     Cover  Frequency
    <char>       <num>     <num>      <num>
1:    Temp 0.570072012 0.2958229 0.24836601
2:    Wind 0.348230653 0.3419576 0.24183007
3: Solar.R 0.058795502 0.1571072 0.30718954
4:     Day 0.019529985 0.1779925 0.16993464
5:   Month 0.003371847 0.0271197 0.03267974
sqrt(mean((data$Ozone - pred)^2)) # RMSE
[1] 17.89918
data_xg = xgb.DMatrix(data = as.matrix(scale(data[,-1])), label = data$Ozone)

One important strength of xgboost is that we can directly do a cross-validation (which is independent of the boosted regression tree itself!) and specify its properties with the parameter “n-fold”:

set.seed(123)

brt = xgboost(data_xg, nrounds = 5L)
[1] train-rmse:39.724624 
[2] train-rmse:30.225761 
[3] train-rmse:23.134840 
[4] train-rmse:17.899179 
[5] train-rmse:14.097785 
brt_cv = xgboost::xgb.cv(data = data_xg, nfold = 3L,
                         nrounds = 3L, nthreads = 4L)
[1] train-rmse:39.895106+2.127355   test-rmse:40.685477+5.745327 
[2] train-rmse:30.367660+1.728788   test-rmse:32.255812+5.572963 
[3] train-rmse:23.446237+1.366757   test-rmse:27.282435+5.746244 
print(brt_cv)
##### xgb.cv 3-folds
  iter train_rmse_mean train_rmse_std test_rmse_mean test_rmse_std
 <num>           <num>          <num>          <num>         <num>
     1        39.89511       2.127355       40.68548      5.745327
     2        30.36766       1.728788       32.25581      5.572963
     3        23.44624       1.366757       27.28244      5.746244

Annotation: The original data set is randomly partitioned into \(n\) equal sized subsamples. Each time, the model is trained on \(n - 1\) subsets (training set) and tested on the left out set (test set) to judge the performance.

If we do three-folded cross-validation, we actually fit three different boosted regression tree models (xgboost models) on \(\approx 67\%\) of the data points. Afterwards, we judge the performance on the respective holdout. This now tells us how well the model performed.

Important hyperparameters:

Hyperparameter Explanation
eta learning rate (weighting of the sequential trees)
max depth maximal depth in the trees (small = low complexity, large = high complexity)
subsample subsample ratio of the data (bootstrap ratio)
lambda regularization strength of the individual trees
max tree maximal number of trees in the ensemble

5.4 Exercise - Trees

Question: Regression Trees

We will use the following code snippet to understand the hyperparameter mincut and thus the predictive performance.

library(tree)
set.seed(123)

data = airquality
rt = tree(Ozone~., data = data,
          control = tree.control(mincut = 1L, nobs = nrow(data)))

plot(rt)
text(rt)
pred = predict(rt, data)
plot(data$Temp, data$Ozone)
lines(data$Temp[order(data$Temp)], pred[order(data$Temp)], col = "red")
sqrt(mean((data$Ozone - pred)^2)) # RMSE

Try different mincut parameters and see what happens. (Compare the root mean squared error for different mincut parameters and explain what you see. Compare predictions for different mincut parameters and explain what happens.) What was wrong in the snippet above?

library(tree)
set.seed(123)

data = airquality[complete.cases(airquality),]

doTask = function(mincut){
  rt = tree(Ozone~., data = data,
            control = tree.control(mincut = mincut, nobs = nrow(data)))

  pred = predict(rt, data)
  plot(data$Temp, data$Ozone,
       main = paste0(
         "mincut: ", mincut,
         "\nRMSE: ", round(sqrt(mean((data$Ozone - pred)^2)), 2)
      )
  )
  lines(data$Temp[order(data$Temp)], pred[order(data$Temp)], col = "red")
}

for(i in c(1, 2, 3, 5, 10, 15, 25, 50, 54, 55, 56, 57, 75, 100)){ doTask(i) }

Approximately at mincut = 15, prediction is the best (mind overfitting). After mincut = 56, the prediction has no information at all and the RMSE stays constant.

Mind the complete cases of the airquality data set, that was the error.

Question: Random forest

We will use the following code snippet to explore a random forest:

library(randomForest)
set.seed(123)

data = airquality[complete.cases(airquality),]

rf = randomForest(Ozone~., data = data)

pred = predict(rf, data)
importance(rf)
        IncNodePurity
Solar.R      17969.59
Wind         31978.36
Temp         34176.71
Month        10753.73
Day          15436.47
cat("RMSE: ", sqrt(mean((data$Ozone - pred)^2)), "\n")
RMSE:  9.507848 
plot(data$Temp, data$Ozone)
lines(data$Temp[order(data$Temp)], pred[order(data$Temp)], col = "red")

Try different values for the nodesize describe how the predictions depend on this parameter.

library(randomForest)
set.seed(123)

data = airquality[complete.cases(airquality),]


for(nodesize in c(1, 15, 50, 100)){
  for(mtry in c(1, 3, 5)){
    rf = randomForest(Ozone~., data = data, nodesize = nodesize)
    
    pred = predict(rf, data)
    
    plot(data$Temp, data$Ozone, main = paste0(
        "    nodesize: ", nodesize,
        "\nRMSE: ", round(sqrt(mean((data$Ozone - pred)^2)), 2)
      )
    )
    lines(data$Temp[order(data$Temp)], pred[order(data$Temp)], col = "red")
  }
}

Nodesize affects the complexity. In other words: The bigger the nodesize, the smaller the trees and the more bias/less variance.

Question: Boosted regression trees 
library(xgboost)
library(animation)
set.seed(123)

x1 = seq(-3, 3, length.out = 100)
x2 = seq(-3, 3, length.out = 100)
x = expand.grid(x1, x2)
y = apply(x, 1, function(t) exp(-t[1]^2 - t[2]^2))


image(matrix(y, 100, 100), main = "Original image", axes = FALSE, las = 2)
axis(1, at = seq(0, 1, length.out = 10),
     labels = round(seq(-3, 3, length.out = 10), 1))
axis(2, at = seq(0, 1, length.out = 10),
     labels = round(seq(-3, 3, length.out = 10), 1), las = 2)


model = xgboost::xgboost(xgb.DMatrix(data = as.matrix(x), label = y),
                         nrounds = 500L, verbose = 0L)
pred = predict(model, newdata = xgb.DMatrix(data = as.matrix(x)),
               ntreelimit = 10L)

saveGIF(
  {
    for(i in c(1, 2, 4, 8, 12, 20, 40, 80, 200)){
      pred = predict(model, newdata = xgb.DMatrix(data = as.matrix(x)),
                     ntreelimit = i)
      image(matrix(pred, 100, 100), main = paste0("Trees: ", i),
            axes = FALSE, las = 2)
      axis(1, at = seq(0, 1, length.out = 10),
           labels = round(seq(-3, 3, length.out = 10), 1))
      axis(2, at = seq(0, 1, length.out = 10),
           labels = round(seq(-3, 3, length.out = 10), 1), las = 2)
    }
  },
  movie.name = "boosting.gif", autobrowse = FALSE
)

Run the code above and play with different values for max_depth and describe what you see!

Tip: have a look at the boosting.gif.

library(xgboost)
library(animation)
set.seed(123)

x1 = seq(-3, 3, length.out = 100)
x2 = seq(-3, 3, length.out = 100)
x = expand.grid(x1, x2)
y = apply(x, 1, function(t) exp(-t[1]^2 - t[2]^2))

image(matrix(y, 100, 100), main = "Original image", axes = FALSE, las = 2)
axis(1, at = seq(0, 1, length.out = 10),
     labels = round(seq(-3, 3, length.out = 10), 1))
axis(2, at = seq(0, 1, length.out = 10),
     labels = round(seq(-3, 3, length.out = 10), 1), las = 2)

for(max_depth in c(3, 6, 10, 20)){
  model = xgboost::xgboost(xgb.DMatrix(data = as.matrix(x), label = y),
                           max_depth = max_depth,
                           nrounds = 500, verbose = 0L)

  saveGIF(
    {
      for(i in c(1, 2, 4, 8, 12, 20, 40, 80, 200)){
        pred = predict(model, newdata = xgb.DMatrix(data = as.matrix(x)),
                       ntreelimit = i)
        image(matrix(pred, 100, 100),
              main = paste0("eta: ", eta,
                            "    max_depth: ", max_depth,
                            "    Trees: ", i),
              axes = FALSE, las = 2)
        axis(1, at = seq(0, 1, length.out = 10),
             labels = round(seq(-3, 3, length.out = 10), 1))
        axis(2, at = seq(0, 1, length.out = 10),
             labels = round(seq(-3, 3, length.out = 10), 1), las = 2)
      }
    },
    movie.name = paste0("boosting_", max_depth, "_", eta, ".gif"),
    autobrowse = FALSE
  )
}

We see that for high values of max_depth, the predictions “smooth out” faster. On the other hand, with a low max_depth (low complexity of the individual trees), more trees are required in the ensemble to achieve a smooth prediction surface.

?xgboost::xgboost

Just some examples:

Question: Hyperparameter tuning of random forest
Hyperparameter Explanation
mtry Subset of features randomly selected in each node (from which the algorithm can select the feature that will be used to split the data).
minimum node size Minimal number of observations allowed in a node (before the branching is canceled)
max depth Maximum number of tree depth

Combing back to the titanic dataset from the morning, we want to optimize min node size in our RF using a simple CV.

Prepare the data:

library(EcoData)
library(dplyr)

Attaching package: 'dplyr'
The following object is masked from 'package:xgboost':

    slice
The following object is masked from 'package:randomForest':

    combine
The following objects are masked from 'package:stats':

    filter, lag
The following objects are masked from 'package:base':

    intersect, setdiff, setequal, union
library(missRanger)
data(titanic_ml)
data = titanic_ml
data = 
  data %>% select(survived, sex, age, fare, pclass)
data[,-1] = missRanger(data[,-1], verbose = 0)

data_sub =
  data %>%
    mutate(age = scales::rescale(age, c(0, 1)),
           fare = scales::rescale(fare, c(0, 1))) %>%
    mutate(sex = as.integer(sex) - 1L,
           pclass = as.integer(pclass - 1L))
data_new = data_sub[is.na(data_sub$survived),] # for which we want to make predictions at the end
data_obs = data_sub[!is.na(data_sub$survived),] # data with known response
data_sub$survived = as.factor(data_sub$survived)
data_obs$survived = as.factor(data_obs$survived)

Hints:

  • adjust the ‘type’ argument in the predict(…) method (the default is to predict classes)
  • when predicting probabilities, the randomForest will return a matrix, a column for each class, we are interested in the probability of surviving (so the second column)

Bonus:

  • tune min node size (and mtry)
  • use more features
library(ranger)
data_obs = data_sub[!is.na(data_sub$survived),] 
set.seed(42)

cv = 3
hyper_minnodesize = ...

tuning_results =
    sapply(1:length(hyper_minnodesize), function(k) {
        auc_inner = NULL
        for(j in 1:cv) {
          inner_split = as.integer(cut(1:nrow(data_obs), breaks = cv))
          train_inner = data_obs[inner_split != j, ]
          test_inner = data_obs[inner_split == j, ]
          
          model = ranger(survived~.,data = train_inner, min.node.size = hyper_minnodesize[k], probability = TRUE)
          predictions = predict(model, test_inner)$predictions[,2]
          
          auc_inner[j]= Metrics::auc(test_inner$survived, predictions)
        }
      return(mean(auc_inner))
    })

results = data.frame(minnodesize = hyper_minnodesize, AUC = tuning_results)

print(results)
library(ranger)

Attaching package: 'ranger'
The following object is masked from 'package:randomForest':

    importance
data_obs = data_sub[!is.na(data_sub$survived),] 
set.seed(42)

cv = 3
hyper_minnodesize = sample(300, 20)

tuning_results =
    sapply(1:length(hyper_minnodesize), function(k) {
        auc_inner = NULL
        for(j in 1:cv) {
          inner_split = as.integer(cut(1:nrow(data_obs), breaks = cv))
          train_inner = data_obs[inner_split != j, ]
          test_inner = data_obs[inner_split == j, ]
          model = ranger(survived~.,data = train_inner, min.node.size = hyper_minnodesize[k], probability = TRUE)
          predictions = predict(model, test_inner)$predictions[,2]
          
          auc_inner[j]= Metrics::auc(test_inner$survived, predictions)
        }
      return(mean(auc_inner))
    })

results = data.frame(minnodesize = hyper_minnodesize, AUC = tuning_results)

print(results)
   minnodesize       AUC
1           49 0.8269769
2          153 0.8186182
3           74 0.8191470
4          228 0.8139092
5          146 0.8176753
6          122 0.8190757
7          300 0.8080344
8          128 0.8202545
9           24 0.8296999
10          89 0.8206236
11         165 0.8170405
12         110 0.8198368
13          20 0.8302728
14         291 0.8090011
15         283 0.8105243
16         109 0.8209075
17           5 0.8282860
18         212 0.8122884
19         259 0.8125252
20         292 0.8081344

Make predictions:

model = ranger(survived~.,data = data_obs, min.node.size = results[which.max(results$AUC),1], probability = TRUE)

write.csv(data.frame(y = predict(model, data_new)$predictions[,1]), file = "Max_titanic_rf.csv")
Question: Hyperparameter tuning of boosted regression trees

Important hyperparameters:

Hyperparameter Explanation
eta learning rate (weighting of the sequential trees)
max depth maximal depth in the trees (small = low complexity, large = high complexity)
subsample subsample ratio of the data (bootstrap ratio)
lambda regularization strength of the individual trees
max tree maximal number of trees in the ensemble

Combing back to the titanic dataset from the morning, we want to optimize max depth and the eta parameter in xgboost.

Prepare the data:

library(EcoData)
library(dplyr)
library(missRanger)
data(titanic_ml)
data = titanic_ml
data = 
  data %>% select(survived, sex, age, fare, pclass)
data[,-1] = missRanger(data[,-1], verbose = 0)

data_sub =
  data %>%
    mutate(age = scales::rescale(age, c(0, 1)),
           fare = scales::rescale(fare, c(0, 1))) %>%
    mutate(sex = as.integer(sex) - 1L,
           pclass = as.integer(pclass - 1L))
data_new = data_sub[is.na(data_sub$survived),] # for which we want to make predictions at the end
data_obs = data_sub[!is.na(data_sub$survived),] # data with known response
library(xgboost)
set.seed(42)
data_obs = data_sub[!is.na(data_sub$survived),] 
cv = 3

outer_split = as.integer(cut(1:nrow(data_obs), breaks = cv))

# sample minnodesize values (must be integers)
hyper_depth = ...
hyper_eta = ...

tuning_results =
    sapply(1:length(hyper_minnodesize), function(k) {
        auc_inner = NULL
        for(j in 1:cv) {
          inner_split = as.integer(cut(1:nrow(data_obs), breaks = cv))
          train_inner = data_obs[inner_split != j, ]
          test_inner = data_obs[inner_split == j, ]
          
          data_xg = xgb.DMatrix(data = as.matrix(train_inner[,-1]), label = train_inner$survived)
          
          model = xgboost(data_xg, nrounds = 16L, eta = hyper_eta[k], max_depth = hyper_depth[k], objective = "reg:logistic")
          predictions = predict(model, newdata = as.matrix(test_inner)[,-1])
          
          auc_inner[j]= Metrics::auc(test_inner$survived, predictions)
        }
      return(mean(auc_inner))
    })

results = data.frame(depth = hyper_depth, eta = hyper_eta, AUC = tuning_results)

print(results)
library(xgboost)
set.seed(42)
data_obs = data_sub[!is.na(data_sub$survived),] 
cv = 3

outer_split = as.integer(cut(1:nrow(data_obs), breaks = cv))

# sample minnodesize values (must be integers)
hyper_depth = sample(200, 20)
hyper_eta = runif(20, 0, 1)


tuning_results =
    sapply(1:length(hyper_minnodesize), function(k) {
        auc_inner = NULL
        for(j in 1:cv) {
          inner_split = as.integer(cut(1:nrow(data_obs), breaks = cv))
          train_inner = data_obs[inner_split != j, ]
          test_inner = data_obs[inner_split == j, ]
          
          data_xg = xgb.DMatrix(data = as.matrix(train_inner[,-1]), label = train_inner$survived)
          
          model = xgboost(data_xg, nrounds = 16L, eta = hyper_eta[k], max_depth = hyper_depth[k], objective = "reg:logistic")
          predictions = predict(model, newdata = as.matrix(test_inner)[,-1])
          
          auc_inner[j]= Metrics::auc(test_inner$survived, predictions)
        }
      return(mean(auc_inner))
    })
[1] train-rmse:0.342302 
[2] train-rmse:0.299191 
[3] train-rmse:0.268716 
[4] train-rmse:0.242803 
[5] train-rmse:0.213241 
[6] train-rmse:0.192588 
[7] train-rmse:0.182530 
[8] train-rmse:0.174108 
[9] train-rmse:0.168385 
[10]    train-rmse:0.163249 
[11]    train-rmse:0.157466 
[12]    train-rmse:0.154353 
[13]    train-rmse:0.150252 
[14]    train-rmse:0.146819 
[15]    train-rmse:0.144884 
[16]    train-rmse:0.140921 
[1] train-rmse:0.355586 
[2] train-rmse:0.303011 
[3] train-rmse:0.279725 
[4] train-rmse:0.259973 
[5] train-rmse:0.244192 
[6] train-rmse:0.227948 
[7] train-rmse:0.213377 
[8] train-rmse:0.203723 
[9] train-rmse:0.195484 
[10]    train-rmse:0.189591 
[11]    train-rmse:0.185054 
[12]    train-rmse:0.180021 
[13]    train-rmse:0.173629 
[14]    train-rmse:0.168810 
[15]    train-rmse:0.164530 
[16]    train-rmse:0.162001 
[1] train-rmse:0.355397 
[2] train-rmse:0.312393 
[3] train-rmse:0.288034 
[4] train-rmse:0.262297 
[5] train-rmse:0.242050 
[6] train-rmse:0.231974 
[7] train-rmse:0.215598 
[8] train-rmse:0.206428 
[9] train-rmse:0.197788 
[10]    train-rmse:0.190170 
[11]    train-rmse:0.183465 
[12]    train-rmse:0.177776 
[13]    train-rmse:0.171649 
[14]    train-rmse:0.165496 
[15]    train-rmse:0.161950 
[16]    train-rmse:0.157418 
[1] train-rmse:0.344811 
[2] train-rmse:0.301841 
[3] train-rmse:0.269651 
[4] train-rmse:0.241837 
[5] train-rmse:0.223290 
[6] train-rmse:0.207494 
[7] train-rmse:0.195355 
[8] train-rmse:0.186628 
[9] train-rmse:0.174448 
[10]    train-rmse:0.168487 
[11]    train-rmse:0.159994 
[12]    train-rmse:0.156250 
[13]    train-rmse:0.151392 
[14]    train-rmse:0.146957 
[15]    train-rmse:0.143999 
[16]    train-rmse:0.141612 
[1] train-rmse:0.357827 
[2] train-rmse:0.304876 
[3] train-rmse:0.276290 
[4] train-rmse:0.257492 
[5] train-rmse:0.242544 
[6] train-rmse:0.229373 
[7] train-rmse:0.216883 
[8] train-rmse:0.207260 
[9] train-rmse:0.198043 
[10]    train-rmse:0.191355 
[11]    train-rmse:0.186312 
[12]    train-rmse:0.180745 
[13]    train-rmse:0.174197 
[14]    train-rmse:0.169893 
[15]    train-rmse:0.165370 
[16]    train-rmse:0.163342 
[1] train-rmse:0.357557 
[2] train-rmse:0.311745 
[3] train-rmse:0.286003 
[4] train-rmse:0.254900 
[5] train-rmse:0.241741 
[6] train-rmse:0.226210 
[7] train-rmse:0.214579 
[8] train-rmse:0.203467 
[9] train-rmse:0.197486 
[10]    train-rmse:0.189492 
[11]    train-rmse:0.183023 
[12]    train-rmse:0.177307 
[13]    train-rmse:0.172381 
[14]    train-rmse:0.167487 
[15]    train-rmse:0.163035 
[16]    train-rmse:0.160142 
[1] train-rmse:0.479481 
[2] train-rmse:0.461296 
[3] train-rmse:0.444457 
[4] train-rmse:0.429080 
[5] train-rmse:0.415456 
[6] train-rmse:0.402267 
[7] train-rmse:0.390538 
[8] train-rmse:0.379364 
[9] train-rmse:0.369656 
[10]    train-rmse:0.360619 
[11]    train-rmse:0.352002 
[12]    train-rmse:0.344516 
[13]    train-rmse:0.337348 
[14]    train-rmse:0.330759 
[15]    train-rmse:0.325420 
[16]    train-rmse:0.320087 
[1] train-rmse:0.480985 
[2] train-rmse:0.463953 
[3] train-rmse:0.447759 
[4] train-rmse:0.432252 
[5] train-rmse:0.418343 
[6] train-rmse:0.405816 
[7] train-rmse:0.394936 
[8] train-rmse:0.385023 
[9] train-rmse:0.376260 
[10]    train-rmse:0.368393 
[11]    train-rmse:0.361204 
[12]    train-rmse:0.353261 
[13]    train-rmse:0.346243 
[14]    train-rmse:0.339924 
[15]    train-rmse:0.334026 
[16]    train-rmse:0.329024 
[1] train-rmse:0.480786 
[2] train-rmse:0.463307 
[3] train-rmse:0.446951 
[4] train-rmse:0.432430 
[5] train-rmse:0.419509 
[6] train-rmse:0.407269 
[7] train-rmse:0.396934 
[8] train-rmse:0.386748 
[9] train-rmse:0.377648 
[10]    train-rmse:0.368996 
[11]    train-rmse:0.361208 
[12]    train-rmse:0.354212 
[13]    train-rmse:0.347979 
[14]    train-rmse:0.342421 
[15]    train-rmse:0.337069 
[16]    train-rmse:0.331923 
[1] train-rmse:0.391331 
[2] train-rmse:0.335947 
[3] train-rmse:0.305201 
[4] train-rmse:0.281717 
[5] train-rmse:0.265914 
[6] train-rmse:0.252585 
[7] train-rmse:0.237463 
[8] train-rmse:0.223741 
[9] train-rmse:0.214639 
[10]    train-rmse:0.207821 
[11]    train-rmse:0.199719 
[12]    train-rmse:0.193863 
[13]    train-rmse:0.186355 
[14]    train-rmse:0.182046 
[15]    train-rmse:0.177708 
[16]    train-rmse:0.173773 
[1] train-rmse:0.399989 
[2] train-rmse:0.347930 
[3] train-rmse:0.314260 
[4] train-rmse:0.293303 
[5] train-rmse:0.276249 
[6] train-rmse:0.264806 
[7] train-rmse:0.253870 
[8] train-rmse:0.244993 
[9] train-rmse:0.237339 
[10]    train-rmse:0.229343 
[11]    train-rmse:0.222446 
[12]    train-rmse:0.216332 
[13]    train-rmse:0.211237 
[14]    train-rmse:0.206756 
[15]    train-rmse:0.203657 
[16]    train-rmse:0.199263 
[1] train-rmse:0.399253 
[2] train-rmse:0.349250 
[3] train-rmse:0.323812 
[4] train-rmse:0.303647 
[5] train-rmse:0.291414 
[6] train-rmse:0.278053 
[7] train-rmse:0.269020 
[8] train-rmse:0.255987 
[9] train-rmse:0.247910 
[10]    train-rmse:0.239493 
[11]    train-rmse:0.229426 
[12]    train-rmse:0.221866 
[13]    train-rmse:0.216462 
[14]    train-rmse:0.211906 
[15]    train-rmse:0.206869 
[16]    train-rmse:0.203552 
[1] train-rmse:0.412613 
[2] train-rmse:0.356528 
[3] train-rmse:0.324341 
[4] train-rmse:0.302757 
[5] train-rmse:0.286213 
[6] train-rmse:0.272936 
[7] train-rmse:0.259950 
[8] train-rmse:0.251537 
[9] train-rmse:0.242169 
[10]    train-rmse:0.235786 
[11]    train-rmse:0.225925 
[12]    train-rmse:0.220411 
[13]    train-rmse:0.214880 
[14]    train-rmse:0.208503 
[15]    train-rmse:0.201849 
[16]    train-rmse:0.198086 
[1] train-rmse:0.419435 
[2] train-rmse:0.369414 
[3] train-rmse:0.334817 
[4] train-rmse:0.313577 
[5] train-rmse:0.296743 
[6] train-rmse:0.284304 
[7] train-rmse:0.273065 
[8] train-rmse:0.264564 
[9] train-rmse:0.252990 
[10]    train-rmse:0.246305 
[11]    train-rmse:0.239789 
[12]    train-rmse:0.233099 
[13]    train-rmse:0.229367 
[14]    train-rmse:0.224895 
[15]    train-rmse:0.220131 
[16]    train-rmse:0.216076 
[1] train-rmse:0.418746 
[2] train-rmse:0.369491 
[3] train-rmse:0.336421 
[4] train-rmse:0.316537 
[5] train-rmse:0.302093 
[6] train-rmse:0.289140 
[7] train-rmse:0.278160 
[8] train-rmse:0.268178 
[9] train-rmse:0.261453 
[10]    train-rmse:0.254947 
[11]    train-rmse:0.249780 
[12]    train-rmse:0.241135 
[13]    train-rmse:0.234711 
[14]    train-rmse:0.227473 
[15]    train-rmse:0.222749 
[16]    train-rmse:0.217501 
[1] train-rmse:0.347540 
[2] train-rmse:0.303745 
[3] train-rmse:0.273687 
[4] train-rmse:0.248077 
[5] train-rmse:0.223700 
[6] train-rmse:0.207197 
[7] train-rmse:0.193925 
[8] train-rmse:0.186242 
[9] train-rmse:0.178182 
[10]    train-rmse:0.173075 
[11]    train-rmse:0.166573 
[12]    train-rmse:0.162239 
[13]    train-rmse:0.158070 
[14]    train-rmse:0.149938 
[15]    train-rmse:0.144231 
[16]    train-rmse:0.142591 
[1] train-rmse:0.360274 
[2] train-rmse:0.309340 
[3] train-rmse:0.279982 
[4] train-rmse:0.257875 
[5] train-rmse:0.242942 
[6] train-rmse:0.224303 
[7] train-rmse:0.212563 
[8] train-rmse:0.206164 
[9] train-rmse:0.199549 
[10]    train-rmse:0.193388 
[11]    train-rmse:0.186543 
[12]    train-rmse:0.182867 
[13]    train-rmse:0.177301 
[14]    train-rmse:0.173397 
[15]    train-rmse:0.169897 
[16]    train-rmse:0.166461 
[1] train-rmse:0.359929 
[2] train-rmse:0.311848 
[3] train-rmse:0.286379 
[4] train-rmse:0.264875 
[5] train-rmse:0.247836 
[6] train-rmse:0.228336 
[7] train-rmse:0.218550 
[8] train-rmse:0.209393 
[9] train-rmse:0.199119 
[10]    train-rmse:0.191081 
[11]    train-rmse:0.183728 
[12]    train-rmse:0.179117 
[13]    train-rmse:0.172885 
[14]    train-rmse:0.168394 
[15]    train-rmse:0.164605 
[16]    train-rmse:0.160796 
[1] train-rmse:0.402424 
[2] train-rmse:0.344908 
[3] train-rmse:0.311936 
[4] train-rmse:0.293220 
[5] train-rmse:0.276182 
[6] train-rmse:0.263648 
[7] train-rmse:0.251038 
[8] train-rmse:0.241413 
[9] train-rmse:0.228978 
[10]    train-rmse:0.220899 
[11]    train-rmse:0.211078 
[12]    train-rmse:0.204254 
[13]    train-rmse:0.199894 
[14]    train-rmse:0.194752 
[15]    train-rmse:0.191196 
[16]    train-rmse:0.186992 
[1] train-rmse:0.410115 
[2] train-rmse:0.358492 
[3] train-rmse:0.326234 
[4] train-rmse:0.301400 
[5] train-rmse:0.286322 
[6] train-rmse:0.273809 
[7] train-rmse:0.262014 
[8] train-rmse:0.253343 
[9] train-rmse:0.245150 
[10]    train-rmse:0.235882 
[11]    train-rmse:0.231059 
[12]    train-rmse:0.224277 
[13]    train-rmse:0.217895 
[14]    train-rmse:0.212065 
[15]    train-rmse:0.208399 
[16]    train-rmse:0.204358 
[1] train-rmse:0.409393 
[2] train-rmse:0.360259 
[3] train-rmse:0.328650 
[4] train-rmse:0.307963 
[5] train-rmse:0.293266 
[6] train-rmse:0.282228 
[7] train-rmse:0.272837 
[8] train-rmse:0.265002 
[9] train-rmse:0.255129 
[10]    train-rmse:0.248240 
[11]    train-rmse:0.241525 
[12]    train-rmse:0.236144 
[13]    train-rmse:0.229078 
[14]    train-rmse:0.223361 
[15]    train-rmse:0.219269 
[16]    train-rmse:0.214754 
[1] train-rmse:0.352912 
[2] train-rmse:0.305237 
[3] train-rmse:0.276568 
[4] train-rmse:0.250577 
[5] train-rmse:0.231958 
[6] train-rmse:0.212720 
[7] train-rmse:0.203744 
[8] train-rmse:0.195451 
[9] train-rmse:0.184679 
[10]    train-rmse:0.175920 
[11]    train-rmse:0.170978 
[12]    train-rmse:0.164906 
[13]    train-rmse:0.161527 
[14]    train-rmse:0.159188 
[15]    train-rmse:0.155948 
[16]    train-rmse:0.151337 
[1] train-rmse:0.365109 
[2] train-rmse:0.309687 
[3] train-rmse:0.280797 
[4] train-rmse:0.261068 
[5] train-rmse:0.241774 
[6] train-rmse:0.227370 
[7] train-rmse:0.215068 
[8] train-rmse:0.207891 
[9] train-rmse:0.200634 
[10]    train-rmse:0.192557 
[11]    train-rmse:0.187296 
[12]    train-rmse:0.181621 
[13]    train-rmse:0.178007 
[14]    train-rmse:0.174441 
[15]    train-rmse:0.170647 
[16]    train-rmse:0.168465 
[1] train-rmse:0.364647 
[2] train-rmse:0.319787 
[3] train-rmse:0.295209 
[4] train-rmse:0.275162 
[5] train-rmse:0.259882 
[6] train-rmse:0.244643 
[7] train-rmse:0.227988 
[8] train-rmse:0.217011 
[9] train-rmse:0.209808 
[10]    train-rmse:0.200657 
[11]    train-rmse:0.194938 
[12]    train-rmse:0.187570 
[13]    train-rmse:0.183875 
[14]    train-rmse:0.179330 
[15]    train-rmse:0.173863 
[16]    train-rmse:0.169534 
[1] train-rmse:0.362176 
[2] train-rmse:0.311901 
[3] train-rmse:0.280702 
[4] train-rmse:0.260316 
[5] train-rmse:0.239976 
[6] train-rmse:0.224227 
[7] train-rmse:0.212696 
[8] train-rmse:0.204659 
[9] train-rmse:0.197053 
[10]    train-rmse:0.186446 
[11]    train-rmse:0.180915 
[12]    train-rmse:0.176416 
[13]    train-rmse:0.171562 
[14]    train-rmse:0.167962 
[15]    train-rmse:0.162565 
[16]    train-rmse:0.158046 
[1] train-rmse:0.373483 
[2] train-rmse:0.323919 
[3] train-rmse:0.297025 
[4] train-rmse:0.271697 
[5] train-rmse:0.258279 
[6] train-rmse:0.245115 
[7] train-rmse:0.232653 
[8] train-rmse:0.222997 
[9] train-rmse:0.214197 
[10]    train-rmse:0.205852 
[11]    train-rmse:0.201917 
[12]    train-rmse:0.194916 
[13]    train-rmse:0.189427 
[14]    train-rmse:0.186183 
[15]    train-rmse:0.182289 
[16]    train-rmse:0.177717 
[1] train-rmse:0.372885 
[2] train-rmse:0.325785 
[3] train-rmse:0.296763 
[4] train-rmse:0.279764 
[5] train-rmse:0.265770 
[6] train-rmse:0.253805 
[7] train-rmse:0.239162 
[8] train-rmse:0.228320 
[9] train-rmse:0.219802 
[10]    train-rmse:0.210701 
[11]    train-rmse:0.203631 
[12]    train-rmse:0.197762 
[13]    train-rmse:0.192693 
[14]    train-rmse:0.186841 
[15]    train-rmse:0.183264 
[16]    train-rmse:0.178558 
[1] train-rmse:0.355068 
[2] train-rmse:0.306965 
[3] train-rmse:0.278270 
[4] train-rmse:0.253591 
[5] train-rmse:0.230023 
[6] train-rmse:0.216835 
[7] train-rmse:0.206119 
[8] train-rmse:0.196151 
[9] train-rmse:0.188044 
[10]    train-rmse:0.182430 
[11]    train-rmse:0.174891 
[12]    train-rmse:0.166900 
[13]    train-rmse:0.164163 
[14]    train-rmse:0.160384 
[15]    train-rmse:0.156842 
[16]    train-rmse:0.154538 
[1] train-rmse:0.367055 
[2] train-rmse:0.312446 
[3] train-rmse:0.282541 
[4] train-rmse:0.263486 
[5] train-rmse:0.247370 
[6] train-rmse:0.229723 
[7] train-rmse:0.222708 
[8] train-rmse:0.212426 
[9] train-rmse:0.203696 
[10]    train-rmse:0.196197 
[11]    train-rmse:0.191197 
[12]    train-rmse:0.185815 
[13]    train-rmse:0.181837 
[14]    train-rmse:0.177194 
[15]    train-rmse:0.174400 
[16]    train-rmse:0.171251 
[1] train-rmse:0.366555 
[2] train-rmse:0.321138 
[3] train-rmse:0.297371 
[4] train-rmse:0.277279 
[5] train-rmse:0.257130 
[6] train-rmse:0.246154 
[7] train-rmse:0.230870 
[8] train-rmse:0.222219 
[9] train-rmse:0.214729 
[10]    train-rmse:0.203458 
[11]    train-rmse:0.195316 
[12]    train-rmse:0.190983 
[13]    train-rmse:0.184888 
[14]    train-rmse:0.177407 
[15]    train-rmse:0.173036 
[16]    train-rmse:0.169503 
[1] train-rmse:0.413003 
[2] train-rmse:0.356993 
[3] train-rmse:0.324332 
[4] train-rmse:0.302632 
[5] train-rmse:0.286470 
[6] train-rmse:0.273611 
[7] train-rmse:0.260870 
[8] train-rmse:0.253180 
[9] train-rmse:0.243336 
[10]    train-rmse:0.236671 
[11]    train-rmse:0.228742 
[12]    train-rmse:0.221460 
[13]    train-rmse:0.214504 
[14]    train-rmse:0.208995 
[15]    train-rmse:0.203657 
[16]    train-rmse:0.198992 
[1] train-rmse:0.419792 
[2] train-rmse:0.369830 
[3] train-rmse:0.335221 
[4] train-rmse:0.313943 
[5] train-rmse:0.297096 
[6] train-rmse:0.285331 
[7] train-rmse:0.274201 
[8] train-rmse:0.265720 
[9] train-rmse:0.257348 
[10]    train-rmse:0.249674 
[11]    train-rmse:0.242869 
[12]    train-rmse:0.237043 
[13]    train-rmse:0.230208 
[14]    train-rmse:0.226608 
[15]    train-rmse:0.223174 
[16]    train-rmse:0.217211 
[1] train-rmse:0.419104 
[2] train-rmse:0.369898 
[3] train-rmse:0.336688 
[4] train-rmse:0.316724 
[5] train-rmse:0.302290 
[6] train-rmse:0.289893 
[7] train-rmse:0.278490 
[8] train-rmse:0.270668 
[9] train-rmse:0.264460 
[10]    train-rmse:0.256828 
[11]    train-rmse:0.251322 
[12]    train-rmse:0.244360 
[13]    train-rmse:0.239434 
[14]    train-rmse:0.232481 
[15]    train-rmse:0.228628 
[16]    train-rmse:0.223731 
[1] train-rmse:0.367975 
[2] train-rmse:0.316612 
[3] train-rmse:0.285462 
[4] train-rmse:0.263922 
[5] train-rmse:0.245271 
[6] train-rmse:0.230708 
[7] train-rmse:0.221704 
[8] train-rmse:0.207412 
[9] train-rmse:0.198099 
[10]    train-rmse:0.191893 
[11]    train-rmse:0.187489 
[12]    train-rmse:0.178444 
[13]    train-rmse:0.173196 
[14]    train-rmse:0.165931 
[15]    train-rmse:0.164044 
[16]    train-rmse:0.160759 
[1] train-rmse:0.378739 
[2] train-rmse:0.319681 
[3] train-rmse:0.293201 
[4] train-rmse:0.270693 
[5] train-rmse:0.256631 
[6] train-rmse:0.240604 
[7] train-rmse:0.232202 
[8] train-rmse:0.224702 
[9] train-rmse:0.214182 
[10]    train-rmse:0.207344 
[11]    train-rmse:0.200937 
[12]    train-rmse:0.196956 
[13]    train-rmse:0.192187 
[14]    train-rmse:0.188495 
[15]    train-rmse:0.184100 
[16]    train-rmse:0.181208 
[1] train-rmse:0.378086 
[2] train-rmse:0.327091 
[3] train-rmse:0.297425 
[4] train-rmse:0.279148 
[5] train-rmse:0.265884 
[6] train-rmse:0.248846 
[7] train-rmse:0.240555 
[8] train-rmse:0.228152 
[9] train-rmse:0.220610 
[10]    train-rmse:0.213126 
[11]    train-rmse:0.204972 
[12]    train-rmse:0.201485 
[13]    train-rmse:0.194622 
[14]    train-rmse:0.188000 
[15]    train-rmse:0.182221 
[16]    train-rmse:0.178003 
[1] train-rmse:0.499000 
[2] train-rmse:0.498010 
[3] train-rmse:0.497026 
[4] train-rmse:0.496047 
[5] train-rmse:0.495073 
[6] train-rmse:0.494104 
[7] train-rmse:0.493140 
[8] train-rmse:0.492181 
[9] train-rmse:0.491226 
[10]    train-rmse:0.490275 
[11]    train-rmse:0.489330 
[12]    train-rmse:0.488389 
[13]    train-rmse:0.487453 
[14]    train-rmse:0.486522 
[15]    train-rmse:0.485596 
[16]    train-rmse:0.484675 
[1] train-rmse:0.499072 
[2] train-rmse:0.498161 
[3] train-rmse:0.497256 
[4] train-rmse:0.496356 
[5] train-rmse:0.495460 
[6] train-rmse:0.494569 
[7] train-rmse:0.493684 
[8] train-rmse:0.492802 
[9] train-rmse:0.491926 
[10]    train-rmse:0.491055 
[11]    train-rmse:0.490188 
[12]    train-rmse:0.489326 
[13]    train-rmse:0.488468 
[14]    train-rmse:0.487616 
[15]    train-rmse:0.486768 
[16]    train-rmse:0.485924 
[1] train-rmse:0.499062 
[2] train-rmse:0.498143 
[3] train-rmse:0.497230 
[4] train-rmse:0.496317 
[5] train-rmse:0.495408 
[6] train-rmse:0.494476 
[7] train-rmse:0.493582 
[8] train-rmse:0.492622 
[9] train-rmse:0.491667 
[10]    train-rmse:0.490716 
[11]    train-rmse:0.489771 
[12]    train-rmse:0.488830 
[13]    train-rmse:0.487895 
[14]    train-rmse:0.486964 
[15]    train-rmse:0.486038 
[16]    train-rmse:0.485116 
[1] train-rmse:0.353172 
[2] train-rmse:0.305445 
[3] train-rmse:0.276743 
[4] train-rmse:0.250790 
[5] train-rmse:0.232189 
[6] train-rmse:0.212974 
[7] train-rmse:0.204002 
[8] train-rmse:0.195040 
[9] train-rmse:0.183821 
[10]    train-rmse:0.175506 
[11]    train-rmse:0.172033 
[12]    train-rmse:0.165681 
[13]    train-rmse:0.159818 
[14]    train-rmse:0.156798 
[15]    train-rmse:0.153866 
[16]    train-rmse:0.150766 
[1] train-rmse:0.365344 
[2] train-rmse:0.309894 
[3] train-rmse:0.281018 
[4] train-rmse:0.261295 
[5] train-rmse:0.242028 
[6] train-rmse:0.227639 
[7] train-rmse:0.215355 
[8] train-rmse:0.208188 
[9] train-rmse:0.200927 
[10]    train-rmse:0.192864 
[11]    train-rmse:0.187600 
[12]    train-rmse:0.181933 
[13]    train-rmse:0.178311 
[14]    train-rmse:0.174738 
[15]    train-rmse:0.170801 
[16]    train-rmse:0.168272 
[1] train-rmse:0.364877 
[2] train-rmse:0.319963 
[3] train-rmse:0.295365 
[4] train-rmse:0.275363 
[5] train-rmse:0.260114 
[6] train-rmse:0.244910 
[7] train-rmse:0.228209 
[8] train-rmse:0.215883 
[9] train-rmse:0.209065 
[10]    train-rmse:0.201187 
[11]    train-rmse:0.192461 
[12]    train-rmse:0.185839 
[13]    train-rmse:0.178898 
[14]    train-rmse:0.173226 
[15]    train-rmse:0.167694 
[16]    train-rmse:0.163567 
[1] train-rmse:0.498143 
[2] train-rmse:0.496315 
[3] train-rmse:0.494504 
[4] train-rmse:0.492711 
[5] train-rmse:0.490936 
[6] train-rmse:0.489174 
[7] train-rmse:0.487429 
[8] train-rmse:0.485702 
[9] train-rmse:0.483991 
[10]    train-rmse:0.482297 
[11]    train-rmse:0.480619 
[12]    train-rmse:0.478958 
[13]    train-rmse:0.477314 
[14]    train-rmse:0.475685 
[15]    train-rmse:0.474073 
[16]    train-rmse:0.472476 
[1] train-rmse:0.498277 
[2] train-rmse:0.496595 
[3] train-rmse:0.494930 
[4] train-rmse:0.493283 
[5] train-rmse:0.491651 
[6] train-rmse:0.490036 
[7] train-rmse:0.488438 
[8] train-rmse:0.486855 
[9] train-rmse:0.485289 
[10]    train-rmse:0.483737 
[11]    train-rmse:0.482203 
[12]    train-rmse:0.480684 
[13]    train-rmse:0.479142 
[14]    train-rmse:0.477613 
[15]    train-rmse:0.476101 
[16]    train-rmse:0.474643 
[1] train-rmse:0.498258 
[2] train-rmse:0.496562 
[3] train-rmse:0.494873 
[4] train-rmse:0.493133 
[5] train-rmse:0.491486 
[6] train-rmse:0.489723 
[7] train-rmse:0.487977 
[8] train-rmse:0.486248 
[9] train-rmse:0.484536 
[10]    train-rmse:0.482840 
[11]    train-rmse:0.481160 
[12]    train-rmse:0.479496 
[13]    train-rmse:0.477849 
[14]    train-rmse:0.476217 
[15]    train-rmse:0.474602 
[16]    train-rmse:0.473010 
[1] train-rmse:0.450145 
[2] train-rmse:0.412699 
[3] train-rmse:0.382944 
[4] train-rmse:0.359949 
[5] train-rmse:0.341843 
[6] train-rmse:0.326485 
[7] train-rmse:0.314051 
[8] train-rmse:0.302660 
[9] train-rmse:0.292461 
[10]    train-rmse:0.284359 
[11]    train-rmse:0.277805 
[12]    train-rmse:0.271235 
[13]    train-rmse:0.265533 
[14]    train-rmse:0.260534 
[15]    train-rmse:0.255269 
[16]    train-rmse:0.250541 
[1] train-rmse:0.453901 
[2] train-rmse:0.420021 
[3] train-rmse:0.389930 
[4] train-rmse:0.367231 
[5] train-rmse:0.346715 
[6] train-rmse:0.331994 
[7] train-rmse:0.320548 
[8] train-rmse:0.311274 
[9] train-rmse:0.300817 
[10]    train-rmse:0.292880 
[11]    train-rmse:0.285697 
[12]    train-rmse:0.279500 
[13]    train-rmse:0.273323 
[14]    train-rmse:0.268322 
[15]    train-rmse:0.263914 
[16]    train-rmse:0.259803 
[1] train-rmse:0.453446 
[2] train-rmse:0.418830 
[3] train-rmse:0.391201 
[4] train-rmse:0.368218 
[5] train-rmse:0.351641 
[6] train-rmse:0.336232 
[7] train-rmse:0.323923 
[8] train-rmse:0.315101 
[9] train-rmse:0.306496 
[10]    train-rmse:0.300159 
[11]    train-rmse:0.294056 
[12]    train-rmse:0.288296 
[13]    train-rmse:0.284111 
[14]    train-rmse:0.278262 
[15]    train-rmse:0.273759 
[16]    train-rmse:0.268695 
[1] train-rmse:0.347479 
[2] train-rmse:0.303696 
[3] train-rmse:0.273634 
[4] train-rmse:0.248019 
[5] train-rmse:0.223632 
[6] train-rmse:0.207121 
[7] train-rmse:0.193844 
[8] train-rmse:0.186157 
[9] train-rmse:0.178096 
[10]    train-rmse:0.172990 
[11]    train-rmse:0.166492 
[12]    train-rmse:0.162156 
[13]    train-rmse:0.157989 
[14]    train-rmse:0.149858 
[15]    train-rmse:0.144153 
[16]    train-rmse:0.142515 
[1] train-rmse:0.360220 
[2] train-rmse:0.309292 
[3] train-rmse:0.279935 
[4] train-rmse:0.257812 
[5] train-rmse:0.242870 
[6] train-rmse:0.224231 
[7] train-rmse:0.212490 
[8] train-rmse:0.206091 
[9] train-rmse:0.199476 
[10]    train-rmse:0.193314 
[11]    train-rmse:0.186470 
[12]    train-rmse:0.182794 
[13]    train-rmse:0.177225 
[14]    train-rmse:0.173323 
[15]    train-rmse:0.169823 
[16]    train-rmse:0.166387 
[1] train-rmse:0.359876 
[2] train-rmse:0.311802 
[3] train-rmse:0.286337 
[4] train-rmse:0.264818 
[5] train-rmse:0.247766 
[6] train-rmse:0.228261 
[7] train-rmse:0.218473 
[8] train-rmse:0.209313 
[9] train-rmse:0.199035 
[10]    train-rmse:0.190997 
[11]    train-rmse:0.183643 
[12]    train-rmse:0.179042 
[13]    train-rmse:0.172667 
[14]    train-rmse:0.169350 
[15]    train-rmse:0.164097 
[16]    train-rmse:0.159964 
[1] train-rmse:0.377154 
[2] train-rmse:0.322996 
[3] train-rmse:0.288625 
[4] train-rmse:0.268110 
[5] train-rmse:0.249516 
[6] train-rmse:0.239117 
[7] train-rmse:0.224216 
[8] train-rmse:0.216819 
[9] train-rmse:0.204899 
[10]    train-rmse:0.199450 
[11]    train-rmse:0.192756 
[12]    train-rmse:0.186917 
[13]    train-rmse:0.178957 
[14]    train-rmse:0.175691 
[15]    train-rmse:0.171456 
[16]    train-rmse:0.163826 
[1] train-rmse:0.387078 
[2] train-rmse:0.328616 
[3] train-rmse:0.299080 
[4] train-rmse:0.282273 
[5] train-rmse:0.266443 
[6] train-rmse:0.254206 
[7] train-rmse:0.241954 
[8] train-rmse:0.230499 
[9] train-rmse:0.225849 
[10]    train-rmse:0.220852 
[11]    train-rmse:0.215028 
[12]    train-rmse:0.207753 
[13]    train-rmse:0.202053 
[14]    train-rmse:0.198234 
[15]    train-rmse:0.193598 
[16]    train-rmse:0.190045 
[1] train-rmse:0.386369 
[2] train-rmse:0.335963 
[3] train-rmse:0.313351 
[4] train-rmse:0.293987 
[5] train-rmse:0.278649 
[6] train-rmse:0.265678 
[7] train-rmse:0.255343 
[8] train-rmse:0.247244 
[9] train-rmse:0.232596 
[10]    train-rmse:0.225452 
[11]    train-rmse:0.218798 
[12]    train-rmse:0.210346 
[13]    train-rmse:0.204773 
[14]    train-rmse:0.197562 
[15]    train-rmse:0.192926 
[16]    train-rmse:0.188129 
[1] train-rmse:0.414606 
[2] train-rmse:0.358917 
[3] train-rmse:0.326077 
[4] train-rmse:0.303940 
[5] train-rmse:0.287606 
[6] train-rmse:0.274322 
[7] train-rmse:0.263745 
[8] train-rmse:0.254184 
[9] train-rmse:0.244747 
[10]    train-rmse:0.237209 
[11]    train-rmse:0.230712 
[12]    train-rmse:0.224574 
[13]    train-rmse:0.216699 
[14]    train-rmse:0.209627 
[15]    train-rmse:0.204999 
[16]    train-rmse:0.199288 
[1] train-rmse:0.421260 
[2] train-rmse:0.371556 
[3] train-rmse:0.337535 
[4] train-rmse:0.315164 
[5] train-rmse:0.297948 
[6] train-rmse:0.286094 
[7] train-rmse:0.273805 
[8] train-rmse:0.266394 
[9] train-rmse:0.259330 
[10]    train-rmse:0.249049 
[11]    train-rmse:0.242956 
[12]    train-rmse:0.236996 
[13]    train-rmse:0.233201 
[14]    train-rmse:0.229789 
[15]    train-rmse:0.224309 
[16]    train-rmse:0.220606 
[1] train-rmse:0.420579 
[2] train-rmse:0.371585 
[3] train-rmse:0.338361 
[4] train-rmse:0.318275 
[5] train-rmse:0.303723 
[6] train-rmse:0.291123 
[7] train-rmse:0.279848 
[8] train-rmse:0.272339 
[9] train-rmse:0.264819 
[10]    train-rmse:0.258476 
[11]    train-rmse:0.253214 
[12]    train-rmse:0.245536 
[13]    train-rmse:0.239850 
[14]    train-rmse:0.235902 
[15]    train-rmse:0.229998 
[16]    train-rmse:0.226427 
[1] train-rmse:0.404374 
[2] train-rmse:0.347051 
[3] train-rmse:0.313732 
[4] train-rmse:0.294703 
[5] train-rmse:0.276964 
[6] train-rmse:0.263043 
[7] train-rmse:0.251020 
[8] train-rmse:0.242893 
[9] train-rmse:0.234263 
[10]    train-rmse:0.226243 
[11]    train-rmse:0.217929 
[12]    train-rmse:0.211913 
[13]    train-rmse:0.204746 
[14]    train-rmse:0.199364 
[15]    train-rmse:0.194590 
[16]    train-rmse:0.189230 
[1] train-rmse:0.411898 
[2] train-rmse:0.360447 
[3] train-rmse:0.328007 
[4] train-rmse:0.302999 
[5] train-rmse:0.286952 
[6] train-rmse:0.274726 
[7] train-rmse:0.262966 
[8] train-rmse:0.255573 
[9] train-rmse:0.248647 
[10]    train-rmse:0.242319 
[11]    train-rmse:0.235167 
[12]    train-rmse:0.228224 
[13]    train-rmse:0.223082 
[14]    train-rmse:0.219313 
[15]    train-rmse:0.215510 
[16]    train-rmse:0.210092 
[1] train-rmse:0.411181 
[2] train-rmse:0.362094 
[3] train-rmse:0.330702 
[4] train-rmse:0.309895 
[5] train-rmse:0.297513 
[6] train-rmse:0.286619 
[7] train-rmse:0.274702 
[8] train-rmse:0.266107 
[9] train-rmse:0.256906 
[10]    train-rmse:0.249981 
[11]    train-rmse:0.240635 
[12]    train-rmse:0.232569 
[13]    train-rmse:0.227588 
[14]    train-rmse:0.221208 
[15]    train-rmse:0.215105 
[16]    train-rmse:0.210351 
results = data.frame(depth = hyper_depth, eta = hyper_eta, AUC = tuning_results)

print(results)
   depth         eta       AUC
1     49 0.988891729 0.8042100
2     65 0.946668233 0.7959009
3    153 0.082437558 0.8104643
4     74 0.514211784 0.8083882
5    146 0.390203467 0.8134649
6    122 0.905738131 0.7986125
7    200 0.446969628 0.8107391
8    128 0.836004260 0.8063349
9     47 0.737595618 0.8053308
10    24 0.811055141 0.8078353
11    71 0.388108283 0.8091617
12   100 0.685169729 0.8074317
13    89 0.003948339 0.7979312
14   165 0.832916080 0.8050775
15   110 0.007334147 0.7999174
16    20 0.207658973 0.8208386
17   154 0.906601408 0.7997587
18   114 0.611778643 0.8133862
19   111 0.379559241 0.8097394
20   131 0.435771585 0.8082372

Make predictions:

data_xg = xgb.DMatrix(data = as.matrix(data_obs[,-1]), label = data_obs$survived)

model = xgboost(data_xg, nrounds = 16L, eta = results[which.max(results$AUC), 2], max_depth = results[which.max(results$AUC), 1], objective = "reg:logistic")

predictions = predict(model, newdata = as.matrix(data_new)[,-1])

# Single predictions from the ensemble model:
write.csv(data.frame(y = predictions), file = "Max_titanic_xgboost.csv")
Bonus: Implement a BRT on your own!

You can easily implement a BRT or boosted linear model using the rpart package or the lm function.

Go through the code line by line and try to understand it. Ask, if you have any questions you cannot solve.

Let’s try it:

data = model.matrix(~. , data = airquality)

model = get_boosting_model(x = data[,-2], y = data[,2], n_trees = 5L )
pred = predict(model, newdata = data[,-2])
plot(data[,2], pred, xlab = "observed", ylab = "predicted")