I am using R and applying MART (Alg. for multiple additive regression trees) on a training set to build prediction models. When I look at the ROC curve, it is not monotone. I would be grateful if someone can help me with how I should fix this. I am guessing the issue is that initially, MART generates n trees and if these trees are not the same for all the models I am building, the results will not be comparable.

Here are the steps I take:

1) Fix the false-negative cost, c_fn. Let cost = c(0, 1, c_fn, 0).

2) use the following line to build the mart model:

mart(x, y, lx, martmode='class', niter=2000, cost.mtx=cost)

where x is the matrix of training set variables, y is the observation matrix, lx is the matrix which specifies which of the variables in x is numerical, which one categorical.

3) I predict the test set observations using the mart model found in step 2 using this line:

y_pred = martpred(x_test, probs=T)

4) I compute the false-positive and false-negative errors as follows:

t = 1/(1+c_fn) %threshold based on Bayes optimal rule where c_fp=1 and c_fn.

p_0 = length(which(y_test==1))/dim(y_test)[1]

p_01 = sum(1*(y_pred[,2]>t & y_test==0))/dim(y_test)[1]

p_11 = sum(1*(y_pred[,2]>t & y_test==1))/dim(y_test)[1]

p_fp = p_01/(1-p_0)

p_tp = p_11/p_0

5) repeat step 1-4 for a new false-negative cost.