Ggtcf

I'm trying to build decision tree classifier for binary classification problem. My dataset was unbalanced (1=173 and 0= 354) and I used the resample approach to increase minority class and make them balanced. I build a model using gridSearchCv and here my code

x=df_balanced["x"]
y=df_balanced['y']

X_train, X_test, Y_train, Y_test = model_selection.train_test_split( x, y, stratify=y, random_state=42,test_size=0.25)

pipeline = Pipeline([
 ('vectorizer',CountVectorizer(stop_words='english')),
 ('classifier', DecisionTreeClassifier(random_state=42))])


grid = 
 'vectorizer__ngram_range': [(1, 1), (1, 2)],
 'vectorizer__analyzer':('word', 'char'),
 'classifier__max_depth':[15,20,25,30]


grid_search = GridSearchCV(pipeline, param_grid=grid, scoring='accuracy', n_jobs=-1, cv=5)
grid_search.fit(X_train,Y_train)
print(grid_search.best_estimator_,"n")

best_parameters = grid_search.best_estimator_.get_params()
for param_name in sorted(list(grid.keys())):
 print("t0: 1".format(param_name, best_parameters[param_name]))
best_model = grid_search.best_estimator_
y_pred=best_model.predict(X_test)

confusion=confusion_matrix(Y_test, y_pred)
report=classification_report(Y_test,y_pred)
false_positive_rate, true_positive_rate, thresholds = roc_curve(Y_test, y_pred)
roc_auc = auc(false_positive_rate, true_positive_rate)
print("Confusion matrix n",confusion,"n")
print("Classification_report n ",report,"n")
print("Train Accuracy",accuracy_score(Y_train, best_model.predict(X_train)))
print("Test Accuracy:",accuracy_score(Y_test,y_pred))
print("roc_auc_score",roc_auc)

and the output.

Confusion matrix 
 [[82 7]
 [13 75]] 

Classification_report 
 precision recall f1-score support

 0 0.86 0.92 0.89 89
 1 0.91 0.85 0.88 88

 micro avg 0.89 0.89 0.89 177
 macro avg 0.89 0.89 0.89 177
weighted avg 0.89 0.89 0.89 177


Train Accuracy 0.9510357815442562
Test Accuracy: 0.8870056497175142
roc_auc_score 0.8868105209397344

To check if I fail in overfitting problem I calculated the train and test accuracy, and I think I am not too much overfitting.

Then I plot the depth of tree that may lead to overfitting.
the code

 #Setup arrays to store train and test accuracies
dep = np.arange(1, 50)
train_accuracy = np.empty(len(dep))
test_accuracy = np.empty(len(dep))

# Loop over different values of k
for i, k in enumerate(dep):

 model=best_model.fit(X_train,Y_train)
 y_pred = model.predict(X_test)

 #Compute accuracy on the training set
 train_accuracy[i] = model.score(X_train,Y_train)

 #Compute accuracy on the testing set
 test_accuracy[i] = model.score(X_test, Y_test)

# Generate plot
plt.title('clf: Varying depth of tree')
plt.plot(dep, test_accuracy, label = 'Testing Accuracy')
plt.plot(dep, train_accuracy, label = 'Training Accuracy')
plt.legend()
plt.xlabel('Depth of tree')
plt.ylabel('Accuracy')
plt.show()

The plot is very strange and I can not explain it.

Any help Plz

Looking closely at your for loop, you will realize that you always just fit the same model; the following line:

model=best_model.fit(X_train,Y_train)

does not depend in any way from your k and does not affect at all the max_depth parameter, as you actually intend to do.

Consequently, all the values of your (training & testing) accuracy are the same, hence the "strange" straight lines (i.e. constant values).

What I guess you want, is to get the performance metrics for the best parameters you have found from your CV and different depths; but the issue here is that max_depth is already included in your best_parameters, so your methodology looks rather vague...