Ensemble with Gradient Boosting in Python

We use the data from sklearn library, and the IDE is sublime text3. Most of the code comes from the book: https://www.goodreads.com/book/show/32439431-introduction-to-machine-learning-with-python?from_search=true

from sklearn.ensemble import GradientBoostingClassifier
import matplotlib.pyplot as plt 
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_breast_cancer

cancer=load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
	cancer.data, cancer.target, random_state=0)
###Gradient boosted regression trees is another ensemble method that combines multiple 
###decision trees to a more powerful model. Despite the “regression” in the name, 
###these models can be used for regression and classification, with strong pre-pruning.

gbrt=GradientBoostingClassifier(random_state=0)
gbrt.fit(X_train,y_train)
print("accuracy on training%f"%gbrt.score(X_train,y_train))
print('\n'"accuracy on test%f"%gbrt.score(X_test,y_test))
###accuracy on training1.000000
###accuracy on test0.965035

###to avoid orverfitting, limit the maximum depth or lower the learning rate
###learning_rate controls how strongly each tree corrects the mistakes of the previous trees

gbrt = GradientBoostingClassifier(random_state=0, max_depth=1)
gbrt.fit(X_train, y_train)
print('\n'"accuracy on training set: %f" % gbrt.score(X_train, y_train))
print('\n'"accuracy on test set: %f" % gbrt.score(X_test, y_test))
###accuracy on training set: 0.990610
###accuracy on test set: 0.972028

gbrt = GradientBoostingClassifier(random_state=0, learning_rate=0.01)
gbrt.fit(X_train, y_train)
print('\n'"accuracy on training set: %f" % gbrt.score(X_train, y_train))
print('\n'"accuracy on test set: %f" % gbrt.score(X_test, y_test))
###accuracy on training set: 1.000000
###accuracy on test set: 0.965035
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s