Stacking调参总结

Stacking调参总结,第1张

Stacking调参总结 1. 回归

训练了两个回归器,GBDT和Xgboost,用这两个回归器做stacking

使用之前已经调好参的训练器

gbdt_nxf = GradientBoostingRegressor(learning_rate=0.06,n_estimators=250,
min_samples_split=700,min_samples_leaf=70,max_depth=6,
max_features='sqrt',subsample=0.8,random_state=75)
xgb_nxf = XGBRegressor(learning_rate=0.06,max_depth=6,n_estimators=200,random_state=75)

  

事先建好stacking要用到的矩阵

from sklearn.model_selection import KFold,StratifiedKFold
kf = StratifiedKFold(n_splits=5,random_state=75,shuffle=True) from sklearn.metrics import r2_score train_proba = np.zeros((len(gbdt_train_data),2))
train_proba = pd.DataFrame(train_proba)
train_proba.columns = ['gbdt_nxf','xgb_nxf'] test_proba = np.zeros((len(gbdt_test_data),2))
test_proba = pd.DataFrame(test_proba)
test_proba.columns = ['gbdt_nxf','xgb_nxf']

  

reg_names = ['gbdt_nxf','xgb_nxf']

for i,reg in enumerate([gbdt_nxf,xgb_nxf]):
pred_list = []
col = reg_names[i]
for train_index,val_index in kf.split(gbdt_train_data,gbdt_train_label):
x_train = gbdt_train_data.loc[train_index,:].values
y_train = gbdt_train_label[train_index]
x_val = gbdt_train_data.loc[val_index,:].values
y_val = gbdt_train_label[val_index] reg.fit(x_train,y_train)
y_vali = reg.predict(x_val)
train_proba.loc[val_index,col] = y_vali
print('%s cv r2 %s'%(col,r2_score(y_val,y_vali))) y_testi = reg.predict(gbdt_test_data.values)
pred_list.append(y_testi)
test_proba.loc[:,col] = np.mean(np.array(pred_list),axis=0)

r2值最高为0.79753,效果还不是特别的好

然后用五折交叉验证,每折都预测整个测试集,得到五个预测的结果,求平均,就是新的预测集;而训练集就是五折中任意四折预测该折的训练集得到的标签的集合

因为有两个训练器,GBDT和Xgboost,所以我们得到了两列的train_proba

最后对新的训练集和测试集做回归,得到我们的结果

#使用逻辑回归做stacking
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
scalar = StandardScaler()
# train_proba = train_proba.values
# test_proba = test_proba.values scalar.fit(train_proba)
train_proba = scalar.transform(train_proba)
test_proba = scalar.transform(test_proba) lr = LogisticRegression(tol=0.0001,C=0.5,random_state=24,max_iter=10) kf = StratifiedKFold(n_splits=5,random_state=75,shuffle=True)
r2_list = []
pred_list = []
for train_index,val_index in kf.split(train_proba,gbdt_train_label):#训练集的标签还是一开始真实的训练集的标签
x_train = train_proba[train_index]
y_train = gbdt_train_label[train_index]
x_val = train_proba[val_index]
y_val = gbdt_train_label[val_index] lr.fit(x_train,y_train)
y_vali = lr.predict(x_val)
print('lr stacking cv r2 %s'%(r2_score(y_val,y_vali))) r2_list.append(r2_score(y_val,y_vali)) y_testi = lr.predict(test_proba)
pred_list.append(y_testi) print(lr.coef_,lr.n_iter_)#过拟合很严重

2. 分类

  经过对每个单模型进行调参之后,我们可以把这些模型进行 stacking 集成。


  如上图所示,我们将数据集分成均匀的5部分进行交叉训练,使用其中的4部分训练,之后将训练好的模型对剩下的1部分进行预测,同时预测测试集;经过5次cv之后,我们可以得到训练集每个样本的预测值,同时得到测试集的5个预测值,我们将测试集的5个测试集进行平均。


有多少个基模型,我们会得到几组不同的预测值;最后使用一个模型对上一步得到预测结果再进行训练预测,得到stacking结果。


stacking模型一般使用线性模型。


  stacking 有点像神经网络,基模型就像底层的神经网络对输入数据进行特征的提取,如下图所示:

首先我们先定义一个DataFrame 格式数据结构荣来存储中间预测结果:

train_proba = np.zeros((len(train), 6))
train_proba = pd.DataFrame(train_proba)
train_proba.columns = ['rf','ada','etc','gbc','sk_xgb','sk_lgb'] test_proba = np.zeros((len(test), 6))
test_proba = pd.DataFrame(test_proba)
test_proba.columns = ['rf','ada','etc','gbc','sk_xgb','sk_lgb']

定义基模型,交叉训练预测

rf = RandomForestClassifier(n_estimators=700, max_depth=13, min_samples_split=30,\
min_weight_fraction_leaf=0.0, random_state=24, verbose=0) ada = AdaBoostClassifier(n_estimators=450, learning_rate=0.1, random_state=24) gbc = GradientBoostingClassifier(learning_rate=0.08,n_estimators=150,max_depth=9,
min_samples_leaf=70,min_samples_split=900,
max_features='sqrt',subsample=0.8,random_state=10) etc = ExtraTreesClassifier(n_estimators=290, max_depth=12, min_samples_split=30,random_state=24) sk_xgb = XGBClassifier(learning_rate=0.05,n_estimators=400,
min_child_weight=20,max_depth=3,subsample=0.8, colsample_bytree=0.8,
reg_lambda=1., random_state=10) sk_lgb = LGBMClassifier(num_leaves=31,max_depth=3,learing_rate=0.03,n_estimators=600,
subsample=0.8, colsample_bytree=0.9, objective='binary',
min_child_weight=0.001, subsample_freq=1, min_child_samples=10,
reg_alpha=0.0, reg_lambda=0.0, random_state=10, n_jobs=-1,
silent=True, importance_type='split') kf = StratifiedKFold(n_splits=5,random_state=233,shuffle=True) clf_name = ['rf','ada','etc','gbc','sk_xgb','sk_lgb']
for i,clf in enumerate([rf,ada,etc,gbc,sk_xgb,sk_lgb]):
pred_list = []
col = clf_name[i]
for train_index, val_index in kf.split(train,label):
X_train = train.loc[train_index,:].values
y_train = label[train_index]
X_val = train.loc[val_index,:].values
y_val = label[val_index] clf.fit(X_train, y_train)
y_vali = clf.predict_proba(X_val)[:,1]
train_proba.loc[val_index,col] = y_vali
print("%s cv auc %s" % (col, roc_auc_score(y_val, y_vali))) y_testi = clf.predict_proba(test.values)[:,1]
pred_list.append(y_testi) test_proba.loc[:,col] = np.mean(np.array(pred_list),axis=0)

使用逻辑回归做最后的stacking  

scaler = StandardScaler()
train_proba = train_proba.values
test_proba = test_proba.values scaler.fit(train_proba)
train_proba = scaler.transform(train_proba)
test_proba = scaler.transform(test_proba) lr = LogisticRegression(tol=0.0001, C=0.5, random_state=24, max_iter=10) kf = StratifiedKFold(n_splits=5,random_state=244,shuffle=True)
auc_list = []
pred_list = []
for train_index, val_index in kf.split(train_proba,label):
X_train = train_proba[train_index]
y_train = label[train_index]
X_val = train_proba[val_index]
y_val = label[val_index] lr.fit(X_train, y_train)
y_vali = lr.predict_proba(X_val)[:,1]
print("lr stacking cv auc %s" % (roc_auc_score(y_val, y_vali))) auc_list.append(roc_auc_score(y_val, y_vali)) y_testi = lr.predict_proba(test_proba)[:,1]
pred_list.append(y_testi) print(lr.coef_, lr.n_iter_)

最终各个基模型和stacking模型的 auc 得分如下图所示:  

分别为 0.8415,0.8506,0.8511,0.8551,0.8572,0.8580,0.8584。


欢迎分享,转载请注明来源:内存溢出

原文地址: http://outofmemory.cn/zaji/585855.html

(0)
打赏 微信扫一扫 微信扫一扫 支付宝扫一扫 支付宝扫一扫
上一篇 2022-04-12
下一篇 2022-04-12

发表评论

登录后才能评论

评论列表(0条)

保存