python - ensemble stacking: meta model results in worsen output than base models -
i doing ensemble stacking in hope meta model (lr) gives better results base model's predictions, measured auc. didn't! below similar example data made explain it:
(1) layer1 y1 y2 predictions , groundtruth y are: layer1_y1 layer1_y2 y 0 0.600 3.010000e-01 1 1 0.530 1.000000e-08 1 2 0.528 5.000000e-03 0 3 0.500 1.000000e-06 1 4 0.400 1.000000e-05 0 (2) logit loss of layer1_y1 vs y is: 0.6200905987848956 auc of layer1_y1 vs y is: 0.833333333333 (3) apply stacking (lr on layer1 y1 y2), resulted stacked_y , : layer1_y1 layer1_y2 y stacked_y 0 0.600 3.010000e-01 1 0.903171 1 0.530 1.000000e-08 1 0.590732 2 0.528 5.000000e-03 0 0.593332 3 0.500 1.000000e-06 1 0.543660 4 0.400 1.000000e-05 0 0.385901 (4) logit loss of stacked_y vs y improves: 0.5250046291474983 auc of stacked_y vs y decreases to: 0.666666666667 layer1_y1 stacked_y, order of row 0,3,4 stay unchanged, order of row 1,2 swapped, makes auc decrease, unexpectedly! expect stacked_y perform @ least not worse layer1_y1 , layer1_y2. why lr fails that?
from layer1_y1 stacked_y, order of row 0,3,4 stay unchanged, order of row 1,2 swapped, makes auc decrease 0.833333333333 0.666666666667, unexpectedly! expect stacked_y perform @ least not worse layer1_y1 , layer1_y2. since lr fails that, should still believe stacking never worsen improve outputs? or should still trust lr meta model job?
-- here codes reproduce above.
import tensorflow tf import numpy np sklearn import metrics import pandas pd data_ = pd.dataframe({'layer1_y1':[0.6,0.53,0.528,0.50,0.40],'layer1_y2':[0.301,1e-8,0.005,1e-6,1e-5], 'y':[1,1,0,1,0] }) print('(1) layer1 y1 y2 predictions , groundtruth y are:\n', data_ ) def get_cost(y_,x_): tmp1 = -(1-y_)*np.log(1-x_) -(y_)*np.log(x_) c = (tmp1) / len(y_) return np.sum(c) print('(2) logit loss of layer1_y1 vs y is:\t', get_cost(data_['y'],data_['layer1_y1']) ) print(' auc of layer1_y1 vs y is:\t\t', metrics.roc_auc_score(data_['y'],data_['layer1_y1'])) sklearn import linear_model stack_model = linear_model.logisticregressionstack_model = linear_model.logisticregression ( c= 180.0, penalty= 'l2', tol= 1e-8) stack_model.fit(data_[ ['layer1_y1','layer1_y2'] ], data_['y']) # print(stack_model.coef_, stack_model.intercept_ ) data_['stacked_y'] = stack_model.predict_proba(data_[ ['layer1_y1','layer1_y2'] ]) [:,1] print('(3) apply stacking (lr on layer1 y1 y2), resulted stacked_y , :\n', data_ ) print('(4) logit loss of stacked_y vs y improves:\t', get_cost(data_['y'],data_['stacked_y']) ) print(' auc of stacked_y vs y decreases to:\t', metrics.roc_auc_score(data_['y'],data_['stacked_y']),'\n') print('from layer1_y1 stacked_y, order of row 0,3,4 stay unchanged, order of row 1,2 swapped, makes auc decrease, unexpectedly!', 'i expect stacked_y perform @ least not worse layer1_y1 , layer1_y2. why lr fails that?')
appreciate opinions!
Comments
Post a Comment