python - ensemble stacking: meta model results in worsen output than base models -


i doing ensemble stacking in hope meta model (lr) gives better results base model's predictions, measured auc. didn't! below similar example data made explain it:

(1) layer1 y1 y2 predictions , groundtruth y are:     layer1_y1     layer1_y2  y 0      0.600  3.010000e-01  1 1      0.530  1.000000e-08  1 2      0.528  5.000000e-03  0 3      0.500  1.000000e-06  1 4      0.400  1.000000e-05  0 (2) logit loss of layer1_y1 vs y is:     0.6200905987848956     auc of layer1_y1 vs y is:        0.833333333333 (3) apply stacking (lr on layer1 y1 y2), resulted stacked_y , :     layer1_y1     layer1_y2  y  stacked_y 0      0.600  3.010000e-01  1   0.903171 1      0.530  1.000000e-08  1   0.590732 2      0.528  5.000000e-03  0   0.593332 3      0.500  1.000000e-06  1   0.543660 4      0.400  1.000000e-05  0   0.385901 (4) logit loss of stacked_y vs y improves:   0.5250046291474983     auc of stacked_y vs y decreases to:  0.666666666667   layer1_y1 stacked_y, order of row 0,3,4 stay unchanged, order of row 1,2 swapped, makes auc decrease, unexpectedly! expect stacked_y perform @ least not worse layer1_y1 , layer1_y2. why lr fails that? 

from layer1_y1 stacked_y, order of row 0,3,4 stay unchanged, order of row 1,2 swapped, makes auc decrease 0.833333333333 0.666666666667, unexpectedly! expect stacked_y perform @ least not worse layer1_y1 , layer1_y2. since lr fails that, should still believe stacking never worsen improve outputs? or should still trust lr meta model job?

-- here codes reproduce above.

import tensorflow tf import numpy np   sklearn import metrics import pandas pd data_ = pd.dataframe({'layer1_y1':[0.6,0.53,0.528,0.50,0.40],'layer1_y2':[0.301,1e-8,0.005,1e-6,1e-5], 'y':[1,1,0,1,0] }) print('(1) layer1 y1 y2 predictions , groundtruth y are:\n', data_ )  def get_cost(y_,x_):     tmp1 = -(1-y_)*np.log(1-x_) -(y_)*np.log(x_)     c = (tmp1) / len(y_)     return np.sum(c)  print('(2) logit loss of layer1_y1 vs y is:\t', get_cost(data_['y'],data_['layer1_y1']) ) print('    auc of layer1_y1 vs y is:\t\t', metrics.roc_auc_score(data_['y'],data_['layer1_y1']))   sklearn import linear_model stack_model = linear_model.logisticregressionstack_model = linear_model.logisticregression ( c= 180.0, penalty= 'l2', tol= 1e-8) stack_model.fit(data_[ ['layer1_y1','layer1_y2'] ], data_['y']) # print(stack_model.coef_,  stack_model.intercept_ ) data_['stacked_y']  = stack_model.predict_proba(data_[ ['layer1_y1','layer1_y2'] ]) [:,1] print('(3) apply stacking (lr on layer1 y1 y2), resulted stacked_y , :\n', data_ )  print('(4) logit loss of stacked_y vs y improves:\t', get_cost(data_['y'],data_['stacked_y']) ) print('    auc of stacked_y vs y decreases to:\t', metrics.roc_auc_score(data_['y'],data_['stacked_y']),'\n')   print('from layer1_y1 stacked_y, order of row 0,3,4 stay unchanged, order of row 1,2 swapped, makes auc decrease, unexpectedly!',      'i expect stacked_y perform @ least not worse layer1_y1 , layer1_y2. why lr fails that?') 

appreciate opinions!


Comments

Popular posts from this blog

resizing Telegram inline keyboard -

command line - How can a Python program background itself? -

php - "cURL error 28: Resolving timed out" on Wordpress on Azure App Service on Linux -