2021科大讯飞-车辆贷款违约预测挑战赛Top1--方案学习

公众号：DS数说

作者：xihuishaw

简介

车贷违约预测问题，目的是建立风险识别模型来预测可能违约的借款人。预测结果为借款人是否可能违约，属于二分类问题。

偏数据挖掘的比赛，关键点是如何基于对数据的理解抽象归纳出有用的特征。

站在大佬的视角，尝试学习总结，站在巨人的肩膀上，也许看得会更远一些。

直接进入主题，开始学习套路，芜湖~

特征工程

1、常用库、数据导入

import pandas as pd
import numpy as np
import lightgbm as lgb
import xgboost as xgb
from sklearn.metrics import roc_auc_score, auc, roc_curve, accuracy_score, f1_score
from sklearn.model_selection import StratifiedKFold
from sklearn.preprocessing import StandardScaler, QuantileTransformer, KBinsDiscretizer, LabelEncoder, MinMaxScaler, PowerTransformer

from tqdm import tqdm
import pickle
import logging
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)
import os

后半部分用了一些工具：

• tqdm：一个优雅的进度条显示，方便观测跑数进度以及速度；
• pickle：将对象以文件的形式存放在磁盘上，几乎所有的数据类型都可以用pickle来序列化，一般先dump，后load，类似于写出、导入的意思；作用是，一次结果多次复用，避免重复做功，hhh，比如说A列数据处理得花2h，每次修改过后需重跑其他列数据，但无须修改A列数据，就可以用pickle解决这个问题，快速调取之前的结果；
• logging：控制台输出日志，方便查看运行状态；

logging.info('data loading...')
train = pd.read_csv('../xfdata/车辆贷款违约预测数据集/train.csv')
test = pd.read_csv('../xfdata/车辆贷款违约预测数据集/test.csv')

2、特征工程

2.1 构造特征

针对训练集、测试集：

1. 根据业务理解，计算新的特征；
2. 对某些比例特征进行等宽分箱（cut），对某些数值特征进行等频分箱（qcut），还有一些数值特征进行自定义分箱，划分bin的范围；

def gen_new_feats(train, test):
    '''生成新特征：如年利率/分箱等特征'''
    # Step 1: 合并训练集和测试集
    data = pd.concat([train, test])

    # Step 2: 具体特征工程
    # 计算二级账户的年利率
    data['sub_Rate'] = (data['sub_account_monthly_payment'] * data['sub_account_tenure'] - data[
        'sub_account_sanction_loan']) / data['sub_account_sanction_loan']

    # 计算主账户的年利率
    data['main_Rate'] = (data['main_account_monthly_payment'] * data['main_account_tenure'] - data[
        'main_account_sanction_loan']) / data['main_account_sanction_loan']

    # 对部分特征进行分箱操作
    # 等宽分箱
    loan_to_asset_ratio_labels = [i for i in range(10)]
    data['loan_to_asset_ratio_bin'] = pd.cut(data["loan_to_asset_ratio"], 10, labels=loan_to_asset_ratio_labels)
    # 等频分箱
    data['asset_cost_bin'] = pd.qcut(data['asset_cost'], 10, labels=loan_to_asset_ratio_labels)
    # 自定义分箱
    amount_cols = [
                   'total_monthly_payment',
                   'main_account_sanction_loan',
                   'main_account_disbursed_loan',
                   'sub_account_sanction_loan',
                   'sub_account_disbursed_loan',
                   'main_account_monthly_payment',
                   'sub_account_monthly_payment',
                   'total_sanction_loan'
                ]
    amount_labels = [i for i in range(10)]
    for col in amount_cols:
        total_monthly_payment_bin = [-1, 5000, 10000, 30000, 50000, 100000, 300000, 500000, 1000000, 3000000, data[col].max()]
        data[col + '_bin'] = pd.cut(data[col], total_monthly_payment_bin, labels=amount_labels).astype(int)

    # Step 3: 返回包含新特征的训练集 & 测试集
    return data[data['loan_default'].notnull()], data[data['loan_default'].isnull()]

2.2 编码-Target Encoding

Target encoding是一种结合目标值进行特征编码的方式。

在二分类中，对于特征i，target encoding在该特征取值为k时的编码值为类别k对应的目标值期望E(y|xi=xik)。

在样本集中一共有10条记录，其中3条记录中特征Trend的取值为Up，我们关注这3条记录。在k=Up时，目标值的期望为2/3 ≈ 0.66，所以将Up编码为0.66。

大佬后面主要是针对id特征进行target encoding。

def gen_target_encoding_feats(train, test, encode_cols, target_col, n_fold=10):
    '''生成target encoding特征'''
    # for training set - cv
    tg_feats = np.zeros((train.shape[0], len(encode_cols)))
    kfold = StratifiedKFold(n_splits=n_fold, random_state=1024, shuffle=True)
    for _, (train_index, val_index) in enumerate(kfold.split(train[encode_cols], train[target_col])):
        df_train, df_val = train.iloc[train_index], train.iloc[val_index]
        for idx, col in enumerate(encode_cols):
            target_mean_dict = df_train.groupby(col)[target_col].mean()
            df_val[f'{col}_mean_target'] = df_val[col].map(target_mean_dict)
            tg_feats[val_index, idx] = df_val[f'{col}_mean_target'].values

    for idx, encode_col in enumerate(encode_cols):
        train[f'{encode_col}_mean_target'] = tg_feats[:, idx]

    # for testing set
    for col in encode_cols:
        target_mean_dict = train.groupby(col)[target_col].mean()
        test[f'{col}_mean_target'] = test[col].map(target_mean_dict)

    return train, test

说实话，这段代码还没完全看明白~先用小本本记着，用的时候先直接掏出来，hhh

2.3 近邻欺诈特征

对于风控账户来说，存在风险的账户可能存在同批大量的注册情况，所以id可能是连着的。

这里大佬构建了近邻欺诈特征，就是每个账号的前后10个账户的lable取均值，也就代表着概率，意为可能违约账户聚集的概率，在一定程度上代表该账户可能违约的相关性。

def gen_neighbor_feats(train, test):
    '''产生近邻欺诈特征'''
    if not os.path.exists('../user_data/neighbor_default_probs.pkl'):
        # 该特征需要跑的时间较久，因此将其存成了pkl文件
        neighbor_default_probs = []
        for i in tqdm(range(train.customer_id.max())):
            if i >= 10 and i < 199706:
                customer_id_neighbors = list(range(i - 10, i)) + list(range(i + 1, i + 10))
            elif i < 199706:
                customer_id_neighbors = list(range(0, i)) + list(range(i + 1, i + 10))
            else:
                customer_id_neighbors = list(range(i - 10, i)) + list(range(i + 1, 199706))

            customer_id_neighbors = [customer_id_neighbor for customer_id_neighbor in customer_id_neighbors if
                                     customer_id_neighbor in train.customer_id.values.tolist()]
            neighbor_default_prob = train.set_index('customer_id').loc[customer_id_neighbors].loan_default.mean()
            neighbor_default_probs.append(neighbor_default_prob)

        df_neighbor_default_prob = pd.DataFrame({'customer_id': range(0, train.customer_id.max()),
                                                 'neighbor_default_prob': neighbor_default_probs})
        save_pkl(df_neighbor_default_prob, '../user_data/neighbor_default_probs.pkl')
    else:
        df_neighbor_default_prob = load_pkl('../user_data/neighbor_default_probs.pkl')
    train = pd.merge(left=train, right=df_neighbor_default_prob, on='customer_id', how='left')
    test = pd.merge(left=test, right=df_neighbor_default_prob, on='customer_id', how='left')

    return train, test

2.4 特征工程结果输出

TARGET_ENCODING_FETAS = [
                            'employment_type',
                             'branch_id',
                             'supplier_id',
                             'manufacturer_id',
                             'area_id',
                             'employee_code_id',
                             'asset_cost_bin'
                         ]


# 特征工程
logging.info('feature generating...')
train, test = gen_new_feats(train, test)
train, test = gen_target_encoding_feats(train, test, TARGET_ENCODING_FETAS, target_col='loan_default', n_fold=10)
train, test = gen_neighbor_feats(train, test)

特征的后续处理，比如一些转换后特征的数据类型转换，一些率值特征的简化，方便后续的模型学习，增强模型的鲁棒性。

# 保存的最终特征名称列表
SAVE_FEATS = [
                 'customer_id',
                 'neighbor_default_prob',
                 'disbursed_amount',
                 'asset_cost',
                 'branch_id',
                 'supplier_id',
                 'manufacturer_id',
                 'area_id',
                 'employee_code_id',
                 'credit_score',
                 'loan_to_asset_ratio',
                 'year_of_birth',
                 'age',
                 'sub_Rate',
                 'main_Rate',
                 'loan_to_asset_ratio_bin',
                 'asset_cost_bin',
                 'employment_type_mean_target',
                 'branch_id_mean_target',
                 'supplier_id_mean_target',
                 'manufacturer_id_mean_target',
                 'area_id_mean_target',
                 'employee_code_id_mean_target',
                 'asset_cost_bin_mean_target',
                 'credit_history',
                 'average_age',
                 'total_disbursed_loan',
                 'main_account_disbursed_loan',
                 'total_sanction_loan',
                 'main_account_sanction_loan',
                 'active_to_inactive_act_ratio',
                 'total_outstanding_loan',
                 'main_account_outstanding_loan',
                 'Credit_level',
                 'outstanding_disburse_ratio',
                 'total_account_loan_no',
                 'main_account_tenure',
                 'main_account_loan_no',
                 'main_account_monthly_payment',
                 'total_monthly_payment',
                 'main_account_active_loan_no',
                 'main_account_inactive_loan_no',
                 'sub_account_inactive_loan_no',
                 'enquirie_no',
                 'main_account_overdue_no',
                 'total_overdue_no',
                 'last_six_month_defaulted_no'
            ]


# 特征工程 后处理
# 简化特征
for col in ['sub_Rate', 'main_Rate', 'outstanding_disburse_ratio']:
     train[col] = train[col].apply(lambda x: 1 if x > 1 else x)
     test[col] = test[col].apply(lambda x: 1 if x > 1 else x)

# 数据类型转换
train['asset_cost_bin'] = train['asset_cost_bin'].astype(int)
test['asset_cost_bin'] = test['asset_cost_bin'].astype(int)
train['loan_to_asset_ratio_bin'] = train['loan_to_asset_ratio_bin'].astype(int)
test['loan_to_asset_ratio_bin'] = test['loan_to_asset_ratio_bin'].astype(int)

# 存储包含新特征的数据集
logging.info('new data saving...')
cols = SAVE_FEATS + ['loan_default', ]
train[cols].to_csv('./train_final.csv', index=False)
test[cols].to_csv('./test_final.csv', index=False)

模型构建

1、模型训练-交叉验证

采用lightgbm、xgboost两种梯度提升树模型，这里不多解释了，下面代码都成了“标准”，DDDD~

def train_lgb_kfold(X_train, y_train, X_test, n_fold=5):
    '''train lightgbm with k-fold split'''
    gbms = []
    kfold = StratifiedKFold(n_splits=n_fold, random_state=1024, shuffle=True)
    oof_preds = np.zeros((X_train.shape[0],))
    test_preds = np.zeros((X_test.shape[0],))

    for fold, (train_index, val_index) in enumerate(kfold.split(X_train, y_train)):
        logging.info(f'############ fold {fold} ###########')
        X_tr, X_val, y_tr, y_val = X_train.iloc[train_index], X_train.iloc[val_index], y_train[train_index], y_train[val_index]
        dtrain = lgb.Dataset(X_tr, y_tr)
        dvalid = lgb.Dataset(X_val, y_val, reference=dtrain)

        params = {
            'objective': 'binary',
            'metric': 'auc',
            'num_leaves': 64,
            'learning_rate': 0.02,
            'min_data_in_leaf': 150,
            'feature_fraction': 0.8,
            'bagging_fraction': 0.7,
            'n_jobs': -1,
            'seed': 1024
        }

        gbm = lgb.train(params,
                        dtrain,
                        num_boost_round=1000,
                        valid_sets=[dtrain, dvalid],
                        verbose_eval=50,
                        early_stopping_rounds=20)

        oof_preds[val_index] = gbm.predict(X_val, num_iteration=gbm.best_iteration)
        test_preds += gbm.predict(X_test, num_iteration=gbm.best_iteration) / kfold.n_splits
        gbms.append(gbm)

    return gbms, oof_preds, test_preds



def train_xgb_kfold(X_train, y_train, X_test, n_fold=5):
    '''train xgboost with k-fold split'''
    gbms = []
    kfold = StratifiedKFold(n_splits=10, random_state=1024, shuffle=True)
    oof_preds = np.zeros((X_train.shape[0],))
    test_preds = np.zeros((X_test.shape[0],))

    for fold, (train_index, val_index) in enumerate(kfold.split(X_train, y_train)):
        logging.info(f'############ fold {fold} ###########')
        X_tr, X_val, y_tr, y_val = X_train.iloc[train_index], X_train.iloc[val_index], y_train[train_index], y_train[val_index]
        dtrain = xgb.DMatrix(X_tr, y_tr)
        dvalid = xgb.DMatrix(X_val, y_val)
        dtest = xgb.DMatrix(X_test)

        params={
            'booster':'gbtree',
            'objective': 'binary:logistic',
            'eval_metric': ['logloss', 'auc'],
            'max_depth': 8,
            'subsample':0.9,
            'min_child_weight': 10,
            'colsample_bytree':0.85,
            'lambda': 10,
            'eta': 0.02,
            'seed': 1024
        }

        watchlist = [(dtrain, 'train'), (dvalid, 'test')]

        gbm = xgb.train(params,
                        dtrain,
                        num_boost_round=1000,
                        evals=watchlist,
                        verbose_eval=50,
                        early_stopping_rounds=20)

        oof_preds[val_index] = gbm.predict(dvalid, iteration_range=(0, gbm.best_iteration))
        test_preds += gbm.predict(dtest, iteration_range=(0, gbm.best_iteration)) / kfold.n_splits
        gbms.append(gbm)

    return gbms, oof_preds, test_preds

def train_xgb(train, test, feat_cols, label_col, n_fold=10):
    '''训练xgboost'''
    for col in ['sub_Rate', 'main_Rate', 'outstanding_disburse_ratio']:
        train[col] = train[col].apply(lambda x: 1 if x > 1 else x)
        test[col] = test[col].apply(lambda x: 1 if x > 1 else x)

    X_train = train[feat_cols]
    y_train = train[label_col]
    X_test = test[feat_cols]
    gbms_xgb, oof_preds_xgb, test_preds_xgb = train_xgb_kfold(X_train, y_train, X_test, n_fold=n_fold)

    if not os.path.exists('../user_data/gbms_xgb.pkl'):
        save_pkl(gbms_xgb, '../user_data/gbms_xgb.pkl')

    return gbms_xgb, oof_preds_xgb, test_preds_xgb


def train_lgb(train, test, feat_cols, label_col, n_fold=10):
    '''训练lightgbm'''
    X_train = train[feat_cols]
    y_train = train[label_col]
    X_test = test[feat_cols]
    gbms_lgb, oof_preds_lgb, test_preds_lgb = train_lgb_kfold(X_train, y_train, X_test, n_fold=n_fold)

    if not os.path.exists('../user_data/gbms_lgb.pkl'):
        save_pkl(gbms_lgb, '../user_data/gbms_lgb.pkl')

    return gbms_lgb, oof_preds_lgb, test_preds_lgb

输出模型训练结果：

# 读取原始数据集
logging.info('data loading...')
train = pd.read_csv('../xfdata/车辆贷款违约预测数据集/train.csv')
test = pd.read_csv('../xfdata/车辆贷款违约预测数据集/test.csv')

# 特征工程
logging.info('feature generating...')
train, test = gen_new_feats(train, test)
train, test = gen_target_encoding_feats(train, test, TARGET_ENCODING_FETAS, target_col='loan_default', n_fold=10)
train, test = gen_neighbor_feats(train, test)

train['asset_cost_bin'] = train['asset_cost_bin'].astype(int)
test['asset_cost_bin'] = test['asset_cost_bin'].astype(int)
train['loan_to_asset_ratio_bin'] = train['loan_to_asset_ratio_bin'].astype(int)
test['loan_to_asset_ratio_bin'] = test['loan_to_asset_ratio_bin'].astype(int)
train['asset_cost_bin_mean_target'] = train['asset_cost_bin_mean_target'].astype(float)
test['asset_cost_bin_mean_target'] = test['asset_cost_bin_mean_target'].astype(float)

# 模型训练：linux和mac的xgboost结果会有些许不同，以模型文件结果为主
gbms_xgb, oof_preds_xgb, test_preds_xgb = train_xgb(train.copy(), test.copy(),
                                                    feat_cols=SAVE_FEATS,
                                                    label_col='loan_default')
gbms_lgb, oof_preds_lgb, test_preds_lgb = train_lgb(train, test,
                                                    feat_cols=SAVE_FEATS,
                                                    label_col='loan_default')

2、划分阈值

因为是0-1二分类，最终分类的均值，可近似理解为取到loan_default=1的概率。
再通过对cv的预测结果排序，取分位数（1-P(loan_default=1)）对应的概率为预测正负样本的划分的临界点。

为了让结果更精准，采取小步长遍历临界点附近的点，找到局部最优的概率阈值。

def gen_thres_new(df_train, oof_preds):
    df_train['oof_preds'] = oof_preds
    # 可看作训练集取到loan_default=1的概率
    quantile_point = df_train['loan_default'].mean() 
    thres = df_train['oof_preds'].quantile(1 - quantile_point) 
    # 比如 0,1,1,1 mean=0.75 1-mean=0.25,也就是25%分位数取值为0

    _thresh = []
     #  按照理论阈值的上下0.2范围，0.01步长，找到最佳阈值，f1分数最高对应的阈值即为最佳阈值
    for thres_item in np.arange(thres - 0.2, thres + 0.2, 0.01):
        _thresh.append(
            [thres_item, f1_score(df_train['loan_default'], np.where(oof_preds > thres_item, 1, 0), average='macro')])

    _thresh = np.array(_thresh)
    best_id = _thresh[:, 1].argmax() # 找到f1最高对应的行
    best_thresh = _thresh[best_id][0] # 取出最佳阈值

    print("阈值: {}\n训练集的f1: {}".format(best_thresh, _thresh[best_id][1]))
    return best_thresh

3、模型融合

对xgb、lgb的模型cv结果的分位数进行加权求和，再去找融合后的模型0-1的概率阈值。

xgb_thres = gen_thres_new(train, oof_preds_xgb)
lgb_thres =  gen_thres_new(train, oof_preds_lgb)


# 结果聚合
df_oof_res = pd.DataFrame({'customer_id': train['customer_id'],
                            'loan_default':train['loan_default'],
                            'oof_preds_xgb': oof_preds_xgb,
                            'oof_preds_lgb': oof_preds_lgb})

# 模型融合
df_oof_res['xgb_rank'] = df_oof_res['oof_preds_xgb'].rank(pct=True) # percentile rank,返回的是排序后的分位数
df_oof_res['lgb_rank'] = df_oof_res['oof_preds_lgb'].rank(pct=True)

df_oof_res['preds'] = 0.31 * df_oof_res['xgb_rank'] + 0.69 * df_oof_res['lgb_rank']

# 融合后的模型，概率阈值
thres = gen_thres_new(df_oof_res, df_oof_res['preds'])

预测

按照融模后训练集的概率阈值，对测试集预测结果进行0-1划分，输出最终预测提交结果。

def gen_submit_file(df_test, test_preds, thres, save_path):
    # 按最终模型融合后的阈值进行划分
    df_test['test_preds_binary'] = np.where(test_preds > thres, 1, 0)  
    df_test_submit = df_test[['customer_id', 'test_preds_binary']]
    df_test_submit.columns = ['customer_id', 'loan_default']
    print(f'saving result to: {save_path}')
    df_test_submit.to_csv(save_path, index=False)
    print('done!')
    return df_test_submit



df_test_res = pd.DataFrame({'customer_id': test['customer_id'],
                                'test_preds_xgb': test_preds_xgb,
                                'test_preds_lgb': test_preds_lgb})

df_test_res['xgb_rank'] = df_test_res['test_preds_xgb'].rank(pct=True)
df_test_res['lgb_rank'] = df_test_res['test_preds_lgb'].rank(pct=True)
df_test_res['preds'] = 0.31 * df_test_res['xgb_rank'] + 0.69 * df_test_res['lgb_rank']

# 结果产出
df_submit = gen_submit_file(df_test_res, df_test_res['preds'], thres,
                            save_path='../prediction_result/result.csv')

总结

大佬的代码风格清晰、简洁，看代码非常流畅，思路也非常清晰，可以好好学习这些工程化的代码，可拓展性强，方便debug。

从赛题角度看，对业务的思考后从id集中度上做了一个“近邻欺诈特征”；在融模操作上，按预测值的ranking值分位数加权。这些小技巧都是可直接复用的~（也是大佬提到的上分点）

下面2个问题，估计很多同学和我一样也都会有些疑惑，我就从b乎直接截图出来：

源码：https://github.com/WangliLin/...

另外，我也整理了个ipynb，方便学习，需要的同学公众号后台回复“1208”获取

---

参考：

1. logging模块
2. pickle模块
3. tqdm模块
4. Target Encoding公式
5. Target Encoding
6. https://zhuanlan.zhihu.com/p/...

---

欢迎关注个人公众号：DS数说