基于Transformer编码器和残差网络的信贷违约预测模型  

Credit default prediction model based on Transformer encoder and residual network

作　　者：张瑶娜 卓佩妍 刘自金 刘炜 宋友[1] ZHANG Yaona;ZHUO Peiyan;LIU Zijin;LIU Wei;SONG You(College of Software,Beihang University,Beijing 100191,China)

机构地区：[1]北京航空航天大学软件学院,北京100191

出　　处：《计算机应用》2024年第S01期324-329,共6页journal of Computer Applications

基　　金：北航渤海大数据联合创新风控金融科技二期项目。

摘　　要：针对传统信贷违约预测模型对高维稀疏类别特征缺乏有效处理,性能受到人工特征工程影响较大的问题,提出一种基于Transformer编码器和残差网络的信贷违约预测模型(TE-ResNet)。该模型首先利用嵌入层对类别特征进行处理,将它们转化为低维度的稠密向量;然后将连续特征和嵌入后的类别特征连接,输入到堆叠的Transformer编码器中进行特征提取,捕捉输入特征之间的关系,得到有用信息的高层特征表示;最后使用结合了通道注意力机制的一维残差网络模型进行违约预测。在训练过程中,模型采用加权交叉熵损失函数,以解决信贷数据不平衡的问题。实验结果表明,与8种主流基准模型的最佳表现相比,TE-ResNet在LendingClub数据集、天池贷款数据集上的各项指标均有提升:AUC指标分别提升了0.58%和2.85%,F1-Score指标分别提升了0.85%和11.92%,G-mean指标分别提升了2.94%和16.19%。TE-ResNet能够提高信贷违约预测的性能,减少人工特征工程,实现端到端的学习。因此,TE-ResNet模型具有实际应用的潜力,并可为信贷业务提供更加精确和可靠的风险评估服务。Aiming at the problem that the traditional credit default prediction models lack effective processing of high-dimensional sparse category features and the performance is greatly affected by artificial feature engineering,therefore a credit default prediction model based on Transformer Encoder and Residual Network(TE-ResNet)was proposed.Firstly,the embedding layer was used to process the categorical features into low-dimensional dense vectors.Then the continuous features and embedded categorical features were connected,and the stacked Transformer encoders were used for feature extraction to capture the relationships between input features and obtain high-level feature representation of useful information.Finally,a one-dimensional residual network model with channel attention mechanism was used to default prediction.During training,weighted cross-entropy was used in loss function to address the imbalanced credit data.Experimental results show that,compared with the best performance of 8 mainstream benchmark models,TE-ResNet improves all metrics on Lending Club dataset and Tianchi credit dataset.Specifically,TE-ResNet increases the AUC indicators by 0.58%and 2.85%respectively,and increases the F1-Score indicators by 0.85%and 11.92%respectively,and increases the G-mean indicators by 2.94%and 16.19%respectively.TE-ResNet can improve the performance of credit default prediction,reduce manual feature engineering,and realize end-to-end learning.Therefore,TE-ResNet model has the potential of practical application and can provide more accurate and reliable risk assessment services for credit business.

关 键 词：深度学习 残差网络 TRANSFORMER 注意力机制 信贷违约预测 

分 类 号：TP389.1[自动化与计算机技术—计算机系统结构]