Overfitting effect of artificial neural network based nonlinear equalizer: from mathematical origin to transmission evolution  

作　　者：Zheng YANG Fan GAO Songnian FU Ming TANG Deming LIU 

机构地区：[1]Wuhan National Laboratory for Optoelectronics,School of Optical and Electronic Information,Huazhong University of Science and Technology,Wuhan 430074,China [2]Alibaba Infrastructure Service,Alibaba Group,Hangzhou 311121,China

出　　处：《Science China(Information Sciences)》2020年第6期63-75,共13页中国科学（信息科学）（英文版）

基　　金：supported by National Key R&D Program of China(Grant No.2018YFB1801301);National Natural Science Foundation of China(Grant No.61875061);Key Project of R&D Program of Hubei Province(Grant No.2018AAA041)。

摘　　要：Overfitting effect of artificial neural network(ANN)based nonlinear equalizer(NLE)leads to a trap of bit error ratio(BER)overestimation in optical fiber communication system,especially when the performance is evaluated by the commonly-used pseudo-random binary sequence(PRBS).First,we mathematically investigate the PRBS generation and Gray code mapping rules,in comparison with the use of Mersenne Twister random sequence(MTRS).Under the condition of a symbol erasure channel,we identify that ANN can recognize both the PRBS generation and symbol mapping rules,by increasing the weights of NLE at specific positions,whereas the MTRS is currently safe owing to the limited input length of current ANN based NLE.Then,we design four channel models of fiber optical transmission to experimentally examine various impairments on the evolution of overfitting effect.When both the additive white Gaussian noise(AWGN)channel and the bandwidth limited channel are considered,the mitigation of overfitting becomes possible by the use of pruned PRBS(P-PRBS)training set with removing the generation and mapping rules determined input symbols.However,as for both the chromatic dispersion(CD)uncompensated channel and the CD managed channel,the overfitting effect becomes serious,because both CD and fiber nonlinearity induced inter-symbol interference(ISI)is beneficial for ANN to identify the PRBS symbol rules.Finally,possible solutions to mitigate the overfitting effect are summarized.

关 键 词：artificial neural network nonlinear equalizer pseudo-random binary sequence OVERFITTING 

分 类 号：TN715[电子电信—电路与系统] TP183[自动化与计算机技术—控制理论与控制工程]