深度学习中对比散度算法的有偏性分析  

Biasness of contrastive divergence in deep learning

作　　者：吴海佳[1] 张雄伟[1] 孙蒙[1] 杨吉斌[1] 

机构地区：[1]解放军理工大学指挥信息系统学院,江苏南京210007

出　　处：《解放军理工大学学报（自然科学版）》2015年第3期224-230,共7页Journal of PLA University of Science and Technology(Natural Science Edition)

基　　金：国家自然科学基金资助项目(NSFC61471394;NSFC61402519);江苏省自然科学基金资助项目(BK2012510);江苏省青年基金资助项目(BK20140071;BK20140074)

摘　　要：为了给对比散度算法的进一步优化提供理论指导,尝试从理论上分析对比散度算法的收敛性。首先从仅含4个结点的玻尔兹曼机入手,利用单纯形表征模型的概率空间,以及流形表征概率空间与模型参数的关系,形象地表示了对比散度算法和极大似然算法的收敛过程,并从理论上推导出对比散度算法的收敛集与极大似然算法的收敛集之差不为空,从而证明了对比散度算法的有偏性。基于该结论,设计了一种先利用对比散度算法进行预训练,再利用极大似然算法调优的训练策略。实验结果表明,在应用该策略获得同等收敛效果的条件下,训练迭代步骤降低了83.3%。Some theoretical problems on the convergence property of the contrastive divergence （CD）algo-rithm were investigated,providing theoretical guidance for optimizing the CD algorithm.Simplex was used to represent the probability space of the model,and manifold used to represent the relationship between the probability space and parameters of the model.Both of them help to reveal the convergence process vis-ually.Compared with the results from normal maximum likelihood estimation （MLE）for a Boltzmann ma-chine with only 4 nodes,the CD algorithm actually has biasness.Based on this conclusion,a new training strategy of CD pre-training followed by MLE fine-tuning was designed.The experimental results show that,in the same convergence condition,the procedure of the algorithm with the new strategy is reduced by 83.3% compared with the traditional algorithm.

关 键 词：深度学习 对比散度 受限玻尔兹曼机 极大似然估计 

分 类 号：TP181[自动化与计算机技术—控制理论与控制工程]