Multimodal Dependence Attention and Large-Scale Data Based Offline Handwritten Formula Recognition  

作　　者：刘汉超 董兰芳 张信明 Han-Chao Liu;Lan-Fang Dong;Xin-Ming Zhang(School of Computer Science and Technology,University of Science and Technology of China,Hefei 230022,China;CCF;IEEE)

机构地区：[1]School of Computer Science and Technology,University of Science and Technology of China,Hefei 230022,China [2]CCF [3]IEEE

出　　处：《Journal of Computer Science & Technology》2024年第3期654-670,共17页计算机科学技术学报（英文版）

基　　金：supported by the National Key Research and Development Program of China under Grant No.2020YFB1313602.

摘　　要：Offline handwritten formula recognition is a challenging task due to the variety of handwritten symbols and two-dimensional formula structures.Recently,the deep neural network recognizers based on the encoder-decoder frame-work have achieved great improvements on this task.However,the unsatisfactory recognition performance for formulas with long LTeX strings is one shortcoming of the existing work.Moreover,lacking sufficient training data also limits the capability of these recognizers.In this paper,we design a multimodal dependence attention(MDA)module to help the model learn visual and semantic dependencies among symbols in the same formula to improve the recognition perfor-mance of the formulas with long LTeX strings.To alleviate overfitting and further improve the recognition performance,we also propose a new dataset,Handwritten Formula Image Dataset(HFID),which contains 25620 handwritten formula images collected from real life.We conduct extensive experiments to demonstrate the effectiveness of our proposed MDA module and HFID dataset and achieve state-of-the-art performances,63.79%and 65.24%expression accuracy on CROHME 2014 and CROHME 2016,respectively.

关 键 词：handwritten formula recognition multimodal dependence attention semantic dependence visual dependence Handwritten Formula Image Dataset 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]