TextFormer: A Query-based End-to-end Text Spotter with Mixed Supervision  

作　　者：Yukun Zhai Xiaoqiang Zhang Xiameng Qin Sanyuan Zhao Xingping Dong Jianbing Shen 

机构地区：[1]School of Computer Science,Beijing Institute of Technology,Beijing 100081,China [2]Department of Computer Vision Technology,Baidu Inc.,Beijing 100193,China

出　　处：《Machine Intelligence Research》2024年第4期704-717,共14页机器智能研究（英文版）

基　　金：supported by the National Natural Science Foundation of China(No.61902027).

摘　　要：End-to-end text spotting is a vital computer vision task that aims to integrate scene text detection and recognition into a unified framework.Typical methods heavily rely on region-of-interest(Rol)operations to extract local features and complex post-processing steps to produce final predictions.To address these limitations,we propose TextFormer,a query-based end-to-end text spotter with a transformer architecture.Specifically,using query embedding per text instance,TextFormer builds upon an image encoder and a text decoder to learn a joint semantic understanding for multitask modeling.It allows for mutual training and optimization of classification,segmentation and recognition branches,resulting in deeper feature sharing without sacrificing flexibility or simplicity.Additionally,we design an adaptive global aggregation(AGG)module to transfer global features into sequential features for reading arbitrarilyshaped texts,which overcomes the suboptimization problem of Rol operations.Furthermore,potential corpus information is utilized from weak annotations to full labels through mixed supervision,further improving text detection and end-to-end text spotting results.Extensive experiments on various bilingual(i.e.,English and Chinese)benchmarks demonstrate the superiority of our method.Especially on the TDA-ReCTS dataset,TextFormer surpasses the state-of-the-art method in terms of 1-NED by 13.2%.

关 键 词：End-to-end text spotting arbitrarily-shaped texts transformer mixed supervision multitask modeling. 

分 类 号：TP181[自动化与计算机技术—控制理论与控制工程] TP391.41[自动化与计算机技术—控制科学与工程] R318[医药卫生—生物医学工程]