基于目标检测的图形用户界面控件识别方法  

Graphical user interface widget extraction based on object detection

作　　者：林灏昶 秦云川 蔡宇辉 李肯立 唐卓 Lin Haochang;Qin Yunchuan;Cai Yuhui;Li Kenli;Tang Zhuo(College of Electrical and Information Engineering,Hunan University,Changsha,410082,China;College of Computer Science and Electronic Engineering,Hunan University,Changsha,410082,China)

机构地区：[1]湖南大学电气与信息工程学院,长沙410082 [2]湖南大学信息科学与工程学院,长沙410082

出　　处：《南京大学学报（自然科学版）》2022年第6期1012-1019,共8页Journal of Nanjing University（Natural Science）

基　　金：国家重点研发项目(2020YFB2104005);湖南省高新技术产业科技创新引领计划科技攻关项目(2020GK2037)。

摘　　要：传统机器人流程自动化(Robot Process Automation,RPA)主要使用操作系统和应用程序提供的接口获取图形用户界面(Graphical User Interface,GUI)控件,受操作系统和应用程序接口限制,但一些系统,如Linux,不提供获取控件信息的接口.提出一种基于神经网络对图形界面控件进行识别的方法,利用目标检测模型提取图形用户界面控件特征,在不使用操作系统接口的前提下识别图形用户界面内控件类别和几何信息,减少RPA对于系统与程序接口的依赖.同时,针对桌面端图形用户界面数据集缺失的问题,提出一种针对RPA领域桌面端图形用户界面目标检测数据集的生成方法.在该图形用户界面数据集上使用各类目标检测模型进行测试,结果显示,常用目标检测模型在识别用户界面控件的类别和几何信息时均能获得92%以上的准确率.Traditional RPA(Robot Process Automation)extracts GUI(Graphical User Interface)widget category and location information through the interface provided by operating system or application programs.However,the application scope of RPA is limited by operating system and application program interfaces.For example,some operation systems,such as Linux,have inadequate extraction interface access to widget category and location information.This paper proposes a method to extract GUI widget features based on neural network and extract the widget information by object detection.Object detection can fetch and understand the widget in desktop platform feature then further extract the widget category and location information without using any operation system or application program interface,which reduce the dependence of RPA on the operation system and application program interface.Meanwhile,aiming at the lack of the user interface database in desktop platform,an auto database generation method for target detection in RPA field is also proposed.The object detection models are tested on this user interface database in desktop platform.Experimental results show that the object detection models can obtain more than 92%mAP in extracting widget category and location information in desktop platform.

关 键 词：目标检测 机器人流程自动化 深度学习 图形用户界面自动生成 自动化测试 

分 类 号：TP311[自动化与计算机技术—计算机软件与理论]