基于机器视觉的Web应用页面元素识别及可视化脚本生成  

Web Application Page Element Recognition and Visual Script Generation Based on Machine Vision

作　　者：李子东 姚怡飞 王微微[1] 赵瑞莲[1] LI Zi-dong;YAO Yi-fei;WANG Wei-wei;ZHAO Rui-lian(School of Information Science and Technology,Beijing University of Chemical Technology,Beijing 100029,China)

机构地区：[1]北京化工大学信息科学与技术学院,北京100029

出　　处：《计算机科学》2022年第11期65-75,共11页Computer Science

基　　金：国家自然科学基金(62077003,61872026)。

摘　　要：为了给用户提供丰富的交互响应,Web应用的可视化元素越发复杂多样,传统基于DOM的测试已不能满足Web应用的测试新需求。新一代基于机器视觉的测试方法为Web应用复杂元素的测试提供了一种有效途径。目前,此类方法主要依赖于模版匹配技术识别Web页面元素,以生成可视化测试脚本对Web应用进行测试。然而,页面元素外观的细微变化可导致模版匹配技术失效,从而无法识别Web页面元素,更无法生成可视化测试脚本。因此,如何提高基于机器视觉的Web页面元素识别的准确性,使其在复杂条件中仍然适用是一项具有挑战性的工作。基于深度学习的目标检测是当前计算机视觉和机器学习领域的研究热点,可通过大样本学习得到深层的数据特征表示,以准确识别目标,其泛化能力相比模板匹配更强。针对Web应用,从页面元素的视觉特征出发,提出了一种基于深度学习的Web页面元素识别方法,即利用基于深度学习的目标检测算法YOLOv3构建Web页面元素识别模型,自动定位元素的位置和边界,识别Web页面元素类型及功能描述;在此基础上,自动为Web应用生成可视化测试脚本,提升Web应用的测试效率。为了验证基于机器视觉的Web页面元素识别的准确性,针对同一Web应用的不同版本及不同Web应用分别进行实验,结果表明,基于机器视觉的Web页面元素识别模型的平均召回率为75.6%,可有效辅助Web应用可视化测试脚本生成。In order to provide richer interactive response effect,the visualization elements of the Web application is becoming more complex and diverse.The traditional test based on DOM cannot match the new requirement to test Web application.A new generation test based on computer vision provides an efficient way for the complex elements in Web application.This test for the Web based on computer vision mainly depends on template matching technique to recognize the page elements,so that it can ge-nerate visualization test script.However,the subtle changes of the page elements’appearance can lead to the failure of template matching technique,so that the Web page elements cannot be recognized and the visualization test script cannot be generated.How to improve the accuracy of Web page element recognition based on machine vision and make it still applicable in complex conditions is a challenging task.Object detection based on deep learning is a research hotspot in the field of today’s computer vision and machine learning.It has been shown from the deep data characteristics gained through the large sample learning that it can recognize the target accurately and has stronger generalization ability.Therefore,this paper targets the Web application,starts from the visual characteristics of the page elements,and proposes a Web page elements recognition method based on deep lear-ning.This method uses YOLOv3 algorithmic structure model based on the target testing to automatically localize the position and boundary of an element,recognize the type of Web page elements as well as describe its function.On the base of the Web page elements recognition,it can automatically generate visualization test script for the Web application.To verify the accuracy of the page elements recognition based on computer vision,experiments are set to test between different versions of the same Web application,and between different Web applications to analyze its accuracy.The results show that the average recall rate of machine vision-based Web page ele

关 键 词：WEB应用测试 Web应用页面元素识别 可视化测试脚本 测试脚本自动生成 机器视觉 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]