基于大语言模型的移动应用可访问性增强方法  

Large Language Model-based Method for Mobile App Accessibility Enhancement

作　　者：马琦珉 李向民 周雅倩[1] MA Qimin;LI Xiangmin;ZHOU Yaqian(School of Computer Science and Technology,Fudan University,Shanghai 200438,China;Shanghai Key Laboratory of Data Science(Fudan University),Shanghai 200438,China)

机构地区：[1]复旦大学计算机科学技术学院,上海200438 [2]上海市数据科学重点实验室(复旦大学),上海200438

出　　处：《计算机科学》2024年第12期223-233,共11页Computer Science

摘　　要：移动应用可访问性(Mobile Application Accessibility)是指移动应用程序设计和实现的程度,目的是确保任何用户都能够轻松地访问和使用该应用。国内移动应用市场上的海量应用中支持无障碍功能的应用少之又少,与数量庞大且与日俱增的老年群体和视觉障碍群体追求享受数字时代红利、打破数字鸿沟的愿景产生矛盾。大规模语言模型(Large Language Model,LLM)在实现人类水平的智能方面表现出了巨大的潜力,通过提示词工程引导可以进行简单的逻辑推理和决策判断。此外,缩短交互路径是一种最为直观的移动应用可访问性增强方法。受到上述事实的启发,提出一种基于大规模语言模型的移动应用可访问性增强方法,创新性地应用可访问性服务和大语言模型,兼顾安全性、自动化和智能化。实现了一种移动应用可访问性辅助工具AccessLink,在非侵入式和用户授权的前提下,感知和操作移动应用的图形化用户界面,由此实现了基于自动化方法的数据集构建方法,并在构建的数据集上使用大模型GPT-3.5、GPT-4.0、通义千问和百川进行实验,证明了所提方法的有效性。Mobile application accessibility refers to the degree to which mobile applications are designed and implemented to ensure that any user can easily access the application.However,only a small fraction of the vast number of applications in the domestic mobile application market support accessibility features,which contradicts to the vision of breaking the digital divide and enjoying the benefits of the digital age for the growing elderly and visually impaired population.Large language models(LLMs)have demonstrated significant potential for achieving human-level intelligence.Through prompts guidance,they can engage in simple logical reasoning and decision-making.In addition,shortening the interactive pathway is an intuitive strategy for enhancing mobile application accessibility.Inspired by the aforementioned facts,we propose an innovative method for enhancing mobile application accessibility based on LLMs.This method creatively applies accessibility services and LLMs,aiming to improve security,automation,and intelligence.We have implemented a mobile application accessibility tool called AccessLink.Under the premise of non-invasiveness and user authorization,AccessLink perceives and interacts with the graphical user interface of mobile applications.Additionally,we have developed a dataset construction approach based on automated methods.Experimental validation is conducted using the constructed dataset with large models such as GPT-3.5,GPT-4.0,QianWen and Baichuan,demonstrating the effectiveness of the proposed method.

关 键 词：大语言模型 安卓 移动应用 可访问性 自然语言处理 

分 类 号：TP311[自动化与计算机技术—计算机软件与理论]