一种基于强化学习的持续集成环境中测试用例排序技术  

Test Case Prioritization Technique in Continuous Integration Based on Reinforcement Learning

作　　者：赵逸凡 郝丹 ZHAO Yi-Fan;HAO Dan(Key Laboratory of High Confidence Software Technologies of Ministry of Education(Peking University),Beijing 100871,China;School of Computer Science,Peking University,Beijing 100871,China)

机构地区：[1]高可信软件技术教育部重点实验室(北京大学),北京100871 [2]北京大学计算机学院,北京100871

出　　处：《软件学报》2023年第6期2708-2726,共19页Journal of Software

基　　金：国家自然科学基金(61872008)。

摘　　要：在软件交付越来越强调迅速、可靠的当下,持续集成成为一项备受关注的技术.开发人员不断将工作副本集成到代码主干完成软件演化,每次集成会通过自动构建测试来验证代码更新是否引入错误.但随着软件规模的增大,测试用例集包含的测试用例越来越多,测试用例的覆盖范围、检错效果等特征也随着集成周期的延长而变化,传统的测试用例排序技术难以适用.基于强化学习的测试排序技术可以根据测试反馈动态调整排序策略,但现有的相关技术不能综合考虑测试用例集中的信息进行排序,这限制了它们的性能.提出一种新的基于强化学习的持续集成环境中测试用例排序方法——指针排序方法:方法使用测试用例的历史信息等特征作为输入,在每个集成周期中,智能体利用指针注意力机制获得对所有备选测试用例的关注程度,由此得到排序结果,并从测试执行的反馈得到策略更新的方向,在“排序-运行测试-反馈”的过程中不断调整排序策略,最终达到良好的排序性能.在5个规模较大的数据集上验证了所提方法的效果,并探究了使用的历史信息长度对方法性能的影响,方法在仅含回归测试用例的数据集上的排序效果,以及方法的执行效率.最后,得到如下结论:(1)与现有方法相比,指针排序方法能够随着软件版本的演化调整排序策略,在持续集成环境下有效地提升测试序列的检错能力.(2)指针排序方法对输入的历史信息长度有较好的鲁棒性,少量的历史信息即可使其达到最优效果.(3)指针排序方法能够很好地处理回归测试用例和新增测试用例.(4)指针排序方法的时间开销不大,结合其更好、更稳定的排序性能,可以认为指针排序方法是一个非常有竞争力的方法.As software delivery increasingly emphasizes speed and reliability,continuous integration(CI)has attracted more and more attention these years.Developers continue to integrate working copies into the mainline to realize software evolution.Each integration involves automated tests to verify whether the update introduces faults.However,as the scale of software increases,test suites contain more and more test cases.As software evolves,the coverage and fault detection ability of test cases also change among different CI cycles.As a result,the traditional test case prioritization techniques may be inapplicable.Techniques based on reinforcement learning can adjust prioritization strategies dynamically according to test feedback.But the existing techniques based on reinforcement learning proposed in recent years do not comprehensively consider information in the test suite during prioritization,which limits their effectiveness.This study proposes a new test case prioritization method in CI,called pointer ranking method.The method uses features like history information of test cases as inputs.In each CI cycle,the agent uses the attention mechanism to gain attention to all candidate test cases,and then obtains a prioritization result.After test execution,it obtains the updating direction from the feedback.It constantly adjusts its prioritization strategy in the process“prioritization,test execution,test feedback”and finally achieves satisfied prioritization performance.This study verifies the effectiveness of the proposed method on five large-scale datasets,and explores the impact of history length on method performance.Besides,it explores the model’s effectiveness on datasets which only contain regression test cases and the model’s execution efficiency.Finally,the study comes to the following conclusions.First,compared to existing techniques,pointer ranking method can adjust its strategy along with the evolution of the software,and effectively enhance the fault detection ability of test sequence in CI.Second,poin

关 键 词：持续集成 测试用例排序 强化学习 

分 类 号：TP311[自动化与计算机技术—计算机软件与理论]