Behaviour-diverse automatic penetration testing:a coverage-based deep reinforcement learning approach  

作　　者：Yizhou YANG Longde CHEN Sha LIU Lanning WANG Haohuan FU Xin LIU Zuoning CHEN 

机构地区：[1]Zhongguancun Laboratory,Beijing 100081,China [2]Zhejiang Lab,Hangzhou 311121,China [3]National Research Centre of Parallel Computer Engineering and Technology,Wuxi 214000,China [4]Faculty of Geographical Science,Beijing Normal University,Beijing 100875,China [5]Department of Earth System Science,Tsinghua University,Beijing 100084,China

出　　处：《Frontiers of Computer Science》2025年第3期15-24,共10页计算机科学前沿(英文版)

基　　金：supported by te Key Research Project of Zhejiang Lab(No.2021PB0AV02)。

摘　　要：Reinforcement Learning(RL)is gaining importance in automating penetration testing as it reduces human effort and increases reliability.Nonetheless,given the rapidly expanding scale of modern network infrastructure,the limited testing scale and monotonous strategies of existing RLbased automated penetration testing methods make them less effective in practical application.In this paper,we present CLAP(Coverage-Based Reinforcement Learning to Automate Penetration Testing),an RL penetration testing agent that provides comprehensive network security assessments with diverse adversary testing behaviours on a massive scale.CLAP employs a novel neural network,namely the coverage mechanism,to address the enormous and growing action spaces in large networks.It also utilizes a Chebyshev decomposition critic to identify various adversary strategies and strike a balance between them.Experimental results across various scenarios demonstrate that CLAP outperforms state-of-the-art methods,by further reducing attack operations by nearly 35%.CLAP also provides enhanced training efficiency and stability and can effectively perform pen-testing over large-scale networks with up to 500 hosts.Additionally,the proposed agent is also able to discover pareto-dominant strategies that are both diverse and effective in achieving multiple objectives.

关 键 词：network security penetration testing reinforcement learning artificial intelligence 

分 类 号：TP393.08[自动化与计算机技术—计算机应用技术]