基于DPPO的双智能体协同渗透测试方法研究
网络安全与数据治理
李科,霍朝宾,贺敏超,杨继
华北计算机系统工程研究所
摘要： 基于强化学习的自动化网络渗透测试方法近年来受到广泛关注，但现有研究多局限于中小规模网络环境，难以应对实际网络中主机数量庞大、拓扑复杂的挑战。提出一种基于DPPO(Dualagent Proximal Policy Optimization)的双智能体协同渗透测试框架，通过分工协作机制提升大规模网络环境下的攻击决策效率。该框架包含目标决策智能体(TargetDecider)与攻击执行智能体(AttackExecutor)，分别负责目标节点筛选与具体漏洞利用执行，二者通过共享环境状态进行协同，以协作奖励机制引导，从而实现分工合作。实验在基于CyberBattleSim构建的仿真网络环境中进行，并与传统的DQN、PPO、SAC等智能体方法进行了比较，结果表明DPPO框架攻陷关键目标所需的步骤数更少、波动更小，显示出更高的攻击效率与策略稳定性。同时，其敏感主机攻陷比率维持在较高水平，表明该框架能可靠完成核心渗透任务。在累计奖励方面，DPPO亦表现出持续优化与稳健收敛的趋势。实验结果表明，该框架能够有效适应大规模静态仿真网络中的渗透测试任务。该框架对动态防御和真实网络环境的适用性仍需在后续研究中进一步验证。
中圖分類號：TP393.08文獻標志碼：ADOI:10.19358/j.issn.2097-1788.2026.06.001中文引用格式：李科,霍朝賓,賀敏超，等.基于DPPO的雙智能體協同滲透測試方法研究［J］.網絡安全與數據治理，2026，45（6）：1-8.
英文引用格式：Li Ke,Huo Chaobin,He Minchao,et al.Research on dualagent collaborative penetration testing method based on DPPO［J］.Cyber Security and Data Governance，2026，45（6）：1-8.
英文引用格式：Li Ke,Huo Chaobin,He Minchao,et al.Research on dualagent collaborative penetration testing method based on DPPO［J］.Cyber Security and Data Governance，2026，45（6）：1-8.
Research on dual-agent collaborative penetration testing method based on DPPO
Li Ke,Huo Chaobin,He Minchao,Yang Ji
National Computer System Engineering Research Institute of China
Abstract： The automated network penetration testing method based on reinforcement learning has received widespread attention in recent years.However,the existing research is mostly limited to small and mediumsized network environments,making it difficult to address the challenges posed by the vast number of hosts and complex topology in realworld networks.This study proposes a dualagent collaborative penetration testing framework based on DPPO (Dualagent Proximal Policy Optimization),which enhances the efficiency of attack decisionmaking in largescale network environments through a division of labor and collaboration mechanism.The framework comprises a target decisionmaking agent (TargetDecider) and an attack execution agent (AttackExecutor),responsible for target node selection and specific vulnerability exploitation execution,respectively.The two agents collaborate through shared environmental states,guided by a collaborative reward mechanism,to achieve division of labor and cooperation.Experiments were conducted in a simulated network environment built based on CyberBattleSim,and compared with traditional agent methods such as DQN,PPO,and SAC.The results show that the DPPO framework requires fewer steps and exhibits less fluctuation to compromise key targets,demonstrating higher attack efficiency and strategic stability.At the same time,its sensitive host compromise rate remains at a high level,indicating that the framework can reliably complete core penetration tasks.In terms of cumulative reward,DPPO also shows a trend of continuous optimization and robust convergence.The results indicate that the framework can effectively adapt to penetration testing tasks in largescale static simulated networks,while its applicability to dynamic defense scenarios and real network e
Key words : automated penetration testing; dualagent collaboration; deep reinforcement learning
引言
隨著網絡攻擊手段日趨復雜，傳統靜態防御體系已難以有效應對新型威脅。自動化滲透測試作為主動安全評估的重要手段，能夠通過模擬攻擊行為提前發現系統脆弱性，但其傳統實現模式高度依賴專家經驗，存在成本高、適應性差等問題［1］。近年來，強化學習技術因其在序列決策問題上的優勢，被引入自動化滲透測試領域，通過將網絡環境建模為馬爾可夫決策過程［2］，使智能體能夠自主學習攻擊策略。然而，現有方法在面對大規模復雜網絡時仍面臨顯著挑戰：狀態與動作空間隨網絡規模指數增長導致探索效率低下，以及滲透測試任務中獎勵信號稀疏導致的收斂困難。
針對上述問題，本文提出一種“目標決策—攻擊執行”雙智能體協同訓練框架。該框架將滲透測試任務分解為宏觀目標選擇和微觀攻擊執行兩個層次，分別由目標決策智能體(TargetDecider)和攻擊執行智能體(AttackExecutor)承擔，二者通過狀態共享與協同獎勵機制實現高效分工協作。本文主要貢獻包括：(1)設計了面向大規模網絡的雙智能體分層強化學習架構，有效降低了學習復雜度；(2)提出了與分工機制相匹配的協同訓練方法及分層獎勵函數；(3)在CyberBattleSim仿真環境中驗證了方法的有效性，實驗表明本框架在攻擊效率與穩定性方面優于傳統單智能體方法。
本文詳細內容請下載：
http://www.tom3567.com/resource/share/2000007120
作者信息：
李科,霍朝賓,賀敏超,楊繼
(華北計算機系統工程研究所，北京100083)
此內容為AET網站原創，未經授權禁止轉載。