Time Table

This research aims to achieve practical-level penetration testing automation, including reconnaissance, vulnerability analysis, exploitation, and report generation. Our strength lies in being part of a security business company with numerous in-house penetration testers, allowing us to collaborate with a team possessing on-site expertise to build evaluation systems and conduct practical verification. Building upon the knowledge of semi-autonomous agents gained from last year’s joint research with Meiji University’s Takagi Lab (arXiv:2502.15506v1), we have evolved this into a multi-agent system built with modern Agentic AI.

This presentation aims to show the audience the extent to which AI can automate practical penetration testing as of 2025. In the opening of the 45-minute session, we will provide an overview of the trends in automated penetration testing research from the advent of LLMs in 2023 to the present, tailored for beginners. In the main part, we will report on the performance evaluation of the constructed system compared to conventional automation tools and manual diagnostics. The evaluation will use HackTheBox challenges in addition to environments simulating real-world scenarios, quantitatively comparing reproducibility, coverage, and time/cost reduction effects. We will also touch upon the possibilities of automation using local LLMs.

Through this presentation, participants will understand the effects and challenges of introducing AI agents and gain specific guidelines for applying them to their own company’s security diagnostics.