Main Article Content

Serhii Semenov
Cao Weilin
Liqiang Zhang
Serhii Bulba


The article developed a method for automated penetration testing using deep machine learning technology. The main purpose of the development is to improve the security of computer systems. To achieve this goal, the analysis of existing penetration testing methods was carried out and their main disadvantages were identified. They are mainly related to the subjectivity of assessments in the case of manual testing. In cases of automated testing, most authors confirm the fact that there is no unified effective solution for the procedures used. This contradiction is resolved using intelligent methods of analysis. It is proposed that the developed method be based on deep reinforcement learning technology. To achieve the main goal, a study was carried out of the Shadov system's ability to collect factual data for designing attack trees, as well as the Mulval platform for generating attack trees. A method for forming a matrix of cyber intrusions using the Mulval tool has been developed. The Deep Q - Lerning Network method has been improved for analyzing the cyber intrusion matrix and finding the optimal attack trajectory. In the study, according to the deep reinforcement learning method, the reward scores assigned to each node, according to the CVSS rating, were used. This made it possible to shrink the attack trees and identify an attack with a greater likelihood of occurring. A comparative study of the automated penetration testing method was carried out. The practical possibility of using the developed method to improve the security of a computer system has been revealed.

Article Details

How to Cite
Semenov, S., Weilin, C., Zhang, L., & Bulba, S. (2021). AUTOMATED PENETRATION TESTING METHOD USING DEEP MACHINE LEARNING TECHNOLOGY. Advanced Information Systems, 5(3), 119–127.
Methods of information systems protection
Author Biographies

Serhii Semenov, National Technical University "Kharkiv Polytechnic Institute", Kharkiv, Ukraine

Doctor of Technical Sciences, Professor, Head of Computer Engineering and Programming Department

Cao Weilin, Neijiang Normal University, Neijiang, China

teacher, Department of IT information Centre

Liqiang Zhang, Neijiang Normal University, Neijiang, China

teacher, College of Computer Science

Serhii Bulba, National Technical University "Kharkiv Polytechnic Institute", Kharkiv, Ukraine

Candidate of Technical Sciences, Associate Professor of Computer Engineering and Programming Department


Hoffmann, J. (2011), “The Metric-FF Planning System: Translating ”Ignoring Delete Lists” to Numeric State Variables”, Journal of Artificial Intelligence Research, vol. 20, pp. 291–341.

Obes, J. L., Sarraute, C. and Richarte, G. G. (1999), “Attack planning in the real world”, Cryptography and Security, 2013.

Schneier, B. (1999), “Attack trees - modeling security threats” , Dr.Dobb’s Journal, vol. 24.

Camtepe, S. and Yener, B. (1999), A Formal Method for Attack Modeling and Detection, URL:

McDermott, J.P. (2001), “Attack Net Penetration Testing”, New Security Paradigms, ACM Press, New York, pp. 15–21.

Ou, X., Govindavajhala, S. and Appel, A.W. (2005), “MulVAL: A logic-based network security analyzer”, 14th USENIX Security Symposium, Baltimore, MD, USA, URL: http://www.cis.ksu.Edu/~xou/publications/mulval_sec05.pdf.

Phillips, C. and Swiler, L. A (1998), “Graph-Based System for Network-Vulnerability Analysis”, Proceedings of the New Security Paradigms Workshop, Charlottesville, VA.

Sheyner, O. (2004), Scenario Graphs and Attack Graphs, Ph.D. diss., Carnegie Mellon University, Pittsburgh, PA, USA.

Yousefi, M., Mtetwa, N., Zhang, Y. and Tianfield, H. (2018), “A reinforcement learning approach for attack graph analysis”, 12th IEEE Int. Conf. On Big Data Science And Engineering (TrustCom/BigDataSE), pp. 212–217.

Zhenguo & Beuran, Hu, Razvan & Tan, Yasuo (2020), Automated Penetration Testing Using Deep Reinforcement Learning, pp. 2-10, DOI:

Yousefi, Mehdi & Mtetwa, Nhamoinesu & Zhang, Yan & Tianfield, Hua (2017), “A novel approach for analysis of attack graph”, IEEE, DOI:

Sh. R. Davlatov, P. V. Kuchynski (2020) Extending the basic functionality of MALTEGO based on the canari framework and SHODAN search engine / Journal of the Belarusian state university. Physics. 2020;1:34 – 40

Madelyn, Bacon (2020), CVSS (Common Vulnerability Scoring System), URL:

Riansanti, O., Ihsan, M. and Suhaimi, D. (2017), “Connectivity algorithm with depth first search (DFS) on simple graphs”, Journal of Physics: Conf. Series, Vol. 948, ICE-STEM, Jakarta, Indonesia.