Volume 18, Number 3

A Systematic Review of Obfuscated Malware Detection: From Traditional Analysis to Deep Learning

  Authors

Mohammed bin Shamlan¹, Mohammed Fadhl Abdullah¹, Khaled Hassan Balhaf¹, Ahmed Saleh Khaled¹ and Makarem Mohamed Bamatraf², ¹University of Science and Technology, Yemen, ²Hadhramaut University, Yemen

  Abstract

Obfuscation has been increasingly difficult in the subject of cybersecurity, since malware developers use it to change code appearance without changing its malicious behavior. As a result, signature-based and basic heuristic detection systems are easily bypassed by these techniques. This article reviews recent and ongoing research in the analysis and detection of obfuscated malware, giving special attention to methods that were recently developed to address this problem. The reviewed methods are divided into five major classes: static analysis, dynamic analysis, hybrid analysis, machine learning, and deep learning. thirty-six recent research papers from 2018 to 2025 are analyzed, with a detailed summary of each, including merits and demerits. The review is intended to generate a broad picture of the research field, point out strengths and weaknesses in each category, and identify the way forward, especially for the area of hybrid and deep learning-oriented memory analysis.

  Keywords

Obfuscated Malware, Static and Dynamic Analysis, Malware Detection, Memory Analysis, Cybersecurity, Explainable Artificial Intelligence.