Metaheuristic Optimization Algorithms for Cybersecurity: A Multi-Domain Experimental Study on Intrusion Detection, Cryptographic Key Optimization, and Malware Classification

Authors

https://doi.org/10.48313/maa.v2i3.52

Abstract

The escalating sophistication of cyber threats demands adaptive, intelligent security mechanisms that transcend the limitations of conventional rule-based and signature-driven approaches. This paper presents a comprehensive metaheuristic-based security optimization framework that addresses three critical cybersecurity problems simultaneously: 1) Network Intrusion Detection System (NIDS) feature selection and classifier optimization using Genetic Algorithm (GA), Particle Swarm Optimization (PSO), Grey Wolf Optimizer (GWO), Whale Optimization Algorithm (WOA), and Harris Hawks Optimization (HHO), 2) cryptographic Substitution-Box (S-box) generation and key scheduling optimization for symmetric ciphers, and 3) malware classification via metaheuristic-optimized ensemble learning. Extensive experiments were conducted on four benchmark datasets — NSL-KDD, UNSW-NB15, and CICIDS-2017 for intrusion detection, and Malimg for malware classification — under rigorous experimental conditions including 10-fold cross-validation and 30 independent runs per configuration. In the intrusion detection domain, GWO-Random Forest (RF) achieved the highest accuracy of 99.41% on NSL-KDD with a 78.0% feature reduction, selecting only 9 of 41 original features. For cryptographic S-box generation, HHO produced S-boxes with an average nonlinearity score of 112 (maximum possible: 120), approaching the quality of the Advanced Encryption Standard (AES) standard S-box while exhibiting a differential uniformity of 6. In the malware classification domain, PSO-optimized ensemble classifiers attained an F1-score of 98.76% on the Malimg dataset. Statistical significance was confirmed via Friedman test (χ² = 18.93, p < 0.001) and pairwise Wilcoxon signed-rank tests. This study provides the first comprehensive multi-domain comparison of modern metaheuristic algorithms across the cybersecurity spectrum, offering practitioners evidence-based guidance for algorithm selection in diverse security applications.

Keywords:

Metaheuristic algorithms, Cybersecurity, Intrusion detection, Feature selection, Cryptography

References

  1. [1] Morgan, S. (2020). Cybercrime to cost the world $10.5 trillion annually by 2025. Cybercrime magazine, 13(11), 2020. https://cybersecurityventures.com/hackerpocalypse-cybercrime-report-2016/
  2. [2] Roesch, M. (1999). Snort: Lightweight intrusion detection for networks. Lisa (Vol. 99, No. 1, pp. 229-238). USENIX Association. https://www.usenix.org/legacy/event/lisa99/full_papers/roesch/roesch.pdf
  3. [3] Ambusaidi, M. A., He, X., Nanda, P., & Tan, Z. (2016). Building an intrusion detection system using a filter-based feature selection algorithm. IEEE transactions on computers, 65(10), 2986–2998. https://doi.org/10.1109/TC.2016.2519914
  4. [4] Farah, M. A. Ben, Farah, A., & Farah, T. (2020). An image encryption scheme based on a new hybrid chaotic map and optimized substitution box. Nonlinear dynamics, 99(4), 3041–3064. https://doi.org/10.1007/s11071-019-05413-8
  5. [5] Ucci, D., Aniello, L., & Baldoni, R. (2019). Survey of machine learning techniques for malware analysis. Computers & security, 81, 123–147. https://doi.org/10.1016/j.cose.2018.11.001
  6. [6] Heidari, A. A., Mirjalili, S., Faris, H., Aljarah, I., Mafarja, M., & Chen, H. (2019). Harris Hawks optimization: Algorithm and applications. Future generation computer systems, 97, 849–872. https://doi.org/10.1016/j.future.2019.02.028
  7. [7] Mirjalili, S., Mirjalili, S. M., & Lewis, A. (2014). Grey wolf optimizer. Advances in engineering software, 69, 46–61. https://doi.org/10.1016/j.advengsoft.2013.12.007
  8. [8] Tavallaee, M., Bagheri, E., Lu, W., & Ghorbani, A. A. (2009). A detailed analysis of the kdd cup 99 data set. 2009 IEEE symposium on computational intelligence for security and defense applications (pp. 1–6). IEEE. https://doi.org/10.1109/CISDA.2009.5356528
  9. [9] Moustafa, N., & Slay, J. (2015). UNSW-nb15: A comprehensive data set for network intrusion detection systems (UNSW-NB15 network data set). 2015 military communications and information systems conference (MILCIS) (pp. 1–6). IEEE. https://doi.org/10.1109/MilCIS.2015.7348942
  10. [10] Khammassi, C., & Krichen, S. (2017). A GA-LR wrapper approach for feature selection in network intrusion detection. Computers & security, 70, 255–277. https://doi.org/10.1016/j.cose.2017.06.005
  11. [11] Elmasry, W., Akbulut, A., & Zaim, A. H. (2020). Evolving deep learning architectures for network intrusion detection using a double PSO metaheuristic. Computer networks, 168, 107042. https://doi.org/10.1016/j.comnet.2019.107042
  12. [12] Varzaneh, Z. A., & Hosseini, S. (2024). An improved equilibrium optimization algorithm for feature selection problem in network intrusion detection. Scientific reports, 14(1), 18696. https://www.nature.com/articles/s41598-024-67488-7
  13. [13] Almomani, A., Alweshah, M., Al Khalayleh, S., Al-Refai, M., & Qashi, R. (2019). Metaheuristic algorithms-based feature selection approach for intrusion detection. In Machine learning for computer and cyber security (pp. 184–208). CRC Press. https://doi.org/10.1201/9780429504044-8
  14. [14] Kasongo, S. M., & Sun, Y. (2020). A deep learning method with wrapper based feature extraction for wireless intrusion detection system. Computers & security, 92, 101752. https://doi.org/10.1016/j.cose.2020.101752
  15. [15] Koryshev, N., Hodashinsky, I., & Shelupanov, A. (2021). Building a fuzzy classifier based on whale optimization algorithm to detect network intrusions. Symmetry, 13(7), 1211. https://doi.org/10.3390/sym13071211
  16. [16] Thaher, T., Heidari, A. A., Mafarja, M., Dong, J. S., & Mirjalili, S. (2019). Binary Harris Hawks optimizer for high-dimensional, low sample size feature selection. In Evolutionary machine learning techniques: Algorithms and applications (pp. 251–272). Springer. https://doi.org/10.1007/978-981-32-9990-0_12
  17. [17] Daemen, J., & Rijmen, V. (2002). The design of Rijndael. Springer. https://doi.org/10.1007/978-3-662-60769-5
  18. [18] Alzaidi, A. A., Ahmad, M., Doja, M. N., Al Solami, E., & Beg, M. M. S. (2018). A new 1D chaotic map and $beta $-hill climbing for generating substitution-boxes. IEEE access, 6, 55405–55418. https://doi.org/10.1109/ACCESS.2018.2871557
  19. [19] Kuznetsov, O., Poluyanenko, N., Frontoni, E., Arnesano, M., & Smirnov, O. (2024). Evolutionary approach to s-box generation: Optimizing nonlinear substitutions in symmetric ciphers. https://arxiv.org/abs/2407.03510
  20. [20] Ali, R. S., Hasoun, R. K., Tayyeh, H. K., & Mohammed, M. Q. (2024). A comprehensive review on s-box generation methods. AIP conference proceedings (Vol. 3207, p. 20001). AIP Publishing LLC. https://doi.org/10.1063/5.0234892
  21. [21] Ahmad, M., Khaja, I. A., Baz, A., Alhakami, H., & Alhakami, W. (2020). Particle swarm optimization based highly nonlinear substitution-boxes generation for security applications. IEEE access, 8, 116132–116147. https://doi.org/10.1109/ACCESS.2020.3004449
  22. [22] Nataraj, L., Karthikeyan, S., Jacob, G., & Manjunath, B. S. (2011). Malware images: Visualization and automatic classification. Proceedings of the 8th international symposium on visualization for cyber security (pp. 1-7). Association for Computing Machinery (ACM). https://doi.org/10.1145/2016904.2016908
  23. [23] Panda, B., Bisoyi, S. S., & Panigrahy, S. (2023). An ensemble approach for imbalanced multiclass malware classification using 1D-CNN. PeerJ computer science, 9, e1677. https://doi.org/10.7717/peerj-cs.1677
  24. [24] Moujoud, L., Ayache, M., & Belmekki, A. (2024). Ensemble learning for malware detection. International conference on smart applications and data analysis (pp. 233–245). Cham: Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-77040-1_17
  25. [25] Holland, J. H. (1992). Adaptation in natural and artificial systems: An introductory analysis with applications to biology, control, and artificial intelligence. MIT Press. https://mitpress.mit.edu/9780262581110/adaptation-in-natural-and-artificial-systems/
  26. [26] Eberhart, R., & Kennedy, J. (1995). Particle swarm optimization. Proceedings of the IEEE international conference on neural networks (Vol. 4, pp. 1942–1948). IEEE. https://doi.org/10.1109/ICNN.1995.488968
  27. [27] Mirjalili, S., & Lewis, A. (2016). The whale optimization algorithm. Advances in engineering software, 95, 51–67. https://doi.org/10.1016/j.advengsoft.2016.01.008
  28. [28] Sharafaldin, I., Lashkari, A. H., & Ghorbani, A. A. (2018). Toward generating a new intrusion detection dataset and intrusion traffic characterization. ICISSp, 1(2018), 108–116. https://doi.org/10.5220/0006639801080116

Downloads

Published

2025-06-18

Issue

Vol. 2 No. 3 (2025): Metaheuristic Algorithms with Applications

Section

Articles

How to Cite

Ekbatanifard, A. (2025). Metaheuristic Optimization Algorithms for Cybersecurity: A Multi-Domain Experimental Study on Intrusion Detection, Cryptographic Key Optimization, and Malware Classification. Metaheuristic Algorithms With Applications, 2(3), 309–323. https://doi.org/10.48313/maa.v2i3.52

Similar Articles

11-17 of 17

You may also start an advanced similarity search for this article.