Scalable AI-based Big Data Framework of Real-time Intrusion Detection and Threat Analytics based on Behavior
Main Article Content
Abstract
Exponential growth in the amount of data and cyber threats is creating a challenge to the conventional security systems, which are not in a position to process and analyze large networks of heterogeneous data in real time. This study proposes Scalable Architecture for Big Data-based Attack Detection Applications, a modular and comprehensive framework that aims at facilitating intelligent detecting of cyber-attacks in distributed settings which uses artificial intelligence and deep learning to detect cyber-attacks. Proposed framework enables to process very large volumes of data that are very heterogeneous and they are coming out of all kinds of different sources network traffic, or cloud platforms, or IoT devices, or enterprise logs uses This architecture incorporates the distributed processing technology such as Apache spark and HDFS to allow a scalable intake, transformation, and real-time analysis of data. It has a deep learning-driven detection that is incorporated as part of the processing pipeline that employs neural networks to detect anomalous behaviors and emergent threats. In contrast to traditional rule-based systems, the Scalable Big data based architecture allows an adaptive learning and pattern recognition involving both historical data and streaming data. The framework is tested on benchmark datasets such as CICIDS2017 and UNSW-NB15 and proves to have high accuracies, poor false positives rates and perform efficiently. It offers the capability of proactive monitoring of cybersecurity in a complex environment with a powerful, scalable, and future proof solution courtesy of its big data infrastructure and smart, AI-driven analytics
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
IJCERT Policy:
The published work presented in this paper is licensed under the Creative Commons Attribution 4.0 International (CC BY 4.0) license. This means that the content of this paper can be shared, copied, and redistributed in any medium or format, as long as the original author is properly attributed. Additionally, any derivative works based on this paper must also be licensed under the same terms. This licensing agreement allows for broad dissemination and use of the work while maintaining the author's rights and recognition.
By submitting this paper to IJCERT, the author(s) agree to these licensing terms and confirm that the work is original and does not infringe on any third-party copyright or intellectual property rights.
References
S. Morgan, “Cybercrime to cost the world $10.5 trillion annually by 2025,” Cybersecurity Ventures, 2020.
M. Roesch, “Snort—lightweight intrusion detection for networks,” in Proc. 13th Systems Administration Conf. (LISA), pp. 229–238, 1999.
H. Hu, Y. Wen, T. S. Chua, and X. Li, “Toward scalable systems for big data analytics: A technology tutorial,” IEEE Access, vol. 2, pp. 652–687, 2014.
T. T. Thi, K. Zhang, and M. Li, “XFedHunter: An explainable federated learning framework for advanced persistent threat detection in SDN,” IEEE Transactions on Network and Service Management, vol. 20, no. 3, pp. 2318 2332, 2023.
Latif, M. Ali, and R. Khan, “Securing federated learning with intrusion detection systems: A deep learning perspective,” Journal of Cybersecurity Research, vol. 12, no. 1, pp. 45–58, 2025.
S. Vinayakumar, K. P. Soman, and P. Poornachandran, “Applying convolutional neural network for network intrusion detection,” in Proc. ICACCI, pp. 1222–1228, 2017.
K. Kim, Y. Kim, and H. Kim, “LSTM-based system call language modeling and robust ensemble method for designing host-based intrusion detection systems,” IEEE Access, vol. 7, pp. 162894–162907, 2019.
X. Yuan, C. Li, and X. Li, “DeepDefense: Identifying DDoS attack via deep learning,” in IEEE SMARTCOMP, pp. 1–8, 2017.
R. Vinayakumar, K. P. Soman, P. Poornachandran, and S. Sathya, “Evaluating deep learning approaches to intrusion detection,” in IEEE ICICCS, pp. 141–146, 2018.
Sharafaldin, A. H. Lashkari, and A. A. Ghorbani, “Toward generating a new intrusion detection dataset and 6 K.Prasanthi et al. / Int. J. Comput. Eng. Res. Trends, 12(5),1-7, 2025 intrusion traffic characterization,” in Proc. ICISSP, pp. 108 116, 2018.
H. Hindy, D. Brosset, E. Bayne, et al., “A taxonomy of network threats and the effect of current datasets on intrusion detection systems,” IEEE Access, vol. 8, pp. 104650–104675, 2020.
K. Prasanthi, K. Sandhya Rani, and P. Venkata Krishna, “BACADA: Big Data Architecture for Cyber Security Attack Detection Applications,” African Journal of Biological Sciences, vol. 6, 2024
C. Yin, Y. Zhu, S. Liu, and J. Fei, “An enhanced capsule network for intrusion detection,” IEEE Access, vol. 7, pp. 49699–49710, 2019.
H. Xiao, R. Li, Y. Li, and H. Wang, “Efficient detection of DDoS attacks with ensemble learning,” IEEE Access, vol. 9, pp. 47636–47644, 2021.
S. Shone, V. N. Ngoc, and Q. Phan, “A deep learning approach to network intrusion detection,” IEEE Trans. Emerging Topics in Computational Intelligence, vol. 2, no. 1, pp. 41–50, 2018.
Y. Zhang, L. Wu, F. Guo, and C. Wang, “A deep learning-based framework for network intrusion detection,” IEEE Access, vol. 6, pp. 29085–29092, 2018.
H. Gao, B. Liu, J. Li, and L. Zhang, “An attention based LSTM-CNN hybrid model for anomaly detection in network traffic,” IEEE Access, vol. 9, pp. 106650–106660, 2021.
F. Roy, K. Verma, and P. K. Gupta, “A comprehensive survey on deep learning-based methods for cybersecurity,” ACM Computing Surveys, vol. 55, no. 4, pp. 1–36, 2023.
Y. Chen, M. Zhao, and W. Pan, “EdgeAI-NIDS: A lightweight edge computing-based deep learning approach for intrusion detection,” IEEE Internet of Things Journal, vol. 10, no. 1, pp. 710–721, 2023.
J. L. Hernandez-Ramos, M. Fernandez, and A. Skarmeta, “Federated learning for anomaly detection in industrial control systems: A cybersecurity perspective,” Computers & Security, vol. 125, p. 102958, 2023.
W. Lin, C. Yang, L. Zhang, and C. Xu, “A temporal convolutional network-based approach for intelligent cyber threat detection,” IEEE Transactions on Information Forensics and Security, vol. 16, pp. 2386–2399, 2021.
M. Schmitt, “AI-enabled malware and intrusion detection for smart infrastructures,” Journal of Digital Security, vol. 5, no. 2, pp. 83–96, 2024.