A Brief Study: Anomaly Detection of Network Traffic Using Isolation Forest in a Big Data Platform Using the Kafka Pipeline
DOI:
https://doi.org/10.14738/tecs.1406.12653Keywords:
Anomaly detection, Streaming Data, Zeek Data, UWF-ZeekData22, Kakfa, Big DataAbstract
In today's digital world, data is continuously being produced at an extremely fast rate. Data that is generated in a continuous process is referred to as streaming data. The challenge in working with streaming data is that it requires an environment that can process and store massive amounts of data efficiently. This work addresses the challenges of using streaming data using Kakfa, a distributed streaming platform allowing real-time data processing in the Big Data environment. Of course streaming data can also contain anomalies. Hence, in this work, a machine learning algorithm, Isolation Forest, is integrated with Kafka to detect anomalies in streaming data.
References
[1]. https://kafka.apache.org/
[2]. A. Carnero, C. Martín, G. Jeon, and M. Díaz, “Online learning and continuous model upgrading with data streams through the Kafka-ML framework,” Future Generation Computer Systems, vol. 160, pp. 251–263, 2024.
[3]. R. Sharma and M. Grover, “Enhancing Cybersecurity with Machine Learning: Evaluating the Efficacy of Isolation Forests and Autoencoders in Anomaly Detection,” in 2024 International Conference on Circuit Power and Computing Technologies (ICCPCT), pp. 1017–1021, 2024.
[4]. Amazon Web Services, “What is Pub/Sub Messaging?,” [Online]. Available: https://aws.amazon.com/what-is/pub-sub-messaging/. [Accessed: Aug. 22, 2024].
[5]. F. T. Liu, K. M. Ting, and Z.-H. Zhou, “Isolation-based Anomaly Detection,” in ACM Transactions on Knowledge Discovery from Data, vol. V, no. N.
[6]. M. S. Kareem and L. A. Muhammed, "Anomaly Detection in Streaming Data using Isolation Forest," in 2024 Seventh International Women in Data Science Conference at Prince Sultan University (WiDS PSU), pp. 223–228, 2024.
[7]. D. Xu, Y. Wang, Y. Meng, and Z. Zhang, "An Improved Data Anomaly Detection Method Based on Isolation Forest," in 2017 10th International Symposium on Computational Intelligence and Design, pp. 287–291, 2017.
[8]. C. Melquiades and F. B. de Lima Neto, “Isolation Forest-based Semi-supervised Anomaly Detection of Multiple Classes,” in 2022 17th Iberian Conference on Information Systems and Technologies (CISTI), Madrid, Spain, 2022.
[9]. Moomtaheen, F., Bagui, S.S., Bagui, S.C. and Mink, D. (2024). Extended Isolation Forest for Intrusion Detection in Zeek Data. Information, 15, 404. https://doi.org/10.3390/info15070404
[10]. Bagui, S.S., Mink, D., Bagui, S.C., Ghosh, T., Plenkers, R., McElroy, T., Dulaney, S. & Shabanali, S. (2023). Introducing UWF-ZeekData22: A Comprehensive Network Traffic Dataset Based on the MITRE ATT&CK Framework. Data, 2023, 8, 18. https://doi.org/10.3390/ data8010018.
[11]. Datasets.uwf.edu.edu
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Sikha S. Bagui, Ben Yanick, Dustin Mink, Subhash C. Bagui

This work is licensed under a Creative Commons Attribution 4.0 International License.