A Brief Study: Anomaly Detection of Network Traffic Using Isolation Forest in a Big Data Platform Using the Kafka Pipeline

Authors

  • Sikha S. Bagui Department of Computer Science, The University of West Florida, Pensacola, Florida, USA
  • Ben Yanick Department of Computer Science, The University of West Florida, Pensacola, Florida, USA
  • Dustin Mink Department of Cybersecurity, The University of West Florida, Pensacola, Florida, USA
  • Subhash C. Bagui Department of Mathematics and Statistics, The University of West Florida, Pensacola, Florida, USA https://orcid.org/0000-0001-6140-5384

DOI:

https://doi.org/10.14738/tecs.1406.12653

Keywords:

Anomaly detection, Streaming Data, Zeek Data, UWF-ZeekData22, Kakfa, Big Data

Abstract

In today's digital world, data is continuously being produced at an extremely fast rate. Data that is generated in a continuous process is referred to as streaming data. The challenge in working with streaming data is that it requires an environment that can process and store massive amounts of data efficiently. This work addresses the challenges of using streaming data using Kakfa, a distributed streaming platform allowing real-time data processing in the Big Data environment. Of course streaming data can also contain anomalies. Hence, in this work, a machine learning algorithm, Isolation Forest, is integrated with Kafka to detect anomalies in streaming data.

References

[1]. https://kafka.apache.org/

[2]. A. Carnero, C. Martín, G. Jeon, and M. Díaz, “Online learning and continuous model upgrading with data streams through the Kafka-ML framework,” Future Generation Computer Systems, vol. 160, pp. 251–263, 2024.

[3]. R. Sharma and M. Grover, “Enhancing Cybersecurity with Machine Learning: Evaluating the Efficacy of Isolation Forests and Autoencoders in Anomaly Detection,” in 2024 International Conference on Circuit Power and Computing Technologies (ICCPCT), pp. 1017–1021, 2024.

[4]. Amazon Web Services, “What is Pub/Sub Messaging?,” [Online]. Available: https://aws.amazon.com/what-is/pub-sub-messaging/. [Accessed: Aug. 22, 2024].

[5]. F. T. Liu, K. M. Ting, and Z.-H. Zhou, “Isolation-based Anomaly Detection,” in ACM Transactions on Knowledge Discovery from Data, vol. V, no. N.

[6]. M. S. Kareem and L. A. Muhammed, "Anomaly Detection in Streaming Data using Isolation Forest," in 2024 Seventh International Women in Data Science Conference at Prince Sultan University (WiDS PSU), pp. 223–228, 2024.

[7]. D. Xu, Y. Wang, Y. Meng, and Z. Zhang, "An Improved Data Anomaly Detection Method Based on Isolation Forest," in 2017 10th International Symposium on Computational Intelligence and Design, pp. 287–291, 2017.

[8]. C. Melquiades and F. B. de Lima Neto, “Isolation Forest-based Semi-supervised Anomaly Detection of Multiple Classes,” in 2022 17th Iberian Conference on Information Systems and Technologies (CISTI), Madrid, Spain, 2022.

[9]. Moomtaheen, F., Bagui, S.S., Bagui, S.C. and Mink, D. (2024). Extended Isolation Forest for Intrusion Detection in Zeek Data. Information, 15, 404. https://doi.org/10.3390/info15070404

[10]. Bagui, S.S., Mink, D., Bagui, S.C., Ghosh, T., Plenkers, R., McElroy, T., Dulaney, S. & Shabanali, S. (2023). Introducing UWF-ZeekData22: A Comprehensive Network Traffic Dataset Based on the MITRE ATT&CK Framework. Data, 2023, 8, 18. https://doi.org/10.3390/ data8010018.

[11]. Datasets.uwf.edu.edu

[12]. https://attack.mitre.org/

Downloads

Published

2026-11-20

How to Cite

Bagui, S. S., Yanick, B., Mink, D., & Bagui, S. C. (2026). A Brief Study: Anomaly Detection of Network Traffic Using Isolation Forest in a Big Data Platform Using the Kafka Pipeline. Transactions on Engineering and Computing Sciences, 14(06), 01–08. https://doi.org/10.14738/tecs.1406.12653