ML-Based Real-Time Anomaly Identification in AWS CloudTrail Logs
Manikanth Sarisa, Mohit Surender Reddy, Siddharth Konkimalla
DOI: 10.63665/ijmlaidse-y1f2a004
View / Download Full Article (PDF)Abstract
Therefore, with an increasing use of AWS Cloud infrastructure, the need to monitor the events related to security in real-time becomes greater and urgent. AWS CloudTrail provides a very detailed record of every API call and activity that occurs within the account. However, the number of log types for static rule-based analysis and human review to identify potential threats in a timely manner is too large. The work presented here proposes a machine learning-based framework for the real-time detection of anomalies in AWS CloudTrail data. We used AWS built-in services to create a scalable pipeline for log intake, model prediction, and feature engineering tasks. We apply several unsupervised and semi-supervised learning approaches, such as Autoencoders and Isolation Forest, to detect anomalous behavior indicative of a security vulnerability. Given the scalability and accuracy of finding issues in real-time, it would appear that our approach may be a good fit for cloud security monitoring. We also discuss ways to do better and considerations while doing so.
Keywords
Cloud Security, AWS CloudTrail, Real-Time Anomaly Detection, Machine Learning, Behavioral Analytics, Unsupervised Learning, Cyber Threat Detection, Log Analysis, AWS Lambda, Data Streaming.
References
[1] Chandola, V., Banerjee, A., & Kumar, V. (2009). Anomaly Detection: A Survey. ACM Computing Surveys.
[2] Ahmed, M., Mahmood, A. N., & Hu, J. (2016). A Survey of Network Anomaly Detection Techniques. Journal of Network and Computer Applications.
[3] Liu, F. T., Ting, K. M., & Zhou, Z. H. (2008). Isolation Forest. IEEE ICDM.
[4] Hawkins, S., He, H., Williams, G., & Baxter, R. (2002). Outlier Detection Using Replicator Neural Networks. ICDM.
[5] Kravchik, M., & Shabtai, A. (2018). Detecting Cyber Attacks in Industrial Control Systems Using CNN. CPS Security Workshop.
[6] Amazon Web Services. (2023). AWS Kinesis Data Streams Developer Guide.
[7] Han, J., Kamber, M., & Pei, J. (2011). Data Mining: Concepts and Techniques. Elsevier.
[8] Ghosh, S., & Reilly, D. (1994). Credit Card Fraud Detection with a Neural Network.
[9] Harshaw, C., et al. (2020). Autoencoder-Based Anomaly Detection in Cloud Application Logs. IEEE Big Data.
[10] Kim, Y., Lee, S., & Lee, H. (2020). AWS CloudTrail Log Analysis for Intrusion Detection.
[11] Mitrokotsa, A., & Douligeris, C. (2008). Intrusion Detection for Wireless Networks Using Hybrid GA-SVM.
[12] Saxe, J., & Berlin, K. (2015). Deep Neural Network Based Malware Detection.
[13] Kwon, D., et al. (2018). Survey of AI Techniques for Cybersecurity.
[14] Gangineni, V. N., et al. (2022). Efficient Framework for Forecasting Auto Insurance Claims Utilizing Machine Learning.
[15] Tyagadurgam, M. S. V., et al. (2022). Intelligent Cybersecurity Intrusion Identify Framework Using ML in Cloud.
[16] Chalasani, R., et al. (2022). Machine Learning-Based Anomaly Detection in Cybersecurity Network Traffic.
[17] Bhumireddy, J. R., et al. (2022). Big Data-Driven Time Series Forecasting for Financial Market Prediction.
[18] Vangala, S. R., et al. (2022). AI Algorithms for Risk Prediction in Life Insurance.
[19] Chundru, S. K., et al. (2022). ML Approaches for Intrusion Identification of DDoS Attacks.
[20] Polu, A. R., et al. Blockchain Technology as a Tool for Cybersecurity.
[21] Nandiraju, S. K. K., et al. (2022). AI-Based Predictive Models for Diagnosis of Alzheimer’s Disease.
[22] Gangineni, V. N., et al. (2023). AI-Enabled Big Data Analytics for Climate Change Prediction.
[23] Pabbineedi, S., et al. (2023). Deep Learning Algorithms with Big Data for Predictive Maintenance.
[24] Bhumireddy, J. R., et al. (2023). Predictive Models for Early Detection of Chronic Diseases.
[25] Polam, R. M. (2023). Predictive ML Strategies for Clinical Diagnosis.
[26] Bhumireddy, J. R. (2023). Hybrid Approach for Melanoma Classification.
[27] Gupta, A. K., et al. (2024). Deep Learning Models for Intrusion Detection Systems.
[28] Narra, B., et al. (2024). AI in Software Development: Trends and Future Prospects.
[29] Achuthananda, R. P., et al. (2024). Machine Learning for Personalized Movie Recommendations.
[30] Polu, A. R., et al. Analytics in Insurance Risk Management.
[31] Gangineni, V. N., et al. (2024). AI-Powered Cybersecurity Risk Scoring for Financial Institutions.
[32] Vangala, S. R., et al. (2024). Machine Learning Framework for Predicting Student Outcomes.