A large, global Data Management and AI software company with over 5,000 customers across more than 100 countries had seen unprecedented growth (more than 30% year over year) in telemetry data from their multi-cloud infrastructure being sent to the Elasticsearch Observability and SIEM Platform. The growth of this data contributed to a multi-million dollar price tag for Elasticsearch.

A Global 1000 E-commerce company struggled with the rapid growth in telemetry data that their security team analyzes with Splunk, Grafana, and other Observability tools in the cloud. Specifically, the increase in VPC Flow log and Firewall log volumes caused a spike in Splunk costs on certain data sets and triggered daily indexing limit overage fees. As this deluge of data began piling up in block storage within their Splunk index, the team saw corresponding spikes in storage costs.

Amazon Web Services (AWS) VPC Flow Logs is a feature designed to capture and provide information about the IP traffic that flows to and from network interfaces within your Virtual Private Cloud (VPC). This data can be published to various destinations, including AWS CloudWatch Logs, AWS S3, or AWS Kinesis Data Firehose. Flow logs serve several important purposes, such as diagnosing security group rule issues, monitoring incoming and outgoing traffic, and determining traffic directions.

A firewall is a network security device or software that is used to monitor and control incoming and outgoing network traffic based on predetermined security rules. Firewall Logs contain valuable information about network and security events. These logs are essential for security and infrastructure monitoring for enterprises. While this data is critical to securing enterprise networks, they are also one of the most voluminous data types security teams use to monitor and secure their networks.