Establishing a proactive security posture involves a data-driven approach to threat detection, investigation, and response. In the past, this was challenging because there wasn’t a centralized way to collect and analyze security data across sources, but with Amazon Security Lake it is much simpler.

Whether you’re a security company improving and refining your threat intelligence for customers, or you’re investigating security threats within your own environment, there are a few important things you need to know. This blog will cover the tools, frameworks and data types you’ll need to threat hunt in Amazon Security Lake.

Understanding Amazon Security Lake

Amazon Security Lake is based on Amazon S3 cloud object storage, and provides a way to centralize enterprise security data across AWS environments, SaaS providers, on-prem environments, and cloud sources. Doing so helps AWS customers act faster on security data and simplify security data management, even in a complex multi-cloud environment.

Amazon Security Lake uses the Open Cybersecurity Schema Framework (OCSF), an open standard to normalize security data into a common format. Data normalization has been a big challenge for security operations centers in the past, given the wide variety of tools available for cybersecurity detection and response (and their varied data formats). Integration with Amazon Security Lake is simple, which is particularly important if you’d like to use other tools to conduct security analysis on your data, including threat hunting.

Preparing for Threats to Your Business Data

Amazon Security Lake centralizes security data from AWS and other third-party sources into a data lake stored in your AWS S3 account. You can configure Amazon Security Lake for your AWS account using the Getting Started steps in the AWS user guide. If you’d like to threat hunt in Amazon Security Lake, you’ll need to connect your S3 bucket with an external partner that can help you with the data analysis.

For example, integrating Amazon Security Lake with ChaosSearch involves allowing ChaosSearch to index Security Lake S3 files. Doing so will allow you to monitor and analyze your security content at scale, while reducing operational costs of text search and relational analytics.

Many teams analyze data such as VPC Flow Logs to hunt for potential threats on network traffic. Using a VPC Flow Dashboard in ChaosSearch, like the one pictured below, you can monitor network traffic. The underlying VPC Flow Log data coming from Amazon Security Lake. If suspicious patterns in network activity pop up in these dashboards, you can use ChaosSearch to investigate further.

An example of a VPC Flow Logs dashboard in ChaosSearch, which can detect anomalies in network traffic.

Turning Your Security Operations Team into Threat Hunters

A successful threat hunt lies in the clear definition of roles and responsibilities within the security team. This clarity ensures a coordinated effort in identifying and mitigating potential threats. It’s important to equip the team with the necessary skills and knowledge to navigate Amazon Security Lake and associated tools. This skill set enables the team to fully leverage the platform's capabilities to find security events among patterns in your data. The next step is establishing and using a threat hunting framework.

How Mature is Your Threat Detection Process?

While many teams develop frameworks to identify and analyze attacker behavior, few follow through: A threat hunting survey revealed that while 73% of organizations have adopted a defined structure, only 38% actually follow it. For example, established methodologies like the MITRE ATT&CK Framework can tell your organization which cyber threat groups to watch out for, while specific techniques or software programs might be used to target your business, so your framework would describe how to detect and mitigate against the adversarial tactics.

Starting off, security operations teams will organize their searches to look for signs of potential compromises or security incidents. These indicators could include things they've learned about from sources like threat intelligence feeds or other cyber attack data.

As organizations become more experienced, they'll start to refine their Indicators of Compromise (IoCs) by adding relevant information from security information and event management (SIEM) and threat intelligence reports to make their data more accurate. Some will also begin to use machine learning to spot unusual activities, helping them get a clearer picture of what's normal for their environment. The most advanced organizations will engage in hypothesis-driven hunting. They use a mix of unusual activity detection patterns and specific threat indicators as clues to build a deeper understanding of their unique risk factors.

Executing Threat Hunts in Amazon Security Lake

The execution phase is where the theoretical aspects of a threat hunt come to life. Using tools like ChaosSearch for data analysis within Amazon Security Lake, your team can dive deep into the security data. Advanced data analysis capabilities can help you parse through historical log and telemetry data at a far lower cost, enhancing the threat detection process. Having access to long-term data enables you to detect potential attackers lurking in your system, such as advanced persistent threats.

Querying and investigating security data involves a deeper examination of the data to identify anomalies and patterns that may indicate potential security threats. Analyzing your findings is a critical step, as it involves confirming or dismissing the initial hypotheses you formulated during the threat hunting framework development phase.

Searching for cyber threats and engaging in a threat hunt using Amazon Security Lake enables your team to stay ahead of security risks. By creating a focused team, building a solid plan, and choosing the best tools, companies can greatly improve their protection against cyber attacks. Find complex threats so you can stop them in their tracks, making your company's cybersecurity posture stronger!

Pairing Amazon Security Lake alongside tools like ChaosSearch gives companies a way to instantly interrogate their security data without worrying about retention costs or the complexities of moving data in and out of Amazon S3. As cyberattacks increase in volume and sophistication, proactive threat detection will be key to beating threat actors at their own game.