Operations | Monitoring | ITSM | DevOps | Cloud

Data is massively growing. Cleaning it up and getting it where it needs to be is difficult, and so expensive. You can't easily find what you need, when you need it. And that's baaaaaad for business.

On July 17th, 2023, more than 400 Cribl users came together at The Mirage in Las Vegas to celebrate each other and the power of learning at CriblCon. The theme of our conference, “Do Different,” resonated throughout the day, emphasizing our commitment to innovation and highlighting the distinctive approach our customers and employees bring to every aspect of their work.

The increasing demand for Cribl Stream as an internal service is a testament to its effectiveness in improving operations and enhancing security measures. With the rise of ITOps, SecOps, SRE, DevOps, and other teams embracing Cribl Stream, we are excited to offer Cribl Stream Projects, which enables the secure expansion of Stream usage to more users within organizations. This enhances collaboration and provides deeper insights, resulting in a more personalized user experience. With Stream Projects, Cribl is the first product in the industry enabling organizations to allow teams to manage their own data without needing to understand the infrastructure or service being used to collect and route it.

Cribl Edge and Cribl Search provide a powerful and comprehensive observability solution that allows observability engineers to quickly access and analyze logs and metrics from edge nodes without needing to ingest data into other security or analytics tools.

In this customer story, Ryan Sanders, lead security engineer at SpyCloud, shares his experience using Cribl to centralize and store data for account takeover protection and online fraud prevention. Ryan discusses the challenges he faced in managing data across multiple platforms and the solutions Cribl provided. Cribl acts as the Swiss Army knife for observability engineers, empowering them to collect data from various sources and perform custom integrations.

In the second video of our series, we delve into the nuts and bolts of configuring Cribl Search to access the data that we've stored in the S3 bucket. The video guides you step-by-step through the process of configuring the Search S3 dataset provider by using the Stream Data Lake destination as a model for the authentication information. From there, we proceed to walk through the process of creating a Dataset to access the Provider that we've just established. To wrap things up, we demonstrate how to search through the test data that we've previously stored in the S3 bucket.

In the very first video of the series, we delve into the concept of a data loop and why it is beneficial to use Cribl Search and Cribl Stream to optimize the use of a data lake. The video gives a concise overview of Cribl Search and Cribl Stream, and how they work in tandem to create a data loop. We then provide step-by-step instructions on how to configure the Cribl Stream "Amazon S3 Data Lake" Destination to transfer data from Stream to an S3 bucket that has been optimized specifically for Cribl Search's access. Finally, we demonstrate sending sample data to the S3 bucket and present a before-and-after view of the bucket to showcase the impact of the test data.

In the second video of our series, we delve into the nuts and bolts of configuring Cribl Search to access the data that we've stored in the S3 bucket. The video guides you step-by-step through the process of configuring the Search S3 dataset provider by using the Stream Data Lake destination as a model for the authentication information. From there, we proceed to walk through the process of creating a Dataset to access the Provider that we've just established. To wrap things up, we demonstrate how to search through the test data that we've previously stored in the S3 bucket.

The third video of our series focuses on utilizing Cribl Stream to manage data. The presenter takes us through the process of configuring the Cribl Stream in_cribl_http source in tandem with the Cribl Search send operator to collect data. We are able to witness live data results being sent from Search to Stream. Afterward, we demonstrate creating a Route in Stream to direct the incoming data from Search (via the in_cribl_http) Source to the Data Lake by using the Amazon S3 Data Lake Destination. This step employs a passthrupipeline to ensure that the data is not altered in transit.

The final section of our video series showcases how to put the data loop to use with a real-world dataset. We utilize the public domain “Boss of the SOC v3” dataset, which is readily available on GitHub. First, we employ Cribl Search to sift through and explore the BOTSv3 data that is stored in an S3 bucket to locate some specific data.