I have always found data collection to be a fascinating area of work at Sumo Logic. Collecting data is a critical first step for all the solutions we develop for our customers. After all, to observe the health and performance of your applications, you must first collect all relevant data. It's also an area that has seen some significant activities by the open-source community over the years, which is completely changing the landscape of observability as we know it.

In today’s highly interconnected worlds, CISOs face a dual challenge: protecting data and reporting to the Board of Directors. Log management has long been a tool in the CISO’s back pocket, helping gain insight into potential security issues. However, the rise of cloud-based infrastructures changes this, making log management increasingly difficult.

Consider the following question: Why do most teams face pressure to rethink traditional logging and observability approaches? Asking this question to most engineers would likely result in answers centered on the challenges posed by microservices apps. Because microservices are more complex than monoliths and involve more moving parts, they require more sophisticated, granular log collection, correlation, and analysis.

We all know the importance of cataloging, organizing, and breaking down the data in your logs. That process, parsing, makes information easier to find and simplifies subsequent analysis. Now, with Logz.io’s upgraded self-parsing tool, custom parsing rules, and log organization is easier than ever. What’s important is parsing that data out correctly. The better parsed, the easier to query.

Tucker Callaway is the CEO of LogDNA. He has more than 20 years of experience in enterprise software with an emphasis on developer and DevOps tools. Tucker drives innovation, experimentation, and a culture of collaboration at LogDNA, three ingredients that are essential for the type of growth that we've experienced over the last few years.

Which data sources do DevOps teams need in order to achieve observability? At a high level, that’s an easy question to answer. Concepts like the “three pillars of observability”—logs, metrics, and traces—may come to mind. Or, you may think in terms of techniques like the RED Method or Google’s Golden Signals, which are other popular frameworks for defining which types of data teams should collect for monitoring and observability purposes.

Logging is a fundamental part of software development. While an app is being developed, we rely on logging to confirm our inputs and outputs match our expectations. In production, logging can be an invaluable resource for tracking down bugs or measuring how users interact with the app. We can also consider logs as a sort of time-series value, where a timestamp is associated with a user’s specific action. These logs can be structured, gathered, and analyzed to provide teams with more information.

Many companies in Asia Pacific (APAC) were caught in a digital tailspin when Covid-19 hit, sacrificing security practices in their hurry to adjust to the new reality of remote work. Two years on, hybrid work is still the norm as the pandemic continues and seems to be a new way of life moving forward. Catalyzed by the coronavirus, firms big and small are now adopting cloud technologies as we tread deeper into a new data age.

We are happy to provide an open-source Terraform module enabling automatic delivery of CloudTrail logs from S3 to a Kinesis Data Stream.