When it comes to centralizing logs to Elasticsearch, the first log shipper that comes to mind is Logstash. People hear about it even if it’s not clear what it does: – Bob: I’m looking to aggregate logs – Alice: you mean… like… Logstash? When you get into it, you realize centralizing logs often implies a bunch of things, and Logstash isn’t the only log shipper that fits the bill.

In an earlier blog post, Log monitoring and unstructured log data, moving beyond tail -f, we talked about collecting and working with unstructured log data. We learned that it’s very easy to add data to the Elastic Stack. So far the only parsing we did was to extract the timestamp from this data, so older data gets backfilled correctly. We also talked about searching this unstructured data toward the end of the blog.

With the growing adoption of remote and distributed application development including micro-services, cloud-native applications, serverless, and more, it is becoming challenging more than ever before for developers to troubleshoot issues within a reasonable time, and that is a bottleneck. That in a sense contradicts the objectives of Agile and DevOps through fast feedback loops, continuous delivery, quick MTTR (mean time to resolution of defects), etc.

Grafana Loki is designed to be cost effective and easy to operate for DevOps and SRE teams, but running queries in Loki can be confusing for those who are new to it. Loki is a horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus. It doesn’t index the content of the logs, but rather a set of labels for each log stream.

We Just celebrated 10 year birthday to Prometheus last month. Prometheus was the second project to join the Cloud Native Computing Foundation after Kubernetes in 2016, and has quickly become the de-facto way to monitor Kubernetes workloads. The plug-and-play experience, just putting Prometheus server and starting to see metrics flowing in tagged with Kubernetes labels, was a compelling offer.

When working with log messages, it’s critical that the timestamp of the log message is accurate. Incorrect timestamps can cause problems when trying to find log messages at a specific date/time or may cause alerts to not function properly. A common cause of incorrect timestamps for log messages is a mismatch of time zones between the log source (device sending the log) and log destination (device receiving the log, such as Graylog).

Monitoring the performance of an application is not a strange concept to most developers. At one point or another, we’ve all had to do some performance debugging of our own. Usually, it happens when there’s a big issue affecting the user’s experience or cost implications. Only then do we make time to look at how the app performs in different scenarios.