Welcome to part two of a three-part series on trend analysis of log event data. Today, we will explore how to perform, using Graylog, a few of the types of trend analysis discussed previously.

BOSTON and TEL AVIV, June 13, 2018 — Logz.io, the leading provider of AI-powered log analytics, releases a specialized Docker Logging Plugin, enabling users to easily ship container logs to Logz.io. The Docker Logging Plugin was created to relieve many of the common issues developers experience when shipping container logs such as complex configuration, accidental pausing of container shipments, and potential loss of data.

This article explores integrating Google Pub/Sub with the world’s most popular open source log analysis platform — the ELK Stack, for deeper analysis and investigation.

The Linux Audit framework is a kernel feature (paired with userspace tools) that can log system calls. For example, opening a file, killing a process or creating a network connection. These audit logs can be used to monitor systems for suspicious activity.

I continue to be intrigued by the evolution of software architectures and their impact on business. In my 20+ year career, I’ve participated in four of these architecture transitions – the shift from client-server to the internet, the rise of 3-tier architectures underpinning rich internet applications, virtualization that upended the dominance of hardware providers, and now the shift to microservices-based architectures based on cloud infrastructure and software automation.

If you’re building a new application from scratch and are responsible for maintaining its availability and performance, you might wonder whether you should be monitoring logs or metrics. For us, it’s a no-brainer that you’ll want both: metrics are fast and efficient for proactively monitoring the health of your system, while logs are essential for helping to troubleshoot the details of the issue itself to find the root cause.

Graphite Metrics are one of the most common metrics formats in application monitoring today. Originally designed in 2006 by Chris Davis at Orbitz and open-sourced in 2008, Graphite itself is a monitoring tool now used by many organizations both large and small.

“Why does the ‘docker logs’ command fail?“, is one of our frequently asked questions. The answer is simple and mentioned in the Docker documentation: “The docker logs command is not available for drivers other than json-file and journald.”

Centralized log collection has become a necessity for many organizations. Much of the data we need to run our operations and secure our environments comes from the logs generated by our devices and applications. Centralizing these logs creates a large repository of data that we can query to enable various types of analysis. The most common types are conditional analysis and trend analysis. They both have their place, but trend analysis is perhaps the more often underutilized source of information.

Production logs can help ensure application security, reveal business insights and find and understand errors, crashes, and exceptions. But as useful as logs are, they’re difficult to manage and hard to keep track of. Making matters worse is that as log data volume grows, so does the difficult task of maintaining and managing them. It’s for this reason that developers, DevOps engineers, and CTOs turn to log management tools.