In Elasticsearch parlance, a document is serialized JSON data. In a typical ELK setup, when you ship a log or metric, it is typically sent along to Logstash which groks, mutates, and otherwise handles the data, as defined by the Logstash configuration. The resulting JSON is indexed in Elasticsearch.

This recipe is similar to the previous rsyslog + Redis + Logstash one, except that we’ll use Kafka as a central buffer and connecting point instead of Redis. You’ll have more of the same advantages.

Starting from Elasticsearch 5.0, you’re able to define pipelines within it that process your data, in the same way you’d normally do it with something like Logstash. We decided to take it for a spin and see how this new functionality (called Ingest) compares with Logstash filters in both performance and functionality. Is it worth sending data directly to Elasticsearch or should we keep Logstash?

Since its first release in 2007, Splunk quickly became one of the leading log management solutions. Its focus on enterprise grade log analysis and security incident and event management (SIEM) made it the de facto choice for organizations generating large volumes of log files and machine data. But over the past decade, the log management landscape has changed drastically.

Grafana v5.3 comes with a new visual query editor for the PostgreSQL datasource. The query editor makes it easier for users to explore time-series data by improving the discoverability of data stored in PostgreSQL. Users can use drop-down menus to formulate their queries with valid selections and macros to express time-series specific functionalities, all without a deep knowledge of the database schema or the SQL language.

Most software and systems generate audit logs. They are a means to examine what activities have occurred on the system and are typically used for diagnostic performance and error correction. System Administrators, network engineers, developers, and help desk personnel all use this data to aid them in their jobs and maintain system stability. Audit logs have also taken on new importance for cybersecurity and are often the basis of forensic analysis, security analysis, and criminal prosecution.

We want you to get started on the right foot when setting up your Web Application Monitoring, so we put together this list of 12 things for you to remember when recording your transactions.

This week the Grafana Labs team was busy prepping for the Grafana v5.3 stable release! Grafana v5.3 adds Stackdriver as a core datasource, a new graphical query editor for Postgres, enhancements to TV and kiosk mode, and a lot more. See the release section for a list of all the new features.