Operations | Monitoring | ITSM | DevOps | Cloud

July 2020

How to maximize span ingestion while limiting writes per second to a Scylla backend with Jaeger tracing

Jaeger primarily supports two backends: Cassandra and Elasticsearch. Here at Grafana Labs we use Scylla, an open source Cassandra-compatible backend. In this post we’ll look at how we run Scylla at scale and share some techniques to reduce load while ingesting even more spans. We’ll also share some internal metrics about Jaeger load and Scylla backend performance. Special thanks to the Scylla team for spending some time with us to talk about performance and configuration!

How blocks storage in Cortex reduces operational complexity for running Prometheus at massive scale

Cortex is a long-term distributed storage for Prometheus. It provides horizontal scalability, high availability, multi-tenancy and blazing fast query performances when querying high cardinality series or large time ranges. Today, there are massive Cortex clusters storing tens to hundreds of millions of active series with a 99.5 percentile query latency below 2.5s.

How we're using 'dogfooding' to serve up better alerting for Grafana Cloud

At Grafana Labs, we’re big fans of putting ourselves in the shoes of our customers. So when it comes to building a product, dogfooding is a term we throw around constantly. In short, what it means is that we actually use the products we create throughout their entire life cycle. And I really mean the whole life cycle.

What recent optimizations in the Prometheus storage engine, TSDB, will enable in the future

At the recent PromCon Online, I gave a review of developments in the space of the Prometheus storage engine, TSDB. In this blog post I am going to recap a bit of the talk and add more insights into what these developments will enable us to do in the future. While the talk contained some of the near-future features, I will be diving even further ahead. You can watch the talk here.

Introducing the new and improved New Relic plugin for Grafana

It’s been a while, but the Kelly and Regis of Grafana Labs (a.k.a. Christine and Eldin of Solutions Engineering) are back to report on another Grafana Enterprise plugin: New Relic! The latest version of this plugin will be just one of the many topics we’ll cover during today’s webinar, All about Grafana plugins: Visualizing disparate data sources in one place. We’ll be hosting a great conversation around plugin updates, use cases, and the best way to make coffee.

Loki tutorial: How to send logs from EKS with Promtail to get full visibility in Grafana

Amazon Elastic Kubernetes Service (Amazon EKS) is the fully managed Kubernetes service on AWS. If you’re using it and wondering how to query all your logs in one place, Loki is the answer. With this tutorial, you’ll learn how to set up Promtail on EKS to get full visibility into your cluster logs while using Grafana. We’ll start by forwarding pods logs then nodes services and finally Kubernetes events.

How the Cortex and Thanos projects collaborate to make scaling Prometheus better for all

Cortex and Thanos are two brilliant solutions to scale out Prometheus, and many companies are now running them in production at scale. These two projects, both in the CNCF Sandbox, initially started with different technical approaches and philosophies: Cortex has been designed for scalability and high performances since day zero, while Thanos was originally focused on operational simplicity and cost-effectiveness.

Gardener, SAP's Kubernetes-as-a-service open source project, is moving its logging stack to Loki

Kristian Zhelyazkov is a developer at SAP working on Gardener, the SAP-driven Kubernetes-as-a-service open source project. In this guest blog post, he explains why the project is moving its logging stack to Loki.

Loki tutorial: How to set up Promtail on AWS EC2 to find and analyze your logs

Amazon’s Elastic Compute Cloud (AWS EC2) is one of the most popular ways to run applications in the cloud, but finding logs for a given instance is a common struggle. That’s where Loki can help. With Loki aggregation, you can group all your logs from all your virtual machines in one place, and with its search capabilities, you can quickly find and analyze them. It’s a great way to gain visibility in your cloud deployment.

Where did all my spans go? A guide to diagnosing dropped spans in Jaeger distributed tracing

Nothing is more frustrating than feeling like you’ve finally found the perfect trace only to see that you’re missing critical spans. In fact, a common question for new users and operators of Jaeger, the popular distributed tracing system, is: “Where did all my spans go?” In this post we’ll discuss how to diagnose and correct lost spans in each element of the Jaeger span ingestion pipeline.

Grafana and NGINX are partnering to give the open source community a turnkey experience for visibility

Over the past few years, NGINX users have naturally gravitated toward Grafana, and vice versa. These days, it’s not uncommon to see these two open source tools used together in the wild. And for good reason. F5, which acquired NGINX last year, is prioritizing building visibility across the entire product set, to make it easy for customers to quickly gain the insights that they need. Meanwhile, Grafana has evolved into the primary visualization and analysis tool in the open source market.

Grafana Loki sneak peek: Generate Ad-hoc metrics from your NGINX Logs

Get a sneak preview of a future version of Grafana Loki that enables you to generate ad-hoc metrics from your log data. This video features a Loki-based web analytics dashboard, which uses the access logs of the popular open-source web server NGINX. Every panel on this dashboard uses ad-hoc metrics created with Loki, well, besides the Log panel obviously. Would this be useful for your use-case? Let us know in the comments.

New Enterprise features in Grafana 7.0: Usage insights and user presence indicator

Dashboard sprawl is a real problem whether you’re using Grafana or any other tool. When growing to thousands of users – and as many dashboards – you’ll eventually want more information about how the tool is being used in your organization. After all, dashboards don’t help anyone if they aren’t being used. Managing large installations is one of the areas where Grafana Enterprise improves Grafana, and our launch of usage insights in 7.0 is a key part of that.

Getting started with the Grafana Cloud Agent, a remote_write-focused Prometheus agent

Hi folks! Éamon here. I’m a recent-ish addition to the Solutions Engineering team and just getting my feet wet on the blogging side so bear with me. :) Back in March, we introduced the Grafana Cloud Agent, a remote_write-focused Prometheus agent. The Grafana Cloud Agent is a subset of Prometheus without any querying or local storage, using the same service discovery, relabeling, WAL, and remote_write code found in Prometheus.

Why optimizing for MTTR over MTBF is better for business

The classic debate when running a software as a service (SaaS) business is between release frequency vs. stability and availability. In other words, are you Team MTTR (mean time to recovery) or Team MTBF (mean time between failure)? In this blog post, I argue for MTTR, which encourages you to push more frequently, embrace the instability this may introduce, and invest in training and tooling to deal with the pursuing outages.