Jaeger primarily supports two backends: Cassandra and Elasticsearch. Here at Grafana Labs we use Scylla, an open source Cassandra-compatible backend. In this post we’ll look at how we run Scylla at scale and share some techniques to reduce load while ingesting even more spans. We’ll also share some internal metrics about Jaeger load and Scylla backend performance. Special thanks to the Scylla team for spending some time with us to talk about performance and configuration!

Cortex is a long-term distributed storage for Prometheus. It provides horizontal scalability, high availability, multi-tenancy and blazing fast query performances when querying high cardinality series or large time ranges. Today, there are massive Cortex clusters storing tens to hundreds of millions of active series with a 99.5 percentile query latency below 2.5s.

At Grafana Labs, we’re big fans of putting ourselves in the shoes of our customers. So when it comes to building a product, dogfooding is a term we throw around constantly. In short, what it means is that we actually use the products we create throughout their entire life cycle. And I really mean the whole life cycle.

At the recent PromCon Online, I gave a review of developments in the space of the Prometheus storage engine, TSDB. In this blog post I am going to recap a bit of the talk and add more insights into what these developments will enable us to do in the future. While the talk contained some of the near-future features, I will be diving even further ahead. You can watch the talk here.

During PromCon, I gave a talk titled “The Future of Prometheus and its Ecosystem.” I want to share the key highlights with you.

It’s been a while, but the Kelly and Regis of Grafana Labs (a.k.a. Christine and Eldin of Solutions Engineering) are back to report on another Grafana Enterprise plugin: New Relic! The latest version of this plugin will be just one of the many topics we’ll cover during today’s webinar, All about Grafana plugins: Visualizing disparate data sources in one place. We’ll be hosting a great conversation around plugin updates, use cases, and the best way to make coffee.

Amazon Elastic Kubernetes Service (Amazon EKS) is the fully managed Kubernetes service on AWS. If you’re using it and wondering how to query all your logs in one place, Loki is the answer. With this tutorial, you’ll learn how to set up Promtail on EKS to get full visibility into your cluster logs while using Grafana. We’ll start by forwarding pods logs then nodes services and finally Kubernetes events.

The 7.1 stable release builds on the major developments in Grafana 7.0. Here are some features we’re excited about in Grafana 7.1.

Cortex and Thanos are two brilliant solutions to scale out Prometheus, and many companies are now running them in production at scale. These two projects, both in the CNCF Sandbox, initially started with different technical approaches and philosophies: Cortex has been designed for scalability and high performances since day zero, while Thanos was originally focused on operational simplicity and cost-effectiveness.

Kristian Zhelyazkov is a developer at SAP working on Gardener, the SAP-driven Kubernetes-as-a-service open source project. In this guest blog post, he explains why the project is moving its logging stack to Loki.

Amazon’s Elastic Compute Cloud (AWS EC2) is one of the most popular ways to run applications in the cloud, but finding logs for a given instance is a common struggle. That’s where Loki can help. With Loki aggregation, you can group all your logs from all your virtual machines in one place, and with its search capabilities, you can quickly find and analyze them. It’s a great way to gain visibility in your cloud deployment.

Nothing is more frustrating than feeling like you’ve finally found the perfect trace only to see that you’re missing critical spans. In fact, a common question for new users and operators of Jaeger, the popular distributed tracing system, is: “Where did all my spans go?” In this post we’ll discuss how to diagnose and correct lost spans in each element of the Jaeger span ingestion pipeline.

Over the past few years, NGINX users have naturally gravitated toward Grafana, and vice versa. These days, it’s not uncommon to see these two open source tools used together in the wild. And for good reason. F5, which acquired NGINX last year, is prioritizing building visibility across the entire product set, to make it easy for customers to quickly gain the insights that they need. Meanwhile, Grafana has evolved into the primary visualization and analysis tool in the open source market.

Dashboard sprawl is a real problem whether you’re using Grafana or any other tool. When growing to thousands of users – and as many dashboards – you’ll eventually want more information about how the tool is being used in your organization. After all, dashboards don’t help anyone if they aren’t being used. Managing large installations is one of the areas where Grafana Enterprise improves Grafana, and our launch of usage insights in 7.0 is a key part of that.

Hi folks! Éamon here. I’m a recent-ish addition to the Solutions Engineering team and just getting my feet wet on the blogging side so bear with me. :) Back in March, we introduced the Grafana Cloud Agent, a remote_write-focused Prometheus agent. The Grafana Cloud Agent is a subset of Prometheus without any querying or local storage, using the same service discovery, relabeling, WAL, and remote_write code found in Prometheus.

The classic debate when running a software as a service (SaaS) business is between release frequency vs. stability and availability. In other words, are you Team MTTR (mean time to recovery) or Team MTBF (mean time between failure)? In this blog post, I argue for MTTR, which encourages you to push more frequently, embrace the instability this may introduce, and invest in training and tooling to deal with the pursuing outages.