Stefano Mitchell is a customer support engineer at SkySilk Cloud Services. It’s no secret that there is a correlation between a team having quick access to metrics and swift resolutions. Accurate monitoring metrics displayed in a clear and efficient manner help your teams respond to alerts and issues as they arise in real time. SkySilk Cloud Services, a cloud services provider, uses Grafana dashboards internally to maintain a strong overview of regional system health.

Grafana dashboards can do a lot, but do you know how much more you can get out of them by configuring them as code? That was the topic of a recent FOSDEM 2020 talk by Grafana software developer Malcolm Holmes and Julien Pivotto, an open source consultant at Inuits. In their presentation, the pair discussed Grafonnet (a Jsonnet library to generate Grafana dashboards), provided tips and tricks about how to use it efficiently, and explained how to fully manage your Grafana instances from code.

In my previous blog post, “How to Explore Prometheus with Easy ‘Hello World’ Projects”, I described three projects that I used to get a better sense of what Prometheus can do. In this post, I’d like to share how I got more familiar with Prometheus Alertmanager and how I set up alert notifications for Slack, PagerDuty, and Gmail.

In a previous post we showed how to install Prometheus and Grafana using the prometheus-ksonnet library along with Tanka. This is great for getting a well-managed monitoring install going, but sometimes it isn’t enough for monitoring larger clusters. If you have multiple clusters that you want to monitor on a single dashboard, or need long-term storage, or need a high-availability setup for your monitoring data, then this installation won’t be sufficient on its own.

People think technical writing is boring, but sometimes documenting software is an adventure. It’s not an adventure like “whee, got my sword and shield, adventure time!” No, it’s more like taking a nice stroll down a path to an unfamiliar-but-known destination when the ground suddenly opens up under your feet. As you’re falling down into the depths, that’s when you realize you are about to have an adventure. I’m a technical writer at Grafana Labs.

Several months ago, Bryan Boreham introduced a few changes to Cortex that massively reduced its storage requirements. The changes were quite simple and altogether had a nice benefit of using almost 3x less data storage than prior versions. Since Loki shares a lot of code with Cortex, could we use these ideas to the same effect? (Spoiler alert: Yes, we can!)

Last week on Slack: Eldin: Hey Christine, do you remember the first time you viewed a log file? Christine: Oh yes. I used Splunk as a support engineer and I remember. You? Eldin: I believe it was early 2000s. I was installing Slackware and a few network cards for a DIY router, and logs were critical. Hello again! We are Eldin and Christine from Solutions Engineering – a team at Grafana that is passionate about connecting people to our products – reporting back for duty.

I have been a Grafana power user since almost the day it was conceived. During this time, I’ve gotten acquainted with a few quirks but also many features, some of which are rather obscure. One of these features that few know about but I absolutely love is annotations.

Hashicorp’s Consul service is a distributed, highly available system that provides a service mesh solution, including service discovery, configuration, and segmentation functionality. Cortex uses Consul’s KV store to share information that’s necessary for distributing data to its components. While writing to Consul has been useful at Grafana Labs, we’ve found that as we expanded the operations, problems started arising.

As mentioned in a previous post, at Grafana Labs we make heavy use of Tanka and the Jsonnet programming language to manage our Kubernetes infrastructure. One of the benefits of the use of Jsonnet is the depth of collaboration that it allows with others outside of your company. For example, the open source prometheus-ksonnet library can be used to install both Prometheus and Grafana.

Recently, Loki v1.3.0 was released. It included many changes, but I’d like to talk about one in particular: the query frontend. This new component in the Loki architecture is a drop-in addition. What does that mean? Loki can run with or without it. In fact, the query frontend both produces and consumes the Loki API, meaning to a consumer there’s no difference.

PromQL is the querying language that is part of Prometheus. In addition to PromQL, Prometheus provides a scraper that fetches metrics from instances (any application providing metrics) and a time series database (TSDB), which stores these metrics over time. This introduction to PromQL will be largely decoupled from specific tools and the non-PromQL parts of Prometheus, in order to focus on the features of the language itself.

Greetings! This is Christine and Eldin reporting from Solutions Engineering, a team at Grafana Labs dedicated to helping users maximize what our Enterprise and Cloud products can do for your orgs. In a previous life, Eldin managed the customer experience department for several different companies and still remembers the pain of running daily reports so his team could have visibility into their ticket queues.