In the old days, the most senior members of an engineering team were the best debuggers. They had built up such an extensive knowledge about their systems that they instinctively knew the right questions to ask and the right places to look. They even wrote detailed runbooks in an attempt to identify and solve every possible issue and possible permutation of an issue.
If you've stumbled (or purposefully landed) on this blog post, chances are you are new to—or diving deeper—into the observability space, o11y for short. Suffice it to say, you’re not in Kansas anymore. Honeycomb in a lot of ways can serve as a yellow brick road into o11y, and this article should serve as an introduction into how Honeycomb facilitates implementing o11y into applications and distributed services.
How do you go from A to Z with observability and OpenTelemetry? This post answers a question we hear often: “How do I get started on instrumentation with OpenTelemetry, while also following best practices for the long-term?” This article is all about taking you from A to Z on instrumentation. This will help you: We will use a simple greeting service application written in Node.js to understand the journey. You can find the pre-instrumented state here.
At Honeycomb, we love involving our customers when we can. We asked: “What was the first thing you found value from with Honeycomb?” Here are some of our favorite answers.
Since 2013, Monitorama has been a community-driven conference, bringing together open source development and operations engineers to focus on pushing the boundaries of monitoring software and practices. It’s chock full of thought-provoking content in the conference talks. The casual atmosphere also makes the hallway track a great way to network with fellow engineers and vendors alike to pick up on new developments in the monitoring space.
A while ago, I wrote about how we track on-call health, and I heard from various people about how “expecting to be woken up” can be extremely unhealthy, or how tracking the number of disruptions would actually be useful. I took that feedback to heart and wanted to address the issues they raised, and also provide some numbers that explain the position I took with these metrics on alerts.
It is only possible to come to an understanding of a system of interest by trying to change it. Here, Jackson contrasts action research with old-style hard science, which tries to study a system from the outside. Laboratories draw a line between experiment and scientist. In the social world, there is no outside: we participate in the systems we study. I’ve noticed this in code: when I come to an existing codebase, I get a handle on it by changing stuff.
Are you overspending on monitoring and APM tools? Forrester’s Total Economic Impact analysis of Honeycomb identified significant ROI in customers using us to reduce spend on less efficient APM workflows. But this isn’t about budget reallocation to a newly branded set of similar but shinier tools.
Remember the old days where if you had an uptime of 99.9 you could be fairly confident everyone was having a good experience with your application? That’s not really how it works anymore. Modern, distributed systems are so complex they typically fail unpredictably, making it much harder to diagnose issues. Traditional monitoring grew out of those early days, allowing you to check the health of simpler systems.
Honeycomb uses AWS Lambda as a core part of our query execution architecture; Lambda’s ability to quickly allocate lots of resources and charge us only for use is invaluable to keeping Honeycomb fast and affordable. Our total Lambda bill is easily accessible in the AWS Console, but how do we know which customers or application areas dominate this bill? How do we judge the cost of changes we make to our own software?
