When you have questions about your software, telemetry data is there for you. Over time, you make friends with your data, learning what queries take you right to the error you want to see, and what graphs reassure you that your software is serving users well. You build up alerts based on those errors. You set business goals as SLOs around those graphs.

At Honeycomb, we are very concerned about privacy and data sovereignty—it’s something we take very seriously, and in an effort to serve our customers better, we’re thrilled to announce that we now offer data residency in Europe. This new instance will allow Honeycomb customers to store their data in the US, in Europe, or both. Let’s talk about the details.

The cost of services is on everybody’s mind right now, with interest rates rising, economic growth slowing, and organizational budgets increasingly feeling the pinch. But I hear a special edge in people’s voices when it comes to their observability bill, and I don’t think it’s just about the cost of goods sold.

Good alerting hygiene consists of a few components: chasing down alert conditions, reflecting on incidents, and thinking of what makes a signal good or bad. The hope is that we can get our alerts to the stage where they will page us when they should, and they won’t when they shouldn’t. However, the reality of alerting in a socio-technical system must cater not only to the mess around the signal, but also to the longer term interpretation of alerts by people and automation acting on them.

There’s plenty of literature on the mechanics of instrumenting code with OpenTelemetry and delivering it to Honeycomb. However, I’ve not found many guides on the craft of instrumenting code in order to have a good observability experience in your system. A lot of focus is placed on automatic instrumentation—which is great, particularly if you’re new to observability or retrofitting—but it misses the power of good instrumentation at the application level.

Picture this: It’s 2 p.m. and you’re sipping on coffee, happily chugging away at your daily routine work. The security team shoots you a message saying the latest pentest or security scan found an issue that needs quick remediation. On the surface, that’s not a problem and can be considered somewhat routine, given the pace of new CVEs coming out. But what if you look at your tooling and find it lacking when you start remediating the issue?

The OpenTelemetry Collector is a core part of telemetry pipelines, which makes it one of the parts of your infrastructure that must be as secure as possible. The general advice from the OpenTelemetry teams is to build a custom Collector executable instead of using the supplied ones when you’re using it in a production scenario. However, that isn’t an easy task, and that prompted me to build something.

For developers, understanding the performance of shipped code is crucial. Through the last decade, a tablestake function in software monitoring and observability solutions has been to save and track app metrics. Engineers love tools that get out of your way and just work, and the appeal of today’s best-in-class application performance monitoring (APM) suites lies in a seamless day zero experience with drop-in agent installs, button click integrations, and immediate metrics collection.

I’m currently working on a small team within Honeycomb where we’re building an ambitious new feature. We’re excited—heck, the whole company is—and even our customers are knocking on our door. The energy is there. With all this excitement, I’ve been thinking about a risk that—if I'm not careful—could severely hinder my team's ability to ship on time, celebrate success, and continue work after launch: burnout.

In a keynote at AI.Dev, Robert Nishihara (CEO, Anyscale) described the shift: A year ago, the people working with ML models were ML experts. Now, they’re developers. A year ago, the process was to experiment with building a model, then put a product on top of it. Now, it’s ship a product, find the market fit, then create customized models. The general-purpose generative AI models available to all of us today (such as ChatGPT) change the way work is done.