Should we build and run the full observability stack in pre-prod? How much realism vs. waiting for prod? Answer: Yes. You absolutely want observability in pre-production environments—local, dev, performance test, staging, CI, everywhere.
With more than 1.5M room nights booked per day, Booking.com requires a solid infrastructure that’s constantly monitored. And indeed, Booking.com now has a footprint of 50,000+ physical servers running across four data centers and six additional points of presence. The sheer size of this server fleet makes it viable for Booking.com to have dedicated teams specializing into looking only at the reliability of those servers.
You need not fear a long-lived streaming workload. A few simple tricks can transform a request that may not ever terminate for hours or days into something you can get regular health and status updates on. We in fact have one of those continuous processing services—Beagle, our Service Level Objective stream processor—which we’ve instrumented in this fashion.
We know that you value collaboration. That’s why we share incident reviews and learnings—because we believe the entire community benefits by working together transparently. In the spirit of working better together, we invited ecosystem partners from ApolloGraph, Cloudflare, LaunchDarkly, and PagerDuty to present at Honeycomb Developer Week, a three-day event filled with snackable, time-efficient learning sessions to help you uplevel your observability skills.
TL;DR: Use auto-instrumentation from OpenTelemetry. Traces will happen. Then your code can use global library functions to customize those traces with your specific important data.
In the past, we’ve written about what instrumentation is and the insights it provides. Instrumenting your code generates telemetry that shows you how your system is performing, and whether your system is healthy. Like with most other companies, at Honeycomb we don’t write all of the code that runs in our systems.
Oh goody, I’m so tickled to get this one. *rubs hands gleefully* Funny story, back in 2016–2017 we thought we were building Honeycomb primarily for DB use cases. The use cases are that killer. I’ve never seen another tool do the kinds of things you can do on the fly with Honeycomb and databases.
The overall theme is high-performing engineering teams are generally the ones that humanize the process. Whether you’re trying to increase productivity or release better-quality code, the biggest piece of advice is to lead with empathy.
It’s harder to understand and operate production systems in 2021 than it was in 2001. Why is that? Shouldn’t we have gotten better at this in the past two decades? There are valid reasons why it’s harder: The architecture of our systems has gotten a lot more sophisticated and complex over the past 20 years. We’re not running monoliths on a few beefy servers these days.
