First, I’d like to say that pager duty isn’t something we should treat like chronic pain or diabetes, where you just constantly manage symptoms and tend to flare-ups day and night. Being paged out of hours is as serious as a fucking heart attack. It should be RARE and taken SERIOUSLY. Resources should be mustered, product cycles should be reassigned, until the problem is fixed.
From September to early October, Honeycomb declared five public incidents. Internally, the whole month was part of a broader operational burden, where over 20 different issues interrupted normal work. A fraction of them had noticeable public impact, but most of the operational work was invisible. Because we’re all about helping everyone learn from our experiences, we decided to share the behind-the-scenes look of what happened.
The moment of launching something new at a game studio (titles, experiences, features, subscriptions) is a blockbuster moment that hangs in the balance. The architecture—distributed and complex, designed by a multitude of teams, to be played across a variety of devices in every corner of the world—is about to meet a frenzy of audience anticipation, along with the sky-high expectations of players, executives, and investors.
The best way to be sure that you keep a secret is not to know it in the first place. Managing secrets is a notoriously difficult engineering problem. Across our industry, secrets are stored in a bewildering variety of secure (and sometimes notoriously insecure) systems of varying complexity. Engineers are often trying to balance the least worst set of tradeoffs. At Honeycomb, we asked: What if we didn’t need to know your secrets to begin with?
All of the business of software, but especially the delivery of product capabilities, is inextricably bound up in questions about time. What’s the estimate? If we have N people working on it, how long will it take? When will we ship?
Stand-up meetings. Is anyone happy with them at this point? They were supposed to help teams work in a more agile manner but they were already controversial in the before times and moving to fully distributed teams hasn’t made things any better. The same old habits, the same tired questions. There must be something better, right? We began meandering syncs to replace stand-ups as an experiment at Honeycomb, but loved the results so much that we have adopted it across the engineering org.
Today, we announced that Honeycomb has raised $50M in Series C funding, in a round led by Insight Partners and joined by all existing investors from our Series B. We’re using this investment to support the growth of our customers and community, ensure the benefits of observability can be realized by all engineering teams, and expand the ways we can better serve you.
You might already know that OpenTelemetry is the future of instrumentation. It’s an open-source and vendor-neutral instrumentation framework that frees you from the trap of using proprietary libraries simply to understand how your code is behaving. Best of all, you can instrument your applications just once and then take that instrumentation to any other backend system of your choice. This blog shows you exactly how to use OpenTelemetry to ✨break the vendor lock-in cycle.✨
As product developers, our responsibility continues beyond shipping code. To keep our software running, we need to notice whether it’s working in production. To make our product smoother and more reliable, we need to understand how it’s working in production. We can do this by making the software tell us what we need to know. How can we notice when the software is running smoothly? Make it tell us!
In the recent past, most engineering teams had a vague notion of what Service Level Agreements (SLAs) and Service Level Objectives (SLOs) were—mainly things that their more business-focused colleagues talked about at length during contract negotiations. The success or failure of SLAs were tallied via magic calculations (what is “available” anyway?!) at the end of the month or quarter, and adjustments were made in the form of credits or celebrations in the break room.
