Operations | Monitoring | ITSM | DevOps | Cloud

With the introduction of Environments & Services, we’ve seen a dramatic increase in the creation of new datasets. These new datasets are smaller than ones created with Honeycomb Classic, where customers would typically place all of their services under a single, large dataset. This change has presented some interesting scaling challenges, which I’ll detail in this post, along with the solution we used, and how we leveraged Honeycomb’s own telemetry to scale Honeycomb.

Over the course of two decades, Adform grew from a dream between friends huddled in a basement to a leading advertising tech platform powering more than 25,000 clients worldwide. Success brought external accolades, but it also created the need for internal innovation to support the company’s continued growth. In 2018, Adform was still operating in startup mode, which meant developers and teams cherry-picked the tools that worked best for them.

Intercom’s mission is to build better communication between businesses and their customers. With that in mind, they began their journey away from metrics alone and towards complete observability. The first step was tooling, and they learned quickly that trying to work with multiple solutions was not the answer.

If you’re writing software today, then you likely use a CI/CD pipeline to build and test your code before deploying it to production. Having a fast and efficient build pipeline saves you development time, shortens feedback loops, and helps you ship features faster. Conversely, slow and unreliable build pipelines are full of lost productivity and sadness.

An observability pipeline is a tool or process that centralizes data ingestion, transformation, correlation, and routing across a business. Production engineers across ITOps, Development, and Security teams use them to more efficiently and cost-effectively transform their telemetry data to drive critical decisions. Businesses of all sizes can enjoy several benefits and gain a significant competitive advantage by implementing an observability pipeline.