Operations | Monitoring | ITSM | DevOps | Cloud

The BigQuery Sink connector is a critical piece of Kafka infrastructure that allows you to offload your Kafka topic data into BigQuery in real time. It is the third most-used connector among Kafka users (after the Google Cloud Managed Service for Apache Kafka and the original WePay sink connector), but it's not without its fair share of plot twists. Here's the story of how this connector switched hands three times and we ultimately ended up helping to re-build it.

As the Senior Manager of Finance & Accounting here at CloudZero, and with a career in FP&A that includes tenures at large public companies, I’ve spent a significant amount of time observing the interactions between the folks who plan the company’s budget and those who spend it. While engineering and operations teams focus on the execution side, my team ensures that the company has the resources required to make each endeavor a success.

For years, you had an unspoken agreement. Your networks were built to be resilient, and your applications were, for the most part, forgiving. You sent emails, transferred files, and backed up data. If a few packets went missing along the way, the protocols would quietly clean up the mess. A little bit of packet loss was just background noise, an expected imperfection in a system that was, by and large, incredibly robust. You could tolerate it.

Enterprises face spiraling observability costs. Gartner reports a 20% year-over-year rise in spending, with the median spend per observability tool reaching $800,000 annually. The average organization using BigPanda coordinates data from ~20 different observability solutions, each taking up an ever-larger portion of IT budgets.