Get Kafka-Nated Episode 10
Kyle McCullough, Co-Founder & CTO at OpsHelm, former Head of Infrastructure Engineering at ProdPerfect and Lead Engineer at Vivid Seats, joins host Hugh Evans to explore what it takes to build real-time, multi-cloud streaming infrastructure at scale.
As Co-Founder and CTO of OpsHelm, Kyle shares how his team processes hundreds of terabytes of cloud events daily, maintaining sub-second visibility while reducing streaming costs by 78% after migrating from MSK and NATS to Aiven Diskless Kafka.
We’ll dive into lessons learned from three generations of streaming architecture, the realities of data loss and scaling challenges, and how Diskless Kafka enabled a 5× cost reduction, zero manual ops, and simplified multi-cloud deployment.
What we’ll cover:
🔷 Why OpsHelm is described as “the git log for cloud infrastructure”
🔷 Key lessons from migrating MSK → NATS → Diskless Kafka
🔷 How Diskless Kafka delivers durability and scale without EBS or cross-AZ fees
🔷 The trade-offs between latency, durability, and cost in real-time systems
Timestamps:
00:00 – Introduction & Guest Focus: Kyle McCullough
01:20 – OpsHelm Overview: “The git log for cloud infrastructure”
03:45 – Lessons from three generations of streaming architecture
08:10 – Migrating MSK → NATS → Diskless Kafka
12:50 – How Diskless Kafka enables durability and scale without EBS
16:30 – Reducing costs by 78% & achieving zero manual ops
20:15 – Trade-offs: latency, durability, and cost in real-time systems
23:50 – Closing thoughts & next steps
Useful links:
Learn more about Aiven Diskless Kafka: https://aiven.io/kafka/diskless
Explore the Aiven platform for Apache Kafka and other open-source services: https://aiven.io
Watch more Get Kafka-Nated episodes: https://www.youtube.com/@Aiven_io
#apachekafka #DisklessKafka #aiven #cloudstreaming #realtimedata
Connect With Us
Website: http://aiven.io
LinkedIn: https://www.linkedin.com/company/aiven/
GitHub: https://github.com/aiven
X: https://twitter.com/aiven_io