Operations | Monitoring | ITSM | DevOps | Cloud

TL;DR Managing Kafka consumer lag manually is time-consuming and error-prone. Today, we'll show you how to create a fully automated consumer lag monitoring agent in just 5 minutes using Aiven Managed Agents. This agent will monitor your Kafka services and send real-time alerts to Slack when consumer lag is high.

When we first introduced people to Aiven Runtime, before it even had that name, we kept getting asked: "Can we schedule things?" If you come from a Unix background, that tends to cause a reflexive reach for the cron tool, which provides basic scheduling at fixed intervals. But if you've spent more time with distributed software, you're more likely to think of Apache Airflow, which can do a lot more (although, yes, you can use Airflow to schedule tasks at fixed intervals if you want).

That was the question posed at the Women in Open Source+ Panel I had been asked to moderate. The panelists, from Snowflake, AWS, and NVIDIA, gave similar answers: We have our jobs today because of the community, and I don’t see that changing any time soon. I felt the same. When I joined Aiven, its direct callout to open source was a deciding factor.

TL;DR All scripts in this post live in the companion repository dkrautschick/pg-logical-migration. Every step references a numbered script you can run against your own Aiven project, and the listings/ folder has each step as a standalone snippet for copy-paste self-tests. The demo database is beerdb, a small craft-beer database, because every example database should contain at least one Doppelbock.

Aiven Managed Agents: Your agents, your data, fully managed and open-sourced.

Say you want to build an agent that resolves support tickets against your own data. Not a demo against a toy dataset, an agent that actually runs in production. The agent itself is maybe a day of work.

TL;DR Part 0 of a three-part series on building a documentation assistant with Aiven. This part gives you the vocabulary. Part 1 is the architecture. Part 2 builds the thing. Almost everything written about large language models explains how models get built. Very little explains what happens the other billion times, when a finished model sits there answering questions. That second thing is called inference, and it is where practically all of the engineering work actually lives.