Operations | Monitoring | ITSM | DevOps | Cloud

TL;DR Managing Kafka consumer lag manually is time-consuming and error-prone. Today, we'll show you how to create a fully automated consumer lag monitoring agent in just 5 minutes using Aiven Managed Agents. This agent will monitor your Kafka services and send real-time alerts to Slack when consumer lag is high.

Improving an AI agent means knowing which behaviors to change and where those changes will have the greatest impact. Insights in Datadog Agent Observability improves AI agents by analyzing agent traces to identify recurring cost and reliability problems. It then explains the causes of those problems, assesses their impact, and recommends fixes backed by evidence. At Datadog, we use Insights to improve the agent behind Insights.

When we first introduced people to Aiven Runtime, before it even had that name, we kept getting asked: "Can we schedule things?" If you come from a Unix background, that tends to cause a reflexive reach for the cron tool, which provides basic scheduling at fixed intervals. But if you've spent more time with distributed software, you're more likely to think of Apache Airflow, which can do a lot more (although, yes, you can use Airflow to schedule tasks at fixed intervals if you want).

That was the question posed at the Women in Open Source+ Panel I had been asked to moderate. The panelists, from Snowflake, AWS, and NVIDIA, gave similar answers: We have our jobs today because of the community, and I don’t see that changing any time soon. I felt the same. When I joined Aiven, its direct callout to open source was a deciding factor.

TL;DR All scripts in this post live in the companion repository dkrautschick/pg-logical-migration. Every step references a numbered script you can run against your own Aiven project, and the listings/ folder has each step as a standalone snippet for copy-paste self-tests. The demo database is beerdb, a small craft-beer database, because every example database should contain at least one Doppelbock.

Aiven Managed Agents: Your agents, your data, fully managed and open-sourced.

Say you want to build an agent that resolves support tickets against your own data. Not a demo against a toy dataset, an agent that actually runs in production. The agent itself is maybe a day of work.