Signal vs. Spend: Building Cost-Aware Observability at Slack - O11yCon 2026

It started with a single log line taking up a massive amount of volume: 500 million emissions per hour. Pulling that thread led Emma and Steven into Slack's broader logging pipeline: 311 billion logs per day at 4.4M/sec peak, with no volume limits, no per-service attribution, and no feedback to the teams generating the noise.

They built a logging MCP server and a Claude skill that hunted down the worst offenders (unsampled hot paths, whole-object dumps, timer-based chatter, and logs redundant with metrics) then gave Claude source-code access so it could open PRs with the data attached rather than file tickets. To keep the problem from growing back, they added a log review agent on every PR, so engineers see the cost impact of their log changes before they ship.

This resulted in an over 30% cost reduction Slack-wide (some services hit 50%), retention extended from five to seven days, and seven-figure annual savings. The AI analysis itself cost under $500.

As Martin Thwaites put it: "Management hadn't reviewed the talk, and were in the audience. Luckily, Emma and Steven still have a job. We think the 30% cost reduction helped."