Operations | Monitoring | ITSM | DevOps | Cloud

AI Agents Write Broken Code 49% of the Time #speedscale #AI #Coding #Tech #DevOps

AI agents write broken code nearly 50% of the time. By adding a traffic-based deterministic evaluation, Speedscale boosted unsupervised bug-fixing quality from 51% to 77% in just 5 minutes. This helped slash token costs and eliminate rework without human intervention. Learn more: speedscale.com.

The Three Pillars Were Built for Humans

It was 2am and I was paying for the privilege. Something was on fire in production, and I’d done the modern thing: I pointed an AI agent at it. It ingested the dashboards. It read the logs. It walked the traces. Then it handed me back a beautifully formatted paragraph that said, in effect, “latency is elevated on the checkout path.” I knew that. The page told me that.

Which Bugs AI Agents Fix Better With Traffic

In the first experiment, I wanted a baseline: if an AI coding agent gets the same production signal a human would get, can it fix bugs in a codebase it has never seen? Yes, but only when I gave it better context. With only an alert, the agent passed 51% of the runtime tests. When I added captured traffic, the actual request and response for the failing call, it climbed to 77%. This post is the second pass.
Sponsored Post

Five things your logs will never tell you

A customer escalation hit my queue when I was on the customer smoke jumpers team at an observability vendor. My team was the group that parachutes into Fortune 500 accounts one bad week from churning and usually after a big customer outage. The customer had filed a billing dispute three weeks earlier and their on-call engineers were stuck. They had our full stack: logs, metrics, traces, end-to-end instrumentation, every product we sold and some we didn't. They could see the request came in. They could see it returned a 500. They could not see the body. The trace was sampled out. The log line was truncated at 4KB.

Beware of PII in Testing Data: The Security Iceberg and Where PII Actually Hides

If you run a platform tools or security team, you have likely heard this request from developers: “I just need a copy of the production database for staging so I can run realistic load and integration tests.” It is a completely reasonable request. Production traffic and data contain the actual request shapes, real-world value distributions, long-tail anomalies, and timing patterns that make tests useful.