Operations | Monitoring | ITSM | DevOps | Cloud

AI workloads have a distinctive infrastructure profile that's exposing the cost and architectural assumptions baked into public cloud over the last decade.

Most organizations know cloud costs are rising. Fewer realize that some of what they're paying isn't for infrastructure at all; it's a penalty for not being in control of it. That penalty has a name: Sovereignty Tax. It isn't a line item on your invoice. It won't appear in your cloud dashboard. But it's accumulating quietly, in egress fees, outage exposure, audit blind spots, and the creeping realization that leaving your current provider would be harder, and more expensive, than you ever anticipated.

Most organizations get the build versus buy decision wrong in the same way. They underestimate the cost of building while overestimating the cost of buying. In the recent Konstruct monthly webinar with M R Rishi (Platform Engineer at Civo), we explored the discussion surrounding whether you should build or buy your platform. If you want to watch the full discussion, watch the recording here.

A Kubernetes Operator is a method of packaging, deploying, and managing a Kubernetes application. It extends the native Kubernetes API by combining custom resources (CRDs) with a dedicated controller: a custom control loop that continuously watches the state of those resources. The primary purpose of an operator is to automate complex, stateful applications (like databases, message queues, or monitoring suites) that require human operational knowledge to maintain.

At Datadog, our broad Kubernetes footprint amplifies the significance of a familiar autoscaling tradeoff: Overprovisioning wastes cloud spend, while underprovisioning threatens reliability. We built Datadog Kubernetes Autoscaling (DKA) to help teams rightsize their workloads by generating intelligent resource recommendations and automating multidimensional workload scaling. Across Datadog, adopting DKA has eliminated more than $3 million in annualized idle compute costs while reducing reliability risks.

Here's a scenario that's playing out in engineering teams across the industry right now. A developer uses AI to rapidly prototype a microservice. The code works. They deploy it to production. Six months later, something breaks. The system is under load, a database connection pools, and the service starts failing in subtle ways. The engineer pulls up the code, but here's the problem, they didn't write it. An AI assistant did. They don't understand the flow deeply. They don't know where to look first.

Today, we’re launching Kubex support for the KAI Scheduler and automated GPU sharing for inference workloads. As AI inference moves into production, platform teams are being asked to serve more models, support more teams, and control GPU costs at the same time. But many inference workloads do not need an entire GPU all the time. When teams reserve full GPUs or oversized GPU fractions to stay safe, expensive capacity can sit idle across the cluster.