Operations | Monitoring | ITSM | DevOps | Cloud

Table of Contents Autoscaling and policy-driven resource management are two sides of the same coin. Autoscaling adjusts capacity as demand changes, while policies define the boundaries it operates within: who can use how much, which workloads can be changed, and what safeguards must be respected. Without autoscaling, clusters are either overprovisioned or overwhelmed. Without policies, autoscaling can create runaway costs, noisy neighbors, or disruptive changes to critical services.

Table of Contents Resource contention happens when workloads compete for more capacity than a node or cluster can provide. For CPU and memory, the symptoms are familiar: CPU throttling, OOM kills, and noisy neighbors slowing latency-sensitive services. These are largely solved problems, addressed by accurate requests and limits, Quality of Service classes, and autoscalers like VPA and HPA that adjust sizing and replicas as demand changes. GPU contention is different, and far more expensive to get wrong.

Table of Contents GPU sharing with NVIDIA hardware is becoming easier to adopt in Kubernetes but it hasn’t been easier to observe. Time-slicing lets multiple workloads share the same GPU. MPS allows CUDA workloads to execute concurrently. Schedulers like KAI make it easier to manage these shared environments. But sharing a GPU introduces a problem that is easy to underestimate.

Table of Contents A few months ago, we introduced Kubex support for the KAI Scheduler to improve GPU sharing for production inference workloads. The basic model is simple: The KAI Scheduler handles placement and GPU sharing. Kubex continuously observes usage and adjusts those allocations as demand changes. KAI provides the scheduling foundation. It lets multiple workloads share a GPU while accounting for the amount of GPU each workload requests. Kubex then closes the loop.

Table of Contents For most of Kubernetes’ history, memory management has been a blunt instrument. Cross your limit, and the kernel kills your container. There has been no equivalent to CPU throttling, no graceful backpressure, just a hard stop. With Kubernetes 1.37, that changes: Memory QoS, built on cgroups v2, graduates to Beta and is enabled by default.

Table of Contents Kubernetes resource optimization appears to be a single problem, but the platforms that address it disagree on almost every design decision, starting with how they analyze workload demands. Some set CPU and memory requests from live signals, while others learn a workload’s historical pattern and provision ahead of it.

Serving large language models at scale is fundamentally a distributed systems problem. A single GPU, or even a single node, is rarely enough once you need multiple models, multiple replicas, tensor-parallel sharding across GPUs, or high-availability rollouts. Kubernetes solves general container orchestration well, but it has no native concept of a GPU-aware, actor-based compute cluster.

The default Kubernetes scheduler was built for stateless services: web servers, APIs, databases. It schedules a pod, checks that a node has enough of whatever resources were requested, and binds it. For CPU and memory, that model works fine. For GPUs, it falls apart in three specific ways. First, GPUs are treated as an opaque integer resource.

Google has quietly rolled out a new beta feature for Google Kubernetes Engine (GKE): the MultidimPodAutoscaler, or MPA. Announced in early July 2026, MPA gives GKE users a single custom resource that scales both horizontally and vertically at the same time.

Coding agents changed implementation economics faster than they changed confidence. They let us produce more code, more quickly, but they did not make reviewers any better at understanding system-wide consequences. In our Kubernetes automation stack, that gap became impossible to ignore once AI started generating meaningful amounts of controller code.