Operations | Monitoring | ITSM | DevOps | Cloud

The rise of large-language models (LLMs) that can understand and create text, and reason about problems has opened the door for software agents that use the reasoning power of LLMs to interact with the wider world and solve problems. At Kubex, our bread and butter is compute and GPU resource optimization, so it’s natural for us to ask how agents could help our customers more effectively optimize their resources.

Kubernetes workload rightsizing is one of those tasks everyone agrees is important and almost nobody enjoys doing by hand. Set requests too high and you pay for capacity that sits idle; set them too low and you invite CPU throttling, out-of-memory kills, and late-night pages. It’s no surprise, then, that many teams reach for the Vertical Pod Autoscaler (VPA) as their first step toward automated rightsizing.

Table of Contents Autoscaling and policy-driven resource management are two sides of the same coin. Autoscaling adjusts capacity as demand changes, while policies define the boundaries it operates within: who can use how much, which workloads can be changed, and what safeguards must be respected. Without autoscaling, clusters are either overprovisioned or overwhelmed. Without policies, autoscaling can create runaway costs, noisy neighbors, or disruptive changes to critical services.

Table of Contents Resource contention happens when workloads compete for more capacity than a node or cluster can provide. For CPU and memory, the symptoms are familiar: CPU throttling, OOM kills, and noisy neighbors slowing latency-sensitive services. These are largely solved problems, addressed by accurate requests and limits, Quality of Service classes, and autoscalers like VPA and HPA that adjust sizing and replicas as demand changes. GPU contention is different, and far more expensive to get wrong.

Table of Contents GPU sharing with NVIDIA hardware is becoming easier to adopt in Kubernetes but it hasn’t been easier to observe. Time-slicing lets multiple workloads share the same GPU. MPS allows CUDA workloads to execute concurrently. Schedulers like KAI make it easier to manage these shared environments. But sharing a GPU introduces a problem that is easy to underestimate.

Table of Contents A few months ago, we introduced Kubex support for the KAI Scheduler to improve GPU sharing for production inference workloads. The basic model is simple: The KAI Scheduler handles placement and GPU sharing. Kubex continuously observes usage and adjusts those allocations as demand changes. KAI provides the scheduling foundation. It lets multiple workloads share a GPU while accounting for the amount of GPU each workload requests. Kubex then closes the loop.

Table of Contents For most of Kubernetes’ history, memory management has been a blunt instrument. Cross your limit, and the kernel kills your container. There has been no equivalent to CPU throttling, no graceful backpressure, just a hard stop. With Kubernetes 1.37, that changes: Memory QoS, built on cgroups v2, graduates to Beta and is enabled by default.

Table of Contents Kubernetes resource optimization appears to be a single problem, but the platforms that address it disagree on almost every design decision, starting with how they analyze workload demands. Some set CPU and memory requests from live signals, while others learn a workload’s historical pattern and provision ahead of it.