Kubernetes Load Test Tutorial
In this blog post we use podtato-head to demonstrate how to load test kubernetes microservices and how Speedscale can help understand the relationships between them. No, that’s not a typo,
podtato-head is an example microservices app from the CNCF Technical Advisory Group for Application Delivery, along with instructions on how to deploy it in numerous different ways:
- Direct deployment with kubectl
- Templated deployment with kustomize, helm, etc.
- Using gitops with argocd, flux
There are more than 10 delivery examples, you will surely learn something by going through the project. We liked it so much we forked the repo to contribute our improvements here:
First off, let’s install minikube and then run it with options to enable pod-to-pod network traffic. I’m using an Intel Mac, so I’ll download that binary.
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-darwin-amd64 sudo install minikube-darwin-amd64 /usr/local/bin/minikube
Next, we’ll start minikube with the container networking interface (CNI) networking option enabled. Without this, traffic between pods within the Kubernetes network won’t flow by default. Again, because I’m using a Mac, I’m installing the hyperkit driver to run Kubernetes inside of a virtual machine. Tune the memory, CPUs, and disk size allocated to the VM to suit your needs and hardware.
<span data-selectable-paragraph="">brew install hyperkit minikube start --cni=true --vm=true --memory=8g --cpus=4 --disk-size=128g</span>
Simple Delivery Option
I picked the
kubectl option because:
- To interact with Kubernetes, you need
kubectl(so no additional tools have to be installed)
- There are also no dependencies on the cluster
This is all it takes to deploy onto my minikube cluster:
kubectl apply -f https://raw.githubusercontent.com/speedscale/podtato-head/main/delivery/kubectl/manifest.yaml
Double check that your pods are running:
kubectl get pods
This automatically deployed a load balancer, but you need to note it is http and it is listening on port
LB=$(minikube ip) echo$LB
Now, on my Mac I was able to see it in the browser with:
You should see a friendly face with all four of his limbs!
Install Speedscale on podtato
If you are curious how the microservices are built, you can use Speedscale to intercept the calls and understand what is calling what. If you don’t have Speedscale already you can sign up for a Free Trial here. Once you have
speedctl on your machine, you can use it to install the control plane on your cluster:
Follow the default prompts to put the sidecar on everything in the podtato-kubeclt namespace:
Deployments patched. Patched default/podtato-head-entry Patched default/podtato-head-hat Patched default/podtato-head-left-arm Patched default/podtato-head-left-legp Patched default/podtato-head-right-arm Patched default/podtato-head-right-leg
Once you exercise your application, you should see the various services on your home page like so:
Drill Down for API Visibility
When you click into the
podtato-head-entry you can see a map which shows the relationship of the various services. The entry service is responsible for fetching images from all of the other services which you can clearly see in the service map:
Going down into the table you can see exactly what the call is from the entry service to each outbound API. It is actually fetching the SVG of the image that is shown on the home page. In your own apps, you can use this drill down to inspect what is being sent and received between your internal and external services. Perhaps a SaaS endpoint changed the body of responses and your current libraries aren’t up-to-date – Speedscale’s traffic viewer can help diagnose the issue.
By using this observed traffic, we ran replay as a load and performance test scenario to understand the limits of our kubernetes cluster. Click on the Snapshot button and this portion of traffic will be analyzed and stored for future use.
Load Test Kubernetes with Traffic Replay
Now running a replay of that snapshot is as simple as applying a patch file. Note that this patch refers to the original traffic snapshot (yours will have a specific unique id), and it also references a test config to run through the traffic at 10x what was originally recorded. Feel free to try out a few variations of the test config for different load patterns. These patterns are stored in customizable test configurations that can help you load test your application, or transform traffic to generate chaos tests for test and staging environments.
apiVersion: apps/v1 kind: Deployment metadata: name: podtato-head-entry annotations: test.speedscale.com/scenarioid: UUID test.speedscale.com/testconfigid: performance_10replicas test.speedscale.com/cleanup: "false" sidecar.speedscale.com/inject: "false"
Running the replay is as simple as:
kubectl patch deployment podtato-head-entry --patch-file patch.yaml
We can check for the Speedscale pods during our run by checking the namespace. These will disappear once testing is done.
kubectl get pods
After replaying the traffic, the testing pods will disappear and we should be able to see our replay from our traffic screen. They may take a moment to process depending on how much traffic you sent. First though, at the top of the page, you can see how requests for the different parts of the image were sent to your various backend services. This can be useful when trying to track down service dependencies in your own application; maybe there’s an old application that you rely on that everyone thought was decommissioned, but it’s still sending and receiving traffic.
Next, clicking on the replay will take you to a replay report.
Here, you can see metrics such as the average latency of requests, throughput, and a breakdown by endpoint tested. Memory and CPU usage are collected if the Kubernetes metrics server is installed and configured, which we did not do in this demo. Using latency, throughput, and hardware metrics can be helpful in identifying slow services or endpoints, or for identifying the best cost-to-performance hardware for your application.
Hopefully this demonstration is useful in understanding how quickly you can load test kubernetes microservices and how Speedscale can help understand the relationships between them! We encourage you to sign up for a free trial and test it out in your own environment today!