Operations | Monitoring | ITSM | DevOps | Cloud

Service mesh observability guide: the metrics Istio, Linkerd and Envoy proxies emit, what ambient mode changes, and why tracing still needs app changes. Sejal Pandey works on content and growth at Last9, writing about observability, reliability, and SRE practices.

The world of Kubernetes networking can sometimes be confusing. What’s a CNI? A service mesh? Do I need one? Both? And how do they interact in my cluster? The questions can go on and on. Even for seasoned platform engineers, making sense of where these two components overlap and where the boundaries of responsibility end can be challenging. Seemingly bewildering obstacles can stand in the way of getting the most out of their complementary features.

For years, platform teams have known what a service mesh can provide: strong workload identity, authorization, mutual TLS authentication and encryption, fine-grained traffic control, and deep observability across distributed systems. In theory, Istio checked all the boxes. In practice though, many teams hit a wall. Across industries like financial services, media, retail, and SaaS, organizations told a similar story. They wanted mTLS between services to meet regulatory or security requirements.

Managing and deploying applications across multiple Kubernetes clusters presents significant challenges, especially as the number of clusters grows. Traditional methods, like manually applying Helm charts or manifests per cluster, become cumbersome, error-prone, and difficult to scale or maintain consistency for Day 2 operations. While Rancher allows managing Helm chart repositories and apps, this is done on a per-cluster basis via the UI.

Service meshes like Istio have become an essential way to securely and reliably distribute network traffic, especially with ephemeral, service-based architectures such as Kubernetes. However, their constantly shifting nature can interfere with targeting specific services for resilience tests. Infrastructure-based testing is designed to target specific IP addresses, allowing precision testing of applications, VMs, and nodes.

Service mesh is a tool for adding observability, security, and traffic management capabilities at the application layer. A service mesh is intended to help developers and site reliability engineers (SREs) with service-to-service communication within Kubernetes clusters. The challenges involved in deploying and managing microservices led to the creation of the service mesh, but service mesh solutions themselves introduce complexities and challenges.