Since Google open sourced Kubernetes in 2014, it has become one of the most popular open source projects. Adopted by all major cloud providers, Kubernetes has undoubtedly become the de facto container orchestrator. But what is Kubernetes and container orchestration really about? Without the technical background and knowledge about the technology developments that preceded it, it can be challenging to wrap your head around it.

Almost every Kubernetes tutorial speaks about how to quickly deploy a container in a pod and expose it as a service. Mostly these tutorials focus on stateless services and ignore a deeper explanation of state management in Kubernetes. But Kubernetes supports both types of deployments, the stateless deployments and stateful deployments and they have somewhat different operational requirements.

This is the first in a series of tutorials on setting up a secure production-grade CI/CD pipeline. We’ll use Kublr to manage our Kubernetes cluster, Jenkins, Nexus, and your cloud provider of choice or a co-located provider with bare metal servers. A common goal of SRE and DevOps practitioners is to enable development and QA teams. We developed a list of tools and best practices that will allow them to iterate quickly, get instant feedback on their builds and failures, and experiment.

Not only does it deploy and manage containers, Kubernetes autoscaling enables users to automatically scale the overall solution in numerous ways. This is a tremendous asset, especially in the modern cloud, where costs are based on the resources consumed. Not only does Kubernetes have the capacity to deploy and manage containers, it can also automatically scale the overall solution in numerous ways.

Kublr and Kubernetes can help make your favorite data science tools easier to deploy and manage. Hadoop Distributed File System (HDFS) carries the burden of storing big data; Spark provides many powerful tools to process data; while Jupyter Notebook is the de facto standard UI to dynamically manage the queries and visualization of results.

Once you get your head around the concept of containers, and subsequently the need for management and orchestration with tools like Kubernetes, what started off as a weekend project suddenly starts to raise more questions than answers. Kubernetes removes much of the complexity of managing the interaction between applications and the underlying infrastructure. It is designed to let developers focus on the applications and solutions rather than worrying about the complexity of the hosting platform.