Many companies and organizations have adopted CI/CD processes in order to help deliver applications running on Kubernetes quickly, transparently, and with automated tests. While this is a desirable goal, it gets more complex when developing a management layer on top of k8s, especially when both images and Helm charts are involved.

In developing IBM Cloud Private, we have implemented a CI/CD process that automates promotion through a series of quality gates where we 1) Ensure that all charts (43) and images (135) are in sync across three supported architectures, 2) Deploy instances of ICP clusters with different topologies, management services, and infrastructure, and 3) Run automated and manual functional and security regression tests against those clusters. Since implementation, we have been able to iterate more quickly by discovering issues earlier in the development process.