Breaking Business Applications Deployment with Kustomize - Lessons Learned

Explore a conference talk detailing how a human error in ArgoCD Kustomize configuration led to business application deployment failures at Bank Hapoalim. Learn about the pitfalls and best practices of using Kustomize, and discover the lessons learned from this incident. Gain insights into managing the desired state of multiple OpenShift clusters using GitOps and ArgoCD. Understand the challenges of assessing the impact of Kustomize base/overlay changes on overall configuration. Examine the implementation of a merge request pipeline with validations and approval mechanisms to prevent future mistakes. Discover strategies for protecting a monorepo that affects multiple clusters from potentially harmful changes.

Intro
This Session
Background: GitOps Managed Desired State
Initial Repo Structure - ManyRepo Many Branch
Branch Per Cluster/Environment
Helm
Why We Chose Kustomize
How We Manage ArgoCD Configuration
Overlay kustomization.yaml
What Happened on 13.11.22
Change In Overlay argocd-cm
Tests Before Merge
How We Fixed The Problem
The Problem - Expected vs Actual Final argocd-cm Manifest
The Problem Impact
Overlaying File Contents
Patching Multiline Yaml
Understanding the Impact of a Change
Git Repo Protection
MR Pipeline Implementation
Human Approval for Critical Cluster Change MR
Human Approval for Base Change MR
Impact of a Change - Diffing New and Existing
Aftermath
Summary