The failure mode
Copying a working repository is the fastest way to launch a second region, and the slowest way to run one. Each copy starts identical, then absorbs a local fix — a different ingress class here, a tighter memory limit there — and none of those fixes travel back. After a while nobody can answer the simple question: is this service configured the same way everywhere?
If the answer to "why is production different here?" is "history", the layout is wrong, not the people.
One contract, many values
The rule we settled on is that a region may differ in configuration, never in structure. Every regional repository has the same directory contract, so a person or a script that understands one understands all of them.
apps/
checkout/
base/ # chart reference + shared values
Chart.yaml
values.yaml
envs/
staging.yaml # overrides only
production.yaml
search/
base/
envs/
platform/
ingress-nginx/
cert-manager/
external-secrets/
base/values.yaml holds everything that should be true everywhere: probes,
security context, the shape of the deployment. The files under envs/ hold only
what genuinely varies — replica counts, resource sizing, hostnames, feature flags. A review
is then easy to reason about, because the diff is the difference.
The control plane stays singular
A separate Argo CD per region sounds like isolation, but in practice it means four upgrade windows, four RBAC configurations and four places to look during an incident. We run one management cluster whose Argo CD registers the regional clusters as targets, with an app-of-apps root per region.
| Concern | Per-region Argo CD | Single control plane |
|---|---|---|
| Upgrades | Four times the work | One window |
| Blast radius | Smaller | Needs sync windows and projects |
| Visibility | Four dashboards | One view of every region |
| Access control | Duplicated | Projects scoped per team and region |
The honest trade-off is blast radius: one control plane is one thing to lose. We mitigate it two ways — the control plane itself is declarative and can be rebuilt from Git in minutes, and regional clusters keep running whatever they last synced if it goes away. A control plane outage stops deployments; it does not stop traffic.
Sync windows are not optional
Regions have different peak hours. Left alone, a merge at 10:00 in one time zone lands in the middle of another region's busiest period. Argo CD sync windows let each region declare when automated syncs may run, which turns a scheduling argument into a config file:
spec:
syncWindows:
- kind: deny
schedule: "0 11-14 * * 1-5" # regional peak, weekdays
duration: 4h
applications:
- "*"
manualSync: true # humans can still override
Promotion is a pull request
Nothing deploys because someone clicked a button. A successful build writes a new image tag
into the staging values file; promotion to production is a pull request that moves that tag
forward. The benefits are unremarkable and exactly what you want: an audit trail, a review,
a revert that is one git revert away.
What I would do differently
- Set the directory contract before the second region, not after the fourth. Retrofitting a layout is a month of careful, invisible work.
- Fail the pipeline on structural drift. A short CI check that asserts every service has the expected files catches divergence on day one.
- Keep platform components in the same repository as the apps. Splitting them felt tidy and only made ordering problems harder to see.
Where it landed
Onboarding a service to a new region is now a values file and a review, measured in minutes rather than a day of copy-paste. More importantly, the question that started this — is this configured the same way everywhere? — is answered by reading one directory.
Working on something similar? I am always happy to compare repository layouts — hello@haongo.dev.