The one percent only tells you anything if it is a random sample. Once users are pinned to a datacenter, it isn’t.
Your canary works because of a property nobody states out loud. The one percent of users you route to the new version is a random sample of the other ninety-nine.
That is the whole load-bearing assumption. Not the percentages. Not the gradual ramp. The users you hurt first are statistically indistinguishable from the users you have not hurt yet, which is what lets you turn „the canary looks fine“ into „the full rollout will be fine.“ You are doing inference, and the sample is what makes the inference legal.
Now put your app in four regions.
You cannot pick a random one percent of global users anymore, because a user in Frankfurt is served by the Frankfurt deployment. Users are not a pool you sample from. They are pinned to infrastructure by geography, and geography is exactly what makes them different from each other. Different data. Different regional providers. Different traffic shape, because your European peak is your American 3am. Different regulatory paths through your code. Maybe a different database topology, if you partitioned instead of replicating.
Your one percent is not a sample. It is a subgroup.
The reflex at this point is to deploy to all four regions at once, from the same pipeline, because CI already knows how. That is the fastest way to break all four. There are two ways to get the safety back instead, they defend against different failures, and you need both.
how you got here without noticing
Nobody chooses this. It arrives.
You start in one region because that is where your customers are. Say us-east-1. It works, and it works for longer than you expect, because a single region is genuinely the easiest way to run software. One database. One place to look when something is on fire. One deploy.
Then two things land in the same quarter. A customer in Frankfurt asks where their data physically lives and will not accept „us-east-1“ as an answer. And your Singapore users start describing the app as sluggish, which is the polite version of what they mean, because every click they make crosses an ocean twice before anything renders.
So you go multi-region. Europe, Asia, another US region. And somewhere around the second or third deploy, the thing you were relying on to catch bad releases stopped working. Not broken, not misconfigured. Gone by construction, and no amount of tuning your rollout config brings it back.
The canary technique itself survives. Run it inside one region and it still works, because within a region your users really are a sample of that region’s users. What you lost is the ability to apply it globally in one shot. Your canary was never measuring the code in isolation. It was measuring the code against one environment, and when you had one environment that was the same thing. Now you have four, and passing in one of them proves the code works there.
four copies of one region
I understand the appeal of the fan-out, and I want to be clear about what it costs. You have just built the most efficient global outage delivery mechanism available to a company of your size.
Single-region, a bad deploy is bounded. Your app is down, which is terrible, and there is a real chance you catch it in the ramp before most users see it. Fan out in parallel and a bad deploy is down everywhere at once, with every one of your carefully separated failure domains failing at the same instant because you personally connected them with a pipeline.
Everything you spent that migration quarter on, the isolation, the blast radius reduction, the entire argument you made to your CEO about resilience, gets undone by one YAML file with four targets in it. Regional isolation is not a property of your infrastructure diagram. It is a property of your deploy process, and if your deploy process treats all regions as one unit, you do not have regional isolation. You have four copies of one region.
There is a version of this that is defensible, and it is worth naming so you can check whether you are in it. If your service is stateless, your regions run identical config against the same schema, and you can roll back globally in the time it takes to notice, then fan-out is a reasonable bet. You are trading a small blast radius for a fast one, and that is a real trade, not a mistake.
The trouble is that most teams believe they are in that case and are not. The regional differences that break you are the ones nobody wrote down. So here is the honest test: name what differs between your regions right now, without opening a terminal. If you cannot, you are not in the defensible case.
Otherwise, no parallel fan-out. Which means the fix has to separate two things you have been calling by one name.
deploy is a regional problem, release is a user problem
Most of the confusion I see about this topic is people using „deploy“ to mean both things in the same sentence. Getting a binary onto machines and exposing a behavior to users are different acts with different failure modes, and once you separate the words, the entire strategy falls out of the definitions.
Cohorts protect you from the change being bad in a place. Bad interaction with a regional dependency, config that was wrong in one environment, a database that behaves differently under the load it actually sees in Singapore. Regional problems need a regional rollout, and no flag catches them, because the flag is on in all four regions equally.
Flags protect you from the change being bad for people. The feature works fine, deploys clean everywhere, and then real users touch it and the conversion rate drops or the enterprise tier hits a code path nobody modeled. Region has nothing to do with it, so a cohort rollout sails through and tells you nothing.
Run cohorts on the binary and flags on the behavior. That is the answer, and the rest of this is how each half works and what it costs you.
ramp regions, not users
The first half keeps the shape of the canary and changes what you are ramping.
You stop ramping percentages of users and start ramping regions. A cohort is one batch of regions that gets the new version together, and the point is that you finish one batch, completely, before you start the next.
Cohort one is a single region. One. Deploy the new version there, run your canary inside it exactly as you always did, one percent to fifty percent, watch your metrics, run your tests. That inner canary is still valid, because within a region your users are a real sample of that region’s users.
Region green? Cohort two is a different region. Same treatment, full inner canary, full soak. Then cohort three is maybe two regions together, cohort four is three or four, and you widen as your confidence grows, because at that point the version has survived multiple independent environments and the marginal information from the next one is smaller.
If a cohort fails, you stop. Not „stop and think about whether to proceed.“ Stop. The remaining regions stay on the old version, which is the version you know works, and they keep serving customers while you fix the problem and start the whole sequence over from cohort one with a new build.
That halt is the entire point of the design. Every other part is scheduling.
And the honest limitation, which follows directly from the same sampling argument: cohorts protect you from bad code, but not from bad code that only breaks in the region you deploy to last. You are sampling environments now, and four is a small sample.
the flag gives your sample back and charges you for it
The second half gives up on staging the deploy at all.
Ship the new version everywhere. All regions, no cohorts, no drama. It is safe to do this because the new feature is behind a flag that is off, so what you actually rolled out was a binary that behaves identically to the one it replaced. Then, separately, you turn the feature on. One percent of users. Then five. Then more.
Here is what makes this work for the multi-region problem. Your flag targeting does not care about geography. You can enable a feature for one percent of users worldwide, or for internal accounts everywhere, or for one enterprise customer who agreed to be your guinea pig. You got your random sample back. The thing regions took away, flags hand back, because the flag evaluates per request and does not care which datacenter the request landed in.
The bill comes due somewhere else. Every flag is a branch in production, and n flags is 2^n possible states of your application, most of which nobody has ever run. Flags that were supposed to live for two weeks are still there fourteen months later, and now they interact, and you get the bug that only appears when flag A is on and flag B is off for users in one plan tier. Your test matrix is not a matrix anymore, it is a cloud. The rollout is safe. Your codebase is where the mess accumulates.
The flag system itself also has to be boringly reliable, because it now sits in the request path of every user you have. If the flag service goes down, you had better fail closed to the old behavior, and you had better have tested that path, because the day you find out your fallback is broken is the day the flag service is already down.
And flags cannot cover everything you will want to ship. A schema change is not flaggable. A dependency upgrade is not flaggable. A rewrite of your request routing is technically flaggable and you will regret it. Anything that changes the shape of the binary rather than the behavior of a code path still needs a staged rollout, which means flags do not let you delete your deploy strategy. They sit on top of one.
Which is also why the two mechanisms should not be tangled together. Roll the binary through cohorts with everything dark, get it everywhere, then start the flag ramp as a separate decision on a separate timeline. If you try to coordinate „cohort three plus flag at twenty percent,“ you have built a state machine you will not be able to reason about during an incident.
split one region out this month
If you are already multi-region and fanning out in parallel, that is the thing to fix. Split one region out of your pipeline and make it cohort one. You do not need a platform team or a new tool. You need your deploy to stop treating four failure domains as one.
Order the cohorts deliberately after that. Smallest blast radius first is the boring answer and mostly the right one, though there is a real argument for putting your most architecturally weird region early, the one with the different database setup or the compliance-specific code path, because that is where a version dies if it is going to die, and you would rather learn it in cohort one than cohort four.
The cost is honest and it is time. A change that used to reach all users in an afternoon now takes days, because each cohort needs a real soak window, and a real soak window means long enough to see your traffic pattern come around. For a daily-cycle app that means a full day, and there is no way to shorten it without giving up exactly what you are buying.
Slow rollouts also mean three versions of your service are live at once. That is the same backward compatibility discipline you already need for zero-downtime deploys, expand-then-contract migrations and all, except the window is now measured in days instead of minutes. If your team has been getting away with sloppiness there, cohorts will find it.
If you are still single-region and thinking about expanding, the sequencing is the other way around. Get feature flags working first, while you still have one place to debug them. Retrofitting a flag system across four regions during a live incident is a bad week. And decide your data story before your deploy story, because whether you replicated, partitioned by geography, or wired up change data capture determines how much your regions differ from each other, and how much they differ determines how paranoid your cohorts need to be.
If a two-day rollout sounds intolerable, sit with why. It usually means something upstream is broken, batches that are too big, a test suite you do not trust, or a habit of shipping fixes for the last deploy. Those are worth fixing on their own. The rollout speed is a symptom.
You will hear that the big companies work this way, cohort-style rollouts attributed to one set of names and flag-driven releases to another. Treat that as directionally true and not as evidence, because the argument does not rest on it. Design from the property you lost, not from the org chart of a company you do not work at.
And the property you lost is specific. A canary is an inference, and the moment your users stopped being interchangeable, the inference stopped being valid. You rebuild the sample in two places or you do not rebuild it at all: per-region for the binary, per-user for the behavior.
Here is the uncomfortable version. Multi-region does not make your system more reliable. It gives you the parts to build something more reliable, and hands you a much better way to break everything at once in the meantime. Your deploy pipeline decides which one you get, not your architecture diagram.