Channels and the release pipeline

A channel is a set of devices that should run the same release. Channels chain, and a release moves down the chain only when the channel it is leaving can prove it is safe to leave. That proof is a gate, and it is the whole idea.

The pipeline

A channel with no upstream — dev — is given releases directly: you pick a distribution set and it goes out. A channel with an upstream — beta naming dev, prod naming beta — cannot be given one. It can only receive a release by promotion from its upstream.

dev  ──promote──▶  beta  ──promote──▶  prod
 │                  │                   │
 no upstream        gate                gate + approval
 given directly

That restriction is deliberate and it is what makes the pipeline mean anything. If you could hand prod a set directly, the gate would be decoration.

Getting devices into a channel

Three ways, and they are checked in this order:

1. Its centre. If a device's centre is in a channel, the device is in that channel. Nothing overrides this — see Centres.
2. The channel's rule. A query such as attribute.ring==beta. A device that matches it and is in no channel yet joins by itself. It never steals a device from another channel — adoption is for the unclaimed.
3. By hand. Pick devices and put them in.

Gates and approvals

A gate is four conditions on the upstream, plus one more when the release carries a manifest. All of them must hold before the release may be promoted:

Condition	Asks	Stops you from
minDevices	At least this many devices of the upstream run it.	promoting on the evidence of three machines.
minSuccess	At least this percentage of the upstream runs it.	promoting while a fifth of beta is still behind.
soakMinutes	It has been in the upstream this long.	promoting a bug that takes an hour to show.
not halted	The upstream's release has not halted on failures.	promoting something that is already going wrong.
orchestrator done	The upstream's systems finished too.	promoting a release that never actually reached half the fleet.

You can ask the gate without promoting anything. It answers line by line, with the real numbers against the required ones — so "it is not ready" is never a mystery:

curl -u admin:$PW "$Q/qawk/v1/fleets/$PROD/gate?from=$BETA"

✓ app:1.2.0 in beta is not halted
✓ 131 devices of beta run app:1.2.0 (at least 20)
✗ 94% of beta runs it (at least 95%)
✓ it has been in beta for 180 minutes (at least 120)
✓ the orchestrator took beta's systems with device-system-2.0: finished

Forcing a closed gate

Sometimes you have to ship anyway. Forcing needs the APPROVE_ROLLOUT permission and a written reason, and the reason is kept in the release's history and the audit log permanently.

curl -u admin:$PW -X POST -H 'Content-Type: application/json' \
  -d '{"from": '$BETA', "force": true, "reason": "hotfix for the scoring crash"}' \
  $Q/qawk/v1/fleets/$PROD/promote

Approvals: four eyes

A channel with gate.approvalRequired does not release on promotion. The release goes to waiting_for_approval and sits in the queue until someone with APPROVE_ROLLOUT decides. Not the person who asked — Qawk refuses that, and it refuses it for the administrator too.

curl -u admin:$PW $Q/qawk/v1/releases?status=waiting_for_approval
curl -u someone-else:$PW2 -X POST -H 'Content-Type: application/json' \
  -d '{"note": "checked with the centre manager"}' $Q/qawk/v1/releases/44/approve

Promoting by itself

A channel with autoPromote promotes as soon as its gate opens, through the same gate, the same approval and the same audit trail a person would go through — it just does not wait for a person to notice. It asks once per release of its upstream, so a promotion already asked for, approved or denied is not asked for again.

By default this is off. Most fleets want someone to look at the gate and decide.

Waves and error thresholds

Two independent safety mechanisms. One limits how fast, the other stops entirely.

Waves — how fast

wavePercent is the share of the channel's members given the release at a time. 0 means all at once. The next wave starts when the current one is done, or after waveTimeoutMinutes if some devices never answered — a switched-off machine must not hold up the fleet for ever.

With wavePercent: 25, a channel of 400 goes out in four waves of 100. If 100 devices are all offline, the fifth wave still starts an hour later.

Error threshold — when to stop

errorThreshold is the percentage of started devices that may fail before the release halts. Halting means no further device is sent it. The devices that already have it keep it; the ones under way finish.

halted: 12 of 140 devices failed (8%), over the 5% threshold

Resuming marks the failures already counted as seen, so only new failures can halt it again. Without that, resuming would halt again within ten seconds on the same devices, and you would press the button for ever.

Freezes and maintenance

A freeze stops every release reaching a channel, with a reason that is shown to anyone who tries. It can start and end at set times, so a change window can be arranged in advance:

curl -u admin:$PW -X PUT -H 'Content-Type: application/json' \
  -d '{"reason": "league nights, no updates until Monday", "until": 1759000000000}' \
  $Q/qawk/v1/fleets/$PROD/freeze

curl -u admin:$PW -X DELETE $Q/qawk/v1/fleets/$PROD/freeze

Maintenance windows, which are different

A freeze is about a channel. A maintenance window is about a device: it downloads whenever, but installs only inside a window — a cron expression, a duration and a time-zone offset, sent with the deployment. A device outside its window answers that it is skipping, and installs when the window opens.

Temporary channels

A machine goes to a trade show. It needs the demo build, not prod's, and it has to go home afterwards with nobody remembering which channel it came from.

Mark a channel temporary. Devices lent to it remember where they came from, and return sends them back — where they get that channel's release again automatically.

curl -u admin:$PW -X PUT -H 'Content-Type: application/json' \
  -d '["expo-demo-01"]' $Q/qawk/v1/fleets/$EXPO/targets

curl -u admin:$PW -X POST -H 'Content-Type: application/json' \
  -d '{}' $Q/qawk/v1/fleets/$EXPO/return

A temporary channel is also the only way to move a device out of its centre's channel by hand. A centre is never put in one: a centre is a place, and places do not go to trade shows.

What a release actually is

A release carries two things:

• a distribution set, for the devices that stand alone;
• optionally a manifest, for the channel's systems — because a lane computer and the terminal under it cannot share one set.

Both travel together when the release is promoted. "orchestrator": false in the promotion leaves the manifest behind, if you want the set to move on but the systems to stay where they are.

And a release is complete only when both halves are done. Until then the channel says so, and the next gate will not open — which is how a release cannot be promoted out of a channel it never fully reached.

Doing all of this from a script

Every button above is an endpoint. See the Qawk API — create a channel, ask the gate, promote, approve, freeze — each with the call in curl, Python, JavaScript, Go and PowerShell.