Engineering1 min read

You cannot test every flag combination. Test these instead

Ten boolean flags make 1,024 combinations, and you will not test them all. You do not need to. Test the defaults, the targets, and the pairs that touch the same code.

Jordan Hale

Jordan Hale

Co-founder & CTO ·

Engineering cover: You cannot test every flag combination. Test these instead

Combinatorics is undefeated: 2^n grows faster than any CI budget. The way out is not more tests, it is choosing which combinations can actually happen and actually interact.

What to test

  1. Everything off. This is what every new user, every outage of the flag service and every forgotten default looks like.
  2. Everything at its production value. Snapshot today’s flags into the test run, so CI tests what users get.
  3. Each flag flipped alone against the production snapshot. Linear, not exponential.
  4. Pairs that share code. If two flags are read in the same module, test their four combinations together.

Pin flags in tests

checkout.test.tsTypeScript
// What an edge node receives: a diff against the version it holds.
interface RulesetDiff {
    version: number
    previous: number
    changed: Record<string, FlagRules>
    removed: string[]
}

function apply(diff: RulesetDiff) {
    if (diff.previous !== store.version) return resync() // missed one: fetch a snapshot
    store.merge(diff.changed, diff.removed)
    store.version = diff.version
}

Find the pairs automatically

Driftwire’s code references know which flags each file reads. driftwire flags pairs --changed lists the pairs that share a file touched by your branch, which is the list CI should care about today.

TestingCIEngineering
Jordan Hale

Written by

Jordan Hale

Co-founder & CTO

Jordan co-founded Driftwire after one too many Friday rollbacks. Writes about release engineering, trunk-based development and paying down flag debt.

More posts by Jordan Hale