Setting a Hydration Performance Budget in CI

A hydration number that nobody enforces becomes a historical artefact within two sprints. The fix is a build that fails, but only on metrics a build can measure honestly. This walkthrough sets up that gate, using the instrumentation from measuring hydration and interactivity metrics as its input.

Prerequisites

Implementation Steps

Gate These, Watch Those Transferred bytes per island and per route are deterministic and belong in CI. Blocking time on a fixed route is measurable in CI with a median and a variance gate. Field percentiles for interaction latency and hydration delta cannot be attributed to a commit and belong on a dashboard, where they set the thresholds the CI gate enforces. Two lists, and the arrow runs one way GATE IN CI bytes per island module — deterministic bytes per route — deterministic blocking time, median of 5 — needs care attributable to one commit, available in minutes WATCH IN FIELD interaction latency at p75 hydration delta distribution activation failures per session sets the thresholds; judges whether the work mattered

Step 1 — Emit the numbers as build output 

// build/report-islands.mjs — runs after the bundle, writes one JSON file.
import { readdirSync, statSync, writeFileSync, readFileSync } from 'node:fs';
import { gzipSync } from 'node:zlib';

const dir = 'dist/js/islands';
const sizes = Object.fromEntries(readdirSync(dir).map((f) => {
  const name = f.replace(/\.[a-f0-9]{8}\.js$/, '');   // strip the content hash
  return [name, gzipSync(readFileSync(`${dir}/${f}`)).length];
}));
writeFileSync('dist/island-sizes.json', JSON.stringify(sizes, null, 2));

Step 2 — Compare against a committed budget 

// build/check-budget.mjs — the whole gate.
const budget = JSON.parse(readFileSync('island-budgets.json', 'utf8'));
const actual = JSON.parse(readFileSync('dist/island-sizes.json', 'utf8'));

const failures = Object.entries(actual)
  .filter(([name, size]) => budget[name] != null && size > budget[name])
  .map(([name, size]) => `${name}: ${size} B exceeds budget ${budget[name]} B ` +
                         `(+${size - budget[name]} B)`);

const unbudgeted = Object.keys(actual).filter((n) => budget[n] == null);
if (unbudgeted.length) failures.push(`no budget set for: ${unbudgeted.join(', ')}`);

if (failures.length) { console.error(failures.join('\n')); process.exit(1); }
console.log(`✓ ${Object.keys(actual).length} island budgets respected`);

The unbudgeted check matters as much as the comparison: a new island with no budget is exactly the thing that would otherwise slip through, and requiring a budget line makes adding one a deliberate act.

Step 3 — Add the timing gate carefully, or not at all

Timing in CI is noisy. If you gate on blocking time, take a median of five runs, discard a run set whose interquartile range exceeds fifteen per cent of the median, and set the threshold with real headroom. Many teams are better served by gating only on bytes and watching timing on a dashboard, because bytes are the causal variable anyway.

Step 4 — Make the failure message do the work

Name the island, the old value, the new value, the budget and the delta. A message that says only “performance budget exceeded” produces a re-run; a message that says “island search: 11 208 B exceeds budget 12 000 B” produces a fix.

Verification

Testing the Gate Itself A budget check can be wrong in three ways: it can pass when it should fail, fail when nothing changed, or measure something other than what ships. Each is tested directly by inducing the condition — adding a large import, re-running with no changes, and comparing the reported size against the file the browser actually downloads. FAILURE OF THE GATE TEST passes when it should fail add a large import on a branch; the build must fail fails when nothing changed re-run on the same commit ten times; ten passes measures the wrong artefact compare against the bytes the browser downloads

The third test catches a subtle and common error: reporting the uncompressed size while the browser downloads a compressed one, or measuring an intermediate build artefact rather than the file served by the CDN. Load the page, read the transferred size from the network panel, and confirm it matches the number in the report within a few per cent. A mismatch means the budget is enforcing a number nobody experiences.

Once the gate is trusted, add its output to the pull request as a comment: a short table of islands whose size changed, with deltas. Most changes will show nothing, which is the point — the signal is visible precisely because it is rare.

Choosing Thresholds That Hold

Budgets set at exactly the current value fail on the next legitimate change and get raised reflexively until nobody reads them. Budgets set far above the current value never fire. The useful setting is derived rather than chosen.

Start from the field. If interaction latency at the seventy-fifth percentile is currently acceptable and the route ships forty kilobytes of island code, then forty kilobytes is a value that produces acceptable results on real devices. Set the route budget slightly above it — enough headroom for one ordinary feature — and distribute it across islands in proportion to their current sizes.

Revisit the distribution rather than the total when a new island is added. A route budget that stays fixed while its islands are rebalanced is a stronger constraint than per-island budgets that each drift upward, because it forces the trade to be explicit: a new eight-kilobyte island means eight kilobytes have to come from somewhere. That conversation is the whole value of the mechanism, and it is the same discipline described in islands performance optimization applied at the point where code is written rather than at review time.

Budgets Belong in the Repository A committed budget file makes raising a threshold a reviewable diff. The same number stored in a dashboard drifts upward without anyone deciding to allow it. Where the number lives decides whether it holds in a dashboard changed by whoever notices the alert; no record of why in a committed file raising it is a diff with a reason, reviewed like any change the review conversation is the mechanism, not the number

Troubleshooting

Should CI gate on field metrics?

No — field data describes visitors, arrives days late and cannot be attributed to a pull request. Gate on lab metrics that respond immediately and deterministically to a code change: bytes per island module and blocking time on a fixed route. Use field data to set the thresholds and to judge whether the work mattered, which is the division of labour that keeps both useful.

How do I stop the check being flaky?

Measure bytes wherever possible, because they are deterministic; a byte budget never flakes. For timing metrics, take a median of at least five runs on a machine class you pin, add a variance gate that discards a run set whose spread is too wide, and set the threshold with headroom rather than at the current value. A check that fails randomly is worse than no check, because the team learns to re-run it without looking.

What happens when a regression is intentional?

The budget file changes in the same pull request, with the reason in the commit message. That is the entire point of committing budgets rather than storing them in a dashboard: raising one becomes a reviewable decision instead of an invisible drift. A reviewer seeing "island search: 11 KB to 19 KB — adds offline suggestions" can agree or push back, which is a conversation that otherwise never happens.

← Back to Measuring Hydration and Interactivity Metrics