Performance tests: adaptive stats, thresholds, perf flags, duration fix (#6) #20

orchestrator · 2026-10-12T20:24:38Z

orchestrator commented

2026-10-12 20:24:38 +00:00

Per #6.

Fixed the total wallclock display (testrunnerview.tsx summed ms but rendered s).
assert.medianRuntimeLessThan now uses a Criterion-style adaptive stopping rule (relative-error/CI target, min/max runs, time budget) and reports a CI; legacy call shape preserved.
New perf threshold registry (perf.setThreshold(name, ms), perf.defaultThreshold) so budgets live in one place (per-file scope, documented).
Headless launches chromium with --js-flags=--expose-gc --enable-precise-memory-info automatically.
Validation of tuning params; docs + tests.

Verification

Self-test 32/32 on chromium + firefox (+ webkit in integration); perf flags confirmed via a Playwright probe.

Closes #6

Base: integration/wave2 is the integration point this branch was based on, so this PR's diff shows only this issue's changes.

Per #6. - Fixed the total wallclock display (`testrunnerview.tsx` summed ms but rendered `s`). - `assert.medianRuntimeLessThan` now uses a **Criterion-style adaptive stopping rule** (relative-error/CI target, min/max runs, time budget) and reports a CI; legacy call shape preserved. - New `perf` threshold registry (`perf.setThreshold(name, ms)`, `perf.defaultThreshold`) so budgets live in one place (per-file scope, documented). - Headless launches chromium with `--js-flags=--expose-gc --enable-precise-memory-info` automatically. - Validation of tuning params; docs + tests. ## Verification - Self-test 32/32 on chromium + firefox (+ webkit in integration); perf flags confirmed via a Playwright probe. Closes #6 > **Base:** `integration/wave2` is the integration point this branch was based on, so this PR's diff shows **only this issue's changes**.

orchestrator added 4 commits

2026-10-12 20:24:38 +00:00

testrunner: fix total wallclock duration units e3b2f99549

The run summary summed per-test durations (milliseconds) but rendered
them as `Time: <n>.000s` — a 300ms run showed as 0.300s. Reuse the
shared millis() formatter so the total shows in ms, consistent with the
per-test durations.

headless: launch chromium with perf flags automatically dd40c8123e

medianRuntimeLessThan needs --expose-gc (to force GC between samples) and
--enable-precise-memory-info (for accurate performance.memory). Set both
automatically when driving chromium, merging with --remote-debugging-port
instead of replacing it. Firefox/WebKit are left untouched.

perf: adaptive median with CI stopping rule + named thresholds f85511274e

Replace the fixed numRuns=29 median with an adaptive measurement: keep
sampling until the median's confidence interval is tight enough
(halfWidth/|median| <= targetRelError), bounded by minRuns/maxRuns and a
wallclock budget. Report the estimate and its error (e.g. 95% CI) as
test-info so results are interpretable.

Add a small, documented thresholds API (perf.setThreshold / named budgets
/ defaultThreshold) usable as assert.medianRuntimeLessThan('name', ...),
plus an options overload. The legacy call shape still works. Stats helpers
(mean/stddev/inverse-normal/median CI) live in shared/utils (browser-safe).
Document both in the README and cover with tests.

perf: address review — scope docs, validation, finite CI 38742f8589

- Document that the perf registry is per test-file (per iframe realm) in
  both the README and perf.ts, since a module singleton cannot be shared
  across realms (globalThis would not help).
- Validate confidence in (0,1), finite targetRelError > 0, finite
  minRuns/maxRuns, and guard the legacy numRuns (reject Infinity) so a
  high-variance measurement can never loop unbounded.
- Report 'CI unavailable (n=N < 2)' instead of \u00b1Infinityms for
  numRuns: 1.
- Doc nits: 'Wallet-clock' -> 'Wall-clock'; note timeBudgetMs starts
  before the 500ms warm-up; type perf with PerfConfig so the export is used.
- README: third-argument numRuns (not fourth).

sufian reviewed

2026-10-13 19:54:45 +00:00

README.md

					
				@ -383,0 +429,4 @@

				});

				```

				#### Named thresholds

This is overly prescriptive, I think we can drop this feature. If users want this, they can easily write a wrapper function around assert.medianRuntimeLessThan

This is overly prescriptive, I think we can drop this feature. If users want this, they can easily write a wrapper function around `assert.medianRuntimeLessThan`

This pull request can be merged automatically.

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin wip/issue-06:wip/issue-06

git switch wip/issue-06

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch integration/wave2

git merge --no-ff wip/issue-06

git switch wip/issue-06

git rebase integration/wave2

git switch integration/wave2

git merge --ff-only wip/issue-06

git switch wip/issue-06

git rebase integration/wave2

git switch integration/wave2

git merge --no-ff wip/issue-06

git switch integration/wave2

git merge --squash wip/issue-06

git switch integration/wave2

git merge --ff-only wip/issue-06

git switch integration/wave2

git merge wip/issue-06

git push origin integration/wave2

Sign in to join this conversation.

No reviewers

No labels

No milestone

No project

No assignees

2 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

legend/gooey-test!20

Rows
Columns