benchmark_baseline.ts

comparison options including regression threshold and staleness warning

comparison result from benchmark_baseline_compare

comparison result from benchmark_baseline_compare

Minimum speedup ratio to consider a regression. For example, 1.05 means only flag regressions that are 5% or more slower. Default: 1.0 (any statistically significant slowdown is a regression)

Number of days after which to warn about stale baseline. Default: undefined (no staleness warning)

Minimum percentage difference to consider meaningful, as a ratio. Passed through to benchmark_stats_compare. See BenchmarkCompareOptions. Default: 0.10 (10%)

Coefficient of variation (std_dev / mean) at or above which a comparison is flagged with noise_warning: true. Doesn't affect bucketing — the task still routes to regressions / improvements / unchanged as the Welch math dictates — but tells the formatter (and any custom consumer) that the underlying measurement is noisy enough that a "significant" call should be read with skepticism. The default is calibrated for system-noise/thermal/background-load contamination, not microbenchmark floor effects; lower it (e.g. 0.15) for sub-microsecond functions where any cv signal matters, raise it for inherently noisy workloads where 0.3 fires on every run. Default: 0.30

Outlier ratio (fraction of iterations rejected by MAD outlier removal) at or above which the comparison is flagged with noise_warning: true, OR-gated with the cv check. Catches the case where the post-cleaning cv looks tight because outlier removal already deflated the spread — a third of the iterations being tail events is itself a noise signal, even if the surviving samples cluster cleanly. Set higher (e.g. 0.2) for benchmarks where outliers are expected (allocator-bound, async I/O), lower (e.g. 0.05) for tight CPU loops. Default: 0.10

Whether a baseline was found

Timestamp of the baseline

Git commit of the baseline

Age of the baseline in days

Whether the baseline is considered stale based on staleness_warning_days option

Node.js version recorded in the baseline (null if no baseline).

Node.js version of the process that produced the current results.

True if the baseline's node version differs from the current process. Informational only — surfaced in the comparison header so readers can apply the caveat across all task comparisons; does not affect per-task classification (Node bumps affect every task uniformly).

The metadata bag persisted in the baseline file, or null if the baseline didn't have one (or no baseline was found). Returned verbatim — fuz_util doesn't validate the shape or diff it against any "current run" metadata. If the consumer needs a diff, they pass options.metadata (current-run context) and walk the two records themselves.

Individual task comparisons for tasks where Welch's math is meaningful — i.e., budget unchanged between baseline and current. Methodology-changed tasks are NOT in here; they live in methodology_changed. Excluding them here keeps aggregate reads of c.comparison.faster / c.comparison.speedup_ratio / c.comparison.recommendation safe — those fields are still computed for methodology-changed rows but answer a counterfactual question (the math is correct over mismatched sample sizes) and would mislead consumers iterating this array.

Tasks that regressed (slower with statistical significance), sorted by effect size (largest first)

Tasks that improved (faster with statistical significance), sorted by effect size (largest first)

Tasks with no significant change

Tasks whose effective budget differs between baseline and current. Excluded from regressions/improvements/unchanged because the Welch comparison is contaminated by sample-size or warmup differences — the math is correct but answers a different question than the reader thinks. Re-save the baseline after intentional methodology changes to surface any genuine drift that was masked.

Tasks in current run but not in baseline

Tasks in baseline but not in current run

Directory to load baseline from (default: '.gro/benchmarks')

Directory to store baselines (default: '.gro/benchmarks')

Git commit hash (auto-detected if not provided)

Git branch name (auto-detected if not provided)

Opt-in passthrough for run-level context (corpus identity, dependency versions, binary sizes, hardware notes). Round-trips on _load and is accessible via BenchmarkBaselineComparisonResult.baseline_metadata, but is *not* interpreted by _compare — consumers decide what to do with mismatches. Keep values JSON-serializable; this is written verbatim to the baseline file.

Welch comparison of baseline vs. current means.

When the row is in methodology_changed, every field of this object is contaminated by sample-size or warmup differences — not just recommendation. The Welch math runs end-to-end and populates faster, speedup_ratio, significant, p_value, percent_difference, effect_size, effect_magnitude, ci_overlap, and recommendation, but it's comparing distributions produced under different methodologies. Treat the result as diagnostic ("how dramatic is the contamination?"), not authoritative.

The built-in formatters (benchmark_baseline_format, benchmark_baseline_format_json) never render comparison.* for methodology-changed rows for this reason — they show only the budget diff. Custom consumers reading this field directly should check methodology_changed first.

True if the effective time budget differs between baseline and current. When set, the task is routed to methodology_changed instead of regressions/improvements/unchanged, and the comparison field's contents become unreliable per the doc above.

True when measurement noise is high enough to undermine the significance call on this row. OR-gated across two signals: - max(baseline.cv, current.cv) >= noise_warning_cv_threshold where cv = std_dev_ns / mean_ns (post-outlier-removal). - `max(baseline.outlier_ratio, current.outlier_ratio) >= noise_warning_outlier_ratio_threshold`, since outlier removal deflates cv — a high outlier ratio is itself a noise signal even when the cleaned cv looks tight. Independent of methodology_changed. The Welch math still ran; this flag tells the reader to take comparison.significant with skepticism on this row. Cv is recomputed at comparison time from persisted mean_ns/std_dev_ns; outlier_ratio is persisted directly.

The larger of the two coefficients of variation across baseline and current, exposed so consumers can render the actual noise level. Recomputed at comparison time from persisted mean_ns and std_dev_ns — not a persisted field.

The larger of the two outlier ratios across baseline and current. Read directly from the persisted entries. Exposed so consumers can render the actual outlier rate alongside noise_warning.