benchmark_stats.ts

Benchmark-specific statistical analysis. Uses the general stats utilities from stats.ts for timing/performance analysis. All timing values are in nanoseconds.

view source

Declarations
#

6 declarations

benchmark_stats_compare
#

benchmark_stats.ts view source

(a: BenchmarkStatsComparable, b: BenchmarkStatsComparable, options?: BenchmarkCompareOptions | undefined): BenchmarkComparison

import {benchmark_stats_compare} from '@fuzdev/fuz_util/benchmark_stats.js';

Compare two benchmark results for practical and statistical significance. Uses percentage difference for effect magnitude classification, with Welch's t-test for statistical confidence. Cohen's d is computed as an informational metric but does not drive classification — its thresholds (0.2/0.5/0.8) are calibrated for social science and produce false positives in benchmarking where within-run variance is tight.

`a`

first benchmark stats (or any object with required properties)

type BenchmarkStatsComparable

`b`

second benchmark stats (or any object with required properties)

type BenchmarkStatsComparable

`options?`

type BenchmarkCompareOptions | undefined

optional

returns

BenchmarkComparison

comparison result with significance, effect size, and recommendation

examples

const comparison = benchmark_stats_compare(result_a.stats, result_b.stats);
if (comparison.significant) {
  console.log(`${comparison.faster} is ${comparison.speedup_ratio.toFixed(2)}x faster`);
}

BenchmarkCompareOptions
#

benchmark_stats.ts view source

BenchmarkCompareOptions import type {BenchmarkCompareOptions} from '@fuzdev/fuz_util/benchmark_stats.js';

Options for benchmark comparison.

`alpha?`

Significance level for hypothesis testing (default: 0.05)

type number

`min_percent_difference?`

Minimum percentage difference to consider practically meaningful, as a ratio. Below this threshold, differences are classified as 'negligible' and significant is forced to false, regardless of p-value. This prevents the t-test's oversensitivity at large sample sizes from flagging system-level noise (thermal throttle, OS scheduler, cache pressure) as meaningful differences.

Effect magnitude thresholds scale from this value: negligible < min, small < min*3, medium < min*5, large >= min*5.

Default: 0.10 (10%).

type number

BenchmarkComparison
#

benchmark_stats.ts view source

BenchmarkComparison import type {BenchmarkComparison} from '@fuzdev/fuz_util/benchmark_stats.js';

Result from comparing two benchmark stats.

`faster`

Which benchmark is faster ('a', 'b', or 'equal' if difference is negligible)

type 'a' | 'b' | 'equal'

`speedup_ratio`

How much faster the winner is (e.g., 1.5 means 1.5x faster)

type number

`significant`

Whether the difference is both statistically and practically significant

type boolean

`p_value`

P-value from Welch's t-test (lower = more confident the difference is real)

type number

`percent_difference`

Percentage difference between means as a ratio (0.05 = 5%, 1.0 = 100%)

type number

`effect_size`

Cohen's d effect size (informational — not used for classification)

type number

`effect_magnitude`

Interpretation of practical significance based on percentage difference

type EffectMagnitude

`ci_overlap`

Whether the 95% confidence intervals of the two means overlap.

Informational only — do not use this field to infer significance. Two means with overlapping 95% CIs can still differ significantly at p<0.05 (overlap of up to ~25% of CI width is consistent with significance); conversely, non-overlapping CIs are slightly *stronger* than the standard p<0.05 bar but the relationship is not strict. Use significant and p_value for classification. This field is exposed for consumers who want to render CIs side-by-side and need the visual overlap check.

Note: the underlying CIs are computed with a z-score (1.96) rather than the strict t-score, so at small n the CIs are slightly narrow — overlap is reported less often than a t-based CI would. Effect is ~2-3% at the n=30 floor after Bessel's correction on std_dev_ns.

type boolean

`recommendation`

Human-readable interpretation of the comparison

type string

BenchmarkStats
#

benchmark_stats.ts view source

import {BenchmarkStats} from '@fuzdev/fuz_util/benchmark_stats.js';

Complete statistical analysis of timing measurements. Includes outlier detection, descriptive statistics, and performance metrics. All timing values are in nanoseconds.

Outliers are treated asymmetrically, because high and low outliers in timing data mean different things:

Upper-tail order statistics (max_ns, p75_ns–p99_ns) are computed over the raw valid timings — high outliers (GC pauses, slow paths) are real latency events, so the tail stays honest and p99 reflects real events rather than a pre-stripped distribution.
min_ns is computed over the MAD-cleaned timings — nothing runs faster than its true cost, so a low outlier is an invalid measurement, not a fast run, and reporting it as the best case would mislead.
Central-tendency statistics (mean_ns, std_dev_ns, cv, confidence_interval_ns, ops_per_second) are computed over the MAD-cleaned timings so the Welch's-t comparison keeps a stable mean.

p50_ns (median) uses raw timings to stay paired with the percentile family; being robust, it's unaffected in practice. outlier_ratio reports how heavy the tail was either way. sample_size is the cleaned count behind central tendency and min_ns; the upper-tail order statistics use all valid timings (sample_size + outliers_ns.length); raw_sample_size is the total input count including the failed_iterations invalid values that were filtered out.

`mean_ns`

Mean (average) time in nanoseconds (over MAD-cleaned samples)

type number

readonly

`p50_ns`

50th percentile (median) time in nanoseconds (over raw samples)

type number

readonly

`std_dev_ns`

Standard deviation in nanoseconds (over MAD-cleaned samples)

type number

readonly

`min_ns`

Minimum time in nanoseconds (over MAD-cleaned samples)

type number

readonly

`max_ns`

Maximum time in nanoseconds (over raw samples)

type number

readonly

`p75_ns`

75th percentile in nanoseconds (over raw samples)

type number

readonly

`p90_ns`

90th percentile in nanoseconds (over raw samples)

type number

readonly

`p95_ns`

95th percentile in nanoseconds (over raw samples)

type number

readonly

`p99_ns`

99th percentile in nanoseconds (over raw samples)

type number

readonly

`cv`

Coefficient of variation (std_dev / mean)

type number

readonly

`confidence_interval_ns`

95% confidence interval for the mean in nanoseconds

type [number, number]

readonly

`outliers_ns`

Array of detected outlier values in nanoseconds

type Array<number>

readonly

`outlier_ratio`

Ratio of outliers to total samples

type number

readonly

`sample_size`

Number of valid samples after outlier removal (population for central tendency and min_ns)

type number

readonly

`raw_sample_size`

Number of input samples before filtering invalid/outlier values

type number

readonly

`ops_per_second`

Operations per second (NS_PER_SEC / mean_ns)

type number

readonly

`failed_iterations`

Number of failed iterations (NaN, Infinity, or negative values)

type number

readonly

`constructor`

type new (timings_ns: number[]): BenchmarkStats

`timings_ns`

type number[]

`toString`

Format stats as a human-readable string.

type (): string

returns string

BenchmarkStatsComparable
#

benchmark_stats.ts view source

BenchmarkStatsComparable import type {BenchmarkStatsComparable} from '@fuzdev/fuz_util/benchmark_stats.js';

Minimal stats interface for comparison. This allows comparing stats from different sources (e.g., loaded baselines).

`mean_ns`

type number

`std_dev_ns`

type number

`sample_size`

type number

`confidence_interval_ns`

type [number, number]

EffectMagnitude
#

benchmark_stats.ts view source

EffectMagnitude import type {EffectMagnitude} from '@fuzdev/fuz_util/benchmark_stats.js';

Effect size magnitude interpretation (Cohen's d).