Statistical confidence

What it is

Energy measurements vary even when the source code stays the same. CPU boost, background work, cache state, temperature, and counter quantization all affect a run. Statistical confidence is the practice of showing that variation instead of treating one sample as the truth.

DoL8 measures an idle baseline, runs a configurable warm-up that is discarded, then performs repeated measured runs. It reports the mean, sample standard deviation, coefficient of variation (CoV), run count, and a confidence label.

Why it matters

A single energy number cannot tell you whether a change came from the code or from normal system noise. It has no estimate of spread, so a small apparent improvement can be indistinguishable from run-to-run variation.

Repeats make the uncertainty visible. A low CoV means the observed samples were more consistent; it does not guarantee that the system was isolated or that RAPL is an exact process-level meter.

How it works

Before each target, DoL8 measures five seconds of idle energy by default. It subtracts estimated idle watts times the target duration from each result and floors adjusted energy at zero. Unless disabled, one warm-up is run and discarded; use --warmup N to choose the count or --no-warmup to skip it.

For adjusted samples x, the mean is sum(x) / n; the standard deviation is the sample standard deviation; CoV is stdev / abs(mean). Current confidence labels are HIGH below 5% CoV, MEDIUM below 15%, otherwise LOW. One run is always LOW because spread is unknown.

mean_j = statistics.mean(samples)
stdev_j = statistics.stdev(samples) if len(samples) > 1 else 0
cov = stdev_j / abs(mean_j) if mean_j else 0

confidence = "LOW" if len(samples) < 2 else classify(cov)

The report includes mean and standard deviation, while the JSON statistics also include CoV and confidence. The current report does not expose min/max in its aggregate summary; those are useful additions when reviewing outliers.

What to build

1. Measure the idle window. In dol8/stats.py, capture energy and temperature before the workload and retain actual elapsed duration.
2. Separate warm-up from samples. Keep warm-up runs out of the measured list; record the configured count in warmup_discarded.
3. Adjust and aggregate consistently. Subtract the estimated idle rate, floor negative estimates at zero, and calculate sample standard deviation only when at least two measurements exist.
4. Keep failures explicit. Mark each run's completion status before aggregation. Do not silently mix incomplete or aborted runs into a clean-sample statistic.
5. Report the spread. Include mean, standard deviation, CoV, min/max, sample count, and a clear low-confidence state for one sample; test thresholds and zero-mean behavior.

How to use it

Choose enough measured runs to observe spread. This sample report is illustrative only; real values depend on workload and hardware.

python -m dol8.cli run workload.py --runs 5 --warmup 1 --json run.json

CONFIDENCE: 5 runs, mean 412.500 J +- 6.200 J, CoV 1.50%, HIGH

Edge cases & gotchas

• A one-run result has no measured sample spread. DoL8 labels it LOW rather than inferring confidence from a zero standard deviation.
• Baseline subtraction is an estimate. Idle power can change between baseline and target, and short workloads are especially sensitive to that error.
• Repeated measurements can still be biased by thermal drift, background processes, or changing power policy. Repeats do not replace a controlled setup.
• When mean energy is zero, CoV needs special handling to avoid division by zero.
• Do not discard outliers without a stated rule; preserve the raw samples so a reviewer can inspect them.

Related