CI energy gates

What it is

An energy gate compares a pull request's measured workload against a saved reference. It turns an energy regression into a build status so teams can notice a cost increase before it becomes the new normal.

DoL8 stores a baseline mean, sample standard deviation, and run count as JSON. A measured run can compare against that file with a percentage threshold and return a non-zero exit code when a significant increase crosses the threshold.

Why it matters

Measurements on physical machines are noisy. A gate that fails on any positive delta will turn ordinary variation into flaky red builds, teaching developers to ignore the signal.

The gate should require both a meaningful percentage increase and evidence that the increase exceeds the recorded noise. For useful results, run on a stable bare-metal Linux machine with readable RAPL counters, not a shared hosted runner or virtual machine that hides them.

How it works

Create a baseline from a DoL8 result using dol8 gate save. During a later run, the evaluator calculates percentage change from the baseline mean and combined deviation sqrt(current_stdev^2 + baseline_stdev^2). A positive change is considered beyond noise only when both groups have at least two samples and the increase exceeds twice that combined deviation.

The current gate fails only when the increase is above the configured percentage threshold and also beyond the noise guard. Improvements do not fail the gate. The evaluator exposes whether the change is beyond noise; the CLI currently prints PASS or FAIL rather than a separate INCONCLUSIVE label.

python -m dol8.cli run tests/fixtures/heavy.py \
  --runs 5 --json current.json \
  --gate-baseline energy-baseline.json --threshold 15

For a physical measurement workflow, pin the baseline to a trusted reference and ensure the CI worker is not concurrently running unrelated workloads.

What to build

1. Save a reproducible baseline. Use dol8 gate save result.json --out energy-baseline.json after collecting multiple runs on the intended machine.
2. Compare distributions. In dol8/gate.py, compare means, standard deviations, sample counts, and threshold; avoid treating an unknown one-run deviation as evidence of significance.
3. Make outcomes explicit. Distinguish pass, regression, and inconclusive/noise in structured output. Current CLI text reports only PASS or FAIL even when the noise guard prevents failure.
4. Use a stable worker. Configure a self-hosted Linux bare-metal runner with RAPL available, fixed power settings where practical, and no competing benchmark jobs.
5. Version the reference deliberately. Review baseline updates like code, record the workload and machine context, and refresh only after an intentional performance/energy change.

name: Energy check
on: [pull_request]
jobs:
  measure:
    runs-on: [self-hosted, linux, rapl]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: python -m pip install -e .
      - run: >-
          python -m dol8.cli run tests/fixtures/heavy.py
          --runs 5 --json current.json
          --gate-baseline energy-baseline.json --threshold 15

Commit a reviewed energy-baseline.json beside the workflow. A GitHub-hosted VM usually cannot provide the physical RAPL interface this check needs.

How to use it

Generate the initial result on the reference machine, save it, and then use that file on subsequent runs. Illustrative commands:

python -m dol8.cli run tests/fixtures/heavy.py --runs 5 --json reference.json
python -m dol8.cli gate save reference.json --out energy-baseline.json
python -m dol8.cli run tests/fixtures/heavy.py --runs 5 \
  --gate-baseline energy-baseline.json --threshold 15

ENERGY GATE: baseline 100.000 J, current 103.000 J, delta +3.00%, threshold 15.00%: PASS

Edge cases & gotchas

• Unknown sample counts default conservatively to one; a single run is not sufficient for the noise guard to declare a significant increase.
• A passing gate does not prove energy stayed identical. It means the configured regression conditions were not both met.
• Only a positive regression above threshold is a failure; decreases and changes within the noise band pass in the current implementation.
• Changing CPU, kernel, cooling, power limits, workload inputs, or runner contention can invalidate a saved baseline.
• Do not regenerate the baseline from every pull request. That would make regressions self-approving.

Related