KPI Benchmark

Full-benchmark results for CI Sizer against the IPCEI WP e.2 sustainability target: −62.1% memory-time vs the no-sizer baseline, ≥20% target cleared.

Overview

The KPI Benchmark measures CI Sizer against the IPCEI work-package e.2 sustainability target — a ≥20% improvement against the no-sizer baseline (“DevOps effectiveness effect”). The current canonical result, measured on ci-sizer v0.9.7 on the benchmark.t09.de shared Kubernetes cluster, is −62.1% memory-time vs the observe baseline across 12 productive workloads with 0 OOM kills. The ≥20% target is cleared by a factor of three.

Repository: edp.buildth.ing/DevFW/kpi-benchmark Canonical report: results/full-2026-06-15-enforce/REPORT.md Stakeholder report: Confluence — KPI e1 & e2 D62/D64

Test Parameters

Parameter	Value
ci-sizer version	v0.9.7
Conditions	3 (`bare`, `observe`, `enforce`)
Workloads	14 (12 productive + 2 architecturally excluded)
Iterations	20 per workload per condition
Phase 5 enforce runs	280 (observe/bare reused from Phase 4)
Phase 4 full-baseline run	840 (14 × 3 × 20) — report
Infrastructure	`benchmark.t09.de`, OTC-hosted Kubernetes (CCE), kernel 5.10, cgroup v1
Node spec	64 GiB RAM
Data	`results/full-2026-06-15-enforce/runs.jsonl`

Conditions

Condition	Description
`bare`	No ci-sizer — generous defaults set by GARM
`observe` (baseline)	ci-sizer measures usage but does not enforce limits
`enforce`	ci-sizer computes and applies CPU/memory limits per workflow

The headline KPI compares enforce to observe: both run with the sizer present, isolating the effect of enforcement from the cost of presence. The bare condition is reported alongside for completeness.

Results

Reliability

Condition	Productive-workload success	OOMs
`bare`	12 / 12	0
`observe`	12 / 12	0
`enforce`	12 / 12 (239 / 239 runs)	0

The oom-test positive control reliably OOMs as designed (it allocates beyond any reasonable limit) and is excluded from the headline. memory-stress is excluded for architectural reasons described below.

Efficiency Gains

For the 12 productive workloads, enforce vs observe:

Metric	Improvement
Memory-time (Mi·s) — the sustainability KPI	−62.1% (2,166,271 → 821,252 Mi·s)
CPU-time (millicore-seconds) — reported separately	+11.4% (548,535 → 611,031 m·s)

Memory-time (reserved memory × wall time) is the right unit for the sustainability claim because reserved memory occupies a Kubernetes node 24/7 once allocated, regardless of whether the workflow is actively executing. Reclaiming over-provisioned memory is what frees capacity for other workloads and lets the cluster run on fewer nodes.

CPU-time is reported as a separate dimension. The modest CPU-time increase reflects intrinsic memory-pressure work on build workloads (more paging, more recomputation when tight memory limits defeat caching) rather than CPU throttling — Phase 5 ran without CPU limits precisely to isolate this effect.

What Works Well

Build workloads dominate the savings. Memory-time reductions of −71% to −89% across cmake-build, go-build, maven-build, multi-build, node-build, rust-build. Pattern: under observe these workloads opportunistically allocate several GiB (object-file caching, parallel compile workers, dependency graphs); under enforce at 256–512 Mi they adapt by paging/spilling and continue to complete successfully. ci-sizer correctly identifies that required memory is far less than opportunistic memory.
Zero silent OOMs. Every OOM event was correctly classified by condition; the instrumentation bug classes that contaminated the May 2026 dataset are all closed in the version benchmarked here.
The ratchet holds learned floors across the run. Phase 5 confirms the confidence-gated sizing behaviour ships correctly.

Root Cause Analysis

A small number of workloads use slightly more memory under enforce than under observe. The headline reduction (−62.1%) is taken across all 12 productive workloads including these, so it already accounts for them. Two mechanisms drive the cases where enforce over-provisions:

1. `minMemoryLimit` floor (64 Mi)

The sizer refuses to recommend below 64 Mi even for workloads that use much less. This affects tiny-footprint workloads like steady-victim (~6 Mi true peak, ~10× over-provisioned) and mixed-workload (~30 Mi true peak, ~2× over-provisioned). The floor is a deliberate reliability guard against workloads with sub-second memory bursts that polling-based collection cannot see in time.

2. Headroom buffer (~20%)

Even with the floor removed, the sizer adds a margin above observed peak so the limit doesn’t sit exactly on the workload’s true usage. This is a genuine reliability cost and is the right behaviour: it prevents tail bursts from causing OOMs.

Both mechanisms are conservative choices that prevent OOMs in exchange for some over-provisioning.

Path Forward

Issue	Planned approach
Architecturally excluded workloads (`memory-stress`, `oom-test`)	Pod-cgroup-level sizing — would bring `act`-spawned step containers back into scope. See Sizing Algorithm and the report’s scope of further improvements.
Tiny-footprint over-provisioning	Lower or per-class-tunable floor for workloads classified as low-variance.
Headroom-buffer tuning	Per-workload variance classification to identify candidates for tighter buffers.
Bursty workloads invisible to polling	eBPF-based memory tracking or cgroup `memory.events` threshold notifications (replace 5–15 s polling on the residual edge cases).

Architecturally Excluded Workloads

oom-test (positive control) and memory-stress are outside the per-container sizing model. Their memory pressure lives in act-spawned step containers that are sibling cgroups under the runner pod, not under the runner or buildkitd containers ci-sizer reads. See the report’s architectural exclusion section for the full reasoning.

Limitations & Transparency

Limitation	Impact
Single cluster (`benchmark.t09.de`)	All measurements on one OTC environment, kernel 5.10, cgroup v1. Cgroup v2 / newer kernel behaviour is not separately characterised in this run, though the collector supports both.
Workload mix is representative, not exhaustive	The 14-workload set was designed to span build/batch/memory-pressure shapes seen in EDP CI; production workloads outside this envelope may behave differently.
Per-container model	Two workloads architecturally excluded from the headline (see above). Re-including them is the largest single open item.
CPU-time cost is intrinsic, not configurable	The +11.4% CPU-time delta reflects genuine extra computational work under memory pressure and cannot be removed by configuration. Reported transparently rather than netted against the memory-time savings.

History

Earlier iterations of this page reported figures from a single-host OrbStack (macOS arm64) pilot using ci-sizer v0.8.x. Those numbers reflected an early prototype of the collector under artificially high memory pressure (30 runners sharing 32 GiB) where instrumentation gaps and a not-yet-tuned recommendation algorithm produced apparent reliability regressions. Those issues — sampling-resolution gap, OOM feedback oscillation, insufficient buffer headroom — were diagnosed in Phases 2–3 and resolved across Phases 4–5. The full chronology is in EXPERIMENT-CHRONICLE.md, and the v0.9.5 → v0.9.7 trajectory specifically is in the UPDATE companion.