Metadata-Version: 2.4
Name: pytest-perfguard
Version: 0.1.2
Summary: Automatic, low-overhead performance regression checks for pytest
Author: community-of-python
License-Expression: MIT
License-File: LICENSE
Classifier: Framework :: Pytest
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Dist: pytest>=7
Requires-Python: >=3.10
Project-URL: Homepage, https://github.com/community-of-python/pytest-perfguard
Project-URL: Issues, https://github.com/community-of-python/pytest-perfguard/issues
Description-Content-Type: text/markdown

# pytest-perfguard

`pytest-perfguard` records CPU and memory metrics for ordinary pytest tests and
compares them with compatible historical results. It does not require benchmark
fixtures, decorators, or markers.

This repository currently contains a proof of concept. Its goals are:

- measure the existing test run instead of repeating tests;
- use Linux hardware instruction counters when permitted;
- fall back explicitly to process CPU time when counters are unavailable;
- collect process RSS with no sampling thread or binary dependency;
- keep history in GitLab job artifacts rather than an external service;
- catch both single-test jumps and suite-wide accumulation;
- retain a fixed anchor so gradual drift remains visible after the rolling
  baseline adapts;
- keep performance regressions non-blocking without masking functional failures.

## How it compares

Perfguard is a broad regression smoke detector for an existing test suite. It
does not replace a controlled microbenchmark harness or an allocation profiler;
those tools trade more setup or runtime for a more precise answer.

| Tool | Test changes and run model | Signals and history | Best fit and trade-off |
| --- | --- | --- | --- |
| `pytest-perfguard` | No test changes; measures one normal pytest run | CPU instructions when available, process CPU time, and RSS; rolling and fixed baselines in GitLab artifacts | Broad, non-blocking CI coverage without a service; measurements are coarser than isolated benchmarks or allocation tracing |
| [`pytest-benchmark`](https://pytest-benchmark.readthedocs.io/) | Uses the `benchmark` fixture and calibrated repetitions | Wall-clock distributions; JSON or Elasticsearch storage, comparisons, and fail expressions | Precise microbenchmarks; requires benchmark-specific tests and adds repeated execution |
| [`pytest-codspeed`](https://codspeed.io/docs/reference/pytest-codspeed) | Uses a benchmark marker or fixture and instrumented benchmark runs | CPU simulation, wall time, or memory measurements with managed history and a dashboard | Stable hosted comparison workflow; requires selected benchmarks and the CodSpeed service |
| [`asv`](https://asv.readthedocs.io/) | Uses a dedicated benchmark suite and can run it across commits and environment matrices | Timing, object or peak memory, and arbitrary numeric values; JSON history and a static web UI | Long-term package and interpreter comparisons; more setup, storage, and CI time |
| [`pytest-monitor`](https://github.com/CFMTech/pytest-monitor) | Automatically records ordinary tests after installation | Duration, CPU usage, and memory in local SQLite or an optional server | The closest annotation-free collector; comparison and alert policy must be built around its recorded data |
| [`pytest-memray`](https://pytest-memray.readthedocs.io/) | Traces existing tests with `--memray`; memory limits use markers | Allocation and peak-memory reports, per-test limits, and increase checks | Deep memory diagnosis; native allocation tracing costs more and threshold gates are explicit |
| [`pytest-leaks`](https://github.com/abalkin/pytest-leaks) | Repeats each test through a command-line option | CPython reference-count and memory-block leaks; no historical baseline | Targeted reference-leak detection; requires a debug Python build and substantially extends the run |

In practice, use Perfguard when the first requirement is coverage of unchanged
tests with local GitLab history and low additional CI work. Use
`pytest-benchmark` or [`pyperf`](https://pyperf.readthedocs.io/) for carefully
isolated hot paths, CodSpeed when a managed dashboard is desirable, and ASV for
performance across many commits, Python versions, or dependency versions.
Memray and `pytest-leaks` are better follow-up tools when a broad RSS signal
needs an allocation or reference-leak diagnosis.

## Install

Add the package to the same environment as pytest:

```console
uv add --dev pytest-perfguard==0.1.1
```

The plugin is automatically discovered through pytest's `pytest11` entry point.
CI enables collection with:

```console
PERFGUARD=1 pytest
```

It writes one artifact per GitLab test job under
`.perfguard/current/<CI_JOB_NAME_SLUG>-<CI_JOB_ID>.json`, avoiding collisions
when GitLab downloads several jobs' artifacts. With pytest-xdist, workers send
their results to the controller and the controller writes one job-level file.

## Non-blocking GitLab design

Do not put `allow_failure: true` on the normal test job: that would also make
functional test failures non-blocking.

Instead, retain the normal blocking test job and add small report jobs. The
default-branch job keeps the rolling database, while merge-request output
expires:

```yaml
.perfguard-report:
  stage: report
  needs:
    - job: test
      artifacts: true
  script:
    - uvx --from "pytest-perfguard==${PERFGUARD_VERSION}" pytest-perfguard gitlab
  when: always
  allow_failure: true
  artifacts:
    when: always
    paths:
      - .perfguard/results.json
      - .perfguard/junit.xml
    reports:
      junit: .perfguard/junit.xml

perfguard:
  extends: .perfguard-report
  artifacts:
    expire_in: never
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: always

perfguard-mr:
  extends: .perfguard-report
  artifacts:
    expire_in: 30 days
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      when: always
```

The complete example is in [`examples/gitlab-ci.yml`](examples/gitlab-ci.yml).
Projects with several test jobs list each producer under `needs`.
On a merge request, the command downloads the latest successful target-branch
`perfguard` artifact using `CI_JOB_TOKEN`. On the default branch it records the
current samples into rolling history while preserving the fixed anchor. No
GitLab credentials enter the application test container.

The result job returns:

- `0` when no confirmed regression is found, including warm-up and persistent
  drift;
- `1` when a regression is found;
- `2` when artifact preparation or comparison fails.

Because only this reporting job has `allow_failure: true`, all three outcomes
remain visible without weakening the test gate.

The command prints the largest regressions and persistent drifts directly in the
job log and writes a JUnit report. Confirmed regressions are failures; drift is
a non-failing skipped testcase. Node IDs, metrics, baselines, and limits
therefore appear in GitLab without downloading JSON.

## Baseline policy

Results are compared only when architecture, CPU model, measurement backends,
xdist worker count, and runner key match. On GitLab the runner key defaults to
`CI_RUNNER_DESCRIPTION`, then `CI_RUNNER_ID`; `PERFGUARD_RUNNER_KEY` can group a
known-homogeneous pool or separate another CI provider's machines. This
prevents different shared-runner hosts from teaching one noisy baseline.
Python version, lock hash, and commit SHA are recorded as diagnostic context
rather than compatibility keys, so interpreter and dependency upgrades are
compared.

Each compatible series keeps two baselines with different purposes:

- a rolling history estimates current noise with median and MAD;
- a fixed anchor is created automatically after three successful default-branch
  samples and never moves during normal recording.

A measurement that exceeds the rolling limit is a confirmed regression. If it
exceeds only the fixed-anchor limit, it is reported as persistent drift without
a failing exit code. Requiring rolling corroboration prevents a short, unusually
fast anchor from making every later stable run flaky, while the fixed anchor
still prevents gradual degradation from disappearing silently as history moves.
To intentionally rebase after an understood change, run one successful
default-branch pipeline with either:

```console
pytest-perfguard gitlab --accept-baseline
```

or `PERFGUARD_ACCEPT_BASELINE=1`. Acceptance is rejected on merge-request and
other compare-only pipelines. Do not leave the variable enabled permanently.

The initial thresholds require both a relative increase and an absolute floor:

| Metric | Relative increase | Absolute floor |
| --- | ---: | ---: |
| CPU instructions (per test and summed job) | 15% | 1,000,000 instructions |
| Process CPU time (summed job and session) | 15% | 5 ms |
| RSS boundary growth (summed positive growth and per-test attribution) | 20% | 16 MiB |
| Process end and peak RSS | 20% | 16 MiB |

Three compatible historical samples are required before gating. Thresholds and
history length can be changed by CLI options.

Job aggregates are compared only when the set of passed test node IDs matches.
They catch regressions that are individually below the absolute floor, such as
1,000 tests each becoming 2 ms slower or several tests each retaining 8 MiB.
Individual baselines use the source of the test function, so editing a different
test in the same file does not reset them.

Per-test process CPU and RSS growth remain in the raw artifact, but become
regression attribution only when the corresponding job aggregate also exceeds
its rolling limit. RSS attribution is corroborated specifically by summed
positive RSS growth; process end and peak RSS remain independent job-level
guards. Shared runners can move unrelated process work, garbage collection,
lazy initialization, and allocator growth between individual pytest calls;
allowing an isolated call spike to fail the report produced false positives in
real xdist trials. Per-test instruction counts remain independently guarded.

## Measurement scope and limitations

- Linux instruction counting uses `perf_event_open`. Each pytest process runs
  one brief startup calibration and validates the counter's scheduled time on
  every read. Access denial, a zero/nonfunctional counter, no scheduled time,
  or less than 90% scheduled time causes an automatic `process_time` fallback;
  the artifact records the exact unavailable reason. This avoids both false
  zero-instruction baselines and noisy heavily multiplexed baselines.
- CPU time is process CPU consumed during each pytest call. Hardware
  instructions count only the thread running that call. The artifact records
  these scopes explicitly. Summed job and session CPU remain guarded so
  threaded work is not silently ignored; individual CPU-time outliers are
  reported only as attribution for a corroborated job regression.
- RSS uses `/proc/self/statm` on Linux, working set on Windows, and peak RSS as a
  fallback. With xdist, process RSS is summed across the controller and workers,
  and per-test CPU is summed across workers; the artifact records the process
  count.
- RSS growth is a regression signal, not proof of a memory leak. Allocation
  tracing and repeated workloads remain diagnostic follow-ups.
- The proof of concept measures the pytest process. Separate application
  services require a service- or cgroup-level collector.
- Shared runners with different CPU models build separate history buckets.
- The default-branch `.perfguard/results.json` is the baseline database and uses
  `expire_in: never`; deleting it restarts warm-up. Merge-request comparison
  artifacts expire after 30 days in the example.
