Metadata-Version: 2.1
Name: pytest-history
Version: 0.2.0
Summary: Pytest plugin to keep a history of your pytest runs
Home-page: https://github.com/Nicoretti/one-piece/tree/grand-line/python/pytest-history
License: MIT
Keywords: pytest,sqlite,sql,history
Author: Nicola Coretti
Author-email: nico.coretti@gmail.com
Requires-Python: >=3.8,<4.0
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Requires-Dist: pytest (>=7.4.3,<8.0.0)
Project-URL: Repository, https://github.com/Nicoretti/one-piece/tree/grand-line/python/pytest-history
Description-Content-Type: text/markdown

<h1 align="center">pytest-history</h1>
<p align="center">
Pytest plugin to keep history of your pytest runs
</p>

<p align="center">

<a href="https://github.com/Nicoretti/one-piece/python/pytest-history">
    <img src="https://img.shields.io/github/checks-status/nicoretti/crc/master" alt="Checks Master">
</a>
<a href="https://opensource.org/licenses/MIT">
    <img src="https://img.shields.io/pypi/l/pytest-history" alt="License">
</a>
<a href="https://pypi.org/project/pytest-history/">
    <img src="https://img.shields.io/pypi/pyversions/pytest-history" alt="Supported Python Versions">
</a>
<a href="https://pypi.org/project/pytest-history/">
    <img src="https://img.shields.io/pypi/v/pytest-history" alt="PyPi Package">
</a>
</p>

## Overview

`pytest-history` enables the tracking of test statuses and other metadata across multiple test runs, providing
additional insights into test behavior.

Initially, this plugin was developed specifically to identify potentially flaky tests (approximately 200) within a test
suite containing over 1000 tests, where various tests exhibited inconsistent behavior by failing on alternate runs.

## Purpose

- **Tracking Test History:** Capturing and storing historical test results, encompassing pass, fail, and other pertinent
  metadata.
- **Identifying Flaky Tests:** Enabling the identification of flaky tests by scrutinizing historical data, detecting
  irregularities or recurring patterns in test outcomes.
- **Facilitating Debugging:** Offering developers and testers insights into test stability, thereby assisting in
  debugging efforts and enhancing overall test reliability.

## Usage

1. Install the plugin using `pip install pytest-history`.
2. Utilize the historical data stored in the `.test-results.db` SQLite database using either the `pytest-history` CLI or an SQLite client.

## CLI
The `pytest-history` command utilizes a SQLite database (default: `.test-results.db`) to provide analysis and information about past test runs and their results. It offers two main subcommands: `list` and `flakes`.

### Subcommands

#### `list`
- `pytest-history list` offers insights into historical data and provides two additional subcommands: `results` and `runs`.
  
##### subcommand `results`
- `pytest-history list results <id>`: Lists historic test results for a specific test run identified by its ID.
  
##### subcommand  `runs`
- `pytest-history list runs`: Lists historic test runs.

#### `flakes`
- `pytest-history flakes`: Lists all flaky tests identified in the test suite.

### Usage

```bash
pytest-history [options] <subcommand> [suboptions]

options
    -h, --help: Show the help message and exit.
    --db DB: Specify the database to be used for analyzing data (default: .test-results.db).
```

### Examples

List historic test runs
```bash
pytest-history list runs
```

List historic test results for a specific run
```bash
pytest-history list results <id>
```

List all flaky tests
```bash
pytest-history flakes
```

## Example Queries

Example: To find flaky tests between two distinct test runs, execute the following SQL query:

```sql
SELECT t1.testcase, t1.test_run, t2.test_run, t1.outcome, t2.outcome
FROM "test.results" t1
         JOIN "test.results" t2 on t1.testcase = t2.testcase AND (t1.test_run = 1 AND t2.test_run = 2)
WHERE (t1.outcome = 'passed' AND t2.outcome = 'failed')
   OR (t1.outcome = 'failed' AND t2.outcome = 'passed')
GROUP BY t1.testcase
ORDER BY t1.testcase;
```

