Metadata-Version: 2.5
Name: pixls-preprints
Version: 1.0.0
Summary: Access and store metadata from different APIs offering scholarly preprints
Project-URL: Homepage, https://codeberg.org/pixls/pixls-preprints
Author-email: Benjamin Wolff <wolff@zbmed.de>
License-Expression: MIT
License-File: LICENSE
Keywords: biorxiv,harvesting,medrxiv,metadata,open science,osf,preprints,scholarly communication
Classifier: Development Status :: 5 - Production/Stable
Classifier: Framework :: AsyncIO
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: <4.0,>=3.11
Requires-Dist: aiohttp<4.0.0,>=3.9.5
Requires-Dist: aiolimiter<2.0.0,>=1.1.0
Requires-Dist: python-dotenv<2.0.0,>=1.0.0
Description-Content-Type: text/markdown

# pixls-preprints

[![Python](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/)
[![Version](https://img.shields.io/badge/version-1.0.0-informational.svg)](https://codeberg.org/pixls/pixls-preprints/src/branch/main/pyproject.toml)
[![License: MIT](https://img.shields.io/badge/code%20license-MIT-green.svg)](https://codeberg.org/pixls/pixls-preprints/src/branch/main/LICENSE)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://black.readthedocs.io/)

Access and store metadata from different APIs offering preprint publications.

`pixls-preprints` is an asynchronous command line harvester for preprint metadata. It
queries [bioRxiv](https://api.biorxiv.org/), [medRxiv](https://api.medrxiv.org/) and
[OSF](https://developer.osf.io/) for a range of dates and writes the results as
[JSON Lines](https://jsonlines.org/): one document per line, one file per source.

```console
$ python -m pixls_preprints --start 2024-07-14 --dest data --servers biorxiv osf
$ wc -l data/biorxiv/2024-07-14.json
49 data/biorxiv/2024-07-14.json
```

## Why use it

- **Fast.** Everything is asynchronous: all pages of a query are fetched concurrently,
  and so are the sources. You get as much speed as your rate limit allows, and an OSF
  token raises that ceiling.
- **Several sources, one command.** bioRxiv, medRxiv and OSF sit behind a single uniform
  CLI, each writing into its own output file.
- **Polite by default.** Every source has its own rate limit (requests per time window)
  and concurrency cap, so you stay within what the upstream API grants you.
- **Survives flaky APIs.** `429`, `502`, `503`, `504` and dropped connections are retried
  with a back-off, honouring `Retry-After`. Whatever still fails is logged, counted and
  reported at the end of the run, not fatal.
- **Streams to disk.** Each page is written as soon as it arrives, so a long harvest does
  not accumulate in memory and partial results survive an interruption.
- **JSON Lines output.** Results can be appended, read line by line without loading a
  whole file, and fed straight into a downstream indexing pipeline.
- **Layered configuration.** Config file, then environment variables, then defaults.
  Tokens are read *only* from the environment, never from the config file.
- **Easy to extend.** A new preprint server means one new subclass of `Source`. The
  HTTP client, rate limiting, retries and file writing are inherited.

## Requirements

- Python 3.11 or newer (the project is developed against 3.14, see [`.python-version`](https://codeberg.org/pixls/pixls-preprints/src/branch/main/.python-version))
- [uv](https://docs.astral.sh/uv/) for dependency management
- An [OSF API token](https://osf.io/settings/tokens), which is optional but raises the
  rate limit OSF grants you

## Installation

```bash
git clone https://codeberg.org/pixls/pixls-preprints.git
cd pixls-preprints
uv sync
```

Then create the two local files. Both are optional and both are ignored by git, so every
clone needs its own copy:

```bash
cp config.toml.example config.toml
cp .env.example .env          # then put your OSF token in it
```

Without `config.toml` the tool falls back to the defaults shipped inside the package,
which hold the same values as `config.toml.example`. `.env` is loaded automatically at
startup, so there is no need to `source` it, and variables already set in your shell
take precedence over it.

## Usage

Run the package as a module:

```bash
uv run python -m pixls_preprints -d data -srv biorxiv osf
```

Installing the package instead provides a `pixls-preprints` command, which takes the
same options and needs no config file to start:

```bash
uv pip install pixls-preprints
pixls-preprints -d data -srv biorxiv osf
```

Without a date, yesterday is harvested. The destination directory gets one subdirectory
per source, and the output file is named after the requested range.

**A single day from bioRxiv:**

```bash
uv run python -m pixls_preprints -s 2024-07-14 -d data -srv biorxiv
# -> data/biorxiv/2024-07-14.json
```

**A whole month from two sources, into one file each:**

```bash
uv run python -m pixls_preprints -s 2024-07-01 -e 2024-07-31 -d data -srv medrxiv osf
# -> data/medrxiv/2024-07-01_-_2024-07-31.json
# -> data/osf/2024-07-01_-_2024-07-31.json
```

**The same month, but one file per day, overwriting what is already there:**

```bash
uv run python -m pixls_preprints -s 2024-07-01 -e 2024-07-31 -S -F -d data -srv biorxiv
# -> data/biorxiv/2024-07-01.json … data/biorxiv/2024-07-31.json
```

**A custom file name and a config file from somewhere else:**

```bash
uv run python -m pixls_preprints -s 2024-07-14 -d data -f july.json -c /etc/pixls/config.toml
```

Existing output files are never silently overwritten: the source is skipped with a note
unless you pass `--force`. Log records are appended to `pixls_preprints_cli.log` in the
directory you ran the command from.

If requests are given up on after their retries, the output file is short. The run still
finishes and still exits `0`, but prints one warning per source saying how many requests
were lost:

```console
$ uv run python -m pixls_preprints -s 2024-07-14 -d data -srv biorxiv osf
WARNING: biorxiv: 3 requests failed permanently, output may be incomplete
```

### Options

| Option | Description |
| --- | --- |
| `-s`, `--start` | Start date (`YYYY-MM-DD`), defaults to yesterday |
| `-e`, `--end` | End date (`YYYY-MM-DD`), defaults to the start date |
| `-d`, `--dest` | **Required.** Destination directory, gets one subdirectory per source |
| `-srv`, `--servers` | **Required.** Sources to fetch from: `biorxiv`, `medrxiv`, `osf` (one or more) |
| `-f`, `--file` | Output file name, defaults to a name built from the dates |
| `-S`, `--split-daily` | Write one output file per day instead of one for the whole range |
| `-F`, `--force` | Overwrite already existing output files |
| `-c`, `--config` | Path to the config file, defaults to `config.toml` in the directory you run from |

An end date without a start date is an error. Note that bioRxiv and medRxiv select by
publication date, while OSF selects by *modification* date.

## Configuration

Per-source settings are resolved from three places, in this order of precedence:

1. the TOML config file
2. environment variables named `PIXLS_PREPRINTS_<SOURCE>_<SETTING>`
3. the defaults shipped with the package, in `pixls_preprints/config_defaults.toml`

The config file is the one given with `-c`, or `config.toml` in the directory you run
from. A path given with `-c` has to exist, otherwise the run stops with
`Config file not found`. Without a config file the run uses the packaged defaults, and
says so in the log.

```toml
# config.toml
[biorxiv]
max_rate = 1          # requests allowed per time_period
time_period = 0.4     # length of the rate limit window, in seconds
max_concurrency = 7   # requests in flight at most

[osf]
max_rate = 1
time_period = 1.5
max_concurrency = 10
page_size = 50        # documents requested per page
```

Tokens are the exception: they are read **only** from the environment. A `token` entry in
the config file is ignored and produces a warning.

```bash
# .env
export PIXLS_PREPRINTS_OSF_TOKEN=your-token-here
```

See [`config.toml.example`](https://codeberg.org/pixls/pixls-preprints/src/branch/main/config.toml.example) and [`.env.example`](https://codeberg.org/pixls/pixls-preprints/src/branch/main/.env.example) for
the full set of keys.

## Output format

One JSON document per line, in whatever shape the upstream API returns it. bioRxiv and
medRxiv records look like this (abridged):

```json
{"doi": "10.1101/001891", "title": "Population genomics of …", "authors": "Carlotta De Filippo;Monica Di Paola;…", "date": "2014-01-17", "version": "1", "type": "New Results", "license": "cc_by_nc_nd", "category": "Evolutionary Biology", "abstract": "…", "server": "biorxiv"}
```

OSF records follow the [JSON:API](https://jsonapi.org/) shape of the OSF v2 API, with
bibliographic contributors and license embedded so no extra request per preprint is
needed.

## Development

The [`Makefile`](https://codeberg.org/pixls/pixls-preprints/src/branch/main/Makefile) wraps the common tasks:

```bash
make test              # run the test suite, skipping tests that need network access
make test-online       # run only the tests that talk to the real APIs
make check-formatting  # black, in check mode
make formatting        # black, rewriting files
make check-code        # flake8
make check-coverage    # pytest with coverage
make audit             # formatting + lint + tests, the pre-merge gate
```

Tests are marked with `medium`, `large` and `online` (see [`pytest.ini`](https://codeberg.org/pixls/pixls-preprints/src/branch/main/pytest.ini)), so
you can deselect the slow or network-dependent ones, for example
`uv run pytest -m "not online" tests/`.

### Project layout

```
pixls_preprints/
├── cli.py            # argument parsing, date handling, task orchestration
├── config.py         # config file + environment + defaults
├── fetcher.py        # rate-limited async HTTP client with retries
└── sources/
    ├── source.py     # abstract base class shared by all sources
    ├── biorxiv.py    # bioRxiv and medRxiv (cursor paging)
    └── osf.py        # OSF (page-number paging, optional token)
```

To add a source, subclass `Source`, implement `url`, `fetch()` and `fetch_preprints()`,
and wire it into `_gather_tasks()` and the `--servers` choices in
[`pixls_preprints/cli.py`](https://codeberg.org/pixls/pixls-preprints/src/branch/main/pixls_preprints/cli.py).

## Getting help

- **Something broken, or a source missing?** Open an issue at
  [codeberg.org/pixls/pixls-preprints/issues](https://codeberg.org/pixls/pixls-preprints/issues).
- **Questions about the upstream APIs:** see the
  [bioRxiv/medRxiv API summary](https://api.biorxiv.org/) and the
  [OSF API documentation](https://developer.osf.io/).
- **Runtime detail:** every run appends to `pixls_preprints_cli.log`, which records the
  arguments used, each request and every retry. It is the first place to look when a
  harvest comes back short.

## Contributing

Contributions are welcome. Please read [`CONTRIBUTING.md`](https://codeberg.org/pixls/pixls-preprints/src/branch/main/CONTRIBUTING.md) for
the workflow, coding conventions and what `make audit` expects before you open a pull
request.

## Maintainers

Developed and maintained by Benjamin Wolff (<wolff@zbmed.de>) at
[ZB MED – Information Centre for Life Sciences](https://www.zbmed.de/), as part of the
PIXLS project.

## Funding

<a href="https://gepris.dfg.de/project/492813820/2?lang=en" style="border: 2px solid #26509E;">
  <img src="https://codeberg.org/pixls/pixls-preprints/raw/branch/main/img/dfg-funded.jpg" alt="Funded by the Deutsche Forschungsgemeinschaft (DFG)" width="70%">
</a>

Developed in [PIXLS – Preprint Information eXtraction for Life Sciences](https://gepris.dfg.de/project/492813820/2?lang=en),
funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation).

## License

Source code is licensed under the [MIT License](https://codeberg.org/pixls/pixls-preprints/src/branch/main/LICENSE). Documentation and other content
are licensed under [CC BY 4.0](https://codeberg.org/pixls/pixls-preprints/src/branch/main/LICENSES/CC-BY-4.0.txt), and configuration and generated
files under [CC0 1.0](https://codeberg.org/pixls/pixls-preprints/src/branch/main/LICENSES/CC0-1.0.txt). See [`LICENSING.md`](https://codeberg.org/pixls/pixls-preprints/src/branch/main/LICENSING.md) for
details.
