Metadata-Version: 2.4
Name: tc4v-tasq
Version: 0.1.1
Summary: A task queue for parallel processing
Author-email: Théo Cavignac <theo.cavignac@gmail.com>
License-Expression: BSD-3-Clause
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# tc4v-tasq

A parallel safe task queue on disk.

## About

`tasq` maintains a database of tasks and let you claim and mark tasks as done
in a multi-process safe way. It has two backends, a sqlite based one for general
single machine multi-process use, and a file based one for multi-machine multi-process
use (on a shared filesystem).

## Installation

```bash
pip install tc4v-tasq
```

or

```bash
git clone https://git.sr.ht/~lattay/tasq.py
cd tasq.py
pip install .
```

## Usage

A task is only defined as a string of text, or name, that should be unique. The interpretation of the name is up to the user.

To create a new task queue, use `init` and provide the first list of task from a file or stdin:
```bash
$ tasq init tasks.db - <<EOF
./work/task1 3
./work/task2 9

# comments and empty lines are ignored
./work/task3 0 # inline comments work too
EOF
```

Tasks can be added later in a similar fashion with `add`:

```bash
$ tasq add tasks.db - <<EOF
./work/task4 blublublu
./work/task5 "it uses shlex so spaces are ok"
./work/task6 blablabla
EOF
```

To see the current state of the queue, use `list`:

```bash
$ tasq list tasks.db
a6cc64a4401b4e0a94489b43bd22ac72 pending 2025-11-22 21:53:31 ['./work/task1', '3']
390dda92b89f4b4a9ed519fc86356fd7 pending 2025-11-22 21:53:31 ['./work/task2', '9']
a2380735393e4f07a7b4b67eb7a969eb pending 2025-11-22 21:53:31 ['./work/task3', '0']
482955a8421940a5b4874d4987affb30 pending 2025-11-22 21:53:37 ['./work/task4', 'blublublu']
1d875235a5a64d16921fa9d5e0d54309 pending 2025-11-22 21:53:37 ['./work/task5', 'it uses shlex so spaces are ok']
b030397203a94303b1a6a1960ba78aaf pending 2025-11-22 21:53:37 ['./work/task6', 'blablabla']
```

The hex string at the begining is a unique identifier for the task that is used
to interact with it.

Tasks can be claimed using `claim`:

```bash
$ tasq claim tasks.db -n 2 --data
390dda92b89f4b4a9ed519fc86356fd7 ["./work/task2", "9"]
a6cc64a4401b4e0a94489b43bd22ac72 ["./work/task1", "3"]
```

and marked done with `done`:

```bash
$ tasq done tasks.db a6cc64a4401b4e0a94489b43bd22ac72
```
See also `tasq help` for more commands.

There is also a second script `pll` that does something similar to GNU parallel
but for a task queue.

For example the following command:
```bash
$ pll tasks.db -n 2 ./process.sh '{0}' '{1}'
```

will process all tasks calling the command `./process.sh` with the task
arguments (placeholder follow `str.format` syntax) using two parallel workers.
The outputs will be printed concatenates with `# <TASK_ID>` line above each
outputs.
Because it uses the task queue, you can have more than one `pll` running in
parallel (for example if using slurm, each slurm job can use `pll`).
