Metadata-Version: 2.4
Name: ftmq
Version: 5.0.0
Summary: followthemoney query dsl and io helpers
License: AGPLv3+
License-File: LICENSE
License-File: NOTICE
Author: Simon Wörpel
Author-email: simon.woerpel@pm.me
Requires-Python: >=3.11,<3.15
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Provides-Extra: aleph
Provides-Extra: api
Provides-Extra: duckdb
Provides-Extra: lake
Provides-Extra: level
Provides-Extra: postgres
Provides-Extra: search
Provides-Extra: sql
Requires-Dist: alephclient (>=2.6.0,<3.0.0) ; extra == "aleph"
Requires-Dist: anystore (>=1.2.6,<2.0.0)
Requires-Dist: deltalake (>=1.6.3,<2.0.0) ; extra == "lake"
Requires-Dist: duckdb (>=1.5.5,<2.0.0) ; extra == "duckdb"
Requires-Dist: duckdb (>=1.5.5,<2.0.0) ; extra == "lake"
Requires-Dist: duckdb-engine (>=0.17.0,<0.18.0) ; extra == "duckdb"
Requires-Dist: fastapi (>=0.139.2,<0.140.0) ; extra == "api"
Requires-Dist: followthemoney (>=4.10.2,<5.0.0)
Requires-Dist: furl (>=2.1.4,<3.0.0) ; extra == "aleph"
Requires-Dist: furl (>=2.1.4,<3.0.0) ; extra == "api"
Requires-Dist: granian (>=2.7.9,<3.0.0) ; extra == "api"
Requires-Dist: nomenklatura (>=4.14.0,<5.0.0)
Requires-Dist: normality (>=3.1.0,<4.0.0)
Requires-Dist: orjson (>=3.10.18,<4.0.0)
Requires-Dist: pandas (>=3.0.5,<4.0.0) ; extra == "lake"
Requires-Dist: plyvel (>=1.5.1,<2.0.0) ; extra == "level"
Requires-Dist: psycopg[pool] (>=3.2.9,<4.0.0) ; extra == "postgres"
Requires-Dist: pyarrow (>=25.0.1,<26.0.0) ; extra == "lake"
Requires-Dist: pyrql (>=0.7.11,<1.0.0)
Requires-Dist: rigour (>=2.3.1,<3.0.0)
Requires-Dist: sqlalchemy (>=2.0.51,<3.0.0) ; extra == "duckdb"
Requires-Dist: sqlalchemy (>=2.0.51,<3.0.0) ; extra == "postgres"
Requires-Dist: sqlalchemy (>=2.0.51,<3.0.0) ; extra == "sql"
Requires-Dist: tantivy (>=0.26.0,<0.27.0) ; extra == "search"
Requires-Dist: typer (>=0.26.8,<1)
Project-URL: Documentation, https://docs.investigraph.dev/lib/ftmq
Project-URL: Homepage, https://docs.investigraph.dev/lib/ftmq
Project-URL: Issues, https://github.com/dataresearchcenter/ftmq/issues
Project-URL: Repository, https://github.com/dataresearchcenter/ftmq
Description-Content-Type: text/markdown

[![Docs](https://img.shields.io/badge/docs-live-brightgreen)](https://docs.investigraph.dev/lib/ftmq/)
[![ftmq on pypi](https://img.shields.io/pypi/v/ftmq)](https://pypi.org/project/ftmq/)
[![PyPI Downloads](https://static.pepy.tech/badge/ftmq/month)](https://pepy.tech/projects/ftmq)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/ftmq)](https://pypi.org/project/ftmq/)
[![Python test and package](https://github.com/dataresearchcenter/ftmq/actions/workflows/python.yml/badge.svg)](https://github.com/dataresearchcenter/ftmq/actions/workflows/python.yml)
[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit)](https://github.com/pre-commit/pre-commit)
[![Coverage Status](https://coveralls.io/repos/github/dataresearchcenter/ftmq/badge.svg?branch=main)](https://coveralls.io/github/dataresearchcenter/ftmq?branch=main)
[![AGPLv3+ License](https://img.shields.io/pypi/l/ftmq)](./LICENSE)
[![Pydantic v2](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/pydantic/pydantic/main/docs/badge/v2.json)](https://pydantic.dev)

# ftmq

This library provides methods to query and filter entities formatted as [Follow The Money](https://followthemoney.tech) data, either from a json file/stream or using a statement-based store backend from [nomenklatura](https://github.com/opensanctions/nomenklatura).

It also provides a `Query` class that can be used in other libraries to work with SQL store queries or api queries.

`ftmq.Query` is the central query hub of the [OpenAleph](https://openaleph.org) ecosystem: one backend-agnostic query object that filters FtM streams and stores, translates to SQL, and bridges to the OpenAleph API.

To get familiar with the _Follow The Money_ ecosystem, you can have a look at [this pad here](https://pad.investigativedata.org/s/0qKuBEcsM#).

## Installation

Minimum Python version: 3.11

    pip install ftmq

## Usage

### Command line

```bash
cat entities.ftm.json | ftmq -q 'filter:schema=Company&filter:properties.country=de&filter:gte:properties.incorporationDate=2023' -o s3://data/entities-filtered.ftm.json
```

### Python Library

```python
from ftmq import Query, M, P, G, smart_read_proxies

# Legal entities in the `companies` dataset that are based in Germany, or in
# Austria and incorporated since 2020, but never the dissolved ones, return the
# 5 most recent incorporated ones:
q = Query().where(
    M(dataset="companies"),
    M(schemata="LegalEntity"),
    G(countries="de") | (G(countries="at") & P(incorporationDate__gte=2020)),
    ~P(status__ilike="%dissolved%"),
).order_by("incorporationDate", ascending=False)[:5]

for proxy in smart_read_proxies("s3://data/entities.ftm.json"):
    if q.apply(proxy):
        yield proxy
```

### Full-text search

`ftmq.search` (formerly the standalone `ftmq-search` package) indexes entities into simple full-text search stores, backed by SQLite FTS5 or [Tantivy](https://github.com/quickwit-oss/tantivy) (`pip install ftmq[search]`):

```bash
cat entities.ftm.json | ftmq search transform | ftmq search --uri sqlite:///ftmqs.db index
ftmq search --uri sqlite:///ftmqs.db "jane doe"
```

### HTTP API

`ftmq.api` (formerly the standalone `ftmq-api` package) exposes a store and its search index as a read-only FastAPI application (`pip install ftmq[api]`), speaking the Aleph filter grammar:

```bash
granian --interface asgi ftmq.api.app:app
curl "localhost:8000/entities?filter:schema=Payment&filter:gte:properties.date=2023"
```

## Documentation

https://docs.investigraph.dev/lib/ftmq

## Support

This project is part of [investigraph](https://investigraph.dev)

In 2023, development of `ftmq` was supported by [Media Tech Lab Bayern batch #3](https://github.com/media-tech-lab)

<a href="https://www.media-lab.de/en/programs/media-tech-lab">
    <img src="https://raw.githubusercontent.com/media-tech-lab/.github/main/assets/mtl-powered-by.png" width="240" title="Media Tech Lab powered by logo">
</a>

## License and Copyright

`ftmq`, (C) 2023 Simon Wörpel
`ftmq`, (C) 2024-2025 investigativedata.io
`ftmq`, (C) 2025 [Data and Research Center – DARC](https://dataresearchcenter.org)

`ftmq` is licensed under the AGPLv3 or later license.

Prior to version 0.8.0, `ftmq` was released under the MIT license.

see [NOTICE](./NOTICE) and [LICENSE](./LICENSE)

