Metadata-Version: 2.4
Name: otb-kbo-opendata-pyclient
Version: 1.0.0rc1
Summary: Client for discovering and downloading KBO Open Data files from the Belgian KBO SFTP server.
Project-URL: Homepage, https://github.com/openthebox/kbo-opendata-pyclient
Project-URL: Repository, https://github.com/openthebox/kbo-opendata-pyclient
Project-URL: Issues, https://github.com/openthebox/kbo-opendata-pyclient/issues
Project-URL: Changelog, https://github.com/openthebox/kbo-opendata-pyclient/blob/main/CHANGELOG.md
Author-email: openthebox <vincent.kox@openthebox.be>
License-Expression: MIT
License-File: LICENSE
Keywords: belgium,cbe,kbo,opendata,sftp
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Utilities
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: boto3>=1.28
Requires-Dist: paramiko>=3.0
Description-Content-Type: text/markdown

# otb-kbo-opendata-pyclient

[![CI](https://github.com/openthebox/kbo-opendata-pyclient/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/openthebox/kbo-opendata-pyclient/actions/workflows/ci.yml)
[![Release](https://github.com/openthebox/kbo-opendata-pyclient/actions/workflows/release.yml/badge.svg)](https://github.com/openthebox/kbo-opendata-pyclient/actions/workflows/release.yml)
[![Coverage](https://img.shields.io/endpoint?url=https%3A%2F%2Fgist.githubusercontent.com%2Fv-kox%2F878ef07a4cc1e80bfc028ff100dbcb5b%2Fraw%2Fcoverage.json)](https://github.com/openthebox/kbo-opendata-pyclient/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/otb-kbo-opendata-pyclient.svg)](https://pypi.org/project/otb-kbo-opendata-pyclient/)
[![Python versions](https://img.shields.io/pypi/pyversions/otb-kbo-opendata-pyclient.svg)](https://pypi.org/project/otb-kbo-opendata-pyclient/)
[![Licence](https://img.shields.io/pypi/l/otb-kbo-opendata-pyclient.svg)](https://github.com/openthebox/kbo-opendata-pyclient/blob/main/LICENSE)

Python client for the SFTP server of the Belgian Crossroads Bank for Enterprises (KBO/BCE), which
publishes daily KBO Open Data ZIP files. Use it as a library or through the `kbo-opendata` command.

## Features

- Discover the latest publication, or look one up by index or by date
- List everything on the server, filter by index or date range, and spot skipped indices
- Download by filename, index, date, or "the latest", choosing update files, full files, or both
- Write to a local directory or straight to S3, streaming rather than buffering whole files
- Host-key verification against `known_hosts` by default
- Fully typed, with a `py.typed` marker

## Installation

```bash
pip install otb-kbo-opendata-pyclient
```

## Getting access

KBO Open Data is free but not anonymous. Two steps are needed before this client can connect:

1. Register on the [KBO Open Data portal](https://kbopub.economie.fgov.be/kbo-open-data) and accept
   the [licence](https://economie.fgov.be/sites/default/files/Files/Entreprises/BCE/Licence-BCE-Open-Data-Conditions-d-utilisation.pdf).
   This alone gives you manual downloads through the website.
2. Request SFTP access separately, in advance, by emailing
   [kbo-bce-webservice@economie.fgov.be](mailto:kbo-bce-webservice@economie.fgov.be). Portal
   registration does not grant it.

The credentials you receive are what this package uses. There is no test or sandbox server.

## Credentials

The client needs the username and password issued by KBO. The library accepts them directly; the
CLI reads them from the environment.

| Variable | Purpose |
| --- | --- |
| `KBO_OPENDATA_USERNAME` | SFTP account name, which also names the remote directory |
| `KBO_OPENDATA_PASSWORD` | SFTP password |

## About the published files

KBO snapshots its database daily and publishes two ZIP files per snapshot:

- a **full** file — every active registered entity and establishment unit at the moment of the snapshot
- an **update** file — the differences between the last full file and the one before it

**Files are kept for 31 days only.** Anything older is gone from the server, so `list_all()`,
`missing_indices()` and date lookups only ever see roughly the last month. Download the full file
first, then keep up with either update files or a periodic full file.

The `XXXX` in a filename is the `ExtractNumber`, incremented by one per publication. It is not a
date: KBO can skip a calendar day, which is why `missing_indices()` reports gaps in the sequence
rather than missing dates.

### What is inside a ZIP

This package downloads and stores the ZIP files; it does not extract or parse them. Each archive
holds CSV files — `meta.csv`, `code.csv`, `enterprise.csv`, `establishment.csv`, `denomination.csv`,
`address.csv`, `contact.csv`, `activity.csv` and `branch.csv` — joined on the enterprise number,
establishment number or branch id. The CSV conventions are a comma delimiter, double-quoted text,
a full stop as decimal point, and `dd-mm-yyyy` dates.

`meta.csv` carries `SnapshotDate`, `ExtractTimestamp`, `ExtractType` (`full` or `update`),
`ExtractNumber` and the format `Version`, which is the reliable way to confirm what an archive
actually contains.

An update archive splits each table into a `_delete` and an `_insert` file. Applying one means
deleting every row for the listed entity numbers, then inserting the rows from the `_insert` file —
the insert file repeats all current rows for a changed entity, not only the changed ones. The files
carry no history: only the current state of active entities.

### Documentation

- [Cookbook](https://economie.fgov.be/sites/default/files/Files/Entreprises/BCE/Cookbook-BCE-Open-Data.pdf)
  — file structure, CSV field descriptions and the update procedure
- Data catalogue, the reusable data fields — no English version exists, only
  [Dutch](https://economie.fgov.be/sites/default/files/Files/Entreprises/KBO/Gegevenscatalogus-hebruikbare-gegevens-KBO-Open-Data.pdf)
  and [French](https://economie.fgov.be/sites/default/files/Files/Entreprises/BCE/Catalogue-des-donnees-reutilisables-BCE-opendata.pdf)
- [KBO Open Data page](https://economie.fgov.be/en/themes/enterprises/crossroads-bank-enterprises/services-everyone/public-data-available-reuse/cbe-open-data)

## Library usage

```python
import datetime as dt

from kbo_opendata import KboOpenDataClient, LocalDestination, S3Destination

with KboOpenDataClient(username="...", password="...") as client:
    latest = client.latest()
    print(latest.update_filename, latest.full_filename)

    pair = client.get_by_index(423)  # None when the index was skipped
    pair = client.get_by_date(dt.date(2026, 8, 7))

    client.download_latest(LocalDestination("./downloads"))
    client.download_index(423, S3Destination("my-bucket", "kbo/"), kinds=["full"])
```

`KboOpenDataClient.from_env()` builds the same client from the environment variables above.

### Queries

| Method | Returns |
| --- | --- |
| `latest()` | The pair with the highest index, or `None` |
| `latest_n(count)` | The most recent pairs, newest last |
| `get_by_index(index)` | The pair for that index, or `None` |
| `get_by_date(date)` | The pair for that date, or `None` |
| `list_all()` | Every pair, oldest first |
| `list_by_index_range(start, end)` | Pairs within inclusive index bounds |
| `list_by_date_range(start, end)` | Pairs within inclusive date bounds |
| `missing_indices()` | Indices skipped between the lowest and highest present |
| `exists(filename)` | Whether the server holds that file |
| `stat(filename)` | Size and modification time |

A `KboFilePair` carries `update_filename` and `full_filename`, either of which is `None` when the
server holds only one of the two. Both are bare filenames, without a path.

### Downloads

| Method | Downloads |
| --- | --- |
| `download(filenames, destination)` | The named files |
| `download_index(index, destination)` | The files for one index |
| `download_date(date, destination)` | The files for one date |
| `download_latest(destination)` | The most recent files |

Every download method accepts `kinds` (defaults to update and full), `overwrite` (defaults to
`False`, so existing files are skipped) and `dry_run`. They return a `DownloadResult` whose
`written`, `skipped`, `missing` and `planned` tuples say what happened to each file.

The index and date variants raise `RemoteFileNotFoundError` when nothing matches; `download` reports
unknown names as `missing` instead, so a multi-file request always reports on every name.

### S3

```python
S3Destination("my-bucket", "kbo/", extra_args={"ServerSideEncryption": "AES256"})
```

A boto3 client is created from the ambient AWS configuration on first use. Pass `client=` to supply
one built from your own session, and `extra_args` to forward parameters to the upload.

## Command line usage

```bash
export KBO_OPENDATA_USERNAME=...
export KBO_OPENDATA_PASSWORD=...

kbo-opendata latest
kbo-opendata index 0423
kbo-opendata date 2026-08-07
kbo-opendata list --from-index 400 --latest 10
kbo-opendata gaps
kbo-opendata check

kbo-opendata download-latest --dest ./downloads
kbo-opendata download-index 423 --dest ./downloads --kind full
kbo-opendata download-date 2026-08-07 --s3-bucket my-bucket --s3-prefix kbo/
kbo-opendata download KboOpenData_0423_2026_08_07_Full.zip --dest ./downloads
```

Add `--json` for machine-readable output, `--overwrite` to replace existing files, `--dry-run` to
see what would be transferred, and `-v`/`-vv`/`-q` to adjust logging.

### Exit codes

| Code | Meaning |
| --- | --- |
| `0` | Success |
| `1` | Nothing matched the request |
| `2` | Usage error |
| `3` | Configuration, authentication or connection failure |
| `4` | The destination could not be written to |

## Host keys

The client verifies the server's host key against `~/.ssh/known_hosts` and refuses to connect to an
unknown host. Add the server once with `ssh-keyscan`, or pass `accept_unknown_host_key=True`
(`--accept-unknown-host-key` on the CLI) to trust it on first use.

## Logging

The package logs through the standard `logging` module under the `kbo_opendata` logger and installs
no handlers of its own. The CLI configures a stderr handler for its own process.

## Development

```bash
uv sync --group dev
uv run ruff check . && uv run ruff format --check .
uv run mypy
uv run coverage run -m pytest && uv run coverage report
```

The test suite is fully offline: the SFTP transport and the download destinations sit behind typed
protocols, and the tests drive real in-memory implementations of them rather than mocks.

## Reusing the data

The data is covered by the KBO Open Data licence, which you accept at registration — separate from
this package's MIT licence, which covers only the code. One restriction is worth stating plainly:
**personal data from these files may not be reused for direct marketing purposes.** See the
[licence](https://economie.fgov.be/sites/default/files/Files/Entreprises/BCE/Licence-BCE-Open-Data-Conditions-d-utilisation.pdf)
and the [CBE privacy statement](https://economie.fgov.be/en/themes/enterprises/crossroads-bank-enterprises/crossroads-bank-enterprises-0).

## Licence

MIT
