Metadata-Version: 2.5
Name: otb-cbso-webservice-pyclient
Version: 1.0.0
Summary: Download annual account data from the Belgian National Bank's Central Balance Sheet Office webservices
Project-URL: Homepage, https://github.com/openthebox/cbso-webservice-pyclient
Project-URL: Issues, https://github.com/openthebox/cbso-webservice-pyclient/issues
Project-URL: Changelog, https://github.com/openthebox/cbso-webservice-pyclient/releases
Author: openthebox
License: Proprietary
Keywords: annual accounts,belgium,cbso,nbb,xbrl
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: boto3>=1.34
Requires-Dist: urllib3>=2.0
Provides-Extra: dev
Requires-Dist: coverage[toml]>=7.4; extra == 'dev'
Requires-Dist: mypy>=1.9; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Description-Content-Type: text/markdown

# otb-cbso-webservice-pyclient

[![CI](https://github.com/openthebox/cbso-webservice-pyclient/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/openthebox/cbso-webservice-pyclient/actions/workflows/ci.yml)
[![tests](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/v-kox/a87a49513a178883c268da2d01e08bf7/raw/tests.json)](https://github.com/openthebox/cbso-webservice-pyclient/actions/workflows/ci.yml)
[![coverage](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/v-kox/a87a49513a178883c268da2d01e08bf7/raw/coverage.json)](https://github.com/openthebox/cbso-webservice-pyclient/actions/workflows/ci.yml)
![python](https://img.shields.io/badge/python-3.10%20%7C%203.11%20%7C%203.12%20%7C%203.13-blue)
[![ruff](https://img.shields.io/badge/lint-ruff-261230)](https://docs.astral.sh/ruff/)
[![mypy](https://img.shields.io/badge/types-mypy%20strict-2a6db2)](https://mypy-lang.org/)

Download annual account data from the Belgian National Bank's Central Balance
Sheet Office (CBSO) [webservices][webservices], from Python or from the command
line.

The CBSO publishes the annual accounts that Belgian legal entities file, as
XBRL, JSON-XBRL and PDF, along with the corrections the bank applies to a
selection of them. This package speaks all five of its products, files what it
downloads under the enterprise that filed it, and picks up where it left off
when a run is repeated.

[webservices]: https://www.nbb.be/en/central-balance-sheet-office/consultation/web-services

- Two dependencies: `urllib3` for pooled, retrying HTTP, and `boto3`, loaded
  only when an `s3://` destination is used.
- Typed throughout, with a `py.typed` marker.
- Each product is usable on its own, so one subscription is enough to start.

## Installation

```console
pip install otb-cbso-webservice-pyclient
```

Python 3.10 or later.

## The five products

The CBSO is not one API but five, each a separate subscription on the bank's
developer portal with a subscription key of its own. A key issued for one will
not authenticate against another.

| Product | What it serves | Key |
|---------|----------------|-----|
| `authentic` | One deposit at a time, as filed | `CBSO_KEY_AUTHENTIC` |
| `extracts` | Daily batches of the accounts as filed | `CBSO_KEY_EXTRACTS` |
| `authentic-archive` | The same batches, any date in the last three years | `CBSO_KEY_AUTHENTIC_ARCHIVE` |
| `improved` | The bank's corrections and conversions | `CBSO_KEY_IMPROVED` |
| `improved-archive` | Improved batches, any date in the last three years | `CBSO_KEY_IMPROVED_ARCHIVE` |

A key is read when a product is contacted, not when a client is built. You only
need keys for the products you actually use, and the error naming a missing one
arrives at the call that needed it.

## Configuration

Set the key of each product you hold a subscription for:

```console
export CBSO_KEY_EXTRACTS=...
export CBSO_KEY_AUTHENTIC=...
```

Any key can also be passed in code, which wins over the environment.

## Command line

```console
cbso-fetch daily --destination ./data
```

Four subcommands:

- `daily` fetches recent days from the daily products
- `backfill` fetches a date range from the archive products
- `deposit` fetches the documents of one deposit
- `legal-entity` lists what one enterprise has filed

```console
# The last three days, JSON only, into a local directory
cbso-fetch daily --destination ./data --days 3 --formats json

# A month from the archive, into S3
cbso-fetch backfill --destination s3://my-bucket/cbso \
    --from 2026-01-01 --to 2026-01-31

# See what a run would fetch, without fetching it
cbso-fetch daily --destination ./data --dry-run

# One deposit, printed to stdout. One form at a time: stdout has no way to
# say where one document ends and the next begins
cbso-fetch deposit 2021-00000148
cbso-fetch deposit 2021-00000148 --formats xbrl,pdf --destination ./data

# An improvement instead of the account as filed. Written under the name the
# package reads back, so `-corrected` becomes `-aa-corrected` or `-sb-corrected`
cbso-fetch deposit 2021-00000148 --improvement corrected --destination ./data

# What an enterprise has filed, by enterprise number or VAT number
cbso-fetch legal-entity 0203201340 --year 2024
cbso-fetch legal-entity BE0203201340
```

Production is the default. `--env test`, or its shorthand `--test`, points at
the bank's UAT2 deployment instead, which needs its own keys. The two cannot
both be given. Run with `-v` to see which host was resolved.

Exit codes: `0` success, `1` other failure, `2` usage error, `3` a subscription
key is missing, `4` the service refused, could not be reached, or answered with
something unreadable — or the destination stopped answering. Only `2` means the
command itself was wrong, so a wrapper can tell what is worth retrying from what
needs a person.

## Library

Each product has a client that stands alone:

```python
from datetime import date

from cbso_webservice_client import AuthenticClient, ExtractsClient

# One deposit
accounts = AuthenticClient().accounting_data("2021-00000148")

# A whole day, as a zip archive
archive = ExtractsClient().accounting_data(date(2026, 8, 3))
```

Keys can be passed instead of being read from the environment, and the test
environment is one argument away:

```python
from cbso_webservice_client import Environment, ExtractsClient

client = ExtractsClient(key="...", environment=Environment.TEST)
```

### Enterprise numbers

The webservices address a legal entity by its KBO/BCE enterprise number, and
accept nothing else: pass a VAT number and the service refuses it. This package
takes either, along with the dots and spaces a number is usually written with,
and converts to the enterprise number before making the request:

```python
from cbso_webservice_client import parse_number

parse_number("BE 0203.201.340").value  # '0203201340'
parse_number("0203201340").vat  # 'BE0203201340'
```

A number is checked before any request is made: ten ASCII digits, a leading `0`
or `1`, and check digits equal to `97 - (first eight mod 97)`. A value that
fails raises `InvalidNumberError` naming the rule it broke, rather than costing
a round trip answered with a bare 400. Digits from another script are refused
rather than sent: they satisfy `str.isdigit` without necessarily converting, and
one that does convert would still be percent-encoded into a URL the service
cannot answer.

`CbsoClient` gathers all five, building each on first use:

```python
from cbso_webservice_client import CbsoClient

client = CbsoClient()
references = client.authentic.legal_entity_references("0203201340")
```

### Reading references

A reference is the record tying a deposit to the enterprise that filed it:

```python
from cbso_webservice_client import AuthenticClient

reference = AuthenticClient().reference("2021-00000148")

reference.enterprise_number  # '0203201340'
reference.exercise_dates  # Period(start_date=..., end_date=...)
reference.model_type.schema_type  # SchemaType.FULL
reference.from_pdf  # whether no XBRL exists for it
```

The query operations answer in PascalCase and the JSON inside the batch
archives is camelCase. Both parse through the same path, so you never have to
know which one you are holding.

### Downloading

`Downloader` walks a range of dates and stores everything under a destination:

```python
from cbso_webservice_client.download import Downloader, recent_dates
from cbso_webservice_client.outputs import writer_for

report = Downloader(writer_for("./data")).run(recent_dates(7))

print(report.summary())
```

A destination is a local directory or an `s3://bucket/prefix` URL. Documents
are filed by artefact and enterprise:

```text
references/0203201340/2021-00000148.json
jsons/0203201340/2021-00000148.json
xbrls/0203201340/2021-00000148.xbrl
improved-references/0203201340/2021-00000148-aa-corrected.json
improved-jsons/0203201340/2021-00000148-aa-corrected.json
_state/extracts/2026-08-03.zip-jsonxbrl.json
```

The last of those is a marker rather than data: it is what a rerun reads to know
the day is done. Nothing downstream is expected to depend on this layout.

An `s3://` destination needs `s3:GetObject` and `s3:PutObject` beneath the
prefix. `s3:ListBucket` is not required, but granting it makes the run's
skipping sharper: without it S3 answers a check for an object that is not there
with `403` rather than `404`, which cannot be told apart from a check that was
refused for some other reason. A run establishes which case it is in once, by
asking whether it may list at all, and then treats `403` accordingly. A
destination that stops answering altogether stops the run rather than reading as
empty, since a run treats "not there" as work already done.

What a run does is controlled by `DownloadOptions`:

```python
from cbso_webservice_client.api import Representation
from cbso_webservice_client.download import DownloadOptions, Downloader, recent_dates
from cbso_webservice_client.outputs import writer_for

options = DownloadOptions(
    representations=(Representation.ZIP_JSONXBRL,),
    include_improved=False,
    concurrency=8,
)

Downloader(writer_for("s3://my-bucket/cbso"), options=options).run(recent_dates(7))
```

Pass `archive=True` to `run()` to ask the archive products, which reach back
three years, rather than the daily ones.

### Resuming

Each batch is recorded as done in the destination, per product, per date and
per representation. A repeated run skips what it already has, and a run that
stopped part-way repeats only the step that failed. Because the record lives
with the data, a run against S3 resumes correctly from a different machine.

A day the service had nothing for is recorded as done as well, but only once
its batches were due: the bank publishes a day from 05:00 Belgian time on the
following day, so an empty answer before then is asked for again on the next
run rather than written off. The report says which of the two happened.

### When a deposit cannot be attributed

Batch files are named after the deposit alone, so the enterprise comes from
that day's references. A document whose reference is missing has its reference
fetched on its own; if that finds nothing, the document is set aside under
`pending/` rather than being lost. It is retried once the run's remaining
batches have been read, since those are what may now name it, and by later runs
after that.

A document named after an improvement the bank has only just started publishing
goes there too, since its name still says which deposit it belongs to.

What no later run can place is kept under `unfiled/` instead — a reference
naming no enterprise, a reference that will not parse, an improvement whose kind
has no name here. Nothing retries those; they are kept because the day is
recorded as done either way, so a batch entry not written somewhere now is one
nothing would ever ask for again. Both roots are listed in the report's
`unresolved`, as the path each record was written to.

### Errors

Everything the package raises descends from a small set of types:

| Exception | Meaning |
|-----------|---------|
| `MissingKeyError` | No subscription key for the product being contacted |
| `NotFoundError` | The service holds nothing at that address |
| `HttpError` | The service answered with an error status |
| `UnreachableError` | The service never answered |
| `ReferenceParseError` | A reference document could not be read |
| `ArchiveError` | A batch payload could not be read as a zip archive |
| `InvalidNumberError` | A value is not a well-formed enterprise number |
| `OutputError` | The destination could not say what it holds |

`ArchiveError`, `ReferenceParseError` and `InvalidNumberError` all subclass
`ValueError`, which is worth knowing if you catch that. Only the last of the
three means the call was made wrongly; the other two mean what came back could
not be read, which is why `cbso-fetch` exits `4` for them. `NotFoundError` is a
subclass of `HttpError`, and both descend from `TransportError`. Server-side statuses and
connection failures are retried with doubling backoff, honouring a `Retry-After`
header when the service sends one, before any of them is raised; a missing key
never is.

A key missing for a product a run only wanted to ask about single deposits is
reported instead of raised: those operations belong to the authentic and improved
products, and a run holding only the archive keys does less rather than stopping
on every rerun at the same deposit. The report's `missing` names what went
unasked.

Connections are pooled and kept alive, which matters because filling the gaps a
batch left makes several requests per deposit. Raise `max_connections` on the
transport above the `concurrency` a run uses, or the surplus connections are
opened and discarded again.

## Licence

Proprietary. © openthebox.
