Metadata-Version: 2.4
Name: persistence-kit
Version: 3.10.1
Summary: Reusable persistence and repository toolkit
License: MIT
License-File: LICENSE
Keywords: repository,persistence,mongodb,sqlalchemy,async
Author: Andres Felipe Serrano Barrios
Author-email: andresfserrano1@gmail.com
Requires-Python: >=3.11,<4.0
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Database
Classifier: Typing :: Typed
Provides-Extra: all
Provides-Extra: api
Provides-Extra: dynamodb
Provides-Extra: restclient
Provides-Extra: security
Provides-Extra: security-cognito
Provides-Extra: storage-routes
Provides-Extra: storage-s3
Provides-Extra: testing
Requires-Dist: asyncpg (>=0.30.0,<1.0.0)
Requires-Dist: boto3 (>=1.35.0,<2.0.0) ; extra == "storage-s3" or extra == "security-cognito" or extra == "dynamodb" or extra == "all"
Requires-Dist: fastapi (>=0.115.0,<1.0.0) ; extra == "api" or extra == "storage-routes" or extra == "security" or extra == "security-cognito" or extra == "testing" or extra == "all"
Requires-Dist: httpx (>=0.28.0,<1.0.0) ; extra == "restclient" or extra == "testing" or extra == "all"
Requires-Dist: motor (>=3.7.1,<4.0.0)
Requires-Dist: pydantic-settings (>=2.3.0,<3.0.0)
Requires-Dist: pyjwt[crypto] (>=2.10.1,<3.0.0) ; extra == "security" or extra == "security-cognito" or extra == "testing" or extra == "all"
Requires-Dist: python-multipart (>=0.0.31,<1.0.0) ; extra == "api" or extra == "storage-routes" or extra == "security" or extra == "security-cognito" or extra == "testing" or extra == "all"
Requires-Dist: sqlalchemy[asyncio] (>=2.0.43,<3.0.0)
Requires-Dist: typing-extensions (>=4.12.0,<5.0.0)
Project-URL: Documentation, https://github.com/andresfserrano/persistence-kit#readme
Project-URL: Homepage, https://pypi.org/project/persistence-kit/
Project-URL: Repository, https://github.com/andresfserrano/persistence-kit
Description-Content-Type: text/markdown

# persistence-kit

The plumbing every one of our services rewrites: repositories, settings, object
storage, authentication, an HTTP client and a cache. You describe your entities
once and the kit gives you an async repository for them, backed by memory,
MongoDB, PostgreSQL or DynamoDB, without your domain code knowing which one.

Nothing here knows about your business. Roles, permissions, and domain rules stay
in your application.

- Python 3.11+ · async everywhere · type hints shipped (`py.typed`)
- Optional dependencies stay out of the graph until you ask for them

## Five minutes

```bash
pip install persistence-kit
```

```python
from dataclasses import dataclass, field
from uuid import UUID, uuid4

from persistence_kit import Database
from persistence_kit.repository_factory import get_repo, register_entity


@dataclass
class User:
    email: str
    name: str
    id: UUID = field(default_factory=uuid4)


register_entity("user", {
    "entity": User,
    "collection": "users",
    "database": Database.MEMORY,
    "unique": {"email": "email"},
})

repo = get_repo("user")
await repo.add(User(email="ada@example.org", name="Ada"))
ada = await repo.get_by_index("email", "ada@example.org")
```

The repository protocol is small and the same for every backend: `add`, `get`,
`update`, `delete`, `get_by_index`, `list`, `list_by_fields`, `count` and
`count_by_fields`.

Switch `database` to `Database.POSTGRES` and the same code talks to PostgreSQL.
That is the whole point of the kit.

## The one idea: the entity registry

Everything else hangs off a single dictionary. You register each entity once,
usually in a `register_defaults()` function in your app, and from then on you ask
for repositories by key.

```python
from persistence_kit.repository_factory import set_registry_initializer

set_registry_initializer(register_defaults)   # called during startup
```

What goes in a registration:

| Key | Meaning |
| --- | --- |
| `entity` | The dataclass this key maps to |
| `collection` | Table or collection name in the backing store |
| `database` | `Database.MEMORY`, `MONGO`, `POSTGRES` or `DYNAMODB`. Defaults to `REPO_DATABASE` |
| `unique` | Indexed lookups, `{"index_name": "field"}`, used by `get_by_index` |
| `relations` | How this entity joins to others, including many-to-many through a pivot |

Relations are what `get_repo_view(...)` reads to return an entity with its
related rows already populated, so you do not write joins by hand. The shapes and
the traps are in [`docs/repositories_and_relations.md`](docs/repositories_and_relations.md).

Four ways to reach a repository, all equivalent:

```python
get_repo("user")                    # plain repository
get_repo_view("user")               # repository that populates relations
provide_repo("user")                # FastAPI dependency
provide_view_repo("user")           # FastAPI dependency, populated
```

## Installation and extras

The base install stays light. Each capability that needs a heavy dependency
lives behind an extra, so a service that only talks to Mongo never installs
`boto3` or `fastapi`.

```bash
pip install "persistence-kit[api,security,restclient]"
```

| Extra | Gives you | Pulls in |
| --- | --- | --- |
| `api` | FastAPI exceptions, pagination, route loading, error handlers | fastapi, python-multipart |
| `security` | Memory identity provider and JWT verifier | fastapi, pyjwt, python-multipart |
| `security-cognito` | The above plus AWS Cognito and JWKS | + boto3 |
| `storage-s3` | S3 object storage adapter | boto3 |
| `storage-routes` | FastAPI route to serve local exports | fastapi, python-multipart |
| `dynamodb` | DynamoDB repository backend | boto3 |
| `restclient` | The REST client and its cache | httpx |
| `testing` | Everything the test suite needs | fastapi, pyjwt, httpx, python-multipart |
| `all` | Every optional capability | all of the above |

Importing `persistence_kit` never loads an optional dependency by itself. Ask for
something you did not install and you get a message telling you which extra to
add, not an obscure `ImportError`.

## What is in the box

| Module | What it does |
| --- | --- |
| `contracts/` | The protocols: `Repository`, `ViewRepository`, `ObjectStorage`, `IdentityProvider` |
| `repository/` | One implementation per backend: memory, Mongo, SQLAlchemy, DynamoDB |
| `repository_factory/` | The entity registry, the factory, and the view repository that populates relations |
| `settings/` | `RepoSettings` and `PersistenceKitSettings`, plus shared enums and parsers |
| `storage/` | Object storage: local directory or S3, with presigned URLs |
| `security/` | Identity providers and JWT verifiers, memory or Cognito |
| `restclient/` | HTTP client with pluggable auth, endpoint resolution, retries and DTO decoding |
| `cache/` | Key-value cache with TTL: memory, Mongo or DynamoDB |
| `resilience/` | Circuit breaker, shared by anything that calls out |
| `api/` | Reusable FastAPI pieces: error handlers, pagination, rate limiting |
| `bootstrap/` | Startup helpers, configuration registry, seed orchestration |
| `utils/` | Small transversal helpers such as upserts |

Import from `persistence_kit` when the public facade is enough. Reach into a
subpackage only when you need something implementation-specific. The root
exports about a hundred names, so let your editor complete them rather than
keeping a list here.

## Object storage

Writing generated files without your domain layer knowing where they land.

```python
from persistence_kit.storage import LocalObjectStorage

storage = LocalObjectStorage(
    base_dir=".local",
    public_base_url="http://localhost:8000",
    signing_secret="dev-secret",
)

key = await storage.upload("exports/report.csv", b"id,name\n1,Ada\n", "text/csv")
url = await storage.generate_presigned_url(key)
```

`LocalObjectStorage` writes to a directory and signs its own download URLs.
`S3ObjectStorage` uploads to S3 and returns AWS presigned URLs. Pick one with
`get_export_storage(settings)`, which reads your settings and caches the adapter.

FastAPI apps can mount `build_local_export_storage_router(...)` to serve local
files, passing their settings provider, an optional current-user dependency, and
an authorization callback.

Needs `[storage-s3]` for S3 and `[storage-routes]` for the route.

## Security

```python
from persistence_kit.security import MemorySecurityProvider, MemoryJwtVerifier

identity = MemorySecurityProvider(
    jwt_secret="dev-secret-with-enough-length",
    jwt_issuer="local-sandbox",
    seed_role_users=True,
    seed_role_codes=("admin", "operator"),
    seed_user_domain="example.org",
)
verifier = MemoryJwtVerifier(secret="dev-secret-with-enough-length", issuer="local-sandbox")
```

Two protocols, `IdentityProvider` and `TokenVerifier`, with a memory pair for
tests and local work and a Cognito pair for real deployments. Registration,
login, and password-reset results come back as dataclasses, not raw dicts.
`get_identity_provider(settings)` and `get_token_verifier(settings)` build the
right pair from your settings.

Your roles, authorization policies and route permission matrices stay in your
application. The kit only answers "who is this".

Needs `[security]`, or `[security-cognito]` for Cognito.

## REST client and cache

For calling somebody else's HTTP API without writing the same client again.
Register a service once, then ask for its client:

```python
from persistence_kit.restclient import get_rest_client, register_rest_service, ServiceConfig

register_rest_service(
    "billing",
    base_url="https://billing.example.org/api",
    config=ServiceConfig(
        default_headers={"X-Api-Key": settings.billing_key},
        timeout_seconds=10,
        cacheable=True,
        cache_ttl_seconds=300,
    ),
)

client = get_rest_client("billing")
```

It brings pluggable authentication (`NoAuth`, `ApiKeyAuth`, `BasicAuth`,
`BearerAuth`, `OAuth2ClientCredentials`, `LoginTokenAuth`), endpoint resolution
that can read a service directory, a retry policy, DTO decoding through
`decode`, and optional response caching. `MemoryRestClient` stands in for the
real one in tests.

Base URLs can be overridden per environment with `REST_SERVICE_URLS`, a JSON map
of service name to URL, without touching the registration.

The cache is a plain key-value store with TTL, chosen by `CACHE_BACKEND` the
same way repositories are chosen by `REPO_DATABASE`. Set `CACHE_NAMESPACE` when
several apps share one DynamoDB table or Mongo collection so their keys do not
collide.

Both are covered in depth in [`docs/restclient_and_cache.md`](docs/restclient_and_cache.md).

Needs `[restclient]`.

## Resilience

`CircuitBreaker` is domain-agnostic and shared by anything that calls out: after
enough consecutive failures it opens, fails fast with `CircuitOpenError` instead
of hammering a service that is already down, and closes again after a cooldown.
`RetryPolicy`, which lives with the REST client, handles the retry side.

## Settings

`PersistenceKitSettings` gathers what most services need: auth, storage,
observability, AWS and job-service settings. Inherit from it and override only
what is yours.

```python
from persistence_kit import Database, PersistenceKitSettings


class Settings(PersistenceKitSettings):
    service_name: str = "my-api"
    key_status_history_database: Database = Database.MONGO
    memory_seed_role_codes: tuple[str, ...] = ("admin", "operator")
```

The factories read that object: `get_identity_provider`, `get_token_verifier`,
`get_export_storage`, `get_cache`.

### Environment variables

| Variable | Meaning |
| --- | --- |
| `REPO_DATABASE` | `memory`, `mongo`, `postgres` or `dynamodb`. Default backend for every entity |
| `CACHE_BACKEND` | `memory`, `mongo` or `dynamodb`. Defaults to `memory` |
| `CACHE_NAMESPACE` | Key prefix, so apps sharing a cache store do not collide |
| `REST_SERVICE_URLS` | JSON map `{"service": "base_url"}` overriding registered REST base URLs |
| `MONGO_DSN`, `MONGO_DB` | Mongo connection |
| `POSTGRES_DSN` | Full DSN, if you prefer it to the pieces below |
| `POSTGRES_USER`, `POSTGRES_PASSWORD`, `POSTGRES_HOST`, `POSTGRES_PORT`, `POSTGRES_DB` | Postgres connection |
| `POSTGRES_SSL` | Set when the server requires TLS |
| `DYNAMODB_REGION`, `DYNAMODB_TABLE_PREFIX` | DynamoDB connection |

## Wiring it into an application

1. Define your entities as dataclasses.
2. Register them in a `register_defaults()` in your app.
3. Call `set_registry_initializer(register_defaults)` at startup.
4. Resolve repositories with `get_repo(...)`, `get_repo_view(...)` or the FastAPI providers.
5. Use `ConfigRegistry` and `SeederProvider` as bootstrap infrastructure only. What gets registered and seeded stays in your app.

## Trying a change against an app on your machine

This is the everyday loop, and it needs no release, no PR and no TestPyPI. It
assumes the kit and the app sit next to each other:

```
aux programación udea/
├── persistence_kit/          <- this repository
└── api_store_manager_v1/     <- the app that consumes it
```

Install the kit into the app's environment in editable mode, from the app:

```bash
cd api_store_manager_v1
.venv/Scripts/pip install -e ../persistence_kit      # Linux/macOS: .venv/bin/pip
```

From that moment the app imports your working copy. Edit the kit, rerun the app
or its tests, and the change is already there. No reinstall between edits.

To go back to the published version:

```bash
.venv/Scripts/pip uninstall persistence-kit
poetry install
```

Two things that will confuse you if nobody warns you:

- **The reported version lies.** In editable mode `importlib.metadata.version("persistence-kit")`
  keeps returning whatever the old `dist-info` said, even though the code running
  is your source. Trust `persistence_kit.__file__`, not the version string.
- **The app pins an exact version.** `api_store_manager_v1` asks for a pinned
  `persistence-kit` with its extras, so `poetry install` will happily undo your
  editable install. Redo it after any dependency change.

Run the kit's own tests with the app's interpreter, which already has every extra
installed:

```bash
"../api_store_manager_v1/.venv/Scripts/python.exe" -m pytest -q
```

Only when the change has to be tried from a machine that is not yours does it
make sense to publish a preview, which is the next section.

## Adding something to the kit

Every capability here is built the same way. Follow the shape and your module
will look like it was always part of the kit.

**1. Start with the contract.** Declare what the thing does before writing how.
The repository pair uses `ABC`; everything else uses `Protocol`, so an
implementation only has to match the shape, not inherit from anything.

```python
from typing import Protocol


class Mailer(Protocol):
    async def send(self, *, to: str, subject: str, body: str) -> None: ...
```

Everything is `async` and arguments are keyword-only. Both rules exist so callers
read clearly at the call site and so adding a parameter never breaks anyone.

**2. Write two implementations, not one.** A memory one and a real one. The
memory one is not a toy: it is what tests and local development run against, and
having it forces the contract to stay honest. `InMemoryTTLCache` next to the
Mongo cache, `MemorySecurityProvider` next to Cognito, `MemoryRestClient` next to
the httpx one.

**3. Choose between them with a factory driven by settings.** Never with an `if`
scattered around the app. The factory reads an enum from settings, caches with
`lru_cache`, and imports the provider lazily inside the builder so an unused
backend never loads its dependency.

```python
@lru_cache
def get_mailer(settings):
    if settings.mailer_backend is MailerBackend.SES:
        from persistence_kit.mailer.ses import SesMailer   # imported only if used
        return SesMailer(region=settings.aws_region)
    return InMemoryMailer()
```

**4. Give the package its own error hierarchy.** One base exception and specific
subclasses, the way `storage/errors.py` and `restclient/errors.py` do it. Callers
should be able to catch the whole family or one precise case.

**5. Export it lazily.** If it needs an optional dependency, add it to
`_OPTIONAL_EXPORTS` in the root `__init__.py` and declare its extra in
`pyproject.toml`. Someone who imports `persistence_kit` without that extra must
get a message naming the extra, not an `ImportError`. `tests/test_capabilities.py`
enforces this and will fail if you skip it.

**6. Add tests next to the feature.** Async tests carry `@pytest.mark.asyncio`.

**Getting it merged.** `main` is protected, so it goes through a pull request.
Branch off `main`, keep the change to one capability, run the suite, and open the
PR. Anything domain-specific, roles, business rules, product settings, belongs in
the application that consumes the kit, not here.

## Development

From the kit's root, with its own environment:

```bash
poetry lock
poetry install --with dev --all-extras
poetry run pytest -q
```

Current baseline: **434 tests passing** (version 3.9.1). Async tests use
`pytest-asyncio` in strict mode, so each one carries `@pytest.mark.asyncio`.

`tests/test_capabilities.py` guards the lazy-import promise: it fails if merely
importing `persistence_kit` drags in fastapi, pyjwt or boto3.

## Releasing

The normal path, and the only one that produces an official version:

1. Merge to `main`. The branch is protected, so it goes through a pull request.
2. Bump `version` in `pyproject.toml`.
3. Publish a GitHub Release with tag `vX.Y.Z` pointing at `main` HEAD.
4. `.github/workflows/publish-pypi.yml` checks the tag matches `main`, builds, and
   publishes to PyPI through Trusted Publishing. No tokens stored anywhere.

For previews that others need to install before there is a release, use a
`X.Y.Z.devN` version:

- From GitHub: run the `Publish Preview Package` workflow, enter the version, and
  pick `testpypi`. It patches the version inside the CI run only, so no commit.
- From your machine: `bash ./scripts/publish-local.sh 3.9.2.dev1 testpypi`, with a
  TestPyPI token exported as `TWINE_PASSWORD`.

Then, in the consuming project:

```bash
pip install --index-url https://test.pypi.org/simple/ \
            --extra-index-url https://pypi.org/simple persistence-kit==3.9.2.dev1
```

Neither PyPI nor TestPyPI lets you re-upload a version, so bump the `devN` on
every iteration.

## Further reading

- [`docs/repositories_and_relations.md`](docs/repositories_and_relations.md): relations, pivots, and how the view repository populates them.
- [`docs/restclient_and_cache.md`](docs/restclient_and_cache.md): the REST client, its auth strategies, and the cache.

Author: Andres Felipe Serrano Barrios · [github.com/AndresFSerrano/persistence-kit](https://github.com/AndresFSerrano/persistence-kit)

