Metadata-Version: 2.5
Name: s3-avldb
Version: 0.1.0
Summary: S3 storage backend for avldb
Project-URL: Documentation, https://github.com/anuradhawick/s3-avldb#readme
Project-URL: Issues, https://github.com/anuradhawick/s3-avldb/issues
Project-URL: Repository, https://github.com/anuradhawick/s3-avldb
Author-email: Anuradha Wickramarachchi <anuradhawick@gmail.com>
License-Expression: Apache-2.0 OR GPL-3.0-only
License-File: LICENSE
License-File: LICENSE-APACHE
License-File: LICENSE-GPL
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: avldb<0.2,>=0.1.1
Requires-Dist: boto3<2,>=1.35
Description-Content-Type: text/markdown

# s3-avldb

[![PyPI](https://img.shields.io/pypi/v/s3-avldb.svg)](https://pypi.org/project/s3-avldb/)
[![Python versions](https://img.shields.io/pypi/pyversions/s3-avldb.svg)](https://pypi.org/project/s3-avldb/)
[![CI](https://github.com/anuradhawick/s3-avldb/actions/workflows/ci.yml/badge.svg)](https://github.com/anuradhawick/s3-avldb/actions/workflows/ci.yml)
[![License: Apache-2.0 OR GPL-3.0-only](https://img.shields.io/badge/license-Apache--2.0%20OR%20GPL--3.0--only-green.svg)](https://github.com/anuradhawick/s3-avldb/blob/main/LICENSE)

`s3-avldb` provides an `S3Backend` for [`avldb`](https://pypi.org/project/avldb/) — a typed
embedded document database with Rust-backed AVL indexes. With `S3Backend` you can use any
AWS S3 bucket (or S3-compatible store such as MinIO or LocalStack) as durable storage.

## Installation

```console
pip install s3-avldb
```

Python 3.10+ and CPython are supported.

## Quick start

```python
import boto3
from avldb import Collection, Document
from s3avldb import S3Backend


class User(Document):
    name: str
    age: int


backend = S3Backend(bucket="my-bucket", prefix="users/")

with Collection(User, backend=backend) as users:
    users.ensure_index("age")
    users.insert(User(name="Ada", age=36))
    users.insert(User(name="Grace", age=29))

    adults = users.find({"age": {"$gte": 18}}).sort({"age": -1}).all()
```

## Configuration

```python
S3Backend(
    bucket="my-bucket",       # S3 bucket name
    prefix="myapp/users/",    # optional key prefix (no leading slash)
    client=None,              # inject a pre-configured boto3 S3 client, or None to create one
    endpoint_url=None,        # override endpoint for MinIO / LocalStack / other S3-compatible stores
    max_fetch_workers=None,   # default: all available logical CPUs; pass an int to limit
)
```

When `client` is `None`, `S3Backend` creates a boto3 S3 client on first use; standard boto3
credential discovery applies (environment variables, `~/.aws/credentials`, instance profiles, etc.).
Document reads use a bounded thread pool and preserve the backend's iteration order. By
default its size is the number of logical CPUs available to the process; set
`max_fetch_workers` to a positive integer to override it. When
injecting your own boto3 client, configure its `max_pool_connections` to at least the
resolved worker count; internally created clients are configured automatically.

## How it works

`S3Backend` follows the same `StorageBackend` / `BackendView` contract as `avldb`'s built-in
`DiskBackend` and `MemoryBackend`. The S3 object layout is:

```
{prefix}/manifest.json               ← atomic commit marker
{prefix}/content/{uuid}.jsonl        ← immutable JSONL content segments
{prefix}/indexes/{field_hash}/{uuid}.jsonl  ← immutable JSONL index segments
```

Atomicity is achieved via S3 **conditional writes** (`If-Match` on `PutObject`):
the manifest's ETag is used as an optimistic concurrency token. A conflicting commit from
another writer raises `WriteConflictError` exactly as it would with `DiskBackend`.

Indexes are loaded into in-memory AVL trees at `open()` time. Index segment objects are
downloaded concurrently using the configured fetch worker count, then decoded and replayed
in manifest order to preserve update semantics. Documents are fetched lazily through
concurrent byte-range `GetObject` calls.

No distributed lock is acquired — multiple processes may safely open the same prefix
concurrently using optimistic concurrency.

## Compaction

```python
with Collection(User, backend=S3Backend("my-bucket", "users/")) as users:
    users.backend.compact()
```

Compaction consolidates all live data into a new content segment and updates the manifest,
reducing the objects that future opens need to read. Previous segments are retained because
an existing immutable view—or another process—may still reference them. Configure an S3
lifecycle rule if you want to reclaim superseded segments after a retention period suitable
for your application.

## Non-AWS S3 stores

```python
backend = S3Backend(
    bucket="test-bucket",
    endpoint_url="http://localhost:9000",  # MinIO
)
```

Or inject a pre-built client:

```python
import boto3
client = boto3.client("s3", endpoint_url="http://localhost:9000")
backend = S3Backend(bucket="test-bucket", client=client)
```

## License

Copyright 2026 Anuradha Wickramarachchi.

Like [`avldb`](https://pypi.org/project/avldb/), `s3-avldb` is available under your choice
of the [Apache License 2.0](https://github.com/anuradhawick/s3-avldb/blob/main/LICENSE-APACHE)
or the [GNU General Public License v3.0 only](https://github.com/anuradhawick/s3-avldb/blob/main/LICENSE-GPL).
See the [dual-license notice](https://github.com/anuradhawick/s3-avldb/blob/main/LICENSE)
for details.
