Metadata-Version: 2.4
Name: diskstructs
Version: 0.1.1
Summary: High-performance disk-based data structures for Python
Author: walker-21
Author-email: walker-21 <57685058+walker-21@users.noreply.github.com>
License-Expression: MIT
License-File: LICENSE
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Python: >=3.14
Project-URL: Homepage, https://github.com/walker-21/diskstructs
Project-URL: Issues, https://github.com/walker-21/diskstructs/issues
Description-Content-Type: text/markdown

# diskstructs

**High-performance, disk-backed data structures for Python.**

`diskstructs` is a lightweight, plug-and-play Python package designed for managing persistent, file-based data structures with minimal overhead and zero external database dependencies.

The initial implementation focuses on a **Multi-Producer, Multi-Consumer JSON File Queue** built specifically for low-latency, cross-process task distribution.

---

## Key Features & Architecture

* **Atomic File Movement:** Uses OS-level atomic moves (`os.replace`) to eliminate partial-read race conditions between concurrent producers and consumers.
* **Workspace & Queue Namespaces:** Isolates structures into named directories (`.diskstructs/queues/<queue_name>/`), allowing multiple data structures to coexist under a single workspace.
* **Sub-Second Latency:** Low-latency execution ($\le 1\text{s}$) powered by lightweight directory polling and recovery logic.
* **Metadata & Object Inspection:** Built-in workspace discovery (`list_objects`), metrics tracking (`stats`), and object lifecycle control (`clear`, `delete`).
* **Built for Package Extensibility:** Modular layout designed to easily incorporate future disk-backed data structures (such as disk-backed stacks, key-value stores, and priority queues).

---

## Directory Architecture

Each named queue maintains an isolated workspace layout:

```text
.diskstructs/
└── queues/
    └── <queue_name>/
        ├── metadata.json   # Object tracking and creation metadata
        ├── temp/           # Non-blocking, isolated atomic payload writes
        ├── pending/        # Active queued items ordered by microsecond timestamps
        └── processing/     # Items claimed by active consumers
```

## Quickstart

### Installation & Initialization

```Python
import diskstructs as ds
# Configure global workspace directory
config = ds.DiskConfig(root_dir="./.diskstructs")
```

### Produce Items (Sender)

```Python
producer = ds.QueueProducer(config, name="orders_queue")

# Non-blocking atomic push
item_id = producer.send({"order_id": 1001, "status": "pending"})
print(f"Pushed item: {item_id}")
```

### Consume Items (Receiver)

```Python
consumer = ds.QueueConsumer(config, name="orders_queue")

# Poll for a single item
item = consumer.poll_once()

if item:
    print("Processing payload:", item.payload)
    consumer.acknowledge(item)  # Remove from queue upon success
```

### Workspace Management & Listing

```Python
# List all diskstructs objects and real-time metrics across the project
objects = ds.list_objects(config)
print(objects)

# Object-level operations via QueueManager
manager = ds.QueueManager(config, name="orders_queue")
print(manager.stats())

# Clear pending/processing items without removing queue metadata
manager.clear()

# Completely purge queue from disk
# manager.delete()
```

## Future Roadmap

* Disk-backed Stacks (LIFO)
* Disk-backed Key-Value Store
* Priority Queues
* Cross-Platform OS file-watcher integrations (watchdog / inotify)
