Metadata-Version: 2.4
Name: zeptodb
Version: 0.0.3
Summary: Python client and DataFrame integration for ZeptoDB — ultra-low latency columnar time-series database
Project-URL: Homepage, https://zeptodb.io
Project-URL: Repository, https://github.com/ZeptoDB/ZeptoDB
Project-URL: Documentation, https://docs.zeptodb.io
Author: ZeptoDB Contributors
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: HFT,analytics,columnar,database,finance,timeseries
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Financial and Insurance Industry
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Topic :: Office/Business :: Financial
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.9
Provides-Extra: all
Requires-Dist: duckdb>=0.9; extra == 'all'
Requires-Dist: numpy>=1.24; extra == 'all'
Requires-Dist: pandas>=2.0; extra == 'all'
Requires-Dist: polars>=0.20; extra == 'all'
Requires-Dist: pyarrow>=14.0; extra == 'all'
Provides-Extra: arrow
Requires-Dist: pyarrow>=14.0; extra == 'arrow'
Provides-Extra: dev
Requires-Dist: duckdb>=0.9; extra == 'dev'
Requires-Dist: numpy>=1.24; extra == 'dev'
Requires-Dist: pandas>=2.0; extra == 'dev'
Requires-Dist: polars>=0.20; extra == 'dev'
Requires-Dist: pyarrow>=14.0; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Provides-Extra: duckdb
Requires-Dist: duckdb>=0.9; extra == 'duckdb'
Provides-Extra: pandas
Requires-Dist: numpy>=1.24; extra == 'pandas'
Requires-Dist: pandas>=2.0; extra == 'pandas'
Provides-Extra: polars
Requires-Dist: polars>=0.20; extra == 'polars'
Description-Content-Type: text/markdown

<div align="center">

# ⚡ ZeptoDB

### In-Memory Time-Series Database for High-Throughput Workloads

*Ingest millions of events per second. Analyze them in microseconds.*

[![C++20](https://img.shields.io/badge/C%2B%2B-20-blue?logo=cplusplus)](https://en.cppreference.com/w/cpp/20)
[![LLVM 19](https://img.shields.io/badge/LLVM-19-orange?logo=llvm)](https://llvm.org/)
[![Highway SIMD](https://img.shields.io/badge/SIMD-Highway-green)](https://github.com/google/highway)
[![Tests](https://img.shields.io/badge/tests-830%2B%20passing-brightgreen?logo=googletest)](tests/)
[![License](https://img.shields.io/badge/License-BSL_1.1-blue)](LICENSE)
[![Docs](https://img.shields.io/badge/docs-docs.zeptodb.io-blue?logo=readthedocs)](https://docs.zeptodb.io)

<!-- Uncomment when ready:
[![Discord](https://img.shields.io/discord/DISCORD_SERVER_ID?color=5865F2&logo=discord&logoColor=white&label=Discord)](https://discord.gg/zeptodb)
[![Docker Pulls](https://img.shields.io/docker/pulls/zeptodb/zeptodb?logo=docker)](https://hub.docker.com/r/zeptodb/zeptodb)
[![PyPI](https://img.shields.io/pypi/v/zeptodb?logo=python&logoColor=white)](https://pypi.org/project/zeptodb/)
-->

[Quick Start](#-quick-start) · [Performance](#-performance) · [SQL Examples](#-sql-examples) · [Docs](https://docs.zeptodb.io) · [Contributing](CONTRIBUTING.md)

</div>

---

<!-- TODO: Replace with actual GIF recording (asciinema or screen capture)
     Record: docker run → INSERT → SELECT → Web UI query editor
     Tool: asciinema rec demo.cast && agg demo.cast demo.gif
-->
<!--
<div align="center">
  <img src="docs/assets/demo.gif" alt="ZeptoDB Demo" width="720">
  <br><em>From zero to query results in 30 seconds</em>
</div>
-->

## What is ZeptoDB?

ZeptoDB is an in-memory columnar database purpose-built for time-series analytics at scale.

It handles **high-throughput ingestion** and **real-time analytical queries** simultaneously — without trade-offs between the two.

The engine is hardware-software co-optimized: Highway SIMD vectorization, LLVM JIT compilation, lock-free ring buffers, NUMA-aware allocation, and UCX/RDMA networking — all working together to eliminate unnecessary copies, allocations, and cache misses.

```
┌─────────────────────────────────────────────────────────────┐
│  Clients: HTTP API · Python DSL · C++ API · Arrow Flight    │
├─────────────────────────────────────────────────────────────┤
│  SQL Engine: Parser (1.5μs) · AST Optimizer · Executor      │
├─────────────────────────────────────────────────────────────┤
│  Execution: Highway SIMD · LLVM JIT · Partition-parallel    │
│  ASOF JOIN · Window JOIN · xbar · EMA · VWAP                │
├─────────────────────────────────────────────────────────────┤
│  Ingestion: Lock-free MPMC Ring Buffer · WAL · Feed Handlers│
├─────────────────────────────────────────────────────────────┤
│  Storage: Arena Allocator · Column Store · Tiered (→S3)     │
├─────────────────────────────────────────────────────────────┤
│  Cluster: Consistent Hashing · RF=2 · Auto Failover         │
├─────────────────────────────────────────────────────────────┤
│  Security: TLS · JWT/OIDC · RBAC · Audit (SOC2/MiFID II)   │
└─────────────────────────────────────────────────────────────┘
```

---

## 🚀 Quick Start

### Docker (fastest)

```bash
docker run -p 8123:8123 zeptodb/zeptodb:0.0.1

# Insert data
curl -X POST http://localhost:8123/ \
  -d "INSERT INTO trades VALUES (1, 1714000000000000000, 185.50, 100)"

# Query
curl -X POST http://localhost:8123/ \
  -d "SELECT vwap(price, volume), count(*) FROM trades WHERE symbol = 'AAPL'"
```

### Build from Source

```bash
# Dependencies (Amazon Linux 2023 / Fedora)
sudo dnf install -y clang19 clang19-devel llvm19-devel \
  highway-devel numactl-devel ucx-devel ninja-build lz4-devel

mkdir -p build && cd build
cmake .. -G Ninja -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER=clang-19 -DCMAKE_CXX_COMPILER=clang++-19
ninja -j$(nproc)

./zepto_http_server --port 8123
```

### Python

```python
import zeptodb

db = zeptodb.Pipeline()
db.start()
db.ingest(symbol=1, price=185.50, volume=100)
db.drain()

# Zero-copy numpy access (522ns)
prices = db.get_column(symbol=1, name="price")
```

📖 Full guide: [Quick Start](https://docs.zeptodb.io/getting-started/QUICK_START/) · [Python Reference](docs/api/PYTHON_REFERENCE.md) · [SQL Reference](docs/api/SQL_REFERENCE.md)

---

## 📊 Performance

Single node. End-to-end latencies including SQL parsing. No cherry-picking.

| Operation | Latency | Notes |
|-----------|---------|-------|
| **Ingestion throughput** | **5.52M events/sec** | Lock-free MPMC ring buffer |
| Filter 1M rows | **272μs** | Highway SIMD vectorized scan |
| VWAP 1M rows | **532μs** | Fused price×volume aggregation |
| GROUP BY (8 threads) | **248μs** | Partition-parallel scatter/gather |
| EMA 1M rows | **2.2ms** | Streaming exponential moving average |
| Window SUM 1M rows | **1.36ms** | Prefix-sum O(n) algorithm |
| xbar (1M → 3,334 bars) | **11ms** | Time-bucketed OHLCV |
| SQL parse | **1.5–4.5μs** | Recursive descent, zero allocation |
| Python column access | **522ns** | Zero-copy shared memory |
| Indexed lookup (g#/p#) | **3.3μs** | 274× faster than full scan |
| HDB flush to disk | **4.8 GB/s** | LZ4 compressed |
| Partition routing | **2ns** | Consistent hash ring |

---

## 💡 SQL Examples

```sql
-- 5-minute OHLCV candlestick bars
SELECT xbar(timestamp, 300000000000) AS bar,
       first(price) AS open, max(price) AS high,
       min(price) AS low, last(price) AS close,
       sum(volume) AS volume
FROM trades WHERE symbol = 'AAPL'
GROUP BY xbar(timestamp, 300000000000)

-- ASOF JOIN (point-in-time lookup)
SELECT t.price, q.bid, q.ask
FROM trades t
ASOF JOIN quotes q
ON t.symbol = q.symbol AND t.timestamp >= q.timestamp

-- EMA with delta
SELECT symbol, price,
       EMA(price, 20) OVER (PARTITION BY symbol ORDER BY timestamp) AS ema20,
       DELTA(price) OVER (ORDER BY timestamp) AS price_change
FROM trades

-- Window JOIN (time-range aggregation)
SELECT t.price, wj_avg(q.bid) AS avg_bid
FROM trades t
WINDOW JOIN quotes q ON t.symbol = q.symbol
AND q.timestamp BETWEEN t.timestamp - 5000000000 AND t.timestamp + 5000000000

-- Materialized view (incremental, updated on ingest)
CREATE MATERIALIZED VIEW ohlcv_5min AS
  SELECT symbol, xbar(timestamp, 300000000000) AS bar,
         first(price) AS open, max(price) AS high,
         min(price) AS low, last(price) AS close,
         sum(volume) AS vol
  FROM trades
  GROUP BY symbol, xbar(timestamp, 300000000000)

-- Storage tiering
ALTER TABLE trades SET STORAGE POLICY
  HOT 1 HOURS WARM 24 HOURS COLD 30 DAYS DROP 365 DAYS
```

Full SQL reference: [SQL_REFERENCE.md](docs/api/SQL_REFERENCE.md) — INSERT, UPDATE, DELETE, CASE WHEN, LIKE, UNION, CTE, subqueries, and more.

---

## 🏗️ Use Cases

| Domain | Why ZeptoDB | Key Features |
|--------|------------|--------------|
| **Finance / HFT** | Sub-ms tick processing, kdb+-class perf | ASOF JOIN, xbar, EMA, VWAP |
| **Quant Research** | Backtest in Python, execute in C++ | Zero-copy numpy, Polars DSL |
| **Crypto / DeFi** | 24/7 multi-exchange streaming | Binance feed handler, real-time agg |
| **IoT / Manufacturing** | High-frequency sensor ingestion | DELTA/RATIO, time-bar agg, LZ4 |
| **Autonomous Vehicles** | Sensor fusion, driving log replay | ASOF JOIN, Parquet HDB, parallel scan |
| **Observability** | High-cardinality metrics | SQL + Grafana, TTL + S3 tiering |

---

## ⚙️ Optimization Stack

<table>
<tr><th>Hardware</th><th>Software</th></tr>
<tr><td>

- **Highway SIMD** — 256/512-bit vectorized scans
- **NUMA-aware** — memory pinned to local node
- **UCX / RDMA** — kernel-bypass networking
- **Arena allocator** — zero GC, zero fragmentation

</td><td>

- **LLVM JIT** — runtime expression compilation
- **Lock-free MPMC** — zero-contention ingestion
- **Columnar storage** — cache-friendly sequential access
- **Partition-parallel** — 3.48× scaling at 8 threads

</td></tr>
</table>

---

## 🔒 Enterprise Security

| Feature | Details |
|---------|---------|
| TLS/HTTPS | OpenSSL 3.2, cert/key PEM |
| Authentication | API Key (SHA256) + JWT/OIDC (HS256/RS256, JWKS auto-fetch) |
| Authorization | RBAC: 5 roles + symbol-level ACL + multi-tenancy |
| Rate Limiting | Token bucket per-identity + per-IP |
| Secrets | Vault KV v2 → K8s secrets → env var (priority chain) |
| Audit Log | 7-year retention, SOC2/EMIR/MiFID II compliant |

---

## 🚢 Deployment

```bash
# Docker
docker run -p 8123:8123 zeptodb/zeptodb

# Helm
helm install zeptodb ./deploy/helm/zeptodb

# Bare-metal (systemd)
./deploy/scripts/install_service.sh
```

Guides: [Production Deployment](docs/deployment/PRODUCTION_DEPLOYMENT.md) · [Kubernetes](docs/operations/KUBERNETES_OPERATIONS.md) · [Bare-metal Tuning](docs/deployment/BARE_METAL_TUNING.md)

---

## 🔄 Migration

Migrate from existing databases with built-in tooling:

```bash
./zepto-migrate --source kdb+ --hdb-path /data/hdb --target localhost:8123
```

Supported: **kdb+** (HDB loader, q→SQL) · **ClickHouse** (DDL/query conversion) · **DuckDB** (Parquet) · **TimescaleDB** (hypertable conversion)

---

## 🤝 Contributing

We welcome contributions! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

<!-- Uncomment when Discord is set up:
## 💬 Community

- [Discord](https://discord.gg/zeptodb) — questions, discussions, help
- [GitHub Discussions](https://github.com/zeptodb/zeptodb/discussions) — design proposals, RFCs
- [Twitter/X](https://twitter.com/zeptodb) — announcements
-->

---

## 📄 License

[Business Source License 1.1](LICENSE) — Production use permitted, except offering as a commercial DBaaS. Changes to Apache 2.0 on 2030-04-01.

For commercial licensing: skswlsaks@gmail.com
