Metadata-Version: 2.4
Name: latent-insights
Version: 0.1.0
Summary: Parallel agent sensemaking tool — collaborative async data analysis with LLM threads
License: MIT
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: FastAPI
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.12
Requires-Dist: duckdb>=1.2
Requires-Dist: fastapi>=0.115
Requires-Dist: openai>=1.60
Requires-Dist: pydantic>=2.10
Requires-Dist: python-multipart>=0.0.18
Requires-Dist: sse-starlette>=2.2
Requires-Dist: uvicorn[standard]>=0.34
Provides-Extra: dev
Requires-Dist: httpx>=0.28; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.25; extra == 'dev'
Requires-Dist: pytest>=8.3; extra == 'dev'
Requires-Dist: ruff>=0.9; extra == 'dev'
Description-Content-Type: text/markdown

# Latent Insights

Parallel-agent sensemaking for collaborative data analysis.
For any uploaded dataset, the system discovers questions, spawns analytical threads, executes LLM orchestrated tool calls, and builds insights with you.

The agent is designed to follow steps from a sensemaking process such as foraging for evidence, framing the hypothesis, investigating the data, and synthesizing the results. 

## Install

```bash
pip install latent-insights
```

## Quick start

### OpenRouter (default)

```bash
export LLM_API_KEY=<your-key>
latent-insights
```

### Ollama (local, free)

```bash
export LLM_PROVIDER=ollama
latent-insights
```

Override individual models if needed:

```bash
export LLM_PROVIDER=ollama
export MODEL_WORKER=gemma3:4b
latent-insights
```

### As a library

```python
from latent_insights import AppConfig
from latent_insights.core.llm import LLMClient
from latent_insights.core.queue import Queue
from latent_insights.core.state import StateStore
from latent_insights.core.tracing import TraceStore
from latent_insights.db.connection import Database
from latent_insights.orchestration.session import SessionFlow

config = AppConfig.from_env()
llm = LLMClient(
    api_key=config.llm_api_key,
    base_url=config.llm_base_url,
    app_name=config.app_name,
    app_url=config.app_url,
)
db = Database(data_dir=config.data_dir)
queue = Queue()
state = StateStore(data_dir=config.data_dir)
trace = TraceStore(data_dir=config.data_dir)

flow = SessionFlow(config, llm, db, queue, state, trace)
flow.create(session_id, "path/to/data.csv")
```

## Development

```bash
git clone https://github.com/karthikbadam/latent-insights.git
cd latent-insights
uv sync --extra dev

# Run dev server with hot reload
uv run uvicorn latent_insights.main:app --reload

# Run tests
uv run pytest                    # all tests
uv run pytest -m "not live"      # skip API-calling tests
uv run ruff check .
```

## Architecture

```
POST /api/sessions (upload CSV)
         │
         ▼
┌─────────────-────┐
│     Session      │
│  Profiler → Scout│──── schema summary + seed questions
└────────┬─────-───┘
         │ spawns N threads
         ▼
┌─────────────-────┐   ┌────────────-─────┐   ┌─────────────-────┐
│    Thread 1      │   │    Thread 2      │   │    Thread N      │
│                  │   │                  │   │                  │
│  ┌────────────┐  │   │  ┌────────────┐  │   │  ┌────────────┐  │
│  │Coordinator │◄─┤   │  │Coordinator │◄─┤   │  │Coordinator │◄─┤
│  │  (judge)   │  │   │  │  (judge)   │  │   │  │  (judge)   │  │
│  └─────┬──────┘  │   │  └─────┬──────┘  │   │  └─────┬──────┘  │
│        │ decide  │   │        │         │   │        │         │
│        ▼         │   │        ▼         │   │        ▼         │
│  ┌────────────┐  │   │  ┌────────────┐  │   │  ┌────────────┐  │
│  │   Worker   │  │   │  │   Worker   │  │   │  │   Worker   │  │
│  │  LLM+SQL   │  │   │  │  LLM+SQL   │  │   │  │  LLM+SQL   │  │
│  └────────────┘  │   │  └────────────┘  │   │  └────────────┘  │
│                  │   │                  │   │                  │
│  Steps: SCOPE    │   │  Steps: FORAGE   │   │  Steps: FRAME    │
│  → FORAGE        │   │  → INTERROGATE   │   │  → INTERROGATE   │
│  → FRAME         │   │  → SYNTHESIZE    │   │  → SYNTHESIZE    │
│  → INTERROGATE   │   │  ✓ DONE          │   │  ? STUCK (human) │
│  → SYNTHESIZE    │   │                  │   │                  │
│  ✓ DONE          │   │                  │   │                  │
└────────────────-─┘   └───────────────-──┘   └───────────────-──┘
         │                    │                       │
         └────────────────────┴───────────────────────┘
                              │
                    GET /api/sessions/{id}/events (SSE)
                    ← llm_call, tool_call, step, complete
```

### Sensemaking moves


| Move            | Purpose                                                      |
| --------------- | ------------------------------------------------------------ |
| **SCOPE**       | Define data slice, narrow to relevant subset                 |
| **FORAGE**      | Exploratory analysis — distributions, correlations, outliers |
| **FRAME**       | Propose tentative hypothesis as testable claim               |
| **INTERROGATE** | Stress-test the frame — contradictions, confounds            |
| **SYNTHESIZE**  | Thread conclusion — finding, confidence, limitations         |


The coordinator picks moves freely based on data — no fixed order.

## API


| Endpoint                           | Description                                                   |
| ---------------------------------- | ------------------------------------------------------------- |
| `GET /health`                      | Health check                                                  |
| `POST /api/sessions`               | Create session (upload CSV + profile + scout + spawn threads) |
| `GET /api/sessions`                | List all sessions with metadata and thread counts             |
| `GET /api/sessions/{id}`           | Full session state with threads and steps                     |
| `POST /api/sessions/{id}/threads`  | Create custom thread with a question                          |
| `POST /api/sessions/{id}/continue` | Resume stuck threads + scout new questions                    |
| `GET /api/threads/{id}`            | Get single thread with steps and events                       |
| `POST /api/threads/{id}/messages`  | Reply to stuck thread, resuming it                            |
| `GET /api/sessions/{id}/events`    | SSE event stream (llm_call, tool_call, step, complete)        |
| `GET /api/system/stats`            | Session and thread counts                                     |


### Per-session config

`POST /api/sessions` accepts optional per-session overrides via a `config` object. All fields are optional — omitted fields use server defaults from environment variables.

```bash
curl -X POST http://localhost:8000/api/sessions \
  -F "file=@data/samples/cars.csv" \
  -F 'config={"max_threads": 20, "num_scout_seed_questions": 12, "initial_questions": ["What is an interesting visual insight in this dataset?", "What factors most influence fuel efficiency?", "Are there regional differences in car specifications?"]}'
```

Available config fields:


| Field                      | Type       | Description                               |
| -------------------------- | ---------- | ----------------------------------------- |
| `model_profiler`           | `string`   | Model for dataset profiling               |
| `model_scout`              | `string`   | Model for question discovery              |
| `model_coordinator`        | `string`   | Model for thread coordination             |
| `model_worker`             | `string`   | Model for SQL analysis                    |
| `model_worker_fallback`    | `string`   | Fallback model after worker retries       |
| `temp_profiler`            | `float`    | Temperature for profiler                  |
| `temp_scout`               | `float`    | Temperature for scout                     |
| `temp_coordinator`         | `float`    | Temperature for coordinator               |
| `temp_worker`              | `float`    | Temperature for worker                    |
| `max_threads`              | `int`      | Cap on total threads spawned              |
| `max_worker_retries`       | `int`      | Worker retries before fallback model      |
| `max_consecutive_errors`   | `int`      | SQL errors before forcing summary         |
| `max_repeated_moves`       | `int`      | Repeated coordinator moves before abort   |
| `llm_timeout`              | `float`    | LLM call timeout in seconds               |
| `num_scout_seed_questions` | `int`      | Number of questions scout should discover |
| `initial_questions`        | `string[]` | Seed questions to start alongside scout   |


## Publishing to PyPI

```bash
# Build the package
uv build

# Publish (requires PyPI API token)
uv publish

# Or test with TestPyPI first
uv publish --publish-url https://test.pypi.org/legacy/
```

## Docs

- [docs/SPEC.md](docs/SPEC.md) — architecture spec
- [docs/PROMPTS.md](docs/PROMPTS.md) — agent prompt designs

