Metadata-Version: 2.4
Name: nrchd
Version: 0.1.0
Summary: Local-first LLM enrichment pipeline for CSV files
License: MIT
License-File: LICENSE
Keywords: ai,csv,enrichment,llm,streamlit
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.10
Requires-Dist: aiofiles>=23.0.0
Requires-Dist: anthropic>=0.28.0
Requires-Dist: chromadb>=0.5.0
Requires-Dist: google-genai>=1.0.0
Requires-Dist: groq>=0.9.0
Requires-Dist: httpx>=0.27.0
Requires-Dist: numpy>=1.26.0
Requires-Dist: openai>=1.30.0
Requires-Dist: openpyxl>=3.1.0
Requires-Dist: pandas>=2.0.0
Requires-Dist: pypdf>=4.0.0
Requires-Dist: python-docx>=1.1.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: scikit-learn>=1.4.0
Requires-Dist: sentence-transformers>=3.0.0
Requires-Dist: streamlit>=1.35.0
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.23.0; extra == 'dev'
Requires-Dist: pytest>=8.0.0; extra == 'dev'
Description-Content-Type: text/markdown

# nrchd — LLM Enrichment Pipeline for CSV Files

> Give me any CSV, tell me what you want to know or generate per row,
> and I'll run it through an LLM at scale and give you back an enriched CSV.

**nrchd** is a local-first, resumable LLM enrichment pipeline with a Streamlit UI. Upload a CSV, define a task and output schema, optionally attach a knowledge base, and run structured LLM enrichment across every row — in parallel, with crash recovery, and with no data leaving your machine unless you choose a cloud LLM.

---

## Features

- **Any CSV → enriched CSV** — define your own prompt and output schema per task
- **Structured output** — schema validation and type coercion per field (bool, string, int, float)
- **Resumable** — crash or stop mid-run; re-run picks up exactly where it left off
- **Parallel + rate-limited** — configurable concurrency and calls/minute
- **RAG (Retrieval-Augmented Generation)** — attach PDF/DOCX/TXT/MD/CSV knowledge bases; relevant chunks are injected per row
- **Semantic analysis** — local dedup, clustering, outlier detection, cross-file similarity, semantic search (no API needed)
- **Multiple LLM backends** — Claude, GPT-4o-mini, Groq Llama, Ollama (local)
- **Task templates** — built-in templates for common tasks; save your own
- **No cloud dependency** — use Ollama + local sentence-transformers for a fully offline pipeline

---

## Quick Start

### Prerequisites

- Python 3.10+
- [uv](https://docs.astral.sh/uv/) (recommended) — install with:
  ```bash
  curl -LsSf https://astral.sh/uv/install.sh | sh
  ```
- At least one API key (Anthropic, OpenAI, or Groq) — or [Ollama](https://ollama.com) for local inference

### Setup

```bash
git clone https://github.com/mortalapps/nrchd
cd nrchd
make setup        # installs deps, creates .env from .env.example
```

Edit `.env` and add your API key(s):

```bash
ANTHROPIC_API_KEY=sk-ant-...
# OPENAI_API_KEY=sk-...
# GROQ_API_KEY=gsk_...
```

### Run

```bash
make run          # opens http://localhost:8501
```

### Docker (alternative)

```bash
docker compose up
```

---

## Supported Models

### LLM Backends

| Provider | Model | Key needed | Notes |
|----------|-------|-----------|-------|
| Claude (Haiku) | claude-haiku-4-5 | `ANTHROPIC_API_KEY` | Fast, cost-effective |
| Claude (Sonnet) | claude-sonnet-4-6 | `ANTHROPIC_API_KEY` | Best quality |
| GPT-4o-mini | gpt-4o-mini | `OPENAI_API_KEY` | Good balance |
| Groq Llama 3.1 70B | llama-3.1-70b | `GROQ_API_KEY` | Free tier, very fast |
| Ollama | any local model | None | Fully local |

### Embedding Models (local, no API)

| Model | Speed | Quality |
|-------|-------|---------|
| all-MiniLM-L6-v2 | Fast | Good (default) |
| all-mpnet-base-v2 | Medium | Better |

---

## Built-in Templates

| Template | Description |
|----------|-------------|
| `question_validator` | Validate MCQ questions for quality, clarity, difficulty |
| `sentiment_analyser` | Classify sentiment, extract themes, draft responses |
| `support_ticket_triage` | Categorise, prioritise, and draft replies to support tickets |

Load templates from the **Task** tab. Save your own with the "Save as template" button.

---

## Workflow Overview

```
1. Data tab     — upload CSV, select ID column, filter rows
2. Analyse tab  — dedup, cluster, find outliers, cross-file similarity (optional)
3. Knowledge tab — ingest documents into a named RAG knowledge base (optional)
4. Task tab     — write prompt, define output schema, test on 3 rows
5. Run tab      — start pipeline; watch real-time progress
6. Results tab  — view stats, filter, semantic search, download
```

For a full step-by-step guide, see [HELP.md](HELP.md).

---

## Use Cases

| Input CSV | Knowledge Base | Task |
|-----------|---------------|------|
| Question bank | Textbook / lecture notes | Validate answers, generate hints |
| Product reviews | Product specs | Sentiment + suggested response |
| Support tickets | Policy docs | Categorise, prioritise, draft reply |
| Legal clauses | Regulation PDFs | Flag risk, plain-English summary |
| Research abstracts | Domain taxonomy | Tag topics, novelty score |
| Job descriptions | Skills ontology | Enrich with skills, seniority |

---

## Configuration

All settings can be overridden in `.env`:

```bash
ANTHROPIC_API_KEY=
OPENAI_API_KEY=
GROQ_API_KEY=
OLLAMA_BASE_URL=http://localhost:11434
DEFAULT_BATCH_SIZE=100
DEFAULT_MAX_CONCURRENT=10
DEFAULT_MAX_PER_MINUTE=10
```

---

## Project Structure

```
nrchd/
├── app.py              # Streamlit entry point
├── tabs/               # UI tabs (one file per tab)
├── engine/             # Core logic (no Streamlit dependency)
│   ├── batch_runner.py # Async batch executor
│   ├── llm_clients.py  # Claude / OpenAI / Groq / Ollama wrappers
│   ├── rag.py          # ChromaDB ingest + retrieval
│   ├── semantic.py     # Sentence Transformer features
│   ├── response_parser.py
│   ├── rate_limiter.py
│   └── utils.py
├── templates/          # Built-in + user task templates (JSON)
├── knowledge_bases/    # Persisted ChromaDB indexes (gitignored)
├── outputs/            # Generated CSVs (gitignored)
└── tests/              # pytest test suite
```

---

## Contributing

- To add a new LLM provider: implement `async def complete(prompt, system, max_tokens, temperature)` and add it to `LLMClient` in `engine/llm_clients.py`
- To add a built-in template: create a JSON file in `templates/` following the schema in [docs/technical_architecture.md](docs/technical_architecture.md)
- To add a new document type to RAG: extend `_read_file()` in `engine/rag.py`

---

## License

MIT with Attribution — free to use, copy, and modify. Any use or derivative work must credit:

**nrchd by [Mortalapps.com](https://mortalapps.com)**

See [LICENSE](LICENSE) for the full terms.
