Metadata-Version: 2.5
Name: insightsmith
Version: 0.7.3
Summary: Agentic data consultant that runs on your own hardware
Project-URL: Homepage, https://github.com/SofianeOuaari/insightsmith
Project-URL: Repository, https://github.com/SofianeOuaari/insightsmith
Project-URL: Issues, https://github.com/SofianeOuaari/insightsmith/issues
Project-URL: Changelog, https://github.com/SofianeOuaari/insightsmith/blob/main/CHANGELOG.md
Author: Sofiane Ouaari
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: cli,data-analysis,data-profiling,eda,polars
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: charset-normalizer>=3.3
Requires-Dist: httpx>=0.27
Requires-Dist: polars>=1.0
Requires-Dist: psutil>=5.9
Requires-Dist: rich>=13.7
Requires-Dist: tomli>=2.0; python_version < '3.11'
Requires-Dist: typer>=0.12
Provides-Extra: all
Requires-Dist: fastexcel>=0.11; extra == 'all'
Requires-Dist: kaleido>=0.2; extra == 'all'
Requires-Dist: matplotlib>=3.8; extra == 'all'
Requires-Dist: pandas>=2.0; extra == 'all'
Requires-Dist: plotly>=5.20; extra == 'all'
Requires-Dist: pyarrow>=15; extra == 'all'
Requires-Dist: scikit-learn>=1.4; extra == 'all'
Requires-Dist: scipy>=1.11; extra == 'all'
Requires-Dist: statsmodels>=0.14; extra == 'all'
Provides-Extra: excel
Requires-Dist: fastexcel>=0.11; extra == 'excel'
Provides-Extra: pandas
Requires-Dist: pandas>=2.0; extra == 'pandas'
Requires-Dist: pyarrow>=15; extra == 'pandas'
Provides-Extra: stats
Requires-Dist: scikit-learn>=1.4; extra == 'stats'
Requires-Dist: scipy>=1.11; extra == 'stats'
Requires-Dist: statsmodels>=0.14; extra == 'stats'
Provides-Extra: viz
Requires-Dist: kaleido>=0.2; extra == 'viz'
Requires-Dist: matplotlib>=3.8; extra == 'viz'
Requires-Dist: plotly>=5.20; extra == 'viz'
Description-Content-Type: text/markdown

<div align="center">

<img src="https://raw.githubusercontent.com/SofianeOuaari/insightsmith/main/assets/logo.png" alt="insightsmith" width="200">

# insightsmith

**Forging insight from raw data.**

An agentic data consultant that runs on your own machine.

</div>

---

> ### What is here today
>
> **Format detection, loading and profiling** (`ismith look`),
> **hardware probing with model-fit recommendation** (`ismith doctor`), the
> **provider layer** routing roles to local or cloud models (`ismith models`),
> the **dataset card plus ideation** (`ismith look --ideas`), **sandboxed code
> execution** answering real questions (`ismith ask`), **charts** with a
> validated palette (`ismith ask --chart`), and a **statistical critic** that
> prints what is wrong with an answer before you act on it.
>
> Nothing sends your data anywhere unless you configure a remote provider
> yourself.

---

## What it does today

```bash
pip install insightsmith
ismith look data/sales.csv
```

```
╭──────────────── source ─────────────────╮
│ format    csv                           │
│ encoding  cp1252                        │
│ dialect   delimiter=';' decimal=','     │
│ confidence 95%                          │
│ assumed   read as cp1252; charset-      │
│           normalizer suggested cp775    │
╰─────────────────────────────────────────╯
umsatz.csv: 5 rows x 3 columns
        columns
┏━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┓
┃ column ┃ dtype   ┃ semantic    ┃ nulls ┃ unique ┃ detail                ┃
┡━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━┩
│ region │ String  │ categorical │     - │      4 │ Nord (2) · Süd (1)    │
│ menge  │ Int64   │ numeric     │     - │      5 │ min 4 · med 12 · max… │
│ umsatz │ Float64 │ numeric     │     - │      5 │ min 40.75 · med 120.5 │
└────────┴─────────┴─────────────┴───────┴────────┴───────────────────────┘
candidate keys: menge
```

Add `--json` for the same profile as machine-readable output.

Columns holding **numbers stored as text** are marked as such. A JSON document
carries no types, so `"92"` arrives as a string and any arithmetic on it fails
with nothing in the schema to have warned anyone. A document keyed by record id,
`{"Abomasnow": {...}, "Starly": {...}}`, is read as the table it is rather than
as one row a thousand columns wide.

### Which local model fits your machine

```bash
ismith doctor
```

Probes CPU, RAM, disk and GPU, then sizes each catalogued model against what it
finds, **per role**, because routing and planning are different jobs and your
machine may afford one but not the other.

The arithmetic is the point. Weights come from the quantisation's bytes-per-param;
the KV cache from `2 × layers × n_kv_heads × head_dim × context × 2 bytes`. Using
`n_kv_heads` rather than the attention-head count is what makes it right: on a
4 GB laptop GPU at 8k context, `qwen3:8b` (8 KV heads) needs 1.21 GB of cache
against 4.92 GB of weights, while `deepseek-coder:6.7b` (same size, but 32 KV
heads and no grouped-query attention) needs **4.29 GB of cache against 4.20 GB
of weights**. A rule of thumb based on parameter count alone cannot tell you that.

Models that don't fit outright get a partial-offload layer count rather than a
shrug, and anything larger than your RAM is excluded with a reason.

### Getting an actual answer

```bash
ismith ask data/sales.csv "total sales by product type?"
```

The model writes a Polars snippet, it runs, and you get the number, plus the
code that produced it, so the result is checkable rather than taken on trust.
When the snippet fails, the traceback goes back to the model and it tries again,
up to three times, then reports the failure honestly instead of inventing an
answer.

**The model does not have to remember the Polars API.** Every model has read far
more pandas than Polars, so left alone it reaches for `groupby` and
`sort_values` and spends a retry discovering they do not exist. A Polars
reference ships inside the package, and each question pulls the two or three
sections that bear on it: `group_by` for an aggregation, `.over()` for a
per-group total, the pitfalls list for a pandas habit. On a retry the traceback
becomes the query and outweighs the question: `no attribute 'groupby'` names the
mistake, where the question only named the goal. The retrieval is BM25 over the
guide's sixty-odd sections in pure stdlib, with no embedding model, no second round
trip, no index to rebuild. `--no-guide` turns it off.

**A failure says what to use, not only what broke.** A traceback names the
mistake and stops there, so a model can spend every remaining attempt
rediscovering the same wrong name. Where the failure implies its own fix, the
retry carries it:

| The snippet wrote | What comes back |
|---|---|
| `df.groupby(...)` | Polars spells it `group_by` |
| `df.sort_values(...)` | the Polars equivalent is `.sort()` |
| `pl.sqrt(x)` | there is no `pl.sqrt()`; every expression has `.sqrt()` |
| `.group_by("k").mean("x")` | name the column in `agg` instead |
| a column that does not exist | the columns that do |
| arithmetic on a text column | cast it first |

Every one of those is checked against the installed polars before it is offered,
so a suggestion can never name a method the library does not have. Replies that
arrive double-escaped, with newlines as a literal backslash and an `n`, are
repaired rather than retried: Python reads them as a line continuation, and
every retry would reproduce it.

**The code runs in a separate process behind six layers of defence** (design doc
§7): an allowlist AST gate that refuses `eval`, `exec`, `open`, `getattr`,
dunder attributes and every import outside the analysis stack; an isolated
interpreter with a scrubbed environment; CPU, memory and file-size limits;
a Parquet copy of the data in a scratch directory rather than a path into your
tree; and `--approve` to see each snippet before it runs.

**Every answer is checked before you see it.** A number is easy to produce and
hard to trust, so each result is reviewed and the caveats printed under it:

```text
0.11683640472865998

qualified (confidence 0.70) · read the caveats
  • Pearson correlation assumes a linear relationship and no heavy tails, but
    avg_rating and num_published_lectures carry outliers. A few extreme rows can
    create or hide this number entirely. Check it against method="spearman".
```

Almost everything the critic knows it **measures** rather than asks: a
correlation reported on outliered columns, a mean on a skewed one, groups of
three described as a comparison, forty p-values with no correction, a division
that produced an infinity, missing values the code never acknowledged, and an
answer computed on a sampled file rather than the whole of it. Those are
arithmetic, and asking a small model whether statistics are sound produces
confident prose with nothing behind it, which is the failure this is meant to
reduce rather than reproduce.

It also catches the commonest way an answer goes wrong: **answering a different
question**. Ask for compliance violations *per year* and get back one overall
number, and that is not the answer, however correct the arithmetic. Where the
question names a real column to group by and the result is a single row, that is
settled by counting rather than by opinion, and the snippet goes back to the
coder to be written properly. Asked to judge the same results, a small model
called every one of them sound.

What genuinely cannot be computed is whether a result answers a question in
looser phrasing, and that is the only part a model is given. Statistical caveats
never trigger a retry, because rewriting a snippet cannot make the data less
skewed.

The confidence figure is a derived index, not a probability that the answer is
right. It falls by a fixed weight per caveat so more or worse findings always
score lower, and identical inputs always score the same. Read the caveats; they
say something specific. `--no-critique` turns the whole pass off.

Add `--chart` and the answer is drawn as well as printed:

```bash
ismith ask data/sales.csv "total sales by product type?" --chart
```

![A ranked horizontal bar chart of total sales by product type](https://raw.githubusercontent.com/SofianeOuaari/insightsmith/main/assets/chart-example.png)

**The model picks the chart, it does not draw it.** It returns a form and which
column fills which role; the renderer draws that spec. So charts are consistent,
reproducible for a given spec, and no plotting code written by a model ever runs.
A spec naming a column the result doesn't have is discarded, exactly as an idea
would be.

The palette was validated with a colour-blindness and contrast checker rather
than chosen by eye, and two of its measurements are enforced in code: slots are
assigned in a fixed order and a ninth series raises instead of inventing a hue,
and scatter caps at three series because all-pairs forms fail the separation
floor at four. Bars are ranked, long tails fold into "other", and one series
gets one hue, because colouring every bar differently implies a distinction the
data does not contain.

When a result cannot be drawn, the reason says which way it cannot. A single
correlation coefficient is a number with nothing to plot it against, which is a
different problem from having no numeric column at all, and being told the wrong
one sends you looking in the wrong place.

Each run saves a PNG and a self-contained interactive HTML next to a manifest
recording the question, the code and the card hash, so a figure found later can
be traced back to the data behind it.

**Read [SECURITY.md](SECURITY.md) before pointing this at anything sensitive.**
It is defence in depth against a model erring by accident, which is the
realistic failure, and explicitly *not* a security boundary against a deliberately
malicious prompt. The resource limits are POSIX-only; on Windows the gate and
the timeout are all there is.

### Asking a model what's worth analysing

```bash
ismith look data/sales.csv --ideas    # ranked analyses
ismith look data/sales.csv --card     # exactly what the model will be shown
```

**The dataframe is never pasted into a prompt.** Everything an agent sees arrives
through a *dataset card*: a compact JSON summary of schema, per-column statistics,
quality flags, a correlation shortlist, and a few stratified example rows with
obvious PII masked. Three things follow, and they are the reason for the design:

- **Token cost is flat regardless of file size.** A 589 KB, 4248-row, 20-column
  file produces a 4.8 KB card, and so would a 40 GB one. That is what makes an
  8B local model workable.
- **No raw records leave the machine.** Sensitive columns are redacted by name,
  recognisable values by pattern.
- **The card hashes**, so the same data yields the same plan and results cache.

Ideas come back ranked, each naming the columns it needs, and **any idea
referencing a column the card doesn't contain is discarded before you see it**.
That one check removes most hallucination for the price of a set-membership test.

![Eight ranked analysis ideas for a sales dataset, each naming the columns it needs](https://raw.githubusercontent.com/SofianeOuaari/insightsmith/main/assets/ideas-example.png)

Every column named above (`Market`, `Product Type`, `State`, `Marketing`,
`Total Expenses`) exists in the file. Anything else was dropped before it
reached the table.

The quality notes travel on the card too, so a model hedges where the data
warrants it: given a date column flagged as ambiguous, it proposes parsing it
"with caution" rather than trusting it.

### Wiring up a model

```bash
ismith models
```

Shows what each role resolves to, whether it stays on your machine, and **how it
will be asked for structured output**. Configure it in `~/.insightsmith/config.toml`:

```toml
[roles]
planner = "ollama/qwen3:8b"
coder   = "ollama/qwen2.5-coder:7b"
cheap   = "ollama/qwen3:4b"

[budget]
max_usd_per_session = 0.50
local_only = true
```

One class covers every backend that speaks the OpenAI wire format: OpenAI,
OpenRouter, DeepInfra, Together, Groq, Fireworks, Mistral, Gemini's compat
endpoint, and local vLLM / llama.cpp / LM Studio. They differ only by
base URL and key. Ollama is written natively instead, since `/api/show`,
`/api/ps` and `keep_alive` are the whole reason to run locally.

**Capabilities are read, not assumed.** Ollama reports whether a model supports
tool-calling, so the router picks its approach up front rather than failing
mid-run: tool-calling where available, JSON mode next, and otherwise prompted
JSON with a bounded retry that feeds the parse failure back. Small models wrap
JSON in prose and code fences no matter how firmly told not to, so that path is
the common case, not an edge case.

**`local_only = true` is a hard failure, not a warning.** Point a role at a
remote provider with it set and loading the config raises, naming the offending
role. A privacy switch that only warned would not be a privacy switch.

**Format detection doesn't trust the extension.** A three-stage cascade of
extension hint, then magic bytes, then a text-dialect probe, where each stage can veto the
one before it. A Parquet file named `.csv` is loaded as Parquet, and you're told
the extension lied. Every result carries a confidence score and the list of
assumptions behind it; below 80% those assumptions are printed rather than hidden.

It is built for the files that actually turn up: semicolon-delimited cp1252 CSVs
with decimal commas and thousands separators, BOMs, comment preambles, quoted
fields containing the delimiter, gzip and single-member zip wrappers.

**Profiling** reports per-column dtype and semantic type, null rates, cardinality,
numeric summaries with outlier counts by two different methods, temporal ranges,
text lengths, category frequencies, candidate keys, and quality notes: duplicate
and near-duplicate rows, constant and near-constant columns, runaway cardinality,
class imbalance.

### As a library

```python
from insightsmith import load, profile, sniff

spec = sniff("data/sales.csv")
print(spec.format, spec.encoding, spec.confidence, spec.warnings)

frame = load(spec)  # a Polars LazyFrame; nothing read yet
result = profile(spec)
print(result.summary())
for issue in result.issues:
    print(issue.severity.value, issue.column, issue.message)
```

Polars `LazyFrame` is the internal representation throughout, so Parquet, Arrow,
NDJSON and UTF-8 CSV are scanned rather than loaded.

**Formats loadable today:** csv, tsv, xlsx/xlsm (with `[excel]`), xls, parquet,
feather/arrow, json, jsonl/ndjson. Detected-but-not-yet-loadable formats (sqlite,
duckdb, xml, html, ods, orc, hdf5, spss, stata, sas) say so, and name the release
that will handle them.

## Install

```bash
pip install insightsmith            # csv, tsv, parquet, arrow, json, jsonl
pip install insightsmith[excel]     # + xlsx / xls
pip install insightsmith[viz]       # + charts (matplotlib, plotly)
pip install insightsmith[pandas]    # + a .to_pandas() escape hatch
pip install insightsmith[stats]     # + scipy / statsmodels / scikit-learn
```

The base install is six dependencies: polars, typer, rich, charset-normalizer,
psutil and httpx (plus a TOML backport on Python 3.10 only). No torch, no pandas,
no agent framework. Extras stay optional on purpose.

## What it won't do

Worth stating plainly, in advance:

- **Large files are profiled on a sample.** Above a size threshold the profile is
  built from a strided sample of the rows, and every affected statistic is marked
  `estimated`. Row counts remain exact; distributions are approximate. An answer
  computed on such a file carries a caveat saying so.
- **The critic reduces statistical nonsense; it does not remove it.** It checks a
  fixed list of things that are computable, so it cannot catch a confounder, a
  survivorship bias, or a question that was the wrong question to ask. A clean
  verdict means nothing on the list fired, not that the analysis is sound.
- **`--chart` often has nothing to draw.** Plenty of good questions have a single
  number for an answer, and a correlation coefficient or an overall average
  cannot be plotted against anything. Roughly two answers in five come back as
  one value, and the chart is skipped with the reason given rather than a
  meaningless figure produced.
- **Encoding detection is a guess on small files.** Single-byte codepages are
  genuinely ambiguous in a few hundred bytes. Sparse non-ASCII text is read as
  cp1252 and the substitution is reported, but it can still be wrong.
- **Statistics have edges.** The IQR fence is degenerate when the middle 50% of a
  column is a single value, so outliers are counted by a MAD-based modified
  z-score as well, and both numbers are shown. They disagree usefully.
- **"Near-duplicate" means one specific thing**: identical once string columns
  are trimmed and case-folded. It is not fuzzy matching.
- **PII masking is best-effort, not a guarantee.** It catches values that look
  like contact details or identifiers and blanks columns whose names say they
  hold personal data. It cannot recognise a person's name in free text, an
  address split across columns, or an identifier in a format it hasn't seen. If
  data must not leave the machine, set `local_only`. Masking is defence in
  depth, not a substitute for keeping it local. `--card` shows exactly what would
  be sent, and `include_examples=False` omits sample values entirely.
- **Dates are inferred, and `04/01/10` is genuinely ambiguous.** Date columns load
  as strings and the format is inferred afterwards, so a file polars cannot parse
  still loads. Where day-first and month-first fit equally well, the chosen format
  is reported as a quality note rather than picked silently. Check it before
  trusting anything grouped by that column.
- **Thousands separators are detected but not stripped.** polars has no option for
  them, so such columns may load as strings.
- **No XML row-unit discovery, and zstd payloads aren't inspected** (no stdlib
  decompressor before Python 3.14).
- **Throughput figures are estimates, not benchmarks.** `ismith doctor` derives
  tok/s from published peak memory bandwidth, which no real decode loop reaches.
  Devices missing from that table report `unknown` rather than a plausible
  substitute, and the catalog only contains models whose layer and KV-head counts
  were read from a running Ollama, never guessed.

- **LLMs write wrong code confidently.** `ismith ask` prints the code for exactly
  that reason, so check it. The retry loop fixes code that *crashes*; it cannot
  tell that a snippet ran cleanly and answered the wrong question. The
  statistical critic that catches some of that arrives in 0.7.0.
- **The sandbox is defence in depth, not a security boundary.** See
  [SECURITY.md](SECURITY.md). Resource limits are POSIX-only.

## License

Apache-2.0. See [LICENSE](LICENSE).
