Metadata-Version: 2.4
Name: valuekit
Version: 0.2.0
Summary: An immutable map for pipeline data, plus disk memoisation for pure functions.
Author-email: Ian Sheret <ian.sheret@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/iansheret/valuekit
Project-URL: Repository, https://github.com/iansheret/valuekit
Project-URL: Issues, https://github.com/iansheret/valuekit/issues
Project-URL: Changelog, https://github.com/iansheret/valuekit/blob/main/CHANGELOG.md
Keywords: immutable,memoization,caching,pipeline,pure-function,content-addressed
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Dynamic: license-file

# valuekit

Disk memoisation for pure functions, plus an immutable map for the
pipeline data they run over. Both apply the same idea: pipeline data as
immutable *values*, identified by content. The two parts are independent;
use either without the other.

The usual ways of caching a pipeline fail in one of two directions. If the
key is too coarse (a file path, a manual version tag), results go stale silently and hits stop
being trusted; if it is too broad (whole-argument hashes), one config edit
recomputes everything and hits stop happening. Either way the cache ends
up cleared before every run that matters, at which point it saves nothing.

valuekit is built so the cache can stay on. Invalidation follows what each
call actually read (change one config key and only the steps that read it
recompute) and what the code actually is (edit a helper and everything
that depends on it recomputes). What the tracking cannot see is a short
documented list, each entry with a remedy, and an uncertain match
recomputes rather than risk a stale result. Caching stays on under a
debugger too: the cached prefix replays in milliseconds, the step under
the breakpoint executes and stops there, and nothing done while paused
enters the cache.

## `@pure`

```python
from valuekit import pure, set_cache_dir

set_cache_dir("~/.cache/mypipeline")     # nothing is cached until this is called

@pure
def calculate_geometry(obs, config):
    order = config["geometry"]["order"]
    ...
    return {"az": az, "el": el}          # the returned dict is the diff

obs = obs | calculate_geometry(obs, config)
```

`obs` and `config` are `ImmutableMap`s here, and that is what buys the
per-key tracing: reads of a map argument are recorded individually, so a
change to a key the function never read does not invalidate it. Any other
argument — including a plain `dict` — is hashed whole. `ImmutableMap` has
its own section below.

Nothing else about the call changes. Arguments arrive as the objects you
passed, and the result is the object the function built, so a cache hit
differs from a miss only in that the body did not run.

`@pure` states a *contract*: the function's output depends only on what it
reads from its arguments, and it has no effects that matter. valuekit does
not verify this; it memoises to disk on the assumption that it holds. The
decorator takes no options, so there is nothing to configure per function.

Note that on a cache hit the function body does not run. Prints, plots,
progress bars, and file writes inside a `@pure` function will not happen on
replays. If a side effect matters, it does not belong in a pure function.

### The contract

The guarantee: a cache hit returns exactly what executing the current
definition on the current arguments would return. The user's promise: the
result depends only on what the function reads from its arguments, plus its
definition. "The definition" means everything reachable by name from the
function's code; if go-to-definition in an IDE can reach it from the
function, it is part of the function's *fingerprint*. Names are resolved at
the function's first call, once the module is fully loaded, so definition
order does not matter and mutual recursion works.

A result is recomputed when any of these change:

| What changed | Why it is tracked |
|---|---|
| a key the call read (or probed and found absent) in a map argument | each call records a trace of exactly what it read |
| the content of any non-map argument | arguments are hashed whole |
| the function's code, or any user function it calls, recursively (helpers, lambdas, methods of user classes, other `@pure` functions) | the recursive code hash |
| an immutable module constant it uses (numbers, strings, tuples, frozensets, read-only arrays) | constants are part of the definition; `x / SPEED_OF_LIGHT` and `x / 299792458.0` invalidate identically |
| default and closure values | part of the definition |
| the version of an installed package it uses, or the Python version | package and standard-library boundaries contribute version markers |

Whitespace, comments, and the function's name are not changes.

A stale result is served when the change was invisible to the fingerprint.
This is the user's responsibility, by design:

| Invisible to the fingerprint | Remedy |
|---|---|
| mutable globals (lists, dicts, sets, writeable arrays), whether rebound, mutated, or edited in source | make them constant (tuple, frozenset, `arr.flags.writeable = False`) or pass them as arguments |
| dispatch through data: `getattr(mod, name)()`, registries, callables stored in structures | pass the function as an argument; functions are hashed by fingerprint, so lambdas work |
| file contents read inside the function | pass the data, or its path and a version, as arguments |
| runtime purity violations: unseeded RNG or clock reads that reach the result, mutation of arguments or globals | none; these break the promise |

The remedy column repeats one idea: arguments are always tracked, so moving
a dependency into the arguments makes it visible. If something invisible
changed anyway, clear it. `clear_cache(fn)` means "`fn` has changed" and
behaves as if it had: it deletes the recorded results of `fn` and of every
`@pure` function that computed through it (callers, transitively, and uses
of `fn` as an argument), across processes, using a small on-disk dependency
index. Matching is conservative: clearing too much means recomputing, while
clearing too little would mean wrong results, so ties resolve towards
clearing more. `clear_cache()` deletes everything.

Tunables belong in config maps rather than in module globals. A traced
config read is exact per call (change an unread key and hits are kept),
while a module constant is definition-wide (edit it and every function
naming it recomputes).

Decoration emits no warnings. Side effects in a `@pure` function (logging,
progress bars, metrics) are permitted by the contract precisely because
they will not happen on a hit; whether that is acceptable is the user's
decision.

## The immutable map

`@pure` does not require the map — any hashable argument works — but the
map is how a function opts an argument into per-key invalidation. There
are three places it is worth using.

The first is granularity, and it is the reason the other two matter under
`@pure`. A plain `dict` argument is a single opaque value: nothing observed
which keys the function used, so any edit anywhere in it invalidates the
result. The same dict as an `ImmutableMap` is traced key by key.

The second is config. Tunables belong in a config passed as an argument,
where every read is traced; the same tunables in a module-level dict are a
mutable global, the first row of the stale-results table above. An
`ImmutableMap` config removes that hazard, since nothing can edit it in
place, and it makes derivation the way to vary settings: an override is
`cfg | {"gain": 2.0}`, a sweep is `[cfg.assoc("order", n) for n in orders]`,
and each variant is a distinct value that recomputes only the steps that
read the changed key.

The third is the data flowing between steps. Frozen state means no step
can mutate another's input, by design or by accident, and each step
returns a derived map instead of editing a shared one:

```python
from valuekit import ImmutableMap

ctx = ImmutableMap({"raw": signal, "fs": 1000.0})

ctx2 = ctx | {"scaled": ctx["raw"] * gain}   # derive; ctx is unchanged
ctx3 = ctx2.assoc("window", "hann")          # single-key derivation
ctx4 = ctx3.dissoc("tmp")                    # drop keys
```

Values are frozen on entry: numpy arrays become read-only (copied only if
writeable; set `arr.flags.writeable = False` beforehand to share without a
copy), sets become frozensets, nested dicts become ImmutableMaps, and
unknown mutable types are rejected with a `TypeError`. The rejection is
deliberate: a type must be registered (`register_type`) before it can be
stored, so nothing mutable gets in by accident. Deriving with `|` shares
unchanged values by reference, so adding one key to a 2 GB context copies
one dict, not 2 GB of data.

Freezing is the map's behaviour, and only the map's: `@pure` never converts
an argument or a result. Putting a value into a map is where you ask for it.

Using the map activates nothing else: no cache, no decorator, no
configuration.

## Read granularity

- `config["filter"]["order"]` records a dependency on that one leaf. Taking
  `f = config["filter"]` and then iterating, printing, or comparing `f`
  observes the whole subtree and records a whole-map dependency. Anything
  that looks at all keys (`len`, iteration, `==`, `keys()`) is a whole-map
  read: correct, but coarser.
- Deriving inside a `@pure` function works exactly as it does outside: `|`,
  `assoc` and `dissoc` are all available on the map you were passed, and
  return a plain `ImmutableMap`. Each copies every key, so each is a
  whole-map read; derive from the narrowest map you can.
- Absence is a dependency. `config.get("detrend", 0)` on a map without
  `"detrend"` records the absence; adding that key later invalidates, and
  adding other keys does not.
- Conditional reads produce separate traces. A function that reads different
  keys on different branches accumulates one trace per observed read-set,
  each matched independently.
- Every other argument (plain dicts, lists, arrays, scalars, tuples,
  lambdas) keys the cache by content hash, whole. Nothing observed how the
  function used it, so any change to it invalidates.
- A map passed from one `@pure` call into a nested one is traced in both:
  the inner call gets its own per-key trace, and the outer stays valid only
  for maps that would drive the inner the same way.

## Debugging

Caching stays on while a debugger is attached. A hit is bypassed, and the
function runs, only when a live breakpoint intersects the function or
anything in its user-code dependency closure. Set a breakpoint in a step or
in one of its helpers and that step executes; clear the breakpoint and hits
resume. Through nested `@pure` calls this applies to the path from the
breakpoint to the root: a breakpoint in an inner function also forces its
`@pure` callers to execute, since a cached caller would otherwise skip the
breakpoint, while sibling stages inside a forced caller are unaffected and
continue to hit and to record. Forced runs never write to the cache, and a
recording whose execution contained a forced run (e.g. a breakpoint added
while paused mid-pipeline) is discarded rather than stored, so nothing done
in a debug session, such as evaluating expressions or modifying locals, can
enter the cache.

Supported debuggers: pydevd (PyCharm, and VS Code's debugpy) and anything
built on `bdb` (pdb, ipdb). Their breakpoint tables are internal APIs, so
access is defensive: if a debugger is detected but its table cannot be
read, valuekit behaves as if there were breakpoints everywhere, which costs
cache hits but never skips a breakpoint. Coverage tools and profilers are
recognised as non-debuggers and do not disable caching.

Manual overrides, from narrowest to broadest:

```python
step.uncached(obs, cfg)   # call the raw function; the cache is untouched
VALUEKIT_ALWAYS_RUN=1     # env var: execute everything, write nothing
clear_cache(step)         # "step changed": deletes step's results and its callers'
clear_cache()             # or delete the cache directory; always safe
```

## The store

The cache directory holds content-addressed files: read-only arrays as
`.npy`, reloaded as memory maps that `freeze` shares without copying (a hit
on a function returning a 2 GB read-only array copies nothing), writeable
arrays as `.npyw`, and everything else in a small structural format in which
composite values reference their children by hash, so an array shared by
many results is stored once. There is no pickle anywhere. Cacheable return
values are a fixed set: `None`, `bool`, `int`, `float`, `complex`, `str`,
`bytes`, `range`, numpy scalars and arrays, tuples, lists, sets, frozensets,
dicts and `ImmutableMap`s of the same.

A stored value reloads as an equal value of the same type, which is what
lets a hit stand in for the call. That is also why the content hash
distinguishes a list from a tuple, two dicts that differ only in order, and
a writeable array from a read-only one: a hash has to identify a value
exactly for a content-addressed store to be able to hand it back.

Writes are atomic; directories can be shared between processes; a missing
or corrupt entry is treated as a miss. Deleting the cache is always safe. A
call that raises caches nothing. There is no eviction in this version: the
cache is a directory, so check its size with `du -sh` and delete it when it
grows too large.

## Parallelism

``run_all(fn, inputs)`` runs a module-level function over a batch of
inputs in parallel and returns a ``BatchResult`` of per-input outcomes, in
input order. Each input runs in its own process, spawned per task with at
most ``max_workers`` at once. Isolation is the point: a timeout kills
exactly one process, a segfault loses exactly one input, and neither
affects the other inputs or the capacity available to the rest of the
batch. The cost is one process start per input (roughly 0.4 s including a
numpy import). Starts overlap across workers, and for inputs that take
seconds or more the cost does not matter; for very small inputs, batch
them inside ``fn``. Each worker takes the parent's cache directory and
shares the cache: value writes are idempotent and trace writes are atomic
appends, so concurrent writers cannot drop each other's results.

Every input is processed, and every failure is recorded against the input
that caused it. An exception raised by ``fn`` carries the string-form
traceback captured in the worker. ``timeout=`` limits the seconds each
input may spend running; a breach kills that input's process promptly and
records a ``TimeoutError``. A process that dies without raising (a
segfault or an out-of-memory kill) records a ``RuntimeError`` naming the
input and the exit code.

```python
result = run_all(process_scenario, session_ids)

result.values                 # plain list of results; raises an
                              # ExceptionGroup if any input failed
for sid, exc in result.failures:
    ...                       # explicit handling; the batch completed
result[i].input               # the input that produced outcome i
result[i].result()            # the value, or re-raises the exception
```

Use ``.values`` by default: it is the plain list of results when
everything succeeded, and it raises when something failed, so failures
cannot be dropped by accident. ``.failures`` is for callers that handle
failures explicitly and continue.

Nothing is replayed automatically. To debug a failure, call the function
on that one input yourself:

```python
process_scenario(sid)         # the cached prefix replays in milliseconds;
                              # the failing step executes and raises here
```

with a live stack and a working REPL. Choosing the input yourself is
deliberate: which input fails first in a parallel batch differs from run
to run, so an automatic replay would pick one arbitrarily.

One debugger accommodation remains, because breakpoints do not reach
worker processes. If a live breakpoint intersects anything reachable by
name from ``fn``, the whole batch runs sequentially in this process, where
breakpoints fire and the usual debugger rules apply. The sequential
fallback does not enforce the timeout. Merely having a debugger attached
changes nothing on its own.

Two rules for using other pools (joblib, dask, a bare executor) around
``@pure`` code: parallelise in the driver, between ``@pure`` calls, never
inside a ``@pure`` function's body (reads performed in worker processes are
not recorded, which produces traces with missing dependencies and therefore
stale results); and call ``set_cache_dir`` at module top level, since a call
inside an ``if __name__ == "__main__":`` block, or in a notebook, does not
reach spawn-based workers. (``run_all`` is exempt: it passes the cache
directory to each worker explicitly.) To drive the location from the
environment, read the variable yourself, at top level:

```python
import os
from valuekit import set_cache_dir

set_cache_dir(os.environ.get("VALUEKIT_CACHE"))   # None disables caching
```

Nothing is cached until `set_cache_dir` is called: importing valuekit has no
effect on its own.

## Install

```
pip install valuekit        # Python >= 3.11; depends only on numpy
```

## Development

```
pip install -e ".[dev]"    # quoted: zsh globs the brackets
pytest
```
