Metadata-Version: 2.4
Name: fabcontext
Version: 0.1.1
Summary: Harvest a Microsoft Fabric tenant into a ranked context layer, published as Delta tables in a lakehouse.
Author: mim
License: MIT
Project-URL: Homepage, https://github.com/djouallah/fabric-context-layer
Project-URL: Repository, https://github.com/djouallah/fabric-context-layer
Project-URL: Issues, https://github.com/djouallah/fabric-context-layer/issues
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Intended Audience :: Developers
Classifier: Topic :: Database
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: duckdb>=1.4
Requires-Dist: deltalake>=1.2
Requires-Dist: requests>=2
Requires-Dist: azure-identity>=1.25
Requires-Dist: azure-storage-file-datalake>=12.14
Provides-Extra: test
Requires-Dist: pytest>=7; extra == "test"
Dynamic: license-file

# context-layer

There is a lot of talk about context layers, and the best way to learn one is to build
one. This is a toy, but the core ideas turned out to be simple.

- A data platform is full of signals: query history, who opens what, what is certified,
  what is refreshed. That is enough to rank definitions.
- Nobody will maintain a knowledge graph by hand, and the ones that exist drift. The
  graph has to build itself.
- A stateless agent is attractive: the ranking happens before the question, so the model
  behind the agent can change without losing anything.
- A context layer is there to be useful, not to hold the ultimate truth.

The bet is that a context built automatically, at least to start, and ranked on how the
platform already behaves gets better as the platform is used. I do not know yet whether
that holds. What I do like is that it uses the semantic models already there and asks for
no new modelling.

So far the industry splits on who settles a conflict: the platform, by scoring it,
which scales with usage, or a person, by reviewing it, which scales with reviewer time; I
prefer the first, time will tell.

The harvest delivers one thing: a knowledge graph of the tenant - every term, its
competing definitions ranked, and what feeds what. That graph is the context. It runs
nightly on its own, an agent asks whenever, and the two meet at the graph without ever
calling each other.

**[See it on a real tenant](https://djouallah.github.io/fabric-context-layer/)**

![Inside the platform, the harvest side reads a Fabric workspace, builds and ranks the graph, and publishes it as the context. Outside it, any stateless agent - on a laptop, in a notebook, in CI, in a chat - reads that context as one markdown file, runs rank 1 as DAX on the model that owns it, and answers with the number, its source and a confidence.](docs/how-it-works-dark.svg)


```python
!pip install fabcontext
import fabcontext
url = fabcontext.harvest("Workspace_A")
```

The first call creates the lakehouse; every later one updates it. It returns the URL an
agent asks against.

The ranking, what is harvested, the schema, how to ask, the limits: **[docs/guide.md](docs/guide.md)**.

## Licence

MIT - see [LICENSE](LICENSE).
