Metadata-Version: 2.5
Name: glean-indexing-sdk
Version: 1.0.0
Summary: SDK for building custom Glean indexing integrations
Project-URL: Source Code, https://github.com/gleanwork/glean-indexing-sdk
Project-URL: Issues, https://github.com/gleanwork/glean-indexing-sdk/issues
Project-URL: Documentation, https://developers.glean.com/libraries/indexing-sdk
Author-email: Steve Calvert <steve.calvert@glean.com>
License: MIT
License-File: LICENSE
Requires-Python: <4.0,>=3.10
Requires-Dist: click>=8.0
Requires-Dist: glean-api-client<0.14.0,>=0.13.0
Requires-Dist: jinja2>=3.1.2
Requires-Dist: pydantic>=2.0
Requires-Dist: python-dotenv>=1.1.0
Requires-Dist: pyyaml>=6.0.2
Provides-Extra: aws
Requires-Dist: boto3<2,>=1.34.0; extra == 'aws'
Requires-Dist: watchtower<4,>=3.3.0; extra == 'aws'
Provides-Extra: dev
Requires-Dist: build>=1.0.0; extra == 'dev'
Requires-Dist: commitizen>=4.4.1; extra == 'dev'
Requires-Dist: pip-audit>=2.6.0; extra == 'dev'
Requires-Dist: twine>=7.0.0; extra == 'dev'
Provides-Extra: gcp
Requires-Dist: google-cloud-logging<4,>=3.11.0; extra == 'gcp'
Requires-Dist: google-cloud-monitoring<3,>=2.22.0; extra == 'gcp'
Requires-Dist: google-cloud-secret-manager<3,>=2.20.0; extra == 'gcp'
Provides-Extra: lint
Requires-Dist: ruff>=0.5; extra == 'lint'
Provides-Extra: test
Requires-Dist: pytest-asyncio>=0.23.2; extra == 'test'
Requires-Dist: pytest-cov>=4.1.0; extra == 'test'
Requires-Dist: pytest-httpx>=0.35.0; extra == 'test'
Requires-Dist: pytest-socket>=0.7.0; extra == 'test'
Requires-Dist: pytest-watcher>=0.3.4; extra == 'test'
Requires-Dist: pytest>=7.4.3; extra == 'test'
Provides-Extra: typing
Requires-Dist: pyright>=1.1.370; extra == 'typing'
Description-Content-Type: text/markdown

# Glean Indexing SDK

[![GA](https://img.shields.io/badge/-GA-F6F3EB?style=flat-square&logo=data:image/svg+xml;base64,PHN2ZyB2aWV3Qm94PSIwIDAgMzIgMzIiIGZpbGw9Im5vbmUiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyI+CjxwYXRoIGQ9Ik0yNC4zMDA2IDIuOTU0MjdMMjAuNzY1NiAwLjE5OTk1MUwxNy45MDI4IDMuOTk1MjdDMTMuNTY1MyAxLjkzNDk1IDguMjMwMTkgMy4wODQzOSA1LjE5Mzk0IDcuMDA5ODNDMS42NTg4OCAxMS41NjQyIDIuNDgzIDE4LjExMzggNy4wMzczOCAyMS42NDg5QzguNzcyMzggMjIuOTkzNSAxMC43ODkzIDIzLjcwOTIgMTIuODI3OSAyMy44MTc3QzE2LjE0NjEgMjQuMDEyOCAxOS41MDc3IDIyLjYyNDggMjEuNjc2NSAxOS44MDU1QzI0LjczNDQgMTUuODggMjQuNTE3NSAxMC40MTQ4IDIxLjQ1OTYgNi43Mjc4OUwyNC4zMDA2IDIuOTU0MjdaTTE4LjExOTcgMTcuMDUxMkMxNi4xMDI4IDE5LjYzMiAxMi4zNzI1IDIwLjEwOTEgOS43NzAwMSAxOC4wOTIyQzcuMTg5MTkgMTYuMDc1MiA2LjcxMjA3IDEyLjMyMzMgOC43MjkwMSA5Ljc0MjQ2QzkuNzA0OTQgOC40ODQ1OCAxMS4xMTQ2IDcuNjgyMTQgMTIuNjc2MSA3LjQ4Njk2QzEzLjA0NDggNy40NDM1OCAxMy40MTM1IDcuNDIxOSAxMy43ODIyIDcuNDQzNThDMTQuOTc1IDcuNTA4NjUgMTYuMTI0NCA3Ljk0MjM5IDE3LjA3ODcgOC42Nzk3N0MxOS42NTk1IDEwLjcxODQgMjAuMTM2NiAxNC40NzAzIDE4LjExOTcgMTcuMDUxMloiIGZpbGw9IndoaXRlIi8+CjxwYXRoIGQ9Ik0yNC41MTc2IDIxLjY5MjJDMjMuOTMyIDIyLjQ1MTMgMjMuMjgxNCAyMy4xMjM2IDIyLjU2NTcgMjMuNzUyNUMyMS44NzE3IDI0LjMzODEgMjEuMTEyNyAyNC44ODAzIDIwLjMxMDIgMjUuMzM1N0MxOS41Mjk1IDI1Ljc2OTUgMTguNjgzNyAyNi4xMzgyIDE3LjgzNzggMjYuNDIwMUMxNi45OTIgMjYuNzAyIDE2LjEwMjggMjYuODk3MiAxNS4yMTM3IDI3LjAwNTdDMTQuMzI0NSAyNy4xMTQxIDEzLjQzNTMgMjcuMTU3NSAxMi41MjQ0IDI3LjA5MjRDMTEuNjEzNSAyNy4wMjczIDEwLjcyNDMgMjYuODc1NSA5Ljg1Njg0IDI2LjY1ODdMOS42NjE2NSAyNy4zNzQzTDguNzcyNDYgMzAuOTk2MkM5LjkwMDIxIDMxLjI5OTggMTEuMDQ5NyAzMS40NzMzIDEyLjIyMDggMzEuNTZDMTIuMjY0MiAzMS41NiAxMi4zMjkyIDMxLjU2IDEyLjM3MjYgMzEuNTZDMTMuNTAwMyAzMS42MjUxIDE0LjY0OTggMzEuNTgxNyAxNS43NTU4IDMxLjQ1MTZDMTYuOTI3IDMxLjI5OTggMTguMDk4MSAzMS4wMzk1IDE5LjIyNTggMzAuNjcwOEMyMC4zNTM2IDMwLjMwMjIgMjEuNDU5NyAyOS44MjUgMjIuNTAwNyAyOS4yMzk1QzIzLjU2MzQgMjguNjUzOSAyNC41NjEgMjcuOTM4MiAyNS40OTM1IDI3LjE1NzVDMjYuNDQ3OCAyNi4zNTUgMjcuMzE1MyAyNS40NDQyIDI4LjA3NDQgMjQuNDQ2NUMyOC4xODI4IDI0LjMxNjQgMjguMjY5NSAyNC4xNjQ2IDI4LjM3OCAyNC4wMTI4TDI0Ljc3NzkgMjEuMzQ1MkMyNC42Njk0IDIxLjQ1MzcgMjQuNjA0NCAyMS41ODM4IDI0LjUxNzYgMjEuNjkyMloiIGZpbGw9IndoaXRlIi8+Cjwvc3ZnPg==&labelColor=343CED)](https://github.com/gleanwork/.github/blob/main/docs/repository-stability.md#ga)
[![PyPI version](https://badge.fury.io/py/glean-indexing-sdk.svg)](https://badge.fury.io/py/glean-indexing-sdk)

Build custom Glean connectors in Python. The SDK handles fetching, transforming, batching, and uploading your data to Glean's indexing APIs, so you write only the parts that are specific to your source.

📖 **[Full documentation](https://developers.glean.com/libraries/indexing-sdk)** on the Glean Developer site.

Repository examples cover [advanced usage](docs/advanced.md) and
[streaming connectors](docs/streaming-connectors.md).

## Build one with an agent

The fastest path is to let a coding agent build the connector. Your agent installs it straight from this repository, which doubles as the plugin marketplace.

**Claude Code**

```bash
claude plugin marketplace add gleanwork/glean-indexing-sdk
claude plugin install glean-connector-builder@glean-indexing-sdk
```

**Codex**

```bash
codex plugin marketplace add gleanwork/glean-indexing-sdk
codex plugin add glean-connector-builder@glean-indexing-sdk
```

**Cursor** has no plugin CLI — open Dashboard → Plugins → Add Marketplace → Import from Repo, point it at `gleanwork/glean-indexing-sdk`, then install from Customize.

Then describe your source:

```text
I want to push my Webex data to Glean. Build a connector for me.
```

The agent explores the source's API, confirms a plan with you, generates the connector against this SDK, and tests it. See the [Indexing SDK overview](https://developers.glean.com/libraries/indexing-sdk) for what it does and what to review in the output.

## Or write it yourself

### Requirements

- Python >= 3.10
- A Glean instance and an [indexing API token](https://developers.glean.com/indexing/tag/Authentication/)

### Installation

```bash
pip install glean-indexing-sdk

# Optional cloud integrations
pip install "glean-indexing-sdk[aws]"   # Beta CloudWatch logs + metrics
pip install "glean-indexing-sdk[gcp]"   # Beta Cloud Logging/Monitoring + deployment Secret Manager
```

## Quickstart

Every connector has two parts: a **data client** that fetches from your source, and a **connector** that transforms the result into Glean documents. The flow is **fetch → transform → upload**; you implement `get_source_data()` and `transform()`, and the SDK does the rest.

Set your credentials:

```bash
export GLEAN_SERVER_URL="https://your-company-be.glean.com"
export GLEAN_INDEXING_API_TOKEN="your-indexing-api-token"
export WIKI_API_TOKEN="your-wiki-token"
```

Then define and run a connector:

```python snippet=non_streaming/complete.py
import os
from datetime import datetime
from typing import Any, List, Optional, Sequence, TypedDict

from glean.indexing.connectors import BaseDataClient, BaseDatasourceConnector
from glean.indexing.models import (
    ContentDefinition,
    CustomDatasourceConfig,
    DocumentDefinition,
    IndexingMode,
    UserReferenceDefinition,
)


class WikiPage(TypedDict):
    id: str
    title: str
    content: str
    author: str
    updated_at: str
    url: str


class WikiDataClient(BaseDataClient[WikiPage]):
    """Fetches pages from the source system. Replace the body with a real API call."""

    def __init__(self, base_url: str, api_token: str):
        self.base_url = base_url
        self.api_token = api_token

    def get_source_data(self, since: Optional[str] = None, **kwargs: Any) -> Sequence[WikiPage]:
        return [
            {
                "id": "page_123",
                "title": "Engineering Onboarding Guide",
                "content": "Welcome to the engineering team...",
                "author": "jane.smith@company.com",
                "updated_at": "2026-02-01T14:30:00Z",
                "url": f"{self.base_url}/pages/123",
            }
        ]


class CompanyWikiConnector(BaseDatasourceConnector[WikiPage]):
    """Transforms wiki pages into Glean documents."""

    configuration = CustomDatasourceConfig(
        name="company_wiki",
        display_name="Company Wiki",
        url_regex=r"https://wiki\.company\.com/.*",
        is_user_referenced_by_email=True,
    )

    def transform(self, data: Sequence[WikiPage]) -> List[DocumentDefinition]:
        return [
            DocumentDefinition(
                id=page["id"],
                title=page["title"],
                datasource=self.name,
                view_url=page["url"],
                body=ContentDefinition(mime_type="text/plain", text_content=page["content"]),
                author=UserReferenceDefinition(email=page["author"]),
                # created_at / updated_at are epoch seconds, not ISO strings.
                updated_at=int(
                    datetime.fromisoformat(page["updated_at"].replace("Z", "+00:00")).timestamp()
                ),
            )
            for page in data
        ]


def create_connector() -> CompanyWikiConnector:
    """Build the production connector after runtime secrets are loaded."""
    return CompanyWikiConnector(
        name="company_wiki",
        data_client=WikiDataClient(
            base_url="https://wiki.company.com",
            api_token=os.environ["WIKI_API_TOKEN"],
        ),
    )


if __name__ == "__main__":
    connector = create_connector()
    connector.configure_datasource()
    connector.index_data(mode=IndexingMode.FULL)
```

Test it without touching the network:

```python
from glean.indexing.testing import StaticDataClient, run_connector

result = run_connector(CompanyWikiConnector("company_wiki", StaticDataClient([...])))
result.assert_documents_posted(count=1, datasource="company_wiki")
```

The module-level `create_connector()` function is the production construction seam for CLI and
cloud execution. It keeps the connector constructor dependency-injectable for tests while letting
runtime code build real clients after environment-backed secrets are loaded. Enable it in generated
deployments with `glean-idx deploy init --cloud gcp --connector-factory create_connector` (or
`--cloud aws`). Existing projects with a zero-argument connector class remain supported without a
factory.

## What's in the box

| Capability | What it gives you |
|---|---|
| [Connector types](https://developers.glean.com/libraries/indexing-sdk/concepts/connector-types) | Four base classes: in-memory, sync streaming, async streaming, and people/identity. |
| [Pull integrations](https://developers.glean.com/libraries/indexing-sdk/pull/http-client) | `PullHttpClient` with retries and backoff, link/offset/cursor pagination, and token-bucket rate limiting. |
| [Push & indexing](https://developers.glean.com/libraries/indexing-sdk/push/uploader) | `PushUploader` for documents, users, groups, memberships, and employees, with parallel batch uploads. |
| [Permissions](https://developers.glean.com/libraries/indexing-sdk/permissions) | Per-document ACLs and datasource identities so results respect who can see what. |
| [Testing](https://developers.glean.com/libraries/indexing-sdk/testing/overview) | Three phases: fully mocked, real-source-with-record/replay, and live end-to-end. |
| [Observability](https://developers.glean.com/libraries/indexing-sdk/observability) | Structured logging and metrics, with optional CloudWatch and Google Cloud plugins. |
| [Status & debugging](https://developers.glean.com/libraries/indexing-sdk/status-and-debugging) | `StatusClient` and `glean-idx document status` to answer "why isn't my document in search?" |
| [Deployment](https://developers.glean.com/libraries/indexing-sdk/deployment/overview) | `glean-idx deploy` generates Docker and Terraform for AWS or GCP. |
| [Connector Builder](https://developers.glean.com/libraries/indexing-sdk) | An agent plugin that builds a connector from a description of your source. |

### Employee replacement compatibility

`BasePeopleConnector` and `PushUploader.bulk_index_employees()` preserve the current full-replacement employee workflow in the 1.x SDK. The upstream `/bulkindexemployees` endpoint is deprecated and scheduled for removal after October 15, 2026. Until an equivalent replacement contract is available, the SDK constrains `glean-api-client` to the verified `0.13.x` line so dependency resolution cannot silently remove that generated-client surface. Removing or materially changing this workflow would require a new SDK major version.

## The CLI

One command, `glean-idx`, covers the whole loop.

```bash
glean-idx doctor                    # are my credentials right?
glean-idx validate ./my-connector   # is the plan complete, before writing code?
glean-idx test --phase all          # mocked, then real source, then live
glean-idx run                       # crawl for real
glean-idx datasource status --datasource my-source
glean-idx document status --datasource my-source --document Article doc-1
glean-idx deploy init --cloud gcp   # Docker and Terraform for a CronJob
```

Commands split into two kinds, and `glean-idx --help` says which is which.

Most need only `GLEAN_SERVER_URL` and `GLEAN_INDEXING_API_TOKEN`, so they run
anywhere, including with no install at all:

```bash
uvx --from glean-indexing-sdk glean-idx doctor
```

`run`, `test`, and `datasource configure` import your connector, so they run
inside the connector project with the SDK installed alongside your code:

```bash
uv run glean-idx run
```

Every command takes `--output json` for a stable envelope, `--yes` to skip
confirmations unattended, and returns a documented exit code — `3` for a
missing precondition, `4` for a Glean error, `5` for a validation failure. In
JSON mode the envelope goes to stdout whether it succeeded or not, so there is
one stream to read. This contract begins after Click parses the invocation:
Click usage and parse errors (such as unknown options, invalid values, or
missing required options) happen before JSON command handling and remain text
diagnostics on stderr with exit code `2`, even when `--output json` is present.

```bash
glean-idx datasource status --datasource my-source --output json | jq .data.documents
```

`glean-idx schema document` prints the JSON Schema your `transform()` has to
produce, and `glean-idx completion zsh` sets up tab completion.

## Indexing modes

```python
connector.index_data(mode=IndexingMode.FULL)         # re-index everything
connector.index_data(mode=IndexingMode.INCREMENTAL)  # only changes since the last crawl
```

A **full** crawl replaces the indexed state: documents absent from the run are deleted as stale. **Incremental** passes a `since` timestamp to your data client, but the SDK does not persist checkpoints — override `_get_last_crawl_timestamp()` on your connector to supply one. See [Indexing modes](https://developers.glean.com/libraries/indexing-sdk/concepts/indexing-modes).

## Contributing

This project uses [mise](https://mise.jdx.dev/) for toolchain management and `uv` for Python dependencies. See [CONTRIBUTING.md](CONTRIBUTING.md).

```bash
mise run setup    # create venv and install dependencies
mise run test     # run all tests
mise run lint     # ruff, pyright, markdown-code
mise run lint:fix # auto-fix and format
```

## License

[MIT](LICENSE)
