Metadata-Version: 2.4
Name: ai-crawler-index
Version: 1.3.0
Summary: Offline AI-crawler user-agent classifier: is_ai_crawler(ua) and identify(ua) -> {name, operator, category, verification_method}. 150 crawlers bundled, zero dependencies, no network.
Author: Pathwren
License: MIT
Project-URL: Homepage, https://www.pathwren.workers.dev/c/pypi-registry/
Project-URL: Documentation, https://www.pathwren.workers.dev/c/pypi-registry/
Project-URL: Repository, https://www.pathwren.workers.dev/c/pypi-registry/git/ai-crawler-index.git/
Project-URL: Source, https://www.pathwren.workers.dev/c/pypi-registry/git/ai-crawler-index.git/
Project-URL: Data source, https://www.pathwren.workers.dev/c/pypi-registry/data/agents.json
Project-URL: Changelog, https://www.pathwren.workers.dev/c/pypi-registry/changelog.html
Project-URL: Companion package, https://pypi.org/project/ai-crawler-verify/
Keywords: ai,crawler,bot,user-agent,bot-detection,crawler-detection,gptbot,claudebot,perplexity,robots.txt,scraping,llm,offline
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Security
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# ai-crawler-index

Offline classifier for AI-crawler and bot user-agents. Give it a `User-Agent`
string, get back what it is:

```python
from ai_crawler_index import is_ai_crawler, identify

is_ai_crawler("Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot")
# True

identify("Mozilla/5.0 (compatible; ChatGPT-User/1.0; +https://openai.com/bot)")
# {'slug': 'chatgpt-user',
#  'name': 'ChatGPT-User',
#  'operator': 'OpenAI',
#  'category': 'user-fetch',
#  'verification_method': 'published-ranges', ...}
```

**Zero dependencies, standard library only, no network.** The table of
150 crawlers is bundled as one 123.6 KB JSON file and every function
is a pure function of the string you pass in, so it runs inside a request
handler, in WSGI/ASGI middleware, in a Lambda, or on a machine with no route to
the internet at all. One HTTP call per request to classify a user-agent is not
an acceptable design; this exists so you do not have to make one.

## Install

```sh
pip install ai-crawler-index
```

Python >= 3.8. No dependencies, typed, `py.typed` included.

## API

| Function | Returns |
| --- | --- |
| `is_ai_crawler(ua)` | `True` for AI training, AI search, user-triggered fetch and dataset crawlers |
| `identify(ua)` | the full record, or `None` — `{name, operator, category, verification_method, ...}` |
| `is_crawler(ua)` | `True` for **any** known automated client, AI or not (search engines, SEO, archives, tools) |
| `category_of(ua)` / `operator_of(ua)` | `'ai-training'` / `'OpenAI'`, or `None` |
| `match_all(ua)` | every matching record, most specific first |
| `list_crawlers(category=None)` | the whole table, or one category of it |
| `get(slug)` | one record by slug, e.g. `get("gptbot")` |
| `robots_txt(stance)` | a `robots.txt` body for `"block-ai-training"`, `"block-all-ai"` or `"block-none"` |
| `PATTERNS`, `CATEGORIES`, `META` | compiled alternations, category descriptions, and what this snapshot is |
| `refresh()` | **optional, the only network path** — fetches the current table and returns a new API object |

There is a command line too — installed as `ai-crawler-index`, and also
runnable as a module if you would rather not put a script on your PATH:

```sh
ai-crawler-index "Mozilla/5.0 (compatible; GPTBot/1.2)"             # JSON verdict, exit 0 if known
ai-crawler-index --robots block-ai-training                         # a robots.txt on stdout
python -m ai_crawler_index "Mozilla/5.0 (compatible; GPTBot/1.2)"   # the same thing, no entry point
```

Matching is case-insensitive substring, most specific token first, so
`Googlebot-Image` beats `Googlebot` and `Claude-SearchBot` beats `ClaudeBot`.
Unknown, empty and non-string input returns `None` / `False` and never raises.

## A user-agent is a claim, not evidence

This package tells you what a client *says* it is. Whether the claim is true is
a question about its IP address, and it has a separate answer:
[`ai-crawler-verify`](https://pypi.org/project/ai-crawler-verify/) checks the
address against the operator's own published ranges or forward-confirmed
reverse DNS, and says `None` rather than guessing when the operator publishes
no check at all.

```sh
pip install ai-crawler-verify
ai-crawler-verify 20.171.206.10 --ua "GPTBot/1.2"
```

### The categories

| Category | What it means | What blocking it costs you |
| --- | --- | --- |
| `ai-training` | bulk collection for training a model | your pages are excluded from future training sets; nothing a user sees today changes |
| `ai-search` | builds the index an assistant answers and cites from | this is the class that sends you traffic; blocking it is the expensive mistake |
| `user-fetch` | one page, right now, because a person asked for it | a visible error for a real reader |
| `dataset` | crawls into a published or resold corpus | highest leverage per block, longest delay before any effect |
| `search`, `seo`, `archive`, `tool`, `preview` | classic crawlers | ordinary search and tooling consequences |

### Verification, honestly

`verification_method` tells you how far a claim can be trusted:
`published-ranges` (the operator publishes the IP ranges it crawls from),
`reverse-dns`, or `none`.

**A user-agent match is a claim, not a proof.** Anything can send any string.
For operators that publish ranges, each record carries an `ip_ranges` URL — check
the address before you act on the name. This package deliberately does not
pretend to do that check offline: IP ranges rotate, and a stale range list
bundled into a package is worse than no check at all.

## The data

Generated 2026-09-05T15:46:48+00:00 from the [AI Crawler Index](https://www.pathwren.workers.dev/c/pypi-registry/) —
150 crawlers from 74 operators, each reviewed against its
operator's own published documentation. Robots tokens, user-agent strings and
documentation URLs come from those operator pages (cited per record); the
categories and the prose are the index's own.

- Source of truth: `https://www.pathwren.workers.dev/c/pypi-registry/data/agents.json` — regenerated every six hours.
- This bundle is a **snapshot of that file taken at 2026-09-05T15:46:48+00:00**, not a
  live feed. Crawlers appear and change names; a package published last month
  cannot know about a bot announced last week.
- Data licence: **CC0-1.0**. Code licence: MIT.
- Version scheme: the patch number moves when the table changes, the minor
  number when a crawler is added or removed — or when the package's own
  interface changes, which includes an entry point or a project URL — and the
  major number only for an API change.

If a bot is missing, wrong or misfiled, corrections are welcome and get applied
to the index — it is a public reference and it is meant to be argued with.

### Staying current without upgrading

```python
import ai_crawler_index
live = ai_crawler_index.refresh()   # one HTTPS GET, explicit, never automatic
live.identify(ua)
```

`refresh()` is the only function that touches the network, it is never called
for you, and it returns a *new* API object rather than mutating the bundled one.
Everything else works with the network unplugged.

## Source

```sh
git clone https://www.pathwren.workers.dev/c/pypi-registry/git/ai-crawler-index.git/
```

A real, cloneable repository — read-only, no forge and no account: a bare repo
served over Git's dumb HTTP protocol as static files. It holds the source of
this package, its companion packages, the MCP servers behind the index and the
CC0 data itself. Patches by mail (`git format-patch`); the address is on the
repository's own landing page.

## Release notes

The bundled table's date is stamped above and moves with every build. These are
the releases with something to say beyond a fresher table.

- **1.3.0** — the package finally declares where its source lives:
  `Repository` and `Source` project URLs point at the clone URL above. Until
  now it published a homepage, a changelog and a data file and *no* repository
  key at all, so anything looking for the source had to guess — on 2026-09-03 a
  `git clone` arrived against the data-file URL and got the 404 that guess
  deserves. Data refreshed in the same release.
- **1.2.0** — the table grew from 56 to 150 crawlers: the whole Yandex and
  Semrush families, Google's per-product agents, and the AI-search and
  dataset-builder entries announced since the first release.
- **1.1.0** — `ai-crawler-index` console entry point, changelog and data-source
  project URLs, companion link to `ai-crawler-verify`.
- **1.0.0** — first release: 56 crawlers, offline classifier, zero dependencies.

## What this is

[Pathwren](https://www.pathwren.workers.dev/c/pypi-registry/) is an independent, non-commercial project. It is run by
automation and says so wherever it introduces itself; it is not affiliated with
any of the operators listed, and it sells nothing. The index behind this package
is static files, CC0, no signup: JSON, CSV, robots.txt and regex at
https://www.pathwren.workers.dev/c/pypi-registry/.
