Metadata-Version: 2.4
Name: scrapeland
Version: 1.0.0
Summary: Rotating proxy and web data extraction API client for Python, with a drop-in Zyte-compatible adapter.
Author-email: scrapeland <support@scrape.land>
License: MIT
Project-URL: Homepage, https://scrape.land
Project-URL: Documentation, https://scrape.land/docs
Project-URL: Source, https://github.com/scrape-land/scrapeland-python
Project-URL: Issues, https://github.com/scrape-land/scrapeland-python/issues
Keywords: proxy,rotating proxy,scraping,zyte,smart proxy manager,scrapy
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Internet :: Proxy Servers
Classifier: Typing :: Typed
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.20
Provides-Extra: scrapy
Requires-Dist: scrapy>=2.6; extra == "scrapy"
Dynamic: license-file

# scrapeland (Python)

Rotating proxy API client for Python: one API key, thousands of self-validated
geo-targeted exit IPs, sticky sessions and country targeting. Ships with a
**drop-in Zyte-compatible adapter** so teams on Zyte API can switch in one line.

```bash
pip install scrapeland      # or: pip install -e clients/python
```

`requests` is the only dependency.

## Quick start

```python
from scrapeland import ScrapelandClient

client = ScrapelandClient("pb_live_YOURKEY")          # or $SCRAPELAND_API_KEY
r = client.get("https://api.ipify.org")
print(r.status_code, r.text)                          # a fresh exit IP each call

# country targeting + a sticky session (same IP across calls)
r = client.get("https://example.com", country="de", session="job42")
```

### Use it with your own `requests`/`httpx`

`.proxies()` returns a ready `{"http", "https"}` mapping, so you keep your existing code:

```python
import requests
proxies = ScrapelandClient("pb_live_YOURKEY").proxies(country="us")
requests.get("https://api.ipify.org", proxies=proxies, timeout=30)
```

## Fetch and extract (data API)

Besides the raw proxy, the client can call the Scrapeland data extraction API
(a different host from the proxy gateway). `fetch()` returns a rendered page;
`extract()` pulls structured fields with CSS selectors.

```python
from scrapeland import ScrapelandClient

client = ScrapelandClient("pb_live_YOURKEY")          # or $SCRAPELAND_API_KEY

# fetch the page HTML (or pass format="text" for plain text)
page = client.fetch("https://example.com", country="de")
print(page["status"], page["html"][:200])

# extract structured data; each field is a CSS selector or {css, attr, all}
data = client.extract("https://example.com", {
    "title": "h1",
    "links": {"css": "a", "attr": "href", "all": True},
})
print(data["data"])                                   # {"title": ..., "links": [...]}
```

### Browser rendering and response headers

`fetch()` and `extract()` accept extra options. Set `render=True` to render the
page in a real browser, `wait_for="<css selector>"` to wait for an element before
capturing (render mode), and `headers=True` to include the response headers.

```python
# render the page, wait for a selector, and ask for the response headers
page = client.fetch(
    "https://example.com",
    render=True,
    wait_for="#app",
    headers=True,
)
print(page["status"], page["headers"])                # headers included on request

# the same options apply to extract()
data = client.extract(
    "https://example.com",
    {"title": "h1"},
    render=True,
    wait_for=".loaded",
    headers=True,
)
```

`device="mobile"`, `actions=`, `fingerprint=True` and `block_resources=True` are
live too. Rendering bills 5 request-units instead of the 1 a plain fetch costs (10 with a
screenshot, 1 with `block_resources`), and it downloads everything the page asks
for — measured at 1–300× the bytes (median ~17×, image-heavy shops worst), so it
is noticeably slower. Unless you need a screenshot
or the images themselves, pass `block_resources=True`: images, fonts and media are
skipped, the DOM your selectors read is identical. If every browser slot is busy
you get a `503` with `Retry-After` — retry, it is not a failure and is not billed.

### AI extraction, lists, batch, and discovery

Describe fields in plain English instead of selectors, fetch many URLs at once,
pull page metadata/links, or ask the deployment what it supports:

```python
# AI extraction — no selectors; add schema to pin the shape, model to route
data = client.extract(
    "https://books.toscrape.com/",
    prompt="the first book title and price as a number",
    schema={"title": "string", "price": "number"},
    model="fast",
)
# a "list every item" prompt comes back under data["data"]["items"]

# Asking a QUESTION rather than listing fields? structured=False returns prose in
# "answer" (and no "data" key), so you don't have to guess the keys a model invented.
ans = client.extract(
    "https://scrape.land/",
    prompt="what's this business about?",
    structured=False,
)
print(ans["answer"])   # "scrape.land is a web data-extraction API that ..."
# schema=/extract_type= pin a JSON shape, so neither can be combined with
# structured=False — that pair is rejected with a 400.

# many URLs in one call (up to 20); each billed like a single fetch
out = client.batch(["https://a.example/", "https://b.example/"], format="text")
for r in out["results"]:
    ...

# page metadata (title/OpenGraph/JSON-LD) and every link as an absolute URL
page = client.fetch("https://example.com", metadata=True, links=True)
print(page["metadata"], page["links"])

# what can THIS key do? (ai.enabled reflects your plan — AI extraction needs Scale+)
caps = client.capabilities()
print(caps["ai"]["enabled"], caps["ai"]["models"], caps["render"]["enabled"])

# your key's plan, remaining quota, prepaid credit, rate limit, key budget
acct = client.account()
print(acct["plan"], acct["included_requests_remaining"])

# POST a body (JSON APIs / forms), or cap exit latency
client.fetch("https://api.example/submit", method="POST",
             body='{"q":"x"}', send_headers={"Content-Type": "application/json"})
client.fetch("https://example.com", max_latency=3000)   # only fast exits
```

### Rank links, stream large batches, and handle errors

```python
# rank() — score a page's links by relevance to a goal, so you can pick which
# sub-pages to fetch next instead of crawling everything.
r = client.rank("https://news.example/", "articles about interest rates", top_k=10)
for link in r["links"]:
    print(link["score"], link["url"], "-", link["reason"])
# AI transparency: on an AI failure rank() does NOT raise — you get the links back in
# document order and a ranking_error you can check, so your code never breaks.
if r.get("ranking_error"):
    print("ranking degraded:", r["ranking_error"])

# batch_iter() — any number of URLs, auto-chunked into <=20-URL calls, yielded in
# order as chunks complete (no 20-URL cap to manage, no holding every response).
for res in client.batch_iter(all_urls, fields={"title": "h1"}):
    if "error" in res:
        log.warning("failed %s: %s", res["url"], res["error"])
    else:
        save(res["url"], res["data"])

# Typed errors — branch on the failure kind instead of parsing strings.
from scrapeland import RateLimitError, PaymentRequiredError, ScrapelandError
try:
    client.fetch("https://example.com")
except RateLimitError as e:
    time.sleep(e.retry_after or 5)      # honor the server's Retry-After
except PaymentRequiredError:
    alert("out of quota / credit")
except ScrapelandError as e:            # base class; .status_code, .detail, .path
    log.error("scrapeland %s: %s", e.status_code, e.detail)
```

The API base URL defaults to `https://scrape.land`; override it with the
`api_base=` argument or `$SCRAPELAND_API_BASE` (falls back to `$PROXYBANK_API_BASE`).

## Scrapy

Route a whole crawl through scrapeland with one settings block, no per-request
boilerplate (`pip install "scrapeland[scrapy]"`):

```python
# settings.py
DOWNLOADER_MIDDLEWARES = {
    "scrapeland.scrapy.ScrapelandMiddleware": 740,   # before HttpProxyMiddleware (750)
}
SCRAPELAND_API_KEY = "pb_live_YOURKEY"
SCRAPELAND_COUNTRY = "us"          # optional defaults applied to every request
```

Per-request overrides (or bypass) via `Request.meta`:

```python
yield scrapy.Request(url, meta={"scrapeland": {"country": "de", "session": "job42"}})
yield scrapy.Request(url, meta={"scrapeland": False})   # don't proxy this one
```

## Migrating from Zyte

Already using the [`zyte-api`](https://github.com/zytedata/python-zyte-api)
client? Change the import and the key. The query/response shapes match:

```diff
- from zyte_api import ZyteAPI
- client = ZyteAPI(api_key="ZYTE_KEY")
+ from scrapeland.zyte import ZyteAPI
+ client = ZyteAPI(api_key="pb_live_YOURKEY")

  result = client.get({"url": "https://toscrape.com", "httpResponseBody": True})
  from base64 import b64decode
  body = b64decode(result["httpResponseBody"])
```

Geolocation, custom request headers, sticky sessions, batching and async all map across:

```python
from scrapeland.zyte import ZyteAPI, decode_body

client = ZyteAPI("pb_live_YOURKEY")
result = client.get({
    "url": "http://ip-api.com/json",
    "httpResponseBody": True,
    "httpResponseHeaders": True,
    "geolocation": "AU",                              # -> exit country AU
    "customHttpRequestHeaders": [{"name": "Accept", "value": "application/json"}],
})
print(result["statusCode"], decode_body(result))

# batch: yields as they complete (a failed URL yields {"error": ...}, never aborts)
for res in client.iter([{"url": u, "httpResponseBody": True} for u in urls]):
    ...
```

Async mirrors `zyte_api.AsyncZyteAPI`:

```python
import asyncio
from scrapeland.zyte import AsyncZyteAPI

async def main():
    client = AsyncZyteAPI("pb_live_YOURKEY")
    res = await client.get({"url": "https://toscrape.com", "httpResponseBody": True})
    async for r in client.iter(queries):
        ...

asyncio.run(main())
```

### Compatibility and differences

| Zyte field | scrapeland |
|---|---|
| `url`, `httpRequestMethod`, `httpRequestText` | yes |
| `httpResponseBody` (base64), `httpResponseHeaders` | yes |
| `customHttpRequestHeaders` / `requestHeaders` | yes |
| `geolocation` (country code) | yes (exit country) |
| `sessionContext` (first `id`) / `session` | yes (sticky session) |
| `browserHtml`, `screenshot`, auto-extraction (`product`, `article`, ...) | no (raises `UnsupportedQuery`) |

scrapeland tunnels **raw HTTP** (it does not render pages or run extraction), so
browser/AI-extraction fields are intentionally unsupported. Everything in Zyte
API's HTTP mode works.

## Configuration

| Argument | Env var | Default |
|---|---|---|
| `api_key` | `SCRAPELAND_API_KEY` (the Zyte adapter also reads `ZYTE_API_KEY`) | (required) |
| `gateway` | `SCRAPELAND_GATEWAY` | `http://gateway.scrape.land:8080` |
| `retries` | (none) | `2` (transient failures retry with backoff; each retry rotates IP) |

Point `gateway` at `http://localhost:8180` to test against a local stack. Because
the Zyte adapter falls back to `ZYTE_API_KEY`, an existing `ZyteAPI()` call that
relied on that env var keeps working after the import swap.

### Blocked port? Use :443

The gateway answers on `gateway.scrape.land:443` as well as `:8080`, for corporate,
university and hotel networks that filter high ports. Identical behaviour and
billing — only the port differs, and the scheme stays `http://` with your key in
the username:

```python
client = ScrapelandClient("pb_live_YOURKEY",
                          gateway="http://gateway.scrape.land:443")  # or $SCRAPELAND_GATEWAY
```

## Plan limits

`client.account()` returns your own numbers; these are the shapes to code against.

| | Free | Starter | Growth | Scale | Business+ | Pay as you go |
|---|---|---|---|---|---|---|
| Rate limit | 5/s | 50/s | 100/s | 200/s | 1,000/s | none |
| Max response | 2 MB | 2 MB | 5 MB | 10 MB | 25 MB | 5 MB |
| AI extraction | — | — | — | yes | yes | — |

Over the rate limit you get a `429` with `Retry-After`. A response larger than your
cap is refused whole with a `413` naming the size, the limit and your plan — never
truncated, and never billed. `prompt=` / `schema=` / `extract_type=` on a plan below
Scale return `403`; check `client.capabilities()["ai"]["enabled"]` first.

Past your included requests, overage draws on **prepaid credit** and floors at
zero — at zero you get a `402`, never an invoice after the fact.

Full HTTP/curl/Node/Scrapy docs: <https://scrape.land/docs>.
