Metadata-Version: 2.4
Name: scrapeland
Version: 0.3.1
Summary: Rotating proxy and web data extraction API client for Python, with a drop-in Zyte-compatible adapter.
Author-email: scrapeland <support@scrape.land>
License: MIT
Project-URL: Homepage, https://scrape.land
Project-URL: Documentation, https://scrape.land/docs
Project-URL: Source, https://github.com/scrape-land/scrapeland-python
Project-URL: Issues, https://github.com/scrape-land/scrapeland-python/issues
Keywords: proxy,rotating proxy,scraping,zyte,smart proxy manager,scrapy
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Internet :: Proxy Servers
Classifier: Typing :: Typed
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.20
Provides-Extra: scrapy
Requires-Dist: scrapy>=2.6; extra == "scrapy"
Dynamic: license-file

# scrapeland (Python)

Rotating proxy API client for Python: one API key, thousands of self-validated
geo-targeted exit IPs, sticky sessions and country targeting. Ships with a
**drop-in Zyte-compatible adapter** so teams on Zyte API can switch in one line.

```bash
pip install scrapeland      # or: pip install -e clients/python
```

`requests` is the only dependency.

## Quick start

```python
from scrapeland import ScrapelandClient

client = ScrapelandClient("pb_live_YOURKEY")          # or $SCRAPELAND_API_KEY
r = client.get("https://api.ipify.org")
print(r.status_code, r.text)                          # a fresh exit IP each call

# country targeting + a sticky session (same IP across calls)
r = client.get("https://example.com", country="de", session="job42")
```

### Use it with your own `requests`/`httpx`

`.proxies()` returns a ready `{"http", "https"}` mapping, so you keep your existing code:

```python
import requests
proxies = ScrapelandClient("pb_live_YOURKEY").proxies(country="us")
requests.get("https://api.ipify.org", proxies=proxies, timeout=30)
```

## Fetch and extract (data API)

Besides the raw proxy, the client can call the Scrapeland data extraction API
(a different host from the proxy gateway). `fetch()` returns a rendered page;
`extract()` pulls structured fields with CSS selectors.

```python
from scrapeland import ScrapelandClient

client = ScrapelandClient("pb_live_YOURKEY")          # or $SCRAPELAND_API_KEY

# fetch the page HTML (or pass format="text" for plain text)
page = client.fetch("https://example.com", country="de")
print(page["status"], page["html"][:200])

# extract structured data; each field is a CSS selector or {css, attr, all}
data = client.extract("https://example.com", {
    "title": "h1",
    "links": {"css": "a", "attr": "href", "all": True},
})
print(data["data"])                                   # {"title": ..., "links": [...]}
```

### Browser rendering and response headers

`fetch()` and `extract()` accept extra options. Set `render=True` to render the
page in a real browser, `wait_for="<css selector>"` to wait for an element before
capturing (render mode), and `headers=True` to include the response headers.

```python
# render the page, wait for a selector, and ask for the response headers
page = client.fetch(
    "https://example.com",
    render=True,
    wait_for="#app",
    headers=True,
)
print(page["status"], page["headers"])                # headers included on request

# the same options apply to extract()
data = client.extract(
    "https://example.com",
    {"title": "h1"},
    render=True,
    wait_for=".loaded",
    headers=True,
)
```

### AI extraction, lists, batch, and discovery

Describe fields in plain English instead of selectors, fetch many URLs at once,
pull page metadata/links, or ask the deployment what it supports:

```python
# AI extraction — no selectors; add schema to pin the shape, model to route
data = client.extract(
    "https://books.toscrape.com/",
    prompt="the first book title and price as a number",
    schema={"title": "string", "price": "number"},
    model="fast",
)
# a "list every item" prompt comes back under data["data"]["items"]

# many URLs in one call (up to 20); each billed like a single fetch
out = client.batch(["https://a.example/", "https://b.example/"], format="text")
for r in out["results"]:
    ...

# page metadata (title/OpenGraph/JSON-LD) and every link as an absolute URL
page = client.fetch("https://example.com", metadata=True, links=True)
print(page["metadata"], page["links"])

# what does this deployment support? (render/AI on?, models, limits — no key needed)
caps = client.capabilities()
print(caps["ai"]["enabled"], caps["ai"]["models"], caps["render"]["enabled"])

# your key's plan, remaining quota, prepaid credit, rate limit, key budget
acct = client.account()
print(acct["plan"], acct["included_requests_remaining"])

# POST a body (JSON APIs / forms), or cap exit latency
client.fetch("https://api.example/submit", method="POST",
             body='{"q":"x"}', send_headers={"Content-Type": "application/json"})
client.fetch("https://example.com", max_latency=3000)   # only fast exits
```

The API base URL defaults to `https://scrape.land`; override it with the
`api_base=` argument or `$SCRAPELAND_API_BASE` (falls back to `$PROXYBANK_API_BASE`).

## Scrapy

Route a whole crawl through scrapeland with one settings block, no per-request
boilerplate (`pip install "scrapeland[scrapy]"`):

```python
# settings.py
DOWNLOADER_MIDDLEWARES = {
    "scrapeland.scrapy.ScrapelandMiddleware": 740,   # before HttpProxyMiddleware (750)
}
SCRAPELAND_API_KEY = "pb_live_YOURKEY"
SCRAPELAND_COUNTRY = "us"          # optional defaults applied to every request
```

Per-request overrides (or bypass) via `Request.meta`:

```python
yield scrapy.Request(url, meta={"scrapeland": {"country": "de", "session": "job42"}})
yield scrapy.Request(url, meta={"scrapeland": False})   # don't proxy this one
```

## Migrating from Zyte

Already using the [`zyte-api`](https://github.com/zytedata/python-zyte-api)
client? Change the import and the key. The query/response shapes match:

```diff
- from zyte_api import ZyteAPI
- client = ZyteAPI(api_key="ZYTE_KEY")
+ from scrapeland.zyte import ZyteAPI
+ client = ZyteAPI(api_key="pb_live_YOURKEY")

  result = client.get({"url": "https://toscrape.com", "httpResponseBody": True})
  from base64 import b64decode
  body = b64decode(result["httpResponseBody"])
```

Geolocation, custom request headers, sticky sessions, batching and async all map across:

```python
from scrapeland.zyte import ZyteAPI, decode_body

client = ZyteAPI("pb_live_YOURKEY")
result = client.get({
    "url": "http://ip-api.com/json",
    "httpResponseBody": True,
    "httpResponseHeaders": True,
    "geolocation": "AU",                              # -> exit country AU
    "customHttpRequestHeaders": [{"name": "Accept", "value": "application/json"}],
})
print(result["statusCode"], decode_body(result))

# batch: yields as they complete (a failed URL yields {"error": ...}, never aborts)
for res in client.iter([{"url": u, "httpResponseBody": True} for u in urls]):
    ...
```

Async mirrors `zyte_api.AsyncZyteAPI`:

```python
import asyncio
from scrapeland.zyte import AsyncZyteAPI

async def main():
    client = AsyncZyteAPI("pb_live_YOURKEY")
    res = await client.get({"url": "https://toscrape.com", "httpResponseBody": True})
    async for r in client.iter(queries):
        ...

asyncio.run(main())
```

### Compatibility and differences

| Zyte field | scrapeland |
|---|---|
| `url`, `httpRequestMethod`, `httpRequestText` | yes |
| `httpResponseBody` (base64), `httpResponseHeaders` | yes |
| `customHttpRequestHeaders` / `requestHeaders` | yes |
| `geolocation` (country code) | yes (exit country) |
| `sessionContext` (first `id`) / `session` | yes (sticky session) |
| `browserHtml`, `screenshot`, auto-extraction (`product`, `article`, ...) | no (raises `UnsupportedQuery`) |

scrapeland tunnels **raw HTTP** (it does not render pages or run extraction), so
browser/AI-extraction fields are intentionally unsupported. Everything in Zyte
API's HTTP mode works.

## Configuration

| Argument | Env var | Default |
|---|---|---|
| `api_key` | `SCRAPELAND_API_KEY` (the Zyte adapter also reads `ZYTE_API_KEY`) | (required) |
| `gateway` | `SCRAPELAND_GATEWAY` | `http://gateway.scrape.land:8080` |
| `retries` | (none) | `2` (transient failures retry with backoff; each retry rotates IP) |

Point `gateway` at `http://localhost:8180` to test against a local stack. Because
the Zyte adapter falls back to `ZYTE_API_KEY`, an existing `ZyteAPI()` call that
relied on that env var keeps working after the import swap.

Full HTTP/curl/Node/Scrapy docs: <https://scrape.land/docs>.
