Metadata-Version: 2.4
Name: web-scout-ai
Version: 1.7.0
Summary: Agentic web research tool — smarter than search, faster than deep research. Search, scrape, and synthesize web content using LLMs.
License: MIT
License-File: LICENSE
Keywords: web-research,ai-agent,web-scraping,llm,search,rag,crawling,document-extraction,pdf-extraction,playwright,docling,synthesis,openai-agents,serper,duckduckgo,litellm,grounded-research
Author: Riccardo Soldan
Requires-Python: >=3.10,<3.14
Classifier: Development Status :: 4 - Beta
Classifier: Framework :: Pydantic :: 2
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Natural Language :: English
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Markup :: HTML
Classifier: Typing :: Typed
Requires-Dist: boto3 (>=1.43.61,<2.0.0)
Requires-Dist: docling (>=2.0)
Requires-Dist: docling-core (>=2.0)
Requires-Dist: httpx (>=0.24)
Requires-Dist: litellm (>=1.30)
Requires-Dist: markdownify (>=0.11)
Requires-Dist: openai-agents (>=0.13)
Requires-Dist: pydantic (>=2.0)
Requires-Dist: pydantic-settings (>=2.0)
Requires-Dist: scrapling[fetchers] (>=0.4.15)
Project-URL: Repository, https://github.com/RSO9192/web-scout-ai
Description-Content-Type: text/markdown

# web-scout-ai

![web-scout-ai logo](assets/web-scout-logo.svg)

[![PyPI Version](https://img.shields.io/pypi/v/web-scout-ai)](https://pypi.org/project/web-scout-ai/)
[![Downloads](https://static.pepy.tech/badge/web-scout-ai/month)](https://pepy.tech/project/web-scout-ai)
[![Python Versions](https://img.shields.io/pypi/pyversions/web-scout-ai)](https://pypi.org/project/web-scout-ai/)
[![License](https://img.shields.io/github/license/RSO9192/web-scout-ai)](LICENSE)

**Web research that actually reads the sources.**

`web-scout-ai` is an agentic Python research pipeline that searches the web, opens and extracts the sources it finds, follows up when evidence is incomplete, and returns a cited answer together with a full audit trail of what it could — and could not — read.

Give it a question or a URL:

```python
from web_scout import run_web_research

result = await run_web_research(
    "Indonesia biodiversity hotspots current status and recent trend"
)

print(result.synthesis)
print(f"Read {len(result.scraped)} sources across {len(result.queries)} searches")
```

Instead of treating search snippets as evidence, `web-scout-ai` tries to **read the underlying source itself** before allowing it to support the final answer.

<details open>
<summary><strong>Example output</strong> (excerpt)</summary>

Query: `Indonesia biodiversity hotspots current status and recent trend`

> Indonesia remains one of the world's most important centres of biodiversity,
> but the available evidence indicates a negative overall status and trend. The
> UNEP country fiche describes Indonesia as one of 17 megadiverse countries,
> containing two globally recognised biodiversity hotspots; it reports
> approximately 31,750 plant species, 732 mammals, 1,711 birds, 750 reptiles,
> 403 amphibians and 1,236 freshwater fish.
> [Biodiversity / Indonesia | Interactive Country Fiches](https://dicf.unepgrid.ch/indonesia/biodiversity)
>
> A 2024 review reports that, according to Ministry of Environment and Forestry
> data for 2020, Indonesia retained 86.9 million hectares of forest, including
> 45.3 million hectares of primary forest. In Sundaland, 70.6% of the region
> had an intense human-footprint score (2009 dataset); among 308 lowland-forest
> bird species assessed, 273, or 88.6%, experienced a reduction in suitable
> range between 2000 and 2015.
> [Forest fragmentation in Indonesia](https://www.sciencedirect.com/science/article/pii/S2351989424001227),
> [Severe human pressures in the Sundaland biodiversity hotspot](https://conbio.onlinelibrary.wiley.com/doi/10.1111/csp2.169)
>
> The most recent forest-loss signal is mixed but concerning. WRI/University of
> Maryland analysis reports that Indonesia's tropical primary-forest loss
> increased 14% from 2024 to 2025, although it remained below the high levels
> of the mid-2010s. A global land-use analysis covering 1995–2022 found that
> Indonesia accounted for 22% of global biodiversity losses attributed to
> land-use change, the largest national share in the study.
> [Tropical rainforest loss slowed in 2025](https://gfr.wri.org/latest-analysis-deforestation-trends),
> [Biodiversity impacts of recent land-use change](https://www.nature.com/articles/s41893-024-01433-4)
>
> Comparisons should be made cautiously because the sources use different
> indicators and years: human-footprint data end in 2009, bird-range data in
> 2015, several forest assessments in 2020–2022, and primary-forest-loss data
> in 2025.

This is a fraction of what that call returned. The full synthesis was 10,055 characters; `result.scraped` also held extracted evidence from all 9 sources (~34,000 characters)—including a ScienceDirect review, a Conservation Biology paper, a Nature paper, a CEPF ecosystem profile, an interactive UNEP country fiche, and WRI's 2025 forest-loss analysis—plus an audit trail of the 3 searches, 1 irrelevant fetch, and 13 snippet-only hits that were not used as citations. The run took 188.7 seconds. Results and timing vary with the query, sources, model, and network.

</details>

## What it actually does

`web-scout-ai` is an async Python research pipeline, not a search-result wrapper. It searches, opens the selected sources, routes each response to the right extractor, evaluates whether the collected evidence is sufficient, and synthesizes only from sources it successfully read.

| Input encountered | What the pipeline does |
| --- | --- |
| Static HTML | Uses a fast HTTP fetch and converts the page to clean Markdown |
| JS-heavy pages and SPA shells | Escalates to a stealth Chromium browser and can interact with page controls |
| Cloudflare or similar bot challenges | Retries with Scrapling's stealth browser and Cloudflare challenge handling |
| PDF, DOCX, PPTX, XLSX | Downloads and converts the document with Docling |
| Scanned PDFs, charts, maps, images | Uses the configured vision model when text extraction is insufficient |
| JSON endpoints | Converts structured payloads into readable evidence |
| List, index, and database pages | Ranks detail links, follows them, and can take one pagination hop |
| Thin or incomplete evidence | Scrapes promising backlog URLs or generates targeted follow-up searches |

The browser fallback can pass many JS-gated and Cloudflare-protected pages, but it is not a guarantee: sites can still block automation. Those URLs are reported in `result.bot_detected` instead of disappearing or being presented as evidence. Use the package only where you have permission and in accordance with the source site's terms.

## Quick start

### 1. Install

```bash
pip install web-scout-ai
web-scout-setup
```

`web-scout-setup` installs the Patchright-managed Chromium browser and its system dependencies. It may request `sudo` for OS-level browser libraries.

### 2. Configure keys

The default research models run on Amazon Bedrock Mantle. Vision fallback still uses Gemini, and open-web discovery uses Serper:

```bash
export AWS_BEARER_TOKEN_BEDROCK="your-bedrock-api-key"
export GEMINI_API_KEY="your-gemini-api-key"
export SERPER_API_KEY="your-serper-api-key"
```

`BEDROCK_MANTLE_API_KEY` is accepted as an alias for the bearer token. OpenAI models on Mantle, including GPT-5.6 and GPT-6 Luna, use LiteLLM's route in `us-east-1`.

To use the Exa search backend instead (`search_backend="exa"`):

```bash
export EXA_API_KEY="your-exa-api-key"
```

Direct-URL mode does not use a search backend, so it only needs the API key for your configured model provider.

### 3. Run research

```python
import asyncio

from web_scout import run_web_research


async def main():
    result = await run_web_research(
        query="What are the main threats to coral reefs worldwide?",
        cache=True,
    )

    print(result.synthesis)

    print("\nSources read:")
    for source in result.scraped:
        print(f"- {source.title or source.url}: {source.url}")

    if result.bot_detected or result.scrape_failed:
        print(
            f"\nCould not read "
            f"{len(result.bot_detected) + len(result.scrape_failed)} source(s)"
        )


asyncio.run(main())
```

## Three ways to use it

### Open-web research

Generate several searches, read the strongest results in parallel, evaluate coverage, and search again if important evidence is missing.

```python
result = await run_web_research(
    query="What is driving the global adaptation finance gap?",
)
```

### Domain-restricted research

Keep discovery and hub deepening focused on authoritative domains.

```python
result = await run_web_research(
    query="Latest evidence on sea-level rise",
    include_domains=["ipcc.ch", "nasa.gov"],
)
```

### Direct URL extraction

Skip search and start from a page, document, API endpoint, image, or database listing.

```python
result = await run_web_research(
    query="Extract the recommended adaptation measures and supporting evidence",
    direct_url="https://example.org/report.pdf",
)
```

For a document, the pipeline reads that document without wandering into site navigation. For a list or database page, it can rank and follow relevant records and their linked primary documents.

## The return value is an audit trail

`run_web_research()` returns a typed `WebResearchResult`:

```python
class WebResearchResult(BaseModel):
    synthesis: str
    scraped: list[UrlEntry]
    scrape_failed: list[UrlEntry]
    blocked_by_policy: list[UrlEntry]
    source_http_error: list[UrlEntry]
    scraped_irrelevant: list[UrlEntry]
    bot_detected: list[UrlEntry]
    snippet_only: list[UrlEntry]
    queries: list[SearchQuery]
```

| Field | Meaning |
| --- | --- |
| `synthesis` | Final answer with inline Markdown citations |
| `scraped` | Sources successfully read and extracted; `content` contains the query-relevant evidence |
| `scrape_failed` | Extraction attempts that failed for an unclassified reason |
| `blocked_by_policy` | Sources skipped because their domain is in `exclude_domains` |
| `source_http_error` | Source-side HTTP or network failures |
| `scraped_irrelevant` | Pages fetched successfully but not useful for the query |
| `bot_detected` | Sources that still returned a bot-protection wall |
| `snippet_only` | Search results discovered but not opened; snippets are never valid citation targets |
| `queries` | Every search query executed, its result count, and domain restrictions |

`UrlEntry` contains `url`, `title`, and `content`. `SearchQuery` contains `query`, `num_results_returned`, and `domains_restricted`.

The synthesizer is instructed to use scraped evidence only. A deterministic final check rejects citations to invented or snippet-only URLs and retries the synthesis with feedback.

## Research depth

```python
# Faster default
await run_web_research(query="...", research_depth="standard")

# More searches and sources, with a stricter coverage threshold
await run_web_research(query="...", research_depth="deep")
```

| Budget | Standard | Deep |
| --- | ---: | ---: |
| Maximum search iterations | 2 | 3 |
| Initial search queries | 3 | 5 |
| Follow-up search queries | 2 | 4 |
| URLs selected in the first round | 6 | 12 |
| URLs selected in a follow-up round | 4 | 8 |
| Hub/detail-page cap | 10 | 15 |

The coverage evaluator can stop early when the evidence already answers the question. You can add your own acceptance conditions with `coverage_criteria`:

```python
result = await run_web_research(
    query="Compare national methane policies",
    coverage_criteria="Include at least one primary government source per country.",
)
```

## Custom extractor guidance

Use `extractor_guidance` to refine what each per-source extractor keeps and how it
organizes evidence. The guidance applies only to extraction; it does not alter search
query generation, coverage evaluation, or final synthesis.

```python
result = await run_web_research(
    query="Tunisia agricultural trends",
    extractor_guidance="""
Country of interest: Tunisia

- Keep findings explicitly about Tunisia.
- Also keep regional findings whose stated scope includes Tunisia, even when the
  source does not name Tunisia; label them with the source's scope, such as
  "North Africa:".
- Discard purely global findings and findings about geographies that do not
  include Tunisia.
""",
)
```

Guidance augments rather than replaces WebScout's extractor instructions. The base
contract always takes precedence for source grounding, tool constraints, output
fields, page-type handling, and the exact no-evidence sentinel.

## Configuration

The defaults are Bedrock Mantle GPT-6 Luna for research, extraction, and follow-up selection, and `gemini/gemini-3.7-flash` for vision. Model IDs follow [LiteLLM provider naming](https://docs.litellm.ai/docs/providers), so each stage can use a different provider or model.

```python
models = {
    # Used as the fallback for query generation, coverage, and synthesis
    "web_researcher": "openai/gpt-4o-mini",

    # Reads and extracts individual sources
    "content_extractor": "gemini/gemini-2.0-flash",

    # Optional stage-specific choices
    "query_generator": "openai/gpt-4o-mini",
    "coverage_evaluator": "openai/gpt-4o-mini",
    "synthesiser": "openai/gpt-4o-mini",
    "followup_selector": "openai/gpt-4o-mini",
    "vision_fallback": "gemini/gemini-2.0-flash",
}

result = await run_web_research(query="...", models=models)
```

Provider credentials are read from their standard environment variables, such as `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GEMINI_API_KEY`, or AWS credentials for Bedrock.

### Public API

```python
result = await run_web_research(
    query="latest IPCC findings on sea-level rise",
    models=None,                       # optional; GPT-6 Luna defaults, Gemini vision
    search_backend="serper",          # "serper" (Google via serper.dev) or "exa" (exa.ai)
    research_depth="standard",        # "standard", "deep", or a custom dict
    include_domains=["ipcc.ch"],       # optional discovery restriction
    direct_url=None,                   # optional; skips search when set
    domain_expertise="climate science",  # optional prompt context
    exclude_domains=None,              # optional blocklist; nothing is blocked
                                       # by default. See "Domain policy" below.
    max_pdf_pages=50,                  # pages converted from each PDF
    max_content_chars=30_000,          # characters passed to the extractor per source
    cache=False,                       # process-local source cache
    coverage_criteria=None,            # extra evidence requirements
    extractor_guidance=None,           # optional per-source extraction guidance
)
```

### Domain policy

Nothing is blocked by default: `exclude_domains=None` means every domain is eligible for search and scraping. To block domains, pass a list explicitly. A curated list of platforms that waste the scrape budget (social/video sites with no scrapeable text, bot-blocked search engines, hard-paywalled publishers) is exported as `RECOMMENDED_EXCLUDE_DOMAINS`:

```python
from web_scout import run_web_research, RECOMMENDED_EXCLUDE_DOMAINS

# Opt in to the curated list
result = await run_web_research(
    query="...",
    exclude_domains=RECOMMENDED_EXCLUDE_DOMAINS,
)

# Curated list plus your own additions
result = await run_web_research(
    query="...",
    exclude_domains=[*RECOMMENDED_EXCLUDE_DOMAINS, "statista.com"],
)
```

Exclusions are applied natively by search backends that support them (Exa's `excludeDomains`); backends without native support (Serper) rely on the post-search URL filter, which always runs. Contradictions fail fast: a domain appearing in both `include_domains` and `exclude_domains`, or a `direct_url` whose host is excluded, raises `ValueError`.

### Source caching

With `cache=True`, successful raw source artifacts are reused by later `run_web_research()` calls in the same Python process. Pages and documents are not fetched or converted again, but query-specific extraction and synthesis still run each time.

The cache is in-memory only. Failed scrapes, final answers, query-specific summaries, and click-driven browser sessions are not cached.

## Pipeline, in one view

```text
question
   │
   ├─ generate targeted searches ─ search in parallel ─ select diverse URLs
   │                                                     │
   │                                                     ▼
   │      static HTML ────────┐                    fetch in parallel
   │      JS / bot challenge ─┤                          │
   │      documents ──────────┼─ route + extract ◀───────┘
   │      JSON / images ──────┘          │
   │                                    ▼
   └─ follow-up search ◀── evaluate evidence coverage
                                        │ sufficient
                                        ▼
                              grounded synthesis
                                        │
                                        ▼
                               citation validation
```

## Where it fits

Use `web-scout-ai` when your application needs the contents of real pages and documents—not only search snippets—and you want the source successes and failures returned as structured data.

It is intentionally a bounded research component. If you only need search links, use a search API directly. If you need an open-ended autonomous research process with human checkpoints, put this package inside a broader agent workflow.

## Requirements

- Python 3.10–3.13
- An API key for the configured LLM provider
- A Serper or Exa API key for search mode
- Chromium setup for rendered pages, interactive sites, and browser fallbacks

## Contributing

The main extension point is [`SearchBackend`](src/web_scout/search_backends.py). New backends should implement the async `search()` contract and return normalized results (title, url, snippet).

Bug reports and focused pull requests are welcome at [github.com/RSO9192/web-scout-ai](https://github.com/RSO9192/web-scout-ai).

## License

[MIT](LICENSE)

