Metadata-Version: 2.4
Name: langchain-wtmapi
Version: 0.1.0
Summary: LangChain Document Loader for WTM API — convert any webpage to Markdown
Author-email: Filippo Tedeschi <filippotedeschi98@gmail.com>
License: MIT
Project-URL: Homepage, https://wtmapi.com
Project-URL: Documentation, https://wtmapi.com/docs
Project-URL: Repository, https://github.com/filtede98/langchain-wtmapi
Keywords: langchain,document-loader,markdown,web-scraping,rag,llm,wtm-api
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Text Processing :: Markup
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: langchain-core>=0.2.0
Dynamic: license-file

# langchain-wtmapi

LangChain Document Loader for [WTM API](https://wtmapi.com) — convert any webpage to clean, structured Markdown.

## Installation

```bash
pip install langchain-wtmapi
```

## Quick Start

```python
from langchain_wtmapi import WTMApiLoader

# Get your free API key at https://wtmapi.com
loader = WTMApiLoader(
    urls=[
        "https://en.wikipedia.org/wiki/Artificial_intelligence",
        "https://developer.mozilla.org/en-US/docs/Web/JavaScript",
    ],
    api_key="wtm_your_api_key",
)

# Load all documents
docs = loader.load()

for doc in docs:
    print(f"Source: {doc.metadata['source']}")
    print(f"Length: {doc.metadata['length']} chars")
    print(f"Response time: {doc.metadata['response_time_ms']}ms")
    print(doc.page_content[:500])
    print("---")
```

## Lazy Loading

For large batches, use lazy loading to process documents one at a time:

```python
for doc in loader.lazy_load():
    # Process each document as it's loaded
    process_document(doc)
```

## Use with RAG Pipeline

```python
from langchain_wtmapi import WTMApiLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import FAISS

# Load web pages as Markdown
loader = WTMApiLoader(
    urls=["https://docs.python.org/3/tutorial/index.html"],
    api_key="wtm_your_api_key",
)
docs = loader.load()

# Split into chunks
splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
chunks = splitter.split_documents(docs)

# Create vector store
embeddings = OpenAIEmbeddings()
vectorstore = FAISS.from_documents(chunks, embeddings)

# Query
results = vectorstore.similarity_search("How do I use lists in Python?")
```

## Options

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `urls` | `list[str]` | required | List of URLs to convert |
| `api_key` | `str` | required | Your WTM API key |
| `include_links` | `bool` | `True` | Include links in output |
| `include_images` | `bool` | `True` | Include images in output |
| `api_url` | `str` | `https://wtmapi.com/api/v1/convert` | API endpoint |

## Document Metadata

Each loaded document includes metadata:

```python
{
    "source": "https://example.com",        # Original URL
    "length": 15234,                         # Markdown length in chars
    "response_time_ms": 523,                 # API response time
    "usage_used": 5,                         # Monthly calls used
    "usage_limit": 50,                       # Monthly call limit
    "plan": "free",                          # Current plan
}
```

## Get Your API Key

Sign up at [wtmapi.com](https://wtmapi.com) — free tier includes 50 calls/month, no credit card required.

## License

MIT
