Metadata-Version: 2.5
Name: finscrape
Version: 0.1.0
Summary: Sign in to retail brokerages, scrape positions and trades, and build a portfolio snapshot workbook.
Project-URL: Homepage, https://github.com/ljohri/finscrape
Project-URL: Repository, https://github.com/ljohri/finscrape
Project-URL: Issues, https://github.com/ljohri/finscrape/issues
Project-URL: Changelog, https://github.com/ljohri/finscrape/blob/main/CHANGELOG.md
Author: Lokesh Johri
License: MIT
License-File: LICENSE
Keywords: brokerage,etrade,fidelity,playwright,portfolio,robinhood,schwab,scraping,wells-fargo,xlsx
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Financial and Insurance Industry
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Office/Business :: Financial :: Investment
Requires-Python: >=3.11
Requires-Dist: beautifulsoup4>=4.12
Requires-Dist: lxml>=5.3
Requires-Dist: openpyxl>=3.1
Requires-Dist: pandas>=2.2
Requires-Dist: playwright>=1.49
Provides-Extra: dev
Requires-Dist: mypy>=1.13; extra == 'dev'
Requires-Dist: pandas-stubs>=2.2; extra == 'dev'
Requires-Dist: pytest>=8.3; extra == 'dev'
Requires-Dist: ruff>=0.8; extra == 'dev'
Provides-Extra: robinhood
Requires-Dist: httpx>=0.27; extra == 'robinhood'
Requires-Dist: mcp>=1.9; extra == 'robinhood'
Requires-Dist: pydantic>=2.9; extra == 'robinhood'
Description-Content-Type: text/markdown

# finscrape

Sign in to your retail brokerage accounts, scrape positions and trades, and build a
consolidated portfolio snapshot as an XLSX workbook.

Supported: **E\*TRADE**, **Fidelity**, **Charles Schwab**, **Wells Fargo Advisors**, and
**Robinhood**. Four are driven with Playwright against the real websites; Robinhood uses
its Agentic Trading MCP API over OAuth.

```console
$ finscrape run -c accounts.json
[1/3] robinhood (robinhood)
robinhood: 1 account(s), 8 holding(s), $12,450.00 (6.2s)
[2/3] schwab_main (schwab)
schwab_main: 1 account(s), 12 holding(s), $85,200.00 (51.8s)
[3/3] wellsfargo (wellsfargo)
wellsfargo: 2 account(s), 15 holding(s), $104,770.00 (94.1s)

TOTAL: $202,420.00 (cash $18,900.00 + holdings $183,520.00)
Workbook: output/portfolio.xlsx
```

---

## Contents

- [Why this exists](#why-this-exists)
- [Install](#install)
- [Quick start](#quick-start)
- [The output workbook](#the-output-workbook)
- [Configuration reference](#configuration-reference)
- [Per-institution options](#per-institution-options)
- [Python API](#python-api)
- [Sample application](#sample-application)
- [CLI reference](#cli-reference)
- [How scraping works](#how-scraping-works)
- [Troubleshooting](#troubleshooting)
- [Extending: add your own broker](#extending-add-your-own-broker)
- [Security notes](#security-notes)
- [Limitations](#limitations)
- [Development](#development)
- [License](#license)

---

## Why this exists

Most brokerages will show you one account at a time. If your money is spread across five
institutions, answering "what do I actually own, and what is it worth?" means five logins
and a manual spreadsheet.

finscrape automates the collection and normalizes wildly different page layouts — HTML
tables, ARIA role grids, AG Grid virtual scrollers, and a JSON-RPC API — into one data
model, then writes a workbook you can diff, chart, or feed into your own analysis.

It is **read-only by design**. Nothing in this library places, modifies, or cancels an
order. The Robinhood adapter actively penalizes any MCP tool whose name or description
suggests it writes.

## Install

```bash
pip install finscrape
python -m playwright install chromium
```

With [uv](https://docs.astral.sh/uv/):

```bash
uv add finscrape
uv run playwright install chromium
```

Robinhood needs an extra, since it pulls in the MCP SDK:

```bash
pip install 'finscrape[robinhood]'
```

Python 3.11 or newer is required.

## Quick start

**1. Create a config.**

```bash
finscrape init accounts.json
```

**2. Edit it.** Enable the accounts you want and reference environment variables instead
of writing secrets into the file:

```json
{
  "version": 1,
  "output": { "workbook": "output/portfolio.xlsx", "data_dir": "output/data" },
  "accounts": [
    {
      "key": "schwab_main",
      "institution": "schwab",
      "sheet_key": "SCHWAB",
      "credentials": {
        "username": "${SCHWAB_USERNAME}",
        "password": "${SCHWAB_PASSWORD}"
      }
    }
  ]
}
```

**3. Export your credentials and check the config** without opening a browser:

```bash
export SCHWAB_USERNAME='...' SCHWAB_PASSWORD='...'
finscrape validate -c accounts.json
```

**4. Run it.**

```bash
finscrape run -c accounts.json
```

A Chromium window opens for each browser-based account. finscrape fills the login form;
**you complete the two-factor prompt yourself**, and it waits for you. Sessions persist in
a per-account browser profile, so subsequent runs usually skip 2FA entirely.

## The output workbook

Three tabs, rewritten on every run except where noted.

### `Consolidated`

One row per account, plus an `ALL ACCOUNTS` total.

| Account | Institution | Cash | Holdings Value | Cost Basis | Unrealized | Total Portfolio | Account Number |
| --- | --- | --- | --- | --- | --- | --- | --- |
| SCHWAB | schwab | 5,200.00 | 80,000.00 | 60,000.00 | 20,000.00 | 85,200.00 | 1234 |
| WFA_8472 | wellsfargo | 12,450.00 | 92,320.00 | 90,000.00 | 2,320.00 | 104,770.00 | 8472 |
| **ALL ACCOUNTS** | | **17,650.00** | **172,320.00** | **150,000.00** | **22,320.00** | **189,970.00** | |

### `Portfolio_History`

Append-only, newest first, **one row per calendar day**. Re-running on the same day
replaces that day's row rather than adding a duplicate, so day-over-day comparisons stay
meaningful. This is the only tab that survives across runs — everything else is rebuilt.

| Run Datetime | Total Portfolio | Cash | Holdings Value | Cost Basis | Unrealized | Accounts |
| --- | --- | --- | --- | --- | --- | --- |
| 2025-08-21 16:04:11 | 189,970.00 | 17,650.00 | 172,320.00 | 150,000.00 | 22,320.00 | 2 |
| 2025-08-20 16:02:55 | 188,410.00 | 17,650.00 | 170,760.00 | 150,000.00 | 20,760.00 | 2 |

### `Tickers`

Every holding rolled up by symbol across all accounts, largest position first.

| Ticker | Description | Total Shares | Current Price | Current Value | Accounts |
| --- | --- | --- | --- | --- | --- |
| MSFT | MICROSOFT CORP | 120 | $425.50 | 51,060.00 | WFA_8472 |
| SPY | SPDR S&P 500 ETF | 80 | $515.75 | 41,260.00 | WFA_8472 |
| AAPL | Apple Inc | 50 | $190.00 | 9,500.00 | FIDELITY, SCHWAB |

**This tab deliberately has no gain columns** — no cost basis, no unrealized, no realized.
It answers "how much of each symbol do I hold and what is it worth", which is the question
that does not depend on lot accounting. See [Limitations](#limitations).

Any extra sheets you add to the workbook by hand are preserved; finscrape only rewrites
the three tabs it owns, and always writes through a temp file so an interrupted save
cannot destroy your accumulated history.

### Side outputs

Alongside the workbook, each account writes to `output/data/<institution>/<account_key>/`:

```
positions/positions_<sheet_key>.csv    one row per holding, plus a CASH row
trades/trades_<sheet_key>.csv          normalized transactions
html/<label>_<timestamp>.html          page snapshots, for debugging
_session/errors.log                    warnings and errors, per run
account_meta_<sheet_key>.json          what was scraped and from where
```

The HTML snapshots are the single most useful artifact when a broker changes their layout
and a scrape starts returning nothing. Disable them with `"save_html_snapshots": false`.

## Configuration reference

### Top level

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `version` | int | required | Config schema version. Currently `1`. |
| `output` | object | see below | Where artifacts are written. |
| `defaults` | object | `{}` | Defaults applied to every account. |
| `accounts` | array | required | At least one account. |

### `output`

| Field | Default | Description |
| --- | --- | --- |
| `workbook` | `output/portfolio_positions.xlsx` | XLSX destination. |
| `data_dir` | `output/data` | Root for CSVs, snapshots, and logs. |
| `profile_dir` | `.finscrape-profiles` | Persistent browser profiles. |
| `write_csv` | `true` | Write the per-account CSV side outputs. |
| `save_html_snapshots` | `true` | Save page HTML for debugging. |

Relative paths resolve against the **config file's directory**, so a config can move
between machines without editing paths.

### `defaults`

| Field | Default | Description |
| --- | --- | --- |
| `headless` | `false` | Run the browser headless. Rarely works — see below. |
| `timeout_ms` | `60000` | Navigation timeout per page. |

> **On headless mode:** brokerage sites fingerprint aggressively, and 2FA needs a human.
> Leave `headless` at `false` unless you have a warm session and have tested that
> particular broker.

### `accounts[]`

| Field | Required | Description |
| --- | --- | --- |
| `key` | yes | Unique identifier. Names the output directory. |
| `institution` | yes | One of `etrade`, `fidelity`, `robinhood`, `schwab`, `wellsfargo`. |
| `sheet_key` | no | Label on the Consolidated tab. Defaults to `key` upper-cased. |
| `enabled` | no | Default `true`. Set `false` to keep an entry without running it. |
| `credentials` | no | `{ "username": ..., "password": ... }`. Omit for Robinhood. |
| `headless` | no | Overrides `defaults.headless`. |
| `timeout_ms` | no | Overrides `defaults.timeout_ms`. |
| `profile_dir` | no | Overrides the per-account browser profile location. |
| `options` | no | Institution-specific — see the next section. |

### Environment variable interpolation

Any string value supports `${VAR}` and `${VAR:-default}`:

```json
"credentials": { "username": "${SCHWAB_USERNAME}", "password": "${SCHWAB_PASSWORD}" },
"options": { "auth_file": "${RH_AUTH:-~/.finscrape/robinhood_auth.json}" }
```

An unset variable with no default expands to an empty string rather than failing, so a
config listing five brokers still loads when you have only set up two. The missing
credential surfaces later as a `CredentialsError` that names the offending account.

**This is what makes the config file safe to commit.**

## Per-institution options

### `wellsfargo`

One sign-on covers every account. The adapter returns one snapshot per account, with
sheet keys like `WFA_8472`.

| Option | Default | Description |
| --- | --- | --- |
| `account_suffixes` | all discovered | Last 4 digits of the accounts to scrape, e.g. `["8472", "3916"]`. |
| `trades` | `true` | Also scrape posted transactions. |

### `etrade`

One sign-on can cover a stock plan *and* a linked brokerage account. Enable either or
both; each produces its own snapshot.

| Option | Default | Description |
| --- | --- | --- |
| `stock_plan` | `false` | Scrape the stock-plan (SAR/RSU) grants grid. |
| `brokerage` | `true` | Scrape the ordinary brokerage positions and transactions. |
| `brokerage_account_number` | — | Last 4 digits. Selects the account on the transactions page. |
| `stock_plan_sheet_key` | `<sheet_key>_SP` | Sheet key for the stock-plan snapshot. |
| `brokerage_sheet_key` | derived | Sheet key for the brokerage snapshot. |
| `stock_plan_symbol` | — | Fallback ticker when the grants grid omits it. |
| `login_url`, `stock_plan_url`, `positions_url`, `transactions_url` | E\*TRADE defaults | Override if the site moves. |

Stock-plan accounts report **sellable quantity**, not granted quantity, and always report
zero cash — the linked brokerage account holds the sweep.

### `schwab`

| Option | Default | Description |
| --- | --- | --- |
| `lot_details` | `true` | Open per-symbol lot modals for lot-level cost basis. Set `false` for a much faster, position-level scrape. |
| `trades` | `true` | Also scrape transaction history. |
| `account_number` | auto | Override the detected account number. |
| `positions_url`, `history_url` | Schwab defaults | Override if the site moves. |

### `fidelity`

| Option | Default | Description |
| --- | --- | --- |
| `positions_url` | Fidelity's positions page | Override if you land somewhere else. |
| `allow_empty` | `false` | Do not fail when the account has no positions. |

Positions only — Fidelity's portfolio page exposes no machine-readable transaction
history. "Pending Activity" is added to the cash balance so the total matches the site.

### `robinhood`

No username or password. First run opens a browser page to approve OAuth access; the
token is cached and refreshed silently after that.

| Option | Default | Description |
| --- | --- | --- |
| `auth_file` | `<data_dir>/robinhood_auth.json` | Where the OAuth token is cached (mode `0600`). |
| `mcp_url` | `https://agent.robinhood.com/mcp/trading` | MCP endpoint. |
| `callback_port` | `3030` | Local port for the OAuth redirect. |
| `open_browser` | `true` | Open the approval URL automatically. |
| `account_number` | auto | Pick a specific account; otherwise the agentic-enabled one wins. |

## Python API

### One call

```python
from finscrape import run_from_config

result = run_from_config("accounts.json")
print(result.report())
```

### Step by step

```python
from finscrape import load_config, scrape_portfolio, write_workbook

config = load_config("accounts.json")
run = scrape_portfolio(config, only=["schwab", "fidelity"])

for failure in run.failed:
    print(f"{failure.account_key} failed: {failure.error}")

write_workbook(run.portfolio, "portfolio.xlsx")
```

`scrape_portfolio` never raises for a broker failure. Each account runs independently, and
you inspect `run.failed`, `run.succeeded`, or `run.ok`.

### Working with the data

```python
portfolio = run.portfolio

totals = portfolio.totals()
print(f"Net worth: ${totals.total_value:,.2f}")

for account in portfolio:
    print(f"{account.sheet_key:<16} ${account.total_value:>12,.2f}")

for rollup in portfolio.ticker_rollup()[:5]:
    weight = rollup.value / totals.holdings_value
    print(f"{rollup.symbol:<6} {rollup.quantity:>10,.2f} sh  {weight:>6.1%}")
```

### Core types

| Type | Purpose |
| --- | --- |
| `Holding` | One position row. `market_value`, `total_cost`, `unrealized` are derived. |
| `Trade` | One transaction. `activity` is normalized to `Buy` / `Sell` / `Reinvest Dividend` / `Other`. |
| `AccountSnapshot` | One account at a point in time: cash, holdings, trades. |
| `Portfolio` | All accounts from a run. `totals()`, `ticker_rollup()`, `account(sheet_key)`. |
| `RunResult` | Portfolio plus per-account success/failure and timings. |

`Holding` accepts either `cost_basis` (total) or `cost_per_share`, and derives the other.
Holdings with no quotable price fall back to cost basis for market value, so illiquid
positions do not silently vanish from your totals.

### Logging

finscrape configures no handlers — your application decides.

```python
import logging

logging.basicConfig(level=logging.INFO, format="%(asctime)s  %(levelname)-7s  %(message)s")
```

Use `logging.DEBUG` to see per-strategy navigation decisions, which is what you want when
a broker's site has changed.

### Exceptions

All inherit from `FinscrapeError`.

| Exception | Raised when |
| --- | --- |
| `ConfigError` | The config file is missing, malformed, or invalid. |
| `CredentialsError` | An enabled account has no username or password. |
| `UnknownInstitutionError` | `institution` is not a registered adapter. |
| `NavigationError` | The adapter could not reach a required page. |
| `ExtractionError` | The page loaded but no data could be parsed from it. |
| `DependencyError` | An optional dependency is missing (e.g. the MCP SDK). |

## Sample application

`examples/portfolio_snapshot/` is a working application meant as a starting point, not a
toy. It shows the shape of real usage: inspect the plan, scrape tolerantly, write the
workbook, then do something with the data.

```bash
cd examples/portfolio_snapshot
cp accounts.example.json accounts.json    # then enable your accounts
export SCHWAB_USERNAME='...' SCHWAB_PASSWORD='...'

uv run python snapshot.py --config accounts.json
uv run python snapshot.py --config accounts.json --only schwab   # one broker at a time
```

| File | What it demonstrates |
| --- | --- |
| `accounts.example.json` | Every supported institution, fully commented by example, all disabled, zero secrets. |
| `snapshot.py` | Load → describe the plan → scrape → write → analyze. Ends with a position-concentration report built from `ticker_rollup()`. |
| `add_account.py` | Interactively append an account to a config, prompting for that broker's options and writing `${ENV_VAR}` placeholders rather than secrets. |

Adding your second broker:

```bash
uv run python add_account.py --config accounts.json --institution fidelity --key fidelity_ira
```

## CLI reference

```
finscrape run -c CONFIG [--only KEY ...] [-o WORKBOOK] [--json] [--fail-fast]
finscrape validate -c CONFIG
finscrape institutions [--json]
finscrape init [PATH] [--force]
```

| Command | Purpose |
| --- | --- |
| `run` | Scrape and write the workbook. Exits `0` on partial success unless `--fail-fast`. |
| `validate` | Parse the config, resolve env vars, report missing credentials. No browser. |
| `institutions` | List adapters with their transport and trade support. |
| `init` | Write a starter config. |

Global flags: `-v/--verbose` for debug logging, `--version`.

## How scraping works

Each adapter handles one broker, but they share a pipeline: launch a persistent browser
profile → sign in → navigate → extract → normalize into `AccountSnapshot`.

**Accounts run independently and in a deliberate order.** API-based brokers go first —
they are fast and need nobody at the keyboard — so their data is already collected before
a long interactive login begins. A broker that fails is recorded and the run continues.

**Two-factor authentication is manual, by design.** finscrape fills the username and
password, then waits for the target page to appear while you complete the challenge in the
browser. Because each account has its own persistent Chromium profile, the session
normally survives and later runs skip the prompt.

Broker-specific notes worth knowing:

- **Wells Fargo** is a single-page app behind a per-session token. The adapter tries four
  navigation strategies in order (direct URL, the SPA router, a sidebar click, a hidden
  form POST), *verifies* it landed on the right page before extracting, and re-mints a
  stale token by returning to the overview.
- **Fidelity** renders positions in an AG Grid, which has no `<table>` at all and splits
  one logical row across pinned and scrolling containers. Rows are merged by `row-index`
  and classified using the grid's own `posweb-row-*` classes.
- **Schwab** logs in inside an iframe with React-controlled inputs that silently reject a
  plain `fill()`, so the adapter verifies the field actually took the value and retries.
- **E\*TRADE** stock plans report *sellable* quantity, not granted quantity.
- **Robinhood** discovers its MCP tools at runtime rather than hardcoding names, and
  scores candidates so anything resembling an order-placing tool is never selected.

Prices come from the brokers themselves. finscrape makes no market data calls.

## Troubleshooting

**A scrape returns no positions.**
Read the HTML snapshot in `output/data/<institution>/<key>/html/`. It is exactly what the
page looked like when parsing failed, and usually shows either a login wall or a changed
layout. For Fidelity you can re-parse one offline:

```python
from finscrape.institutions.fidelity import parse_positions_html

holdings, cash, diagnostics = parse_positions_html("output/data/fidelity/fid/html/positions_….html")
print(diagnostics)  # row counts and the column ids actually seen
```

**Login hangs or 2FA never completes.**
Complete the challenge in the browser window that opened; the adapter is waiting for the
destination page and logs what it is waiting for. If the window is gone, delete that
account's profile directory under `.finscrape-profiles/` and start fresh.

**`NavigationError` from Wells Fargo.**
All four strategies failed. Run with `-v` to see each attempt and why verification failed,
and check `_session/errors.log`.

**Robinhood: `DependencyError`.**
`pip install 'finscrape[robinhood]'`.

**Robinhood: authorization loops.**
Delete the `auth_file` and re-run to force a fresh OAuth approval. If port 3030 is taken,
set `options.callback_port`.

**`CredentialsError` even though the variable is set.**
Confirm the shell running finscrape exported it: `finscrape validate -c accounts.json`
reports exactly which accounts resolved to empty credentials.

## Extending: add your own broker

Subclass `BrowserInstitution` (or `Institution` for an API), implement one method, and
register it. Your adapter is then usable from a config like any built-in.

```python
from finscrape import AccountSnapshot, Holding, register
from finscrape.institutions import BrowserInstitution
from finscrape.parsing.holdings import ColumnSpec, extract_holdings
from finscrape.parsing.tables import tables_from_page


@register
class MyBroker(BrowserInstitution):
    name = "mybroker"
    display_name = "My Broker"

    def collect(self, page):
        page.goto("https://mybroker.example/login")
        username, password = self.account.credentials.require(self.account.key)
        page.fill("#user", username)
        page.fill("#pass", password)
        page.click("#submit")
        page.wait_for_selector("#positions")
        self.context.snapshot(page, "positions")

        holdings, cash = extract_holdings(tables_from_page(page), ColumnSpec())
        return [
            AccountSnapshot(
                account_key=self.account.key,
                institution=self.name,
                sheet_key=self.sheet_key(),
                cash=cash,
                holdings=holdings,
            )
        ]
```

Return a **list** — one sign-on may expose several accounts. Helpers available to you:

| Helper | Use |
| --- | --- |
| `finscrape.parsing.tables` | HTML tables and ARIA role grids. |
| `finscrape.parsing.aggrid` | AG Grid row merging. |
| `finscrape.parsing.holdings` | Generic position/trade extraction driven by a `ColumnSpec`. |
| `finscrape.parsing.numbers` | `parse_number`, `parse_date`, `normalize_symbol`, `canonical_activity`. |
| `self.context.snapshot(page, label)` | Save page HTML for debugging. |
| `finscrape.browser.wait_for_manual_step` | Block for 2FA until a condition holds. |

## Security notes

- **No secrets in the config.** Use `${ENV_VAR}` so the file is safe to commit.
- **`Credentials.__repr__` masks the password**, so it cannot leak through a traceback or
  a debug log.
- **Browser profiles hold live sessions.** `.finscrape-profiles/` is as sensitive as a
  logged-in browser. Never commit it.
- **The Robinhood token file is written mode `0600`** and holds a refreshable OAuth token.
- **HTML snapshots contain your account data.** They are diagnostics, not artifacts to
  share; turn them off with `"save_html_snapshots": false`.
- **Read-only.** Nothing here places, modifies, or cancels orders.

You are responsible for complying with each broker's terms of service. Automating access
to your own accounts may or may not be permitted; check before you rely on this.

## Limitations

- **No realized gains, anywhere.** Computing them requires replaying full transaction
  history through a lot-matching engine (FIFO, specific-ID, average cost). finscrape
  reports what brokers state. `Consolidated` therefore shows cost basis and *unrealized*
  gain, both broker-reported, and `Tickers` carries no gain columns at all.
- **No market data.** Prices are whatever the broker displayed at scrape time.
- **Fidelity has no trade history.** Positions and cash only.
- **Options, bonds, and crypto** are extracted only where the broker lists them in the
  ordinary positions table; they are not modeled as distinct instrument types.
- **Scrapers are brittle by nature.** A broker redesign will break an adapter. HTML
  snapshots and layered navigation strategies are there to make the fix quick.

## Development

```bash
git clone git@github.com:ljohri/finscrape.git
cd finscrape
uv sync --extra dev --extra robinhood
uv run playwright install chromium

uv run pytest          # offline: no browser, no network, no credentials
uv run ruff check .
uv run ruff format --check .
uv run mypy src
```

The test suite feeds saved broker markup and API-shaped payloads directly into the parsing
functions, so it runs anywhere in a couple of seconds.

### Using this as a git submodule

From the parent repository that should embed finscrape:

```bash
git submodule add git@github.com:ljohri/finscrape.git fin_institution_scraping
git submodule update --init --recursive
uv add --editable ./fin_institution_scraping
```

Cloning that parent repository later:

```bash
git clone --recurse-submodules <parent-repo-url>
# or, if it is already cloned:
git submodule update --init --recursive
```

Pulling upstream changes into the parent:

```bash
git submodule update --remote fin_institution_scraping
git add fin_institution_scraping && git commit -m "Bump finscrape"
```

## License

MIT. See [LICENSE](LICENSE).
