Metadata-Version: 2.4
Name: amex-israel-scraper
Version: 0.1.0
Summary: Scrape American Express Israel (DigitalV3), in the israeli-bank-scrapers result shape
Project-URL: Homepage, https://github.com/Adam-Gold/amex-israel-scraper
Project-URL: Issues, https://github.com/Adam-Gold/amex-israel-scraper/issues
Author: Adam Gold
License: MIT License
        
        Copyright (c) 2026 Adam Gold
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: american-express,amex,finance,israel,israeli-bank-scrapers,scraper
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Office/Business :: Financial
Requires-Python: >=3.11
Requires-Dist: camoufox[geoip]>=0.4
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Description-Content-Type: text/markdown

# amex-israel-scraper

Scrape your own **American Express Israel** transactions, returning the same shape
[`israeli-bank-scrapers`](https://github.com/eshaham/israeli-bank-scrapers) returns.

## Why this exists

Amex Israel moved to a platform called **DigitalV3** (`web.americanexpress.co.il`) behind Cloudflare.
`israeli-bank-scrapers` drives Puppeteer, and headless Chrome is refused by that WAF before it
reaches a login form — so its `amex` scraper cannot reach the new platform at all.

Two things turned out to matter:

1. **The browser fingerprint, not the behaviour.** Cloudflare rejects headless Chrome on sight.
   [Camoufox](https://github.com/daijro/camoufox) — a patched Firefox — gets through. Slowing down,
   moving the mouse, and adding waits do not help a browser that is refused before it acts.
2. **Underneath the WAF, the site is a clean JSON API.** Nothing here parses HTML. The browser exists
   only to hold a session Cloudflare trusts; every call is a `fetch()` evaluated inside it.

The result is a scraper that is browser-based but not page-scraping: fast, and stable across the
site's visual redesigns.

## Install

```bash
pip install amex-israel-scraper
python -m camoufox fetch        # one-off: downloads the patched Firefox
```

Python 3.11+. The Camoufox fetch is a few hundred MB and only needs doing once.

From a clone instead:

```bash
pip install -e .
```

## Use it

```python
from amex_israel import scrape

result = scrape(
    {
        "id": "...",  # ת"ז
        "card6Digits": "...",  # first 6 digits of the card
        "password": "...",  # the permanent password (no OTP: Amex IL does not send one)
    }
)

if not result.success:
    raise SystemExit(f"{result.errorType}: {result.errorMessage}")

for account in result.accounts:
    print(account.accountNumber, len(account.txns))
    for txn in account.txns:
        print(txn.date, txn.description, txn.chargedAmount)
```

Or from the shell — credentials come from the environment, never from arguments, because an argument
lands in your shell history and in the process list:

```bash
export AMEX_ID=... AMEX_CARD6=... AMEX_PASSWORD=...
python -m amex_israel --session ~/.amex-session.json > result.json
```

### History

By default you get the current open billing cycle. To walk closed statements back:

```python
result = scrape(credentials, start_date="2024-01-01")
```

One request per card per month, so it is slow. DigitalV3 stops serving somewhere around two years
back regardless of what you ask for.

### Session reuse — do this if you run on a schedule

Logging in is the WAF-prone step; the API calls afterwards are not. `result.deviceTrustData` is the
browser session, and handing it back skips the login entirely:

```python
first = scrape(credentials)
session = first.deviceTrustData  # persist this

later = scrape(credentials, device_trust_data=session)  # no login at all
```

A dead session falls back to a normal login, so this never becomes a failure mode. **The session
file holds live authenticated cookies — treat it exactly like the password.** The CLI writes it
`0600`.

## Where it runs

**Amex's WAF is materially more permissive from an Israeli residential IP.** From a datacenter — any
mainstream cloud — expect to be challenged or blocked, with correct credentials and a correct
fingerprint. This was the single biggest factor in getting a reliable scrape, above every browser
setting.

If you are seeing `BLOCKED_BY_WAF`, the IP is the first thing to change, not the code. Run with
`headless=False` to see what the WAF is actually showing you.

## Output

The same structures `israeli-bank-scrapers` produces, under the same names:

| here | there |
| --- | --- |
| `ScraperScrapingResult` | `interface.d.ts` |
| `TransactionsAccount` | `transactions.d.ts` |
| `Transaction` | `transactions.d.ts` |
| `TransactionTypes` / `TransactionStatuses` | `transactions.d.ts` |
| `ScraperErrorTypes` | `scrapers/errors.d.ts` |
| `FutureDebit` | `interface.d.ts` |
| `deviceTrustData` | `interface.d.ts` |

Including the convention that trips people up: **a purchase is negative**, a refund positive, exactly
as `originalAmount: -txn.dealSum` does there. Amex's API reports a purchase as positive; the mapping
negates.

`result.to_dict()` gives a plain JSON-able dict with enums flattened and empty fields dropped, so the
payload matches what the JS library emits.

## Beyond `israeli-bank-scrapers`

Everything above is drop-in compatible. These are **additions** — extra fields on otherwise-identical
structures, so a consumer that ignores them sees exactly the standard shape.

### `Transaction.stableId` — an id you can key a database on

`identifier` is the issuer's voucher number, and **Amex does not assign one until a charge posts**. A
fresh charge has no identifier at all. Key on it directly and every un-posted charge in a scrape
collides on the same empty key — then *changes* key once the voucher lands, which reads as a delete
plus an insert.

`stableId` is the voucher when there is one, otherwise a hash of (card, date, amount, merchant),
prefixed so the two kinds can never be confused. Expect exactly one transition per charge, from
content key to voucher key; de-dup on (date, amount, description) across that boundary if it matters
to you.

### `Transaction.billingMonth` — which statement it landed on

Not derivable from the purchase date. **Amex bills around the 9th**, so a purchase on the 8th and one
on the 10th belong to different statements despite being two days apart. Derive the cycle from the
purchase date and boundary charges land in the wrong month, and a monthly total stops matching the
bill you were actually sent. This field is what the issuer says, not what we inferred — and
`processedDate` is anchored to it.

### `Transaction.isPosted` — whether the issuer has given it an identity

Distinct from `status`. Pending/Completed describes settlement; this describes whether a voucher
number exists yet. An un-posted charge is real and will be billed — it just cannot be keyed on its
voucher, and the merchant may still revise it.

### `result.statementTotals` — check the scrape against the issuer's own number

Amex publishes the official upcoming charge (`futureDebits`) separately from the transactions that
make it up. `statementTotals` sums what was actually fetched, per card and month, so the two can be
compared:

```python
for total in result.statementTotals:
    print(total.accountNumber, total.billingMonth, total.total, total.transactionCount)
```

Equal means nothing was missed. A gap means the statement fetch dropped rows — which is otherwise a
**silent** failure, because a short list of real transactions looks exactly like a correct one.

### `ScraperErrorTypes.BLOCKED_BY_WAF`

Its own type because the remedy is unlike every other failure: nothing about the credentials is
wrong, and retrying from the same IP will keep failing. See *Where it runs*.

### The full statement, not the recent window

Not a field, but the reason the numbers reconcile. DigitalV3 offers a simpler-looking
`GetLatestTransactions`, and it is a trap: it returns a narrow recent window that omits the monthly
portions of installment plans and anything charged early in the cycle. Totals built from it look
reasonable and quietly fail to match the bill. This library reads `GetTransactionsList` — the full
statement — including the `outOfStatementChargeDateVouchers` bucket, which holds charges billed on a
statement but dated outside it.

## Errors

`scrape()` does not raise; a failure comes back as `success=False` with an `errorType`, matching the
reference library.

| type | meaning |
| --- | --- |
| `INVALID_PASSWORD` | still on the login page after submitting — check all three fields |
| `BLOCKED_BY_WAF` | Cloudflare refused the session; change the IP, not the code |
| `TIMEOUT` | the run exceeded its ceiling, usually a challenge page spinning |
| `GENERIC` | an API call returned non-200 or a non-JSON body (usually an expired session) |
| `GENERAL_ERROR` | anything else, including a driver crash |

## Tests

```bash
pytest
```

The mapping — signs, dates, installments, the stable id, statement anchoring — is tested against
recorded API rows with no browser and no network. That is where a mistake is silent, so that is
where the tests are.

## Scope and limits

- **Cards only.** Amex Israel is a card issuer here; there is no bank account to read.
- **`balance` is always `None`.** A card has a statement, not a balance. The field exists for shape
  parity.
- **No OTP path.** Amex Israel's flow uses a permanent password. There is no two-factor step to
  implement, and `TWO_FACTOR_RETRIEVER_MISSING` has no analogue.
- **Hebrew descriptions.** `description` is whatever the merchant registered, usually Hebrew.
- Not affiliated with, endorsed by, or supported by American Express.

## Using this responsibly

This reads **your own account, with your own credentials**, the same data the website shows you when
you log in. That is the only use it is built for. Keep the request volume near what a person would
generate — polling in a loop is both rude and the fastest way to get an account flagged. Your
credentials and the session file stay on your machine; nothing here transmits them anywhere.

## Contributing

Issues and pull requests are welcome. The mapping (`amex_israel/mapping.py`) is deliberately free of
any browser import, so its tests run anywhere with no network, no browser and no Amex account —
`pytest` is enough, and CI runs it on 3.11 and 3.13.

The live scrape cannot be tested in CI, for the reason in *Where it runs*: it needs an Israeli
residential IP and a real account. If you change the browser or login path, say in the PR how you
verified it.

## Licence

MIT — see [LICENSE](LICENSE).
