Metadata-Version: 2.4
Name: pooled-job-scraper
Version: 0.1.3
Summary: Generic job listings scraper with baseline dedupe and integration with the open jobpool
Author: lramos0
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown

# Pooled Job Scraper

`pooled-job-scraper` is published on PyPI and provides a CLI for scraping business careers pages, extracting listings, and generating a unique delta against the April 2026 baseline dataset.

PyPI package:

- https://pypi.org/project/pooled-job-scraper/

## Platform Links

- [mewannajob.com](https://www.mewannajob.com/)
- [jobpool.live](https://jobpool.live/)

### How They Fit Together

- `jobpool.live` is the open data pool and hydration surface.  
  Use this scraper to discover and normalize job listings, then review unique delta rows before promoting data into the pool workflow.
- `mewannajob.com` is the consumer-facing experience for browsing and using listings data.  
  Data prepared through the pool process ultimately supports downstream job discovery use cases there.

### Documentation Surfaces

- Public pool context and hydration navigation: [jobpool.live](https://jobpool.live/)
- Hydration docs in this repository: `pool/hydration/docs/`
- Scraper implementation in this repository: `scripts/generic_job_listings_scraper.py`

## Install

### Windows

```powershell
py -m pip install --upgrade pooled-job-scraper
```

### WSL / Linux / macOS

```bash
python3 -m pip install --upgrade pooled-job-scraper
```

## Usage

```bash
job-scraper \
  --business-url https://mossyhonda.hireology.careers/ \
  --company-name "Mossy Honda" \
  --output output/mossy-scraped.csv \
  --unique-output output/mossy-unique.csv
```

## Live Progress + Rate Controls

The CLI now shows a live status bar with:

- percent complete
- ETA
- observed request rate (req/s)
- derived safe request rate from observed site limits (`429`, `Retry-After`, and `X-RateLimit-*` headers)

During runs, a persistent prompt stays active until completion:

```text
rate-control>
```

Supported prompt commands:

- `rate <rps>` (example: `rate 1.2`)
- `delay <seconds>` (example: `delay 0.8`)
- `auto` (return to adaptive pacing)
- `status` (show current derived limits/rate)
- `help`

Disable the interactive prompt when needed:

```bash
job-scraper \
  --business-url https://mossyhonda.hireology.careers/ \
  --company-name "Mossy Honda" \
  --no-control-prompt \
  --output output/mossy-scraped.csv \
  --unique-output output/mossy-unique.csv
```

## Field Enrichment + Limits

The scraper now enriches records when source pages are sparse:

- Derives `job_summary` when absent.
- Derives `job_posted_date` from available text/URL patterns, with ingest date fallback.
- Derives `job_industries` from curated company+industry hints, baseline company patterns, and keyword hints.
- Applies sensible per-column word caps for listing-style data quality.

## Default Cache Behavior

By default, unique rows are posted to:

- `https://jobpool.live/api/scrape-cache`

```bash
job-scraper \
  --business-url https://mossyhonda.hireology.careers/ \
  --company-name "Mossy Honda" \
  --output output/mossy-scraped.csv \
  --unique-output output/mossy-unique.csv
```

The scraper infers `user_name` from local environment or git config and sends:

- `user_name`
- `request_timestamp`
- `source_business_urls`
- `listings` (standard listing fields plus any additional discovered fields)

## Disable Cache Submission

Use the only cache-related flag when you need to skip cache persistence:

```bash
job-scraper \
  --business-url https://mossyhonda.hireology.careers/ \
  --company-name "Mossy Honda" \
  --disable-cache \
  --output output/mossy-scraped.csv \
  --unique-output output/mossy-unique.csv
```

## Cache API

- `POST /api/scrape-cache` stores a scrape request payload.
- `GET /api/scrape-cache?limit=25&user_name=<name>` returns recent cached submissions.
- `GET /api/scrape-cache?leaderboard=1&leaderboard_limit=20` returns GitHub user leaderboard data with `preprod_records` and `prod_records`.

## Publishing Flow

Publishing is automated through:

- `.github/workflows/publish-pypi.yml`

Behavior:

- Triggers on push/merge to `main`.
- Builds distributions from `pyproject.toml`.
- Checks whether the current version already exists on PyPI.
- Publishes only when the version is new.
- Skips cleanly when that version already exists.

To release a new version:

1. Bump `project.version` in `pyproject.toml`.
2. Merge to `main`.
3. Wait for the publish workflow to complete.
