Metadata-Version: 2.4
Name: bio_scraper
Version: 0.1.0
Summary: Scrape GEO dataset IDs from ExomiRHub and download datasets from NCBI GEO.
Author: conite
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: selenium
Requires-Dist: requests
Requires-Dist: tqdm
Requires-Dist: webdriver-manager
Requires-Dist: tenacity
Requires-Dist: pandas
Dynamic: author
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# biomedical_scraper

A Python module to extract GEO dataset IDs from [ExomiRHub](http://www.biomedical-web.com/exomirhub/download) and download the corresponding datasets from NCBI GEO.

The site is a Vue.js SPA with an ag-Grid table — Selenium is required.

---

## Requirements

- Python 3.10+
- Google Chrome installed

---

## Installation

**From PyPI:**
```bash
pip install bio_scraper
```

**From source:**
```bash
git clone <repo-url>
cd biomedical_scraper
pip install -e .
```

---

## Usage

### As a CLI tool

```bash
# Fetch all GEO IDs, save to CSV, download datasets 0–4
bio-scraper --save-metadata --start 0 --end 5

# Download only the first dataset
bio-scraper --start 0 --end 1

# Run with a visible browser window (useful for debugging)
bio-scraper --no-headless --start 0 --end 1
```

| Flag | Default | Description |
|------|---------|-------------|
| `--start` | `0` | Start index of GEO IDs to download |
| `--end` | `5` | End index (exclusive) |
| `--save-metadata` | off | Save all GEO IDs to `geo_ids.csv` |
| `--headless` | `True` | Run Chrome headlessly |

---

### As a Python module

#### Fetch all GEO IDs

```python
from biomedical_scraper import BiomedicalScraper

with BiomedicalScraper(headless=True) as scraper:
    geo_ids = scraper.get_geo_ids()
    print(geo_ids)
# ['GSE144777', 'GSE115572', 'GSE115114', ...]
```

#### Get a download link for a specific dataset

```python
from biomedical_scraper import BiomedicalScraper

with BiomedicalScraper() as scraper:
    url = scraper.get_download_link("GSE144777")
    print(url)
# https://ftp.ncbi.nlm.nih.gov/geo/series/GSE144nnn/GSE144777/...
```

#### Fetch IDs + download datasets

```python
from biomedical_scraper import BiomedicalScraper
from biomedical_scraper.downloader import GEODownloader
from biomedical_scraper.utils import select
from biomedical_scraper.metadata import save_metadata

with BiomedicalScraper(headless=True) as scraper:
    geo_ids = scraper.get_geo_ids()       # all 191 IDs
    save_metadata(geo_ids)                # writes geo_ids.csv

    subset = select(geo_ids, start=0, end=5)
    downloader = GEODownloader()          # saves to data/raw/geo/

    for geo_id in subset:
        url = scraper.get_download_link(geo_id)
        if url:
            downloader.download_from_url(url, geo_id)
```

#### Save metadata only (no download)

```python
from biomedical_scraper import BiomedicalScraper
from biomedical_scraper.metadata import save_metadata

with BiomedicalScraper() as scraper:
    geo_ids = scraper.get_geo_ids()

save_metadata(geo_ids, path="my_geo_ids.csv")
```

---

## Output

| File | Description |
|------|-------------|
| `geo_ids.csv` | All GEO IDs scraped from ExomiRHub |
| `data/raw/geo/<ID>.tar` | Downloaded dataset archives |
| `debug_screenshot.png` | Auto-saved on timeout (debugging only) |
| `debug_page.html` | Auto-saved on timeout (debugging only) |

---

## Project structure

```
biomedical_scraper/
├── biomedical_scraper/
│   ├── scraper.py      # Selenium scraper (get_geo_ids, get_download_link)
│   ├── downloader.py   # Resumable file downloader
│   ├── cli.py          # bio-scraper entry point
│   ├── config.py       # URLs, timeouts, paths
│   ├── metadata.py     # CSV export
│   └── utils.py        # ID selection helpers
└── setup.py
```
