Metadata-Version: 2.3
Name: histotuner
Version: 0.4.9
Summary: Add your description here
Author: Ajit Johnson Nirmal
Author-email: Ajit Johnson Nirmal <ajitjohnson.n@gmail.com>
Requires-Dist: anndata>=0.12.2
Requires-Dist: cellpose>=4.0.6
Requires-Dist: dask>=2024.11.2
Requires-Dist: geopandas>=1.1.1
Requires-Dist: leidenalg>=0.12.0
Requires-Dist: magicgui>=0.10.1
Requires-Dist: matplotlib>=3.10.6
Requires-Dist: napari>=0.7.0
Requires-Dist: numpy>=2.3.3
Requires-Dist: opencv-python>=4.11.0.86
Requires-Dist: openslide-bin>=4.0.0.8
Requires-Dist: openslide-python>=1.4.2
Requires-Dist: pandas>=2.3.3
Requires-Dist: pillow>=11.3.0
Requires-Dist: pip>=25.2
Requires-Dist: psutil>=7.1.0
Requires-Dist: pyqt6>=6.11.0
Requires-Dist: python-igraph>=1.0.0
Requires-Dist: pyyaml>=6.0.3
Requires-Dist: scikit-learn>=1.7.2
Requires-Dist: shapely>=2.1.2
Requires-Dist: spatialdata>=0.8.0
Requires-Dist: tifffile>=2025.9.30
Requires-Dist: timm>=1.0.20
Requires-Dist: torch>=2.11
Requires-Dist: torchvision>=0.26
Requires-Dist: tqdm>=4.67.1
Requires-Dist: transformers>=4.57.1
Requires-Dist: umap-learn>=0.5.7
Requires-Dist: wandb>=0.22.2
Requires-Dist: zarr>=3
Requires-Python: >=3.12
Description-Content-Type: text/markdown

## histotuner

### Linux And O2 Install

PyPI does not allow published package metadata to hard-code direct CPU wheel
URLs for PyTorch, so `histotuner` cannot enforce a CPU-only Linux PyTorch build
at package metadata level. The clean pattern is to install the PyTorch flavor
you want first, then install `histotuner` without re-resolving dependencies.

Recommended default CPU install on Linux:

```bash
pip install torch==2.11.0 torchvision==0.26.0 --index-url https://download.pytorch.org/whl/cpu
pip install --no-deps histotuner
```

# upload to O2
```
python upload_melanoma_to_o2.py \
  --csv "test.csv" \
  --destination /n/scratch/users/a/ajn16/melanoma_uploads \
  --offset 0 \
  --report-csv upload_report.csv \
  --dry-run
```

# Run full pipeline in O2 (new)

```
python submit_full_pipeline_jobs.py \
  --image-root /n/scratch/users/a/ajn16/melanoma_cohort \
  --output-dir /n/scratch/users/a/ajn16/full_pipeline_jobs \
  --template-config /n/scratch/users/a/ajn16/embedder.yaml \
  --conda-env-embed histotuner_embed \
  --conda-env-analysis histotuner \
  --cpus 16 \
  --time 0-06:00 \
  --partition gpu_quad \
  --gres gpu:l40s:1 \
  --mem 128G \
  --overwrite-existing-tables \
  --continue-on-error \
  --run-mode api \
  --submit \
  --limit 20

  ```

Recommended HMS O2 / Linux GPU install for CUDA `12.8`:

```bash

conda create -n histotuner python=3.13 -y
conda activate histotuner
conda config --env --set channel_priority strict

conda install -c rapidsai -c conda-forge -c nvidia -c nodefaults \
  "cuda-version=12.8" "cuda-bindings>=12.8,<13" "cuda-python>=12.8,<13" \
  "cuda-core>=0.3,<0.4" "cuda-nvrtc=12.*" "cuda-nvrtc-dev=12.*" \
  "cuda-cudart=12.*" "cuda-cudart-dev=12.*" \
  cupy cuml=26.06 cudf=26.06 cugraph=26.06 rmm pylibraft timm transformers \
  numpy scipy=1.16.2 scikit-learn numba umap-learn \
  anndata dask geopandas leidenalg matplotlib pandas pillow psutil pyqt pyyaml \
  python-igraph shapely spatialdata tifffile tqdm zarr \
  -y

  

pip install histotuner --no-deps --upgrade

python -m pip install torch==2.11.0 torchvision==0.26.0 --index-url https://download.pytorch.org/whl/cu128

pip install spatialdata==0.8.0

python -c "import torch; print('GPU Available:', torch.cuda.is_available())"


# multi embeder
module load gcc/14.2.0
conda activate histotuner-embed
python embed/scripts/run_batch_he_embedder_multimodel.py \
  --image-folder embed/ \
  --template-config embed/scripts/he_embedder.example_multimodel.yaml

python o2/batch_he_embedder_multimodel/submit_batch_he_embedder_multimodel_jobs.py \
  --image-root /path/on/o2/embed \
  --output-dir /path/on/o2/embed_jobs \
  --template-config embed/scripts/he_embedder.example_multimodel.yaml
  --time 0-10:00
  --mem 256G
  --submit


# run cell_sep scoring on O2

python o2/melanocyte_dbscan/submit_melanocyte_dbscan_jobs.py \
  --root-dir /n/scratch/users/a/ajn16/test \
  --output-dir /n/scratch/users/a/ajn16/test/jobs_reader_check \
  --limit 1 \
  --submit


python -u .\o2\melanocyte_dbscan\run_token_umap_melanocyte_dbscan.py `
  "C:\Users\aj\Downloads\test" `
  --recursive `
  --continue-on-error `
  --summary-json "C:\Users\aj\Downloads\test\melanocyte_dbscan_summary.json"


  # sort by slrum status
  python o2/sort_folders_by_slurm_status.py \
  --manifest /path/to/jobs/submission_manifest.csv \
  --source-root /path/to/sample/root \
  --report-csv job_status_folder_sort_report.csv \
  --dry-run

# remove tables
python -m histotuner.delete_tables --image-folder "C:\Users\aj\Downloads\test" --tables hoptimus1_tokens,uni2h_tokens 

python - <<'PY'
mods = ["histotuner", "spatialdata", "anndata", "zarr", "cupy", "cuml", "cudf", "cugraph", "umap", "torch"]
for m in mods:
    try:
        mod = __import__(m)
        print(m, "OK", getattr(mod, "__version__", "unknown"))
    except Exception as e:
        print(m, "FAIL", type(e).__name__, e)

import cupy, torch, histotuner as ht
print("cupy devices:", cupy.cuda.runtime.getDeviceCount())
print("torch cuda:", torch.cuda.is_available(), torch.cuda.device_count())
print(ht.umap_backend_status())
PY



```

To use a different GPU PyTorch build, replace the PyTorch index, for example
`cu126` or `cu128`, before installing `histotuner --no-deps`.

If you do use a plain `pip install histotuner`, the exact PyTorch wheel flavor
chosen on Linux depends on pip resolution and the available package indexes in
that environment.

### GPU UMAP and Clustering on Linux

`histotuner` installs CPU UMAP support through `umap-learn`. GPU UMAP is
optional because it depends on the local CUDA driver/toolkit stack and should be
installed separately from the package dependencies in `pyproject.toml`.

GPU UMAP uses RAPIDS cuML when both `cuml` and `cupy` are available in the
active Python environment and a CUDA-capable NVIDIA GPU is visible.

The same optional GPU stack is also used by native clustering:

- `ht.umap(...)`
- `ht.leiden(...)`
- `ht.dbscan(...)`
- `histotuner-leiden`
- `histotuner-dbscan`

Check the active environment from Python:

```python
import histotuner as ht

ht.umap_backend_status()
```

Expected GPU-ready output has `gpu_available: True`, with both `gpu_cuml` and
`gpu_cupy` set to `True`.

Recommended install path on Linux is to create a RAPIDS-compatible environment
with the official RAPIDS install selector:

https://docs.rapids.ai/install/

To request an interactive O2 GPU session for testing:

```bash
srun --pty -p gpu_quad --gres=gpu:l40s:1 -c 8 --mem=64G --time=0-03:00 /bin/bash
```

Inside the allocation, activate the same environment used by batch jobs and
verify both the Slurm GPU allocation and the Python RAPIDS stack:

```bash
module purge
module load gcc/14.2.0
source "$HOME/miniconda3/etc/profile.d/conda.sh"
conda activate histotuner
nvidia-smi
python - <<'PY'
import histotuner as ht
print(ht.umap_backend_status())
PY
```

If `nvidia-smi` sees a GPU but `ht.umap_backend_status()` reports
`gpu_cupy=False`, `gpu_cuml=False`, or `gpu_device_count=0`, the conda
environment is not GPU-ready. Inspect the CUDA/RAPIDS stack:

```bash
conda list | egrep 'cuda-bindings|cuda-python|cuda-core|cuda-cudart|cuda-version|cuml|cudf|cupy'
python -m pip show cuda-bindings cuda-python cuda-core
```

A common failure is a pip-installed CUDA 13 `cuda-bindings` package mixed with
conda CUDA 12/RAPIDS packages, for example `cuda-bindings 13.x` with
`cuda-python 12.9` and RAPIDS `cuda12`. Fix it by removing the pip binding and
reinstalling CUDA bindings from conda:

```bash
python -m pip uninstall -y cuda-bindings
conda install -c conda-forge -c nvidia \
  "cuda-bindings>=12.9,<13" "cuda-python=12.9.*" \
  --force-reinstall
```

Then re-test:

```bash
python - <<'PY'
import cupy
print("cupy devices", cupy.cuda.runtime.getDeviceCount())
import cuml
print("cuml", cuml.__version__)
import histotuner as ht
print(ht.umap_backend_status())
PY
```

Keep CUDA/RAPIDS packages conda-managed in this environment; avoid installing
`cuda-*` packages with pip after RAPIDS is installed.

For pip-based RAPIDS installs, choose wheels matching the installed CUDA major
version and use the RAPIDS selector for the exact command:

https://docs.rapids.ai/install/

Then run UMAP with:

```python
ht.umap(
    sdata=zarr_path,
    tableKeys=["mstar_tokens", "virchow2_tokens"],
    sample_n=25000,
    prefer_gpu="auto",  # uses GPU if RAPIDS is available, otherwise CPU
)
```

To require GPU and fail loudly if RAPIDS is not available:

```python
ht.umap(
    sdata=zarr_path,
    tableKeys=["mstar_tokens", "virchow2_tokens"],
    sample_n=25000,
    prefer_gpu="gpu",
)
```

Native clustering uses the same `prefer_gpu` switch:

```python
ht.leiden(
    sdata=zarr_path,
    tableKeys="tokens",
    obsm_key="X_umap",
    prefer_gpu="auto",
    target_col="leiden",
)
```

```python
ht.dbscan(
    sdata=zarr_path,
    tableKeys="tokens",
    obsm_key="X_umap",
    prefer_gpu="gpu",
    target_col="dbscan",
)
```

CLI examples:

```bash
histotuner-leiden /path/to/sample.zarr \
  --tables tokens \
  --obsm-key X_umap \
  --prefer-gpu auto \
  --target-col leiden
```

```bash
histotuner-dbscan /path/to/sample.zarr \
  --tables tokens \
  --obsm-key X_umap \
  --prefer-gpu gpu \
  --target-col dbscan
```

Notes:

- RAPIDS requires Linux or WSL2; native Windows Python environments generally
  cannot install/use cuML, cuGraph, or cuDF directly.
- GPU UMAP uses `cuml` plus `cupy`.
- GPU Leiden uses `cudf`, `cugraph`, `cuml`, and `cupy`.
- GPU DBSCAN uses `cuml` plus `cupy`.
- CUDA package suffixes must match the CUDA toolkit/driver stack in the
  environment. If installation fails, generate a fresh command from the RAPIDS
  selector for the specific Linux, Python, CUDA, and RAPIDS versions.

### Supported token-extraction backends

`histotuner` can append multiple model-specific token tables into the same
SpatialData Zarr while keeping shared geometry layers model-agnostic.

Currently supported token extractors:

- `hf-hub:bioptimus/H-optimus-1`
- `hf-hub:MahmoodLab/UNI2-h`
- `hf-hub:paige-ai/Virchow2`
- `hf-hub:Wangyh/mSTAR`
- `hf-hub:prov-gigapath/prov-gigapath`
- `owkin/phikon-v2`
- `MahmoodLab/conchv1_5`
- `WenchuanZhang/Patho-CLIP-L`
- `majiabo/GPFM`
- `kaiko-ai/vitl14`
- `xiangjx/musk`

### Token-grid semantics

All currently supported models export a unified `14x14` token grid so token
tables can be compared directly across models.

- `phikon-v2` exports a native `14x14` patch-token grid.
- `hf-hub:bioptimus/H-optimus-1`, `hf-hub:Wangyh/mSTAR`, and
  `hf-hub:prov-gigapath/prov-gigapath` export native `14x14` grids.
- `hf-hub:MahmoodLab/UNI2-h` and `hf-hub:paige-ai/Virchow2` have native
  `16x16` patch-token grids after special tokens are stripped, and `histotuner`
  adaptively average-pools them to `14x14`.
- `conchv1_5` is special:
  - the native vision encoder runs at `448x448` with `patch16`
  - that produces a native `28x28` patch-token grid
  - `histotuner` average-pools each non-overlapping `2x2` token neighborhood
    to export a compatibility `14x14` token grid
- `Patho-CLIP-L` is also special:
  - the native CLIP-L/14 vision encoder produces a `24x24` patch-token grid at
    `336x336` input resolution
  - `histotuner` adaptively average-pools that native `24x24` grid to export a
    compatibility `14x14` token grid
- `GPFM` is also special:
  - the native DINOv2 ViT-L/14 encoder produces a `16x16` patch-token grid at
    `224x224` input resolution
  - `histotuner` adaptively average-pools that native `16x16` grid to export a
    compatibility `14x14` token grid
- `kaiko-ai/vitl14` is also special:
  - the native Kaiko ViT-L/14 encoder produces a `16x16` patch-token grid at
    `224x224` input resolution
  - `histotuner` uses the Kaiko preprocessing defaults (`mean=std=0.5`) and
    adaptively average-pools that native `16x16` grid to export a compatibility
    `14x14` token grid
- `xiangjx/musk` is also special:
  - the native MUSK patch16 vision encoder produces a `24x24` patch-token grid
    at `384x384` input resolution
  - `histotuner` uses the MUSK preprocessing defaults (`mean=std=0.5`) and
    adaptively average-pools that native `24x24` grid to export a compatibility
    `14x14` token grid
  - MUSK is gated on Hugging Face and requires the optional official `musk`
    package

That pooling choice is deliberate so downstream single-cell workflows can
consume every supported model through the same `14x14` token layout. For the
pooled models, this is a compatibility semantic rather than the model's native
tokenization:

- `UNI2-h` and `Virchow2`: pooled from native `16x16`
- `conchv1_5`: pooled from native `28x28`
- `Patho-CLIP-L`: pooled from native `24x24`
- `GPFM`: pooled from native `16x16`
- `kaiko-ai/vitl14`: pooled from native `16x16`
- `xiangjx/musk`: pooled from native `24x24`

### Not yet supported for token extraction

- none from the current requested set

### O2 batch job generation

To generate one `embedder.yaml` and one `embed_cluster.sh` per sample folder on
O2:

```bash
python generate_o2_jobs.py \
  --root-dir /n/scratch/users/a/ajn16/histotuner/full \
  --template-yaml embedder.yaml \
  --template-shell embed_cluster.sh \
  --output-dir /n/scratch/users/a/ajn16/histotuner/generated_jobs

  python generate_o2_jobs.py \
  --root-dir /n/scratch/users/a/ajn16/histotuner/heonly \
  --template-yaml embedder_HEonly.yaml \
  --template-shell embed_cluster_HEonly.sh \
  --output-dir /n/scratch/users/a/ajn16/histotuner/generated_jobs


```

To preview the `sbatch` submissions for the generated job scripts:

```bash
python submit_generated_jobs.py \
  --generated-dir /n/scratch/users/a/ajn16/histotuner/generated_jobs \
  --dry-run
```

### Melanocyte UMAP/DBSCAN token pipeline

The scripts in `o2/melanocyte_dbscan/` find SpatialData `.zarr` stores under a
folder, map tokens to cells, compute a global UMAP for native token tables,
map broad phenotype labels onto token tables, compute a second melanocyte-only
UMAP, compute cell-type separability from `phenotype_broad`, and run DBSCAN on
`X_umap_melanocytes` for tokens where
`phenotype_broad == "Melanocytes"`. They then generate thumbnail PDFs for
`dbscan_melanocytes_umap` using an HE image auto-detected beside each `.zarr`,
and save UMAP plots for each token table/model:

- `X_umap/phenotype_broad.png`, excluding `-1` and `0`
- `X_umap_melanocytes/phenotype_broad.png`, subset to `Melanocytes`
- `X_umap_melanocytes/dbscan_melanocytes_umap.png`, excluding `-1` and `nan`

Run a local dry-run first:

```powershell
python .\o2\melanocyte_dbscan\run_token_umap_melanocyte_dbscan.py `
  "C:\Users\aj\Downloads\test" `
  --recursive `
  --dry-run
```

Run the local pipeline and write a summary:

```powershell
python -u .\o2\melanocyte_dbscan\run_token_umap_melanocyte_dbscan.py `
  "C:\Users\aj\Downloads\test" `
  --recursive `
  --continue-on-error `
  --summary-json "C:\Users\aj\Downloads\test\melanocyte_dbscan_summary.json"
```

The script is intentionally a thin wrapper around histotuner package APIs. It
uses native histotuner readers and writers directly and does not monkey-patch
SpatialData, AnnData, UMAP, DBSCAN, or plotting internals. O2/runtime errors
should be fixed in the package rather than patched inside this wrapper.

The script uses native histotuner token-table selection. By default,
`tokenCellMapper` and `phenotypeCellMapper` auto-detect token tables, while UMAP
and DBSCAN use the native `tokens` selector. Melanocyte DBSCAN uses
`--dbscan-min-samples 100` by default. Pass `--no-thumbnail-pdfs` to skip PDF
generation, or `--thumbnail-image-path /path/to/image.ome.tiff` to provide an
explicit image for a single-sample run.
Pass `--no-umap-plots` to skip the saved UMAP plots.
The pipeline runs `ht.repairSpatialDataTableRegistry(...)` before any per-zarr
pipeline step, after global UMAP writes, after melanocyte UMAP writes, and after
DBSCAN writes so on-disk tables are re-registered before downstream plotting/PDF steps. The
repair function now detects Zarr v2 stores, writes Zarr v3 metadata with
`zarr.metadata.migrate_v3.migrate_v2_to_v3(...)`, and then applies the Zarr v3 table
registry repair. Existing Zarr v3 stores skip migration and go straight to the
v3 repair path.
Cell-type separability is enabled by default with the batch-notebook defaults:
`phenotype_broad`, `max_tokens_per_cell_type=10000`,
`max_comparison_tokens_per_cell_type=10000`, `initial_sample_size=200`,
`bootstrap_repeats=100`, `target_relative_ci_width=0.05`, and
`distance_metric="cosine"`. Per-sample output is written to
`<sample>/<sample>_cell_type_separability.csv`. Pass
`--no-cell-type-separability` to skip this step.

### O2 parallel melanocyte DBSCAN jobs

If you are not pulling the full repo on O2, upload both files from
`o2/melanocyte_dbscan/` together:

- `run_token_umap_melanocyte_dbscan.py`
- `submit_melanocyte_dbscan_jobs.py`

The submitter copies the uploaded pipeline script into
`<output-dir>/scripts/run_token_umap_melanocyte_dbscan.py` and points every
generated Slurm job at that staged copy. This avoids accidentally running an
older script from a previous upload or from a different folder.

On O2, generate one Slurm script per sample/zarr without submitting:

```bash
python o2/melanocyte_dbscan/submit_melanocyte_dbscan_jobs.py \
  --root-dir /n/scratch/users/a/ajn16/he_embed \
  --output-dir /n/scratch/users/a/ajn16/melanocyte_dbscan_jobs
```

Submit a single test job:

```bash
python o2/melanocyte_dbscan/submit_melanocyte_dbscan_jobs.py \
  --root-dir /n/scratch/users/a/ajn16/he_embed \
  --output-dir /n/scratch/users/a/ajn16/melanocyte_dbscan_jobs_test \
  --limit 1 \
  --submit
```

Submit all jobs:

```bash
python o2/melanocyte_dbscan/submit_melanocyte_dbscan_jobs.py \
  --root-dir /n/scratch/users/a/ajn16/he_embed \
  --output-dir /n/scratch/users/a/ajn16/melanocyte_dbscan_jobs \
  --submit
```

Submit all jobs with a smaller resource request:

```bash
python o2/melanocyte_dbscan/submit_melanocyte_dbscan_jobs.py \
  --root-dir /n/scratch/users/a/ajn16/he_embed \
  --output-dir /n/scratch/users/a/ajn16/melanocyte_dbscan_jobs \
  --cpus 8 \
  --mem 64G \
  --time 0-03:00 \
  --submit
```

Replace `/n/scratch/users/a/ajn16/he_embed` with the O2 path containing the
sample folders or `.zarr` stores. Each submitted job runs the melanocyte DBSCAN
pipeline on one sample folder, so samples run in parallel through Slurm.
The `--output-dir` folder stores generated Slurm scripts, logs, per-job summary
JSON files, and `submission_manifest.csv`. The main analysis outputs are written
beside each sample `.zarr` under `--root-dir`, and each `.zarr` is updated
in-place.
After submitting, the job log should include:

```text
[pipeline] Version: melanocyte_dbscan_native_package_v1_2026_07_24
```

If those lines are absent, the job is still using an old pipeline script.
Regenerate the jobs by rerunning `submit_melanocyte_dbscan_jobs.py`; already
submitted Slurm jobs will not change retroactively.
Thumbnail PDFs are generated by default in each sample job; pass
`--no-thumbnail-pdfs` to
`o2/melanocyte_dbscan/submit_melanocyte_dbscan_jobs.py` to disable them. UMAP
plots are also generated by default; pass `--no-umap-plots` to disable them.
Cell-type separability is generated by default; pass
`--no-cell-type-separability` to disable it. The submission manifest is written to
`/n/scratch/users/a/ajn16/melanocyte_dbscan_jobs/submission_manifest.csv`.
