Metadata-Version: 2.4
Name: medgaze
Version: 0.1.0
Summary: Gaze heads for medical vision-language models: discover the attention heads that look where you point, steer them, and calibrate them on a few labelled cases.
Author: GazeHeads-Medical-VLMs authors
Project-URL: Homepage, https://github.com/aj-das-research/GazeHeads-Medical-VLMs
Project-URL: Source, https://github.com/aj-das-research/GazeHeads-Medical-VLMs
Keywords: vision-language,medical imaging,attention,interpretability,steering
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Intended Audience :: Science/Research
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: torch>=2.1
Requires-Dist: transformers>=4.57
Requires-Dist: accelerate>=1.0
Requires-Dist: numpy
Requires-Dist: pillow
Provides-Extra: metrics
Requires-Dist: sacrebleu; extra == "metrics"
Requires-Dist: rouge-score; extra == "metrics"
Requires-Dist: pycocoevalcap; extra == "metrics"
Requires-Dist: bert-score; extra == "metrics"
Provides-Extra: seg
Requires-Dist: torchxrayvision; extra == "seg"
Requires-Dist: scikit-image; extra == "seg"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"

# medgaze

Gaze heads for medical vision-language models: **discover** the attention heads
that look where a question points, **steer** them at decode time, and
**calibrate** their gains on a few labelled cases. The model stays frozen; a
saved "gaze file" is a few kilobytes.

Works with any model an adapter supports: Qwen2.5-VL and Qwen3-VL families and
their medical tunes (Lingshu, MedVLThinker, HuatuoGPT-Vision, QoQ-Med, MediX-R1,
Chiron-o1, Citrus-V, ...), Gemma 3 / MedGemma, and LLaVA-style fixed-grid models.
Models must be loaded with eager attention (the adapters do this) so attention
can be read.

```bash
pip install medgaze                      # or: pip install -e . from the repo root
```

## The short version: adapt a model in two lines

```python
from medgaze import GazeModel

gm = GazeModel.from_pretrained("Qwen/Qwen2.5-VL-7B-Instruct", gaze="auto")
gm.generate(image, "Is the liver enlarged? Answer with yes or no.")   # no region needed
```

`gaze="auto"` attaches the released gaze adapter for that model: a decode-only rank-8 update on the
query rows of its 100 discovered gaze heads, under a megabyte, trained with the answer loss plus a
mask-supervised attention term. With it attached the model looks at the anatomy the question names
without being pointed at it, and answers closed questions better.

`gm.describe()` returns what was *measured* for this model and where the adapter does not help:

```python
{'model_id': 'Qwen/Qwen2.5-VL-7B-Instruct', 'n_parameters': 847872, 'n_heads': 100,
 'measured': {'yesno': {'frozen': 0.671, 'adapter': 0.914, 'delta_points': 24.3, 'n_questions': 210},
              'organ-choice': {'frozen': 0.633, 'adapter': 0.733, 'delta_points': 10.0},
              'reading': {'attention_on_named_organ': {'off': 0.061, 'on': 0.226},
                          'gaze_heads_peaking_inside': {'off': 0.04, 'on': 0.28}}},
 'caveats': ["Trained on SLAKE closed questions: the accuracy gain is in-domain and does not
              transfer to a different corpus (no effect on the VQA-RAD test split).",
             "The accuracy gain alone does not require the discovered heads ...",
             "Organ-choice accuracy falls for models that are already strong on it ..."]}
```

Read the caveats before using it: the gain is in-domain, and for models already strong on
organ-choice the adapter can cost accuracy there. What it does in every model tested is change
*where* the model looks.

The adapters live in `release/gaze-adapters/` in this repository (set `$GAZEMED_ADAPTERS` to use a
copy elsewhere); `release/gaze-adapters/index.json` lists every model, its checksum and its numbers.

## Three calls

```python
from medgaze import GazeModel, CalibrationItem

gm = GazeModel.from_pretrained("Qwen/Qwen2.5-VL-7B-Instruct")

# 1. discover: images with >= 2 named regions (masks from any segmenter, or boxes)
gm.discover([(image, {"left lung": mask_l, "right lung": mask_r, "heart": mask_h}), ...],
            top_k=100)

# 2. (optional) calibrate one gain per head on a handful of labelled cases
items = [CalibrationItem(image, "Is the liver enlarged? Answer with yes or no.",
                         region=(x0, y0, x1, y1), positive="No", negatives=("Yes",)), ...]
gm.fit(items, steps=40)

# 3. answer, steered to a region (mask or pixel box)
gm.generate(image, "Is the heart enlarged? Answer with yes or no.", region=mask_h)
gm.score(image, "Describe the finding.", "cardiomegaly", region=mask_h)   # log P(phrase)

gm.save("qwen25-7b.gaze.json")      # heads + gains; load with heads="qwen25-7b.gaze.json"
```

A trained GazeLoRA adapter (Stage 2, `scripts/46_train_gaze_lora.py`) loads with
`gm.attach_lora("adapter.pt")`; it then steers the gaze heads by itself on every
`generate`/`score` call, with no region required.

## Command line

```bash
medgaze models                       # the model families this build supports and the adapters it can find
medgaze describe --model lingshu-medical-mllm/Lingshu-7B     # what the released adapter was measured to do
medgaze discover --model <id> --items items.jsonl --out heads.gaze.json
medgaze fit      --model <id> --items cal.jsonl --heads heads.gaze.json --out gains.gaze.json
medgaze answer   --model <id> --heads gains.gaze.json --gaze auto \
                 --image cxr.png --prompt "Is the heart enlarged?" --region 120,80,300,260
```

`models` and `describe` answer questions about the package rather than about a checkpoint, so they
run without torch installed and without loading any weights. Every other subcommand takes `--gaze`
(a path to an adapter, or `auto` for the released one for that model id). Item files are JSON lines;
`medgaze discover --help` prints their shape.

## Install and test

```bash
pip install -e .                       # from the repository root
pip install -e '.[dev]' && pytest -q   # 20 CPU-only tests: no weights, no GPU
```

The tests cover the lazy import boundary (importing `medgaze` must not pull in torch), the CLI,
released-adapter resolution including the short-name fallback, the gaze file format, token-to-region
assignment, and the geometry of the two hard grounding controls.

## What the intervention is

For the top-`k` gaze heads, an additive bias is put on the attention logits at
every decode step: `+g_h` on the image tokens inside the region, `-g_h` on the
other image tokens; the prompt is never re-encoded. The first generated token is
steered too (`steered_generate`), which plain `model.generate` cannot do with
decode-only hooks. `fit` learns the per-head gains `g_h` by gradient descent on
the candidate answers' first-token cross-entropy; at initialisation every gain
equals the fixed bias, so the learned model is a strict generalisation of the
fixed one.

## What it does, measured (pilot, SLAKE, held-out images)

- Calibrated gains: +2.2 points closed-question accuracy pooled over four models
  (p = 0.003) against +0.4 for the same heads at a fixed bias; the fit turns about
  a third of the heads off. 1–3 minutes per model.
- Gaze adapter (`GazeLoRA`, 0.5–0.9 M parameters, decode-only, keys/values and
  the prompt cache untouched): +5.1 points with no region given (p < 1e-4);
  attention of the gaze heads on the named organ at the answer step rises 3–5×
  and the answer's dependence on the organ rises. Whole-image attention
  amplification (PAI-style) on the same models hurts by 6.5 points.
  Details, ablations and caveats: the paper's "Learning to look" section.

## Beyond VQA

`medgaze.imagenome_caption` turns Chest ImaGenome into two generation tasks that use the same
mechanism: region-conditioned captioning (each anatomical region with the verbatim report sentences
describing it) and report generation (each gold study's findings section). Both read data you
already hold under `data/raw/`; neither ships text with the package. `scripts/63_region_caption.py`
and `scripts/64_report_generation.py` run them.

Both corpora are credentialed MIMIC-CXR derivatives. The loaders return text; nothing in this
repository writes a reference sentence outside `data/raw/`, and none of it may go to an external
service.

## Status

Research code accompanying the GazeHeads paper; the API is small and stable but
the package is pre-1.0. See the repository for the experiments and the paper.
