Metadata-Version: 2.4
Name: xcheck
Version: 0.1.2
Summary: Python package for 'XCheck: A Consistency-based validation framework of englacial layers annotations' across consecutive and intersecting radargram pairs using dot-product similarity.
Author: Muhammad Behroze Hassan, Bayu Adhi Tama, Sanjay Purushotham, Vandana P. Janeja
Maintainer-email: Muhammad Behroze Hassan <mhassan4@umbc.edu>
License-Expression: MIT
Project-URL: Paper, https://doi.org/10.1109/ICDMW69685.2025.00015
Keywords: radargram,englacial layers,ice sheet,radar,annotation validation,glaciology
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Physics
Classifier: Topic :: Scientific/Engineering :: Image Processing
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Requires-Dist: numpy
Requires-Dist: matplotlib
Requires-Dist: scipy
Requires-Dist: pandas
Requires-Dist: cartopy

# XCheck: A Consistency-Based Validation Framework for Englacial Layer Annotations

This python package repository provides the implementation for **"XCheck: A Consistency-Based Validation Framework for Englacial Layer Annotations"** ([Paper](https://ieeexplore.ieee.org/document/11415920)), accepted for publication at IEEE ICDMW 2025 proceedings. We introduce a dot-product-based layer consistency framework that validates annotated englacial layers between two radargrams observing the same ice volume — either two **consecutive** flight segments (overlap detected via GPS time) or two **intersecting** flight paths (crossing point supplied via a pre-computed CSV).

---

## Introduction

Ice-penetrating radar surveys produce radargrams — 2-D depth profiles of englacial structure. Human or automated annotators trace internal reflection horizons (layers) that represent isochrones of ice deposition. When two flight lines observe the same ice at a crossing or overlap point, consistent annotations should agree in depth. XCheck operationalizes this constraint: it extracts binary depth masks from annotated layers, aligns them geometrically at the shared observation point, and uses a sliding dot-product test to find matching layer pairs. A high match count indicates annotation consistency; a low count flags potential mislabelling or pick drift.

XCheck handles two radargram relationship types:

- **Consecutive** — two sequential segments of the same flight line share a tail/head overlap detectable via GPS timestamp matching.
- **Non-consecutive (intersecting)** — two flight lines from different dates or paths cross at a known geographic point supplied by a pre-computed intersections CSV.

---

## Methodology

### Algorithm Steps

**Requires:** Binary Masks M1, M2 · GPS time sequences T1, T2 · Flight paths P1, P2 · Surface elevations S1, S2 · Depth axis D · Window size w


| Step | Name | Description |
|---|---|---|
| **1** | Temporal & Spatial Relationship Detection | For consecutive pairs: find overlapping GPS time intervals between T1 and T2 and derive the crossing point from the midpoint of the overlap region using flight paths P1, P2. For intersecting pairs: read the crossing point directly from the intersections CSV. |
| **2** | Surface Normalization | Align each radargram column vertically using surface elevation S1/S2 relative to the depth axis D, correcting for topographic variation before comparison. |
| **3** | Midpoint Column Extraction | Extract a window of w columns around the crossing point from each binary mask, centered on the closest trace to the crossing coordinate. |
| **4** | Dot Product Matching | For each candidate layer row pair (i, j) within `max_distance` pixels, compute the dot product of their column windows. A pair is matched if the score meets `dp_threshold`. |
| **5** | Aggregate Matches | Collect all one-to-one matched layer pairs and report total match count and alignment percentage. |


---

## Installation

```bash
pip install xcheck
```

### NDH_PythonTools (required for data loading)

XCheck uses [NDH_PythonTools](https://github.com/nholschuh/NDH_PythonTools) for `.mat` file loading and radar data processing. It must be installed manually since it is not a standard pip package:

```bash
git clone https://github.com/nholschuh/NDH_PythonTools.git
```

Add its **parent directory** to `PYTHONPATH` before running:

```bash
# Linux / macOS
export PYTHONPATH="/path/to/parent/of/NDH_PythonTools"

# Windows PowerShell
$env:PYTHONPATH = "C:\path\to\parent\of\NDH_PythonTools"
```

The pure matching algorithm (`match_layers_with_dot_product`) does **not** require NDH_PythonTools and works standalone on any NumPy arrays.

---

## Directory Structure

```
pypi package/
├── src/xcheck/
│   ├── xcheck.py          # Core algorithms: loading, masking, GPS overlap,
│   │                      #   column extraction, layer matching, plotting, CLI
│   └── __init__.py        # Public API exports
├── pyproject.toml         # Package metadata and dependencies
├── README.md
└── dist/                  # Built wheel and source distribution
```

---

## Quickstart

### 1. Command Line Interface

**Consecutive radargrams** (crossing derived automatically from GPS overlap):

```bash
xcheck --r1 Data_20120429_01_026.mat --r2 Data_20120429_01_027.mat \
       --layer_dir ./Nick-layer-data-mat/ \
       --radar_dir ./Nick-raw-radargram-mat/
```

**Intersecting radargrams** (crossing looked up from CSV):

```bash
xcheck --r1 Data_20120330_01_004.mat --r2 Data_20120511_01_054.mat \
       --intersections_csv ./Intersections_2012.csv \
       --layer_dir ./Nick-layer-data-mat/ \
       --radar_dir ./Nick-raw-radargram-mat/
```

**Batch mode** (all pairs in the intersections CSV):

```bash
xcheck --intersections_csv ./Intersections_2012.csv \
       --layer_dir ./Nick-layer-data-mat/ \
       --radar_dir ./Nick-raw-radargram-mat/
```

### 2. Python API

**Layer matching only (no NDH_PythonTools needed):**

```python
import numpy as np
from xcheck import match_layers_with_dot_product

m1 = np.zeros((6, 5)); m1[[1, 3, 5], :] = 1
m2 = np.zeros((6, 5)); m2[[1, 3, 5], :] = 1

matches = match_layers_with_dot_product(m1, m2, max_distance=1,
                                        window_size=3, dp_threshold=1)
print(matches)  # [(1, 1), (3, 3), (5, 5)]
```

**Full pipeline (requires NDH_PythonTools + data):**

```python
from xcheck import (
    load_and_process_radargram,
    load_log_gps_time,
    find_overlapping_intervals,
    extract_and_adjust_mask_columns_for_location,
    match_layers_with_dot_product,
)

r1, m1, d1 = load_and_process_radargram(
    "Data_20120429_01_026.mat",
    layer_dir="./Nick-layer-data-mat/",
    radar_dir="./Nick-raw-radargram-mat/")
r2, m2, d2 = load_and_process_radargram(
    "Data_20120429_01_027.mat",
    layer_dir="./Nick-layer-data-mat/",
    radar_dir="./Nick-raw-radargram-mat/")

log_t1 = load_log_gps_time("./Nick-raw-radargram-mat/Data_20120429_01_026.mat")
log_t2 = load_log_gps_time("./Nick-raw-radargram-mat/Data_20120429_01_027.mat")
overlaps1, overlaps2 = find_overlapping_intervals(log_t1, log_t2)

for iv1, iv2 in zip(overlaps1, overlaps2):
    mid1 = (iv1[0] + iv1[1]) // 2
    mid2 = (iv2[0] + iv2[1]) // 2
    mc1 = extract_and_adjust_mask_columns_for_location(
        r1['Longitude'][mid1], r1['Latitude'][mid1], r1, m1, d1, 20)
    mc2 = extract_and_adjust_mask_columns_for_location(
        r2['Longitude'][mid2], r2['Latitude'][mid2], r2, m2, d2, 20)
    matches = match_layers_with_dot_product(mc1, mc2)
    print(f"Matched layers: {matches}")
```

---

## Parameters

| Argument | Default | Description |
|---|---|---|
| `--r1`, `--r2` | — | Single-pair mode: filenames of the two radargrams |
| `--lat`, `--lon` | — | Explicit crossing coordinates for single-pair mode |
| `--csv_path` | — | Batch CSV with columns `Radargram 1`, `Radargram 2`, `Latitude`, `Longitude` |
| `--intersections_csv` | — | Pre-computed intersections CSV (used alone or to look up crossing coordinates) |
| `--layer_dir` | required | Directory of layer `.mat` files |
| `--radar_dir` | required | Directory of raw radargram `.mat` files |
| `--layer_prefix` | `Layer_` | Filename prefix prepended to layer files |
| `--cols_side` | 20 / 3 | Columns extracted each side of crossing (consecutive / non-consecutive) |
| `--max_distance` | 4 / 3 | Max depth-pixel gap between candidate layer rows |
| `--window_size` | 7 / 3 | Dot product window width (must be odd) |
| `--dp_threshold` | 4 / 1 | Minimum dot product score to confirm a match |
| `--plot_overlaps` | off | Plot GPS overlap regions on a map |

Defaults shown as **consecutive / non-consecutive** — selected automatically based on whether a crossing point is available.

---

## Default Thresholds by Radargram Type

| Parameter | Consecutive | Non-Consecutive |
|---|---|---|
| `cols_side` | 20 (41 columns) | 3 (7 columns) |
| `max_distance` | 4 px | 3 px |
| `window_size` | 7 | 3 |
| `dp_threshold` | 4 | 1 |

---

## Methodological Note

GPS overlap detection compares **`log(GPS_time)`** between two radargrams with an absolute tolerance of `1e-8`. Because `d(log t) ≈ dt/t` and GPS times are ~1×10⁹ seconds, this tolerance corresponds to matching raw timestamps within roughly 10 seconds. Consecutive flight segments share a small tail/head of overlapping traces where the crossing point is derived. Non-consecutive intersecting radargrams do not share GPS timestamps — their crossing point must be supplied via `--intersections_csv` or `--lat`/`--lon`.

---

## Citation

Cite our work.

```bibtex
@INPROCEEDINGS{11415920,
  author={Hassan, Muhammad Behroze and Tama, Bayu Adhi and Purushotham, Sanjay and Janeja, Vandana P.},
  booktitle={2025 IEEE International Conference on Data Mining Workshops (ICDMW)}, 
  title={XCheck: A Consistency-Based Validation Framework for Englacial Layers Annotations}, 
  year={2025},
  volume={},
  number={},
  pages={76-85},
  keywords={Surveys;Radar remote sensing;Annotations;Ice sheets;Radar;Benchmark testing;Radar tracking;Robustness;Data mining;Remote sensing;Ice sheets;englacial layers;radargrams spatiotemporal relationship;validation metric;AI-ready dataset},
  doi={10.1109/ICDMW69685.2025.00015}
}

```

---

## References

M. B. Hassan, B. A. Tama, S. Purushotham and V. P. Janeja, "XCheck: A Consistency-Based Validation Framework for Englacial Layers Annotations," 2025 IEEE International Conference on Data Mining Workshops (ICDMW), Washington, DC, USA, 2025, pp. 76-85, doi: 10.1109/ICDMW69685.2025.00015.

---

## License

MIT
