Metadata-Version: 2.4
Name: UnifyVcf
Version: 0.2.0
Summary: Unify SV, CNV, tandem-repeat and ROH calls into one structural VCF for Geneyx.
Author: Bar Cohen
Author-email: Bar Cohen <bar@geneyx.com>
License: MIT License
        
        Copyright (c) 2026 Geneyx
        
        Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Project-URL: homepage, https://dev.azure.com/geneyx/Geneyx/_git/geneyx-unify-vcf
Project-URL: repository, https://dev.azure.com/geneyx/Geneyx/_git/geneyx-unify-vcf
Project-URL: documentation, https://geneyx.com/docs
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# UnifyVcf

Unifies the separate structural call files a sequencing pipeline produces into the
single VCF Geneyx reads as an SV VCF.

The files it unifies are:

- **SV** – structural variants VCF
- **CNV** – copy number variants VCF
- **Repeats** – tandem repeat variants VCF
- **ROH** – Regions of Homozygosity (DRAGEN BED, or an already-converted VCF)

It does not only concatenate them. Each caller writes slightly different files, and
Geneyx needs specific fields to read a record properly — so records are rewritten on
the way out and the result is sorted and bgzipped. **Run the entry point that matches
the pipeline your VCFs came from.**

This package is the single implementation behind both the customer-facing scripts in
[geneyx.analysis.api](https://github.com/geneyx/geneyx.analysis.api) and the Geneyx
primary pipeline, so both produce the same unified file.

## Installation

```bash
pip install UnifyVcf
```

Python 3.9–3.13 on **Linux, macOS or Windows**, with no external tools required and
no dependencies outside the standard library. Sorting and BGZF compression are pure
Python; repeat filtering uses `bedtools` when it is installed and computes the same
intersection itself when it is not.

Wheels are published per platform and Python version because the implementation ships
compiled — see [How this is distributed](#how-this-is-distributed). If `pip install`
reports no matching distribution, your platform or Python version is outside the
published set; tell us which and we will add it.

## Command line

```
dragen-unify-vcf  -o OUTPUTPATH [-s SVPATH] [-c CNVPATH] [-r REPEATPATH] [-d ROH_BED_FILE]
ont-unify-vcf     -o OUTPUTPATH [-s SVPATH] [-c CNVPATH] [-r REPEATPATH] [-modify]
pacbio-unify-vcf  -o OUTPUTPATH [-s SVPATH] [-c CNVPATH] [-r FULLREPEATPATH] [-b REPEATLOCATIONSBEDFILEPATH]
```

| Argument  | Meaning                                                                                                                                                                                                       |
| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `-o`      | Path to the unified VCF, including its name. The output is compressed, so the name should end in `.vcf`, not `.gz`.                                                                                           |
| `-s`      | Structural variants (SV) VCF. Gzipped or not.                                                                                                                                                                 |
| `-c`      | Copy number variants (CNV) VCF. Gzipped or not.                                                                                                                                                               |
| `-r`      | Tandem repeats VCF. Gzipped or not.                                                                                                                                                                           |
| `-d`      | DRAGEN only. The ROH BED produced by DRAGEN when run with `--vc-enable-roh`.                                                                                                                                  |
| `-modify` | ONT only, and no longer needed. Forces `;SVTYPE=REP` onto every repeat call. Without it, a repeat call that carries no `SVTYPE` is marked anyway and one that already has one is left alone.                  |
| `-b`      | PacBio only. A BED restricting which repeat loci are unified. **Without `-b` the repeats are left out of the unified file entirely**, because the full TRGT catalog is far wider than a report can interpret. |

A catalog of pathogenic hg38 STR loci ships with the package:

```python
from UnifyVcf import default_repeat_catalog_path
```

```bash
pacbio-unify-vcf -o unified.vcf -s sv.vcf.gz \
  -r sample.GRCh38.trgt.sorted.vcf.gz \
  -b "$(python -c 'from UnifyVcf import default_repeat_catalog_path as p; print(p())')"
```

## Python

```python
from UnifyVcf import run

run(output_path="unified.vcf", sv_path="sv.vcf.gz", cnv_path="cnv.vcf.gz",
    repeat_path="repeats.vcf.gz", roh_bed_path="roh.bed")
```

`run` keeps the signature the standalone `UnifyVcf.py` script published, so code that
did `from UnifyVcf import run` works unchanged once this package is installed.

A caller that already owns a sorting step — the Geneyx pipeline does — skips `run` and
composes the pieces instead, so it keeps its own sorting and indexing:

```python
from UnifyVcf import UnifyOptions, read_struct_lines, unify_lines

options = UnifyOptions(drop_sv_filters=("InvBreakpoint",))  # exclude by FILTER
unify_lines(read_struct_lines(paths, options), "unified.vcf", options)   # unsorted
```

Nothing about the _shape_ of the inputs has to be declared. Whether a repeat record
needs `;SVTYPE=REP` is read off the record, and whether ROH arrived as a BED or an
already-converted VCF is read off the file — so the same call is correct for DRAGEN,
ONT and PacBio input. `UnifyOptions` still accepts `add_repeat_svtype` and
`roh_format` as overrides for a caller that knows better than its files do.

### How the pieces fit

| Function                                    | Does                                                                                                          |
| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| `read_struct_lines(paths, options)`         | Reads each input (gzipped or not), drops excluded SV records, and skips inputs that are empty or header-only. |
| `unify_lines(lines, output_path, options)`  | Writes the unified, **unsorted** VCF. Returns `None` when there was nothing to write.                         |
| `sort_and_compress(vcf_path)`               | Sorts by position and bgzips; returns the `.gz` path.                                                         |
| `filter_repeats_vcf(repeat_path, bed_path)` | Intersects a repeat VCF with a locus catalog, via bedtools or in Python.                                      |

PacBio has two front doors over these, because repeat filtering is separable:
`pacbio_filter_and_unify` filters the full TRGT VCF against a catalog and then
unifies, which is what `pacbio-unify-vcf` runs; `pacbio_unify` unifies repeats that
are **already** filtered, for a caller that owns that step itself.

The first present input in SV → CNV → repeats order donates the header; the rest
contribute records only.

## Platform support

Everything runs on Linux, macOS and Windows. Two things used to be Linux-only and are
no longer:

**Sorting and compression.** The output has to be sorted by position and compressed as
**BGZF** — a gzip variant written in <64KB blocks — because Geneyx indexes it with
tabix, and tabix cannot index plain gzip. Compression is always pure Python. Sorting
has two engines: a pure-Python one that needs no `sort` binary, and one that shells out
to a real `sort -k1,1 -k2,2n` — several times faster past a few hundred thousand
records, which a PacBio long-read callset can reach. `sort_and_compress` prefers the
`sort` binary when it is on PATH, the same way repeat filtering below prefers
`bedtools`. Either way the ordering matches what `sort -k1,1 -k2,2n` produced under the
previous shell implementation, including GNU sort's whole-line tie-break, so files keep
the order Geneyx has always received — `tests/test_sort_parity.py` pins the two engines
against each other to prove it.

**Repeat filtering.** `bedtools intersect` is used where bedtools is on PATH. Where it
is not, the same intersection is computed in Python: a record spans `[POS-1, END)`,
taking `END` from INFO where present — repeat callers emit symbolic `<STR>` alleles
whose length is not the length of REF — and falling back to `POS-1 + len(REF)`. The
engine that ran is logged. The CI runs the suite on both Linux and Windows, and the
Linux leg installs bedtools so the two engines are compared against each other rather
than one being skipped.

Force one or the other when you need to:

```python
from UnifyVcf import BEDTOOLS, BUILTIN, filter_repeats_vcf

filter_repeats_vcf(repeats, catalog, engine=BUILTIN)
```

```python
from UnifyVcf import GNU_SORT, PYTHON_SORT, sort_and_compress

sort_and_compress(unified_vcf_path, engine=PYTHON_SORT)
```

## How this is distributed

The wheel ships **compiled** modules (`.pyd` on Windows, `.so` elsewhere), not `.py`
files. What it publishes instead is the interface:

- `UnifyVcf/*.pyi` type stubs carrying every public signature and its description,
  generated from the source by `tools/generate_stubs.py`
- the same docstrings inside the compiled modules, so `help(UnifyVcf.run)` works
- `py.typed`, so mypy and IDEs read the stubs
- this README

So autocomplete, type checking and `help()` behave normally; the function bodies are
just not there to read. `__init__.py` stays interpreted deliberately — it is only
re-exports, and it is the surface this README documents.

Consequences worth knowing:

- **No sdist is published.** An sdist would carry the sources, undoing the point.
  `pip download --no-binary :all:` therefore has nothing to fetch.
- **One wheel per platform and Python version**, built by cibuildwheel in CI, rather
  than the single pure-Python wheel a normal package gets.
- **Import from `UnifyVcf`, not from its submodules.** Pylint will not import an
  extension module to see what it exports, and it does not read the stubs, so a
  perfectly good `from UnifyVcf.cli import dragen_from_cli` is reported as
  `E0611: No name 'dragen_from_cli' in module 'UnifyVcf.cli'`. `__init__.py` stays
  interpreted, so pylint reads the names straight out of it and
  `from UnifyVcf import dragen_from_cli` analyses cleanly — which is why everything
  public is re-exported there. A caller that does need a submodule directly wants
  this in its pylint config:

  ```ini
  [MAIN]
  extension-pkg-allow-list=UnifyVcf
  ```

  mypy and Pylance need neither, they read the stubs.

- **This is a deterrent, not a control.** A compiled extension can still be analysed,
  and string literals — log messages, `;SVTYPE=REP` and the like — remain visible in
  the binary because the code has to use them at run time. It stops casual reading and
  copy-paste reuse; it is not a security boundary.

### Working on the package

The sources live in this repository and run interpreted, so development and the test
suite are unaffected:

```bash
pip install -e .            # needs a C compiler; builds the extensions in place
python -m pytest tests
python tools/generate_stubs.py          # after changing any public signature
python tools/generate_stubs.py --check  # what CI enforces
```

`tests/test_published_surface.py` fails if a stub drifts from the source, so the
published description cannot silently stop matching the code.

## Per-caller differences

These are options rather than separate implementations:

|                          | DRAGEN   | ONT      | PacBio                  | Geneyx pipeline               |
| ------------------------ | -------- | -------- | ----------------------- | ----------------------------- |
| `;SVTYPE=REP` on repeats | detected | detected | detected                | detected                      |
| ROH input                | detected | –        | –                       | detected                      |
| Repeat filtering         | –        | –        | `bedtools` against `-b` | done as its own pipeline step |
| SV `FILTER` exclusions   | none     | none     | none                    | `InvBreakpoint` (PacBio)      |

The first two rows used to be per-caller options and are now read off the files, so
those callers no longer differ at all. The third is the same `filter_repeats_vcf`
call on both sides, made at a different point — see the two PacBio front doors above.

The last row is a real difference and not one this package can resolve: a customer
unifying their own PacBio VCFs keeps `InvBreakpoint` records, and the pipeline drops
them, so the same SV callset yields a different reported call set depending on which
path unified it. Whether a breakpoint half of an inversion is a reportable event is a
product decision, and it is listed here to stay visible until it is made.

The PacBio row and the pipeline row differ on `InvBreakpoint` deliberately: the Geneyx
pipeline drops those records from the unified file it analyses, and the customer-facing
script does not. Aligning them would change what customers upload, so it is a product
decision rather than a packaging one.

## Known defect carried over

Pinned by a characterization test rather than fixed, so that packaging stayed a
refactor. Worth its own change:

**`is_valid_line` misses single-sample no-calls.** The GT is read from the last
tab-separated cell, which on a single-sample record still carries the trailing newline,
so `"./.\n" != "./."` and the no-call is kept. Multi-sample records take the intended
path. The neighbouring `gt_index == '0'` compares an `int` to a `str` and is never true.

Two other defects in the original shell-based sorter are gone as a consequence of
rewriting it for Windows, rather than by choice:

- It swallowed its own failures — a failed sort or `bgzip` was logged and execution
  continued, leaving an unsorted, uncompressed `.vcf` where the caller expected
  `.vcf.gz`. Failures now raise.
- Its temp directory was `vcf-temp` in the current working directory, so two runs in
  one directory deleted each other's temp files. Temporary files are now written beside
  the output and removed in a `finally`.

## Tests

```bash
pytest tests
```

No bedtools, bgzip or network needed — the suite is file manipulation only. One test
compares the two repeat-filtering engines and skips where bedtools is not installed.
