Metadata-Version: 2.4
Name: cwb-vrt
Version: 0.3.0
Summary: Tools for processing VRT files
Home-page: https://github.com/ausgerechnet/cwb-vrt
Author: Philipp Heinrich
Author-email: philipp.heinrich@fau.de
License: GPL-3.0-or-later
Classifier: Development Status :: 3 - Alpha
Classifier: Operating System :: Unix
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: >=3.10.0
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pandas>=2.0
Requires-Dist: tqdm>=4.70.0
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license
Dynamic: license-file
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# cwb-vrt #
[![Build](https://github.com/ausgerechnet/cwb-vrt/actions/workflows/build-test.yml/badge.svg?branch=main)](https://github.com/ausgerechnet/cwb-vrt/actions/workflows/build-test.yml?query=branch%3Amain)
[![PyPI version](https://badge.fury.io/py/cwb-vrt.svg)](https://badge.fury.io/py/cwb-vrt)
[![PyPI Downloads](https://img.shields.io/pypi/dm/cwb-vrt)](https://img.shields.io/pypi/dm/cwb-vrt)
[![License](https://img.shields.io/pypi/l/cwb-vrt.svg)](https://github.com/ausgerechnet/cwb-vrt/blob/main/LICENSE)

**cwb-vrt** is a Python 3 module and command line interface for processing VRT files.

- [Installation](#installation)
- [VRT files](#vrt-files)
  - [VRT files for CQPweb](#vrt-files-for-cqpweb)
- [Using the CLI](#using-the-cli)

## Installation ##

Requires Python 3.10+.

via PyPI
```bash
pip install cwb-vrt
```
via git (latest, unreleased changes)
```bash
pip install git+https://github.com/ausgerechnet/cwb-vrt.git
```

## VRT files ##

VRT files are XML-like files containing verticalised text and are used as an import (and export) format of the [IMS Open Corpus Workbench (CWB)](http://cwb.sourceforge.net/).  The CWB distinguishes **positional attributes** (p-atts) on a token-level, which are stored in tab-separated lines, and **structural attributes** (s-atts) stored in XML elements (i.e. matching pairs of start and end tags).
```xml
<?xml version="1.0" encoding="ISO-8859-1" standalone="yes" ?>
<!-- A Thrilling Experience -->
<story num="4" title="A Thrilling Experience">
<p>
<s>
Tick	NN	tick
.	SENT	.
</s>
<s>
A	DT	a
clock	NN	clock
.	SENT	.
</s>
<s>
Tick	VB	tick
,	,	,
tick	VB	tick
.	SENT	.
</s>
</p>
...
</story>
...
```

The VRT file above contains three p-atts: by default, the first or *primary* layer is called `word` — the other p-atts here could e.g. be named `pos` and `lemma`. Note that the names of p-atts are usually not explicitly encoded in VRT files.

There are also three s-atts encoded in XML-elements: `story`, `p`, and `s`. `story` has two attribute-value pairs:
```xml
<story num="4" title="A Thrilling Experience">
```
cwb-vrt refers to the name of the XML element (e.g. `story`) usually as "level" of the s-att and stores it alongside the other key-value pairs in a dictionary. Note that in the CWB, each attribute is stored separately (here: `story_num` and `story_title`) with **annotation** (`story` itself is not encoded as an s-att).  `p` and `s` do not have any annotation.

Unlike well-formed XML, s-attribute regions in a VRT file don't have to nest properly — they may overlap (also called "crossing" or "interleaved" annotations). For example, a `<s>` spanning a `<p>` boundary is valid VRT but not valid XML: `<p>...<s>...</p><p>...</s>...</p>`.

### VRT files for CQPweb ###

Not all VRT files accepted by the CWB can be used for CQPweb:
- there has to be an XML-element called "text" with unique IDs `<text id="...">`
- only a relatively small number of `<text>`s are possible (~ 10,000,000)
- meta data stored in `<text>`-attributes can be used for subcorpus creation and restricted queries, but they have to be marked as **categorical** in CQPweb — and this only works if all values are valid MySQL-identifiers

## Using the CLI ##

Installing cwb-vrt puts a single `cwb-vrt` command on your PATH, with one subcommand per tool. Run `cwb-vrt --help` for the full list, or `cwb-vrt <subcommand> --help` for a subcommand's options.
```
$ cwb-vrt --help
usage: cwb-vrt [-h] [-v]
               {bio-repair,convert,cqpweb,dedup,encode,group,merge,meta,remove}
               ...

cwb-vrt: tools for processing VRT files

positional arguments:
  {bio-repair,convert,cqpweb,dedup,encode,group,merge,meta,remove}
    bio-repair          repair (potentially invalid) BIOES token tagging into
                        valid BIO tagging
    convert             render VRT file(s) as HTML (or plain text)
    cqpweb              make VRT file compatible with CQPweb
    dedup               find duplicate regions, or remove them from the VRT
    encode              create CWB import script from VRT file
    group               conflate texts according to meta data into cohorts
    merge               merge several VRT files of cohorts into one VRT
    meta                create TSV table of meta data stored in s-atts
    remove              remove structural and/or positional attributes from
                        VRT file

options:
  -h, --help            show this help message and exit
  -v, --version         output version information and exit
```

`cwb-vrt bio-repair`: repair (potentially invalid) BIOES token tagging into valid BIO tagging, storing it as an s-attribute
```bash
cwb-vrt bio-repair ner-bioes.vrt.gz --col 2 --level ner
```

`cwb-vrt convert`: render VRT file(s) as HTML (or plain text), optionally highlighting spans annotated with a given s-attribute (e.g. `<sub>` for sensitive data)
```bash
cwb-vrt convert tagesschau-mini-cqpweb.vrt.gz --highlight sub
```

`cwb-vrt cqpweb`: make VRT file compatible with CQPweb
```bash
cwb-vrt cqpweb tagesschau-mini.vrt.gz --level article
```

`cwb-vrt dedup`: find regions enclosed by level that are duplicates (`--mode report`, default: writes a TSV table), or remove them directly (`--mode deduplicate`: writes a cleaned copy of the VRT)
```bash
cwb-vrt dedup tagesschau-mini.vrt.gz --level s
cwb-vrt dedup tagesschau-mini.vrt.gz --level s --mode deduplicate
```

`cwb-vrt encode`: create CWB import script from VRT file
```bash
cwb-vrt encode tagesschau-mini.vrt.gz
```

`cwb-vrt group`: conflate texts according to meta data into cohorts
```bash
cwb-vrt group -m tagesschau-mini.vrt.gz -c month rubrik --level-old article --level-new article
```

`cwb-vrt merge`: merge several VRT files of cohorts into one VRT
```bash
cwb-vrt merge 'tagesschau-mini-cohorts/*.vrt.gz' --name TAGESSCHAU
```

`cwb-vrt meta`: create TSV table of meta data stored in s-atts
```bash
cwb-vrt meta tagesschau-mini.vrt.gz --level article
```

`cwb-vrt remove`: remove structural and/or positional attributes from VRT file, or just selected keys from a structural attribute's annotation
```bash
cwb-vrt remove tagesschau-mini.vrt.gz --s_atts p --p_atts 1 --s_att_keys article:fname
```
