Metadata-Version: 2.4
Name: echemdb-ecdata
Version: 0.9.2
Summary: a database for published electrochemical data inferred from SVG and raw CSV files.
Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: LICENSE-CC-BY-4.0
License-File: LICENSE-ODC-By-1.0
Requires-Dist: pydantic<3,>=2.12.5
Requires-Dist: svgdigitizer<0.15,>=0.14.5
Requires-Dist: unitpackage<0.14,>=0.13.1
Dynamic: license-file

# Electrochemistry-data

<p align="center">
  <img src="https://img.shields.io/badge/Code_License-GPL_3.0_or_later-blue.svg" alt="Code License: GPL 3.0 or later">
  <a href="https://creativecommons.org/licenses/by/4.0/"><img src="https://img.shields.io/badge/Data_License-CC--BY--4.0-blue.svg" alt="Data License: CC-BY-4.0"></a>
  <a href="https://opendatacommons.org/licenses/by/"><img src="https://img.shields.io/badge/Data_License-ODC--By--1.0-blue.svg" alt="Data License: ODC-By-1.0"></a>
  <a href="https://doi.org/10.5281/zenodo.20723429"><img src="https://zenodo.org/badge/DOI/10.5281/zenodo.20723429.svg" alt="DOI"></a>
</p>

This repository hosts a dataset for interfacial electrochemistry data,
focusing on cyclic voltammograms of well-defined single crystal electrodes
recorded in aqueous and non-aqueous electrolytes. It consists of published
data that is either retraced from source PDFs or provided by the authors, and
can be browsed on [echemdb.org/cv](https://www.echemdb.org/cv).

![Cyclic voltammograms of select transition metal electrodes.](doc/figures/echemdb_overview.png)

The data is provided as frictionless-based [`unitpackages`](https://echemdb.github.io/unitpackage/)
and originates from two kinds of input:

* **Digitized data** retraced from figures in publications, stored as SVG and
  YAML files and converted with [`svgdigitizer`](https://echemdb.github.io/svgdigitizer/).
* **Directly submitted data** provided by authors as CSV files together with
  YAML metadata.

In both cases a BibTeX (BIB) reference ties the data to its publication, and all
input YAML files and output Data Packages are validated against the [echemdb-metadata schema](https://github.com/echemdb/metadata-schema).
The dataset is generated automatically on each release from the input files in
this repository and is maintained by the [echemdb](https://github.com/echemdb) community.

## Accessing Data

### Direct Download

The generated data archive can be downloaded directly as a ZIP:

* **GitHub release (latest):** [data-0.9.2.zip](https://github.com/echemdb/electrochemistry-data/releases/download/0.9.2/data-0.9.2.zip)
* **Zenodo (archived, citable):** download the ZIP from the *Files* section of the [Zenodo record](https://doi.org/10.5281/zenodo.20723429), which always resolves to the latest version.

All releases are also listed in the [release section](https://github.com/echemdb/electrochemistry-data/releases).

### Unitpackage API

A collection can be created from the the [echemdb module](https://echemdb.github.io/unitpackage/usage/echemdb_usage.html) of the [`unitpackages`](https://echemdb.github.io/unitpackage/) interface
(see [`unitpackages` installation instructions](https://echemdb.github.io/unitpackage/installaton.html)).

```python
from unitpackage.database.echemdb import Echemdb
db = Echemdb.from_remote()
```

### Electrochemistry Data API

Install the latest version of the module.

```sh
pip install git+https://github.com/echemdb/electrochemistry-data.git
```

In your preferred Python environment retrieve the URL with the data via

```py
from echemdb_ecdata.url import ECHEMDB_DATABASE_URL
ECHEMDB_DATABASE_URL
```

## Contributing

You can contribute data in two ways:

* **Digitize data from a publication.** The preparation of the files and the
  extraction of the data from a PDF source is described in the
  [svgdigitizer workflow](https://echemdb.github.io/svgdigitizer/workflow.html). The resulting
  SVG and YAML files go into [`literature/svgdigitizer/`](./literature/svgdigitizer).
* **Submit your own raw data directly.** Add your measurement as a CSV file
  together with a YAML metadata file under
  [`literature/source_data/`](./literature/source_data); no digitization is
  required.

## Development

If you want to work on the data and repository itself, install [pixi](https://pixi.sh)
and clone the repository:

```sh
git clone https://github.com/echemdb/electrochemistry-data.git
cd electrochemistry-data
```

For possible commands run

```sh
pixi run
```

More pixi tasks can be inferred from the [pyproject.toml](pyproject.toml).

### Conversion

The repository converts source data into standardized frictionless datapackages:

```sh
# Convert all data (SVG digitizer + raw data)
pixi run -e dev convert

# Convert only SVG digitizer data (from literature/svgdigitizer/)
pixi run -e dev convert-svg

# Convert only raw data (from literature/source_data/)
pixi run -e dev convert-raw

# Clean generated data before converting
pixi run -e dev clean-data
```

A typical workflow:

```sh
# Clean previous builds and convert all data
pixi run -e dev clean-data && pixi run -e dev convert
```

Generated datapackages are written to `data/generated/svgdigitizer/` and `data/generated/source_data/`.

Both SVG digitization and raw data conversion use a batch approach that imports
heavy dependencies once and processes all files in a single Python process.
This avoids the ~3 s Python startup overhead per file that occurs when spawning
a subprocess for each file, reducing full-rebuild time from ~15 min to ~30-50 s
for 273 SVG files.

Force a full rebuild (ignoring timestamps):

```sh
pixi run -e dev convert-force
```

Verify that the batch conversion produces output identical to existing generated data:

```sh
pixi run -e dev verify-svg   # SVG digitizer output
pixi run -e dev verify-raw   # Source data output
pixi run -e dev verify-all   # Both at once
```

### Validation

All data (input YAML and output JSON) is validated against the [echemdb-metadata schema](https://github.com/echemdb/metadata-schema).
In addition, filenames, identifiers, and bibliography keys are validated for consistency.

Two umbrella tasks cover all checks:

```sh
# Validate all input files (YAML schema, filenames/identifiers, bib keys)
pixi run -e dev validate-input

# Validate all generated files (JSON schema, identifiers)
pixi run -e dev validate-generated
```

These are also used in the CI workflows. You can run individual sub-tasks:

```sh
# Schema validation
pixi run -e dev validate-svgdigitizer-yaml  # Input YAML (svgdigitizer)
pixi run -e dev validate-source-yaml        # Input YAML (source data)
pixi run -e dev validate-svgdigitizer       # Generated JSON (svgdigitizer)
pixi run -e dev validate-raw                # Generated JSON (source data)

# Filename and identifier validation
pixi run -e dev validate-identifiers              # All input filenames
pixi run -e dev validate-svgdigitizer-filenames   # SVG digitizer filenames only
pixi run -e dev validate-source-filenames         # Source data filenames only
pixi run -e dev validate-generated-identifiers    # Generated data identifiers

# Bibliography key validation
pixi run -e dev validate-bib-keys  # Check bib keys match expected identifiers
pixi run -e dev validate-bib-utf8  # Check for LaTeX accent encodings
```

Validate against a specific schema version:

```sh
pixi run -e dev validate-input --version 0.8.0
pixi run -e dev validate-generated --version main
```

### Fix Utilities

```sh
# Lowercase SVG labels and filenames (enforced for Windows compatibility)
pixi run -e dev fix-lowercase          # Apply changes
pixi run -e dev fix-lowercase-dry-run  # Preview only

# Convert LaTeX accent encodings to UTF-8 in bibliography.bib
pixi run -e dev fix-bib-utf8           # Apply changes
pixi run -e dev fix-bib-utf8-dry-run   # Preview only

# Auto-fix identifier mismatches (detects dir name != YAML citationKey)
pixi run -e dev fix-identifiers          # Apply changes
pixi run -e dev fix-identifiers-dry-run  # Preview only

# Rename directories and files after a bib key change (manual)
pixi run -e dev rename-identifiers OLD_NAME NEW_NAME

# Migrate input YAML metadata across breaking metadata-schema releases
# (applies the migration steps shipped with the metadata-schema repository)
pixi run -e dev migrate-metadata          # Apply changes
pixi run -e dev migrate-metadata-dry-run  # Preview only
```

## Reviewing Submissions (Curators)

New literature and source-data pull requests can be reviewed with the help of an
AI assistant. Both assistants run the same checks (input validation, the review
module, and PDF cross-checks against the cited paper) and produce a review report.

- **GitHub Copilot** — invoked as slash commands:
  - `/review-source-data-pr <folder-name>`
  - `/review-literature-pr <folder-name>`
- **Claude Code** — the equivalent skills:
  - `review-source-data-pr <folder-name>`
  - `review-literature-pr <folder-name>`

Here `<folder-name>` is the entry directory (e.g. `droog_1980_oxygen_387`). The
review generates a `REVIEW.md` in the repository root (the parent directory of the
entries). This file is **git-ignored** and must not be committed: the reviewer marks
each finding as accept / reject / comment and adds notes directly in it, and the
assistant then applies the accepted fixes.

For recurring issues indicate by `(add to context)` in the comment, that these aspects should be considered in future reviews.
This will update the review skills.

The canonical, shared instructions for both assistants live in
[`.github/prompts/`](.github/prompts/) (the Claude skills under
[`.claude/skills/`](.claude/skills/) delegate to the same prompt files).

## License

You can redistribute and/or modify the contents of this repository under the
terms of the GNU General Public License (GPL) as published by the Free Software
Foundation; either version 3.0 of the License, or (at your option) any later
version (GPL-3.0-or-later). See https://www.gnu.org/licenses.

The data and generated data, i.e., the contents of
[`literature/`](./literature) and the contents of the generated data archive
that is published with each release, are also available under either of the
following two licenses, at your option:

* The [Creative Commons Attribution 4.0 International
  License](https://creativecommons.org/licenses/by/4.0/) (CC-BY-4.0)
* The [Open Data Commons Attribution
  License](https://opendatacommons.org/licenses/by/) (ODC-By-1.0)

While you should consult the above licenses for all the technical details, the
above roughly translates to the following in practice:

* You may use any of the raw and processed data you find here as long as you
  give credit.
* If you want to incorporate any source code you find here into your published
  product or project it must itself be released under the GPL version 3 or
  later.
