Metadata-Version: 2.4
Name: on-the-list
Version: 0.1.2
Summary: Reads a cosmetic ingredient list and reports what the EU's official annexes say about the ingredients on it.
Author: Waiga Arya
License-Expression: MIT
Project-URL: Homepage, https://github.com/Waiga/on-the-list
Project-URL: Issues, https://github.com/Waiga/on-the-list/issues
Keywords: cosmetics,inci,ingredients,cosing,labelling,cli
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Manufacturing
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# on-the-list

Reads a cosmetic ingredient list and reports what the EU's official annexes say
about the ingredients on it.

Runs entirely on your machine. No account, no API key, no upload, no
dependencies beyond Python itself. The annexes are shipped with the package;
downloading a fresh copy is a separate command you have to type.

What came back when it was pointed at 16,635 real published labels, together
with the hand audits and their limits, is written up in
[An ingredient list cannot tell you most of what you want to know](https://medium.com/@aryawaiga0/an-ingredient-list-cannot-tell-you-most-of-what-you-want-to-know-f3807f357837).

```
$ on-the-list examples/shade-stick.txt

on-the-list 0.1.2 — examples/shade-stick.txt
CosIng annexes II-VI, Commission last update 28/08/2026, 1913 distinct names,
read from the copy shipped with this package

10 ingredients read. 2 prohibited, 1 colourant order, 1 repeated entry.

NAMES THAT MATCH AN ANNEX II ENTRY
  Annex II is the list of substances prohibited in cosmetic products. A match
  is reported here as a match. Whether it means anything about this product
  is for someone with the formulation in front of them.

  Butylphenyl Methylpropional matches Annex II, entry 1666, in the
  Commission's list of substances prohibited in cosmetic products.
      printed as: Butylphenyl Methylpropional
      register name: BUTYLPHENYL METHYLPROPIONAL (read from the 'identified'
      column)
      annex entry: 2-(4-tert-butylbenzyl) propionaldehyde
```

## The thing to understand before anything else

**An ingredient that appears in none of the annexes is not an error, and this
tool never reports it as one.**

Annexes II to VI are lists of *restricted* substances: prohibited, restricted,
permitted colourants, permitted preservatives, permitted UV filters. They are
not an inventory of valid INCI names. Aqua and Glycerin appear in none of them,
because nothing restricts Aqua or Glycerin.

So *"that is not a real INCI name"* is not a check this tool can make, and it
does not attempt one. Absence from the annexes means **these annexes say
nothing about it**, and the report says exactly that, in those words. There is
no `unknown` and no `unrecognised` anywhere in the output.

This is the most likely way to misread the tool, which is why it is the first
thing on the page. The reasoning is set out in full in
[`docs/superpowers/specs/2026-09-09-on-the-list-design.md`](https://github.com/Waiga/on-the-list/blob/main/docs/superpowers/specs/2026-09-09-on-the-list-design.md).

## What it does not do

It never says a product is compliant, safe, legal, permitted or clean. It
compares names printed on a label with names printed in a published annex, and
reports where they coincide. Whether that means anything about a particular
product depends on its concentration, its product type, its route of exposure
and its formulation, none of which an ingredient list states.

That is not modesty. It is the only claim the evidence supports.

## Install

```bash
pip install on-the-list
```

Python 3.11 or newer. Nothing else. To install from a clone instead, `pip install .`
from the repository root.

## Use

```bash
on-the-list label.txt                      # whole pack text; the list is found in it
cat product-page.txt | on-the-list -
on-the-list --ingredients inci.txt         # you already have just the list
on-the-list --ingredients inci.txt --pack-text pack.txt   # runs the warning check too
on-the-list label.txt --format markdown    # to paste into an email
on-the-list label.txt --format json        # for a pipeline
on-the-list label.txt --skip colourant-order
on-the-list --list-checks
on-the-list register                       # which register is in use, and its hashes
on-the-list update-register                # the only command that uses the network
```

Exit codes: `0` nothing found, `1` at least one finding, `2` could not run. A
label whose ingredient list did not parse exits `2`, never `0`, and so does a
run with every check switched off: a green CI job over a label that nothing was
compared to is the one thing this tool exists not to do.

## The four checks

Each one can be switched off on its own with `--skip`, and the report always
prints which ran and which did not, with a reason. A check that did not run must
never read as a check that found nothing.

| | |
|---|---|
| **prohibited** | An ingredient's name matches an entry in **Annex II**. Reported as a match with the entry number, split into entries that say something unconditional about the substance and entries whose own wording sets a condition. |
| **colourant-order** | A `CI NNNNN` colourant is printed before a non-colourant. Article 19(1)(g) allows colourants in any order *after* the other ingredients, so this is a positional observation with no judgement in it. |
| **repeated-entry** | The same name appears twice in the declared list. Needs no register at all. |
| **warning-wording** | An annex attaches a `Contains …` statement to an ingredient on the list, and that statement is not in the pack text you supplied. Reported as a **gap between two documents**, never as a violation: the wording may be printed somewhere the supplied text does not cover, in another language, or the entry's condition may not apply. It needs pack text to search. A whole label file counts as that and `--ingredients` on its own does not. It says which when it does not run. |

Two things that are not checks and change no exit code:

**What the annexes name.** Every entry in Annexes II to VI that names an
ingredient on the list, with what that annex is (prohibited, restricted, a
permitted colourant, preservative or UV filter) and the product types the entry
is limited to. This is the tool's own title, and for a while it was the one
thing the report did not print: a label containing Phenoxyethanol produced no
finding, and the reader was told one of three ingredients was named somewhere
and never told where.

**Coverage.** How many ingredients no annex entry names, labelled *not
restricted by these annexes*.

### Four narrowings that cost coverage on purpose

**The position check uses only bare `CI NNNNN` forms.** Named Annex IV
substances are not used, because most do more than one job: Titanium Dioxide is
an Annex IV colourant, an Annex VI UV filter and an opacifier, and where it
sits in a list proves nothing about which of those it is doing. A CI number is
unambiguous.

**Only `Contains …` statements are searched for in pack text.** The annexes'
wording column mixes label text with conditions that have nothing to do with a
label: *"Purity criteria as set out in Commission Directive 95/45/EC (E 129)"*,
*"Only nanomaterials having the following characteristics are allowed"*.
Treating the whole column as label text would report a missing warning on every
sunscreen containing zinc oxide. 38 of the 627 entries in Annexes III to VI
yield a statement this tool will look for; the rest are shown to the reader and
not searched for, and the report says so.

**A match on part of a printed name is not reported against Annex II.**
`CI 77288 / CHROMIUM` is a colourant printed beside its element name, and the
fragment `CHROMIUM` matches the Annex II entry for chromium metal. Over 16,635
real labels, fragment matches produced no correct Annex II finding at all, so
they are shown under *considered and not counted* with the reason instead. A
name with a bracketed aside removed (`Titanium Dioxide (nano)` → `Titanium
Dioxide`) is still the name of the ingredient, and does match.

**Nothing is built on a concentration.** A label does not state one. Every
maximum-concentration column in the annexes is unused.

## Reproducibility

The register shipped in this package is the European Commission's own CosIng
export, byte for byte. Every report names the version it used, and you can
check it:

```bash
$ on-the-list register
CosIng annexes II-VI, Commission last update 28/08/2026, 1913 distinct names,
read from the copy shipped with this package

CosIng, © European Union, 1995-2026. Annexes II to VI of Regulation (EC)
No 1223/2009, retrieved from the European Commission CosIng API. Reused under
Commission Decision 2011/833/EU (CC BY 4.0). Unmodified.

Annex II — list of substances prohibited in cosmetic products
  1758 entries, Commission last update 28/08/2026
  sha256 b7105a05bf724bb10cebaac3a3813b6146c3153ae1d07097d35bb1f73cd7283b
  https://api.tech.ec.europa.eu/cosing20/1.0/api/annexes/II/export-csv
```

```bash
curl -s https://api.tech.ec.europa.eu/cosing20/1.0/api/annexes/II/export-csv | shasum -a 256
```

If that hash no longer matches, the Commission has republished the annex, and a
result produced against the old one is a result about a document that no longer
exists. `on-the-list update-register` downloads the current files; the report
then names the new hashes instead.

Full manifest, including the corpus used for the measurements below:
[`docs/corpus-manifest.md`](https://github.com/Waiga/on-the-list/blob/main/docs/corpus-manifest.md).

## Measured against real labels

Unit tests pass on the inputs their author imagined, which proves very little.
This was run over **16,635 real published cosmetic labels** from an Open Beauty
Facts export: real packs, real messiness, none of it written by this project.
The selection rule is every record in the export whose `ingredients_text` field
is at least 50 characters. Nothing else is filtered or sampled.

`tools/measure_corpus.py` produces the automated corpus counts and redraws the
seeded samples described here. Human review produces the audit judgments.

| | |
|---|---|
| Records in the export | 64,237 |
| Labels selected | 16,635 |
| Labels whose list parsed | 16,535 |
| Crashes | 0 |
| Ingredients parsed | 298,423 |
| **Annex II matches** | **2,408** on 2,037 labels |
| of which the annex entry is unconditional | 1,066 on 942 labels |
| of which the annex entry sets a condition | 1,342 on 1,167 labels |
| Colourant position | 975 on 645 labels |
| Repeated entries | 712 on 291 labels |
| Warning wording not found | 1,170 on 1,027 labels |
| Considered and not counted | 554 |
| Labels with pack prose inside the ingredient panel | 4,288 (25.8%) |
| Labels whose ingredient field carries a section heading | 1,631 (9.8%) |

Those automated counts come out of `tools/measure_corpus.py`, including the
per-entry and per-statement breakdowns quoted below. `--samples` redraws the
seeded selections with the seed and size the manifest names. The judgments do
not come from the script.

### What the hand audits found

Hand-audited against the published annex text or the label's own list. These
are the numbers, not a summary of them.

**prohibited.** All 19 Annex II entries behind the 1,066 unconditional findings
in the pinned corpus group were read against the annex text. One overbroad
mapping was responsible for 99 of those findings. The other 18 mappings were
supported: Butylphenyl Methylpropional (651 findings), Zinc
Pyrithione (105), Hydroxyisohexyl 3-Cyclohexene Carboxaldehyde (100),
Pentasodium Pentetate (65), Ergocalciferol and Cholecalciferol (11), borates
(7), 4-Methylbenzylidene Camphor (5), Formaldehyde (4), and a tail of one to
three each. The overbroad mapping is the tool's largest single known false
positive: **Annex II entry 1388 is `Octamethylcyclotetrasiloxane; D4`, and the
Commission's own identified-ingredients column gives its INCI name as
`CYCLOMETHICONE`**. Cyclomethicone names a *mixture* of cyclic siloxanes, so a
label printing it has not said it contains D4. That produced 99 findings. There
is no mechanical signal separating this from a correct match, so it is reported
here rather than patched around.

The 1,342 conditional findings are counted neither right nor wrong, because an
ingredient list cannot settle them. They are dominated by entries whose
condition no label states: petrolatum *"except if the full refining history is
known"*, the furocoumarin entry that names ordinary citrus oils *"except for
normal content in natural essences used"*, and the colourants prohibited only
*"when used as a substance in hair dye products"*. Each is printed with its own
wording so a reader can see the condition and decide.

**colourant-order: 4 of 30 wrong.** Thirty findings drawn by
`tools/measure_corpus.py --samples out.json` (seed 11, one row per label), each
read against the label's own list. Two failures are pack prose that the parser
kept: Spanish marketing copy after the colourants, and an OCR'd panel where
the list has no commas at all. One is a make-up palette declaring several
shades in one field. One is an OCR'd label whose text before `INGREDIENTS` is
unreadable. This sample result is not a population rate.

The underlying weakness is deeper than that rate suggests, and it is stated in
[`docs/limitations.md`](https://github.com/Waiga/on-the-list/blob/main/docs/limitations.md): the Regulation *permits*
colourants after the other ingredients but does not *require* it, so a colourant
in weight order is not necessarily out of place. The check reports where a name
is printed and nothing more.

> **Correction, 11 September 2026.** An earlier version described 16 of 30
> repeated entry findings as false positives and presented 53 percent as if it
> described the corpus. The row level evidence for that statement was incomplete.
> A fresh audit of the same seeded sample found 21 false positives and 9 findings
> that correctly described a repeated normalized name in the recorded text. This
> is a result for 30 sampled findings. It is not a rate for the full corpus, all
> repeated entry findings, clean single product lists, or future inputs.

The 21 false positives comprised nine multi product or multi component fields,
seven OCR, transcription, or parser artifacts, and five fields that were not one
cosmetic ingredient declaration. The 9 sound findings correctly described a
repeated normalized name in the historical recorded text. Not every row had a
usable package image, and two sound rows were different barcodes carrying the
same recorded formula. The [row level audit](https://github.com/Waiga/on-the-list/blob/main/docs/audits/repeated-entry-audit.md)
records the judgment and evidence basis for every finding.

**warning-wording: accuracy not measured.** Open Beauty Facts carries no field
holding the text printed on a pack, so every finding in the corpus run is a gap
by construction and there is no accuracy result. What the run does establish is
that the check fires on the right population: of the 1,170 findings, 692 are
`Contains sodium fluoride` on fluoride toothpastes, 132
`Contains sodium monofluorophosphate`, 127 `Contains hydrogen peroxide` on
developers, 93 `Contains ammonia` on hair colour and 48
`Contains Benzophenone-3` on sunscreens. Its accuracy is untested, and this
README will say so until a corpus with real pack text exists.

### What the measurement found that the tests did not

Every one of these was a real defect, found only by meeting 16,635 real labels
or by reading the annexes against their own text:

- **Feeding `csv.reader` a list of lines destroys newlines inside quoted
  fields.** The annexes put ten-line warning blocks in one cell.
  `Contains selenium disulphide\nAvoid contact with eyes` silently became one
  string. The extractor found 32 statements instead of 38, and 19 of the ones it
  did find ran on into the sentence after them. No error, all tests green. The
  reader uses `io.StringIO`.
- **The identified-ingredients column is comma-separated and its values contain
  commas.** A plain `split(",")` gives a different answer from the parser on 52
  rows, and on 38 of them it leaves a one-character fragment: a substance called
  `N`, from `N,N-DIETHYL-m-AMINOPHENOL`.
- **Annex II entry 1725 is `Styrene/Acrylates copolymer (nano)` and lists the
  ordinary INCI name beside it.** 421 labels printing the ordinary polymer
  matched a prohibited entry that is not about it. They are now reported as
  considered and not counted, with the reason.
- **HTML entities were resolved after splitting, not before.** The `;` inside
  `&lt;` is a separator: one Russian pack produced three ingredients called
  `&lt`, an American one produced three called `FD`, and each set was reported
  as a repeated ingredient.
- **`[+/- CI 77491, CI 77492]` closes its bracket after the colourants, not
  after the marker.** A pattern that expected `[+/-]` as a unit matched none of
  it, so a whole shade-range block was read as declared ingredients.
- **Toothpastes print `Contient du fluorure de sodium (1450 ppm de fluor)`
  immediately after the colourants with no break.** Read as an ingredient, it
  was the commonest reason a colourant appeared to be printed in front of a
  non-colourant.
- **`Cl 77492` with a lowercase L is endemic in scanned panels**, and appears
  beside a correctly read `CI` in the same list.
- **`Styrene / Acrylates Copolymer` is one name, not two.** Treating a spaced
  slash as a synonym separator left the fragment `Styrene`, which matches the
  styrene monomer in Annex II, on 35 real labels. Requiring a space on both
  sides of the slash left 21 of them; a polymer names its monomers with slashes,
  and the rule now says so, which leaves none.
- **Hydroquinone is prohibited by Annex II "with the exception of entry 14 in
  Annex III"**, where it is allowed in professional nail products. Reporting
  only the prohibition is a half-truth, so a match that also appears in another
  annex now says which.
- **Only 314 of Annex II's 1,758 rows carry an INCI name at all.** The rest are
  chemical or CAS identifiers with no cosmetic-glossary equivalent. The
  prohibited check can only see 18% of the prohibited list.
- **A file handed to `--ingredients` usually starts with the word
  "Ingredients:".** It was not stripped on that path, so the first ingredient
  became `Ingredients: Formaldehyde`, matched nothing, and the run exited 0,
  while the same text through the whole-label path reported the match.
- **`Chromium (CI 77288)` was reported as prohibited and `CI 77288 / CHROMIUM`
  was not.** Same substance, two print orders, opposite answers. What survives a
  bracket being removed is the name only if it is at least half the words.
- **`Glycerin +/- 0.5%` opened a shade-range block.** Every declared ingredient
  after a printed tolerance dropped out of both order-dependent checks, and the
  report said only that the list had a "may contain" block.
- **The position check was quadratic.** 20,000 colour index numbers took 64
  seconds; the hostile-input tests never called the checks, only the parser.
- **A colourant found only in Annex II was described as "listed in Annex IV".**
  Four are: CI 12150, CI 20170, CI 27290 and CI 45425, each prohibited in hair
  dye. The finding now names the entries the register actually holds.
- **A malformed `--register` directory raised a traceback and exited 1**, which is the
  code for "findings were reported", and the message about the Commission
  having changed the export format was unreachable.
- **`--skip` on all four checks exited 0.** Nothing was compared, and the report
  said "nothing found".

The corpus itself is not redistributed here. Open Beauty Facts is ODbL 1.0,
which is share-alike and incompatible with this repository's MIT licence. Only
the measurements are published, with the export's hash so you can obtain the
same file.

## What it misses

Stated plainly, because a checking tool that hides its blind spots is worse than
none. The long version is in [`docs/limitations.md`](https://github.com/Waiga/on-the-list/blob/main/docs/limitations.md).

- **It sees 18% of Annex II.** Most rows have no INCI name, so most prohibited
  substances cannot be matched against a label at all.
- **It cannot see concentration.** Most annex restrictions are limits, and a
  limit is not something a name can breach.
- **It cannot see product type.** A restriction on hair dye says nothing about a
  face cream, and an ingredient list does not say which the product is.
- **It matches exact names only, after folding.** No fuzzy matching, no
  substring search, no edit distance. A misspelling, a supplier's trade name, or
  a name the Commission writes differently will not match, and the tool will not
  tell you it missed one. It does look a printed name up under a bracketed or
  slash-separated part of itself (that is how `CI 77891` is found inside
  `Titanium Dioxide (CI 77891)`), but **only the Annex II check refuses a match
  found that way**; the other checks and the coverage count accept it.
- **It reads five annexes.** Annex I (the safety report) and Annex VII are not
  read, and neither is any national requirement, retailer standard, or rule
  outside the EU.
- **A pack panel inside the ingredient field derails everything.** A quarter of
  the corpus had prose in the panel. The tool says so when it can tell.
- **A "may contain" block is excluded from the position check** and is printed
  across a whole shade range, so nothing positional can be said about it.
- **Crowd-sourced label data can be wrong.** In the measurement above, a finding
  may be about a mistyped record rather than about a pack.

## Licence

MIT, for the code. See `LICENSE`.

The annex data in `on_the_list/data/` is European Commission material: CosIng,
© European Union, 1995-2026, Annexes II to VI of Regulation (EC) No 1223/2009,
retrieved from the CosIng API and redistributed unmodified. Reuse is governed by
Commission Decision 2011/833/EU; the Commission's legal notice states that
content it owns is licensed under CC BY 4.0.

Nothing in this repository is legal or regulatory advice.
