Metadata-Version: 2.4
Name: lingity
Version: 0.1.0
Summary: Deterministic governed text analysis
Author: phrocker
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/phrocker/lingitylingity
Project-URL: Repository, https://github.com/phrocker/lingitylingity
Project-URL: Issues, https://github.com/phrocker/lingitylingity/issues
Keywords: linguistics,readability,governance,nlp,deterministic
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Typing :: Typed
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: jsonschema<5,>=4.23
Requires-Dist: markdown-it-py<4,>=3
Requires-Dist: nltk<4,>=3.9
Requires-Dist: spacy<3.9,>=3.8
Provides-Extra: dev
Requires-Dist: mypy<2,>=1.13; extra == "dev"
Requires-Dist: pytest<9,>=8.3; extra == "dev"
Requires-Dist: trove-classifiers>=2024.10.0; extra == "dev"
Requires-Dist: types-jsonschema<5,>=4.23; extra == "dev"
Provides-Extra: release
Requires-Dist: lingity[dev]; extra == "release"
Requires-Dist: build<2,>=1.2; extra == "release"
Requires-Dist: twine<7,>=5; extra == "release"
Dynamic: license-file

# Lingity

Lingity currently provides deterministic, governed text analysis for
LLM-authored or human-authored content. This milestone ships two local CLI
commands: `analyze` produces a reproducible JSON analysis artifact, and
`verify` validates and replays that artifact.

There are no network calls or LLM calls in the current runtime. Provider
interfaces and schemas exist for planned rewrite-proposal and semantic-drift
adapters, but they are not invoked by `analyze` or `verify`.

## Core contract

1. The source text is immutable.
2. Every score is reproducible from a versioned profile and analyzer.
3. The current runtime must not emit success-shaped fallback results.
4. Protected facts, identifiers, quantities, modality, negation, citations,
   ownership, and governance status must survive unchanged.
5. Planned rewrite candidates must improve the configured linguistic
   thresholds without introducing a hard-gate violation.
6. Planned rewrite runs must surface semantic uncertainty as `needs_human`,
   not disguise it as success.

## Current workflow

```text
source text
  -> deterministic analysis
  -> schema-valid analysis artifact
  -> deterministic verification replay
```

## Planned workflow

```text
source text
  -> deterministic analysis
  -> bounded LLM proposal
  -> invariant validation
  -> deterministic rescoring
  -> semantic-drift challenge
  -> accept, iterate, reject, or require human review
```

Four profiles ship. `architecture-review` reads recommendations, ADR summaries,
findings, risks, and review decisions that must remain precise while reading
like professional human communication. `product-strategy` reads need
statements, value propositions, positioning, and go-to-market plans.
`web-copy` reads public-facing prose: landing pages, product descriptions,
job listings. `resume-review` reads resume and CV accomplishment bullets.

A strategy document fails differently from an architecture review. It claims
something unfalsifiable, or it claims it without naming who acts, so
`product-strategy` weights agency and lexical clarity highest and structure
lowest.

It also sets `require_responsible_actor`. Under that threshold a directive must
name an actor the profile recognises, rather than any noun at all, and
`product-strategy` omits "market", "industry", and "space" from its actor
terms. "The market should prioritize retention" therefore reports
`LING-ACTOR-001`, because a sentence whose only actor is the market names
nobody who can act.

The difference is measurable. On the same hyped paragraph,
`architecture-review` scores 88.65 and reports no jargon at all, while
`product-strategy` scores 69.53 and reports five jargon findings. Neither
profile penalises prose that names a number, an actor, and a limit.

A resume fails differently again. A bullet drops its subject by convention, so
`resume-review` sets `allow_implied_first_person`: a directive that carries no
subject at all is read as the author rather than as missing agency. The reading
is narrow on purpose. It suppresses the missing-subject finding and nothing
else, so "Must be completed before the release" still reports both
`LING-AGENCY-001` and `LING-PASSIVE-001` — a resume written in the passive
hides the work, which is the defect this profile exists to find.

`resume-review` also sets `count_repetition_across_blocks`. Redundancy is
otherwise counted within a single block, which a bullet list defeats: every
bullet is its own block, so a verb opening six of them never repeats inside
one. The flag compares the document as one bucket, and widens which tokens are
compared rather than which are read.

Without the threshold the actor rule taxes the wrong lines. "Cut checkout
latency from 1.2 seconds to 300 milliseconds" reports `LING-ACTOR-001` under
both prose profiles, while "Responsible for the migration of the reporting
platform" reports nothing at all under either. The measured effect on a six
bullet resume: `architecture-review` scores the weak version 91.56 and the
strong version 94.95, `product-strategy` scores them 89.44 and 93.69, and
`resume-review` scores them 66.43 and 100.00. Only `resume-review` ranks the two documents the way a reader would.

## Installation

Lingity is not yet published to a package index, so install it from a clone.
Three commands are required, and the second and third are not optional:

```text
python -m pip install .
python -m pip install https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl
python -m nltk.downloader wordnet omw-1.4
```

The linguistic model and the WordNet corpus are installed separately because
neither can be declared as a dependency. `en_core_web_sm` is not on a package
index, so naming it in `pyproject.toml` would require a direct URL reference,
and a public index rejects any distribution whose metadata carries one. WordNet
is corpus data rather than a Python package, so `nltk` ships the downloader and
not the corpus.

Skipping either step does not degrade an analysis quietly. The model loader
requires exactly the pinned version and raises `LinguisticModelError` on any
other, and canonicalization raises `WordNetDataError` when the corpus is
absent. Both name the command that fixes them.

## CLI

```text
lingity analyze review.md --profile architecture-review
lingity verify analysis.json

lingity critique review.md --output brief.json
lingity judge review.md --candidate rewrite.md
lingity improve review.md --provider subagent --candidate rewrite.md
```

`analyze` emits a deterministic, schema-valid JSON artifact containing located
findings, the attributed Human Readability Index, protected-element manifests,
and content/profile hashes. `verify` validates the schema and hashes, resolves
the recorded profile, and replays the analysis; altered or non-reproducible
artifacts fail explicitly. Both commands are pure and offline.

`critique`, `judge`, and `improve` drive rewriting. `critique` emits an
improvement brief — the ranked defects and the elements a rewrite may not
change. `judge` decides a single candidate. `improve` runs the bounded loop,
feeding each rejection back into the next brief. All three exit `0` on success,
`1` on a reasoned rejection, and `2` on an error, so a host agent can branch on
the exit code alone.

The current analyzer is a versioned English dependency-parse model covering
every deterministic signal published in the [DESIGN.md](DESIGN.md) dimension
table:

- **Sentence load** — words, clauses, punctuation depth, and actions per
  sentence.
- **Morphology** — nominalization density and weak verb constructions.
- **Noun stacking** — consecutive noun modifiers and hyphenated compound depth.
  A stack must be contiguous, and a named entity counts as one unit, so
  `Azure Kubernetes Service cluster` is two units rather than four and a
  person's name is never reported as a stack. Detection reads the dependency
  relation rather than the part-of-speech tag, because the tagger reads
  `messaging` in `messaging loss hypotheses` as a noun in one sentence and a
  verb in another. The finding reports `words` for the span and `units` for the
  naming units the threshold compares.
- **Agency** — agentless directives and missing explicit actor-action pairs.
- **Voice** — passive constructions and indirect predicates. Passive detection
  is structural: it requires a passive auxiliary or passive subject relation
  (`auxpass`/`nsubjpass`), so the active perfect (`has expired`) can never be
  mistaken for the passive (`has been approved`).
- **Lexical clarity** — jargon, uncommon compounds, and undefined abbreviation
  density.
- **Structure** — paragraph length, list suitability, and mixed-purpose
  sentences.
- **Redundancy** — repeated qualifiers, duplicated recommendations, and filler
  phrases. Lingity counts a repeated content word within one block, not across
  the whole text. Governance prose has to call one concept by one name in every
  section, so a term that recurs between sections shows consistency. Counting
  document-wide made a finding depend on wording far away from it. Joining clear
  paragraphs then manufactured findings that no paragraph had alone.

Every rule is block-scoped: the findings for a document are exactly the findings
of its blocks. A passage therefore scores the same alone as it does inside the
document that contains it.

A finding quotes source text the way the parser read it. The parser joins a
block's wrapped lines with a single space, so an observed value never carries a
line break or a list marker's indentation.

Noun stacking findings are reported under the `morphology` dimension and voice
findings under `agency`, so the score always resolves to the six weighted
dimensions.

Lingity reads Markdown structure before it parses prose. Headings, list items,
blockquotes, and paragraphs carry prose. Fenced code, indented code, tables, and
thematic breaks do not, so the analyzer never reads them. Each prose block
parses on its own, so a heading cannot glue itself onto the paragraph beneath it
and a table row cannot register as one long sentence. An identifier written
inside a code span is a name a rewrite must not change, so a finding falling
wholly inside one is dropped.

Each block is parsed as one unit, so a sentence wrapped across two source lines
stays one sentence and neither the wrap point nor the line-ending style changes
a score.

Block structure comes from `markdown-it-py`, which is CommonMark compliant and
tested against the specification's own suite. The parser is part of the analysis
contract exactly as the linguistic model is: its identity is published in every
artifact under `ingest`, and a major-version change is refused rather than
silently re-segmented. The artifact also publishes `unresolved_lines` and
`uncovered_lines`, so text that left the analysis is counted rather than lost in
silence. `verify` replays the segmentation.

Every finding carries a rule ID, severity, character location, observed value,
threshold, and remediation. Overlapping spans within a dimension are
de-duplicated so a single defect is not penalised twice.

Rules read sentence structure rather than surface strings, so detection
generalises to unseen wording. They inspect predicates, subjects, auxiliaries,
negation, and modifier chains. The parse is part of the analysis contract, so
Lingity pins the pipeline to `en_core_web_sm` at an exact version and loads it
fail-closed. It records the parser name, version, runtime, and digest as
`linguistic_model` inside the hashed artifact. `verify` refuses any artifact
produced by a different pipeline instead of silently re-analysing it.

The Human Readability Index weights those six dimensions and converts each
dimension's deducted points into a score with a half-life decay, so worse text
never scores higher than better text. The exact arithmetic is published in the
artifact's `score.formula` field.

Provider protocols exist for future proposal and semantic-challenge adapters,
but `analyze` and `verify` themselves perform no network or LLM calls.

See [DESIGN.md](DESIGN.md), the
[AgenticTuner comparison](docs/agentictuner-comparison.md), and the
[implementation plan](docs/implementation-plan.md).

## Rewriting

A model may *propose* a rewrite. Only deterministic code decides whether to
accept one. A candidate is accepted when, and only when, all of the following
hold:

- protected meaning is equivalent to the source,
- the Human Readability Index strictly improves,
- no new high-severity finding appears,
- and no semantic-drift challenge raised material doubt.

Lingity never accepts a regression, never accepts a tie, and never accepts an
unresolved meaning comparison. When nothing qualifies, it returns the source
text unchanged together with the reasons every candidate failed. It rejects a
candidate that scores a perfect 100 but drops a protected claim. A higher score
never buys a change in meaning.

Rejections are actionable. Every verdict carries `protected_delta`, naming the
exact elements dropped, introduced, or left unresolved, so the next attempt can
restore them by name instead of guessing:

```text
$ lingity judge source.txt --candidate shorter.txt
accepted False   70.46 -> 89.50
  reason: protected meaning is changed: 9 protected element(s) dropped
  MISSING quantity:count:2
  MISSING governance:term:ratify
  MISSING order:sequence:earlier=require closure evidence govern recommendation;later=target architecture return human decision
  ...
```

The gate compares meaning as propositions rather than as wording. It parses
each sentence into a claim signature: action, actor, target, modality,
polarity, and status. It also records the ordering relations between claims.
"Close the findings before sign-off" therefore agrees with "sign-off happens
only after the findings are closed". "Approve" and "ratify" do not agree. The
gate reads linking verbs as state claims. "The fix is complete and
fail-closed" therefore disagrees with "the fix is incomplete and fail-open".
No profile contains protected sentence patterns.

A held-out corpus of 32 pairs measures how well the gate generalises. It shares
no wording with any profile or fixture. The corpus documents the eight pairs
the gate does not resolve, with the linguistic reason for each. Every one of
them answers `unresolved` or `changed` rather than `equivalent`. The corpus is
evidence, not proof: it once masked a false `equivalent` on copular text behind
an unrelated coverage failure. Treat a passing corpus as a floor.

Providers are transports, never authorities:

- **`subagent`** (default) — no network and no API key. The host agent, such as
  Agency, *is* the model: Lingity hands it a brief, the host writes a candidate,
  and Lingity judges the result. Use `critique` and `judge` interactively, or
  pass `--candidate` files to `improve`.
- **`openai`** and **`anthropic`** — direct API calls over the standard library.
  Credentials come only from `OPENAI_API_KEY` and `ANTHROPIC_API_KEY`, and are
  never accepted as arguments, logged, or written to an artifact. There is no
  default model: omitting `--model` is an error rather than a guess.

A drift challenger may only *raise* doubt. It can block an acceptance, but it
can never clear a deterministic failure, and an unparseable challenge response
is an error rather than a quiet `no_material_change`.

## Development

```text
python -m pip install -e '.[dev]'
python -m pip install https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl
python -m nltk.downloader wordnet omw-1.4
python -m pytest
python -m mypy
python -m compileall -q lingity tests
```

These are the commands CI runs, in this order, on Python 3.11 and 3.12, for
every push to `main` and every pull request; see `.github/workflows/ci.yml`.
`tests/test_documentation.py` compares this block against the workflow and fails
if the two diverge. The command strings are identical; CI differs only in when
it runs the second one, skipping the corpora download when its cache restores
`~/nltk_data`. Locally that step is unconditional, and repeating it once the
corpora are present is a no-op. Each tool is invoked through `python -m` so it
runs under the interpreter that has the package installed rather than whichever
console script happens to be first on `PATH`, and the extras spec is quoted
because `zsh` treats the brackets as a glob. `pytest` and `mypy` take their settings from `pyproject.toml`, so no flags
are needed.

Analysis needs the spaCy model and the WordNet corpora present locally. Both are
install-time steps on purpose: nothing downloads anything at analysis time, so a
run cannot silently depend on the network or quietly change behaviour when a
corpus is missing. Missing data is an error, not a fallback.

The model is the pinned `en_core_web_sm` 3.8.0 wheel, installed by explicit URL
in its own step. It is deliberately not a declared dependency: it is not on a
package index, so declaring it would require a direct URL reference, and a
public index rejects any distribution whose metadata carries one. Do not
substitute `python -m spacy download en_core_web_sm`: that resolves whatever
model version is current at the time, and `lingity/nlp.py` rejects anything but
3.8.0. WordNet is not a Python distribution and stays a separate download.

WordNet drives morphology — deriving the verb behind a nominalization
("ratification" → "ratify") and separating a word from its antonyms — rather
than a hand-maintained suffix list.

## Prior art

The rule families follow published work on requirements and plain-language
quality:

- Femmer, Méndez Fernández, Wagner, Eder, *Rapid Quality Assurance with
  Requirements Smells* (Journal of Systems and Software, 2017) — the
  smell-detection framing behind nominalization, passive voice, and vague-term
  rules.
- INCOSE-TP-010-009, *Guide to Writing Requirements* (2019) — rules on
  imperatives, ambiguity, and quantification.
- U.S. Federal Plain Language Guidelines (PLAIN) — actor-first sentences, active
  voice, and short sentence targets.

No existing package was found that detects nominalizations, noun stacks, hidden
agency, or bureaucratic phrasing as attributed findings, or that gates a rewrite
on preserved governed meaning, so those are implemented here.

## License

Apache License 2.0. The full text is in [LICENSE](LICENSE), and [NOTICE](NOTICE)
carries the copyright statement and the attribution required by section 4(d).

The two data artifacts Lingity needs are not redistributed with it and are not
declared as dependencies. Each is downloaded by the installing user under its
own terms: the `en_core_web_sm` spaCy model under the MIT License, and the
NLTK WordNet corpus under the WordNet 3.0 License. `NOTICE` records both.

## Release

Publishing is one command, run from a machine whose twine is already
authenticated. No credential is read, written, or passed as an argument: twine
resolves them itself from `~/.pypirc`, the system keyring, or `TWINE_*`
environment variables.

```
python -m pip install -e ".[release]"
python scripts/release.py --repository testpypi --dry-run
python scripts/release.py --repository testpypi
python scripts/release.py --repository pypi
```

`--repository` is required. There is no default, because the difference between
the two indexes is not something to get wrong by omission.

The script refuses to publish anything it cannot verify. It stops on an unclean
working tree, a failing guard test, a version the index already holds, a direct
URL dependency, missing licence metadata, or an unknown classifier. Three of
those are invisible to `twine check`, which validates README rendering and
nothing else, and are otherwise answered for the first time by an HTTP 400 from
the index.

An unreachable index stops the release rather than reading as an absent version.
Uploads cannot be undone and a version number can never be reused, so a check
that cannot run must say so rather than fall silent.

`--dry-run` performs every check and builds the artifacts without uploading.
