Metadata-Version: 2.5
Name: mule-discovery
Version: 2.0.0
Summary: Scan Mule applications for migration complexity assessment
Project-URL: Homepage, https://github.com/KongHQ-CX/mule-discovery
Author: Stephen Brown
License-Expression: MIT
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.10
Requires-Dist: pyyaml>=6.0
Provides-Extra: anypoint
Requires-Dist: anypoint-sdk>=0.2.0; extra == 'anypoint'
Provides-Extra: dev
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Description-Content-Type: text/markdown

# mule-discovery

Scan Mule applications for migration complexity assessment.

Parses Mule 4 (and 3) XML source files, POM dependencies, DataWeave scripts, and API specifications to produce a structured migration readiness report with complexity scoring.

## Estate Analysis

The output produced by `mule-discover` (JSON or YAML) can be fed into the [estate-analyzer](https://github.com/KongHQ-CX/kong-ps-agent-skills/tree/main/mule-analysis/estate-analyzer) agent skill to generate pre-sales migration reports. The estate-analyzer processes discovery output across your entire Mule application estate to produce complexity summaries, connector frequency analysis, PoC candidate recommendations, and migration sizing reports.

## Quick Start (uv)

No install required - just run from the project directory:

```bash
cd mule-discovery

# Discover all Mule apps under a directory
uv run mule-discover /path/to/apps --output-dir ./inventory

# JSON output instead of YAML
uv run mule-discover /path/to/apps --json --output-dir ./inventory
```

`uv run` reads `pyproject.toml`, resolves dependencies into an ephemeral environment, and runs the command. Nothing is installed globally.

## Installation

### From PyPI with `uv` (recommended)

Install as a global CLI tool - `uv` puts the entry points on your `PATH`:

```bash
uv tool install mule-discovery

# Then run directly from anywhere
mule-discover /path/to/apps --output-dir ./inventory
```

Upgrade or remove later with:

```bash
uv tool upgrade mule-discovery
uv tool uninstall mule-discovery
```

For Anypoint Platform integration (policy scanning), install with the extra:

```bash
uv tool install "mule-discovery[anypoint]"
```

### From PyPI with `pip`

```bash
pip install mule-discovery

# Then run directly
mule-discover /path/to/apps --output-dir ./inventory
```

For Anypoint Platform integration (policy scanning):

```bash
pip install mule-discovery[anypoint]
```

### From source (development)

```bash
uv sync --extra dev
```

Requires Python 3.10+.

## CLI Tools

### `mule-discover`

Recursively find all Mule applications under a directory and produce migration complexity reports for each.

```bash
# Discover all apps, write YAML inventories (default) to ./inventory
uv run mule-discover /path/to/apps --output-dir ./inventory

# JSON output
uv run mule-discover /path/to/apps --json --output-dir ./inventory

# Suppress progress output
uv run mule-discover /path/to/apps -o ./inventory -q

# Custom flow-inventory complexity thresholds
uv run mule-discover /path/to/apps --flow-low 8 --flow-medium 18 --flow-high 30

# Custom scoring weights (bands, divisors, catalogue)
uv run mule-discover /path/to/apps --weights ./my-weights.yaml -o ./inventory

# Resolve parent POMs and external flow-refs against a customer's shared frameworks
uv run mule-discover /path/to/apps --common-libs /path/to/common_libs_home -o ./inventory

# Score an Exchange-hosted API spec the operator downloaded
uv run mule-discover /path/to/apps --specs /path/to/exchange_specs -o ./inventory

# Or pull Exchange-hosted specs with the Anypoint SDK first, in one step
uv run mule-discover /path/to/apps --specs-from-anypoint -o ./inventory
```

#### `--common-libs`

Customers often ship their Mule apps alongside a separate "common libs" / shared frameworks directory containing parent POMs (which the apps inherit dependencies from) and shared Mule projects defining flows that the apps reference via `flow-ref`. Without this directory, the discovery tool only sees what's local to each app and reports an empty connector list and unresolved flow-refs.

When you pass `--common-libs PATH`, the tool will:

- **Resolve parent POM chains.** Each app's `<parent>` coordinates are looked up in the directory; if found, the parent's `<dependencies>` are merged into the app's connector inventory (with `source_file` indicating which parent contributed each connector). Multi-level chains are walked.
- **Resolve external flow-refs.** Every `flow-ref` whose target is not defined inside the app is checked against an index of `<flow>` and `<sub-flow>` definitions found anywhere under the common-libs directory. Resolved refs include the library name, source file, and line number; unresolved ones are reported separately so they can be flagged for follow-up.
- **Pull in only what's referenced.** The common-libs directory may be large; the tool does not blanket-include everything from it. Only parent POMs the apps actually inherit from and flows the apps actually reference are surfaced.

Expected directory layout (one Mule project per top-level subdirectory):

```
common_libs_home/
├── hvcp-mule4-common-framework-handler/
│   ├── pom.xml
│   └── src/main/mule/*.xml
├── hvcp-mule4-common-messaging-framework/
│   └── ...
└── shared-parent-poms/
    └── pom.xml
```

The discovery output gains two new fields under the per-app inventory:

- `resolved_parent_poms`: list of parent POMs in the inheritance chain, each with `groupId`, `artifactId`, `version`, `resolved` (boolean), `source_file`, and the `dependencies` declared there.
- `external_flow_refs`: `{"resolved": [...], "unresolved": [...]}`. Each resolved ref shows the library and source file/line of the matching definition.

#### `--specs`

Customers often keep the RAML or OAS contract for an API in Anypoint Exchange rather than committing it to the app's source tree. When an `apikit:config`'s `raml`/`api` attribute does not resolve on disk, the tool checks the app's `pom.xml` (and its resolved parent chain, when `--common-libs` is also given) for a dependency shaped like an Exchange asset - an explicit `raml`, `oas`, `wsdl` or `raml-fragment` classifier, or a `zip`-typed dependency whose groupId looks like an Exchange organisation id (a UUID) or ends in `.exchange`. Without `--specs`, that becomes a specific, recoverable `api-spec-in-exchange` unknown naming the coordinates; a recoverable unknown is still listed and counted but does not widen the band.

Pass `--specs PATH` to score the spec instead. For each Exchange dependency found, the directory is searched, in order, for:

```
specs_home/
├── <artifact_id>/                              # unzipped asset directory
│   └── api.raml                                # or the single *.raml / *.yaml / *.json
├── <artifact_id>-<version>/                     # same, version-qualified
├── <artifact_id>-<version>-<classifier>.zip      # opened in place, not unzipped to disk
└── <artifact_id>.raml                            # (or .yaml / .json) bare file
```

The entry-point file is `api.raml` when present, else the single `.raml`, else the single `.yaml` or `.json`; its `!include` chain resolves inside the same directory or zip. A resolved spec is scored exactly like a RAML/OAS committed to the source tree. When the directory does not have the asset, the `api-spec-in-exchange` unknown returns, and its detail names the directory that was searched.

##### Specs from Exchange

When the specs directory doesn't exist yet, the Anypoint SDK the tool already uses for policy work can build it directly, in one of two ways.

**Two-step**: download once, reuse the directory across scans and other apps:

```bash
export ANYPOINT_CLIENT_ID=...
export ANYPOINT_CLIENT_SECRET=...

uv run mule-download-specs /path/to/apps --output-dir ./exchange_specs
uv run mule-discover /path/to/apps --specs ./exchange_specs -o ./inventory
```

`mule-download-specs` finds every app under `search_path` (and, with `--common-libs`, their resolved parent-chain poms too), collects the Exchange-hosted specs their poms declare, downloads each distinct one into `--output-dir` in the same layout `--specs` reads, and prints a JSON summary (coordinates, `status` - `downloaded`, `cached` or `failed` - and `path` per asset) to stdout. A directory that already has an asset is left alone and reported `cached`, so re-running the command after adding apps only downloads what's new. `ANYPOINT_ORG_ID` is not required here - Exchange downloads are addressed by group id, not organisation.

**One-step**: `--specs-from-anypoint` downloads into `<output-dir>/specs` and scores in the same command, as if `--specs` had named that directory:

```bash
export ANYPOINT_CLIENT_ID=...
export ANYPOINT_CLIENT_SECRET=...

uv run mule-discover /path/to/apps --specs-from-anypoint -o ./inventory
```

`--specs-from-anypoint` and `--specs` are mutually exclusive. Both require the `anypoint` extra; without it, the CLI exits with the same install hint `mule-download-policies` gives. Each per-app inventory's `summary.specs_source` records which of the three states applied: `"anypoint"`, `"directory"` or `"none"`.

**Naming an asset directly**: `--asset GROUP:ARTIFACT:VERSION[:CLASSIFIER]` (repeatable) downloads one Exchange asset by its coordinates, without it being a pom dependency - useful for an asset a demo organisation or a customer published but never wired into a pom, or to exercise the resolved path on an app whose pom lacks it. `CLASSIFIER` defaults to `raml`; accepted values are `raml`, `oas`, `wsdl`, `raml-fragment`. With `mule-download-specs`, `search_path` becomes optional once at least one `--asset` is given:

```bash
uv run mule-download-specs --asset 68ef9520-24e9-4cf2-b2f5-620025690913:combined-api:1.0.2
uv run mule-download-specs /path/to/apps --asset 68ef9520-24e9-4cf2-b2f5-620025690913:combined-api:1.0.2:oas --output-dir ./exchange_specs
```

`--specs-from-anypoint` takes the same `--asset` flag, so the one-step form can name an asset too:

```bash
uv run mule-discover /path/to/apps --specs-from-anypoint --asset 68ef9520-24e9-4cf2-b2f5-620025690913:combined-api:1.0.2 -o ./inventory
```

An `--asset` spec is merged with the pom-derived ones and deduplicated on the same coordinates; the JSON summary row for each asset carries `"source": "asset-flag"` or `"source": "pom"` so it's clear which named it.

#### Connector discovery without a parent POM

When an app's `pom.xml` declares no dependencies (typical when everything is inherited from a parent POM that isn't on disk), the tool falls back to deriving connectors from the `xmlns:` declarations in the Mule XML files themselves. These entries are tagged with `notes: "derived from XML namespaces - parent POM not resolved"` and `source_file: "(xml-namespaces)"` so the source of each connector remains traceable. Pass `--common-libs` to replace this fallback with proper parent-POM resolution where possible.

Each per-app report includes:
- Flow inventory with complexity levels (LOW / MEDIUM / HIGH / VERY_HIGH)
- DataWeave transformation analysis and classification
- HTTP listener and scheduled job detection
- Connector inventory with migration weights
- API specification detection (OpenAPI, WSDL)
- External dependency and out-of-scope item tracking
- AWS service usage (SQS, S3, DynamoDB)
- SOAP/WSDL service detection
- HTTP request-config inventory and connector authentication metadata (`request_configs`, `connector_auth`)
- Effort points, band (S / M / L / XL, with a `+` when unknowns widen the reading), feature and feature-zero breakdown, and the unknowns list, each marked `recoverable` when a known input (`--specs`, `--common-libs`) would resolve it
- Cross-app hops: outbound HTTP hosts resolved through the property files and matched against the other applications in the same scan
- Per feature: `closure` (its flow and sub-flow names, including router wrappers and global-handler flows), `shared_flows` (the closure names also reached by another feature) and `outbound` (one row per distinct outbound call - `kind`, `config_ref`, and, for HTTP, the matched hop's `host`, `target_app` and `external`)
- `summary.outbound_system_count`: the distinct outbound hosts and connector configs across every feature
- The 0 to 100 compatibility index and its SMALL / MEDIUM / LARGE / XLARGE label, derived from the points

### `mule-scan-policies`

Scan Anypoint Platform for API policies on deployed applications. Requires the `anypoint` extra.

```bash
pip install -e ".[anypoint]"

export ANYPOINT_CLIENT_ID=...
export ANYPOINT_CLIENT_SECRET=...
export ANYPOINT_ORG_ID=...
export ANYPOINT_ENV_ID=...

uv run mule-scan-policies
uv run mule-scan-policies --format json
```

### `mule-download-policies`

Download custom policies from Anypoint Exchange. Requires the `anypoint` extra.

```bash
export ANYPOINT_CLIENT_ID=...
export ANYPOINT_CLIENT_SECRET=...
export ANYPOINT_ORG_ID=...

uv run mule-download-policies --output-dir ./custom_policies
```

### `mule-download-specs`

Download every Exchange-hosted API spec an app estate's poms declare, into the layout `mule-discover --specs` reads. Requires the `anypoint` extra. See "Specs from Exchange" above for the full two-step and one-step forms.

```bash
export ANYPOINT_CLIENT_ID=...
export ANYPOINT_CLIENT_SECRET=...

uv run mule-download-specs /path/to/apps --output-dir ./exchange_specs
```

## Complexity Scoring

Each application is sized in **effort points**: the work in each feature,
plus a one-off foundation, plus a count of the things the scanner can see
but cannot size. The full model, including every catalogue entry and the
reasoning behind it, is in [docs/scoring-model.md](docs/scoring-model.md).

The shape of the number is:

```
app points = sum(feature points) + feature zero points
```

- A **feature** is one unit of migration work: one top-level resource
  family for an APIkit API (`/orders` covers `get:\orders`, `post:\orders`
  and `get:\orders\(id)\lines`), or one listener, scheduler or subscriber
  for everything else. Each feature owns the transitive set of flows its
  entrypoints reach through `flow-ref`.
- Every artefact a feature reaches - the flow XML itself, DataWeave, Java,
  scripts, SQL, WSDL and XSD, the RAML or OAS - is measured the same way:
  effective lines over a per-type divisor, plus one point per known-bad
  construct found in it.
- Fixed per-item points are added for each endpoint, outbound call,
  error-handler branch and cross-app hop.
- **Feature zero** is the one-off foundation: connector configs, property
  keys, the global error handler, the API contract, shared DataWeave
  modules and the parent POM chain, plus a fixed base added once per app
  for the skeleton, CI and connector clients a migration builds before its
  first feature. An app with no flow, sub-flow or connector config gets no
  base.

### Bands

The band is read straight off the app total:

| Band | Points | Recommendation |
|---|---|---|
| S | under 15 | SMALL |
| M | 15 to under 40 | MEDIUM |
| L | 40 to under 90 | LARGE |
| XL | 90 and over | XLARGE |

A band gains a `+` (for example `L+`) when the unknowns count exceeds
`unknown_plus_threshold`, or more than a quarter of the feature count. The
plus means the reading is wider than the number: there are constructs the
scanner recognised but could not size.

### Unknowns

An unknown never adds points; it is listed with `file:line` and counted.
Dynamic `flow-ref`s, flow-refs that resolve nowhere, `http:request` hosts
that no property file resolves to a literal, DataWeave modules and Java
classes named but not found, unmeasured scripting engines, undefined
property keys and unresolved parent POMs all land here.

### The compatibility index

`score_result.score` is still a 0 to 100 number and
`score_result.recommendation` is still SMALL / MEDIUM / LARGE / XLARGE, so
reports written against the old output keep reading. The index is now
derived from the points on a fixed piecewise-linear curve (100 at 0 points,
75 at the S/M boundary, 50 at M/L, 25 at L/XL, 0 at 200) rather than being
the headline in its own right.

### `--weights`

Every number the model uses - bands, divisors, per-item points,
feature-zero points and the whole known-bad catalogue - lives in one
versioned YAML file shipped with the tool. Override it per run:

```bash
uv run mule-discover /path/to/apps --weights ./my-weights.yaml -o ./inventory
```

An override file only needs the top-level keys it changes; every key it
omits falls back to the shipped default. The `weights_version` in force is
stamped into every output file and into `summary.weights_version`. Two
outputs are comparable only when their weights versions match.

### Flow Complexity Thresholds

The flow inventory keeps its own component-count label, separate from the
effort points (configurable via CLI flags):

| Components | Complexity |
|---|---|
| 6 or fewer | LOW |
| 7 to 14 | MEDIUM |
| 15 to 25 | HIGH |
| more than 25 | VERY_HIGH |

### DataWeave Classification

Each DataWeave inventory entry keeps its `complexity` and `classification`
fields, now read off the same measurement the score uses:

| Field | Value | Criteria |
|---|---|---|
| complexity | LOW / MEDIUM / HIGH | under `dataweave_labels.medium` points / under `dataweave_labels.high` / at or above it |
| classification | business_logic | a user-defined `fun`, or one of the trap constructs |
| classification | field_level_logic | any other catalogue hit (`lookup`, `readUrl`, `java!`, a non-core `import`) |
| classification | simple_mapping | no catalogue hit |

The two boundaries are points, so they only mean anything against the
DataWeave divisor they were drawn for. Both live beside that divisor in
`data/weights.yaml` under `dataweave_labels`, and a `--weights` override
moves them with it. The shipped pair is 0.15 and 0.4, which reads the
reference corpus as roughly 64 percent LOW, 28 percent MEDIUM and 8
percent HIGH.

## Package Structure

```
src/mule_discovery/
├── __init__.py                # Main discover_mule_app() orchestrator
├── constants.py               # XML namespaces, element classifications, connector weights
├── xml_helpers.py             # XML utility functions
├── models/                    # Data models (dataclasses)
│   ├── result.py              # DiscoveryResult (top-level container)
│   ├── flows.py               # FlowInfo, BatchInfo, ChoiceInfo, ScatterGatherInfo, ...
│   ├── connectors.py          # ConnectorInfo, SpringDependency
│   ├── dataweave.py           # DataWeaveInfo
│   ├── listeners.py           # HttpListenerInfo, ScheduledJobInfo
│   ├── dependencies.py        # ExternalDependencyInfo, SourceFiles, OutOfScopeItem
│   ├── schemas.py             # ApiSpecInfo (OpenAPI, WSDL)
│   └── scoring.py             # ComplexityThresholds, ScoreResult
├── data/
│   └── weights.yaml           # Default scoring weights and known-bad catalogue
├── parsers/                   # File IO → models
│   ├── file_discovery.py      # find_mule_apps(), find_mule_xml_files()
│   ├── mule_xml.py            # Mule XML parsing (flows, listeners, jobs)
│   ├── pom.py                 # POM parsing (app name, version, connectors, parent-POM chain resolution)
│   ├── common_libs.py         # Indexing of customer-supplied common-libs (flows + parent POMs)
│   ├── http_auth.py           # HTTP auth config extraction
│   ├── dataweave.py           # DataWeave script parsing
│   ├── soap.py                # SOAP/WSDL service detection
│   ├── aws.py                 # AWS service detection (SQS, S3, DynamoDB)
│   ├── openapi.py             # OpenAPI spec detection
│   └── wsdl.py                # WSDL parsing utilities
├── analysis/                  # Models → models (pure functions)
│   ├── classification.py      # Flow type and source category constants
│   ├── complexity.py          # Flow complexity, DataWeave inventory labels
│   ├── patterns.py            # Pattern detection (async, scatter-gather, choice, ...)
│   └── dependencies.py        # External dependency and out-of-scope extraction
├── scoring/                   # The effort-points model (docs/scoring-model.md)
│   ├── weights.py             # The versioned calibration file loader
│   ├── tokenize.py            # Effective lines, comment and string stripping
│   ├── measure.py             # The one rule: size points plus known-bad hits
│   ├── features.py            # Entrypoints, resource families, flow closure
│   ├── artefacts.py           # Closure to the files it depends on
│   ├── properties.py          # Property files and placeholder resolution
│   ├── hops.py                # Cross-app hops resolved through properties
│   ├── feature_zero.py        # The one-off foundation, sized per app
│   ├── unknowns.py            # Everything seen but not sizeable
│   ├── specs.py               # Exchange-hosted spec recognition + --specs resolution
│   └── assemble.py            # Points, bands, the compatibility index
├── output/                    # Models → formatted strings
│   ├── yaml_output.py         # YAML
│   ├── json_output.py         # JSON
│   └── text_output.py         # Human-readable text summary
├── anypoint/                  # Anypoint Platform integration (optional)
│   ├── policies.py            # Policy scanning
│   ├── exchange.py            # Custom policy download
│   └── specs.py               # Exchange-hosted API spec download
└── cli/                       # CLI entry points (thin wrappers)
    ├── discover.py            # mule-discover
    ├── scan_policies.py       # mule-scan-policies
    ├── download_policies.py   # mule-download-policies
    └── download_specs.py      # mule-download-specs
```

### Design Principles

- **No function does both IO and computation.** Parsers read files → return models. Analysis takes models → returns models. Output takes models → returns strings.
- **All data models are plain dataclasses** with typed fields - no methods with side effects.
- **All analysis functions are standalone** - no class methods, no inheritance.
- **Each output format is a separate module.**

## Testing

```bash
make test
```

Or directly:

```bash
uv run --extra dev python -m pytest
```

Coverage is enforced at 70% (branch coverage) via `pyproject.toml`.
