Metadata-Version: 2.4
Name: hiddenshadow
Version: 0.11.1
Summary: License + SBOM + dependency analyzer for OSS source trees.
Author: DarkShadowSec LLC
License: Proprietary - DarkShadowSec LLC Internal
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click>=8.1
Requires-Dist: rich>=13.0
Requires-Dist: pyyaml>=6.0
Provides-Extra: test
Requires-Dist: pytest>=7; extra == "test"
Provides-Extra: web
Requires-Dist: flask>=3.0; extra == "web"
Requires-Dist: openpyxl>=3.1; extra == "web"
Requires-Dist: python-pptx>=1.0; extra == "web"
Requires-Dist: pillow>=10.0; extra == "web"
Dynamic: license-file

# HiddenShadow

**License + SBOM + CVE + dependency analyzer for OSS source trees.**

A standalone tool that scans any directory tree of cloned source
repositories and produces:

- **License detection** — finds LICENSE/COPYING/NOTICE files; identifies
  SPDX licenses; classifies them per a configurable policy
  (permissive / weak-copyleft / strong-copyleft / network-copyleft /
  source-available / public-domain).
- **SBOM** — emits CycloneDX 1.5 JSON per repo from package-manager
  manifests (npm, PyPI, Cargo, Go, Maven, Gradle, Composer).
- **Library rollup** — aggregates every direct dependency across all
  repos; flags top recurring deps for license review.
- **Network license lookup** — resolves declared licenses for thousands
  of deps via the deps.dev v3 API (with persistent cache).
- **CVE scan** — queries OSV.dev for vulnerabilities against every
  (package, version) pair in the rollup.
- **Dual-license detection** — recognizes sibling LICENSE+COPYING pairs
  with different SPDXs and SPDX-License-Identifier expressions inside
  COPYING.md narratives, demoting findings appropriately.
- **METADATA-aware suppression** — when a fork has a METADATA.yaml that
  documents the disposition of a non-permissive file, scan findings are
  demoted to a "Documented in METADATA" info entry. Enables CI to surface
  only NEW findings since fork import.
- **Cleartext-secret detection** — flags hardcoded credentials in
  source/config files: private-key blocks, AWS / GitHub / Google /
  Stripe / Slack / Twilio / SendGrid / npm / PyPI tokens, connection
  strings with embedded passwords, JWTs, and generic
  `password = "..."` / `api_key: '...'` style assignments. Matches are
  redacted in the findings file (only first/last 3 chars kept) so the
  artifact itself is safe to share. Disable with `--no-secrets`.
- **Findings + recommendations** — every observed issue is recorded
  with severity (`critical`/`high`/`medium`/`low`/`info`), location,
  detail, and a concrete recommended fix.

Owned and maintained by **DarkShadowSec LLC**. Renamed in v0.4 and
split into a standalone tool.

## Install

```powershell
cd "E:\New Apps\HiddenShadow"
pip install -e .
```

Or run directly without install:

```powershell
python -m hiddenshadow --help
```

## Commands

```
hiddenshadow scan <path>          # license + SBOM + finding pipeline
hiddenshadow lookup-licenses ...  # resolve unknown licenses via deps.dev
hiddenshadow scan-cves ...        # OSV vulnerability scan
hiddenshadow classify <SPDX>      # show policy decision for one SPDX id
hiddenshadow licenses             # full policy table
hiddenshadow findings ...         # print last-scan findings
```

## Common runs

```powershell
# Scan the DarkShadowSec source mirror
hiddenshadow scan "E:\New Apps\DarkShadowSec Source Apps"

# Scan with strict CI-mode — exit non-zero on any new high/critical
hiddenshadow scan "E:\path\to\third_party" --fail-on high

# Force the raw view (don't suppress METADATA-documented findings)
hiddenshadow scan "E:\path\to\third_party" --ignore-metadata

# Resolve licenses for the rollup (network — uses deps.dev)
hiddenshadow lookup-licenses --rollup .\library-rollup.csv

# CVE scan via OSV.dev (network)
hiddenshadow scan-cves --rollup .\library-rollup.csv
```

## Layout the scanner expects

```
<source_root>/
├── <category>/
│   ├── <repo-1>/         (with LICENSE / METADATA.yaml / .git etc.)
│   └── <repo-2>/
├── <repo-3>/             (or a repo can sit directly under source_root)
└── ...
```

A repo is detected by the presence of any of: `.git`, `METADATA.yaml`,
`LICENSE`, `LICENSE.txt`, `LICENSE.md`, `COPYING`, `BUILD.darkshadow`.

## Output

After a scan, `<output_root>` (defaults to `<source_root>`) contains:

```
<output_root>/
├── _logs/
│   ├── scan-findings.json     # full findings + summaries
│   ├── scan-summary.csv       # per-repo row
│   ├── library-rollup.csv     # every direct dep across all repos
│   └── cve-findings.json      # if scan-cves was run
├── sboms/
│   ├── <repo>.cdx.json
│   └── ...
└── _cache/                    # network-fetch caches; safe to delete
    ├── depsdev/
    └── osv-dev/
```

## Severity guide

| Severity | When |
|---|---|
| **critical** | network-copyleft (AGPL) or source-available (BUSL/SSPL) license inside an authoritative location of a permissive-licensed repo |
| **high** | strong-copyleft (GPL) license in an authoritative location; source files claim non-permissive license; LICENSE missing entirely |
| **medium** | weak-copyleft (MPL/LGPL) license inside an authoritative path; root LICENSE present but unrecognized |
| **low** | reserved |
| **info** | non-permissive license in a docs/test path (likely fixture/data); deps needing license lookup; mixed-but-compatible SPDX headers; **METADATA-documented dispositions** |

## Configuration

The license policy table lives at `hiddenshadow/licenses.py`. To
support a new license, add a regex pattern to `LICENSE_PATTERNS` and a
classification entry to `LICENSE_CLASS`. Restart any installed
process after editing.

## Extending

| Module | Responsibility |
|---|---|
| `hiddenshadow.licenses` | SPDX patterns + classification policy |
| `hiddenshadow.manifests` | package-manager manifest parsers |
| `hiddenshadow.dual_license` | dual-license pattern recognition |
| `hiddenshadow.metadata_aware` | METADATA.yaml-driven finding suppression |
| `hiddenshadow.findings` | analysis rules — license, SBOM, dep |
| `hiddenshadow.secrets` | cleartext-credential / API-key detection |
| `hiddenshadow.sbom` | CycloneDX 1.5 emission |
| `hiddenshadow.network` | shared HTTP+cache for network calls |
| `hiddenshadow.registry` | deps.dev lookup |
| `hiddenshadow.osv` | OSV.dev vulnerability scan |
| `hiddenshadow.scan` | orchestration |
| `hiddenshadow.cli` | click-based CLI |

## Tests

```powershell
cd "E:\New Apps\HiddenShadow"
pip install -e ".[test]"
pytest tests/
```

## Known limitations

- **No registry-resolution for transitives.** We only parse direct deps
  declared in package-manager manifests. For full transitive coverage,
  run a real SBOM tool (Syft) against compiled artifacts and feed
  `hiddenshadow scan-cves` with the resulting SBOM.
- **Maven dep resolution is shallow.** Real Maven resolves
  `<dependencyManagement>` and properties; we just enumerate explicit
  `<dependency>` blocks.
- **SPDX header sampling capped at 200 files per repo** for performance.
- **Heuristic license detection** — custom licenses (MITRE, DRL) are
  recognized but unusual ones may not be — they show as "Root LICENSE
  present but unrecognized" findings, prompting manual review.

## License

HiddenShadow itself is proprietary to **DarkShadowSec LLC** — see the
`LICENSE` file at the repo root. The OSS projects it scans retain their
own licenses; HiddenShadow does not change them.
