Metadata-Version: 2.4
Name: netscraper
Version: 0.1.0
Summary: Automated network pentest check orchestration -- port-gates a target list, runs the relevant checks (TLS, SSH, SMB, LDAP, RPC, DNS, MSSQL, SNMP, FTP, HTTP, DIRSEARCH, CORS), and produces per-finding POC screenshots plus a styled Excel report.
Author: Wati Cyber
License: MIT
Keywords: pentest,penetration-testing,security,vulnerability-scanner,network-security,recon,tls,ssh,smb,cors
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Information Technology
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: MacOS
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Security
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: playwright>=1.40
Requires-Dist: ansi2html>=1.9
Requires-Dist: ssh-audit>=3.2
Requires-Dist: impacket>=0.12.0
Requires-Dist: openpyxl>=3.1
Requires-Dist: dirsearch>=0.4.3
Dynamic: license-file

# NetScraper

Automates network pentest checks against a list of IPs, hostnames, or CIDR
subnets, and produces per-finding POC screenshots sorted into folders —
ready to drop straight into a report appendix.

```bash
pip install netscraper
playwright install chromium   # one-time, for POC screenshots
netscraper --targets targets.txt
```

For authorized penetration testing and security research only — see the
legal banner the tool itself prints on every run. See "Requirements" below
for the external command-line tools (`sslscan`, `nmap`, etc.) each check
shells out to; `pip install` alone gets you the `netscraper` command and
its Python dependencies, not those OS-level tools.

Runs **vuln-first, not host-first**: every check's required port (TLS's own
port, SSH/22, SMB/445, LDAP/389, RPC/445, MSSQL/1433, FTP/21, HTTP/80) gets
port-gated (a fast parallel TCP connect check) across the *whole* target
list in one unified scan, up front, before anything else happens. UDP-only
checks (DNS, SNMP) skip the TCP pre-gate entirely — there's nothing
meaningful to "nc -z" for a UDP service, so the real protocol probe itself
is the gate.

That port scan's real, live open/closed counts then drive an interactive
menu — pick TLS, SSH, SMB, ..., any combination, or All — so you run
exactly the check(s) that actually apply to this target list instead of
always running all ten. Different engagements need different things: a
quick TLS-only re-test after a cert renewal doesn't need to sit through
SMB/LDAP/RPC checks against hosts that don't even run those services. Once
you've picked, only hosts that actually answered on the relevant port get
the real protocol-specific probe — a host without SMB open never gets an
SMB check attempted, it's recorded `N/A`, not scanned, not guessed at, and
never lumped in with a "safe" result it didn't earn.

When you kick off a scan it immediately tells you how many targets were
loaded, then shows the port-scan results, then the mode picker. During a
**live** scan it pauses 5 seconds between checks once one finishes and
before the next starts (`--mode-pause`, default 5, set to 0 to disable) —
a courtesy pause between check types, not between individual hosts.

For scripted/automated use, skip the interactive picker entirely with
`--modes` (comma-separated, e.g. `--modes tls,ssh`, or `--modes all`).

## Scans several hosts at once, automatically

Once a check's port-gating is done, the actual per-host work (running the
real tool, saving evidence, rendering the POC) runs against **several hosts
at the same time** instead of one at a time — this is what keeps a target
list of dozens or hundreds of hosts from taking as long to scan as it would
running host-by-host. There is no flag, prompt, or setting for this
anywhere: the moment you give the tool a target list, it works out how many
hosts to run in parallel from the machine it's running on (RAM and CPU
count) and just does it. You'll see a line like:

```
[+] Parallel scan workers: 6 (auto-sized for this machine)
```

right before each check starts, and results stream in as each host
finishes — not all at once at the end, and not silently, so a long-running
scan against 100+ hosts still shows visible progress the whole way through
instead of looking stuck.

This has to be a pool of **processes**, not threads: the pty-based capture
used for sslscan/ssh-audit output (`core/terminal_capture.py`) calls
`os.fork()` directly, which is unsafe to do from a multi-threaded process,
and Playwright's browser automation API is single-thread-only. A process
pool sidesteps both, since every worker is a fully independent OS process.
Each worker process launches its own Chromium instance for POC rendering
the first time it needs one, then reuses that same instance for every
remaining screenshot it renders for the rest of the run — the same "launch
once, reuse many times" design as a single-process run (see the POCs
section below), just scoped per worker process instead of per whole run,
since a Playwright browser object can't be handed from one process to
another.

The worker count is picked once per run and capped in both directions: at
least 2, never more than 8 regardless of how much RAM a machine reports
(more than that mostly just means more processes competing for the same
network link and the same target host's own connection limits, not real
extra throughput), and never more than the number of CPU cores actually
available (oversubscribing a 2-core box with 6+ worker processes measurably
*slowed things down* in testing — more workers than cores buys you context-
switching overhead, not parallelism). Within that range it's sized from
total system RAM (roughly one worker per 1.5 GB, since each worker can be
holding open a Chromium instance alongside whatever external tool that
check shells out to) — a fairly typical 8–16 GB laptop lands in the 5–8
worker range this way. If total RAM can't be detected at all (unsupported
platform), it falls back to a conservative 8 GB assumption rather than
guessing high.

A single worker pool is created once per run and reused across every check
you pick and every round of the interactive check-menu loop below — so a
worker's already-launched Chromium instance keeps paying off as you run
TLS, then SSH, then SMB against the same target list, instead of being
torn down and relaunched between checks.

## The tool never just exits on you

When run interactively (bare `netscraper`, or with everything except a
target given up front), finishing a check drops you straight back into the
**same check menu, against the same target list** — not back to the
Individual Mode / Bulk Scanning / Exit menu, and not back to your shell.
Run TLS, look at the results, then pick SSH next against the exact same
hosts without re-entering a target or a file path a second time.
`summary.csv` and `PT_Scan_Report.xlsx` accumulate across every round in
that same output folder, so by the time you're done picking checks, one
report has everything you ran in this session — not one report per check.

When you're actually done with this target list, either type **`back`** (or
its number, always the entry right after "All checks") at that check menu,
or just hit Ctrl-C there — both take you back to the Individual Mode / Bulk
Scanning / Exit menu so you can point the tool at something else. The only
way the tool actually quits the whole process is choosing **Exit** at *that*
menu (or Ctrl-C *at* that exact menu).

Anywhere else — entering a target, picking check modes, or a scan that's
actively running — Ctrl-C means "go back", not "crash". Hit it mid-scan and
you'll see whatever results were already collected get saved (both
`summary.csv` and `PT_Scan_Report.xlsx`, same as a normal finish) before
you're bounced back to the check menu for this target list. The same
save-first behavior also covers the truly unexpected case — a POC render
hiccup, or anything else that wasn't already turned into a clean per-host
`ERROR` row — so one bad screenshot on host #47 of a 200-host run costs you
that one row, never the rest of the scan or everything found before it.

(This looping behavior is for interactive use only. A scripted invocation
with `--targets`/`--target` and `--modes` given on the command line runs
that fixed set of checks once and exits, same as always — no menu, no loop
— since there's no interactive session to loop back into.)

## What it checks

**TLS check** (`SSL_Scan/`, gates on each target's own port, default 443):
- **Expired SSL Certificate** — certificate `notAfter` date is in the past.
- **Deprecated SSL/TLS Protocol(s) Enabled** — SSLv2/SSLv3/TLSv1.0/TLSv1.1 accepted.
- **Weak Cipher Suite(s) Accepted** — sub-128-bit, RC4/DES/3DES/NULL/EXPORT/MD5/anon-*,
  or legacy SHA-1 MAC suites (`*-SHA`, not `*-SHA256`/`*-SHA384`).
- **Weak Certificate Key Strength** — RSA/DSA under 2048 bits, EC under 224 bits.
- **Certificate Hostname Mismatch** — none of the cert's CN/SAN entries match
  the scanned hostname (skipped when scanning by bare IP).
- **Untrusted / Self-Signed SSL Certificate** — a genuinely self-signed cert
  *or* a complete chain that resolves to a CA the system doesn't trust.
- **Incomplete SSL/TLS Certificate Chain** — the *server* fails to send the
  intermediate certificate(s) needed to build a chain (see "False-positive
  avoidance" below for how this is told apart from the untrusted-CA case).
- Zero findings → **SAFE** POC in `SSL_Scan/Secure_POC/`.

**SSH check** (`SSH/`, gates on port 22):
- **Weak / Deprecated Cryptographic Configuration (SSH)** — anchored on
  `ssh-audit`'s own "critical" recommendation tier, not its "warning" tier
  (which also flags purely forward-looking things like post-quantum
  readiness on an already-patched server).
- **SSH Banner / Version Disclosure** — near-universal, low-severity finding
  (any reachable SSH server discloses this pre-auth). A host can't get a
  pure SAFE verdict from this check for that reason alone.

**SMB check** (`SMB/`, gates on port 445):
- **SMBv1 Enabled** — `nmap smb-protocols` reports the legacy NT LM 0.12
  (SMBv1) dialect accepted.
- **SMB Signing Not Enforced — NTLM Relay Exposure** — read directly off
  `nmap`'s own `smb-security-mode` (legacy/SMB1) or `smb2-security-mode`
  (SMB2/3) verdict text: "disabled", or "enabled but not required".
- **SMB Null Session / Anonymous Access** — a real `smbclient -L //host -N`
  null-session share *listing* actually succeeds (not just an anonymous
  session being established, which is normal SMB protocol behavior on
  almost any server — see below).
- Signing status the scripts didn't recognize → **Needs Manual
  Verification**, not a guessed Safe/Vulnerable.

**LDAP check** (`LDAP/`, gates on port 389):
- **LDAP Anonymous Bind Enabled** — an unauthenticated `ldapsearch` bind +
  base search actually succeeds.
- **LDAP Signing Not Enforced** — a `ldapsearch` simple bind with a bogus
  DN/password over plaintext gets **Invalid credentials (49)** (the server
  evaluated the credentials over an unsigned channel) rather than
  **Confidentiality required (13)** / **Stronger authentication required
  (8)** (the server refused before even looking at credentials — signing
  enforced). Works with zero valid creds; see "False-positive avoidance".
- Unrecognized bind result → **Needs Manual Verification**.

**RPC check** (`RPC/`, gates on port 445):
- **Exposed MSRPC Services / Endpoint Mapper** — `rpcdump.py` (Impacket)
  enumerates 1+ interfaces registered with the endpoint mapper via an
  unauthenticated query.
- **MS-SCMR and atsvc Exposed over RPC** — among whatever's registered,
  specifically the Service Control Manager Remote Protocol (MS-SCMR) or
  Task Scheduler/AT-Scheduler (MS-TSCH/atsvc) interfaces.
- A genuine transport failure (as opposed to a clean, well-formed "no
  endpoints registered" answer) → **Needs Manual Verification**.
- **Honesty note**: this check's tooling and parsing are real (validated
  against a real Samba fileserver's genuine DCERPC responses), but a
  *populated* endpoint list (i.e., actually seeing MS-SCMR/MS-TSCH
  registered) requires a real Windows host or AD DC, which wasn't available
  to stand up in this build environment. Test against a real Windows target
  before relying on this for a report.

**DNS check** (`DNS/`, UDP, no TCP pre-gate, port 53):
- **DNS Recursion Enabled** — a `dig` query for an arbitrary non-local
  domain comes back with the `ra` (Recursion Available) flag set.
- No usable response at all → **Needs Manual Verification**.

**MSSQL check** (`MSSQL/`, gates on port 1433):
- **MSSQL Service Information Disclosure** — `nmap ms-sql-info` gets back
  instance name / version / TCP port / clustering info pre-auth.
- **Honesty note**: no real MS SQL Server was available in this build
  environment to validate the *positive* detection path live; the negative
  path (closed/non-MSSQL port → no disclosure) was validated for real.
  Verify against a genuine MSSQL instance before relying on this check.

**SNMP check** (`SNMP/`, UDP, no TCP pre-gate, port 161):
- **Weak / Default SNMP Community String** — `onesixtyone` against a
  bundled wordlist of common default/weak community strings
  (`wordlists/snmp_communities.txt`) gets a real reply for 1+ of them.

**FTP check** (`FTP/`, gates on port 21):
- **Anonymous FTP Access** — a real scripted `ftp` client session logging
  in as `anonymous` gets back `230 Login successful.` (not `530 Login
  incorrect.`).

**HTTP check** (`HTTP/`, gates on port 80):
- **Server / Service Version Disclosure** — the `Server:` response header
  contains an actual version number (`nginx/1.18.0`, not just `nginx`).
- **Host Header Injection** — request sent with `Host: bing.com`; flagged
  only when the response is a redirect (3xx) whose `Location` header
  reflects `bing.com` back (proves the app builds absolute URLs from the
  attacker-controlled Host header instead of a fixed canonical hostname).
- **Cross-Site Tracing (XST)** — a `TRACE` request carrying a unique
  `X-PT-XST-Marker` header; flagged only when that exact header is echoed
  back verbatim in the response body (proves TRACE is enabled and reflects
  requests — the classic XST primitive for defeating HttpOnly via a
  co-hosted script).

**DIRSEARCH check** (`Dirsearch/`, port(s) you choose, not fixed):
- Unlike every other check, DIRSEARCH has no single fixed port — after you
  select it (interactively, or via `--modes dirsearch`), you're asked for
  one or several ports (e.g. `80` or `80,443,8080`; scripted runs pass
  `--dirsearch-ports` instead of being prompted), and only *those* get a
  quick TCP gate across the target list. Every host:port that answers gets
  a real `dirsearch` brute-force run in quiet mode, already asked (via
  dirsearch's own `-i`/`--exclude-sizes` flags) for only status
  `200`/`301`/`302` and never a 0-byte response — then the real captured
  output is independently re-checked against that exact same filter, so
  nothing slips through just because a flag didn't do what it was supposed
  to. **No POC screenshot is rendered for this check** — the filtered
  "Found" lines are saved straight to a `.txt` evidence file (and rolled up
  into `PT_Scan_Report.xlsx` the same as every other check: one red
  "Vulnerable" row per host:port with anything found, one green "Not
  Vulnerable" row per host:port with nothing).

**CORS check** (`CORS/`, port(s) you choose, not fixed):
- Same "no fixed port" shape as DIRSEARCH — pick it (interactively, or via
  `--modes cors`), then give it one or several ports (`--cors-ports` for a
  scripted run); only those get a quick TCP gate across the target list.
- **CORS Misconfiguration — Arbitrary Origin Reflected** — every qualifying
  host:port gets a real `curl` request carrying `Origin: https://evil.com`
  (a domain that could never legitimately be on any real allow-list).
  Flagged **only** when the server's own `Access-Control-Allow-Origin`
  response header echoes that exact value back verbatim — a static `*`
  wildcard, a different real allow-listed origin, or no CORS header at all
  are all correctly left Safe, since none of those is the server actually
  reflecting our attacker-controlled input. If
  `Access-Control-Allow-Credentials: true` is also present, the finding is
  raised to **high** severity (credentialed cross-origin reads — the
  account-takeover-class variant); without it, **medium**.
- **No POC screenshot is rendered for this check either** — the evidence
  `.txt` always includes the exact `curl` command, already filled in with
  the real host/port, so a tester can re-run it by hand and capture their
  own screenshot for the report if they want one.

## "Needs Manual Verification"

A few checks (SMB signing when nmap can't get a clean read, LDAP
signing/anonymous-bind on an unrecognized bind result, RPC endpoint-mapper
reachability on a genuine transport failure, DNS recursion on no response)
are **behaviorally ambiguous by nature** — forcing them to a binary
Safe/Vulnerable verdict from an inconclusive signal would just manufacture
false positives or false negatives. These get their own `Needs_Manual_
Verification/<Check_Name>/` subfolder under the check's output directory,
with a yellow **NEEDS MANUAL VERIFICATION** badge on the POC instead of the
usual red/green — same real captured evidence, honestly labeled as
inconclusive rather than guessed at. `summary.csv` marks these rows
`MANUAL_VERIFICATION`, distinct from `VULNERABLE`/`SAFE`/`N/A`/`ERROR`. A
host can carry confirmed findings *and* manual-verification items from the
same check at once (e.g. LDAP: anonymous bind confirmed vulnerable,
signing flagged for manual verification, both real, independent signals).

## False positives are avoided by anchoring on the tool's own real signal, not a guess

- **SSH weak crypto** anchors on `ssh-audit`'s "critical" bucket specifically
  — its own maintained security database, not a heuristic reinvented here.
- **Untrusted CA vs. incomplete chain** are told apart by counting how many
  certificates the server actually sent (`-showcerts`): 1 (leaf only) and
  validation fails = the server's fault (incomplete chain); 2+ (a real
  chain) and validation still fails = the chain is fine but the root isn't
  trusted here (untrusted CA).
- **SMB anonymous access** is judged by whether the null-session share
  *listing* actually succeeded, not merely whether an anonymous session was
  established — establishing one is normal SMB protocol behavior on almost
  any server (validated against a real hardened local `smbd`: "Anonymous
  login successful" still printed even though the subsequent listing was
  denied with `NT_STATUS_ACCESS_DENIED`). Judging on session-establishment
  alone would false-positive on virtually every host.
- **LDAP signing** is judged by the server's own real LDAP result code from
  a bind attempt with bogus credentials — `Invalid credentials (49)` proves
  the server evaluated the credentials unprotected; `Confidentiality
  required (13)` / `Stronger authentication required (8)` proves it refused
  before ever looking at them. No valid credentials are needed either way.
- **RPC endpoint-mapper reachability** distinguishes a genuine, well-formed
  "no endpoints registered" DCERPC response (`ept_s_not_registered` — the
  mapper answered, there's just nothing there) from an actual transport
  failure (connection refused, timeout) — only the latter goes to manual
  verification; the former is a real Safe result.
- **SNMP / MSSQL** absence-of-a-hit is a real Safe result for what those
  specific checks claim ("weak/default community string" / "info
  disclosed"), not a blanket "this service is fully secure" claim.
- **Host Header Injection / XST** are deterministic, binary tests per an
  exact specified method (Host: bing.com → redirect Location reflects it;
  TRACE → unique marker header reflected in the body) — no heuristics.
- Every verdict is backed by the tool's own real output — nothing here
  reformats or reinterprets what the underlying tool said.

## Requirements

- Python 3.9+, on macOS or Linux (uses a Unix pty for ANSI-preserving
  terminal capture where relevant; no Windows support).
- `pip install netscraper && playwright install chromium` (once) — pulls in
  `ssh-audit`, `impacket` (for `rpcdump.py`), `dirsearch`, `playwright`
  (renders POC screenshots), `ansi2html` (real captured terminal colors →
  matching HTML), and `openpyxl` (writes `PT_Scan_Report.xlsx`) as
  dependencies automatically, and puts the `netscraper` command on your
  PATH. Running from a source checkout instead of the published package?
  `pip install -e . && playwright install chromium` from the repo root does
  the same thing against your local copy.
- Command-line tools this toolkit shells out to (install via your package
  manager — Debian/Ubuntu names shown):
  - `sslscan`, `openssl` — TLS check (`apt install sslscan`; openssl is
    already on virtually every Mac/Linux box).
  - `nmap` — SMB and MSSQL checks (`apt install nmap`).
  - `smbclient` — SMB check (`apt install smbclient`).
  - `ldapsearch` — LDAP check (`apt install ldap-utils`).
  - `rpcdump.py` — RPC check, comes from `pip install impacket` (already
    in requirements.txt), lands on PATH automatically.
  - `dig` — DNS check (`apt install dnsutils`).
  - `onesixtyone` — SNMP check (`apt install onesixtyone`); uses the
    bundled `wordlists/snmp_communities.txt`. **Optional on macOS**: this is
    an apt-only tool with no real Mac package, so if it's not found on
    PATH the SNMP check automatically falls back to a built-in,
    dependency-free Python SNMPv1 prober (`core/snmp_pure_probe.py`) that
    does the exact same community-string check over a real UDP socket —
    same wordlist, same logic, no install needed. The POC always discloses
    honestly which of the two actually ran.
  - `ftp` — FTP check (`apt install ftp`).
  - `curl` — HTTP check (usually already present).
  - `dirsearch` — DIRSEARCH check; already installed by `pip install -r
    requirements.txt` above (or clone https://github.com/maurosoria/
    dirsearch and add it to PATH yourself if you'd rather not pip-install it).
- Real network reach to whatever you're scanning. Run this from a machine
  or jump box that's actually inside/adjacent to the target scope — a
  sandboxed or heavily-proxied environment (anything that TLS-intercepts
  outbound traffic) will give you false results for the TLS check
  specifically, since it'll see its own proxy's certificate instead of the
  real target's. (Every check in this toolkit was instead validated during
  development against real, purpose-built local test servers — real
  `sslscan`/`ssh-audit`/`nmap`/`smbclient`/`ldapsearch`/`rpcdump.py`/
  `dig`/`onesixtyone`/`ftp`/`curl` output, not synthetic data, with one
  deliberately-vulnerable and one hardened instance per check wherever
  that distinction matters — see `fixtures/` and the per-check "Honesty
  note" callouts above for the two checks where full live parity with a
  real Windows/AD or MSSQL target wasn't achievable in that environment.)

## POCs are real terminal captures, not summaries

Each POC `.png` is the *actual* tool output — `sslscan`, `ssh-audit`,
`nmap`, `smbclient`, `ldapsearch`, `rpcdump.py`, `dig`, `onesixtyone`,
`ftp`, or `curl`, whichever check it's from (the exceptions are DIRSEARCH,
which by design saves its filtered "Found" results straight to a `.txt`
file, and CORS, which saves the ready-to-run `curl` PoC command instead —
neither ever renders a screenshot) — captured through a
pseudo-terminal (`core/terminal_capture.py`) for the two tools that
colorize their output (`sslscan`, `ssh-audit`) so coloring isn't suppressed
by the lack of a real TTY; every other tool in this toolkit doesn't
colorize its output at all, so a plain capture is already the real thing.
The captured bytes go through unmodified: real colors map 1:1 to matching
HTML colors (`osx` palette — macOS Terminal.app's default ANSI colors) and
nothing in the text itself is summarized, reworded, or filtered. The POC's
title bar, prompt line, and footer all show the **exact real command** that
was run — never a hardcoded tool name — so every screenshot honestly
reflects what actually produced it.

The only thing added on top is an outline box around the specific line(s)
relevant to that finding — red for a confirmed Vulnerable finding, drawn
*around* the real text without ever touching it. A Needs-Manual-
Verification POC gets a yellow badge instead of red/green. The Secure_POC
card shows the same real output with no box at all.

One Chromium browser instance is launched **once** and reused for every
single POC rendered from that same process — not relaunched per
screenshot. A run against a large target list can easily produce several
hundred POCs (one per finding, plus one per clean host per check), and
launching a fresh browser process for each one doesn't scale: it's slower,
and it's a real reliability risk under sustained load. In a single-process
run, `core/poc_renderer.py`'s `browser_session()` launches that one browser
for the whole run; when scanning runs in parallel (see "Scans several
hosts at once, automatically" above), each worker *process* instead calls
`get_process_browser()` to lazily launch its own instance the first time it
needs one, then keeps reusing that same instance for the rest of the run —
a Playwright browser object can't be shared across process boundaries, so
the reuse is scoped per worker process rather than per whole run, but it's
still one Chromium launch per worker instead of one per screenshot. Either
way, the browser is always closed on the way out, exception or not, so a
render failure partway through a long scan can never leak the browser
process.

## Usage

```bash
# Scan a list of targets (IPs, hostnames, and/or CIDR subnets, one per line)
# --output defaults to ./results if omitted. With no --modes flag, you'll be
# prompted to pick which check(s) to run once the port scan finishes.
netscraper --targets targets.txt

# Scan a single target or a whole subnet on the command line
netscraper --target 192.168.1.10
netscraper --target 192.168.1.0/28

# Custom port (used by the TLS check; every other check checks its own
# fixed port regardless, e.g. SSH always 22, SMB always 445)
netscraper --target 192.168.1.10:8443

# Skip the interactive picker and run specific check(s) non-interactively
netscraper --targets targets.txt --modes tls,ssh
netscraper --targets targets.txt --modes smb,ldap,rpc
netscraper --targets targets.txt --modes dns,mssql,snmp,ftp,http
netscraper --targets targets.txt --modes all

# DIRSEARCH has no fixed port -- give it one (or several, comma-separated)
# with --dirsearch-ports when running it non-interactively; in the
# interactive picker you're simply asked for it right after choosing DIRSEARCH.
netscraper --targets targets.txt --modes dirsearch --dirsearch-ports 80,443,8080

# CORS is the same shape -- no fixed port, give it one or several with
# --cors-ports (or you'll be asked interactively after picking CORS).
netscraper --targets targets.txt --modes cors --cors-ports 443,8443

# Skip the 5s pause between checks during a live scan, or change it
netscraper --targets targets.txt --mode-pause 0
netscraper --targets targets.txt --mode-pause 10
```

At the very start of every run, before anything is scanned, the tool prints
how many targets were loaded and exactly which folder this run's evidence
is going into, then runs the unified port scan across every check up front,
and — unless you passed `--modes` — asks which check(s) you actually want:

```
============================================================
[+] Target list loaded: 137 IP/host(es) to scan
[+] Scan type: LIVE
[+] Results folder: results/scan_20260907_143022_a1b2c3
============================================================

[+] Scanning required ports across 137 target(s) ...
    TLS           : 121/137 host(s) open
    SSH           : 40/137 host(s) open
    SMB           : 12/137 host(s) open
    LDAP          : 3/137 host(s) open
    RPC           : 12/137 host(s) open
    DNS    (UDP)  : checked directly during the scan (no TCP pre-check possible)
    MSSQL         : 2/137 host(s) open
    SNMP   (UDP)  : checked directly during the scan (no TCP pre-check possible)
    FTP           : 5/137 host(s) open
    HTTP          : 118/137 host(s) open
    DIRSEARCH     : port(s) chosen after you select this mode
    CORS          : port(s) chosen after you select this mode

Select which check(s) to run:

  [1] TLS       -- 121/137 host(s) with the required port open
  [2] SSH       -- 40/137 host(s) with the required port open
  [3] SMB       -- 12/137 host(s) with the required port open
  [4] LDAP      -- 3/137 host(s) with the required port open
  [5] RPC       -- 12/137 host(s) with the required port open
  [6] DNS       -- UDP service, checked directly
  [7] MSSQL     -- 2/137 host(s) with the required port open
  [8] SNMP      -- UDP service, checked directly
  [9] FTP       -- 5/137 host(s) with the required port open
  [10] HTTP     -- 118/137 host(s) with the required port open
  [11] DIRSEARCH -- directory/file brute-force (pick port(s) after selecting)
  [12] CORS   -- CORS misconfiguration check (pick port(s) after selecting)
  [13] All checks

  Select mode(s) (e.g. 1,3 or 'all'):
```

Type a number, several comma-separated (`1,3,7`), or `all`. Only the
check(s) you pick actually run — no more sitting through SMB/LDAP/RPC
checks against a target list that's mostly web servers just because the
tool insists on doing all ten every time. Picking DIRSEARCH or CORS prompts
you for its port(s) right after you make your selection, since those are
the two checks with no fixed port of their own.

### Every run gets its own results folder

By default, each run writes into a uniquely-named subfolder under `--output`
(default root `./results`) instead of a single fixed path — e.g.
`results/scan_20260907_143022_a1b2c3` (a timestamp plus a short random
suffix, so two runs started in the same second never collide). This means
re-scanning the same targets later (a re-test after remediation, a repeat
engagement, etc.) never silently overwrites a previous run's POCs and raw
evidence — every run's evidence is kept, side by side, under `./results/`.

If you're scripting this and want full control over the exact output path
yourself (e.g. in CI), pass `--no-run-id` to write directly into `--output`
with no extra subfolder (this WILL overwrite whatever was already at that
path):

```bash
netscraper --targets targets.txt --output ./results --no-run-id
```

### Replay mode (no live scanning)

For reprocessing previously-captured output, or testing the pipeline
without touching the network:

```bash
netscraper --targets targets_example_tls.txt   --xml-dir ./fixtures --modes tls
netscraper --targets targets_example_ssh.txt   --xml-dir ./fixtures --modes ssh
netscraper --targets targets_example_smb.txt   --xml-dir ./fixtures --modes smb
netscraper --targets targets_example_ldap.txt  --xml-dir ./fixtures --modes ldap
netscraper --targets targets_example_rpc.txt   --xml-dir ./fixtures --modes rpc
netscraper --targets targets_example_dns.txt   --xml-dir ./fixtures --modes dns
netscraper --targets targets_example_mssql.txt --xml-dir ./fixtures --modes mssql
netscraper --targets targets_example_snmp.txt  --xml-dir ./fixtures --modes snmp
netscraper --targets targets_example_ftp.txt   --xml-dir ./fixtures --modes ftp
netscraper --targets targets_example_http.txt  --xml-dir ./fixtures --modes http
```

`--xml-dir` looks for pre-captured evidence instead of invoking the real
tools (per check: `<host>.xml`/`.txt`/`_chain.txt` for TLS, `<host>_ssh.
json`/`.txt` for SSH, `<host>_smb_nmap.txt`/`_smbclient.txt` for SMB,
`<host>_ldap_anon.txt`/`_ldap_signing.txt` for LDAP, `<host>_rpcdump.txt`
for RPC, `<host>_dig.txt` for DNS, `<host>_mssql_nmap.txt` for MSSQL,
`<host>_onesixtyone.txt` for SNMP, `<host>_ftp.txt` for FTP, and
`<host>_http_version.txt`/`_http_hostheader.txt`/`_http_trace.txt` for
HTTP). `fixtures/` contains real captured output from purpose-built local
test servers — a genuinely self-signed cert, a genuinely incomplete chain,
real weak vs. hardened SSH/SMB/LDAP/SNMP/FTP/HTTP configurations, etc. Demo
target lists are split by check because each fixture host is illustrative
of one specific check; a real targets file lists every host once and lets
port-gating sort out which checks actually apply to it (that's the whole
point of vuln-first scanning).

## Output layout

Everything below lives under this run's unique folder (e.g.
`results/scan_20260907_143022_a1b2c3/`, or directly under `--output` if you
passed `--no-run-id`):

```
scan_<timestamp>_<id>/
  SSL_Scan/       Expired_Certificate/ Deprecated_TLS_Protocols/ Weak_Ciphers/
                  Weak_Key_Strength/ Hostname_Mismatch/ Untrusted_Self_Signed/
                  Incomplete_Certificate_Chain/ Secure_POC/
  SSH/            Weak_Deprecated_Crypto/ Banner_Disclosure/ Secure_POC/
  SMB/            SMBv1_Enabled/ SMB_Signing_Not_Enforced/
                  Null_Session_Anonymous_Access/
                  Needs_Manual_Verification/SMB_Signing/ Secure_POC/
  LDAP/           LDAP_Anonymous_Bind_Enabled/ LDAP_Signing_Not_Enforced/
                  Needs_Manual_Verification/LDAP_Anonymous_Bind/
                  Needs_Manual_Verification/LDAP_Signing/ Secure_POC/
  RPC/            Exposed_MSRPC_Endpoint_Mapper/ MS-SCMR_atsvc_Exposed/
                  Needs_Manual_Verification/RPC_Endpoint_Mapper/ Secure_POC/
  DNS/            DNS_Recursion_Enabled/
                  Needs_Manual_Verification/DNS_Recursion/ Secure_POC/
  MSSQL/          MSSQL_Service_Information_Disclosure/ Secure_POC/
  SNMP/           Weak_Default_SNMP_Community/ Secure_POC/
  FTP/            Anonymous_FTP_Access/
                  Needs_Manual_Verification/Anonymous_FTP/ Secure_POC/
  HTTP/           Server_Version_Disclosure/ Host_Header_Injection/
                  Cross-Site_Tracing_XST/ Secure_POC/
  Dirsearch/      Found_Paths/ Secure_POC/     (no .png here -- .txt evidence only)
  CORS/           CORS_Misconfiguration/ Secure_POC/   (no .png here -- .txt evidence only)
  raw/<check>/<host>_<artifact>.txt    <- always saved, every check, every host
  summary.csv                           <- one row per (host, check)
  PT_Scan_Report.xlsx                   <- one row per finding, styled report
```

Each finding folder holds `<host>.png` (the POC screenshot) and `<host>.txt`
(the plain-text evidence). `summary.csv` columns are `host, port, check,
status, findings` — `status` is one of `VULNERABLE`, `SAFE`, `N/A` (check's
required port wasn't open on this host), `ERROR` (the tool itself failed),
or `MANUAL_VERIFICATION`.

### PT_Scan_Report.xlsx

A styled Excel report, written alongside `summary.csv` at the end of every
run (partial results are still written if you Ctrl-C mid-scan — see below).
Columns are `S.No`, `Vulnerability`, `Host`, `Status` — a blue bold header,
one row per row that's actually a real Vulnerable/Not-Vulnerable verdict:

- One row per **confirmed finding**, `Vulnerability` = the exact finding
  name (e.g. "Expired SSL Certificate"), `Status` = **Vulnerable** (red). A
  host with 3 findings from one check gets 3 separate rows.
- One row per (host, check) that came back completely **clean**,
  `Vulnerability` = `"<CHECK> -- No Vulnerabilities Found"`, `Status` =
  **Not Vulnerable** (green).
- `N/A` (port not open), `ERROR` (tool failure), and `NEEDS MANUAL
  VERIFICATION` results are deliberately left **out** of this file — none
  of them is a real binary Vulnerable/Not-Vulnerable verdict, and forcing
  one in would misrepresent an inconclusive result as a finding. They're
  still fully recorded in `summary.csv` and in their own POC/evidence
  files regardless.

## Project layout (for extending with new checks)

```
pyproject.toml            — packaging metadata (this is what `pip install netscraper` builds from)
LICENSE / MANIFEST.in
src/netscraper/
  __init__.py              — package version
  cli.py                   — orchestration: port-gate each family, route results to folders
  wordlists/               — small bundled wordlists (currently: SNMP default communities), shipped as package data
  core/
    targets.py             — expands IPs / hostnames / CIDR subnets into targets
    port_gate.py            — fast parallel TCP "is this port open" pre-check
    terminal_capture.py     — pty-based real ANSI output capture (reused wherever a tool colorizes)
    poc_renderer.py         — renders any (raw text, Finding, real command) as a terminal-card PNG
    concurrency.py          — auto-sizes the parallel scan worker pool from this machine's RAM/CPU
    scan_worker.py          — picklable per-(family, host) unit of work run inside a pool worker process
    rules.py                — Finding dataclass + the TLS/cert-config rule set
    families.py             — one run_xxx_family() per family; cli.py stays generic
    sslscan_runner.py / sslscan_parser.py                              — TLS family, sslscan half
    cert_chain_runner.py / cert_chain_parser.py / cert_chain_rules.py  — TLS family, chain-of-trust half
    ssh_runner.py / ssh_parser.py / ssh_rules.py                       — SSH family
    smb_runner.py / smb_parser.py / smb_rules.py                       — SMB family
    ldap_runner.py / ldap_parser.py / ldap_rules.py                    — LDAP family
    rpc_runner.py / rpc_parser.py / rpc_rules.py                       — RPC family
    dns_runner.py / dns_parser.py / dns_rules.py                       — DNS family
    mssql_runner.py / mssql_parser.py / mssql_rules.py                 — MSSQL family
    snmp_runner.py / snmp_parser.py / snmp_rules.py                    — SNMP family
    ftp_runner.py / ftp_parser.py / ftp_rules.py                       — FTP family
    http_runner.py / http_parser.py / http_rules.py                    — HTTP family
    dirsearch_runner.py / dirsearch_parser.py / dirsearch_rules.py     — DIRSEARCH family (no fixed port; render_poc=False)
    cors_runner.py / cors_parser.py / cors_rules.py                    — CORS family (no fixed port; render_poc=False)
fixtures/                 — captured evidence for offline/replay testing (dev only, not shipped in the package)
```

Adding a new family means: a runner (gathers evidence, live + replay), a
parser (raw → plain struct), a rules module (struct → `Finding`s, reusing
`core.rules.Finding`, returning `(confirmed_findings, manual_verification_
findings)`), and one `run_xxx_family()` in `families.py` registered in
`FAMILIES` with its required port (a plain int for a fixed TCP port, `None`
to gate on each target's own `.port`, `"udp:<port>"` for a UDP-only service
with no TCP pre-gate, or `"prompt"` for a family with no fixed port at all —
see DIRSEARCH/CORS and `cli.py`'s `_resolve_prompt_gated_ports()`). A family entry
can also set `"render_poc": False` to skip POC screenshots entirely and
save only the `.txt` evidence (again, DIRSEARCH). `port_gate.py`,
`terminal_capture.py`, `poc_renderer.py`, and all of `cli.py`'s
orchestration are already generic and reusable as-is.
