Metadata-Version: 2.4
Name: dvc-helper
Version: 0.1.0
Summary: An intelligent CLI assistant for Data Version Control (DVC)
Author: dvc-helper contributors
License: MIT
Project-URL: Homepage, https://github.com/anomalyco/dvc-helper
Project-URL: Repository, https://github.com/anomalyco/dvc-helper
Project-URL: Documentation, https://github.com/anomalyco/dvc-helper#readme
Keywords: dvc,mlops,data-version-control,cli,machine-learning
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Topic :: Software Development :: Version Control
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: typer>=0.9.0
Requires-Dist: rich>=13.0.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: questionary>=2.0.0
Requires-Dist: platformdirs>=4.0.0
Requires-Dist: networkx>=3.0
Requires-Dist: InquirerPy>=0.3.4
Requires-Dist: packaging>=23.0

# dvc-helper — Intelligent CLI Assistant for Data Version Control (DVC)

> **Version 0.1.0** | [PyPI](https://pypi.org/project/dvc-helper/) | [GitHub](https://github.com/anomalyco/dvc-helper)

---

## Table of Contents

1. [What is DVC?](#1-what-is-dvc)
2. [Why DVC? The Advantages](#2-why-dvc-the-advantages)
3. [Enter dvc-helper](#3-enter-dvc-helper)
4. [Installation](#4-installation)
5. [Quick Start](#5-quick-start)
6. [Command Reference](#6-command-reference)
   - [Global Commands](#61-global-commands)
   - [Stage Commands](#62-stage-commands)
   - [Experiment Commands](#63-experiment-commands)
   - [Pipeline Commands](#64-pipeline-commands)
   - [Metrics, Params, Plots](#65-metrics-params-plots)
   - [Remote & Cache](#66-remote--cache)
   - [AI Assistance](#67-ai-assistance)
7. [Architecture](#7-architecture)
8. [Development](#8-development)
9. [Publishing to PyPI](#9-publishing-to-pypi)
10. [FAQ](#10-faq)

---

## 1. What is DVC?

**DVC** (Data Version Control) is an open-source version control system designed specifically for machine learning and data science projects. It extends Git with capabilities for managing:

- **Large data files** — datasets, images, audio, video files that don't belong in Git
- **ML models** — trained model binaries, checkpoints, and artifacts
- **Pipeline stages** — reproducible sequences of data processing and training steps
- **Experiments** — systematic tracking of hyperparameters, metrics, and results
- **Metrics and plots** — quantitative evaluation across runs

DVC works on top of Git, meaning you keep using your familiar Git workflow while gaining ML-specific superpowers. It is language-agnostic, works with any ML framework (PyTorch, TensorFlow, scikit-learn, XGBoost, etc.), and is 100% open source.

### How DVC Works

At its core, DVC replaces large files in your Git repository with lightweight pointer files (`.dvc` files or entries in `dvc.yaml`). The actual data is stored in a **cache** (local or remote — S3, GCS, Azure, SSH, etc.) and is pulled on demand. Your `dvc.yaml` file defines the pipeline: stages, commands, dependencies, outputs, parameters, metrics, and plots.

```
Project/
├── .git/               # Git metadata
├── .dvc/               # DVC cache & config
├── dvc.yaml            # Pipeline definition
├── dvc.lock            # Locked dependency hashes
├── params.yaml         # Hyperparameters
├── data/
│   ├── raw.csv.dvc     # Pointer to cached data
│   └── processed.csv   # Generated by pipeline
├── models/
│   └── model.pkl       # Generated artifact
├── metrics.json        # Evaluation results
└── src/
    ├── preprocess.py
    ├── train.py
    └── evaluate.py
```

---

## 2. Why DVC? The Advantages

### 2.1 Version Control for Large Files

Git cannot handle files larger than ~100 MB effectively. DVC stores only lightweight pointers in Git while the actual data lives in external storage (S3, GCS, local filesystem, etc.). Every commit in Git corresponds to a specific version of your data, models, and pipeline configuration — full reproducibility with zero bloat.

### 2.2 Reproducible Pipelines

DVC pipelines are defined as directed acyclic graphs (DAGs) in `dvc.yaml`. Each stage declares its command, dependencies, and outputs. DVC tracks the checksums of every dependency and output, so it knows exactly which stages need to be re-run when something changes. Running `dvc repro` executes only the outdated stages.

### 2.3 Experiment Management

DVC experiments allow you to run, queue, compare, and apply variations of your pipeline without branching your Git repository. Change a parameter, run an experiment, compare metrics across runs, and apply the best one — all within a single Git branch.

### 2.4 Metric & Plot Tracking

DVC natively tracks metrics (JSON, YAML) and plots (CSV, images). Compare metrics across experiments with `dvc metrics diff`, generate comparison plots with `dvc plots diff` — making it trivial to track model performance over time.

### 2.5 Cloud-Agnostic Remote Storage

DVC supports S3, GCS, Azure Blob, SSH, MinIO, Google Drive, WebDAV, HDFS, and local filesystems as remote storage backends. Switch between them without changing your workflow.

### 2.6 CI/CD Integration

DVC integrates seamlessly with CI/CD pipelines (GitHub Actions, GitLab CI, etc.). Pull data from remote storage, reproduce the pipeline, and publish metrics — all automated.

### 2.7 Framework Agnostic

DVC does not care what framework you use. Whether it's PyTorch, TensorFlow, JAX, scikit-learn, XGBoost, CatBoost, LightGBM, or raw NumPy — DVC tracks files, commands, and parameters without any framework-specific code.

### 2.8 Git-Native Workflow

DVC commands like `dvc push`, `dvc pull`, `dvc fetch`, `dvc status` mirror Git's semantics. There is no new paradigm to learn — just new commands that feel familiar.

> **Bottom line:** DVC turns ad-hoc ML projects into disciplined, reproducible, auditable pipelines — without forcing you to change how you write code.

---

## 3. Enter dvc-helper

While DVC is powerful, its command-line interface can be verbose and error-prone. Manually editing `dvc.yaml` to define stages, specifying dependencies and outputs with exact paths, remembering flags like `--force-downstream` or `--allow-missing` — these create friction.

**dvc-helper** is an intelligent CLI layer on top of DVC that:

- **Eliminates memorization** — no need to remember complex DVC commands
- **Automates discovery** — analyzes your Python scripts using AST to detect dependencies, outputs, parameters, metrics, and plots automatically
- **Provides interactive wizards** — asks only what it needs, infers the rest
- **Prevents errors** — validates stages, detects circular dependencies, checks for missing files
- **Beautiful output** — uses Rich for colored, formatted terminal output with tables, panels, progress bars, and syntax highlighting
- **Supports both interactive and non-interactive modes** — for human-driven exploration and scripted automation

### The Core Philosophy

> **A user should be able to create an entire DVC pipeline by answering only two questions:**
> - Stage Name
> - Command
>
> Everything else — dependencies, outputs, parameters, metrics, plots — should be automatically discovered, validated, and suggested.

### Example

Instead of manually editing `dvc.yaml`:

```bash
dvc-helper stage create
```

```
Stage Name:
> train

Command:
> python src/train.py --epochs 100 --lr 0.001
```

dvc-helper analyzes `src/train.py`, detects file read/write operations, examines CLI arguments, scans the project for `params.yaml` and metric files, and prompts you to confirm before saving. The result is a valid `dvc.yaml` entry:

```yaml
stages:
  train:
    cmd: python src/train.py --epochs 100 --lr 0.001
    deps:
      - src/train.py
      - src/model.py
      - params.yaml
      - data/train.csv
    outs:
      - models/model.pt
      - predictions.csv
    params:
      - params.yaml
    metrics:
      - metrics.json
    plots:
      - plots/loss.csv
```

No manual YAML editing. No memorized commands. No broken pipelines.

---

## 4. Installation

### Prerequisites

- **Python 3.10 or later**
- **Git** — install from [git-scm.com](https://git-scm.com/)
- **DVC** — `pip install dvc` (optional for some features, required for `repro`, `status`, `exp`, `remote`, `cache`)

### Install from PyPI

```bash
pip install dvc-helper
```

### Install from Source

```bash
git clone https://github.com/anomalyco/dvc-helper.git
cd dvc-helper
pip install -e ".[dev]"
```

### Verify Installation

```bash
dvc-helper doctor
```

Example output:

```
dvc-helper Doctor
  OS: Windows 10
  Python: 3.10.11
  dvc-helper: 0.1.0
  DVC: 3.58.0
  Git: git version 2.54.0.windows.1
  DVC available: True
  Git available: True
  Project root: /path/to/project
  All dependencies available.
```

---

## 5. Quick Start

### 5.1 Initialize a Project

```bash
cd my-ml-project
dvc-helper init
```

This will:
1. Check for Git, initialize if missing
2. Check for DVC, initialize if missing
3. Create a standard folder structure: `data/`, `models/`, `src/`, `notebooks/`, `plots/`, `config/`, `metrics/`, `logs/`
4. Create `params.yaml` with sensible defaults
5. Create an initial `dvc.yaml`
6. Update `.gitignore` with DVC entries

### 5.2 Create Your First Stage

```bash
dvc-helper stage create
```

```
Stage Name:
> preprocess

Command:
> python src/preprocess.py --input data/raw.csv
```

dvc-helper analyzes `src/preprocess.py` and presents detected dependencies, outputs, and parameters for confirmation.

### 5.3 Create a Training Stage

```bash
dvc-helper stage create
```

```
Stage Name:
> train

Command:
> python src/train.py --epochs 50 --lr 0.001
```

### 5.4 Run the Pipeline

```bash
dvc-helper repro
```

### 5.5 Check Status

```bash
dvc-helper status
```

### 5.6 View the DAG

```bash
dvc-helper dag --format mermaid
```

---

## 6. Command Reference

### 6.1 Global Commands

---

#### `dvc-helper version`

Display the installed version of dvc-helper.

```bash
dvc-helper version
```

```
dvc-helper v0.1.0
```

---

#### `dvc-helper init`

Initialize a new DVC project with recommended defaults.

```bash
dvc-helper init [OPTIONS]
```

| Option | Alias | Description |
|--------|-------|-------------|
| `--skip-git` | | Skip Git initialization |
| `--skip-dvc` | | Skip DVC initialization |
| `--skip-dirs` | | Skip folder structure creation |
| `--skip-params` | | Skip `params.yaml` creation |
| `--force` | `-f` | Overwrite existing files |

**What `dvc-helper init` does:**

1. **Verifies Git** — checks if Git is installed
2. **Initializes Git** — runs `git init` if `.git` does not exist
3. **Verifies DVC** — checks if DVC is installed
4. **Initializes DVC** — runs `dvc init` if `.dvc` does not exist
5. **Creates folder structure**:
   - `data/raw/`, `data/processed/`, `data/interim/`, `data/external/`
   - `models/`
   - `notebooks/`
   - `src/`
   - `config/`
   - `reports/figures/`
   - `plots/`
   - `metrics/`
   - `logs/`
6. **Creates `params.yaml`** with template parameters
7. **Creates `dvc.yaml`** with empty stages section
8. **Updates `.gitignore`** with DVC-related patterns

**Example:**

```bash
# Full initialization
dvc-helper init

# Quick initialization (only DVC, no Git)
dvc-helper init --skip-git

# Force overwrite existing config
dvc-helper init --force
```

---

#### `dvc-helper doctor`

Run system diagnostics to verify that all dependencies are available.

```bash
dvc-helper doctor
```

Checks:
- Operating system version
- Python version
- dvc-helper version
- DVC version and availability
- Git version and availability
- Project root detection
- All Python dependency imports

---

#### `dvc-helper config`

Get, set, or list dvc-helper configuration values.

```bash
dvc-helper config [KEY] [VALUE] [OPTIONS]
```

| Argument | Description |
|----------|-------------|
| `KEY` | Configuration key to get or set |
| `VALUE` | Value to set (omit to get current value) |

| Option | Alias | Description |
|--------|-------|-------------|
| `--list` | `-l` | List all configuration values |

**Examples:**

```bash
# List all config
dvc-helper config --list

# Get a specific value
dvc-helper config default_params_file

# Set a value
dvc-helper config auto_discover false
```

Available configuration keys:
- `debug` (bool) — Enable debug mode
- `color` (bool) — Enable colored output
- `interactive` (bool) — Enable interactive prompts
- `confirm_before_write` (bool) — Confirm before writing files
- `backup_before_write` (bool) — Create backups before overwriting
- `default_params_file` (str) — Default parameters file path
- `auto_discover` (bool) — Auto-discover dependencies and outputs
- `strict_validation` (bool) — Enable strict validation
- `telemetry_enabled` (bool) — Enable telemetry

---

#### `dvc-helper completion`

Install shell completion for dvc-helper commands.

```bash
dvc-helper completion [SHELL]
```

| Argument | Description (Default: `auto`) |
|----------|-------------------------------|
| `SHELL` | Shell type: `bash`, `zsh`, `fish`, `powershell`, or `auto` |

`auto` detects the current shell. On Windows, defaults to `powershell`.

**Example:**

```bash
# Auto-detect and install
dvc-helper completion

# Explicit PowerShell
dvc-helper completion powershell
```

After installation, restart your shell or source your config file. Tab-completion will work for all dvc-helper commands, subcommands, and options.

---

### 6.2 Stage Commands

Stage commands are the core of dvc-helper. They allow you to create, read, update, delete, validate, and visualize pipeline stages without manually editing YAML files.

---

#### `dvc-helper stage create`

Create a new pipeline stage with intelligent auto-detection.

```bash
dvc-helper stage create [STAGE_NAME] [COMMAND] [OPTIONS]
```

| Argument | Description |
|----------|-------------|
| `STAGE_NAME` | Name of the stage (alphanumeric, underscores, hyphens) |
| `COMMAND` | Shell command to execute |

| Option | Alias | Description |
|--------|-------|-------------|
| `--non-interactive` | `-n` | Skip interactive prompts |
| `--dep` | `-d` | Dependency path (repeatable) |
| `--out` | `-o` | Output path (repeatable) |
| `--param` | `-p` | Params file path (repeatable) |
| `--metric` | `-m` | Metric file path (repeatable) |
| `--plot` | | Plot file path (repeatable) |
| `--desc` | | Stage description |

**Interactive Mode (default):**

```bash
dvc-helper stage create
```

You are prompted for:
1. **Stage Name** — must start with a letter or underscore
2. **Command** — the full shell command to execute

After analyzing the command and its script, dvc-helper presents:

```
Detected Dependencies
  ✔ src/train.py
  ✔ src/model.py
  ✔ src/utils.py
  ✔ params.yaml
  ✔ config/config.yaml
  ✔ data/train.csv

Detected Outputs
  ✔ models/model.pt
  ✔ metrics.json
  ✔ predictions.csv

Detected Parameters
  ✔ params.yaml

Detected Metrics
  ✔ metrics.json

Detected Plots
  ✔ plots/loss.csv
```

You can edit each list before confirming. Then optionally add a description and set up a matrix stage.

**Non-Interactive Mode:**

```bash
dvc-helper stage create train "python src/train.py --epochs 50" \
  --dep src/train.py \
  --dep src/model.py \
  --dep params.yaml \
  --out models/model.pt \
  --param params.yaml \
  --metric metrics.json
```

**Auto-Detection in Detail:**

When you provide a command that runs a Python script, dvc-helper performs:

1. **AST Analysis** — parses the Python script's abstract syntax tree
   - Detects all imports (standard library, third-party, local modules)
   - Identifies function calls for I/O operations
   - Extracts class and function definitions

2. **Dependency Detection** — identifies all files read by the script
   - File open operations (`open()`, `Path().open()`)
   - Pandas reads (`pd.read_csv()`, `pd.read_parquet()`, etc.)
   - NumPy loads (`np.load()`, `np.loadtxt()`)
   - Torch loads (`torch.load()`)
   - Joblib/Pickle loads (`joblib.load()`, `pickle.load()`)
   - JSON/YAML reads (`json.load()`, `yaml.safe_load()`)
   - Database connections (`sqlite3.connect()`, `duckdb.connect()`)
   - Glob patterns (`glob.glob()`, `Path().glob()`)
   - Directory listing (`os.listdir()`, `os.walk()`)

3. **Output Detection** — identifies all files written by the script
   - Pandas writes (`df.to_csv()`, `df.to_parquet()`, etc.)
   - Torch saves (`torch.save()`)
   - Joblib/Pickle dumps (`joblib.dump()`, `pickle.dump()`)
   - Plot saves (`plt.savefig()`, `fig.savefig()`)
   - JSON/YAML writes (`json.dump()`, `yaml.dump()`)
   - NumPy saves (`np.save()`, `np.savetxt()`)

4. **Parameter Detection** — detects CLI arguments and params.yaml usage
   - Detects `argparse`, `click`, `typer`, `fire` argument definitions
   - Parses actual CLI arguments from the command string
   - Includes `params.yaml` if the script uses it

5. **Metrics Detection** — scans for metric files
   - `metrics.json`, `metrics.yaml`, `results.json`
   - `evaluation.json`, `scores.json`, `accuracy.json`

6. **Plot Detection** — scans for plot files
   - `loss.csv`, `accuracy.csv`, `confusion_matrix.png`
   - `roc_curve.csv`, `pr_curve.csv`, `feature_importance.png`

**Matrix Stage Builder:**

After the basic configuration, dvc-helper asks:

```
Create Matrix Stage? [Y/N]:
> Y

Matrix Parameter (empty to finish):
> learning_rate

Values for learning_rate (comma-separated):
> 0.001, 0.01, 0.1

Matrix Parameter (empty to finish):
> epochs

Values for epochs (comma-separated):
> 10, 50, 100

Matrix Parameter (empty to finish):
>
```

This generates a DVC matrix stage:

```yaml
stages:
  train:
    foreach:
      - learning_rate
      - epochs
    matrix:
      learning_rate: [0.001, 0.01, 0.1]
      epochs: [10, 50, 100]
    do:
      cmd: python src/train.py --lr ${item.learning_rate} --epochs ${item.epochs}
```

---

#### `dvc-helper stage update`

Update an existing stage's configuration.

```bash
dvc-helper stage update STAGE_NAME [OPTIONS]
```

| Argument | Description |
|----------|-------------|
| `STAGE_NAME` | Name of the stage to update (required) |

| Option | Alias | Description |
|--------|-------|-------------|
| `--non-interactive` | `-n` | Skip interactive prompts |
| `--cmd` | | New command |
| `--dep` | `-d` | New dependency (repeatable, replaces all) |
| `--out` | `-o` | New output (repeatable, replaces all) |

**Interactive Mode:**

```bash
dvc-helper stage update train
```

```
Updating stage: train

Stage Configuration Summary:
┌──────────────────────────────────────────────────────────────┐
│                     Stage: train                              │
│   Name: train                                                 │
│   Command: python src/train.py --epochs 50 --lr 0.001         │
│   Dependencies: src/train.py, params.yaml, data/train.csv     │
│   Outputs: models/model.pt                                    │
│   Params: params.yaml                                         │
│   Metrics: metrics.json                                       │
│   Plots: plots/loss.csv                                       │
└──────────────────────────────────────────────────────────────┘

Press Enter to keep current value.

Command [python src/train.py --epochs 50 --lr 0.001]:
> python src/train.py --epochs 100 --lr 0.0001

Dependencies (comma-separated) [src/train.py, params.yaml, data/train.csv]:
>
```

Only fields that change are updated. Unchanged fields are preserved.

**Non-Interactive Mode:**

```bash
dvc-helper stage update train --cmd "python train.py --lr 0.01"
```

---

#### `dvc-helper stage delete`

Delete a stage from the pipeline.

```bash
dvc-helper stage delete STAGE_NAME [OPTIONS]
```

| Argument | Description |
|----------|-------------|
| `STAGE_NAME` | Name of the stage to delete (required) |

| Option | Alias | Description |
|--------|-------|-------------|
| `--force` | `-f` | Skip confirmation prompt |

**Examples:**

```bash
# Interactive (with confirmation)
dvc-helper stage delete train

# Force delete without confirmation
dvc-helper stage delete train --force
```

This removes the stage from `dvc.yaml` and also cleans up the corresponding entry in `dvc.lock` if it exists.

---

#### `dvc-helper stage rename`

Rename an existing stage.

```bash
dvc-helper stage rename OLD_NAME NEW_NAME [OPTIONS]
```

| Argument | Description |
|----------|-------------|
| `OLD_NAME` | Current stage name |
| `NEW_NAME` | New stage name |

| Option | Alias | Description |
|--------|-------|-------------|
| `--force` | `-f` | Skip confirmation prompt |

**Example:**

```bash
dvc-helper stage rename train trainer
```

All internal references to the stage are updated. The `dvc.lock` file is also updated if it contains the old stage name.

---

#### `dvc-helper stage duplicate`

Duplicate an existing stage under a new name.

```bash
dvc-helper stage duplicate SOURCE TARGET [OPTIONS]
```

| Argument | Description |
|----------|-------------|
| `SOURCE` | Source stage name to duplicate |
| `TARGET` | Target stage name for the copy |

| Option | Alias | Description |
|--------|-------|-------------|
| `--force` | `-f` | Skip confirmation prompt |

**Example:**

```bash
dvc-helper stage duplicate train train_gpu
```

This creates a deep copy of the source stage configuration under the new name. Useful for creating variations of a stage (e.g., CPU vs GPU training).

---

#### `dvc-helper stage list`

List all stages in the pipeline.

```bash
dvc-helper stage list [OPTIONS]
```

| Option | Alias | Description |
|--------|-------|-------------|
| `--all` | `-a` | Show full configuration details for each stage |

**Default view (table):**

```bash
dvc-helper stage list
```

```
┌──────────────────────────────────────────────────────────────────┐
│                         Pipeline Stages                           │
├───────────┬────────────────────────────────┬────────────┬────────┤
│ Name      │ Command                        │ Outputs     │ Frozen │
├───────────┼────────────────────────────────┼────────────┼────────┤
│ preprocess│ python src/preprocess.py ...   │ data/      │        │
│ train     │ python src/train.py --epoch... │ models/... │        │
│ evaluate  │ python src/evaluate.py         │ metrics... │   ❄    │
└───────────┴────────────────────────────────┴────────────┴────────┘
```

**Detailed view:**

```bash
dvc-helper stage list --all
```

Shows complete configuration for each stage using Rich panels.

---

#### `dvc-helper stage show`

Display the full configuration of a specific stage.

```bash
dvc-helper stage show STAGE_NAME
```

| Argument | Description |
|----------|-------------|
| `STAGE_NAME` | Name of the stage to display |

**Example:**

```bash
dvc-helper stage show train
```

```
┌──────────────────────────────────────────────────────────────────┐
│                         Stage: train                              │
├──────────────────────────────────────────────────────────────────┤
│   Name:         train                                             │
│   Command:      python src/train.py --epochs 100                  │
│   Frozen:       No                                                │
│   Dependencies: src/train.py, src/model.py, params.yaml           │
│   Outputs:      models/model.pt                                   │
│   Params:       params.yaml                                       │
│   Metrics:      metrics.json                                      │
│   Plots:        plots/loss.csv                                    │
│   Description:  Training stage for the model                      │
└──────────────────────────────────────────────────────────────────┘
```

---

#### `dvc-helper stage validate`

Validate stage configuration for common issues.

```bash
dvc-helper stage validate [STAGE_NAME]
```

| Argument | Description |
|----------|-------------|
| `STAGE_NAME` | Stage name to validate (omitting validates all stages) |

**Validation Checks:**

- **Missing dependencies** — files listed as deps that do not exist
- **Missing outputs** — outputs that should exist but don't
- **Duplicate outputs** — the same output path produced by multiple stages
- **Circular dependencies** — cycles in the stage dependency graph (detected via `networkx`)
- **Invalid commands** — executables not found in PATH
- **Missing scripts** — script files referenced in commands that don't exist
- **Empty commands** — stages with no command defined
- **Duplicate stage names** — multiple stages with the same name

**Example:**

```bash
dvc-helper stage validate
```

```
Validation Results:
✖ [WARNING] Dependency 'data/raw.csv' not found.
  Stage: preprocess
  Suggestion: Ensure 'data/raw.csv' is generated by an upstream stage.
✖ [ERROR] Executable 'python3' not found in PATH.
  Stage: train
  Suggestion: Install python3 or update the command.
⚠ [WARNING] Output 'models/model.pt' is produced by multiple stages: train, train_gpu
  Suggestion: Ensure only one stage produces this output.
✔ No issues found for stage: evaluate
```

---

#### `dvc-helper stage freeze`

Freeze a stage so it is skipped during `dvc repro`.

```bash
dvc-helper stage freeze STAGE_NAME
```

| Argument | Description |
|----------|-------------|
| `STAGE_NAME` | Name of the stage to freeze |

A frozen stage is treated as unchanged even if its dependencies have changed. This is useful for stages that are known to be stable and would waste time re-running.

```bash
dvc-helper stage freeze evaluate
```

---

#### `dvc-helper stage unfreeze`

Unfreeze a previously frozen stage.

```bash
dvc-helper stage unfreeze STAGE_NAME
```

| Argument | Description |
|----------|-------------|
| `STAGE_NAME` | Name of the stage to unfreeze |

```bash
dvc-helper stage unfreeze evaluate
```

---

#### `dvc-helper stage graph`

Visualize the stage dependency graph.

```bash
dvc-helper stage graph [STAGE_NAME] [OPTIONS]
```

| Argument | Description |
|----------|-------------|
| `STAGE_NAME` | Show upstream/downstream for a specific stage |

| Option | Alias | Description |
|--------|-------|-------------|
| `--upstream` | `-u` | Show only upstream dependencies |
| `--downstream` | `-d` | Show only downstream dependents |
| `--format` | `-f` | Output format: `ascii`, `mermaid`, `graphviz` (default: `ascii`) |

**Examples:**

```bash
# Full ASCII graph
dvc-helper stage graph
```

```
  preprocess (root)
    ↑ data/raw.csv

  train
    ↑ preprocess (via data/processed.csv)

  evaluate
    ↑ train (via models/model.pt)
```

```bash
# Mermaid graph
dvc-helper stage graph --format mermaid
```

```mermaid
graph TD;
  preprocess[preprocess];
  train[train];
  evaluate[evaluate];
  preprocess --> train;
  train --> evaluate;
```

```bash
# Graphviz format
dvc-helper stage graph --format graphviz
```

```dot
digraph G {
  rankdir=TB;
  node [style=rounded];
  "preprocess";
  "train";
  "evaluate";
  "preprocess" -> "train";
  "train" -> "evaluate";
}
```

```bash
# Upstream/downstream for a specific stage
dvc-helper stage graph evaluate --upstream
```

```
Upstream of 'evaluate':
  ← train
  ← preprocess
```

---

#### `dvc-helper stage doctor`

Run diagnostics on stages (alias for `stage validate`).

```bash
dvc-helper stage doctor [STAGE_NAME]
```

Identical in behavior to `dvc-helper stage validate`.

---

### 6.3 Experiment Commands

DVC experiments let you run, compare, and manage variations of your pipeline without creating Git branches.

---

#### `dvc-helper exp run`

Run a DVC experiment.

```bash
dvc-helper exp run [ARGS]...
```

| Argument | Description |
|----------|-------------|
| `ARGS` | Extra arguments passed directly to `dvc exp run` |

**Example:**

```bash
# Run experiment with modified parameters
dvc-helper exp run --set-param training.lr=0.01

# Queue an experiment
dvc-helper exp run --queue
```

---

#### `dvc-helper exp queue`

Queue experiments for execution.

```bash
dvc-helper exp queue [ARGS]...
```

---

#### `dvc-helper exp show`

List all experiments.

```bash
dvc-helper exp show [ARGS]...
```

---

#### `dvc-helper exp compare`

Compare experiments side-by-side.

```bash
dvc-helper exp compare [ARGS]...
```

Shows metrics for all experiments in a table, making it easy to identify the best-performing configuration.

---

#### `dvc-helper exp apply`

Apply an experiment's changes to the workspace.

```bash
dvc-helper exp apply EXP_NAME
```

| Argument | Description |
|----------|-------------|
| `EXP_NAME` | Name of the experiment to apply |

---

#### `dvc-helper exp remove`

Remove an experiment.

```bash
dvc-helper exp remove EXP_NAME
```

| Argument | Description |
|----------|-------------|
| `EXP_NAME` | Name of the experiment to remove |

---

#### `dvc-helper exp branch`

Create a Git branch from an experiment.

```bash
dvc-helper exp branch EXP_NAME BRANCH_NAME
```

| Argument | Description |
|----------|-------------|
| `EXP_NAME` | Experiment name |
| `BRANCH_NAME` | Name for the new Git branch |

---

### 6.4 Pipeline Commands

---

#### `dvc-helper repro`

Reproduce the pipeline (or a specific stage).

```bash
dvc-helper repro [STAGE] [OPTIONS]
```

| Argument | Description |
|----------|-------------|
| `STAGE` | Stage to reproduce (reproduces entire pipeline if omitted) |

| Option | Alias | Description |
|--------|-------|-------------|
| `--downstream` | `-d` | Reproduce downstream stages as well |
| `--dry` | `-n` | Dry run — show what would be executed without running |
| `--force-downstream` | | Force reproduction of downstream stages even if unchanged |

**Examples:**

```bash
# Reproduce entire pipeline
dvc-helper repro

# Reproduce a single stage
dvc-helper repro train

# Reproduce a stage and all downstream stages
dvc-helper repro preprocess --downstream

# Dry run to see what would change
dvc-helper repro --dry
```

The dry run mode shows exactly why each stage will re-run:

```
$ dvc-helper repro --dry

Stage 'preprocess' is unchanged.
Stage 'train' will be reproduced:
  - deps: params.yaml changed
Stage 'evaluate' will be reproduced:
  - deps: models/model.pt changed (generated by train)
```

---

#### `dvc-helper status`

Show pipeline status (which stages have changed).

```bash
dvc-helper status [STAGE] [OPTIONS]
```

| Argument | Description |
|----------|-------------|
| `STAGE` | Specific stage to check |

| Option | Alias | Description |
|--------|-------|-------------|
| `--cloud` | `-c` | Check status against remote storage |
| `--cache` | | Check cache status |

**Example:**

```bash
dvc-helper status
```

```
preprocess — ✔ unchanged
train — ✖ changed (params.yaml modified)
evaluate — ✔ unchanged
```

---

#### `dvc-helper dag`

Display the full pipeline DAG.

```bash
dvc-helper dag [OPTIONS]
```

| Option | Alias | Description |
|--------|-------|-------------|
| `--format` | `-f` | Output format: `ascii`, `mermaid`, `graphviz` (default: `ascii`) |

Same visualization as `dvc-helper stage graph`, but at the pipeline level.

---

#### `dvc-helper pipeline dag`

Alternative command for displaying the pipeline DAG.

```bash
dvc-helper pipeline dag [OPTIONS]
```

| Option | Alias | Description |
|--------|-------|-------------|
| `--format` | `-f` | Output format: `ascii`, `mermaid`, `graphviz` |

---

### 6.5 Metrics, Params, Plots

---

#### `dvc-helper metrics`

Show or compare DVC metrics.

```bash
dvc-helper metrics [ACTION]
```

| Argument | Description (Default: `show`) |
|----------|-------------------------------|
| `ACTION` | `show` — display current metrics, `diff` — compare with previous commit |

**Examples:**

```bash
dvc-helper metrics show
dvc-helper metrics diff
```

---

#### `dvc-helper params`

Show or compare DVC parameters.

```bash
dvc-helper params [ACTION]
```

| Argument | Description (Default: `show`) |
|----------|-------------------------------|
| `ACTION` | `show` — display current params, `diff` — compare with previous commit |

**Examples:**

```bash
dvc-helper params show
dvc-helper params diff
```

---

#### `dvc-helper plots`

Show or compare DVC plots.

```bash
dvc-helper plots [ACTION] [OPTIONS]
```

| Argument | Description (Default: `show`) |
|----------|-------------------------------|
| `ACTION` | `show` — display plots, `diff` — compare with previous commit |

| Option | Alias | Description |
|--------|-------|-------------|
| `--template` | `-t` | Plot template (e.g., `linear`, `confusion`, `scatter`) |

**Examples:**

```bash
dvc-helper plots show
dvc-helper plots diff --template confusion
```

---

### 6.6 Remote & Cache

---

#### `dvc-helper remote`

Manage DVC remote storage configurations.

```bash
dvc-helper remote [ACTION] [NAME] [OPTIONS]
```

| Argument | Description (Default: `list`) |
|----------|-------------------------------|
| `ACTION` | `add`, `list`, `remove` |
| `NAME` | Remote name (required for `add` and `remove`) |

| Option | Alias | Description |
|--------|-------|-------------|
| `--url` | `-u` | Remote URL/path |
| `--type` | `-t` | Remote type (s3, gcs, azure, ssh, local, minio, gdrive) |
| `--interactive` | `-i` | Interactive setup wizard |

**Interactive Mode:**

```bash
dvc-helper remote --interactive
```

```
Remote Setup Wizard
Remote name [myremote]:
> production

Remote type (s3/gcs/azure/ssh/local/minio/gdrive) [s3]:
> s3

Remote URL/path:
> s3://my-bucket/dvc-storage

Set as default remote? [Y/n]:
> Y
```

**Non-Interactive Examples:**

```bash
# List remotes
dvc-helper remote list

# Add a remote
dvc-helper remote add myremote --url s3://bucket/path --type s3

# Add with default
dvc-helper remote add myremote --url /local/path --type local
```

---

#### `dvc-helper cache`

Manage DVC cache.

```bash
dvc-helper cache [ACTION] [TARGET] [OPTIONS]
```

| Argument | Description (Default: `status`) |
|----------|---------------------------------|
| `ACTION` | `gc` — garbage collect, `checkout` — restore files from cache, `commit` — record files to cache, `verify` — verify cache integrity, `clean` — clean cache |
| `TARGET` | Target stage or file (for `checkout`, `commit`) |

| Option | Alias | Description |
|--------|-------|-------------|
| `--force` | `-f` | Force the action |

**Examples:**

```bash
# Garbage collect unused cache
dvc-helper cache gc

# Checkout files from cache
dvc-helper cache checkout

# Commit current data to cache
dvc-helper cache commit

# Verify cache integrity
dvc-helper cache verify

# Clean temporary cache files
dvc-helper cache clean
```

---

### 6.7 AI Assistance

---

#### `dvc-helper ai`

AI-powered analysis and suggestions for your DVC pipeline.

```bash
dvc-helper ai [ACTION] [PATH]
```

| Argument | Description (Default: `analyze`) |
|----------|----------------------------------|
| `ACTION` | `analyze` — analyze a file, `suggest` — suggest pipeline stages, `project` — analyze the full project |
| `PATH` | File path to analyze (required for `analyze`) |

**Analyze a file:**

```bash
dvc-helper ai analyze src/train.py
```

```
AI Analysis: train
  Confidence: 85%
  Command: python src/train.py
  Explanation: Detected 12 imports; Detected 3 dependencies; Detected 2 outputs; Detected 5 I/O operations

Dependencies:
  ✔ src/train.py
  ✔ src/model.py
  ✔ params.yaml

Outputs:
  ● models/model.pt
  ● predictions.csv

Params:
  ● params.yaml

Metrics:
  ● metrics.json

Plots:
  ● plots/loss.csv
```

**Get pipeline suggestions:**

```bash
dvc-helper ai suggest
```

Scans all Python files in the project and suggests stages for each one, with detected dependencies, outputs, and metrics.

**Analyze the full project:**

```bash
dvc-helper ai project
```

```
Project Analysis
  Structure: src, data, models, config, notebooks
  Python files: 8
  Config files: 3
  Data files: 12
  Existing stages: 3
  Git: yes
  DVC: yes
```

---

## 7. Architecture

### 7.1 Package Structure

```
dvc_helper/                        # Main package
│
├── __init__.py                    # Version and metadata
│
├── cli/
│   ├── __init__.py
│   └── main.py                    # Typer CLI: 34+ commands across 3 apps
│
├── commands/                      # Reserved for future command plugins
│   └── __init__.py
│
├── parser/                        # Intelligent analysis engine
│   ├── __init__.py
│   ├── ast_parser.py              # Python AST parser (imports, calls, I/O)
│   ├── dependency_detector.py     # Auto-detect files read by scripts
│   ├── output_detector.py         # Auto-detect files written by scripts
│   ├── metrics_detector.py        # Scan for metric files
│   ├── params_detector.py         # Detect CLI args and params.yaml
│   ├── plot_detector.py           # Scan for plot files
│   ├── argument_detector.py       # Detect argparse/click/typer/fire args
│   └── matrix_detector.py         # Matrix stage builder
│
├── stage/                         # Stage lifecycle management
│   ├── __init__.py
│   ├── base.py                    # DvcYamlManager (YAML CRUD + serialization)
│   ├── create.py                  # Stage creation with interactive wizard
│   ├── update.py                  # Stage update
│   ├── delete.py                  # Stage deletion
│   ├── rename.py                  # Stage rename
│   ├── duplicate.py               # Stage duplication
│   ├── validate.py                # Validation + circular dep detection
│   └── graph.py                   # DAG (networkx, ASCII, Mermaid, Graphviz)
│
├── dvc/                           # DVC CLI integration
│   ├── __init__.py
│   └── integration.py             # DvcInit, DvcRepro, DvcStatus, DvcExp,
│                                  # DvcRemote, DvcCache, DvcMetrics,
│                                  # DvcParams, DvcPlots, DvcPipeline
│
├── ai/                            # AI analysis
│   ├── __init__.py
│   └── analyzer.py                # AiAnalyzer, ProjectAnalyzer
│
├── config/                        # Configuration
│   ├── __init__.py
│   ├── models.py                  # Pydantic models (StageConfig, DvcYaml, etc.)
│   └── settings.py                # User settings (platformdirs, JSON)
│
└── utils/                         # Shared utilities
    ├── __init__.py
    ├── helpers.py                 # CLI parsing, file ops, DVC/Git checks
    ├── display.py                 # Rich terminal (tables, panels, progress)
    └── logging.py                 # Logging (file + colored console)
```

### 7.2 Design Principles

- **SOLID Principles** — Single responsibility, open/closed, Liskov substitution, interface segregation, dependency inversion
- **Clean Architecture** — Separation of concerns between CLI, analysis, stage management, and DVC integration
- **Pydantic Models** — All configuration and analysis results are type-checked Pydantic models
- **Rich Terminal** — Beautiful output with colors, tables, panels, progress bars, syntax highlighting
- **Cross-Platform** — Works on Windows, Linux, and macOS
- **Plugin-Friendly** — Modular architecture allows extending with new parsers, detectors, and commands

### 7.3 Data Flow

```
User Input (CLI)
      │
      ▼
  [Typer CLI] ──► Parses args/options
      │
      ├──► stage create ──► StageCreator
      │                          │
      │                          ├──► analyze_command()
      │                          │       ├──► DependencyDetector
      │                          │       ├──► OutputDetector
      │                          │       ├──► MetricsDetector
      │                          │       ├──► ParamsDetector
      │                          │       ├──► PlotDetector
      │                          │       ├──► ArgumentDetector
      │                          │       └──► AstParser
      │                          │
      │                          ├──► interactive prompts (questionary)
      │                          │
      │                          └──► DvcYamlManager.save()
      │                                  └──► write_yaml() + clean_for_yaml()
      │
      ├──► repro ──► DvcRepro.run()
      │                  └──► subprocess("dvc repro")
      │
      ├──► ai analyze ──► AiAnalyzer
      │                       └──► uses all detectors + AstParser
      │
      └──► stage graph ──► StageGraph
                              └──► networkx → ASCII/Mermaid/Graphviz
```

---

## 8. Development

### 8.1 Setup

```bash
git clone https://github.com/anomalyco/dvc-helper.git
cd dvc-helper
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate
pip install -e ".[dev]"
```

### 8.2 Running Tests

```bash
pytest tests/ -v
```

All 118+ tests should pass.

### 8.3 Code Style

```bash
pip install black isort
black dvc_helper/ tests/
isort dvc_helper/ tests/
```

The project follows:
- **Black** for code formatting (line length: 100)
- **isort** for import sorting (profile: black)
- **Type hints** everywhere (Python 3.10+ syntax)
- **Pydantic** for all data models
- **SOLID** principles throughout

### 8.4 Adding a New Command

1. Add a function with `@app.command()` or `@stage_app.command()` in `cli/main.py`
2. Implement the business logic in the appropriate module under `commands/` or `stage/`
3. Add tests in `tests/`
4. Run `pytest` to verify

### 8.5 Adding a New Detector

1. Create a new file under `parser/` (e.g., `parser/foo_detector.py`)
2. Implement a class with `detect_from_command()` and `deduplicate()` methods
3. Integrate it into `StageCreator._analyze_command()` in `stage/create.py`
4. Add tests in `tests/test_parser/`

---

## 9. Publishing to PyPI

### 9.1 Prerequisites

```bash
pip install build twine
```

Create a PyPI account at [pypi.org/account/register](https://pypi.org/account/register/).

### 9.2 Build

```bash
python -m build
```

This creates `dist/dvc-helper-0.1.0.tar.gz` and `dist/dvc_helper-0.1.0-py3-none-any.whl`.

### 9.3 Upload to Test PyPI (Recommended)

```bash
twine upload --repository testpypi dist/*
```

Test install:

```bash
pip install --index-url https://test.pypi.org/simple/ dvc-helper
```

### 9.4 Upload to PyPI

```bash
twine upload dist/*
```

You will be prompted for your PyPI username and password. Alternatively, use environment variables:

```bash
export TWINE_USERNAME=__token__
export TWINE_PASSWORD=pypi-xxxxxxxxxxxxxxxxxxxx
twine upload dist/*
```

### 9.5 Version Bumping

Update the version in:

1. `dvc_helper/__init__.py` — `__version__ = "0.2.0"`
2. `pyproject.toml` — `version = "0.2.0"`

Then rebuild and re-upload.

---

## 10. FAQ

### What is the difference between dvc-helper and DVC itself?

DVC is the underlying data version control system. dvc-helper is a CLI assistant that sits on top of DVC, making it easier to create and manage pipelines through auto-detection, interactive wizards, and validation. dvc-helper generates valid `dvc.yaml` files that are 100% compatible with DVC.

### Do I need DVC installed to use dvc-helper?

Most features require DVC. `dvc-helper init` will initialize DVC for you if it's installed. However, stage management commands (`stage create`, `stage list`, `stage show`, etc.) only require `dvc.yaml` and work without DVC being installed.

### Can I use dvc-helper with existing DVC projects?

Yes. dvc-helper reads and writes standard `dvc.yaml` files. It will load existing stages, and you can use `stage create`, `stage update`, `stage delete`, etc. to manage them. dvc-helper never overwrites unrelated stages.

### Does dvc-helper work on Windows?

Yes. dvc-helper is tested on Windows, Linux, and macOS. All file paths are normalized to use forward slashes.

### Can I use dvc-helper in CI/CD pipelines?

Yes. All commands support non-interactive mode with `--non-interactive` or `-n` flags, making them suitable for automated pipelines.

### How does the AST analysis work?

dvc-helper parses Python scripts using the built-in `ast` module. It walks the AST tree to identify imports, function calls, class definitions, and assignments. It recognizes known I/O patterns like `pd.read_csv()`, `torch.save()`, `plt.savefig()`, etc. No external dependencies are required for AST parsing.

### What file formats are supported for metrics and plots?

- **Metrics**: JSON, YAML
- **Plots**: CSV, PNG, JPG, SVG, PDF

### Can I customize the auto-detection?

Yes. In non-interactive mode, you can explicitly specify dependencies, outputs, params, metrics, and plots using the `--dep`, `--out`, `--param`, `--metric`, and `--plot` options.

### Is dvc-helper production-ready?

dvc-helper is currently in alpha (v0.1.0). The core features are implemented and tested, but you may encounter edge cases. Please report issues on the [GitHub repository](https://github.com/anomalyco/dvc-helper/issues).

---

## License

MIT License. See `LICENSE` file for details.

## Contributors

- dvc-helper contributors — [GitHub](https://github.com/anomalyco/dvc-helper/graphs/contributors)

---

*Built with Python, Typer, Rich, Pydantic, and ❤️ for the MLOps community.*
