Metadata-Version: 2.4
Name: knot-migrate
Version: 26.8.1
Summary: Reliable migration tool for S3-compatible and Swift object storage systems
License-Expression: Apache-2.0
Project-URL: Homepage, https://www.nhncloud.com/kr/service/storage/object-storage
Project-URL: Source, https://github.com/nhn/knot-migrate.nhncloud
Keywords: s3,swift,openstack,object-storage,migration,nhn-cloud
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Intended Audience :: System Administrators
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: System :: Archiving :: Mirroring
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: boto3>=1.28.0
Requires-Dist: python-swiftclient>=4.0.0
Requires-Dist: keystoneauth1>=5.0.0
Requires-Dist: python-keystoneclient>=5.0.0
Requires-Dist: PyYAML>=6.0
Requires-Dist: colorlog>=6.0.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-mock>=3.10.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: moto[s3]>=4.0.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Dynamic: license-file

# knot-migrate

English | [한국어](https://github.com/nhn/knot-migrate.nhncloud/blob/main/README.ko.md)

knot-migrate is a command-line tool for migrating objects between Amazon S3 and NHN Object Storage (OpenStack Swift). It transfers objects in parallel across multiple processes, resumes after an interruption, and verifies integrity with ETags.

## Why KNOT?

**KNOT's Not Object Transfer — it's object migration.**

- Reliable migration with retry, resume, and integrity verification
- Metadata-aware transfer between heterogeneous object storage systems
- Designed for live migration with minimal operational disruption

A knot ties two distant ends together — just as KNOT connects the S3 and Swift ecosystems. The name also nods to the Red Knot, a migratory bird that reliably travels thousands of kilometers across continents. And finally… why not?

## Installation

Requires Python 3.10 or later.

```bash
pip install knot-migrate
knot-migrate --help
```

### From source (development)

Clone the repository and run the following in the repository root:

```bash
python3 -m venv .venv
source .venv/bin/activate
pip install -e "."
```

## Configuration

The example below sets up an S3 → Swift migration.

```yaml
source:
  type: s3
  config:
    region: "ap-northeast-2"
    bucket: "YOUR_BUCKET"
    access_key: "YOUR_ACCESS_KEY" # Optional (supports AWS credential chain)
    secret_key: "YOUR_SECRET_KEY" # Optional

destination:
  type: swift
  config:
    auth_url: "https://api-identity.infrastructure.cloud.toast.com/v2.0"
    tenant_id: "YOUR_TENANT_ID"
    username: "YOUR_USERNAME"
    password: "YOUR_PASSWORD"
    container: "YOUR_CONTAINER"
    region: "KR1"
    auth_version: "2"

execution:
  workers: 8
  threads_per_worker: 5
  batch_size: 100
  segment_size: "5MB"
  max_consecutive_failures: 10
```

If the `execution` section is omitted, the defaults are used (`workers`: 8, `threads_per_worker`: 5, `batch_size`: 100, `segment_size`: `"5MB"`, `max_consecutive_failures`: 10).
For S3 authentication, you can specify a `profile` instead of the `access_key`/`secret_key` pair, or omit the credentials entirely — in which case the boto3 credential chain is used automatically.
To migrate in the opposite direction, swap the `type` values of `source` and `destination`.

See [docs/en/configuration.md](https://github.com/nhn/knot-migrate.nhncloud/blob/main/docs/en/configuration.md) for the full list of configuration options.

## Usage

```bash
# Run in the background
knot-migrate start config.yaml -d
```

```text
Starting migration in background...
  Job ID: knot-migrate-a3f8d9c2-20260210_143000
  PID: 12345
  Log file: ~/.knot-jobs/knot-migrate-a3f8d9c2-20260210_143000/main.log
```

```bash
# Check the job status
knot-migrate status <job-id>

# Follow the log in real time
tail -f ~/.knot-jobs/<job-id>/main.log

# Pause and resume (omitting the config reuses the settings saved at start)
knot-migrate pause <job-id>
knot-migrate resume <job-id>

# Inspect and retry failed objects
knot-migrate failures <job-id>
knot-migrate retry <job-id>

# List jobs
knot-migrate list
```

The options in the `execution` section of config.yaml can be overridden from the command line.

```bash
knot-migrate start config.yaml --workers 8 --segment-size 100MB -d
```

## CLI Commands

| Command | Description |
| --- | --- |
| `knot-migrate start <config> [-d]` | Starts a new migration. |
| `knot-migrate pause <job-id>` | Pauses a running job. |
| `knot-migrate resume <job-id> [-c <config>]` | Resumes a paused job. Omitting the config reuses the settings saved at start. |
| `knot-migrate retry <job-id> [-c <config>]` | Retries failed objects. Omitting the config reuses the original job's settings. |
| `knot-migrate status <job-id>` | Shows the status of a job. |
| `knot-migrate list` | Lists all jobs. |
| `knot-migrate failures <job-id>` | Lists failed objects. |
| `knot-migrate update <job-id> --description TEXT` | Updates the job description. |
| `knot-migrate delete <job-id>` | Deletes a completed job. |
| `knot-migrate clear` | Deletes all completed jobs at once. |

- `-d` / `--daemon`: Runs the job in the background.
- `-v` / `--verbose`: Enables DEBUG logging.
- `--json`: Prints output in JSON format (`status`, `list`, `failures`).

See [docs/en/cli-reference.md](https://github.com/nhn/knot-migrate.nhncloud/blob/main/docs/en/cli-reference.md) for the full list of command options.

## Logs and Job Data

Job data is stored under `~/.knot-jobs/<job-id>/` by default.
The location can be changed with the `KNOT_MIGRATE_JOB_DIR` environment variable.

```text
~/.knot-jobs/<job-id>/
├── status.json          ← job status and progress counters
├── failed.jsonl         ← failed object records
├── listing_cursor.json  ← object listing progress
├── main.log             ← logs at INFO level and above
├── main-debug.log       ← logs at DEBUG level and above
└── workers/<uuid>/      ← per-worker logs
```

See [docs/en/job-management.md](https://github.com/nhn/knot-migrate.nhncloud/blob/main/docs/en/job-management.md) for details.
