Metadata-Version: 2.4
Name: mkpipe-extractor-mariadb
Version: 0.5.0
Summary: MariaDB extractor for mkpipe.
Author: Metin Karakus
Author-email: metin_karakus@yahoo.com
License: Apache License 2.0
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: Apache Software License
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: mkpipe
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: license
Dynamic: license-file
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# mkpipe-extractor-mariadb

MariaDB extractor plugin for [MkPipe](https://github.com/mkpipe-etl/mkpipe). Reads MariaDB tables via JDBC.

## Documentation

For more detailed documentation, please visit the [GitHub repository](https://github.com/mkpipe-etl/mkpipe).

## License

This project is licensed under the Apache 2.0 License - see the [LICENSE](LICENSE) file for details.

---

## Connection Configuration

```yaml
connections:
  mariadb_source:
    variant: mariadb
    host: localhost
    port: 3306
    database: mydb
    user: myuser
    password: mypassword
```

---

## Table Configuration

```yaml
pipelines:
  - name: mariadb_to_pg
    source: mariadb_source
    destination: pg_target
    tables:
      - name: orders
        target_name: stg_orders
        replication_method: full
        fetchsize: 100000
```

### Incremental Replication

```yaml
      - name: orders
        target_name: stg_orders
        replication_method: incremental
        iterate_column: updated_at
        iterate_column_type: datetime
        partitions_column: id
        partitions_count: 4
        fetchsize: 50000
```

### Custom SQL

```yaml
      - name: orders
        target_name: stg_orders
        replication_method: full
        custom_query: "SELECT id, user_id, total FROM orders WHERE {query_filter}"
```

---

## Read Parallelism

Set `partitions_column` and `partitions_count` to read in parallel via multiple JDBC connections. Partitioning only applies to incremental replication.

---

## All Table Parameters

| Parameter | Type | Default | Description |
|---|---|---|---|
| `name` | string | required | MariaDB table name |
| `target_name` | string | required | Destination table name |
| `replication_method` | `full` / `incremental` | `full` | Replication strategy |
| `iterate_column` | string or list | — | Column(s) for incremental watermark. String for single column or SQL expression, list for multi-column OR logic |
| `iterate_column_type` | `int` / `datetime` | — | Type of `iterate_column` |
| `partitions_column` | string | same as `iterate_column` | Column to split JDBC reads on |
| `partitions_count` | int | `10` | Number of parallel JDBC partitions |
| `fetchsize` | int | `100000` | Rows per JDBC fetch |
| `custom_query` | string | — | Override SQL with `{query_filter}` placeholder |
| `custom_query_file` | string | — | Path to SQL file (relative to `sql/` dir) |
| `tags` | list | `[]` | Tags for selective pipeline execution |
| `pass_on_error` | bool | `false` | Skip table on error instead of failing |



