Metadata-Version: 2.4
Name: core-cdc
Version: 2.0.2
Summary: This library provides the mechanism to implement Change Data Capture (CDC) services...
Author-email: Alejandro Cora González <alek.cora.glez@gmail.com>
Maintainer: Alejandro Cora González
License-Expression: MIT
Project-URL: Homepage, https://gitlab.com/bytecode-solutions/core/core-cdc
Project-URL: Repository, https://gitlab.com/bytecode-solutions/core/core-cdc
Project-URL: Documentation, https://core-cdc.readthedocs.io/en/latest/
Project-URL: Issues, https://gitlab.com/bytecode-solutions/core/core-cdc/-/issues
Project-URL: Changelog, https://gitlab.com/bytecode-solutions/core/core-cdc/-/blob/master/CHANGELOG.md
Classifier: Intended Audience :: Developers
Classifier: Development Status :: 5 - Production/Stable
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Utilities
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.9
Description-Content-Type: text/x-rst
License-File: LICENSE
Requires-Dist: core-mixins>=2.2.4
Provides-Extra: all
Requires-Dist: core-db[all]>=3.0.1; extra == "all"
Requires-Dist: mysql-replication>=1.0.9; extra == "all"
Provides-Extra: dev
Requires-Dist: core-dev-tools>=1.0.1; extra == "dev"
Requires-Dist: core-tests>=2.0.3; extra == "dev"
Provides-Extra: mongo
Requires-Dist: core-db[mongo]>=3.0.1; extra == "mongo"
Provides-Extra: mysql
Requires-Dist: core-db[mysql]>=3.0.1; extra == "mysql"
Requires-Dist: mysql-replication>=1.0.9; extra == "mysql"
Provides-Extra: snowflake
Requires-Dist: core-db[snowflake]>=3.0.1; extra == "snowflake"
Dynamic: license-file

core-cdc (CDC a.k.a Change Data Capture)
===============================================================================

It provides the core mechanism and required resources to 
implement "Change Data Capture" services...

===============================================================================


.. image:: https://img.shields.io/pypi/pyversions/core-cdc.svg
    :target: https://pypi.org/project/core-cdc/
    :alt: Python Versions

.. image:: https://img.shields.io/badge/license-MIT-blue.svg
    :target: https://gitlab.com/bytecode-solutions/core/core-cdc/-/blob/main/LICENSE
    :alt: License

.. image:: https://gitlab.com/bytecode-solutions/core/core-cdc/badges/release/pipeline.svg
    :target: https://gitlab.com/bytecode-solutions/core/core-cdc/-/pipelines
    :alt: Pipeline Status

.. image:: https://readthedocs.org/projects/core-cdc/badge/?version=latest
    :target: https://readthedocs.org/projects/core-cdc/
    :alt: Docs Status

.. image:: https://img.shields.io/badge/security-bandit-yellow.svg
    :target: https://github.com/PyCQA/bandit
    :alt: Security

|


Installation
===============================================================================

Install from PyPI using pip:

.. code-block:: bash

    pip install core-cdc
    uv pip install core-cdc  # Or using UV...


Features
===============================================================================

**Multi-Database CDC Support**
    - MySQL Binary Log (BinLog) based change capture
    - MongoDB Change Streams for real-time event streaming
    - Extensible processor architecture for additional database engines

**Comprehensive Event Handling**
    - DML operations: INSERT, UPDATE, DELETE
    - DDL operations: CREATE, ALTER, DROP (schemas and tables)
    - Configurable event filtering by operation type

**Flexible Target Replication**
    - MySQL target for database-to-database replication
    - Snowflake target for data warehouse integration
    - Support for multiple simultaneous targets

**Standardized Data Format**
    - Common Record structure for cross-service integration
    - Includes metadata: timestamps, transaction IDs, source position
    - JSON serialization support for streaming and messaging systems

**Production-Ready Features**
    - Built-in error handling and retry mechanisms
    - Comprehensive logging for monitoring and debugging
    - Optional event timestamp column for UPSERT/MERGE operations


Quick Start
===============================================================================

Installation
-------------------------------------------------------------------------------

Install the package:

.. code-block:: bash

    pip install core-cdc
    uv pip install core-cdc     # Or using UV...
    pip install -e ".[dev]"     # For development...


Setting Up Environment
-------------------------------------------------------------------------------

1. Install required libraries:

.. code-block:: bash

    pip install --upgrade pip
    pip install virtualenv

2. Create Python virtual environment:

.. code-block:: bash

    virtualenv --python=python3.12 .venv

3. Activate the virtual environment:

.. code-block:: bash

    source .venv/bin/activate

Install packages
-------------------------------------------------------------------------------

.. code-block:: bash

    pip install .
    pip install -e ".[dev]"

Optional libraries
-------------------------------------------------------------------------------

.. code-block:: bash

    pip install '.[all]'  # For all...
    pip install '.[mysql]'
    pip install '.[mongo]'
    pip install '.[snowflake]'


Check tests and coverage
-------------------------------------------------------------------------------

.. code-block:: bash

    python manager.py run-tests
    python manager.py run-tests --test-type integration
    python manager.py run-coverage

    # Having the docker containers up and running you can execute the functional
    # tests that ensure the CDC services are working as expected...
    python manager.py run-tests --test-type functional --pattern "*.py"


Implemented CDC Engines
===============================================================================

The following database engines have CDC implementations:

Fully Implemented
-------------------------------------------------------------------------------

**MySQL** - Binary Log (BinLog) based CDC
    - Uses ``mysql-replication`` library
    - Captures INSERT, UPDATE, DELETE operations
    - Supports DDL events (CREATE, ALTER, DROP)
    - Fallback mechanism for column name resolution
    - See: `core_cdc/processors/mysql/ <core_cdc/processors/mysql/>`_

**MongoDB** - Change Streams based CDC
    - Uses native MongoDB Change Streams
    - Captures INSERT, UPDATE, DELETE operations
    - Requires replica set configuration
    - Real-time event streaming
    - See: `core_cdc/processors/mongo/ <core_cdc/processors/mongo/>`_

In Development
-------------------------------------------------------------------------------

**MS SQL Server** - Abstract base class defined
    - See: `core_cdc/processors/mssql.py <core_cdc/processors/mssql.py>`_

**Oracle** - Abstract base class defined
    - See: `core_cdc/processors/oracle.py <core_cdc/processors/oracle.py>`_


Contributing
===============================================================================

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Write tests for new functionality
4. Ensure all tests pass: ``pytest -n auto``
5. Run linting: ``pylint core_cdc``
6. Run security checks: ``bandit -r core_cdc``
7. Submit a pull request


License
===============================================================================

This project is licensed under the MIT License. See the LICENSE file for details.


Links
===============================================================================

* **Documentation:** https://core-cdc.readthedocs.io/en/latest/
* **Repository:** https://gitlab.com/bytecode-solutions/core/core-cdc
* **Issues:** https://gitlab.com/bytecode-solutions/core/core-cdc/-/issues
* **Changelog:** https://gitlab.com/bytecode-solutions/core/core-cdc/-/blob/master/CHANGELOG.md
* **PyPI:** https://pypi.org/project/core-cdc/


Support
===============================================================================

For questions or support, please open an issue on GitLab or contact the maintainers.


Authors
===============================================================================

* **Alejandro Cora González** - *Initial work* - alek.cora.glez@gmail.com
