Metadata-Version: 2.3
Name: redis_canal
Version: 0.2.0
Summary: redis_canal
Project-URL: Source, https://github.com/wh1isper/redis_canal
Author-email: wh1isper <jizhongsheng957@gmail.com>
License: BSD 3-Clause License
License-File: LICENSE
Keywords: redis_canal
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.10
Requires-Dist: anyio
Requires-Dist: click
Requires-Dist: loguru
Requires-Dist: pluggy
Requires-Dist: pydantic>=2
Requires-Dist: redis>=4.2.0rc1
Provides-Extra: all
Requires-Dist: boto3; extra == 'all'
Requires-Dist: google-cloud-pubsub; extra == 'all'
Provides-Extra: docs
Requires-Dist: autodoc-pydantic; extra == 'docs'
Requires-Dist: pydata-sphinx-theme; extra == 'docs'
Requires-Dist: sphinx; extra == 'docs'
Requires-Dist: sphinx-click; extra == 'docs'
Provides-Extra: pubsub
Requires-Dist: google-cloud-pubsub; extra == 'pubsub'
Provides-Extra: sqs
Requires-Dist: boto3; extra == 'sqs'
Provides-Extra: test
Requires-Dist: boto3; extra == 'test'
Requires-Dist: boto3-stubs[sqs]; extra == 'test'
Requires-Dist: docker; extra == 'test'
Requires-Dist: google-cloud-pubsub; extra == 'test'
Requires-Dist: pytest; extra == 'test'
Requires-Dist: pytest-asyncio; extra == 'test'
Requires-Dist: pytest-cov; extra == 'test'
Description-Content-Type: text/markdown

![](https://img.shields.io/github/license/wh1isper/redis-canal)
![](https://img.shields.io/github/v/release/wh1isper/redis-canal)
![](https://img.shields.io/docker/image-size/wh1isper/redis-canal)
![](https://img.shields.io/pypi/dm/redis-canal)
![](https://img.shields.io/github/last-commit/wh1isper/redis-canal)
![](https://img.shields.io/pypi/pyversions/redis-canal)
[![codecov](https://codecov.io/gh/Wh1isper/redis-canal/graph/badge.svg?token=DI8L42sAMw)](https://codecov.io/gh/Wh1isper/redis-canal)

# redis-canal

Proxy redis stream from one to another through global queue service.

Motivation: Redis provided on the cloud is usually only available within a VPC and does not provide external access. We want to provide a way to synchronize messages across clouds without going through a VPN.

![Architecture Overview](./assets/Architecture.png)

## Supported queue service

- [x] AWS SQS
- [x] Google PubSub

Welcome to contribute more queue service, see [adapter/impl](./redis_canal/adapter/impl/) for more details.

We also support the plugin system, the adapter can also be imported as a third-party library. See [example](./example/extension/custom-adapter/) for more details.

## Install

`pip install redis-canal[all]` for all components.

Or use docker image

`docker pull wh1isper/redis-canal`

## Usage

Use CLI

- `redis-canal stream-to-queue` to start a daemon to consume redis stream and push to global queue
- `redis-canal queue-to-stream` to pull from global queue and push to redis stream

CLI args support `envvar`, which you can refer to [envs.py](./redis_canal/envs.py).

## TODO

- More adapter for Google PubSub, Kafka, etc
- Allows users to more easily configure the adapter(Now only support environment variables, within `Adapter`)

## Develop

Install pre-commit before commit

```
pip install pre-commit
pre-commit install
```

Install package locally

```
pip install -e .[test]
```

Run unit-test before PR

```
pytest -v tests
```

### Develop sqs adapter

You need to configure your AWS account first. See [AWS CLI docs](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) for more details.

Make sure your account has `sqs:*` permission. Then you can run `pytest -v tests/adapter/test_sqs_adapter.py` to test it.
