Metadata-Version: 2.4
Name: meltano-tap-cratedb
Version: 0.0.0
Summary: A Singer tap / Meltano extractor for CrateDB, built with the Meltano SDK, and based on the Meltano PostgreSQL tap.
Author-email: CrateDB Engineering <office@crate.io>
Project-URL: changelog, https://github.com/crate/meltano-tap-cratedb/blob/main/CHANGES.md
Project-URL: documentation, https://github.com/crate/meltano-tap-cratedb
Project-URL: homepage, https://github.com/crate/meltano-tap-cratedb
Project-URL: repository, https://github.com/crate/meltano-tap-cratedb
Keywords: CrateDB,data-loading,data-processing,data-toolkit,data-transfer,data-transformation,ELT,ETL,io,Meltano,Meltano SDK,PostgreSQL,Singer
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Customer Service
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Education
Classifier: Intended Audience :: End Users/Desktop
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Manufacturing
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: System Administrators
Classifier: Intended Audience :: Telecommunications Industry
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: Unix
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Adaptive Technologies
Classifier: Topic :: Communications
Classifier: Topic :: Database
Classifier: Topic :: Documentation
Classifier: Topic :: Education
Classifier: Topic :: Internet
Classifier: Topic :: Office/Business
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development :: Pre-processors
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Software Development :: Version Control
Classifier: Topic :: System :: Archiving
Classifier: Topic :: System :: Benchmark
Classifier: Topic :: System :: Clustering
Classifier: Topic :: System :: Distributed Computing
Classifier: Topic :: System :: Logging
Classifier: Topic :: System :: Monitoring
Classifier: Topic :: System :: Networking
Classifier: Topic :: System :: Systems Administration
Classifier: Topic :: Text Processing
Classifier: Topic :: Utilities
Requires-Python: <3.12,>=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: crate[sqlalchemy]<1,>=0.34
Requires-Dist: cratedb-toolkit<0.0.42
Requires-Dist: importlib-resources; python_version < "3.9"
Requires-Dist: meltanolabs-tap-postgres==0.0.6
Provides-Extra: develop
Requires-Dist: black<24; extra == "develop"
Requires-Dist: mypy==1.7.1; extra == "develop"
Requires-Dist: poethepoet<1; extra == "develop"
Requires-Dist: pyproject-fmt<2; extra == "develop"
Requires-Dist: ruff<0.15; extra == "develop"
Requires-Dist: validate-pyproject<1; extra == "develop"
Provides-Extra: release
Requires-Dist: build<2; extra == "release"
Requires-Dist: twine<5; extra == "release"
Provides-Extra: test
Requires-Dist: faker<21.0.0,>=18.5.1; extra == "test"
Requires-Dist: pendulum~=2.1; extra == "test"
Requires-Dist: pytest<8; extra == "test"
Requires-Dist: pytest-cov<5; extra == "test"
Requires-Dist: pytest-mock<4; extra == "test"
Requires-Dist: singer-sdk[testing]; extra == "test"
Dynamic: license-file

# Singer tap / Meltano extractor for CrateDB

[![Tests](https://github.com/crate/meltano-tap-cratedb/actions/workflows/main.yml/badge.svg)](https://github.com/crate/meltano-tap-cratedb/actions/workflows/main.yml)
[![Test coverage](https://img.shields.io/codecov/c/gh/crate/meltano-tap-cratedb.svg)](https://codecov.io/gh/crate/meltano-tap-cratedb/)
[![Python versions](https://img.shields.io/pypi/pyversions/meltano-tap-cratedb.svg)](https://pypi.org/project/meltano-tap-cratedb/)

[![License](https://img.shields.io/github/license/crate/meltano-tap-cratedb.svg)](https://github.com/crate/meltano-tap-cratedb/blob/main/LICENSE)
[![Status](https://img.shields.io/pypi/status/meltano-tap-cratedb.svg)](https://pypi.org/project/meltano-tap-cratedb/)
[![PyPI](https://img.shields.io/pypi/v/meltano-tap-cratedb.svg)](https://pypi.org/project/meltano-tap-cratedb/)
[![Downloads](https://pepy.tech/badge/meltano-tap-cratedb/month)](https://pepy.tech/project/meltano-tap-cratedb/)


## About

A [Singer] tap for [CrateDB], built with the [Meltano SDK] for custom extractors
and loaders, and based on the [Meltano PostgreSQL tap].

In Singer ELT jargon, a "tap" conceptually wraps a data source, where you
"extract" data from.

In order to learn more about Singer, Meltano, and friends, please
navigate to the [Singer Intro].

## Install

Usually, you will not install this package directly, but rather on behalf
of a Meltano project. A corresponding snippet is outlined in the next section.

After adding it to your `meltano.yml` project definition file, you can install
all defined components and their dependencies with a single command.

```shell
meltano install
```

## Usage

You can run the CrateDB Singer tap `tap-cratedb` by itself, or
in a pipeline using Meltano.

### Meltano

Using the `meltano add` subcommand, you can add the plugin to your
Meltano project.
```shell
meltano add extractor tap-cratedb
```
NB: It will only work like this when released and registered on Meltano Hub.
    In the meanwhile, please add the configuration snippet manually.


#### CrateDB Cloud

In order to connect to [CrateDB Cloud], configure the `sqlalchemy_url` setting
within your `meltano.yml` configuration file like this.
```yaml
- name: tap-cratedb
  namespace: cratedb
  variant: cratedb
  pip_url: meltano-tap-cratedb
  config:
    sqlalchemy_url: "crate://admin:K4IgMXNvQBJM3CiElOiPHuSp6CiXPCiQYhB4I9dLccVHGvvvitPSYr1vTpt4@example.aks1.westeurope.azure.cratedb.net:4200?ssl=true"}
```


#### On localhost
In order to connect to a standalone or on-premise instance of CrateDB, configure
the `sqlalchemy_url` setting within your `meltano.yml` configuration file like this.
```yaml
- name: tap-cratedb
  namespace: cratedb
  variant: cratedb
  pip_url: meltano-tap-cratedb
  config:
    sqlalchemy_url: crate://crate@localhost/
```

Then, invoke the pipeline by using `meltano run`, similar like this.
```shell
meltano run tap-cratedb target-csv
```

### Standalone

You can also invoke the adapter standalone by using the `tap-cratedb` program.
This example demonstrates how to export data from the database into a file.

Define the database connection string including credentials in SQLAlchemy format.
```shell
export TAP_CRATEDB_SQLALCHEMY_URL='crate://admin:K4IgMXNvQBJM3CiElOiPHuSp6CiXPCiQYhB4I9dLccVHGvvvitPSYr1vTpt4@example.aks1.westeurope.azure.cratedb.net:4200?ssl=true'
```

Discover all available database streams.
```shell
tap-cratedb --config ENV --discover
```

Export CrateDB's `sys.summits` table.
```shell
tap-cratedb --config ENV --catalog tests/resources/cratedb-summits.json
```

## Development

In order to work on this adapter dialect on behalf of a real pipeline definition,
link your sandbox to a development installation of [meltano-tap-cratedb], and
configure the `pip_url` of the component to point to a different location than the
[vanilla package on PyPI].

Use this URL to directly point to a specific Git repository reference.
```yaml
pip_url: git+https://github.com/crate/meltano-tap-cratedb.git@main
```

Use a `pip`-like notation to link the CrateDB Singer tap in development mode,
so you can work on it at the same time while running the pipeline, and iterating
on its definition.
```yaml
pip_url: --editable=/path/to/sources/meltano-tap-cratedb
```


[600+ connectors]: https://hub.meltano.com/
[Apache Lucene]: https://lucene.apache.org/
[CrateDB]: https://cratedb.com/product
[CrateDB Cloud]: https://console.cratedb.cloud/
[ELT]: https://en.wikipedia.org/wiki/Extract,_load,_transform
[ETL]: https://en.wikipedia.org/wiki/Extract,_transform,_load
[Meltano]: https://meltano.com/
[meltano | Hub]: https://hub.meltano.com/
[Meltano SDK]: https://github.com/meltano/sdk
[Meltano PostgreSQL tap]: https://github.com/MeltanoLabs/tap-postgres
[meltano-tap-cratedb]: https://github.com/crate/meltano-tap-cratedb
[Singer]: https://www.singer.io/
[Singer Spec]: https://hub.meltano.com/singer/spec/
[PipelineWise]: https://transferwise.github.io/pipelinewise/
[PipelineWise Taps]: https://transferwise.github.io/pipelinewise/user_guide/yaml_config.html
[SQLAlchemy]: https://www.sqlalchemy.org/
[vanilla package on PyPI]: https://pypi.org/project/meltano-tap-cratedb/
