Metadata-Version: 2.1
Name: geolibs-dator
Version: 0.0.6
Summary: GeoLibs Dator - A data extractor
Home-page: https://github.com/GeographicaGS/GeoLibs-Dator
License: MIT
Author: Geographica
Author-email: hello@geographica.com
Requires-Python: >=3.6,<4.0
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Requires-Dist: PyYAML (>=5.1,<6.0)
Requires-Dist: carto (>=1.4,<2.0)
Requires-Dist: cartoframes (>=0.9.2,<0.10.0)
Requires-Dist: google-cloud-bigquery (>=1.11,<2.0)
Requires-Dist: marshmallow (>=2.19,<3.0)
Requires-Dist: numpy (>=1.17,<2.0)
Requires-Dist: pandas (>=0.24.2,<0.25.0)
Requires-Dist: psycopg2-binary (>=2.8.2,<3.0.0)
Requires-Dist: sqlalchemy (>=1.1.15,<2.0.0)
Description-Content-Type: text/markdown

# GeoLibs-Dator
Dator, a data extractor (ETL as a library), that uses Pandas' DataFrames as in memory temporal storage.

### Features
| Source | Extract | Transform | Load |
| --- | --- | --- | --- |
| BigQuery | Y | Y |  |
| CARTO | Y | Y | Y* |
| CSV | Y |  | Y |
| Pandas |  | Y |  |
| PostgreSQL | Y | Y | Y |

_* Note:_ We are waiting for the append feature on [CARTOframes](https://github.com/CartoDB/cartoframes), because the one we are using is a _ñapa_.

### Configuration
Create a `config.yml` file using the `config.example.yml` one as guide. You can find in that one all the possible ETL cases.

If you are using BigQuery in your ETL process, you need to add a `GOOGLE_APPLICATION_CREDENTIALS` environment variable with the path to your Google Cloud's `credentials.json` file.

You can test them with the `example.py` file.

#### Example

*dator_config.yml*

```yml
datastorages:
  bigquery_input:
    type: bigquery
    data:
      query: SELECT * FROM `dataset.table` WHERE updated_at >= '2019-05-04T00:00:00Z' AND updated_at < '2019-06-01T00:00:00Z';

  carto_input:
    type: carto
    credentials:
      url: https://domain.com/user/user/
      api_key: api_key
    data:
      table: table

  postgresql_input:
    credentials:
      ...
    data:
      query: SELECT * FROM somewhere;
      types:
        - name: timeinstant
          type: datetime
        - name: fillinglevel
          type: float
        - name: temperature
          type: int
        - name: category
          type: str

  carto_output:
    type: carto
    credentials:
      url: https://domain.com/user/user/
      api_key: api_key
    data:
      table: table
      append: false

transformations:
  bigquery_agg:
    type: bigquery
    time:
      field: updated_at
      start: "2019-05-02T00:00:00Z"  # As string or YAML will parse them as DateTimes
      finish: "2019-05-03T00:00:00Z"
      step: 5 MINUTE
    aggregate:
      by:
        - container_id
        - updated_at
      fields:
        field_0: avg
        field_1: max

extract: bigquery_input
transform: bigquery_agg
load: carto_output
```

### How to use

This package is designed to accomplish ETL operations in three steps:

#### Extract

The extract method is a default method, this means although this method can be overwritten, by default, it must work via config.

(This section under construction)

#### Transform

(This section under construction)

#### Load

The load method is a default method, this means although this method can be overwritten, by default, it must work via config. It can receive 2 parameters, the Pandas dataframe and a dictionary with extra info.

#### Example

*app.py*

```python
from dator import Dator

dator = Dator('/usr/src/app/dator_config.yml')
df = dator.extract()
df = dator.transform(df)
dator.load(df)
```

*app.py* with extra info

```python
from dator import Dator

def upsert_method:
  pass

dator = Dator('/usr/src/app/dator_config.yml')
df = dator.extract()
df = dator.transform(df)
dator.load(df, {'method': upsert_method})
```

### TODOs
- Better doc.
- Tests.

