Metadata-Version: 2.1
Name: pydatafabric
Version: 0.4.39
Summary: SHINSEGAE DataFabric Python Package
Home-page: https://github.com/emartddt/dataplaltform-python-dist
Author: SHINSEGAE DataFabric
Author-email: admin@shinsegae.ai
License: MIT License
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.8,<3.11
Description-Content-Type: text/markdown
Requires-Dist: thrift-sasl ==0.4.3
Requires-Dist: hvac ==0.11.2
Requires-Dist: pyhive[hive] ==0.6.5
Requires-Dist: pyarrow ==6.0.1
Requires-Dist: pandas ==1.3.5
Requires-Dist: db-dtypes <2.0.0,>=0.4.0
Requires-Dist: slackclient >=2.9.0
Requires-Dist: httplib2 >=0.20.0
Requires-Dist: click
Requires-Dist: PyGithub
Requires-Dist: pycryptodome
Requires-Dist: tabulate >=0.8.7
Requires-Dist: grpcio ==1.44.0
Requires-Dist: grpcio-status ==1.44.0
Requires-Dist: sqlalchemy ==1.4.48
Requires-Dist: packaging
Requires-Dist: tqdm >=4.63.0
Requires-Dist: ipywidgets
Requires-Dist: hmsclient-hive-3
Requires-Dist: dvc[s3] ==2.9.5
Requires-Dist: gcsfs
Requires-Dist: google-cloud-bigtable <3.0.0,>=2.0.0
Requires-Dist: google-cloud-monitoring <3.0.0,>=2.0.0
Requires-Dist: google-cloud-datacatalog <4.0.0,>=3.0.0
Requires-Dist: redis
Requires-Dist: testresources
Requires-Dist: python-dateutil >=2.8.2
Requires-Dist: requests <3.0.0,>=2.26.0
Requires-Dist: protobuf <4.0.0,>=3.0.0
Requires-Dist: google-cloud-bigquery <4.0.0,>=3.0.0
Requires-Dist: google-cloud-storage <3.0.0,>=2.0.0
Requires-Dist: google-auth <3.0.0,>=2.0.0
Requires-Dist: google-auth-oauthlib <0.5,>=0.4.1
Requires-Dist: google-api-core <3.0.0,>=2.0.0
Requires-Dist: google-api-python-client <3.0.0,>=2.34.0
Requires-Dist: google-cloud-core <3.0.0,>=2.0.0
Requires-Dist: google-cloud-common <2.0.0,>=1.0.0
Requires-Dist: googleapis-common-protos <2.0.0,>=1.55.0
Provides-Extra: emart
Requires-Dist: seaborn ==0.11.2 ; extra == 'emart'
Requires-Dist: matplotlib ==3.5.1 ; extra == 'emart'
Requires-Dist: openpyxl ==3.0.9 ; extra == 'emart'
Requires-Dist: xgboost ==1.5.2 ; extra == 'emart'
Requires-Dist: scikit-learn ==1.0.2 ; extra == 'emart'
Requires-Dist: bayesian-optimization ==1.2.0 ; extra == 'emart'
Requires-Dist: scipy <1.8.0,>=1.7.3 ; extra == 'emart'
Requires-Dist: numpy <1.22.2,>=1.15.0 ; extra == 'emart'

# SHINSEGAE DataFabric Python Package

[![Linter && Formatting](https://github.com/emartdt/datafabric-python-dist/actions/workflows/Flake8.yml/badge.svg)](https://github.com/emartdt/datafabric-python-dist/actions/workflows/Flake8.yml)
[![Publish to TestPyPI](https://github.com/emartdt/datafabric-python-dist/actions/workflows/TestPyPI.yml/badge.svg)](https://github.com/emartdt/datafabric-python-dist/actions/workflows/TestPyPI.yml)
[![Publish to PyPI](https://github.com/emartdt/datafabric-python-dist/actions/workflows/PyPI.yml/badge.svg)](https://github.com/emartdt/datafabric-python-dist/actions/workflows/PyPI.yml)

This is highly site dependent package. Resources are abstracted into package structure.

## Usage

Get pandas dataframe from parquet file in hdfs
```python
from pydatafabric.ye import parquet_to_pandas

pandas_df = parquet_to_pandas(hdfs_path)
```

Save pandas dataframe as parquet in hdfs
```python
from pydatafabric.ye import get_spark
from pydatafabric.ye import pandas_to_parquet

spark = get_spark()
pandas_to_parquet(pandas_df, hdfs_path, spark)  # we need spark for this operation
spark.stop()
```

Work with spark
```python
from pydatafabric.ye import get_spark

spark = get_spark()
# do with spark session
spark.stop()
```

Work with spark-bigquery-connector
```python
# SELECT
from pydatafabric.gcp import bq_table_to_pandas

pandas_df = bq_table_to_pandas("dataset", "table_name", ["col_1", "col_2"], "2020-01-01", "cust_id is not null")
# INSERT 
from pydatafabric.gcp import pandas_to_bq_table

pandas_to_bq_table(pandas_df, "dataset", "table_name", "2022-02-22")
```

Send slack message
```python
from pydatafabric.ye import slack_send

text = 'Hello'
username = 'airflow'
channel = '#leavemealone'
slack_send(text=text, username=username, channel=channel)
# Send dataframe as text
df = pd.DataFrame(data={'col1': [1, 2], 'col2': [3, 4]})
slack_send(text=df, username=username, channel=channel, dataframe=True)
```

Get bigquery client
```python
from pydatafabric.gcp import get_bigquery_client

bq = get_bigquery_client(project="emart-datafabric")
bq.query(query)
```

IPython BigQuery Magic
```python
from pydatafabric.gcp import import_bigquery_ipython_magic

import_bigquery_ipython_magic()

query_params = {
    "p_1": "v_1",
    "dataset": "common_dev",
}
```
```python
%% bq --params $query_params

SELECT c_1 
FROM {dataset}.user_logs
WHERE c_1 = @p_1
```

Use NES CLI
```bas
nes input_notebook_url -p k1 v1 -p k2 v2 -p k3 v3
```

Use github util
```python
from pydatafabric.ye import get_github_util

g = get_github_util
# query graphql
res = g.query_gql(graph_ql)
# get file in github repository
byte_object = g.download_from_git(github_url_path)
```

## Installation

```sh
$ pip install pydatafabric --upgrade
```

If you would like to install submodules for Emart Inc.

```sh
$ pip install pydatafabric[emart] --upgrade
```


