Metadata-Version: 2.1
Name: fsqlfly
Version: 5.0.1
Summary: Flink SQL Job Management Website
Home-page: https://github.com/mrzhangboss/fsqlfly
Author: mrzhangboss
Author-email: 2529450174@qq.com
License: UNKNOWN
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.6.0
Description-Content-Type: text/markdown
Requires-Dist: tornado (<=6.0.4)
Requires-Dist: terminado (<=0.8.3)
Requires-Dist: python-dotenv (<=0.12.0)
Requires-Dist: logzero (<=1.5.0)
Requires-Dist: PyYAML (<=5.3.1)
Requires-Dist: Jinja2 (<=2.11.1)
Requires-Dist: python-magic (<=0.4.15)
Requires-Dist: requests (<=2.23.0)
Requires-Dist: attrs (<=19.3.0)
Requires-Dist: sqlalchemy (<=1.3.15)
Requires-Dist: sqlalchemy-utils (<=0.36.3)
Provides-Extra: airflow
Requires-Dist: apache-airflow ; extra == 'airflow'
Provides-Extra: all
Requires-Dist: PyMySQL (<=0.9.3) ; extra == 'all'
Requires-Dist: psycopg2 (<=2.8.4) ; extra == 'all'
Requires-Dist: canal-python (<=0.4) ; extra == 'all'
Requires-Dist: kafka-python (<=1.4.7,>=1.3.4) ; extra == 'all'
Requires-Dist: protobuf (<=3.11.3) ; extra == 'all'
Requires-Dist: sqlalchemy (<=1.3.15) ; extra == 'all'
Requires-Dist: apache-airflow ; extra == 'all'
Requires-Dist: PyHive (<=0.6.2) ; extra == 'all'
Requires-Dist: thrift (<=0.13.0) ; extra == 'all'
Requires-Dist: sasl (<=0.2.1) ; extra == 'all'
Requires-Dist: thrift-sasl (<=0.4.2,>0.4.1) ; extra == 'all'
Requires-Dist: elasticsearch (<8.0.0,>=7.0.0) ; extra == 'all'
Requires-Dist: happybase (<=1.2.0) ; extra == 'all'
Provides-Extra: canal
Requires-Dist: canal-python (<=0.4) ; extra == 'canal'
Requires-Dist: kafka-python (<=1.4.7,>=1.3.4) ; extra == 'canal'
Requires-Dist: protobuf (<=3.11.3) ; extra == 'canal'
Requires-Dist: sqlalchemy (<=1.3.15) ; extra == 'canal'
Provides-Extra: es
Requires-Dist: elasticsearch (<8.0.0,>=7.0.0) ; extra == 'es'
Provides-Extra: hbase
Requires-Dist: happybase (<=1.2.0) ; extra == 'hbase'
Provides-Extra: hive
Requires-Dist: PyHive (<=0.6.2) ; extra == 'hive'
Requires-Dist: thrift (<=0.13.0) ; extra == 'hive'
Requires-Dist: sasl (<=0.2.1) ; extra == 'hive'
Requires-Dist: thrift-sasl (<=0.4.2,>0.4.1) ; extra == 'hive'
Provides-Extra: mysql
Requires-Dist: PyMySQL (<=0.9.3) ; extra == 'mysql'
Provides-Extra: postgresql
Requires-Dist: psycopg2 (<=2.8.4) ; extra == 'postgresql'
Provides-Extra: test
Requires-Dist: TorMySQL ; extra == 'test'

# Flink SQL Job Management Website


## Display




> require

1. python3.6+
2. flink 1.9.0+ installed (need set `FSQLFLY_FLINK_BIN_DIR` in ~/.fsqlfly or in env)

ps: if you want run multi fsqlfly in one computer, you can set `FSQLFLY` in env , like

    export FSQLFLY=/path/where/you/like
    fsqlfly comand

you can generate a env template by `fsqlfly echoenv [filename]`


> install

    pip install fsqlfly

> init database

    fsqlfly initdb 

> run website

    fsqlfly webserver [--jobdaemon]

ps: if you want daemon all flink sql job(need set publish and available), add `--jobdaemon` in commands


> reset database (warning it'll delete all data)

    fsqlfly resetdb



> support canal consumer(load mysql log data to kafka)

**require install [canal v1.1.4+](https://github.com/alibaba/canal)** 

    pip install fsqlfly[canal]

    fsqlfly runcanal [name or id]

# settings

you can change by write in `env file` (~/.fsqlfly) or just in environment variables (`eg: export name=value`)


Name | Description|Default
---- | --- | ---
FSQLFLY_PASSWORD|admin password(if not set use a random password)|password
FSQLFLY_DB_URL|database connection url(if you set then other is ignore) |None
FSQLFLY_STATIC_ROOT|the dir of static file(if not set then it will be fsqlfly/static) |None
FSQLFLY_FLINK_BIN_DIR|the dir of flink bin dir |/opt/flink/bin
FSQLFLY_FLINK_MAX_TERMINAL|the max value of living terminal  |1000
FSQLFLY_DEBUG| set web debug(if set then set True else False)   |None
FSQLFLY_DEBUG| set web debug(if set then set True else False)   |None
FSQLFLY_WEB_PORT|set http port   |8082
FSQLFLY_FINK_HOST|  flink REST api host  | http://localhost:8081
FSQLFLY_JOB_DAEMON_FREQUENCY| each job check damon time second           | 30
FSQLFLY_JOB_DAEMON_MAX_TRY_ONE_DAY| each job maximum try times in one day            | 3
FSQLFLY_JOB_LOG_DIR| flink job damon log file            | /tmp/fsqlfly_job_log
FSQLFLY_UPLOAD_DIR| upload dir            | ~/.fsqlfly_upload
FSQLFLY_SAVE_MODE_DISABLE| if set then support delete or otherwise            | False 
FSQLFLY_MAIL_ENABLE| send email or not |false
FSQLFLY_MAIL_HOST| smt email host|None
FSQLFLY_MAIL_USER| smt email user|None
FSQLFLY_MAIL_PASSWORD| smt email password |None
FSQLFLY_MAIL_RECEIVERS| smt email receivers sep by ,|None







> connection url detail in [FSQLFLY_DB_URL](https://docs.sqlalchemy.org/en/13/core/engines.html)

ps: the admin token value is `FSQLFLY_PASSWORD` md5 hex value, you can generate it by 

        import hashlib
        md5 = hashlib.md5()
        md5.update(b'password')
        token = md5.hexdigest()


if you want control all flink sql job start and stop by api, you can add token in url or header without login


## API

> need login by token(in request params `token`)

- jobs

        url: /api/job
        method: get


- job control 

      url: /api/transform/<mode(status|start|stop|restart)>/<id or job name>
      method: post


**Beta** you can set `pt` in request body(json format), then will create a unique job 
name for job, if you sql need other format value, we support `jinja2` format 
eg: `insert into tablea select * from table where pt_field = {{ pt }};`
you can send `pt` value in request body.I recommend you control daily job by `airflow`.
If you want kill all `pt` job add `kill_all_pt` in request json body.

PS: `pt` only can contain '0-9a-zA-Z_-' 
PS: `status` api if no `last_run_job_id` or multi running job , api will return `FAILED`  
PS: we will generate a dt variable in your sql template, you can set `execution_date` in request body(json format), eg: (`2020-05-11 23:22:07` or `2020-05-11`)

## Airflow Support

> use dag operator in `fsqlfly.airflow_plugins.FSQLFlyOperator`

example:

    from airflow.models import DAG
    from fsqlfly.airflow_plugins import FSQLFlyOperator

    dag = DAG(
        dag_id='flink_hive_process',
        default_args=args,
        schedule_interval="2 1 * * *",
        dagrun_timeout=timedelta(minutes=60),
        max_active_runs=8,
        concurrency=8
    )

    data = dict(pt="{{ ds_nodash }}")
    http_conn_id = "fsqlplatform"
    token = '{{ var.value.fsqlfly_token }}'
    start_flink_job = FSQLFlyOperator(
        task_id='fink_job',
        job_name='flik_run_in_fsql_fly',
        token=token,
        http_conn_id=http_conn_id,
        data=data,
        dag=dag,
        method='start',  # support restart | start | stop  
        daemon=True,
        parallelism=0,   # if parallelism set not zero then will control the max running job one time
        poke_interval=5,
    )


`token`: fsqlfly token, you can real token, also you can save in `variable` in airflow
`HOST`: airflow connection id , see more in [detail](https://airflow.apache.org/docs/stable/howto/connection/index.html)
`data`: args in flink job

if you want control `connector` by airflow you can use `fsqlfly.airflow_plugins.FSQLFlyConnectorOperator` same usage as upper.



## Quick Start


1. unzip Flink 1.10.0 to /opt/flink
2. `pip install fsqlfly`
3. `fsqlfly echoenv ~/.fsqlfly`
4.  change the value `FSQLFLY_FLINK_BIN_DIR` in `~/.fsqlfly` to your flink bin dir  like `/opt/flink/bin`
5.  `fsqlfly initdb`
6. `fsqlfly runwebserver`
7. open your browser in `http://localhost:8082` the password is `password`


