helpers package

Submodules

helpers.connections module

helpers.connections.execute_sql(pg_str, sql)

Execute a sql query with psycopg2.

Args:

  1. pg_str: connection string using helper function psycopg_connection_string, returning:host= port= user= dbname= password=

  2. sql: SQL string in triple quotes:

    ```CREATE TABLE foo (bar text)```
Returns:
Executed sql with conn.cursor().execute(sql)
helpers.connections.get_config(full_path)

Get config file with all login credentials, port numbers, etc.

Args:
full_path: provide the full path to the config.ini file, for example authentication/config.ini
Returns:
The entire configuration file to use them with config.get(config_name, 'AUTHURL')
helpers.connections.objectstore_connection(config_full_path, config_name, print_config_vars=None)

Get an objectsctore connection.

Args:
  1. config_full_path: /path_to_config/config.ini or config.ini if in root.
  2. config_name: objectstore
  3. print_config_vars: if set to True: print all variables from the config file
Returns:
An objectstore connection session.
helpers.connections.postgres_engine_pandas(config_full_path, db_config_name)

Pandas uses SQLalchemy, this is the config wrapper to insert config parameters in to_sql queries.

Args:
  1. config_full_path: location of the config.ini file including the name of the file, for example authentication/config.ini
  2. db_config_name: dev or docker to get the ip user/password and port values.
Returns:
The postgres pandas engine to do sql queries with.
helpers.connections.psycopg_connection_string(config_full_path, db_config_name)

Postgres connection string for psycopg2.

Args:
  1. config_full_path: location of the config.ini file including the name of the file, for example authentication/config.ini
  2. db_config_name: dev or docker to get the ip user/password and port values.
Returns:
Returns the psycopg required connection string: ‘PG:host= port= user= dbname= password=’

helpers.files module

helpers.files.create_dir_if_not_exists(directory)

Create directory if it does not yet exists.

Args:
Specify the name of directory, for example: dir/anotherdir
Returns:
Creates the directory if it does not exists, of return the error message.
helpers.files.save_file(data, output_folder, filename)

save_file currently works with: csv, txt, geojson and json as suffixes. It reads the filename suffix and saves the file as the appropriate type.

Args:
  1. data: list of flattened dictionary objects for example: [{id:1, attr:value, attr2:value}, {id:2, attr:value, attr2:value}]
  2. filename: data_output.csv or data_output.json
  3. output_folder: dir/anotherdir
Returns:
Saved the list of objects to the given geojson or csv type.
helpers.files.unzip(path, filename_as_folder=False)

Find all zip files and unzip in root.

Args:
  1. path: set the folder to check for zip files.
  2. filename_as_folder:Set it to True to unzip to subfolders with name of zipfile instead of in the root folder.
Returns:
Unzipped files in the path directory or in the path/name of the zip file.

helpers.json_dict_handlers module

helpers.json_dict_handlers.clean_dict(dictionary, key_name)

Remove a field from a dict based on key name. Args:

  1. dictionary: {id:1, dates:2018-12-02}
  2. key_name: ‘dates’
Returns:
{id:1}
helpers.json_dict_handlers.flatten_json(json_object)

Flatten nested json Object. Args:

1 json_object, for example: {“key”: “subkey”: { “subsubkey”:”value” }}
Returns:
{“key.subkey.subsubkey”:”value”}
Source:
https://towardsdatascience.com/flattening-json-objects-in-python-f5343c794b10
helpers.json_dict_handlers.joinByKeyNames(geojson, dataset, key1, key2)

Insert data from dataset to geojson where key1 from dataset matches key2 from geojson

helpers.json_dict_handlers.jsonPoints2geojson(df, latColumn, lonColumn)

Convert JSON with lat/lon columns to geojson. https://gis.stackexchange.com/questions/220997/pandas-to-geojson-multiples-points-features-with-python

helpers.json_dict_handlers.openJsonArrayKeyDict2FlattenedJson(fileName)

Open json and return array of objects without object value name. For example: [{‘container’:{…}}, {‘container’:{…}}] returns now as [{…},{…}])

helpers.logging module

helpers.logging.logger()

Setup basic logging for console.

Usage:
Initialize the logger by adding the code at the top of your script: logger = logger()

TODO: add log file export

helpers.xml_handlers module

Module contents