Metadata-Version: 2.4
Name: al-data-utils
Version: 0.3.1
Summary: Reusable utilities for data engineering (Airflow, Azure, Snowflake, SQL, config).
Author-email: AL Data Labs <pypi@aldatalabs.com>
License-Expression: MIT
Project-URL: Homepage, https://aldatalabs.com
Keywords: airflow,data-engineering,azure,snowflake,sql
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# al-data-utils

Reusable Python utilities for data and automation workflows.

This repository contains independent utility modules for sync operations, git automation, SQL work, logging helpers, and MSSQL schema snapshot generation.

## Python and Setup

- Python: 3.10+
- Packaging: setuptools (src layout)

Install locally in editable mode:

```bash
pip install -e .
```

## Project Structure

- src/al_data_utils
- examples
- development_guides
- tools

Main utility groups under src/al_data_utils:

- airflow_utils
- backup_utils
- config_utils
- files_sync_utils
- geo_utils
- git_utils
- logging_utils
- ones_api_utils
- sql_utils
- tg_utils

## Examples

Runnable examples are available in the `examples` folder for the main utility groups.
Use them as reference implementations for local scripts and automation jobs.

## MSSQL Schema Snapshot

The snapshot tool exports SQL objects from MSSQL into a filesystem layout and syncs staging output to destination folders.

Current behavior highlights:

- Supports single DB and multi-DB runs in one execution.
- Always writes outputs into per-DB folders.
- Uses staging first, then syncs to final output.
- Supports cleanup of staging folders.
- Supports sync log retention cleanup.
- Supports include and exclude filters per object type.
- Supports DataGrip template override files.

Output shape:

```text
<MSSQL_OUT_DIR>/<DatabaseName>/schemas/<schema>/
	tables/
	views/
	routines/
	sequences/
	synonyms/
```

### Snapshot Environment Variables

Core connection and output:

- MSSQL_OUT_DIR
- MSSQL_SERVER
- MSSQL_DATABASE
- MSSQL_DATABASES
- MSSQL_AUTH_MODE
- MSSQL_USER
- MSSQL_PASSWORD
- MSSQL_DRIVER

Snapshot behavior:

- MSSQL_ENABLE_TABLE_DEFINITIONS
- MSSQL_ENABLE_INDEXES
- MSSQL_ENABLE_FOREIGN_KEYS
- MSSQL_ENABLE_VIEWS
- MSSQL_ENABLE_STORED_PROCEDURES
- MSSQL_ENABLE_FUNCTIONS
- MSSQL_ENABLE_SEQUENCES
- MSSQL_ENABLE_SYNONYMS
- MSSQL_CLEANUP_STAGING
- MSSQL_SYNC_LOG_RETENTION_DAYS

Filters:

- MSSQL_INCLUDE_SCHEMAS
- MSSQL_EXCLUDE_SCHEMAS
- MSSQL_INCLUDE_TABLES
- MSSQL_EXCLUDE_TABLES
- MSSQL_INCLUDE_VIEWS
- MSSQL_EXCLUDE_VIEWS
- MSSQL_INCLUDE_STORED_PROCEDURES
- MSSQL_EXCLUDE_STORED_PROCEDURES
- MSSQL_INCLUDE_FUNCTIONS
- MSSQL_EXCLUDE_FUNCTIONS
- MSSQL_INCLUDE_SEQUENCES
- MSSQL_EXCLUDE_SEQUENCES
- MSSQL_INCLUDE_SYNONYMS
- MSSQL_EXCLUDE_SYNONYMS

Notes:

- If MSSQL_DATABASES is set, it is treated as a comma-separated list.
- If MSSQL_DATABASES is empty, MSSQL_DATABASE is used.
- If MSSQL_DATABASE contains commas, it is also treated as a list for backward compatibility.

## File Sync Utilities

`files_sync_utils` provides:

- Inventory building
- Plan generation
- Copy and update execution
- Extra destination cleanup rules
- Activity CSV logging
- Log retention cleanup

## Git Utilities

`git_utils` wraps common git automation operations with consistent behavior and error handling.

## Development Guides

- development_guides/dev_guide.txt
- development_guides/auth_key_guide.txt

These guides contain release flow notes and auth-related instructions.

## Changelog

See CHANGELOG.md for release notes.
