Metadata-Version: 2.4
Name: DataOpsHub
Version: 6.6.12
Summary: A Python toolkit for streamlined data operations and database management
Author-email: Jon Ander Gabantxo <jagabantxo@gmail.com>
License: MIT License
        
        Copyright (c) 2024 DataOpsHub
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE. 
Project-URL: Homepage, https://github.com/EusDancerDev/DataOpsHub
Project-URL: Documentation, https://github.com/EusDancerDev/DataOpsHub#readme
Project-URL: Repository, https://github.com/EusDancerDev/DataOpsHub.git
Project-URL: Bug Reports, https://github.com/EusDancerDev/DataOpsHub/issues
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: sqlalchemy>=2.0.0
Requires-Dist: numpy>=2.2.3
Requires-Dist: pandas>=2.2.3
Requires-Dist: filewise>=3.14.0
Requires-Dist: pygenutils>=17.1.0
Requires-Dist: paramlib>=3.5.0
Requires-Dist: email_validator>=2.0.0
Requires-Dist: selenium>=4.0.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: black>=22.0.0; extra == "dev"
Requires-Dist: flake8>=5.0.0; extra == "dev"
Dynamic: license-file

# DataOpsHub

[![Python Version](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/downloads/)
[![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
[![PyPI Version](https://img.shields.io/pypi/v/dataopshub.svg)](https://pypi.org/project/dataopshub/)

**DataOpsHub** is a comprehensive Python toolkit designed specifically for healthcare data operations and management. It provides a robust set of tools for managing medical records, user data, form validation, database operations, and web automation in healthcare environments. The package emphasises data integrity, security, and compliance with healthcare standards including HL7 v2 compatibility.

## Features

- **Healthcare Data Models**:
  - Comprehensive user management with medical data relationships
  - Vital signs tracking and monitoring
  - Laboratory results management
  - Medication history and prescription tracking
  - HL7 v2 compatibility for healthcare data interchange

- **Database Operations**:
  - Advanced SQLAlchemy-based database handling
  - Multi-database support (MySQL, PostgreSQL, SQLite)
  - Automated file-to-database loading (CSV, Excel, ODS)
  - Comprehensive CRUD operations with error handling
  - Data filtering and querying with temporal range support

- **Form Validation and Data Entry**:
  - WTForms-based form validation for healthcare applications
  - Comprehensive field validation (email, medical IDs, dates, demographics)
  - Spanish DNI validation support
  - JSON and form data validation with error reporting
  - Medical data entry forms with specialised validations

- **Security and Configuration**:
  - Automated secure password generation
  - Comprehensive security configuration management
  - Custom exception hierarchy for precise error handling
  - Environment-based configuration with security best practices

- **Web Automation**:
  - Selenium WebDriver setup and testing utilities
  - Headless browser configuration for automated testing
  - Support for Firefox and Chrome browsers in CI/CD environments

## Installation

### Prerequisites

Before installing, please ensure the following dependencies are available on your system:

- **External Tools** (required for full functionality):
  - Database server (MySQL, PostgreSQL, or SQLite)
  - Web browser (Firefox/Chrome for web automation features)

- **Required Third-Party Libraries**:

  ```bash
  pip install sqlalchemy pandas numpy scipy wtforms email-validator selenium
  ```

  Or via Anaconda (recommended channel: `conda-forge`):

  ```bash
  conda install -c conda-forge sqlalchemy pandas numpy scipy
  pip install wtforms email-validator selenium
  ```

- **Database Drivers** (choose based on your database):

  ```bash
  # For MySQL
  pip install pymysql

  # For PostgreSQL
  pip install psycopg2-binary

  # SQLite is included with Python
  ```

- **Internal Package Dependencies**:

  ```bash
  pip install filewise paramlib
  pip install pygenutils                    # Core functionality
  pip install pygenutils[arrow]             # With arrow support (optional)
  ```

### For regular users (from PyPI)

```bash
pip install dataopshub
```

### For contributors/developers (with latest Git versions)

```bash
# Install with development dependencies (includes latest Git versions)
pip install -e .[dev]

# Alternative: Use requirements-dev.txt for explicit Git dependencies
pip install -r requirements-dev.txt
pip install -e .
```

**Benefits of the new approach:**

- **Regular users**: Simple `pip install dataopshub` with all dependencies included
- **Developers**: Access to latest Git versions for development and testing
- **PyPI compatibility**: All packages can be published without Git dependency issues

### Troubleshooting

**If you encounter import errors:**

1. **For PyPI users**: The package should install all dependencies automatically. If you get import errors, try:

   ```bash
   pip install --upgrade dataopshub
   ```

2. **For developers**: Make sure you've installed the development dependencies:

   ```bash
   pip install -e .[dev]
   ```

3. **Common issues**:
   - **Missing dependencies**: For regular users, all dependencies are included. For developers, use `pip install -e .[dev]`
   - **Python version**: Ensure you're using Python 3.10 or higher
   - **Database drivers**: Install the appropriate database driver for your database system

### Verify Installation

To verify that your installation is working correctly:

```python
try:
    import DataOpsHub
    from filewise.general.introspection_utils import get_type_str
    from pygenutils.arrays_and_lists.data_manipulation import flatten_list
    from paramlib.config_params import DB_ERROR_CODE_DICT
    
    print("✅ All imports successful!")
    print(f"✅ DataOpsHub version: {DataOpsHub.__version__}")
    print("✅ Installation is working correctly.")
    
except ImportError as e:
    print(f"❌ Import error: {e}")
    print("💡 For regular users: pip install dataopshub")
    print("💡 For developers: pip install -e .[dev]")
```

## Usage

### Basic Example - User Management

```python
from DataOpsHub.models.user import User
from DataOpsHub.services.user_service import UserService
from DataOpsHub.databases.database_handler import init_db
from DataOpsHub.security.config import DATABASE_CREDENTIALS

# Initialize database
engine, Session = init_db(DATABASE_CREDENTIALS)
session = Session()

# Create user service
user_service = UserService(session)

# Create a new user
user_data = {
    'user_id': 'U001',
    'first_name': 'John',
    'last_name': 'Doe',
    'email': 'john.doe@example.com',
    'date_of_birth': '1990-01-01',
    'gender': 'Male'
}

result = user_service.create_user(user_data)
print(result['message'])  # "User created successfully"
```

### Advanced Example - Medical Data Management

```python
from DataOpsHub.models.user import VitalSigns, LabResults
from datetime import datetime

# Add vital signs for a user
vital_signs = VitalSigns(
    user_id='U001',
    heart_rate=72,
    blood_pressure='120/80',
    temperature=98.6,
    oxygen_saturation=98
)

# Add lab results
lab_result = LabResults(
    user_id='U001',
    test_name='Complete Blood Count',
    result='Normal',
    unit='cells/mcL',
    reference_range='4,500-11,000'
)

session.add(vital_signs)
session.add(lab_result)
session.commit()

# Get user with all medical data
user_data = user_service.get_user_with_related_data('U001')
print(f"User: {user_data['user'].first_name}")
print(f"Vital signs: {len(user_data['vital_signs'])}")
print(f"Lab results: {len(user_data['lab_results'])}")
```

### Form Validation Example

```python
from DataOpsHub.data_entry_forms.forms.forms import RegistrationForm
from DataOpsHub.data_entry_forms.validators.form_field_validations import validate_field

# Validate a registration form
form_data = {
    'email': 'user@example.com',
    'first_name': 'John',
    'last_name': 'Doe'
}

form = RegistrationForm(data=form_data)
if form.validate():
    print("Form validation successful!")
else:
    print("Validation errors:", form.errors)

# Validate individual fields
email_result = validate_field('email', 'test@example.com', {}, [])
print(f"Email valid: {not email_result['has_error']}")
```

### Database File Upload Example

```python
from DataOpsHub.databases.database_handler import upload_file_data

# Upload CSV data to database
upload_file_data(
    input_file_list='medical_data.csv',
    config=DATABASE_CREDENTIALS,
    database_type='mysql',
    if_exists='append',
    dtype_dict={'patient_id': 'str', 'age': 'int64', 'temperature': 'float64'}
)
```

### Security Example

```python
from DataOpsHub.security.automated_passwords import generate_automated_password
from DataOpsHub.security.exceptions import ValidationError

# Generate secure password
password = generate_automated_password(33, 127, 16)
print(password)

# Handle custom exceptions
try:
    # Some operation that might fail
    pass
except ValidationError as e:
    print(f"Validation failed: {e.message}")
```

## Project Structure

The package is organised into specialised sub-packages for healthcare operations:

```text
DataOpsHub/
├── constants/
│   ├── error_messages.py         # Standardised error messages
│   └── table_mappings.py         # Database table field mappings
├── data_entry_forms/
│   ├── forms/
│   │   └── forms.py              # WTForms-based form classes
│   └── validators/
│       └── form_field_validations.py  # Comprehensive validation functions
├── databases/
│   ├── database_handler.py       # Core database operations and utilities
│   └── upload_data.py           # File-to-database upload utilities
├── models/
│   ├── base.py                  # Base model with HL7 compatibility
│   └── user.py                  # User and medical data models
├── security/
│   ├── automated_passwords.py   # Secure password generation
│   ├── config.py               # Security configuration management
│   └── exceptions.py           # Custom exception hierarchy
├── services/
│   └── user_service.py         # User management service layer
└── web_automation/
    └── webdriver_Firefox-Chrome_setup_test.py  # WebDriver testing utilities
```

## Key Functions

### Database Operations

- `init_db()` - Initialize database with healthcare models
- `upload_file_data()` - Load CSV/Excel/ODS files into database tables
- `filter_data()` - Advanced data filtering with temporal ranges
- `create_user()`, `update_user()`, `delete_user()` - User CRUD operations

### Medical Data Models

- `User` - Comprehensive user model with medical relationships
- `VitalSigns` - Vital signs tracking and monitoring
- `LabResults` - Laboratory test results management
- `MedicationsHistory` - Medication and prescription tracking

### Form Validation

- `RegistrationForm` - User registration with validation
- `validate_field()` - Individual field validation
- `validate_date_range()` - Medical date range validation
- `verify_spanish_dni()` - Spanish ID validation

### Security Functions

- `generate_automated_password()` - Secure password generation
- Custom exception classes for precise error handling
- Environment-based security configuration

### Web Automation

- `test_firefox_webdriver()` - WebDriver setup testing
- `configure_firefox_options()` - Headless browser configuration

## Healthcare Standards Compliance

DataOpsHub is designed with healthcare standards in mind:

- **HL7 v2 Compatibility**: All models include HL7 v2 fields for healthcare data interchange
- **Medical Data Integrity**: Comprehensive validation for medical identifiers and data
- **Audit Trails**: Automatic timestamp tracking for all medical records
- **Security**: Healthcare-grade security configuration and password policies
- **Data Relationships**: Proper medical data relationships between users and their health records

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.

### Development Guidelines

- Follow healthcare data security best practices
- Ensure all medical data models maintain HL7 compatibility
- Add comprehensive validation for any new form fields
- Include proper error handling and custom exceptions
- Write tests for database operations and data validation

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## Acknowledgments

- SQLAlchemy team for robust database operations
- WTForms community for form validation framework
- Healthcare standards organisations for HL7 specifications
- Open-source contributors to the healthcare data ecosystem

## Contact

For any questions or suggestions, please open an issue on GitHub or contact the maintainers.

## Version

Current version: 6.6.12

For detailed changelog, see [CHANGELOG.md](CHANGELOG.md).

## Security Notice

This package handles sensitive healthcare data. Please ensure:

- Use secure database connections in production
- Implement proper access controls and authentication
- Follow HIPAA compliance guidelines where applicable
- Regularly update security configurations and dependencies
- Use environment variables for sensitive configuration data
