Architecture
flowerpower-io is designed with a modular and extensible architecture, making it easy to integrate with various data sources and formats. The core design principles revolve around separation of concerns, flexibility, and ease of use.
Core Design Principles
- Modularity: The library is divided into distinct modules for loaders, savers, and common utilities, allowing for independent development and maintenance.
- Extensibility: New data formats or sources can be easily added by implementing specific loader or saver classes.
- Abstraction: Users interact with high-level interfaces, abstracting away the complexities of underlying data handling mechanisms.
- Performance: Where possible, operations leverage efficient libraries (e.g., Polars, PyArrow) for optimized data processing.
Loaders
Loaders are responsible for reading data from various sources and converting it into a standardized format (typically a Polars DataFrame). Each loader is designed to handle a specific data source or format.
Supported Loader Types
- CSV
- DeltaTable
- DuckDB
- JSON
- MQTT
- MSSQL
- MySQL
- Oracle
- Parquet
- PostgreSQL
- Pydala
- SQLite
Savers
Savers are responsible for writing data from the standardized format (Polars DataFrame) to various destinations or formats. Each saver is designed to handle a specific data destination or format.
Supported Saver Types
- CSV
- DeltaTable
- DuckDB
- JSON
- MQTT
- MSSQL
- MySQL
- Oracle
- Parquet
- PostgreSQL
- Pydala
- SQLite