API Reference
Complete API reference for flowerpower-io library
API Reference
Welcome to the flowerpower-io API reference documentation. This comprehensive guide covers all public classes, functions, and methods available in the library.
Overview
The flowerpower-io library provides a unified interface for data I/O operations across multiple formats and storage systems. It supports:
- File Formats: CSV, JSON, Parquet, Delta Lake
- Databases: SQLite, PostgreSQL, MySQL, Oracle, MSSQL, DuckDB
- Data Processing: Polars, Pandas, PyArrow, DataFusion
- Cloud Storage: AWS S3, Google Cloud Storage, Azure Blob Storage
- Message Queues: MQTT
Usage Patterns
Basic Data Loading
from flowerpower_io.loader import CSVFileReader, ParquetFileReader
# Read CSV file
csv_reader = CSVFileReader("data.csv")
df = csv_reader.to_pandas()
# Read Parquet file
parquet_reader = ParquetFileReader("data.parquet")
df = parquet_reader.to_polars()Basic Data Writing
from flowerpower_io.saver import CSVFileWriter, DeltaTableWriter
# Write CSV file
csv_writer = CSVFileWriter("output.csv")
csv_writer.write(df)
# Write Delta Lake table
delta_writer = DeltaTableWriter("delta_table/")
delta_writer.write(df)Database Operations
from flowerpower_io.loader import PostgreSQLReader
from flowerpower_io.saver import PostgreSQLWriter
# Read from PostgreSQL
pg_reader = PostgreSQLReader(
table_name="users",
host="localhost",
port=5432,
username="user",
password="password",
database="mydb"
)
df = pg_reader.to_pyarrow_table()
# Write to PostgreSQL
pg_writer = PostgreSQLWriter(
table_name="users",
host="localhost",
port=5432,
username="user",
password="password",
database="mydb"
)
pg_writer.write(df)Metadata Extraction
from flowerpower_io.metadata import get_dataframe_metadata, get_quality_metadata
# Get basic metadata
metadata = get_dataframe_metadata(df)
# Get quality metadata
quality = get_quality_metadata(df)Advanced Features
Custom Storage Options
from flowerpower_io.loader import ParquetFileReader
# Read from S3
reader = ParquetFileReader(
path="s3://bucket/data.parquet",
storage_options={
"key": "your-access-key",
"secret": "your-secret-key",
"client_kwargs": {
"region_name": "us-east-1"
}
}
)
df = reader.to_polars(lazy=True)Delta Lake Advanced Features
from flowerpower_io.saver import DeltaTableWriter
# Write with partitioning and schema evolution
writer = DeltaTableWriter("delta_table/")
writer.write(
df,
mode="append",
partition_by=["date"],
schema_mode="merge",
predicate="active = true"
)Database Connection Management
Error Handling
The library provides comprehensive error handling:
from flowerpower_io.loader import CSVFileReader
try:
reader = CSVFileReader("data.csv")
df = reader.to_duckdb()
except FileNotFoundError:
print("File not found")
except Exception as e:
print(f"Error reading file: {e}")Performance Considerations
- Use lazy loading with large datasets where possible
- Consider partitioning for large Delta Lake tables
- Use appropriate compression settings for file formats
- Leverage connection pooling for database operations
Best Practices
- Use appropriate data types for your data to optimize storage and performance
- Implement proper error handling for production applications
- Use connection pooling for database operations
- Consider data partitioning for large datasets
- Use metadata functions for data quality monitoring
- Leverage lazy evaluation for large datasets when possible
Contributing
If you find any issues or have suggestions for improvements, please refer to the main documentation for contribution guidelines.
License
This library is licensed under the MIT License. See the LICENSE file for details.