Savers

Savers in flowerpower-io are responsible for writing data from a standardized data structure (typically a Polars DataFrame, but also supporting Pandas DataFrames and PyArrow Tables) to various destinations and formats. All saver classes inherit from BaseFileWriter or BaseDatabaseWriter, providing a consistent API for data output.

Available Saver Classes

Here’s a list of the currently supported saver classes:

  • CSVFileWriter and CSVDatasetWriter: For writing data to CSV files and CSV datasets (e.g., partitioned CSVs).
  • DeltaTableWriter: For writing data to Delta Lake tables.
  • DuckDBWriter: For writing data to DuckDB databases.
  • JsonFileWriter and JsonDatasetWriter: For writing data to JSON files and JSON datasets.
  • MSSQLWriter: For writing data to Microsoft SQL Server databases.
  • MySQLWriter: For writing data to MySQL databases.
  • OracleDBWriter: For writing data to Oracle databases.
  • ParquetFileWriter and ParquetDatasetWriter: For writing data to Parquet files and Parquet datasets.
  • PostgreSQLWriter: For writing data to PostgreSQL databases.
  • PydalaDatasetWriter: For writing data using Pydala’s dataset capabilities.
  • SQLiteWriter: For writing data to SQLite databases.
  • MQTTSaver: For publishing data to MQTT topics.

Examples

CSV Saver (CSVFileWriter)

The CSVFileWriter allows you to easily save data to a CSV file.

Show the code 
import pandas as pd
import os
from flowerpower_io.saver import CSVFileWriter

# Create a dummy DataFrame
df_to_save = pd.DataFrame({'id': [1, 2, 3], 'name': ['Alice', 'Bob', 'Charlie']})
output_csv_path = "output_data.csv"

# Save data to CSV
writer = CSVFileWriter(path=output_csv_path)
writer.write(df_to_save)

print(f"Data saved to {output_csv_path}")
print(f"File exists: {os.path.exists(output_csv_path)}")

# Clean up
os.remove(output_csv_path)

JSON Saver (JsonFileWriter)

The JsonFileWriter is used to save data to JSON files.

Show the code 
import pandas as pd
import os
from flowerpower_io.saver import JsonFileWriter

# Create a dummy DataFrame
df_products = pd.DataFrame([
  {"product": "Laptop", "price": 1200},
  {"product": "Mouse", "price": 25},
  {"product": "Keyboard", "price": 75}
])
output_json_path = "output_products.json"

# Save data to JSON
writer = JsonFileWriter(path=output_json_path)
writer.write(df_products)

print(f"Data saved to {output_json_path}")
print(f"File exists: {os.path.exists(output_json_path)}")

# Clean up
os.remove(output_json_path)

SQLite Saver (SQLiteWriter)

The SQLiteWriter enables writing data to SQLite databases.

Show the code 
import pandas as pd
import os
import sqlite3
from flowerpower_io.saver import SQLiteWriter

# Create a dummy DataFrame
df_orders = pd.DataFrame({'order_id': [1, 2, 3], 'item': ['Book', 'Pen', 'Book'], 'quantity': [2, 5, 1]})
db_path = "output_sales.db"

# Initialize SQLiteWriter
writer = SQLiteWriter(path=db_path, table_name="orders", mode="replace")

# Save data to SQLite
writer.write(df_orders)

print(f"Data saved to {db_path} in table 'orders'.")
print(f"Database file exists: {os.path.exists(db_path)}")

# Verify data by reading it back (optional)
conn = sqlite3.connect(db_path)
df_verify = pd.read_sql("SELECT * FROM orders", conn)
print("\nData verified from SQLite:")
print(df_verify)
conn.close()

# Clean up
os.remove(db_path)