Metadata-Version: 2.4
Name: analysis3054
Version: 0.4.9
Summary: Advanced time-series analytics and forecasting toolkit for commodity and power trading
Author-email: Hoff <you@example.com>
License: Apache-2.0
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pandas>=2.3.3
Requires-Dist: numpy<2.4.0,>=1.25.0
Requires-Dist: pycountry>=24.6.1
Requires-Dist: plotly>=6.5.0
Requires-Dist: statsmodels>=0.14.6
Requires-Dist: scipy>=1.16.3
Requires-Dist: holidays>=0.87
Requires-Dist: chronos-forecasting==2.2.2
Requires-Dist: autogluon.timeseries==1.5.0
Requires-Dist: joblib>=1.3.0
Requires-Dist: requests>=2.32.5
Requires-Dist: httpx>=0.28.1
Requires-Dist: beautifulsoup4>=4.12.3
Requires-Dist: Pillow>=10.0.0
Requires-Dist: openpyxl>=3.1.2
Requires-Dist: pandas_datareader>=0.10.0
Requires-Dist: pdfplumber>=0.11.8
Requires-Dist: pypdf>=3.17.0
Requires-Dist: h2>=4.1.0
Requires-Dist: selenium>=4.27.0
Requires-Dist: stealthenium>=1.1.1
Requires-Dist: websocket-client>=1.8.0
Provides-Extra: stats
Requires-Dist: pmdarima>=2.1.1; extra == "stats"
Requires-Dist: arch>=8.0.0; extra == "stats"
Provides-Extra: ml
Requires-Dist: scikit-learn>=1.8.0; extra == "ml"
Requires-Dist: xgboost>=3.1.2; extra == "ml"
Requires-Dist: lightgbm>=4.6.0; extra == "ml"
Requires-Dist: catboost>=1.2.8; extra == "ml"
Provides-Extra: dl
Requires-Dist: tensorflow>=2.20.0; extra == "dl"
Requires-Dist: transformers>=4.50.0; extra == "dl"
Provides-Extra: prophet
Requires-Dist: prophet>=1.2.1; extra == "prophet"
Requires-Dist: neuralprophet>=0.8.0; extra == "prophet"
Provides-Extra: physics
Requires-Dist: torch>=2.9.1; extra == "physics"
Requires-Dist: torchdiffeq>=0.2.5; extra == "physics"
Requires-Dist: PyWavelets>=1.9.0; extra == "physics"
Provides-Extra: tbats
Requires-Dist: tbats>=1.1.3; extra == "tbats"
Provides-Extra: autogluon
Requires-Dist: autogluon==1.5.0; extra == "autogluon"
Provides-Extra: plot
Requires-Dist: plotly>=6.5.0; extra == "plot"
Provides-Extra: playwright
Requires-Dist: playwright>=1.57.0; extra == "playwright"
Provides-Extra: seleniumbase
Requires-Dist: seleniumbase>=4.45.0; extra == "seleniumbase"
Provides-Extra: gemini
Requires-Dist: google-genai>=1.56.0; extra == "gemini"
Provides-Extra: windows
Requires-Dist: pywin32>=306; extra == "windows"
Provides-Extra: all
Requires-Dist: analysis3054[autogluon,dl,gemini,ml,physics,playwright,plot,prophet,seleniumbase,stats,tbats,windows]; extra == "all"
Dynamic: license-file

# Analysis3054

Analysis3054 is a data pipeline + dashboard toolkit for energy, commodities, and demand planning. The analysis code lives in this repo; dashboards + update scripts live in a separate dashboard workspace.

## Paths & environment

```bash
export ANALYSIS3054_CODEX_HOME=~/Analysis3054-Codex
export ANALYSIS3054_DASHBOARD_HOME=~/analysis3054-dashboards
```

## Workspace layout

```
$ANALYSIS3054_DASHBOARD_HOME/
  scripts/
  dashboard_repos/<dashboard>/
    data/
      build_dashboard.py
      update_data.py
      requirements.txt
      *.csv (inputs)
      *.html (outputs)
```

## Automation scripts

Automated update scripts live in `$ANALYSIS3054_DASHBOARD_HOME/scripts`. Finder-friendly wrappers live alongside them in `$ANALYSIS3054_DASHBOARD_HOME/` (`run_update_*.command`).

## Install (repo)

```bash
python -m venv .venv
source .venv/bin/activate
pip install -e .
```

Per-dashboard dependencies live in each `requirements.txt`:

```bash
pip install -r $ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/kayross/data/requirements.txt
```

All dashboards load Plotly JS from the Plotly 3.3.1 CDN (export HTML inlines Plotly for offline use).

## Dashboards

### 1) Vehicle Miles & Fuel Usage (Kayross + DTN)

Folder: `$ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/kayross/data/`

Inputs (4 CSVs):
- `kayross_us.csv` columns: `VALUE_DATE`, `VALUE_DIESEL`, `VALUE_GASOLINE`, `REGION`
- `kayross_eu.csv` columns: `VALUE_DATE`, `COUNTRY`, `VALUE_DIESEL_CONSUMPTION`, `VALUE_GASOLINE_CONSUMPTION`
- `dtn_refined_fuels.csv` (PADD) columns: `effectiveDateTime`, `region`, `grade`, `sumNetVolume`, `coverageFactor`
- `dtn_refined_fuels_rack.csv` (rack/state) columns: `effectiveDateTime`, `rackId`, `rackName`, `state`, `region`, `grade`, `sumNetVolume`

Build:
```bash
python $ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/kayross/data/build_dashboard.py \
  --na-csv $ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/kayross/data/kayross_us.csv \
  --eu-csv $ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/kayross/data/kayross_eu.csv \
  --padd-csv $ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/kayross/data/dtn_refined_fuels.csv \
  --rack-csv $ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/kayross/data/dtn_refined_fuels_rack.csv
```

Outputs:
- `$ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/kayross/data/kayross_dashboard.html`
- `$ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/kayross/data/kayross_dashboard_export.html` (inline Plotly)

Chronos2 forecast:
```bash
KAYROSS_ENABLE_CHRONOS2=1 python $ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/kayross/data/build_dashboard.py
```

DTN updater (optional):
```bash
python $ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/kayross/data/update_data.py --update-dtn
```
DTN auth (any one):
- `DTN_API_KEY` (preferred)
- `DTN_ACCESS_TOKEN`
- or OAuth: `DTN_CLIENT_ID` + `DTN_CLIENT_SECRET` (+ optional `DTN_AUDIENCE`)

Pandas inputs:
- `build_dashboard.py` accepts DataFrames via `build_payload(...)`. If you want in-memory DataFrames, copy that script and call `build_payload`/`build_html` directly.

Notes:
- DTN diesel grades: `#1 Diesel`, `#2 Diesel` (plus common variants).
- DTN gasoline grades: `Premium`, `Regular`.
- PADD volumes apply coverage factor then divide by 42000 (KBD). Rack/state data skips coverage factor and still divides by 42000.

### 2) CFTC Disaggregated COT Dashboard

Folder: `$ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/cftc_disagg/data/`

Build (downloads data if cache missing):
```bash
python $ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/cftc_disagg/data/build_dashboard.py
```

Outputs:
- `$ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/cftc_disagg/data/cftc_disagg_dashboard.html`
- `$ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/cftc_disagg/data/cftc_disagg_dashboard_export.html`
- `$ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/cftc_disagg/data/cftc_disagg_dashboard_data.json`
- `$ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/cftc_disagg/data/cftc_disagg_filtered.csv`

### 3) Port TEU Dashboard

Folder: `$ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/port_teu/data/`

Inputs:
- `port_of_la_container_stats_2021_present.csv`
- `port_long_beach_teu.csv`
- `port_ny_teu.csv`
- `port_houston_teu.csv`
- `port_savannah_teu.csv`

Build:
```bash
python $ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/port_teu/data/build_dashboard.py --data-dir $ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/port_teu/data
```

Update (Gemini disabled; Houston/Savannah are manual):
```bash
python $ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/port_teu/data/update_data.py \
  --houston-csv /path/to/port_houston.csv \
  --savannah-csv /path/to/port_savannah.csv
```

### 4) Rail Traffic Dashboard

Folder: `$ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/rail_traffic/data/`

Inputs (manual):
- `rail_traffic_north_american.csv`
- `rail_traffic_us.csv`
- `rail_traffic_canada.csv`
- `rail_traffic_mexico.csv`

Build:
```bash
python $ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/rail_traffic/data/build_dashboard.py --data-dir $ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/rail_traffic/data
```

Update:
```bash
python $ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/rail_traffic/data/update_data.py \
  --north-american-csv /path/to/north_american.csv \
  --us-csv /path/to/us.csv \
  --canada-csv /path/to/canada.csv \
  --mexico-csv /path/to/mexico.csv
```

### 5) Macro Drivers Dashboard

Folder: `$ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/macro/data/`

Inputs:
- FRED snapshots: `fred_diesel_monthly.csv`, `fred_diesel_quarterly.csv`
- Macro focus: `fred_macro_focus_*`
- EIA snapshots: `eia_distillate_product_supplied.csv`, `eia_steo_monthly.csv`, `eia_steo_quarterly.csv`
- Forecast snapshots: `distillate_forecast*.csv`

Update (FRED + optional EIA refresh):
```bash
FRED_API_KEY=... python $ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/macro/data/update_data.py
```
Skip EIA refresh:
```bash
SKIP_EIA_FETCH=true python $ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/macro/data/update_data.py
```

Build:
```bash
python $ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/macro/data/build_dashboard.py
```

Forecast model override:
```bash
DISTILLATE_FORECAST_MODEL=chronos2 python $ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/macro/data/build_dashboard.py
```

### 6) Europe Energy Dashboard

Folder: `$ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/europe_energy/data/`

Update (Spain + UK sources):
```bash
python $ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/europe_energy/data/update_data.py
```

Build:
```bash
python $ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/europe_energy/data/build_dashboard.py
```

### 7) DFO Generator Dashboard

Folder: `$ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/dfo_generators/data/`

Inputs: EIA-860/923 raw tabs (place into this folder). The update helper rebuilds:
- `dfo_generators_inventory.csv`
- `dfo_generators_costs.csv`

Update + Build:
```bash
python $ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/dfo_generators/data/update_data.py
```

### 8) EIA Electricity Runner (EIA-860/923)

Folder: `$ANALYSIS3054_CODEX_HOME/analysis3054/data/eia_electricity/`

This runner refreshes the raw EIA-860/923 CSVs and can rebuild the DFO dashboard.

Run:
```bash
python $ANALYSIS3054_CODEX_HOME/analysis3054/data/eia_electricity/update_data.py
```

Options:
```bash
python $ANALYSIS3054_CODEX_HOME/analysis3054/data/eia_electricity/update_data.py --output-dir $ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/dfo_generators/data
python $ANALYSIS3054_CODEX_HOME/analysis3054/data/eia_electricity/update_data.py --year-860 2024 --year-923 2024
python $ANALYSIS3054_CODEX_HOME/analysis3054/data/eia_electricity/update_data.py --skip-dfo
```

### 9) Refinery Product Mix Dashboard

Folder: `$ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/refinery/data/`

Inputs:
- `la_refinery_latest.csv` (optional updater)
- `tx_refineries_2021_present.csv`
- `refinery_data.csv`
- `refinery_forecast.csv`
- `refinery_product_forecast.csv`
- `tx_refinery_name_map.csv`

Update:
```bash
python $ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/refinery/data/update_data.py --refresh-la
```

Build:
```bash
python $ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/refinery/data/build_dashboard.py
```

### 10) TX Refinery Receipts & Deliveries

Folder: `$ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/tx_refineries_transfers/data/`

Inputs:
- `tx_refineries_received_2021_present.csv`
- `tx_refineries_delivered_2021_present.csv`
- `tx_refineries_2021_present.csv`
- `tx_refinery_name_map.csv`

Build:
```bash
python $ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/tx_refineries_transfers/data/build_dashboard.py
```

Update:
```bash
python $ANALYSIS3054_DASHBOARD_HOME/dashboard_repos/tx_refineries_transfers/data/update_data.py \
  --received-csv /path/to/tx_received.csv \
  --delivered-csv /path/to/tx_delivered.csv
```

## Notes

- Each dashboard folder is designed to be copied as-is.
- Update scripts avoid Gemini-based parsing; provide manual CSVs where noted.
- If you want offline HTML, use the `*_export.html` outputs where available.
- Legacy `analysis3054/data/*` folders in the code repo are no longer used by automation; remove them only after confirming no one relies on module default paths or the EIA electricity runner.
