Metadata-Version: 2.4
Name: apophenia-trend
Version: 0.1.0
Summary: Find absurd, statistically significant correlations between unrelated datasets
Author-email: Oliver Bigbee <oliverbigbee@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/sectersion/apophenia
Project-URL: Repository, https://github.com/sectersion/apophenia
Project-URL: Bug Tracker, https://github.com/sectersion/apophenia/issues
Keywords: correlation,statistics,data-analysis,absurdity,fred,google-trends
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pandas>=2.0
Requires-Dist: numpy>=1.24
Requires-Dist: matplotlib>=3.7
Requires-Dist: seaborn>=0.12
Requires-Dist: scipy>=1.10
Requires-Dist: fastapi>=0.100
Requires-Dist: uvicorn[standard]>=0.22
Requires-Dist: pydantic>=2.0
Requires-Dist: pydantic-settings>=2.0
Requires-Dist: httpx>=0.24
Requires-Dist: python-dotenv>=1.0
Requires-Dist: pytrends>=4.9
Requires-Dist: fredapi>=0.5
Requires-Dist: statsmodels>=0.14
Requires-Dist: openai>=1.0
Requires-Dist: sentence-transformers>=2.2
Requires-Dist: diskcache>=5.6
Requires-Dist: rich>=13.0
Requires-Dist: requests>=2.31
Requires-Dist: beautifulsoup4>=4.12
Requires-Dist: lxml>=4.9
Requires-Dist: yfinance>=0.2.31
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: ruff>=0.1; extra == "dev"
Requires-Dist: mypy>=1.0; extra == "dev"
Dynamic: license-file

# Apophenia

[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
[![Code style: ruff](https://img.shields.io/badge/code%20style-ruff-ff69b4.svg)](https://docs.astral.sh/ruff/)
[![FastAPI](https://img.shields.io/badge/API-FastAPI-009688.svg)](https://fastapi.tiangolo.com/)
[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](https://github.com/sectersion/apophenia/pulls)

> *"The human tendency to perceive meaningful patterns within random data."*

**Apophenia** is a data pipeline that finds absurd, statistically significant correlations between **completely unrelated datasets** — FRED economic indicators, Google Trends searches, OSHA workplace injuries, USPTO patent filings, and ESPN sports viewership. It scores each result on *absurdity* and generates clickbait headlines.

---

## Features

- **Multi-source data ingestion** — FRED, Google Trends, OSHA, USPTO patents, ESPN, Yahoo Finance
- **Rigorous statistics** — Pearson / Spearman correlation, lagged cross-correlation (up to 30 days), rolling correlation, cointegration (Engle-Granger)
- **Absurdity scoring** — Weighted combination of domain distance, statistical strength, and directional inconsistency
- **Clickbait headline generator** — "Economists baffled: SP500 skyrocketed 80% while Bigfoot searches crashed 65%"
- **CLI + REST API** — Use from the terminal or via HTTP
- **Disk-cached** — No redundant API calls
- **LLM support** — Optional OpenAI integration for AI-powered absurdity scoring

---

## Quick Start

```bash
# Install
pip install apophenia-trend

# Or from source
git clone https://github.com/sectersion/apophenia.git
cd apophenia
pip install -e .

# Set up your API keys (optional — most sources work without them)
cp .env.example .env
# Edit .env to add FRED_API_KEY and/or OPENAI_API_KEY
```

### CLI

```bash
# Find the most absurd correlations
apophenia --start 2020-01-01 --end 2024-01-01 --min-absurdity 0.6

# Output as JSON
apophenia --start 2022-01-01 --end 2023-01-01 --json
```

### API

```bash
# Start the server
uvicorn apophenia.api:app --reload

# Or use the CLI shortcut
python -c "from apophenia.api import serve; serve()"
```

```bash
curl "http://localhost:8000/correlate?start=2020-01-01&end=2024-01-01&min_absurdity=0.7"
```

---

## Data Sources

| Source | Category | Requires Key | Description |
|--------|----------|-------------|-------------|
| [FRED](https://fred.stlouisfed.org/) | Economics | Key (free) | GDP, unemployment, fed funds, CPI, yield curve, VIX, housing starts, industrial production... |
| [Google Trends](https://trends.google.com/) | Culture | No | Search volume for 20+ queries ("bitcoin", "quiet quitting", "aliens", "wordle"... ) |
| [OSHA](https://www.osha.gov/) | Health | No | Workplace injury / fatality enforcement data (falls, electrocutions, amputations...) |
| [USPTO Patents](https://developer.uspto.gov/) | Innovation | No | Patent filings for 13 categories (AI, blockchain, gene editing, spacecraft...) |
| [ESPN](https://www.espn.com/) | Sports | No | Scraped viewership / interest across 10 leagues (NFL, NBA, MLB, UFC, golf...) |
| [Yahoo Finance](https://finance.yahoo.com/) | Finance | No | 30+ tickers (SPY, QQQ, GLD, BTC-USD, TLT, sector ETFs, VIX...) |

---

## Architecture

```
src/apophenia/
├── __init__.py          # Cache directory setup
├── config.py            # Pydantic settings (env vars)
├── cli.py               # Rich-formatted terminal UI
├── api.py               # FastAPI REST server
├── core/
│   ├── engine.py        # Orchestration pipeline
│   ├── correlation.py   # Statistical computation
│   ├── absurdity.py     # Absurdity scoring
│   └── headlines.py     # Clickbait headline generator
└── sources/
    ├── base.py          # Abstract source + caching
    ├── fred.py          # Federal Reserve data
    ├── trends.py        # Google Trends
    ├── osha.py          # OSHA enforcement
    ├── patents.py       # USPTO filings
    ├── espn.py          # ESPN sports data
    └── yfinance_source.py  # Yahoo Finance market data
```

### Pipeline

```
Sources ─► Fetch ─► Align ─► Correlate ─► Score ─► Headline ─► Results
  │          │          │           │          │           │
  │    diskcache    dropna     Pearson    domain     random
  │    + rate      + union    Spearman   distance   templates
  │    limits       index      lag(1..30) stat.strength
  │                            rolling   direction
  │                            coint
```

---

## Absurdity Score

Each correlation gets a score `[0, 1]` based on three components:

| Component | Weight | Description |
|-----------|--------|-------------|
| **Category Distance** | 30% | How unrelated the domains are (e.g., sports × health = 0.5, sports × economics = 0.8) |
| **Statistical Strength** | 40% | Magnitude of `r`, significance of `p`, with bonus for extreme values |
| **Direction Absurdity** | 30% | Whether contemporaneous and lagged correlations point in opposite directions |

The higher the score, the more absurd (and interesting) the result.

---

## Example Output

```
┌─────────────────────────────────────────────────────────────────────┐
│               Apophenia Report — Top 5 Correlations                │
├───┬──────────┬────────────────────────────────────────┬──────┬─────┤
│ # │ Absurdity │ Headline                              │ r    │ p   │
├───┼──────────┼────────────────────────────────────────┼──────┼─────┤
│ 1 │     0.89 │ SP500 skyrocketed 83% — Meanwhile,     │ 0.92 │ 2e-4│
│   │          │ Bigfoot searches plummeted 72%         │      │     │
│ 2 │     0.84 │ Can Electrocutions predict NFT hype?   │ 0.87 │ 5e-3│
│   │          │ Crashed 61% foreshadows Surging 78%    │      │     │
│ 3 │     0.81 │ Coincidence? Federal Funds Rate soared │ 0.85 │ 1e-3│
│   │          │ and Divorce Lawyer searches tanked     │      │     │
│ 4 │     0.78 │ As 3D Printing jumped 67%, NBA         │ 0.81 │ 8e-3│
│   │          │ viewership collapsed 54%               │      │     │
│ 5 │     0.72 │ Economists baffled: Unemployment       │-0.79 │ 3e-2│
│   │          │ nosedived 62% correlates with Keto     │      │     │
│   │          │ Diet searches exploding 91%            │      │     │
└───┴──────────┴────────────────────────────────────────┴──────┴─────┘
```

---

## Development

```bash
# Install dev dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Lint
ruff check src/

# Type check
mypy src/
```

---

## Configuration

All settings via `.env` or environment variables:

| Variable | Required | Description |
|----------|----------|-------------|
| `FRED_API_KEY` | Optional | FRED API key ([free signup](https://fred.stlouisfed.org/docs/api/api_key.html)) |
| `OPENAI_API_KEY` | Optional | OpenAI key for LLM-based absurdity scoring |

---

## Why "Apophenia"?

> *Apophenia* is the tendency to perceive meaningful connections between unrelated things. The term was coined by neurologist Klaus Conrad in 1958. This project leans *hard* into that tendency — with statistics.

---

## License

MIT
