Metadata-Version: 2.4
Name: salcon
Version: 0.1.1
Summary: S-ALCoN: Soft-Gated Adaptive Layer Condensation Networks - Learning Dynamic Compression Ratios for Hierarchical Feature Selection
Author-email: Ananda Jana <ajana2006@gmail.com>
License: MIT
Project-URL: Homepage, https://github.com/anandajana/salcon
Project-URL: Documentation, https://github.com/anandajana/salcon#readme
Project-URL: Repository, https://github.com/anandajana/salcon
Project-URL: Bug Tracker, https://github.com/anandajana/salcon/issues
Keywords: salcon,s-alcon,alcnet,deep-learning,neural-networks,compression,feature-selection,adaptive-architecture,pytorch,machine-learning
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=1.9.0
Requires-Dist: numpy>=1.19.0
Requires-Dist: tqdm>=4.50.0
Requires-Dist: matplotlib>=3.3.0
Requires-Dist: scikit-learn>=0.24.0
Provides-Extra: dev
Requires-Dist: pytest>=6.0; extra == "dev"
Requires-Dist: pytest-cov>=2.12; extra == "dev"
Requires-Dist: black>=22.0; extra == "dev"
Requires-Dist: flake8>=3.9; extra == "dev"
Requires-Dist: mypy>=0.910; extra == "dev"
Provides-Extra: viz
Requires-Dist: matplotlib>=3.3.0; extra == "viz"
Provides-Extra: examples
Requires-Dist: torchvision>=0.10.0; extra == "examples"
Dynamic: license-file
Dynamic: requires-python

# S-ALCoN: Soft-Gated Adaptive Layer Condensation Networks

[![Python 3.8+](https://img.shields.io/badge/python-3.8%2B-blue.svg)](https://www.python.org/downloads/)
[![PyTorch](https://img.shields.io/badge/PyTorch-2.0%2B-ee4c2c.svg)](https://pytorch.org/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

> **Learning Dynamic Compression Ratios for Hierarchical Feature Selection**

**S-ALCoN** (Soft-Gated Adaptive Layer Condensation Networks) introduces an end-to-end differentiable mechanism that learns optimal layer-wise compression ratios ($\rho^{(i)}$) as trainable network parameters directly during standard backpropagation.

Rather than relying on rigid, human-designed layer dimensions or computationally expensive Neural Architecture Search (NAS), S-ALCoN dynamically filters uninformative activations layer-by-layer based on task complexity.

---

## Why S-ALCoN?

Standard deep networks use fixed hidden dimensions (e.g., `[784, 256, 128, 64, 10]`) set before training begins. This traditional workflow presents several challenges:

* **Manual Over-Engineering:** Fixed architectures often retain redundant hidden neurons on simple tasks or bottleneck feature flow on complex ones.
* **Costly Search Pipelines:** Post-hoc structural pruning or Neural Architecture Search (NAS) require multi-stage training, discrete search spaces, or heavy GPU resources.
* **Lack of Dynamic Adaptability:** Fixed layers cannot automatically prune non-informative representations on a sample- or dataset-adaptive basis during training.

### How S-ALCoN Solves This

S-ALCoN inserts a **Soft Median-Centered Gating Mechanism** into each hidden layer:

1. **Dynamic Selection:** Each neuron's activation is evaluated relative to the median activation of its layer.
2. **Trainable Retention Ratios ($\rho$):** Target compression ratios are reparameterized into unconstrained logit space ($\theta_\rho$), enabling gradient-based updates to target retention levels.
3. **Task Complexity Discovery:** Simple datasets automatically learn aggressive early compression, while complex representations dynamically retain higher feature density.

---

## Mathematical Formulation

### 1. Forward Pass & Soft Median Gating

For a hidden layer $i$ receiving input $x$, the un-gated pre-activations $z$ and post-ReLU activations $h$ are computed as:

$$z = Wx + b$$

$$h = \text{ReLU}(z)$$

To compute individual feature survival probabilities without hard non-differentiable thresholds, S-ALCoN calculates a median-referenced selection score $s_j \in (0,1)$ for neuron $j$:

$$s_j = \sigma\left(\alpha \cdot \left(h_j - \text{median}(h)\right)\right)$$

where:
* $\text{median}(h)$ is the median activation value across neurons in the layer.
* $\alpha$ is the sharpness scaling factor controlling the steepness of the gate (default: $\alpha = 10.0$).
* $\sigma(\cdot)$ is the standard logistic sigmoid function.

The gated activation $a_j$ passed to the next layer is obtained via element-wise soft filtering:

$$a_j = h_j \cdot s_j$$

### 2. Parameterization of Compression Ratios ($\rho$)

The target retention ratio $\rho^{(i)} \in (0,1)$ represents the fraction of active features to retain in layer $i$.

To allow standard unconstrained gradient updates, $\rho^{(i)}$ is parameterized through its inverse-sigmoid logit $\theta_\rho^{(i)}$:

$$\rho^{(i)} = \sigma\left(\theta_\rho^{(i)}\right)$$

### 3. Loss Functions

S-ALCoN optimizes a joint multi-objective loss consisting of three components:

1. **Task Loss ($L_{\text{task}}$):** Standard Cross-Entropy for classification or Mean Squared Error (MSE) for regression.
2. **Mean Sparsity Regularizer ($L_{\text{sparse}}$):** Encourages overall network efficiency by minimizing average selection scores across all $N$ filterable layers:

$$L_{\text{sparse}} = \frac{1}{N} \sum_{i=1}^{N} \left( \frac{1}{d_i} \sum_{j=1}^{d_i} s_j^{(i)} \right)$$

3. **Ratio Matching Penalty ($L_{\text{ratio}}$):** Encourages layer selection scores to match the learned target compression ratios:

$$L_{\text{ratio}} = \frac{1}{N} \sum_{i=1}^{N} \left( \frac{1}{d_i} \sum_{j=1}^{d_i} s_j^{(i)} - \rho^{(i)} \right)^2$$

### Total Loss

$$L_{\text{total}} = L_{\text{task}} + \lambda_{\text{sparse}} L_{\text{sparse}} + \lambda_{\text{ratio}} L_{\text{ratio}}$$

where $\lambda_{\text{sparse}}$ and $\lambda_{\text{ratio}}$ are user-defined hyperparameter weights.

**Defaults:**
* $\lambda_{\text{sparse}} = 0.001$
* $\lambda_{\text{ratio}} = 0.01$

---

## Benchmark Results

Evaluated across canonical machine learning and computer vision datasets, S-ALCoN achieves competitive validation accuracy while reducing active feature footprints.

| Dataset | Type | Architecture | Best Val Acc (%) | Avg. Retention Ratio ($\bar{\rho}$) |
| :--- | :--- | :--- | ---: | ---: |
| **MNIST** | Image Classification | `[784, 256, 128, 64, 10]` | **98.24%** | 0.42 |
| **FashionMNIST** | Image Classification | `[784, 256, 128, 64, 10]` | **88.51%** | 0.51 |
| **California Housing** | Binarized Tabular | `[8, 6, 4, 2]` | **85.49%** | 0.38 |
| **Wine** | Tabular Classification | `[13, 10, 8, 3]` | **97.22%** | 0.31 |
| **Iris** | Tabular Classification | `[4, 6, 4, 3]` | **96.67%** | 0.28 |

> **Note on California Housing:** In alignment with the paper's experimental protocol, continuous tabular target values are binarized at the median target threshold into a classification task (`[8, 6, 4, 2]` ending in 2 binary logits), evaluated via Cross-Entropy and classification accuracy.

---

## Installation

### Install from PyPI

```bash
pip install salcon
