Metadata-Version: 2.5
Name: cosq
Version: 0.1.0
Summary: Chain-of-Self-Questioning for selective factual answering with LLMs
Project-URL: Homepage, https://github.com/senolali/cosq
Project-URL: Repository, https://github.com/senolali/cosq
Project-URL: Issues, https://github.com/senolali/cosq/issues
Author-email: Ali Senol <alisenol@tarsus.edu.tr>
License: MIT License
        
        Copyright (c) 2026 Ali Şenol
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: abstention,hallucination,llm,question-answering,selective-prediction
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: numpy<2.4,>=1.24
Requires-Dist: pyyaml>=6.0
Requires-Dist: requests>=2.31
Requires-Dist: scipy>=1.10
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == 'dev'
Requires-Dist: mypy<2,>=1.11; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest<9,>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Requires-Dist: twine>=5.0; extra == 'dev'
Requires-Dist: types-pyyaml; extra == 'dev'
Provides-Extra: hf
Requires-Dist: accelerate>=0.30; extra == 'hf'
Requires-Dist: bitsandbytes>=0.43; extra == 'hf'
Requires-Dist: datasets>=2.18; extra == 'hf'
Requires-Dist: torch>=2.0; extra == 'hf'
Requires-Dist: transformers>=4.40; extra == 'hf'
Provides-Extra: openai
Requires-Dist: openai>=1.0; extra == 'openai'
Description-Content-Type: text/markdown

# CoSQ: Chain-of-Self-Questioning

CoSQ is a Python framework for **selective factual answering** with large language models. Instead of forcing a model to answer every question, CoSQ asks the model to decompose the question into required knowledge items, assess whether those items are supported, and then either answer or abstain with `I don't know`.

The framework is designed for reproducible experiments on hallucination, abstention, answered accuracy, and risk-coverage trade-offs.

## Why CoSQ?

Standard accuracy rewards guessing: an abstention is usually scored the same as a wrong answer. That is a poor fit for settings where a wrong answer is more costly than saying "I don't know". CoSQ treats factual answering as a selective prediction problem and reports:

- **Answered accuracy**: accuracy among parseable committed answers.
- **Coverage**: the fraction of questions the model answers.
- **Hallucination rate**: wrong committed answers divided by all questions.
- **Abstention rate**: explicit `I don't know` responses.
- **Unparseable rate**: outputs that cannot be mapped to the benchmark answer space.

## Installation

From PyPI, after release:

```bash
pip install cosq
```

For local development:

```bash
git clone https://github.com/senolali/cosq.git
cd cosq
python -m venv .venv
.venv\Scripts\activate
pip install -e ".[dev]"
```

Optional backends:

```bash
pip install -e ".[hf]"      # local Hugging Face / transformers models
pip install -e ".[openai]"  # OpenAI API backend
```

## Quick Start

Run a no-network smoke test with the deterministic mock backend:

```bash
cosq run --config configs/experiment/smoke_mock.yaml --backend mock --allow-dirty
```

The command writes a run directory under `results/runs/`. Then score and report it:

```bash
cosq evaluate results/runs/<RUN_DIR>
cosq analyze results/runs/<RUN_DIR>
cosq report results/runs/<RUN_DIR>
```

## Using OpenAI Models

Set your API key:

```bash
set OPENAI_API_KEY=sk-...        # Windows cmd
$env:OPENAI_API_KEY="sk-..."     # PowerShell
export OPENAI_API_KEY=sk-...     # macOS/Linux
```

Probe the model:

```bash
cosq probe --model configs/model/openai_gpt4o_mini.yaml
```

Run a small open-ended TruthfulQA pilot:

```bash
cosq run --config configs/experiment/pilot_truthfulqa_open_openai.yaml --allow-dirty
```

You can copy `configs/model/openai_template.yaml` and change `id` to another model available in your OpenAI account.

## Using Hugging Face Models

Install optional dependencies:

```bash
pip install -e ".[hf]"
```

For gated models, set `HF_TOKEN` and accept the model license on Hugging Face. Then copy one of the examples in `configs/model/`, replace `revision` with an immutable Hugging Face commit SHA, and run:

```bash
cosq probe --model configs/model/hf_llama3_8b_instruct.yaml
cosq run --config configs/experiment/pilot_truthfulqa_open.yaml --allow-dirty
```

Pinned revisions are required for local Hugging Face models so that a run refers to stable weights.

## Experiment Configuration

```yaml
name: smoke_mock
model: configs/model/mock.yaml
data:
  name: jsonl
  path: tests/fixtures/mini.jsonl
  n: 3
  seed: 1002
strategies:
  - name: direct
  - name: cot
  - name: cot_abstain
  - name: cosq
  - name: cosq_graded_gate
    label: cosq_graded_gate_mean060
    threshold: 0.60
    aggregator: mean
    on_empty: abstain
repeats: 1
open_ended: false
```

## Built-in Strategies

- `direct`: answer directly.
- `cot`: reason step by step, then answer.
- `cot_abstain`: CoT with permission to answer `I don't know`.
- `cosq`: binary Chain-of-Self-Questioning with a strict conjunctive gate.
- `cosq_gate`: CoSQ gate followed by CoT-style answer generation.
- `cosq_graded`: item-level 0-100 confidence scores with thresholding.
- `cosq_graded_gate`: graded gate followed by CoT-style answer generation.

## Data

CoSQ ships with a tiny JSONL fixture for tests. Benchmark datasets are loaded at runtime or supplied as JSONL files. Local dataset payloads are ignored by git to keep the package lightweight.

A custom JSONL dataset should contain:

```json
{"id":"q1","question":"What is the capital of France?","options":["Paris","Lyon"],"gold_index":0}
```

## Caching and Re-scoring

Runs can use a SQLite cache:

```bash
cosq run --config configs/experiment/pilot_truthfulqa_open_openai.yaml --cache results/cache.sqlite
```

Raw model outputs are written before scoring. You can improve parsers or metrics and re-run `evaluate`, `analyze`, or `report` without making new model calls.

## Development and PyPI Release

```bash
pip install -e ".[dev]"
pytest
ruff check .
python -m build
python -m twine check dist/*
```

To publish to PyPI:

```bash
python -m build
python -m twine upload dist/*
```

## Repository Layout

```text
src/cosq/
  backends/       # mock, local Hugging Face, OpenAI API
  strategies/     # direct, CoT, CoSQ, graded CoSQ variants
  decision/       # binary and confidence-based gates
  data/           # dataset adapters
  eval/           # offline scoring and metrics
  report/         # tables and statistical summaries
  prompts/        # versioned prompt templates
configs/
  experiment/     # runnable experiment YAML files
  model/          # model backend examples
tests/            # no-network test suite
```

## Citation

If you use CoSQ in academic work, please cite the accompanying paper once available. A BibTeX entry will be added after publication.

## License

MIT License.
