Metadata-Version: 2.2
Name: hical
Version: 0.2.2
Summary: Python-first active learning engine backed by libhicalengine
Keywords: active-learning,continuous-active-learning,information-retrieval,ranking,legal-tech
Author: Mustafa Abualsaud
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: C++
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Project-URL: Homepage, https://github.com/gathera/CALEngine
Project-URL: Documentation, https://github.com/gathera/CALEngine/blob/main/README.md
Project-URL: Repository, https://github.com/gathera/CALEngine
Project-URL: Issues, https://github.com/gathera/CALEngine/issues
Project-URL: Releases, https://github.com/gathera/CALEngine/releases
Requires-Python: >=3.10
Requires-Dist: PyYAML>=6.0
Requires-Dist: snowballstemmer>=2.0
Requires-Dist: tqdm>=4.60
Provides-Extra: datasets
Requires-Dist: ir_datasets>=0.5.11; extra == "datasets"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Description-Content-Type: text/markdown

# hical

[![CI](https://github.com/gathera/CALEngine/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/gathera/CALEngine/actions/workflows/ci.yml)
[![Package](https://github.com/gathera/CALEngine/actions/workflows/package.yml/badge.svg?branch=main)](https://github.com/gathera/CALEngine/actions/workflows/package.yml)
[![PyPI](https://img.shields.io/pypi/v/hical)](https://pypi.org/project/hical/)

`hical` helps you review large document collections more efficiently. Instead
of working through a corpus uniformly, you give it a few relevant examples and
review feedback, and it keeps reranking the remaining documents so the next
batch is more likely to matter.

Its primary use is helping reviewers find more relevant documents faster. It is
also useful for building labeled datasets and evaluation or test collections
without judging the entire corpus by hand.

In practice, `hical` gives you a Python workflow to build a corpus dataset from
raw documents or `ir_datasets`, open it, and run interactive review sessions.

The source lives in the `CALEngine` repository. The intended user-facing
surface is the `hical` Python package.

## Installation

Install the package from PyPI:

```bash
python -m pip install hical
```

If you want `ir_datasets` support:

```bash
python -m pip install "hical[datasets]"
```

If you are building from a source checkout, see
[docs/developer/development.md](docs/developer/development.md).

## Quickstart

Create a tiny JSONL corpus:

```bash
cat > docs.jsonl <<'EOF'
{"id": "doc-1", "title": "Florida citrus", "body": "Oranges and groves across central Florida."}
{"id": "doc-2", "title": "Coastal cleanup", "body": "Shoreline cleanup and beach restoration projects."}
EOF
```

Create a minimal config:

```bash
cat > corpus.yaml <<'EOF'
input:
  format: jsonl
  path: ./docs.jsonl
  doc_id_field: id
  text_fields:
    - title
    - body
output:
  path: ./docs.bin
  min_df: 1
  build_threads: 2
  parallel_docs_per_chunk: 50000
  optimize_for_fast_load: true
EOF
```

Build the corpus:

```bash
hical-build-corpus --config corpus.yaml
```

Open the dataset and start reviewing:

```python
import hical

dataset = hical.open_dataset("docs.bin")
session = dataset.start_session(
    relevant_seeds=["florida oranges"],
    review_batch_size=2,
    retraining="auto",
    retrain_every_n_judgments=2,
    session_id="demo-review",
)

batch = session.next_batch()
for item in batch:
    print(item.doc_id, item.score)

session.judge_relevant(batch[0])
```

The normal flow is:

1. build a `.bin`
2. open it as a `Dataset`
3. start a `Session`
4. fetch documents and record judgments

For the purpose and workflow at a higher level, see
[docs/overview.md](docs/overview.md).

## Common Tasks

### Build your own corpus

Use `hical-build-corpus` with JSONL, CSV, TSV, archive, or `ir_datasets`
input. For working configs and sample inputs, see:

- [docs/build-your-own-corpus.md](docs/build-your-own-corpus.md)
- [preprocessing/python/config_examples/](preprocessing/python/config_examples/)
- [examples/custom_data/](examples/custom_data/)

### Use with `ir_datasets`

Build directly from a dataset id:

```bash
hical-build-ir-dataset --dataset-id cranfield --output ./cranfield.bin
```

Inspect fields first when the document type has multiple useful fields:

```bash
hical-build-ir-dataset --dataset-id beir/msmarco --list-fields
```

Then choose specific fields to combine:

```bash
hical-build-ir-dataset \
  --dataset-id beir/msmarco \
  --text-field title \
  --text-field text \
  --output ./msmarco.bin
```

For more, see [docs/ir-datasets.md](docs/ir-datasets.md).

### Use the Python API

The main public entry points are:

- `hical.build_corpus`
- `hical.build_ir_dataset_corpus`
- `hical.inspect_ir_dataset`
- `hical.build_fast_load_index`
- `hical.open_dataset`
- `dataset.start_session`
- `session.next_batch`

For the fuller dataset/session workflow, see [docs/python-api.md](docs/python-api.md).

## Documentation

- [docs/README.md](docs/README.md): user guide index
- [docs/overview.md](docs/overview.md): what `hical` is for and how the review loop works
- [docs/concepts.md](docs/concepts.md): core objects and terminology
- [docs/build-your-own-corpus.md](docs/build-your-own-corpus.md): corpus build workflows and config examples
- [docs/ir-datasets.md](docs/ir-datasets.md): direct `ir_datasets` corpus building
- [docs/python-api.md](docs/python-api.md): dataset and session usage from Python
- [docs/cli.md](docs/cli.md): `bmi_cli` usage
- [docs/troubleshooting.md](docs/troubleshooting.md): setup and runtime issues
- [examples/README.md](examples/README.md): runnable examples

## Supported Platforms

Published wheels are smoke-tested on:

- Linux `x86_64`
- macOS `x86_64`
- macOS `arm64`

## Contributing

If you want to work on the repository internals rather than just use the Python
package, start here:

- [docs/developer/README.md](docs/developer/README.md)
- [docs/developer/development.md](docs/developer/development.md)
- [docs/developer/architecture.md](docs/developer/architecture.md)
