---
myst:
  html_meta:
    title: AutoRAG - BM25
    description: Learn about BM25 module in AutoRAG
    keywords: AutoRAG,RAG,Advanced RAG,retrieval,BM25
---
# BM25

The `BM25` is the most popular TF-IDF method for retrieval, which reflects how important a word is to a document. It is often called sparse retrieval. It is different with dense retrieval, which is using embedding model and similarity search. Dense retrieval search passage using semantic similarity, but sparse retrieval uses word counts. If you use documents in specific domains, `BM25` can be more useful than `VectorDB`. It uses the BM25Okapi algorithm for scoring and ranking the passages. 

## **Module Parameters**

- **bm25_tokenizer**: You can select which tokenize method you use for bm25.
  The default method is 'porter_stemmer'.
  And you can choose between 'ko_kiwi', 'space', and huggingface AutoTokenizer name.

### porter_stemmer

The default method to tokenize. It is optimized for English. It divides sentences to word, and extract stem.

It means, stemmer can change 'studying', 'studies' to 'study'.

### ko_kiwi

It uses kiwi tokenizer for Korean language.
We highly recommend to use it for Korean documents.
You can check more information about kiwi at [here](https://github.com/bab2min/Kiwi).

### space

It is simple method to divide words into just space.
It is simple, but it can be a great choice for multilingual documents.

### Huggingface AutoTokenizer

You can use any `AutoTokenizer` from huggingface, like gpt2 or mistralai/Mistral-7B-Instruct-v0.2.
Just type huggingface repo path, and you can use the tokenizer.

## **Example config.yaml**
```yaml
modules:
  - module_type: bm25
    bm25_tokenizer: [ porter_stemmer, ko_kiwi, space, gpt2 ]
```
