============
filedb notes
============

TBD.

Files created
=============

<revision_number>.toc
    The "master" file containing information about the index and its segments.

The index directory will contain a set of files for each segment. A segment is like a mini-index -- when you add documents to the index, semlix creates a new segment and then searches the old segment(s) and the new segment to avoid having to do a big merge every time you add a document. When you get enough small segments semlix will merge them into larger segments or a single segment.

<segment_number>.dci
    Contains per-document information (e.g. field lengths). This will grow linearly with the number of documents.

<segment_number>.dcz
    Contains the stored fields for each document.

<segment_number>.tiz
    Contains per-term information. The size of file will vary based on the number of unique terms.

<segment_number>.pst
    Contains per-term postings. The size of this file depends on the size of the collection and the formats used for each field (e.g. storing term positions takes more space than storing frequency only).

<segment_number>.fvz
    contains term vectors (forward indexes) for each document. This file is only created if at least one field in the schema stores term vectors. The size will vary based on the number of documents, field length, the formats used for each vector (e.g. storing term positions takes more space than storing frequency only), etc.

