Metadata-Version: 2.4
Name: tokensplit
Version: 0.1.1
Summary: String-separated values with user-defined multi-character delimiters
Author-email: lost_0 <l05t_0@proton.me>
License-Expression: MIT
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Classifier: Topic :: File Formats
Requires-Python: >=3.7
Description-Content-Type: text/markdown
License-File: LICENSE.txt
Dynamic: license-file

# TokenSplit — Token-Separated Values

A lightweight Python package for reading and writing `.toks` files: a plain-text tabular format where you choose your own multi-character delimiter string.

---

## Why?

CSV uses a single character (`,`) as a separator, which means commas in your data need escaping or quoting.  
Toks lets you pick any string — `/---/`, `:::`, `<<SEP>>` — that you know won't appear in your data, keeping files simple and unambiguous without any escape sequences.

---

## File format

```
/---/
Alice/---/30/---/Engineer/---/
Bob/---/25/---/Designer/---/
```

- **Line 1** — the delimiter string (written automatically by the writer)
- **Every other line** — values separated by the delimiter, with the line ending on `<delimiter><newline>`

Newlines *inside* a value are preserved because rows end only on the `<delimiter><newline>` sequence, not on bare newlines.

---

## Installation

```bash
pip install tokensplit           # once published to PyPI
# or, from source:
pip install .
```

---

## Quick start

### Writing

```python
import tokensplit

# Convenience function
tokensplit.write("people.toks", [
    ["name", "age", "role"],
    ["Alice", "30", "Engineer"],
    ["Bob",   "25", "Designer"],
], delimiter="/---/")
```

```python
# Streaming writer — useful for large files
with open("people.toks", "w") as f:
    writer = tokensplit.ToksWriter(f, delimiter="/---/")
    writer.writerow(["name", "age", "role"])   # header
    writer.writerow(["Alice", "30", "Engineer"])
    writer.writerow(["Bob",   "25", "Designer"])
```

### Reading

```python
import tokensplit

# Convenience function — returns list of rows
rows = tokensplit.read("people.toks")
# [["name", "age", "role"], ["Alice", "30", "Engineer"], ...]

# Streaming reader — one row at a time (memory-efficient)
with open("people.toks") as f:
    reader = tokensplit.ToksReader(f)
    print("delimiter:", reader.delimiter)   # "/---/"
    for row in reader:
        print(row)
```

---

## Choosing a delimiter

Any non-empty string without a newline character works. Good choices:

| Delimiter | Good when data contains… |
|-----------|--------------------------|
| `/---/`   | General text |
| `\|\|\|`  | Paths, URLs |
| `<<<>>>`  | Code snippets |
| `,,,,`    | Numeric CSVs being converted |
| `:::`     | Short labels / IDs |

**Two rules enforced by the writer:**

1. A value must not *contain* the delimiter string.
2. A value must not end with a prefix of the delimiter in a way that creates an ambiguous sequence when written (e.g. value `"aa"` with delimiter `"aaa"` would produce `"aaaaa"` which embeds an extra delimiter). A `ValueError` is raised in both cases.

---

## API reference

### `tokensplit.write(filepath, rows, delimiter)`
Write `rows` (list of lists of strings) to `filepath`.

### `tokensplit.read(filepath) → List[List[str]]`
Read all rows from `filepath`. Returns a list of lists of strings.

### `tokensplit.ToksWriter(file_obj, delimiter)`
Streaming writer. Call `.writerow(row)` or `.writerows(rows)`.  
The delimiter is written to line 1 of the file on construction.

### `tokensplit.ToksReader(file_obj)`
Streaming reader. Iterate with `for row in reader`.  
`.delimiter` attribute exposes the delimiter read from line 1.

---

## Reading algorithm

The reader uses a **forward-only sliding window** of exactly `len(delimiter)` characters:

```
content:   h e l l o / - - - / w o r l d / - - - / \n
window:    [     5     ]
                  → slides one character at a time
                        match! → emit token, jump window past delimiter
```

- **Time:** O(n) — every character is visited once; one slice emitted per match  
- **Extra space:** O(d) — only the current window lives in memory beyond the content string  
- No regex, no `str.split`, no backtracking

---

## Running tests

```bash
python -m pytest tests/
```
