Metadata-Version: 2.4
Name: rag-semantic-chunker
Version: 0.1.0
Summary: Semantic code chunking for LLM processing with policy-based configuration
Project-URL: Homepage, https://github.com/zthanos/SemanticCodeChunker
Project-URL: Documentation, https://github.com/zthanos/SemanticCodeChunker#readme
Project-URL: Repository, https://github.com/zthanos/SemanticCodeChunker
Project-URL: Issues, https://github.com/zthanos/SemanticCodeChunker/issues
Author: Thanos Zikas
License: MIT
License-File: LICENSE
Keywords: code-chunking,llm,rag,semantic-search
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.9
Requires-Dist: pyyaml>=6.0
Requires-Dist: tree-sitter-languages>=1.8.0
Requires-Dist: tree-sitter>=0.23.2
Provides-Extra: dev
Requires-Dist: black>=23.0; extra == 'dev'
Requires-Dist: mypy>=1.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Description-Content-Type: text/markdown

# 🧩 Semantic Code Chunker

<div align="center">

**Semantic code chunking library optimized for LLM processing and RAG systems**

[![PyPI version](https://badge.fury.io/py/rag-semantic-chunker.svg)](https://pypi.org/project/rag-semantic-chunker/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)

</div>

---

## 📖 Table of Contents

- [Overview](#overview)
- [Features](#features)
- [Installation](#installation)
- [Quick Start](#quick-start)
- [Supported Languages](#supported-languages)
- [Configuration](#configuration)
- [API Reference](#api-reference)
- [Advanced Usage](#advanced-usage)
- [Architecture](#architecture)
- [Contributing](#contributing)
- [License](#license)

---

## 🎯 Overview

**Semantic Code Chunker** is a Python library that intelligently splits source code into meaningful chunks based on Abstract Syntax Tree (AST) analysis. Unlike naive character-based chunking, this library understands the structure of your code and ensures that:

- ✅ Functions are not split in half
- ✅ Classes stay together with their methods
- ✅ Context is preserved for LLM understanding
- ✅ Configurable per-language policies

Perfect for **RAG systems**, **code analysis tools**, and **AI-powered development assistants**.

---

## ✨ Features

| Feature | Description |
|---------|-------------|
| 🌳 **AST-Based Parsing** | Uses tree-sitter for accurate code structure understanding |
| 🎯 **Semantic Chunking** | Chunks by functions, classes, methods - not arbitrary positions |
| ⚙️ **Policy-Driven** | YAML configuration per language with fine-grained control |
| 🌍 **Multi-Language** | Supports 12+ languages including legacy systems (COBOL, JCL) |
| 🔧 **Fallback Mode** | Regex-based parsing when tree-sitter unavailable |
| 📊 **Rich Metadata** | Each chunk includes type, line numbers, function names |
| ⚡ **High Performance** | Optimized for large codebases |

---

## 📦 Installation

### Using pip

```bash
pip install rag-semantic-chunker
```

### Using uv (recommended - faster)

```bash
uv add rag-semantic-chunker
```

### Development Installation

```bash
git clone https://github.com/zthanos/SemanticCodeChunker.git
cd SemanticCodeChunker
uv venv .venv
source .venv/bin/activate  # Linux/macOS
.venv\Scripts\activate     # Windows
uv pip install -e ".[dev]"
```

---

## 🚀 Quick Start

### Basic Example

```python
from semantic_chunker import SemanticCodeChunker

# Initialize with policy file
chunker = SemanticCodeChunker("policy.yaml")

# Sample Python code
code = """
def hello_world():
    print("Hello World")

class Calculator:
    def add(self, a, b):
        return a + b
    
    def subtract(self, a, b):
        return a - b
"""

# Chunk the code (max 500 chars per chunk for demo)
chunks = chunker.chunk_code(
    language="python", 
    source_code=code,
    context_size=500
)

print(f"Created {len(chunks)} chunks:")
for i, chunk in enumerate(chunks):
    print(f"\n--- Chunk {i+1} ---")
    print(chunk[:200] + "..." if len(chunk) > 200 else chunk)
```

### Output

```
Created 2 chunks:

--- Chunk 1 ---
def hello_world():
    print("Hello World")

--- Chunk 2 ---
class Calculator:
    def add(self, a, b):
        return a + b
    
    def subtract(self, a, b):
        return a - b
```

---

## 🌍 Supported Languages

| Language | Parser | Target Nodes | Notes |
|----------|--------|--------------|-------|
| Python | `python` | functions, classes, decorators | Full support |
| Java | `java` | methods, classes, interfaces | Enterprise ready |
| JavaScript | `javascript` | functions, arrow funcs, classes | ES6+ supported |
| TypeScript | `typescript` | All JS + types, interfaces | Type-aware |
| C | `c` | functions, structs, enums | Systems programming |
| C++ | `cpp` | Classes, templates, namespaces | OOP features |
| C# | `c_sharp` | Methods, properties, events | .NET ecosystem |
| COBOL | `cobol` | Paragraphs, sections | Legacy/mainframe ⚠️ |
| JCL | `jcl` | Jobs, EXEC, DD statements | Mainframe jobs ⚠️ |
| CLIST | `clist` | Procedures, functions | TSO/E commands ⚠️ |
| Visual Basic | `vbnet` | Subs, functions, classes | VB.NET support |
| Elixir | `elixir` | def, modules, protocols | Functional paradigm |

> **⚠️ Note**: Legacy languages (COBOL, JCL, CLIST) use fallback regex parsing when tree-sitter grammars unavailable.

---

## ⚙️ Configuration

### Policy File Structure

Create a `policy.yaml` file to configure chunking behavior:

```yaml
defaults:
  max_context_size: 2048      # Default chars per chunk
  min_chunk_size: 100         # Minimum chunk size
  include_comments: true      # Include comments in chunks
  
languages:
  python:
    parser_name: "python"
    target_nodes:
      - "function_definition"
      - "class_definition"
      - "method_definition"
    
    metadata_fields:
      - "node_type"
      - "line_number"
      - "function_name"
      
  java:
    parser_name: "java"
    target_nodes:
      - "method_declaration"
      - "class_declaration"
```

### Programmatic Configuration

```python
from semantic_chunker.config import PolicyConfig, LanguageConfig

# Create custom configuration
config = PolicyConfig(
    max_context_size=4096,
    languages={
        "python": LanguageConfig(
            parser_name="python",
            target_nodes=["function_definition", "class_definition"]
        )
    }
)

chunker = SemanticCodeChunker(config=config)  # Pass config directly
```

---

## 📚 API Reference

### `SemanticCodeChunker`

Main class for chunking code.

#### Constructor

```python
def __init__(self, policy_path: str | None = None, config: PolicyConfig | None = None):
    """
    Initialize the chunker.
    
    Args:
        policy_path: Path to YAML policy file (optional)
        config: Direct PolicyConfig object (optional)
        
    Raises:
        FileNotFoundError: If policy_path doesn't exist
        ValueError: If configuration is invalid
    """
```

#### `chunk_code()` Method

```python
def chunk_code(
    self, 
    language: str, 
    source_code: str, 
    context_size: int | None = None
) -> List[str]:
    """
    Chunk source code semantically.
    
    Args:
        language: Language identifier (e.g., 'python', 'java')
        source_code: Source code string to chunk
        context_size: Override default max chunk size
        
    Returns:
        List of chunk strings respecting semantic boundaries
        
    Example:
        >>> chunks = chunker.chunk_code("python", code, context_size=2048)
        >>> len(chunks)  # Number of chunks created
        5
    """
```

### `ParsedChunk` Data Class

Detailed chunk information with metadata.

```python
@dataclass
class ParsedChunk:
    code: str                    # The actual code content
    metadata: ChunkMetadata      # Metadata about the chunk
    
# Accessing metadata
chunk = chunks[0]
print(chunk.metadata.node_type)     # "function_definition"
print(chunk.metadata.line_number)   # 42
print(chunk.metadata.function_name) # "calculate_sum"
```

---

## 🔬 Advanced Usage

### Getting Detailed Chunk Information

```python
from semantic_chunker import SemanticCodeChunker
from semantic_chunker.parser import parse_and_chunk

# Get chunks with full metadata
chunks = parse_and_chunk(
    source_code=code,
    language="python",
    target_nodes=["function_definition", "class_definition"],
    max_context_size=2048
)

for chunk in chunks:
    print(f"Type: {chunk.metadata.node_type}")
    print(f"Lines: {chunk.metadata.line_number}-{chunk.metadata.end_line}")
    print(f"Function: {chunk.metadata.function_name}")
    print(f"Code:\n{chunk.code}\n")
```

### Processing Large Files

```python
def process_large_file(file_path: str, language: str) -> List[str]:
    """Process large files in streaming fashion."""
    
    with open(file_path, 'r', encoding='utf-8') as f:
        source_code = f.read()
        
    chunker = SemanticCodeChunker("policy.yaml")
    
    # Process with larger context for big files
    chunks = chunker.chunk_code(
        language=language,
        source_code=source_code,
        context_size=4096  # Larger chunks for complex codebases
    )
    
    return chunks

# Usage
chunks = process_large_file("src/main.py", "python")
```

### Custom Language Support

```python
from semantic_chunker.parser import ASTParser

class RustParser(ASTParser):
    """Custom parser for Rust (not yet in tree-sitter-languages)."""
    
    def _get_fallback_patterns(self) -> Dict[str, str]:
        return {
            "function": r'\bfn\s+(\w+)\s*\(',
            "struct": r'\bstruct\s+(\w+)',
            "impl_block": r'\bimpl\s+',
        }

# Use custom parser
rust_parser = RustParser("rust")
```

---

## 🏗️ Architecture

### System Diagram

```
┌─────────────────────────────────────────────────────────────────┐
│                      Semantic Code Chunker                       │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐      │
│  │   Policy     │    │    AST       │    │   Chunk      │      │
│  │   Config     │───▶│    Parser    │───▶│   Extractor  │      │
│  │  (policy.yaml)│    │(tree-sitter) │    │              │      │
│  └──────────────┘    └──────────────┘    └──────────────┘      │
│         │                  │                    │               │
│         ▼                  ▼                    ▼               │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │                   SemanticCodeChunker                    │   │
│  │                                                         │   │
│  │  chunk_code(language, source_code, context_size)       │   │
│  │                                                         │   │
│  └─────────────────────────────────────────────────────────┘   │
│                              │                                  │
│                              ▼                                  │
│                    ┌───────────────────┐                       │
│                    │  List[ParsedChunk]│                       │
│                    │                   │                       │
│                    │  - code: str      │                       │
│                    │  - metadata       │                       │
│                    └───────────────────┘                       │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘
```

### Data Flow

1. **Load Policy** → Read YAML configuration for target language
2. **Parse AST** → tree-sitter creates Abstract Syntax Tree
3. **Query Nodes** → Extract functions, classes, methods based on policy
4. **Group Chunks** → Combine nodes respecting context_size limit
5. **Extract Metadata** → Add line numbers, names, types to each chunk

---

## 🤝 Contributing

We welcome contributions! Here's how you can help:

### Setting Up Development Environment

```bash
# Clone repository
git clone https://github.com/zthanos/SemanticCodeChunker.git
cd SemanticCodeChunker

# Create virtual environment
uv venv .venv
source .venv/bin/activate  # Linux/macOS

# Install with dev dependencies
uv pip install -e ".[dev]"

# Run tests
pytest tests/

# Format code
black src/semantic_chunker tests/

# Type checking
mypy src/semantic_chunker
```

### Adding Support for New Language

1. Add parser name to `LANGUAGE_PARSERS` in `parser.py`
2. Add default config in `config.py` → `get_default_languages()`
3. Add target nodes in example `policy.yaml`
4. Write tests in `tests/test_<language>.py`

### Running Tests

```bash
# All tests
pytest

# Specific test file
pytest tests/test_python_chunking.py

# With coverage
pytest --cov=src/semantic_chunker --cov-report=html

# Verbose output
pytest -v
```

---

## 📄 License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

---

## 📞 Support & Contact

- **Issues**: [GitHub Issues](https://github.com/zthanos/SemanticCodeChunker/issues)
- **Discussions**: [GitHub Discussions](https://github.com/zthanos/SemanticCodeChunker/discussions)

---

<div align="center">
Made with ❤️ for the AI and Developer Community<br><br>
⭐ Star this repo if you find it helpful!
</div>