# V1.1 NL Descriptions Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Add LLM-generated natural language descriptions to code chunks that lack docstrings, improving semantic search quality by ~12% for NL-to-code queries.

**Architecture:** Pluggable `DescriptionProvider` ABC with Anthropic Claude as default implementation. Descriptions are generated inline during indexing, cached by content hash in SQLite, and prepended to chunk content before embedding (leveraging voyage-code-3's docstring-code pair training). The provider is wired through `factory.py` like all other components.

**Tech Stack:** anthropic SDK (async), SQLite for caching, tree-sitter for docstring extraction

---

## Design Decisions

| Decision | Choice | Rationale |
|----------|--------|-----------|
| LLM provider | Pluggable ABC, Claude default | User requested pluggability |
| Search integration | Prepend description to content before embedding | Greptile data: +12% similarity; matches voyage-code-3 docstring-code training distribution |
| Generation timing | Inline during indexing | Simpler architecture, no re-embedding pass needed |
| Cache key | `(content_hash, provider_model)` | Different models → different descriptions; prompt changes within model → manual cache invalidation |
| Fallback on failure | Skip description, log warning, index chunk without it | Graceful degradation — indexing must not fail due to LLM errors |
| Docstring detection | tree-sitter AST extraction in PythonChunker | Already have AST; docstring is first `expression_statement > string` child of function/class body |

---

### Task 1: Add `docstring` field to `CodeEntity` and extract from AST

**Files:**
- Modify: `clew/chunker/strategies.py`
- Test: `tests/unit/test_strategies.py`

**Step 1: Write the failing tests**

In `tests/unit/test_strategies.py`, add tests for docstring extraction. The `PythonChunker` already has tests — add these alongside existing ones:

```python
class TestDocstringExtraction:
    """Test docstring extraction from AST nodes."""

    def test_function_with_docstring(self, python_chunker, parser):
        source = '''def greet(name: str) -> str:
    """Say hello to someone."""
    return f"Hello, {name}"
'''
        tree = parser.parse(source.encode("utf-8"))
        entities = list(python_chunker.extract_entities(tree, source))
        assert len(entities) == 1
        assert entities[0].docstring == "Say hello to someone."

    def test_function_without_docstring(self, python_chunker, parser):
        source = '''def add(a, b):
    return a + b
'''
        tree = parser.parse(source.encode("utf-8"))
        entities = list(python_chunker.extract_entities(tree, source))
        assert len(entities) == 1
        assert entities[0].docstring is None

    def test_class_with_docstring(self, python_chunker, parser):
        source = '''class User:
    """Represents a user account."""
    name: str
'''
        tree = parser.parse(source.encode("utf-8"))
        entities = list(python_chunker.extract_entities(tree, source))
        assert len(entities) == 1
        assert entities[0].docstring == "Represents a user account."

    def test_method_with_multiline_docstring(self, python_chunker, parser):
        source = '''class Calculator:
    def multiply(self, a, b):
        """Multiply two numbers.

        Args:
            a: First number.
            b: Second number.
        """
        return a * b
'''
        tree = parser.parse(source.encode("utf-8"))
        entities = list(python_chunker.extract_entities(tree, source))
        method = [e for e in entities if e.entity_type == "method"][0]
        assert method.docstring is not None
        assert method.docstring.startswith("Multiply two numbers.")

    def test_function_with_string_literal_not_docstring(self, python_chunker, parser):
        source = '''def example():
    x = "not a docstring"
    return x
'''
        tree = parser.parse(source.encode("utf-8"))
        entities = list(python_chunker.extract_entities(tree, source))
        assert entities[0].docstring is None
```

Note: The tests use existing `python_chunker` and `parser` fixtures. Check `conftest.py` for their definitions — if they don't exist, you'll need fixtures like:

```python
@pytest.fixture
def parser():
    from tree_sitter_languages import get_parser
    return get_parser("python")

@pytest.fixture
def python_chunker():
    return PythonChunker()
```

**Step 2: Run tests to verify they fail**

Run: `pytest tests/unit/test_strategies.py::TestDocstringExtraction -v`
Expected: FAIL — `CodeEntity` has no `docstring` attribute

**Step 3: Add `docstring` field to `CodeEntity`**

In `clew/chunker/strategies.py`, modify the `CodeEntity` dataclass:

```python
@dataclass
class CodeEntity:
    """A parsed code entity (class, function, method)."""

    entity_type: str
    name: str
    qualified_name: str
    content: str
    line_start: int
    line_end: int
    parent_class: str | None
    docstring: str | None = None
```

**Step 4: Add `_extract_docstring` helper and wire it into `_node_to_entity`**

Add this method to `PythonChunker`:

```python
def _extract_docstring(self, node: Any, source_bytes: bytes) -> str | None:
    """Extract docstring from a function/class definition node.

    Docstrings are the first expression_statement containing a string
    in the body block.
    """
    for child in node.children:
        if child.type == "block":
            for stmt in child.children:
                if stmt.type == "expression_statement":
                    for expr in stmt.children:
                        if expr.type == "string":
                            raw = source_bytes[expr.start_byte : expr.end_byte].decode("utf-8")
                            # Strip triple-quote delimiters and clean whitespace
                            stripped = raw.strip("\"'").strip()
                            return stripped if stripped else None
                # Only the first statement can be a docstring
                if stmt.type not in ("comment", "newline", "expression_statement"):
                    break
    return None
```

Then update all three `yield CodeEntity(...)` calls in `_node_to_entity` to include `docstring=self._extract_docstring(node, source_bytes)` for classes and functions, and `docstring=self._extract_docstring(method, source_bytes)` for methods.

**Step 5: Run tests to verify they pass**

Run: `pytest tests/unit/test_strategies.py -v`
Expected: ALL PASS (new + existing)

**Step 6: Run full test suite to check for regressions**

Run: `pytest --tb=short -q`
Expected: 349 passed (the new `docstring` field defaults to `None` so existing code is unaffected)

**Step 7: Commit**

```bash
git add clew/chunker/strategies.py tests/unit/test_strategies.py
git commit -m "feat: extract docstrings from AST in PythonChunker"
```

---

### Task 2: Create `DescriptionProvider` ABC and `AnthropicDescriptionProvider`

**Files:**
- Create: `clew/clients/description.py`
- Test: `tests/unit/test_description_provider.py`

**Step 1: Write the failing tests**

Create `tests/unit/test_description_provider.py`:

```python
"""Tests for description generation providers."""

from __future__ import annotations

from unittest.mock import AsyncMock, MagicMock, patch

import pytest

from clew.clients.description import AnthropicDescriptionProvider


class TestAnthropicDescriptionProvider:
    """Test Anthropic Claude description provider."""

    def test_model_name(self):
        provider = AnthropicDescriptionProvider(api_key="test-key")
        assert provider.model_name == "claude-sonnet-4-5-20250929"

    def test_custom_model(self):
        provider = AnthropicDescriptionProvider(
            api_key="test-key", model="claude-haiku-4-5-20251001"
        )
        assert provider.model_name == "claude-haiku-4-5-20251001"

    @pytest.mark.asyncio
    async def test_generate_description(self):
        provider = AnthropicDescriptionProvider(api_key="test-key")
        mock_response = MagicMock()
        mock_response.content = [MagicMock(text="Validate an email address format.")]

        with patch.object(
            provider._client.messages, "create", new_callable=AsyncMock
        ) as mock_create:
            mock_create.return_value = mock_response
            result = await provider.generate_description(
                code='def validate_email(email: str) -> bool:\n    return "@" in email',
                language="python",
                entity_type="function",
                name="validate_email",
            )

        assert result == "Validate an email address format."
        mock_create.assert_called_once()
        call_kwargs = mock_create.call_args[1]
        assert call_kwargs["model"] == "claude-sonnet-4-5-20250929"
        assert call_kwargs["max_tokens"] == 150

    @pytest.mark.asyncio
    async def test_generate_description_strips_whitespace(self):
        provider = AnthropicDescriptionProvider(api_key="test-key")
        mock_response = MagicMock()
        mock_response.content = [MagicMock(text="  Check user permissions.  \n")]

        with patch.object(
            provider._client.messages, "create", new_callable=AsyncMock
        ) as mock_create:
            mock_create.return_value = mock_response
            result = await provider.generate_description(
                code="def check_perms(): pass",
                language="python",
                entity_type="function",
                name="check_perms",
            )

        assert result == "Check user permissions."

    @pytest.mark.asyncio
    async def test_generate_description_returns_none_on_error(self):
        provider = AnthropicDescriptionProvider(api_key="test-key")

        with patch.object(
            provider._client.messages, "create", new_callable=AsyncMock
        ) as mock_create:
            mock_create.side_effect = Exception("API error")
            result = await provider.generate_description(
                code="def broken(): pass",
                language="python",
                entity_type="function",
                name="broken",
            )

        assert result is None

    @pytest.mark.asyncio
    async def test_generate_description_returns_none_on_empty(self):
        provider = AnthropicDescriptionProvider(api_key="test-key")
        mock_response = MagicMock()
        mock_response.content = [MagicMock(text="   ")]

        with patch.object(
            provider._client.messages, "create", new_callable=AsyncMock
        ) as mock_create:
            mock_create.return_value = mock_response
            result = await provider.generate_description(
                code="def empty(): pass",
                language="python",
                entity_type="function",
                name="empty",
            )

        assert result is None

    @pytest.mark.asyncio
    async def test_generate_batch(self):
        provider = AnthropicDescriptionProvider(api_key="test-key")
        mock_response = MagicMock()
        mock_response.content = [MagicMock(text="A description.")]

        with patch.object(
            provider._client.messages, "create", new_callable=AsyncMock
        ) as mock_create:
            mock_create.return_value = mock_response
            items = [
                {"code": "def a(): pass", "language": "python", "entity_type": "function", "name": "a"},
                {"code": "def b(): pass", "language": "python", "entity_type": "function", "name": "b"},
            ]
            results = await provider.generate_batch(items)

        assert len(results) == 2
        assert all(r == "A description." for r in results)
```

**Step 2: Run tests to verify they fail**

Run: `pytest tests/unit/test_description_provider.py -v`
Expected: FAIL — `ModuleNotFoundError: No module named 'clew.clients.description'`

**Step 3: Write the implementation**

Create `clew/clients/description.py`:

```python
"""Description generation providers for NL chunk descriptions."""

from __future__ import annotations

import asyncio
import logging
from abc import ABC, abstractmethod

logger = logging.getLogger(__name__)


class DescriptionProvider(ABC):
    """Abstract base class for NL description generators."""

    @property
    @abstractmethod
    def model_name(self) -> str:
        """Return the model identifier."""
        ...

    @abstractmethod
    async def generate_description(
        self,
        code: str,
        language: str,
        entity_type: str,
        name: str,
    ) -> str | None:
        """Generate a natural language description for a code chunk.

        Returns None if generation fails or produces empty output.
        """
        ...

    async def generate_batch(
        self,
        items: list[dict[str, str]],
    ) -> list[str | None]:
        """Generate descriptions for multiple chunks.

        Default implementation calls generate_description sequentially.
        Subclasses may override for batched API calls.
        """
        results: list[str | None] = []
        for item in items:
            result = await self.generate_description(
                code=item["code"],
                language=item["language"],
                entity_type=item["entity_type"],
                name=item["name"],
            )
            results.append(result)
        return results


_DESCRIPTION_PROMPT = """Generate a concise 1-2 sentence natural language description of this {entity_type}.
Focus on WHAT it does and WHY, not HOW. Write as if for a code search index.

{entity_type}: {name}
Language: {language}

```{language}
{code}
```

Description:"""


class AnthropicDescriptionProvider(DescriptionProvider):
    """Generate descriptions using Anthropic Claude."""

    def __init__(
        self,
        api_key: str,
        model: str = "claude-sonnet-4-5-20250929",
        max_tokens: int = 150,
        max_concurrent: int = 5,
    ) -> None:
        import anthropic

        self._client = anthropic.AsyncAnthropic(api_key=api_key)
        self._model = model
        self._max_tokens = max_tokens
        self._semaphore = asyncio.Semaphore(max_concurrent)

    @property
    def model_name(self) -> str:
        return self._model

    async def generate_description(
        self,
        code: str,
        language: str,
        entity_type: str,
        name: str,
    ) -> str | None:
        prompt = _DESCRIPTION_PROMPT.format(
            code=code,
            language=language,
            entity_type=entity_type,
            name=name,
        )
        try:
            async with self._semaphore:
                response = await self._client.messages.create(
                    model=self._model,
                    max_tokens=self._max_tokens,
                    messages=[{"role": "user", "content": prompt}],
                )
            text = response.content[0].text.strip()
            return text if text else None
        except Exception:
            logger.warning("Description generation failed for %s", name, exc_info=True)
            return None

    async def generate_batch(
        self,
        items: list[dict[str, str]],
    ) -> list[str | None]:
        """Generate descriptions concurrently with semaphore throttling."""
        tasks = [
            self.generate_description(
                code=item["code"],
                language=item["language"],
                entity_type=item["entity_type"],
                name=item["name"],
            )
            for item in items
        ]
        return list(await asyncio.gather(*tasks))
```

**Step 4: Run tests to verify they pass**

Run: `pytest tests/unit/test_description_provider.py -v`
Expected: ALL PASS

**Step 5: Commit**

```bash
git add clew/clients/description.py tests/unit/test_description_provider.py
git commit -m "feat: add DescriptionProvider ABC with Anthropic implementation"
```

---

### Task 3: Add description cache to `CacheDB`

**Files:**
- Modify: `clew/indexer/cache.py`
- Test: `tests/unit/test_cache.py`

**Step 1: Write the failing tests**

Add to the existing `tests/unit/test_cache.py`:

```python
class TestDescriptionCache:
    """Test NL description caching."""

    def test_get_description_missing(self, cache_db):
        result = cache_db.get_description("abc123", "claude-sonnet-4-5-20250929")
        assert result is None

    def test_set_and_get_description(self, cache_db):
        cache_db.set_description("abc123", "claude-sonnet-4-5-20250929", "Validates email format.")
        result = cache_db.get_description("abc123", "claude-sonnet-4-5-20250929")
        assert result == "Validates email format."

    def test_description_keyed_by_model(self, cache_db):
        cache_db.set_description("abc123", "model-a", "Description A")
        cache_db.set_description("abc123", "model-b", "Description B")
        assert cache_db.get_description("abc123", "model-a") == "Description A"
        assert cache_db.get_description("abc123", "model-b") == "Description B"

    def test_description_upsert(self, cache_db):
        cache_db.set_description("abc123", "model-a", "Old")
        cache_db.set_description("abc123", "model-a", "New")
        assert cache_db.get_description("abc123", "model-a") == "New"
```

Note: Use the existing `cache_db` fixture from `conftest.py`. If it doesn't exist, create one:

```python
@pytest.fixture
def cache_db(tmp_path):
    return CacheDB(tmp_path)
```

**Step 2: Run tests to verify they fail**

Run: `pytest tests/unit/test_cache.py::TestDescriptionCache -v`
Expected: FAIL — `CacheDB` has no `get_description` method

**Step 3: Add description_cache table and methods**

In `clew/indexer/cache.py`, add to `CACHE_SCHEMA` (append before the closing `"""`):

```sql
CREATE TABLE IF NOT EXISTS description_cache (
    content_hash TEXT NOT NULL,
    provider_model TEXT NOT NULL,
    description TEXT NOT NULL,
    created_at TEXT DEFAULT (datetime('now')),
    PRIMARY KEY (content_hash, provider_model)
);

CREATE INDEX IF NOT EXISTS idx_description_cache_model
ON description_cache(provider_model);
```

Add two methods to `CacheDB`:

```python
def get_description(self, content_hash: str, model: str) -> str | None:
    """Get cached NL description or None if not found."""
    with self._get_cache_conn() as conn:
        row = conn.execute(
            "SELECT description FROM description_cache "
            "WHERE content_hash = ? AND provider_model = ?",
            (content_hash, model),
        ).fetchone()
        return row["description"] if row else None

def set_description(
    self,
    content_hash: str,
    model: str,
    description: str,
) -> None:
    """Cache an NL description."""
    with self._get_cache_conn() as conn:
        conn.execute(
            "INSERT OR REPLACE INTO description_cache "
            "(content_hash, provider_model, description) "
            "VALUES (?, ?, ?)",
            (content_hash, model, description),
        )
```

**Step 4: Run tests to verify they pass**

Run: `pytest tests/unit/test_cache.py -v`
Expected: ALL PASS (new + existing)

**Step 5: Commit**

```bash
git add clew/indexer/cache.py tests/unit/test_cache.py
git commit -m "feat: add description_cache table to CacheDB"
```

---

### Task 4: Add config and environment support

**Files:**
- Modify: `clew/config.py`
- Modify: `clew/models.py`
- Test: `tests/unit/test_config.py` (if exists), `tests/unit/test_models.py` (if exists)

**Step 1: Write the failing tests**

Test the new config fields:

```python
# In test for models.py
def test_indexing_config_nl_description_defaults():
    config = IndexingConfig()
    assert config.nl_description_enabled is False
    assert config.nl_description_model == "claude-sonnet-4-5-20250929"
    assert config.nl_description_max_concurrent == 5


def test_indexing_config_nl_description_custom():
    config = IndexingConfig(
        nl_description_enabled=True,
        nl_description_model="claude-haiku-4-5-20251001",
        nl_description_max_concurrent=10,
    )
    assert config.nl_description_enabled is True
    assert config.nl_description_model == "claude-haiku-4-5-20251001"


# In test for config.py
def test_environment_anthropic_api_key(monkeypatch):
    monkeypatch.setenv("ANTHROPIC_API_KEY", "test-key-123")
    env = Environment()
    assert env.ANTHROPIC_API_KEY == "test-key-123"


def test_environment_anthropic_api_key_default():
    env = Environment()
    # Should be empty string when not set (not raise)
    assert isinstance(env.ANTHROPIC_API_KEY, str)
```

**Step 2: Run tests to verify they fail**

Run: `pytest tests/unit/test_models.py::test_indexing_config_nl_description_defaults -v`
Expected: FAIL — `IndexingConfig` has no `nl_description_enabled` field

**Step 3: Add config fields**

In `clew/models.py`, add to `IndexingConfig`:

```python
class IndexingConfig(BaseModel):
    """Indexing pipeline configuration."""

    overlap_tokens: int = Field(default=200, ge=0, le=1000)
    fallback_max_tokens: int = Field(default=3000, ge=500, le=8000)
    embedding_provider: str = Field(default="voyage")
    embedding_model: str = Field(default="voyage-code-3")
    nl_description_enabled: bool = Field(default=False)
    nl_description_model: str = Field(default="claude-sonnet-4-5-20250929")
    nl_description_max_concurrent: int = Field(default=5, ge=1, le=20)
```

In `clew/config.py`, add to `Environment`:

```python
ANTHROPIC_API_KEY: str = os.environ.get("ANTHROPIC_API_KEY", "")
```

**Step 4: Run tests to verify they pass**

Run: `pytest tests/unit/test_models.py tests/unit/test_config.py -v`
Expected: ALL PASS

**Step 5: Commit**

```bash
git add clew/models.py clew/config.py tests/unit/test_models.py tests/unit/test_config.py
git commit -m "feat: add NL description config fields"
```

---

### Task 5: Wire `DescriptionProvider` into factory and pipeline

**Files:**
- Modify: `clew/factory.py`
- Modify: `clew/indexer/pipeline.py`
- Test: `tests/unit/test_factory.py`
- Test: `tests/unit/test_pipeline.py` (or existing `test_indexer.py`)

**Step 1: Write the failing tests for factory**

Add to `tests/unit/test_factory.py`:

```python
def test_components_has_description_provider_none_by_default(monkeypatch, mock_env):
    """Description provider is None when NL descriptions are disabled."""
    components = create_components()
    assert components.description_provider is None


def test_components_has_description_provider_when_enabled(monkeypatch, mock_env):
    """Description provider is created when enabled and API key is set."""
    monkeypatch.setenv("ANTHROPIC_API_KEY", "test-key")
    # Need to provide a config with nl_description_enabled=True
    # This depends on how factory loads config — may need a config file fixture
    # or patch ProjectConfig
    ...
```

Note: The exact test structure depends on how the existing `test_factory.py` mocks dependencies. Follow the existing pattern — the factory tests likely patch `QdrantManager`, `create_embedding_provider`, etc.

**Step 2: Update `Components` dataclass and `create_components()`**

In `clew/factory.py`:

Add import:
```python
from clew.clients.description import AnthropicDescriptionProvider, DescriptionProvider
```

Add field to `Components`:
```python
@dataclass
class Components:
    """Container for all wired components."""

    config: ProjectConfig
    qdrant: QdrantManager
    embedder: EmbeddingProvider
    cache: CacheDB
    search_engine: SearchEngine
    indexing_pipeline: IndexingPipeline
    enhancer: QueryEnhancer | None
    reranker: RerankProvider | None
    description_provider: DescriptionProvider | None = None
```

Add to `create_components()`, before the indexing pipeline creation:

```python
# NL Description provider (optional)
description_provider: DescriptionProvider | None = None
if config.indexing.nl_description_enabled and env.ANTHROPIC_API_KEY:
    description_provider = AnthropicDescriptionProvider(
        api_key=env.ANTHROPIC_API_KEY,
        model=config.indexing.nl_description_model,
        max_concurrent=config.indexing.nl_description_max_concurrent,
    )

# Indexing
indexing_pipeline = IndexingPipeline(
    qdrant=qdrant,
    embedder=embedder,
    description_provider=description_provider,
    cache=cache,
)
```

Update the `return Components(...)` to include `description_provider=description_provider`.

**Step 3: Update `IndexingPipeline` constructor and `_embed_and_upsert`**

In `clew/indexer/pipeline.py`:

Update imports:
```python
if TYPE_CHECKING:
    from clew.clients.base import EmbeddingProvider
    from clew.clients.description import DescriptionProvider
    from clew.clients.qdrant import QdrantManager
    from clew.indexer.cache import CacheDB
```

Update constructor:
```python
def __init__(
    self,
    qdrant: QdrantManager,
    embedder: EmbeddingProvider,
    description_provider: DescriptionProvider | None = None,
    cache: CacheDB | None = None,
    batch_size: int = 100,
    max_tokens: int = 3000,
) -> None:
    self._qdrant = qdrant
    self._embedder = embedder
    self._description_provider = description_provider
    self._cache = cache
    self._batch_size = batch_size
    self._max_tokens = max_tokens
    self._parser = ASTParser()
```

Update `_embed_and_upsert` — after metadata extraction and before creating the payload, add description generation:

```python
async def _embed_and_upsert(self, chunks: list[Chunk], collection: str) -> None:
    """Embed a batch of chunks and upsert to Qdrant."""
    # Generate NL descriptions if provider available
    descriptions = await self._generate_descriptions(chunks)

    # Prepend descriptions to content for embedding
    texts = []
    for i, chunk in enumerate(chunks):
        desc = descriptions[i] if descriptions else None
        if desc:
            texts.append(f"# Description: {desc}\n\n{chunk.content}")
        else:
            texts.append(chunk.content)

    embeddings = await self._embedder.embed(texts, input_type="document")

    points: list[models.PointStruct] = []
    for i, (chunk, embedding) in enumerate(zip(chunks, embeddings)):
        sparse = text_to_sparse_vector(chunk.content)
        file_path_str = chunk.file_path

        # Extract metadata
        entity_type = chunk.metadata.get("entity_type", "section")
        qualified_name = chunk.metadata.get("qualified_name", "")
        app_name = detect_app_name(file_path_str)
        layer = classify_layer(file_path_str)
        signature = extract_signature(entity_type, chunk.content)
        chunk_id = build_chunk_id(
            file_path_str,
            entity_type,
            qualified_name,
            content=chunk.content,
        )

        payload = {
            "content": chunk.content,
            "chunk_id": chunk_id,
            "file_path": file_path_str,
            "language": _detect_language(file_path_str),
            "chunk_type": entity_type,
            "class_name": chunk.metadata.get("parent_class", ""),
            "function_name": chunk.metadata.get("name", ""),
            "signature": signature,
            "app_name": app_name,
            "layer": layer,
            "line_start": chunk.metadata.get("line_start", 0),
            "line_end": chunk.metadata.get("line_end", 0),
            "is_test": _is_test_file(file_path_str),
            "source_type": collection,
            "embedding_model": self._embedder.model_name,
            "indexed_at": datetime.now(tz=timezone.utc).isoformat(),
        }

        # Add NL description to payload if available
        desc = descriptions[i] if descriptions else None
        if desc:
            payload["nl_description"] = desc

        # Also add docstring if available in chunk metadata
        docstring = chunk.metadata.get("docstring")
        if docstring:
            payload["docstring"] = docstring

        point_id = str(uuid.uuid5(CHUNK_UUID_NAMESPACE, chunk_id))

        points.append(
            models.PointStruct(
                id=point_id,
                vector={
                    "dense": embedding,
                    "bm25": models.SparseVector(
                        indices=sparse.indices,
                        values=sparse.values,
                    ),
                },
                payload=payload,
            )
        )

    self._qdrant.upsert_points(collection, points)
```

Add the `_generate_descriptions` helper:

```python
async def _generate_descriptions(self, chunks: list[Chunk]) -> list[str | None] | None:
    """Generate NL descriptions for chunks that lack docstrings.

    Returns None if no description provider is configured.
    Returns a list parallel to chunks with descriptions or None per chunk.
    """
    if not self._description_provider:
        return None

    import hashlib

    results: list[str | None] = [None] * len(chunks)
    to_generate: list[tuple[int, dict[str, str]]] = []

    for i, chunk in enumerate(chunks):
        docstring = chunk.metadata.get("docstring")
        if docstring:
            # Already has docstring — no need to generate
            continue

        content_hash = hashlib.sha256(chunk.content.encode()).hexdigest()

        # Check cache first
        if self._cache:
            cached = self._cache.get_description(
                content_hash, self._description_provider.model_name
            )
            if cached:
                results[i] = cached
                continue

        to_generate.append((
            i,
            {
                "code": chunk.content,
                "language": _detect_language(chunk.file_path),
                "entity_type": chunk.metadata.get("entity_type", "section"),
                "name": chunk.metadata.get("qualified_name", chunk.metadata.get("name", "")),
                "_content_hash": content_hash,
            },
        ))

    if not to_generate:
        return results

    # Generate descriptions in batch
    items = [item for _, item in to_generate]
    descriptions = await self._description_provider.generate_batch(items)

    for (idx, item), desc in zip(to_generate, descriptions):
        results[idx] = desc
        # Cache successful descriptions
        if desc and self._cache:
            self._cache.set_description(
                item["_content_hash"],
                self._description_provider.model_name,
                desc,
            )

    return results
```

**Step 4: Write pipeline integration tests**

Add to existing pipeline tests (e.g., `tests/unit/test_pipeline.py` or wherever `IndexingPipeline` is tested):

```python
@pytest.mark.asyncio
async def test_pipeline_without_description_provider(mock_qdrant, mock_embedder):
    """Pipeline works normally without description provider."""
    pipeline = IndexingPipeline(qdrant=mock_qdrant, embedder=mock_embedder)
    # ... existing test patterns

@pytest.mark.asyncio
async def test_pipeline_with_description_provider(mock_qdrant, mock_embedder, tmp_path):
    """Pipeline generates descriptions and prepends to content."""
    mock_desc_provider = AsyncMock()
    mock_desc_provider.model_name = "test-model"
    mock_desc_provider.generate_batch.return_value = ["Validates email format."]
    cache = CacheDB(tmp_path)

    pipeline = IndexingPipeline(
        qdrant=mock_qdrant,
        embedder=mock_embedder,
        description_provider=mock_desc_provider,
        cache=cache,
    )
    # ... test that embed is called with prepended description
    # ... test that payload includes nl_description field

@pytest.mark.asyncio
async def test_pipeline_skips_description_when_docstring_exists(mock_qdrant, mock_embedder, tmp_path):
    """Pipeline skips description generation for chunks with docstrings."""
    mock_desc_provider = AsyncMock()
    mock_desc_provider.model_name = "test-model"
    cache = CacheDB(tmp_path)

    pipeline = IndexingPipeline(
        qdrant=mock_qdrant,
        embedder=mock_embedder,
        description_provider=mock_desc_provider,
        cache=cache,
    )
    # Create chunk with docstring in metadata
    # Verify generate_batch NOT called for that chunk

@pytest.mark.asyncio
async def test_pipeline_caches_descriptions(mock_qdrant, mock_embedder, tmp_path):
    """Generated descriptions are cached in SQLite."""
    mock_desc_provider = AsyncMock()
    mock_desc_provider.model_name = "test-model"
    mock_desc_provider.generate_batch.return_value = ["A description."]
    cache = CacheDB(tmp_path)

    pipeline = IndexingPipeline(
        qdrant=mock_qdrant,
        embedder=mock_embedder,
        description_provider=mock_desc_provider,
        cache=cache,
    )
    # ... index a chunk
    # ... verify cache.get_description returns the description
    # ... index same chunk again
    # ... verify generate_batch not called a second time (cache hit)
```

**Step 5: Run tests to verify they pass**

Run: `pytest tests/ -v --tb=short`
Expected: ALL PASS

**Step 6: Commit**

```bash
git add clew/factory.py clew/indexer/pipeline.py tests/unit/test_factory.py tests/unit/test_pipeline.py
git commit -m "feat: wire description provider into factory and indexing pipeline"
```

---

### Task 6: Propagate docstring from `CodeEntity` through chunk metadata

**Files:**
- Modify: `clew/chunker/fallback.py` (if `Chunk` metadata needs updating)
- Modify: `clew/chunker/strategies.py` (already done in Task 1)
- Modify: Wherever `CodeEntity` → `Chunk` conversion happens

**Context:** The `PythonChunker` yields `CodeEntity` with a `docstring` field (Task 1). But the `IndexingPipeline` receives `Chunk` objects from `split_file()`. We need to ensure the docstring flows through: `CodeEntity.docstring` → `Chunk.metadata["docstring"]`.

**Step 1: Trace the flow**

Read `clew/chunker/fallback.py` to find where `CodeEntity` is converted to `Chunk`. The `split_file()` function calls `PythonChunker.extract_entities()` and wraps results in `Chunk` objects. Find the exact conversion point.

**Step 2: Add docstring to Chunk metadata**

At the point where `CodeEntity` becomes a `Chunk`, add:

```python
metadata["docstring"] = entity.docstring
```

This should happen in the `split_file()` function in `fallback.py`, in the branch that handles AST-parsed entities.

**Step 3: Write a test**

```python
def test_split_file_preserves_docstring():
    source = '''def greet(name):
    """Say hello."""
    return f"Hello, {name}"
'''
    chunks = split_file("test.py", source, max_tokens=3000, parser=ASTParser())
    assert chunks[0].metadata.get("docstring") == "Say hello."
```

**Step 4: Run tests**

Run: `pytest tests/ -v --tb=short`
Expected: ALL PASS

**Step 5: Commit**

```bash
git add clew/chunker/fallback.py tests/unit/test_fallback.py
git commit -m "feat: propagate docstring from CodeEntity through chunk metadata"
```

---

### Task 7: Add `anthropic` dependency to `pyproject.toml`

**Files:**
- Modify: `pyproject.toml`

**Step 1: Add dependency**

Add `anthropic` to the project's dependencies in `pyproject.toml`. Since NL descriptions are optional, consider adding it as an optional dependency:

```toml
[project.optional-dependencies]
nl = ["anthropic>=0.40.0"]
```

Or if simpler, add to main dependencies — check how `httpx` and `voyageai` are listed and follow the same pattern.

**Step 2: Install**

Run: `pip install -e ".[nl]"` (or `pip install -e .` if added to main deps)

**Step 3: Verify import works**

Run: `python3 -c "from clew.clients.description import AnthropicDescriptionProvider; print('OK')"`
Expected: `OK`

**Step 4: Commit**

```bash
git add pyproject.toml
git commit -m "chore: add anthropic SDK dependency for NL descriptions"
```

---

### Task 8: Add `--nl-descriptions` flag to CLI index command

**Files:**
- Modify: `clew/cli.py`
- Test: `tests/unit/test_cli.py`

**Step 1: Write the failing test**

```python
def test_index_with_nl_descriptions_flag(runner, mock_components):
    """--nl-descriptions flag enables NL description generation."""
    result = runner.invoke(app, ["index", "/tmp/test", "--nl-descriptions"])
    assert result.exit_code == 0
    # Verify that create_components was called or that
    # indexing_pipeline received a description provider
```

**Step 2: Add the flag**

In `clew/cli.py`, add `--nl-descriptions` flag to the `index` command:

```python
@app.command()
def index(
    project_root: ...,
    full: ...,
    nl_descriptions: bool = typer.Option(
        False,
        "--nl-descriptions",
        help="Generate NL descriptions for code chunks without docstrings (requires ANTHROPIC_API_KEY)",
    ),
):
```

When `nl_descriptions=True`, the CLI should set `config.indexing.nl_description_enabled = True` before passing to the factory, or pass it as a parameter that overrides the YAML config.

**Step 3: Run tests**

Run: `pytest tests/unit/test_cli.py -v`
Expected: ALL PASS

**Step 4: Commit**

```bash
git add clew/cli.py tests/unit/test_cli.py
git commit -m "feat: add --nl-descriptions flag to CLI index command"
```

---

### Task 9: Update CLAUDE.md and docs

**Files:**
- Modify: `CLAUDE.md`
- Modify: `docs/IMPLEMENTATION.md`

**Step 1: Update CLAUDE.md**

- Add V1.1 status line under Project State
- Add `clients/description.py` to the module inventory
- Add note about `ANTHROPIC_API_KEY` env var
- Update module/test counts

**Step 2: Update IMPLEMENTATION.md**

- Mark V1.1 NL Descriptions as complete
- Document the cache schema addition
- Document the config fields

**Step 3: Commit**

```bash
git add CLAUDE.md docs/IMPLEMENTATION.md
git commit -m "docs: update CLAUDE.md and IMPLEMENTATION.md for V1.1"
```

---

### Task 10: Full verification

**Step 1: Run full test suite**

Run: `pytest --cov=clew -v`
Expected: All tests pass, coverage >= 90%

**Step 2: Run linters**

Run: `ruff check . && ruff format --check . && mypy clew/`
Expected: Clean

**Step 3: Final commit if any fixups needed**

```bash
git add -A && git commit -m "fix: linting and type checking fixups for V1.1"
```
