Mithril Providers — How to Add & Understand Providers

What it does

Takes a conversation history (list of messages with roles: system, user, assistant) and returns the model's complete response as a single string.

The flow (same for ALL providers)

1. Convert messages to the provider's native format
2. Build HTTP request with auth headers and JSON payload
3. POST to the provider's API endpoint
4. Parse response — extract the assistant's text from the provider-specific JSON
5. Return the text as a String

What differs per provider

What it does

Same as chat() but tokens arrive one by one as they're generated. Used by the TUI and API to show responses in real-time.

The flow

1. Same request building as chat(), but add "stream": true to the payload
2. Open SSE connection — server sends chunks as Server-Sent Events
3. For each chunk: parse the delta token, send it through the tx channel
4. Accumulate all tokens into the final complete response
5. Return the full text (the caller also got each token via the channel)

What differs per provider

• Gemini: Uses ?alt=sse query param and a different endpoint path (streamGenerateContent)
• OpenAI/Groq: Standard SSE with data: {json} lines. Delta in choices[0].delta.content
• Anthropic: SSE with event types: content_block_delta contains the token
• Local: Tokens come from llama.cpp one by one via callback (no HTTP, no SSE)

What it does

Sends messages + a list of available tools. The model can either respond with text OR request that specific tools be executed with specific arguments.

The flow

1. Convert tool definitions to the provider's native format (JSON Schema for most)
2. Add tools to the request payload alongside messages
3. POST to the provider
4. Parse response: check if it's a text reply or tool_calls
5. If tool_calls: return structured ToolCall {name, arguments} objects
6. If text: return the text response directly

What differs per provider

To add a new LLM provider (e.g. Mistral, Cohere, DeepSeek cloud), follow this template. Create a new file src/providers/your_provider.rs:

use anyhow::Result;
use async_trait::async_trait;
use crate::providers::{ChatMessage, ChatProvider, ChatResponse, ToolDefinition};
use tokio::sync::mpsc::Sender;

pub struct YourProvider {
    api_key: String,
    model: String,
    client: reqwest::Client,
}

impl YourProvider {
    pub fn new(api_key: String, model: &str) -> Self {
        Self {
            api_key,
            model: model.to_string(),
            client: reqwest::Client::new(),
        }
    }
}

#[async_trait]
impl ChatProvider for YourProvider {
    fn name(&self) -> &str { "your_provider" }
    fn model(&self) -> &str { &self.model }

    async fn chat(&self, messages: &[ChatMessage]) -> Result<String> {
        // 1. Convert messages to your API's format
        let payload = build_payload(messages);
        
        // 2. POST to your API
        let response = self.client
            .post("https://api.yourprovider.com/v1/chat")
            .bearer_auth(&self.api_key)
            .json(&payload)
            .send().await?
            .error_for_status()?
            .json::<YourResponse>().await?;
        
        // 3. Extract and return the assistant's text
        Ok(response.choices[0].message.content.clone())
    }

    async fn chat_stream(
        &self, messages: &[ChatMessage], tx: Sender<String>
    ) -> Result<String> {
        // Same as chat() but with "stream": true
        // Read SSE chunks, send each token via tx.send(token)
        // Return the full accumulated text
        todo!("Implement streaming")
    }

    async fn chat_with_tools(
        &self, messages: &[ChatMessage], tools: &[ToolDefinition]
    ) -> Result<ChatResponse> {
        // Same as chat() but include tool definitions in payload
        // Parse response: if tool_calls present, return them
        // Otherwise return text
        todo!("Implement tool calling")
    }

    async fn is_available(&self) -> bool {
        !self.api_key.is_empty()
    }
}

Then register it in the factory (src/providers/mod.rs):

// In create_provider_with_model():
"your_provider" => {
    let key = config.get_credential("your_provider")?
        .ok_or_else(|| anyhow!("API key not set"))?;
    Ok(Box::new(YourProvider::new(key, model)))
}

That's it. The orchestrator, API layer, CLI, and TUI all work automatically with your new provider. No other files need changes.

• HTTP boilerplate: Every cloud provider repeats: build_payload → client.post() → error_for_status() → parse. A shared helper could save ~40 lines per provider.
• Struct fields: api_key + model + client are identical across 4 providers. HttpProviderBase exists but isn't used yet.
• Groq is 50% larger than others (436 vs ~280 LOC) without extra features. Likely has verbose error handling that could be simplified.

These are tracked in the refactoring roadmap. The architecture is correct — the duplication is cosmetic and doesn't affect functionality or extensibility.