Metadata-Version: 2.4
Name: llm-neuralwatt
Version: 0.0.1
Summary: A plugin to add support for the Neuralwatt AI inference service to llm
Author: Chris Adams
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/mrchrisadams/llm-neuralwatt
Project-URL: Changelog, https://github.com/mrchrisadams/llm-neuralwatt/releases
Project-URL: Issues, https://github.com/mrchrisadams/llm-neuralwatt/issues
Project-URL: CI, https://github.com/mrchrisadams/llm-neuralwatt/actions
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: llm
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Dynamic: license-file

# llm-neuralwatt

[![PyPI](https://img.shields.io/pypi/v/llm-neuralwatt.svg)](https://pypi.org/project/llm-neuralwatt/)
[![Changelog](https://img.shields.io/github/v/release/mrchrisadams/llm-neuralwatt?include_prereleases&label=changelog)](https://github.com/mrchrisadams/llm-neuralwatt/releases)
[![Tests](https://github.com/mrchrisadams/llm-neuralwatt/actions/workflows/test.yml/badge.svg)](https://github.com/mrchrisadams/llm-neuralwatt/actions/workflows/test.yml)
[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/mrchrisadams/llm-neuralwatt/blob/main/LICENSE)

A plugin to add support for the OpenAI compatible Neuralwatt inference service, to run inference against various open weights models, and return the direct energy measurements in the response as well as logging it locally in llm's logs.db sqlite file.

## Installation

Install this plugin in the same environment as [LLM](https://llm.datasette.io/).
```bash
llm install llm-neuralwatt
```

## Usage

Once you have llm-neuralwatt installed, you should see new models available when you call `llm models`:

```
Neuralwatt: neuralwatt/deepseek-coder-33b-instruct (aliases: neuralwatt-deepseek-coder)
Neuralwatt: neuralwatt/gpt-oss-20b (aliases: neuralwatt-gpt-oss)
Neuralwatt: neuralwatt/Qwen3-Coder-480B-A35B-Instruct (aliases: neuralwatt-qwen3-coder)
```

You will need to set a key with 

```
llm keys set neuralwatt
```

You can sign up for Neuralwatt, and get an API key from https://portal.neuralwatt.com.

### Energy Consumption Logging

This plugin automatically captures and logs energy consumption data from Neuralwatt API responses. Energy data is stored in the `response_json` field of the llm logs database.

To view energy consumption for your requests:
```bash
# View recent logs with energy data
llm logs --model neuralwatt-gpt-oss --json | jq '.[-5:].response_json.energy'

# Query specific energy metrics
llm logs --model neuralwatt-deepseek-coder --json | jq -r '.[] | select(.response_json.energy != null) | "\(.datetime_utc): \(.response_json.energy.energy_joules) joules, \(.response_json.energy.energy_kwh) kWh"'
```

Each energy measurement includes:
- `energy_joules`: Energy consumption in joules
- `energy_kwh`: Energy consumption in kilowatt-hours  
- `avg_power_watts`: Average power consumption in watts
- `duration_seconds`: Duration of the API call
- `attribution_method`: How energy was attributed
- `attribution_ratio`: Ratio of energy attribution

You can read more about how energy consumption is attributed to a single use in the [Neuralwatt docs](https://portal.neuralwatt.com/docs/energy-methodology)

### Known issues with Streaming vs Non-Streaming Requests

**Important**: Energy consumption data is currently only available when using non-streaming responses (`--no-stream` flag). Due to how the OpenAI client library handles streaming responses, energy data chunks are filtered out during streaming.

In streaming mode, Neuralwatt sends energy data as a special chunk just before the `[DONE]` marker in streaming responses (more in the docs)(https://portal.neuralwatt.com/docs/guides/streaming), but the OpenAI client does not preserve these non-standard chunks.

To ensure energy data is captured:
```bash
# ✅ This will capture energy data
llm "Explain quantum computing" -m neuralwatt-gpt-oss --no-stream

# ❌ This will NOT capture energy data due to streaming limitation
llm "Explain quantum computing" -m neuralwatt-gpt-oss
```

We're investigating ways to work around this limitation in future versions of the plugin.

## Development

To set up this plugin locally, first checkout the code. Then create a new virtual environment:

```bash
cd llm-neuralwatt
python -m venv venv
source venv/bin/activate
```
Now install the dependencies and test dependencies:
```bash
python -m pip install -e '.[test]'
```
To run the tests:
```bash
python -m pytest
```
