Metadata-Version: 2.4
Name: vllm-mblt
Version: 0.3.0
Summary: An out-of-tree vLLM plugin for Mobilint NPU runtime integration.
Author-email: "Mobilint Inc." <tech-support@mobilint.com>
License-Expression: Apache-2.0
Project-URL: Home, https://www.mobilint.com/
Project-URL: Repository, https://github.com/mobilint/vllm-mblt
Keywords: vllm,mobilint,mblt,npu,inference,llm,vlm,aries
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: vllm<=0.11.2,>=0.11.2
Requires-Dist: mblt-model-zoo[transformers]>=2.3.0
Dynamic: license-file

# vLLM MBLT

<div align="center">

<a href="https://github.com/vllm-project/vllm" target="_blank">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/mobilint/vllm-mblt/refs/heads/main/assets/header-dark.png">
    <img src="https://raw.githubusercontent.com/mobilint/vllm-mblt/refs/heads/main/assets/header-light.png" alt="vLLM × Mobilint" width="720">
  </picture>
</a>

[![PyPI - Version](https://img.shields.io/pypi/v/vllm-mblt?logo=pypi)](https://pypi.org/project/vllm-mblt/)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/vllm-mblt?logo=python)](https://pypi.org/project/vllm-mblt/)
[![vLLM](https://img.shields.io/badge/vLLM-0.11.2-blue)](https://github.com/vllm-project/vllm)
[![Mobilint](https://img.shields.io/badge/Mobilint-NPU-green)](https://www.mobilint.com/)
[![PyPI Downloads](https://static.pepy.tech/badge/vllm-mblt?period=total&units=INTERNATIONAL_SYSTEM&left_color=BLACK&right_color=GREEN&left_text=downloads)](https://clickpy.clickhouse.com/dashboard/vllm-mblt)

</div>

**vllm-mblt** is an out-of-tree [vLLM](https://github.com/vllm-project/vllm) plugin that integrates
[Mobilint](https://www.mobilint.com/) NPU runtime support into the vLLM serving and benchmarking stack.

It provides a custom vLLM platform, worker, and model registry hooks so Mobilint-optimized LLM/VLM artifacts
can be served through familiar vLLM commands and OpenAI-compatible APIs.

## Highlights

- **Out-of-tree vLLM plugin**: registers the `mblt` platform without patching vLLM itself.
- **Mobilint NPU worker**: dispatches text-generation and multimodal execution to Mobilint runtime models.
- **Model registry integration**: supports Mobilint wrappers for Llama, HyperCLOVAX, EXAONE/EXAONE4, Qwen2/3,
  and Qwen2/3-VL families.
- **Runtime-aware scheduling**: reads model-configured `npu_prefill_chunk_size` and `max_batch_size` values to
  tune chunked prefill and scheduler concurrency automatically.
- **vLLM benchmark compatibility**: works with `vllm serve`, `vllm bench serve`, and `vllm bench throughput`.

## Requirements

- Python 3.10+
- `vllm==0.11.2`
- `mblt-model-zoo[transformers] >= 2.1.0`
- A Mobilint NPU environment. If you are not yet a Mobilint customer, please contact
  [tech-support@mobilint.com](mailto:tech-support@mobilint.com).

The package pins vLLM for compatibility:

```text
vllm>=0.11.2,<=0.11.2
```

## Installation

Install from PyPI:

```bash
pip install vllm-mblt
```

Or install the latest source checkout:

```bash
git clone https://github.com/mobilint/vllm-mblt.git
cd vllm-mblt
python -m venv .venv
source .venv/bin/activate
pip install -U pip
pip install -e .
```

## Quick Start

### 1. Verify Plugin Registration

After installation, run:

```bash
vllm --help
```

You should see plugin logs indicating that the Mobilint `mblt` platform plugin has been discovered and activated.

### 2. Serve a Text Model

```bash
vllm serve mobilint/Llama-3.2-1B-Instruct --trust-remote-code
```

Then query the OpenAI-compatible endpoint:

```bash
curl http://127.0.0.1:8000/v1/models
```

### 3. Serve a VLM Model

Qwen2-VL and Qwen3-VL Mobilint models can be loaded through the same vLLM server path:

```bash
vllm serve mobilint/Qwen2-VL-2B-Instruct --trust-remote-code
```

```bash
vllm serve mobilint/Qwen3-VL-2B-Instruct --trust-remote-code
```

Current Mobilint Qwen2/3-VL notes:

- The worker loads VLMs through `AutoModelForImageTextToText`.
- Image inputs are processed through vLLM's multimodal pipeline and merged into Mobilint language-model prompt
  embeddings inside the custom worker.
- The NPU path currently supports exactly one image in the initial multimodal request.
- Subsequent turns in the same session must be text-only or reuse the same image-token position.
- Video inputs are not supported by the current Mobilint Qwen2/3-VL NPU path.

## Runtime Tuning

### Runtime Layout Overrides

By default, `vllm-mblt` follows the runtime layout encoded in the Mobilint model artifact/config. Use
`--model-loader-extra-config` only when you intentionally want to override runtime placement or testing knobs.

Runtime settings such as `dev_no`, `target_cores`, `target_clusters`, `core_mode`, and `max_batch_size` can be
provided through `--model-loader-extra-config`.
For detailed `core_mode` and multicore runtime layout guidance, see the
[Mobilint multicore documentation](https://docs.mobilint.com/v1.2/en/multicore.html).

```bash
vllm serve mobilint/Llama-3.2-1B-Instruct \
  --trust-remote-code \
  --model-loader-extra-config '{"dev_no": 0, "target_cores": ["1:0"]}'
```

For VLMs such as `mobilint/Qwen3-VL-2B-Instruct`, shared runtime layout keys are applied to both Mobilint
submodules by forwarding them as model-zoo VLM subconfig keys (`vision_*` and `text_*`). Use explicit prefixed
keys when the vision encoder and text model need different placement:

```bash
vllm serve mobilint/Qwen3-VL-2B-Instruct \
  --trust-remote-code \
  --model-loader-extra-config '{"dev_no": 0, "core_mode": "global4", "vision_core_mode": "single"}'
```

For VLM-specific MXQ path overrides, use `vision_mxq_path` and/or `text_mxq_path`; a single top-level `mxq_path`
is only meaningful for single-module text models.

### Chunked Prefill Auto-Tuning

If a model config includes `npu_prefill_chunk_size`, `vllm-mblt` uses it to tune vLLM chunked prefill.

- Integer values are used directly.
- Dict values are selected by `core_mode`.
- `core_mode` is resolved from `--model-loader-extra-config` first, then from the model config default.
- The selected value is applied to vLLM's `max_num_batched_tokens` for chunked prefill.
- A `default` (or `DEFAULT`) key is used when the resolved `core_mode` has no entry.
- `core_mode: "auto"` is never a dict key. When the resolved `core_mode` is `"auto"` (or is absent) and the
  dict holds exactly one usable entry, that entry is used — a global-scheme batch mxq ships only the mode it
  was compiled for. Multiple entries under `"auto"` are not guessed.
- If no matching value is found, `vllm-mblt` falls back to `128`.
- For batch-compiled models with `max_batch_size > 1`, the effective chunked prefill limit is clamped to `128`
  to match the qbruntime batch execution limit used by the worker.

Example model config:

```json
{
  "npu_prefill_chunk_size": {
    "single": 64,
    "global4": 256,
    "global8": 512
  }
}
```

With this command, `vllm-mblt` selects `256` for `global4`:

```bash
vllm serve mobilint/YourModel \
  --trust-remote-code \
  --model-loader-extra-config '{"dev_no": 0, "core_mode": "global4", "target_clusters": [0]}'
```

If you also pass `--max-num-batched-tokens`, the effective value becomes the smaller of the user-provided value
and the model-configured `npu_prefill_chunk_size`.

Use `--block-size` only when you intentionally want to override the model-configured/default block size:

```bash
vllm serve mobilint/Llama-3.2-1B-Instruct \
  --trust-remote-code \
  --block-size 64
```

### Model-Configured Batch Capacity

If a model config includes `max_batch_size`, `vllm-mblt` uses that value to support batch-compiled Mobilint models.

- The worker uses `max_batch_size` for KV cache memory sizing.
- The platform applies it to vLLM `max_num_seqs` automatically.
- You do not need to pass `--max-num-seqs` unless you intentionally want a smaller scheduler cap.
- `max_batch_size` also supports the same `core_mode` keyed dict form as `npu_prefill_chunk_size`.
- For local testing, `--model-loader-extra-config '{"max_batch_size": 32}'` overrides the model config value.

Example:

```bash
vllm serve mobilint/Llama-3.2-1B-Instruct-Batch32 --trust-remote-code
```

For batch-compiled MXQs such as `mobilint/Llama-3.2-1B-Instruct-Batch32`, the plugin also caps the effective
chunked prefill limit to `128`, even when the model config advertises a larger `npu_prefill_chunk_size`.

## Benchmarking

This repository includes `sonnet.txt`, which can be used with vLLM benchmark commands.

### Serve Benchmark

Terminal 1:

```bash
vllm serve --model mobilint/Llama-3.2-1B-Instruct --trust-remote-code
```

Terminal 2:

```bash
vllm bench serve --model mobilint/Llama-3.2-1B-Instruct \
  --trust-remote-code \
  --port 8000 \
  --num-warmups 1 \
  --dataset-name sonnet \
  --dataset-path sonnet.txt \
  --num-prompts 10
```

### Throughput Benchmark

```bash
vllm bench throughput --model mobilint/Llama-3.2-1B-Instruct \
  --trust-remote-code \
  --dataset-name sonnet \
  --dataset-path sonnet.txt \
  --num-prompts 10
```

Notes:

- `vllm bench serve` uses a separate server process; `vllm bench throughput` runs the engine directly.
- `vllm bench serve --max-concurrency` is a benchmark client load setting, not the server-side scheduler limit.
- Reported latency and throughput are environment-dependent. Capture results from your target board for documentation
  or performance comparisons.

## NPU Event Tracing

The worker can record NPU activity as a Chrome Tracing log through qbruntime's
event tracer, so you can see where time actually goes on the accelerator. It is
wired into vLLM's standard worker profiler hook, which means the usual controls
apply -- there is no MBLT-specific flag or endpoint.

Set `VLLM_TORCH_PROFILER_DIR` to a directory before starting the server. This is
the switch: without it the OpenAI server does not register the profile routes,
and the worker reports that tracing is not enabled.

```bash
export VLLM_TORCH_PROFILER_DIR=/tmp/mblt_traces
vllm serve mobilint/Llama-3.2-1B-Instruct --trust-remote-code
```

Then bracket the work you care about:

```bash
curl -X POST http://localhost:8000/start_profile
# send the requests you want to trace
curl -X POST http://localhost:8000/stop_profile
```

For an offline run, `LLM.start_profile()` / `LLM.stop_profile()` do the same, and
`vllm bench serve --profile` brackets the benchmark for you.

Each window writes `{hostname}_{pid}.mblt_npu_rank{rank}.{time_ns}.json` into
that directory. Open it at <https://ui.perfetto.dev/>. The name follows the
convention `torch.profiler.tensorboard_trace_handler` uses for its own traces,
which vLLM and vllm-ascend both build on: the nanosecond timestamp is what
keeps successive windows from clashing, the pid separates concurrent processes
on a host, and the hostname separates containers that share a mounted trace
directory but not a pid namespace. vLLM's front-end trace lands beside it as
`{hostname}_{pid}.async_llm.{time_ns}.pt.trace.json.gz`. The events are the runtime's own device-level
spans -- `infer`, `run npu`, `copy to npu`, `lock core`, `read device` and the
like -- so a window shows what each inference step spent on the accelerator.

vLLM writes its own front-end CPU trace (`*.async_llm.*.pt.trace.json.gz`) into
the same directory, because `VLLM_TORCH_PROFILER_DIR` also enables the API
server's `AsyncLLM` profiler. The two are complementary: that file covers
CPU-side scheduling, the MBLT file covers NPU execution.

Notes:

- Trace a short window. qbruntime buffers the whole log in the process and
  writes it only when tracing stops, so leaving a trace on for the life of a
  server grows memory and produces a file too large to be useful. If the worker
  shuts down while a trace is running it is stopped first so the window is not
  lost.
- Only one qbruntime trace can record per process. A start while one is
  already recording is refused with a warning rather than cutting the first
  window short. qbruntime has no way to report whether a trace is running or
  who owns it, so this covers traces started through this plugin and, when its
  module is already imported, through `mblt_model_zoo`'s benchmark helpers. A
  trace started by any other client cannot be detected.
- If a trace cannot be started, or its log cannot be written, `/start_profile`
  and `/stop_profile` fail rather than reporting success for a trace that will
  not be on disk. A repeated start or a stop with nothing running is not a
  failure and only logs. During shutdown a trace that cannot be written is
  logged and skipped so the rest of the teardown still runs.
- A `/stop_profile` with no trace running answers `500`. That comes from vLLM's
  front-end `AsyncLLM` profiler, which raises when stopped before it was
  started; the worker-side trace is unaffected and only logs that there was
  nothing to stop.
- A torch profiler on the worker would show nothing useful here: the model runs
  on the NPU through qbruntime rather than through torch ops, which is why this
  hook records a qbruntime trace instead.

## Supported Model Families

`vllm-mblt` registers Mobilint model wrappers for:

| Family | Registry class |
| --- | --- |
| Llama / HyperCLOVAX-compatible text models | `MobilintLlamaForCausalLM` |
| EXAONE | `MobilintExaoneForCausalLM` |
| EXAONE4 | `MobilintExaone4ForCausalLM` |
| Qwen2 | `MobilintQwen2ForCausalLM` |
| Qwen3 | `MobilintQwen3ForCausalLM` |
| Qwen2-VL | `MobilintQwen2VLForConditionalGeneration` |
| Qwen3-VL | `MobilintQwen3VLForConditionalGeneration` |

Model artifacts are available through Mobilint model repositories such as the
[Mobilint Hugging Face Hub](https://huggingface.co/mobilint).

## Cache Behavior

`MbltWorker` uses snapshot-based KV cache reuse with these policies:

- Event-driven dump, not every step.
- Reuse live cache for same-request continuous decode.
- Keep finished-session snapshots for prefix reuse.
- Evict finished snapshots with an LRU cap of 16 sessions.
- Load matched snapshots only when the worker-side cost model expects the
  one-cache-id load to beat recomputing the matched prefix.

The prefix-cache load threshold is enabled by default. During model warmup the
worker tries to measure optimistic one-cache-id prefill costs for 1, 2, 4, and
8 KV blocks; for batch MXQs this still submits exactly one active `cache_id`.
Real snapshot loads and dumps update per-`cache_id` EWMA timings. A snapshot is
loaded only when `load_ms < prefill_ms * 0.9`; otherwise the snapshot remains
stored and the prompt is recomputed. The policy can be adjusted with environment
variables or equivalent model loader extra config keys:

- `VLLM_MBLT_PREFIX_CACHE_AUTO_THRESHOLD` / `prefix_cache_auto_threshold`
  defaults to enabled. Set `0` to disable measured thresholding.
- `VLLM_MBLT_PREFIX_CACHE_MIN_HIT_TOKENS` / `prefix_cache_min_hit_tokens`
  sets a manual minimum matched-token count before any snapshot load.
- `VLLM_MBLT_PREFIX_CACHE_LOAD_MARGIN` / `prefix_cache_load_margin` defaults
  to `0.9`.
- `VLLM_MBLT_PREFIX_CACHE_CALIBRATE` defaults to enabled. Set `0` to skip
  startup prefill calibration.

The worker also tracks how many tokens each live runtime cache (or batch
`cache_id` slot) actually holds. A request may continue from a live cache
without reloading only when that count matches the scheduler's
`num_computed_tokens`. On a mismatch the worker logs a warning naming the
request and `cache_id`, drops the ownership claim, and rebuilds the prefix
instead of decoding against another sequence's KV. Seeing
`MBLT runtime cache token count holds fewer tokens than the scheduler expects` in the log
means measurements taken from that server should be re-checked.

### Sampling Penalties

`frequency_penalty`, `presence_penalty`, and `repetition_penalty` are applied
by default on the CPU-hosted MBLT sampler. Set
`VLLM_MBLT_ENABLE_SAMPLING_PENALTIES=0` to ignore them instead; the worker then
logs once which penalties it dropped. Released MBLT packages ship
`repetition_penalty` in `generation_config.json`, so ignoring them makes NPU
output generated under different sampling than a GPU reference run.

VLM prefix caching currently covers the language-model KV cache only. The
worker may load a compatible LM KV prefix snapshot and run only the uncached
text/embedding suffix. Image/video feature extraction is not cached by this
layer: image requests still rebuild vision features through the model's
multimodal feature hooks before the LM prefill/decode step.

Batch-compiled VLM text backends (`max_batch_size > 1`) are supported for
Mobilint Qwen2-VL and Qwen3-VL model types. With `mblt-model-zoo>=2.3.0`,
Qwen3-VL dynamic-vision Batch16 artifacts such as
`mobilint/Qwen3-VL-8B-Instruct-Batch16` forward packed text embeddings plus
the matching packed RoPE and deepstack tensors to the 3-input text MXQ.
Unsupported multimodal model types fail before runtime inference with a clear
error.

Implementation file: [`vllm_mblt/mblt_worker.py`](vllm_mblt/mblt_worker.py)

## Tests

```bash
python -m pytest tests
```

## Project Structure

```text
vllm_mblt/
├── __init__.py                 # vLLM plugin and model registration entry points
├── mblt_platform.py            # platform config overrides and runtime-aware defaults
├── mblt_worker.py              # custom worker, prefill/decode flow, KV snapshot logic
├── tracing.py                  # qbruntime NPU event tracing behind vLLM's profiler hook
└── models/                     # Mobilint model wrappers for LLM/VLM families

tests/
├── test_kv_cache_swap_spec.py
├── test_mblt_platform_prefill.py
├── test_mblt_tracing.py
└── test_mblt_worker_optimizations.py
```
