liteinfer benchmarks
Each config is scored against the one it improves on (vs base), against the start of its lineage (vs first), and against vLLM at the same batch width (vs vLLM). Above 1.00x is better. Each row names the config it improves on; vLLM rows are references, not competitors. Bars show standing on the headline metric of each table.
model Llama-3.2-1B-InstructISL 128OSL 256prompts 50 / 200decoding greedy, forced lengthlast run 2026-08-29T19:12:54+00:00
latency · Llama-3.2-1B-Instruct · ISL=128 OSL=256
| config | ITL p50 | ITL p95 | TTFT p50 | TTFT p95 | E2E p50 | vs base | vs first | vs vLLM |
|---|---|---|---|---|---|---|---|---|
| liteinfer-nocacheNo KV cache: every step re-feeds the whole sequence · reference point | 16.5 | 16.5 | 14.8 | 16.1 | 4,215.6 | — | — | 0.32x |
| liteinfer-eagerKV cache via transformers DynamicCache · improves on liteinfer-nocache | 13.7 | 13.9 | 15.7 | 16.3 | 3,521.2 | 1.20x | 1.20x | 0.38x |
| liteinfer-native-eagerKV cache as plain tensors, no DynamicCache · improves on liteinfer-eager | 13.7 | 13.9 | 14.8 | 16.1 | 3,518.5 | 1.00x | 1.20x | 0.38x |
| liteinfer-pagedPaged KV cache: fixed-size blocks from a pool · improves on liteinfer-native-eager | 15.3 | 15.3 | 19.3 | 21.1 | 3,910.7 | 0.90x | 1.08x | 0.34x |
| liteinfer-eager-b4Static batching, B=4, eager cache · improves on liteinfer-eager | 14.0 | 14.1 | 15.8 | 17.7 | 3,596.6 | 0.98x | 1.17x | 0.37x |
| liteinfer-native-eager-b4Static batching, B=4, native eager cache · improves on liteinfer-native-eager | 13.9 | 13.9 | 14.9 | 16.7 | 3,561.1 | 0.99x | 1.18x | 0.38x |
| liteinfer-paged-b4Static batching, B=4, paged cache · improves on liteinfer-paged | 15.0 | 15.0 | 19.0 | 20.9 | 3,832.2 | 1.02x | 1.10x | 0.35x |
| liteinfer-continuousContinuous batching, up to 32 concurrent sequences · improves on liteinfer-paged-b4 | 14.9 | 14.9 | 19.0 | 21.0 | 3,807.6 | 1.01x | 1.11x | 0.35x |
| vllmvLLM, one sequence at a time · reference point | 5.2 | 5.2 | 22.7 | 27.4 | 1,354.8 | — | — | — |
| vllm-b4vLLM, up to 4 concurrent sequences · reference point | 5.2 | 5.2 | 22.6 | 27.0 | 1,353.9 | — | — | — |
| vllm-continuousvLLM, up to 32 concurrent sequences · reference point | 5.2 | 5.2 | 23.2 | 32.2 | 1,356.2 | — | — | — |
throughput · Llama-3.2-1B-Instruct · ISL=128 OSL=256
| config | tok/s | req/s | wall (s) | vs base | vs first | vs vLLM |
|---|---|---|---|---|---|---|
| liteinfer-nocacheNo KV cache: every step re-feeds the whole sequence · reference point | 60.5 | 0.2 | 846.7 | — | — | 0.32x |
| liteinfer-eagerKV cache via transformers DynamicCache · improves on liteinfer-nocache | 73.4 | 0.3 | 697.9 | 1.21x | 1.21x | 0.39x |
| liteinfer-native-eagerKV cache as plain tensors, no DynamicCache · improves on liteinfer-eager | 72.7 | 0.3 | 704.6 | 0.99x | 1.20x | 0.39x |
| liteinfer-pagedPaged KV cache: fixed-size blocks from a pool · improves on liteinfer-native-eager | 66.8 | 0.3 | 766.7 | 0.92x | 1.10x | 0.35x |
| liteinfer-eager-b4Static batching, B=4, eager cache · improves on liteinfer-eager | 281.6 | 1.1 | 181.8 | 3.84x | 4.66x | 0.39x |
| liteinfer-native-eager-b4Static batching, B=4, native eager cache · improves on liteinfer-native-eager | 277.6 | 1.1 | 184.4 | 3.82x | 4.59x | 0.38x |
| liteinfer-paged-b4Static batching, B=4, paged cache · improves on liteinfer-paged | 252.9 | 1.0 | 202.4 | 3.79x | 4.18x | 0.35x |
| liteinfer-continuousContinuous batching, up to 32 concurrent sequences · improves on liteinfer-paged-b4 | 1,268.3 | 5.0 | 40.4 | 5.01x | 20.98x | 0.28x |
| vllmvLLM, one sequence at a time · reference point | 188.4 | 0.7 | 271.8 | — | — | — |
| vllm-b4vLLM, up to 4 concurrent sequences · reference point | 724.0 | 2.8 | 70.7 | — | — | — |
| vllm-continuousvLLM, up to 32 concurrent sequences · reference point | 4,466.6 | 17.4 | 11.5 | — | — | — |