moonshotai/Kimi-K2.6

Overview

Kimi K2.6 is an open-source, native multimodal agentic model built through continual pretraining on approximately 15 trillion mixed visual and text tokens atop Kimi-K2-Base. It seamlessly integrates vision and language understanding with advanced agentic capabilities, instant and thinking modes, as well as conversational and agentic paradigms.

Prerequisites

• vLLM version: >= 0.25.0 nightly for the optimized B300 EAGLE3 and native CPU KV offload path documented below
• Hardware (INT4): 8x H200 GPUs (verified), or equivalent aggregate VRAM (~640 GB)
• Hardware (NVFP4): 4x Blackwell GPUs; the optimized B300 path below was verified on vllm/vllm-openai:nightly-09663abde0f50944a8d5ea30120666024b503faa
• AMD support: 8x MI300X / MI325X / MI355X with ROCm 7.2.1 and Python 3.12

NVIDIA B300: NVFP4 with Eagle3

The following text-only TP4 command mirrors the B300 configuration validated by InferenceX PR #2158. It uses the Kimi K2.6 Eagle3 MLA draft, TOKENSPEED_MLA attention, TRT-LLM ragged MLA prefill, FP8 KV cache, and full-and-piecewise CUDA graphs.

export VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm

vllm serve nvidia/Kimi-K2.6-NVFP4 \
  --tensor-parallel-size 4 \
  --trust-remote-code \
  --language-model-only \
  --kv-cache-dtype fp8 \
  --block-size 64 \
  --gpu-memory-utilization 0.90 \
  --attention-backend TOKENSPEED_MLA \
  --attention-config '{"mla_prefill_backend":"TRTLLM_RAGGED","use_prefill_query_quantization":true}' \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' \
  --max-cudagraph-capture-size 2048 \
  --max-num-batched-tokens 16384 \
  --stream-interval 10 \
  --enable-prefix-caching \
  --speculative-config '{"method":"eagle3","model":"lightseekorg/kimi-k2.6-eagle3-mla","num_speculative_tokens":4,"rejection_sample_method":"synthetic","synthetic_acceptance_length":3.24}'

Native CPU KV offload

SimpleCPUOffloadConnector extends the prefix cache into host DRAM. The feature toggle uses a conservative 8 GiB starter capacity. Size cpu_bytes_to_use for the host and divide the aggregate budget across TP ranks. The verified B300 TP4 run used 1,199 GiB total (299.75 GiB per rank):

export VLLM_USE_SIMPLE_KV_OFFLOAD=1
CPU_OFFLOAD_BYTES=$((1199 * 1024 * 1024 * 1024))

vllm serve nvidia/Kimi-K2.6-NVFP4 \
  --tensor-parallel-size 4 \
  --trust-remote-code \
  --language-model-only \
  --kv-cache-dtype fp8 \
  --block-size 64 \
  --gpu-memory-utilization 0.90 \
  --attention-backend TOKENSPEED_MLA \
  --attention-config '{"mla_prefill_backend":"TRTLLM_RAGGED","use_prefill_query_quantization":true}' \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' \
  --max-cudagraph-capture-size 2048 \
  --max-num-batched-tokens 16384 \
  --stream-interval 10 \
  --enable-prefix-caching \
  --speculative-config '{"method":"eagle3","model":"lightseekorg/kimi-k2.6-eagle3-mla","num_speculative_tokens":4,"rejection_sample_method":"synthetic","synthetic_acceptance_length":3.24}' \
  --disable-hybrid-kv-cache-manager \
  --kv-transfer-config "{\"kv_connector\":\"SimpleCPUOffloadConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"cpu_bytes_to_use\":${CPU_OFFLOAD_BYTES},\"lazy_offload\":false}}"

Decode context parallelism

For higher concurrency, TP4/DCP4 was validated both with and without native CPU KV offload. DCP is intentionally guide-only rather than exposed as a command-builder option. Do not combine DCP with the Eagle3/TOKENSPEED_MLA flags above until vLLM PR #48180 lands. For the current pinned image, remove --attention-backend TOKENSPEED_MLA and --speculative-config, then add:

--decode-context-parallel-size 4

The successful agentic sweep covered these B300 points:

Serving path	Parallelism	Native CPU KV offload	Tested concurrency
Eagle3	TP8	No	1
Eagle3	TP4	No	2, 4, 8
Eagle3	TP4	Yes	8, 16, 32
DCP	TP4/DCP4	No	32, 64, 128
DCP	TP4/DCP4	Yes	64, 128, 256

AMD MI300X/MI325X

On 8x MI300X or MI325X (gfx942), use the standard W4A16 MoE path with AITER and INT4 QuickReduce.

export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4

vllm serve moonshotai/Kimi-K2.6 \
  --host 0.0.0.0 \
  --port 8000 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --tool-call-parser kimi_k2 \
  --enable-auto-tool-choice \
  --reasoning-parser kimi_k2 \
  --mm-encoder-tp-mode data

AMD MI350X/MI355X

On 8x MI350X or MI355X (gfx950), add --moe-backend flydsl to use the optimized FlyDSL W4A16 MoE kernel. Keep LoRA disabled for this path.

export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4

vllm serve moonshotai/Kimi-K2.6 \
  --tensor-parallel-size 8 \
  --trust-remote-code \
  --mm-encoder-tp-mode data \
  --moe-backend flydsl \
  --compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}'

Notes:

• The FlyDSL INT4 MoE path does not support expert parallelism; do not add --enable-expert-parallel.
• Keep --compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}'; it is required for this FlyDSL path on MI350X / MI355X.
• vLLM has tuned MI350X/MI355X FlyDSL configs for this Kimi shape at TP=8 and TP=4.
• Keep vLLM's default block size unless you are tuning long-context throughput; --block-size 64 is safe to try.

Client Usage

Once the vLLM server is running, consume it via the OpenAI-compatible API:

import time
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:8000/v1",
    timeout=3600
)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
                }
            },
            {
                "type": "text",
                "text": "Read all the text in the image."
            }
        ]
    }
]

start = time.time()
response = client.chat.completions.create(
    model="moonshotai/Kimi-K2.6",
    messages=messages,
    max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")

Troubleshooting

• OOM errors: Lower --gpu-memory-utilization or adjust TP/EP to match your GPU count.
• Vision encoder performance: Use --mm-encoder-tp-mode data to run the vision encoder in data-parallel mode. The encoder is small, so TP adds communication overhead with little gain.
• Unique multimodal inputs: Pass --mm-processor-cache-gb 0 to avoid caching overhead. For repeated inputs, --mm-processor-cache-type shm uses host shared memory for better performance at high TP settings.
• MoE kernel tuning: Use the benchmark_moe script from vLLM to tune Triton kernels for your specific hardware.
• Async scheduling: Enabled by default for better throughput. Disable if you encounter issues, and file a bug report to vLLM.
• Eagle3 with DCP: The current pinned image does not support the combination. Disable Eagle3/TOKENSPEED_MLA for DCP until vLLM PR #48180 is merged and available in the image.

References

• Kimi-K2.6 on Hugging Face
• NVIDIA Kimi-K2.6-NVFP4 on Hugging Face
• InferenceX Kimi-K2.6 B300 agentic sweep
• vLLM SimpleCPUOffloadConnector
• vLLM DCP + Eagle support PR
• vLLM multimodal inputs guide
• vLLM Expert Parallelism docs
• vLLM NixlConnector usage guide