In the previous chapter, we established a layered mental model of vLLM v1, knowing that a request starts from the API Server, passes through EngineCore, and finally reaches the Worker for execution. But how does a JSON string in an HTTP request body become an object inside the engine that can be scheduled, tracked, and interrupted? This is the question the Request class must answer.

The specification system of KV Cache: from KVCacheSpec to the registry

Request solves the problem of "who wants to compute," whileKVCacheSpecsolves the problem of "where to compute." In the world of PagedAttention, the KV cache of each model layer needs to be precisely described: how many heads it has, how large each head is, how many tokens a block can store, and whether quantization is needed. This information is encoded inKVCacheSpec's inheritance system.

Intuitive model: KVCacheSpec is the "floor plan" of GPU memory

Without this specification system, KV cache allocation could only rely on hardcoded assumptions and could not support the diverse model requirements from standard MHA to MLA, from full attention to sliding window, and from FP16 to FP8 quantization.

Data structure: the inheritance tree and key fields of KVCacheSpec

KVCacheSpecis the base class of all specs, and it is a@dataclass(frozen=True) 📎 vllm/v1/kv_cache_interface.py:150-152. frozen means that once a spec object is created, it is immutable—this ensures that multiple components (scheduler, Worker, KV Cache Manager) see the same spec and that inconsistency is not caused by modification somewhere.

The base class defines three abstract properties that must be implemented by subclasses:num_heads、tokens_per_state、state_content_size_bytes 📎 vllm/v1/kv_cache_interface.py:182-183. Together, these three properties determinepage_size_bytes—that is, the number of bytes occupied by one block.

AttentionSpecis the most core subclass, and it introducesnum_kv_heads、head_size、dtype、kv_quant_modeand other fields📎 vllm/v1/kv_cache_interface.py:485-498. Among them, thetokens_per_statefield is especially ingenious in design: the default value is 1, meaning one state corresponds to one token; but it can be set to an integer greater than 1 (such as DeepSeek-V4's sparse MLA compressing multiple tokens into one state), or to a fraction less than 1 (such as Whisper's block pooling usingFraction(1, block_pool_size)to indicate that one token corresponds to multiple states)📎 vllm/v1/kv_cache_interface.py:501-501。

FullAttentionSpecOn the basis ofAttentionSpec,sliding_windowandattention_chunk_size 📎 vllm/v1/kv_cache_interface.py:566-566are added. Note that its docstring explains an important design decision: when the hybrid allocator is disabled, sliding window attention layers are treated as full attention in the KV Cache Manager (allocating blocks for all tokens), but at model runtime they are still computed as sliding window📎 vllm/v1/kv_cache_interface.py:540-545. This is aconservative allocation, precise computationstrategy.

MLAAttentionSpecis a key spec for the DeepSeek series of models. It setshead_size_vto 0 by default📎 vllm/v1/kv_cache_interface.py:670, because MLA stores only one latent vector and has no independent V.alignmentThe field is used for page alignment padding📎 vllm/v1/kv_cache_interface.py:646-652, which is crucial for backends such as FlashMLA that require specific alignment.

MambaSpecdoes not follow the attention route at all. It usesshapesanddtypestuples to describe the shape of the state tensor📎 vllm/v1/kv_cache_interface.py:1027-1028，state_content_size_bytesis the sum of all state tensor sizes📎 vllm/v1/kv_cache_interface.py:1048-1052. Mamba'smax_memory_usage_byteshas three different calculation methods depending onmamba_cache_mode📎 vllm/v1/kv_cache_interface.py:1073-1084, which reflects the complexity of Mamba state management—it does not grow linearly like attention, but has a fixed state size.

Scenario-driven: conversion from specs to memory layout

When the engine starts, it needs to convert theKVCacheSpecof all layers into the actual memory layout. This process is completed byKVCacheTensorandcreate_kv_cache_views.

KVCacheTensordescribes the position of a group of same-shaped layers in KV cache allocation📎 vllm/v1/kv_cache_interface.py:1406-1427. Its core fields arelayer_strideandblock_stride: the former is the byte distance between adjacent layers, and the latter is the byte distance between adjacent blocks. The docstring explains in detail two layout modes: layer-outermost layout gives each layer a contiguous region, and block-outermost layout makes each block contain the pages of all layers📎 vllm/v1/kv_cache_interface.py:1416-1416。

flowchart LR
    subgraph spec["KVCacheSpec 层"]
        fas["FullAttentionSpec<br/>num_kv_heads=32<br/>head_size=128<br/>block_size=16"]
    end
    subgraph tensor["KVCacheTensor 层"]
        kt["KVCacheTensor<br/>size=2GB<br/>layer_stride=page*num_blocks<br/>block_stride=page"]
    end
    subgraph view["torch.Tensor 视图"]
        v1["layer_0: [B, H, N, C]"]
        v2["layer_1: [B, H, N, C]"]
        v3["layer_N: [B, H, N, C]"]
    end
    fas -->|"compute_layer_kv_cache_shape_bytes()"| kt
    kt -->|"create_kv_cache_views()"| v1
    kt -->|"create_kv_cache_views()"| v2
    kt -->|"create_kv_cache_views()"| v3

create_kv_cache_viewsThe function is the core of this process📎 vllm/v1/kv_cache_interface.py:353-417. It receives a flat int8 buffer and, throughtorch.as_strided, creates a 4D view for each layer[B, H, N, C]. The key parameter isstrides, which is calculated bycompute_layout_strides📎 vllm/v1/kv_cache_interface.py:314-350. This function follows the dimension order specified bylayout.stride_orderand computes the byte stride of each dimension in reverse starting from the innermost dimension.

There is a noteworthy boundary check here: when kernel_block_size is smaller than spec.block_size (that is, one manager block is split into multiple kernel blocks), the code verifies whether block_stride is equal to dense_page_size📎 vllm/v1/kv_cache_interface.py:381-382. If not, it indicates that there is padding in the layout and it cannot be evenly split, and a ValueError with a clear fix suggestion is thrown.

Design thinking: registry pattern and extensibility

KVCacheSpecRegistryis a key design for vLLM extensibility📎 vllm/v1/kv_cache_spec_registry.py:39-40. It maintains two global dictionaries:_REGISTRY_KVCACHESPEC_LISTstores the mapping from spec classes to metadata,_REGISTRY_ROLE_MANAGERSstores the mapping from roles to managers📎 vllm/v1/kv_cache_spec_registry.py:35-36。

get_manager_classThe method demonstrates the core lookup logic of the registry: it traverses upward along the spec class's MRO (method resolution order) and finds the first registered base class📎 vllm/v1/kv_cache_spec_registry.py:129-130. This means that a customCustomFullAttentionSpec, if not registered separately, will automatically inherit the manager ofFullAttentionSpec. Thisinheritance-based lookupmakes it possible to register only the differing parts when adding a new spec type.

check_kv_cache_spec_registryThe method verifies at startup that the specs of all layers are registered📎 vllm/v1/kv_cache_spec_registry.py:165-174. Note that it usesraise ValueErrorinstead ofassert, and the comment explicitly states that this is to also take effect in production environments📎 vllm/v1/kv_cache_spec_registry.py:165-174. This is an important engineering decision: Python's-Oflag removes asserts, but configuration errors in production environments must be exposed at startup rather than crashing only at runtime.

Chapter Summary

This chapter analyzed two core data structures of vLLM v1.Requestis the lifecycle carrier of a request inside the engine. Through its dual token lists, asynchronous scheduling counters, and block hash mechanism, it supports the two core features of continuous batching and prefix caching.KVCacheSpecand its inheritance hierarchy define the memory layout specification for the KV cache, ranging from the standardFullAttentionSpectoMLAAttentionSpec、MambaSpec, covering diverse model architecture requirements. The registry pattern allows new spec types to be added without modifying core code, ensuring system extensibility.

At this point, we have seen how a Request is transformed from an EngineCoreRequest, and how it supports scheduling decisions through state counters, block hashes, and other mechanisms. But how exactly does an external request traverse the API Server, chat template, and multimodal processing to ultimately become an EngineCoreRequest? The next chapter will enter the request entry layer and fully trace this path from HTTP/CLI to EngineCore.

In the previous chapter, we analyzed the two core data structures inside the engine, Request and KVCacheSpec, and understood how logical sequences are decoupled from physical memory blocks. But how exactly does an HTTP request body or a Python string traverse the API Server, chat template, and multimodal processing to ultimately become an EngineCoreRequest? This chapter will fully trace this path and reveal how the three entry paths—synchronous CLI, asynchronous API, and offline LLM class—converge onto the same engine core.

3.1 The Convergence Point of Three Entry Paths: AsyncLLMEngine and LLMEngine

Before diving into request parsing, we must first understand the topology of the three entry paths. vLLM provides three usage modes:vllm servethe OpenAI-compatible HTTP service started by, the command-linevllmtool, and directly instantiating theLLMclass in Python for offline inference. They appear independent but actually share the same engine core.

Let's first look at the alias mechanism of the asynchronous API path.

📎 vllm/engine/async_llm_engine.py:7-7

This file is so short it barely looks like a module—it does only one thing: aliasingAsyncLLMEngineto point tovllm.v1.engine.async_llm.AsyncLLM. This is a typical trace of architectural migration. In the vLLM v0 era,AsyncLLMEnginewas a large and complex class. After the v1 architecture rewrite, the newAsyncLLMtook on the same responsibilities. To avoid breaking existing user code, vLLM retained the old module path as a compatibility layer.

Now let's look at the entry point of the offline path.

📎 vllm/entrypoints/llm.py:344-346

LLM.__init__ultimately callsLLMEngine.from_engine_args, passing inUsageContext.LLM_CLASS. ThisUsageContextenum is the key to distinguishing entry paths—it lets the engine know whether it is running in offline batch mode or online serving mode, thereby adjusting logging, metrics, and resource management strategies.

📎 vllm/entrypoints/llm.py:357-359

Note the assignment ofself.renderer = self.llm_engine.rendererandself.input_processor = self.llm_engine.input_processorhere. The offlineLLMclass does not implement chat template rendering itself, but reuses the engine's internalrenderer. This means the chat template parsing logic is the same code on both offline and online paths, only the invocation timing differs.

The convergence relationship of the three paths can be represented by the following data flow diagram.

flowchart LR
    subgraph entry["入口层"]
        http["HTTP 请求体<br/>ChatCompletionRequest"]
        cli["CLI 参数<br/>vllm serve / vllm chat"]
        offline["Python 调用<br/>LLM.chat(messages)"]
    end

    subgraph parse["解析层"]
        chat_utils["chat_utils.parse_chat_messages<br/>-> ConversationMessage + mm_data"]
        renderer["renderer<br/>apply_chat_template -> token_ids"]
    end

    subgraph engine["引擎层"]
        async_llm["AsyncLLM<br/>add_request()"]
        llm_engine["LLMEngine<br/>add_request()"]
        core["EngineCore<br/>input_queue"]
    end

    http --> chat_utils
    cli --> chat_utils
    offline --> chat_utils
    chat_utils --> renderer
    renderer --> async_llm
    renderer --> llm_engine
    async_llm --> core
    llm_engine --> core

This diagram reveals a key design: regardless of whether the request comes from HTTP, CLI, or Python,chat_utilsis the sole entry point for multimodal and chat template processing. It unifies heterogeneous input formats into aConversationMessagelist plusMultiModalDataDict, then hands them to the renderer to generate token sequences.

3.2 chat_utils: From Heterogeneous Messages to Unified Conversation Structure

chat_utils.pyis the most complex module in the entire request entry layer. Its 2264 lines of code handle OpenAI-compatible formats, custom extensions, multimodal embeddings, tool calls, and all other input forms. Its core responsibility can be summarized in one sentence: normalize any message list passed by the user into aConversationMessagelist that the chat template can understand, while extracting multimodal data into a separateMultiModalDataDict.

Intuitive Model: Translator and Baggage Sorter

Think ofchat_utilsas an airport translator and baggage sorter. Passengers (users) come from different countries (OpenAI format, custom format, Harmony format), speaking different languages. The translator first translates everyone's words into a unified working language (ConversationMessage), while sorting the passenger's checked baggage (images, audio, video) onto independent conveyor belts (MultiModalDataDict), attaching tags (UUID), and finally loading both the person and the baggage onto the same airplane (engine).

Without this layer, the engine would have to understand the details of every input format, the extraction logic for multimodal data would be scattered across various entry points, and adding any new format would require modifying the engine core.

Data structures: dual-class collaboration between trackers and parsers

chat_utilsThe core of is the collaboration between two groups of classes:BaseMultiModalItemTrackerand its subclasses are responsible for "tracking" multimodal items,BaseMultiModalContentParserand its subclasses are responsible for "parsing" content parts.

First, let's look at the field layout of the tracker.

📎 vllm/entrypoints/chat_utils.py:598-601

_items_by_modalityis adefaultdict[str, list[_T]], storing pending items grouped by modality (image, audio, video, etc.)._modality_orderspecifically records, for thevision_chunkmodality, the original modality of each chunk (image or video), because the unified vision chunk model maps both tovision_chunk, but subsequent processing needs to know the original type.

📎 vllm/entrypoints/chat_utils.py:613-615

use_unified_vision_chunk_modalityis acached_property, reading theuse_unified_vision_chunkflag from the HuggingFace configuration. Usingcached_propertyinstead of a regular attribute is because this check is triggered on everyaddcall, and caching avoids repeatedgetattroverhead.

The tracker'saddmethod is the core entry point.

📎 vllm/entrypoints/chat_utils.py:656-684

addThe method first calls_validate_addfor validation, then stores items under different keys depending on whether the unified vision chunk modality is used. Note the special handling ofprompt_embeds: it directly appends to_items_by_modality["prompt_embeds"]and returnsNone, because precomputed embeddings do not go through the HF processor and have no placeholder string.

_validate_addThe validation logic in is worth a closer look.

📎 vllm/entrypoints/chat_utils.py:686-721

There is a subtle branch here: whenenable_mm_embeds=Trueand the per-prompt limit for that modality is 0 and the original modality ends with_embeds, skip the count validation. This is to allow embedding inputs to bypass the count limit of the original modality—embeddings are precomputed and do not consume the processing resources of the original modality.

Scenario-driven: how a chat request with an image is parsed

Suppose the user sends a chat request containing an image URL and text.parse_chat_messagesis the entry point for the synchronous path.

📎 vllm/entrypoints/chat_utils.py:2161-2197

parse_chat_messagescreatesMultiModalItemTracker, iterates over each message and calls_parse_chat_message_content, and finally calls_postprocess_messagesto process tool call parameters, then materializes multimodal data throughmm_tracker.resolve_items().

_parse_chat_message_contentis responsible for parsing a single message.

📎 vllm/entrypoints/chat_utils.py:2007-2029

It first normalizes content:Nonebecomes an empty list, and a string becomes a single text part. Then it calls_parse_chat_message_content_parts, where thewrap_dictsparameter is determined bycontent_format == "openai"—this determines whether the output is a list of structured dictionaries or a concatenated string.

_parse_chat_message_content_partsiterates over each part.

📎 vllm/entrypoints/chat_utils.py:1814-1853

Each part is processed by_parse_chat_message_content_part. Ifwrap_dicts=False, it ultimately concatenates text and placeholders into a single string; ifwrap_dicts=True, it returns a list of structured dictionaries.

_parse_chat_message_content_partis the core of dispatch.

📎 vllm/entrypoints/chat_utils.py:1875-1884

For pure text parts, it first performs a placeholder-preservation check, then decides the return format based onwrap_dicts. For structured parts, it calls_parse_chat_message_content_mm_partto extract the type and content.

📎 vllm/entrypoints/chat_utils.py:1690-1723

_parse_chat_message_content_mm_partlooks up the corresponding parsing function throughMM_PARSER_MAP. Note the condition ofuuid is None—if the user provides a UUID, it means the media data may not be in the request body (it has been uploaded through other means), so it goes to the direct URL field branch below.

📎 vllm/entrypoints/chat_utils.py:1731-1733

Whenpart_type is Noneoruuid is not None, the code tries to directly extract the URL field from the part. This "lenient parsing" is to be compatible with clients that do not strictly follow the OpenAI format.

Returning to_parse_chat_message_content_part, parts of media types are dispatched to the correspondingmm_parsermethods.

📎 vllm/entrypoints/chat_utils.py:1923-1968

Each media type calls the correspondingparse_*method, and these methods internally calltracker.addto add the item to the tracker and return a placeholder string. Finally, based oninterleave_strings, it decides whether to return the placeholder orNone。

📎 vllm/entrypoints/chat_utils.py:1984-1999

prompt_embedsis handled specially: regardless ofinterleave_strings, it returnsPROMPT_EMBEDS_PLACEHOLDER_TOKEN. The comment explains the reason—prompt_embeds are concatenated at token offsets, and position matters; if it went throughmissing_placeholders's pre-padding logic, the order would be disrupted.

Differences in the asynchronous path

The asynchronous path usesAsyncMultiModalItemTrackerandAsyncMultiModalContentParser. The core difference is inresolve_items。

📎 vllm/entrypoints/chat_utils.py:906-952

The asynchronous version usesasyncio.gatherto concurrently await all modality items. The comment explicitly points out: each tracked item is already an independent awaitable, and the asynchronous connector offloads blocking decoding work to a thread pool, so serially awaiting one modality and then the next would unnecessarily increase latency.return_exceptions=Truelets all tasks complete or fail before throwing uniformly, avoiding giving up on network requests still in progress just because the first one fails.

Design reflection: why trackers and parsers are separated

3.3 From messages to tokens: the handoff between renderer and EngineCore

chat_utilsTheConversationMessagelist andMultiModalDataDictproduced by still need to go through chat template rendering before they can become token sequences. This step is completed by the renderer, after which the request truly enters the engine.

Scenario-driven: chat template rendering and request submission

parse_chat_messagesAfter returning, the caller (such asOpenAIServingChat) will passconversationandmm_datato the renderer. The renderer applies the chat template, renders theConversationMessagelist into text, and then tokenizes it into a token ID sequence. Multimodal placeholders (such as<##IMAGE##>) are replaced with model-specific placeholder tokens after tokenization.

After rendering is complete, the request is encapsulated asEngineCoreRequestand delivered to EngineCore's input queue viaAsyncLLM.add_request()orLLMEngine.add_request().

📎 vllm/entrypoints/llm.py:420-484

OfflineLLM.generateThe method demonstrates this chain: it first validatesrunner_type, obtains default sampling parameters, then calls_run_completion。_run_completionInternally, it calls the renderer to render the prompt, then delivers the request viallm_engine.

📎 vllm/entrypoints/llm.py:615-708

LLM.chatThe method demonstrates the chat path: it receives themessageslist, calls_run_chat, which internally callsparse_chat_messagesand the renderer.

Design consideration: Why is the renderer inside the engine

Error recovery and production pitfalls

_postprocess_messagesThe tool call parameter handling in

📎 vllm/entrypoints/chat_utils.py:2118-2158

is a typical production environment trap.tool_callsWhen an assistant message containsarguments, the field may be a JSON string, a dictionary, or invalid JSON. The code attempts to parse the JSON string; if it fails, it logs a warning and forcibly converts it to an empty object. The comment explains the reason: malformedargumentsexists in the conversation history, and if the request is failed here, every subsequent turn will fail, and the conversation will be unable to recover. This is a deliberate fault-tolerance design—better to let the model see empty tool parameters than to let the entire conversation get stuck.

Another trap is injection protection for reserved placeholders.

📎 vllm/entrypoints/chat_utils.py:1856-1872

Whenenable_prompt_embedsis enabled,PROMPT_EMBEDS_PLACEHOLDER_TOKENis registered as an indivisible special token. If the user text happens to contain this literal sequence, the tokenizer will encode it as the same token ID, and the renderer will mistakenly think this is a splice point, allowing the caller to move or inject the splice position through plain text content._reject_reserved_placeholder_in_textrejects this kind of input during text part parsing, closing this security hole.

📎 vllm/entrypoints/chat_utils.py:1889-1892

Note that this check is called in both theisinstance(part, str)branch and the structured text branch, ensuring that all text paths are protected.

Chapter summary

This chapter traced the first segment of the path by which a request enters the system from the outside. The three entry paths—HTTP API, CLI, and offlineLLMclass—all ultimately converge onchat_utils's multimodal parsing layer.BaseMultiModalItemTrackeris responsible for state management,BaseMultiModalContentParseris responsible for content extraction, and the separation of the two allows the synchronous and asynchronous paths to share tracing logic.parse_chat_messagesnormalizes heterogeneous messages into aConversationMessagelist andMultiModalDataDict, then hands them to the engine-internal renderer to complete chat template rendering and tokenization. Finally, the request is encapsulated asEngineCoreRequestand delivered to EngineCore's input queue.

Chapter review and self-test

The next chapter will enter the scheduler to see how EngineCore orchestrates these requests using continuous batching and memory-aware strategies.

At this point, the request has completed the normalized transformation from external input to EngineCoreRequest and has reached the entrance of the engine core. But after the request enters, it is not executed immediately—the engine needs to decide which requests to process at each step and how to allocate limited GPU memory resources. The next chapter will delve into EngineCore's scheduling loop, analyzing how the Scheduler balances throughput and latency in continuous batching, and how chunked prefill, prefix caching, and KV block allocation work together.

After requests enter EngineCore's input queue, they are not executed immediately. Which requests to process at each step, how many token budgets to allocate to each request, and who to sacrifice first when GPU memory is insufficient—these decisions are all concentrated in theScheduler.schedule()method. This chapter starts from the scheduler's data structures and traces how a singleschedule()call organizes the waiting queue, running list, and KV cache pool into an executable batch.

4.1 Scheduler Data Structures: Three Queues and One Memory Pool

The core question the scheduler must answer is:Under limited token budget and KV block budget, which requests should advance by how many tokens at this step?To understand it, we must first see clearly what state it holds.

The scheduler maintains three types of request containers.self.requestsis a global dictionary,req_id -> Request, the single source of truth for all active requests📎 vllm/v1/core/sched/scheduler.py:208-209。self.waitingandself.skipped_waitingare two priority queues; the former holds requests normally waiting to be scheduled, while the latter holds requests that temporarily cannot be scheduled due to asynchronous dependencies or constraints (such as waiting for remote KV or waiting for structured output grammar compilation)📎 vllm/v1/core/sched/scheduler.py:208-209。self.runningis an ordinary list, storing requests that have already entered the running state and hold KV blocks📎 vllm/v1/core/sched/scheduler.py:208-209。

There is a design here that is easy to overlook:max_num_running_reqsandmax_num_active_reqsare two different upper limits. The former comes frommax_num_seqs, determining the number of slots for the model runner; the latter comes frommax_num_active_seqs, only limiting the number of requests that can enter RUNNING, and by default equal to the former📎 vllm/v1/core/sched/scheduler.py:123-131. This separation allows reducing the actual concurrent decode batch size without shrinking CUDA graph capture capacity.

The memory side is uniformly managed byKVCacheManager, which internally holdsBlockPool。BlockPoolThe core of isself.blocks(a list of allKVCacheBlock) andfree_block_queue(a doubly linked list of free blocks arranged in eviction order)📎 vllm/v1/core/block_pool.py:171-177. Note the existence ofnull_block: it is the first block popped from the head of the free queue,is_null=True, reference counting does not participate in regular maintenance, and it is specifically used as a placeholder📎 vllm/v1/core/block_pool.py:183-187. When a certain token position of a request does not need a real KV block (for example, a position skipped by the sliding window), this null block is filled into the block table.

The index structure for prefix caching isBlockHashToBlockMap, which mapsBlockHashWithGroupIdto aKVCacheBlockor a{block_id: KVCacheBlock}dictionary📎 vllm/v1/core/block_pool.py:56-59. Why use a union type? The comment gives the answer: most hashes correspond to only one block, and using a dictionary would cause unnecessary GC overhead; only when the same hash is shared by multiple blocks does it upgrade to a dictionary📎 vllm/v1/core/block_pool.py:56-59. This is a typical trade-off of type complexity for runtime overhead.

KVCacheBlocksis the interface object between the scheduler and the KV cache manager, hiding the internal data structures. Itsblocksfield istuple[Sequence[KVCacheBlock], ...], the outer dimension is the KV cache group, and the inner dimension is the block sequence📎 vllm/v1/core/kv_cache_manager.py:41-54. The comment explicitly explains why blocks are not used as the outer dimension: that would assume all groups have the same number of blocks, whereas in the future different groups may be configured with different block sizes📎 vllm/v1/core/kv_cache_manager.py:43-48。

flowchart LR
    subgraph Sched["Scheduler 状态"]
        W["waiting<br/>RequestQueue"]
        SW["skipped_waiting<br/>RequestQueue"]
        R["running<br/>list[Request]"]
        REQ["requests<br/>dict[str, Request]"]
    end
    subgraph KV["KVCacheManager"]
        BP["BlockPool.blocks<br/>list[KVCacheBlock]"]
        FQ["free_block_queue<br/>FreeKVCacheBlockQueue"]
        MAP["cached_block_hash_to_block<br/>BlockHashToBlockMap"]
    end
    W -->|"admit + allocate_slots"| R
    R -->|"preempt"| W
    R -->|"free / pop_blocks_for_free"| FQ
    FQ -->|"get_new_blocks"| BP
    BP -->|"cache_full_blocks"| MAP
    MAP -->|"get_cached_block"| W

This diagram anchors the data flow between the scheduler and the memory pool: requests in the waiting queue enter running throughallocate_slots, running requests return to waiting when preempted, freed blocks return to the free queue, and the prefix caching hash table is the entry point for waiting requests to hit the cache.

4.2 schedule() main flow: running first, waiting supplement, preemption as fallback

schedule()is the core method of the entire scheduler, and it returns aSchedulerOutput, describing what to execute in this step. The comment at the beginning of the method points out the design philosophy: there is no distinction between the "decode phase" and the "prefill phase" in the scheduler; each request only hasnum_computed_tokensandnum_tokens_with_spec, and the scheduler's task is to let the former catch up with the latter📎 vllm/v1/core/sched/scheduler.py:559-568. This unified perspective is the foundation for chunked prefill, prefix caching, and speculative decoding to coexist.

4.2.1 Budget initialization and threshold calculation

Before entering the main loop, the scheduler first sets two budgets:token_budgetinitialized tomax_num_scheduled_tokens，input_budgetinitialized tomax_num_batched_tokens 📎 vllm/v1/core/sched/scheduler.py:577-580. The two are usually equal, but when the model may append tokens within a batch (such as speculative decoding),max_num_scheduled_tokenswill be less thanmax_num_batched_tokens, and the difference is the space reserved for draft tokens.

long_prefill_token_thresholdThe handling of is worth looking at separately. Its purpose is to prevent a long prefill from starving other requests, but if there is only one request currently, no one will be starved, so the threshold is set to zero📎 vllm/v1/core/sched/scheduler.py:606-616. Whenadaptive_long_prefill_thresholdis enabled, the threshold is also raised toinput_budget // num_eligible_reqs, ensuring that a single request's budget is not squeezed below its fair share📎 vllm/v1/core/sched/scheduler.py:617-622。

4.2.2 Scheduling loop for running requests

The main loop traverses from the head ofself.running,req_indexis the cursor📎 vllm/v1/core/sched/scheduler.py:624-627. For each request, a series of skip checks are performed first:

• Under asynchronous scheduling, if the request's output placeholder indicates that it has reachedmax_tokens, skip to avoid running an extra step📎 vllm/v1/core/sched/scheduler.py:631-645。
• In the V2 + PP + asynchronous scenario, if the current step has not yet reachednext_decode_eligible_step, skip to match the sampling token broadcast rhythm on the worker side📎 vllm/v1/core/sched/scheduler.py:647-651。
• When DP prefill balancing is enabled, prefill chunks on non-rhythm-aligned steps are postponed📎 vllm/v1/core/sched/scheduler.py:653-657。

After passing the skip checks, calculate how many tokens this request can advance in this step:

num_new_tokens = request.num_tokens_with_spec
               + request.num_output_placeholders
               - request.num_computed_tokens

Then it is constrained in turn bylong_prefill_token_threshold、token_budget、input_budget - draft_slotsandmax_model_len. If the request carries encoder input, it also needs to be adjusted by📎 vllm/v1/core/sched/scheduler.py:670-688_try_schedule_encoder_inputsNext is the most critical step: allocating KV blocks.📎 vllm/v1/core/sched/scheduler.py:700-712。

is wrapped in aallocate_slotsloopwhile True. If it returns📎 vllm/v1/core/sched/scheduler.py:742-747, it means there is not enough memory, and the scheduler begins preemption: select a victim according to the policy (the PRIORITY policy selects the lowest-priority one, and the FCFS policy selects the one at the end of the running list)None, call📎 vllm/v1/core/sched/scheduler.py:761-767to kick it back to the waiting queue, and then retry allocation_preempt_request. If the victim is the current request itself, it means there is no object left to preempt, so break out of the loop, and the current request cannot be scheduled either📎 vllm/v1/core/sched/scheduler.py:801-806There is a subtle detail in the preemption logic: under the PRIORITY policy, if the preempted request is already in📎 vllm/v1/core/sched/scheduler.py:807-813。

(that is, resources have already been allocated for it in this step), its token budget, blocks, speculative tokens, and encoder budget all need to be returnedscheduled_running_reqs. This ensures the consistency of the budget ledger.📎 vllm/v1/core/sched/scheduler.py:779-797After successful allocation, the request is added to

, recording the block and token counts, and deducting the budgetscheduled_running_reqs. Tokens related to speculative decoding are trimmed and recorded here📎 vllm/v1/core/sched/scheduler.py:815-8234.2.3 Admission of waiting requests📎 vllm/v1/core/sched/scheduler.py:825-841。

After the running loop ends, if no preemption occurred in this step and the scheduler is not paused, start processing the waiting queue

. Before admission, check two upper limits first:📎 vllm/v1/core/sched/scheduler.py:868-872andmax_num_active_reqsScheduling waiting requests has one more prefix cache lookup step than running requests. Wheninput_budget 📎 vllm/v1/core/sched/scheduler.py:873-879。

, callrequest.num_computed_tokens == 0to look up local cache hits_get_local_prefix_cache_hit. If a KV connector is configured, remote cache hits are also queried📎 vllm/v1/core/sched/scheduler.py:932-939Here there is a delicate logic for handling conflicts between local and remote hits. A local hit may not be block-aligned (📎 vllm/v1/core/sched/scheduler.py:942-954。

), and if the remote hit strictly exceeds the local complete hit, discard the local sub-block tail and let the remote load overwrite it, avoiding copy-on-writepartial_tail. Otherwise, keep the local tail and do not load external📎 vllm/v1/core/sched/scheduler.py:977-988After successful admission, the request is popped from the waiting queue, its state is set to RUNNING, and it is added to the running list📎 vllm/v1/core/sched/scheduler.py:989-995。

. If it is still in prefill after this step (📎 vllm/v1/core/sched/scheduler.py:1263-1319), add it to thenum_computed_tokens + num_new_tokens < request.num_tokensset_inflight_prefillsCopy📎 vllm/v1/core/sched/scheduler.py:1326-1328。

flowchart TD
    start["schedule() 开始"] --> init["初始化 token_budget / input_budget"]
    init --> run_loop{"running 循环<br/>req_index < len(running)<br/>且 token_budget > 0?"}
    run_loop -->|是| skip_check{"跳过条件?<br/>max_tokens 已达 /<br/>decode_eligible / defer_prefills"}
    skip_check -->|跳过| run_inc["req_index += 1"]
    run_inc --> run_loop
    skip_check -->|不跳过| calc["计算 num_new_tokens<br/>受多约束裁剪"]
    calc --> alloc{"allocate_slots<br/>返回 None?"}
    alloc -->|成功| admit_run["加入 scheduled_running_reqs<br/>扣减预算"]
    admit_run --> run_inc
    alloc -->|失败| can_preempt{"有可抢占请求?<br/>_request_blocks_can_be_freed"}
    can_preempt -->|否| break_run["跳出 running 循环"]
    can_preempt -->|是| preempt["_preempt_request<br/>踢回 waiting"]
    preempt --> alloc
    break_run --> wait_loop{"无抢占且未暂停?<br/>waiting 非空且 token_budget > 0?"}
    run_loop -->|否| wait_loop
    wait_loop -->|是| blocked{"blocked 状态?<br/>_is_blocked_waiting_status"}
    blocked -->|是且无法提升| skip_wait["移入 skipped_waiting"]
    skip_wait --> wait_loop
    blocked -->|否| prefix{"num_computed_tokens == 0?<br/>查找前缀缓存"}
    prefix -->|命中| alloc_wait["allocate_slots<br/>带 new_computed_blocks"]
    prefix -->|未命中| alloc_wait
    alloc_wait --> wait_ok{"分配成功?"}
    wait_ok -->|是| admit_wait["加入 running<br/>状态设为 RUNNING"]
    admit_wait --> wait_loop
    wait_ok -->|否| break_wait["跳出 waiting 循环"]
    wait_loop -->|否| build["构建 SchedulerOutput"]
    break_wait --> build

's two major loops and the preemption branch. Note the preemption retry path afterschedule()fails in the running loop, and the blocked-state requests in the waiting loop being moved intoallocate_slots 失败后的抢占重试路径，以及 waiting 循环中 blocked 状态请求被移入 skipped_waitingbypass.

4.3 The Core of Memory Awareness: allocate_slots and Preemption

allocate_slotsis the gate between the scheduler and GPU memory. Its parameter list is itself a memory ledger:num_new_tokensis the number of tokens to be newly computed,num_new_computed_tokensis the number of tokens newly hit in the prefix cache,num_external_computed_tokensis the number of external hits provided by the connector,num_lookahead_tokensis the slots reserved for speculative decoding.📎 vllm/v1/core/kv_cache_manager.py:371-383。

The comment at the beginning of the method precisely describes the block layout with an ASCII diagram.📎 vllm/v1/core/kv_cache_manager.py:417-438：

| < comp > | < new_comp > | < ext_comp >  | < new >  | < lookahead > |
                                          |   < to be computed >     |
                        |            < to be allocated >           |

compis already-computed tokens,new_compis prefix cache hits,ext_compis external hits,newis newly computed in this step,lookaheadis speculative reservation. Allocation is divided into three stages: first release unneeded blocks and check whether there are enough free blocks, then process prefix tokens, and finally allocate blocks for newly computed tokens.📎 vllm/v1/core/kv_cache_manager.py:458-461。

4.3.1 Watermark and Admission Control

allocate_slotsThere are two admission gates in .full_sequence_must_fit: when enabled, it first checks whether the entire request sequence (not just the first chunk) can fit, and if not, directly returnsNone 📎 vllm/v1/core/kv_cache_manager.py:515-531. This prevents excessive admission under chunked prefill from causing KV cache thrashing.

The second is the watermark.watermark_blocksIt only takes effect when the request state is WAITING or PREEMPTED and some request has already been scheduled.📎 vllm/v1/core/kv_cache_manager.py:506-513It requires that at least a certain proportion of free blocks be retained after allocation, avoiding frequent eviction and preemption.reserved_blocksIt is used for asynchronous KV loading scenarios to ensure that the reserved blocks for in-flight prefill are not consumed by new requests.📎 vllm/v1/core/kv_cache_manager.py:564-570。

4.3.2 The Cost and Recovery of Preemption

Preemption also handles the "stale output" problem under asynchronous scheduling.num_stale_output_tokensis set tonum_in_flight_tokens, marking all in-flight outputs as stale.📎 vllm/v1/core/sched/scheduler.py:1571-1574. These tokens will still be delivered (discarding them would perturb the speculative decoding acceptance rate), but they will not modify the reset counters.drop_stale_outputThe flag determines whether to discard or deliver.📎 vllm/v1/core/sched/scheduler.py:1539-1547。

4.3.3 Delayed Release: The Read-After-Write Risk of Asynchronous Connectors

When a KV connector is used and there are multiple in-flight batches,defer_block_freeis set toTrue 📎 vllm/v1/core/sched/scheduler.py:175-181. The reason is that a step may still be writing the KV blocks of an already released request, while a consumer connector may reallocate and fill those blocks through a load that is not ordered with that write.

Delayed release is implemented throughdeferred_freesa double-ended queue, where each entry is(fence_seq, blocks) 📎 vllm/v1/core/sched/scheduler.py:388-390。_free_request_blockschecks_request_blocks_can_be_freed. If the request's last scheduling step has not yet been processed, the blocks are placed into the delayed queue.📎 vllm/v1/core/sched/scheduler.py:2679-2688。_drain_deferred_freesis advanced inupdate_from_outputand then called to release blocks whose fence has been satisfied.processed_step_seq4.4 Prefix Cache Hit Determination and Block Lifecycle📎 vllm/v1/core/sched/scheduler.py:2701-2706。

The lookup entry point for the prefix cache is

. It first checks whether the cache is enabled and whether the request is marked to skip reading.KVCacheManager.get_computed_blocks. Then it calls📎 vllm/v1/core/kv_cache_manager.py:286-287, passing incoordinator.find_longest_cache_hitandrequest.block_hashesWhymax_cache_hit_length = request.num_tokens - 1 📎 vllm/v1/core/kv_cache_manager.py:295-300。

? The comment explains: when all tokens hit the cache, the last token must still be recomputed to obtain logits.num_tokens - 1. This is an easily overlooked boundary: even if the prefix is fully hit, at least one token must still be computed.📎 vllm/v1/core/kv_cache_manager.py:289-294The lifecycle of a block is managed by

.BlockPoolpops a block from the head of the free queue. If caching is enabled, it first callsget_new_blocksto clear its hash metadata, and then increments the reference count._maybe_evict_cached_blockThen, depending on whether the block has a hash, it is placed back at the head or tail of the queue: blocks without a hash are reused LIFO (better GPU locality), and blocks with a hash are reused FIFO (LRU eviction behavior).📎 vllm/v1/core/block_pool.py:683-702。free_blocksis the moment when a block is written into the prefix cache hash table. It traverses newly full blocks, skips null blocks and masked blocks, computes a hash for each block, and inserts it into📎 vllm/v1/core/block_pool.py:785-805。

cache_full_blocks. If a block already has a hash (the scenario where a partial block is upgraded to a full block), first remove the old hash and then insert the new hash.cached_block_hash_to_block 📎 vllm/v1/core/block_pool.py:272-300The method handles reference counting on cache hits: if the block is in the free queue (📎 vllm/v1/core/block_pool.py:285-293。

touch), first remove it from the queue, and then increment the reference count.ref_cnt == 0. This ensures that a hit block will not be evicted.📎 vllm/v1/core/block_pool.py:754-770Design Considerations

[Design Inference and Architectural Trade-offs]

Chapter Summary

The core of the scheduler is theschedule()two loops in the method: the running loop prioritizes advancing already-running requests, while the waiting loop admits new requests when budget allows. When VRAM is insufficient, space is freed by preempting the lowest-priority request in the running list. The preempted request'snum_computed_tokensis reset to 0, but prefix caching can offset part of the recomputation cost.allocate_slotsis the VRAM gate, throughfull_sequence_must_fit, watermark, andreserved_blocksthree-tier admission control to prevent over-allocation. Prefix caching enables cross-request sharing through block hash indexing, with hit determination capped atnum_tokens - 1to ensure at least one token is computed to obtain logits.

Chapter Review and Self-Test

The scheduler's outputSchedulerOutputclarifies the execution content of this step: block IDs for new requests, number of cached tokens for cached requests, speculative tokens, encoder inputs, etc. The next chapter will trace how this output is consumed by the ModelRunner, fromSchedulerOutputall the way to the GPU forward pass.

In the previous chapter, we saw how the Scheduler, in each step's scheduling loop, determines which requests enter the running queue, which are preempted, and which wait due to insufficient VRAM, ultimately producing a SchedulerOutput—which describes what to compute in this step: which requests, how many tokens each, and which KV blocks to use. But this list is only a logical intent; the GPU needs physical tensors. This chapter traces how SchedulerOutput is distributed by the Executor to Workers, then translated by GPUModelRunner into GPU-executable inputs such as input_ids, positions, slot_mapping, and block table, and finally injects the cross-layer shared batch description into each model layer through forward_context, completing the leap from scheduling decisions to forward propagation.

5.1 Executor: Delivering Scheduling Results to Every Card

Intuitive Model

Executoris the "herald" between EngineCore and GPU Workers. Without it, EngineCore would have to know by itself how many cards are in the cluster, which process each card is in, and how toSchedulerOutputSerializing the past—scheduling logic would become entangled with the distributed topology.ExecutorExtract this responsibility: EngineCore only needs to callexecute_model(scheduler_output), and the rest—"whom to send to, how to send, how many results to collect"—is decided by the Executor.

Class hierarchy and fields

Executoris an abstract base class whose class-level fields directly encode backend capabilities📎 vllm/v1/executor/abstract.py:48-49：

uses_ray: bool = False  # whether the executor uses Ray for orchestration.
supports_pp: bool = False  # whether the executor supports PP

These two flags are not decorative—upper-layer code reads them to decide whether to enable certain optimization paths.__init__In , the following are initialized:sleeping_tags、kv_output_aggregator、ec_output_aggregatorthree state fields📎 vllm/v1/executor/abstract.py:119-120, used respectively for sleep-mode tag tracking, KV connector output aggregation, and encoder connector output aggregation.

Backend selection:get_classbranch routing in

get_classis a static factory that returns the concrete Executor class based on thedistributed_executor_backendconfiguration📎 vllm/v1/executor/abstract.py:51-96. Its branching structure is worth examining closely:

• If the configuration itself is atype, validate whether it is aExecutorsubclass and then use it directly📎 vllm/v1/executor/abstract.py:52-61；
• "ray"Under the branch there are further secondary branches:VLLM_USE_RAY_V2_EXECUTOR_BACKENDwhen true, useRayExecutorV2, otherwise useRayDistributedExecutor 📎 vllm/v1/executor/abstract.py:64-72；
• "mp"maps toMultiprocExecutor，"uni"maps toUniProcExecutor 📎 vllm/v1/executor/abstract.py:73-80；
• Custom backends in string form are dynamically resolved throughresolve_obj_by_qualname📎 vllm/v1/executor/abstract.py:85-90。

flowchart TD
    start["Executor.get_class(vllm_config)"] --> check_type{"backend 是 type?"}
    check_type -->|是| verify_sub{"issubclass(Executor)?"}
    verify_sub -->|否| err_type["raise TypeError"]
    verify_sub -->|是| use_direct["executor_class = backend"]
    check_type -->|否| check_ray{"backend == 'ray'?"}
    check_ray -->|是| ray_v2{"VLLM_USE_RAY_V2?"}
    ray_v2 -->|是| use_rayv2["RayExecutorV2"]
    ray_v2 -->|否| use_ray["RayDistributedExecutor"]
    check_ray -->|否| check_mp{"backend == 'mp'?"}
    check_mp -->|是| use_mp["MultiprocExecutor"]
    check_mp -->|否| check_uni{"backend == 'uni'?"}
    check_uni -->|是| use_uni["UniProcExecutor"]
    check_uni -->|否| check_ext{"backend == 'external_launcher'?"}
    check_ext -->|是| use_ext["ExecutorWithExternalLauncher"]
    check_ext -->|否| check_str{"backend 是 str?"}
    check_str -->|是| resolve["resolve_obj_by_qualname"]
    check_str -->|否| err_unknown["raise ValueError"]

Step-by-Step: the call flow of oneexecute_model

Set the scenario: EngineCore completes one scheduling step and obtainsSchedulerOutput, then callsexecutor.execute_model(scheduler_output)。

Executor.execute_modelThe implementation is extremely minimal📎 vllm/v1/executor/abstract.py:237-238：

def execute_model(
    self, scheduler_output: SchedulerOutput, non_block: bool = False
) -> ModelRunnerOutput | None | Future[ModelRunnerOutput | None]:
    output = self.collective_rpc(
        "execute_model", args=(scheduler_output,), non_block=non_block
    )
    return output[0]

sample_tokensfollows the same pattern📎 vllm/v1/executor/abstract.py:257-258, but the return type does not includeNone—sampling necessarily produces a result. The division of labor between these two methods corresponds to vLLM v1's "execution-sampling separation" design:execute_modelmay returnNone(indicating that the forward pass has been submitted but sampling is deferred), in which case the state is temporarily stored inExecuteModelState.

Design considerations

collective_rpcis declared as@abstractmethod 📎 vllm/v1/executor/abstract.py:186-192, meaning different backends must implement "how to send the RPC to the Worker" themselves.MultiprocExecutoruses shared-memory queues,RayDistributedExecutoruses Ray actor calls, andUniProcExecutordirectly calls locally. This abstraction means upper-layer code does not need to care about distributed details at all.

An easily overlooked detail:supported_tasksis marked as@cached_property 📎 vllm/v1/executor/abstract.py:306-309, and the comment explicitly says "avoid unnecessary RPC calls." Becauseget_supported_tasksrequires cross-process communication, while the task list does not change during the model lifecycle, caching is a correct and necessary optimization.

5.2 GPUModelRunner: from SchedulerOutput to input tensors

Intuitive model

GPUModelRunneris a "translator": it translates the logical description inSchedulerOutput(request ID, token count, block ID) into physical tensors that the GPU can directly consume. Without it, the model layer would have to handle questions like "which KV slot contains the 7th token of the 3rd request"—a disastrous leak of concerns.

Core state and memory layout

GPUModelRunnerinherits from three Mixins📎 vllm/v1/worker/gpu_model_runner.py:479-480：LoRAModelRunnerMixin、KVConnectorModelRunnerMixin、ECConnectorModelRunnerMixin, which respectively provide LoRA adaptation, KV connector, and encoder connector capabilities.

__init__caches all configuration objects📎 vllm/v1/worker/gpu_model_runner.py:488-498, and initializes several key flags:

• check_ep_fault: only when data parallelism > 1 and the model is MoE, query whether the EP all2all manager supports fault tolerance📎 vllm/v1/worker/gpu_model_runner.py:507-509；
• is_pooling_model: determined byrunner_type == "pooling"📎 vllm/v1/worker/gpu_model_runner.py:515；
• enable_prompt_embeds: whether prompt embedding input is enabled📎 vllm/v1/worker/gpu_model_runner.py:516。

ExecuteModelStateis aNamedTuple, carrying the temporary state betweenexecute_model()andsample_tokens(). Its field design reveals the essence of execution-sampling separation:📎 vllm/v1/worker/gpu_model_runner.py:463-476is the forward-pass product,logits、hidden_states、sample_hidden_statesis the metadata still needed during the sampling stage. The comment explicitly says this is "temporary cached state passed after execute_model() returns None"spec_decode_metadata、slot_mappingsHow to synchronize cached state📎 vllm/v1/worker/gpu_model_runner.py:464-464。

Step-by-Step：_update_statesSet the scenario: the scheduler decides that this step processes request A (new request), B (continuation of the previous decode step), and C (resumed after preemption), while request D has already completed.

Step 1: clean up completed requests.

iterates over, pops the state from thefinished_req_idsdictionary, and removesself.requestsfrominput_batch. Note the edge case pointed out by the comment:📎 vllm/v1/worker/gpu_model_runner.py:1202-1217andfinished_req_idsmay overlap—when a request is aborted and then resubmitted with the same ID, they are treated as two different requestsscheduled_req_idsStep 2: zero out newly allocated KV blocks.📎 vllm/v1/worker/gpu_model_runner.py:1211-1215。

Ifis non-empty, callnew_block_ids_to_zeroto zero the GPU memory, preventing stale NaNs from contaminating attention or SSM computation_zero_block_ids. This is the safety prerequisite for PagedAttention block reuse.📎 vllm/v1/worker/gpu_model_runner.py:1219-1222Step 3: compute the set of unscheduled requests.

This is the most error-prone stepCopy📎 vllm/v1/worker/gpu_model_runner.py:1238-1247：

scheduled_req_ids = scheduler_output.num_scheduled_tokens.keys()
cached_req_ids = self.input_batch.req_id_to_index.keys()
resumed_req_ids = scheduler_output.scheduled_cached_reqs.resumed_req_ids
unscheduled_req_ids = cached_req_ids - (scheduled_req_ids - resumed_req_ids)

rather than directlyscheduled_req_ids - resumed_req_ids: usuallyscheduled_req_idsandcached_req_idsare disjoint, but in forced preemption scenarios triggered byresumed_req_ids, resumed requests need to be removed from the persistent batch first and then re-addedreset_prefix_cacheStep 4: handle new requests.📎 vllm/v1/worker/gpu_model_runner.py:1241-1246。

For each, constructscheduled_new_reqs. If the sampling type isCachedRequestState 📎 vllm/v1/worker/gpu_model_runner.py:1295-1308, create a seededRANDOM_SEED. If the model uses M-RoPE, calltorch.Generator 📎 vllm/v1/worker/gpu_model_runner.py:1277-1284to precompute positions_init_mrope_positionsStep 5: update running requests.📎 vllm/v1/worker/gpu_model_runner.py:1319-1321。

For each, updatescheduled_cached_reqs, handling block ID appends or replacementsnum_computed_tokens 📎 vllm/v1/worker/gpu_model_runner.py:1402. If the request is not in the persistent batch (📎 vllm/v1/worker/gpu_model_runner.py:1437-1448), add it toreq_index is NoneStep 6: compaction and reordering.reqs_to_add 📎 vllm/v1/worker/gpu_model_runner.py:1450-1465。

第六步：压缩与重排。 condense()Fill the holes left by removal requests📎 vllm/v1/worker/gpu_model_runner.py:1511-1512，_may_reorder_batchLet the attention backend rearrange on demand📎 vllm/v1/worker/gpu_model_runner.py:1513-1514，refresh_metadata()Refresh batch metadata📎 vllm/v1/worker/gpu_model_runner.py:1515-1516。

Input tensor preparation:_prepare_input_idsasynchronous fast path

_prepare_input_idshandles a subtle issue: under asynchronous scheduling, the sampled token from the previous step is still on the GPU, and this step'sinput_idsneeds to fill them in📎 vllm/v1/worker/gpu_model_runner.py:1767-1772。

Normal path (prev_sampled_token_ids is None) directly copies CPU tensors to GPU📎 vllm/v1/worker/gpu_model_runner.py:1788-1794. The asynchronous path iterates over requests, computing the index of each request's last token in the flattenedinput_ids.📎 vllm/v1/worker/gpu_model_runner.py:1809-1836The comment gives a concrete example:cu_num_tokens = [2, 5, 8]、draft_tokens = [1, 2, 2]whensample_flattened_indices = [0, 2, 5]，spec_flattened_indices = [1, 3, 4, 6, 7] 📎 vllm/v1/worker/gpu_model_runner.py:1820-1822。

there is a key optimization📎 vllm/v1/worker/gpu_model_runner.py:1859-1868：

if common_indices_match and max_flattened_index == (num_common_tokens - 1):
    self.input_ids.gpu[:num_common_tokens].copy_(
        self.input_batch.prev_sampled_token_ids[:num_common_tokens, 0],
        non_blocking=True,
    )
    return

When the batch is unchanged and there is no rearrangement, the indices are0..N-1the same permutation, so a single slice copy can be used directly, avoiding scatter overhead. This is a direct manifestation of the persistent batch optimization.

slot_mappingand block table

_get_slot_mappingsreturns two formats📎 vllm/v1/worker/gpu_model_runner.py:4078-4078: indexed by KV cache groupdict[int, torch.Tensor]for attention metadata use, and indexed by layer namedict[str, torch.Tensor]forForwardContextuse. For encoder-only KV cache groups, slot mapping is an all-zero tensor📎 vllm/v1/worker/gpu_model_runner.py:4096-4115; otherwise slice fromblock_table.slot_mapping.gpu.📎 vllm/v1/worker/gpu_model_runner.py:4107-4109Unused tail padding-1, the comment explains this isreshape_and_cacherequired in full CUDA graph mode📎 vllm/v1/worker/gpu_model_runner.py:4118-4122。

_get_block_tableobtains device tensors for each KV cache group📎 vllm/v1/worker/gpu_model_runner.py:2319-2335, and usesNULL_BLOCK_IDto fill CUDAGraph padding rows—block 0 is reserved for padding📎 vllm/v1/worker/gpu_model_runner.py:2332-2334。

5.3 forward_context: batch description shared across layers

Intuitive model

forward_contextis a "unified notice board" posted at the front of the classroom: every model layer can look up and see the seating arrangement (attention metadata) and rules (slot mapping) for this exam, without having to ask individually. Without it, every attention layer would have to receive this information from parameters—and the model layer'sforwardsignature is fixed, so parameters cannot be passed separately for each layer.

Data structure

ForwardContextis a@dataclass 📎 vllm/forward_context.py:141-202, core fields:

• no_compile_layers: copied fromstatic_forward_context, marks layers that do not participate in compilation📎 vllm/forward_context.py:132-137；
• attn_metadata: mapping from layer name to attention metadata; in DBO mode it is a list of length 2 (one per microbatch)📎 vllm/forward_context.py:144-152；
• slot_mapping: mapping from layer name to slot mapping tensor📎 vllm/forward_context.py:145；
• cudagraph_runtime_mode: runtime CUDA graph mode, defaultNONE 📎 vllm/forward_context.py:155-157；
• batch_descriptor: batch descriptor, used for CUDA graph dispatch📎 vllm/forward_context.py:158；
• is_padding: boolean mask on the token axis,Trueindicates padding rows📎 vllm/forward_context.py:162-165。

BatchDescriptoris another@dataclass(frozen=True) 📎 vllm/forward_context.py:30-57, and the field design follows the "minimize description items" principle:num_tokens、num_reqs(can be None in PIECEWISE mode),uniform(all requests have the same number of tokens),has_lora、num_active_loras. The comment explainsnum_active_lorasthe reason for its existence: whencudagraph_specialize_lora_countis enabled, each LoRA count value captures an independent CUDA graph, because the grid size of kernels such asfused_moe_loradepends on this value📎 vllm/forward_context.py:60-64。

Global singleton and context management

_forward_contextis a module-level global variable📎 vllm/forward_context.py:199-201, throughoverride_forward_contextthe context manager saves the old value on entry and restores it on exit📎 vllm/forward_context.py:263-274。set_forward_contextis a higher-level wrapper📎 vllm/forward_context.py:277-394, which additionally handles DP metadata construction, automatic batch descriptor creation, and platform-specific kwargs injection.

Step-by-Step: fromexecute_modelto model forward

Scenario:GPUModelRunner.execute_modelall input tensors are ready, and the model is about to be called.

Inexecute_model,set_forward_contextis called📎 vllm/v1/worker/gpu_model_runner.py:4408-4420：

with (
    set_forward_context(
        attn_metadata,
        self.vllm_config,
        num_tokens=num_tokens_padded,
        num_tokens_across_dp=num_tokens_across_dp,
        cudagraph_runtime_mode=cudagraph_mode,
        batch_descriptor=batch_desc,
        ubatch_slices=ubatch_slices_padded,
        slot_mapping=slot_mappings,
        skip_compiled=has_encoder_input,
        is_padding=is_padding,
    ),
    ...
):
    model_output = self._model_forward(...)

set_forward_contextinternally first constructsDPMetadata(if DP or sequence-parallel MoE is enabled)📎 vllm/forward_context.py:299-328, then callscreate_forward_contextto constructForwardContextinstance📎 vllm/forward_context.py:347-358, and finally sets the global variable throughoverride_forward_context📎 vllm/forward_context.py:361-362。

Model layers readget_forward_context()through📎 vllm/forward_context.py:208-214. If not set, the assertion fails and prompts to useset_forward_context。

sequenceDiagram
    participant EC as EngineCore
    participant EX as Executor
    participant W as Worker
    participant MR as GPUModelRunner
    participant FC as ForwardContext
    participant M as Model Layers

    EC->>EX: execute_model(SchedulerOutput)
    EX->>W: collective_rpc("execute_model", args)
    W->>MR: execute_model(scheduler_output)
    MR->>MR: _update_states(scheduler_output)
    MR->>MR: _prepare_inputs(...)
    MR->>MR: _get_slot_mappings(...)
    MR->>FC: set_forward_context(attn_metadata, slot_mapping, ...)
    FC-->>MR: context manager entered
    MR->>M: _model_forward(input_ids, positions, ...)
    M->>FC: get_forward_context()
    FC-->>M: ForwardContext
    M-->>MR: hidden_states
    MR->>MR: compute_logits(sample_hidden_states)
    MR-->>W: ExecuteModelState / None
    W-->>EX: ModelRunnerOutput
    EX-->>EC: output[0]

Design considerations

is_paddingThe design of the📎 vllm/forward_context.py:162-165field is noteworthy: the comment says "consumers can use it to skip work on padding tokens." This is an optimization in the CUDA graph scenario—padding rows participate in graph capture but should not produce actual computation.

all_moe_layersandmoe_layer_indexare a clever pair of workarounds📎 vllm/forward_context.py:170-195. The comment explains the problem in detail:vllm.moe_forwardcustom operators hardcode the layer name string into the graph, causing torch.compile cold start time to be too long. The solution is to store the layer name list inForwardContext, and the custom operator pops strings in order and increments a counter. The comment also admits that this relies on the assumption that "custom operators execute in order and torch.compile will not reorder"📎 vllm/forward_context.py:182-184。

Design considerations and production pitfalls

State consistency under asynchronous scheduling. _update_statesUnder asynchronous speculative decoding,output_token_idsadopts an "optimistic assumption" strategy: assume all draft tokens from the previous step are accepted, first expand📎 vllm/v1/worker/gpu_model_runner.py:1376-1384, then register a deferred correction function📎 vllm/v1/worker/gpu_model_runner.py:1509-1510. The correction function is called after the model forward startsnum_computed_tokens 📎 vllm/v1/worker/gpu_model_runner.py:1547-1558, reads the actual accepted count from the GPU, and rolls back

_may_reorder_batch. The brilliance of this design is that the correction happens after "the batch has started," without blocking the forward pass, preserving the continuity of the asynchronous pipeline.Trigger condition forkv_cache_groups. This method first checks📎 vllm/v1/worker/gpu_model_runner.py:1131-1132whetheris_attention_freeThe Mamba model is also attention-free, but it uses a KV cache to store internal state📎 vllm/v1/worker/gpu_model_runner.py:1116-1139. Only models that truly have no KV cache group skip the reordering.

_prepare_input_idsindexing calculation pitfalls.When the batch contains both decode requests from the previous step and new requests,num_common_tokens < total_without_spec, you need to first copy the CPU tensor before scattering📎 vllm/v1/worker/gpu_model_runner.py:1849-1854. Ifnum_common_tokens == 0, it means no request overlaps with the previous step, so return directly📎 vllm/v1/worker/gpu_model_runner.py:1855-1858. Distinguishing these two branches is critical—missing either one will causeinput_idssome parts to remain uninitialized.

AsyncGPUModelRunnerOutputstream synchronization.The output copy is performed on a separate CUDA stream📎 vllm/v1/worker/gpu_model_runner.py:308-328, usingblocking=True's Event to avoid busy-polling the CUDA driver lock📎 vllm/v1/worker/gpu_model_runner.py:296-298。get_output(), first synchronize then release the device tensor reference📎 vllm/v1/worker/gpu_model_runner.py:336-340, the order cannot be reversed—otherwise the tensor may be reclaimed before the copy completes.

Chapter Summary

This chapter tracedSchedulerOutput's complete path from EngineCore to GPU forward pass.ExecutorThroughcollective_rpc, the scheduling results are broadcast to all Workers,GPUModelRunner's_update_statessynchronizes cache state,_prepare_inputsconstructs input tensors,_get_slot_mappingsgenerates KV slot mappings, and finallyset_forward_contextinjects the batch description into the global context for consumption by each model layer. The asynchronous scheduling path maintains pipeline continuity through optimistic assumptions + deferred correction, whileForwardContext's global singleton design resolves the contradiction between fixed model-layer signatures and cross-layer metadata injection.

Chapter Review Questions

At this point, we have completed the full path from SchedulerOutput to GPU forward propagation: Executor dispatch, Worker execution, GPUModelRunner translating the logical manifest into physical tensors, and injecting the batch description into each layer through forward_context. However, the most time-consuming part of model forward propagation—the attention computation—has not yet been unfolded. The next chapter will dive into the attention backend, examining how the block table and slot mapping in attn_metadata are consumed by the PagedAttention kernel, and how different backends such as FlashAttention, FlashInfer, and Triton are selected and scheduled through a unified interface.

In the previous chapter, we saw how GPUModelRunner translates scheduling results into physical tensors such as input_ids, slot_mapping, and block_table, and injects them into each layer through forward_context. But the real heavyweight that consumes GPU time—attention computation—is still up in the air. Who exactly consumes those tensors in attn_metadata? Why can implementations like FlashAttention, FlashInfer, and Triton be swapped under the same set of model code? The answer lies in the AttentionBackend abstraction layer. It decouples "how attention is computed" from "how the model calls it": the model layer only holds an AttentionImpl reference and calls the unified forward(query, key, value, kv_cache, attn_metadata, output); the specific backend is responsible for translating block_table, slot_mapping, seq_lens into parameters that its own kernel can consume. This chapter uses FlashAttentionBackend as the main thread because it simultaneously covers the richest branches, including PagedAttention's gather semantics, CUDA Graph compatibility, cascade attention, and DCP distributed context. Once you understand it thoroughly, other backends are just variants of parameter mapping. The design motivation for this "backend registration + unified interface" is straightforward: attention kernels evolve extremely quickly (FA2→FA3→FA4, FlashInfer iterations, in-house Triton), and if the model layer directly depended on a specific kernel, every kernel upgrade would require changing the model code. The abstraction layer isolates change behind a single factory method, get_impl_cls().

Backend Selection: Capability Declaration and Metadata Construction

Intuitive Model

Think ofAttentionBackendas a job posting: it does not do the work itself, it only declares "which dtypes, which head_sizes, which KV cache quantization formats, and which attention types I can handle." The scheduler takes the model configuration and matches it; if matching fails, it moves on to the next candidate. Without this layer of declaration, the system would only discover at runtime that "this head_size is not supported by the kernel" and crash immediately.

Capability Matrix: Fields Are Contracts

FlashAttentionBackendThe class attributes of are its capability boundary.supported_dtypesrestricts fp16/bf16📎 vllm/v1/attention/backends/flash_attn.py:287-287；supported_kv_cache_dtypesadditionally allows the fp8 family📎 vllm/v1/attention/backends/flash_attn.py:298-299. But "declaring support" does not equal "unconditional support"—supports_kv_cache_dtypefor quantized KV, it further delegates toflash_attn_supports_kv_cache_dtypeto make device-related judgments📎 vllm/v1/attention/backends/flash_attn.py:431-438。

Even more fine-grained issupports_combination: it receives a whole set of combined parameters such as head_size, dtype, block_size, use_mla, has_sink, and returnsNoneto indicate availability, or returns a string to indicate the reason for rejection📎 vllm/v1/attention/backends/flash_attn.py:454-507. For example, sink is rejected on compute capability < 9.0📎 vllm/v1/attention/backends/flash_attn.py:467-468, and on SM90, FP8 KV with mm_prefix must go through Triton📎 vllm/v1/attention/backends/flash_attn.py:472-472. This "return a reason string" design allows upper layers to produce diagnosable errors rather than silently falling back.

The choice of block_size is likewise capability-driven. By default it returnsMultipleOf(16), but SM90 FP8-KV forces 64📎 vllm/v1/attention/backends/flash_attn.py:297-324, and FA4's head_size=256 kernel forcesFA4_HD256_PAGE_SIZE 📎 vllm/v1/attention/backends/flash_attn.py:326-352. This explains why the KV cache block size is not arbitrary—it is inversely constrained by the kernel's TMA tile size.

Metadata Structure: Field Layout of FlashAttentionMetadata

FlashAttentionMetadatais a dataclass, and its fields fall into four groups📎 vllm/v1/attention/backends/flash_attn.py:511-566：

The first group is the basic batch description:num_actual_tokens(the real number of tokens after removing padding),max_query_len、query_start_loc(prefix sums, used by varlen kernels to locate the start and end of each sequence),seq_lens、block_table、slot_mapping 📎 vllm/v1/attention/backends/flash_attn.py:520-526. Note the ASCII diagram in the source comments📎 vllm/v1/attention/backends/flash_attn.py:512-518, which precisely distinguishescontext_len(historical KV),query_len(newly added in this step),seq_len(the sum of the two)—this is the key to understanding varlen kernel parameters.

The second group is cascade attention fields:use_cascade、common_prefix_len、cu_prefix_query_lens, etc.📎 vllm/v1/attention/backends/flash_attn.py:528-533。

The third group is DCP (Decode Context Parallel) fields:max_dcp_context_kv_len、dcp_context_kv_lens, as well as counters distinguishing the number of decode/prefill requests📎 vllm/v1/attention/backends/flash_attn.py:535-544。

The fourth group is optional scheduling and special masks:scheduler_metadata(used by FA3 AOT scheduling),causal(can be bool or a tensor, supporting per-sequence causality),mm_prefix_query_range_tensor(multimodal bidirectional ranges), R-SWA related fields📎 vllm/v1/attention/backends/flash_attn.py:546-566。

Step-by-Step of build()

Scenario: a mixed batch, 3 decode sequences + 2 prefill sequences, no cascade, no DCP.

Step one, fromcommon_attn_metadataunpack the basic tensors📎 vllm/v1/attention/backends/flash_attn.py:824-832. Step two, decide whether to enable AOT scheduling:aot_schedule = self.aot_schedule and not fast_build and not envs.VLLM_BATCH_INVARIANT 📎 vllm/v1/attention/backends/flash_attn.py:836-838。self.aot_scheduleIn__init__determined byget_flash_attn_version() == 3— only FA3 supports precomputed scheduling metadata. Step three, lazily populate on first build📎 vllm/v1/attention/backends/flash_attn.py:709-709: iterate over allaot_sliding_windowlayers to collect sliding window configurations; if the configuration is unique, adopt it; if there is more than one, disable AOTFlashAttentionImplStep four, compute📎 vllm/v1/attention/backends/flash_attn.py:848-851。

. Default is 0 (let FA3 use heuristics); only set it when full CUDA graph is enabled and the token count falls within the capture rangemax_num_splits. The comment explains why:self.max_num_splits 📎 vllm/v1/attention/backends/flash_attn.py:856-866will allocatenum_splits > 1intermediate buffers, which is expensive in GPU memory, and is only worth it in CUDA graph scenarios[num_splits, num_heads, num_tokens, head_size]Step five, take the non-cascaded non-DCP branch, call📎 vllm/v1/attention/backends/flash_attn.py:862-865。

to generate FA3's scheduling metadata_get_scheduler_metadata. Step six,📎 vllm/v1/attention/backends/flash_attn.py:976-986handles the CUDA graph scenario: copy the new metadata into the preallocated buffer, and zero out the remaining part_store_scheduler_metadata. The zeroing step is critical — the comment explicitly points out that otherwise some thread blocks will read invalid metadata and overwrite the output buffer📎 vllm/v1/attention/backends/flash_attn.py:671-684Step seven, construct📎 vllm/v1/attention/backends/flash_attn.py:671-672。

and returnFlashAttentionMetadataCopy📎 vllm/v1/attention/backends/flash_attn.py:992-1015。

flowchart TD
    start["build(common_prefix_len, common_attn_metadata)"] --> unpack["解包 query_start_loc / seq_lens / block_table / slot_mapping"]
    unpack --> aot{"aot_schedule 且非 fast_build 且非 BATCH_INVARIANT?"}
    aot -->|是| sw_check{"aot_sliding_window 已初始化?"}
    aot -->|否| maxsplit
    sw_check -->|否, 首次| collect["_get_sliding_window_configs 收集层滑窗"]
    collect --> sw_unique{"配置数量 == 1?"}
    sw_unique -->|是| set_sw["设置 aot_sliding_window"]
    sw_unique -->|否, >1| disable_aot["self.aot_schedule = False"]
    set_sw --> maxsplit
    disable_aot --> maxsplit
    sw_check -->|是| maxsplit["计算 max_num_splits"]
    maxsplit --> cg_check{"use_full_cuda_graph 且 tokens <= max_cudagraph_size?"}
    cg_check -->|是| set_splits["max_num_splits = self.max_num_splits"]
    cg_check -->|否| zero_splits["max_num_splits = 0"]
    set_splits --> branch
    zero_splits --> branch
    branch{"dcp_world_size > 1?"}
    branch -->|是| dcp_path["计算 dcp_context_kv_lens, 可能 skip"]
    branch -->|否| cascade_check{"common_prefix_len > 0?"}
    cascade_check -->|是| cascade_path["构造 prefix/suffix 双份 scheduler_metadata"]
    cascade_check -->|否| normal_path["_get_scheduler_metadata 单份"]
    dcp_path --> store
    cascade_path --> store
    normal_path --> store
    store["_store_scheduler_metadata: CUDA graph 时拷入预分配缓冲并清零尾部"] --> build_meta["构造 FlashAttentionMetadata"]
    build_meta --> mm_check{"mm_req_doc_ranges 非空?"}
    mm_check -->|是| fill_mm["fill_mm_prefix_query_ranges + 拷贝到 GPU"]
    mm_check -->|否| rswa_check
    fill_mm --> rswa_check{"rswa_window 非空?"}
    rswa_check -->|是| copy_rswa["拷贝 prefix_lens 到持久缓冲"]
    rswa_check -->|否| done
    copy_rswa --> done["返回 attn_metadata"]

---

Intuitive model

is the backend's "final assembly workshop": it takes the Q/K/V computed by the model layers, the KV cache tensors, and the metadata built in the previous step, adjusts the physical layout of the KV cache into the shape expected by the kernel, and then dispatches to the specific kernel. Without this step, the kernel would read the wrong memory layout and produce silent errors in the output — harder to debug than a crash.

forward()KV cache memory layout transformation

vLLM's KV cache physical shape is

— K and V are concatenated in the last dimension[num_blocks, num_kv_heads, block_size, 2 * head_size]. But the FlashAttention kernel expects K and V to be separate, with layout📎 vllm/v1/attention/backends/flash_attn.py:1246-1247The transformation happens at the beginning of[num_blocks, block_size, num_kv_heads, head_size]。

:forward()turnskv_cache.transpose(1, 2).split(self.head_size, dim=-1) 📎 vllm/v1/attention/backends/flash_attn.py:1310-1310。transpose(1,2)into[blocks, heads, block_size, 2D]splitting K and V along the last dimension. Note that[blocks, block_size, heads, 2D]，splitonly changes strides without moving data, so subsequent kernels must support non-contiguous access.transposeImmediately after that is

. The comment clarifies the motivation: whencanonicalize_singleton_dim_strides 📎 vllm/v1/attention/backends/flash_attn.py:1310-1310(common in TP scenarios), the stride of size-1 dimensions is degenerate, while FA3/FA4 on H100+ use TMA, which requires strides to be at least 16-byte alignednum_kv_heads=1. This is a typical "logically equivalent, physically invalid" trap.📎 vllm/v1/attention/backends/flash_attn.py:1310-1310Parameter flow in the non-cascaded path

After entering the

branch, parameters are mapped one by oneif not attn_metadata.use_cascadetakes📎 vllm/v1/attention/backends/flash_attn.py:1326-1342：cu_seqlens_q = query_start_loc，seqused_k = seq_lens，block_table = attn_metadata.block_table。descale_shape, used for scale broadcasting in FP8 quantization — the comment states that flash-attn expects the descale shape to be(batch_size, num_kv_heads), using(num_sequences, num_kv_heads)to avoid copying.expand()Then comes the symmetrization of the sliding window.📎 vllm/v1/attention/backends/flash_attn.py:1258-1258。

The logic of_maybe_symmetrize_window: causal sliding window(w, 0)in non-causal scenarios must become(w, w), allowing bidirectional queries to look in both directions📎 vllm/v1/attention/backends/flash_attn.py:587-589. The comment also emphasizes that "a layer's own window takes precedence over the group's window," because a single KV cache group may simultaneously contain windowed layers and global layers (e.g., when Gemma-3 disables the hybrid KV cache manager)📎 vllm/v1/attention/backends/flash_attn.py:1362-1365。

Mask branch: mm_prefix and R-SWA

Whenmm_prefix_query_rangesis non-empty and the FA4 + static causal conditions are met, the code constructs CuTE-DSL'smask_mod 📎 vllm/v1/attention/backends/flash_attn.py:1374-1407. The key actions arecausal = Falseandsliding_window_size = None 📎 vllm/v1/attention/backends/flash_attn.py:1406-1407. The comment explains why: the semantics of mm_prefix is(causal ∧ window) ∨ bidirectional-range, not a subset of causal; after FA #155, setting mask_mod no longer automatically clears causal/local, and the caller must explicitly disable them, otherwise the built-in causal path will short-circuit mask_mod📎 vllm/v1/attention/backends/flash_attn.py:1402-1405。

_make_mm_prefix_mask_modusesfunctools.cacheto cache📎 vllm/v1/attention/backends/flash_attn.py:1793-1802. The comment gives a hardcore reason: FA4'shash_callablewill mix the closure unit'srepr()into the compilation key, and nested_load_q_rangehave different addresses on each call, causing a full JIT recompilation on every forward📎 vllm/v1/attention/backends/flash_attn.py:1793-1802. This is a typical example of a production environment performance trap.

There is a coordinate conversion detail inside the mask: FA4 passes localq_idx(0-based within the current prefill chunk), whilekv_idxis an absolute position. The code usesq_abs = q_idx + seqlen_k - seqlen_qto recover the absolute position📎 vllm/v1/attention/backends/flash_attn.py:1859-1865。__vec_size__ = 1The setting of_load_q_rangealso has its rationale:📎 vllm/v1/attention/backends/flash_attn.py:1897-1897。

reads lane 0, and a single call cannot span query rowscausal & (in_prefix | in_window) 📎 vllm/v1/attention/backends/flash_attn.py:1945-1948R-SWA's mask_mod is similar, but its semantics isuse_fast_sampling = True, and📎 vllm/v1/attention/backends/flash_attn.py:1950-1950。

lets FA4 skip KV blocks that are fully masked, without loading their data

Special handling for FA4 hd256self.fa4_hd256Whennum_pages = cdiv(max_seqlen_k, FA4_HD256_PAGE_SIZE)，max_seqlen_kis true, the code enforces page alignment:block_tablerounds up to the page boundary,num_splits = 1 📎 vllm/v1/attention/backends/flash_attn.py:1442-1448truncates to the exact number of pages,

. The comment states that the hd256 kernel requires page-aligned length, an exact-width block table, and does not support SplitKV._FA4_DENSE_ATTENTION_KERNEL(...)Finally calls📎 vllm/v1/attention/backends/flash_attn.py:1450-1475。

, passing q, k, v, out, cu_seqlens_q, seqused_k, block_table, softcap, mask_mod, aux_tensors, etc. all together

forward()KV cache write: do_kv_cache_updatedo_kv_cache_updateonly reads the KV cache; writes are done byreshape_and_cache_flash. It callsslot_mapping, using📎 vllm/v1/attention/backends/flash_attn.py:1532-1541to scatter the newly computed K/V into the cachekey/value. The comment points out:slot_mappingNo, but manual slicing is not needed, because the op usesslot_mapping's shape to determine the actual token count📎 vllm/v1/attention/backends/flash_attn.py:1527-1531. No stride normalization is done here, because no TMA kernel is involved📎 vllm/v1/attention/backends/flash_attn.py:1520-1521。

sequenceDiagram
    participant Model as 模型层 Attention
    participant Impl as FlashAttentionImpl
    participant KVC as kv_cache 张量
    participant Kernel as flash_attn_varlen_func
    Model->>Impl: forward(query, key, value, kv_cache, attn_metadata, output)
    Impl->>Impl: output_scale 非空? 抛 NotImplementedError
    Impl->>Impl: attn_metadata is None? 返回 output.fill_(0)
    Impl->>KVC: transpose(1,2).split(head_size)
    KVC-->>Impl: key_cache, value_cache
    Impl->>Impl: canonicalize_singleton_dim_strides(key_cache)
    Impl->>Impl: use_cascade?
    alt 非级联
        Impl->>Impl: 映射 cu_seqlens_q / seqused_k / block_table
        Impl->>Impl: _maybe_symmetrize_window
        Impl->>Impl: mm_prefix 或 R-SWA? 构造 mask_mod
        Impl->>Kernel: _FA4_DENSE_ATTENTION_KERNEL(q, k, v, out, ...)
        Kernel-->>Impl: output 就地写入
    else 级联
        Impl->>Kernel: cascade_attention(prefix + suffix 两次调用)
        Kernel-->>Impl: merge_attn_states 合并
    end
    Impl-->>Model: output

---

Design thinking: why it is written this way

CUDA Graph compatibility is an invisible constraint on metadata design。_store_scheduler_metadata's "copy in + zero the tail" pattern📎 vllm/v1/attention/backends/flash_attn.py:671-684repeatedly appears in the R-SWA persistent buffer📎 vllm/v1/attention/backends/flash_attn.py:787-798and the mm_prefix staging area📎 vllm/v1/attention/backends/flash_attn.py:800-813. The common pattern is: preallocate the maximum-size persistent buffer in__init__, and only copy without allocating inbuild(). The reason is stated in the comments - no allocation operations are allowed during CUDA graph capture📎 vllm/v1/attention/backends/flash_attn.py:1044-1046。

Mutual exclusion between DCP and fused draft decode。supports_draft_decode_metadata_update = self.dcp_world_size == 1 📎 vllm/v1/attention/backends/flash_attn.py:742-742. The comments explain: fused draft decode reuses the captured metadata object across draft steps, but DCP's build-time host-side decisions (such asskip_dcp_context_attention()) change the metadata shape, and these Python fields are not refreshed in place between graph replays📎 vllm/v1/attention/backends/flash_attn.py:736-741. This is a typical trade-off of "choosing correctness when performance optimization conflicts with correctness."

Heuristic thresholds for cascade attention。use_cascade_attentionuses a series of threshold filters: common_prefix_len < 256 is rejected directly📎 vllm/v1/attention/backends/flash_attn.py:1967-1967, alibi/sliding_window/local_attention are not supported📎 vllm/v1/attention/backends/flash_attn.py:1978-1979, request count < 8 is rejected📎 vllm/v1/attention/backends/flash_attn.py:1982-1984, and the DCP scenario is disabled📎 vllm/v1/attention/backends/flash_attn.py:1985-1987. After passing, a rough performance model is still used to compare the CTA count and wave count of cascade and FlashDecoding📎 vllm/v1/attention/backends/flash_attn.py:2011-2029. The comments admit that this model is "very rough"📎 vllm/v1/attention/backends/flash_attn.py:2009-2010。

Production pitfalls：forward()contains a prominent comment warning that under piece-wise CUDA graph this method executes in eager mode,view/sliceand methods that appear to have no GPU operations are actually very slow, so changes must be benchmarked📎 vllm/v1/attention/backends/flash_attn.py:1277-1284. This explains why the code heavily uses[:num_actual_tokens]slicing rather than more "elegant" forms - every place is the result of a performance trade-off.

---

Chapter summary

This chapter followsFlashAttentionBackendthrough the complete lifecycle of the attention backend: capability declaration (supports_*series) -> metadata construction (build()translatesCommonAttentionMetadataintoFlashAttentionMetadata) -> kernel invocation (forward()transforms the KV cache layout, constructs masks, and dispatches to the FA kernel). The core mechanisms include: the KV cachetranspose+splitlayout transformation, normalization of degenerate strides, the persistent buffer pattern under CUDA graph, CuTE-DSL mask construction for mm_prefix/R-SWA, and heuristic decisions for cascade attention.

Key design principles: separation of capability declaration and implementation, CUDA graph compatibility driving metadata preallocation, and prioritizing correctness when performance optimization conflicts with correctness (DCP disables fused draft decode).

The next chapter turns to sampling and output:logitshow

becomes tokens through the processor chain (temperature, top-p, penalties), how structured output constrains decoding, and how streaming return cooperates with the scheduler.

At this point, the complete chain from the abstract interface to the kernel implementation of the attention backend has been connected: the model layer calls uniformly through AttentionImpl, the backend is responsible for translating metadata such as block_table and slot_mapping into concrete kernel parameters, and FlashAttentionBackend's PagedAttention implementation demonstrates the gather semantics under paged KV Cache and the CUDA Graph compatibility strategy. But the attention computation produces only hidden states, and what the model ultimately needs to output is the next token. How do these hidden states become logits, and how do the logits go through sampling and post-processing to finally be returned to the client as streaming text? The next chapter will trace this last mile.

In the previous chapter, we traced how the attention backend translates the block table into kernel parameters and completes gather-style attention computation on non-contiguous memory. But attention produces only hidden states—what the model truly needs to deliver to the user is the text of the next token. This chapter traces this last mile: after hidden states are projected into logits by lm_head, how do they pass through a carefully ordered processor chain (temperature, penalties, top-k/top-p, structured constraints), get sampled into token ids, and then be restored to text by the detokenizer and pushed in streaming fashion. If any step in this chain is out of order or leaks state, output quality will silently degrade.

Sampler: The order of the processor chain is correctness

Intuitive model: The Sampler is like an assembly line, and logits are the rough workpiece to be processed. Each station (processor) on the line modifies the workpiece, and the order of the stations directly determines the finished product—cutting first and then grinding is not the same as grinding first and then cutting. Without this chain, the model could only output the raw probability distribution, and what the user gets would be "bare sampling" with no ability to control temperature, suppress repetition, or constrain format.

Data structures and memory layout

The Sampler itself isnn.Module, but its core state is extremely thin: it only holds thetopk_topp_samplersubmodule,logprobs_modeand theuse_fp64_gumbelflag📎 vllm/v1/sample/sampler.py:61-64. The real batch-level state is all encapsulated inSamplingMetadataand passed in through the forward parameters. This design of "stateless Sampler + external metadata" is deliberate: the Sampler instance is created only once during the engine lifecycle, while the batch composition of each decode step is changing, so externalizing the state is the only way to let the Sampler be safely replayed after being captured by CUDA Graph.

The key constant is_SAMPLING_EPS = 1e-5 📎 vllm/v1/sample/sampler.py:18. It simultaneously serves two semantics: a temperature below this value is treated as greedy, andapply_temperatureprovides the fallback to prevent division by zero.

Step-by-Step Walkthrough

Scenario: a batch mixes greedy requests and random sampling requests, and some requests also enable logprobs.

Step one, snapshot the original logprobs.Before applying any penalty or temperature, if the request needs logprobs, first followlogprobs_modeto determine the snapshot content📎 vllm/v1/sample/sampler.py:84-93. Note that the comment explicitly points out the difference from V0: V1 usesraw logits(before penalties and temperature) to compute top-k logprobs.📎 vllm/v1/sample/sampler.py:72-77. This is the semantic contract—the logprob the user sees should reflect the model's true distribution, not a distribution distorted by penalties.

Step two, unify to float32. 📎 vllm/v1/sample/sampler.py:95-96Regardless of whether the input is bf16 or fp16, upcast to float32. The reason is that subsequent log_softmax, top-k, and cumulative probabilities accumulate errors under low precision, especially when the vocab reaches 150,000.

Step three, the non-argmax-invariant processor chain. apply_logits_processorsApply in sequence: allowed token whitelist mask, bad words exclusion,non_argmax_invariantprocessors, penalty terms📎 vllm/v1/sample/sampler.py:391-404. The classification here is the core design—non_argmax_invariantrefers to thosethat change the greedy resultprocessors (such as min_tokens, logit_bias), which must take effect before greedy sampling; whileargmax_invariantprocessors (such as min_p) do not change the argmax and can be deferred until after temperature.

Step four, sampling. sampleThe method first determines whether it is fully random📎 vllm/v1/sample/sampler.py:256-271: ifall_greedy, directly return argmax; otherwise first compute the greedy result for later use, then apply temperature, argmax-invariant processors, top-k/top-p📎 vllm/v1/sample/sampler.py:275-291. Finally usetorch.whereto select between the greedy and random results according to the temperature threshold📎 vllm/v1/sample/sampler.py:305-306, and reuse thegreedy_sampledtensor as the output buffer to avoid extra allocation.

Step five, collect logprobs and package the output.According tonum_logprobs, there are three cases: None returns only the logprobs of the specified token; -1 returns the full unsorted logprobs; otherwise top-k📎 vllm/v1/sample/sampler.py:120-131. Finally, the token id is converted to int32 to compress the size, and expanded to[num_requests, 1]a two-dimensional tensor📎 vllm/v1/sample/sampler.py:138-148。

flowchart TD
    in_logits["logits (bf16/fp16)"] --> snap{"需要 logprobs?"}
    snap -->|是| raw["compute_logprobs / clone<br/>raw_logprobs 快照"]
    snap -->|否| f32
    raw --> f32["logits.to(float32)"]
    f32 --> proc["apply_logits_processors"]
    proc --> mask{"allowed_token_ids_mask?"}
    mask -->|是| fill["masked_fill_(-inf)"]
    mask -->|否| bad
    fill --> bad{"bad_words_token_ids?"}
    bad -->|是| apply_bad["apply_bad_words"]
    bad -->|否| noninv
    apply_bad --> noninv["non_argmax_invariant 处理器"]
    noninv --> pen["apply_penalties"]
    pen --> sample["sample()"]
    sample --> allg{"all_greedy?"}
    allg -->|是| greedy["greedy_sample (argmax)"]
    allg -->|否| temp["apply_temperature"]
    temp --> arginv["argmax_invariant 处理器"]
    arginv --> topp["topk_topp_sampler"]
    topp --> where["torch.where(temp < EPS)"]
    greedy --> out
    where --> out["SamplerOutput<br/>sampled_token_ids"]

Design reflections and pitfalls

Why must penalty terms come before temperature?Temperature is a scaling of the distribution, while penalties are additions or subtractions to specific tokens. If scaling is done first and then penalties are applied, the absolute magnitude of the penalty will be amplified or reduced by the temperature, causing the same set of penalty parameters to behave inconsistently at different temperatures. V1 fixes penalties before temperature, ensuring the stability of parameter semantics.

mark_unbackedcompilation trap.Ingather_logprobs,batched_count_greater_thanis compiled, and when the batch dimension changes from 1 to ≥2, it triggers dynamo's 0/1 specialization recompilation📎 vllm/v1/sample/sampler.py:345-348。mark_unbackedmarks that dimension as fully symbolic to avoid this recompilation. In production, if you see a sudden one-time stall after the first decode request, it is very likely this kind of recompilation.

gpu_sync_allowedsynchronization boundary. batched_count_greater_thanmay internally trigger GPU synchronization, and vLLM usesgpu_sync_allowed(first_only=True)context to explicitly declare "synchronization is allowed here, but only for the first time"📎 vllm/v1/sample/sampler.py:345-348. If synchronization unexpectedly occurs inside a CUDA Graph capture region, it will cause capture failure—this is a key clue for troubleshooting graph capture issues.

Structured output: dual-track state machine of bitmask and grammar

Intuitive model: structured output is like putting a pair of "grammar glasses" on the sampler—at each step it can only see tokens that conform to the JSON schema or grammar. Without it, the model may generate syntactically invalid JSON, and the downstream parser crashes directly. The essence of vLLM's implementation is that the grammar state machine advances on the CPU side, while constraints are passed to the GPU-side sampler in the form of a bitmask.

Data structures and memory layout

StructuredOutputManageris an engine-level singleton, holdingbackend(one of xgrammar/guidance/outlines/lm-format-enforcer),reasoner_clsand two thread pools📎 vllm/v1/structured_output/__init__.py:39-98。

The bitmask is the core data structure:_grammar_bitmaskis an int32 tensor of shape[max_batch_size * (1 + max_num_spec_tokens), vocab_size/32]📎 vllm/v1/structured_output/__init__.py:327-336. Each bit corresponds to whether a token is legal._full_mask = torch.tensor(-1, dtype=torch.int32)represents "all 1s"—all tokens are legal📎 vllm/v1/structured_output/__init__.py:59。

The two thread pools have clear division of labor:executoris responsible for grammar compilation (CPU-intensive, with the number of workers being half the number of CPUs)📎 vllm/v1/structured_output/__init__.py:71-78；executor_for_fillmaskis responsible for parallel filling of large-batch bitmasks, enabled only when the batch exceeds 128📎 vllm/v1/structured_output/__init__.py:62-69。

Step-by-Step Walkthrough

Grammar initialization.When a request enters for the first time,grammar_initis called📎 vllm/v1/structured_output/__init__.py:115-176. If the backend is not initialized, select an implementation according to the configuration📎 vllm/v1/structured_output/__init__.py:130-165. Then submit the compilation task: by default it uses asynchronousexecutor.submit, but inexternal_launchermode it must be synchronous📎 vllm/v1/structured_output/__init__.py:167-176。

Bitmask generation.At each decode step,grammar_bitmaskgenerates masks for all structured requests in the batch📎 vllm/v1/structured_output/__init__.py:314-442. Large batches use the parallel path: submitted to the thread pool in groups of 16📎 vllm/v1/structured_output/__init__.py:346-373. Small batches use the serial path, advancing the grammar state token by token📎 vllm/v1/structured_output/__init__.py:374-433。

Mask alignment under speculative decoding.This is the most ingenious part. When there are draft tokens, each request needs1 + max_num_spec_tokensrows of masks. The serial path processes token by token: if a draft token is rejected by the grammar, recordfailed_index, and subsequent rows directly copy that row's mask📎 vllm/v1/structured_output/__init__.py:396-418. This ensures that "after a draft is rejected, the constraint state of subsequent positions rolls back to the rejection point."

State rollback.During bitmask filling, the grammar state has been advanced bystate_advancementssteps, but the draft token has not yet been truly accepted, so it mustgrammar.rollback(state_advancements)roll back📎 vllm/v1/structured_output/__init__.py:422-430. True acceptance happens ataccept_tokens 📎 vllm/v1/structured_output/__init__.py:444-466。

sequenceDiagram
    participant Sched as Scheduler
    participant Mgr as StructuredOutputManager
    participant Pool as executor_for_fillmask
    participant Gram as StructuredOutputGrammar
    participant GPU as GPU Runner

    Sched->>Mgr: grammar_bitmask(requests, ids, spec_tokens)
    Mgr->>Mgr: allocate_token_bitmask(max_batch*(1+spec))
    alt batch > 128 且无投机
        Mgr->>Pool: _async_submit_fill_bitmask(batch)
        Pool->>Gram: fill_bitmask(bitmask, index)
        Gram-->>Pool: 写入合法 token 位
        Pool-->>Mgr: Future.result()
    else 小 batch 或含投机
        loop 每个 req 的每个 spec token
            Mgr->>Gram: fill_bitmask(bitmask, cumulative_index)
            Mgr->>Gram: accept_tokens(req_id, [token])
            Gram-->>Mgr: True/False
            Note over Mgr: 失败则记录 failed_index<br/>后续行复制该行
        end
        Mgr->>Gram: rollback(state_advancements)
    end
    Mgr-->>Sched: bitmask.numpy() (NDArray int32)
    Sched->>GPU: 传入采样内核

Design reflections and pitfalls

Why must external_launcher compile synchronously?The comment gives the precise reason: asynchronous compilation causesWAITING_FOR_STRUCTURED_OUTPUT_GRAMMAR → WAITINGstate transitions to occur at different times on different TP ranks, breaking the determinism assumption that external_launcher relies on📎 vllm/v1/structured_output/__init__.py:47-56This is a typical case of the conflict between distributed determinism and asynchronous optimization.

The constraint starting point under the reasoning model. _get_constraint_startDetermines from which token to begin applying grammar constraints.📎 vllm/v1/structured_output/__init__.py:220-292For models with a chain of thought, the reasoning phase should not be subject to JSON constraints; it only starts after reasoning ends.enable_in_reasoningWhen True, directly returns 0 (constrain throughout).📎 vllm/v1/structured_output/__init__.py:235-236If the reasoner supportsfind_reasoning_end_offsetuse it to precisely locate📎 vllm/v1/structured_output/__init__.py:261-267otherwise fall back to token-by-token backtracking search📎 vllm/v1/structured_output/__init__.py:287-291。

validate_tokensprefix semantics.During speculative decoding, draft tokens may violate the grammar,validate_tokensreturns the "longest legal prefix"📎 vllm/v1/structured_output/__init__.py:294-312Note that it first strips speculative padding (-1), then computes the constraint starting point, and finally performs grammar validation only on tokens within the constrained interval.

Detokenizer: The boundary game between incremental decoding and stop strings

Intuitive modelThe detokenizer is like a scribe copying character by character, translating token ids into human-readable text. The difficulty lies in: tokens and characters are not one-to-one (a token may correspond to only half a UTF-8 character), and a stop string may span multiple tokens. Without incremental decoding, the entire sequence must be decoded from scratch at each step, and the O(n²) overhead would cripple throughput.

Data structures and memory layout

IncrementalDetokenizerThe base class only holdstoken_idslist📎 vllm/v1/engine/detokenizer.py:32-33。BaseIncrementalDetokenizeradds stop-related fields:stoplist,min_tokens、include_stop_str_in_output、stop_buffer_lengthand_last_output_text_offset 📎 vllm/v1/engine/detokenizer.py:70-94。

stop_buffer_lengthis key: when the stop string is not included in the output, it equals the longest stop string length minus one📎 vllm/v1/engine/detokenizer.py:87-90This "rollback buffer" ensures that streaming output does not prematurely emit characters that might be a prefix of a stop string.

Two implementation paths:FastIncrementalDetokenizerUse the tokenizers library'sDecodeStream 📎 vllm/v1/engine/detokenizer.py:166-246；SlowIncrementalDetokenizerUse the Python-sidedetokenize_incrementally 📎 vllm/v1/engine/detokenizer.py:249-305The choice is based on tokenizers version ≥ 0.22.0 and matching tokenizer type📎 vllm/v1/engine/detokenizer.py:32-33📎 vllm/v1/engine/detokenizer.py:61-63。

Step-by-Step Walkthrough

incremental decoding. updateReceives new token ids and thestop_terminatedflag📎 vllm/v1/engine/detokenizer.py:96-142If stop terminates and does not include the stop string, the last token is excluded from decoding📎 vllm/v1/engine/detokenizer.py:107-111Then calls token by tokendecode_nextaccumulates text📎 vllm/v1/engine/detokenizer.py:117-122。

stop string detection. check_stop_stringsSearches only within the range of newly added characters📎 vllm/v1/engine/detokenizer.py:308-360The search starting point is1 - new_char_count - stop_string_len 📎 vllm/v1/engine/detokenizer.py:338this offset ensures that stop strings spanning token boundaries can also be captured. When multiple stop strings match simultaneously, choosethe one that completes earliest📎 vllm/v1/engine/detokenizer.py:342-347。

streaming output slicing. get_next_output_textBased ondeltathe parameter determines whether to return the full amount or the increment📎 vllm/v1/engine/detokenizer.py:148-163When incomplete, retainsstop_buffer_lengthcharacters without emitting📎 vllm/v1/engine/detokenizer.py:145-146uses_last_output_text_offsetto record the sent position📎 vllm/v1/engine/detokenizer.py:148-163。

exception recovery. FastIncrementalDetokenizer._protected_stepHandles two types of exceptions: OverflowError/TypeError logs and returns None📎 vllm/v1/engine/detokenizer.py:225-229for "Invalid prefix" errors,rebuilds DecodeStreamand retries📎 vllm/v1/engine/detokenizer.py:222-246The latter addresses the edge case where the tokenizer produces non-monotonic UTF-8 output.

Design considerations and pitfalls

The trade-off of stop_buffer_length.The longer the buffer, the greater the streaming latency (the time before users see text is delayed), but the less likely it is to miss a stop string spanning tokens. Taking "the longest stop string length minus one" is an exact lower bound: the prefix of any stop string is at most this long.

min_tokens and stop_check_offset.When the number of output tokens has not reachedmin_tokensstop_check_offsetis continuously pushed to the end of the text📎 vllm/v1/engine/detokenizer.py:120-122meaning this text will not undergo stop detection. This prevents the model from hitting a stop string right at the beginning and producing empty output.

The added_token_ids cache of the Fast path.Whenspaces_between_special_tokensis False, spaces between special tokens need to be suppressed📎 vllm/v1/engine/detokenizer.py:192-207The code cachesadded_token_idson the tokenizer object📎 vllm/v1/engine/detokenizer.py:195-200avoiding rebuilding the dictionary on every decode.

Design considerations

The three modules share one design philosophy:separate state advancement from constraint checking, letting the GPU side perform only stateless tensor operationsThe Sampler is stateless; the state is inSamplingMetadatathe grammar state machine advances on the CPU side, and the GPU only consumes the bitmask; the detokenizer's_last_output_text_offsetis the only streaming cursor. This separation allows every GPU-side component to be captured by CUDA Graph.

Another main thread isorder is semanticsThe order of the Sampler's processor chain, the constraint starting point of structured output, and the stop detection offset of the detokenizer—an error in any one of these orders will not crash, but will silently produce incorrect results—this is precisely what makes this kind of code hardest to debug.

Chapter summary

• The Sampler's processor chain is strictly ordered: raw logprobs snapshot → float32 → whitelist/bad words → non-argmax-invariant → penalties → temperature → argmax-invariant → top-k/top-p.
• Structured output uses a bitmask to pass CPU-side grammar state to the GPU; under speculative decoding, consistency is ensured throughfailed_indexcopying androllback.
• The Detokenizer usesstop_buffer_lengthfallback buffering to balance streaming latency and cross-token stop string detection; the Fast path relies on tokenizers ≥ 0.22.0'sDecodeStream。

Chapter Review and Self-Test

At this point, the full inference pipeline on a single machine has been connected: from attention computation to sampling output, every step directly affects the quality of the final delivered text. But when model size exceeds single-card capacity, this pipeline must span multiple devices working together. In the next chapter we leave the single machine and enter distributed parallelism: how TP, PP, and EP partition the model, and how communication primitives synchronize these sampling results across ranks.

In the previous chapter we completed the last mile of a single inference lifecycle, from logits sampling to streaming output. But when the model is too large to fit on a single card, this pipeline must be split across multiple devices for coordinated execution. The first-order question in distributed inference is not "how to partition the model," but "after partitioning, who talks to whom and in what way." vLLM assigns these two questions to the process group topology in parallel_state.py and the communicator implementation in custom_all_reduce.py, respectively. This chapter follows the chain of "group creation → partitioning → communication → load rebalancing" to unpack the parallel strategies and underlying communication primitives of TP, PP, and EP layer by layer.

8.1 Process Group Topology: How a Rank Grid Is Carved into TP/PP/DP/EP

Intuitive Model

Think of 8 GPUs as a long table with 8 seats. Tensor Parallelism (TP) requires "people at the same table to raise their glasses simultaneously," Pipeline Parallelism (PP) requires "adjacent seats to pass dishes in relay," Data Parallelism (DP) requires "different tables eat separately but reconcile at the end," and Expert Parallelism (EP) requires "tokens to be triaged by department." Without a unified seating arrangement, each module wouldnew_group, a communication misalignment occurs: "I thought you were in the TP group, but you're actually in the DP group" — once any rank is absent from a collective communication, NCCL will hang indefinitely rather than raise an error.

Data Structures and Memory Layout

GroupCoordinatoris the carrier for all of this. Its field design directly corresponds to "a single process's multiple identities across multiple parallel dimensions":

• rankis the global rank,ranksis the list of global ranks of members in this group,world_sizeis the group size📎 vllm/distributed/parallel_state.py:434-436。
• local_rankis used to bind the device,rank_in_groupis the intra-group index — the source code uses a table to precisely distinguish the two: in a 4-GPU group spanning two nodes, rank 2'slocal_rankis 0 (it is the first GPU on node 1), butrank_in_groupis 2📎 vllm/distributed/parallel_state.py:437-445。
• cpu_groupanddevice_groupexist as a pair: the former uses gloo for metadata/object communication, the latter uses NCCL for tensor communication📎 vllm/distributed/parallel_state.py:446-447。

There is a key design here:Why does every group need to maintain a CPU group?Becausebroadcast_object、send_objectoperations like this transmit Python objects (serialized bytes); using NCCL would both waste VRAM and potentially pollute the current CUDA device.barrier()The comments state this very plainly: NCCL's barrier internally is a broadcast, which secretly creates GPU tensors and can easily mess up the current device, so a CPU group must be used📎 vllm/distributed/parallel_state.py:1355-1362。

Step-by-Step：initialize_model_parallelHow to slice the grid

Consider a concrete scenario: 8 GPUs, TP=2, PP=4, DP=1. The core is to reshape the one-dimensional rank sequence into a multi-dimensional grid, then slice along each dimension.

Step one, construct the rank grid. The layout order is explicitly defined asExternalDP x DP x PP x PCP x TP 📎 vllm/distributed/parallel_state.py:2045-2060：

all_ranks = torch.arange(world_size).reshape(
    -1, data_parallel_size, pipeline_model_parallel_size,
    prefill_context_model_parallel_size, tensor_model_parallel_size,
)

Step two, slice the TP group: view the grid as(-1, tp_size)then unbind, obtaining[g0,g1],[g2,g3],... 📎 vllm/distributed/parallel_state.py:2065-2077. Note that the TP group additionally passesuse_message_queue_broadcaster=True, because the TP group needs shared-memory broadcast to distribute metadata.

Step three, slice the PP group:all_ranks.transpose(2, 4)Move the PP dimension to the last dimension before slicing, obtaining[g0,g2,g4,g6],[g1,g3,g5,g7] 📎 vllm/distributed/parallel_state.py:2175-2188. This is exactly the example given in the docstring📎 vllm/distributed/parallel_state.py:1997-1997。

Step four, slice the DP group:transpose(1, 4)then slice📎 vllm/distributed/parallel_state.py:2195-2202。

Step five, slice the EP group — there is an easily overlooked detail here: the EP group is only created under MoE models; dense models skip it entirely📎 vllm/distributed/parallel_state.py:2210-2241. The EP group's rank set is the product ofDP x PCP x TP, meaning EP reuses the physical GPUs of DP and TP rather than being an independent dimension.

flowchart TD
    start["initialize_model_parallel()"] --> grid["all_ranks = arange(world_size).reshape(-1, DP, PP, PCP, TP)"]
    grid --> tp["TP: view(-1, tp_size).unbind(0)"]
    grid --> pp["PP: transpose(2,4).reshape(-1, pp_size)"]
    grid --> dp["DP: transpose(1,4).reshape(-1, dp_size)"]
    grid --> ep_check{"model_config.is_moe?"}
    ep_check -->|是| ep["EP: transpose(1,2).reshape(-1, DP*PCP*TP)"]
    ep_check -->|否| skip["_EP 保持 None"]
    ep --> eplb_check{"enable_eplb?"}
    eplb_check -->|是| eplb["EPLB: 与 EP 同 rank 集，独立 PG"]
    eplb_check -->|否| no_eplb["_EPLB 保持 None"]
    tp --> done["logger.info_once 打印各维度 rank"]
    pp --> done
    dp --> done
    ep --> done
    skip --> done
    eplb --> done
    no_eplb --> done

Design Considerations and Pitfalls

Why does EPLB need an independent process group?The comments provide the answer: to isolate EPLB communication from the collective communication of MoE forward passes, preventing "execution-time torch.distributed" and "EPLB's torch.distributed" from deadlocking each other📎 vllm/distributed/parallel_state.py:2243-2246. This is a classic trade-off of "trading an independent communication domain for determinism" — the cost is the VRAM overhead of one extra PG, and what you get in return is that forward passes won't get stuck during weight transfers.

Synchronization Constraints of the DP Groupis the most commonly encountered pitfall in production: all ranks within the same DP group must callgeneratesimultaneously, otherwise deadlock📎 vllm/distributed/parallel_state.py:2048-2051. This is because the DP group performs all-reduce on gradients/sampling results, and any absent rank will cause the collective communication to block forever.

Destruction OrderAlso has its subtleties.destroy()First destroy the device communicator, then destroy the device_group and cpu_group📎 vllm/distributed/parallel_state.py:1380-1393. The comments explain why: the device communicator may hold collective communication workspaces that depend on these PGs (such as the FlashInfer PCIe IPC barrier), so it must be released first📎 vllm/distributed/parallel_state.py:1377-1377。

8.2 Communication Primitives: How Custom all-reduce Bypasses NCCL

Intuitive Model

NCCL's all-reduce is a "general-purpose truck" — it can carry any cargo and take any road, but its startup overhead and protocol overhead are fixed. When you need to repeatedly perform small-tensor all-reduce on an 8-GPU NVLink fully-connected machine (every attention/MLP layer in TP needs it), the "toll" of the general-purpose truck becomes non-negligible. Custom all-reduce is a "dedicated handcart": it is only enabled on the same machine, with full NVLink interconnect, and suitable tensor sizes, using a singlecudaMemcpyto replace NCCL's handshake and protocol overhead.

Data Structures and Memory Layout

CustomAllreduceThe initialization of is a combination of "capability probing + resource pre-allocation". Key fields:

• _SUPPORTED_WORLD_SIZES = [2, 4, 6, 8, 16]: only supports these group sizes📎 vllm/distributed/device_communicators/custom_all_reduce.py:113-129。
• meta_ptrs: synchronization metadata + intermediate result buffer, sizeops.meta_size() + max_size 📎 vllm/distributed/device_communicators/custom_all_reduce.py:291-294。
• buffer_ptrs: pre-registered IPC buffer; in eager mode, input tensors are first copied in before computation📎 vllm/distributed/device_communicators/custom_all_reduce.py:298-305。
• rank_data: an 8MB uint8 tensor storing the IPC buffer pointer tuples of all ranks📎 vllm/distributed/device_communicators/custom_all_reduce.py:309-315。

Why do buffers need to be pre-registered?Because CUDA Graph capture requires all addresses to be fixed at capture time.register_graph_buffersAt the end of capture, broadcast all used buffer addresses to all ranks and register them📎 vllm/distributed/device_communicators/custom_all_reduce.py:474-491。

Step-by-Step: The Decision Flow of a Single all-reduce

Consider the scenario: a certain MLP layer's output within the TP group needs all-reduce, and the input is a 4MB bf16 tensor.

Step one,custom_all_reducecheck whether it is disabled, whether it satisfiesshould_custom_ar 📎 vllm/distributed/device_communicators/custom_all_reduce.py:529-533。

Step two,should_custom_arFilter item by item: reject if world_size > 8; dtype must be fp32/fp16/bf16; byte count must be a multiple of 16; must be weakly contiguous; only continue if world_size==2 or fully interconnected📎 vllm/distributed/device_communicators/custom_all_reduce.py:493-508。

Step three, branch based on whether in CUDA Graph capture: during capture useregistered=True(address already fixed), otherwiseregistered=False(need to memcpy to pre-registered buffer first)📎 vllm/distributed/device_communicators/custom_all_reduce.py:529-545。

Step four, actually callops.all_reduce, passing inbuffer_ptrs[rank]andmax_size 📎 vllm/distributed/device_communicators/custom_all_reduce.py:519-527。

flowchart TD
    call["custom_all_reduce(input)"] --> disabled{"self.disabled?"}
    disabled -->|是| ret_none["return None → 回退 NCCL"]
    disabled -->|否| should{"should_custom_ar(input)?"}
    should -->|否| ret_none
    should -->|是| capturing{"self._IS_CAPTURING?"}
    capturing -->|是| stream_cap{"is_current_stream_capturing()?"}
    stream_cap -->|是| reg["all_reduce(registered=True)"]
    stream_cap -->|否| mimic["return empty_like(input) 模拟分配"]
    capturing -->|否| eager["all_reduce(registered=False) 先 memcpy"]
    reg --> out["返回 out 张量"]
    eager --> out

Design thinking and pitfalls

The degradation path for multi-node scenariosis the most elegant part of this code.same_nodeWhen is false,mnnvl_onlyset to true📎 vllm/distributed/device_communicators/custom_all_reduce.py:198-199, then check MNNVL (Multi-Node NVLink) capability. If not every GPU in the group supports MNNVL, directly disable custom collective communication📎 vllm/distributed/device_communicators/custom_all_reduce.py:228-233。_group_can_attempt_mnnvlUse a single CPU all-reduce (MIN operation) to ensure all ranks follow the same control flow📎 vllm/distributed/device_communicators/custom_all_reduce.py:59-73—this is the key safeguard in heterogeneous clusters to avoid "some ranks entering the MNNVL path while others go through NCCL" causing hangs.

The cost of P2P checks：_can_p2pwill iterate over all peers doinggpu_p2p_access_check, the comment says the first computation is expensive but will be cached📎 vllm/distributed/device_communicators/custom_all_reduce.py:278-278. In production, if startup is found to be slow, you can setVLLM_SKIP_P2P_CHECKto skip, directly trusting the driver's P2P report📎 vllm/distributed/device_communicators/custom_all_reduce.py:86-100。

Three-tier backend selection for reduce-scatteris worth looking at separately:_select_reduce_scatter_backendreturns by prioritymnnvl_multimem > mnnvl_lamport > legacy 📎 vllm/distributed/device_communicators/custom_all_reduce.py:601-636. The multimem path requires world_size to be in(2,4,8)and device capability to be (10,0) or (10,3) (Blackwell-class)📎 vllm/distributed/device_communicators/custom_all_reduce.py:103-104. Note thatVLLM_BATCH_INVARIANTwill disable the multimem path📎 vllm/distributed/device_communicators/custom_all_reduce.py:628—because multimem's reduction order is nondeterministic, which would break batch invariance.

8.3 EPLB: Scheduling logic for expert load rebalancing

Intuitive model

In a MoE model, 256 logical experts are distributed across 32 GPUs, 8 per GPU. But under real traffic, some "hot experts" (e.g., those handling common syntactic structures) get routed a large number of tokens, causing the GPU holding them to become a bottleneck while other GPUs sit idle. EPLB (Expert Parallel Load Balancer) is essentially "adding replicas for hot experts": copying the weights of hot experts to idle GPUs so tokens can be diverted there. Without it, MoE's actual throughput would be locked to the slowest GPU.

Data structures and memory layout

EplbModelStateuses three mapping tables to describe the "logical expert ↔ physical expert" relationship:

• physical_to_logical_map: shape(num_moe_layers, num_physical_experts), each physical slot stores the logical expert id it carries📎 vllm/distributed/eplb/eplb_state.py:105-120。
• logical_to_physical_map: shape(num_moe_layers, num_logical_experts, max_replicas+1), sparse matrix, -1 means no mapping📎 vllm/distributed/eplb/eplb_state.py:123-146。
• logical_replica_count: how many replicas each logical expert has📎 vllm/distributed/eplb/eplb_state.py:147-161。

expert_load_windowis a sliding window, shape(window_size, num_moe_layers, num_physical_experts) 📎 vllm/distributed/eplb/eplb_state.py:180-187. The comment specifically notes: now it records the load of all physical experts rather than only local experts, to ensure consistent statistics across different dispatch methods (naive all-to-all, DeepEP); under naive all-to-all, each DP rank contributes the same token set, so the load gets multiplied by dp_size📎 vllm/distributed/eplb/eplb_state.py:180-187。

Step-by-Step: The complete chain of one rebalancing

Scenario:expert_rearrangement_stepreaches the threshold, triggeringrearrange()。

Step one, map physical load back to logical experts. Usescatter_add_to aggregate byphysical_to_logical_map, invalid slots (<0) are filled into theinvalid_idxbucket and discarded at the end📎 vllm/distributed/eplb/eplb_state.py:794-816。

Step two, cross-rank all-reduce to get global logical load._allreduce_listconcatenates the loads of multiple models then does one all-reduce and splits them back, avoiding multiple communications📎 vllm/distributed/eplb/eplb_state.py:1045-1068。

Step three, call the policy to compute the new mapping.policy.rebalance_expertsruns on host, so both the load window and the current mapping must be copied back to CPU📎 vllm/distributed/eplb/eplb_state.py:859-867。

Step four, ROCm-specific "skip rebalancing" check: if the new mapping improves rank load imbalance by less than 5%, skip this rebalancing📎 vllm/distributed/eplb/eplb_state.py:869-923. This is a pragmatic optimization—rebalancing itself has communication cost, so if the benefit isn't enough, don't do it.

Step five, perform weight transfer and commit the new mapping📎 vllm/distributed/eplb/eplb_state.py:925-942。

sequenceDiagram
    participant Main as 主线程 step()
    participant Policy as DefaultEplbPolicy
    participant Comm as EplbCommunicator
    participant Async as async_worker 线程
    Main->>Main: expert_rearrangement_step >= interval
    Main->>Main: scatter_add_ 物理负载→逻辑负载
    Main->>Main: _allreduce_list 跨 rank 聚合
    Main->>Policy: rebalance_experts(load, replicas, groups, nodes, gpus, map)
    Policy-->>Main: new_physical_to_logical_map
    alt 同步模式
        Main->>Comm: rearrange_expert_weights_inplace()
        Comm-->>Main: 权重搬运完成
        Main->>Main: _commit_eplb_maps()
    else 异步模式
        Main->>Main: eplb_stats = EplbStats(...); rebalanced = True
        Main->>Async: rearrange_event.record()
        Async->>Comm: 后台搬运权重到 expert_buffer
        Async-->>Main: pending_result 就绪
        Main->>Main: _move_to_workspace() 提交
    end

Design thinking and pitfalls

Synchronization primitives for async modeis the most subtle part of this code.rebalancedThe flag relies on the GIL to synchronize between the main thread and the async worker📎 vllm/distributed/eplb/eplb_state.py:194-203. But the comment warns:rebalancedmust remain consistent across all ranks, otherwise_all_ranks_result_readythe all-reduce inside will hang📎 vllm/distributed/eplb/eplb_state.py:664-665。_all_ranks_result_readyPrefer using the CPU group for all-reduce, because the CPU group is more reliable📎 vllm/distributed/eplb/eplb_state.py:1024-1043。

The sliding window's "early recording" optimization：_should_record_current_steponly enables recording when the distance to the next rebalancing is no more thanwindow_sizesteps📎 vllm/distributed/eplb/eplb_state.py:689-709. The comment explains: the data of thestep_interval - window_sizesteps before each rebalancing cycle will be overwritten by the sliding window, so recording it is wasted effort and wastes GPU compute📎 vllm/distributed/eplb/eplb_state.py:1196-1199。should_record_tensoris the same scalar tensor shared by all layers, onefill_updates all layers📎 vllm/distributed/eplb/eplb_state.py:272-278。

Capacity reservation for elastic EP：enable_elastic_epwhen,physical_expert_capacityreserve byelastic_ep_max_dp_size, the mapping table fills extra slots with -1📎 vllm/distributed/eplb/eplb_state.py:375-386. This way, scaling up doesn't require reallocating GPU memory, just filling the -1 slots with real experts.reconfigure_physical_expert_slotsis responsible for refreshing the view during scale-up/scale-down📎 vllm/distributed/eplb/eplb_state.py:1135-1160。

_commit_eplb_maps's pin memory handling: whenPIN_MEMORYis enabled and the source is on CPU, first copy to pinned memory thennon_blocking=Trueasynchronously copy to GPU📎 vllm/distributed/eplb/eplb_state.py:1392-1400. This is to avoid H2D copies blocking the main thread—the mapping table is updated every layer every round, and synchronous copies would become a bottleneck.

Design thinking

The three pieces of code share one design philosophy:Trade capability detection for deterministic degradation。GroupCoordinatorWhenworld_size == 1directly bypass all collective communication📎 vllm/distributed/parallel_state.py:736-738；CustomAllreducereturn when any condition is not metNonelet the caller fall back to NCCL📎 vllm/distributed/device_communicators/custom_all_reduce.py:532-533; EPLB skips rearrangement when the improvement is less than 5%📎 vllm/distributed/eplb/eplb_state.py:916. This "fail fast + graceful degradation" pattern allows the same code to run across the full spectrum of hardware from a single GPU to multi-machine MNNVL, without needing to write branches for every configuration.

Another commonality iscontrol-flow consistency takes priority over performance。_group_can_attempt_mnnvluse CPU all-reduce to force all ranks onto the same branch📎 vllm/distributed/device_communicators/custom_all_reduce.py:59-73，_all_ranks_result_readysimilarly📎 vllm/distributed/eplb/eplb_state.py:1024-1043. In distributed systems, "some ranks take the fast path while others take the slow path" is far more dangerous than "all ranks take the slow path" - the former hangs, while the latter is merely slow.

Chapter Summary

• GroupCoordinatorReshape the one-dimensional rank sequence into aExternalDP x DP x PP x PCP x TPgrid, and partition TP/PP/DP/EP/EPLB process groups along each dimension; each group simultaneously maintains two PGs: CPU (gloo) and device (NCCL).
• CustomAllreduceUse capability detection (same machine, NVLink full interconnect, tensor size, dtype, 16-byte alignment) to decide whether to take over all-reduce, and degrade to MNNVL or NCCL in multi-machine scenarios.
• EPLB uses three mapping tables to describe the logical/physical expert relationships, counts load through a sliding window, computes a new mapping via a strategy, and moves weights through a communicator, supporting both synchronous and asynchronous modes.
• The shared design principles of the three: capability detection + deterministic degradation + control-flow consistency first.

Chapter Review Questions

At this point, we have clarified the group formation, partitioning, and load rebalancing mechanisms for inter-GPU communication. However, the communication challenges of distributed inference go beyond a single instance—when prefill and decode are split across different instances, the KV Cache needs to be transferred across nodes. In the next chapter, we will leave "inter-GPU communication" and enter "inter-instance communication": how KV Cache is transferred between prefill and decode instances in disaggregated deployment, and how the KV Connector abstraction unifies transfer backends such as NIXL and Mooncake.

In the previous chapter, we focused our view inside a single inference instance: how TP/PP/DP/EP process groups are formed, how tensors are partitioned across GPUs, and how EPLB performs expert rebalancing at the MoE layer. But all these mechanisms are built on the same premise—prefill and decode run in the same instance, and the KV Cache stays in local GPU memory from beginning to end. Disaggregated deployment (Prefill-Decode Disaggregation, or PD disaggregation) breaks this premise. It splits prefill and decode into two independent vLLM instances: the prefill instance only performs the forward computation for the prompt, produces the KV Cache, and hands it to the decode instance; the decode instance takes this KV Cache and continues autoregressive generation. The benefit is that resources can be configured independently according to the characteristics of each phase—prefill is compute-intensive and suits large TP and large batches; decode is memory-access-intensive and suits small batches and low-latency scheduling. The two no longer drag each other down. The cost is that the KV Cache must be transferred across instances. This is the protagonist of this chapter—the KV Connector. The file header comment of vllm/distributed/kv_transfer/kv_connector/v1/base.py already lists the core primitives of the entire abstraction: the Scheduler side is responsible for binding metadata, querying remote cache hits, and deciding whether to asynchronously release blocks; the Worker side is responsible for the actual KV loading and saving. The design goal of this interface is to completely decouple the upper-layer scheduling logic from the underlying transfer backends (NIXL, Mooncake, MoRIIO). From an engineering perspective, the biggest risk of PD disaggregation is not slow transfer, but state inconsistency: the prefill instance believes the KV has been sent, but the decode instance does not receive it; or the decode instance releases the block early while prefill is still writing to it. What this chapter aims to clarify is exactly how this connector system uses handshake protocols, leases, heartbeats, and failure recovery mechanisms to cover these boundary cases.

1. KVConnectorBase_V1: Dual-role abstraction and metadata contract

Intuitive model

The KV Connector is like a courier system between two branch stores. The Prefill store has calculated the semi-finished product (KV Cache), packages it, and ships it to the Decode store for further processing. But a courier system cannot have only the action of "shipping"—it needs a waybill (metadata) explaining what is being sent and where it is going; it needs a receipt mechanism to confirm that the other party has received it; and it also needs a set of timeout rules to prevent packages from being stuck on the road forever and occupying shelf space.

Without this abstraction, every transfer backend (NIXL, Mooncake) would have to implement its own scheduling logic, and vLLM's Scheduler would have to write a set of adaptation code for each backend. The value of KVConnectorBase_V1 is to fix this contract in place.

Dual roles: Scheduler side and Worker side

📎 vllm/distributed/kv_transfer/kv_connector/v1/base.py:137-142defines the two roles of the connector:

class KVConnectorRole(enum.Enum):
    # Connector running in the scheduler process
    SCHEDULER = 0
    # Connector running in the worker process
    WORKER = 1

This division is not arbitrary. The Scheduler process is responsible for global scheduling decisions—which requests need transfer and when blocks can be released; the Worker process is responsible for the actual data movement. The two communicate throughKVConnectorMetadatacommunication.

📎 vllm/distributed/kv_transfer/kv_connector/v1/base.py:153-158defines the base class for metadata in the Scheduler-to-Worker direction:

class KVConnectorMetadata(ABC):  # noqa: B024
    """Abstract Metadata used to communicate
    Scheduler KVConnector -> Worker KVConnector.
    """
    pass

In the reverse Worker-to-Scheduler direction,📎 vllm/distributed/kv_transfer/kv_connector/v1/base.py:161-176definesKVConnectorWorkerMetadata, which requires implementing theaggregatemethod—because in one engine step, multiple workers may each return metadata, which needs to be aggregated before being handed to the Scheduler.

Core data structure: KVConnectorTransferResults

📎 vllm/distributed/kv_transfer/kv_connector/v1/base.py:87-96defines the snapshot structure of transfer results:

@dataclass
class KVConnectorTransferResults:
    finished_sending: set[str] = field(default_factory=set)
    finished_recving: set[str] = field(default_factory=set)
    failed_recving: set[str] = field(default_factory=set)

Note the key design in the comments:Failed receives also appear infinished_recvingin. This is to allow the Scheduler to release requests from the "waiting for transfer" state—even if the transfer fails, the request must not be stuck forever. Failure information is passed separately throughfailed_recving, and the Scheduler decides whether to retry or degrade based on it.

Lifecycle hooks: from request to release

The entire connector lifecycle revolves around several key hooks. On the Scheduler side:

• get_num_new_matched_tokens 📎 vllm/distributed/kv_transfer/kv_connector/v1/base.py:485-518: Queries how many tokens the remote cache can hit. The comment specifically emphasizes "should only consider the actually available maximum prefix"—if some tokens cannot be obtained due to connection issues or eviction, they must not be counted.
• update_state_after_alloc 📎 vllm/distributed/kv_transfer/kv_connector/v1/base.py:520-544: Updates state after block allocation. There is an easy pitfall in the comment—whether to load should be determined bynum_external_tokens, not byblockswhether it is empty, because non-selected sub-connectors of MultiConnector also receive real blocks.
• request_finished 📎 vllm/distributed/kv_transfer/kv_connector/v1/base.py:579-598: Called when the request completes, returningTrueindicates that the connector takes over the responsibility for asynchronous release of the block.

Worker side:

• start_load_kv / wait_for_layer_load: Loads layer by layer, supporting pipelining.
• save_kv_layer / wait_for_save: Saves layer by layer.
• get_transfer_results 📎 vllm/distributed/kv_transfer/kv_connector/v1/base.py:396-397: Returns the completion status of asynchronous transfers.

📎 vllm/distributed/kv_transfer/kv_connector/v1/base.py:192-201There is also an easily overlooked but critical design—requires_kv_deliveryproperty:

@property
def requires_kv_delivery(self) -> bool:
    """Whether this connector hands off KV that must be reliably delivered.
    ...
    """
    return self._kv_transfer_config.is_kv_producer

The comment explains the motivation: if a request is preempted before the KV handoff is complete, it should be recomputed rather than allowed to complete and hand off blocks that have already been released by preemption. Only the producer role requires reliable delivery; a lost best-effort cache is just a future cache miss.

Handshake metadata

📎 vllm/distributed/kv_transfer/kv_connector/v1/base.py:145-150defines the base class for handshake metadata:

class KVConnectorHandshakeMetadata(ABC):  # noqa: B024
    """Metadata used for out of band connector handshake between
    P/D workers. This needs to serializable.
    """
    pass

"out of band" means the handshake does not go through the normal request path, but communicates directly between P/D workers. This lays the groundwork for NIXL's ZMQ handshake protocol.

---

II. NIXL Connector: Handshake, Registration, and Descriptor Construction

Intuitive model

NIXL (NVIDIA Inference Xfer Library) is a low-level transfer library provided by NVIDIA, supporting multiple backends such as UCX and GDS. The role of NixlBaseConnectorWorker is like the sorting center of a courier company—it first needs to establish a dedicated line with the other sorting center (handshake), register its own shelf layout (register KV Cache memory regions), and only then can it efficiently pick up and deliver goods by address.

Without this mechanism, every transfer would require renegotiating addresses and reestablishing connections, and the latency would be unacceptably high.

Memory layout: Region and Descriptor

NIXL's core concepts areregion(memory region) anddescriptor(descriptor). Each KV Cache layer is registered in NIXL as one or more regions, and each region has a base address, block length, and block stride.

📎 vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py:740-751lists the core fields related to regions:

# Number of NIXL regions. Currently one region per cache
# (so 1 per layer for MLA, otherwise 2 per layer)
self.num_regions = 0
self.region_mem_types: list[str] = []
self.region_group_ids: list[int] = []
self._uses_region_group_mapping = False
self.region_names: list[str] = []
self.region_num_blocks: list[int] = []
self._mixed_mem_types = False

📎 vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py:897-900further explains the source of the block stride:

# Per-region block stride in bytes. Taken from the registered tensor's
# stride(0) so it stays correct under layouts that interleave layers
# within a block (BLHNC/BHLNC), where stride > block_len.
self.block_stride_per_layer = list[int]()

The key insight here is:block_stride is not equal to block_len. Under interleaved layouts such as BLHNC/BHLNC, the actual span of a block may be larger than its valid data length. If block_len is used directly as the stride, the wrong address will be read.

Handshake protocol: ZMQ + compatibility hash

The handshake is the most complex part of the NIXL connector.📎 vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py:974-1128The_nixl_handshakemethod of

fully demonstrates this process.📎 vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py:988-998The first step is to set the CUDA device context.

# the first time we connect to a remote agent.
# be careful, the handshake happens in a background thread.
# it does not have an active cuda context until any cuda runtime
# call is made. when UCX fails to find a valid cuda context, it will
# disable any cuda ipc communication, essentially disabling any NVLink
# communication.
if not self.use_host_buffer:
    current_platform.set_device(self.device_id)

explains the reason:

Copy📎 vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py:1029-1036：

msg = msgspec.msgpack.encode(
    (GET_META_MSG, remote_pp_rank, remote_rank)
)
# Set receive timeout to 5 seconds to avoid hanging on dead server
sock.setsockopt(zmq.RCVTIMEO, 5000)  # milliseconds
start_time = time.perf_counter()
sock.send(msg)
reply_parts = sock.recv_multipart()

The second step is to send a metadata query via ZMQ.📎 vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py:1042-1045Copy

The 5-second timeout prevents waiting indefinitely after the peer dies. At the same time, the code uses RTT to estimate the clock offset📎 vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py:1063-1080：

assert self.compat_hash is not None
if (
    self.enforce_compat_hash
    and handshake_payload.compatibility_hash != self.compat_hash
):
    raise RuntimeError(
        f"NIXL compatibility hash mismatch. "
        ...
    )

The third step is compatibility verification.📎 vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py:1372-1376Copy

self.compat_hash = compute_nixl_compatibility_hash(
    self.vllm_config,
    self.backend_name,
    transfer_mode=self._TRANSFER_MODE,
)

:transfer_modeCopy📎 vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py:163-166Note that

also participates in the hash—

The comment in📎 vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py:824-835：

self._handshake_initiation_executor = ThreadPoolExecutor(
    # NIXL is not guaranteed to be thread-safe, limit 1 worker.
    max_workers=1,
    thread_name_prefix="vllm-nixl-handshake-initiator",
)
self._ready_requests = queue.Queue[tuple[ReqId, ReqMeta]]()
self._handshake_futures: dict[
    EngineId, Future[tuple[dict[tuple[int, int], str], float]]
] = {}
# Protects _handshake_futures and _remote_agents.
self._handshake_lock = threading.RLock()

max_workers=1Asynchronous handshake scheduling_handshake_lockThe handshake is asynchronous and executed through a thread pool._handshake_futuresCopy_remote_agentsis because NIXL does not guarantee thread safety.

_ensure_handshake 📎 vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py:1257-1317protects

and

the two dictionaries.📎 vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py:172-310implements idempotent handshake initiation: if the handshake has already succeeded, return None directly; if a handshake is in progress, return the existing Future; otherwise, submit a new task and register a callback._compute_desc_idsDescriptor construction: from block ID to NIXL descriptor

After the handshake is complete, descriptors need to be constructed for each request.📎 vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py:226-262The

# NOTE (NickLucche) With HMA, every kv group has the same number of layers
# and layers from different groups share the same kv tensor.
# eg block_ids=[[1, 2], [3]]->blocks [1, 2] need to be
# read across all regions, same for [3], but group0-group1 blocks will
# always differ (different areas). Therefore we can just flatten the
# block_ids and compute the descs ids for all groups at once.

is the core.📎 vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py:285-304：

elif _is_ssm_spec(spec_type):
    # NOTE (NickLucche) SSM and Attention block regions can
    # be exchanged arbitrarily by manager.  Therefore, descs
    # are laid out as:
    #   [descs_fa (all regions) | descs_ssm (all regions)].
    # num_fa_descs offset must be computed per-engine since
    # P and D can have different num_blocks (and thus
    # different FA desc counts).

. The comment explains the handling in the HMA scenario:

Copy📎 vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py:2130-2178For hybrid SSM models, the descriptor layout is more complexadd_remote_agentCopy

When D.world_size > P.world_size, multiple D workers read different KV head shards from the same P worker. The documentation gives a concrete example: D TP=4, P TP=2, tp_ratio=2. D-Worker0 reads the first half of KV heads from P-Worker0, and D-Worker1 reads the second half.

For MLA models, the KV Cache is replicated across TP workers, so rank_offset is always 0.

Lease and heartbeat: preventing blocks from being released prematurely

This is one of the most ingenious designs of the NIXL connector. After the Prefill instance sends KV, it cannot immediately release the block—because the decode instance may still be reading. But if it never releases, GPU memory will leak.

The solution is a lease.📎 vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py:528-528：

kv_lease_duration: int = vllm_config.kv_transfer_config.get_from_extra_config(
    "kv_lease_duration", 30
)
# NOTE (NickLucche): For now we use a hardcoded value for a simpler interface.
self._lease_extension = kv_lease_duration * 2 // 3

The default lease is 30 seconds, extended by 20 seconds (2/3) on each heartbeat.

Heartbeat handling is in📎 vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py:3014-3034：

def _handle_heartbeat(self, payload: str) -> None:
    new_expiry = time.perf_counter() + self._lease_extension
    for req_id in payload.split(","):
        if req_id in self._reqs_to_send:
            old = self._reqs_to_send[req_id]
            self._reqs_to_send[req_id] = max(old, new_expiry)

Notemax(old, new_expiry)—heartbeats can only extend the lease, not shorten it.

Reclamation after lease expiration is in📎 vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py:2986-3012：

def _reap_expired_send_leases(self, done_sending: set[str]) -> None:
    """Reclaim expired send-side KV leases into ``done_sending``.

    ``_reqs_to_send`` is not ordered by expiry: heartbeats update the
    deadline in place, and mixed TTLs share the map, so a live head
    entry can sit in front of already-expired ones. Scan every entry
    rather than stopping at the first still-live request.
    """

The comment points out an easy mistake: you cannot stop scanning just because you encounter the first non-expired request, because heartbeats update the expiration time in place, causing the map to not be sorted by expiration time.

Transfer state machine and failure recovery

The lifecycle of a transfer is managed through_pop_done_transfers 📎 vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py:3036-3086:

for handle in handles:
    try:
        xfer_state = self.nixl_wrapper.check_xfer_state(handle)
        if xfer_state == "DONE":
            res = self.nixl_wrapper.get_xfer_telemetry(handle)
            self.xfer_stats.record_transfer(res)
            self.nixl_wrapper.release_xfer_handle(handle)
        elif xfer_state == "PROC":
            in_progress.append(handle)
        else:
            self._log_failure(
                failure_type="transfer_failed",
                req_id=req_id,
                xfer_state=xfer_state,
            )

NIXL transfers have three states:DONE(completed),PROC(in progress), and others (failed).

Failure handling is in📎 vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py:3103-3127：

def _handle_failed_transfer(
    self,
    req_id: str,
    handle: int | None,
    failed_req_ids: set[str] | None = None,
    record_failed_transfer: bool = True,
) -> bool:
    if record_failed_transfer:
        self.xfer_stats.record_failed_transfer()
    if failed_req_ids is not None:
        failed_req_ids.add(req_id)
    return handle is None or self._try_release_xfer_handle(req_id, handle)

_try_release_xfer_handle 📎 vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py:3088-3101The comment in is critical:

except Exception as e:
    # A status error does not guarantee that the backend stopped DMA.
    self._log_failure(
        failure_type="transfer_release_failed",
        msg="Retaining handle and blocks until release succeeds",
        ...
    )
    return False

A state error does not guarantee that the backend has stopped DMA. If release fails, the handle and block must be retained until release succeeds. This is a typical "rather leak than misuse" design.

Block handling for failed requests

When reception fails,📎 vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py:2876-2891shows the handling logic:

for req_id in done_recving:
    meta = self._recving_metadata.pop(req_id, None)
    assert meta is not None, f"{req_id} not found in recving_metadata list"

    # Skip KV sync and post-processing for failed requests
    if req_id in failed_recv_reqs:
        self._pending_recv_notifs.pop(req_id, None)
        # TODO (NickLucche) handle failed transfer for HMA.
        if not self._is_hma_required:
            self._invalid_block_ids.put(set(meta.local_block_ids[0]))
        logger.warning(
            "Skipping KV post-processing for failed request %s",
            req_id,
        )
        continue

The failed block ID is placed into the_invalid_block_idsqueue, and the Scheduler retrieves it throughget_block_ids_with_load_errors 📎 vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py:3491-3504to decide whether to retry.

TTL eviction of remote engines

Long-running instances will continuously encounter new remote engines, and if not cleaned up, memory will grow indefinitely.📎 vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py:3506-3532The_evict_stale_enginesof implements TTL eviction:

def _evict_stale_engines(self) -> None:
    """Scan for and evict remote engines that have exceeded their TTL.

    Called from the main thread in when a new remote engine appears.
    We can only go OOM as we discover and register a new remote, therefore we make
    sure we clean up stale engine data structures before then.
    """
    if self._engine_ttl <= 0:
        return

    now = time.perf_counter()
    busy = self._engines_with_inflight_transfers()
    for eid, last_active in list(self._engine_last_active.items()):
        if now - last_active > self._engine_ttl and eid not in busy:
            self._cleanup_remote_engine(eid)

The key constraint is thebusyset—engines with in-progress transfers cannot be evicted.📎 vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py:3534-3546The comment in explains the reason:

"""Remote engines a transfer is still reading from.

The timestamp is stamped when a read is issued and not refreshed while
it runs, so a transfer that outlives the TTL leaves its engine looking
idle. A peer that has lost its NIC holds one indefinitely.
"""

If the peer's NIC is broken, the transfer may hang forever, the timestamp will not refresh, and the engine will appear idle.busyThe set explicitly protects against this situation.

Timing of handshake and transfer

The sequence diagram below shows the core interaction from request to transfer completion:

sequenceDiagram
    participant Sched as Scheduler
    participant Worker as NixlWorker
    participant BgThread as 握手后台线程
    participant Remote as 远程 NIXL Agent

    Sched->>Worker: build_connector_meta()
    Worker->>Worker: _ensure_handshake(engine_id)
    alt 已握手
        Worker->>Worker: 直接返回 None
    else 握手中
        Worker->>BgThread: 返回已有 Future
    else 新握手
        Worker->>BgThread: submit(_nixl_handshake)
        BgThread->>Remote: ZMQ GET_META_MSG
        Remote-->>BgThread: NixlHandshakePayload
        BgThread->>BgThread: 校验 compat_hash
        BgThread->>Remote: add_remote_agent()
        BgThread-->>Worker: done_callback 注册 _remote_agents
    end
    Worker->>Remote: prep_xfer_dlist + make_xfer_req
    Worker->>Worker: _recving_transfers[req_id] = handles
    Sched->>Worker: get_transfer_results()
    Worker->>Worker: _pop_done_transfers()
    alt xfer_state == DONE
        Worker->>Remote: release_xfer_handle
        Worker-->>Sched: finished_recving
    else xfer_state == PROC
        Worker->>Worker: 保留 handle 等待下一轮
    else 失败
        Worker->>Worker: _handle_failed_transfer
        Worker-->>Sched: failed_recving + invalid_block_ids
    end

---

III. Design thinking: why it is designed this way

Why should the handshake be asynchronous?

The handshake involves a network round trip and may take tens of milliseconds. If executed synchronously, it would block the Scheduler's main loop and affect the scheduling of all requests. Asynchronous handshaking allows the Scheduler to process other requests first, and notify via callback after the handshake completes.

But asynchrony also brings complexity:_handshake_futuresThe dictionary needs lock protection, the callback must handle both success and failure cases, and duplicate handshakes must be prevented.

Why use leases instead of reference counting?

Reference counting requires the decode instance to explicitly notify prefill that "I have finished reading." But if the decode instance crashes, the notification will never arrive, and the prefill's block will leak forever.

Leases are a more robust solution: even if decode crashes, prefill automatically reclaims after the lease expires. The heartbeat mechanism ensures lease renewal under normal conditions.

Why retain the handle on failure?

📎 vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py:3088-3101The comment in makes it very clear: a state error does not guarantee that DMA has stopped. If the handle is released at this point, DMA may still be writing data to the released memory, causing data corruption or a crash. It is better to leak temporarily than to take this risk.

Why should TTL eviction check busy?

📎 vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py:3534-3546The comment in reveals a hidden bug scenario: the timestamp is set when the read is initiated and is not refreshed during the read. If the transfer time exceeds the TTL, the engine appears idle, but it is actually still being read. If it is evicted at this point, the in-progress transfer will fail.

Pitfalls in production environments

1. CUDA context issue: The handshake is executed in a background thread, and must be explicitlyset_device, otherwise UCX will silently disable NVLink.

2. Compatibility hash mismatch: The vLLM version, model, dtype, KV layout, and attention backend of the P/D instances must be completely consistent. When inconsistent, the handshake will fail, and the error message will indicate how to disable the check (but this is not recommended).

3. Lease expiration: If the decode instance is under heavy load, heartbeats may be delayed, causing the lease to expire. A "Releasing expired KV blocks" warning will appear in the logs. You can increasekv_lease_duration。

4. TP mismatch: Heterogeneous TP requires a block-contiguous layout (such as LBHNC). If a non-contiguous layout is used, heterogeneous TP will fail.

5. NIXL UAR exhaustion：📎 vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py:631-636comment warning: Each UCX thread allocates UAR (doorbell pages) through DevX. Excessive NIXL UAR usage will exhaust NIC UAR space, causing NVSHMEM (used by DeepEP kernels) to fail during RDMA initialization.

---

Chapter Summary

This chapter explored the core mechanisms of the KV Connector system:

1. KVConnectorBase_V1It defines the dual-role abstraction for the Scheduler side and Worker side, exchanging metadata and reporting transfer results throughKVConnectorMetadataandKVConnectorTransferResults.

2. NIXL Connectoris the most mature implementation. It establishes connections between P/D instances through a ZMQ handshake protocol, uses compatibility hashing to prevent configuration mismatches, and uses an asynchronous thread pool to avoid blocking the main loop.

3. Lease and Heartbeatmechanisms solve the timing issue of block release: after prefill sends KV, it does not release immediately but waits for decode's heartbeat renewal or lease expiration.

4. Failure Recoveryfollows the principle of "better to leak than to misuse": when release fails, retain the handle, and report the failed block ID to the Scheduler to decide on retry.

5. TTL Evictionprevents unbounded growth of remote engine state during long-running operation, but must protect engines with in-progress transfers.

In the next chapter, we will turn to another direction for eliminating overhead: compilation acceleration and CUDA Graph. After PD separation solves the resource utilization problem, the launch overhead of a single forward pass becomes the new bottleneck—how to use CUDA Graph to compress hundreds or thousands of kernel launches into a single replay.

Chapter Review Questions

At this point, we have seen clearly how the KV Connector establishes a reliable data channel between prefill and decode instances, and how it uses leases, heartbeats, and failure recovery mechanisms to preserve state consistency. But cross-instance transfer is only half the story of PD disaggregation—after the KV Cache reaches the decode instance, the inference engine still needs to efficiently execute each forward computation step within a single instance. And Python scheduling and kernel launch overhead are precisely the next bottleneck constraining single-step latency. The next chapter turns to compilation acceleration and CUDA Graph, examining how vLLM uses torch.compile and the piecewise backend to eliminate these overheads, and how it enables CUDA Graph and dynamic batch shapes to coexist harmoniously.

In the previous chapter, we saw how the KV Connector efficiently moves KV cache between Prefill and Decode engines via connectors such as NIXL and Mooncake, enabling the disaggregated architecture to reduce TTFT while improving resource utilization. But no matter how fast the transfer, autoregressive decoding still has two fixed costs that cannot be eliminated by algorithms: the scheduling overhead of the Python interpreter and the launch overhead of GPU kernels. When the model forward pass is split into hundreds of operators, each requiring a Python function call and a CUDA kernel launch, the CPU-side overhead is enough to leave the GPU idle between computations. This chapter analyzes how vLLM uses torch.compile to fuse operators into a static graph, then uses CUDA Graph to record the entire kernel launch sequence into a single replay, thereby driving these two types of overhead close to zero.

Compilation Cache and Compiler Adaptation Layer: Enabling Cross-Process Reuse of Compilation Results

Intuitive Model

The benefit of compilation acceleration is "compile once, run many times," but the cost is that the first compilation may take several minutes. Without caching, every service restart would require recompilation, and cold start time would be unacceptable.CompilerInterfaceThis layer addresses exactly the problem of "how compilation artifacts are serialized, how they are identified by hash, and how they are precisely hit on the next startup." Without it, the disaster the system faces is not a crash, but degradation to "first run" on every restart—in an auto-scaling production environment, this means newly scaled instances cannot provide low-latency service for several minutes.

Data Structures and Interface Contracts

CompilerInterfacedefines the abstract contract of the compiler adapter, with four core methods:initialize_cacheis responsible for redirecting the compiler's own cache directory under vLLM's cache directory📎 vllm/compilation/compiler_interface.py:36-51；compute_hashcollects compiler-related configuration information to generate a hash📎 vllm/compilation/compiler_interface.py:53-62；compileexecutes compilation and returns a callable object and a handle📎 vllm/compilation/compiler_interface.py:64-95；loadrestores compilation artifacts from the handle📎 vllm/compilation/compiler_interface.py:97-103。

The key design here iscompilereturns a two-tuple(callable, handle)。callableis the compilation result directly callable within the current process;handleis the credential "used to restore on the next startup," and the documentation explicitly requires it to be a "plain Python object, preferably a string or a file path"📎 vllm/compilation/compiler_interface.py:81-81. This separation allows the cache-hit path and the first-compilation path to follow completely different code—on a hit, there is no need forcompile, onlyload。

compile_rangeThe parameter carries the semantics of dynamic shapes. The comment states that it "could be concrete size (if compile_sizes is provided), e.g. [4, 4] or a range [5, 8]," and that "Right now we only support one variable in ranges for all inputs, which is the batchsize (number of tokens) during inference"📎 vllm/compilation/compiler_interface.py:74-74. This is the core constraint of vLLM's compilation strategy: all dynamic shapes are reduced to a single variable—the number of tokens.

Scenario-Driven: The Complete Flow of a Compilation Request

Suppose the service starts for the first time,InductorAdaptor.compileis called. It first increments the compilation counter📎 vllm/compilation/compiler_interface.py:477-489, then enters a carefully constructed patch stack.

The first step is to deep-copy the graph. The comment states that "inductor can inplace modify the graph, so we need to copy it"📎 vllm/compilation/compiler_interface.py:500-502, which is a defensive design—if compilation fails, the original graph can still be used for retry.

The second step is to install a series of monkey-patches.hijacked_compile_fx_innerwraps Inductor's internal compilation function, and after compilation completes, grabs the hash frominductor_compiled_graph._fx_graph_cache_key📎 vllm/compilation/compiler_interface.py:512-536。hijack_compiled_fx_graph_hashintercepts the hash computation function itself📎 vllm/compilation/compiler_interface.py:538-542. Why "hijack" the hash? Because vLLM needs to compile separately outside the Dynamo tracing context, while Inductor's hash computation depends on that context.

The third step is_check_can_cachepatch, it directly returns without performing any checks📎 vllm/compilation/compiler_interface.py:544-551. The comment explains the motivation: "Inductor refuses to cache the graph outside of Dynamo tracing context, and also disables caching for graphs with high-order ops. For vLLM, in either case, we want to cache the graph"📎 vllm/compilation/compiler_interface.py:544-551。

The fourth step is cleaning up the tracing context. This is the most subtle part: vLLM callsPiecewiseCompileInterpreterfrom withincompile_fx, at which point Dynamo'sFakeTensorModeis inconsistent with the subgraph input'sFakeTensorMode,detect_fake_mode()will cause an assertion failure📎 vllm/compilation/compiler_interface.py:615-622. The code savesTracingContextthen sets it to null, and registers a callback to restore it on exit📎 vllm/compilation/compiler_interface.py:623-630。

flowchart TD
    start["InductorAdaptor.compile()"] --> deepcopy["copy.deepcopy(graph)"]
    deepcopy --> patch_stack["ExitStack 安装补丁"]
    patch_stack --> p1["patch compiled_fx_graph_hash"]
    patch_stack --> p2["patch FxGraphCache._get_shape_env"]
    patch_stack --> p3["patch _check_can_cache"]
    patch_stack --> p4["清空 TracingContext"]
    p4 --> call_fx["compile_fx(graph, example_inputs)"]
    call_fx --> check{"hash_str is None?"}
    check -->|"是"| err["RuntimeError: 编译失败<br/>建议删除 torch_compile_cache"]
    check -->|"否"| check2{"file_path is None?"}
    check2 -->|"是"| assert_err["AssertionError"]
    check2 -->|"否"| ret["return (compiled_graph, (hash_str, file_path))"]
    err --> cleanup["ExitStack 退出<br/>恢复 TracingContext"]
    assert_err --> cleanup
    ret --> cleanup

Design considerations: AlwaysHitShapeEnv and cache consistency

AlwaysHitShapeEnvThis class deserves a separate analysis. Its docstring plainly states the motivation: vLLM only runs Dynamo bytecode compilation once, but needs to run Inductor compilation multiple times with different shapes plus one generic shape; shape-specific compilation happens outside the Dynamo context, where no shape environment is available to Inductor, causing Inductor code cache lookup failures📎 vllm/compilation/compiler_interface.py:114-131。

The solution is to provide an "always hit" fake shape environment:evaluate_guards_expressionalways returnsTrue 📎 vllm/compilation/compiler_interface.py:144-145，get_pruned_guardsreturns an empty list📎 vllm/compilation/compiler_interface.py:144-145，produce_guards_expressionreturns an empty string📎 vllm/compilation/compiler_interface.py:147-159. The comment candidly admits these methods are "obtained by trial-and-error until it works"📎 vllm/compilation/compiler_interface.py:137-142—this is a fragile point coupled with PyTorch's internal implementation, and also the most error-prone area when upgrading PyTorch.

The composition of the cache hash is equally critical.get_inductor_factorscollects three categories of factors: system stateCacheBase.get_system(), PyTorch statetorch_key(), and Inductor and functorch configurations📎 vllm/compilation/compiler_interface.py:165-185. Note that functorch configuration is collected within thepatch(_get_vllm_functorch_config())context📎 vllm/compilation/compiler_interface.py:188-189, which ensures "compile-time configuration and cache key are always consistent"—the comment explicitly states this is to keepset_functorch_config()andget_inductor_factors()consistent📎 vllm/compilation/compiler_interface.py:147-159. If these two are inconsistent, there will be a mismatch where "configuration A was used at compile time but the cache key was computed according to configuration B," causing a cache hit that loads the wrong artifact.

Production pitfalls:_patch_standalone_compile_atomic_saveis a backport for torch < 2.10.0📎 vllm/compilation/compiler_interface.py:205-243. It changesCompiledArtifact.save()to usewrite_atomicto write in binary format, with the comment stating the purpose is "preventing corrupt cache files when multiple processes compile concurrently"📎 vllm/compilation/compiler_interface.py:208-210. In scenarios where multiple replicas cold-start simultaneously, multiple processes will concurrently write to the same cache file; non-atomic writes produce truncated files, and subsequent processes reading corrupted artifacts will behave unpredictably.

PiecewiseBackend: shape-bucketed compilation and runtime dispatch

Intuitive model

PiecewiseBackendis the scheduling hub between compilation and execution. It compiles "one FX subgraph" into "callable objects for multiple shape buckets," and at runtime selects the most appropriate one based on the actual token count. Without it, either all shapes go through the same generic compilation (suboptimal performance), or each shape is compiled separately (compilation time explodes).

Data structures: RangeEntry and compilation ranges

The core data structure isRangeEntry, which binds thecompile_range、compiledflag andrunnabletogether📎 vllm/compilation/piecewise_backend.py:80-83。PiecewiseBackendmaintains arange_entries: dict[Range, RangeEntry] 📎 vllm/compilation/piecewise_backend.py:166-171。

The construction of compilation ranges is done in two steps. First, handlecompile_sizes(exact sizes), generating a single-point interval ofRange(start=size, end=size)for each size📎 vllm/compilation/piecewise_backend.py:166-171. Note that for the string"cudagraph_capture_sizes"it directly throwsNotImplementedError, and states "should be handled inpost_init_cudagraph_sizes" 📎 vllm/compilation/piecewise_backend.py:166-171—this is an explicit declaration of responsibility boundaries. Then handlecompile_ranges(intervals), generating one entry per interval📎 vllm/compilation/piecewise_backend.py:173-173。

PiecewiseBackendsupports two mutually exclusive modes, and the constructor enforces this with an XOR assertion📎 vllm/compilation/piecewise_backend.py:117-119: compilation mode (has graph, no compiled_runnables) goes throughcompile_all_ranges() 📎 vllm/compilation/piecewise_backend.py:193-194; precompiled mode (no graph, has compiled_runnables) goes throughload_all_ranges() 📎 vllm/compilation/piecewise_backend.py:193-194. This design allows cold start and warm start to share the same class, with only the data source differing.

Scenario-driven: from compilation to runtime dispatch

Compilation phase：compile_all_rangesiterates over all range entries, calling_log_compile_startfor each uncompiled entry📎 vllm/compilation/piecewise_backend.py:252-256records tracing eventscreate_concrete_args. The key branch is in parameter construction: if it is a single-point size, call📎 vllm/compilation/piecewise_backend.py:258-261to generate a FakeTensor of the specific shapeget_fake_args_from_graph; otherwise call📎 vllm/compilation/piecewise_backend.py:262-263。

create_concrete_argsto directly reuse the placeholder metadata from the graphShapeEnvThe implementation reveals the details of symbolic shape concretization. It constructs aFakeTensorMode 📎 vllm/compilation/piecewise_backend.py:54withSymInt, then iterates over placeholder nodes. For inputs of typeconcretize, usesize 📎 vllm/compilation/piecewise_backend.py:47-52to replace all free symbols withTensor; for typecompute_required_storage_length, it must simultaneously concretize shape, stride, and storage_offset, and useas_stridedto compute the required storage length, then reconstruct the tensor via📎 vllm/compilation/piecewise_backend.py:64-73. Why can't we just change the shape? Because stride and storage_offset may also contain symbols, and the three must be self-consistent, otherwiseas_stridedwill go out of bounds.

Runtime dispatch：__call__is a hot path. Ifsym_shape_indicesexists, retrieve the runtime shape fromargs, then call📎 vllm/compilation/piecewise_backend.py:357-362to look up. The lookup logic has priority: first check whether an exact_find_range_for_shapeis hit; if so, return that single-point rangecompile_sizes; otherwise iterate over📎 vllm/compilation/piecewise_backend.py:342-355to find the range containing that shapecompile_rangesCopy📎 vllm/compilation/piecewise_backend.py:342-355。

flowchart TD
    call["PiecewiseBackend.__call__(*args)"] --> has_sym{"sym_shape_indices 非空?"}
    has_sym -->|"是"| get_shape["runtime_shape = args[sym_shape_indices[0]]"]
    get_shape --> find["_find_range_for_shape(runtime_shape)"]
    find --> exact{"runtime_shape in compile_sizes?"}
    exact -->|"是"| exact_entry["返回 Range(start=shape, end=shape) 的 entry"]
    exact -->|"否"| scan["遍历 compile_ranges 找包含区间"]
    scan --> found{"找到?"}
    found -->|"否"| assert_fail["AssertionError: 形状超出编译范围"]
    found -->|"是"| entry_ok["返回对应 entry"]
    has_sym -->|"否"| static["取唯一已编译 entry"]
    static --> check_count{"compiled_entries 数量 == 1?"}
    check_count -->|"否"| count_err["AssertionError"]
    check_count -->|"是"| entry_ok
    exact_entry --> run["range_entry.runnable(*args)"]
    entry_ok --> run

[Design inference and architectural trade-offs]

During serialization,bundled_autograd_cache 📎 vllm/compilation/piecewise_backend.py:222is also temporarily enabled, which echoes the logic in_get_vllm_functorch_config—whenVLLM_USE_MEGA_AOT_ARTIFACTis not enabled, this config isFalse 📎 vllm/compilation/compiler_interface.py:160-161, while during serialization it is forced toTrue, ensuring the artifacts are packaged.

load_all_rangesis the warm-start path; it asserts that every range can find a corresponding key incompiled_runnables, otherwise it throws an error containing the list of available keys📎 vllm/compilation/piecewise_backend.py:329-339. This error message is designed very practically—it directly lists the available keys, making it easy to troubleshoot cache version mismatches.

CUDA Graph wrapper: capture, replay, and nested dispatch

Intuitive model

CUDA Graph records "a sequence of kernel launches" into a static graph, and thereafter each replay requires only one API call.CUDAGraphWrapperis the executor of recording and replay. Its core challenge is: vLLM's batch size is dynamic, while CUDA Graph requires fixed input addresses. The solution is "capture by batch descriptor tiers"—record one graph per shape tier, and at runtime look up and replay by descriptor.

Data structures: CUDAGraphEntry and dispatch contract

CUDAGraphEntryholds three key fields:batch_descriptoras the dispatch key📎 vllm/compilation/cuda_graph.py:128-135、cudagraphis the captured graph object📎 vllm/compilation/cuda_graph.py:128-135、outputis the output at capture time (stored as a weak reference to save memory)📎 vllm/compilation/cuda_graph.py:128-135。input_addressesis used only in debug mode to verify that input addresses match during replay📎 vllm/compilation/cuda_graph.py:128-135。

CUDAGraphWrapperThe class documentation of precisely describes the dispatch contract: at initialization, allocate a runtime mode (FULL or PIECEWISE)📎 vllm/compilation/cuda_graph.py:158-158; at runtime, receive runtime_mode and batch_descriptor from the forward context and "blindly trust them"📎 vllm/compilation/cuda_graph.py:158-158; if runtime_mode is NONE or does not match, directly call📎 vllm/compilation/cuda_graph.py:158-158; otherwise perform capture or replay📎 vllm/compilation/cuda_graph.py:158-158。

The documentation also specifically declares a boundary: "CUDAGraphWrapper does not store persistent buffers or copy any runtime inputs into that buffers for replay"📎 vllm/compilation/cuda_graph.py:164-164. This means input buffer management is the caller's responsibility—the wrapper is only responsible for the graph itself.

Scenario-driven: one capture and one replay

Capture path: when__call__is triggered and runtime_mode matches, first check whether the forward context is available. If not available (such as the forward pass of a vision encoder), directly call the underlying function📎 vllm/compilation/cuda_graph.py:232-233. This is a key branch in multimodal scenarios—the ViT forward pass does not go through CUDA Graph.

Next, retrievebatch_descriptorandcudagraph_runtime_mode 📎 vllm/compilation/cuda_graph.py:242-244. If mode is NONE or does not match, directly call📎 vllm/compilation/cuda_graph.py:246-256. This "pass through on mismatch" design allows nested wrappers to coexist: the FULL wrapper on the outside and the PIECEWISE wrapper on the inside, with only one activated at runtime.

If the entry'scudagraphis None, enter capture. First callvalidate_cudagraph_capturing_enabled()to validate legality📎 vllm/compilation/cuda_graph.py:279, then record the input address📎 vllm/compilation/cuda_graph.py:281-284, createtorch.cuda.CUDAGraph() 📎 vllm/compilation/cuda_graph.py:285。

There are several key operations in the capture context. Ifgc_disableis enabled, patch outgc.collectandtorch.accelerator.empty_cache 📎 vllm/compilation/cuda_graph.py:288-303. The comment explains the reason: in piecewise mode, each layer must capture a graph, and repeated GC would make capture extremely slow, so "only run gc for the first graph, and disable gc for the rest"📎 vllm/compilation/cuda_graph.py:289-294. Next, set the graph pool id📎 vllm/compilation/cuda_graph.py:305-308, and synchronize the offloader's copy stream📎 vllm/compilation/cuda_graph.py:310-312。

The actual capture is executed in thetorch.cuda.graph(cudagraph, pool=..., stream=...)contextself.runnable(*args, **kwargs) 📎 vllm/compilation/cuda_graph.py:315-321. After capture, callget_offloader().join_after_forward()to avoid unjoined stream errors📎 vllm/compilation/cuda_graph.py:322-326. Ifweak_ref_outputis enabled, convert output to a weak reference to save memory📎 vllm/compilation/cuda_graph.py:327-334. Finally, the entry saves the weak-reference output and the graph object📎 vllm/compilation/cuda_graph.py:338-339, butreturns the original output rather than the weak reference—the comment emphasizes that this is to let PyTorch correctly manage memory during capture📎 vllm/compilation/cuda_graph.py:343-346。

Replay path: if the entry already has a graph, verify that input addresses match in debug mode📎 vllm/compilation/cuda_graph.py:348-357, then synchronize the offloader📎 vllm/compilation/cuda_graph.py:359-361, callentry.cudagraph.replay()and returnentry.output 📎 vllm/compilation/cuda_graph.py:362-363。

Design consideration: Why should the output be a weak reference while the return should be a strong reference

This isCUDAGraphWrapperOne of the most counterintuitive points inoutputis managed by PyTorch's cudagraph pool during capture📎 vllm/compilation/cuda_graph.py:320. If the entry strongly references the output, the GPU memory occupied by this graph can never be released; but if it is converted to a weak reference during capture, PyTorch may reclaim the memory before capture completes, causing capture to fail. Therefore, the code uses a weak reference inside the capture block📎 vllm/compilation/cuda_graph.py:334, stores a weak reference in the entry📎 vllm/compilation/cuda_graph.py:338, but the function return value is a strong reference📎 vllm/compilation/cuda_graph.py:346. This "triple reference state" is a precise balance between memory safety and GPU memory efficiency.

Another noteworthy design is_all_instancesthisWeakSet 📎 vllm/compilation/cuda_graph.py:173-176. It allowsclear_all_graphsto clear all wrapper graphs at once📎 vllm/compilation/cuda_graph.py:173-176, used for emergency reclamation when GPU memory is tight. UsingWeakSetinstead of a normal set is to avoid preventing wrappers from being GC'd - otherwise the wrappers themselves would leak.

Production pitfalls:__getattr__'s implementation throws an error with context for nonexistent attributes in debug mode📎 vllm/compilation/cuda_graph.py:211-217. This seems trivial, but when troubleshooting "why a certain method call failed," being able to see the string description of the runnable wrapped by the wrapper is much more useful than bareAttributeError.

Design consideration: Decoupling compilation from CUDA Graph

The design document clearly records the motivation for this refactor. Early piecewise compilation was intended to support piecewise CUDA Graph capture, excluding operators that do not support CUDA Graph (mainly attention)📎 docs/design/cuda_graphs.md:25. Later, full CUDA Graph support was added, but "this tight coupling between compilation and cudagraph capture led to an all-or-nothing experience with little flexibility"📎 docs/design/cuda_graphs.md:25。

The refactored goals are fourfold: explicitly distinguish prefill/mixed and uniform-decode batches and capture them separately📎 docs/design/cuda_graphs.md:25-25; decouple CUDA Graph capture logic from compilation, so that "capturing piecewise and full cudagraphs using the same compiled graph"📎 docs/design/cuda_graphs.md:25-25; dispatch at runtime based on batch composition📎 docs/design/cuda_graphs.md:25-25; centralize control to reduce complexity📎 docs/design/cuda_graphs.md:25-25。

BatchDescriptoris the core structure of the dispatch key, containingnum_tokens、num_reqs、uniform、has_lorafour fields📎 docs/design/cuda_graphs.md:86-93。uniformThe flag is especially critical - many attention backends only support full CUDA Graph when the batch is uniform📎 docs/design/cuda_graphs.md:95-95. The document also anticipates that this structure may be extended, for example by addinguniform_query_lento support multiple uniform decode lengths📎 docs/design/cuda_graphs.md:95-95。

The dispatch priority isFULL > PIECEWISE > None, and if the dispatch key does not exist, it falls back to NONE mode for eager execution📎 docs/design/cuda_graphs.md:112-115. This "degrade rather than error" strategy ensures that any batch combination can execute, just with different performance.

AttentionCGSupportThe enum quantifies the backend's CUDA Graph capability, with valuesALWAYS=3 > UNIFORM_BATCH=2 > UNIFORM_SINGLE_TOKEN_DECODE=1 > NEVER=0 📎 docs/design/cuda_graphs.md:153-162. Hybrid attention models (such as mamba mixer) take the minimum capability across all backends and downgrade the CUDA Graph mode accordingly📎 docs/design/cuda_graphs.md:173-175. This design decouples "capability declaration" from "mode selection" - adding a new backend only requires declaring capabilities, and the downgrade strategy takes effect automatically.

Chapter summary

Chapter review and self-test

The next chapter will turn to quantization and custom kernels, looking at how vLLM intervenes in precision control starting from the weight loading stage, and uses highly specialized operators to truly convert quantization gains into throughput improvements.

This chapter analyzed the two-layer mechanism of vLLM compilation acceleration. The first layer is CompilerInterface and PiecewiseBackend: the former defines the compiler adaptation contract and cache hashing strategy, using AlwaysHitShapeEnv to bypass the problem of a missing Dynamo context; the latter compiles a single FX subgraph into multiple shape tiers and dispatches at runtime according to the number of tokens. The second layer is CUDAGraphWrapper: it captures CUDA Graphs by tier according to BatchDescriptor, and implements nested dispatch through runtime mode matching, allowing FULL and PIECEWISE modes to coexist on the same compiled graph. The decoupling of the two is the core of this refactor—the compilation artifact can be reused by both CUDA Graph modes, and CUDA Graph can also work independently of compilation. However, compilation and graph capture solve scheduling overhead, while the model's own weight precision and operator efficiency remain another main optimization line. The next chapter will turn to quantization and custom kernels, looking at how vLLM parses quantization configurations, completes format conversion such as FP8/INT4/AWQ/GPTQ during weight loading, and further squeezes hardware performance with the help of _custom_ops and Triton kernels.

In the previous chapter, we saw that torch.compile and CUDA Graph push Python scheduling and kernel launch overhead to the extreme. But no matter how fast scheduling is, if the weights themselves are FP16 and matrix multiplication uses a generic GEMM, hardware compute power is still dragged down by memory bandwidth and inefficient operators. Quantization and custom kernels are another orthogonal main optimization line: the former lowers precision at the weight loading stage, while the latter truly converts quantization gains into throughput. This chapter starts from the parsing entry point of quantization configuration and goes all the way to operator registration in _custom_ops and Triton kernel scheduling.

11.1 Quantization Configuration: From CLI String to QuantKey

Intuitive model

The role of the quantization configuration module is like a restaurant's menu translator. The user says at the front desk, "I want fp8_per_tensor" (CLI string), while the kitchen needs the precise recipe number (QuantKey). The translator must handle three kinds of input: pure CLI shorthand, quantization metadata carried by the checkpoint, and the combined scenario where both are overlaid. Without this layer of translation, the kitchen would receive a bunch of ambiguous strings and be unable to decide which kernel to call.

Data structures and memory layout

The core data structures areQuantSpecandQuantizationConfigArgs. The former describes the weight and activation quantization keys of a single layer type (linear or MoE), while the latter is the user-visible top-level configuration.

📎 vllm/config/quantization.py:73-99

@config
class QuantSpec:
    weight: QuantKeyField = None
    activation: QuantKeyField = None

    def __str__(self) -> str:
        def quant_key_str(quant_key: QuantKey | None) -> str:
            if quant_key is None:
                return "None"
            return next(
                (
                    name
                    for name, known_quant_key in QUANT_KEY_NAMES.items()
                    if known_quant_key == quant_key
                ),
                str(quant_key),
            )
        return quant_key_str(self.weight)

weightandactivationare both optionalQuantKey。NoneThe semantics of is "fall back to the method class's own default value"—typically inherited from the checkpoint, and in online quantization scenarios it means no quantization.📎 vllm/config/quantization.py:74-74。QuantKeyitself is a complex type containingNamedTupleandClassVar[GroupShape]declarations, and pydantic cannot directly introspect it, so the author usedGetPydanticSchemato inject a custom validator_coerce_quant_key, normalizing strings orQuantKeyuniformly.📎 vllm/config/quantization.py:60-69。

QuantizationConfigArgsThe field layout of is worth noting📎 vllm/config/quantization.py:102-126：

• linear / moe: they act on theLinearBaseandFusedMoEFactorylayers respectively;
• ignore: a list of layer names to skip quantization for; online quantization also supports fnmatch wildcards;
• targets: per-layer online quantization overrides; keys can be exact layer names,re:-prefixed regexes, or fnmatch patterns; values are mutually exclusive withlinear/moe.

targetsandlinear/moeare enforced to be mutually exclusive bymodel_validator📎 vllm/config/quantization.py:172-179. This constraint is not formalism:targetsfollows the per-layer override path,linear/moefollows the global default path, and having both at the same time makes "which spec a given layer actually uses" undecidable.

Step-by-Step: one resolution of--quantization fp8_per_tensor

Scenario: the user passes--quantization fp8_per_tensoron the command line, and at the same time specifies activation quantization for MoE layers via--quantization-config.

Step one,resolve_quantization_configis called with the CLI string and the config dict📎 vllm/config/quantization.py:233-235. It first checks whetherquantizationis inONLINE_QUANT_SHORTHAND_NAMES—this tuple contains all shorthand names plus a"online" 📎 vllm/config/quantization.py:216-222。

Step two,fp8_per_tensorhits the shorthand table,baseis resolved to_ONLINE_SHORTHANDS["fp8_per_tensor"], i.e. both linear and moe usekFp8StaticTensorSym 📎 vllm/config/quantization.py:188-190。

Step three,quantization_configis non-empty and is constructed as aQuantizationConfigArgsobject. Then it enters the merge logic📎 vllm/config/quantization.py:267-268: each field is decided byquantization_config.xxx or base.xxx—fields explicitly set by the user take precedence, and unset fields inherit the shorthand defaults. Usingorhere instead ofif is not Noneis intentional:QuantSpecand an empty list are both falsy, so semantically "unset" and "empty" are equivalent.

Step four, ifquantizationis not in the shorthand table (for example it is the checkpoint's ownawq), andquantization_configisNone, the function directly returnsNone 📎 vllm/config/quantization.py:256-257. This means "do not layer on online quantization", and the checkpoint's quantization method remains dominant.

There is an easily overlooked branch:_DEFERRED_ONLINE_SHORTHANDScontainsmxfp4andmxfp8 📎 vllm/config/quantization.py:233-235. These two names are both CLI shorthands and checkpoint quantization method names. When the user passes only--quantization mxfp4withoutquantization_config, the function returnsNonerather thanbase 📎 vllm/config/quantization.py:267-268, deferring the decision to the checkpoint metadata—only when the checkpoint has no quantization information does it fall back to the online shorthand.

flowchart TD
    start["resolve_quantization_config(quantization, quantization_config)"]
    check_shorthand{"quantization in ONLINE_QUANT_SHORTHAND_NAMES?"}
    checkpoint_path{"quantization_config is None?"}
    return_none1["return None (checkpoint 主导)"]
    build_args["QuantizationConfigArgs(**quantization_config)"]
    get_base["base = _ONLINE_SHORTHANDS.get(quantization)"]
    cfg_none{"quantization_config is None?"}
    deferred{"quantization in _DEFERRED_ONLINE_SHORTHANDS?"}
    return_none2["return None (推迟到 checkpoint)"]
    return_base["return base"]
    merge["逐字段合并: cfg.xxx or base.xxx"]
    return_merged["return 合并后的 QuantizationConfigArgs"]

    start --> check_shorthand
    check_shorthand -->|否| checkpoint_path
    checkpoint_path -->|是| return_none1
    checkpoint_path -->|否| build_args
    check_shorthand -->|是| get_base
    get_base --> cfg_none
    cfg_none -->|是| deferred
    deferred -->|是| return_none2
    deferred -->|否| return_base
    cfg_none -->|否| merge
    merge --> return_merged

Design considerations and pitfalls

_coerce_specThe validator handles a subtle scenario: whenlinearormoereceives a string, it first looks up_ONLINE_SHORTHANDS, and if it hits, it takes the spec of the corresponding field; if it misses, it treats it as a singleQuantKeyname📎 vllm/config/quantization.py:130-139. This meanslinear="fp8_per_tensor"andlinear="fp8_per_tensor_static"follow two different paths—the former is a full config shorthand, the latter is a single quantization key. If that field in the shorthand isNone(for exampleint8_per_channel_weight_onlyhas nolinearfield), it throws an explicitValueErrorrather than silently returningNone 📎 vllm/config/quantization.py:130-139。

A common trap in production:targetsregex keys are precompiled and validated in_validate_targets, but fnmatch pattern keys are not validated. If the user writes an fnmatch pattern that will never match any layer, no error is raised; that layer simply remains unquantized—when troubleshooting, you need to check whether the layer names actually match.📎 vllm/config/quantization.py:166-167: operator registration and fake implementations

11.2 _custom_opsIntuitive model

is the adaptation layer between vLLM and the underlying CUDA/C++ operators, like a customs checkpoint. PyTorch's

_custom_ops.pynamespace registers compiled C++ operators, but calling them directly has three problems: different platforms (CUDA/ROCm/CPU/XPU) have different operator sets,torch.ops._Cneeds fake implementations to infer output shapes, and some operators need Python-side argument preprocessing.torch.compileencapsulates these problems uniformly._custom_opsData structures and registration mechanism

When the module loads, it first calls

, giving the platform layer a chance to import its own operator library. Then it definescurrent_platform.import_kernels() 📎 vllm/_custom_ops.py:25-26—underregister_fakeit is an empty decorator, and at runtime it imports fromTYPE_CHECKINGtorch.libraryThe core purpose of the fake implementation is to let📎 vllm/_custom_ops.py:25-26。

know the operator's output shape and dtype during tracing, without actually executing it. Taketorch.compileas an example:scaled_fp4_quantCopy

📎 vllm/_custom_ops.py:90-100

if hasattr(torch.ops, "_C") and hasattr(torch.ops._C, "scaled_fp4_quant"):

    @register_fake("_C::scaled_fp4_quant")
    def _scaled_fp4_quant_fake(
        input: torch.Tensor,
        input_scale: torch.Tensor,
        is_sf_swizzled_layout: bool,
    ) -> tuple[torch.Tensor, torch.Tensor]:
        n = input.shape[-1]
        m = input.numel() // n
        return create_fp4_output_tensors(m, n, input.device, is_sf_swizzled_layout)

guard: the fake implementation is defined only when the platform actually registershasattr. This ensures that importing the module on CPU or older GPUs will not crash due to missing operators._C::scaled_fp4_quantshows the memory layout details of FP4 quantized output

create_fp4_output_tensors. When📎 vllm/_custom_ops.py:69-87, the scale tensor needs to be arranged in the 128x4 tile layout required by Tensor Cores: the number of rows is rounded up to a multiple of 128, the number of columns (is_sf_swizzled_layout=True) is rounded up to a multiple of 4, and every 4 float8_e4m3 values are packed into one int32n // 16. The comment explicitly states that the NVFP4 quantization kernel will explicitly zero all padded scale entries, so a separate zero-initialization kernel is not needed📎 vllm/_custom_ops.py:55-64Step-by-Step: the call flow of one AWQ GEMM📎 vllm/_custom_ops.py:60-61。

Scenario: the model has loaded an AWQ-quantized weight, and during forward propagation it needs to perform matrix multiplication between activations and the quantized weight.

Step one, call

. The function first checks the environment variableawq_gemm 📎 vllm/_custom_ops.py:587-592. If true, it lazily importsVLLM_USE_TRITON_AWQand calls it—this is a pure Triton implementation path, used for platforms that do not support CUDA operators or for debugging scenarios.awq_gemm_tritonStep two, the default path calls

, passing input, qweight, scales, qzeros, andtorch.ops._C.awq_gemmStep three, ifsplit_k_iters 📎 vllm/_custom_ops.py:598-598。

第三步，如果 torch.ops._C.awq_gemmexists, the fake implementation is registered📎 vllm/_custom_ops.py:601-616. The shape returned by fake is(split_k_iters, num_in_feats, qweight.size(1) * 8)then.sum(0)—this precisely simulates the intermediate result shape of split-K and the final shape after reduction.qweight.size(1) * 8From AWQ's packing method: each int32 stores 8 4-bit weights.

Step four,awq_dequantizefollows a similar path📎 vllm/_custom_ops.py:553-559, but the fake implementation's shape derivation differs:out_c = qout_c * 8, because after dequantization the number of columns expands by 8x📎 vllm/_custom_ops.py:587-592。

The repack functions in the Marlin series demonstrate another pattern.gptq_marlin_repackThe fake implementation of computespack_factor = 32 // num_bits, and the output shape is(size_k // 16, size_n * 16 // pack_factor) 📎 vllm/_custom_ops.py:1103-1119. Here16is the Marlin tile size,size_k // 16indicates that the K dimension is split by tile. The MoE version ofgptq_marlin_moe_repackloops over each expert at the Python layer and calls the single-expert repack📎 vllm/_custom_ops.py:1154-1172, and assertssize_k % 16 == 0—this is a hard constraint of the Marlin format.

flowchart LR
    input["input: torch.Tensor (FP16/BF16)"]
    qweight["qweight: torch.Tensor (INT32 packed)"]
    scales["scales: torch.Tensor"]
    qzeros["qzeros: torch.Tensor"]
    check_env{"VLLM_USE_TRITON_AWQ?"}
    triton_path["awq_gemm_triton(input, qweight, scales, qzeros, split_k_iters)"]
    cuda_path["torch.ops._C.awq_gemm(...)"]
    output["output: torch.Tensor (FP16/BF16)"]

    input --> check_env
    qweight --> check_env
    scales --> check_env
    qzeros --> check_env
    check_env -->|是| triton_path
    check_env -->|否| cuda_path
    triton_path --> output
    cuda_path --> output

Design considerations and pitfalls

The fake implementation must exactly match the output shape of the real operator, otherwise the graph traced bytorch.compilewill have shape mismatches at runtime.create_fp4_output_tensorsThe comments in particularly emphasize "Must match the C++ scaled_fp4_quant_func allocation exactly when padded_n is None"📎 vllm/_custom_ops.py:69-74. This is an error-prone point: if the C++ side changes the allocation logic and the fake is not synchronized, the compiled graph will crash during CUDA Graph replay.

Another pitfall istorch.library.custom_op's aliasing rules.safeFusedQuantizeNvThe comments in point out that torch 2.12+ does not allow the output of a custom operator to alias any input, so the author changed the returned tensor to an in-place parameter📎 vllm/_custom_ops.py:4650-4655. This practice of "changing the API form to bypass framework limitations" is very common in operator adaptation layers, and when troubleshooting, you need to pay attention to whether themutates_argsdeclaration is consistent with the actual behavior.

CPUDNNLGEMMHandlerdemonstrates another resource management pattern: the handler pointer is stored in an int64 tensor,__del__and on callsrelease_dnnl_matmul_handlerto release📎 vllm/_custom_ops.py:3708-3717. Storing the pointer in a tensor is to prevent it from being optimized away by Python's integer inlining—this is a classic technique in low-level bindings.

11.3 Triton kernel dispatch:KernelOverrideand cross-module rebinding

Intuitive model

The role of the Triton kernel dispatcher is like a company's job stand-in system. When a platform (such as ROCm) needs to replace a Triton kernel in the vLLM core with its own implementation, it cannot directly modify the core code—that would pollute upstream.dispatcherallows the platform to register a stand-in, and then quietly replaces all references pointing to the original kernel with the stand-in. Without this mechanism, every platform would have to maintain a fork, causing constant conflicts when merging upstream changes.

Data structures and memory layout

The core data structure is the_registrydictionary and theKernelOverrideclass📎 vllm/triton_utils/dispatcher.py:29-36。

KernelOverride's key fields📎 vllm/triton_utils/dispatcher.py:50-61：

• _impl: platform implementation function;
• arg_names: a tuple mirroring the original kernel's parameter names, used for keyword binding at launch time;
• constexprs: constexpr declarations inherited from the original kernel;
• func: points to the implementation function, for warmup introspection;
• _forward_by_name: a boolean flag that determines whether to forward parameters by keyword or by position at launch time.

_forward_by_nameThe computation logic of is: compareinspect.signature(impl).parameterswith the original kernel'sarg_namesto see whether they are exactly equal📎 vllm/triton_utils/dispatcher.py:50-61. If equal, it means the implementation's parameter names match the kernel, and it is safe to forward by keyword; otherwise, it must forward by position in the original kernel's parameter order.

Step-by-Step: oneregister_kernelsrebinding

Scenario: the ROCm platform callsregister_kernels({"vllm.v1.sample.rejection_sampler.expand_kernel": my_expand_impl})。

during initialization. Step one,register_kernelsiterates over overrides, and for each name calls_resolve_kernel 📎 vllm/triton_utils/dispatcher.py:162-166。_resolve_kernelto split the name by the last.into a module name and an attribute name📎 vllm/triton_utils/dispatcher.py:83-94. If the first letter of the last segment of the module name is uppercase, it means the kernel belongs to some class (JIT warmup owner), and you need to first import the parent module and thengetattrto get the class, returning(类, 属性名); otherwise, import the module itself and return(模块, 属性名)。

. Step two, after obtaining the original kernel object, construct theKernelOverridewrapper and record it in_registry 📎 vllm/triton_utils/dispatcher.py:167-169。

. Step three,_rebind_kernelsperforms a full-module scan📎 vllm/triton_utils/dispatcher.py:97-144. It iterates oversys.modulesof all modules in__dict__, and performs an identity comparison on each attribute value—note that it isisrather than==, because some attribute values (such as thePlaceholderModulesentinel) trigger imports or exceptions during hash/eq📎 vllm/triton_utils/dispatcher.py:116-123。

. Step four, for attributes that match the original kernel, directlysetattrreplace them with the wrapper📎 vllm/triton_utils/dispatcher.py:125-135. For the JIT warmup owner (the object whose instance attributekernelpoints to the original kernel), replacevalue.kerneland clear the cached_kernel_arg_names, so that launch binding is re-derived from the wrapper📎 vllm/triton_utils/dispatcher.py:138-139。

. Step five,_rebind_kernelsonly after completion, also replace the attribute at the definition site with the wrapper📎 vllm/triton_utils/dispatcher.py:170-174. The comments explain the importance of the order: if the definition site is replaced first, the original kernel can no longer be found during the scan📎 vllm/triton_utils/dispatcher.py:170-171。

sequenceDiagram
    participant Platform as "ROCm 平台"
    participant Dispatcher as "register_kernels"
    participant Resolver as "_resolve_kernel"
    participant Scanner as "_rebind_kernels"
    participant Modules as "sys.modules"

    Platform->>Dispatcher: register_kernels({"vllm...expand_kernel": my_impl})
    Dispatcher->>Resolver: _resolve_kernel("vllm...expand_kernel")
    Resolver-->>Dispatcher: (module, "expand_kernel")
    Dispatcher->>Dispatcher: KernelOverride(original, my_impl)
    Dispatcher->>Scanner: _rebind_kernels([(original, wrapper)])
    Scanner->>Modules: 遍历所有模块 __dict__
    Modules-->>Scanner: 属性值列表
    Scanner->>Scanner: lookup(value) 身份比较
    Scanner->>Modules: setattr(module, attr, wrapper)
    Scanner->>Modules: value.kernel = wrapper (JIT owner)
    Scanner-->>Dispatcher: 重绑定完成
    Dispatcher->>Modules: setattr(host, attr, wrapper)
    Dispatcher-->>Platform: 注册完成

Design considerations and pitfalls

KernelOverride.__getitem__returnsself._launch, makingkernel[grid](**kwargs), a standard Triton launch syntax, transparent to the wrapper📎 vllm/triton_utils/dispatcher.py:63-74。_launch's forwarding logic has three cases📎 vllm/triton_utils/dispatcher.py:63-74: when there are positional arguments, pass them through directly;_forward_by_namewhen true, forward by keyword; otherwise, check whether kwargs contains parameter names unknown to the original kernel, and if so, raiseRuntimeError; if not, extract values in the original kernel's parameter order and forward by position.

ThisRuntimeErroris an important defense: if the parameter names implemented by the platform are inconsistent with the kernel, and the caller passes parameters that the implementation does not recognize, silently ignoring them will lead to error results that are difficult to troubleshoot. Explicit error reporting exposes the problem at the registration stage.

A production environment pitfall:_rebind_kernelsThe scan is O(number of modules × number of attributes × number of kernels). For large models,sys.modulesthere may be thousands of modules, each with hundreds of attributes. Although it is executed only once during initialization, if many kernels are registered, startup time will increase significantly.lookupThe function uses linear scanning rather than hash lookup, and the comment explains the reason—some attribute values are unhashable📎 vllm/triton_utils/dispatcher.py:116-123. This is a typical "correctness over performance" trade-off.

Another pitfall:_resolve_kernelIt determines whether something is a class attribute by "the first letter of the last segment of the module name being uppercase"📎 vllm/triton_utils/dispatcher.py:83-94. If a module name happens to start with an uppercase letter (which does not conform to Python naming conventions but is syntactically legal), it will be misjudged as a class. This is a convention-over-configuration design that relies on vLLM's internal naming conventions.

Design considerations

The two layers of mechanisms, quantization configuration and operator registration, together form vLLM's "accuracy-performance" adjustment surface.QuantizationConfigArgsThe design reflects the separation of "user intent" and "method default value":NoneIt is not "no quantization," but "let the method class decide for itself." This delayed decision allows the same configuration to adapt to both checkpoint quantization and online quantization scenarios.

_custom_opsThe fake implementation pattern oftorch.compileis standard in the ecosystem, but vLLM's uniqueness lies inhasattrthe widespread use of guards. This allows the same module to be imported on CUDA, ROCm, CPU, and XPU without crashing, at the cost that each operator requires three pieces of code: Python wrapper, fake implementation, and platform guard.

The cross-module rebinding of the Triton dispatcher is a radical approach. It does not rely on Python's import hooks or__getattr__, but directly scans and replaces all references. The advantage of this approach is that it is thorough—no matter how many places the kernel isfrom mod import kernelcopied to, it can be replaced; the disadvantage is that it is fragile—any new way of holding a kernel reference (such as closure capture) may escape the scan.

Chapter summary

Chapter review questions and self-test

, the extracted parameter order will be scrambled, causing the kernel to receive incorrect parameter values.

This chapter dissects the two-layer infrastructure of vLLM quantization and custom kernels. The first layer is quantization configuration parsing: QuantSpec and QuantizationConfigArgs uniformly normalize CLI strings, checkpoint metadata, and per-layer overrides into a QuantKey; resolve_quantization_config handles shorthand expansion and field merging; _DEFERRED_ONLINE_SHORTHANDS resolves name-conflict scenarios. The second layer is operator adaptation: _custom_ops implements cross-platform operator registration through hasattr guards and register_fake, where the fake implementation precisely mirrors the real operator's output shape to support torch.compile; the dispatcher implements platform replacement of Triton kernels through KernelOverride and full-module scanning. Together they support realizing quantization benefits from weight loading to forward computation. Next, we will turn to advanced inference features that improve throughput and reduce latency: how automatic prefix caching reuses KV across requests, how speculative decoding accelerates generation with a draft model, and how LoRA dynamically switches adapters.

In the previous chapter, we went deep into vLLM's quantization system and custom operator infrastructure, seeing how quantization configurations are parsed and corresponding kernels are selected, and how schemes such as FP8, INT4, AWQ, and GPTQ complete conversion during weight loading. At the same time, we explored how _custom_ops registers CUDA operators, the scheduling mechanism of Triton kernels, and how MoE fused kernels reduce memory round trips. These low-level capabilities paved the way for more advanced inference optimization. This chapter will focus on three major advanced inference features of vLLM: automatic prefix caching (APC), speculative decoding, and LoRA. They appear independent, but in fact share the same underlying infrastructure—hashing of KV blocks, slot allocation by the scheduler, and dynamic weight injection during model execution. The key to understanding them is understanding how they push "reuse" to the extreme without breaking the paging semantics of PagedAttention.

12.1 Prefix Caching: How Block Hash Fingerprints a Prefix

Intuitive Model

Prefix caching is like a library's "shared excerpt book for common passages": two students write essays, and both quote the same classical passage at the beginning. The teacher only needs to grade that passage once, and then look separately at the different parts that follow. Without it, every request would need to prefill the entire prompt from scratch, and in long-document question-answering scenarios, compute would be consumed repeatedly several times over.

Data Structure: Mapping from Tokens to Block Hash

The core of prefix caching is "how to determine that two requests have the same prefix." vLLM's answer is: split the token sequence into blocks, and compute a chained hash for each block. Chained means that the hash of the Nth block includes the hashes of the previous N-1 blocks, so a block hash uniquely fingerprints the entire prefix "from the beginning of the sequence to the end of that block."

The carrier of the hash isBlockHash, which is defined asbytesofNewType, rather than a barebytes, in order to prevent misuse at the type level📎 vllm/v1/core/kv_cache_utils.py:59-62. When a block hash needs to be combined with a KV cache group id into a dictionary key, vLLM does not use a tuple, but instead appends the 4-byte big-endian group id directly to the end of the hash bytes📎 vllm/v1/core/kv_cache_utils.py:75-76：

def make_block_hash_with_group_id(block_hash, group_id):
    return BlockHashWithGroupId(block_hash + group_id.to_bytes(4, "big", signed=False))

The hash function itself is provided byhash_block_tokens, which feeds the parent block hash, the tuple of token ids in the current block, and extra keys together into the hash function📎 vllm/v1/core/kv_cache_utils.py:650-680. Note that the parent hash of the first block is notNone, but the globalNONE_HASH：

if not parent_block_hash:
    parent_block_hash = NONE_HASH

📎 vllm/v1/core/kv_cache_utils.py:674-675。NONE_HASHThe choice of seed hides a security design: for cryptographic hashes such as SHA-256, the seed is fixed"vllm-none-hash", so that different vLLM processes compute the same hash for the same content, thereby sharing prefix caches across nodes; while for non-cryptographic hashes such as xxhash, the seed is randomized per process, because a predictable seed would allow attackers to precompute colliding blocks offline📎 vllm/v1/core/kv_cache_utils.py:105-126。resolve_none_hash_seedimplements this fork:PYTHONHASHSEEDEnvironment variables take precedence; otherwise, cryptographic hashes use a fixed seed, and non-cryptographic hashes useos.urandom(32) 📎 vllm/v1/core/kv_cache_utils.py:132-145。

Scenario-driven: block hash computation for a single request

Suppose a request enters with 128 tokens, and the block size is 16.get_request_block_hasherThe returned closure is responsible for incremental computation📎 vllm/v1/core/kv_cache_utils.py:802-861：

The first step is to determine where to start computing.start_token_idx = len(request.block_hashes) * hash_block_size 📎 vllm/v1/core/kv_cache_utils.py:812-812, that is, the number of already-computed blocks multiplied by the block size. If the remaining tokens are fewer than one block, return empty directly📎 vllm/v1/core/kv_cache_utils.py:812-812。

The second step is to handle the multimodal offset. If the starting position falls inside a multimodal input, it is necessary to useget_mm_features_in_windowto repositioncurr_mm_idx 📎 vllm/v1/core/kv_cache_utils.py:823-832. This is because the placeholder token of the multimodal input itself does not carry semantics, so the mm feature identifier and its offset within the block must be mixed into the hash as additional keys.

The third step is to loop over and compute each block.generate_block_hash_extra_keysCollect all additional keys📎 vllm/v1/core/kv_cache_utils.py:611-647, including the LoRA name, multimodal key, cache salt, and prompt embeds hash. Among these, the cache salt only takes effect on the first block📎 vllm/v1/core/kv_cache_utils.py:633-635, and this is intentional: the purpose of the salt is to isolate the entire cache namespace, so it only needs to be injected once at the starting point of the chain.

The fourth step,hash_block_tokenshash the parent hash, token tuple, and additional keys together, and use the result as the parent hash of the next block📎 vllm/v1/core/kv_cache_utils.py:851-857. The chain structure is thus formed.

Granularity conversion for multiple block sizes

When a model has multiple KV cache groups and the block sizes differ, the hash granularity and the block granularity of a group may be inconsistent.BlockHashListWithBlockSizeTo solve this problem: it does not recompute the hash, but instead leverages the property of chained hashing - the hash of a target block is the hash of the last hash block inside it📎 vllm/v1/core/kv_cache_utils.py:2781-2851. For example, when the hash block is 16 and the target block is 32, the hash of tokens 0-31 is the second 16-size hash (which already covers 0-31 through chaining)📎 vllm/v1/core/kv_cache_utils.py:2794-2806。_get_value_atThe implementation isself.block_hashes[(idx + 1) * self.scale_factor - 1] 📎 vllm/v1/core/kv_cache_utils.py:2848-2851。

flowchart TD
    req["Request 到达"] --> check{"剩余 token >= hash_block_size?"}
    check -->|否| empty["返回空列表"]
    check -->|是| mm{"起始位置在多模态窗口内?"}
    mm -->|是| reloc["get_mm_features_in_window 重定位 curr_mm_idx"]
    mm -->|否| extra
    reloc --> extra["generate_block_hash_extra_keys 收集 LoRA/MM/salt/embeds 键"]
    extra --> hash["hash_block_tokens 链式哈希"]
    hash --> append["追加到 new_block_hashes"]
    append --> advance["start_token_idx += hash_block_size"]
    advance --> check

Design considerations and pitfalls

Why use chained hashing instead of independent hashing?Independent hashing cannot distinguish the case where "the same block appears at different prefix positions." Chained hashing makes the block hash uniquely fingerprint the entire prefix, which is exactlyfind_longest_cache_hitthe prerequisite for safely reusing KV.

The cross-process pitfall of non-cryptographic hashing.If xxhash is used andPYTHONHASHSEEDis not set, then each process'sNONE_HASHis different, causing cross-instance prefix caching to fail completely.init_none_hashA warning will be printed📎 vllm/v1/core/kv_cache_utils.py:161-169. In production, if multiple instances are deployed to share a cache,PYTHONHASHSEEDmust be explicitly set or sha256 must be used instead.

The subtlety of multimodal offsets. _gen_mm_extra_hash_keysUse(mm_identifier, offset - start_token_idx)as an additional key📎 vllm/v1/core/kv_cache_utils.py:552. The offset is relative to the start of the block, so when the same mm item appears at different block positions, the hash differs, avoiding false hits.

12.2 Speculative decoding: collaboration between drafting and verification

Intuitive model

Speculative decoding is like a secretary drafting several versions of a reply for the leader first, and the leader only needs to quickly circle which version is usable. The draft model (drafter) predicts multiple candidate tokens at extremely low cost, and the target model (target) verifies these candidates in parallel in a single forward pass, accepting the matching parts. Without it, the target model can only generate token by token serially, and GPU utilization is extremely low during the decode phase.

Data structure: annotation of EAGLE groups

The core issue of speculative decoding in KV cache management is: how should the KV layers of the draft model and the KV layers of the target model be grouped?_annotate_eagle_groupsUse two rules to identify draft groups📎 vllm/v1/core/kv_cache_utils.py:2134-2189：

Rule one is spec-driven:non_causal_multi_token_decodeThe flag is declared onMLAAttentionSpec, set by the draft attention layer running non-causal multi-token decode, and can survive themergeoperation📎 vllm/v1/core/kv_cache_utils.py:2175-2177。

Rule two is positional fallback: MTP drafters (such as DeepseekV4/V4.1 DSpark) reuse the target model's own decoder layers, with no marker on the spec, but their draft attention layers are always registered after all target layers, so the group holding the last registered layer is annotated📎 vllm/v1/core/kv_cache_utils.py:2183-2184. This rule only takes effect when the group exactly partitionskv_cache_specall layers📎 vllm/v1/core/kv_cache_utils.py:2183-2184。

Scenario-driven: KV allocation for speculative decoding

Whenspeculative_configis enabled anduse_eagle_block_drop()is true,_annotate_eagle_groupsis called📎 vllm/v1/core/kv_cache_utils.py:2175-2177. The annotation resultis_eagle_groupaffects the subsequent block allocation strategy - the blocks of the draft group can be discarded after verification.

In the main path ofget_kv_cache_groups, annotation occurs after grouping📎 vllm/v1/core/kv_cache_utils.py:2364-2365. If no group is annotated as a draft group,_warn_if_unannotated_eagle_mambawill issue a warning📎 vllm/v1/core/kv_cache_utils.py:2192-2222。

sequenceDiagram
    participant Sched as Scheduler
    participant Drafter as 草稿模型
    participant Target as 目标模型
    participant KV as KV Cache Manager
    Sched->>Drafter: 请求生成 k 个候选 token
    Drafter->>KV: 分配草稿组 block (is_eagle_group=True)
    Drafter-->>Sched: 返回候选 token 序列
    Sched->>Target: 并行验证候选 (一次前向)
    Target->>KV: 读取目标组 block
    Target-->>Sched: 返回接受/拒绝掩码
    Sched->>KV: 丢弃被拒绝的草稿 block

Design considerations and pitfalls

Why do draft groups need separate annotation?Tokens generated by the draft model may be rejected after verification, and the corresponding KV needs to be discarded. If draft KV and target KV are mixed in the same group, the discard operation will accidentally affect target KV. Annotation allows the scheduler to reclaim precisely.

The fragility of the positional fallback rule.Rule two relies on the convention that "the draft layer is registered last," and the comments explicitly mark this as a hacky check and leave a FIXME📎 vllm/v1/core/kv_cache_utils.py:2158-2159. When the draft's tail cache spans multiple groups, this rule only annotates the group holding the last layer, and it needs to be generalized.

Additional constraints of the Mamba model.If speculative decoding is enabled but no group is recognized as a draft group, and a Mamba group exists, a warning is triggered📎 vllm/v1/core/kv_cache_utils.py:2211-2213. This usually means the spec of the draft layer cannot be distinguished from the target layer, and the model registration order needs to be checked.

12.3 LoRA: Dynamic Adapters Without Reloading the Base

Intuition Model

LoRA is like swapping different phone cases for the same phone: the phone itself (base model) stays unchanged, but changing the case (adapter) gives it a different style. Without it, every fine-tuning task would require loading a full set of weights, which VRAM cannot afford.

Data Structure: Dual LRU Cache and Slot Array

LoRAModelManagerTwo LRU caches are used to manage the adapter lifecycle📎 vllm/lora/model_manager.py:115-120：

self._registered_adapters: AdapterLRUCache[LoRAModel] = AdapterLRUCache(
    self.capacity, self.deactivate_adapter
)
self._active_adapters: AdapterLRUCache[None] = AdapterLRUCache(
    self.lora_slots, self._deactivate_adapter
)

capacityis the total number of adapters that can be cached on the CPU side (max_cpu_loras）📎 vllm/lora/model_manager.py:340-342，lora_slotsis the number of adapters that can be simultaneously active on the GPU side (max_loras）📎 vllm/lora/model_manager.py:345-346。_registered_adaptersWhen removed, it triggers thedeactivate_adaptercallback📎 vllm/lora/model_manager.py:71-74, ensuring that when the CPU cache evicts an entry, the GPU copy is also cleaned up.

lora_index_to_idis an array of lengthlora_slotsthat maps GPU slot indices to adapter ids📎 vllm/lora/model_manager.py:122. This array is the core index used by the punica wrapper for batched LoRA computation.

Scenario-Driven: Adapter Activation

When a request carrying a LoRA adapter comes in,activate_adapteris called📎 vllm/lora/model_manager.py:352-409：

Step one, check if already activated; if so, return directly📎 vllm/lora/model_manager.py:352-354。

Step two, find a free slot. Iterate overlora_index_to_idto find the firstNone 📎 vllm/lora/model_manager.py:362-362. If no free slot exists, throwValueError("No free lora slots") 📎 vllm/lora/model_manager.py:368-368。

Step three, update state and iterate over all wrapped modules, callingmodule.set_lora(index, lora_a, lora_b)to copy weights into the GPU's stacked buffer📎 vllm/lora/model_manager.py:377-401. If a module has no corresponding LoRA weights, callreset_lora(index)to zero them out📎 vllm/lora/model_manager.py:378-385。

Step four, if no weights were applied, print a one-time debug log📎 vllm/lora/model_manager.py:411-416. This is expected behavior under pipeline parallelism or expert parallelism—some ranks do not hold the adapted layers.

Module Wrapping: From nn.Linear to BaseLayerWithLoRA

_create_lora_modulesIterate over all named modules of the model📎 vllm/lora/model_manager.py:462-606. Key logic:

• SkipPPMissingLayer 📎 vllm/lora/model_manager.py:473-474。
• Filter based ontarget_modules: if unspecified, useis_supported_lora_moduleto determine; otherwise use_match_target_modules 📎 vllm/lora/model_manager.py:479-493。
• Handle alias modules: the same underlying module may be accessed through multiple paths (e.g., a MoE gate exists both on the block and inside the runner). In this case, redirect the alias attribute to the same wrapper, but do not register it again, otherwiseactivate_adapterwill callreset_loraon the alias and clear the weights just set📎 vllm/lora/model_manager.py:512-527。
• Usefrom_layerto create the wrapper and replace the original module📎 vllm/lora/model_manager.py:546-553。

Design Considerations and Pitfalls

Slot layout changes trigger mapping updates. set_adapter_mappingNot only compares whether the mapping has changed, but also compareslora_index_to_idthe tuple snapshot of📎 vllm/lora/model_manager.py:1323-1331. The reason is clearly stated in the comments: an out-of-bandadd_lora()may trigger LRU eviction and slot reallocation, while the running batch and its mapping remain unchanged📎 vllm/lora/model_manager.py:1323-1331. If only the mapping is checked, punica metadata will use a stale slot layout.

EP slicing for MoE.When expert parallelism is enabled, the checkpoint holds the weights of all global experts, but each rank only ownslocal_num_expertsof them._stack_moe_lora_weightsFirst reshape byglobal_num_experts, then slice[expert_start:expert_end] 📎 vllm/lora/model_manager.py:966-977. When not using EP, the slicing is a no-op.

Timing of pin_memory.Weight packing (e.g.,pack_moe) may invalidate pin_memory allocations, so pin_memory is performed after all weights are merged📎 vllm/lora/model_manager.py:916-934. The comments explicitly state two reasons: MoE models have a large number of LoRA weights, and pinning too early incurs significant overhead; packing may invalidate allocations📎 vllm/lora/model_manager.py:916-921。

Design Considerations: The Synergy of the Three

The three features converge at the KV cache management layer. Prefix caching reuses KV via block hash; speculative decoding usesis_eagle_groupannotations to distinguish draft KV; LoRA uses_gen_lora_extra_hash_keysto mix the adapter name into the block hash📎 vllm/v1/core/kv_cache_utils.py:568-581, ensuring that identical token sequences from different adapters do not mistakenly hit each other's KV.

generate_block_hash_extra_keysplaces the LoRA key at the front of the extra keys list📎 vllm/v1/core/kv_cache_utils.py:640-642, together with multimodal keys, cache salt, and prompt embeds keys, forming the complete hash input. This guarantees that even if two requests have identical tokens, as long as their LoRA adapters differ, their block hashes will differ, and KV will not be cross-used.

Chapter Summary

Chapter Review Questions

The next chapter turns to the plugin system and extensibility, looking at how vLLM supports diverse deployment forms through platform abstraction, IO processors, and endpoint extensions.

This chapter analyzed the underlying mechanisms of vLLM's three major advanced inference features. The core of prefix caching is chained block hashing: hash_block_tokens hashes the parent hash, token tuple, and extra keys together, and the NONE_HASH seed strategy balances cross-process sharing and collision safety. Speculative decoding distinguishes draft KV groups through is_eagle_group annotation. LoRA manages the adapter lifecycle through a dual LRU cache and slot array, and mixes the adapter name into the block hash to achieve cache isolation. Together, these features demonstrate the depth and flexibility of vLLM in inference optimization. Next, we will turn to vLLM's plugin system and extensibility, looking at how platform plugins adapt to new hardware, how IO processor plugins intervene in multimodal input processing, and how endpoint plugins inject custom API routes. Understanding the loading order of plugin registration and discovery will reveal how to extend vLLM's capabilities without modifying the core code.

In the previous chapter, we saw that advanced features such as prefix caching, speculative decoding, and LoRA are deeply coupled into the core paths of the scheduler, KV management, and model execution. But for an inference engine to truly move into production, performance alone is not enough - it must answer a more difficult question: when the community wants to integrate a new piece of hardware, a new multimodal input format, or a custom HTTP route, how can this be done without forking the core code? This is precisely the significance of the plugin system. vLLM's architecture is inherently multi-process: the API Server frontend process, the EngineCore process, and the Worker process corresponding to each TP/PP rank. If the plugin mechanism simply "executes a piece of code at import time," then it will either execute repeatedly in every process, causing side effects to accumulate, or execute only in the main process, causing Workers to fail to receive the extension. What this chapter will unpack is how vLLM uses Python's standard entry_points mechanism, combined with the triple constraints of group + process boundary + loading timing, to build a plugin system that can both cover all processes and precisely control the exposed surface. We focus on three main lines: platform plugins (adapting to new hardware), IO processor plugins (intervening in multimodal input processing), and endpoint plugins (injecting custom API routes). The loading strategies of the three are completely different. Understanding this difference means understanding vLLM's philosophical trade-off between "extensibility" and "security boundaries."

1. Plugin Discovery and Loading: The Grouping Contract of entry_points

Intuitive model: the plugin's "broadcast channels"

Think of vLLM's plugin system as a set of broadcast channels. When each plugin package is installed, throughsetup.py'sentry_pointsit "registers" its call sign (plugin name) and response function (plugin value) with a certain channel. vLLM scans these channels at startup and decides which channels are "listened to" in which processes.

Without this mechanism, extending vLLM could only be done by modifying source code—every time the community adds a piece of hardware, a fork must be maintained, ultimately leading to version fragmentation. The value of the grouping mechanism lies in:The same plugin package can be registered to only a specific channel, thereby being restricted to loading in specific processes。

Data structures: five group constants and a global flag

vLLM invllm/plugins/__init__.pyThe top defines five entry point group constants, each corresponding to a loading strategy:

📎 vllm/plugins/__init__.py:16-30

DEFAULT_PLUGINS_GROUP = "vllm.general_plugins"
IO_PROCESSOR_PLUGINS_GROUP = "vllm.io_processor_plugins"
PLATFORM_PLUGINS_GROUP = "vllm.platform_plugins"
STAT_LOGGER_PLUGINS_GROUP = "vllm.stat_logger_plugins"
ENDPOINT_PLUGINS_GROUP = "vllm.endpoint_plugins"

The comments hide key information:DEFAULT_PLUGINS_GROUPInAll processesLoad (process0, engine core, worker);IO_PROCESSOR_PLUGINS_GROUP Only in process0；PLATFORM_PLUGINS_GROUPLoad in all processes, but the trigger timing iscurrent_platformWhen first accessed;STAT_LOGGER_PLUGINS_GROUPOnly in process0 and in async mode;ENDPOINT_PLUGINS_GROUPOnly in the API Server frontend process.

Immediately following is a module-level global variableplugins_loaded = False 📎 vllm/plugins/__init__.py:32-33, which is the guard for idempotent loading—the comment explicitly states "make sure one process only loads plugins once".

Step-by-Step: A completeload_plugins_by_groupcall flow

Scenario: the user insetup.pyregisteredvllm.general_pluginsunderregister_dummy_model, now vLLM starts, and some process callsload_general_plugins()。

Step 1: Idempotent guard. load_general_pluginsFirst checksplugins_loaded, if alreadyTruedirectly returns📎 vllm/plugins/__init__.py:77-90. Note a subtlety here: the guard is setbeforeloading, meaning that even if subsequent loading throws an exception, it will not retry. This is intentional—plugin loading failure should not cause the process to repeatedly attempt.

Step 2: Discovery.Entersload_plugins_by_group, throughimportlib.metadata.entry_points(group=group)obtains all installed entry points under that group📎 vllm/plugins/__init__.py:36-45. If empty, logs a debug message and returns an empty dictionary.

Step 3: Log level classification.The source code distinguishes log levels for default and non-default groups:is_default_groupwhen true useslogger.debug, otherwise useslogger.info 📎 vllm/plugins/__init__.py:47-54. The motivation is practical—vllm.general_pluginsusually has a large number of model registration plugins attached, and using INFO would flood the screen; while platform/endpoint plugins are few and important, and deserve INFO visibility.

Step 4: Whitelist filtering.Readsenvs.VLLM_PLUGINS, ifNonethen loads all, otherwise only loads plugins whose names are in the list📎 vllm/plugins/__init__.py:62-70. Noteplugin.load()is wrapped in try/except, so a single plugin loading failure only logs an exception and does not affect other plugins📎 vllm/plugins/__init__.py:68-72。

Step 5: Execution.Returns toload_general_plugins, and directly calls each loaded functionfunc() 📎 vllm/plugins/__init__.py:77-90. This is why the documentation emphasizes that plugin functions must bere-entrant—it may be called multiple times in multiple processes.

The flowchart below depicts the complete decision path ofload_plugins_by_group:

flowchart TD
    start["load_plugins_by_group(group)"] --> discover["entry_points(group=group)"]
    discover --> empty{"len(discovered) == 0?"}
    empty -->|是| ret_empty["返回 {}"]
    empty -->|否| log["按 is_default_group 选 log_level"]
    log --> loop["遍历 discovered_plugins"]
    loop --> check{"allowed_plugins is None<br/>或 plugin.name in allowed?"}
    check -->|否| skip["跳过该插件"]
    check -->|是| load["func = plugin.load()"]
    load --> load_ok{"加载成功?"}
    load_ok -->|否| log_exc["logger.exception 记录"]
    load_ok -->|是| add["plugins[name] = func"]
    skip --> next["下一个插件"]
    log_exc --> next
    add --> next
    next --> loop
    loop --> ret["返回 plugins 字典"]

Design thinking: why use entry_points instead of a configuration file

---

II. Platform plugins: the abstraction layer for hardware adaptation

Intuitive model: the platform is a "hardware dialect translator"

PlatformTheclass is thesole translatorcurrent_platform.get_attn_backend_cls()、current_platform.is_cuda_alike()for all communication between vLLM and hardware. Model code only calls abstract methods such asimport torch.cuda, and never directlyif device == "xpu". Without this layer of abstraction, every new piece of hardware supported would require adding

branches in the model code, eventually turning into spaghetti.

PlatformData structures: field layout of the Platform base classvllm/platforms/interface.pyis a pure class (not used as an instance), and the key class attributes are defined at the beginning of📎 vllm/platforms/interface.py:135-179：

class Platform:
    _enum: PlatformEnum
    device_name: str
    device_type: str
    dispatch_key: str = "CPU"
    ray_device_key: str = ""
    device_control_env_var: str = "VLLM_DEVICE_CONTROL_ENV_VAR_PLACEHOLDER"
    ray_noset_device_env_vars: list[str] = []
    simple_compile_backend: str = "inductor"
    dist_backend: str = ""
    supported_quantization: list[str] = []
    additional_env_vars: list[str] = []
    _global_graph_pool: Any | None = None

_enumisPlatformEnumenum value, determiningis_cuda()、is_rocm()and other judgments📎 vllm/platforms/interface.py:69-78。device_control_env_varis the platform-independent abstraction of "device visibility environment variables"—CUDA isCUDA_VISIBLE_DEVICES, and other platforms each define📎 vllm/platforms/interface.py:151-152。_global_graph_poolis the class-level CUDA graph memory pool cache, lazily initialized throughget_global_graph_pool📎 vllm/platforms/interface.py:1210-1215。

It is worth noting the fallback logic of__getattr__: when accessing an attribute that does not exist on Platform, it tries to forward from the📎 vllm/platforms/interface.py:1189-1208namespace. This allows platform code to writetorch.<device_type>while actually callingcurrent_platform.memory_allocated(). But the source code deliberately excludes dunder methods—otherwise pickle checkingtorch.cuda.memory_allocated()would get__getstate__and try to call itNone📎 vllm/platforms/interface.py:1182-1185。

Step-by-Step: three-namespace conversion of device IDs

The easiest pitfall in platform abstraction isdevice ID namespaces. The source code comments explicitly list three kinds of📎 vllm/platforms/interface.py:275-283：

• logical: vLLM's internal local rank, indexing_assigned_physical_gpu_ids
• visible: the torch/CUDA ordinal after the current process is remapped byCUDA_VISIBLE_DEVICES
• physical: the global GPU ID used by topology APIs such as NVML, unaffected by environment variables

Scenario: a Worker process is assigned physical GPU[4, 5], environment variableCUDA_VISIBLE_DEVICES=4,5, now local rank 0 needs to be converted totorch.device("cuda:0")。

Step 1: logical → physical. device_id_to_physical_device_id(0)First checks_assigned_physical_gpu_ids, if already set then directly indexes and returns4 📎 vllm/platforms/interface.py:296-297. If not set, then splits the comma-separated list fromdevice_control_env_varand takes item 0📎 vllm/platforms/interface.py:305-311. Note that the source code deliberately treatsempty stringas unset—this is a legal configuration when Ray starts the engine on a pure CPU placement group📎 vllm/platforms/interface.py:296-297。

Step 2: physical → visible. logical_device_id_to_visible_device_id(0)After obtaining the physical4, split the environment variable into[4, 5], find the index of4and return0. If the physical ID is not in the visible list, throw📎 vllm/platforms/interface.py:316-339—this is a hard safeguard against cross-process misuse of invisible devices.RuntimeErrorThe idempotent design of

set_assigned_physical_gpu_idsis also worth noting: setting the same value repeatedly is a no-op, while setting a different value throwsRuntimeError 📎 vllm/platforms/interface.py:38-56. This prevents device mappings from being accidentally overwritten in multithreaded environments.

Registration and configuration injection of platform plugins

Platform plugins are registered through thevllm.platform_pluginsgroup, and the plugin function returns the fully qualified name of the platform class (orNoneto indicate that the current environment is not supported)📎 docs/design/plugin_system.md:50-50. The minimal implementation given in the documentation requires📎 docs/design/plugin_system.md:100-100：

• _enumis usually set toPlatformEnum.OOT（out-of-tree）
• device_typereturns the device type string recognized by PyTorch
• check_and_update_configis called early during vLLM initialization,and must be set hereworker_cls
• get_attn_backend_clsreturns the attention backend class name
• get_device_communicator_clsreturns the communicator class name

check_and_update_configis the most critical hook of the platform plugin📎 vllm/platforms/interface.py:583-592. It receives aVllmConfigreference and modifies it in place, and can adjust block size, graph mode, etc. The documentation emphasizes that "the most important thing is that worker_cls must be set here"📎 docs/design/plugin_system.md:105-105—because vLLM needs to know which Worker class to use to instantiate the worker process.

Design consideration: the three-stage strategy for block size alignment

The most complex logic in the platform interface isupdate_block_size_for_backend 📎 vllm/platforms/interface.py:666-708. It is divided into three stages to ensure that the block size is compatible with the attention backend:

Phase 1: if the user has not explicitly specified--block-size, call_preferred_block_size_for_backendsto select the smallest block size supported by all backends📎 vllm/platforms/interface.py:687-697. This function uses LCM (least common multiple) to enumerate candidate values, because some backends (such as CPU_MLA) only accept exact sizes rather than multiples📎 vllm/platforms/interface.py:622-663。

Phase 2: hybrid models (attention + mamba) need to align the block with the mamba page size📎 vllm/platforms/interface.py:699-702。

Phase 3: when multiple KV dtypes share a block pool (such as nvfp4 primary + unquantized skip layers), the primary block needs to be enlarged enough to cover the largest padded spec page📎 vllm/platforms/interface.py:704-708。

---

III. IO Processor and endpoint plugins: input processing and API extension

Intuitive model: IO Processor is a "multimodal translation layer"

The input of a multimodal model (such as LLaVA) is not plain text, but a mixture of text + images. The IO Processor plugin is responsible for converting raw multimodal data into tensors that the model can consume, and then converting the model output back into a human-readable format. It is like a customs translator: incoming foreign languages (images/audio) are translated into the model's native language, and outgoing model native language is translated back into foreign languages.

Step-by-Step: discovery and instantiation of IO Processor

Scenario: loading a model with an HF config containing aio_processor_pluginfield.

Step 1: determine the plugin name. get_io_processorPrefer the explicitly passedplugin_from_init, otherwise readhf_configfrom theio_processor_pluginfield of📎 vllm/plugins/io_processors/__init__.py:42-50. If both are empty, returnNone—indicating that the model does not need an IO processor📎 vllm/plugins/io_processors/__init__.py:52-54。

Step 2: load all installed plugins.Callload_plugins_by_group(IO_PROCESSOR_PLUGINS_GROUP)to get all plugins under that group📎 vllm/plugins/io_processors/__init__.py:59-61。

Step 3: build the loadable mapping.Iterate over each plugin, call its function to getprocessor_cls_qualname, and if it is notNone, record it inloadable_plugins 📎 vllm/plugins/io_processors/__init__.py:66-76. Note that each plugin's function call here is also wrapped in try/except, so a single failure does not affect the others.

Step 4: validate and instantiate.If the number of loadable plugins is 0, throwValueErrorindicating "an IOProcessor plugin is required but none is installed"📎 vllm/plugins/io_processors/__init__.py:66-76. If the plugin name required by the model is not in the loadable list, throwValueErrorand list all available plugin names📎 vllm/plugins/io_processors/__init__.py:80-81. Finally, resolve the class name throughresolve_obj_by_qualnameand instantiate📎 vllm/plugins/io_processors/__init__.py:80-81。

Endpoint plugins: a default-deny security posture

Endpoint plugins are the most special category in this chapter, because theyare not loaded by default。load_endpoint_pluginsThe docstring of explicitly explains the reason: endpoint plugins add HTTP routes to the API Server, expanding the network exposure surface, so a stricter "default deny" posture is adopted thanload_plugins_by_group📎 vllm/plugins/__init__.py:93-94。

The specific rule is: only when the plugin nameexplicitly appears inVLLM_PLUGINS, and itsrequired_tasksisNoneor has an intersection with the tasks supported by the server, is it loaded📎 vllm/plugins/__init__.py:108-108。

Scenario: the user installed an endpoint plugin but forgot to setVLLM_PLUGINS。

Step 1: check whether VLLM_PLUGINS is unset.Ifenvs.VLLM_PLUGINS is None, first discover the plugins under that group, and if any exist, log a warning indicating "must be explicitly allowlisted"📎 vllm/plugins/__init__.py:126-126. Note that the source code comment specifically points out:VLLM_PLUGINS=""is parsed as[""]rather thanNone, so it is treated as an "allowlist that matches no plugins" rather than "unset"📎 vllm/plugins/__init__.py:108-108. This boundary distinction is important—an empty string is an explicit "load nothing", whileNoneis "not configured".

Step 2: load and instantiate.After obtaining the factory function throughload_plugins_by_group, callfactory()one by one to instantiate📎 vllm/plugins/__init__.py:133-141. Instantiation failures are logged as exceptions and continue.

Step 3: task gating.Checkplugin.required_tasks, if it is notNoneand intersects withsupported_tasksNo intersection, skip this plugin📎 vllm/plugins/__init__.py:144-145. This allows the same plugin package to register different endpoints for different tasks (e.g., embedding vs generation).

The following sequence diagram depicts the complete interaction of an endpoint plugin from discovery to loading:

sequenceDiagram
    participant App as "API Server 前端进程"
    participant Loader as "load_endpoint_plugins()"
    participant Env as "envs.VLLM_PLUGINS"
    participant EP as "entry_points(ENDPOINT_PLUGINS_GROUP)"
    participant Factory as "plugin factory()"

    App->>Loader: load_endpoint_plugins(supported_tasks)
    Loader->>Env: 读取 VLLM_PLUGINS
    alt VLLM_PLUGINS is None
        Loader->>EP: entry_points(group)
        EP-->>Loader: discovered plugins
        Loader-->>App: 返回 [] (记 warning)
    else VLLM_PLUGINS 已设置
        Loader->>EP: load_plugins_by_group(group)
        EP-->>Loader: factories 字典
        loop 每个 factory
            Loader->>Factory: factory()
            Factory-->>Loader: EndpointPlugin 实例
            Loader->>Loader: 检查 required_tasks 交集
            alt tasks 不匹配
                Loader->>Loader: 跳过 (记 info)
            else tasks 匹配
                Loader->>Loader: append 到结果列表
            end
        end
        Loader-->>App: 返回 endpoint_plugins 列表
    end

Design consideration: Process boundaries determine loading strategy

The differences in loading strategies for the three types of plugins are essentially a mapping ofprocess boundaries:

Production pitfall: Silent degradation on plugin load failure

load_plugins_by_groupFor each plugin'splugin.load()is wrapped in try/except, and failures only log an exception📎 vllm/plugins/__init__.py:68-72. This meansa broken plugin will not prevent vLLM from starting, but it also will not give an explicit error—users may be confused about "why my plugin isn't taking effect."

Troubleshooting suggestion: Set the log level to DEBUG and search for"Failed to load plugin". If the plugin is under thevllm.general_pluginsgroup, the default log level is DEBUG, and it must be explicitly enabled to see loading details📎 vllm/plugins/__init__.py:49-50。

Another pitfall is the timing of setting theplugins_loadedguard📎 vllm/plugins/__init__.py:77-90: it is set before loadingTrue. If the first load fails for some reason (such as an entry_points scan exception), subsequent calls will return directly without retrying. This can cause the bizarre phenomenon of "the plugin works sometimes and not others" in test environments.

---

Chapter summary

vLLM's plugin system is built on Pythonentry_points, usingfive group constantsto divide extension types, usingprocess boundariesto determine the loading scope, and usingVLLM_PLUGINSan allowlistto control the loading set. Platform plugins use thePlatformbase class to abstract hardware differences, and its three-namespace device ID conversion (logical/visible/physical) is the core of cross-process device management; IO processor plugins are triggered by the HF config'sio_processor_pluginfield and are responsible for translating multimodal inputs; endpoint plugins adopt a "deny by default" posture and are loaded only when explicitly allowlisted and the task matches, in order to control network exposure surface.

The three main lines share the same discovery mechanism, but the differences in loading strategy reflect vLLM's trade-off between "extension convenience" and "security boundaries": plugins that do not expose the network are loaded by default, while plugins that expose the network must be opt-in.

Chapter review questions

---

The next chapter turns to architectural trade-offs, production pitfalls, and future evolution. We will put together the mechanisms dissected in the previous thirteen chapters, examine vLLM's trade-offs among performance, maintainability, and extensibility, and look ahead to the evolution direction of inference engines.

At this point, we have seen clearly how vLLM opens up extensibility while keeping the core code stable through the grouping mechanism of entry_points, process-boundary-aware loading timing, and differentiated strategies for three types of plugins: platforms, IO processors, and endpoints. This plugin system allows new hardware, new input formats, and new API routes to be integrated non-invasively, but extensibility itself also means more dimensions that need to be weighed. The next chapter will conclude the book, systematically sorting out the tensions in vLLM's key design decisions—continuous batching and GPU memory fragmentation, CUDA Graph and dynamic shapes, disaggregated deployment and network overhead—and provide a production environment pitfalls checklist and diagnostic path, while also looking ahead to the evolution trends of the Rust frontend, the IR layer, and heterogeneous hardware directions.

In the previous chapter, we dissected vLLM's plugin-based extension mechanism and saw how platform plugins, IO processor plugins, and endpoint plugins allow the engine to adapt to new hardware, new modalities, and new APIs without modifying the core code. This extensibility allows vLLM to quickly embrace change, but the more extension points there are, the more complex the interaction paths become in production environments. When real problems such as GPU memory fragmentation, NCCL handshake failures, compilation cache invalidation, and network jitter occur simultaneously, the mechanisms introduced in the previous thirteen chapters pull against one another, exposing tensions that were not visible in ideal environments. This chapter does not introduce new core mechanisms, but instead puts these mechanisms together, using the official troubleshooting documentation as an anchor, combining it with the design of the Rust frontend bench tool, examining the trade-offs between performance and operability, and providing an actionable diagnostic path.

I. Optimization Levels: An Explicit Contract Between Startup Time and Runtime Performance

Intuitive Model

Optimization levels are like a camera's "scene modes": auto mode (-O2) suits most scenarios, but when you need to capture quickly (debug), switching to manual mode (-O0) responds immediately at the cost of lower image quality (performance). vLLM turns this trade-off into an explicit four-level contract, rather than hiding it in dozens of boolean flags for users to assemble themselves.

Field Layout of the Four Levels

vLLM provides-O0through-O3four levels📎 docs/design/optimization_levels.md:5-5. The core design principle is:Flags explicitly set by the user take precedence over the optimization level defaults 📎 docs/design/optimization_levels.md:5-5. This means the optimization level is only a set of defaults, not a hard constraint.

-O0turns off everything: no autotuning, no compilation, no cudagraph📎 docs/design/optimization_levels.md:32-33. Specifically, it comes down to four switches:cudagraph_mode=NONE、mode=NONE, all fusion disabled,enable_flashinfer_autotune=False 📎 docs/design/optimization_levels.md:37-40。

-O1is the balance point for development scenarios: enablePIECEWISEcudagraph andVLLM_COMPILEmode📎 docs/design/optimization_levels.md:50-51. Note that there is a subtle detail here:fuse_norm_quantandfuse_act_quantare enabled only when one of the operators uses a custom kernel; otherwise Inductor's automatic fusion works better📎 docs/design/optimization_levels.md:61. This is a typical design judgment of "don't compete with the compiler for work."

-O2is the default, aimed at production📎 docs/design/optimization_levels.md:66-67. On top of-O1it addsFULL_AND_PIECEWISEcudagraph andfuse_allreduce_rms 📎 docs/design/optimization_levels.md:72-73。-O3currently equivalent to-O2, reserving📎 docs/design/optimization_levels.md:80-81。

for more aggressive experimental optimizations in the future.

Scenario-Driven Selection Flowvllm serve model -O1When a user executes

flowchart TD
    start["用户启动 vllm serve -O1"] --> parse["解析 optimization_level=1"]
    parse --> load_defaults["加载 O1 默认值集合"]
    load_defaults --> check_user{"用户是否显式设置了<br/>cudagraph_mode?"}
    check_user -->|是| user_wins["使用用户值<br/>覆盖 O1 默认"]
    check_user -->|否| use_default["使用 O1 默认<br/>PIECEWISE"]
    user_wins --> check_fusion{"fuse_norm_quant<br/>是否涉及自定义 kernel?"}
    use_default --> check_fusion
    check_fusion -->|是| enable_fuse["启用该 fusion"]
    check_fusion -->|否| skip_fuse["跳过，交给 Inductor"]
    enable_fuse --> done["配置完成，进入引擎初始化"]
    skip_fuse --> done

Copycheck_userThe key to this flow is the📎 docs/design/optimization_levels.md:5-5branch: explicit user settings always take precedence

. This avoids hard-to-troubleshoot problems such as "the optimization level silently overrode my debug flag."

Design Considerations and PitfallsThe most common production trap of optimization levels isexcessively long startup time-O0. The documentation explicitly recommends: when startup time is too long, use-O1 📎 docs/design/optimization_levels.md:87or-O0. But there is a hidden cost here—

without cudagraph, the CPU launch overhead of each kernel is exposed, and throughput may drop several times in high-concurrency scenarios.Another trap is。-O2compilation errorsFULL_AND_PIECEWISE. The-O2cudagraph has stronger assumptions about model structure; some custom models fail to compile under-O1but work normally underdebug_dump_path. The documentation recommends using📎 docs/design/optimization_levels.md:88to obtain more debugging information-O0. The troubleshooting path should be: first use-O1、-O2to confirm functional correctness, then gradually upgrade to

---

II. Production Pitfall Checklist: Diagnostic Path from Symptoms to Root Causes

Intuitive Model

Troubleshooting in production is like emergency triage: you cannot run a full battery of tests on every patient. You must first quickly narrow the scope based on symptoms (OOM, hang, crash), then dig deeper in a targeted way. vLLM's troubleshooting documentation is essentially a triage manual.

Symptom Classification and Diagnostic Tools

The documentation divides common issues into several major categories. We will walk through them in order of increasing diagnostic difficulty.

Category 1: Model download/loading hangs.The symptom is no response for a long time after startup. The root cause is usually slow network or slow shared filesystem.📎 docs/usage/troubleshooting.md:11-11. The diagnostic method is--load-format dummySkip weight loading to isolate whether it is download slowness or loading slowness📎 docs/usage/troubleshooting.md:23-23. This is a classic "binary search isolation" technique.

Category 2: GPU memory OOM.The documentation points directly to the conserving_memory configuration doc📎 docs/usage/troubleshooting.md:23. But OOM in production is often not because the model is too large, but because of KV cache fragmentation or concurrency request counts exceeding expectations.

Category 3: Generation quality changes.This is an easily overlooked pitfall. v0.8.0 changed the source of default sampling parameters: from vLLM's neutral defaults to the model author'sgeneration_config.json 📎 docs/usage/troubleshooting.md:23-23. In most cases this improves quality, but for some models the configuration is actually worse📎 docs/usage/troubleshooting.md:23-23. The diagnostic method is to fall back to--generation-config vllmcompare📎 docs/usage/troubleshooting.md:23-23。

Category 4: Hang.This is the hardest category to diagnose. The documentation provides a set of progressive debugging environment variables📎 docs/usage/troubleshooting.md:41-41：

• VLLM_LOGGING_LEVEL=DEBUG: enable verbose logging
• VLLM_LOG_STATS_INTERVAL=1.: high-frequency output of queue and cache hit status
• CUDA_LAUNCH_BLOCKING=1: locate which CUDA kernel is causing the problem
• NCCL_DEBUG=TRACE: enable NCCL verbose logging
• VLLM_TRACE_FUNCTION=1: record all function calls, but it slows things down by more than 100x📎 docs/usage/troubleshooting.md:41

There is an important operational discipline here: after debugging, you must turn off these environment variables, or directly open a new shell, otherwise the residual debugging configuration will continue to slow down the system📎 docs/usage/troubleshooting.md:11-11。

The Process Boundary Trap of Breakpoint Debugging

vLLM's multi-process architecture makes conventionalpdbbreakpoints ineffective — if a breakpoint executes in a child process, it will throwBdbQuit 📎 docs/usage/troubleshooting.md:45-54. Two solutions: useforked-pdb 📎 docs/usage/troubleshooting.md:57-61, or setVLLM_ENABLE_V1_MULTIPROCESSING=0to keep the scheduler in the same process📎 docs/usage/troubleshooting.md:63-68。

Diagnosis of Distributed Communication

There is dedicated diagnostic documentation for distributed deployment. The core recommendations are:Set environment variables at cluster creation time, because variables propagate to all nodes; setting them in the shell only affects the local node📎 docs/serving/distributed_troubleshooting.md:16-16。

A high-frequency issue isNo available node types can fulfill resource request, which occurs even when the cluster has enough GPUs📎 docs/serving/distributed_troubleshooting.md:16-16. The root cause is usually that a node has multiple IPs and vLLM chose the wrong one. The solution is to useVLLM_HOST_IPto explicitly specify, and useray statusto verify📎 docs/serving/distributed_troubleshooting.md:16-16。

Diagnostic Script for NCCL Initialization Failure

The documentation provides a complete diagnostic script that verifies the communication stack layer by layer📎 docs/usage/troubleshooting.md:89-150. Its design is very layered:

flowchart TD
    start["运行诊断脚本"] --> nccl_test["测试 PyTorch NCCL<br/>dist.all_reduce"]
    nccl_test --> nccl_ok{"value == world_size?"}
    nccl_ok -->|否| hw_broken["硬件/驱动故障<br/>联系系统管理员"]
    nccl_ok -->|是| gloo_test["测试 PyTorch GLOO<br/>CPU 通信"]
    gloo_test --> gloo_ok{"value == world_size?"}
    gloo_ok -->|否| gloo_fail["GLOO 配置问题<br/>检查网络接口"]
    gloo_ok -->|是| pynccl_test["测试 vLLM PyNcclCommunicator"]
    pynccl_test --> pynccl_ok{"all_reduce 正确?"}
    pynccl_ok -->|否| pynccl_fail["vLLM NCCL 封装问题"]
    pynccl_ok -->|是| graph_test["测试 CUDA Graph 内 all_reduce"]
    graph_test --> graph_ok{"g.replay() 后正确?"}
    graph_ok -->|否| graph_fail["CUDA Graph 捕获问题<br/>检查 stream 语义"]
    graph_ok -->|是| success["sanity check 成功"]

The brilliance of this script lies in its layer-by-layer isolation: first verify the lowest-level PyTorch NCCL, then verify CPU-side GLOO, then verify vLLM's own PyNcclCommunicator wrapper, and finally verify communication within CUDA Graph📎 docs/usage/troubleshooting.md:90-146. Each layer's failure points to a different root cause.

A noteworthy detail in the script:pynccl.disabled = Falseis for backward compatibility with 0.6.4 and below📎 docs/usage/troubleshooting.md:121-125. 0.6.5+ enables it by default, but keeping this line prevents users reading the latest documentation from being confused.

For multi-node testing, the documentation deliberately uses--rdzv_backend=staticinstead ofc10d, becausec10dwill fail due to DNS resolution failure in multi-node setups📎 docs/usage/troubleshooting.md:168-168. This is a typical "you only know after stepping on the pit" configuration.

Design Thinking and Pitfalls

NCCL Initialization Failure（ncclCommInitRankreporting unhandled system error) usually points to two root causes: missingIPC_LOCKcapability or/dev/shmnot mounted📎 docs/usage/troubleshooting.md:311-311. Both are classic traps in containerized deployment.

CUDA PTX Toolchain Mismatch（the provided PTX was compiled with an unsupported toolchain) indicates that the PTX in the wheel was compiled with a higher version of the CUDA toolkit📎 docs/usage/troubleshooting.md:325-327. The solution is to enable CUDA forward compatibility: add under Docker-e VLLM_ENABLE_CUDA_COMPATIBILITY=1 📎 docs/usage/troubleshooting.md:325-327, and on bare metal install thecuda-compatpackage and setVLLM_CUDA_COMPATIBILITY_PATH 📎 docs/usage/troubleshooting.md:325-327。

Known NCCL Memory Overhead Issue：vLLM >= 0.4.3, <= 0.10.1.1will setNCCL_CUMEM_ENABLE=0to work around an NCCL bug. External processes connecting to vLLM must also set this variable, otherwise they will hang or crash📎 docs/usage/troubleshooting.md:375. After the fix in NCCL 2.22.3, newer versions removed this override to allow performance optimization📎 docs/usage/troubleshooting.md:375. This case shows:The cross-process environment variable contract is an implicit dependency of distributed systems, and must be synchronized during upgrades.

---

III. Rust Frontend: The Zero-Copy Design Philosophy of the bench Tool

Intuitive Model

If the Python frontend is a "fully featured but heavy" Swiss Army knife, the Rust bench tool is a scalpel "built only for stress testing." Its design goal is not feature coverage, but minimizing the client's own overhead under high concurrency, so that the measured numbers truly reflect server-side performance.

Data Structures and Memory Layout

The core data structure of the bench tool isRequestFuncInput 📎 rust/src/bench/src/backends/mod.rs:59-89. It makes heavy use ofArc<str>andArc<[u32]>rather thanString/Vec, which is the core of the zero-copy design.

Look at a few key fields:prompt: Arc<str> 📎 rust/src/bench/src/backends/mod.rs:50-52——Multiple concurrent requests can share the same prompt string, avoiding cloning a copy for each request.prompt_token_ids: Option<Arc<[u32]>> 📎 rust/src/bench/src/backends/mod.rs:77——Precomputed token IDs are sent directly to the server, skipping server-side tokenization📎 rust/src/bench/src/backends/mod.rs:74-76。

The most ingenious part ismulti_modal_content: Option<Arc<[Arc<str>]>> 📎 rust/src/bench/src/backends/mod.rs:81. The comment explains: multimodal content is treated as pre-serialized JSON fragments, and the chat backend directly concatenates them into the payload byte stream, avoiding any parsing or deep copying of base64 image data📎 rust/src/bench/src/backends/mod.rs:78-80. This is a two-layerArcstructure: the outer layerArc<[...]>shares the entire array, and the inner layerArc<str>shares a single fragment.

chat_messages_json: Option<Arc<str>>has the highest priority and is concatenated directly into the payload as-is📎 rust/src/bench/src/backends/mod.rs:82-85。

Zero-allocation deserialization

Parsing SSE streaming responses is another performance-critical point. The comment explicitly states: use typed deserialization to avoid building the completeserde_json::Valuetree, and extract only the needed fields📎 rust/src/bench/src/backends/mod.rs:20-24。

CompletionChunkkeep onlychoicesandusagetwo fields📎 rust/src/bench/src/backends/mod.rs:20-24，ChatChunkSimilarly📎 rust/src/bench/src/backends/mod.rs:33-37。#[serde(default)]makes the missingchoicesfield default to an empty array📎 rust/src/bench/src/backends/mod.rs:20-24, which is a common case for streaming responses.

Scenario-driven request flow

When a stress-test request is sent, how does the data flow? The data flow diagram below shows the transformation from input to output:

flowchart LR
    input["RequestFuncInput<br/>Arc&lt;str&gt; prompt"] --> build["build_headers<br/>+ payload 拼接"]
    build --> send["reqwest::Client<br/>send_request"]
    send --> sse["SSE 流式响应<br/>字节流"]
    sse --> parse["CompletionChunk<br/>类型化反序列化"]
    parse --> output["RequestFuncOutput<br/>ttft/itl/tpot"]

BackendThe enum uses static dispatch to avoid the async trait object problem📎 rust/src/bench/src/backends/mod.rs:150-154。send_requestthroughmatchdispatches to the concrete implementation📎 rust/src/bench/src/backends/mod.rs:158-168。get_backendreturns the corresponding backend according toBackendKind📎 rust/src/bench/src/backends/mod.rs:172-181。

One detail:API_KEYusesOnceLockcaching to avoid making an environment variable syscall on every request📎 rust/src/bench/src/backends/mod.rs:186-188。build_headersinserts Content-Type, Authorization, extra headers, and request-id in sequence📎 rust/src/bench/src/backends/mod.rs:191-215。

Design reflections and pitfalls

RequestFuncOutputThe field design ofttft（time to first token）、itlis also worth noting:tpot（time per output token）📎 rust/src/bench/src/backends/mod.rs:93-105(inter-token latency array),

---

. These three metrics correspond to different performance dimensions: TTFT reflects prefill and queueing latency, ITL reflects the stability of decode, and TPOT reflects overall throughput. If stress testing only looks at average latency, it will mask ITL jitter.

Design reflection: the underlying logic of architectural trade-offs

Continuous batching allows the batch to be reorganized at every step, greatly improving throughput, but the cost is that KV cache allocation and release are extremely frequent. PagedAttention's block table mechanism is precisely designed to handle this high-frequency allocation—fixed-size blocks eliminate external fragmentation, but introduce the indirect addressing overhead of the block table and internal fragmentation (the last block may not be fully filled). This is a typical trade-off of "using an indirection layer to exchange for a lower fragmentation rate," the same idea as virtual memory paging in operating systems.CUDA Graph vs dynamic shapes.PIECEWISECUDA Graph requires static shapes, but the batch size in continuous batching changes at every step. vLLM's solution isFULL_AND_PIECEWISEand📎 docs/design/optimization_levels.md:50,72modes-O0——capture the statically capturable parts as graphs and keep the dynamic parts eager.-O2Completely disabling cudagraph is for debugging,-O1fully enabling it is for production, and the middle

is a compromise.Disaggregated deployment vs network overhead.IPC_LOCK、/dev/shm）📎 docs/usage/troubleshooting.md:311-311KV Connector allows prefill and decode to be separated into different instances, but cross-instance transfer of KV cache introduces network latency. The configuration requirements for GPUDirect RDMA in the documentation (

) show that this path has hard requirements on the infrastructure. Network jitter can cause KV transfer timeouts, which in turn trigger retries or degradation.Operability vs performance.VLLM_TRACE_FUNCTION=1Optimization levels, debugging environment variables, and diagnostic scripts are all costs paid for operability.📎 docs/usage/troubleshooting.md:41can slow things down by 100x

---

, but it is the last resort for locating hang issues. A mature engine must provide these "slow but clear" tools.

Chapter summary

This chapter concludes the book, reexamining the mechanisms from the previous thirteen chapters from a production perspective.-O0Optimization levels (-O3to📎 docs/design/optimization_levels.md:5-5) are an explicit contract between startup time and runtime performance, and user flags always take precedence over level defaultsArc. The production pitfalls checklist covers the complete diagnostic path from model loading, GPU memory OOM, generation quality changes, to distributed communication failures. The core methodology is "binary-search isolation" and "layer-by-layer verification." The Rust bench tool uses

Three core trade-off lines run through the entire book: continuous batching vs. GPU memory fragmentation, CUDA Graph vs. dynamic shapes, and disaggregated deployment vs. network overhead. Understanding these tensions is more important than memorizing any single mechanism—because every tuning decision in production is essentially about finding the balance point among these tensions.

Chapter Review and Self-Assessment

---

At this point, the source code analysis of all fourteen chapters comes to a close. We started from a single API call, passed through the scheduler, KV cache manager, attention backend, and distributed communication layer, finally reached the GPU kernel launch point, and then returned to the production operations diagnostic console. Every design decision in vLLM has clear trade-offs behind it. Only by understanding these trade-offs can you make correct engineering judgments when facing new hardware, new models, and new workloads. The evolution of inference engines will not stop—Rust frontend, IR layer, and heterogeneous hardware support are all advancing rapidly—but the underlying trade-off logic is stable, and this is the core capability this book hopes to convey.

At this point, we have completed the full journey from request entry to GPU Kernel, and have also seen the trade-offs and pitfalls in production environments that turn a system from "able to run" into "runs stably." vLLM's evolution will not stop at the current architecture—more efficient attention implementations, smarter scheduling strategies, and more seamless heterogeneous support are all on the way. But no matter how the future changes, understanding the tensions and trade-offs among these mechanisms will always be the key to mastering inference engines.