ROCm/RDNA4 (gfx1201) vLLM: native HIP kernels, baked-in quantizers and tuner - vLLM 0.29 tree, torch 2.11, ROCm 10.0, Python 3.14.
Quick start
A minimal working compose for serving on RDNA4. Point the volume at your model folder, set the tensor parallel size to your GPU count, and bring it up. The kernels described on this page are on by default and need no flags.
The capture-size ladder steps by num_speculative_tokens + 1 (here 5 + 1 = 6). An MTP decode step submits the drafts plus the verify token, so each sequence contributes 6 tokens; vLLM pads every batch up to the next captured size, so sizes that are multiples of 6 mean every concurrent-request count lands exactly on a captured graph instead of paying padding it never uses. The lone 1 covers plain single-token decode.
One addition most long-context deployments want, appended to the command: block: rope scaling for context past the model's trained length. original_max_position_embeddings is the model's REAL trained length, and factor is simply the context you want divided by that trained length. Here 262144 trained × factor 4.0 serves 1M (set --max-model-len to match). This example is the config for the Qwen3.5 / 3.6 / 3.8 27B models. The remaining values (mrope_section, partial_rotary_factor, rope_theta) are specific to each model architecture and must come from the model's own config, not copied blindly.
docker compose up -d and the OpenAI-compatible API is on port 8077. The rest of this page covers what the image does underneath and the knobs for each piece.
A MoE bigger than VRAM: experts in system RAM
If the model's experts do not fit your cards, add --enable-expert-offload. The experts that do not fit live in system RAM, and each GPU keeps a cache of the ones it uses most, in whatever VRAM is left after the weights and KV cache. The flag alone is enough: the plan sizes the VRAM cache and the system-RAM budget from what is free at boot, and even without the flag, offload turns itself on when the weight census shows the model will not fit the cards. The startup log prints the plan. This is the same compose as above on two GPUs with offload added, for Qwen3.8-Flash-Next on 2× R9700. See Serving → Expert offload for every flag.
--expert-offload-mem N is an optional manual cap on the system RAM the experts may use in total. Leave it out and the plan sizes it from the free RAM.
The same on a 64 GB machine
Qwen3.8-Flash-Next also has a 47.7 GB PLE embedding table that sits in pinned system RAM by default. On a 64 GB machine that leaves no room for the experts, so add --ple-nvme-offload: the table is written to a file next to the model at startup and read from the drive through a small RAM cache. The model folder has to be on an NVMe drive with 48 GB free; the file is deleted on stop unless --ple-cache-reuse true is set. Context and concurrency are lowered to fit.
--ple-nvme-dir puts the file somewhere else. docker run --rm tcclaviger/vllm:dev --flags lists the image's own flags. --recipes --list lists the serve recipes baked into the image, and --recipes FILTER prints the matching ones. Neither needs a GPU.
Adding the KV-scale sidecar to a checkpoint
Serving with --kv-cache-dtype fp8 needs calibrated q/k/v scales. Without them the runtime uses a scale of 1.0. Calibrated checkpoints ship the scales as a tiny sidecar file (model-kvscales.safetensors, a few KB) plus matching entries in the index. If you grabbed a checkpoint before the sidecar existed, here is how to add it by hand, using Qwen3.8-27B-RFI8-AA as the example.
1. Download the sidecar
Open the repo's file list on Hugging Face (the Files and versions tab), find model-kvscales.safetensors, and click the download icon at the end of its row. Drop the file into your model folder, right next to the model-0000N.safetensors shards. That is the whole download; the file only holds 51 scalar values.
2. Tell the index about it
In the same folder, open model.safetensors.index.json in any text editor. It has two parts: a short "metadata" block and a long "weight_map" that maps every tensor name to the shard file that holds it. The loader only reads files the weight_map mentions, so the sidecar is invisible until its tensors are listed.
For each full-attention layer, add three lines pointing at the sidecar. For this model those are layers 0, 3, 7, 11, 15, 19, 23, 27, 31, 35, 39, 43, 47, 51, 55, 59, 63 (layer 0, then every 4th from 3, the hybrid's full-attention slots). The three lines for layer 0 look like this; repeat the pattern for each layer in the list, changing only the layer number:
Paste them anywhere inside the weight_map braces, watch the commas (every line ends with one except the last entry of the whole map), and save. If your editor complains about invalid JSON, a comma is the usual culprit.
3. Serve
Add --kv-cache-dtype fp8 to the serve command. The boot log will show the scales loading with the checkpoint; nothing else changes. If your download already has the weight_map entries (newer snapshots include them), there is nothing to edit, and serving without fp8 KV works either way, the scales are simply ignored.
Stack overview
Attention, GDN linear attention, the quantized and FP8 GEMMs, the MoE experts, the tensor-parallel all-reduce and the per-op kernels run on AOT HIP kernels compiled into the image. Nothing compiles or autotunes at startup for the parts they serve. Anything a kernel does not cover falls back to Triton (or RCCL for communication) automatically.
CLAV_ATTN: full attention, head sizes 64-512, bf16 or fp8 KV cache, sliding window, speculative verify.
GDN: gated-delta-net linear attention for prefill, decode and speculative verify.
RFI / RFA GEMMs: the quantized linear layers at 2/4/6/8-bit, plus the Hadamard rotation.
FP8 GEMM: block-scaled w8a8 for fp8-block checkpoints.
libr4d: MXFP4 and W4A16 MoE experts, MXFP4 dense GEMM, the tensor-parallel all-reduce, MoE routing, the expert cache, DeepSeek-V4 kernels and DFlash drafter kernels.
Per-op kernels: the KV-cache write, the GDN causal conv, the mamba state copies, and the Qwen3.8-Flash-Next PLE and HyperConnection kernels.
On by default on RDNA4 (gfx12x) GPUs, and safe under CUDA-graph capture.
Other GPUs run the stock Triton and RCCL paths.
DISABLE_ALL_CLAV=1 turns off attention, GDN, the RFI/RFA and FP8 GEMMs, the r4d MXFP4 and W4A16 kernels, the all-reduce and all-gather, and the general per-op kernels at once; each part falls back to Triton or RCCL. Each card below lists the switch for its own kernel.
CLAV_ATTN is the native HIP full-attention backend. It is on by default on RDNA4 (gfx12x) GPUs. It sits ahead of Triton in vLLM's backend list, and anything it does not support falls back to Triton automatically, per attention group. The kernels are compiled into the image, so the layers it serves do not compile or autotune at startup.
What it serves
bf16 models with head sizes 64, 128, 256 and 512.
A bf16 KV cache, or an fp8 (e4m3) KV cache with --kv-cache-dtype fp8.
Plain decode, speculative/MTP verify, chunked prefill and long-context prefill.
Sliding-window attention, and the non-causal attention of DFlash drafters.
Attention sinks, ALiBi, logits soft-capping, per-token-head KV quantization, an e5m2 or integer KV cache, non-decoder attention, per-sequence causal masks, and any GPU other than gfx12x. A model with supported and unsupported layers runs each group on its own path.
Qwen3.5-family 27B models use r4d attention
On the Qwen3.5-family 27B geometry (head size 256, 6 query heads per KV head) with a bf16 KV cache, the r4d paged-attention backend is the default instead. With an fp8 KV cache, CLAV_ATTN is used. VLLM_CLAV_ATTN=1 keeps CLAV_ATTN on a bf16 KV cache; an explicit R4D_ATTN value overrides both.
Controls
CLAV_ATTN=0: disable; attention runs on Triton. VLLM_CLAV_ATTN=0 does the same.
CLAV_ATTN_FA1=1: run prefill on the FA1 kernel instead of the default FA2-v2.
VLLM_CLAV_PREFILL_SPLITKV=0: disable split-KV on long-context prefill.
VLLM_CLAV_PREFILL_QSLICE=N: query rows per prefill scratch slice, default 1024.
VLLM_CLAV_DECODE_SEGMENTS=N: upper bound on decode split-KV segments, default 128. The count per batch is computed from the batch shape.
R4D_ATTN=1: put the r4d paged-attention backend ahead of Triton (off by default outside the 27B case above). It serves head size 256 with 6 query heads per KV head, causal, with a bf16 KV cache and no sliding window.
GDN: linear attention
The GDN (gated delta net) linear-attention layers of hybrid models such as Qwen3.5, Qwen3.6 and Qwen3.8 run on fused AOT HIP kernels for prefill, decode and speculative verify. It is on by default on RDNA4 (gfx12x) GPUs.
What it covers
GDN layers with 128/128 head dims.
Plain prefill, plain decode, mixed decode+prefill batches, speculative verify batches, and mixed verify+prefill batches. MTP and DFlash models run every step shape on this path.
Activations in bf16, fp16 or fp32.
The conv and recurrent state use the model's dtype. The conv state layout defaults to DS; an explicit VLLM_SSM_CONV_STATE_LAYOUT is kept.
When it falls back to Triton
Layers with head dims other than 128/128, activation dtypes other than bf16/fp16/fp32, and any GPU other than gfx12x fall back to the fla-Triton kernels.
Controls
CLAV_GDN=0: disable; GDN runs on the fla-Triton kernels. NO_AMD_GDN_HIP=1 does the same.
Based upon the RDNA4 GDN work of Pat Carter (@patcarter883, rdna4-vllm, Apache-2.0).
Quantized GEMMs: RFI / RFA
The RFI and RFA linear layers run on AOT HIP GEMM and Hadamard-rotation kernels at 2, 4, 6 and 8 bits. Each layer picks its path once, at weight load. A layer outside the kernels' contract runs the Triton kernels, and the serve log names it with the reason.
RFI covers 2, 6 and 8 bits (8 by default). RFA is 4-bit.
The compute core is fp16 for layers whose weights fit the fp16 range and bf16 for the rest, decided per layer at load.
RFI's activation path (W8A8 or W8A16) comes from the checkpoint's config.json and can be overridden at serve time; see Quantizer → RFI-8.
MoE expert layers of RFI/RFA checkpoints run the Triton kernels (tuned per card, see Triton Tuner). Dense layers in the same model run on HIP.
Quantization method names registered: rfi, rfa, and rfp458 (an alias of rfa).
Controls
RFI_HIP_KERNELS=0: run the RFI/RFA linear layers on the Triton kernels instead. Read at boot.
RFI_ACT_DTYPE=int8|fp16: override the checkpoint's RFI activation path. Read at boot.
FP8 GEMM: block-scaled w8a8
fp8-block checkpoints (128×128 block-scaled FP8 linear layers) run a native WMMA HIP GEMM on RDNA4 (gfx12x). The kernel picks its tile geometry per call from the batch size. At model load the FP8 weights are reordered once into the order the matrix cores read. Output is bf16.
The kernel order is HIP, then the RDNA4 Triton kernel, then the generic Triton kernel. Checkpoints outside the HIP kernel's contract (block scales other than 128, fp16 output) use Triton automatically.
Controls
VLLM_DISABLE_FP8HIP=1: skip the HIP kernel; use the RDNA4 Triton kernel.
VLLM_DISABLE_RDNA4_FP8_KERNEL=1: skip the RDNA4 Triton kernel as well; use the generic Triton kernel.
How a call moves data
Weights are reordered once at load and load straight to registers. Activations stage through LDS. The scales take a direct gather or the LDS pipeline, depending on batch size.
Based on work from AITER, Radiance VLLM, and existing Triton kernels.
TP all-reduce and all-gather
Tensor-parallel all-reduce runs on libr4d's P2P all-reduce by default at tensor parallel 2, 3, 4 and 8. Each rank pushes its data over the PCIe BAR straight into peer VRAM; the host and RCCL are not involved. Messages outside its size bands go to RCCL.
Requires P2P BAR access between the GPUs: large BAR, and one PID namespace or the ptrace capability for the dmabuf handshake. CLAV_P2P=0 puts the whole tensor-parallel path on RCCL.
How messages are sent
TP2: one-shot push up to 48 MiB. Messages above 128 KiB travel on a 6-bit wire.
TP3, TP4, TP8: small messages go one-shot on the exact wire (up to 160 KiB at TP3, 60 KiB at TP4 and TP8). Larger messages go two-shot (reduce-scatter, then all-gather) on an int8 wire.
Above the library's ceiling for each size, messages go to RCCL.
All-gather
At TP4 and TP8, all-gather runs on libr4d's one-shot push on the same buffers as the all-reduce. Messages larger than one slot, and all TP2 all-gathers, go to RCCL. CLAV_AG=1 adds a second P2P all-gather after libr4d, for all TP sizes.
Controls
R4D_AR=0: run all-reduce on RCCL. Qwen3.8-Flash-Next sets R4D_AR=1 itself; the launch environment does not change it for that model.
R4D_AR_QUANT=0: send every message on the exact bf16 wire.
CLAV_P2P=0: turn off every P2P backend (all-reduce and all-gather); the whole tensor-parallel path runs on RCCL. Use this on hosts without GPU peer-to-peer.
CLAV_AG=1: turn on the additional P2P all-gather (off by default). CLAV_AG_Q8=1 sends it on an 8-bit wire.
Checking P2P is available (Linux)
Large BAR: lspci -vv on each GPU should show a memory region the size of the card's VRAM (e.g. [size=32G]), not 256M. If it shows 256M, enable Above 4G Decoding and Resizable BAR in the BIOS.
IPC mode: leave HSA_ENABLE_IPC_MODE_LEGACY=0 (the image default). Legacy mode (=1) routes IPC through ioctls that only exist in AMD's out-of-tree DKMS driver; on the in-tree amdgpu driver every TP>1 IPC path fails at handle exchange with it set.
PID namespace: the dmabuf handshake uses pidfd_getfd, which needs all TP ranks in one PID namespace. Normal single-container TP is fine; if ranks are split across containers, add --cap-add=SYS_PTRACE.
Confirm it engaged: the serve log prints the dispatch chain at boot, in a line that starts Using [...] all-reduce backends, and r4d all-reduce INSTALLED for each rank. If R4D_AR is absent from the list, the log states the reason.
If the probe still fails: check dmesg for amdgpu/IOMMU errors around boot; on consumer platforms setting the kernel parameter iommu=pt is the usual fix for blocked or slow PCIe peer-to-peer DMA.
Based on the all-reduce from Radiance VLLM by deadcode.
Per-op kernels
Small per-op kernels on the serving path run as AOT HIP kernels. All are on by default on RDNA4 (gfx12x). A shape outside a kernel's contract, or a failed import, falls back to the stock path automatically.
All models
KV-cache write (reshape_and_cache_flash): the cache insert on every full-attention layer, bf16 and fp8 KV. CLAV_RESHAPE_CACHE=0 disables.
GDN causal conv: the conv1d prefill and decode-update pair on GDN models, for the shapes the fused GDN kernels do not cover. CLAV_CONV1D=0 disables.
Batch memcpy: the mamba state batch copy. CLAV_MEMCPY=0 disables.
Fused state copy: the align/spec-decode state copies that MTP hybrid models run every step. CLAV_STATE_COPY=0 disables.
Qwen3.8-Flash-Next and MXFP4 MoE
PLE short conv: the dilated causal conv of the Qwen3.8-Flash-Next PLE layer, one launch per step. Speculative tokens use the torch path. CLAV_PLECONV=0 disables.
HyperConnection mix: the Qwen3.8-Flash-Next HyperConnection chain as one launch, for decode batches of up to CLAV_HC_MAX_M rows (default 8). Larger batches use the standard chain. CLAV_HC=0 disables.
SiLU + fp8 quant: SiluAndMul fused with per-token fp8 quantization between the two grouped GEMMs of the r4d MXFP4 MoE experts. CLAV_SILU_QUANT=0 disables.
libr4d kernels
libr4d is a gfx1201 kernel library shipped in the image. It runs MoE experts, MoE routing, the expert cache, DeepSeek-V4 and DFlash drafter kernels, the tensor-parallel all-reduce, and the Qwen3.8-Flash-Next sparse attention. Every part is on by default and falls back to the stock path when a shape is outside its contract.
MoE and dense GEMMs
MXFP4 dense and MoE GEMMs: MXFP4 weights with fp8 activations, used by MXFP4 checkpoints such as Qwen3.8-Flash-Next-MXFP4-FP8 and the DeepSeek-V4 experts. VLLM_DISABLE_R4D_MXFP4=1 disables.
W4A16 MoE experts: symmetric int4 group-128 GPTQ / AutoRound experts (for example GLM-5.3-Flash W4A16), with the Triton WNA16 kernel as fallback. VLLM_DISABLE_R4D_W4A16=1 disables.
MoE routing: top-k, shared-expert slot and alignment in one launch, for models with 128, 256 or 512 experts and softmax scoring. R4D_ROUTE=0 disables.
Fused shared experts: when the shared expert matches the routed experts in size and quantization, it runs inside the routed grouped GEMM as one extra expert. VLLM_DISABLE_FSE=1 keeps it as a separate MLP.
Expert cache: under --enable-expert-offload, cache management and gather run in one launch per MoE layer. R4D_LRU_FUSED=0 disables.
DeepSeek-V4
Dense fp8 projections: block-fp8 projections of fp4-expert checkpoints run on libr4d. R4D_DSFP8=0 dequantizes them to bf16 at load instead.
Expert activation quant: R4D_ACT_QUANT=0 uses the torch path.
mHC pre/post: R4D_MHC=0 uses the torch path.
Sparse-attention indexer: the prefill and decode scorers for DeepSeek-V4 and GLM-5.3-Flash.
DFlash drafters
Grouped conv of DFlash2 drafters in one launch. R4D_DFLASH_CONV=0 uses the torch path.
Draft-step prep (context K/V, norm, rope and cache writes) in one launch. R4D_DFLASH2_PREP=0 uses the standard chain.
Four weight quantizers (RFI, RFA, MXFP4-16 and the composite router) run from the tcclaviger/vllm:dev image and share one architecture registry: qwen3, qwen3_moe, qwen3_5, qwen3_5_moe (incl. Qwen3.6), qwen4_exp (Qwen3.8-Flash-Next), step3p7, gemma4, muse_glimmer, minimax_m2, minimax_m3, minimax_m3_eagle3, laguna and ling3. An unrecognized architecture is refused.
Device flags are elided in the examples below; every GPU command needs --device /dev/kfd --device /dev/dri --group-add video.
Compute modes (shared)
none = CPU
--gpu = 1 layer per GPU
--gpu-all = all layers round-robin in parallel (needs VRAM for all weights)
--gpu-stream + --load-chunks N (default 4) = prefetch chunks from disk, bounds host RAM to ~2 chunks
Inspection (shared)
--dry-run, --layers a,b,c, --ignore RE, --quant-only
--eval / --no-eval: writes eval_metrics.txt (integrity, coverage, bpw, fidelity); on by default for RFA and MXFP4-16, off for RFI
--savegroupmse
RFI-8: int8 W8A8, 8.25 bpw
Per-token int8 activation × int8 weight on the gfx12 int8-WMMA path. ~44.9 dB fidelity.
2/6/8-bit all share one int8 WMMA path; --bits 8 is the default, so a bare --rfi run is RFI-8.
The stored int8 is the WMMA operand directly, with no dequant step.
Force specific modules back to 8-bit in a mixed run with --bits8 PATTERN.
Activation path, A8 vs A16
RFI checkpoints carry an act_dtype field in config.json selecting how the activation is fed to the GEMM. The quantizer writes int8 by default.
act_dtype: "int8" (W8A8): the activation is quantized per token so the GEMM runs on the int8-WMMA path. The speed option.
act_dtype: "fp16" (W8A16): the weights are dequantized and multiplied against the unquantized activation, with no per-token activation quantization.
At serve time the RFI_ACT_DTYPE environment variable (int8 | fp16) overrides the checkpoint's value without re-quantizing.
Mixed precision via --bits6 PATTERN: substring or re:-regex; granularity is the fused runtime module (gate/up and q/k/v must resolve to the same scheme+bits).
Typically used to push MoE experts down from 8-bit while keeping the dense path at RFI-8.
IQ4-NL 16-point grid, no forced-zero weights, with a block-float group scale: signed int8 mantissa per group of 16 (sign carries polarity) + shared int8 exponent per output channel.
Per-group 16×16 Hadamard rotation on by default, auto-disabled when K % 16 != 0 (--no-rotate to force it off).
--codebook nl|lloyd|<json> picks the 16-point grid: nl (IQ4-NL, default), lloyd, or a JSON list of 16 numbers.
--fp8-attn: block-FP8 (e4m3, 128×128) the attention q/k/v/o_proj instead of copying them; requires the arch spec to declare fp8 attention.
--fp16-experts SPEC: protect MoE experts in source dtype instead of quantizing: inline layer:expert pairs (12:5,30:88) or a JSON file.
--overrideLayers PATTERN: force normally-copied modules into the quant grid (name fragment or re: regex; only 2-D .weight tensors are quantized, others copied with a warning).
One checkpoint mixing RFA / RFI8 / RFI6 / RFI2 / BF16 per layer or tensor pattern, first match wins. The host declares quant_method: rfi carrying both groups; the runtime decodes each module with its own scheme. The map is where the size/quality decision lives: the router mechanics are simple, and the value is in how the bits are assigned to importance.
Planning a MoE map: bit density follows expert usage
A very large MoE can be shrunk far past any uniform width, because expert usage is not uniform. The workflow is to get the activation map first, then write the map against it:
Get the map. Run the expert mapper (Testing Tools) against the model as-shipped: it ranks every routed expert by actual contribution and prints the weight-rank versus activation-rank divergence, which is the number that decides whether per-expert protection can pay at all. When the source model is bigger than you want to sit with, a fast MXFP4-16 pass first is a legitimate cheap proxy: quantize, map, and plan against the result. The usage distribution is what you need, and MXFP4-16 is the quickest format to get one.
Read the distribution. Most MoE traffic concentrates on a minority of experts; the tail is near-dead. The map assigns bit density to that ranking: very low-use experts drop to RFI-2, the bulk of the routed experts take RFA, the top of the usage curve (roughly the top fifth) keeps RFI-8, and the super-experts and outliers, the few that carry a disproportionate share of traffic or range, stay 16-bit.
Protect the structural layers too. The shipped recipes all keep the same components at 8-bit regardless of the expert mix: the attention projections, the GDN/linear-attention projections, the MTP draft, the shared expert, and the first and last few layers. The expert map decides the expert ladder; the architecture decides the rest.
The result is a single --mixedQuant map: a handful of regexes, each naming a component class and its width. Patterns are matched in the order written, so the specific (a layer range, a named expert set) goes before the general (the whole expert bulk).
Planning a dense map: contribution decides the layer width
Dense models have no expert axis, so the map is per-layer. Activation capture ranks the layers by total activation contribution through the residual stream. That ranking sets the per-layer widths: RFA on the lower-contributing layers, RFI-8 or RFI-6 on the higher-contributing layers, and BF16 on the highest. A dense composite map is typically three or four bands: a protected head and tail, a compressed middle, and the attention and projection set.
When the composite run uses --calib, the per-layer report doubles as a second opinion on the map. The block-float scale search records, per layer, how far the shared exponent had to shift and how many group mantissas came back starved, a mantissa that cannot use its bits because the layer's range pushed the exponent up. The two signals paired together mark the layers where the format is working against the data: those are the ones that want protection at a higher bit weight in the next revision of the map, independent of what the activation ranking said. A layer that is quiet on both is the safest thing to drop a width.
Shaped examples from the shipped recipes
The image ships recipe maps for the deployed models, and they show the pattern at full resolution. A 35B MoE composite puts the routed expert bulk and the shared expert in RFA and everything structural (self-attention, linear attention, the MTP draft) in RFI-8. Its 2-bit variant pushes the routed expert bulk down to RFI-2 and the shared expert to RFI-6 for a much larger shrink on the same structural protection. A 2.1B-class model with interleaved linear attention protects the first three layers' experts and projections at 8-bit, takes a middle band of layers to 6-bit, and compresses the remaining expert bulk to RFA. A vision MoE keeps the vision tower and its projector at 8-bit, the early MLPs and the share expert at 6-bit, and the MoE bulk at RFA. In every case the map is short enough to read in a minute, and the order encodes the intent: specific before general.
Mixed-bit + forcing a normally-unquantized component
Scheme-mixing is --mixedQuant above. Bit-mixing and pulling a normally-dense component (lm_head, embeddings, a top-level projection, things the router keeps copied) into the quant grid is the single-format path: --bits6 / --bits2 set the mix, and --overrideLayers names the tensor(s) to force in by component name fragment or re: regex. Only 2-D .weight tensors are quantized; anything else matching is left copied with a warning.
# mixed 8/6-bit RFI (6-bit on the MoE experts), and force the normally-copied
# lm_head into the quant grid by naming the component:
--quantize --rfi --gpu-all \
--bits6 're:\.mlp\.experts\.' \
--overrideLayers 'lm_head' \
--input /m/MyModel --output /m/MyModel-RFI-mixed
# the same override, naming the exact tensor with a regex:
--overrideLayers 're:^model\.lm_head\.weight$'
MXFP4-16: 5.0 bpw
IQ4-NL grid with an fp16 per-group-of-16 scale.
Data-free only. Its --act-* flags are disabled: passing them prints a notice and runs the data-free path. For activation-aware 4-bit, use RFA with --calib.
Activation-aware quantization for RFA, RFI and the composite router (not MXFP4-16). With --calib the tool runs the model over a calibration corpus while it quantizes: activation statistics weight the scale search, and each column's rounding error is passed to the columns not yet quantized. The output checkpoint has the same format as a data-free run.
How a run proceeds
One layer at a time: load the layer, capture its inputs, quantize, then run the data through the quantized layer so the next layer calibrates on what inference feeds it. Only the active layer is resident, so models larger than VRAM work.
Within a layer, gate/up are quantized first, then down_proj is captured on the output of the quantized gate/up. --no-sequential-capture uses one capture per layer.
Resume is on by default: re-running the same command continues from the last finished layer. If the corpus, flags, codebook, feedback width or source checkpoint changed, it refuses to resume. --force-resume overrides that; --no-resume starts clean.
A per-layer report is written to aa_report.json.
Calibration corpus
Bare --calib uses the baked-in aa_calib_2m.jsonl.gz: 3438 chat rows, about 1.9M tokens, over 26 topic domains balanced by token count. --calib PATH takes your own JSONL; image rows are skipped.
Flags
--feedback 16|256|full|auto: how far a rounding error may travel (one scale group, a 256-column block, or all of K). auto (default) sizes it from free VRAM once per run. A forced width that does not fit is an error.
--aa-gpus all|N|0,2,3 (default all): shard the quantize step across GPUs. The output is identical at any GPU count for a fixed feedback width and micro-batch. Mixed gfx targets need --allow-mixed-gpus.
--carried auto|device|host: where the hidden state carried between layers lives. host keeps it in system RAM and streams it in slices. The output is the same either way.
On by default, each with a --no- form: --weighted-scale, --gptq-group.
Other: --gptq-gain (quantizes twice to report what the error feedback changed), --no-row-shard, --stats-out DIR, --no-stats, --no-progress.
# minimum: format, --calib, --gpu, --input. Everything else defaults.
docker run --rm ... -v /models:/m tcclaviger/vllm:dev \
--format rfa --calib --gpu --input /m/MyModel
# pinned feedback width, larger corpus budget, all GPUs
--format rfa --calib --gpu --aa-gpus all --feedback 256 \
--seq-len 8192 --nsamples 240 --micro-batch 4 \
--input /m/MyModel --output /m/MyModel-RFA-AA
# report what the error feedback changed (quantizes twice)
--format rfi --calib --gpu --gptq-gain --layers 0 --input /m/MyModel
Perplexity: --test --ppl
Runs WikiText-2 perplexity against the model the container is serving, with the llama.cpp-compatible chunked methodology, the standard measure of what a quant cost. The usual use is a pair of runs: the bf16 (or reference quant) checkpoint and the new one, and the delta is the number you defend. The test runs inside the serving container via docker exec and takes no configuration at all; everything is derived from the running server.
Note: vLLM launches do not reproduce identical numbers. PPL is deterministic within a given launch (rerunning against the same running server gives the same result), but minor container startup variation can show small drift between launches. Compare quants within one launch each, and read cross-launch deltas with that drift in mind.
# zero configuration: run it in the serving container, everything is derived
docker exec <serving-container> /app/tools/image_entrypoint.sh --test --ppl
There are no options. The endpoint comes from the server's own startup command (/proc/1/cmdline--port becomes http://localhost:<port>, always the current boot). The model id comes from the server's /v1/models answer and doubles as the run label. The tokenizer is read from /app/models, the checkpoint the server loaded, so it can never mismatch (a mismatched tokenizer would produce a wrong-but-plausible PPL, which is why it is not a parameter). Chunk size is fixed at 2048 for comparability with published llama.cpp-style numbers, and the result JSON lands at /app/models/ppl_result.json, next to the weights.
VRAM preflight
Before any request is sent, the preflight sizes the prompt-logprobs spike the test provokes: echo=true scoring makes the server materialize a min(2048, max-num-batched-tokens) × vocab × 8 B logits block per GPU, an allocation outside vLLM's profiled budget. It reads the batched-token limit and --gpu-memory-utilization from the startup command, the vocab from config.json, and live free VRAM from rocm-smi (idle GPUs are ignored). If the spike does not fit in 98% of the worst serving GPU's free memory, the test refuses to start (exit 3), prints the exact --gpu-memory-utilization that would make it fit, and measures nothing, instead of OOMing the server mid-run. Per-chunk progress streams unbuffered to the exec terminal.
Expert mapper: --map-experts
Which MoE experts deserve protection, and at what width. It is the first step of the composite MoE workflow (Quantizer → Composite router): the map is the usage distribution the bit-density ladder is written against, and the run's go/no-go decides whether per-expert differentiation can pay for itself at all.
The model must not be serving while this runs. The mapper boots its own vLLM instance and loads the checkpoint onto the same GPUs.
# default: the corpus baked into the image, map written next to the model
docker run --rm --device /dev/kfd --device /dev/dri --ipc host --shm-size 16g \
-v /your/modeldir/modelname:/app/models \
tcclaviger/vllm:dev --map-experts /app/models
# custom prebuilt corpus instead of the baked one
docker run --rm --device /dev/kfd --device /dev/dri --ipc host --shm-size 16g \
-v /your/modeldir/modelname:/app/models -v /your/datadir:/data \
tcclaviger/vllm:dev --map-experts /app/models --corpus /data/calib.jsonl
# or build the corpus from raw sources at run time
docker run --rm --device /dev/kfd --device /dev/dri --ipc host --shm-size 16g \
-v /your/modeldir/modelname:/app/models -v /your/datadir:/data \
tcclaviger/vllm:dev --map-experts /app/models \
--source math /data/MetaMathQA-395K.json 77 \
--source code /data/magicoder-evol-instruct.jsonl 102
Flags
MODEL (positional)
Required, and the only required argument. The model directory; the map is written next to it.
--corpus FILE
A prebuilt JSONL instead of the baked corpus, one calibration sequence per line, run in file order. Rows carry domain (free-form tag; the per-domain activation breakdown groups by it), dataset (provenance label), messages (chat turns, applied through the served model's own chat template), optional seq_len (per-row token cap), and optional images (a row with images is a vision row). With neither --corpus nor --source, the corpus baked into the image is used.
--source DOMAIN PATH [N] [SHAPE]
Repeatable; builds the corpus at run time instead of the baked one. N defaults to 77. SHAPE is auto-detected (chat / metamath / magicoder / glaive / sharegpt / mmlu / gsm8k / arc / hellaswag / winogrande / gpqa / ifeval / text / raw_text / images) or forced. Messages, not token ids: token ids are tokenizer-specific, and the routing being measured has to be the routing the served model actually does.
-tp N
Tensor parallel size, default 4.
--out-dir DIR
Where the map and sidecar are written; default is the model directory.
--seq-len N
Per-row token cap, default 1024; a corpus row's own seq_len overrides it.
--seed N
Shuffle seed when building from --source, default 0.
--image-root DIR
Where the corpus's images live as visible to the process: needed when a host-built corpus is read inside the container, where the dataset tree is mounted elsewhere. Falls back to the paths in the corpus, then to the corpus file's own directory.
--vision-token-budget N
Token floor for max_num_batched_tokens when the corpus has vision rows, default 16384: an image's placeholder run is only known after the multimodal processor expands it.
--gpu-memory-utilization F
Default 0.85.
--max-model-len N
Default: the model's own limit.
--quantization NAME
Override the quantization scheme vLLM detects from the checkpoint; rarely needed.
--hist-bins N
Histogram resolution in the map's per-expert distributions, default 48.
--seq-detail-cap N
How many sequences get a per-sequence row in the streamed sidecar; default 0, every sequence. Per-domain totals always cover every sequence.
--checkpoint-every N
Rewrite expert_map.json after every N prompts, so a crash keeps the totals so far. Default 1; 0 writes only at the end.
--chunked-prefill TOKENS
Enable chunked prefill with this batched-token budget (for example 2048). Default 0, off: each prompt runs as one forward.
--engine-arg KEY=VALUE
Repeatable; passes an extra vLLM engine argument, for models that only fit with offload, for example --engine-arg enable_expert_offload=true --engine-arg expert_offload_mem=40. The capture settings (eager mode, prefix caching, chunked prefill, max_num_seqs) cannot be overridden.
--limit N
Smoke test: cap total prompts. Verify the load and the write path with --limit 8 before committing to the full corpus.
--dry-run
Build the corpus, then stop before loading the model.
Escape hatches. --no-tp-reduce skips the in-hook TP all-reduce and is debug only: under TP>1 the resulting map is wrong, which is what it says. --single-process collapses driver, engine and worker into one process for debugging, TP=1 only.
How it measures
Ranks every routed expert by actual contribution. The model loads through vLLM, so any quantization this image serves is mapped as-shipped, no dequant, no bf16 source, no transformers/REAP dependency. A fast MXFP4-16 pass over a too-big source model is a legitimate input: the usage distribution is what the map produces, and that format is the quickest to get one.
Writes expert_map.json, expert_map_sequences.json, and the streamed per-sequence sidecar expert_map_sequences.jsonl.gz. Read the schema list rather than assuming column positions.
The divergence between weight-derived rank and act_rank is the headline result: experts the weights say matter but the traffic never routes to (and the reverse) are exactly the misallocations a uniform width makes. The run prints a go/no-go: if routed traffic is near-uniform, per-expert protection cannot pay for itself at any threshold and the model wants a uniform scheme instead.
Engine config is forced (enforce_eager, prefix caching off, max_num_seqs=1): this is a measurement pass, not a serve.
What the map contains
The main file carries, per (layer, expert): the weight-derived importance (the expert's norm-based share of its layer's capacity), the activation-derived importance (the routed traffic it actually received over the corpus, with per-domain breakdowns keyed on the corpus's free-form domain tag), and the ranks of each. The sidecar file carries the sequence-level detail: per-domain totals always cover every sequence, and each sequence (or the first --seq-detail-cap sequences) gets a per-sequence row, showing which prompts drove an expert's number. Under TP>1 the activation columns are all-reduced in-hook so the map is the same at any TP.
KV cache scale generator: --kvcalibration
Measures the q/k/v range of a checkpoint and writes the fp8 KV-cache scales. With no calibrated k_scale / v_scale in the checkpoint, the runtime uses 1.0.
The model must not be serving while this runs. Calibration boots its own vLLM instance and loads the checkpoint onto the same GPUs; stop the model's container first, calibrate, then bring it back up with --kv-cache-dtype fp8.
# normal pass: measures q/k/v over the baked corpus and writes the scale
# shard into the model dir, live for the next serve
docker run --rm --device /dev/kfd --device /dev/dri --ipc host --shm-size 16g \
-v /your/modeldir/modelname:/app/models \
tcclaviger/vllm:dev --kvcalibration --modeldir /app/models
# MTP pass: merges the draft's mtp.* scales into the existing shard
# (normal pass must have run first)
docker run --rm --device /dev/kfd --device /dev/dri --ipc host --shm-size 16g \
-v /your/modeldir/modelname:/app/models \
tcclaviger/vllm:dev --kvcalibration --modeldir /app/models --mtp
Flags
--modeldir DIR
Required, and the only required flag. The checkpoint to calibrate; also the default output location, so the scales are live the moment the run finishes.
--outputdir DIR
Write the scale shard, index entries and report somewhere other than the model dir.
-tp N
Tensor parallel size, default 4.
--pad F
Headroom multiplier over the observed max, default 1.1: cover for outliers the corpus never saw.
--textOnly
Calibrate on the text corpus only, even when config.json declares a vision tower. Without it, 120 vision samples are added automatically on vision-capable checkpoints.
--dryRunLimit N
Smoke test: only the first N prompts. Verifies the load, the probe naming and the write path before committing to the full corpus.
--mtp
The draft pass. The target's scales must exist first (normal pass). Runs the calibration with speculative decoding so the draft actually proposes, decodes a few tokens per prompt, and merges only the draft's mtp.* scales into the existing shard. Its report lands separately in kv_calibration_report_mtp.json.
--mtp-method NAME
Speculative-config method, default mtp; any name in the MTP model types normalizes to mtp.
--mtp-num-tokens N
num_speculative_tokens, default 4; match the serve compose.
--mtp-decode-tokens N
Tokens generated per prompt in --mtp mode, default 32. The speculator only runs on decode steps, so 1 would capture nothing from the draft.
--gpu-memory-utilization F
Default 0.75, not greedier: the pass runs one sequence at one token of generation, so it needs weights plus a token of KV.
--max-model-len N
Default 16384. Rows longer than this are truncated, and the log reports how many; raise it to keep long rows, which reach the high RoPE positions.
--max-num-batched-tokens N
Default 4096, auto-raised to --max-model-len when lower: chunked prefill is off by design (a split sequence breaks one-amax-per-sequence), so every prompt must fit in one batch.
--mm-processor-cache-gb F
Multimodal processor cache budget, default 0.5.
How it measures
--kvcalibration walks the baked corpus through the model one sequence at a time and records the real q/k/v range before the cache write, including the q_scale most checkpoints omit.
Writes a scalars-only safetensors plus its index entries, existing shards are never touched, so a failed run cannot corrupt the checkpoint, and a per-layer report whose spread ratio says whether the corpus explored the range at all.
The recorded scale is (observed_max × 1.1) / range_constant, where the constants are Q/K/V_SCALE_CONSTANT (200/200/100) read at run time so an override cannot desync the checkpoint from the runtime, and the 1.1 is headroom for outliers the corpus never saw (--pad to change).
Corpus, subset and vision sample count are fixed, not flags. The text corpus is the same aa_calib_2m set that --calib uses, trimmed to about 25k tokens per domain, longest rows first. Vision-capable checkpoints add 120 vision rows from the expert-map corpus.
Capture runs at --kv-cache-dtype auto, so it measures unquantized q/k/v; serve the result with --kv-cache-dtype fp8.
Refuses a checkpoint that already declares quantization_config.kv_cache_scheme.
--tune: per-card configs
The tuner writes config files for the Triton kernels. The HIP and libr4d kernels are compiled into the image and need no tuning. What runs on Triton is the RFI/RFA and FP8 MoE expert GEMMs and any layer that falls back from the HIP paths; that is what --tune is for.
Triton GEMM/MoE configs are not tuned per card. --tune detects which schemes a model uses per tensor and tunes each, writing per-card JSON the runtime loads at serve time. Mixed checkpoints get every scheme tuned.
16-bit weights, vision, lm_head, embeddings, scales and norms are skipped.
--tp is the tensor-parallel size you will deploy at, shapes are sharded by it before tuning so the configs match your serving TP. It is independent of --gpucount (the GPUs the sweep itself runs on; default all visible).
Point --outputpath at a folder beside the model, then bind-mount that folder over the runtime's config directory at serve time. The mount hides the tuned configs already shipped in the image; copy any you still need into your folder first. Filenames carry shape, card and scheme; anything unmatched falls back to defaults with a startup warning.
Manual shapes, --shapes
Tune specific GEMMs without trusting derivation. --shapes takes a JSON list-of-lists and must be scoped to one kernel bucket via --only <scheme> + --which linear|moe (not both). --modelpath is still required but its derived shapes are ignored.
When the bucket exposes one kernel, entries are bare shapes [N,K]: e.g. mxfp4_16 / rfa / fp8_w8a8 / rfi linear or moe.
When it exposes several, tag each shape with its kernel as ["<dtype>",N,K]: e.g. rfi linear is rfi_int8 / rfi6_int8 / rfi2_int8.
fp8_w8a8_moe uses a 3-arity shape [E,N,H] (--only fp8_w8a8_moe --which moe).
docker run --rm ... -v /models/MyModel-RFA:/model tcclaviger/vllm:dev --tune \
--modelpath /model --outputpath /model/configs --tp 2 --gpucount 2
# serve with them: bind-mount the tuned folder over the runtime's config dir
-v /models/MyModel/configs:<runtime configs dir>:ro
# manual shapes, single-kernel bucket, bare [N,K]:
--tune --only mxfp4_16 --which linear \
--modelpath /model --outputpath /model/configs --tp 4 \
--shapes '[[1536,4096],[6144,4096]]'
# manual shapes, multi-kernel bucket, tag each with its dtype:
--tune --only rfi --which linear \
--modelpath /model --outputpath /model/configs --tp 4 \
--shapes '[["rfi_int8",1536,384],["rfi6_int8",640,4096]]'
Startup tuners
Two tuners can run at startup. Triton attention autotune is on by default and only tunes attention groups that run on Triton. TunableOp dense-GEMM tuning runs when TunableOp is enabled.
Triton attention autotune
Tunes the Triton attention kernels for the groups that CLAV_ATTN does not serve. CLAV_ATTN_AUTOTUNE=0 uses the built-in per-bucket settings instead. CLAV_ATTN_AUTOTUNE_DUMP_DIR=DIR writes the results to a directory.
TunableOp dense-GEMM warmup
Tunes the dense bf16/fp16 projections at the CUDA-graph capture sizes, split across the TP ranks, between model construction and weight load. Results stay in memory.
PYTORCH_TUNABLEOP_ENABLED=1: turns TunableOp on and runs the warmup. The image sets CLAV_TUNABLEOP_SWEEP=1.
CLAV_TUNABLEOP_SWEEP=0: keep TunableOp on but skip the startup warmup.
PYTORCH_TUNABLEOP_TUNING=1: leave tuning enabled after the warmup. Unset or 0 freezes it.
PYTORCH_TUNABLEOP_FILENAME: PyTorch's own setting for a results CSV, for example one produced by an offline tuning run.
# startup GEMM tuning ON
-e PYTORCH_TUNABLEOP_ENABLED=1 ...
Repetition guards
Two server-wide guards. Degenerate-loop detection (on by default) ends a request whose output has fallen into an exact token loop. The DRY penalty (off by default) penalizes tokens that would repeat an earlier sequence.
Degenerate-loop detection
Ends a request whose output tail is an exactly periodic token loop, frees its KV and scheduler slot, and returns finish_reason repetition. Chat responses get a short "repetition detected" notice in the content.
A loop of period P triggers after max(128, 6×P) periodic tokens; periods up to 100 are checked.
Penalizes a token that would extend a token sequence already present in the context, by multiplier × base^(match length − allowed length). Tokens that do not continue a repeat are not affected.
--dry-multiplier: server default; 0 (the default) is off, a value such as 0.8 turns it on for every request.
--dry-penalty-ramp (0.75, base 1.75), --dry-allowed-length (2), --dry-range (−1, whole context), --dry-sequence-breakers (newline, colon, double quote, asterisk). These match llama.cpp's DRY defaults.
Per request: the SamplingParams dry_* fields, or vllm_xargs such as {"enable_dry": false} or {"dry_multiplier": 0}.
Works with speculative decoding.
Fork behaviors
Small vLLM changes in the image that apply automatically when a specific model, checkpoint or request needs them. The speculative-decoding confidence exit is the one opt-in item.
Speculative decoding
Confidence exit for MTP drafting."draft_confidence_threshold": X in --speculative-config stops drafting for a request once the running product of its draft confidences drops below X, so draft length varies from 1 to num_speculative_tokens. Unset (the default) drafts the full length.
MTP drafts use --hf-overrides. For method: mtp the drafter receives the same --hf-overrides as the target (for example YaRN rope scaling).
Quantized DFlash drafters. DFlash and DFlash2 drafters load from FP8, NVFP4, W4A16, GPTQ and AWQ checkpoints.
Quantized Step MTP drafters. Step MTP drafters load checkpoints with quantized MTP layers (RFI and composite).
Checkpoints and loading
Calibrated KV scales are optional. q/k/v scale tensors are used with an fp8 KV cache and ignored with a bf16/auto KV cache. A bf16 checkpoint served with --kv-cache-dtype fp8 loads its calibrated scales.
compressed-tensors MXFP4 MoE on ROCm. mxfp4-pack-quantized checkpoints (for example Ling-3.0-flash-MXFP4) run on the Triton OCP MX emulation experts: weight-only W4A16, packed fp4 in VRAM.
compressed-tensors block-FP8. Block-FP8 layers of compressed-tensors checkpoints run on the FP8 block GEMM chain, with either weight_scale or weight_scale_inv scale names.
GLM-5.3-Flash quantized MTP. AutoRound checkpoints load their quantized MTP block.
Models and parsers
MRoPE + YaRN correction range. The YaRN correction range uses the model's trained length. Serve with the model's real original_max_position_embeddings (262144 for Qwen3.6-27B).
Step reasoning parser. Honors enable_thinking=false per request.
Non-streaming think blocks. When the prompt closes the think block (thinking off), non-streaming output goes to content, as in streaming.
Expert offload
--enable-expert-offload serves a MoE model whose experts do not fit in VRAM. Experts that do not fit stay in pinned system RAM, and each GPU uses its remaining VRAM as an expert cache. The plan is decided once at startup and printed in the boot log. Supported for Qwen3.8-Flash-Next, DeepSeek-V4 and GLM-5.3-Flash, with MXFP4 or W4A16 experts on libr4d.
Flags
--enable-expert-offload: turn offload on. Any other --expert-* flag also turns it on.
--expert-offload-mem N: GiB of system RAM the offloaded experts may use, across all GPUs. Unset sizes it from the free RAM.
--expert-resident-layers "0,1,2": layers whose experts are kept fully in VRAM (default 0,1,2; "" for none).
--expert-precise-memory: size the cache without the default ~1 GiB per GPU of headroom for compile and graph capture.
--expert-direct-load, --experts-repack-ongpu, --expert-host-mlock: on by default; each has a --no- form.
--expert-packed-dir DIR: load experts from pre-packed files instead of the checkpoint.
--expert-prefetch N (0 to 2, default 0): decode-time prefetch lookahead. --expert-prefetch-k, --expert-prefetch-budget and --expert-prefetch-slots size it.
Routing profile
A model-expertprofile.safetensors file next to the weights holds per-expert routing counts. With it, the most-used experts are kept in VRAM and the cache starts warm. Without it, only the shared experts are kept in VRAM. The file is read only under --enable-expert-offload. Create it from an expert map (see Testing Tools → Expert mapper):
The PLE embedding table sits in pinned system RAM by default. Other placements:
--engram-config '{"cpu_offload": false}': keep the table in VRAM.
--ple-nvme-offload: serve the table from a file on NVMe behind a RAM row cache. --ple-cache-gb N sets the cache (default 4). --ple-nvme-dir DIR sets where the file goes (default beside the checkpoint). --ple-cache-reuse true keeps the file between runs. --ple-nvme-queue-depth (default 512) sets the read queue.
--nvme-engram-direct: read the table from NVMe with no row cache. --ple-cache-gb does not apply.
Tensor parallel 3 (experimental)
--tensor-parallel-size 3 is supported as an experimental feature. Head counts and layer widths that do not divide by 3 are padded at load with zero-weight heads and channels, so models that stock vLLM refuses at TP3 load and serve. DeepSeek-V4-Flash-Vision and GLM-5.3-Flash load at TP3. No flag is needed.
What gets padded
KV heads are raised to a multiple of 3; query heads follow, keeping the GQA ratio.
GDN / linear-attention key and value head pairs are raised the same way.
MLP, MoE and shared-expert intermediate sizes are raised to a multiple of 3 × 128.
The vocabulary padding absorbs the TP size.
A draft model is padded to its target's KV width.
Vision encoders whose heads do not divide by 3 run replicated on each GPU instead.
The padded heads and channels hold zeros and take VRAM and compute like real ones. The boot log prints one TP padding for tensor_parallel_size=3 line per padded setting. Other TP sizes are not padded.
Model support
Beyond stock vLLM on ROCm, the image serves these models on gfx1201 with its own kernels. Working compose files are in the Recipes tab.
Qwen3.8-Flash-Next
Hybrid GDN + QSA sparse attention with HyperConnections, a PLE embedding layer and an MTP drafter. The sparse attention runs on libr4d, then clav, then Triton. An fp8 KV cache requires one of the two HIP sparse-attention paths (both on by default). The model sets R4D_QSA=1, R4D_AR=1 and CLAV_TUNABLEOP_SWEEP=1 itself. MXFP4 checkpoints can offload experts to system RAM; see Serving → Expert offload.
DeepSeek-V4-Flash and DeepSeek-V4-Flash-Vision
Architectures DeepseekV4ForCausalLM and DeepseekV4ForConditionalGeneration. The mHC steps, the sparse-attention indexer, the block-fp8 projections and the fp4 experts run on libr4d; attention runs on Triton. The DSpark drafter's confidence head is included. Experts can be offloaded to system RAM with --enable-expert-offload.
GLM-5.3-Flash
W4A16 AutoRound checkpoints (int4 experts, everything else bf16). The experts and the sparse-attention indexer run on libr4d; sparse MLA runs on Triton. Serve with --kv-cache-dtype auto: there is no fp8 KV kernel for this model in the image. The quantized MTP block loads for mtp speculative decoding.
Qwen3.x 27B
Block-scaled FP8 Qwen3.8-27B on four cards, 524288 context, with a DFlash drafter at 7 speculative tokens. Replace <yourpath> with the model folder and <draftpath> with the DFlash drafter folder (Qwen3.8-27B-DFlash2-FP8). Serves on port 8080.
524288 context is YaRN at factor 2.0 over the model's real 262144 trained length. Drop both --hf-overrides and --max-model-len back to 262144 to serve the native length with no scaling.
Two swaps if DFlash is not what you want, one line each in place of the --speculative-config above:
vLLM's memory profiler books roughly 3.8 GiB of transient driver memory (compile autotune, combo-kernel benchmarking, the encoder profile pass, HSA_NO_SCRATCH_RECLAIM scratch) as permanent, and sizes the KV cache at 5.19 GiB on this config as a result. --kv-cache-memory overrides that number in bytes. Measured on 4× 32 GB cards on 2026-09-15, worst case being perplexity and concurrency running at once:
--kv-cache-memory
Size
Result
8804682956
8.2 GiB, 447K tokens
Safe. 29.8 GiB peak resident of 31.86, 2.1 GiB free.
9341542400
8.7 GiB, 475K tokens
Peak lands exactly on the 0.95 budget.
10350000000
9.64 GiB, 524288 tokens
Do not serve. ~31.2 GiB peak, 0.66 GiB margin, set only to hunt a user-visible OOM.
These are per-config numbers, not constants. Re-measure if the image, the tensor-parallel size, the capture sizes or --max-num-batched-tokens change.
Qwen3.8 Flash Next
Working configurations for Qwen3.8-Flash-Next-MXFP4-FP8-AA, one per machine shape. Replace <yourpath> with the folder holding the checkpoint. The group_add value is the host GID of the video (or render) group; 1101 is an example, run getent group video to find yours. All serve on port 8080.
The TP2 blocks are all the same 2× R9700 configuration, at the system-RAM sizes each was measured at. Each one is complete: pick the block whose footprint matches your machine and paste it whole. The name says which PLE (embedding table) mode it runs and what total system RAM it was observed using while serving.
Two PLE modes appear across them. LRU keeps a RAM row cache in front of the table; --expert-offload-mem and --ple-cache-gb are what move its footprint. Direct reads the table straight off the drive with no cache at all. The flags:
--ple-nvme-offload: serve the PLE table from NVMe behind a RAM row cache (LRU mode).
--nvme-engram-direct: serve the PLE table straight off NVMe with no row cache (direct mode). The LRU flags below do not apply.
--ple-nvme-dir <dir>: directory for the flat table file. Unset puts it beside the checkpoint.
--ple-cache-gb <n>: GiB of mlocked RAM for the row cache. 0 means the default of 4.
--ple-cache-reuse true: keep the table file across boots, so no 38.7 GiB rebuild at every start.
--expert-offload-mem <n>: GiB of system RAM the experts that do not fit in VRAM may use.
One card, for a 64 GB system
TP1 - PLE LRU 61GB
One R9700: PLE table on NVMe behind a 2 GB mlocked row cache (LRU mode, --ple-cache-reuse true), 50 GB expert budget, 131072 context, 2 sequences, 1024 prefill chunk, no speculative decoding. Observed at 61 GB total system RAM in use while serving.
The PLE table is read straight off NVMe with no RAM row cache in front of it. --nvme-engram-direct selects this mode, and --ple-nvme-dir points at the directory holding the flat table file. None of the LRU-mode flags apply: no --ple-cache-gb, no --ple-cache-reuse. A 34 GB expert budget and a 2048 prefill chunk, observed at 50 GB total system RAM in use while serving.
The PLE table is served from a flat file on NVMe behind a RAM row cache, measured at 54 GB total system RAM in use while serving. Three flags select it: --ple-nvme-offload turns the mode on, --ple-nvme-dir points at the directory for the table file, and --ple-cache-reuse true keeps that file across boots so there is no 38.7 GiB rebuild at every start. The directory needs to be a mounted volume on an NVMe drive with room for the table.
The block currently serving here: a 40 GB expert budget and a 4 GB PLE cache, running at about 60 GB total system RAM in use. More experts stay resident in RAM instead of being re-read from the drive, and the wider PLE cache absorbs more of the embedding-table traffic.
The largest of the three: a 60 GB expert budget and a 16 GB PLE cache, measured at 83 GB total system RAM in use while serving. For machines with RAM to spare.
The configuration currently loaded here: a 46 GB expert budget and an 8 GB mlocked PLE row cache, the smallest of the three-card blocks. Observed at 73 GB total system RAM in use while serving. Same 524288 context at YaRN factor 2.0, MTP=3 with a PIECEWISE capture at 4.
The PLE table is served from a flat file on NVMe behind a 16 GB mlocked row cache (--ple-nvme-offload, LRU mode), which is what pulls the footprint down from the host-RAM variant below. Experts offload with a 60 GB budget, and 524288 context comes from YaRN at factor 2.0. Observed at 82 GB total system RAM in use while serving. The plecache volume must be on NVMe with room for the 47.7 GB table file.
With three cards the experts do not all fit in VRAM, so --enable-expert-offload keeps a 50 GB budget in system RAM. No PLE flags appear: the 47.7 GB embedding table stays in pinned host RAM by default, which is what makes the footprint large. Observed at 110 GB total system RAM in use while serving. MTP=3 with a PIECEWISE capture at 4. Head counts and layer widths that do not divide by 3 are padded at load, so no extra flag is needed.
With four cards the experts fit in VRAM, so no expert offload and no NVMe PLE file: the embedding table stays in system RAM by default. That frees the budget for 524288 context via YaRN at factor 2.0 on the model's 262144 trained length, a wider prefill chunk (8192) and MTP=4.
The four-card model at 524288 context: YaRN at factor 2.0 on the model's 262144 trained length, MTP=4 with a capture ladder [5, 10, 20, 40, 80], and a 2048 prefill chunk at 0.975 GPU utilization. The tunableop mount carries the tuned dense-GEMM tables; drop it if you have none.
The capture-size ladder is num_speculative_tokens + 1 throughout: 4 for MTP=3, 5 for MTP=4. Serving with --kv-cache-dtype fp8 needs the checkpoint's calibrated q/k/v scales, see Adding the KV-scale sidecar to a checkpoint.
Muse Glimmer
Block-scaled FP8 Muse-Glimmer-30B on four cards at 1M context, with a DFlash drafter at 15 speculative tokens. Replace <yourpath> with the model folder and <draftpath> with the drafter folder; the DFlash drafter is a separate release artifact and does not ship inside the model folder. Serves on port 8080.
Glimmer's tool calling uses the ATEM protocol: <atem:function_calls> blocks with to=<tool> channel routing, and reasoning riding the to=self channel. The muse_glimmer tool-call and reasoning parsers (PR #51655, in the fork since 2026-08-13) handle both directions, and both flags are set above. Reasoning strength is not a server flag: put Reasoning strength: <low|medium|high|xhigh> in the system prompt. The chat template defaults to high.
Why there is no YaRN here
Extending context on this model is free, and YaRN would be actively wrong. The iRoPE layout is the reason: the 13 full-attention layers are NoPE, so no rotary embedding exists and there are no positions to extrapolate, and the other 39 layers are sliding-window 2048, so relative distance never exceeds 2047 at any total length. The only length-dependent artifact is the RoPE cos/sin table, sized from max_position_embeddings; raising it computes more entries at the same theta (500k) with no frequency scaling. The query pre-scale (3.87/sqrt(128)) is static, not position-dependent, and nothing else in the model reads sequence length. Applying YaRN would compress the within-window distances the sliding layers trained on natively, while the NoPE layers ignore rope entirely.
The KV cost of the extension is mild under the hybrid allocator: full layers run about 6.5 KB per token per rank and the sliding layers stay window-capped. The real risk is empirical, not structural: NoPE quality past the 131K training length is unvalidated, and Glimmer has no Llama4-style attention temperature tuning for long NoPE. Validate with codeneedle at deep positions before trusting it. For a more conservative 262144, lower both values:
Qwen3.6-35B-A3B in bf16 on four cards at its native 262144 context, with MTP at 5 speculative tokens. Replace <yourpath> with the model folder. Serves on port 8080.
The capture-size ladder steps by 6, which is num_speculative_tokens + 1 for MTP=5, so every concurrent-request count lands on a captured graph instead of paying padding.
524288 context with YaRN
Native is 262144. To double it, raise --max-model-len to 524288 and add the override below to the command block.
To serve without a drafter, drop the --speculative-config line.
GLM 5.3 Flash
GLM-5.3-Flash quantized W4A16 with Intel AutoRound (GPTQ-packed int4 experts, everything else bf16) on four cards at 262144 context. The routed experts run through the expert cache (libr4d gemm_moe_w4a16); the shared experts and the MTP block stay in VRAM. Replace <yourpath> with the model folder. Serves on port 8080.
The flat single-service layout is deliberate. Cogferra's model container manager reads this file with a plain serde deserializer and does not resolve YAML anchors or <<: merge keys, so a merged service comes back with no image, volumes, devices or environment, and the launch spec is unusable. Write every key out.
The tool-call and reasoning parsers are the GLM 4.x ones (glm47 and glm45), which 5.3 Flash speaks unchanged. --expert-resident-layers 3 pins the first three layers' experts in VRAM ahead of the offload cache.
Gemma4 31B
Block-scaled FP8 gemma-4-31B-it, a dense Gemma4ForConditionalGeneration with text and image input, on four cards at its native 262144 context. No rope override is needed. Replace <yourpath> with the model folder. Serves on port 8080.
Gemma4 splits across both backends, and that is expected. 50 of the 60 layers are sliding-window (1024) attention at a per-rank shape of 8q/4kv/d256; clav_attn serves these since the SWA clamp landed, running windowed calls on FA1/FA2 at segments=1. The other 10 layers are full attention at global_head_dim=512 with unified K/V (attention_k_eq_v); head size 512 with unified K/V is not instantiated for structural LDS reasons, so that group runs Triton either way. That is why the shared Triton cache stays worth mounting and why the autotune stays available for that group. Do not set VLLM_ATTN_AUTOTUNE=0 here: the d512 group belongs to Triton, so let it tune.
Model-card settings, applied above
Sampling per the card's Best Practices: temperature 1.0, top_p 0.95, top_k 64.
Context 262144 native, no rope override.
Thinking uses <|think|> in the system prompt with <|channel>thought tags. The fork's gemma4 parsers handle both directions: --reasoning-parser gemma4 splits the thought channel into reasoning_content, --tool-call-parser gemma4 parses native tool calls.
Thinking is on by default via --default-chat-template-kwargs enable_thinking=true. A per-request chat_template_kwargs {"enable_thinking": false} overrides it.
preserve_thinking=true (model card §3): history thinking is dropped except on tool-call turns, which the template only keeps when this kwarg is set.
Put images before text in prompts. The card's 70..1120 token budgets are prompt-side, not server flags.
EAGLE3 speculative decoding
There is a published speculator for this model: RedHatAI/gemma-4-31B-it-speculator.eagle3, in speculators format, a 1-layer llama draft with a 32K draft vocab. Put it in ./speculator-eagle3, mount it, and add the line below. The token count comes from the speculator's own proposal config.
Everything in this section was learned writing and measuring the HIP kernels in this image, attention (clav_attn), GDN, and the RFI/RFA quantized GEMMs, all on one part: the Radeon AI PRO R9700, gfx1201, RDNA4. Every conclusion below traces to a recorded measurement in the kernel trees (hipattn/tools/, gdnhip/tools/, rfihip/tools/); where a claim is arithmetic or design rather than measurement, the text says so. Nothing here is quoted from vendor documentation without having been checked against the assembler or the clock.
The hardware, as measured from the host
Resource
Value
How it was established
Compute units
64
rocminfo on the GPU agent. HIP's multiProcessorCount reports 32: that is the WGP count (2 CUs per WGP), not the CU count, and it is a real trap when sizing grids.
SIMDs
2 × SIMD32 / CU
wave32; 16 wave slots per SIMD → 32 waves/CU → 2048 waves device-wide, confirmed empirically by the wave-saturation band landing exactly on multiples of 2048 waves at three different workgroup sizes.
LDS
65536 B / workgroup
Hard cap; a 78272 B tile request fails to instantiate at compile time.
VGPRs
~1536 / SIMD
Consistent with driver-reported co-residency: a 191-VGPR kernel gets exactly 8 waves/SIMD (1536/191 = 8.04); at 193 VGPRs it drops to 7.
Caches
L1 32 KB · L2 8 MB · MALL 64 MB
rocminfo / architecture; the 64 MB MALL is measurable: it holds a 24 MB decode state fully and serves parts of a 64 MB working set at >1 TB/s effective.
HBM bandwidth
~640 GB/s stock, ~670-710 OC
Reference roofline used across all effective-bandwidth figures; best measured kernel rates reach 96-99% of it at saturating batch.
WMMA rate
114.7 TFLOP/s fp16 · 1.97× fp8
Measured, gdnhip/tools/wmma_rate_probe.hip. bf16 matrix rate matches fp16; bf16 has no scalar/vector ALU at all (see Data types).
Occupancy is the overlap mechanism. There is no workable explicit-overlap primitive on this part. Three were built and measured (register-carried prefetch, multi-stage software pipelining, s_prefetch_data) and all three regressed at every point. Cutting LDS so a second workgroup co-resides per CU is what raised a decode kernel from 444 to 610 GB/s. Details and numbers in Optimizing occupancy.
…until the kernel is issue-bound, and then occupancy is worthless. The quantized GEMM reached the Triton peer's exact co-residency (5 WG/CU, 10 waves/SIMD) and got 8-20% slower. Once a kernel is off the bandwidth wall, waves do not remove instructions. Check which resource actually binds before applying any rule.
bf16 is a storage format, not a compute format. The assembler rejects every bf16 ALU and convert opcode; only the matrix core and one dot product take bf16. The software round costs 6.4 VALU per element and was the single largest term in the quantized dequant at every bit width. Details in Data types and the ISA.
Bytes are only the limit when you are actually at the roofline. An effective-bandwidth figure well below the roofline is evidence against a traffic explanation, not for one: a cache-blocking fix applied to a 188 GB/s kernel measured null.
Measure with CUDA graphs, MALL-cold for decode, and let serving arbitrate. Two harness defects (host dispatch inside the timed window, unramped clocks) invalidated every early number, and both flattered the new kernel. Details in Measurement discipline.
What the lessons bought, end to end
27B long-context prefill +42% at 161k over the fully-tuned Triton path (1450 → 2006 TPS); gemma4-31B TTFT at 78k context 111 s → 45.5 s.
Concurrency ahead of stock Triton from 4 streams up, +3.15% per-request retention at 32 streams; decode at parity.
Quantized serving: full HIP stack over all-Triton: prefill 1.04-1.86×, decode 1.01-1.05×, 160K TTFT 127.3 → 78.7 s, at equal perplexity.
Cold start 2:39-3:45 versus ~10 minutes of Triton JIT + autotune.
Three different memory regimes showed up across the kernel work, and they demand opposite optimizations: streaming kernels at the HBM wall (attention decode at length, 8-bit GEMM at large M), MALL-resident kernels where dispatch count dominates (GDN decode), and unpack-bound kernels that look memory-bound but are not (sub-8-bit GEMM). Diagnosing which regime a kernel is in (with a measured effective-GB/s figure against the ~640 GB/s roofline) is the first step; every wrong fix in the record came from skipping it.
At the wall: reduce the bytes, then keep the controller fed
Don't re-read what a group shares. The original attention kernels gridded over query heads, so the gqa query heads sharing one KV head each re-read the same K/V tiles: 8× the KV traffic at GQA 8 (arithmetic, not measurement). Gridding over KV heads with the GQA group folded into the matmul's M dimension serves the whole group from one tile load. This was the single largest attention lever.
Clamp ranges, don't just mask. The per-element sliding-window mask was always correct, and a windowed layer at 512k context still walked every KV tile from key 0 and masked them all: ~1000× the tiles it kept (arithmetic). The tile-aligned KV-loop lower clamp turns the walk into the live window only.
Halve the element. The fp8 KV path keeps K/V in e4m3 from VRAM through the native fp8 WMMA: half the KV bytes with no dequant stage. At head_dim 512, decode measured ~84% of the bandwidth roofline, so the remaining acceleration there is a bytes problem (fp8 KV, weight quant), not a kernel problem.
Coalescing is worth more than elegance. The FWHT rotation's one-channel-per-lane mapping made the butterfly trivial but loaded 2 B per lane: 64 B per wave against the ~1 KB Triton was pulling. Eight consecutive channels per lane makes the load one 16-byte uint4 (512 B/wave) and turns the first three butterfly stages into register swaps. Measured waterfall at K=4096, M=1: 8.64 µs (LDS-staged) → 7.34 (row stays in registers) → 7.14 (loads batched ahead of the butterflies) → 4.26 µs (8 channels/lane), against Triton's 4.37. The GDN prep kernel learned the same lesson: its first thread-per-token cut cost 7.0 ms at T=65536 (more than the three following pipeline stages combined), and coalescing was the whole fix.
Sources: hipattn/README.md, rfihip/README.md (rotation section), gdnhip/csrc + gdnhip/tools A/B records
In the MALL: dispatch count dominates, not bytes
Decode state for the GDN layers is MALL-hot in serving (48 layers × 512 KB fp32 state = 24 MB against the 64 MB MALL), and in that regime cost is dispatch count × fixed kernel cost. The four-kernel HIP decode chain ran 12.7 µs per layer-step against Triton's fused pair at 5.07 µs; fusing conv + split + prep + recurrence into one dispatch brought it to 6.12 µs hot / 8.08 cold against the pair's 6.72 / 9.20 in the same harness: 2.4× over the legacy chain, from deleting launches and the HBM round-trips between them, not from any inner-loop work.
The mirror rule for benchmarking: a MALL-resident measurement overstates what serving sees whenever serving is not MALL-resident. The attention LDS-alias win measured 1.88× with a warm MALL and 1.37× MALL-cold (--hbm); decode attention is measured MALL-cold by rule, while GDN decode is measured MALL-hot because that is its serving regime. The rotation kernel's ~1200 GB/s effective rate is likewise MALL-assisted by design: the point of that kernel is to stay out of the GEMM's HBM budget, and its GB/s column is a comparison figure, not a roofline claim.
The most counter-intuitive measured result in the project: weight compression below 8 bits buys footprint and no decode speed. A 3-format composite streaming 66% of the bytes of the all-8-bit model decodes at exactly the same rate: bytes 66%, effective bandwidth 66%, GEMM time identical to within 1%. The dividing line is not the activation path; it is whether the stored byte is the dot operand. At 8 bits direct_b feeds the int8 WMMA straight from global memory; every other width loads packed bytes, unpacks, stages through LDS and reads fragments back: per-weight-element work that does not shrink when the bytes do.
Source: rfihip/tools/first_serve_throughput.md §9/§9b, all three formats profiled inside the same decode step
The penalty is per weight element, so it divides by M: devastating at decode, nearly gone by M≈32. That amortization signature is what names the mechanism:
Source: rfihip/tools/first_serve_throughput.md §9b, Qwen3.6-711 served all-RFI-6 vs all-RFI-8, same image
Cache-line reuse dies with co-residency: the 2.62× cliff
A shallow K-block (BK=32) consumes a 128-byte weight cache line over 8-16 loop iterations, and silently depends on the line surviving that long. Whether it does is set by resident workgroups × planes: at ~5 WG/CU the live-line footprint (~82 KB/CU for the two-plane 6-bit format) blows the per-CU caches, every visit refetches the line, and weight traffic amplifies 8-16×. Measured: the same config that is the argmin at 160 workgroups (0.0727 ms) runs 0.334 ms at 320 workgroups; split-K and N enter only as workgroup multipliers. The 6-bit format broke first (two planes, half the per-visit granule), 8-bit was immune until ~448 workgroups. The fix is a register-resident prefetch ring that issues each line's touches as one back-to-back burst, and the attempted generalization of that ring to deep K-blocks was itself measured and reverted (−3.5%): at BK=128 the line is consumed over two adjacent iterations, the cache handles that unaided, and the ring's register shifts are cost with no benefit. The ring pays for reuse distance, not granule.
When the traffic explanation is wrong: the swizzle null
The GEMM's raw dispatch order swept the entire 46 MB weight matrix once per M-block: four full HBM passes with no reuse at M=512, 168 GB/s effective. Triton's grouped swizzle fixes exactly this, so it was implemented. It measured null at every rung. The error is in one step of the reasoning: 168 GB/s is nowhere near the ~640 GB/s roofline, so there was never bandwidth to reclaim: the kernel was issue-bound, and cutting the bytes it wasn't waiting on changed nothing. The reusable rule: an effective-GB/s figure well below the roofline is evidence against a traffic explanation, not for one.
Optimizing occupancy
gfx1201 gives a kernel no good way to overlap memory latency within a workgroup. The memory controller overlaps separate workgroups' request streams fine, so on a bandwidth-bound kernel the only working lever is co-residency: more workgroups per CU, more waves per SIMD, until the controller stays fed. This was established the expensive way, by building all three explicit-overlap mechanisms and measuring each one as a regression.
Explicit overlap: three dead ends, one working fix
Sources: hipattn/README.md dead-ends table, hipattn/tools/bench_pipe_probe.hip results, memory rdna4-overlap-via-occupancy. gfx1201 also has no VMEM→LDS direct load (global_load_lds needs a target feature this part lacks; probed via codegen).
Loop-carried V prefetch regressed at every point and got worse with length (0.96× at L=512, 0.74-0.77× at L=32768): the tell that the carried registers cost more occupancy than the latency they hid.
Multi-stage software pipelining (register-staged global reads + LDS ring): every stage count regressed, up to +167% at d256, where the pressure probe had predicted 176-256 B/lane of spill paid every iteration. Wave interleaving at ≥4 waves/SIMD already provides the overlap the pipeline was built to add; the machinery contributes only a second barrier and register pressure.
s_prefetch_data costs no VGPRs, LDS or barriers, and still regressed 17-18%: on a bandwidth-bound kernel the prefetch stream competes with demand loads for the same controller.
What worked: aliasing the FA1 kernel's epilogue accumulator into the K/V tile storage (temporally disjoint) cut LDS from 44.5 KB to 28 KB per workgroup (one to two workgroups per CU) and lifted MALL-cold decode 1.37-1.67× at batch 12, from 444 to 610 GB/s effective (~86% of the overclocked roofline).
Where the fed controller lands
With the alias in place and segments chosen per shape, FA1 decode reaches 93-99% of the ~670 GB/s roofline at batch 8-12 across 32k-131k contexts. Batch 1 remains the hard case: a single sequence exposes the narrow grid, and the best split (s128) reaches 77%:
Source: hipattn/tools/abs_seg_SUMMARY.txt, achieved GB/s, trimmed mean of 4 reps, --hbm, hq6/hkv1/d256. Best configs at b=4-12: 586-763 GB/s (93-99% of roofline; the 114% entries at short context are MALL leakage through the scrub, not real HBM rate).
The wave-saturation band: a scheduling hazard at exactly full occupancy
A launch whose grid lands just below full wave occupancy pays a flat ~15 µs regardless of its work: 18.7 µs against 3.2 µs one workgroup away, on a cache-resident do-nothing body. The band is exactly the four grids [G_sat−3, G_sat] where G_sat is any integer multiple of 2048 waves ÷ waves-per-workgroup, it requires ≥4 co-resident workgroups per CU (at 32 KB LDS/WG it disappears), and it reproduced identically across two independent runs:
The dodge steps over, to G_sat+5: surplus workgroups run zero iterations of a grid-stride loop and retire free. Stepping down instead forces some workgroups to take a second row while the rest idle, measured 1.14× → 0.92× at M=1024, and G_sat+1, while clean, sits one workgroup from a reproducible isolated anomaly at G_sat+2 (22.22 µs in both runs). The mechanism is not established; the extent is. In-band magnitudes do not repeat run to run, so only the band's location and width are facts.
Sources: rfihip/tools/band_results.txt, band_results_run2.txt, abc_results.txt; probe kernels in rfihip/csrc/rfi_probe.hip. Re-verify against any new driver/ROCm before trusting safe_grid().
The counter-law: occupancy is not the lever on an issue-bound kernel
The quantized GEMM contradicts the standing rule, and the contradiction is the most reusable result in the record. Four separate attempts to buy speed with co-residency all failed:
Change
Co-residency
Result
s_out/s_a LDS union
3 → 4 WG/CU (6 → 8 waves/SIMD)
null
Same union at MT=2
3 → 4 WG/CU
−10%
BK=32 (Triton's exact figure)
3 → 5 WG/CU, 10 waves/SIMD
−8…−20% at M ≥ 64
Square tile at kWM=4
3 → 2 WG/CU
+13.6% at M=64 (tile shape, not occupancy)
The arithmetic that explains all four at once, the two M-tiling extremes at M=4096:
grid.y
weight re-streamed
effective rate
MT=1 (fine split, max parallelism)
256
256 × 46 MB = 11.8 GB in 17.09 ms
690 GB/s: pinned at the roofline
MT=8 (fat block)
32
1.47 GB in 7.82 ms
188 GB/s: issue-bound, 2.2× faster
Fine splitting buys parallelism and drives the kernel straight into the HBM wall doing it; fat blocks pull it off the wall, and once off the wall it is issue-bound, where more waves cannot remove an instruction from the stream. The law for this kernel: take the fattest M-block that still fills the machine, then fight on instruction count. Every subsequent win there came from the loop body (see Segments & tiles, and Data types).
Source: rfihip/README.md "occupancy is not the lever" (four measurements), rfihip/tools/trough_*.txt
Which resource binds decides whether prefetch is free
The same one-K-block prefetch idea died in attention and won in the GEMM, and the difference is bookkeeping, not luck. hipattn's three overlap dead ends were all register-bound: the prefetch registers came straight out of occupancy. The GEMM at kAChunk=4 is LDS-bound at 3 WG/CU, which permits far more VGPRs than the ~200 in use: the prefetch registers were already paid for and idle, and the split of decode16 into load + decode measured −2 to −10%, biggest at low M. Before importing either rule, check which resource actually binds the kernel (the runtime tells you: hipFuncGetAttributes + hipOccupancyMaxActiveBlocksPerMultiprocessor, wrapped in rfi_kernel_resources()).
Serialization beats byte-thrift: why the fused GDN kernel lost
The fused GDN prefill held the whole chunk-loop working set in ~62 KB of LDS (one workgroup per CU, 48 blocks on 64 CUs at the deployed shape) and serialized ~100% of prefill FLOPs behind the state recurrence. In serving it cost TTFT +7-17% and tokens/s −4-7% against the fla-Triton control; GDN was the wall the attention layers waited on. The split design pays an h-snapshot HBM round-trip (~130 MB/layer at 4k tokens) to buy chunk-parallel occupancy for the ~75% of FLOPs with no sequential dependency, and the serving A/B (not the idle-GPU microbench, which flatters whichever kernel gets the whole device) is the arbiter that trade wins. fla, the CuTe DSL GDN kernels and AMD's aiter all decompose the same way.
Segments, split-K and tile shape
Splitting one logical reduction across CTAs (split-KV in attention, split-K in the GEMM) is the only way to fill 64 CUs when the batch is small, and a pure cost once the batch fills the device by itself. Both kernels converged on the same structure (fixed-size partials scratch + a separate deterministic reduce) and the same conclusion: the split count is a computed function of the shape, never a tuned constant.
Split-KV: the optimum falls as batch rises
The decode grid is q-blocks × kv-heads × sequences × segments, so segments buy parallelism while the batch cannot fill the device and cost a reduce pass once it can. The full sweep (FA1, t16/w4, hq6/hkv1/d256, warm median ms; * = best per row):
Source: hipattn/README.md segments rescan (36-config grid); raw sweeps in hipattn/tools/abs_*
Three conclusions the data forces:
A fixed value is only right at one end. The serving backend used to hardcode segments=64; every concurrent decode above ~4 streams paid 7-14%: exactly the regime the concurrency benchmarks live in. Replacing the constant with a per-shape table keyed on query rows (so MTP lands in the right bucket) recovered it and produced the +3.15% concurrency lead.
s256 never wins at d256, and is the argmin at d128 batch-1 deep. On the d256 grid above, s256 lost at every measured point: once a segment covers too few KV tiles, the per-segment reduce overhead dominates. But that conclusion is head-dim specific and did not survive the wide-grid re-sweep: at d128, half the bytes per KV tile make thin segments affordable, and s256 becomes the argmin at b=1 long context (L=131072: 0.0795 ms vs s128's 0.0980 on FA1, 0.0975 vs 0.1265 on FA2-v2). A ceiling flat at 128 (validated on d256, applied everywhere) was the single worst cell of the whole 2026-08-13 sweep corpus at +23.3% (FA1) / +29.7% (v2), so the shipped selection law's ceiling is now per head dim: 256 below d256, 128 at d256, where raising it costs nothing (pooled regret 1.96% → 1.86%, no grid regresses). The meta-lesson outranks the constant: a bound measured at one head dim is a fact about that head dim, not about the kernel.
Splitting is safe to vary at runtime only because the scratch is sized at the maximum and smaller segment counts pack into a prefix of the same allocation; a reallocation would invalidate captured CUDA graphs.
Sources: hipattn/clav_attn/backend.py_seg_ceiling() (the per-head-dim ceiling and its measured justification), hipattn/tools/seg_law_worst_cells.txt, wide-grid sweeps in hipattn/tools/abs_wide_*
The reduce is not free, and it was 90% of the decode gap
Profiled at b=1 / L=131072 / s128 over 1068 calls each, the main attention kernel was already at parity with Triton; the reduce was 7.9× behind and carried 90% of the total gap. The rewrite deleted work rather than trading it: two block-uniform passes every thread ran over the whole partials array became one cooperative pass with a shuffle+LDS reduction, the exp2 weight computed once per segment instead of twice per segment per thread, and the accumulation split into four independent FMA chains so the strided loads pipeline:
Kernel (avg µs/call)
clav before
clav after rewrite
Triton
main attention kernel
206.6
206.1
203.2
reduce_segments
34.9
9.2
4.4
Sources: hipattn/tools/rocprof_decode.txt, hipattn/tools/rocprof_redfix.txt (rocprofv3, 1068/1315 calls per kernel)
Split-K in the GEMM: starved vs redundant, decided per shape
At M=1 the wave arithmetic is brutal: total waves = N/16 regardless of tile shape (the tile knob cancels), so N=4096 puts 256 waves on a 2048-slot machine (12.5% occupancy), and splitting K is the only thing that adds workgroups. The shipped law bounds the segment count by want (segments needed to reach 2048 waves) and traffic (segments before the partials stop being cheap: the reduce writes and re-reads nseg×M×N floats, which decays with M and retires split-K without an M threshold), then rounds down to a power of two: odd counts split K into unbalanced bands and leave a ragged grid. The trough diagnostic shows the law at 0.92-1.11× of the measured argmin per rung, with the residual at M ≥ 64 being loop body, not shape selection:
M
law picks
occupancy
law ms
Triton ms
ratio
1
mt1 / sk4
62.5%
0.0915
0.1207
1.32×
8
mt1 / sk4
62.5%
0.0852
0.1026
1.20×
32
mt2 / sk4
62.5%
0.1060
0.1019
0.96×
64
mt4 / sk4
62.5%
0.1558
0.1242
0.80×
128
mt8 / sk2
31.2%
0.2355
0.2088
0.89×
512
mt8 / sk1
62.5%
0.8323
0.6990
0.84×
Source: rfihip/tools/trough_dflt_b8.txt (N=5120 K=8704 bits=8, MALL-cold, graph-timed). The bf16-round change afterwards moved M=1 to 1.45× and M=8 to 1.27×; see Data types.
Tile shape: three measured rules
LDS decides what is instantiable before performance gets a vote. Attention tile 32 at head_dim 256 needs 78272 B of K/V staging against the 65536 B budget; tile 16 (59840 B) is the only option there. Dropping tile 16 because it "looked slower at head_dim 128" would have locked out a deployed model entirely; a lesson recorded after it nearly shipped.
Square wave tiles minimize fragment loads per matrix op. A wave loads (m+n) operand fragments to issue m·n MMAs, so the cost is (m+n)/(m·n): worst for slivers. The GEMM's original 8m×1n sliver ran 1.125 loads/wmma (measured 1.44 LDS fragment reads/wmma against the Triton peer's 0.79); splitting the workgroup 2×2 gives 0.75 at MT=8 with the same eight accumulator fragments, worth −5 to −10% at M ≥ 64. Copying the peer's warpsPerCTA=[4,1] blindly was wrong at MT=4: it reproduced the sliver's ratio from the other side and cost +13.6%.
K-block depth is M-dependent. Shallow blocks win where per-block overhead (barriers, staging bookkeeping) dominates and split-K needs many blocks; deep blocks win once the MMA work amortizes the overhead. Measured at bits=8, law-selected (ms):
M
BK=32
BK=64
BK=128
best
1
0.0892
0.1022
0.0859
128
32
0.1052
0.1112
0.1196
32
128
0.2848
0.2583
0.8153
64
512
1.1005
0.9170
2.6838
64
The BK=128 collapse at M ≥ 128 is not a K-depth effect: it is a register spill at the MT=8 VGPR ceiling (see VGPR vs LDS). BK rides on the MTILES dispatch (1→128, 2→32, ≥4→64), so the per-M choice costs no extra instantiations.
Redundant compute as a splitting tool: it wins and loses on the same kernel
FA2-JUMBO fits head_dim 512 by splitting the head across four CTAs, each redundantly computing the full-width QK product and softmax so all slices derive identical stats and compose with split-KV unchanged. On compute-rich prefill that redundancy is nearly free and the kernel measured 2.8-2.9× tuned Triton at every depth. On bandwidth-bound decode the same design re-streams K four times and measured 3× slower than FA1, which fits the width in one workgroup and reads each K tile once, so decode routes to FA1 and prefill to JUMBO, permanently. The same trade appears in the GDN serial stage, where duplicating the recurrent state in every warp's registers (redundant VALU) replaced cross-warp LDS handoffs and barriers. Redundancy is a currency: spend it where the kernel is compute-slack, never where it is byte-bound.
Row shape routing: FA1 vs FA2-v2 is decided by live rows, not preference
Spec-verify decode (2-6 query rows) measured 2.3-3.4× faster on FA2-v2 than FA1: multi-row batches are architecturally wrong for a kernel whose warps split KV columns of a single 16-row tile. But the win does not extend down to one row: at query_len 1 only gqa of v2's 128 rows are live (95% dead at gqa 6) against 6 of FA1's 16 (62% dead), and FA1 won all four measured decode points by 7-27% at matched segments (b=1 L=32768: FA1 s64 0.083 ms vs v2 s64 0.105). The shipped routing (plain decode on FA1, MTP/spec-verify on v2) is measured on both sides, and a probe run at s128 confirms it: FA1 0.066 ms against v2's best 0.073.
The selection law: TARGET from the ELF
The decode segment selector (hipattn/clav_attn/backend.py: _wgs_per_cu, _seg_target, _snap_segments, _formula_segments) is the most complete result in this section: a runtime tuning knob whose value is derived from the compiled kernel's own resource footprint, validated by pre-registered predictions that landed before the sweeps ran, carrying exactly one empirical constant, and scored against the exact per-cell argmin over six measured grids, 1,029 cells. This card is that derivation, end to end.
The arc: constant → exact table → a formula that beat the table on its own data
A pinned constant (segments=64) taxed every concurrent decode above ~4 streams 7-14%, because the optimum falls as batch rises (previous card).
An exact per-shape lookup table seemed forced: on the 1-workgroup-per-CU kernel no single CTA-fill target reproduced the argmin (64 CTAs at b=1 but 192 at both b=6 and b=12), which read as "nothing generalizes across shapes." That was a symptom, not a law: with one workgroup resident per CU, the effective fill target moved with whatever else was co-resident.
The occupancy fix killed the table's justification. Once the LDS alias made two workgroups per CU deterministic, the re-swept argmin collapsed onto plain machine-filling arithmetic, and a two-constant closed form scored 0.69% time-weighted regret / 19.9% worst against the exact table's 1.99% / 38.6%, on the table's own data. A closed form beating an exact per-shape lookup is the tell that the table had been fitting occupancy noise rather than shape structure.
segments = pow2snap( TARGET / (kv_heads × rows) ) in [16, ceiling(head_dim)]
then halved while a segment would cover fewer than MIN_TPS × 16-token KV tiles
TARGET = 64 CUs × the kernel's own workgroups-per-CU (derived; see below)
MIN_TPS = 2 (the lone empirical constant)
Rows, not sequences: under MTP a sequence contributes 1+num_speculative_tokens query rows, and rows are what widen the grid. The returned value may never exceed the allocated scratch bound: the kernel strides scratch by its template segment count, so a smaller value packs into a prefix of the same allocation, while a larger one walks off a buffer whose pointer CUDA-graph capture has already recorded. Under-shoot costs percent; over-shoot is a fault. Every path clamps.
TARGET is not a constant: it is the compiled footprint, read back
The fitted TARGETs stopped looking like fitted numbers once there were four of them. Workgroups-per-CU is the min of two bounds the toolchain already knows: LDS (65536 ÷ the kernel's static LDS, mirrored arithmetically from the .hip; off by 64 B of padding, which never crosses a bracket) and registers (1536 VGPRs/SIMD ÷ the allocation granule, × 2 SIMDs ÷ waves per CTA). The VGPR counts are compiler output, read off the shipped .so with llvm-readelf --notes: the one input that must be refreshed when a kernel changes shape:
kernel
LDS B
VGPRs
WG/CU
64 × WG/CU
measured best TARGET
FA1, d256
28032
110
2
128
128-192 (plateau)
FA1, d128
15744
110
4
256
256, predicted first
FA2-v2, d256
16896
234
3
192
192
FA2-v2, d128
8704
138
5
320
320, predicted first
Source: hipattn/clav_attn/backend.py (derivation comment block + _KERNEL_VGPRS); VGPRs via llvm-readelf --notes on clav_attn_C*.so
The d128 pair was written down before the sweep ran (2026-08-13), and both landed in the winning bracket: against +5.93%/+9.68% for reusing the d256 value and +6.86%/+9.86% for the old fitted d128 constant. The strongest single datum: the MTP d128 TARGET had been fitted at 1536 on an earlier sweep and shipped; re-measured, the fitted constant scored +9.86% mean regret against the derived 320's +2.89%. It had been fitted where the argmin was flat and never re-checked. A constant per head-dim was always a proxy for a quantity the compiler already knew.
MIN_TPS: the lone empirical constant, and why it had to be re-fit
MIN_TPS floors how few KV tiles a segment may cover before thin-segment reduce overhead dominates: it encodes a property of the reduce kernel, and when that kernel was rewritten 3.8× cheaper (34.9 → 9.2 µs, previous card), the knee the constant encoded moved. The old value (8) then clamped too hard: worst case b=1/L=2048 served s16 where s64 wins, +51.6%. The re-fit swept TARGET × MIN_TPS over the full post-fix grid:
Source: hipattn/tools/abs_wide_SUMMARY.txt (7 batches × 7 lengths × 5 segments × 4 reps, --hbm; full TARGET × MIN_TPS surface at the bottom of the file)
The re-fit moved TP4 from +5.06% mean / +51.6% worst to +1.40% / +13.5%, and TP2 to +0.00%: the exact argmin in all 49 cells. The reusable rule: an empirical constant encodes a property of a specific kernel, and it silently expires when that kernel changes. The reduce rewrite and the MIN_TPS re-fit were one change, not two.
The regret scoreboard: law vs exact argmin, all six grids
grid
cells
mean regret
worst cell
worst-cell cause
TP4 FA1 d256
49
+1.40%
+13.5%
raw over-division at b=12 depth (law s16, argmin s32); the known residual below
TP2 FA1 d256
49
+0.00%
+0.0%
exact argmin, every cell
TP4 v2 d256
294
+1.55%
+17.8%
qlen-1 cells: a shape v2 never serves (FA1 owns it); serving-range regret falls to +1.36% at qlen 6
TP2 v2 d256
294
+1.90%
+18.2%
same qlen-1 artifact
TP4 FA1 d128
49
+1.45%
+23.3%
the flat-128 ceiling at b=1 deep; since fixed by the per-head-dim ceiling (previous card)
TP4 v2 d128
294
+2.95%
+29.7%
same ceiling cell, same fix
Source: hipattn/tools/seg_law_worst_cells.txt; pooled regret over all six grids 1.96% → 1.86% after the per-head-dim ceiling landed, with no grid regressing
The MTP law's depth-invariance claim also graduated from assumption to measurement on this corpus: 35 of 49 (batch, length) cells have an argmin that does not move across qlen 1-6 at all, every mover settles by qlen 5, and shipped-law regret falls monotonically into the serving range (+3.00% at qlen 1 → +1.36% at qlen 6), so one closed form spans MTP depth and no qlen term is needed.
What the law still gets wrong, on purpose
FA1 over-divides at deep KV. Pure CU-fill under-counts past saturation: there is still marginal value in more in-flight segments than dividing by occupancy predicts (the b=12 depth cells, +11.7-13.5%). A depth-dependent floor was built and measured: scoped to FA1 it scores pooled +1.76%, the best of anything measured, but applied to both laws it is a net loss (+1.96% → +2.10%), because v2's grid is sequence-keyed and saturates differently. It is not shipped: it would reintroduce a partition-specific special case into a law whose whole point is deriving everything from the kernel footprint, so it waits for its own measurement pass (a floor swept per partition, not two constants guessed).
TP1 is unvalidated. Every cell ran kv_heads 1 or 2; kv_heads divides raw, so TP1 (kv_heads 4 on the deployed model) shifts the pick two brackets and is not covered. TP8 needs no separate test: kv heads replicate below one per rank, so it measures as kv_heads 1.
The VGPR table is the one manual input. A wrong entry costs accuracy, not correctness: the law degrades to the LDS-only bound. A hipFuncGetAttributes query op would remove it.
The two on-chip storage pools cap occupancy independently: LDS at 65536 B per workgroup, the register file at ~1536 VGPRs per SIMD divided by the kernel's per-wave allocation. On this part the register file is the residency that scales (1536 VGPRs per SIMD against 32 KB of LDS per CU-half), and the single most repeated structural win across all three kernel projects was moving working state out of LDS arenas into WMMA accumulator fragments, spending registers to buy co-residency.
Register residency is what admits wide tiles
Source: hipattn/README.md, measured by hipattn/tools/fa2v2_pressure_probe.hip. Compiled resources for the shipped kernels (hipattn/tools/clav_occupancy.txt): FA1 94 VGPRs at d128 / 110 at d256, FA2-v2 138 at d128, zero spill in every instantiation.
FA2-v2 keeps Q and the score matrix in registers (S = K·Qᵀ puts the query on the accumulator's column axis, so softmax state is lane-resident and the row reduction is one shfl_xor); LDS carries only the K/V tiles. That is what admits BLOCK_M=128 at every head size and the occupancy above: kernel-level 2.2× over staged FA2 at the deployed d256 shape, serving +42% prefill at 161k. The wide tile did not cause the win; it was unlocked by it. The GDN rewrite is the same lesson at project scale: the fla-Triton pipeline it chased runs zero LDS with VGPR 152-256 per kernel, while the old chunked kernels sat at one workgroup per CU behind 23-58 KB LDS arenas; the register-resident r-family measured 1.60-1.65× over the chunked split at T=65536 and closed serving prefill to −2.1% at 160k before the fusion work took it ahead.
Aliasing temporally-disjoint LDS: always correct, only sometimes fast
FA1: the epilogue accumulator staging (16 KB at d256) is dead during the whole KV loop; unioning it with the K/V tiles cut 44.5 → 28 KB, doubled workgroups per CU, and bought the 1.37-1.67× MALL-cold decode win. The cheapest occupancy doubling available on a bandwidth-bound kernel.
The GEMM: the identical union (epilogue spill into the activation tile, 19712 → 15616 B, 3 → 4 WG/CU with scratch still 0) measured null: the kernel is issue-bound, so the extra waves had nothing to fix. The union is kept anyway because it pays the LDS budget for a deeper activation chunk, worth −5 to −9% by halving chunk-loop barriers and fragment re-reads. Same transformation, different reason to keep it, and only the measurement tells them apart.
The VGPR ceiling, and its cliffs
MT
LDS/WG
VGPR
WG/CU
note
1
17152 B
65-73
3
LDS-bound
2
19712 B
87-91
3
LDS-bound
4
19712 B
139-143
3
LDS-bound
8
19712 B
191-195
3
at the register ceiling: 1536/191 = 8.04 waves/SIMD
16
19712 B
210-216
3
spills; excluded by the selection law
Source: rfihip/tools/res_achunk2.txt, from hipFuncGetAttributes and the driver's occupancy query, not a hand model. All four bit widths within ±5 VGPRs of these figures.
Three register cliffs cost real measurements before they were mapped:
Landing exactly on the ceiling is a constraint, not a comfort. At VGPR 191 the kernel gets 8 waves/SIMD; at 193 it gets 7. When a kernel sits there, nothing may cost registers, which is why several null-but-cheap "improvements" were rejected on VGPR grounds alone.
Dynamic indexing into a register array demotes it to scratch. An accumulator array indexed by a runtime value collapsed VGPR counts to a flat 87 at every MT (the fragments silently left the register file) and took M=512 from 0.70 to 2.10 ms. Every accumulator subscript in the shipped kernels is compile-time by construction: full unrolling of the chunk loop is load-bearing, not a hint.
Spill is the BK=128 collapse. The 0.26 → 0.82 ms cliff at M ≥ 128 in the K-depth table is VGPR 256 at MT=8 spilling: the same failure mode that excludes MT=16. A resource probe (WG/CU unchanged, Scratch_Size 0) is part of the gate for any register-spending change.
The A8 and A16 GEMM paths share the GPU, tile shape and bit width: the perfect internal control. Counting the emitted K loop isolates why A16 lagged: the weight LDS round-trip that A8's direct_b skips entirely.
per K-block per wave
W·A8
W·A16
v_wmma
32
32
ds_read
36
50
ds_write
10
76
s_barrier
4
18
LDS bytes / wmma
348
1096
Source: rfihip/README.md ISA account, counted by rfihip/tools/rfi_isa_count.py on the disassembled code object
Two caveats that both produced wrong readings before they were understood: barriers were not the cost (deleting all 256 per-workgroup barriers moved M=1 by 2%; the cost was instruction issue), and the bf16 LDS staging earns its keep: dequantizing straight into fragment registers, tried twice from opposite directions, measured −49% to −197%, because staging turns a per-element scatter into one coalesced store plus a load_matrix_sync.
Data types and the ISA
The most expensive wrong assumption in the record was about a data type. ROCm's precision-support matrix gives bfloat16 a green tick for RDNA4, and its legend defines that tick as "supported natively or with hardware emulation", without saying which. On gfx1201 it is the emulated kind, and the difference was the largest single term in the quantized dequant at every bit width. Do not read a support matrix as an ISA capability list; probe the assembler.
bf16 is matrix-only: the assembler-verified list
accepted
rejected ("instruction not supported on this GPU")
Source: rfihip/tools/bf16_isa_probe.txt, every opcode compiled against the gfx1201 assembler. The emulation goes through f32, not fp16: bf16 is the top half of an f32, so the compiler widens with one shift, computes in f32, and narrows with a ~6-instruction software round-to-nearest-even. fp16 has the full ALU including v_fma_mixlo_f16 (FMA + convert + half-placement in one instruction).
The cost, measured, and the hand-rolled fix
Compiling the same GEMM kernel at fp16 and bf16 and counting the emitted K loop put the conversion at 6.4 VALU per weight element, constant across all four bit widths: it was never the unpack. Hand-rolling the identical round-to-nearest-even (two integer ops per element plus one v_perm_b32 per pair, byte-select doing the pack the compiler spent a shift and an or on) produces the same bits:
Sources: rfihip/tools/isa_lut4reg.s / isa_fastrnd.s counted by rfi_isa_count.py; timing in rfihip/tools/ab_b8.txt et al.
Timed with the interleaved A/B protocol, the hand round is worth −17 to −25% at decode M and −4 to −8% at prefill M, on every rung of every width, with the numeric gate byte-identical between builds. It moved the bits=8 ratio against the tuned Triton peer to 1.45× at M=1. AMD hand-rolls this too, in their own gfx1201 FlashAttention kernel (they truncate; ours stays RNE and bit-identical to torch).
The same fix, three different verdicts: position against the bottleneck decides
Sources: gdnhip/tools/ab_f2bf_A_r1.txt / ab_f2bf_B_r1.txt (prefill_r vs trt referee in both builds), hipattn/tools/ab_f2bf.txt (attention: ratio 1.000-1.037 across all 20 rungs)
One transformation, three measured outcomes: −30 to −35% on the VALU-dense GDN prefill, −17 to −39% VALU (−4 to −25% time) on the GEMM dequant, and nothing on bandwidth-bound attention. A conversion is only worth attacking where it sits on the binding resource.
fp16 compute: exact where it matters, and served
bf16 → fp16 is mantissa-lossless (8 bits into 11); the exposure is range (fp16 saturates at 65504), which no microbenchmark can rule on because it is a property of the served activation distribution. Served and validated: running the quantized GEMM's activation path in fp16 measured a perplexity drift of 0.0001 against a frozen baseline whose own reproducibility band is ±0.0273. For scale, a real accuracy bug elsewhere in the stack (the GDN conv-state anchoring bug) cost +0.013 on the same metric, 130× more. fp16 compute ships as the default, decided at build time.
Source: rfihip/README.md ("SERVED AND VALIDATED"), rfihip/tools/first_serve_throughput.md §9c
fp8, int8, and register codebooks
fp8 WMMA is real and fast: 1.97× the fp16 rate (measured, wmma_rate_probe), with the same fragment lane equations as bf16: the attention fp8-KV path runs it straight from the cache. But it is not free precision: putting the GDN Gram matmuls on fp8 operands cost 25× accuracy for a net 10% slowdown, and the attention kernel's fp8 P-matrix needed an exponent shift (P×2⁸) to keep ordinary attention rows from flushing e4m3 subnormals to zero: self-consistent sparsification that never crashes and only shows up as end-of-prefill KL drift.
int8 with the stored byte as the dot operand is the fastest quantized path on the part: 457 GB/s effective at M=1, 74-96% of roofline, precisely because there is no unpack and no staging (direct_b).
A constexpr table indexed by a runtime value is not immediates. hipcc emitted the frozen 16-entry codebook to memory and kept a per-element global_load_i8: byte-for-byte identical ISA before and after the "optimization". Real register codebooks go through v_perm_b32 byte selects (the 4-entry 2-bit table lives in one dword, the 16-entry table in four). Whether a pre-scaled table pays follows a one-line rule discovered by modeling: build cost is entries/GROUP rounds per element: 4 entries over a 32-wide group won 1.36-1.47×, 16 entries over a 16-wide group is dead even, exactly as measured.
More measurements were invalidated by the harness than by the kernels. Every rule below exists because its absence produced a specific wrong number that steered real work, and each is cheap to follow compared to what it cost to learn.
Graph-time everything; keep the host out of the window
The two defects that invalidated every early attention number, both flattering the new kernel: host dispatch inside the timed window (Triton's Python wrapper costs ~32 µs/call against our ~8 µs, so sub-1k-context "kernel" comparisons were largely two Python wrappers racing), and clocks that never ramped behind a fixed 3-call warm-up. The fix: capture N calls into one CUDA graph and time a single replay, with a wall-clock warm-up running straight into the timed replays, and capture before the warm-up, because warming up and then stopping to capture hands the driver an idle window right before measurement. This is also the serving-relevant number: vLLM captures decode into CUDA graphs in production, so the naive column describes a path serving never takes.
Know your cache regime
The attention LDS-alias win measured 1.88× MALL-warm and 1.37× MALL-cold: the default harness regime overstated a real win by a third. Decode-shaped attention and GEMM rows are measured --hbm (a scrub defeats the 64 MB MALL) by standing rule.
GDN decode is deliberately measured the opposite way, MALL-hot, because its serving working set (24 MB of state) genuinely lives there. The regime follows serving, not a convention.
Effective GB/s above the roofline (the rotation's ~1200, attention's 114%-of-roofline short-context entries) means the scrub did not fully evict: read those as comparison figures, never as roofline claims.
The A/B protocol, and the noise floor
Within-variant spread on this hardware is 0.5-2.5%, so a naive two-rep A/B can report ~5% for a change worth ~3.5%. The protocol that produced every trusted delta: four interleaved reps per variant (A,B,A,B; thermal drift lands on both), discard high and low, average the middle two, and print an identical-code control beside every rung: the Triton peer, compiled into both builds, moved 0.1-0.3% wherever the run was quiet. Any kernel A/B under ~3% without this is not a measurement. Serve comparisons are the exception: a serve carries its own sustained heat and load, is repeatable as a single pair, and interleaving it is pure cost.
Source: rfihip/tools/rfi_ab.sh / raw ab_*.txt reps in all three trees
When the counters read zero, count the ISA, carefully
SQ_INSTS_VALU, SQ_INSTS_LDS and SQ_WAIT_ANY all return 0.0 on gfx1201 in the current rocprof build. The working substitute is static: disassemble the code object (no GPU needed), find the K-loop back-edge structurally, and count the instruction mix per iteration. It found the 1.31× instruction ratio that explained a 1.36× time gap with nothing left over. Two caveats, each of which produced a wrong reading first:
Static counts include both sides of a branch. A "3.5× more instructions" reading was almost entirely a ragged-tail fallback that never executes at deployed shapes; compiling it out measured null on the clock, because a uniform branch already skips it at runtime.
A kernel can contain several K loops, and "the widest backward branch" can select the slow one. The counter prints each loop's wmma count so the hot loop is identifiable.
Check comment claims against emitted code. Two source comments in one tree failed this check (the "constexpr codebook = immediates" claim among them); a claims-checker script now exists precisely because comments drift and the code object does not.
Sources: rfihip/tools/rfi_isa_count.py, rfi_isa_claims.py, rfihip/README.md ISA account
The microbench flatters; serving arbitrates
An idle-GPU harness hands each kernel the whole device: the best case for many-small-dispatch designs and the worst mirror of a loaded serve. The fused GDN kernel won its microbench and lost its serving A/B; the split design that replaced it won serving while looking worse in isolation. Direction and shape transfer; magnitude does not.
Per-kernel µs ratios are sound; multiplying them into a step budget is not. The profile localizes which kernel; only the serve sizes the win.
Throughput records say nothing about output quality. Dozens of zero-fault serving records shipped while a conv-state bug cost +0.013 perplexity and −6 tool-eval on every MTP serve. It passed every per-op gate because the gate's buffers were sized for the no-MTP configuration: the bug lived only at the deployed sizing, in state carried across calls. The companion failure in attention: an exp2 rewrite collapsed serving under MTP while every offline gate stayed green, because no gate ever ran multi-row queries together with split-KV. The rule both incidents force: gates must run the deployed configuration (buffer sizings, batch shapes, and state carried across calls), and a quality metric (PPL/tool-eval) rides along on every serve validation.
