NVIDIA details strategy for fast long-context LLMs
NVIDIA researchers have outlined a hardware-model co-design strategy for LLM attention, giving developers a concrete blueprint to maximize long-context inference speed on modern GPUs.
NVIDIA has published a hardware-model co-design framework to optimize attention mechanisms for fast, interactive long-context inference. As workloads scale, attention dominates processing costs; for instance, in DeepSeek-R1, the attention phase's share of prefill time rises from 18 percent at a 4K context length to 85 percent at 128K. To address this, the framework analyzes how key parameters like group size, head dimension, and sequence length interact with GPU hardware during the compute-bound prefill and memory-bound decode phases, utilizing FP8 for both attention compute and the key-value cache.
For practitioners, the guidelines offer specific architectural targets to maximize throughput. Group size, representing query heads per key-value head, should be pushed high to accelerate decode. While varying group size from 1 to 64 alters prefill runtime by under 1 percent, doubling it halves decode runtime. Raising group size from 1 to 8 yields an 8x gain in decode arithmetic intensity. At a 32K input sequence length, raising group size from 8 to 16 improves arithmetic intensity by under 6 percent, and the decode runtime curve flattens beyond a group size of 16 for a 32K sequence length compared to a 128K sequence length. Additionally, developers should select a head dimension of 128 or 256 to align with 128-byte hardware transfers. A head dimension of 64 still incurs the hardware cost of a 128-wide tile, whereas dimensions of 512 or greater approach tensor memory capacity limits.
Managing sequence length and parallelism is also critical. Prefill work scales quadratically with input sequence length, meaning doubling the length quadruples runtime, whereas decode scales linearly with key-value sequence length. To mitigate these costs, developers can compress the key-value cache or use hybrid architectures like NVIDIA Nemotron 3. Furthermore, tensor parallelism must remain less than or equal to the number of key-value heads to prevent memory-wasting duplication. For models with very few key-value heads, such as Nemotron 3 with two heads or multi-query attention with one, practitioners should scale using alternative strategies in TensorRT-LLM, such as Attention Data Parallelism, KV Parallelism, Wide EP, or Helix Parallelism.
This is our own summary of reporting by NVIDIA Developer Blog
