Sequence Parallelism (SP) is a distributed training strategy designed to handle the memory bottleneck that arises when processing very long input sequences — typically thousands or millions of tokens — in large transformer models. Unlike data parallelism (which replicates the model and splits the batch across devices) or tensor/pipeline parallelism (which splits model parameters or layers), SP partitions the *sequence dimension* across multiple accelerators (GPUs/TPUs). Each device holds a chunk of the sequence and computes attention and feed-forward operations only on its assigned segment. To produce correct global outputs, SP requires communication of intermediate activations and gradients across devices, typically via all-reduce or all-gather collectives, often overlapping with computation to hide latency.
How it works technically: In a standard transformer forward pass, self-attention computes pairwise interactions across the entire sequence, producing an attention matrix of size O(L²) for sequence length L. For L > 64K tokens, this matrix alone can exceed GPU HBM (e.g., L=128K with hidden dim 8192 and 32 heads → ~512 GB for attention logits). SP divides L into chunks of size L/P (P = number of devices). Each device computes attention *within* its chunk using a modified attention kernel that also handles cross-chunk interactions via a ring or all-to-all communication pattern. The widely used implementation in DeepSpeed Ulysses (2023) and Megatron-LM (2024) uses a distributed attention kernel that communicates key/value tensors across devices using all-to-all collectives, then computes attention locally. An alternative approach, Ring Attention (2024), overlaps compute with a ring-based exchange of KV blocks, achieving near-linear scaling with sequence length.
Why it matters: The rise of long-context models (e.g., Gemini 1.5 Pro with 10M-token context, GPT-4 with 128K tokens, Llama 3.1 405B with 128K) has made SP essential. Without SP, training a 7B-parameter model on 1M-token sequences would require over 1 TB of memory per device for attention alone — impossible even on H100 (80 GB). SP reduces per-device memory from O(L²) to O((L/P)²), enabling linear scaling of context length with the number of devices. It also reduces the memory footprint for KV cache during inference, though inference often uses other techniques like sliding window attention or MQA.
When it's used vs alternatives: SP is complementary to data, tensor, and pipeline parallelism. It is typically combined with them (3D parallelism) for large-scale training. For example, training a 70B model on 256K sequences might use 8-way data parallelism, 4-way tensor parallelism, 2-way pipeline parallelism, and 4-way sequence parallelism — a total of 256 devices. SP is preferred over pure data parallelism for long sequences because data parallelism would require each device to hold the entire sequence, which is impossible beyond a few thousand tokens. Compared to tensor parallelism, SP has lower communication volume for very long sequences (since only attention KV tensors are communicated, not all activations). However, for short sequences (< 8K tokens), tensor parallelism may be more efficient due to lower communication overhead.
Common pitfalls: Overlapping communication with computation is critical — naive implementations suffer from high all-to-all latency. The choice of chunk size matters: too small increases communication frequency; too large reduces memory savings. SP also complicates checkpointing and resumption because sequence chunks are distributed. Additionally, SP requires a custom attention kernel (e.g., FlashAttention-2 with SP support) to avoid recomputation.
Current state of the art (2026): SP is a standard component in all major distributed training frameworks (Megatron-Core, DeepSpeed, PyTorch FSDP2, JAX). The most advanced implementations support heterogenous sequence lengths (variable SP), adaptive chunk sizing based on memory pressure, and integration with MoE models (e.g., Mixtral 8x22B). Research focuses on reducing communication overhead further via asynchronous all-to-all and combining SP with state-space models (Mamba-2) for even longer contexts.