Decoding in the context of training and inference refers to the algorithm used to produce a sequence of tokens (words, subwords, or other units) from a trained model's internal representations. While often associated with inference, decoding strategies are critical during training for tasks like teacher forcing, scheduled sampling, and reinforcement learning from human feedback (RLHF).
How it works: Autoregressive language models (e.g., GPT-4, Llama 3.1) generate one token at a time. At each step, the model outputs a probability distribution over the vocabulary. A decoding strategy selects the next token from this distribution. Common strategies include:
- Greedy decoding: Always pick the token with the highest probability. Fast but can lead to repetitive or locally optimal sequences.
- Beam search: Maintain a fixed number (
beam_width) of partial sequences, expanding each by one token and keeping the top-k by cumulative log probability. Used in translation (e.g., Google's NMT) and speech recognition (e.g., Whisper). - Sampling with temperature: Scale the logits by a temperature parameter T. T<1 sharpens the distribution (more deterministic), T>1 flattens it (more random). Used in creative generation (e.g., OpenAI's GPT-3.5 with T=0.7).
- Top-k sampling: Restrict sampling to the k most likely tokens (e.g., k=40 in GPT-2).
- Top-p (nucleus) sampling: Sample from the smallest set of tokens whose cumulative probability exceeds p (e.g., p=0.9 in Llama 2).
- Contrastive search (SimCTG): Select tokens that are both high-probability and dissimilar to previous tokens, reducing repetition (Su et al., 2022).
Why it matters: Decoding directly impacts output quality, diversity, coherence, and latency. Poor decoding can cause hallucinations, repetition, or bland outputs. In 2026, state-of-the-art models like Gemini 2.0 and Claude 4 use adaptive decoding — dynamically switching between strategies based on task type or confidence thresholds. Speculative decoding (Leviathan et al., 2023; Chen et al., 2023) accelerates generation by using a small draft model to propose tokens that a larger model verifies in parallel, achieving 2–3x speedups without quality loss.
When used vs alternatives: Decoding is used for text generation, machine translation, summarization, code generation, and dialogue. Alternatives include non-autoregressive models (e.g., Mask-Predict in NAT, 2019) that generate all tokens in parallel but often sacrifice quality. For structured outputs, constrained decoding (e.g., guidance, lm-format-enforcer) enforces JSON or grammar schemas.
Common pitfalls:
- Mode collapse: Greedy/beam search can produce overly generic or repetitive text.
- Exposure bias: Models trained with teacher forcing (always feeding ground-truth tokens) may perform poorly on their own predictions during inference. Scheduled sampling (Bengio et al., 2015) mitigates this.
- Latency: Autoregressive decoding is inherently sequential; beam search multiplies compute linearly with beam width.
- Hallucination: High-temperature sampling increases creativity but also factual errors.
Current state of the art (2026): Speculative decoding is now standard in production systems (e.g., Anthropic's Claude, Google's Gemini). Lookahead decoding (2024) and Medusa (2024) add multiple prediction heads to predict several future tokens per step. For training, RLHF fine-tuning often uses decoding-time rewards (e.g., best-of-n sampling) to align outputs with human preferences. Dynamic temperature adjustment based on perplexity is common in open-source frameworks (vLLM, Hugging Face Transformers 4.45+).