When AI research reaches production.

An agent pattern that interleaves reasoning traces with tool-use actions, using each observation to refine the next reasoning step.

Self-supervised approach where an LM learns when and how to call external APIs by generating and filtering its own tool-use demonstrations.

Agent framework that converts environment feedback into verbal self-reflection stored in memory, improving performance across trials without weight updates.

Learning reward functions from pairwise human comparisons rather than hand-coded rewards. The direct precursor to RLHF.

Using an LM to generate adversarial prompts that elicit harmful behavior, scaling safety evaluation far beyond human red-teaming.

A three-stage recipe (SFT → reward model from human comparisons → PPO) that aligns LM outputs with human preferences. InstructGPT is the canonical reference.

Training harmless assistants using a written constitution of principles and an AI-generated critique/revision loop rather than human labels for every case.

Aligning LMs to preference data by directly optimizing a closed-form likelihood ratio, eliminating the reward model and RL loop of RLHF.

Using an off-the-shelf LLM to generate preference labels, scaling preference learning without human annotators.

A preference-optimization variant that avoids DPO's over-fitting by adding an explicit regularizer.

Iterative alignment where the LM judges its own outputs using an LLM-as-a-judge prompt, removing human-labeled preferences from the loop.

Alignment method that treats individual completions as binary-good/bad signals (no preference pairs needed) inspired by prospect theory.

An architecture where a router activates only a subset of expert sub-networks per token, scaling parameter count without proportional compute cost.

A sequence-to-sequence architecture that replaces recurrence with scaled dot-product attention, enabling parallel training and long-range context modeling.

A relative-position encoding that rotates query/key vectors in complex space, giving transformers better length extrapolation than absolute sinusoidal embeddings.

An inference-time optimization that groups multiple query heads to share a single key/value head, reducing KV cache memory at minimal quality loss.

A method to extend RoPE-based models to much longer contexts via frequency-dependent interpolation, with minimal fine-tuning data.

A state-space sequence model with input-dependent selection that matches Transformer quality with linear inference cost and unlimited context.

A technique letting tokens skip transformer layers when unnecessary, allocating compute adaptively based on token importance.

A tiled, IO-aware attention kernel that computes exact attention with linear memory by fusing reads/writes to SRAM.

A scheduling technique that adds/removes requests at the iteration level rather than the batch level, dramatically increasing throughput for LLM serving.

Row-wise and vector-wise INT8 quantization with outlier detection that enables zero-degradation 8-bit inference of LLMs.

Post-training quantization to 3-4 bits using second-order information, enabling 175B-scale LLMs to run on single-GPU inference.

An inference technique where a small draft model proposes tokens and a large model verifies them in parallel, yielding 2-3x speedup without quality loss.

4-bit weight quantization that preserves salient weights based on activation magnitudes, matching GPTQ quality with faster inference.

A memory-management scheme for KV cache modeled on OS paging, eliminating fragmentation and enabling high-throughput serving.

A sliding-window attention pattern with preserved initial tokens ("sinks") that enables indefinite streaming generation without quality collapse.

Training sparse autoencoders on residual-stream activations to extract monosemantic, human-interpretable features from transformer internals.

Applying a standard Transformer directly to sequences of image patches, establishing Transformers as the dominant image-recognition backbone.

Dual-encoder model trained on 400M image-caption pairs to align image and text embeddings, enabling zero-shot visual classification.

Diffusion performed in a compressed VAE latent space, making high-resolution image generation tractable on consumer GPUs.

Cross-attention layers interleaved into a frozen LLM that attend to vision features, enabling few-shot visual question answering.

Encoder-decoder Transformer trained on 680k hours of weakly-supervised multilingual speech, setting new robustness benchmarks across accents and noise.

Projecting CLIP features into an LLM's token space via a simple projector + instruction tuning on GPT-4-generated visual conversations.

A prompting technique that elicits step-by-step reasoning by showing exemplars that include intermediate reasoning steps.

Sample multiple CoT completions and take the majority-vote answer, substantially improving reasoning accuracy.

Eliciting step-by-step reasoning without few-shot exemplars, simply by appending a phrase like "let's think step by step".

Reasoning over a tree of intermediate thoughts with explicit look-ahead, backtracking, and self-evaluation, beyond linear CoT.

Reward models trained to score each intermediate reasoning step rather than only the final answer, enabling superior reasoning policy learning.

Allocating more compute at inference (longer reasoning chains, multiple samples + verifier) can outperform scaling parameters — the basis for o1-style reasoning models.

Learned dual-encoder retrieval that outperforms BM25 on open-domain QA by training encoders on question-passage pairs.

Conditioning generation on retrieved passages from a non-parametric memory, combining parametric and retrieval-based knowledge.

Parameter-efficient fine-tuning that injects low-rank decomposition matrices into attention weights, training <1% of parameters.

Fine-tuning a pretrained LM on a mixture of tasks phrased as natural-language instructions, enabling strong zero-shot generalization.

Scaling law showing compute-optimal models use ~20 training tokens per parameter — correcting prior over-parameterization in GPT-3-era models.

Bootstrapping instruction-tuning data by having an LM generate its own instructions, inputs, and outputs from a small seed set.

LoRA fine-tuning on 4-bit quantized base weights, enabling 65B-model fine-tuning on a single 48GB GPU.

Training smaller models on GPT-4-generated explanation traces rather than answer-only demonstrations, closing the capability gap.

Sampling multiple completions, scoring with a reward model, and fine-tuning on the top samples — a simpler alternative to PPO used in Llama 2.