GPT-4o

Ingredient list

1. #1Speculative Decoding

   Invented by Google · 2022-11 · Velocity 3y

   “GPT-4o's speed improvements align with inference optimizations like speculative decoding, though not explicitly confirmed.”

   inferencelow
2. #2FlashAttention

   Invented by Stanford · 2022-05 · Velocity 4y

   “OpenAI's Triton kernels, used for GPT-4, are predecessors to FlashAttention. GPT-4o's speed improvements suggest optimized attention.”

   inferencemedium
3. #3Zero-Shot Chain-of-Thought

   Invented by University of Tokyo · 2022-05 · Velocity 4y

   “GPT-4o exhibits zero-shot reasoning when prompted with 'think step by step', a hallmark of zero-shot CoT.”

   reasoninghigh
4. #4Self-Consistency

   Invented by Google · 2022-03 · Velocity 4y

   “System Card mentions sampling multiple outputs for evaluation, a core self-consistency technique.”

   reasoningmedium
5. #5Chain-of-Thought Prompting

   Invented by Google · 2022-01 · Velocity 4y

   “GPT-4o can be prompted to show step-by-step reasoning, a core CoT capability.”

   reasoninghigh
6. #6Instruction Tuning (FLAN)

   Invented by Google · 2021-09 · Velocity 4y

   “GPT-4o follows instruction-tuned models like InstructGPT, enabling strong zero-shot task following.”

   traininghigh
7. #7Rotary Position Embedding (RoPE)

   Invented by Zhuiyi Technology · 2021-04 · Velocity 5y

   “GPT models since GPT-3 use rotary position embeddings (RoPE). GPT-4o's architecture is a direct evolution.”

   architecturehigh
8. #8Transformer Self-Attention

   Invented by Google · 2017-06 · Velocity 9y

   “GPT-4o is a Transformer-based model, the core architecture of all GPT models.”

   architecturehigh
9. #9Mixture of Experts (Sparse MoE for LLMs)

   Invented by Google · 2017-01 · Velocity 9y

   “GPT-4 is widely reported as a MoE model. GPT-4o is its successor, implying architectural continuity.”

   architecturemedium