Gemini 3 Flash

Ingredient list

1. #1Grouped-Query Attention (GQA)

   Invented by Google · 2023-05 · Velocity 3y

   “Gemini 1.5 models use grouped-query attention (GQA) for efficient inference, as detailed in the technical report.”

   architecturehigh
2. #2FlashAttention

   Invented by Stanford · 2022-05 · Velocity 4y

   “Gemini models use FlashAttention-2 for efficient training and inference, as stated in the Gemini 1.5 technical report.”

   inferencehigh
3. #3Chain-of-Thought Prompting

   Invented by Google · 2022-01 · Velocity 4y

   “Gemini models are trained to reason step-by-step, and Gemini 1.5 Flash was shown to use chain-of-thought reasoning in its technical report.”

   reasoninghigh
4. #4Instruction Tuning (FLAN)

   Invented by Google · 2021-09 · Velocity 4y

   “Gemini models are instruction-tuned, building upon the FLAN instruction-tuning methodology developed by Google.”

   traininghigh
5. #5Rotary Position Embedding (RoPE)

   Invented by Zhuiyi Technology · 2021-04 · Velocity 5y

   “Gemini models use rotary position embeddings (RoPE), as confirmed in the Gemini 1.5 technical report.”

   architecturehigh
6. #6Vision Transformer (ViT)

   Invented by Google · 2020-10 · Velocity 5y

   “Gemini models use a Vision Transformer (ViT) architecture for processing visual inputs, as detailed in the technical report.”

   multimodalhigh
7. #7Transformer Self-Attention

   Invented by Google · 2017-06 · Velocity 9y

   “Gemini models are based on the Transformer architecture, using decoder-only models with self-attention.”

   architecturehigh
8. #8Mixture of Experts (Sparse MoE for LLMs)

   Invented by Google · 2017-01 · Velocity 9y

   “Gemini 1.5 Pro uses a Mixture-of-Experts (MoE) architecture. While Flash is a dense model, the overall Gemini family deploys MoE.”

   architecturemedium