Kimi K2.5

Ingredient list

1. #1YaRN RoPE Context Extension

   Invented by Nous Research · 2023-08 · Velocity 3y

   “To achieve long context windows, models often use YaRN or similar RoPE extension techniques.”

   architecturemedium
2. #2Grouped-Query Attention (GQA)

   Invented by Google · 2023-05 · Velocity 3y

   “As a large-scale model, Kimi K2.5 likely uses GQA to manage KV cache memory efficiently for its 1T parameters.”

   architecturemedium
3. #3LLaVA (Visual Instruction Tuning)

   Invented by University of Wisconsin · 2023-04 · Velocity 3y

   “Kimi K2.5 is a multimodal model with vision capabilities, similar to LLaVA's approach of projecting visual features into LLM token space.”

   multimodalmedium
4. #4FlashAttention

   Invented by Stanford · 2022-05 · Velocity 4y

   “The model card mentions optimizations for efficient inference, which commonly includes FlashAttention for long-context handling.”

   inferencemedium
5. #5Chain-of-Thought Prompting

   Invented by Google · 2022-01 · Velocity 4y

   “Kimi K2.5 demonstrates step-by-step reasoning in its responses, a hallmark of chain-of-thought prompting.”

   reasoningmedium
6. #6Instruction Tuning (FLAN)

   Invented by Google · 2021-09 · Velocity 5y

   “Kimi models are instruction-tuned for conversational ability, aligning with FLAN-style training.”

   trainingmedium
7. #7Rotary Position Embedding (RoPE)

   Invented by Zhuiyi Technology · 2021-04 · Velocity 5y

   “Most modern LLMs use RoPE for position encoding; Kimi K2.5's long-context capability aligns with this.”

   architecturemedium
8. #8Vision Transformer (ViT)

   Invented by Google · 2020-10 · Velocity 5y

   “As a vision-language model, Kimi K2.5 likely uses Vision Transformer (ViT) for image patch encoding.”

   multimodalmedium
9. #9Transformer Self-Attention

   Invented by Google · 2017-06 · Velocity 9y

   “Kimi K2.5 is fundamentally a Transformer-based model, using self-attention as its core architecture.”

   architecturehigh
10. #10Mixture of Experts (Sparse MoE for LLMs)

    Invented by Google · 2017-01 · Velocity 9y

    “The 1 trillion parameter count strongly suggests a Mixture of Experts architecture to manage computational costs.”

    architecturehigh