transformer architecture
30 articles about transformer architecture in AI news
RF-DETR: A Real-Time Transformer Architecture That Surpasses 60 mAP on COCO
RF-DETR is a new lightweight detection transformer using neural architecture search and internet-scale pre-training. It's the first real-time detector to exceed 60 mAP on COCO, addressing generalization issues in current models.
LeCun's NYU Team Unveils Breakthrough in Efficient Transformer Architecture
Yann LeCun and NYU collaborators have published new research offering significant improvements to Transformer efficiency. The work addresses critical computational bottlenecks in current architectures while maintaining performance.
Amazon's T-REX: A Transformer Architecture for Next-Basket Grocery Recommendations
Amazon researchers propose T-REX, a transformer-based model for grocery basket recommendations. It addresses unique challenges like repetitive purchases and sparse patterns through category-level modeling and causal masking, showing significant improvements in offline/online tests.
NVIDIA's DiffiT: A New Vision Transformer Architecture Sets Diffusion Model Benchmark
NVIDIA has released DiffiT, a Diffusion Vision Transformer achieving state-of-the-art image generation with an FID score of 1.73 on ImageNet-256 while using fewer parameters than previous models.
STAR-Set Transformer: AI Finally Makes Sense of Messy Medical Data
Researchers have developed a new transformer architecture that handles irregular, asynchronous medical time series by incorporating temporal and variable-type attention biases, outperforming existing methods on ICU prediction tasks while providing interpretable insights.
SORT: The Transformer Breakthrough for Luxury E-commerce Ranking
SORT is an optimized Transformer architecture designed for industrial-scale product ranking. It overcomes data sparsity to deliver hyper-personalized recommendations, proven to increase orders by 6.35% and GMV by 5.47% while halving latency.
Sam Altman Predicts Next 'Transformer-Level' Architecture Breakthrough, Says AI Models Are Now Smart Enough to Help Find It
OpenAI CEO Sam Altman stated he believes a new AI architecture, offering gains as significant as transformers over LSTMs, is yet to be discovered. He argues current advanced models are now sufficiently capable of assisting in that foundational research.
8 AI Model Architectures Visually Explained: From Transformers to CNNs and VAEs
A visual guide maps eight foundational AI model architectures, including Transformers, CNNs, and VAEs, providing a clear reference for understanding specialized models beyond LLMs.
Sam Altman Teases 'Massive Upgrade' AI Architecture, Compares Impact to Transformers vs. LSTM
OpenAI CEO Sam Altman said a new AI architecture is coming that represents a 'massive upgrade' comparable to the Transformer's leap over LSTM. He also stated current frontier models are now powerful enough to help research these next breakthroughs.
New Pipeline Enables Lossless Distillation of Transformer LLMs into Hybrid xLSTM Architectures
Researchers developed a distillation pipeline that transfers transformer LLM knowledge into hybrid xLSTM models. The distilled students match or exceed teacher models like Llama, Qwen, and Olmo on downstream tasks.
Beyond the Transformer: Liquid AI's Hybrid Architecture Challenges the 'Bigger is Better' Paradigm
Liquid AI's LFM2-24B-A2B model introduces a novel hybrid architecture blending convolutions with attention, addressing critical scaling bottlenecks in modern LLMs. This 24-billion parameter model could redefine efficiency standards in AI development.
Apple's 'Attention to Mamba' Paper Proposes Cross-Architecture Transfer
Apple researchers introduced a two-stage recipe for transferring capabilities from Transformer models to Mamba-based architectures. This could enable efficient models that retain the performance of larger, attention-based predecessors.
QV-Ka: New Research Proposes Eliminating Key Projection from Transformer Attention
A new arXiv paper argues the Key projection in Transformer attention is theoretically redundant. The proposed QV-Ka scheme removes it, simplifying architecture while maintaining performance on language tasks.
Google's TITANS Architecture: A Neuroscience-Inspired Revolution in AI Memory
Google's TITANS architecture represents a fundamental shift from transformer limitations by implementing cognitive neuroscience principles for adaptive memory. This breakthrough enables test-time learning and addresses the quadratic scaling problem that has constrained AI development.
How a Custom Multimodal Transformer Beat a Fine-Tuned LLM for Attribute
LeBonCoin's ML team built a custom late-fusion transformer that uses pre-computed visual embeddings and character n-gram text vectors to predict ad attributes. It outperformed a fine-tuned VLM while running on CPU with sub-200ms latency, offering calibrated probabilities and 15-minute retraining cycles.
NVIDIA Nemotron 3 Super: 120B Hybrid Mamba-Transformer MoE with 1M Context
NVIDIA has released Nemotron 3 Super, a 120B parameter open hybrid Mamba-Transformer Mixture of Experts model with 12B active parameters and 1M token context length. The company claims it delivers up to 7.5x higher throughput than similar open models.
Google's Memory Caching Bridges RNN-Transformer Gap with O(NL) Complexity
Google's 'Memory Caching' method saves RNN memory states at segment boundaries, allowing tokens to reference past checkpoints. This O(NL) approach significantly improves RNN performance on recall tasks, narrowing the gap with Transformers.
Tiny 9M Parameter LLM Tutorial Runs on Colab, Demystifies Transformer Training
A developer shared a complete tutorial for training a ~9M parameter transformer language model from scratch, including tokenizer, training, and inference, all runnable on Google Colab in minutes.
ASI-Evolve: This AI Designs Better AI Than Humans Can — 105 New Architectures, Zero Human Guidance
Researchers built an AI that runs the entire research cycle on its own — reading papers, designing experiments, running them, and learning from results. It discovered 105 architectures that beat human-designed models, and invented new learning algorithms. Open-sourced.
Goal-Aligned Recommendation Systems: Lessons from Return-Aligned Decision Transformer
The article discusses Return-Aligned Decision Transformer (RADT), a method that aligns recommender systems with long-term business returns. It addresses the common problem where models ignore target signals, offering a framework for transaction-driven recommendations.
SteerViT Enables Natural Language Control of Vision Transformer Attention Maps
Researchers introduced SteerViT, a method that modifies Vision Transformers to accept natural language instructions, enabling users to steer the model's visual attention toward specific objects or concepts while maintaining representation quality.
UniMixer: A Unified Architecture for Scaling Laws in Recommendation Systems
A new arXiv paper introduces UniMixer, a unified scaling architecture for recommender systems. It bridges attention-based, TokenMixer-based, and factorization-machine-based methods into a single theoretical framework, aiming to improve parameter efficiency and scaling return on investment (ROI).
LSA: A New Transformer Model for Dynamic Aspect-Based Recommendation
Researchers propose LSA, a Long-Short-term Aspect Interest Transformer, to model the dynamic nature of user preferences in aspect-based recommender systems. It improves prediction accuracy by 2.55% on average by weighting aspects from both recent and long-term behavior.
Luma Labs Launches Uni-1: An Autoregressive Transformer for Image Generation with a Pre-Generation Reasoning Phase
Luma Labs has released Uni-1, a foundational image model that uses an autoregressive transformer to reason about user intent before generating pixels. It aims to address the 'intent gap' common in diffusion models by adding a structured reasoning step.
WiT: Waypoint Diffusion Transformers Achieve FID 2.09 on ImageNet 256×256 in 265 Epochs, Matching JiT-L/16 Efficiency
Researchers introduced WiT, a diffusion transformer that uses semantic waypoints from pretrained vision models to resolve trajectory conflicts in pixel-space flow matching. It matches the performance of JiT-L/16 at 600 epochs in just 265 epochs, achieving an FID of 2.09 on ImageNet 256×256.
A Deep Dive into LoRA: The Mathematics, Architecture, and Deployment of Low-Rank Adaptation
A technical guide explores the mathematical foundations, memory architecture, and structural consequences of Low-Rank Adaptation (LoRA) for fine-tuning LLMs. It provides critical insights for practitioners implementing efficient model customization.
LLM Architecture Gallery Compiles 38 Model Designs from 2024-2026 with Diagrams and Code
A new open-source repository provides annotated architecture diagrams, key design choices, and code implementations for 38 major LLMs released between 2024 and 2026, including DeepSeek V3, Qwen3 variants, and GLM-5 744B.
From Browsing History to Personalized Emails: Transformer-Based Product Recommendations
A technical article outlines a transformer-based system for generating personalized product recommendations from user browsing data, directly applicable to retail and luxury e-commerce for enhancing email marketing and on-site personalization.
Graph Tokenization: A New Method to Apply Transformers to Graph Data
Researchers propose a framework that converts graph-structured data into sequences using reversible serialization and BPE tokenization. This enables standard Transformers like BERT to achieve state-of-the-art results on graph benchmarks, outperforming specialized graph models.
HyperTokens Break the Forgetting Cycle: A New Architecture for Continual Multimodal AI Learning
Researchers introduce HyperTokens, a transformer-based system that generates task-specific tokens on demand for continual video-language learning. This approach dramatically reduces catastrophic forgetting while maintaining fixed memory costs, enabling AI models to learn sequentially without losing previous knowledge.