transformers
30 articles about transformers in AI news
Google Titan: A New Architecture That Could Dethrone Transformers
Google's Titan architecture claims to surpass Transformers on long-context tasks via neural long-term memory, achieving 1.2x-2.5x speedups on benchmarks.
RF-DETR Hits Hugging Face Transformers: SOTA Real-Time Detection
Roboflow's RF-DETR, a SOTA real-time detection model, integrated into Hugging Face Transformers, bridging DETR accuracy with real-time speed.
8 AI Model Architectures Visually Explained: From Transformers to CNNs and VAEs
A visual guide maps eight foundational AI model architectures, including Transformers, CNNs, and VAEs, providing a clear reference for understanding specialized models beyond LLMs.
Graph Tokenization: A New Method to Apply Transformers to Graph Data
Researchers propose a framework that converts graph-structured data into sequences using reversible serialization and BPE tokenization. This enables standard Transformers like BERT to achieve state-of-the-art results on graph benchmarks, outperforming specialized graph models.
WiT: Waypoint Diffusion Transformers Achieve FID 2.09 on ImageNet 256×256 in 265 Epochs, Matching JiT-L/16 Efficiency
Researchers introduced WiT, a diffusion transformer that uses semantic waypoints from pretrained vision models to resolve trajectory conflicts in pixel-space flow matching. It matches the performance of JiT-L/16 at 600 epochs in just 265 epochs, achieving an FID of 2.09 on ImageNet 256×256.
Sam Altman Teases 'Massive Upgrade' AI Architecture, Compares Impact to Transformers vs. LSTM
OpenAI CEO Sam Altman said a new AI architecture is coming that represents a 'massive upgrade' comparable to the Transformer's leap over LSTM. He also stated current frontier models are now powerful enough to help research these next breakthroughs.
How Generative Recommenders Are Redefining RecSys at Scale | NVIDIA
NVIDIA's technical blog details how generative recommenders are redefining RecSys at scale. It highlights a shift from two-stage pipelines to sequence-to-sequence transformers for large-scale platforms.
GigaWorld-Policy-0.5 Hits 85ms on RTX 4090 for Robot Control
GigaWorld-Policy-0.5 runs robot control at 85ms on an RTX 4090, using a Mixture-of-Transformers architecture for real-time local deployment.
MultiHashFormer Brings Hash-Based Autoregression to Causal LMs
MultiHashFormer brings hash-based autoregression to causal LMs, slashing embedding memory and outperforming standard Transformers from 100M to 3B parameters.
Computer Vision Deployments Drive Retail Productivity Gains
Computer vision deployments in retail are driving productivity gains by automating inventory, checkout, and loss prevention. AI News reports that retailers using these systems see measurable operational improvements. The technology leverages vision transformers and cloud platforms like Google Cloud.
Meta's Sapiens2: 1B Human Image ViTs for Pose, Segmentation, Normals
Meta open-sourced Sapiens2 on Hugging Face, a family of vision transformers pretrained on 1 billion human images for pose estimation, segmentation, normal estimation, and point maps. The models target high-resolution human-centric perception.
Google's Memory Caching Bridges RNN-Transformer Gap with O(NL) Complexity
Google's 'Memory Caching' method saves RNN memory states at segment boundaries, allowing tokens to reference past checkpoints. This O(NL) approach significantly improves RNN performance on recall tasks, narrowing the gap with Transformers.
SteerViT Enables Natural Language Control of Vision Transformer Attention Maps
Researchers introduced SteerViT, a method that modifies Vision Transformers to accept natural language instructions, enabling users to steer the model's visual attention toward specific objects or concepts while maintaining representation quality.
Sam Altman Predicts Next 'Transformer-Level' Architecture Breakthrough, Says AI Models Are Now Smart Enough to Help Find It
OpenAI CEO Sam Altman stated he believes a new AI architecture, offering gains as significant as transformers over LSTMs, is yet to be discovered. He argues current advanced models are now sufficiently capable of assisting in that foundational research.
ViTRM: Vision Tiny Recursion Model Achieves Competitive CIFAR Performance with 84x Fewer Parameters Than ViT
Researchers propose ViTRM, a parameter-efficient vision model that replaces a multi-layer ViT encoder with a single 3-layer block applied recursively. It uses up to 84x fewer parameters than Vision Transformers while maintaining competitive accuracy on CIFAR-10 and CIFAR-100.
Vision AI Breakthrough: Automated Multi-Label Annotation Unlocks ImageNet's True Potential
Researchers have developed an automated pipeline to convert ImageNet's single-label training set into a multi-label dataset without human annotation. Using self-supervised Vision Transformers, the method improves model accuracy and transfer learning capabilities, addressing long-standing limitations in computer vision benchmarks.
Kimi Team's 'Attention Residuals' Replace Fixed Summation with Softmax Attention, Boosts GPQA-Diamond by +7.5%
Researchers propose Attention Residuals, a content-dependent alternative to standard residual connections in Transformers. The method improves scaling laws, matches a baseline trained with 1.25x more compute, and adds under 2% inference overhead.
PartRAG Revolutionizes 3D Generation with Retrieval-Augmented Part-Level Control
Researchers introduce PartRAG, a breakthrough framework that combines retrieval-augmented generation with diffusion transformers for precise part-level 3D creation and editing from single images. The system achieves superior geometric accuracy while enabling localized modifications without regenerating entire objects.
SemiAnalysis: Open Models Still Trail Closed Frontier by 1-2 Years
SemiAnalysis argues open models still trail closed frontier by 1-2 years, with post-training and inference-time compute as key differentiators. The gap persists across all training eras.
Massive Activations Found in Hybrid Linear Attention LLMs
New paper finds massive activations in hybrid linear attention LLMs, forming pre-attention spikes and inter-spike plateaus. Code and checkpoints released on Hugging Face; implications for quantization and deployment.
OpenAI Loses GPT-3 GPU Builder Scott Gray; 13 Leaders Out in 2026
Scott Gray, OpenAI's founding GPU engineer, left in 2026—the 13th senior departure. The exodus spans every function, signaling systemic churn beyond the C-suite.
DeepSeek Sparse Attention: DSA Redefines Resource Allocation
SemiAnalysis touts DeepSeek Sparse Attention as a paradigm shift, but lacks data. Sparse attention could cut costs, yet no benchmarks yet.
Virginia Forces AI Data Centers to Fund Grid Upgrades After 76% Rate Hikes
Virginia requires AI data centers to pay for dedicated grid infrastructure after 76% rate hikes. Governor Youngkin says it saves civilians hundreds of millions.
OpenAI's $20B Georgia Data Center Tests Gigawatt Buildout
OpenAI announced $20B Georgia data center. Gigawatt buildout faces execution test amid supply chain and power constraints.
Rimnot, Yuantu Deploy 10K Robots in Server Factories by 2027
Rimnot partners Yuantu for 10K robots in server factories by 2027, achieving 30-minute adaptation at WAIC.
NUS CIMERA Chip Cuts LLM Memory Wall with Compute-in-Interconnect
NUS researchers propose CIMERA, an LLM inference accelerator integrating compute-in-interconnect and memory to mitigate the memory wall, detailed in arXiv:2607.13649 (July 2026).
CacheBlend: 2-4x Faster KV Cache for Multi-Doc Queries
CacheBlend reuses per-document KV caches by recomputing only boundary tokens, achieving 2-4x speedups on multi-document queries. Alibaba data shows 10% of blocks serve 77% of hits.
Airbnb Cuts LLM Eval From Weeks to a Day With Deterministic Caching
Airbnb cut LLM eval from weeks to a day with deterministic caching and micro adapters. The approach trains bug-fix patches in under an hour per GPU.
Soofi S 30B-A3B: German open model tops English, German benchmarks
German consortium releases Soofi S 30B-A3B, an open MoE model beating OLMo 3 and Apertus 70B on English and German benchmarks while activating only 3.2B of 31.6B parameters.
Feature Freshness: The Production Bug That Makes Good Recommenders Look Bad
Jie Li's article reveals that stale features—outdated user signals—can degrade recommender performance by 20-30% in offline metrics, often misdiagnosed as model problems. The piece urges teams to prioritize feature freshness monitoring alongside model tuning.