transformers
30 articles about transformers in AI news
Google Titan: A New Architecture That Could Dethrone Transformers
Google's Titan architecture claims to surpass Transformers on long-context tasks via neural long-term memory, achieving 1.2x-2.5x speedups on benchmarks.
RF-DETR Hits Hugging Face Transformers: SOTA Real-Time Detection
Roboflow's RF-DETR, a SOTA real-time detection model, integrated into Hugging Face Transformers, bridging DETR accuracy with real-time speed.
8 AI Model Architectures Visually Explained: From Transformers to CNNs and VAEs
A visual guide maps eight foundational AI model architectures, including Transformers, CNNs, and VAEs, providing a clear reference for understanding specialized models beyond LLMs.
Graph Tokenization: A New Method to Apply Transformers to Graph Data
Researchers propose a framework that converts graph-structured data into sequences using reversible serialization and BPE tokenization. This enables standard Transformers like BERT to achieve state-of-the-art results on graph benchmarks, outperforming specialized graph models.
WiT: Waypoint Diffusion Transformers Achieve FID 2.09 on ImageNet 256×256 in 265 Epochs, Matching JiT-L/16 Efficiency
Researchers introduced WiT, a diffusion transformer that uses semantic waypoints from pretrained vision models to resolve trajectory conflicts in pixel-space flow matching. It matches the performance of JiT-L/16 at 600 epochs in just 265 epochs, achieving an FID of 2.09 on ImageNet 256×256.
Sam Altman Teases 'Massive Upgrade' AI Architecture, Compares Impact to Transformers vs. LSTM
OpenAI CEO Sam Altman said a new AI architecture is coming that represents a 'massive upgrade' comparable to the Transformer's leap over LSTM. He also stated current frontier models are now powerful enough to help research these next breakthroughs.
MultiHashFormer Brings Hash-Based Autoregression to Causal LMs
MultiHashFormer brings hash-based autoregression to causal LMs, slashing embedding memory and outperforming standard Transformers from 100M to 3B parameters.
Computer Vision Deployments Drive Retail Productivity Gains
Computer vision deployments in retail are driving productivity gains by automating inventory, checkout, and loss prevention. AI News reports that retailers using these systems see measurable operational improvements. The technology leverages vision transformers and cloud platforms like Google Cloud.
Meta's Sapiens2: 1B Human Image ViTs for Pose, Segmentation, Normals
Meta open-sourced Sapiens2 on Hugging Face, a family of vision transformers pretrained on 1 billion human images for pose estimation, segmentation, normal estimation, and point maps. The models target high-resolution human-centric perception.
Google's Memory Caching Bridges RNN-Transformer Gap with O(NL) Complexity
Google's 'Memory Caching' method saves RNN memory states at segment boundaries, allowing tokens to reference past checkpoints. This O(NL) approach significantly improves RNN performance on recall tasks, narrowing the gap with Transformers.
SteerViT Enables Natural Language Control of Vision Transformer Attention Maps
Researchers introduced SteerViT, a method that modifies Vision Transformers to accept natural language instructions, enabling users to steer the model's visual attention toward specific objects or concepts while maintaining representation quality.
Sam Altman Predicts Next 'Transformer-Level' Architecture Breakthrough, Says AI Models Are Now Smart Enough to Help Find It
OpenAI CEO Sam Altman stated he believes a new AI architecture, offering gains as significant as transformers over LSTMs, is yet to be discovered. He argues current advanced models are now sufficiently capable of assisting in that foundational research.
ViTRM: Vision Tiny Recursion Model Achieves Competitive CIFAR Performance with 84x Fewer Parameters Than ViT
Researchers propose ViTRM, a parameter-efficient vision model that replaces a multi-layer ViT encoder with a single 3-layer block applied recursively. It uses up to 84x fewer parameters than Vision Transformers while maintaining competitive accuracy on CIFAR-10 and CIFAR-100.
Vision AI Breakthrough: Automated Multi-Label Annotation Unlocks ImageNet's True Potential
Researchers have developed an automated pipeline to convert ImageNet's single-label training set into a multi-label dataset without human annotation. Using self-supervised Vision Transformers, the method improves model accuracy and transfer learning capabilities, addressing long-standing limitations in computer vision benchmarks.
Kimi Team's 'Attention Residuals' Replace Fixed Summation with Softmax Attention, Boosts GPQA-Diamond by +7.5%
Researchers propose Attention Residuals, a content-dependent alternative to standard residual connections in Transformers. The method improves scaling laws, matches a baseline trained with 1.25x more compute, and adds under 2% inference overhead.
PartRAG Revolutionizes 3D Generation with Retrieval-Augmented Part-Level Control
Researchers introduce PartRAG, a breakthrough framework that combines retrieval-augmented generation with diffusion transformers for precise part-level 3D creation and editing from single images. The system achieves superior geometric accuracy while enabling localized modifications without regenerating entire objects.
LLMs Spontaneously Develop Human-Like Brain Regions for Language, Math
LLMs spontaneously develop human-like brain regions for language, math, physics, and social reasoning, per @LiorOnAI. Two optimization processes converged on the same solution.
Building a Tiny Recommendation Engine with Embeddings Only
A developer created a tiny recommendation engine using only embeddings, demonstrating a lightweight approach to item-to-item recommendations without complex infrastructure.
How Simon Willison Ported a 0.2B Image Model to the Browser with Claude
Simon Willison used Claude Code to port a 0.2B image inpainting model to WebGPU, running it as a parallel side project while his main agent worked on Datasette. The technique? Research with Claude.ai, then hand off to Claude Code with research.md.
PRS 2026: Netflix Workshop Reveals Industry Shift to LLM-Powered
Netflix's 2026 PRS workshop featured DoorDash, LinkedIn, Pinterest, Google DeepMind, and Stanford, showcasing how LLMs are transforming personalization, recommendation, and search. The event underscored the industry's shift toward integrating large language models into core recommendation pipelines.
mlx-vlm v0.6.2 Adds Gemma 4 QAT Support for Local GPUs
mlx-vlm v0.6.2 adds launch-day support for Google DeepMind's Gemma 4 QAT checkpoints, enabling local inference on consumer GPUs and edge devices with video input for the 12B model.
MLLM Raters Show Central Tendency Bias in Clinical Scoring
Study finds GPT-5 and other MLLMs show central tendency bias in clinical scoring, compressing predictions toward scale midpoint despite prompt modifications.
Collider-Bench Tests LLM Agents on LHC Analysis Reproduction
Collider-Bench tests LLM agents on reproducing LHC analyses from papers. No agent beats physicist-in-the-loop, highlighting gaps in scientific reasoning.
Tariffs Threaten $200B AI Data Center Buildout, CSIS Warns
CSIS warns tariffs could raise AI data center costs 20-30%, threatening $200B US hyperscaler buildout through 2028.
AI Data Centers Face 4-Year Post-Approval Delays, PJM Data Shows
PJM data shows AI data centers face 4-year post-approval delays, longer than the queue, threatening $700B CapEx plans.
PJM Data: AI Datacenter Delays Shift to Post-Approval Phase
PJM data shows AI datacenter projects now average 7+ years to go live, with post-approval delays of 4 years outpacing queue times, driven by transmission and supply chain bottlenecks.
Perplexity Claims 3x Blackwell Inference Throughput for 70B Models
Perplexity AI claims 3x inference throughput for 70B models on Nvidia Blackwell GPUs via FP4 and custom scheduling. The gain exceeds Nvidia's own 2x marketing claim.
Nvidia Blackwell CLC Boosts GEMM Tile Scheduling by 15% Over Static Persistence
Nvidia Blackwell CLC delivers up to 15% higher GEMM throughput via dynamic persistent tile scheduling, fixing load imbalance without startup overhead.
Claude Code's HTML Output Beats Markdown for LLM-Readable Docs
Claude Code generates HTML docs that LLMs parse more accurately than Markdown, per Thariq's analysis. Trade-off: harder for humans to edit.
Unsloth × NVIDIA Cut LLM Fine-Tuning ~25% — Three Glue-Code Wins on Blackwell
Daniel & Michael Han at Unsloth, in collaboration with NVIDIA, published a joint guide quantifying three glue-code optimizations that combine for ~25% faster LLM training on B200 Blackwell hardware. The wins target overhead around the main kernels — caching packed-sequence metadata, double-buffered gradient checkpoint reloads, and a cheaper GPT-OSS MoE router using argsort + bincount. All three are merged via public PRs.