pretraining
30 articles about pretraining in AI news
Gemini 4 Pretraining Begins, Google's Most Ambitious Run Yet
Google starts Gemini 4 pretraining, its most ambitious run yet. No details on compute or timeline; competitive pressure from OpenAI and Anthropic.
Alignment Pretraining Could Backfire, LessWrong Post Warns
LessWrong post warns synthetic alignment pretraining data could backfire in capable LLMs, leading to rebel personas.
SemiAnalysis: Pretraining Dead for All but Frontier Labs
@SemiAnalysis_ declares pretraining dead for non-frontier labs, citing 'Pretrainitis' as vanity-driven waste. Prompt engineering offers higher ROI.
GPT-5.5 'Spud' Prioritizes Pretraining Over Chain-of-Thought
A new OpenAI model, Spud (GPT-5.5), focuses on pretraining improvements rather than heavy test-time compute, promising faster and cheaper responses.
OpenAI Finishes GPT-5.5 'Spud' Pretraining, Halts Sora for Compute
OpenAI has finished pretraining its next major model, codenamed 'Spud' (likely GPT-5.5), built on a new architecture and data mix. The company reportedly halted its Sora video generation project entirely, sacrificing a $1B Disney investment, to prioritize compute for Spud's launch.
Why Deduplication Is the Most Underestimated Step in LLM Pretraining
A technical article on Medium argues that data deduplication is a critical, often overlooked step in LLM pretraining, directly impacting model performance and training cost. This is a foundational engineering concern for any team building or fine-tuning custom models.
Cursor Trains GPT-Size Model with 10-20x Compute
Cursor trained a GPT-size model from scratch with 10-20x more compute, announced at Compile. The move shifts from fine-tuning to pretraining for code generation.
Karpathy Joins Anthropic to Lead Recursive Self-Improvement Team
Andrej Karpathy joins Anthropic to lead a new recursive self-improvement team using Claude to accelerate pretraining, per @kimmonismus. The move signals a bet on synthetic data loops over brute-force scaling.
Genesis AI Reveals GENE-26.5: Humanoid Robot Cooks Stir-Fry, Solves Rubik's Cube
Genesis AI released GENE-26.5, a foundation model enabling a humanoid robot to autonomously cook stir-fry, solve Rubik's cubes, and organize cables. The approach uses human data pretraining and simulation closed-loop evaluation.
Geometric Latent Diffusion (GLD) Achieves SOTA Novel View Synthesis, Trains 4.4× Faster Than VAE
GLD repurposes features from geometric foundation models like Depth Anything 3 as a latent space for multi-view diffusion. It trains significantly faster than VAE-based approaches and achieves state-of-the-art novel view synthesis without text-to-image pretraining.
Tencent's Penguin-VL: A New Approach to Compact Multimodal AI
Tencent has launched Penguin-VL, a compact vision-language model that replaces traditional CLIP/SigLIP pretraining with an LLM-initialized vision encoder. The model achieves strong multimodal reasoning capabilities with just 2B and 8B parameter versions, potentially changing how smaller AI systems process images and text.
Brain-OF: The First Unified AI Model That Reads Multiple Brain Signals Simultaneously
Researchers have developed Brain-OF, the first omnifunctional foundation model that jointly processes fMRI, EEG, and MEG brain signals. This unified approach overcomes previous single-modality limitations by integrating complementary spatiotemporal data through innovative architecture and pretraining techniques.
SemiAnalysis: Open Models Still Trail Closed Frontier by 1-2 Years
SemiAnalysis argues open models still trail closed frontier by 1-2 years, with post-training and inference-time compute as key differentiators. The gap persists across all training eras.
Frontis-MA1 35B Beats GPT-5.5 on MLE-Bench Lite
Frontis-MA1, a 35B meta-evolution agent, scored 71.21% on MLE-Bench Lite, beating GPT-5.5 and Codex. The unverified claim suggests recursive self-improvement can rival trillion-scale models.
Decoy Font Tricks AI Vision Models With Dual-Layer Glyphs
Mixfont's Decoy Font hides text from AI vision models by layering two characters into one glyph, exploiting a tokenization blind spot in ChatGPT and Gemini.
Google Ships 3 Flash Models as 3.5 Pro Remains Missing
Google shipped three Gemini Flash models but 3.5 Pro remains delayed. Efficiency gains don't close the frontier gap with OpenAI and Anthropic.
GenCeption: Video Diffusion Backbone Beats Specialists on 5 Vision Tasks
GenCeption uses video diffusion as a vision backbone, matching specialists with 7-500x less data and generalizing from synthetic to real footage.
NVIDIA TwoTower: 2.4x Faster LLM Decoding, 98.7% Quality
NVIDIA TwoTower clones a pretrained LLM into a frozen context tower and trainable denoiser tower, achieving 2.42x faster generation with 98.7% quality on a 30B MoE model.
Four metagaming types need separate fixes or models learn to conceal it
A LessWrong taxonomy classifies AI metagaming into four types requiring separate fixes; blanket mitigation may teach concealment.
Alibaba's Qwen-RobotNav Unifies Robot Navigation in One 2B-8B Model
Alibaba's Qwen-RobotNav unifies VLN, ObjectNav, tracking, and autonomous driving in a 2B-8B model, deploying zero-shot to quadruped robots via a configurable observation protocol.
Epoch AI's EBR-Bench: Top Models Score 30-50% on Experience-Based Reasoning
Epoch AI's EBR-Bench tests experience-based reasoning. Top models score 30-50%, with Google Gemini 3 Pro leading at 48.2%, revealing a gap between pattern matching and true learning.
NanoEuler: GPT-2-Scale 116M Model Built in Pure C/CUDA From Scratch
NanoEuler is a 116M-parameter GPT-2-scale model built in pure C/CUDA from scratch. It provides a complete educational training pipeline for understanding LLMs at the lowest level.
OpenAI shows small doses of beneficial-trait RL improve 44 of 53 safety benchmarks — and the gains generalize
OpenAI researchers Jagadeesh, Saab, Singhal et al. published findings on June 18 showing RL training on traits like honesty and corrigibility improved 44 of 53 safety benchmarks. Gains generalized across domains not used in training, and the model resisted harmful fine-tuning better than the baselin
AI Generates Chest X-Rays Clinicians Cannot Tell Apart From Real Ones
RadiT XL, a 1.3B-parameter rectified flow transformer trained on 1.2 million chest radiographs, produces synthetic images that clinical experts cannot reliably distinguish from real ones — a milestone that could break the data bottleneck limiting medical AI fairness and generalization.
Qwen 2.5 7B Expresses Near-Constant Confidence Whether It Is Right or Wrong, Study Finds
A June 2026 arXiv preprint from University of Minnesota researchers tested Qwen 2.5 7B on structured clinical prediction data and found its verbalized confidence scores are essentially uninformative -- clustering between 0.856 and 0.937 no matter how well or badly the model performs. Combining SHAP-
NVIDIA Blackwell Sweeps MLPerf Training 6.0, GB300 Hits 1.6x Speedup
NVIDIA Blackwell swept MLPerf Training 6.0 across all seven benchmarks. GB300 NVL72 delivered 1.6x speedup over GB200 NVL72 using NVFP4 and 8,192 GPUs.
Google Titan: A New Architecture That Could Dethrone Transformers
Google's Titan architecture claims to surpass Transformers on long-context tasks via neural long-term memory, achieving 1.2x-2.5x speedups on benchmarks.
Prithvi-EO Fails Cross-Country Crop Yield Generalization, Paper Shows
Prithvi-EO and ViT-Base embeddings yield universally negative R² under cross-country maize yield prediction, failing to beat traditional spectral features due to yield distribution shift.
Sakana AI 7B Conductor Hits SOTA on GPQA-Diamond via Orchestration
Sakana AI's 7B Conductor model achieves SOTA on GPQA-Diamond and LiveCodeBench via orchestration of specialized sub-models, accepted at ICLR 2026.
ByteDance GenLIP: ViT Predicts Language Tokens Directly with 8B Samples
ByteDance's GenLIP trains ViTs to predict language tokens directly with a single autoregressive objective, outperforming baselines on 8B samples.