multi modal
30 articles about multi modal in AI news
ClBench-V: New Benchmark Tests Multimodal Contextual Learning in 3 Dimensions
ClBench-V benchmark from @HuggingPapers tests multimodal contextual learning across three dimensions: grounding, application, and knowledge learning. No results disclosed yet.
Scaling Laws Differ for Native Multimodal VLMs
A systematic study reveals distinct scaling laws for native multimodal pre-training, showing vision-language models require different compute-optimal size/token ratios than language-only models.
Google alone ships full any-to-any multimodal models
Mollick notes Google alone ships full any-to-any multimodal models; OpenAI and Anthropic lag. This gives Google a structural advantage in agentic workflows.
Building a Multimodal Vector Search Platform for Product Catalogs
Insider Engineering shares practical lessons from building a multimodal vector search platform for product catalogs, covering multitenancy, GPU economics, and infrastructure surprises. The post provides actionable insights for retail AI teams considering similar systems.
SingGuard: Runtime Guardrails for Multimodal AI Treat Safety as Input
SingGuard treats safety rules as runtime inputs for multimodal AI, achieving SOTA across 6 families and 35 datasets via fast/slow reasoning.
Apple AFM Core Advanced: Sparse, Multimodal, iPhone 17 Pro Only
Apple AFM Core Advanced is sparse, multimodal, and exclusive to iPhone 17 Pro, M3+ Mac, M4+ iPad, while AFM Core is dense for other devices.
Google Gemma 4 12B: Encoder-Free Multimodal Model Launches
Google launched Gemma 4 12B, an encoder-free multimodal model for on-device AI, reducing latency by eliminating the vision encoder.
MiniMax M3: Sparse Attention, 1M Context, Multimodal via Together
MiniMax M3 uses sparse attention for 1M context and multimodality, with Together AI serving fast inference.
ByteDance Open-Sources BAGEL: 7B Multimodal Model for Image Gen, Editing, Understanding
ByteDance open-sourced BAGEL, a 7B multimodal model for image gen, editing, style transfer, and understanding under Apache 2.0.
ByteDance Lance 3B MoE Beats 7B Models on Multimodal Benchmarks
ByteDance released Lance, a 3B multimodal MoE model that beats 7B+ models on benchmarks through multi-task synergy and specialized pathways.
Odyssey Launches Starchild-1, First Real-Time Multimodal World Model
Odyssey AI released Starchild-1, first real-time multimodal world model for video generation targeting embodied AI and robotics.
DataArc-SynData-Toolkit: Open-Source Framework for Multimodal Synthetic Data
DataArc-SynData-Toolkit is an open-source framework for multimodal synthetic data, aiming to lower technical barriers for LLM training. It features a configuration-driven pipeline with visual interface and modular architecture.
Meta Tuna-2: Encoder-Free Multimodal Model Beats VAE-Based Rivals
Meta released Tuna-2, an encoder-free multimodal model that understands and generates images from raw pixels. It beats encoder-based models on fine-grained perception benchmarks, challenging the dominant VAE/vision encoder paradigm.
NVIDIA Nemotron 3 Nano Omni: Open Multimodal Model Unifies Video, Audio, Image, Text
NVIDIA announced Nemotron 3 Nano Omni, an open multimodal model that processes video, audio, images, and text in a unified architecture, expanding accessibility for multimodal AI research.
Tencent Open-Sources HY-World 2.0 Multimodal 3D World Model
Tencent's Hunyuan AI lab has open-sourced HY-World 2.0, a multimodal world model capable of generating, reconstructing, and simulating interactive 3D scenes. This release provides a significant, freely available tool for 3D content creation and embodied AI research.
Indexing Multimodal LLMs for Large-Scale Image Retrieval
A new arXiv paper proposes using Multimodal LLMs (MLLMs) for instance-level image-to-image retrieval. By prompting models with paired images and converting next-token probabilities into scores, the method enables training-free re-ranking. It shows superior robustness to clutter and occlusion compared to specialized models, though struggles with severe appearance changes.
MedGemma 1.5 Technical Report Released, Details Multimodal Medical AI
Google DeepMind has published the technical report for MedGemma 1.5, detailing the architecture and capabilities of its open-source, multimodal medical AI model. This follows the initial Med-PaLM 2 release and represents a significant step in making specialized medical AI more accessible.
JBM-Diff: A New Graph Diffusion Model for Denoising Multimodal Recommendations
A new arXiv paper introduces JBM-Diff, a conditional graph diffusion model designed to clean 'noise' from multimodal item features (like images/text) and user behavior data (like accidental clicks) in recommendation systems. It aims to improve ranking accuracy by ensuring only preference-relevant signals are used.
Building a Multimodal Product Similarity Engine for Fashion Retail
The source presents a practical guide to constructing a product similarity engine for fashion retail. It focuses on using multimodal embeddings from text and images to find similar items, a core capability for recommendations and search.
Google Releases Gemma 4 Family Under Apache 2.0, Featuring 2B to 31B Models with MoE and Multimodal Capabilities
Google has released the Gemma 4 family of open-weight models, derived from Gemini 3 technology. The four models, ranging from 2B to 31B parameters and including a Mixture-of-Experts variant, are available under a permissive Apache 2.0 license and feature multimodal processing.
Uni-SafeBench Study: Unified Multimodal Models Show 30-50% Higher Safety Failure Rates Than Specialized Counterparts
Researchers introduced Uni-SafeBench, a benchmark showing that Unified Multimodal Large Models (UMLMs) suffer a significant safety degradation compared to specialized models, with open-source versions showing the highest failure rates.
mmAnomaly: New Multi-Modal Framework Uses Conditional Latent Diffusion to Achieve 94% F1 Score for mmWave Anomaly Detection
Researchers introduced mmAnomaly, a multi-modal anomaly detection system that uses a conditional latent diffusion model to synthesize expected mmWave spectra from visual context, achieving up to a 94% F1 score for detecting concealed weapons and through-wall anomalies.
Stop Shipping Demo-Perfect Multimodal Systems: A Call for Production-Ready AI
A technical article argues that flashy, demo-perfect multimodal AI systems fail in production. It advocates for 'failure slicing'—rigorously testing edge cases—to build robust pipelines that survive real-world use.
MMM4Rec: A New Multi-Modal Mamba Model for Faster, More Transferable Sequential Recommendations
Researchers propose MMM4Rec, a novel sequential recommendation framework using State Space Duality for efficient multi-modal learning. It claims 10x faster fine-tuning convergence and improved accuracy by dynamically prioritizing key visual/textual information over user interaction sequences.
Training-Free Polynomial Graph Filtering: A New Paradigm for Ultra-Fast Multimodal Recommendation
Researchers propose a training-free graph filtering method for multimodal recommendation that fuses text, image, and interaction data without neural network training. It achieves up to 22.25% higher accuracy and runs in under 10 seconds, dramatically reducing computational overhead.
Gastric-X: New 1.7K-Case Multimodal Benchmark Challenges VLMs on Realistic Gastric Cancer Diagnosis Workflow
Researchers introduce Gastric-X, a comprehensive multimodal benchmark with 1.7K gastric cancer cases including CT scans, endoscopy, lab data, and expert notes. It evaluates VLMs on five clinical tasks to test if they can correlate biochemical signals with tumor features like physicians do.
Multimodal RAG System for Chest X-Ray Reports Achieves 0.95 Recall@5, Reduces Hallucinations with Citation Constraints
Researchers developed a multimodal retrieval-augmented generation system for drafting radiology impressions that fuses image and text embeddings. The system achieves Recall@5 above 0.95 on clinically relevant findings and enforces citation coverage to prevent hallucinations.
VLM2Rec: A New Framework to Fix 'Modality Collapse' in Multimodal Recommendation Systems
New research proposes VLM2Rec, a method to prevent Vision-Language Models from ignoring one data type (like images or text) when fine-tuned for recommendations. This solves a key technical hurdle for building more accurate, robust sequential recommenders that truly understand multimodal products.
AMES: A Scalable, Backend-Agnostic Architecture for Multimodal Enterprise Search
Researchers propose AMES, a unified multimodal retrieval system using late interaction. It enables cross-modal search (text, image, video) within existing enterprise engines like Solr without major redesign, balancing speed and accuracy.
Goal-Driven Data Optimization: Training Multimodal AI with 95% Less Data
Researchers introduce GDO, a framework that optimizes multimodal instruction tuning by selecting high-utility training samples. It achieves faster convergence and higher accuracy using 5-7% of the data typically required. This addresses compute inefficiency in training vision-language models.