batch processing
30 articles about batch processing in AI news
New arXiv Paper Proposes LLM-Generated 'Reference Documents' to Speed Up
A new arXiv preprint introduces a method for efficient LLM-based reranking. It uses LLMs to generate 'reference documents' that help dynamically truncate long ranked lists and optimize batch processing, achieving up to 66% speedup on TREC benchmarks.
mlx-vlm v0.5.0 Adds Continuous Batching, Distributed Inference for Apple Silicon
mlx-vlm v0.5.0 adds continuous batching, speculative decoding, and distributed inference for Apple Silicon. The release supports Qwen3.5, Kimi K2.5, Gemma 4 video, and new models with 21 contributors.
Dflash with Continuous Batch Inference Teased for Draft Models
A developer teased the upcoming release of 'Dflash' with continuous batch inference, targeting current text-only draft models used in speculative execution to speed up LLM inference.
MLX-VLM Adds Continuous Batching, OpenAI API, and Vision Cache for Apple Silicon
The next release of MLX-VLM will introduce continuous batching, an OpenAI-compatible API, and vision feature caching for multimodal models running locally on Apple Silicon. These optimizations promise up to 228x speedups on cache hits for models like Gemma4.
MLX-LM v0.9.0 Adds Better Batching, Supports Gemma 4 on Apple Silicon
Apple's MLX-LM framework released version 0.9.0 with enhanced server batching and support for Google's Gemma 4 model, improving local LLM inference efficiency on Apple Silicon. This update addresses a key performance bottleneck for developers running models locally on Mac hardware.
Stirling-PDF Hits 77K GitHub Stars as Local AI Document Processing Surges
Stirling-PDF, a fully local, open-source PDF toolkit, has surpassed 77,100 GitHub stars and 25M+ downloads. Its growth highlights a major shift toward privacy-first, self-hosted document AI, challenging paid cloud services like Adobe Acrobat.
Pinterest's Request-Level Deduplication
Pinterest's engineering blog details 'request-level deduplication,' a critical efficiency technique for modern recommendation systems. By eliminating redundant processing of massive user sequences, they achieve 10-50x storage compression and significant training speedups, while solving novel training challenges like batch correlation.
OpenAI Details GPT-Live's Continuous Audio Architecture
OpenAI disclosed GPT-Live's continuous audio architecture via @rohanpaul_ai. The design shifts from turn-based voice processing to streaming, targeting latency, though specifics remain undisclosed.
AWS Personalize cuts recommendation latency to under 10 seconds in new
AWS published a real-time e-commerce personalization blueprint that cuts recommendation latency to under 10 seconds using AWS Personalize, Kinesis, and Lambda. The architecture bridges batch and streaming modes for luxury retailers.
AMD-Cerebras Disaggregated Inference: 5× T/s/W, Prompt vs. Decode Split
AMD and Cerebras launched a disaggregated inference platform splitting prompt processing on Helios from decode on WSE, claiming up to 5× T/s/W.
Pinterest's MIQPS: A Data-Driven Approach to URL Normalization for Content
Pinterest's engineering team details the MIQPS algorithm, which dynamically identifies 'important' vs. 'noise' query parameters per domain by testing if their removal changes a page's visual fingerprint. This solves the costly problem of ingesting and processing duplicate product pages from varied merchant URLs.
How I Built a Production RAG Pipeline for Fintech at 1M+ Daily Transactions
A technical case study from a fintech ML engineer outlines the end-to-end design of a Retrieval-Augmented Generation pipeline built for production at extreme scale, processing over a million daily transactions. It provides a rare, real-world blueprint for building reliable, high-volume AI systems.
Hugging Face OCRs 27,000 arXiv Papers to Markdown with Open 5B Model
Hugging Face CEO Clement Delangue announced the OCR conversion of 27,000 arXiv papers to Markdown using an open 5B-parameter model and 16 parallel jobs on L40S GPUs. This demonstrates a scalable, open-source pipeline for large-scale academic document processing.
Pinterest Details 'Request-Level Deduplication' to Scale Massive
Pinterest's engineering team published a detailed technical breakdown of 'request-level deduplication'—a family of techniques that eliminate redundant processing of user data across thousands of candidate items in their recommendation system. This approach was critical to scaling their Foundation Model by 100x while controlling infrastructure costs.
Insanely Fast Whisper CLI Transcribes 2.5 Hours of Audio in 98 Seconds with Flash Attention 2
A new open-source CLI tool called Insanely Fast Whisper achieves 19x speedup over standard Whisper large-v3, transcribing 150 minutes of audio in 98 seconds using Flash Attention 2 and batching with no quality loss.
SpaceX's Starlink Launches First Orbital Data Center Test with AI Compute Module
SpaceX has launched a prototype data center module to orbit aboard a Starlink mission, testing the viability of orbital computing infrastructure for AI and other workloads. This marks the first physical step toward off-planet data processing.
Groq's LPU Inference Engine Demonstrates 500+ Token/s Performance on Llama 3.1 70B
Groq's Language Processing Unit (LPU) inference engine achieves over 500 tokens/second on Meta's Llama 3.1 70B model, demonstrating significant performance gains for large language model inference.
M4 Max Mac Studio Tops GB10 in Local AI Decode Throughput
M4 Max Mac Studio beats GB10 and Strix Halo in local AI decode throughput but memory bandwidth caps large model performance. Tom's Hardware tested llama.cpp across three platforms.
3 MCP Production Failures This Developer Hit Automating Tax Returns (And
MCP's JSON-RPC can't handle binary uploads, 270-tool servers bloat context, and rate limits fail silently. Use CLI scripts for files, allow-list tools, and prompt-based throttling.
7 AI Agent Cost Optimization Strategies That Cut LLM Bills by Up to 90%
The source outlines seven cost optimization strategies for AI agents, including prompt compression and model routing, that can reduce LLM bills by up to 90%. This matters for retail and luxury brands deploying AI at scale where inference costs can become prohibitive.
Drive After Effects from Claude Code: Generate Production Videos via MCP
aftr is an open-source MCP server that lets Claude Code drive After Effects programmatically via JSON commands over WebSocket, enabling automated video rendering without manual work.
PadCaptioner: 3B video caption model beats 7B rivals with parallel decoding
PadCaptioner, a 3B model, beats 7B rivals in dense video captioning via lossless parallel autoregressive decoding, challenging scaling orthodoxy.
Building a Multimodal Vector Search Platform for Product Catalogs
Insider Engineering shares practical lessons from building a multimodal vector search platform for product catalogs, covering multitenancy, GPU economics, and infrastructure surprises. The post provides actionable insights for retail AI teams considering similar systems.
Ollama-OCR Turns Scanned Docs Into Markdown, No Cloud Needed
Ollama-OCR extracts text from scanned docs locally using Ollama vision models. 2.3k stars, no cloud APIs needed.
Instacart Uses PyFixest to Solve High-Cardinality Fixed Effects in
Instacart's tech blog details how PyFixest overcomes O(k³) complexity in high-cardinality fixed-effect regressions for marketplace experiments. This enables scalable treatment effect estimation across 1,000+ geographic regions, directly applicable to retail logistics and delivery optimization.
We Cut Embedding Storage Costs by ~90% — Replacing S3 with PostgreSQL
A team cut embedding storage costs by ~90% by migrating from S3 to PostgreSQL with pgvector, enabling efficient vector search and on-demand retrieval for RAG and recommender systems, with no performance loss.
AMD's Lemonade v10.8 Adds MCP Support, Letting Claude Desktop and Cursor Route Tasks to Local AMD GPUs
AMD-backed Lemonade v10.8, released June 17, now exposes a Model Context Protocol server, letting Claude Desktop, Cursor, and GitHub Copilot route inference tasks to local AMD Ryzen AI NPUs, Radeon GPUs, or plain CPUs — no cloud API required. The update also adds Moonshine speech-to-text, expanded R
MiniMax M3: Sparse Attention, 1M Context, Multimodal via Together
MiniMax M3 uses sparse attention for 1M context and multimodality, with Together AI serving fast inference.
DeepSeek v4 Pricing Cuts 75%: $0.43/M Tokens In
DeepSeek v4 API pricing permanently cut 75% to $0.43/M input, $0.87/M output, enabled by 27% compute and 10% cache vs v3.2.
mlx-audio v0.4.3 Ships 6 New TTS Models, Slimmer Deps
mlx-audio v0.4.3 adds 6 TTS models, server concurrency, and slims dependencies, targeting Apple Silicon developers.