reliability
30 articles about reliability in AI news
AgingBench: AI Agents Lose Reliability Over Time & Memory Fails
UT Austin paper finds AI agents degrade over time via memory errors. Proposes AgingBench to measure reliability decay across sessions.
Building PharmaRAG: A Case Study in Proactive Reliability for RAG Systems
A developer details the architecture of PharmaRAG, a system for querying drug labels, which prioritizes a 'reliability layer' to detect unanswerable questions before any LLM generation. This approach directly tackles the critical problem of AI hallucination in high-stakes domains.
Anthropic Survey of 80,508 Users Reveals AI's Dual Perception: Hope for Work & Growth, Fear of Unreliability & Job Loss
Anthropic's global study of 80,508 users finds people simultaneously hold hope and fear about AI. Top hopes center on work improvement and personal growth, while top concerns are unreliability, job loss, and reduced autonomy.
AI Agents Cross the Reliability Threshold: Karpathy Declares Programming Fundamentally Transformed
Former OpenAI researcher Andrej Karpathy declares programming has become "unrecognizable" as AI agents now reliably complete complex tasks in minutes rather than days. This fundamental shift occurred in late 2026 when agents achieved unprecedented reliability through improved model quality and task persistence.
Beyond Accuracy: Researchers Propose New Framework for Measuring AI Agent Reliability
A new research paper introduces 12 metrics to evaluate AI agent reliability across four dimensions: consistency, robustness, predictability, and safety. The study reveals that despite improving accuracy scores, today's agents remain fundamentally unreliable in practice.
The Quantization Paradox: How Compressing Multimodal AI Impacts Reliability
New research reveals that compressing multimodal AI models through quantization significantly reduces their reliability, making them more likely to produce confidently wrong answers. The study identifies methods to mitigate these effects while maintaining efficiency gains.
Ethan Mollick's 'AI Weirdness Axiom': Why Treating AI Like Standard IT Products Reduces Reliability
Wharton professor Ethan Mollick argues that AI's inherent 'weirdness' must be embraced, not minimized. Attempting to implement AI like conventional software leads to less useful and less reliable systems.
CARE Framework Exposes Critical Flaw in AI Evaluation, Offers New Path to Reliability
Researchers have identified a fundamental flaw in how AI models are evaluated, showing that current aggregation methods amplify systematic errors. Their new CARE framework explicitly models hidden confounding factors to separate true quality from bias, improving evaluation accuracy by up to 26.8%.
ResearchGym Exposes AI's 'Capability-Reliability Gap' in Scientific Discovery
A new benchmark called ResearchGym reveals that while frontier AI agents can occasionally achieve state-of-the-art scientific results, they fail to do so reliably. In controlled evaluations, agents completed only 26.5% of research sub-tasks on average, highlighting critical limitations in autonomous scientific discovery.
MCP Proxy Testing Exposes Silent Client-Side Failures
MCP proxy logging reveals silent client-side failures. Claude Code's 45/45 success rate vs Gemini's 42/45 shows reliability differences. Check tool-call gaps and serverInfo metadata to avoid misattributing failures.
Claude Code Digest — Aug 13–Aug 16
Claude Code is no longer just a prompt box: the week’s strongest signal is that reliability now comes from hard gates, audit trails, and orchestration layers—not smarter prompts.
Meta Paper: LLM Judge Accuracy vs. Golden Data Is Hollow
Meta paper argues LLM judge accuracy vs golden data is insufficient, pushing for reliability metrics like consistency and calibration. Omar Sanseviero flagged it on X.
Claude Code's Architecture Explained
Master Claude Code's 6-layer architecture—harness, context window, subagents, MCP, skills, hooks, CLAUDE.md—to optimize token usage, parallelism, and agent reliability. First sentence: Claude Code's 6 layers (harness, context window, subagents, MCP, skills, hooks, CLAUDE.md) determine your setup's success.
Agents Signal via Filenames, Base64 Attachments
Simon Willison shared agents communicating via filenames with base64 attachments and zz prefixes. This turns the filesystem into a deterministic queue, but has payload and reliability limits.
OrcaMan Open-Sources BoundaryBench for Enterprise RL
OrcaMan open-sourced BoundaryBench, a benchmark for enterprise RL boundary detection. The tool targets reliability gaps in production AI.
Google ADK Go 2.0 Adds Graph Engine, Human-in-Loop for Agents
Google released ADK Go 2.0 on July 2, 2026, adding a graph-based workflow engine and human-in-the-loop for multi-agent orchestration, targeting production reliability.
Caliper: Run Your Claude Code Skills k Times and Get a pass@k Score That
Caliper gives Claude Code users a pass@k reliability score for skills, with a baseline delta showing if the skill beats the base agent. Install via pipx or npx.
Cline v4.0.0 Ships Plugin Marketplace
Cline v4.0.0 introduces a plugin marketplace, queued prompts, and an SDK rewrite. Claude Code users get new extensibility and reliability features.
OpenAI Acquires Cloud Startup Ona to Power Agent Infrastructure
OpenAI acquired cloud startup Ona to support AI agent infrastructure, two days after a $6.6B raise. The deal targets enterprise reliability gaps as OpenAI pivots to B2B.
Stanford, Meta 'Code as Agent Harness' Paper Rethinks AI Agent Design
Stanford and Meta's "Code as Agent Harness" paper proposes code-driven AI agent orchestration, potentially improving reliability over natural language prompts.
Claude Code Quality Drops Post-4.6, Users Report 25% Task Failure Rate
Claude Code quality dropped post-4.6 with ~25% instruction misses. Codex offers 95% reliability but less creativity.
PJM Warns AI Data Center Load Could Break Power Market Assumptions
PJM warns AI data center load could grow 5x to 25 GW by 2035, colliding with queue delays and outdated market rules. Regulators flag reliability and cost risks.
Claude Code Digest — May 11–May 14
Anthropic's agent misalignment fixes cut incidents by 40-60%, redefining AI reliability.
Agentic Commerce: 50% of Online Transactions by 2027, Google Cloud Leads
Agents projected to handle 50% of online transactions by 2027. Payment reliability determines winners in agentic commerce, with Google Cloud leading enterprise rollouts.
Claude Skills: Directive Descriptions Hit 100% Activation in 650-Trial Test
A 650-trial experiment found directive Claude skill descriptions achieve 100% activation vs 37% for passive phrasing. The YAML description field does 90% of the reliability work.
GPT-5.5 Pro Sustains 2-Hour Bug Fixing Sessions
A user reports GPT-5.5 Pro maintains consistent bug-finding performance for 2-hour coding sessions, suggesting improved reliability for long-running tasks.
From Checkout to Trust Layer: How Merchants Can Prepare for Agentic Commerce
The article discusses the evolution of e-commerce from simple checkout processes to a future where AI shopping agents act on behalf of consumers. It argues that success in this 'agentic commerce' era depends on merchants building a robust trust layer with data security, transparency, and reliability at its core.
Your AI Agent Is Only as Good as Its Harness — Here’s What That Means
An article from Towards AI emphasizes that the reliability and safety of an AI agent depend more on its controlling 'harness'—the system of protocols, tools, and observability layers—than on the underlying model. This concept is reportedly worth $2 billion but remains poorly understood by many developers.
Opus 4.7 AI Hallucinates with High Conviction, Developer Reports
A developer reported that Anthropic's Opus 4.7 model repeatedly hallucinated about a test result, insisting the score was unchanged despite evidence. This highlights a critical trust issue where improved benchmarks may not reflect real-world reliability.
From MLOps to AgentOps: A Vision for AI Production in 2026
A forward-looking article argues that by 2026, AI systems will be complex, multi-agent software requiring a new operational discipline called 'AgentOps'. This evolution from MLOps is necessary to manage reliability, safety, and cost at scale.