adversarial attacks
30 articles about adversarial attacks in AI news
Microsoft RAMPART Brings Pytest-Based Safety Testing to AI Agents
Microsoft's RAMPART brings pytest-native safety testing to AI agents, covering adversarial attacks and benign failures, addressing a critical gap in agent development.
New Research Proposes FilterRAG and ML-FilterRAG to Defend Against Knowledge Poisoning Attacks in RAG Systems
Researchers propose two novel defense methods, FilterRAG and ML-FilterRAG, to mitigate 'PoisonedRAG' attacks where adversaries inject malicious texts into a knowledge source to manipulate an LLM's output. The defenses identify and filter adversarial content, maintaining performance close to clean RAG systems.
GPT-Red: OpenAI's LLM Super-Hacker Finds 84% of Attacks, Humans 13%
OpenAI's GPT-Red LLM finds 84% of attacks vs 13% for humans, hardening GPT-5.6 Sol. Automated red-teaming shifts safety paradigm.
AI Teaches Itself to See: Adversarial Self-Play Forges Unbreakable Vision Models
Researchers propose AOT, a revolutionary self-play framework where AI models generate their own adversarial training data through competitive image manipulation. This approach overcomes the limitations of finite datasets to create multimodal models with unprecedented perceptual robustness.
New Training Method Promises to Fortify AI Against Subtle Linguistic Attacks
Researchers propose Distributional Adversarial Training (DAT), a novel approach using diffusion models to generate diverse training samples, addressing LLMs' persistent vulnerability to simple linguistic manipulations like tense changes and translations.
AI Role-Playing Agents Learn to Defend Themselves Through Adversarial Evolution
Researchers have developed a novel framework that enables AI role-playing agents to autonomously strengthen their defenses against jailbreak attacks while maintaining character fidelity. The dual-cycle system creates progressively stronger attacks and distills defensive knowledge without requiring model retraining.
Frontier AI Models Resist Prompt Injection Attacks in Grading, New Study Finds
A new study finds that while hidden AI prompts can successfully bias older and smaller LLMs used for grading, most frontier models (GPT-4, Claude 3) are resistant. This has critical implications for the integrity of AI-assisted academic and professional evaluations.
We Ran Real Attacks Against Our RAG Pipeline. Here’s What Actually Stopped Them.
A practical security analysis of RAG pipelines tested three specific attack vectors and identified the most effective defenses. This is critical for any enterprise using RAG for customer-facing or internal knowledge systems.
POTEMKIN Framework Exposes Critical Trust Gap in Agentic AI Tools
A new paper formalizes Adversarial Environmental Injection (AEI), a threat model where compromised tools deceive AI agents. The POTEMKIN testing harness found agents are evaluated for performance, not skepticism, creating a critical trust gap.
FiMMIA Paper Exposes Broken MIA Benchmarks, Challenges Hessian Theory
A paper accepted at EACL 2026 shows membership inference attack (MIA) benchmarks suffer from data leakage, allowing model-free classifiers to achieve up to 99.9% AUC. The work also challenges the theoretical foundation of perturbation-based attacks, finding Hessian-based explanations fail empirically.
Google DeepMind: Web Environment, Not Model Weights, Is Key AI Agent Attack Surface
Google DeepMind researchers present a systematic framework showing that the web environment itself—not just the model—is a primary attack surface for AI agents. In benchmarks, hidden prompt injections hijacked agents in up to 86% of scenarios, with memory poisoning attacks exceeding 80% success.
Claude Haiku 4.5 Costs $10.21 to Breach, 10x Harder Than Rivals in ACE Benchmark
Fabraix's ACE benchmark measures the dollar cost to break AI agents. Claude Haiku 4.5 required a mean adversarial cost of $10.21, making it 10x more resistant than the next best model, GPT-5.4 Nano ($1.15).
How Large Language Models 'Counter Poisoning': A Self-Purification Battle Involving RAG
New research explores how LLMs can defend against data poisoning attacks through self-purification mechanisms integrated with Retrieval-Augmented Generation (RAG). This addresses critical security vulnerabilities in enterprise AI systems.
REPO: The New Frontier in AI Safety That Actually Removes Toxic Knowledge from LLMs
Researchers have developed REPO, a novel method that detoxifies large language models by erasing harmful representations at the neural level. Unlike previous approaches that merely suppress toxic outputs, REPO fundamentally alters how models encode dangerous information, achieving unprecedented robustness against sophisticated attacks.
AI-Powered Espionage: How Hackers Weaponized Claude to Breach Mexican Government Systems
A hacker used Anthropic's Claude AI chatbot to orchestrate sophisticated cyberattacks against Mexican government agencies, stealing 150GB of sensitive tax and voter data. The incident reveals how advanced AI tools are being weaponized for state-level espionage with minimal technical expertise required.
MAIL Network: A Breakthrough in Efficient and Robust Multimodal Medical AI
Researchers have developed MAIL and Robust-MAIL networks that overcome key limitations in multimodal medical imaging analysis, achieving up to 9.34% performance gains while reducing computational costs by 78.3% and enhancing adversarial robustness.
Mythos AI Agent Called 'Unprecedented Cyberweapon' by Wharton Prof
Ethan Mollick highlighted the Mythos AI agent, stating its capabilities could constitute an 'unprecedented cyberweapon' in adversarial hands. He notes a narrow window where only a few companies have this level of capability.
Decoy Font Tricks AI Vision Models With Dual-Layer Glyphs
Mixfont's Decoy Font hides text from AI vision models by layering two characters into one glyph, exploiting a tokenization blind spot in ChatGPT and Gemini.
AI Breached Real Production Systems, Not Just Sandboxes
AI breached real production systems, not just test environments. First documented case of operational security breach.
MCP Confused Deputy: Protocol Design Lacks Provenance, Enables Injection
MCP has a confused deputy vulnerability: tool results lack provenance, allowing injection. The official fetch server feeds attacker-controlled Markdown to context.
Gemini 3.5 Flash Scores 78.4 on OSWorld, Matching GPT-5.5
Google integrated Computer Use into Gemini 3.5 Flash, scoring 78.4 on OSWorld — matching GPT-5.5 and undercutting on cost.
RIFT-Bench Tests 45 Agentic Systems With Dynamic Red-Teaming
RIFT-Bench evaluates 45 agentic AI systems via a graph-driven two-phase pipeline, enabling unified security comparison across heterogeneous architectures.
Persuasion Techniques Boost LLM Compliance from 35% to 51% in PNAS Study
PNAS study finds persuasion techniques boost LLM compliance from 35% to 51%, with newer models resisting more.
Anthropic Shows Anyone With a Laptop Can Poison Any Major AI Model
Anthropic proved anyone with a laptop can poison any major AI model, challenging assumptions about model security. The attack works on models from OpenAI, Google, and others, but details are scarce.
Embedding distance predicts VLM typographic attack success (r=-0.93)
A new study shows that embedding distance between image text and harmful prompt strongly predicts attack success rate (r=-0.71 to -0.93). The researchers introduce CWA-SSA optimization to recover readability and bypass safety alignment without model access.
Decepticon Open-Sources Autonomous AI Red Team for Full Kill Chain
Decepticon, a new open-source multi-agent AI system, autonomously executes the entire cyber kill chain for red teaming, from reconnaissance to exfiltration, enabling continuous security testing.
SharpAP: New Attack Method Makes Recommender System Poisoning More
Researchers propose SharpAP, a poisoning attack that uses sharpness-aware minimization to generate fake user profiles that transfer better between different recommender system models, posing a more realistic threat.
DNL Method Finds 2 Bits That Crash ResNet-50, Qwen3-30B
Researchers introduced Deep Neural Lesion (DNL), a method to find critical parameters. Flipping just two sign bits reduced ResNet-50 accuracy by 99.8% and Qwen3-30B reasoning to 0%.
LLMs Can De-Anonymize Users from Public Data, Study Warns
Large Language Models can now piece together a person's identity from their public online trail, rendering pseudonyms ineffective. This raises significant privacy and security concerns for internet users.
Google DeepMind Maps AI Attack Surface, Warns of 'Critical' Vulnerabilities
Google DeepMind researchers published a paper mapping the fundamental attack surface of AI agents, identifying critical vulnerabilities that could lead to persistent compromise and data exfiltration. The work provides a framework for red-teaming and securing autonomous AI systems before widespread deployment.