arXiv: Utility Under Attack: Agent Memory Poisoning and the Limits of Content Screening and Provenance Ranking

AI Analysis

A new research paper, Utility Under Attack: Agent Memory Poisoning and the Limits of Content Screening and Provenance Ranking, has been published on arXiv. The paper demonstrates a novel attack vector against AI agents that use long-term memory or retrieval-augmented generation. Specifically, it shows that malicious content embedded in documents, web pages, or user interactions can poison an agent's memory store, causing it to produce incorrect or harmful outputs even when standard content screening and provenance ranking controls are in place. This is not a patch or regulatory update, but a peer-reviewed style technical disclosure that highlights a fundamental limitation in current defensive architectures.

The findings affect any organization deploying AI agents that ingest external data or maintain persistent memory, including customer support chatbots, research assistants, financial analysis tools, and healthcare decision-support systems. Sectors with strict data integrity and audit requirements, such as finance, legal, healthcare, and public administration, are particularly exposed, as poisoned memory could lead to regulatory violations or erroneous decisions that are difficult to trace.

Compliance teams should treat this as a risk signal, not a compliance failure. Immediately review any AI system that uses memory or retrieval functions to identify whether it can be fed untrusted content. Update your AI risk register to include memory poisoning as a distinct threat, and require engineering teams to implement stricter input validation, memory isolation, and periodic memory audits. Until mitigations are proven, consider limiting agent access to external sources or adding human-in-the-loop review for high-stakes outputs. Monitor the paper's follow-up work for practical defenses.