Data Poisoning and Generative AI: What do companies need to know?

Data poisoning refers to the intentional corruption of data used by AI systems to manipulate their behaviour. Attackers inject false information, modify existing data, or plant malicious instructions that AI systems follow. Their aim can be to introduce biases, create backdoors or even steal data.

Between August 2024 and August 2025, security researchers demonstrated successful data poisoning attacks against Microsoft 365 Copilot, ChatGPT Connectors, and multiple RAG-based enterprise AI systems. These attacks achieved data exfiltration, decision manipulation, and persistent misinformation injection, often requiring nothing more than uploading a malicious document to a shared drive.

This article examines real-world data poisoning attacks that have targeted enterprise AI systems over the past year, explains how these attacks work, and provides practical steps you can take to reduce your organisation's exposure to this emerging threat.

Two types of data poisoning

These documented attacks exploit two distinct vulnerabilities in AI systems:

Training-time poisoning affects the data used to build the model itself. Subtle modifications to as little as 0.1% of a massive dataset can plant behavioural ‘triggers’ which can later be exploited. The UK AI Safety Institute's October 2025 study, conducted with Anthropic and the Alan Turing Institute, represents the largest empirical study to date. It found that this type of data poisoning is much more common and easier than previously assumed. Any company training or fine-tuning models on web or public data can unintentionally use content that plants these triggers.

RAG and prompt injection poisoning attacks production systems by manipulating the documents, emails, or knowledge bases that AI assistants retrieve to answer questions. This represents the more immediate threat for most GenAI adopters.

Whilst you cannot control your vendor's training pipeline, you do feed data into the AI workspace systems you use (think of SharePoint, Google Drive, Confluence, Slack, and email). An attacker who can introduce malicious content into these sources can manipulate AI outputs without accessing the underlying model or getting the user to take an action (as often occurs in phishing attacks).

This type of data poisoning an result in degraded performance, biased results, data being exposed to unauthorised parties, and answers that confidently cite non-existent information. These often emerge gradually and may be dismissed, until the business impact becomes large enough for the issue to be investigated.

Documented attacks on enterprise AI systems

AgentFlayer: The poisoned Google Drive file (August 2025)

Researchers demonstrated that a single hidden 300-word payload inside a shared Google Drive file could silently compromise ChatGPT via Connectors and exfiltrate secrets with no further user action required. The AI assistant, granted access to company files to enhance productivity, became a vector for data theft.

Business impact: Any organisation connecting AI assistants to SaaS data stores (Drive, GitHub, calendars, internal wikis) can be compromised via a single malicious file entering shared spaces. OpenAI shipped mitigations following disclosure, but the underlying architectural vulnerability persists: AI agents with broad access permissions represent high-value attack surfaces.

Microsoft 365 Copilot: ASCII smuggling and unauthorised tool invocation (August 2024)

Multiple research teams demonstrated how malicious content in emails or documents could instruct Copilot to auto-invoke tools, retrieve additional emails and SharePoint files, and exfiltrate data through "ASCII smuggling" techniques. In this case, the attacker sends an email that does not need to be opened. Instead,the attack occurs when Copilot processes it in the background, hidden instructions trigger data exfiltration automatically.

Business impact: AI assistants such as Microsoft Copilot reads data from apps such as Email, Teams, OneDrive and SharePoint. Attackers can choose to poison this content and rely on the AI to activate it.  This type of attack does not require user interaction, beyond using an AI assistant that is able to access privileged information.

While Microsoft issued fixes in light of the issue mentioned, similar patterns can affect other enterprise AI tools with comparable architectures. For example, any AI assistant that can read your emails, access your file storage, and take actions on your behalf (such as Slack AI, Google Workspace AI, or custom RAG systems connected to your internal databases) faces the same fundamental risk. Ultimately, any "AI Agent" that has access to sensitive information has the potential to be compromised in this manner.

PoisonedRAG: Small injections with significant influence (May 2025)

Researchers at USENIX Security '25 demonstrated that inserting a small number of malicious texts into a RAG knowledge base can reliably influence outputs through ‘knowledge corruption’ without triggering system failures or obvious alerts.

Business impact: If your AI relies on internal corpora (wikis, support tickets, PDFs), quietly corrupted entries can bias or mislead high-stakes decisions. For example:

• A pricing model that systematically underquotes after someone plants a document suggesting lower margins represent competitive best practice
• A contract review tool that overlooks liability clauses because a poisoned legal memo taught it those terms are standard boilerplate

The danger is not a catastrophic failure, but confidently incorrect answers that compound over time and influence strategic decisions.

Nightshade: Poisoning image models at scale (January 2024 onwards)

University of Chicago's Nightshade tool enables creators to subtly alter images so models trained on them learn incorrect associations. For example, teaching a model that ‘handbag’ should be classified as a ‘toaster’.

Business impact: Organisations training or retraining vision models on scraped or user-submitted content, risk degraded or mislabelled outputs and consequent reputational damage. Examples include:

• A retail company's visual search returning incorrect products
• A medical imaging tool that learned incorrect feature associations
• Quality degradation that proves difficult to diagnose because the training data appears legitimate

Practical mitigation strategies

Whilst you cannot prevent all poisoning attacks, you can significantly reduce your risk and catch problems earlier. We recommend focusing on the following practical steps:

Control who can add content to your AI systems

• Think of the documents and data your AI reads as potential entry points for attackers. Where possible, verify the source of documents before they're indexed.
• Limit who can upload or edit files in the folders your AI systems access.
• Regularly check who has access to add files into any directory that your AI can access (Email, Sharepoint, Teams)
• Ensure that appropriate filters are set for your Email provider

Limit what your AI assistants can do automatically

• Turn off automatic actions by default. Ensure that human approval is required for important tasks
• Set up controls to prevent your AI from making unauthorised connections to external websites (this stops attackers from using your AI to send sensitive data out of your organisation)
• Block your AI from automatically loading images or following links in documents it retrieves, as these can be used to steal information
• Regularly review and restrict the permissions you've granted to AI assistants

The AgentFlayer case demonstrates why this matters. If your AI assistant can automatically perform actions or connect to the internet without oversight, attackers can exploit these capabilities.

Track what your AI is reading and using

• Log which documents influenced each answer. If your AI starts giving incorrect advice or leaking information, you can quickly identify and remove the problematic source document
• Take caution with external documents. Store files from outside your organisation in clearly marked folders with tighter access controls, making it easier to spot when something suspicious enters your system
• Review access permissions quarterly. Check who can add or edit files in your AI-indexed folders and remove permissions for people who've left or changed roles.

Embed human review where needed

Require two-person checks on high-stakes AI outputs:

• Pricing decisions
• Contract approvals
• Financial forecasts
• Customer-facing communications

Establish clear escalation paths (these are scenarios that determine what to do in case of an issue). The goal here is not to eliminate AI use, but to ensure AI mistakes are caught and corrected before they impact your business.

It is not possible to eliminate the risk of data poisoning entirely. When using a generative AI system or an AI agent, it is not possible to control how your AI vendor trains their models. Furthermore, no system is completely secure.  However, the above steps will enable you to safeguard against data poising attacks and significantly reduce the damage when attacks happen. Think of this like any other business risk: insurance, backup plans, and good governance.

Our data protection and cyber-risk team are experienced in helping organisations secure their AI deployments and manage emerging technology risks. Please contact our advisers if you would like to find out more.