ConfusedPilot

A Sneak Attack on AI Systems

OCTOBER 16, 2024

ERICA JAYASUNDERA
MINDVIEW AI

Classification | Confidential-External
Introduction
The digital landscape is rapidly evolving, with Artificial Intelligence (AI) playing an increasingly
prominent role. However, this growing reliance on AI introduces new vulnerabilities, as highlighted by
the recent discovery of the "ConfusedPilot" attack. Researchers at the University of Texas at Austin's
Spark Lab, led by Professor Mohit Tiwari, identified this novel cyberattack method targeting Retrieval-
Augmented Generation (RAG) based AI systems.

What are RAG-based AI systems?

RAG systems combine two techniques: retrieval and generation. They retrieve relevant information
from a vast data pool and then utilize that information to generate human-like text, code, or other
outputs. This makes them highly versatile, with applications ranging from Microsoft 365 Copilot's code
completion to chatbots and automated content creation tools.

How does ConfusedPilot work?

The attack exploits the way RAG systems reference data. Here's the breakdown:

1. Planting the Seed: The attacker introduces seemingly innocuous documents containing
carefully crafted strings into the AI system's data pool. This can be achieved through various
means, such as uploading documents to a shared workspace or exploiting vulnerabilities in data
ingestion processes.
2. Triggering the Response: When a user interacts with the AI system by posing a query, the
system retrieves relevant data to formulate its response. If the crafted strings are present in the
retrieved documents, they can manipulate the AI's output.

Classification | Confidential-External
 3. Misinformation and Flawed Decisions: The attacker's strings can trick the AI into
generating misleading or incorrect responses. This could lead to critical consequences,
such as:
o Financial Losses: If an AI system used for financial analysis is manipulated, it could
provide inaccurate recommendations leading to bad investments or fraudulent
transactions.
o Operational Disruptions: A compromised AI system assisting with logistics or supply
chain management could disrupt entire operations.
o Reputational Damage: Fabricated information generated by a compromised AI used
for customer service could damage an organization's reputation.

The Attack Flow

An adversary attempting ConfusedPilot attack would likely follow these steps:

1. Data Environment Poisoning: An attacker introduces an innocuous document that contains

specifically crafted strings into the target’s environment. This could be achieved by any identity
with access to save documents or data to an environment indexed by the AI copilot.
2. Document used in Query Response: When a user makes a relevant query, the RAG system
retrieves the document containing these strings.
3. AI Copilot interprets strings as user Instructions: The document contains strings that could act
as instructions to the AI system, including:
1. Content Suppression: The malicious instructions cause the AI to disregard other
relevant, legitimate content.
2. Misinformation Generation: The AI generates a response using only the corrupted
information.

Classification | Confidential-External
 3. False Attribution: The response may be falsely attributed to legitimate sources,
increasing its perceived credibility.
4. AI Copilot retains instructions: Even if the malicious document is later removed, the corrupted
information may persist in the system’s responses for a period of time.

A few illustrative examples:

Enterprise knowledge management systems:

If an attacker introduces a malicious document into the company's knowledge base copilot (perhaps
through social engineering or deliberate sabotage), they could manipulate AI-generated responses
across the organisation to spread misinformation. This could potentially influence critical business
decisions.

AI-assisted decision support systems:

In environments where AI systems are used to analyse data and provide recommendations for strategic
decisions, an attacker could inject false information that persists even after the original malicious
content is removed. This could lead to a series of poor decisions over time due to reliance on AI, with
the source of the problem remaining elusive without thorough forensic investigation.

Customer-facing AI services:

For organisations providing AI-powered services to customers, Confused Pilot becomes even more
dangerous. An attacker could potentially inject malicious data that affects the AI's responses to multiple
customers, leading to widespread misinformation, loss of trust, and potential legal liabilities.

End Users Relying on AI-generated Content:

Whether it's employees or executives, any end user using AI assistants for daily tasks or synthesising
AI-generated insights could make critically flawed decisions and unknowingly spread misinformation
throughout the organisation.

Why is ConfusedPilot concerning?

Several factors elevate the concern regarding ConfusedPilot:

• Wide Attack Surface: RAG-based AI systems are increasingly prevalent, making a large
number of organizations potentially vulnerable.
• Low Barrier to Entry: Launching a ConfusedPilot attack requires minimal technical expertise
compared to other cyberattacks.
• Persistence: Even after removing the malicious seed document, the crafted strings might linger
in cached data, making the attack persistent.
• Evasion Tactics: The attack can potentially bypass existing AI security measures designed to
detect anomalies in data or generated responses.

Classification | Confidential-External
Defending Against ConfusedPilot:

While ConfusedPilot presents a challenge, researchers and security professionals are actively
developing mitigation strategies. Here are some potential solutions:

• Data Governance: Implementing stricter controls on data entry and access can minimize the
chances of malicious content entering the system.
• Data Provenance: Tracking the source and history of data used by the AI system can help
identify suspicious or manipulated information.
• Adversarial Training: Training the AI system with examples of manipulated data can help it
recognize and resist manipulated responses.
• Continuous Monitoring: Regularly monitoring AI outputs for inconsistencies and unexpected
trends can flag potential attacks.
• User Awareness: Educating users about the potential for AI manipulation can help them
critically evaluate AI-generated responses.

The Evolving Landscape of AI Security

ConfusedPilot serves as a wake-up call for the AI development and security communities. While AI
offers immense potential, securing these systems is crucial to ensure their reliability and prevent them
from becoming a liability. Ongoing research into attack detection, data integrity, and robust AI
architectures will be essential in building a future where AI can be trusted.

Further Considerations:

This analysis provides a foundational understanding of ConfusedPilot. Here are some additional points
for exploration:

• The ethical implications of AI manipulation: ConfusedPilot highlights the potential for

malicious actors to misuse AI for disinformation campaigns or social engineering attacks.
• Regulations and standards for AI security: There's a growing need for regulations and
standards that ensure the responsible development and deployment of AI systems, with
security being a core consideration.
• The role of user trust: As reliance on AI grows, building user trust is essential. This can be
achieved through transparency about how AI systems work and demonstrably robust security
measures.

The Importance of a Layered Security Approach for RAG Systems

Retrieval-Augmented Generation (RAG) systems, which combine information retrieval and text
generation, have become increasingly prevalent in various industries. However, their growing reliance
on external data sources makes them vulnerable to cyberattacks. A layered security approach is crucial
to protect RAG systems from these threats and ensure their integrity and reliability.

Understanding the Risks

RAG systems are susceptible to several security risks:

Classification | Confidential-External
 • Data Poisoning: Attackers can introduce malicious data into the system's knowledge base,
influencing the AI's responses and potentially leading to misinformation or harmful outputs.
• Model Extraction: Adversaries can extract the underlying model parameters, compromising
the system's intellectual property and potentially creating malicious copies.
• Supply Chain Attacks: Vulnerabilities in the underlying software or hardware components
used by RAG systems can be exploited to gain unauthorized access or control.

The Benefits of a Layered Security Approach

A layered security approach involves implementing multiple security controls at different levels of the
system to create a robust defense. This approach offers several benefits:

• Enhanced Resilience: By combining various security measures, a layered approach makes it

more difficult for attackers to breach the system's defenses.
• Risk Mitigation: Each layer of security can address specific vulnerabilities, reducing the overall
risk of a successful attack.
• Compliance Adherence: Many industries have strict data privacy and security regulations. A
layered security approach can help organizations comply with these requirements.
• Proactive Defense: A layered approach allows for continuous monitoring and adaptation to
emerging threats, ensuring that the system remains protected.

Key Components of a Layered Security Approach

A comprehensive layered security approach for RAG systems should include the following
components:

1. Data Security:
o Input Validation: Implement input validation to filter out malicious or unexpected
data.
o Data Encryption: Encrypt sensitive data both at rest and in transit to protect it from
unauthorized access.
o Data Masking: Mask sensitive data to prevent unauthorized disclosure.
2. Model Security:
o Model Obfuscation: Use techniques like quantization, pruning, or knowledge
distillation to make the model more difficult to reverse engineer.
o Model Monitoring: Continuously monitor the model's behavior for anomalies that
may indicate a compromise.
3. Infrastructure Security:
o Network Security: Implement firewalls, intrusion detection systems, and other
network security measures to protect the system from external threats.
o Access Controls: Restrict access to the system to authorized users only.
o Patch Management: Keep all software and hardware components up-to-date with
the latest security patches.
4. AI Security:
o Adversarial Training: Train the model to be resilient to adversarial attacks, which
aim to manipulate the system's outputs.

Classification | Confidential-External
 o Explainability: Increase the transparency of the model's decision-making process to
identify and mitigate potential biases or vulnerabilities.
5. Incident Response:
o Incident Response Plan: Develop a comprehensive incident response plan to
address security breaches effectively.
o Regular Testing: Conduct regular security testing and penetration testing to identify
vulnerabilities and improve the system's resilience.

By implementing a layered security approach, organizations can significantly enhance the protection of
their RAG systems and mitigate the risks associated with their use.

Reference :

1. https://securityboulevard.com/2024/10/confusedpilot-ut-austin-symmetry-systems-uncover-
novel-attack-on-rag-based-ai-systems/
2. https://www.infosecurity-magazine.com/news/confusedpilot-attack-targets-ai/

Classification | Confidential-External