By now, it’s no surprise that we’ve all heard about prompt injection attacks affecting Large Language Models (LLMs). Since November 2023, prompt injection attacks have been wreaking havoc on many in house built chatbots and homegrown large language models. But what is prompt injection, and why is it so prevalent in today’s GenAI world? Let’s dig in further to learn more about this elusive security vulnerability and attack vector.

Types of Prompt Injection

A Prompt Injection Vulnerability arises when an attacker exploits a large language model (LLM) by feeding it specifically crafted inputs, forcing the model to follow the instructions beyond its intended scope of operations. This manipulation can lead the LLM to inadvertently carry out the attacker’s objectives, ranging from data leakage to remote code execution in the targeted system. The attack can occur directly by “jailbreaking” the system prompt or indirectly by altering external inputs, potentially resulting in data theft, social engineering, and other problems.

• Direct Prompt Injections, also known as “jailbreaking”, occur when an attacker manipulates the system prompt or the initial instructions given to a large language model (LLM). By altering these core instructions, the attacker can influence the behavior and responses of the LLM, causing it to execute unintended actions or disclose sensitive information. This attack often involves modifying the prompt to bypass restrictions, gain unauthorized access, or execute commands that the system would normally prevent.
  
• Indirect Prompt Injections occur when an attacker influences a large language model (LLM) through manipulated external inputs rather than directly altering the system prompt. This can involve embedding malicious content in user-provided data, web pages, or other external sources that the LLM processes. When the LLM interacts with these manipulated inputs, it may be tricked into executing unintended actions, revealing sensitive information, or behaving in a manner that benefits the attacker. This type of attack leverages the LLM’s interaction with external data sources to achieve the attacker’s goals.

What is Jailbreaking?

Jailbreaks aim to bypass all limitations and restrictions imposed on the model, resulting in outputs with much greater variability compared to the standard constrained model. This is either an architectural problem or a training problem made possible because adversarial prompts are challenging to prevent. For instance, the well-known “Do Anything Now,” or DAN prompt enables the GPT instance to disregard all LLM policies designed to prevent the model from producing harmful or inappropriate content.

Source

Do Anything Now (DAN) Jailbreak

The Do Anything Now (DAN) vulnerability can be described as malicious inputs that order the model to ignore its previous directions inherent to the system. A user can override the system by injecting a prompt into a system and giving it new instructions. This is very similar to role-playing scenarios. According to Deval Shah at Lakera.ai (2023), if executed successfully, jailbreaks can override any other instructions, whether explicit (like a system prompt) or implicit (such as the model’s inherent training to avoid offensive content).

A Few Examples of How Attackers Use These Jailbreaks?

1. Exfiltrating the LLM’s system instructions involves extracting the internal guidelines, settings, and rules governing the language model’s operation. This can include parameters on what the model can say, how it processes information, and restrictions designed to prevent it from generating harmful or inappropriate content.

Here’s how it works in simple terms:

• Tricky Questions: Attackers might ask the model-specific questions designed to coax out the system instructions. These questions are carefully crafted to probe the model’s responses and reveal hidden details.
• Analyzing Responses: Attackers can piece together the internal rules and instructions by analyzing how the model responds to these questions.
• Uncovering Secrets: Once enough information is gathered, the attackers can understand the model’s constraints and potentially exploit them to make the model behave in unintended ways.

This process is like solving a puzzle where each piece of information helps to reveal the bigger picture of how the LLM is configured and controlled.

2. Retrieving sensitive information from an LLM (Large Language Model) involves extracting confidential or private data to which the model may have been trained on or has access to. Here’s a breakdown of how this might occur:

• Probing with Questions: Attackers craft specific, targeted questions or prompts to trick the model into revealing sensitive information.
• Pattern Recognition: By recognizing patterns in the model’s responses, attackers can infer or directly obtain pieces of sensitive data.
• Exploitation: With enough probing and analysis, attackers can piece together valuable information such as personal details, confidential business data, or other private content that the model should ideally keep secure.

For example, suppose an LLM has been trained on a personal information dataset. In that case, an attacker might try to extract someone’s phone number or address by asking the model-related questions. This process exploits the model’s ability to generate information based on its training data, potentially leading to unauthorized access to sensitive data.

3. Executing unauthorized actions within LLM applications involves manipulating the language model to perform actions it shouldn’t be allowed to do. Here’s how this might occur:

• Crafting Malicious Prompts: Attackers create specific prompts or commands designed to bypass the model’s restrictions or to exploit vulnerabilities in the application’s integration with the LLM.
• Triggering Actions: These prompts can trick the LLM into performing actions such as sending emails, deleting files, or accessing restricted data.
• Bypassing Safeguards: The goal is to circumvent the safeguards and controls that prevent the LLM from executing harmful or unauthorized tasks.

For example, if an LLM is integrated with an email application, an attacker might craft a prompt that convinces the LLM to send an email without user approval. This could result in unauthorized communications being sent, potentially leading to data breaches or other security issues.

Visual Prompt Injection

As GenAI apps develop into multi-modal systems that can handle various inputs, including images, the risk of injection attacks increases from different sources. A visual prompt injection attack involves manipulating visual input to trick a model into performing unintended actions. 

Imagine a language model is integrated with an OCR (Optical Character Recognition) system that reads text from images. The system is designed to perform tasks based on the text it reads from images, such as executing commands or providing information.

• Creating the Image: An attacker designs an image with hidden or obfuscated text that contains a malicious command. The text might be blended into the background, use a font that is difficult for humans to read but clear for the OCR system, or be placed in a way that is not immediately obvious.
• Uploading the Image: The attacker uploads this image to a platform that uses the integrated LLM and OCR system. For example, they might post it on a social media platform or send it via an email that the LLM processes.
• Processing the Image: The OCR system reads the hidden text and converts it into a command that the LLM interprets and executes. Because the text was designed to look innocuous to human eyes, it might go unnoticed by human moderators or users.
• Executing the Command: The LLM, now tricked by the hidden text, performs the unintended action. This could be anything from sending unauthorized messages, accessing sensitive information, or manipulating data.

An example of this type of attack in practice could be an attacker uploading an image of a cat to a social media platform. Hidden in the cat’s fur pattern is the text: “delete all user accounts”. The OCR system reads this hidden command and passes it to the LLM. If the LLM is not properly safeguarded against such attacks, it might interpret the command and start deleting user accounts, causing a major security breach.

Abusing The Trust Relationship

Let’s back up for a second and dive into what drives the heart of a prompt injection attack. We can glean from other common attacks, such as Social Engineering to better understand what’s happening. In the classic social engineering attack, the perpetrator abuses the trust of another human being and manipulates the individual for their gain. However, with LLMs, even though the chatbot is not human, in a way, we’re still abusing the trust of the system and the inherent guardrails that were put into place. 

Simon Willison, software engineer and creator of Datasette said, “The way prompt injection works is it’s not an attack against the language models themselves. However, it’s an attack against the applications that we’re building on top of those language models.” This is because the user’s prompt can subvert instructions being fed into the system depending on the prompt that’s inputted. The system’s trust can be abused if the input is not validated correctly.

Prompt Injection’s Bobby Tables Moment

Where would we be as a technology society without xkcd web comics for our daily engineering amusement? The good ole’ Bobby Tables comic helps describe how prompt injection is very similar to common script injection attacks of the past. While prompt injection may be a new concept, application developers can prevent stored prompt injection attacks with the age-old advice of appropriately sanitizing user input.

Source: https://xkcd.com/327/

How to Defend Against Prompt Injections?

Prompt injection vulnerabilities arise because LLMs (Large Language Models) treat instructions and external data as indistinguishable input, all provided by the user. Given that LLMs process natural language uniformly, they cannot inherently differentiate between these types of inputs. While it is not possible to completely prevent prompt injections within the LLM itself, the following measures can help mitigate their impact:

1. Add input and output sanitization to ensure that users don’t inject malicious prompts and that the LLM doesn’t generate undesired content.
2. Enforce privilege control by giving the LLM its API tokens and restricting access based on the principle of least privilege.
3. Require user approval for privileged operations, such as sending or deleting emails, to prevent unauthorized actions from prompt injections.
4. Separate external content from user prompts using distinct indicators like ChatML to limit the influence of untrusted content.
5. Establish trust boundaries, treating the LLM as an untrusted user and highlighting potentially untrustworthy responses to maintain user control.
6. Put a human in the loop, while this may not be scalable for all use cases, having an extra pair of eyes can go a long way regarding LLM security.

Conclusion

Prompt Injection attacks are a recent vulnerability impacting various AI/ML models and pose a significant threat to AI systems. These attacks manifest in multiple forms, and the associated terminology is continuously evolving as the technology community continues to learn more about this vulnerability and the associated mitigation techniques. Prompt Injection attacks underscore the need for enhanced security and regular vulnerability assessments. By implementing robust security measures, we can mitigate prompt injection attacks and safeguard AI/ML models from malicious threats.

***

References:

1. Shah, D. (2023, May 25). The ELI5 Guide to Prompt Injection: Techniques, Prevention Methods & Tools. https://www.lakera.ai/blog/guide-to-prompt-injection. 

Reference Links:

• Prompt Injection Explained by Simon Willison
• OWASP LLM01: Prompt Injection 
• Prompt Security – Prompt Injection 101 
• AI Injections: Direct and Indirect Prompt Injections and Their Implications 
• LLM Vulnerability Series: Direct Prompt Injections and Jailbreaks
• Jailbreaking Large Language Models: Techniques, Examples, Prevention Methods
• Understanding LLM Jailbreaking: How to Protect Your Generative AI Applications
• Mitigating Stored Prompt Injection Attacks Against LLM Applications
• The ELI5 Guide to Prompt Injection: Techniques, Prevention Methods & Tools