Prompt injection is an exploit where malicious text inserted into data an agent processes, like a document or web page, is misinterpreted by the model as an instruction. The model then executes these instructions, because it cannot reliably distinguish between commands given by its system prompt and data it is meant to merely observe. This vulnerability is the primary attack vector for agentic systems that operate with real-world permissions and interact with untrusted external content.
Prompt injection is a critical concern for any agent designed to act in the real world. If an agent with email access reads a crafted message, or one that manages files processes a malicious document, it might delete crucial data, send confidential information, or execute unauthorized code. This is not a theoretical flaw; it directly undermines the integrity and security of autonomous systems, leading to tangible financial loss, data breaches, or operational disruption if not robustly defended against.
A common misconception is that a strong system prompt or explicit instructions telling the model to ignore such commands will prevent prompt injection. However, these are merely requests; a sufficiently clever injection can often override them because the model processes all text, including the system prompt and untrusted data, through the same mechanism. Effective defenses against prompt injection reside in the harness: employing least privilege to limit potential damage, strictly separating trusted instructions from untrusted content, requiring human confirmation for consequential actions, and maintaining an immutable log of all operations.