Prompt Injection via Images

Multimodal AI systems can interpret screenshots, scanned documents, charts, and photographs to summarize reports, extract data, or answer questions about an interface. This capability also opens another path for prompt injection.

An image may contain instructions intended for the model rather than the viewer. If treated as authoritative, they can influence its behavior. The danger grows when the model can use existing permissions to access private information, call tools, or take actions.

How an Image Becomes a Prompt

A request may combine system instructions, the user’s message, conversation history, images, OCR text, and documents or tool results. Processing these sources together makes it difficult to maintain a strict boundary between trusted instructions and untrusted visual content.

Imagine an employee asking an assistant to summarize a screenshot from an external website. Hidden text tells it to retrieve an internal document and send the contents outside the organization. If the application permits those operations, the image supplies the intent while the agent supplies the access.

This is indirect prompt injection. NIST describes prompt injection as an attack that exploits the combination of untrusted input with a prompt created by a higher-trust party. OWASP notes that malicious input doesn’t need to be visible or readable to a person to affect a model.

Why Visual Attacks Are Difficult to Detect

Visual attacks may be obvious or easy to overlook. Examples include:

These methods do not work equally well across every model or pipeline. Research on self-interpreting adversarial images has shown that modified pixels can influence responses without ordinary readable instructions.

OCR creates another injection route. An image may look harmless while OCR extracts text that enters the model’s context. Resizing, compression, cropping, and preprocessing can also change what each component detects. If a filter and the production model receive different versions, one may miss content the other interprets. Trail of Bits demonstrated attacks in which downscaling revealed instructions not apparent in the original image.

The consequences depend on the system. An image-description model may give a misleading answer, while an agent connected to email, storage, databases, or browser tools could expose data or perform an unintended action if safeguards allow it.

Prompt Instructions Are Not a Security Boundary

Telling a model to ignore instructions inside images may reduce risk, but it does not create an enforceable boundary. The model must still decide what to analyze and follow, while attackers can vary the wording, placement, formatting, and context. OWASP recommends multiple mitigations because complete prevention is not guaranteed.

Applications should label image-derived content as untrusted and separate it from system instructions where possible. OCR output should keep the same trust status as its source. Labels and delimiters may help distinguish sources, but they do not enforce permissions. A generated tool call remains a proposal, not proof that an action is authorized.

Building Defenses Around Multimodal Inputs

Scanners and prompt filters cannot guarantee that an upload is safe. File validation should restrict formats, dimensions, and sizes, while securely configured, updated libraries address file-handling risks. Detectors can inspect images and OCR output, although they may miss attacks or raise false alarms.

The runtime should preserve whether content came from an image, webpage, or document; limit the agent to the files and operations needed; and separate reading from acting. An image-summary agent does not automatically need permission to send email or modify records.

Before executing a tool call, the system should check the operation, target, data classification, user identity, and authorized scope. High-risk actions may require an authorized reviewer to see the exact action, target, and relevant data. Approval applies only to what was reviewed and does not replace other controls. Outbound communication should also be restricted because an allowed domain may contain an attacker-controlled account.

Audit records should capture source identifiers, relevant model outputs, tool calls, policy decisions, and outcomes. Images or extracted text should be retained only when needed, with access restrictions and retention limits. Microsoft describes indirect prompt-injection defense as combining probabilistic detection with deterministic controls that can block harmful actions or data transfers.

Reducing the Impact

Images should be treated as untrusted even when they look ordinary. Testing should cover visible instructions, low-contrast text, OCR attacks, altered sizes, and malicious content mixed with legitimate documents. It should use the real preprocessing pipeline and check what happens after the model proposes an unsafe action.

Multimodal AI can extract useful visual information, but understanding an image does not give it authority. Enforced permissions, tool validation, outbound controls, provenance tracking, and targeted approval can limit the harm a malicious image causes.