Microsoft AI Red Team example: prompt injection against an email-summary copilot
A scenario from "Lessons From Red Teaming 100 Generative AI Products": a scammer's email hides an instruction that makes an email-summarising copilot output a malicious link.
This record is a scenario that its own source calls hypothetical; it did not happen.
Description
The Microsoft AI Red Team paper "Lessons From Red Teaming 100 Generative AI Products" (arXiv 2501.07238, January 2025) explains its threat model with an example. An LLM-based copilot summarises a user's emails. "One possible attack against this system would be for a scammer to send an email that contains a hidden prompt injection instructing the copilot to 'ignore previous instructions' and output a malicious link." The paper calls this a cross-prompt injection attack (XPIA). It is an illustration, not a report of an attack that took place.
Exhibited failure modes
- Security vulnerability — Insecure patterns, injection flaws, weak cryptography, secret leakage.
- Context pollution — Trajectory drift, hallucination spirals, overlong or off-task context.
Affected systems
- email-summary copilot (generic)
Mitigated by
- Guardrails AI — Output validation framework with a catalog of validators and reask-on-failure loops.
- NeMo Guardrails — Colang-based programmable guardrails for LLM applications.
Sources
Notes
A scenario described by its own source as one possible attack; nothing happened, so harm_class and reproducibility carry the lowest values the schema offers and should not be read as findings. Corrected on 2026-11-21: the second source link returned 404 and was replaced by the blog post Microsoft published with the paper; the quoted instruction was reworded to match the paper; harm_class was "security_breach". A demonstrated case of the same kind is copilot-prompt-injection-github-comments-2025.