Skip to main content

Incidents

Microsoft AI Red Team example: prompt injection against an email-summary copilot

A scenario from "Lessons From Red Teaming 100 Generative AI Products": a scammer's email hides an instruction that makes an email-summarising copilot output a malicious link.

This record is a scenario that its own source calls hypothetical; it did not happen.

Hypothetical scenario 2025-01-13 plausible

Description

The Microsoft AI Red Team paper "Lessons From Red Teaming 100 Generative AI Products" (arXiv 2501.07238, January 2025) explains its threat model with an example. An LLM-based copilot summarises a user's emails. "One possible attack against this system would be for a scammer to send an email that contains a hidden prompt injection instructing the copilot to 'ignore previous instructions' and output a malicious link." The paper calls this a cross-prompt injection attack (XPIA). It is an illustration, not a report of an attack that took place.

Exhibited failure modes

Affected systems

  • email-summary copilot (generic)

Mitigated by

  • Guardrails AI — Output validation framework with a catalog of validators and reask-on-failure loops.
  • NeMo Guardrails — Colang-based programmable guardrails for LLM applications.

Sources

Notes

A scenario described by its own source as one possible attack; nothing happened, so harm_class and reproducibility carry the lowest values the schema offers and should not be read as findings. Corrected on 2026-11-21: the second source link returned 404 and was replaced by the blog post Microsoft published with the paper; the quoted instruction was reworded to match the paper; harm_class was "security_breach". A demonstrated case of the same kind is copilot-prompt-injection-github-comments-2025.

View raw YAML on GitHub →