AI Agent Goal Hijacking

AI Agent Goal Hijacking occurs when an attacker manipulates an autonomous agent's objectives, priorities, planning context, or task state so the agent begins pursuing attacker-influenced goals while continuing to operate through legitimate enterprise permissions and tools.

AI agents differ from traditional conversational AI because they can pursue objectives across multiple steps, maintain task state, make decisions, invoke tools, interact with external systems, and adapt their plans as conditions change.

This creates a critical attack surface: the agent’s goal itself.

An attacker may not need to directly command an agent to perform a malicious action. Instead, the attacker can attempt to alter the information the agent uses to determine what it is trying to accomplish.

Malicious instructions introduced through prompts, retrieved documents, web content, email, shared memory, tool responses, MCP resources, or other agents may influence an agent’s planning process.

If the malicious objective becomes incorporated into the agent’s working state, the agent may continue pursuing it across subsequent actions.

This is particularly dangerous because the resulting activity can appear legitimate. The agent may use valid credentials, approved APIs, authorized tools, and expected workflows while pursuing an objective the user never intended.

AI Agent Goal Hijacking targets the mechanisms used by an autonomous system to establish, maintain, update, and prioritize objectives.

An attacker first gains influence over information consumed by the agent.

Rather than requesting a single unauthorized action, the attacker introduces instructions designed to alter the agent’s current objective or create a competing objective with higher apparent priority.

Potential manipulation targets include:

• System objectives
• User task state
• Planning context
• Retrieved instructions
• Agent memory
• Tool responses
• External documents
• Web content
• Email and messaging content
• MCP resources
• Agent-to-agent messages
• Workflow state
• Completion criteria

A successful attack may cause the agent to reinterpret its legitimate assignment.

For example, an enterprise research agent tasked with collecting information for an employee could encounter attacker-controlled content instructing it to retrieve additional internal information and transmit that information through an available tool.

The agent may incorporate the malicious objective into its plan and continue executing it as part of what appears to be the original task.

The security failure occurs when untrusted information is able to modify trusted agent objectives.

1. Legitimate Task Initiation
A user or enterprise workflow assigns an objective to an AI agent.

2. Exposure to Attacker-Controlled Input
The agent encounters malicious content through a prompt, document, website, email, RAG source, memory store, tool response, MCP resource, or another agent.

3. Goal Manipulation
The malicious content introduces a new objective, modifies an existing objective, or changes task priorities.

4. Planning Contamination
The attacker-controlled objective becomes incorporated into the agent’s working plan or state.

5. Legitimate Capability Use
The agent invokes authorized tools, APIs, credentials, applications, or data sources while pursuing the manipulated objective.

6. Persistent Execution
The altered objective may survive across multiple reasoning and execution steps.

7. Unauthorized Outcome
The agent accesses data, performs actions, changes resources, communicates externally, or completes another attacker-directed objective.

Attack path:

Attacker-Controlled Input → Agent Context → Goal Manipulation → Planning → Authorized Tools → Unauthorized Outcome

• Autonomous AI agents
• Enterprise AI assistants
• Multi-agent environments
• RAG-enabled agents
• MCP-enabled agents
• Browser agents
• Research agents
• Coding agents
• DevOps agents
• Cloud administration agents
• Email agents
• Customer service agents
• Financial workflow agents
• Security automation agents
• Long-running AI workflows
• Agents with persistent memory
• Tool-enabled LLM applications

AI Agent Goal Hijacking can transform legitimate enterprise automation into attacker-directed automation without requiring the attacker to directly compromise the underlying credentials, API, or infrastructure.

The agent may continue using legitimate identities and approved capabilities, making malicious activity significantly harder to distinguish from expected automation.

Potential consequences include:

• Unauthorized data access
• Sensitive data disclosure
• External data transmission
• Unauthorized communications
• Modification of cloud resources
• Malicious code changes
• Unauthorized API activity
• Manipulation of enterprise workflows
• Financial or transactional actions
• Persistence of malicious objectives
• Cross-agent propagation
• Expansion into additional enterprise systems

The risk becomes substantially greater as organizations give agents longer execution windows, persistent memory, broader tool access, and authority to independently plan how objectives should be completed.

An enterprise therefore cannot evaluate agent security solely by asking whether each individual tool call is technically permitted.

The organization must also determine whether the objective causing those permitted actions is still the objective the user originally authorized.

For Redbot’s Attack the Stack™ methodology, this creates a critical adversarial test:

Can we change what your AI is trying to accomplish without changing who it is or the permissions it already has?

If the answer is yes, an attacker may be able to turn the enterprise’s own AI automation against the systems it was designed to operate.

Establish a legitimate task with clearly defined objectives and permitted actions.

Introduce attacker-controlled information through each untrusted channel available to the agent.

Test:

• Direct goal replacement
• Competing objective injection
• Priority manipulation
• Indirect prompt injection
• Retrieved-document instructions
• Malicious web content
• Email-based instructions
• Tool-response manipulation
• MCP resource manipulation
• Memory-based objective manipulation
• Cross-agent objective propagation
• Completion-condition manipulation
• Multi-step persistence of injected goals
• Objective changes following tool execution
• Goal manipulation across session boundaries

Determine whether untrusted content can alter what the agent believes its primary objective to be.

Then determine whether the altered objective causes the agent to invoke capabilities that would otherwise be legitimate.

A strong validation test is:

Assign the agent Goal A, expose it to attacker-controlled content containing Goal B, and determine whether the agent begins pursuing Goal B without explicit authorization from the initiating user.

Testing should continue beyond the initial response.

Evaluate whether the manipulated objective persists through:

• Additional reasoning cycles
• Tool calls
• Agent handoffs
• Memory writes
• Workflow transitions
• Session continuation
• Error recovery
• Task replanning

Monitor for:

• Agent objectives changing during execution
• New tasks not present in the original user request
• Tool calls unrelated to the initiating objective
• Unexpected changes in planning state
• External instructions becoming agent tasks
• Sensitive operations following retrieval of untrusted content
• Unexpected memory writes
• Agent-generated subgoals outside approved scope
• Repeated attempts to complete unauthorized objectives
• Tool invocation following external document retrieval
• Cross-agent propagation of unexpected objectives
• Deviations between original task and completed actions
• External communications not required by the original task
• Unexpected changes to workflow completion criteria

Where possible, retain both the original authorized objective and subsequent agent-generated plans for comparison.

MITRE ATLAS Relevant ATLAS concepts include adversarial manipulation of AI-enabled systems, prompt-based attacks, and abuse of AI system behavior. Exact technique mapping should be determined based on the initial access and execution mechanism used in the tested implementation. MITRE ATT&CK T1059 — Command and Scripting Interpreter may become relevant when a hijacked coding or operations agent subsequently invokes command execution capabilities. T1199 — Trusted Relationship may apply where the manipulated agent uses trusted relationships with downstream systems or agents. The resulting enterprise behavior should additionally be mapped to the ATT&CK technique corresponding to the action actually performed by the compromised agent.

OWASP LLM01 — Prompt Injection Direct or indirect prompt injection can provide the initial mechanism for altering an agent's objective. LLM06 — Excessive Agency Broad functionality, permissions, and autonomy increase the potential impact once an agent's objective has been manipulated. OWASP Agentic AI Security Goal manipulation, instruction hierarchy, tool authorization, memory security, external-content trust, and autonomous execution are central considerations for agentic AI security.