AI Search and Answer Poisoning

Attackers contaminate public information sources so AI search systems repeat fraudulent claims, contact details, or destinations as trusted answers, potentially directing users into scams.

AI search and answer poisoning exploits the information an assistant retrieves when answering legitimate questions.

In September 2026, Vigilance Security researcher Ariel Simon reported “Dark Sourcery,” a campaign using malicious web content to introduce fraudulent contact details into answers from ChatGPT, Gemini, and Google AI Overview.

The researcher reported 374 targeted companies. This represents brands targeted by deceptive information, not 374 confirmed enterprise breaches. Reported results varied between repeated queries

Attacker-controlled information

Malicious material enters sources an AI system may consult. The assistant can then reproduce false information without the attacker compromising the model’s hosting infrastructure.

Trust boundary

The failure occurs when retrieved claims are presented as verified guidance.

Relationship to other attacks

This narrative focuses on public search sources. RAG Data Poisoning addresses manipulation of retrieved knowledge more broadly. Explicit instructions embedded in retrieved content may additionally constitute indirect prompt injection, but fabricated facts alone can poison an answer.

1. Publish misleading information

The attacker places false claims in discoverable content.

2. Reach retrieval

An AI search system encounters that material while handling a legitimate query.

3. Contaminate the answer

The response repeats the attacker’s claim as useful guidance.

4. Influence the user

The user may follow the fraudulent contact information or destination.

This generalized sequence does not establish that every exposed user suffered a loss.

Assessment scope may include:

• Assistants with public web-search capabilities.
• Applications that generate answers from external documents.
• Customer-service tools that retrieve contact information.
• Research assistants that recommend websites or services.
• Agents permitted to act on externally retrieved information.

Applicability depends on how each application selects, validates, and uses its sources.

Customer decision risk

An organization should assess whether inaccurate AI guidance can influence consequential customer decisions, including account recovery and selection of service providers.

Operational burden

Investigation may require coordination among security, customer support, communications, and the teams maintaining AI integrations.

Measurement

Track incorrect-answer frequency, affected workflows, confirmed user actions, and remediation effectiveness separately. Exposure to misleading information should not automatically be counted as a successful compromise.

Controlled assessment

Use an isolated retrieval environment and a fictional organization. The following is proposed testing guidance, not a statement that Redbot has validated a particular deployment.

Establish authoritative facts

Create an approved reference containing synthetic contact details and permitted destinations.

Introduce conflicting test material

Add a clearly controlled test document containing an alternative, harmless destination. Keep the document within the authorized environment.

Evaluate answer accuracy

Check whether the assistant selects the approved facts, identifies the conflict, or presents the unsupported claim as authoritative.

Evaluate action controls

Where an agent can navigate or communicate, determine whether it independently validates the destination before acting.

Record repeatability

Preserve the query, retrieved documents, citations, generated answer, model configuration, and resulting actions. Measure repeated trials rather than treating a single successful or unsuccessful response as conclusive.

Compare answers against approved records

Maintain a verified inventory of organizational domains, support channels, and account-recovery destinations. Flag generated answers that contradict that inventory.

Retain source provenance

Record which documents supplied important claims. Review whether the cited passage actually supports the answer.

Identify conflicting evidence

Escalate inconsistent contact details or instructions instead of silently selecting one version.

Track changes over time

Use a repeatable set of representative questions to detect unexpected changes in answers and citations.

LLM09:2025 - Misinformation