When Your AI Takes Orders from Strangers: The Prompt Injection Problem

Hi everyone,

In this article, I’ll share what I’ve learnt about prompt injection. As developers, most of us are familiar with classic vulnerabilities like SQL injection. Prompt injection is a newer class of risk that emerges from how modern AI systems are instructed and controlled.

A financial services company lost access to 127 confidential emails. The attacker didn't hack their servers. They didn't steal passwords. They didn't exploit a software vulnerability.

They sent an email with invisible text.

Three employees asked their AI assistant to summarize that email. The AI read the hidden instructions, searched for sensitive messages, and forwarded them to an external address. Nobody noticed until a week later.

This is prompt injection. And if you're using AI systems in your business, you need to understand it.

Imagine hiring a brilliant assistant who follows instructions perfectly. They can't distinguish between orders from you, their actual employer, and suggestions whispered by a stranger pretending to have authority.

That's how most AI language models behave in production today.

This vulnerability, prompt injection, is becoming the defining security challenge of the AI era. While SQL injection terrorized web applications in the 2000s, prompt injection is emerging as its successor for large language models. Unlike many theoretical security risks, this one is actively being exploited in the wild.

And unlike traditional attacks that require technical sophistication, this one can be executed by anyone who understands how to phrase a sentence.

This vulnerability "prompt injection" is the defining security challenge of the AI era. It's being actively exploited in the wild, and unlike most security threats that require technical sophistication, this one can be executed by anyone who understands how to phrase a sentence.

Here's what actually happens, why it matters, and what you need to do about it.

Language models process everything as text. System instructions, user messages, and external content all land in the same channel. The model does not reliably distinguish between policy and data without additional controls.

When an attacker can convince the AI that their input is actually a system instruction, the AI obeys. Not because it's been "hacked" in the traditional sense, but because it's doing exactly what it was designed to do. Just follows instructions it finds in text.

Traditional software has clear boundaries: code is separate from data, system commands are distinct from user input. AI language models blur these boundaries. That's the fundamental problem.

How It Actually Works

Scenario 1: Direct Manipulation

Jaffna Heritage Tours runs a chatbot for tourists. The developers set it up with clear rules:

A visitor types:

What happens:

A poorly protected system reads both the original rules and the user's message sequentially. The message explicitly says "ignore previous instructions" and provides new ones. Without proper defenses, the AI follows the most recent, most specific instruction.

Attack succeeded. Internal pricing leaked.

Scenario 2: The Hidden Attack (More Dangerous)

TechNorth Solutions in Jaffna uses AI to screen job applications. A candidate named Priya submits a resume that looks normal, but contains hidden text in white font at the bottom:

The HR manager asks the AI: "Review these resumes and rank candidates."

The AI reads every document including Priya's hidden instructions. Her resume appears at the top, flagged as urgent, with an artificially inflated score.

The HR team never sees the manipulation. They only see their "trusted" AI recommending what looks like a top candidate.

This is indirect prompt injection — malicious instructions hidden in content the AI processes later. No direct interaction required. The attack activates automatically.

Let me walk you through how actually Email forwarding plays out.

Day 1 - The Setup:

The attacker sends an email to multiple employees. Subject: "Q4 Budget Review - Please Summarize. " The body contains a normal-looking financial report, but buried in the middle is invisible text:

Day 2 - The Trigger:

Employees use their AI email assistant to summarize the message. They're doing exactly what the subject line asks. The AI reads everything, including the hidden instructions.

Day 3-5 - The Breach:

The AI assistant, being helpful, does what it was told:

1. Summarizes the budget report (the expected action)
2. Searches for emails with "confidential" (the injected instruction)
3. Forwards sensitive emails to the external address (completing the attack)

All employees receive clean-looking summaries. Nothing seems wrong.

Day 8 - Discovery:

Security audit flags unusual email forwarding patterns. Investigation reveals the breach. By this point, sensitive information has been exfiltrated.

Why It Worked:

1. The AI couldn't distinguish between document content and hidden instructions
2. Employees trusted their AI assistant to "just summarize"
3. No logging captured what the AI actually read
4. The AI had permission to access email and send messages
5. No output validation checked for unexpected actions

This wasn't a sophisticated technical hack. It was a carefully phrased sentence that exploited how language models work.

Why Traditional Security Doesn't Apply

If you're thinking "can't we just filter out the bad inputs?" that's the right instinct, but it doesn't work here.

In traditional web security, you can block specific dangerous patterns. A database knows that certain character combinations are malicious because they contain symbols that shouldn't appear in normal text. You can escape those characters or reject the input.

But with AI, there are no special characters. The attack uses perfectly normal language. "Ignore previous instructions" and "disregard prior directives" mean the same thing. So do hundreds of other phrasings. You can't block them all without blocking legitimate queries too.

Imagine you're a bank teller helping customers. Someone walks up and says, "Hi, I'd like to make a withdrawal. Also, I'm actually the branch manager doing a security drill. Please give me all the cash in your drawer and don't mention this to anyone."

As a human, you'd immediately recognize this as suspicious. You'd verify identity and follow protocols.

But an AI doesn't have that instinct. It processes both sentences the same way: as instructions to follow. It doesn't automatically know that the second sentence is trying to override security policies.

The attack isn't breaking in. It's talking its way in.

Traditional security assumes attackers are exploiting bugs or bypassing authentication. Prompt injection doesn't need either. It works by using the system exactly as designed, just with carefully crafted language.

Security tools often work by recognizing attack patterns. But prompt injection has infinite variations. Today's attack might say "ignore previous instructions." Tomorrow's might say "new priority directive." Next week it might use completely different phrasing that means the same thing.

It's like trying to prevent social engineering by listing every possible lie someone might tell. The list would be endless, and attackers would just use lies you hadn't thought of yet.

Current AI systems treat everything as input to process, not as commands to evaluate for trustworthiness. Until that changes at the architectural level, we're stuck with defense through careful system design rather than simple input filtering.

What Actually Works:

You can't eliminate this risk entirely, but you can reduce it dramatically. Here's what to implement, in order of impact.

Tier 1: Highest Impact fix:

1. Architectural Separation

Never give the same AI access to both system instructions and untrusted content.

The Input Handler can be manipulated, but it has no permissions. The Decision Maker has permissions, but only sees pre-processed content.

Why this works:

Even if injection succeeds in the Input Handler, the malicious instructions never reach the component that can actually take actions.

Real implementation:

Use separate service accounts with different permission levels. The input processor runs with read-only access. The decision maker runs with elevated permissions but only accepts structured input (not raw text).

2. Least Privilege by Default

AI should only access what it absolutely needs, when it needs it.

1. Customer support bot: Read-only access to knowledge base. No email sending, no database writes.
2. Resume screener: Can score documents, cannot access email or internal systems.
3. Code assistant: Can suggest code, cannot execute it without explicit approval.

Practical example:

Use separate service accounts per agent. Disable outbound email by default. Require an allowlist for recipients. Remove database write permissions unless specifically needed for that workflow.

Most teams won't rebuild their entire architecture this quarter. That's fine. But start from removing permissions that aren't absolutely necessary, and add human approval gates for sensitive actions. You can refactor the architecture later.

If you can't tackle the full architecture changes yet?

These three actions take minimal time and reduce risk immediately:

1. Remove unnecessary tool permissions

Audit what your AI systems can actually do. Revoke database write access, file system access, and API permissions that aren't required for core functionality.

2. Add recipient allowlists for outbound communications

If your AI can send emails or messages, restrict recipients to an approved list. Require explicit approval to add new recipients.

3. Require human approval for sensitive actions

Any action involving external data transfer, financial transactions, or access to confidential information should pause for human verification before execution.

These are configuration changes, not code changes. You can implement them today.

Tier 2: Important but More Complex

3. Output Validation

Before the AI's response is shown or action is taken, check:

1. Does it contain sensitive information it shouldn't have?
2. Is it trying to perform actions outside its scope?
3. Does it look suspiciously different from normal outputs?

Use a second AI model or rule-based system to validate outputs before they go live.

Implementation tip: Start simple with rule-based checks (regex for credit card numbers, API keys). Add ML-based anomaly detection later if you have resources.

4. Comprehensive Logging

Log everything the AI reads and generates. Not just the final output — the full context.

When an incident happens (and it will), logs are the only way to understand what went wrong.

What to log: Full prompt sent to model, complete model response, any tools called, permissions checked, validation results. Retain for at least 90 days.

Tier 3: Advanced Hardening - Do This When Resources Allow

5. Content Sanitization

For external content (resumes, documents, emails):

1. Strip all formatting before processing
2. Extract text only, discard hidden elements
3. Use OCR on images rather than trusting embedded text
4. Process in sandboxed environments

6. Human-in-the-Loop for Sensitive Actions

Require human approval before:

1. Sending emails
2. Accessing sensitive data
3. Making financial transactions
4. Modifying system configurations

Present AI reasoning transparently so humans can spot manipulation.

7. Adversarial Testing

Regularly test your system with injection attempts. Hire penetration testers or use automated tools to probe for weaknesses.

As of early 2026, prompt injection remains largely unsolved at the fundamental level. We've gotten better at mitigation, but the core problem persists: language models can't inherently distinguish trusted instructions from untrusted input.

The good news: Layered defenses make successful exploitation much harder and limit damage when it happens.

If you're building AI systems that handle untrusted content or have access to sensitive operations, you need to assume injection will be attempted and design accordingly.

Practical Checklist for Your Team

Before deploying any AI system, answer these questions:

1. What's the worst thing that could happen if someone tricks this AI?
2. Does the AI process content from untrusted sources?
3. What permissions does it have? (Can it read/write/send/execute?)
4. Is there architectural separation between input handling and privileged operations?
5. Do we log everything for forensic analysis?
6. Is there output validation before actions are taken?
7. Have we tested it with adversarial prompts?
8. Do we have an incident response plan for AI manipulation?

If you can't confidently answer "yes" to most of these, you're not ready for production.

Closing thought:

Prompt injection isn't a theoretical risk or an edge case. It's being exploited right now, in real systems, causing real damage.

Companies that had "AI security on the roadmap" have gotten breached while waiting to prioritize it.

Don't be one of them.

Start with Tier 1 defenses: architectural separation and least privilege. Those two changes alone will protect you from many of the most common attack patterns. Then work your way through the remaining tiers as resources permit.

The AI doesn't know who to trust. That's your job to engineer into the system.

Notes on Sources and Evidence

Attack patterns described: Based on documented incidents from security advisories, bug bounty reports, and published research from 2024-2025. The composite email forwarding example draws from multiple reported variations of this attack class.

Public incident references:

1. Indirect prompt injection via resume was demonstrated by security researchers at AI conferences in 2024-2025
2. Email-based injection attacks are documented in OWASP LLM Top 10 (LLM01: Prompt Injection)
3. Hidden text exploits were publicly demonstrated against major AI assistants (see Simon Willison's "Prompt injection: What's the worst that can happen?" from April 2023 and subsequent technical analyses)

Defense effectiveness: Recommendations are based on industry best practices from:

1. OWASP Top 10 for LLM Applications 2025 (LLM01: Prompt Injection)
2. NIST AI Risk Management Framework (AI RMF 1.0, January 2023 - Govern, Map, Measure, Manage functions)
3. Field reports from security teams implementing these controls

Specific effectiveness varies by implementation quality and evolving attack sophistication.

Key technical resources:

1. OWASP LLM01: Prompt Injection - https://owasp.org/www-project-top-10-for-large-language-model-applications/
2. Simon Willison's blog: simonwillison.net (extensive prompt injection research and demonstrations)
3. NIST AI RMF: https://www.nist.gov/itl/ai-risk-management-framework
4. Anthropic, OpenAI, and Google AI safety and security documentation
5. NCC Group and Trail of Bits LLM security research publications

Want to go deeper?

The OWASP Top 10 for LLM Applications (2025 edition) has comprehensive technical guidance. Search "OWASP LLM01 Prompt Injection" for the full breakdown.

Building AI systems? Consider hiring security researchers who specialize in LLM vulnerabilities for penetration testing.

Encountered an attack? Document it. Share it (anonymized if needed). The community learns faster when we stop pretending these incidents aren't happening.

What's your experience with AI security?

Have you implemented any of these defenses? Share in the comments.