Sensitive Data Leakage in AI Applications: Causes and Fixes

Introduction

Employees and AI systems now handle sensitive data every day, and a lot of it leaks. Harmonic Security analyzed 22.4 million enterprise prompts and file uploads in 2025 and detected 579,113 instances of sensitive data, from source code to M&A plans. About 16.9% of those exposures went through personal AI accounts that IT could not see (Harmonic Security, 2026).

Breaches are already happening. In IBM's 2025 Cost of a Data Breach Report, 13% of organizations reported an attack that affected their AI models or applications, and 97% of those lacked proper AI access controls. A high level of shadow AI added an extra USD 670,000 to the global average breach cost (IBM, 2025).

In this guide, you will learn:

• What sensitive data leakage in AI applications means and what data is at risk
• The six paths through which AI systems leak data
• Real incidents that show each path in action
• The business and compliance impact of a leak
• Layered prevention controls, with a code example
• How to match controls to your AI architecture

As an AI & Blockchain Innovation Partner, SotaTek helps enterprises build LLM applications, RAG systems and AI agents with data redaction, permission-aware retrieval and security testing designed in from the start.

What Is Sensitive Data Leakage in AI Applications?

Sensitive data leakage in AI applications is the unintended exposure of protected information through an AI system's inputs, outputs, storage or connected tools. The person who receives the data may be an outside attacker, another user of the same app or a third-party vendor.

OWASP calls this risk LLM02: Sensitive Information Disclosure. It notes that LLMs embedded in applications can expose sensitive data, proprietary algorithms or confidential details through their output (OWASP, 2025).

Data at risk usually falls into four groups:

• Personal data (PII and PHI): names, emails, ID numbers, health records
• Financial data: card details, account numbers, revenue forecasts, deal terms
• Business secrets: source code, contracts, legal advice, product roadmaps
• Credentials: API keys, database passwords, access tokens

AI leakage is different from a classic database breach in one key way. A database returns exactly the rows a query asks for. An LLM generates text from everything in its context and weights, so it can surface data that no one explicitly requested. That makes access control at the data layer, not the model layer, the main line of defense.

6 Paths of Sensitive Data Leakage in the AI Stack

Leaks happen at specific points in the AI stack. Map your system against these six paths before choosing controls.

A few paths need more detail.

Training data memorization is a proven risk. Researchers extracted hundreds of verbatim training examples from GPT-2 with black-box queries, including names, phone numbers and email addresses. Some extracted sequences appeared in only one document in the training set (Carlini et al., 2021). If you fine-tune on customer tickets or contracts, assume the model can repeat them.

Embeddings are not anonymous. Many teams treat vector databases as safe because they store numbers, not text. Researchers showed that an inversion method can recover 92% of 32-token inputs exactly from their embeddings, and it recovered full names from clinical notes (Morris et al., 2023). Protect your vector store like the source documents.

System prompts are not secret stores. OWASP's LLM07 entry says a system prompt should not be treated as a secret or used as a security control, and that credentials and permission rules belong in systems the model cannot access (OWASP, 2025).

Real Incidents of AI Data Leakage

Each path above has already caused real exposure. These four cases show how varied the root causes are.

Sources: CSO Online, OpenAI, Wiz, PromptArmor.

Only the Slack case needed an active attacker. The other three came from normal use, a software bug and a misconfiguration. Most leakage risk sits in engineering and governance gaps, not in the model itself.

The Slack case also shows how leakage and prompt injection combine. If your app reads untrusted content, read SotaTek's guide on prompt injection prevention alongside this one.

Business Impact of Sensitive Data Leakage

A leak through an AI application creates the same costs as any data breach, plus some that are specific to AI.

• Breach cost: IBM puts the 2025 global average breach cost at USD 4.44 million. High shadow AI use added USD 670,000 to that figure (IBM, 2025).
• Regulatory fines: In December 2024, Italy's data protection authority fined OpenAI EUR 15 million. It found ChatGPT processed personal data for training without an adequate legal basis, and that OpenAI did not notify it of the March 2023 breach (WSLS, 2024). Under GDPR, fines can reach EUR 20 million or 4% of global turnover.
• Data you cannot take back: Once data is in a model's weights, you cannot delete one record. You may need to retrain the model, which costs time and compute.
• Lost trust and slower adoption: After its leak, Samsung restricted generative AI tools on company devices. Bans like this slow down the productivity gains AI was meant to deliver.
• Weak governance amplifies all of the above: 63% of organizations in IBM's study had no AI governance policies to manage AI or prevent shadow AI.

The pattern is clear. Leakage is cheaper to prevent at design time than to fix after data has spread into models, logs and vendor systems.

How to Prevent Sensitive Data Leakage in AI Applications

No single tool covers all six paths. Use layered controls so that data missed by one layer is caught by the next.

1. Classify and minimize data before ingestion

Decide which data classes each AI use case may touch. Tag sources as public, internal, confidential or restricted. Exclude restricted data from training sets and indexes unless there is a clear business need. Deduplicate training data, since repeated records are easier for a model to memorize. Strong data foundations make this step easier, as covered in SotaTek's guide to AI-ready data architecture.

2. Redact sensitive data at an AI gateway

Route every model call through one gateway. Detect and mask PII and secrets before prompts leave your network, then log the masked version only. Open-source tools such as Microsoft Presidio combine pattern matching and named entity recognition for this job.

from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine

analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()

def redact(prompt: str) -> str:
    findings = analyzer.analyze(
        text=prompt,
        entities=["PERSON", "EMAIL_ADDRESS", "PHONE_NUMBER", "CREDIT_CARD"],
        language="en",
    )
    return anonymizer.anonymize(text=prompt, analyzer_results=findings).text

safe_prompt = redact(user_prompt)  # send safe_prompt to the LLM

Detection accuracy depends on configuration and language. Test recall on your own data and add custom recognizers for internal IDs, project codes and key formats.

3. Enforce permissions at retrieval time

A RAG assistant should never retrieve a chunk the current user cannot open in the source system. Store access metadata (owner, group, classification) with each chunk and filter by the user's identity in the vector query, before ranking. Sync permissions when they change in the source system. Restrict who can add content to the index, and encrypt the vector store at rest. If you are still choosing between grounding methods, RAG vs fine-tuning compares their data exposure trade-offs.

4. Keep secrets and access logic out of prompts

Store API keys in a secrets manager and inject them only in the tool layer, never in the model context. Enforce authorization in code, not through instructions such as "do not reveal customer data." Assume users can read your full system prompt.

5. Filter outputs and control rendering

Scan model outputs for PII, secrets and restricted terms before they reach the user. Block markdown images and links to unknown domains, because attackers hide stolen data in URL parameters that the chat interface loads automatically. Return structured outputs where possible and validate them in code.

6. Limit what agents can access and send

Give each agent only the tools and data scopes its task needs. Run tool calls with the end user's permissions, not a shared service account. Require human approval before an agent emails external parties, exports files or calls new endpoints. SotaTek's AI agent development guide covers where these controls fit in the build process.

7. Control logs, retention and vendors

• Mask data before logging and set short retention periods for prompts and responses.
• Protect log stores with authentication, encryption and network restrictions.
• Use enterprise API terms that exclude your data from model training, and confirm data residency.
• Test multi-tenant isolation for caches, sessions and conversation history.

8. Govern shadow AI and test continuously

Publish a clear AI use policy, give staff approved tools and monitor traffic to unapproved AI services. Red team every release with extraction prompts, cross-user tests and injected documents. Plant canary values, such as fake customer records, in test data and alert if they ever appear in outputs or external traffic. SotaTek's AI red teaming guide explains how to run these tests.

Match Leakage Controls to Your AI Architecture

Different architectures leak data in different ways. Use this table to prioritize controls for your system.

Most enterprise systems combine several rows. A support agent that uses RAG on a third-party API needs controls from three rows at once. Start with the row that carries the most sensitive data, then add the rest.

Conclusion

Sensitive data leakage in AI applications rarely comes from one dramatic hack. It comes from ordinary prompts, memorized training data, permission-blind retrieval, secrets in system prompts, overpowered agents and unprotected logs. Map your system against these six paths, apply controls at the layer where data enters or leaves, and test continuously. Teams that design for leakage from day one can use AI on sensitive data with confidence.

Planning an AI application that touches customer, financial or proprietary data? Talk to SotaTek's AI team to review your architecture and build data leakage controls that fit your use case.