TL;DR
- Treat the LLM as an untrusted component. It should receive the minimum data needed for one task, never decide who may see what, and never hold secrets.
- Retrieval-augmented generation (RAG) is the default pattern for private data: the model reads selected passages at query time instead of learning your data through training.
- Enforce access control at retrieval time, inside the query that fetches documents, using the end user's identity. Filtering after generation is too late.
- Pick the deployment boundary per data class: a vendor API under contract, a model in your own cloud account (VPC), or a self-hosted model on infrastructure you run.
- Redact or pseudonymize personal data before it reaches components that do not need it, and keep logs, traces and evaluation sets inside the same boundary as production data.
- The same rules apply to engineers using AI coding tools on private repositories: approved tools, scoped access, no secrets in prompts, reviewed output.
Start with a data map, not a model
Before choosing a model or a vector database, write down what the system will touch:
- Data classes. Public content, internal documents, customer data, personal data, special-category data (health, biometrics), credentials and secrets, source code.
- Flows. For each feature: which inputs go in, which systems are queried, where the prompt is assembled, which model processes it, where the output and the logs are stored.
- Who may see what. The existing permission model in the source systems: document ACLs, tenant boundaries, roles, row-level rules.
- Obligations. Contracts with customers, regulations (GDPR, UK GDPR, HIPAA, sector rules), data residency requirements, retention periods.
The output is a one-page diagram with every hop labeled by data class. It decides most of the architecture: which deployment boundary each flow needs, where redaction sits, and which logs must be treated as production data. It is also the document a security review or a data protection impact assessment will ask for first.
Treat the model as an untrusted component
The OWASP Top 10 for LLM Applications (2025) lists sensitive information disclosure as a top risk: models can reveal data from their context or training, and prompt injection can make them do it on purpose. The practical consequence for design:
- Do not rely on the system prompt for security. Instructions like "never reveal customer data" are not a control. Anything placed in the context window can end up in the output.
- Do not let the model make authorization decisions. The model can suggest which tool to call; your code decides whether this user may call it with these arguments.
- Keep secrets out of the context. API keys, connection strings and tokens belong in the tool layer, never in prompts, retrieved documents or few-shot examples.
- Validate outputs before they reach another system: schema checks, allow-lists for actions, and a human approval step where the action is consequential.
Everything below follows from one rule: the model should only ever see data the current user is already entitled to see, and only as much of it as the task needs.
RAG over private data: where the access control lives
For most private-data use cases (search over internal documents, assistants for support or operations teams, drafting from contracts or case files) retrieval-augmented generation is the right default. The model is not trained on your data. Instead, at query time the system retrieves relevant passages and passes them to the model with the question. If you are still deciding between retrieval and fine-tuning, start there: training a model on private data makes deletion, access control and auditing much harder.
A RAG pipeline has four places where private data can leak. Each needs its own control.
1. Ingestion
- Keep the source document's permissions with every chunk: tenant ID, document ID, ACL groups, classification label, retention date.
- Decide per source what gets indexed at all. Some fields (bank details, national identifiers, free-text medical notes) may not need to be searchable.
- Run redaction or pseudonymization here if the use case does not need the identifiers (see the PII section below).
- Make deletion work end to end: when a source document is deleted or its permissions change, the chunks and embeddings must follow. Track lineage from chunk back to source so this is a query, not a project.
2. Retrieval
This is where most access-control bugs live. The common mistake is to retrieve the top results across the whole index and then filter them. Two problems follow: users can infer the existence of documents they cannot open, and when filtering removes most results the answer quality silently drops.
Instead:
- Filter inside the retrieval query, using the authenticated end user's identity, not a service account's. Most vector stores support metadata filters; if your index lives in PostgreSQL with pgvector, row-level security lets the database itself enforce tenant and ACL rules on every query.
- Separate tenants physically (separate indexes, schemas or databases) when the contract or the risk demands it. Logical filters are fine for many cases, but a physical boundary is easier to explain in a security review.
- Re-check permissions at answer time for long-running sessions: a user's access can change between the first question and the fifth.
- OWASP tracks this class of problem separately as vector and embedding weaknesses, including cross-tenant leakage through shared indexes.
3. Prompt assembly and generation
- Send only the retrieved passages the answer needs, not whole documents "for context".
- Mark retrieved content as data, not instructions, and strip or neutralize instructions embedded in documents. Indirect prompt injection arrives through exactly this channel.
- Return citations with every answer, so a user can open the source document and so a reviewer can tell an answer from a guess.
4. Outputs, logs and evaluation data
- Logs and traces contain the same private data as the prompt. Store them in the same boundary, with the same access rules and a retention period you can state.
- Evaluation sets built from real cases are production data too. Use synthetic or redacted cases where possible and keep the rest under access control.
- If the model can call tools, log each call with the user identity, arguments and result. That log is what you will need to answer "who saw what, when".
Where a person has to check an answer before it is used, build that step into the interface rather than leaving it to habit: design human review as part of the product.
Choosing the deployment boundary: vendor API, your VPC or on-prem
Where inference runs is a per-flow decision, not a company-wide one. Our private AI guide for business owners compares the three options in plain language, including running costs. From the engineering side, the questions are:
| Boundary | What it means in practice | Engineering questions to answer |
|---|---|---|
| Vendor API under contract | Prompts go to the model provider's infrastructure under a data processing agreement and the provider's retention terms. | Which retention and zero-retention options apply to your account? Where is data processed? Which product and endpoint does the contract cover? |
| Model in your cloud account (VPC) | A managed model service inside your cloud account and chosen region, reached through private networking. | Private endpoints instead of public internet? Which logs does the cloud service keep, and where? Is the region pinned? On AWS, for example, Amazon Bedrock documents that model providers have no access to customer prompts and completions. |
| Self-hosted (on-prem or your own GPUs) | Open-weight models served on infrastructure you operate, for example with vLLM. | Who patches and monitors the serving stack? Is the model quality enough for this task? How do you scale and fail over? |
Two design choices keep this decision reversible:
- Put a gateway in front of every model. One internal, OpenAI-compatible API for all model calls lets you route by data class (sensitive flows to the private model, non-sensitive ones to a frontier API), apply redaction and logging in one place, and switch models without touching product code.
- Route by classification, not by feature. If the data map says a flow carries special-category data, the gateway sends it to the boundary approved for that class, whatever the feature is called.
From our own work (anonymized): for a US healthcare startup, we kept every model inside the client's environment, with speech recognition and the language model self-hosted, so no patient data reached a third-party AI vendor. For an open-source AI company, we built an OpenAI-compatible API layer over locally run models, with function calling, vision and routing between models, which is the gateway pattern described above.
PII redaction and pseudonymization that survives production
Redaction is not one step. Decide where it runs, what it replaces and whether the original value ever needs to come back.
Where it runs.
- At ingestion, for identifiers the use case never needs (an internal search assistant rarely needs bank account numbers).
- At the gateway, before a prompt leaves your boundary for a vendor API.
- On outputs and logs, as a second line of defense.
What it replaces.
- Masking (
[EMAIL]) is simple and irreversible. It works when the model does not need to distinguish people. - Pseudonymization (
PERSON_1,PERSON_2) keeps references consistent so the model can still reason about "the claimant" and "the witness". The mapping table is personal data: store it inside your boundary and restore real values only after the response comes back, only for users allowed to see them. - Generalization (an age band instead of a birth date) for analytics and evaluation sets.
How it detects. Combine pattern rules (emails, phone numbers, IBANs, national ID formats) with named-entity recognition for names, addresses and organizations. Open-source frameworks such as Microsoft Presidio provide this combination and can be extended with your own recognizers for domain identifiers like policy, patient or case numbers.
How you know it works. Build a labeled test set from your real document types, including the awkward ones: scanned forms, mixed languages, names that are also common words. Measure misses (personal data that got through) separately from over-redaction (useful text removed), because they have different costs. Re-run the set when you change detectors or document sources. We built this kind of loop for a recruitment-sector tool that detects and redacts personal data in CVs before they are shared with employers. The lesson that transfers: detection quality depends far more on representative test documents than on the choice of library.
Redaction reduces exposure; it does not replace access control or a lawful basis for processing. Pseudonymized data is still personal data under GDPR. If your documents arrive as forms, contracts or case files, the same controls sit inside document workflow automation.
Permissions for agents and tools
When the model can act, through tools, MCP servers or API calls, the permission question gets sharper:
- Delegate the user's identity, do not grant the agent its own superpowers. A tool call should run with the end user's permissions (OAuth on-behalf-of, scoped tokens), so the agent can never reach data the user could not.
- Scope each tool narrowly. "Read invoices for this customer" is a tool; "run SQL" is not.
- Require confirmation for writes and external sends. Drafts are cheap to review; an email sent to the wrong customer is not.
- Log every action with arguments and results, linked to the user and session.
For the operating side of these rules, see govern agent actions through permissions and evidence.
Engineering AI enablement: the same rules for your own team
Private data also leaks through the development process. Engineers paste stack traces with customer records into chat tools, and coding agents read repositories that contain secrets. Engineering AI enablement means giving the team AI tools that are better than the unapproved alternatives, and setting up the guardrails once:
- An approved list of tools and accounts, with business or enterprise data terms, so nobody needs a personal account to get work done.
- Repository context without secrets. Keep credentials out of the repository and out of agent context; use environment-scoped, short-lived credentials for any tool access an agent needs.
- Least-privilege tool connections. An agent working on a feature branch does not need production database access.
- Synthetic or redacted fixtures for tests and debugging, so production data never has to be pasted anywhere.
- Review stays mandatory. Generated code goes through the same tests and code review as any other change, with a named person accepting it.
- Usage visibility. Know which tools and models the team uses, on which repositories, so policy is based on what actually happens. Our note on what an engineering team needs around its AI tools covers the delivery side.
A checklist before the first release
- Data map with every hop labeled by data class, signed off by the data owner.
- Deployment boundary chosen per data class, with the vendor or cloud terms that justify it on file.
- Retrieval filters run inside the query, with the end user's identity; cross-tenant tests in CI.
- Deletion and permission changes propagate from source to chunks and embeddings; tested.
- Redaction placed where the data map says, with a labeled test set and tracked miss rate.
- Gateway in front of all model calls: routing, redaction, logging, model switching.
- Logs, traces and evaluation data stored inside the same boundary, with stated retention.
- Tool calls run with delegated user permissions; writes require confirmation.
- Prompt injection test cases, including instructions hidden in retrieved documents.
- Evidence pack ready for security review and DPIA: diagram, controls, test results.
FAQ
How do you use an LLM with private data without training on it? Use retrieval-augmented generation. The model is not trained on your data; at query time the system retrieves passages the user is allowed to see and includes them in the prompt. Deleting or re-permissioning a document then takes effect immediately, which is hard to achieve once data is baked into model weights.
Is RAG safe for confidential documents? It can be, if access control is enforced inside the retrieval query using the end user's identity, tenants are separated, and logs are kept inside the same boundary as the data. Most RAG leaks come from shared indexes and filtering results after retrieval, not from the model itself.
Do I need on-prem or self-hosted LLMs for private data? Not always. Many teams use a model in their own cloud account (VPC) or a vendor API under a data processing agreement for most flows, and reserve self-hosted models for special-category or contractually restricted data. Decide per data class, and put a gateway in front so you can change the decision later.
Should I redact PII before sending prompts to an LLM? Redact or pseudonymize wherever the task does not need the real identifiers, especially before data leaves your boundary for a vendor API. Pseudonymization keeps references consistent so the model can still reason, and real values are restored afterwards only for authorized users. Redaction reduces exposure but does not replace access control.
How do you stop an AI assistant from showing users data they can't access? Never let the model decide permissions. Filter documents in the retrieval query with the user's identity, run tool calls with delegated user credentials, and validate outputs before they reach other systems. Instructions in the system prompt are not a security control.
What is engineering AI enablement? It is the work of helping an engineering team use AI tools productively and safely: approved tools with business data terms, repository context without secrets, least-privilege tool access, mandatory review and visibility into usage. The goal is a better workflow than the unapproved alternatives, measured by accepted work rather than generated code.
Build it with engineers who have done it
Cybergizer builds AI features that work with private data: retrieval with permissions, private model deployments, redaction pipelines and the evaluation and logging around them. We start from your data map and one workflow, and agree how you will accept the result before we build.
Explore AI systems & product development ↗ · Discuss your implementation ↗
For engineering teams adopting AI coding tools on private repositories: AI adoption for engineering teams ↗