Ideas. Experiments. People. Join the Cybergizer community ↗
Cybergizer / BlogGuide

How to use LLMs with private data: an engineering guide

Choosing where the model runs is one decision. Making sure it only sees the data each user is allowed to see, that personal data is stripped where it is not needed, and that you can prove all of this later is the engineering work. This guide covers the architecture we use: retrieval with permissions, deployment boundaries, PII redaction and the team practices around it.

TL;DR

Start with a data map, not a model

Before choosing a model or a vector database, write down what the system will touch:

  1. Data classes. Public content, internal documents, customer data, personal data, special-category data (health, biometrics), credentials and secrets, source code.
  2. Flows. For each feature: which inputs go in, which systems are queried, where the prompt is assembled, which model processes it, where the output and the logs are stored.
  3. Who may see what. The existing permission model in the source systems: document ACLs, tenant boundaries, roles, row-level rules.
  4. Obligations. Contracts with customers, regulations (GDPR, UK GDPR, HIPAA, sector rules), data residency requirements, retention periods.

The output is a one-page diagram with every hop labeled by data class. It decides most of the architecture: which deployment boundary each flow needs, where redaction sits, and which logs must be treated as production data. It is also the document a security review or a data protection impact assessment will ask for first.

Treat the model as an untrusted component

The OWASP Top 10 for LLM Applications (2025) lists sensitive information disclosure as a top risk: models can reveal data from their context or training, and prompt injection can make them do it on purpose. The practical consequence for design:

Everything below follows from one rule: the model should only ever see data the current user is already entitled to see, and only as much of it as the task needs.

RAG over private data: where the access control lives

For most private-data use cases (search over internal documents, assistants for support or operations teams, drafting from contracts or case files) retrieval-augmented generation is the right default. The model is not trained on your data. Instead, at query time the system retrieves relevant passages and passes them to the model with the question. If you are still deciding between retrieval and fine-tuning, start there: training a model on private data makes deletion, access control and auditing much harder.

A RAG pipeline has four places where private data can leak. Each needs its own control.

1. Ingestion

2. Retrieval

This is where most access-control bugs live. The common mistake is to retrieve the top results across the whole index and then filter them. Two problems follow: users can infer the existence of documents they cannot open, and when filtering removes most results the answer quality silently drops.

Instead:

3. Prompt assembly and generation

4. Outputs, logs and evaluation data

Where a person has to check an answer before it is used, build that step into the interface rather than leaving it to habit: design human review as part of the product.

Choosing the deployment boundary: vendor API, your VPC or on-prem

Where inference runs is a per-flow decision, not a company-wide one. Our private AI guide for business owners compares the three options in plain language, including running costs. From the engineering side, the questions are:

BoundaryWhat it means in practiceEngineering questions to answer
Vendor API under contractPrompts go to the model provider's infrastructure under a data processing agreement and the provider's retention terms.Which retention and zero-retention options apply to your account? Where is data processed? Which product and endpoint does the contract cover?
Model in your cloud account (VPC)A managed model service inside your cloud account and chosen region, reached through private networking.Private endpoints instead of public internet? Which logs does the cloud service keep, and where? Is the region pinned? On AWS, for example, Amazon Bedrock documents that model providers have no access to customer prompts and completions.
Self-hosted (on-prem or your own GPUs)Open-weight models served on infrastructure you operate, for example with vLLM.Who patches and monitors the serving stack? Is the model quality enough for this task? How do you scale and fail over?

Two design choices keep this decision reversible:

From our own work (anonymized): for a US healthcare startup, we kept every model inside the client's environment, with speech recognition and the language model self-hosted, so no patient data reached a third-party AI vendor. For an open-source AI company, we built an OpenAI-compatible API layer over locally run models, with function calling, vision and routing between models, which is the gateway pattern described above.

PII redaction and pseudonymization that survives production

Redaction is not one step. Decide where it runs, what it replaces and whether the original value ever needs to come back.

Where it runs.

What it replaces.

How it detects. Combine pattern rules (emails, phone numbers, IBANs, national ID formats) with named-entity recognition for names, addresses and organizations. Open-source frameworks such as Microsoft Presidio provide this combination and can be extended with your own recognizers for domain identifiers like policy, patient or case numbers.

How you know it works. Build a labeled test set from your real document types, including the awkward ones: scanned forms, mixed languages, names that are also common words. Measure misses (personal data that got through) separately from over-redaction (useful text removed), because they have different costs. Re-run the set when you change detectors or document sources. We built this kind of loop for a recruitment-sector tool that detects and redacts personal data in CVs before they are shared with employers. The lesson that transfers: detection quality depends far more on representative test documents than on the choice of library.

Redaction reduces exposure; it does not replace access control or a lawful basis for processing. Pseudonymized data is still personal data under GDPR. If your documents arrive as forms, contracts or case files, the same controls sit inside document workflow automation.

Permissions for agents and tools

When the model can act, through tools, MCP servers or API calls, the permission question gets sharper:

For the operating side of these rules, see govern agent actions through permissions and evidence.

Engineering AI enablement: the same rules for your own team

Private data also leaks through the development process. Engineers paste stack traces with customer records into chat tools, and coding agents read repositories that contain secrets. Engineering AI enablement means giving the team AI tools that are better than the unapproved alternatives, and setting up the guardrails once:

A checklist before the first release

FAQ

How do you use an LLM with private data without training on it? Use retrieval-augmented generation. The model is not trained on your data; at query time the system retrieves passages the user is allowed to see and includes them in the prompt. Deleting or re-permissioning a document then takes effect immediately, which is hard to achieve once data is baked into model weights.

Is RAG safe for confidential documents? It can be, if access control is enforced inside the retrieval query using the end user's identity, tenants are separated, and logs are kept inside the same boundary as the data. Most RAG leaks come from shared indexes and filtering results after retrieval, not from the model itself.

Do I need on-prem or self-hosted LLMs for private data? Not always. Many teams use a model in their own cloud account (VPC) or a vendor API under a data processing agreement for most flows, and reserve self-hosted models for special-category or contractually restricted data. Decide per data class, and put a gateway in front so you can change the decision later.

Should I redact PII before sending prompts to an LLM? Redact or pseudonymize wherever the task does not need the real identifiers, especially before data leaves your boundary for a vendor API. Pseudonymization keeps references consistent so the model can still reason, and real values are restored afterwards only for authorized users. Redaction reduces exposure but does not replace access control.

How do you stop an AI assistant from showing users data they can't access? Never let the model decide permissions. Filter documents in the retrieval query with the user's identity, run tool calls with delegated user credentials, and validate outputs before they reach other systems. Instructions in the system prompt are not a security control.

What is engineering AI enablement? It is the work of helping an engineering team use AI tools productively and safely: approved tools with business data terms, repository context without secrets, least-privilege tool access, mandatory review and visibility into usage. The goal is a better workflow than the unapproved alternatives, measured by accepted work rather than generated code.

Build it with engineers who have done it

Cybergizer builds AI features that work with private data: retrieval with permissions, private model deployments, redaction pipelines and the evaluation and logging around them. We start from your data map and one workflow, and agree how you will accept the result before we build.

Explore AI systems & product development ↗ · Discuss your implementation ↗

For engineering teams adopting AI coding tools on private repositories: AI adoption for engineering teams ↗

Sources