Every business owner we talk to asks the same question before approving an AI project: where does our data go? It is the right question. Customer records, contracts, patient notes and financial data are exactly what makes AI useful inside a company, and exactly what you cannot afford to leak.
The good news: in 2026 you no longer have to choose between using AI and keeping your data in-house. This guide explains, in plain language, what really happens to data sent to public AI tools, the three ways to run "private AI", what they cost, and how to decide.
TL;DR
- Private AI means running large language models (LLMs) so that your sensitive data never leaves an environment you control or a contract you trust.
- There are three options: public APIs with enterprise terms, AI hosted in your own cloud account, and self-hosted open-weight models. Most companies end up with a mix.
- Open-weight models are now good enough for most focused business tasks: document processing, search, summaries, transcription and classification.
- Self-hosting starts at roughly $1,500 per month in GPU rental for one dedicated GPU, plus engineering. It pays off when data is regulated or volumes are high.
- The biggest risk is usually not the AI vendor. It is "shadow AI": employees pasting company data into tools nobody approved.
What happens to your data when you use public AI tools?
It depends on which product you use and which contract you signed. "ChatGPT" can mean very different things for your data.
Business APIs are safer than most owners think. According to OpenAI's API data policy, data sent to the API is not used to train its models unless you opt in. But it is still stored: abuse-monitoring logs are kept for up to 30 days by default. Zero Data Retention is available only to customers OpenAI approves. Regional data residency in Europe also requires approval through sales.
Legal processes can override vendor promises. In 2025, a US court order in the New York Times copyright case required OpenAI to preserve ChatGPT output that would otherwise have been deleted. The order was lifted for new data in October 2025, but logs already saved stayed preserved (Simon Willison). The lesson for a business: data stored on someone else's servers is subject to someone else's legal battles.
The real leak is usually inside your own company. IBM's Cost of a Data Breach Report 2025 found that:
- 63% of the 600 organizations studied had no policy to manage AI or prevent shadow AI.
- A high level of shadow AI added about $670,000 to the average cost of a breach.
- 97% of organizations that had an AI-related security incident lacked proper AI access controls.
In other words, banning AI does not protect your data. Employees will use it anyway. Giving them an approved, private tool that is as good as the public one does protect it.
What is private AI? Three ways to deploy it
Private AI is any setup where you decide where prompts, documents and outputs are processed and stored, and who can access them. In practice there are three levels of control.
| 1. Public API, enterprise terms | 2. AI in your own cloud account | 3. Self-hosted open-weight models | |
|---|---|---|---|
| Example | OpenAI or Anthropic API with a data processing agreement | Azure OpenAI, Amazon Bedrock, Google Vertex AI in a region you choose | Qwen, Mistral, DeepSeek or similar models on your servers or rented GPUs |
| Where data goes | Vendor's servers, under contract | Your cloud provider, inside your account and region | Only your infrastructure |
| Model quality | Best available | Best available (most major models) | Strong for focused tasks, behind the frontier on the hardest ones |
| Setup time | Days | Weeks | Weeks to a few months |
| Cost model | Pay per use | Pay per use, often with reserved capacity | Fixed infrastructure cost + engineering |
| Best for | Non-sensitive data, prototypes | Most regulated companies already on a big cloud | Health, legal, finance, IP-heavy or very high-volume workloads |
Option 1 is fine for marketing copy, public documents and prototypes. Option 2 keeps data within the security perimeter and contracts you already have with your cloud provider. Option 3 is the only one where no third party processes your data at all.
How good are open models now? A mid-2026 review of open-weight models lists several strong, commercially licensed options, such as Qwen (Apache 2.0), Mistral Small (Apache 2.0) and DeepSeek (MIT). It also states plainly that closed frontier models "still lead many hard, ambiguous, high-stakes tasks." That matches what we see in projects. For transcription, extraction, classification, search and summaries, a well-chosen open model is usually enough. For complex reasoning across many documents, a frontier model may still win.
The smart default is hybrid. Route sensitive data to a private model. Send anonymized or non-sensitive tasks to a frontier API. If your application talks to every model through the same OpenAI-compatible interface, you can switch models without rewriting the product.
Is AI compliant with GDPR and HIPAA?
AI can be compliant with both, but compliance is a property of your setup, not of the model. Here is what each regulation means for an AI project in plain terms. (This is general information, not legal advice.)
GDPR (EU and UK). If prompts contain personal data, the AI vendor is your data processor. You need a data processing agreement, a lawful basis for the processing, and a legal mechanism for any transfer outside the EU. Self-hosting inside the EU removes the transfer question entirely and makes data minimization and deletion requests much easier to honor.
HIPAA (US healthcare). Any vendor that touches protected health information must sign a Business Associate Agreement (BAA). Some AI providers offer BAAs for specific products. A local model inside your HIPAA-compliant environment avoids the question, because patient data never reaches a new vendor.
EU AI Act. The obligations for high-risk AI systems were originally due in August 2026. The EU's Digital Omnibus moved them: Annex III high-risk obligations now apply from 2 December 2027, and Annex I from 2 August 2028 (Orrick, July 2026). Use the extra time to document which models you use, for what, and with which data. A private setup makes that documentation much simpler.
One rule covers all three: know where every piece of sensitive data goes. If you can draw that map on one page, an auditor, an investor's due diligence team or an enterprise client's security review will be short.
How much does private AI cost?
A self-hosted model costs from about $1,500 a month in rented GPU time, plus a one-off engineering project. The main line is compute. An NVIDIA H100, the workhorse GPU for running LLMs, rented 24/7 (about 730 hours a month) costs roughly (IntuitionLabs, October 2026):
| Provider type | H100 price per GPU-hour | One GPU, 24/7, per month |
|---|---|---|
| Specialized GPU clouds (Runpod, Lambda, Nebius) | $2.00 – $4.50 | ~$1,450 – $3,300 |
| Hyperscalers (AWS, Azure, Google Cloud) | $6.88 – $12.29 | ~$5,000 – $9,000 |
On top of compute, budget for:
- Setup: choosing and testing the model on your real data, deployment, security, an OpenAI-compatible API layer, monitoring.
- Maintenance: model updates, scaling and cost tuning. Usually a few days a month, not a full-time team.
- Your own hardware (optional): buying GPUs makes sense only with steady, high load. Most companies should rent first.
When does it pay off? Pay-per-use APIs, billed per 1M tokens, are cheaper at low volume. Private hosting wins in three cases:
- Regulated data, where the alternative is not a cheaper API but no AI at all.
- High, steady volume, such as processing every document, call or ticket, where per-token bills grow faster than a fixed GPU cost.
- Enterprise sales, where "your data never leaves our infrastructure" shortens security reviews and wins deals.
A practical tip: smaller models that fit on one GPU often handle most routine business tasks. Matching the model to the task, instead of picking the largest one, is where most savings come from.
Case study: private AI in healthcare, with no patient data leaving the environment
A US healthcare startup came to Cybergizer with a common problem. Caregivers had to document every visit by hand, and the company could not send patient conversations to a public AI service.
What we built. Caregivers dictate notes by voice. AI turns the recording into structured content and generates patient reports, vitals and medication logs automatically.
How we kept it private. Every model runs inside the client's own environment:
- Speech recognition: Whisper, an open speech-to-text model, hosted locally.
- Language model: an open instruct model on NVIDIA GPUs, served through vLLM, a high-performance engine for running LLMs.
- Application: Ruby on Rails and React on top, with PostgreSQL for data.
The result for the business. The product is HIPAA-compliant by design: no protected health information reaches a third-party AI vendor, so there is no extra BAA to negotiate and no vendor data policy to monitor. Caregivers spend their time with patients instead of keyboards, and the company can say "your data never leaves our infrastructure" to every clinic it sells to.
We applied the same approach for an open-source AI company. We built an OpenAI-compatible API layer for locally run models, with function calling, vision and routing between several models. For a business, this means one important thing: your product can switch between a private model and a commercial one without being rewritten.
Which option is right for you? A 7-question checklist
Answer these before you pick a vendor or a model:
- What data will the AI see? Personal data, health records, financials, source code or trade secrets point to option 2 or 3.
- Which regulations apply? GDPR, HIPAA, PCI DSS, sector rules, or clients' own security requirements.
- Where must data be stored? If the answer is "only in the EU" or "only on our servers", that narrows the field quickly.
- How hard is the task? Extraction, search and summaries run well on open models. Complex multi-step reasoning may need a frontier model.
- What is the volume? Hundreds of requests a day favor pay-per-use. Every document, call or ticket in the company favors fixed infrastructure.
- Do your clients ask about AI in security reviews? If yes, private AI is a sales argument, not only a cost.
- Can you switch models later? Insist on an architecture that is not locked to one vendor.
How to start in four steps
- Map your data. List the processes you want to automate and the data each one touches. This takes days, not months.
- Pick one high-value use case. Document processing, internal knowledge search or call summaries are good first candidates.
- Test 2–3 models on your real data. Compare a private open model with a frontier API on quality, speed and cost before committing.
- Launch, measure, expand. Start with one team, track time saved and errors, then roll out.
FAQ
What is private AI? Private AI is a way of using large language models where your company controls where data is processed and stored. Usually this means running open-weight models on your own servers or cloud account, so sensitive data never reaches a third-party AI vendor.
Can I use ChatGPT without sharing my company data? Partly. OpenAI's API does not train on business data by default, but it keeps logs for up to 30 days unless you are approved for Zero Data Retention. For full control, use a model hosted in your own environment, or route only non-sensitive tasks to ChatGPT.
Are open-source LLMs good enough for business? For most focused tasks, yes: transcription, document extraction, classification, search and summarization. For the hardest reasoning tasks, frontier commercial models still lead, which is why many companies use a hybrid setup.
Is a self-hosted LLM GDPR compliant? It can be, and it is often the easiest route. Hosting in the EU removes international transfer issues, and you control retention and deletion. You still need a lawful basis, access controls and documentation.
How much does it cost to run a private LLM? Renting one H100 GPU around the clock costs roughly $1,450–$3,300 per month on specialized GPU clouds and $5,000–$9,000 on hyperscalers (October 2026 prices), plus setup and maintenance.
How long does it take to launch? A focused pilot on one use case typically takes a few weeks. Production rollout depends on integrations and compliance requirements.
Talk to brainiacs, not just coders
Cybergizer builds AI integrations for companies that cannot afford to leak their data, from local speech recognition in healthcare to OpenAI-compatible platforms for private models. We start by understanding your business case, then choose the simplest setup that meets it.
Book a free 30-minute consultation and we will help you map your data, pick the right deployment option and estimate real costs.
Sources
- OpenAI API: Your data
- IBM: 2025 Cost of a Data Breach Report: Navigating the AI rush
- Simon Willison: OpenAI no longer has to preserve all ChatGPT data (Oct 2025)
- Orrick: EU AI Act Update: Digital Omnibus Finalizes 8 Compliance Changes (Jul 2026)
- Kingy AI: Best open-weight AI models in 2026
- IntuitionLabs: H100 rental prices compared (Oct 2026)