ChatGPT Is More Than a Model: LLM Systems Engineering Through a Single Conversation

Table of Contents
Mention ChatGPT, and most people picture a massive Transformer neural network. In production, however, no standalone model can serve a billion users on its own. A cluster containing only model weights and GPUs cannot handle cross-region authentication, high-concurrency session reads and writes, tenant privacy isolation, sandboxed tool execution, or safe version rollouts.
Production capability depends as much on distributed data and privacy architecture, GPU inference scheduling, and model engineering (MLOps and LLMOps) as it does on the model weights themselves. As a systems engineering guide for developers and architects, this article follows a single “retail demand forecasting analysis” request from start to finish—drawing on OpenAI’s disclosed practices around PostgreSQL, Habitat, data privacy, and Prompt Caching—to trace how these subsystems fit together into a continuous operating loop and inform enterprise AI platform design.
1. Full-Stack Architecture from a Single Request #
Whether someone uses ChatGPT, Claude, DeepSeek, Kimi, or Qwen, the surface interaction looks almost identical: open a browser or app, type a prompt, and watch words stream across the screen a few seconds later. For a complex task, the assistant may also search the web, inspect uploaded files, run Python code, and synthesize a final report (this article focuses primarily on ChatGPT because OpenAI has published the most detailed engineering disclosures).
From the user’s perspective, the interaction has only three steps:
- Submit a prompt.
- Wait for processing.
- Read the response.
Under the hood, the software and machine learning workflow is far more involved. Suppose a user asks:
Analyze a set of retail demand forecasting experiments, compare TimesFM, LightGBM, and XGBoost, and explain how each model behaves under promotions, seasonality, and intermittent demand.
To answer this request, ChatGPT may authenticate the user’s identity, read uploaded datasets, assemble model context from conversation history and saved memories, invoke a Python execution sandbox, analyze experimental outputs, and stream text and charts back to the client. Meanwhile, users and enterprise architects naturally ask another set of questions: Where are account settings and chat histories stored? Do uploaded business files enter model training? What happens on the server after a user clicks “Delete chat”?
Connecting these questions highlights a central engineering problem: How does ChatGPT combine distributed software architecture, user data and privacy governance, and large-model inference into an AI system that teams can continuously train, deploy, and operate?
1.1 Three Subsystems and the Cloud Ecosystem #
Answering that question requires breaking production ChatGPT into three interconnected subsystems with distinct engineering goals:
- Distributed Application & Data System: Manages user accounts, chat histories, permissions, privacy compliance, tool execution, and service reliability.
- LLM Inference System: Executes model computation on GPU accelerators to transform input context into output tokens.
- Model Engineering System (MLOps / LLMOps): Governs pretraining, post-training, evaluation, deployment, monitoring, and continuous updates.
OpenAI has not released the complete production source code or blueprint for ChatGPT. This article relies on two categories of material: implementations confirmed in OpenAI’s official engineering blog posts, API documentation, and privacy help center articles; and a reference architecture constructed from those disclosures and standard distributed-systems principles. Where internal details remain undisclosed, this article distinguishes confirmed facts from engineering inference.
Before examining the architecture diagram, it helps to clarify a common question: How do OpenAI, Microsoft, NVIDIA, and Oracle divide responsibilities, and why does Figure 1 use Microsoft Azure as its reference cloud platform? Official announcements show that ChatGPT’s infrastructure is not built by a single vendor; instead, it combines a primary cloud and data foundation, a GPU hardware stack, and multi-cloud compute expansion:
- Microsoft Azure (Strategic Cloud Partner and Disclosed Online Database Foundation): Microsoft has served as OpenAI’s strategic cloud partner since 2019. Under their updated October 2025 agreement, OpenAI’s stateless API services remain exclusive to Azure (while first-party products have multi-cloud flexibility)[1]; OpenAI previously disclosed scaling a research Kubernetes cluster on Azure to 7,500 nodes in 2021[2], and within ChatGPT’s online production stack, confirmed Azure components include the Azure Database for PostgreSQL deployment serving 800 million users[3] and the Azure Cosmos DB fleet backing OpenAI’s unified storage platform Habitat[4][5]. That is why Figure 1 uses the Azure ecosystem (mapping container orchestration, gateways, and search onto AKS, API Management, and AI Search) as its reference architecture foundation.
- NVIDIA (GPU Supercomputing Hardware and Acceleration Stack): From earlier A100, H100, and H200 clusters to Blackwell and Vera Rubin systems, ChatGPT’s training and inference compute relies on NVIDIA GPUs, NVLink and InfiniBand interconnects, and the CUDA software stack. In September 2025, OpenAI and NVIDIA announced a strategic partnership to deploy at least 10 gigawatts (GW) of NVIDIA systems[6].
- Oracle and AWS (Multi-Cloud Compute Expansion): As compute demand outgrew the power and data center capacity of any single cloud provider, OpenAI partnered with Oracle and AWS under the Stargate initiative—including a 4.5 GW capacity agreement with Oracle[7][8]. One common misconception deserves clarification: although Oracle is widely known for its commercial relational database and collaborates with OpenAI to integrate AI models into Oracle enterprise applications, within ChatGPT’s own infrastructure Oracle primarily supplies Oracle Cloud Infrastructure (OCI) supercomputing clusters packed with NVIDIA GPUs to expand training and inference capacity[7]. ChatGPT’s online operational databases remain built on Azure PostgreSQL and Azure Cosmos DB[3][4].

Figure 1: Full-stack ChatGPT reference architecture and MLOps closed loop built on the Azure cloud ecosystem (labels 1–6 on the left mark the six logical tiers; solid lines denote online request and real-time data flows, dashed lines denote model and configuration release control flows, and dotted lines denote telemetry, logging, and feedback loops).
As shown across Layers 1–6 in Figure 1, client channels handle user interaction, the distributed application and data tiers (Layers 2–4) orchestrate requests and persist state, the GPU inference tier (Layer 5) executes compute-intensive token generation, and the MLOps tier (Layer 6) spans online and offline boundaries to manage model and runtime lifecycles. Beyond confirmed components such as PostgreSQL, Cosmos DB, and GPU clusters, the remaining Azure managed services in Figure 1 (such as AKS, API Management, Blob Storage, AI Search, and Data Factory) serve as functional reference mappings; internally, OpenAI also operates its custom Habitat storage platform and Valkey caching layer.
While this layered split resembles conventional web architecture, large language models introduce distinct physical constraints. A typical web API request finishes in milliseconds on CPUs; an LLM request may activate billions of parameters across multiple GPUs, and its output length is unknown when the request arrives. Consequently, engineers must optimize web throughput, database connection pools, and network latency alongside GPU memory capacity, token generation speed, batch scheduling, and model version control.
1.2 Streaming Responses and Server Orchestration #
ChatGPT clients—whether web browsers, mobile apps, or desktop applications—manage interaction state, establish authenticated connections, upload messages and attachments, and progressively render text, code, charts, and tool execution status.
A defining client-side experience is the streaming response. Whereas traditional HTTP APIs return only after computation finishes, an autoregressive LLM can push tokens to the client as soon as they are generated. OpenAI’s public API exposes this capability via incremental server-sent events[9]. If a complete answer takes 20 seconds, a blocking call leaves the user staring at a spinner for 20 seconds, whereas streaming delivers the opening sentence almost immediately. Time to first token, however, is not task completion time: for complex agent workflows, multiple rounds of reasoning, web retrieval, and tool execution may occur before the final answer begins streaming. Moreover, public API support for a given transport protocol does not imply that every internal ChatGPT client uses the exact same transport implementation.
On the server side, a typical request passes through six logical steps (production services may fetch data in parallel, run tasks asynchronously, or persist state at different checkpoints):
- Client Request: The client submits user input and attachments.
- Authentication & Admission Control: The gateway verifies identity, workspace permissions, and rate limits.
- Conversation & Context Assembly: Application services fetch conversation history, long-term memories, and user settings.
- Model Selection & Routing: The router selects the target model variant and an appropriate compute node.
- Inference Service & Tool Execution: The GPU cluster runs model inference; if the model emits a tool call, an external sandbox executes the tool and feeds results back into the inference loop.
- Streaming Response & Client Rendering: The server streams generated tokens back to the client for rendering.
Thus, the backend does far more than forward a raw prompt string. It assembles system and developer instructions, current user input, historical dialogue turns, saved memories, multimodal file references, tool schemas, and security permissions into the active model context. Keeping the architecture clear requires distinguishing three concepts that are frequently conflated:
| Concept | Role | Primary Challenge |
|---|---|---|
| Conversation History | Persist user dialogues | Consistency, ACLs, recovery |
| Model Context | Input fed to current inference | Window length, relevance, cost |
| KV Cache | Reuse attention key/value states | GPU VRAM, hit rate, latency |
Chat history is not stored inside model weights, and persisted conversation records are not the same thing as a GPU’s KV cache. That distinction is the starting point for understanding ChatGPT’s data platform and privacy guarantees.
2. Online Data Platform and Privacy Boundaries #
Focusing solely on the Transformer architecture makes it easy to overlook the distributed data platform and privacy engineering behind ChatGPT. User login, conversation persistence, memory management, permission checks, and cross-region access all depend on massive online storage infrastructure. For a multi-cloud, multi-region system, the hardest problem is rarely replicating stateless web containers—it is governing relational databases, hundreds of petabytes of document storage, global caches, and regulatory compliance boundaries.
2.1 PostgreSQL and the Habitat Storage Foundation #
OpenAI has publicly detailed two foundational storage systems: PostgreSQL for strongly consistent transactional workloads, and Habitat, a unified online storage platform built for planetary-scale state access.
In January 2026, OpenAI published Scaling PostgreSQL to power 800 million ChatGPT users, explaining how a relational database supports traffic from 800 million users[3]. The disclosed architecture uses a single Azure PostgreSQL primary node for transactional writes paired with nearly 50 read replicas distributed across multiple regions, serving millions of queries per second[3]. Sustaining that scale required coordinated governance across the entire access path:
- Connection Pooling: PgBouncer pools database connections so thousands of application containers do not exhaust database resources through repeated connection handshakes[3].
- Cross-Region Read Replicas and Caching: Nearly 50 read replicas and a caching tier absorb the vast majority of high-frequency read traffic[3].
- Cache Leasing Against Stampedes: When a hot cache key misses, cache leasing acquires a lock so only a single worker queries the database and repopulates the cache, preventing concurrent requests from overwhelming PostgreSQL[3].
- Workload Partitioning by Access Pattern: High-write workloads that partition cleanly by key were migrated off PostgreSQL onto Azure Cosmos DB[3].
This design also illustrates a classic distributed-systems failure mode: without timeouts, rate limiting, circuit breakers, and retry budgets, a cache miss that slows the database can trigger unthrottled upstream retries—creating a cascading loop of cache miss → higher database load → request timeout → automatic retry → deeper overload.
As product traffic continued to grow, OpenAI published a two-part series in September and October 2026 detailing Habitat, its unified online storage platform layered over Azure Cosmos DB and other backing stores[4][5]. According to OpenAI, products powered by Habitat serve over 1 billion weekly users, process more than 70 million storage requests per second (note that this measures storage-layer operations, not LLM inference requests), manage over 500 PB of data, and span nearly 40 geographic regions[4].

Figure 2: OpenAI online data platform (PostgreSQL + Habitat) and user data privacy architecture constructed from OpenAI’s engineering posts and privacy documentation (solid lines denote synchronous online reads/writes, dashed lines denote asynchronous event streams and cache invalidation, and red dashed lines mark privacy isolation and the 30-day hard deletion boundary).
As shown in Figure 2, Habitat’s evolution reflects three pragmatic architectural decisions:
- From Client Library to Standalone Platform Service: Habitat began as a Python client library wrapping Cosmos DB calls. As services multiplied, scattered connection management, authorization, serialization, and caching logic became difficult to upgrade or debug consistently. OpenAI evolved Habitat into an independent platform service that centralizes multi-tenant isolation, hot-key throttling, and request priority scheduling[4][5].
- Rewriting in Rust Only After Profiling Real Bottlenecks: OpenAI initially wrote Habitat in Python to maximize iteration speed, scaling the Python service to over 20 million requests per second at peak. Only after the service logic stabilized and CPU and memory overhead became primary bottlenecks did the team rewrite Habitat in Rust in Q2 2026, improving CPU efficiency by roughly 6× and memory efficiency by roughly 15×[4].
- Cutting Cross-Region Latency with Local Caching and Selective Replication: Because compute clusters and storage regions do not always sit in the same location, cross-region network round-trips initially dominated read latency. By broadly replicating only small, critical datasets such as authentication state, deploying regional Valkey caches alongside compute nodes, and using the Cosmos DB Change Feed for sub-second global cache invalidation, OpenAI reduced Habitat’s global median read latency from roughly 120 ms to about 10 ms[5]. End-to-end responsiveness in an AI product depends on storage locality and call-chain depth, not just GPU speed.
2.2 Account Settings, Chat History, and Memory #
Understanding how PostgreSQL and Habitat divide work makes it straightforward to answer a core user question: How are account settings, chat histories, and memories stored?
Different categories of user data exhibit very different read/write frequencies, consistency requirements, and payload sizes. The table below synthesizes OpenAI’s disclosed storage architecture with engineering inferences drawn from product behavior (explicitly labeling each row’s basis)[3][4][5]:
| Data Category | Typical Payload | Storage Tier | Basis |
|---|---|---|---|
| Account & ACLs | Identity, billing, workspaces | PostgreSQL (strong ACID) | Disclosed[3] |
| Chats & Prefs | Message tree, instructions, memory | Habitat / Cosmos DB | Disclosed + Inferred[3][4] |
| Multimodal Files | Docs, images, code outputs | Blob / Object Storage | Inferred |
| Hot Online State | Auth sessions, active state | Regional Valkey Cache | Disclosed[5] |
The first tier covers account and organizational configuration. Login credentials, subscription tiers (Free, Plus, Pro, Enterprise), workspace membership roles, and billing state require strict transactional consistency. They reside in relational databases such as PostgreSQL, backed by global read replicas and caches for low-latency authentication checks[3].
The second tier covers chat history and personalization state. OpenAI has disclosed that high-concurrency, key-partitioned workloads run on Azure Cosmos DB behind Habitat, which handles serialization and multi-tenant isolation[3][4][5]. Based on product behavior, sidebar conversation lists, multi-turn transcripts, Custom Instructions, and Saved Memories belong to this partitioned document tier. Furthermore, because ChatGPT lets users regenerate responses or edit an earlier prompt to fork a conversation branch, a conversation is engineered not as a flat append-only text file, but as a directed message tree linking parent and child message nodes.
The third tier covers large multimodal files (engineering inference). In the retail forecasting scenario, uploaded CSV tables, PDF reports, and charts generated by Python tools are far larger than text turns. Following standard cloud-native design, raw file bytes reside in cloud object storage (such as Azure Blob Storage), while conversation documents store only file reference IDs, metadata, and access control lists.
Users also frequently ask whether ChatGPT “remembers” their profession by writing chat transcripts into the model’s neural weights. The engineering answer is no—but it requires separating online inference from offline training: during live online conversations, user inputs never update the weights of the serving LLM in real time (whether historical transcripts are later used in offline training runs depends on the account type and privacy controls discussed in Section 2.4). Instead, cross-session memory relies on structured context injection maintained by the application and storage layers[10]:
- Custom Instructions: Background profile details and response preferences explicitly entered by the user in settings.
- Saved Memories: Concise factual notes distilled during conversations (for example, “The user works with Python and retail demand forecasting”) and stored as discrete entries in the user’s profile store.
- Reference Chat History: When enabled, the context assembly service retrieves relevant snippets or insights from past conversations and injects them into the current request’s Model Context.
This mechanism creates two practical privacy nuances that users should keep in mind[10]:
- Turning off Memory does not delete existing memories: Disabling the toggle in
Settings -> Personalization -> Memorystops ChatGPT from creating or referencing memories, but leaves previously saved memory entries intact until deleted. - Deleting a chat thread does not automatically erase distilled memory items: If ChatGPT extracted a saved memory from last week’s conversation, deleting that conversation from the sidebar does not remove the entry already written to the Saved Memories list; users must delete the memory entry in settings or ask ChatGPT to forget it[10].
2.3 Data Retention, Deletion, and Temporary Chat #
When users want explicit control over data lifecycles, ChatGPT enforces multi-stage retention and deletion rules[11][12][13]:
- Regular and Archived Chats: Saved under the user’s account until manually deleted. Archiving merely hides a thread from the main sidebar list; it does not delete data or shorten retention[11].
- Deleted Chats and the 30-Day Purge Window: When a user deletes a conversation, it disappears from the UI immediately (and cannot be recovered via support) and is permanently purged from OpenAI’s systems within 30 days[11]. Official documentation notes two exceptions: data that has already been de-identified and dissociated from the account before deletion, and data held under security investigations or binding legal preservation orders (for example, during the 2025 New York Times copyright litigation, a court issued a temporary preservation order requiring isolated retention of certain output logs; that broad preservation requirement ended in late September 2025, restoring the standard 30-day deletion pipeline for new deletion requests)[11][14].
- Temporary Chat: Sessions started in Temporary Chat do not appear in history, do not read or create Saved Memories, and are never used to train models. The server retains a copy for up to 30 days solely for safety and abuse monitoring before permanent purge[12]. In our retail demand forecasting scenario, an analyst using a personal account to test a sanitized script without polluting Saved Memories or contributing to model training can switch on Temporary Chat.
- Standalone Files, Data Export, and Account Deletion: Files saved to a user’s Library or Space are decoupled from individual chat threads, so deleting a chat does not delete independently archived files unless removed from the file view[11]. Users can also request a full data export archive or submit an irreversible account deletion request via
Settings -> Data Controlsor the OpenAI Privacy Portal[13].
2.4 Encryption and Model Training Boundaries #
OpenAI structures its security and compliance posture around three boundaries: cryptographic protection, training data isolation, and regional data residency.
First is encryption in transit and at rest. According to OpenAI’s enterprise privacy documentation, traffic between clients, internal services, and storage layers uses TLS 1.2+, while data at rest is encrypted with AES-256. Enterprise customers with strict governance requirements can enable Enterprise Key Management (EKM) to control encryption keys in their own Key Management Service (KMS), backed by SOC 2 Type II attestation[15]. Within the storage platform itself, OpenAI’s Habitat engineering posts confirm that the Habitat service layer handles serialization and encryption before writing payloads to underlying storage[4].
Second is the default boundary governing whether user data enters model training, which is the sharpest distinction between consumer and commercial accounts[13][15][16]:
| Account / Mode | Default Training | Control & Retention | Compliance & Residency |
|---|---|---|---|
| Consumer | Yes | Opt out anytime in settings | Standard global cloud |
| Temporary Chat | No | Per-session, ≤30d abuse log | Safety audit only |
| Commercial | No | Admin retention & RBAC | Residency, SOC 2, EKM |
| API Platform | No | Org controls, Zero Retention | Residency & audit logs |
Note: Consumer accounts include Free, Plus, and Pro; commercial accounts include ChatGPT Team, Enterprise, and Edu. Policies summarized above reflect OpenAI’s official documentation as of November 2026 and may vary by product update or jurisdiction[13][15][16].
For consumer accounts, “Improve the model for everyone” is enabled by default, but users can turn it off at any time in Data Controls so subsequent conversations are excluded from training pipelines[13]. For data permitted to enter training runs, OpenAI applies automated privacy filters (such as the OpenAI Privacy Filter) and deduplication to strip personally identifiable information (PII), combined with post-training alignment to refuse requests for private personal data. Human review is restricted to a small number of authorized, audited personnel operating under least-privilege access for abuse investigation, user-submitted feedback, legal compliance, or quality evaluation of non-opted-out consumer conversations[16].
Third is Data Residency. For organizations governed by GDPR or sector-specific regulations, ChatGPT Enterprise, Edu, and the API Platform allow administrators to pin data at rest to designated regions (including the European Union, the United Kingdom, the United States, Japan, Singapore, and Australia), with higher tiers also supporting in-region GPU inference (Inference Residency) so both storage and compute remain inside the specified geographic boundary[15]. For enterprises operating in mainland China, compliance boundaries are governed by the Personal Information Protection Law (PIPL), the Data Security Law, the Provisions on Promoting and Regulating Cross-Border Data Flows, and the Interim Measures for the Management of Generative Artificial Intelligence Services: transferring important data or personal information abroad above statutory volume thresholds (where not covered by regulatory exemptions) requires a cross-border security assessment, certification, or standard contract filing; meanwhile, generative AI services offered to the public in mainland China with public-opinion or social-mobilization attributes require algorithm filing and generative AI service registration (internal enterprise R&D or back-office deployments that do not serve the public face different requirements). Consequently, regulated financial, public-sector, and healthcare organizations in China typically deploy retail or operational analytics agents via domestic compliant cloud APIs or private on-premises clusters.
3. GPU Inference Clusters and Agent Workflows #
Once a request passes authentication, fetches conversation state and file metadata, and assembles its Model Context, it enters the second major subsystem: the GPU inference cluster and tool execution environment.
Whereas conventional web services process short, independent requests, an autoregressive LLM predicts one token at a time conditioned on the input context and all previously generated tokens. That sequential dependency ties inference performance simultaneously to compute density, GPU memory capacity, sequence length, and external tool orchestration.
3.1 Prefill, Decode, KV Cache, and Parallelism #
As shown in Layer 5 of Figure 1 (GPU Inference / LLM Serving), a single forward pass of an autoregressive Transformer splits into two computationally distinct phases: Prefill and Decode.
The Prefill phase processes the input context. If a user submits a 20,000-token prompt, the model computes hidden representations and attention states across all input tokens in parallel, making Prefill heavily compute-bound. The Decode phase then generates output tokens one step at a time; for token \(y_t\), the model estimates:
\[ P(y_t\mid x,y_1,\ldots,y_{t-1}) \]where \(x\) denotes the input context and \(y_1,\ldots,y_{t-1}\) denote the tokens generated so far. Because each token depends on its predecessors, adding more GPUs cannot make a single output sequence generate all its tokens in parallel (techniques such as speculative decoding can verify multiple draft tokens per step, but do not remove the underlying sequential dependency). Decode is therefore dominated by memory bandwidth and repeated reads of cached attention states (memory-bandwidth-bound).
To avoid recomputing Key and Value projections for the entire history at every Decode step, the engine stores intermediate attention tensors in the KV Cache. For a Transformer using standard Multi-Head Attention (MHA) or Grouped-Query Attention (GQA), theoretical KV cache memory consumption is given by:
\[ M_{\mathrm{KV}} = 2 \times L \times B \times S \times H_{\mathrm{KV}} \times D \times b \]where \(L\) is the number of Transformer layers, \(B\) is the batch size, \(S\) is the cached sequence length in tokens, \(H_{\mathrm{KV}}\) is the number of KV heads, \(D\) is the head dimension, \(b\) is the bytes per element, and the factor of 2 accounts for Key and Value tensors. Consider a model configuration with the following parameters:
| Parameter | Value |
|---|---|
| Transformer Layers | 48 |
| Batch Size | 16 |
| Context Length | 8,192 |
| KV Heads | 8 |
| Head Dimension | 128 |
| Precision | BF16, 2 Bytes |
Its theoretical KV cache footprint is:
\[ M_{\mathrm{KV}} = 2\times48\times16\times8192\times8\times128\times2\ \mathrm{B} \approx 24\ \mathrm{GiB} \]This 24 GiB figure covers just 16 concurrent sequences of 8K tokens—about 1.5 GiB per sequence—and grows linearly with both batch size and context length. A quick comparison puts that number in perspective: a 48-layer dense model with roughly 30B parameters requires about 60 GiB for its BF16 weights (or about 30 GiB when quantized to FP8). In other words, the KV cache for just 16 concurrent 8K requests (24 GiB) already rivals the quantized model weights. If context length expands to 64K or concurrency doubles, KV cache memory rapidly exceeds the size of the model weights themselves, becoming the primary bottleneck governing maximum concurrency per node.
To serve models with tens of billions to trillions of parameters within finite HBM capacity and bandwidth, modern inference engines rely on two groups of systems techniques:
- Paged Memory Management and State Compression: PagedAttention, introduced by vLLM, borrows virtual memory paging from operating systems to allocate KV cache in fixed-size blocks and eliminate external memory fragmentation[17]. Production deployments further apply FP8 or INT4 quantization to weights and KV cache tensors, while architectures such as DeepSeek use Multi-head Latent Attention (MLA) to compress KV states structurally.
- Multi-GPU Parallelism (TP / PP / EP): When a single GPU cannot hold model weights and active KV caches, Tensor Parallelism (TP) shards individual matrix multiplications across GPUs connected by high-bandwidth NVLink, while Pipeline Parallelism (PP) partitions layers across stages. For Mixture-of-Experts (MoE) architectures—where each token activates only a sparse subset of expert sub-networks—clusters combine TP with Expert Parallelism (EP) to distribute experts across nodes, scaling total parameter capacity without proportionally increasing per-token FLOPs.
3.2 Prefix Caching, Continuous Batching, and PD Disaggregation #
Whereas the KV cache reuses attention state within a single request, Prompt Caching reuses KV states across requests that share a common prefix. In our retail forecasting workflow, system instructions, tool definitions, and uploaded dataset schemas repeat across multiple turns. OpenAI’s Prompt Caching documentation notes that the platform reuses the actual KV tensors of matching prefixes rather than merely caching text: caching activates automatically for prompts of 1,024 tokens or more, routes requests by prefix hash to compute nodes likely holding the cached state (with an optional prompt_cache_key hint), enforces TTL expiration, and strictly isolates caches by organization[18].
At the batch scheduler level, concurrent requests vary widely in output length (request A may generate 20 tokens, request B 200 tokens, and request C 1,000 tokens). Under static batching, where all sequences in a batch must finish before the next batch starts, long requests stall short ones. Continuous Batching uses iteration-level scheduling: as soon as a sequence emits its end-of-sequence token, it exits the batch at the next token step and a waiting request takes its slot, substantially improving GPU utilization and throughput under mixed-length workloads[19].
Even with continuous batching, inserting a new 30,000-token prompt directly into a decoding worker causes a compute-heavy Prefill spike that stalls ongoing Decode steps for other requests in the batch. Modern serving engines address this with Chunked Prefill, splitting long input prompts into smaller chunks interleaved with Decode steps. Large-scale clusters—including architectures disclosed by Kimi (Mooncake) and DeepSeek—go further with Prefill / Decode (PD) disaggregation, separating compute-bound Prefill workers from memory-bandwidth-bound Decode workers into distinct resource pools and streaming KV cache blocks over high-speed RDMA networks so each pool can optimize independently for Time to First Token (TTFT) and Time per Output Token (TPOT).
Engineering teams evaluate an inference cluster by tracking several metrics simultaneously:
| Metric | Meaning |
|---|---|
| TTFT | Time to First Token latency |
| TPOT | Time per Output Token interval |
| E2E Latency | End-to-end request latency |
| Tokens/s | Token generation throughput |
| Queue Time | Scheduler queuing delay |
| GPU Utilization | Accelerator compute utilization |
| KV Cache Utilization | Active KV memory occupancy |
TTFT and TPOT capture user-perceived responsiveness, while throughput and utilization measure platform unit economics. Note that while Continuous Batching, PagedAttention, and PD disaggregation represent standard design patterns in modern LLM serving, citing them does not imply that every internal OpenAI model runs an unmodified open-source engine.
3.3 From Single-Pass Inference to Agent Orchestration #
When a model can search the web, execute Python code, and query enterprise connectors, a conversation is no longer a single forward pass. Instead, it becomes the multi-step distributed workflow shown across Layers 3 and 4 of Figure 1, where an Agent Orchestrator and Tool Router loop between GPU inference and isolated tool sandboxes.
In the retail demand forecasting task, the model may first inspect CSV column headers, run statistical tests in Python, render diagnostic plots, and finally synthesize a written summary. A single user prompt thus triggers multiple model invocations while execution state spans both the LLM context window and external container environments.
If an agent invokes an external API that mutates state—such as updating a database record, sending an email, or triggering a procurement workflow—and encounters a network timeout, the orchestrator faces classic distributed-systems challenges around idempotency, state recovery, and compensating transactions: Did the write take effect? Can the step be retried safely without duplicate side effects? Does the action require explicit human confirmation? Agent orchestration cannot be reduced to a while loop calling an LLM API; a production-grade agent system embeds non-deterministic model decisions inside a constrained, observable, and recoverable distributed workflow.
OpenAI’s ChatGPT Agent System Card also highlights the security risks introduced by external environment access, particularly Prompt Injection[20]. When an agent reads web pages, uploaded documents, or third-party connector responses, untrusted content may contain adversarial instructions attempting to hijack model behavior or exfiltrate private context. Untrusted external data must never inherit the privilege level of system instructions, and high-impact tool actions require strict authorization boundaries, sandbox isolation, and user confirmation gates.
4. From Model Training to Dual-Layer LLMOps #
Chapters 2 and 3 traced the online lifecycle of our retail forecasting request as it authenticated, read state from PostgreSQL and Habitat, and executed GPU inference and sandboxed tools. Behind that online path sits the third major subsystem: the continuous engineering pipeline (MLOps and LLMOps) that governs how models and application configurations reach production safely. Note that OpenAI has not open-sourced its internal training and release orchestration platform; this chapter combines published research papers, official OpenAI incident post-mortems, and industry-standard MLOps/LLMOps practices.
4.1 Pretraining and Post-Training #
As illustrated in Layer 6 of Figure 1 (GPT Model Training & MLOps), delivering a foundation model forms an end-to-end closed loop spanning data ingestion, cleaning and curation, distributed pretraining, post-training alignment, evaluation and red-teaming, model registration, canary rollout, and production telemetry feedback.
Pretraining builds a model’s language representations, world knowledge, and base predictive capabilities over massive corpora. For autoregressive language modeling, the standard objective minimizes expected negative log-likelihood over next tokens:
\[ \theta^* = \arg\min_{\theta} \mathbb{E}_{x\sim D} \left[ -\sum_t\log P_\theta(x_t\mid x_{\lt t}) \right] \]where \(\theta\) denotes the model parameters, \(D\) is the pretraining data distribution, and \(x_{\lt t}\) represents the sequence of context tokens preceding position \(t\). Engineering challenges at this stage center on petabyte-scale data filtering and throughput, multi-dimensional distributed training parallelism, gradient synchronization, automated hardware fault recovery, fast checkpointing, and cluster Goodput.
Post-training shapes instruction following, multi-step reasoning, tool use, and safety alignment. Common techniques include Supervised Fine-Tuning (SFT), preference optimization, and Reinforcement Learning (RL). OpenAI’s early InstructGPT paper demonstrated how Reinforcement Learning from Human Feedback (RLHF) substantially improved instruction following[21], though modern frontier models use evolving post-training recipes that go well beyond any single published paper.
4.2 Joint Versioning and Canary Rollouts #
In traditional machine learning, deploying a model often means registering a serialized weight artifact in a Model Registry and exposing a prediction endpoint. In an LLM system, however, online behavior is determined jointly by six interacting layers:
| Layer | Behavioral Impact |
|---|---|
| Model Weights | Reasoning capability, tone, accuracy |
| Inference Runtime | Latency, VRAM usage, numerical precision |
| Serving Config | Batch size, cache routing, timeouts |
| System Prompt | Instruction adherence and persona |
| Tool Schema | Function arguments and calling policy |
| Application Code | Context assembly, ACLs, workflow state |
Consequently, an online quality regression does not require a weight update—changing a few lines of a System Prompt or modifying a tool’s JSON Schema can shift agent behavior dramatically even when model weights remain frozen. Production LLMOps must therefore place model weights, inference runtime, serving configuration, System Prompts, tool schemas, and application code under joint version control, backed by domain-specific regression suites (in our retail analytics example, testing whether the assistant catches temporal train/test leakage, distinguishes statistical forecast error from inventory decision loss, and recovers gracefully when a Python tool throws an exception).
A real production rollback at OpenAI in late April 2025 illustrates how tightly these layers interact. OpenAI rolled out an update to GPT-4o in ChatGPT aimed at improving personality and responsiveness, but changes in post-training reward signals that over-weighted short-term user thumbs-up/thumbs-down feedback—combined with concurrent candidate updates such as memory—weakened anti-sycophancy guardrails. The deployed model became excessively flattering and agreeable (sycophancy), even validating unreasonable or impulsive user beliefs[22]. Although the candidate had passed existing pre-launch offline evaluations, OpenAI had to roll back the GPT-4o update within days and subsequently overhauled its offline blocking gates, long-term interaction evaluations, and progressive rollout checks[22].
Even when a candidate version scores higher on offline benchmarks, teams cannot cut over 100% of production traffic at once. If a new checkpoint improves reasoning accuracy by 5% but increases per-request cost by 40%, drops tool-schema compliance by 3%, and inflates p95 latency, an unguarded cutover causes a severe production incident. Engineers must evaluate quality, safety, latency, reliability, and cost together through a staged canary deployment pipeline:
- Offline Evaluation: Validate the release candidate against general benchmarks and domain-specific regression suites.
- Staging Verification: Run end-to-end integration and schema contract tests in a pre-production environment.
- Canary Traffic: Route a small percentage of live traffic to the new version.
- Quality, Safety & SLO Gates: Monitor infrastructure metrics alongside semantic response quality; trigger an immediate rollback if any gate breaches its threshold.
- Progressive Rollout: Gradually ramp traffic proportions as statistical confidence accumulates.
- Production Cutover: Complete the rollout and transition to continuous online monitoring.
One operational rule stands out above all others: a healthy service does not guarantee a correct answer. Even when GPUs report zero out-of-memory errors, HTTP endpoints return 200 OK, and latency stays well within SLOs, the model may still hallucinate facts or pass invalid arguments to a tool. LLM observability must pair infrastructure telemetry with continuous semantic quality sampling.
4.3 Foundation Model MLOps vs. Application LLMOps #
Most enterprises also need to draw a clear strategic boundary between Foundation Model MLOps and Application LLMOps, which operate on very different lifecycles:
| Dimension | Foundation Model MLOps | Application LLMOps |
|---|---|---|
| Core Artifact | Model weights & training runs | Prompts, RAG, tools, agents |
| Primary Data | Pretraining & alignment corpora | Enterprise docs, eval sets |
| Release Unit | Checkpoint, runtime, serving config | App code, prompts, tool schemas |
| Key Metrics | Capability, safety, cluster efficiency | Task success, quality, latency, cost |
| Primary Risks | Training divergence, eval regression | Tool errors, ACL leaks, hallucinations |
| Typical Teams | Research / ML Systems / Infra | MLE / Backend / Platform / Product |
Foundation Model MLOps focuses on multi-thousand-GPU training runs, alignment research, and low-level kernel optimization—activities with high capital cost and long iteration cycles. Application LLMOps, by contrast, holds foundation weights fixed and iterates rapidly through context engineering, Retrieval-Augmented Generation (RAG), tool orchestration, permission enforcement, and business evaluation suites. The two lifecycles couple at the model version boundary: every upstream model upgrade requires downstream application teams to re-verify prompt contracts and tool-calling behavior. For the vast majority of organizations that do not pretrain foundation models from scratch—and instead consume capabilities via commercial APIs or open-weight checkpoints—whether an AI product delivers reliable business value depends primarily on the maturity of its Application LLMOps engineering.
5. System Trade-Offs and Enterprise AI Platforms #
Stepping back across the previous four chapters shows that the complexity of a production AI system comes not only from the depth of each component, but from inherent conflicts among subsystem optimization goals.
5.1 Four Core Tensions in Production AI Systems #
Application services optimize for low latency and high availability, data platforms optimize for cache hit rates and compliance isolation, GPU clusters optimize for token throughput and hardware utilization, and MLOps teams optimize for answer quality and safety. Those local objectives create four fundamental engineering tensions:
| Tension | Objective A | Objective B | Balancing Mechanism |
|---|---|---|---|
| Throughput vs. Latency | Larger batches | Shorter queue & TTFT | SLO-capped dynamic batch |
| Cache vs. Load | Route to cached GPU | Spread load evenly | Load-aware prefix routing |
| Locality vs. Compute | Store near users | GPUs bound by power | Small-state sync + cache |
| Quality vs. Cost | Larger models & CoT | Cap cost & latency | Tiered model routing |
- Throughput vs. Latency: Increasing inference batch size raises total cluster tokens per second, but once GPUs approach saturation, larger batches lengthen scheduler queue time and widen per-token decode intervals (TPOT). Teams cannot judge an inference cluster by GPU utilization alone; they must maximize throughput subject to strict TTFT and TPOT service-level objectives (SLOs).
- Cache Affinity vs. Load Balancing: A naive load balancer distributes requests evenly across idle workers, whereas Prompt Caching wants requests sharing a prefix to land on the specific GPU node holding that KV cache. Pure round-robin destroys prefix cache hit rates and forces redundant Prefill computation; rigid affinity overloads hot nodes. Production routers must balance node load watermarks against expected cache savings.
- Data Locality vs. Compute Capacity: Storing data close to users or inside regulatory jurisdictions reduces read latency and satisfies Data Residency mandates, but power-hungry GPU superclusters cannot be deployed uniformly in every region. OpenAI’s Habitat architecture balances these physical constraints by combining broad replication for small authentication records, regional Valkey caches, sub-second Change Feed invalidation, and targeted workspace residency[5][15].
- Model Quality vs. System Cost: Larger frontier models or extended reasoning chains improve accuracy on hard tasks but multiply token cost and end-to-end latency. In a retail forecasting workflow, extracting CSV column names or formatting fields only requires a fast, lightweight model, whereas diagnosing residual errors across promotions, seasonality, and intermittent demand warrants routing to a frontier reasoning model. In production, accuracy gains must justify their marginal operational cost—the best production model is rarely the highest-scoring offline checkpoint, but the one delivering the highest net business value within latency, safety, and budget constraints.
5.2 Architectural Comparison of Leading LLM Systems #
Widening the lens from ChatGPT to Google Gemini and Anthropic Claude shows that although each provider builds on different cloud foundations and silicon, their publicly disclosed system capabilities follow a shared layered logic[1][6][15][18][23][24][25][26]:
| Disclosed Dimension | ChatGPT | Gemini | Claude |
|---|---|---|---|
| Cloud Partners | Azure + OCI / AWS | Google Cloud | AWS + GCP |
| Disclosed Silicon | NVIDIA GPU clusters | Google TPU + GPU | Trainium + TPU + GPU |
| Prefix Caching | Auto Prompt Cache | Implicit + Explicit | Breakpoint Cache |
| Tools & Protocols | Sandbox / Responses | Vertex Extensions | MCP / Computer Use |
| Enterprise Privacy | No training + EKM | No training + CMEK | No training + isolated |
Note: The table above summarizes only dimensions confirmed in official documentation and technical reports as of November 2026 (undisclosed internal online session databases are omitted). Google Gemini is trained and served natively on GCP AI Hypercomputer infrastructure using custom Google TPUs (such as TPU v5p and 6th-gen Trillium) and offers both implicit and explicit Context Caching alongside Customer-Managed Encryption Keys (CMEK) on Vertex AI[23][24]; Anthropic confirmed in October 2025 that it operates a three-platform compute strategy spanning AWS Trainium, Google Cloud TPUs, and NVIDIA GPUs, and open-sourced the Model Context Protocol (MCP)[25][26].
Whether a provider vertically integrates custom silicon on a single cloud (Gemini) or orchestrates heterogeneous accelerators across clouds (ChatGPT and Claude), every frontier AI platform solves the same core systems problems: distributed databases and caches for session state, KV prefix caching to eliminate redundant Prefill, sandboxes and standardized protocols for tool execution, and tenant/key isolation for enterprise compliance. Understanding that shared blueprint does not mean every company should build a 500-petabyte Habitat platform or a 10,000-GPU cluster; rather, the key lesson from frontier labs is to decouple layers and evolve infrastructure around verified bottlenecks.
5.3 Three Principles for Enterprise AI Platforms #
Consider an enterprise building a retail demand forecasting and business analytics agent. That organization does not need to pretrain a foundation language model from scratch; its real engineering challenge is connecting internal databases, existing forecasting models (such as LightGBM or TimesFM), Python execution sandboxes, and business approval workflows so permissions remain enforced, analytical conclusions stay verifiable, and upstream model upgrades do not break production operations.
Enterprises with existing data and machine learning infrastructure can apply the layered decoupling shown in Figure 1 through three practical engineering principles:
- Decouple Business Logic from Models via a Model Gateway: Application code should never hard-code a single vendor’s proprietary SDK or pack all business rules into a monolithic prompt. A unified Model Gateway centralizes routing, rate limiting, audit logging, fallback policies, and multi-model switching while keeping tool execution, permission checks, and workflow state in a controlled orchestration tier. Whether the backend connects to OpenAI, Anthropic, and Gemini or to leading open and commercial models such as DeepSeek, Qwen, and Kimi, a clean gateway contract lets teams route tasks by complexity and compliance without rewriting application logic.
- Weigh Managed APIs Against Self-Hosted GPU Clusters Pragmatically: When an organization’s competitive advantage lies in proprietary domain data and workflows—and commercial enterprise APIs already satisfy “no training on customer data, TLS/AES encryption, and regional data residency”—starting with managed APIs lets engineers focus on high-leverage domain evaluation and agent reliability. Investing in a self-hosted inference platform built on vLLM, SGLang, or NVIDIA Triton Inference Server (for instance, serving open-weight Qwen or DeepSeek models) is warranted only when strict air-gapped regulations, custom fine-tuned weights, high-volume cost curves, or tail-latency requirements justify the operational burden. Teams deploying on non-NVIDIA accelerators (such as Huawei Ascend clusters in domestic financial or public-sector environments) should also evaluate runtime compatibility layers such as
vLLM-Ascendand MindIE behind the gateway. - Unify Governance Standards Rather Than Forcing a Single Runtime: Do not force every AI workload into the same execution engine. Batch tabular forecasting with LightGBM does not need a GPU token scheduler, and an interactive conversational agent should not run on an offline batch pipeline. An enterprise AI platform should standardize identity, data security, experiment tracking, evaluation gates, and observability at the governance layer while allowing classical ML, RAG retrieval, and LLM inference to run on the compute engines best suited to each workload:
| Capability Layer | Candidate Enterprise Reference Stack |
|---|---|
| Web & Service APIs | React, FastAPI, Go |
| Agent & Workflow Orchestration | Temporal, LangGraph, custom state machines |
| Relational DB & Caching | PostgreSQL, Redis / Valkey |
| Model Gateway & Self-Hosted Serving | Internal Model Gateway, vLLM, SGLang, NVIDIA Triton, vLLM-Ascend / MindIE |
| Containers & Delivery | Kubernetes, GitHub Actions, Argo CD |
| Experiments, Observability & Evals | MLflow, OpenTelemetry, Prometheus, Grafana, domain eval suites |
Note: The table above lists representative open-source and managed technologies enterprises can evaluate when building an internal AI platform; it does not imply that OpenAI uses every listed tool internally.
6. Conclusion: AI Engineering Lives at System Boundaries #
Returning to our opening retail demand forecasting prompt, we can combine the publicly disclosed components and reference architecture discussed above into an illustrative end-to-end walkthrough (note that the sequence below is a pedagogical synthesis based on public disclosures and our reference architecture, not an undisclosed internal OpenAI call trace):
- Admission and State Retrieval (synchronous online path): The client connects over TLS through edge and authentication layers, verifying identity and quota against a PostgreSQL read replica or local cache[3]; application services then query Habitat (where global median read latency has been reduced to roughly 10 ms[5]) and object storage to retrieve conversation branches, Saved Memories, and uploaded retail CSV metadata to assemble the active Model Context.
- GPU Inference and Tool Execution Loop (compute and sandbox path): The router dispatches the request by prefix hash and load watermark to a GPU cluster, reusing cached KV states where possible[18], running Prefill, and entering the continuous batching decode loop to stream tokens back to the client[9][19]. When the model emits a Python tool call, the orchestrator executes the script inside an isolated sandbox container to compare TimesFM and LightGBM error distributions, appends the output to context, and starts the next generation turn.
- Persistence, Privacy Governance, and MLOps Feedback (asynchronous and background path): Once the response finishes, session state is serialized, encrypted, and written to backing storage while the Cosmos DB Change Feed invalidates cross-region caches within a second[4][5]. Background pipelines enforce account-specific training isolation, PII scrubbing, and the 30-day deletion window[11][15][16], while sampled quality evaluations and telemetry flow back into MLOps / LLMOps gates to guide the next canary release.
OpenAI’s disclosures around PostgreSQL, Habitat, and data privacy underscore a foundational engineering truth: even with frontier foundation models and massive GPU superclusters, relational databases, partitioned document stores, regional caches, connection pools, encryption, and strict data-control boundaries remain the bedrock of product reliability and user trust. At the same time, the memory bottlenecks and non-deterministic outputs of autoregressive LLMs force traditional distributed systems engineering and machine learning engineering to merge at the system boundary.
Ultimately, the defining capability of production AI engineering is never just running a single model faster in isolation—it is mastering the couplings and constraints across all three subsystems: continuously finding the system-level optimum across task quality, interactive latency, cluster throughput, operational reliability, privacy security, and total cost.
References #
[1] OpenAI. The Next Chapter of the Microsoft–OpenAI Partnership. OpenAI, October 28, 2025. https://openai.com/index/next-chapter-of-microsoft-openai-partnership/
[2] OpenAI. Scaling Kubernetes to 7,500 Nodes. OpenAI Research, January 25, 2021. https://openai.com/index/scaling-kubernetes-to-7500-nodes/
[3] Zhang, B. Scaling PostgreSQL to Power 800 Million ChatGPT Users. OpenAI Engineering, January 22, 2026. https://openai.com/index/scaling-postgresql/
[4] Lee, J., Yu, C., & Ries, B. Rapidly Scaling Online Storage to Serve Over 1 Billion ChatGPT Users (Part I). OpenAI Engineering, September 11, 2026. https://openai.com/index/scaling-storage-one-billion-users-part-one/
[5] Lee, J., Hakobyan, H., & Yu, C. Rapidly Scaling Online Storage to Serve Over 1 Billion ChatGPT Users (Part II). OpenAI Engineering, October 23, 2026. https://openai.com/index/scaling-storage-one-billion-users-part-two/
[6] OpenAI. OpenAI and NVIDIA Announce Strategic Partnership to Deploy 10 Gigawatts of NVIDIA Systems. OpenAI, September 22, 2025. https://openai.com/index/openai-and-nvidia-announce-strategic-partnership-to-deploy-10-gigawatts-of-nvidia-systems/
[7] OpenAI. Stargate Advances with 4.5 GW Partnership with Oracle. OpenAI, July 22, 2025. https://openai.com/index/stargate-advances-with-partnership-with-oracle/
[8] OpenAI. AWS and OpenAI Announce Multi-year Strategic Partnership. OpenAI, November 3, 2025. https://openai.com/index/aws-and-openai-partnership/
[9] OpenAI. Streaming API Responses. OpenAI API Documentation. https://developers.openai.com/api/docs/guides/streaming-responses
[10] OpenAI. Memory FAQ. OpenAI Help Center. https://help.openai.com/en/articles/8590148-memory-faq
[11] OpenAI. Chat and File Retention Policies in ChatGPT. OpenAI Help Center. https://help.openai.com/en/articles/8983130-chat-and-file-retention-policies-in-chatgpt
[12] OpenAI. Temporary Chat FAQ. OpenAI Help Center. https://help.openai.com/en/articles/8914046-temporary-chat-faq
[13] OpenAI. Data Controls FAQ. OpenAI Help Center. https://help.openai.com/en/articles/7730893-data-controls-faq
[14] OpenAI. Addressing the New York Times’ Data Demands. OpenAI, 2025. https://openai.com/index/response-to-nyt-data-demands/
[15] OpenAI. Enterprise Privacy at OpenAI. OpenAI. https://openai.com/enterprise-privacy/
[16] OpenAI. How Your Data Is Used to Improve Model Performance. OpenAI Help Center. https://help.openai.com/en/articles/5722486-how-your-data-is-used-to-improve-model-performance
[17] Kwon, W., et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP, 2023. https://arxiv.org/abs/2309.06180
[18] OpenAI. Prompt Caching. OpenAI API Documentation. https://developers.openai.com/api/docs/guides/prompt-caching
[19] Yu, G.-I., Jeong, J. S., Kim, G.-W., Kim, S., & Chun, B.-G. Orca: A Distributed Serving System for Transformer-Based Generative Models. OSDI, 2022. https://www.usenix.org/conference/osdi22/presentation/yu
[20] OpenAI. ChatGPT Agent System Card. 2025. https://cdn.openai.com/pdf/839e66fc-602c-48bf-81d3-b21eacc3459d/chatgpt_agent_system_card.pdf
[21] Ouyang, L., et al. Training Language Models to Follow Instructions with Human Feedback. NeurIPS, 2022. https://arxiv.org/abs/2203.02155
[22] OpenAI. Sycophancy in GPT-4o: What Happened and What We’re Doing About It. OpenAI, April 29, 2025. https://openai.com/index/sycophancy-in-gpt-4o/
[23] Google DeepMind. Gemini 4 Argon Model Card; Gemini Team. Gemini: A Family of Highly Capable Multimodal Models. 2023–2026. https://deepmind.google/models/model-cards/gemini-4-argon/; https://arxiv.org/abs/2312.11805
[24] Google Cloud. Trillium (TPU v6e), Vertex AI Context Caching & Data Governance Documentation. Google Cloud, 2025–2026. https://cloud.google.com/tpu/docs/v6e; https://cloud.google.com/vertex-ai/generative-ai/docs/context-cache/context-cache-overview; https://cloud.google.com/vertex-ai/generative-ai/docs/data-governance
[25] Anthropic. Expanding Our Use of Google Cloud TPUs and Services. Anthropic News, October 23, 2025. https://www.anthropic.com/news
[26] Anthropic. Prompt Caching with Claude & Model Context Protocol (MCP). Anthropic News, 2024–2025. https://www.anthropic.com/news/prompt-caching; https://www.anthropic.com/news/model-context-protocol