Reference Architecture · Enterprise Model Infrastructure
Inference Fabricv1.0
A dedicated, governed serving estate for operating open-weight models as internal infrastructure — one controlled ingress, inference-aware routing, and capacity that belongs to a workload rather than to whoever asked first.
§1–§10 — why an inference fabric exists, what it buys, what it costs, the major design decisions, and when operating one is the wrong choice. About ten minutes. You can stop at the divider.
§11 onward — serving boundaries, inference routing, model pools, distributed serving, cache management, GPU scheduling, observability, failure handling, and request paths.
§ 1Executive summary
A GPU server running a language model is not an inference platform.
The difference is everything around it: how requests enter, how they find the right accelerator, how capacity is divided between workloads, how model versions move into production, how cache state affects routing, how a failed GPU is removed from service, and how an organization can tell whether the hardware it bought is actually being used efficiently.
This architecture packages those concerns into a dedicated inference fabric. The fabric runs model serving and nothing else. Applications, agents, evaluation systems, and developer tools reach it through one controlled service boundary. Behind that boundary, inference-aware routing selects a model pool and a serving endpoint based on the request, the workload, available capacity, and reusable cache state — rather than treating model servers like interchangeable web replicas.
Large or latency-sensitive deployments may go further and separate prompt processing from token generation, tier KV cache beyond accelerator memory, or schedule different workloads onto different accelerator classes. Those are optimizations beneath the same boundary, not requirements imposed on every deployment.
Models change. The contract around them should not.
1.1At a glance
| What it is | A dedicated platform for serving open-weight models on enterprise accelerator infrastructure |
| Who operates it | An AI infrastructure / platform engineering team |
| Consumers | Applications, agents, RAG systems, evaluation pipelines, batch jobs, and developer tooling |
| Core principle | Model infrastructure is exposed as a service, never as individual GPU endpoints |
| Primary ingress | An enterprise AI Gateway, or another explicitly authorized model-access service |
| Internal routing | Model-, cache-, load-, and workload-aware |
| Serving engines | vLLM, SGLang, TensorRT-LLM, or equivalent |
| Distributed inference | Adopt when workload evidence justifies it; not a baseline requirement |
| Data posture | No direct internet route from serving workloads; model artifacts enter through controlled distribution paths |
| Related architecture | The Enterprise AI Agent Framework may consume this fabric, but is not required to |
§ 2Why this architecture exists
Five problems appear when an organization moves from “we have a GPU server” to “other teams depend on these models.”
2.1Accelerators scale differently from applications
An application replica may start in seconds and cost little while idle. A large model may occupy several accelerators, take substantial time to load, and reserve expensive memory even when nobody is using it.
Treating the two as the same infrastructure produces predictable results: application teams do not understand GPU capacity, infrastructure teams cannot attribute the spend, and model upgrades become application incidents. The fabric gives accelerator capacity its own operating model.
2.2LLM replicas are not interchangeable web servers
A traditional HTTP load balancer can often choose any healthy backend. An inference request is different. One worker may already hold a reusable prefix in its KV cache while another would have to recompute the entire prompt. One may be carrying heavy decode load while another has capacity to spare.
Current distributed-serving systems account for cache locality and active load when selecting workers. Routing is part of inference performance, not merely network plumbing.
SourceNVIDIA Dynamo — KV-aware routing
2.3Different model workloads want different infrastructure
Interactive chat, agent tool loops, embeddings, reranking, evaluation judges, and long-context generation do not share a latency profile. Putting them all behind one undifferentiated pool lets one workload degrade another and makes capacity planning almost meaningless.
The fabric separates workloads where the evidence says they should be separated — and only there.
2.4Model distribution can damage the service it is trying to scale
Large weights have to move somewhere before a replica becomes useful. If every scale event pulls hundreds of gigabytes across the same network carrying inference traffic, adding capacity can temporarily make production slower.
Artifact distribution, local caching, and controlled promotion are therefore part of the serving architecture, not an afterthought in a deployment script.
2.5Hardware health is application health
A degraded accelerator can remain technically reachable while producing thermal throttling, memory errors, or deteriorating latency. GPU health cannot be inferred from HTTP status codes. It has to be measured directly and incorporated into operations.
SourceNVIDIA DCGM exporter — accelerator telemetry in Prometheus form
None of these problems is solved by choosing a better model. A reference architecture should survive replacing the model, the serving engine, and eventually the accelerator generation. This one is designed to do exactly that.
§ 3The architecture at a glance
Four layers, and one boundary that matters more than all of them.
| Layer | Responsibility |
|---|---|
| Access boundary | Receives authorized model traffic from outside the fabric |
| Inference routing | Selects the model, the workload pool, and the serving endpoint |
| Serving layer | Runs the inference engines and manages model execution |
| Accelerator estate | Provides GPU or other accelerator capacity, high-speed networking, and local storage |
Supporting them are model-artifact storage, cache infrastructure, telemetry, and the orchestration control plane. The boundary that matters is the one above the fabric.
§ 4The rules of the fabric
Six invariants carry most of the architecture. Every structural choice in the technical reference exists to make one of them enforceable.
| # | Rule | What it removes |
|---|---|---|
| 1 | There is one model-serving ingress | An unauthenticated, unmetered path to a model worker |
| 2 | The external contract is independent of the serving engine | Applications coupled to this year's inference stack |
| 3 | Routing is inference-aware | Cache thrashing and tail latency from round-robin balancing |
| 4 | Workloads receive deliberate capacity | One workload silently degrading another |
| 5 | Scaling must not degrade live inference | Scale events that consume the network requests are using |
| 6 | The hardware is observable directly | Degraded accelerators that still return HTTP 200 |
The six rules in full, with rationale
1There is one model-serving ingress
Consumers do not address model servers directly. This creates one enforceable boundary for authentication, routing, overload protection, and service-level policy.
It also avoids relying on an inference engine's built-in authentication as the security perimeter. Current vLLM documentation notes that its API-key option does not protect every inference-capable endpoint, and recommends placing the server behind an appropriate proxy when hardening the service.
SourcevLLM OpenAI-compatible server — API key caveats
2The external contract is independent of the serving engine
Where practical, the service exposes a standard application-facing API — OpenAI-compatible chat, responses, or embedding semantics. vLLM currently exposes OpenAI-compatible chat, completions, responses, and embeddings APIs, and NVIDIA Dynamo provides an OpenAI-compatible frontend across supported backends.
Applications should not care whether the implementation underneath is vLLM today and something else next year.
SourceNVIDIA Dynamo documentation
3Routing is inference-aware
At meaningful scale, worker selection considers more than replica count. Cache locality, current token load, request characteristics, model availability, and workload policy may all influence placement.
WhyThis is the single highest-leverage configuration decision in the fabric. A request steered to a worker that already holds the relevant prefix skips work the other worker would have to redo from scratch.
4Workloads receive deliberate capacity
An agentic tool loop should not accidentally compete with a large batch-generation workload simply because both happen to use the same base model. Pools are capacity and performance boundaries, not naming conventions.
5Scaling must not degrade live inference
Weight distribution, cache warming, and node initialization are controlled so that adding capacity does not consume the network or storage resources needed by requests already in flight.
6The hardware is observable directly
GPU memory, utilization, errors, thermals, serving queues, cache state, and token latency are first-class operational signals. An HTTP 200 response is not proof that the accelerator estate is healthy.
§ 5The design decisions
Six decisions determine everything else, each stated with what it buys and what it costs.
1Give inference its own estate
BenefitCapacity, upgrades, networking, security, and cost scale with model demand rather than application demand. A serving-engine rollout cannot evict an unrelated enterprise service, and an application deployment cannot consume accelerator capacity by accident.
CostAnother infrastructure estate exists and needs its own operational practice. That is intentional — expensive hardware should not be hidden inside a general-purpose cluster merely to reduce the cluster count.
2Keep application policy outside the fabric
BenefitThe upstream AI Gateway owns caller identity, model permission, budgets, data-loss controls, and selection between local and external providers. The Inference Gateway owns scheduling. Inference infrastructure can then evolve without becoming the enterprise policy engine.
CostEvery local model request crosses two logical gateways.
3Prefer inference-aware routing over generic load balancing
BenefitBetter cache reuse, better load distribution, and better tail latency. Kubernetes' InferencePool and Endpoint Picker model exists specifically because generative-model endpoints expose scheduling information that normal service load balancing does not use.
CostThe routing layer now holds model-specific state and must itself be operated as critical infrastructure.
4Separate workloads before separating models
BenefitThe useful unit is not always “one pool per model.” The same model may need one deployment optimized for interactive agent traffic and another for high-throughput batch generation. Pool by the behavior the business needs from the model.
CostMore deployments to size, upgrade, and monitor — which is why a pool should exist only when two workloads measurably want different things.
5Treat prefill/decode disaggregation as an optimization
BenefitPrefill is compute-bound and decode is memory-bandwidth-bound, and both llm-d and Dynamo support placing those phases on different workers so each scales on its own curve.
CostThat does not mean every deployment should split them. For short prompts, smaller models, or requests with strong local prefix-cache reuse, an aggregated worker may be faster and considerably simpler. Disaggregate only when benchmarks demonstrate the need (§16).
6Give model servers no ambient internet path
BenefitProduction inference workloads receive their images, model weights, and dependencies from controlled internal sources. They do not fetch arbitrary packages or phone home at runtime.
CostAn internal mirror and artifact pipeline to operate. The model-serving estate should be boring from a network-security perspective, and boring has a maintenance cost.
SourcesGateway API Inference Extension · llm-d prefill/decode disaggregation · Dynamo disaggregated serving
§ 6One inference request, end to end
Consider an internal agent requesting a completion from a locally hosted model.
- The application calls the AI Gateway using the caller's authenticated context.
- The gateway verifies permission — that the caller may use the requested logical model — and checks any applicable quota, budget, or data policy.
- The gateway selects the local fabric and forwards the request to its single ingress.
- The Inference Gateway resolves the logical model to an eligible workload pool.
- The scheduler selects an endpoint using the information available for that deployment — current load, prefix-cache affinity, model capability, and endpoint health.
- The serving engine executes the request. In an aggregated deployment one worker performs both prefill and decode; in a disaggregated deployment the scheduler may select separate workers and coordinate transfer of KV state.
- Tokens stream back through the same path.
- The fabric records time to first token, inter-token latency, token counts, queueing, cache behavior, serving version, accelerator placement, and hardware health.
The consumer sees a model API. The infrastructure team sees everything required to explain how that model API behaved — which is the entire point of putting a boundary there rather than handing out endpoints.
§ 7What the architecture guarantees
Assuming the boundaries are implemented as specified, with a pointer to the part of the technical reference that covers each one.
| Concern | Architectural outcome | Specified in |
|---|---|---|
| Model access | Serving workers are not directly addressable by ordinary application workloads | §11 |
| Engine portability | Consumers depend on the fabric contract rather than on one serving implementation | §12 |
| Workload isolation | Capacity can be assigned independently to latency-sensitive, throughput-oriented, and specialized workloads | §15 |
| Routing efficiency | Worker placement uses inference-specific state instead of generic round-robin balancing | §14 |
| Accelerator isolation | Model workloads operate on a dedicated infrastructure boundary | §11 |
| Artifact control | Production weights and images enter through governed distribution paths | §18 |
| Hardware visibility | Accelerator failures and degradation are observed directly rather than inferred | §20 |
| Cost attribution | Usage can be measured by consumer, model, workload pool, and accelerator class | §20 |
| Failure containment | A model rollout or GPU-node failure does not require changing consuming applications | §21 |
§ 8What this costs to operate
Self-hosting replaces a model-provider invoice with an infrastructure practice. It does not make inference free.
8.1What the enterprise has to own
| Capability | Requirement |
|---|---|
| Accelerators | GPU or equivalent compute, lifecycle management, and a spare-capacity strategy |
| Fabric networking | Sufficient bandwidth and topology for the chosen model and serving mode |
| Serving | One or more supported inference engines |
| Routing | An inference-aware gateway and scheduler |
| Model distribution | Internal artifact path, weight storage, and node-local caching strategy |
| Orchestration | Kubernetes or another platform capable of accelerator-aware placement |
| Telemetry | Model-server, request, and hardware-level metrics |
| Release management | Versioned model promotion, rollback, and compatibility testing |
| Capacity planning | Forecasting against latency, throughput, concurrency, and availability targets |
8.2The ledger
What it buys
- Capacity and model placement the organization owns, rather than buying every token from a third party
- Sensitive prompts that remain inside the controlled estate
- Applications insulated from serving-engine churn
- Model workloads tuned against the actual jobs they perform
- Infrastructure spend that is measurable rather than disappearing into API invoices or shared hardware
- Predictable model availability, on your own upgrade calendar
What it costs
- Accelerators purchased or reserved before they are used
- Idle capacity that is still a bill
- Driver, firmware, serving-engine, and model upgrades as your responsibility
- Redundancy that does no productive work some of the time
- A failed capacity model producing either poor service or very expensive headroom
- A team operating the fabric that does not disappear after launch
8.3The economic question
The relevant comparison is not GPU purchase price versus API token price. It is the fully loaded cost of reliable internal inference at the utilization and availability the organization can actually achieve, versus the equivalent external service.
That calculation should include accelerator depreciation or lease cost, networking, storage, power, redundancy, support, engineering labor, and the opportunity cost of reserved but unused capacity. A utilization assumption is the number most likely to be wrong, and the one the whole case rests on.
§ 9Open decisions
These should be resolved from workload evidence rather than from architectural preference. Each has a stated default that applies if nobody decides.
1Serving engine
vLLM is the default starting point for a Kubernetes-oriented open serving stack, because of its ecosystem and its integration with projects such as llm-d. SGLang and TensorRT-LLM remain credible alternatives, and NVIDIA Dynamo can coordinate several engine backends rather than requiring a permanent choice.
2Distributed inference layer
llm-d and NVIDIA Dynamo represent two strong but different approaches. The architecture requires neither. Choose after benchmarking the models and traffic shapes the organization actually intends to run.
3Prefill/decode topology
Aggregated by default. Disaggregate where long prompt processing, model size, or concurrency demonstrates measurable interference and the fabric has suitable KV-transfer bandwidth.
4Accelerator interconnect
Use the topology required by the model parallelism and KV-transfer strategy. RDMA-capable fabrics become particularly important for cross-node disaggregated inference. The architecture intentionally does not impose a universal link speed.
5Cache hierarchy
Local accelerator cache first. Introduce CPU, NVMe, or remote cache tiers only where measured reuse justifies the additional state and operations.
6Scheduling and quota
Standard Kubernetes placement may be sufficient for simple serving. Topology-aware scheduling and quota systems such as Kueue become useful when multiple teams, accelerator classes, or tightly coupled multi-node workloads share the estate.
7Availability target
N+1 capacity sounds inexpensive until N represents several high-end GPUs. Define the service objective before defining the redundancy. If the objective survives losing a rack or a site, the Multi-Cluster Inference Fabric specifies what that costs.
SourcesDynamo · llm-d disaggregation · Kueue
§ 10When to build this — and when not to
Probably excessive if
- model traffic is small or intermittent;
- hosted APIs satisfy the organization's data requirements;
- there is no existing GPU operations capability;
- only one or two models are required;
- utilization will remain low;
- a temporary provider outage is cheaper than maintaining redundant accelerator capacity.
Appropriate when
- model traffic is large enough that internal capacity is economically defensible;
- prompts or completions must remain within the controlled environment;
- predictable model availability matters;
- multiple internal teams need shared inference;
- workload-specific tuning materially affects user experience;
- open-weight models are a strategic capability rather than an experiment.
The fabric is modular. A team can begin with one model server behind a controlled gateway and add inference-aware routing, specialized pools, cache tiers, and distributed serving as evidence justifies each one.
Complexity should arrive behind demand, not ahead of it.
Technical reference
The remainder specifies how the guarantees above are implemented. It is written for platform, infrastructure, and SRE teams, and assumes the vocabulary those teams already use. Nothing below changes the architecture described in §1–§10.
§ 11Trust boundary and topology
The fabric should be treated as a restricted compute estate rather than another application namespace.
| Zone | Purpose |
|---|---|
| Upstream services | AI Gateway and explicitly authorized internal consumers |
| Fabric ingress | Inference Gateway, TLS, and request admission |
| Serving | Inference scheduler and model-server workloads |
| Accelerator nodes | GPU compute, high-speed intra- and inter-node fabric |
| Artifact plane | Internal image registry, weight store, and local caches |
| Operations | Orchestration control plane, observability, and privileged administration |
The important denied flows are more useful than the allowed ones:
- ordinary applications do not connect to model-server pods
- accelerator workloads do not have unrestricted internet egress
- production nodes do not pull arbitrary model weights directly from public registries
- administrative access does not share the application request path
§ 12The service contract
The fabric exposes logical models rather than physical deployments. A caller asks for something like reasoning-large, agent-fast, or embedding-default, and the fabric decides which implementation currently satisfies that contract.
Where compatible with the workload, use an OpenAI-compatible request surface so consumers are not coupled to a particular serving engine.
Two engines can accept the same request body and still behave differently. Model capabilities, tool-calling behavior, context limits, and structured-output support require explicit platform metadata and validation — a logical model that silently loses tool calling on its next physical implementation is an outage that looks like a model regression.
§ 13Model-serving layer
A reference implementation begins with vLLM. Current releases provide OpenAI-compatible endpoints for chat, completions, responses, and embeddings, and expose a substantial serving configuration surface.
Alternative engines may be selected per workload. The rule is that engine-specific configuration remains behind the fabric boundary: tensor parallelism, context limits, quantization, memory utilization, and batching parameters are pool-level implementation details rather than application concerns.
§ 14Inference Gateway and routing
The Kubernetes InferencePool and Endpoint Picker model is the right abstraction even if the concrete implementation changes. An InferencePool identifies eligible serving endpoints; an endpoint picker or inference scheduler chooses among them using model-serving information a normal load balancer does not possess.
| Routing must understand | At larger scale, also |
|---|---|
| Endpoint health | Prefix / KV-cache affinity |
| Current load | Queued or in-flight token load |
| Model compatibility | Workload class |
| Request priority | Adapter availability |
| Disaggregated-serving role |
The scheduler is now on the critical path and must be horizontally available. A single endpoint picker is a single point of failure for every model in the estate — see §21.
§ 15Workload and model pools
A starting taxonomy might include pool-interactive, pool-agentic, pool-batch, pool-embedding, pool-rerank, and pool-judge. These are examples, not required products.
A pool exists when two workloads need measurably different latency, throughput, capacity, or release behavior. Do not create a taxonomy merely because the configuration makes it easy — every pool is capacity that cannot be borrowed by anything else.
The agentic case is the one most often missed. A tool-calling loop is chatty and latency-sensitive in a different way than long-form chat: small context, high concurrency, and a model tuned for repeated structured tool calls. Sizing it separately is the difference between a responsive agent and one that feels broken.
§ 16Disaggregated serving
Prefill processes the prompt. Decode generates the output. Their hardware behavior differs enough that large-scale serving systems can place them on separate workers and coordinate the transfer of KV state between stages.
The decision should be made from benchmarks using realistic input length, output length, concurrency, and cache-hit distributions. Use disaggregation when those costs buy better service, and not before.
Sourcesllm-d disaggregation architecture · Dynamo disaggregated serving
§ 17KV cache and prefix reuse
Prompt computation already performed is valuable state, and reusing it is the largest single win available in serving throughput.
- Local prefix reuse on the serving worker. The first optimization, and the cheapest.
- Routing new requests toward workers likely to hold reusable state. The second, and the one that requires the scheduler to be cache-aware.
- Offloading or sharing cache through CPU memory, local SSD, or a remote tier. Only after the first two are measured.
Cache is an optimization, not durable state. The platform must remain correct after losing all of it. A design that becomes incorrect — rather than merely slower — when a cache tier is flushed has turned an optimization into a dependency.
§ 18Model lifecycle and weight distribution
Models enter production through the same kind of controlled promotion process expected of executable software. Record at least:
| Recorded | Why it is needed |
|---|---|
| Source and license | Legal review, and knowing what may be redistributed |
| Immutable artifact identity | “The same model” must mean the same bytes |
| Tokenizer and chat template | The most common source of silent output regressions |
| Model configuration | Reproducing a deployment from the registry alone |
| Serving-engine compatibility | Knowing which pools can host it |
| Quantization or transformation | Attributing a quality change to a conversion rather than the model |
| Evaluation evidence | The promotion gate |
| Security approval and promotion history | Audit, and rollback targets |
Public registries are sources, not production dependencies. Mirror approved artifacts internally, and pre-stage large weights where practical so that a scale event does not turn into an uncontrolled network benchmark.
§ 19Capacity, scheduling, and scaling
Capacity planning begins with workload metrics, not GPU count. Measure input tokens, output tokens, concurrent sequences, time to first token, inter-token latency, total request latency, queue depth, cache-hit behavior, tokens per second, accelerator memory, and accelerator utilization.
Autoscaling has a different cost profile from stateless application scaling, because loading model weights can be slow and resource intensive. For predictable workloads, scheduled or forecast capacity may outperform purely reactive scaling — a scale-out that completes after the traffic peak has passed is pure cost.
Topology-aware scheduling and quota systems become useful once multiple teams, accelerator classes, or tightly coupled multi-node workloads share the estate; gang scheduling matters as soon as a single model spans more than one node.
SourceKueue — topology-aware scheduling and mixed batch/serving management
§ 20Observability and SLOs
At minimum, correlate four layers on one request identity.
| Layer | Correlate |
|---|---|
| Request | Logical model, workload class, consumer, input and output token counts, latency, outcome |
| Scheduler | Selected pool, selected endpoint, queueing, cache estimate, routing reason |
| Serving | Engine version, model version, batch state, cache utilization, generation metrics |
| Hardware | GPU utilization, memory, ECC and XID errors, temperature, throttling state |
20.1Service objectives
The useful objectives are generally some combination of availability, time to first token, inter-token latency, end-to-end latency, throughput, and successful request rate.
WhyAverage latency conceals the failures users notice. Tail latency is the number worth defending.
20.2Operational signals
| Signal | Alert on |
|---|---|
| Requests waiting in the serving queue | Sustained backlog → scale out or shed |
| GPU KV-cache utilization | Approaching saturation → out-of-memory risk |
| Time to first token / per output token, p95 | Service-objective breach |
| Cache hit rate per pool | Collapse → routing regression or prompt-shape change |
| Weight-load duration on scale-out | Growth → artifact path or staging regression |
| GPU ECC / XID errors, thermal throttling | Any occurrence → drain the node before applications notice |
| Per-consumer token volume and cost | Attribution drift, and budget conversations |
§ 21Failure modes and availability
Design explicitly for each of these. The fabric's behavior under failure is the part consumers will actually experience.
| Failure | Designed response |
|---|---|
| Model worker failure | Remove it from routing and retry only where request semantics permit |
| GPU degradation | Drain based on hardware telemetry rather than waiting for application errors |
| Scheduler failure | Run redundant routing infrastructure; inference must not depend on one endpoint picker |
| Weight-store outage | Loaded replicas continue serving; scale-out may pause |
| Network degradation | Disaggregated deployments need particular attention, because KV transfer is now on the inference path |
| Bad model release | Promote gradually and preserve rollback capability |
| Capacity exhaustion | Reject or shed load deliberately rather than letting every request degrade simultaneously |
Losing a rack, a cluster, or a region is a larger question than any row above, and it is answered by topology rather than by configuration — see the Multi-Cluster Inference Fabric reference architecture.
§ 22Request paths
F1Aggregated inference
- Consumer
- AI Gateway
- Inference Gateway
- Scheduler
- Model worker
- Response
The default path. One worker performs prefill and decode, and the KV state never crosses the network.
F2Disaggregated inference
- Consumer
- AI Gateway
- Inference Gateway
- Scheduler
- Prefill worker
- KV transfer
- Decode worker
- Response
Same entry as F1. The scheduler now selects two workers and coordinates state transfer between them, which puts fabric bandwidth on the request's critical path.
F3Model promotion
- Approved artifact source
- Internal artifact store
- Validation / evaluation
- Staged weight distribution
- Canary pool
- Production pool
Model changes ship through the same controlled path as code, and evaluation is the gate rather than a follow-up. Staging the weights before the pool needs them is what keeps promotion from competing with live traffic.
F4Telemetry
- Model workers + GPU nodes + routing infrastructure
- Metrics / traces / logs
- Observability platform
Collection is pulled or pushed from inside the fabric outward. Nothing in the observability path grants a route back into the serving estate.
App. ANetwork rules
The fabric should be expressible almost entirely as a small allowlist.
| Source | Destination | Conditions |
|---|---|---|
| Upstream AI Gateway | Inference Gateway | The only flow entering the fabric |
| Inference Gateway / scheduler | Model workers | Cluster-internal; endpoint selected per §14 |
| Prefill worker | Decode worker | Only when disaggregation is enabled; dedicated fabric |
| Model workers | Internal artifact and cache services | Read-only, and only as explicitly required |
| Observability | Exporters / collectors | Scrape or push; no return path into serving |
| Operations | Orchestration and management endpoints | Separate management path |
Everything else is denied by default
- Application workloads → model-server pods
- Model workers → the internet
- Model workers → public registries or model sources
- Consumers → the Inference Gateway, bypassing the AI Gateway
- Administrative access sharing the application request path
App. BReference implementation
A credible current starting point. Named technologies are implementation candidates — the architecture is the boundary and the behavior, not the product list.
| Capability | Candidate |
|---|---|
| Orchestration | Kubernetes |
| GPU lifecycle | NVIDIA GPU Operator, where NVIDIA hardware is used |
| Primary serving engine | vLLM |
| Alternative engines | SGLang, TensorRT-LLM |
| Inference routing | InferencePool + a production endpoint picker, or the llm-d router |
| Distributed-serving alternative | NVIDIA Dynamo |
| Cache extension | LMCache or equivalent, where justified |
| GPU telemetry | DCGM Exporter |
| Metrics | Prometheus-compatible telemetry |
| Model artifacts | Internal object or parallel storage, plus node-local cache |
| Scheduling | Kueue or Volcano for gang scheduling and quota |
| Delivery | GitOps plus internal image and model mirrors |
Hardware and fabric notes
| Component | Notes |
|---|---|
| GPU nodes | H100 / H200 / B200 class. NVLink or NVSwitch intra-node; InfiniBand or RoCEv2 at 400G+ inter-node |
| Model servers | Tensor parallelism sized per model; maximum context length and memory utilization tuned per pool |
| Weights store | Parallel filesystem, or object store plus local NVMe cache. Scale-up pulls must not saturate the fabric network |
| KV cache tier | Offload to CPU RAM or NVMe, introduced only where measured reuse justifies it |
| Telemetry | Per-GPU utilization, streaming-multiprocessor occupancy, ECC, thermals, and XID errors into the metrics pipeline |
App. CSizing methodology
Do not size from parameter count alone. Capture representative traffic and benchmark the candidate deployment across eight dimensions:
- Model and quantization.
- Input-length distribution.
- Output-length distribution.
- Concurrency.
- Cache-hit rate.
- Latency objective.
- Throughput objective.
- Redundancy requirement.
Then determine the smallest topology that satisfies the target at the expected peak, plus the required failure headroom. Re-run the model whenever any of those assumptions changes — a sizing exercise is a perishable artifact, and the one most often quoted long after it stopped being true.
App. DTerminology
Reference only. Nothing in §1–§10 requires any of it.
- AI Gateway
- the enterprise policy boundary deciding who may call which model, and under what budget and data rules
- Inference Gateway
- the fabric-internal routing layer deciding where an authorized inference request executes
- InferencePool
- a logical collection of compatible serving endpoints
- Prefill
- computation over the input prompt that creates attention / KV state
- Decode
- iterative generation of output tokens
- KV cache
- attention state reused during generation and, where possible, between compatible requests
- Aggregated serving
- one worker performs both prefill and decode
- Disaggregated serving
- separate workers perform those phases and transfer the required state
- Workload pool
- accelerator capacity intentionally configured for a particular service profile
- TTFT
- time to first token — how long before a response starts arriving
- TPOT
- time per output token — how fast it streams once started
- EPP
- endpoint picker — the component that chooses which model server receives a request
- TP / PP
- tensor and pipeline parallelism — the two ways a model is split across GPUs
- RDMA
- remote direct memory access — the low-latency transport used for collectives and KV transfer
- DCGM
- NVIDIA's data-center GPU manager — the source of direct accelerator telemetry
- XID error
- an NVIDIA driver-reported hardware fault; a signal to drain a node rather than retry a request
- Gang scheduling
- placing all workers of a tightly coupled job at once, or not at all
The Enterprise AI Agent Framework defines how autonomous software consumes models, enterprise tools, and execution environments safely. It may use this inference fabric for local models, but the two architectures are independently deployable.