Multi-Cluster Inference Fabricv0.1

§ 1Executive summary

A cluster can fail. A request should never know.

1.1At a glance

§ 2Why this architecture exists

Five problems show up whenever a single GPU inference-fabric cluster moves from a pilot into something the business actually depends on. Each is solved by a structural choice, not by better monitoring of the one cluster you have.

2.1A GPU fabric is a hardware fault domain, not just a software one

A model-serving replica is pinned to specific GPUs, a specific NVLink domain, specific IB rails. When a node drains for a driver update, a rack loses power, or a switch ASIC fails, the replicas on it don't reschedule elsewhere in milliseconds — they are gone until that hardware is back. A platform that only scales within one cluster inherits every hardware failure mode of that cluster as an availability outage.

2.2Maintenance and model promotion need somewhere to fail over to

GPU firmware, driver, and NCCL library upgrades happen on a cadence the calling applications don't control, and they are the highest-risk maintenance a platform team performs. Doing them with zero downtime requires a second fabric to shift traffic to — not a rolling update within one cluster's node pool.

2.3Capacity and blast radius scale differently across an organization

One team's traffic spike — a launch, an incident, a bulk evaluation run — can saturate a shared fabric's queue depth for every other consumer. Multiple independent clusters let capacity be planned, and bursts absorbed, per region, per business unit, or per criticality tier, funded and sized independently of one another.

2.4KV cache and parallelism state do not survive a cluster loss for free

This is the problem that breaks naive multi-region designs. A model server's attention state lives in GPU memory on the exact GPUs serving it; a tensor-parallel group is a set of GPUs synchronized every layer over NVLink. None of that state, and none of that synchronization group, can span a cluster boundary or survive that cluster disappearing. A multi-cluster design has to state, explicitly, what happens to in-flight requests and cache locality on failover — silence on this point is where most multi-region inference designs quietly break in production.

2.5Network economics do not scale the way GPU economics do

DGX nodes get materially faster generation over generation — NVLink, then InfiniBand or RoCE. Wide-area links between racks or sites do not improve at the same rate, and their cost scales with distance and bandwidth in a way GPU procurement doesn't. A design that assumes cross-site bandwidth behaves like intra-rack bandwidth will be both economically and technically wrong the moment a second site is added.

§ 3The architecture at a glance

Four areas, and one rule about how they connect. Everything else in this document is detail beneath this picture.

§ 4What makes a cluster safe to lose

These are the part of the architecture worth remembering. If you retain one thing from this document, it should be that no single cluster is allowed to hold anything the others can't route around.

4.1The concerns they answer

4.2The controls

Fabric Router — governs which cluster serves a request

Every inference request, regardless of caller, resolves to a cluster through one routing layer. Routing considers cluster health, capacity, and — where the routing technology supports it — KV or prefix affinity. Nothing calls a fabric cluster's model-server endpoint directly; that is the single highest-leverage rule in this document, and the one most tempting to bypass for a “quick fix.”

Cluster Autonomy — governs what a cluster is allowed to depend on

A fabric cluster may not depend on any other fabric cluster to serve a request: its own scheduler, its own KV cache tier, its own model pool, its own weights cache. Cross-cluster dependencies are exactly the thing that turns N independent fault domains back into one.

Parallelism Containment — governs where model-parallel state is allowed to live

Tensor parallelism never crosses a node's NVLink domain. No parallel group — tensor, pipeline, data, or expert — ever crosses a cluster boundary. Full treatment in §12; this is the technical center of the document.

Failure-Aware Cache Policy — governs what happens to context locality on failover

Prefix and KV cache locality is optimized for the common case — no failure — and explicitly, deliberately allowed to reset in the uncommon case. That tradeoff is stated in §5 rather than discovered in an incident review.

4.3Engineering guarantees

§ 5The design decisions

Six decisions determine everything else. Five are settled; the sixth is open (§9).

§ 6One request, with a cluster down

A request arrives for a model normally served with a warm prefix cache on Cluster A. Cluster A has just failed a health check.

1. The request reaches the Fabric Router. The router already knows Cluster A is unhealthy from continuous background probing — this request is not the thing that discovers the failure.
2. The router selects a healthy cluster. Cluster B, chosen by capacity and whatever affinity signal is available. No session is pinned to a dead cluster waiting for it to come back.
3. The request executes on Cluster B with a cold prefix cache. Time-to-first-token reflects a full prefill instead of a cache hit — the accepted cost from §5, decision 3, not a bug.
4. Cluster A is removed from rotation, not torn down. A distinction that matters: a transient failure and a planned drain both end in “not receiving traffic,” but only one of them should page anyone.
5. When Cluster A returns healthy, the router reintroduces it gradually. A hard cutover back onto a cold cluster reproduces the same cache-miss spike the failover just caused — reintroduction is ramped, not instant.
6. Every hop is on one correlated trace. The routing decision, the health state that drove it, and the resulting cache-miss cost are all attributable after the fact, not inferred from a latency graph (§16).

§ 7What the architecture guarantees

Stated as outcomes, with a pointer to the part of the technical reference that specifies each one.

§ 8What this costs to operate

Every architectural choice has a price. This section states the one attached to this design.

8.1What the enterprise has to own

8.2The ledger

8.3Where the effort concentrates

Three areas hold most of it. The parallelism and networking decision (§12) — a one-time, high-consequence sizing exercise per model family. The Fabric Router (§13) — the one component with no fully standardized off-the-shelf answer, evaluated like §9. Steady-state headroom cost — not an engineering task at all, but an ongoing budget line that needs the same scrutiny as any other standing capacity reservation.

§ 9Open decisions

One decision gates delivery. The rest can be made after the fabric is standing, and are listed so that nobody mistakes silence for consensus.

9.1The gating decision — cross-cluster routing technology

9.2Decisions that can wait

Six choices are deliberately unmade. None of them changes the shape of the fabric, and each has a stated default that applies if nobody decides.

§ 10When to build this — and when not to

This is a substantial capacity commitment. It is worth being explicit about the conditions under which it is the wrong answer, because most organizations reach those conditions gradually rather than starting at them.

10.1Two lists

10.2Assumptions this design rests on

• DGX-class (or equivalent) GPU nodes, homogeneous within a cluster; heterogeneous across clusters is supportable but not the default case sized here.
• Each cluster has, at minimum, 400G-class InfiniBand or RoCEv2 inter-node fabric with GPUDirect RDMA. A genuinely lower-bandwidth fabric materially changes the cross-node parallelism guidance in §12 and should be re-evaluated against it before this design is applied as-is.
• Clusters are independently reachable — separate control planes, separate network paths to the router — not sharing a single upstream failure domain.
• The router is deployed with its own availability plan, inheriting the edge-service treatment the companion agentic-platform reference gives its AI Gateway; specifying that plan is out of scope here.
• No requirement yet for cross-cluster session or cache consistency stronger than “eventually recovers” — see §14.3 if that assumption changes.

Change any of these — particularly the network-generation assumption — and §12 needs re-deriving before the rest of the document still applies.

§ 11Per-cluster fabric topology

11.1What every cluster must provide — the contract with the router

• An OpenAI-compatible inference endpoint per model pool.
• A health and readiness surface reflecting actual GPU health — DCGM signals, XID errors, thermal state — not just process liveness.
• Cache-aware internal routing (prefix or KV affinity), intra-cluster.
• Independent scaling, with no cross-cluster scaling dependency.
• Exactly one inbound path from the router — the cluster-boundary form of the single-ingress principle a well-built fabric already applies at every zone boundary inside it.

11.2Reference implementation

Named products are the current best answers, not requirements — the fabric's contract with the router is a pool of OpenAI-compatible endpoints behind one gateway; substitutions are local decisions.

11.3Per-cluster HA — one layer below the whole-cluster failure this document defends against

Losing one node inside a cluster should not be a cluster-level event. Ray-based clusters get this from GCS fault tolerance backed by an external, HA-configured Redis (Sentinel); Dynamo or LWS-based clusters get it from the underlying Kubernetes scheduler's normal pod replacement plus per-role health checks. Either way, this is intra-cluster resilience — it is what makes §4's “cluster autonomy” claim actually true, not what makes the fabric multi-cluster. A cluster without this layer is not a safe unit to call a replica.

§ 12Parallelism strategy

The section this document exists to make precise.

12.1The rule

Tensor parallelism stays inside a node. Pipeline, data, and expert parallelism may cross nodes within a cluster. Nothing crosses a cluster boundary. The rule follows directly from the bandwidth gap between the two kinds of interconnect involved, not from a preference: NVLink delivers roughly 600 GB/s to 1.8 TB/s of intra-node bandwidth per GPU depending on generation; a single inter-node NIC on a 400G or 800G fabric delivers roughly 50–100 GB/s per GPU. That gap — not the absolute speed of either — is what determines where tensor parallelism, with its every-layer all-reduce, is allowed to live.

12.2Decision table

12.3When cross-node tensor parallelism is acceptable

Stated plainly: cross-node TP — a tensor-parallel degree greater than one node's GPU count — is a real, supported, production-precedented pattern for dense models too large to serve at any workable precision on a single node. Meta's own Llama 3.1 405B release ships a model-parallel-16 configuration built for exactly this, run over a 400G InfiniBand fabric between two 8-GPU nodes. It is a deliberate latency trade, not a default. NVIDIA SHARP in-network computing on the switch fabric materially reduces the cost of the cross-node all-reduce this requires and should be enabled whenever cross-node TP is used (Appendix B) — but even with SHARP enabled, per-token latency is measurably worse than staying within a node. Reserve it for the case where nothing else fits; try pipeline, data, or expert parallelism first.

12.4The one thing this design never permits

No parallel group — tensor, pipeline, data, or expert — crosses a cluster boundary. A TP or PP group spans, at most, every node in one cluster, over that cluster's own RDMA fabric. It never spans two clusters. This is the constraint that makes §4's cluster-autonomy claim and the use of model parallelism at all simultaneously true — a parallel group split across clusters would make both clusters required for a single request to complete, which defeats the entire premise of this document.

EvidenceA gang-scheduling placement that lands any rank of a TP or PP group outside its assigned cluster is a defect, not a scheduling edge case — treat it as a release blocker, not a performance tuning item.

12.5Benchmark gate

Before finalizing a parallelism plan for a specific model and fabric generation, measure the actual per-hop collective cost on the target hardware. The bandwidth figures in Appendix B are a planning input, not a substitute for nccl-tests on the fabric that will actually carry production traffic.

§ 13Cross-cluster control plane

13.1What the Fabric Router must do

• Health-check every cluster continuously, on the same GPU-health signal §16 defines — never a bare TCP or HTTP liveness check, which can pass while the fleet behind it is degraded.
• Route on capacity and, where the technology supports it, prefix or KV affinity — sending a request to the cluster most likely to already hold its context, while a healthy alternative always exists.
• Fail over transparently — a cluster failing removes it from the routing set without surfacing an error to the caller, beyond the latency cost of a cold cache on the receiving cluster.
• Reintroduce a recovered cluster gradually — a hard cutover back onto a cold cluster reproduces the thundering-herd problem it was built to avoid.
• Never become a cluster-specific dependency itself — deployed independently of every fabric cluster it routes to.

13.2Implementation options

Recapped from §9: Gateway API Inference Extension in its multi-cluster form, Istio Ambient Multicluster, or a dedicated cross-cluster control plane. The router's job is identical across all three — only the mechanism differs, and the gate is which one matches infrastructure already standardized on elsewhere.

13.3Active-active traffic shifting

Capacity-aware routing should shift traffic proportionally toward whichever cluster has headroom, not just binary-fail toward survivors on a hard cluster loss. That proportional behavior is what lets a cluster be intentionally drained for maintenance with no special-cased “maintenance mode” anywhere in the router — a planned drain and an unplanned failure are the same mechanism at different speeds.

§ 14Disaggregated serving and KV cache — cross-cluster implications

14.1The intra-cluster case, recapped

Prefill/decode disaggregation via llm-d or Dynamo, KV transfer via NIXL, tiering via LMCache — established practice, specified in §11.2. Nothing below changes that.

14.2The rule for disaggregation across a cluster boundary: never

A prefill pool and its paired decode pool are, functionally, a single parallel-serving unit connected by a KV transfer path that assumes a shared high-speed fabric. Splitting that pair across clusters reintroduces exactly the cross-cluster dependency §12.4 forbids, on the most latency-sensitive path in the system — KV transfer has to complete inside the token-generation budget. Keep every prefill/decode pair inside one cluster, full stop.

14.3The shared-cache exception, stated precisely

For a specific workload with a very long, frequently reused shared prefix — a large system prompt, a static RAG corpus — where cross-cluster cache-hit rate materially changes cost or latency, a shared LMCache tier, backed by an object store or a cross-cluster cache service, can be adopted deliberately as an addition to, not a replacement for, per-cluster caches. This is opt-in per workload, never a platform default: the added latency and consistency cost lands on every request on that path, whether or not a failover ever happens.

§ 15Orchestration: adopt or build (per cluster)

The per-cluster orchestrator is the one component below the router with no fully settled off-the-shelf answer.

15.1The comparison

15.2The gate

This is not a gating decision the way the serving-layer choice is in the companion agentic-platform reference, because the router's contract with a cluster (§11.1) is identical either way. Pick the option that matches existing operational investment — Ray elsewhere in the stack favors KubeRay; heavy NVIDIA-stack investment and a SHARP-capable fabric already in place favors Dynamo — and revisit only if that contract stops being met.

§ 16Observability and failure signals

16.1Operational metrics

§ 17Request paths

Three runtime paths carry essentially all traffic; the narrative walkthrough in §6 is F2 in full. Read the semantic chain first — the N# references beneath each one are cross-references into the connection matrix in Appendix A.

App. ANetwork matrix

Scoped to what's new in a multi-cluster design. Composes with the companion agentic-platform reference's network matrix for everything inside one cluster — this table only adds the cross-cluster edges.

Explicitly denied

• Cluster → cluster, any protocol, for inference or GPU-fabric traffic (N5)
• Caller → cluster, direct, bypassing the Fabric Router
• Fabric Router holding request state that must survive its own restart — the statelessness requirement from 4.3

App. BHardware & fabric appendix

On SHARP

NVIDIA's Scalable Hierarchical Aggregation and Reduction Protocol offloads part of a collective all-reduce into the switch ASIC itself, meaningfully reducing the cost of any cross-node TP, PP, or EP collective on a Quantum-2 (NDR) or Quantum-X800 (XDR) fabric. Enable it wherever the fabric supports it — it's a switch and firmware configuration, not a topology change, and one of the cheaper levers available for cross-node parallelism performance.

On scale-up versus scale-out

On a standard 8-GPU DGX node, any hop beyond that node's own GPUs is necessarily a scale-out hop over the ConnectX fabric — there is no other path. On a GB200 or GB300 NVL72 rack, up to 72 GPUs across 18 physical trays share one NVLink scale-up domain, and ConnectX in that architecture carries only the scale-out link beyond the rack. This matters directly for §12: on NVL72-class hardware, “cross-node” TP within one rack is still NVLink-speed, and the boundary this document actually cares about is the edge of the NVLink domain — not the edge of a physical server chassis.

On sub-400G fabrics

If the target fabric is genuinely HDR-generation (200 Gbps) rather than NDR or XDR, treat every cross-node TP recommendation in §12 as unavailable by default. Scale via pipeline, data, or expert parallelism only, and do not extend tensor parallelism past one node without benchmarking first (12.5). 400G+ is the practical floor for this document's cross-node TP guidance — this is also why 10.2 lists it as a load-bearing assumption rather than a preference.

App. CTerminology

Reference only. Nothing in §1–§10 requires any of it.

TP / PP / DP / EP

tensor, pipeline, data, and expert parallelism — the four ways a model or its serving load is split across GPUs

NVLink / NVSwitch

scale-up GPU interconnect within a node, or, on NVL72-class hardware, within a rack

InfiniBand / RoCEv2

scale-out network fabric connecting nodes within a cluster

SHARP

Scalable Hierarchical Aggregation and Reduction Protocol — switch-offloaded collective communication

GCS fault tolerance

Ray's mechanism for surviving a head-node failure via an externally backed control store

Fabric Router

this document's term for the cross-cluster routing control plane (§13)

Active-active / active-passive

whether all clusters serve traffic simultaneously, or one stands by unused until called on

TTFT

time to first token — how long before a response starts arriving

TPOT

time per output token — how fast it streams once started

KV cache

the key/value attention state a model server keeps per in-flight request; reusing it across requests with a shared prefix is the largest single win in serving throughput

mTLS

mutual TLS — both sides of a connection present a certificate