Inference Fabric & Agentic Platformv0.4

§ 1Executive summary

The agent can reason freely. It cannot act freely.

1.1At a glance

§ 2Why this architecture exists

Five problems show up whenever an organization moves agents from a pilot into shared production. Each one is solved by a specific structural choice rather than by policy or training, because the failures are not the kind a person can be careful enough to avoid.

2.1Agents expand the security boundary

Traditional applications follow predetermined code paths. Agents decide which tools to use, in what sequence, based on content they read at runtime — a ticket, a document, a web page. That content can be written by someone with an interest in what the agent does next. The architecture limits the consequences by placing authorization outside the agent, in components the model cannot influence.

2.2Model-generated code is an untrusted workload

Agents that are useful for real work will generate and run code. There is no prompt that reliably prevents a model from emitting arbitrary shell or Python, so the useful question is not whether the code is safe but what it can reach. Running it inside the application platform would put a bad generation or a successful prompt injection directly next to everything else the platform runs.

2.3Enterprise AI needs accountability

When an agent modifies a ticket, queries a warehouse, executes code, or spends money on a model call, the organization needs to know who initiated it, what happened, why, and what it cost. That is an audit requirement in regulated settings and an operational requirement everywhere else — without it, nobody can debug an agent, and nobody can defend one.

2.4AI infrastructure creates unpredictable cost

Inference is expensive and scales on a different curve from the applications consuming it. Mixed into a general application platform, GPU capacity becomes both the bottleneck and the budget line nobody can attribute. Separating model infrastructure lets capacity, routing, and spend be managed — and charged back — independently of the teams building agents.

2.5Long-running agents must survive failures

Business workflows span minutes, hours, or days, and the valuable ones pause for a human decision somewhere in the middle. The platform has to stop, resume, and recover across restarts and deployments without repeating irreversible actions — which is a durability problem, not an AI problem, and is solved with the same rigor as any other.

§ 3The architecture at a glance

Four areas, and one rule about how they connect. Everything else in this document is detail beneath this picture.

The rule is that the agent sits inside the platform and reaches the other three areas only through a control built for that purpose.

§ 4Four controls around every agent

These are the part of the architecture worth remembering. If you retain one thing from this document, it should be that an agent's authority is held by four brokers rather than by the agent.

4.1The concerns they answer

4.2The controls

AI Gateway — governs model access

Every call to a language model, whether self-hosted or a vendor API, is made by one service on the caller's behalf. Model access is at once a permissions question, a budget question, a data-leakage question, and a cost-attribution question; answering those four in one place is tractable, and answering them at every call site is not. Nothing else in the platform — not an agent, not the evaluation suite, not a debugging script — can reach a model.

Tool Gateway — governs enterprise access

An agent asks for an action against a business system; a tool service decides whether that identity may take it, and performs it. The agent holds no credential to anything, because an agent's behavior is only as predictable as the model driving it, and models can be steered by the documents they read. Agents also cannot see tools they are not permitted to call — an unavailable tool named in a prompt is an invitation to work around it.

Sandbox Broker — governs code execution

Code the model wrote runs somewhere that can be thrown away: a fresh isolated environment per invocation, seeded with only the files that task needs, with no ambient network, destroyed on completion, never reused across teams. Treating generated code as hostile turns an unanswerable question — is this code safe? — into an answerable one: what can it reach?

Approval Broker — governs irreversible actions

Some actions require a person, and the agent is not that person. Which actions those are is configurable, because it is a risk-tolerance decision rather than a technical one. When a run reaches a gated action it pauses, saves its position, releases the machine it was using, and waits — for minutes or for days — then resumes exactly where it stopped once someone decides. The deciding identity is part of the audit record.

4.3Engineering guarantees

Underneath the four controls sit six invariants. They are the load-bearing rules of the platform: every structural choice in the technical reference exists to make one of them enforceable, and every design review of a change should start by asking which of the six it touches.

§ 5The design decisions

Six decisions determine everything else. Five are settled and are stated here with what each one buys and what it costs; the sixth is a delivery question that is still open (§9).

§ 6One agent task, end to end

A single realistic example exercises almost the whole architecture. An engineer asks an agent to investigate an operational issue, analyze the data, and update the corresponding ticket.

1. The user signs in and submits the task. Single sign-on establishes who they are and which team they belong to. That identity is attached to the request and travels with every step that follows — nothing downstream ever infers it.
2. The agent plans its work. It decides what to look at and in what order. This is the part that is genuinely autonomous, and the part the architecture makes no attempt to constrain.
3. Model requests go through the AI Gateway. The gateway checks the team's model permissions and budget, routes to the right model pool, records tokens and cost against the team, and returns the response.
4. The agent requests data through approved tools. It asks for logs and metrics; the tool services authorize the requesting user against each system and use their own scoped credentials to fetch. The agent never sees a credential and cannot reach a system no tool exposes.
5. Generated code runs in an isolated environment. To analyze the results the agent writes a script. The script executes in a disposable sandbox with no network access, which returns its output and is then destroyed.
6. The ticket update stops for approval. Writing to the ticketing system is configured as irreversible. The run pauses, saves its position, and releases the machine it was using. The engineer gets a notification, sees exactly what the agent intends to write, and approves it.
7. The run resumes and finishes. It picks up from where it paused — possibly on a different machine, possibly after the platform was deployed in the interim — and completes the write.
8. Every step is on one record. The plan, each model call and its cost, each tool call and the authorization decision behind it, the sandbox execution, the approval and who gave it, and the final write are all part of a single reconstructable trace.

§ 7What the architecture guarantees

Stated as outcomes, with a pointer to the part of the technical reference that specifies each one.

§ 8What this costs to operate

Every architectural choice has a price. This section states the one attached to this design.

8.1What the enterprise has to own

Stated as capabilities rather than products — the architecture holds with any credible implementation of each row, and specific technology suggestions are in the technical reference (§11.3).

8.2The ledger

8.3Where the effort concentrates

Implementation effort is not spread evenly. Three areas hold most of it: the GPU estate, which is a hardware and capacity practice more than a software one; the tenancy boundary, which is cheap to describe and unforgiving to implement, and where the platform's credibility is won or lost; and the serving layer, which is the one component with no settled answer and is sized in §9 at somewhere between three and eight engineer-weeks depending on how one question resolves.

Everything else — the gateways, the sandbox broker, the approval flow, the observability stack — is integration and configuration of components that exist. Costly in aggregate, but not uncertain.

§ 9Open decisions

One decision gates delivery. The rest can be made after the platform is standing, and are listed so that nobody mistakes silence for consensus.

9.1The gating decision — adopt or build the serving layer

9.2Decisions that can wait

Six choices are deliberately unmade. None of them changes the shape of the platform, and each has a stated default that applies if nobody decides — they are listed so that silence is not mistaken for consensus.

§ 10When to build this — and when not to

This is a substantial platform. It is worth being explicit about the conditions under which it is the wrong answer, because most organizations reach those conditions gradually rather than starting at them.

10.1Two lists

The four controls are separable. An organization that meets only some of these conditions can adopt the ones that apply — a central AI gateway is valuable on its own, and so is an isolated sandbox — without taking on the two-estate topology or the durable serving layer. The architecture is a destination, not a prerequisite.

10.2Assumptions this design rests on

• Single organization, multiple internal teams that do not share a trust boundary with each other but are not mutually adversarial.
• On-premises or colocated GPU hardware; the agent platform on-premises or on managed Kubernetes.
• Not air-gapped, but default-deny egress with a brokered allowlist.
• Prompt and completion payloads are treated as sensitive at rest.
• Vendor model APIs optionally reachable, through the same gateway as local models.
• Single-sign-on authenticated internal users. No anonymous public entry point.

Change any of these — particularly the “not mutually adversarial” assumption — and several conclusions need re-deriving, starting with how deep the separation between teams goes (§9.2).

§ 11Trust zones and topology

The four conceptual areas in §3 are implemented as seven zones. The useful question about a zone boundary is not what sits inside it but why it is drawn at all — each of these separates two things that fail differently, scale differently, or deserve different amounts of trust.

11.1The zones

11.2What enforces each boundary

11.3What runs where

The inventory below is a reference implementation, not a requirement. Named products are the ones we would reach for; the architecture holds if you substitute equivalents, and §8.1 states the same thing as capabilities. The inference fabric has its own section (§13).

11.4Sandbox requirements — Z3 in detail

Dedicated tainted node pool. Nothing else schedules here. Nothing persists here. The broker allocates a fresh sandbox, seeds only the working set the tool needs, applies ceilings, pushes work in, pulls the result out, and destroys it (C9 → C11).

EvidenceDenied egress attempts and sandbox timeout / OOM rates are alerting signals, not debug output. A spike in either means generated code is trying to do something the design says it cannot.

§ 12Multi-tenancy and the security model

This is the section that justifies the harness decision. The tenant boundary is a chain, and it breaks at its weakest link — so it is specified link by link rather than asserted once.

Two of those rows carry more weight than the rest. 404 rather than 403 on a cross-tenant read matters because a 403 confirms the identifier exists and turns the endpoint into an enumeration oracle. Row-level security as a backstop matters because it changes the consequence of an application-layer authorization bug from “another tenant's rows” to “zero rows”.

The database-level half of this — row-level security policies, transaction-scoped tenant context, and the connection-pool constraint the checkpointer imposes — is specified in Appendix B.1.

§ 13Inference fabric

A stateless serving tier on its own estate. No agent logic, no business logic, no tool execution — the fabric answers inference requests and does nothing else. The business case for separating it is §5, decision 2; this section is the engineering contract.

13.1Architecture requirements

These are the properties the design depends on. They should outlive any particular serving stack.

• Model serving runs independently of agent workloads — separate estate, separate scaling, separate upgrade cadence, separate blast radius.
• Routing is cache-aware, not round-robin. Requests are steered to the replica most likely to already hold the relevant attention state. This is the single highest-leverage configuration decision in the fabric.
• Pools are sized per workload shape. An agentic tool-calling loop and a long-form chat session are different workloads and should not share a pool.
• Prefill and decode scale independently, so latency to first token and overall throughput are not traded against each other by accident.
• Weight distribution must not saturate the fabric network — a scale-up event should not degrade in-flight inference.
• GPU health is a first-class operational signal, not something inferred from application errors.
• Exactly one ingress. Only the AI Gateway may originate traffic into the fabric; GPU nodes have no internet route at all.

The tool-calling loop is chatty and latency-sensitive in a different way than long-form chat. Sizing an agentic pool separately — small context, high concurrency, a model tuned for repeated structured tool calls — is the difference between a responsive agent and one that feels broken.

13.2Reference implementation

Named products below are the current best answers, not requirements. The fabric's contract with the rest of the platform is a pool of OpenAI-compatible endpoints behind one gateway; substitutions are local decisions.

§ 14Agent serving layer

This is the one component with no acceptable off-the-shelf answer, and it sits directly on the critical path: nothing in Z2 can be sized until it is settled. This section specifies what the component must do; §15 decides who writes it.

14.1The harness, and why it is not the control plane

Deep Agents on the LangGraph runtime is the agent harness for the entire platform.

The requirement that drove this is multi-tenancy. Self-hosted personal-assistant runtimes assume one trusted operator per gateway; tenancy in those systems is simulated by running one full instance per user, and session identifiers route requests rather than authorize them. That is an operational dead end at organizational scale and a security boundary that does not exist.

Deep Agents gives the same open-ended capability surface — explicit planning, sub-agent delegation, a virtual filesystem, skills, long-horizon execution — on top of a runtime whose state model (threads, checkpointer, store) is ours to scope. Tenancy is enforced by our authorization code and our database, not by instance count.

The corollary matters as much as the decision: Deep Agents is an execution harness, not a control plane. Credential management, model policy, cost governance, tool authorization, and audit do not live in the harness. They live in the AI Gateway, MCP Gateway, Approval Broker, and Vault.

Why a harness that owned governance would have to be trusted. We would rather trust infrastructure we can test.

14.2What is in scope, and what is not

The harness and runtime are not in question. Only the server wrapped around the compiled graph is.

Graph code, tools, MCP integration, and the AI Gateway are unchanged whichever path is taken.

14.3How a run actually executes

14.4Acceptance requirements

Eight guarantees the serving layer must provide. For an adopted server they are the evaluation rubric; for a built one they are the build scope. They are numbered so that spike findings can be recorded against them, and so a later re-evaluation starts from evidence rather than memory.

14.5Tenancy and the checkpointer

The architectural point is short: the platform must establish tenant ownership of a thread in its own tables before that thread ID is passed into LangGraph's independently-pooled checkpointer.

Why the checkpointer manages its own connection pool and executes its own SQL. It does not participate in the request-scoped session and will not carry the request's tenant context, so row-level security cannot be the thing protecting it. Authorize first, then delegate an already-authorized identifier.

This is path-independent: the same interaction has to be resolved whether the serving layer is adopted or written, because it arises from the checkpointer rather than from the server. It is also the reason a built server is an eight-week project rather than a four-week one.

The resolution has four parts:

1. thread_id is a uuid4, unguessable, so a leaked identifier is not enumerable.
2. Every request path resolves thread_id → tenant_id against the platform's own threads table under row-level security before the identifier is handed to the checkpointer.
3. Workers do the same: a leased run carries its tenant_id, revalidated at pickup — never trusted from the queue payload.
4. Optional defense-in-depth: a connection factory for the checkpointer pool that sets the tenant setting per checkout. Adds complexity; only worth it if the threat model includes a compromised worker.

§ 15Adopt or build the serving layer

The commercial option is closed (§14.2). What remains is adopting a permissively licensed open-source server or writing a thin one, decided by a one-week spike against the §14.4 rubric on a single criterion.

15.1The comparison in one table

Every row but one is recoverable with a wrapper. R5 is not, which is why it is the only thing the gate turns on.

15.2Path A — adopt Aegra

Aegra is an Agent Protocol-compliant server for LangGraph agents: FastAPI + PostgreSQL, drop-in compatible with the standard LangGraph SDK, Postgres checkpointing, a Redis job queue with lease-based crash recovery, streaming with reconnection, pluggable JWT / OAuth authentication, bring-your-own Postgres, and the tracing backend of your choice. On paper it satisfies most of §14.4. It is Apache-2.0 rather than MIT — permissive, with an explicit patent grant, which satisfies R8.1.

Spike plan — one week, one engineer

Timeboxed. The output is a filled-in rubric, not a working platform.

Pass / fail criteria

Adopt if R5 is satisfiable through the documented auth extension points without patching Aegra internals, and R2 and R4 hold under induced failure.

Fall back to Path B if tenancy requires modifying core request handling, or the checkpointer / tenancy interaction (§14.5) cannot be resolved cleanly.

If adopted — integration work, 2–3 weeks

1. Fork and vendor immediately. Pin to a commit in your own GitLab, build your own image via the Z1 mirror. Do not consume upstream tags directly into production. This is not distrust of the project; it is what makes R8.3 real.
2. Auth adapter — validate the internal JWT from the Z1 ingress, map claims to Aegra's principal, reject any request whose token lacks tenant_id.
3. RLS migration on the ownership tables, plus the thread-authorization interaction in §14.5.
4. Approval Broker bridge — interrupt payload → broker; decision → resume call.
5. OTLP wiring and baggage propagation.
6. Helm chart — API Deployment, worker Deployment, KEDA ScaledObject on Redis queue depth, pod disruption budget, NetworkPolicy per Appendix A.
7. Failure test suite — kill workers under load, roll deploys mid-run, expire leases, assert zero lost or duplicated runs.

Fork insurance

Aegra is young. The mitigation is not to avoid it — it is to be structurally ready to own it.

• Keep 4–8 weeks of engineering capacity in reserve as fork insurance rather than spending it on Path B upfront.
• Maintain a rebuild-from-scratch estimate, reviewed quarterly.
• Keep all graph and tool code free of Aegra imports. Aegra is a deployment target, not a dependency of your agents. If the only Aegra-aware code is the serving layer and its adapters, replacement is a contained project rather than a rewrite.
• Build to the Agent Protocol wire format so clients are portable across serving layers.

15.3Path B — build a thin agent server

Only if Path A fails its gate. The estimate holds only if the scope does, so the scope is stated as a table rather than a paragraph.

Building to the Agent Protocol spec keeps LangGraph SDK and Agent Chat UI compatibility for free, and makes a later switch to Aegra — or back — a configuration change.

Milestones

Total: ~8 weeks for one engineer; 5–6 with two — M3 and M4 parallelize cleanly. That is the upper bound of the effort, which is why Path A is evaluated first. The service layout, API surface, data model, queue, and streaming design for this path are specified in Appendix B.2.

15.4The decision gate

15.5Work common to both paths

Needed either way — schedule it in parallel with the spike, not after it.

15.6Risks

15.7Recommendation

§ 16Observability and evaluation

An operator has four questions, and the observability design exists to answer them rather than to collect signals:

Why each question maps to exactly one chokepoint. That is not a coincidence — it is the payoff for routing every capability through a broker in §4.

16.1Instrumentation strategy

Instrument at infrastructure chokepoints plus the framework callback, and correlate on one trace ID:

• AI Gateway emits a span per model call — model, tokens, cost, latency, cache hit, tenant. Authoritative for cost, and complete regardless of caller.
• Agent Serving Layer / Deep Agents emits the reasoning tree — graph nodes, sub-agent delegation, planning steps, tool-call decisions, interrupts.
• MCP Gateway emits a span per tool call with the authorization decision attached.
• Sandbox Broker / Z3 emits execution spans — exit code, duration, resource consumption, egress attempts.

Langfuse ingests OTLP over HTTP at /api/public/otel; spans carrying gen_ai.* attributes render as generations with model, token, and cost mapping, and everything else nests around them as regular observations. gRPC is not supported — use HTTP/protobuf.

Propagate trace attributes via OpenTelemetry baggage, not manual per-span tagging. tenant_id, user_id, session_id, thread_id, and experiment_id set once at ingress will then appear on every descendant span across process boundaries.

Why hand-tagging is where trace trees fall apart in practice: one un-instrumented hop and the tree becomes several disconnected fragments that no longer answer any of the four questions.

16.2Evaluation

Deep Agents makes evaluation tractable in a way a self-modifying daemon does not: the graph topology is code, so the trace shape is stable across runs and span-level assertions are meaningful.

Three tiers, each answering a different question, all recorded in Langfuse:

Reproducibility requirements

• Every eval trial runs as a fresh thread under a dedicated eval tenant — no checkpointer or Store state carries between trials.
• Pin model, model version, and sampling parameters explicitly. temperature=0 does not make an agent deterministic, but unpinned model routing makes results meaningless.
• Version the prompt in Langfuse prompt management and record the version on the run, so a score regression can be attributed to a prompt change rather than guessed at.
• The judge model runs on a pinned pool separate from the model under test.

Scoring must be outcome- and trajectory-weighted, not span-diffed. Even with fixed topology, an agent's loop count varies by input. Metrics that survive: task success rate, cost per task, turns to completion, tool error rate, approval-gate violations, and p95 latency.

Evidence“Gated action attempted without approval” is an assertion in the trajectory suite, so a regression in the Approval Broker fails a pull request rather than surfacing in production.

16.3Operational metrics

The last row is the only one on this list with no acceptable non-zero rate. Everything above it describes a system under load; that one describes a system whose central invariant has failed.

§ 17Request paths

Five runtime paths carry essentially all traffic; the narrative walkthrough in §6 is F1, F2, and F3 composed into one task. Read the semantic chain first — the C# references beneath each one are cross-references into the connection matrix in Appendix A, where protocol, port, and authentication for every flow are specified.

App. ANetwork matrix

Every permitted network flow in the platform, in initiator → responder direction. All internal hops are mutual TLS with SPIFFE workload identity unless noted. The C# identifiers are the cross-references used by the figures and by the flows in §17. Edge labels in any rendering of this architecture carry the C# only — protocol and port belong in a tooltip, or the diagram is unreadable at the zoom people actually use.

Explicitly denied

• Z3 → Z2 except C13 telemetry — Z3 initiates nothing else
• Z3 → Z4, Z3 → Z5, Z3 → Z0 direct
• Z2 Agent worker → Z4 direct — must traverse the AI Gateway
• Z2 Agent worker → Z5 direct — must traverse MCP
• Z4 → Z0 — GPU nodes have no internet route
• Any workload → Z0 without the Egress Broker
• Any outbound license-verification or usage-telemetry call from the serving layer (R7.1)

A.1Diagram production notes

Figures 1–4 are the reading views. The full-fidelity version belongs in a layered draw.io document; these are the layers and the rules that keep it readable.

Layout. Z0 top; Z1 as a horizontal band beneath; Z2 as the large center container; Z3 visually distinct (red / hatched) with its own border and a single bidirectional edge (C11) plus one telemetry edge (C13); Z4 as a separate container with exactly one edge entering it (C14); Z5 below or to one side; Z6 as a side rail.

Edge labels use C# IDs only. Protocol and port belong in the tooltip — otherwise the diagram is unreadable at the zoom people actually use.

What the diagram must communicate at a glance:

1. One arrow enters Z4.
2. Z3 is red, initiates nothing, and holds no state.
3. The tenant claim originates at C2 and is present on every edge downstream.

App. BImplementation notes

Two subjects that belong in the hands of whoever writes the code, and nowhere near an architecture review: the database mechanics of tenant-safe checkpointing, and the internals of a built serving layer.

B.1Tenant-safe LangGraph checkpointing

The architectural rule is in §14.5: authorize thread ownership in platform-owned tables before the thread ID reaches the checkpointer. What follows is how that is implemented without the tenant context silently evaporating — which is the failure mode, and it fails quietly, returning empty results rather than errors.

RLS on platform-owned tables — satisfies R5.4

ALTER TABLE app.threads ENABLE ROW LEVEL SECURITY;
ALTER TABLE app.threads FORCE ROW LEVEL SECURITY;   -- policies bypass the owner by default

CREATE POLICY tenant_isolation ON app.threads
  USING (tenant_id = current_setting('app.tenant_id', true))
  WITH CHECK (tenant_id = current_setting('app.tenant_id', true));

FORCE matters because the migration user usually owns the table. WITH CHECK matters because omitting it lets a tenant insert a row attributed to someone else and then lose sight of it.

Set the context transaction-scoped, never session-scoped

async with SessionFactory() as session:
    async with session.begin():
        await session.execute(text("SET LOCAL ROLE tenant_user"))
        await session.execute(
            text("SELECT set_config('app.tenant_id', :tid, true)"),
            {"tid": principal.tenant_id},
        )
        yield session

SET SESSION persists on the pooled connection after the request ends; the next borrower inherits it, and a path that skips setup silently runs as the previous tenant with no error. An f-string here is also injectable from a JWT claim. Under PgBouncer in transaction mode, session-scoped settings are outright broken.

Do not attempt to force row-level security onto the checkpointer's own tables. You do not own that schema, and a migration you do not control will eventually collide with policies you added to it.

B.2Built serving layer — service, API, data model, queue, streaming

Applies only if the gate in §15.4 resolves to Path B. Included because the eight-week estimate is only meaningful against a stated scope.

agent-server/
├── pyproject.toml
└── src/
    ├── common/      app.py  settings.py  database/  deps.py
    │                repository.py  pagination.py  errors.py
    │                security.py  telemetry.py
    ├── routes/      threads.py  runs.py  stream.py  health.py   (HTTP only)
    ├── access/      threads.py  runs.py                        (SQL only)
    ├── schemas/     threads.py  runs.py  events.py             (Pydantic)
    ├── modules/
    │   ├── graphs/     registry — assistant_id → compiled graph
    │   ├── executor/   worker loop, lease management, reaper
    │   └── events/     Redis pub/sub publisher + SSE relay
    └── main.py       mounts routers; worker entrypoint is separate

app.include_router(threads_router, dependencies=[Depends(get_principal)])

claim:     UPDATE runs SET lease_owner=:worker,
                          lease_expires_at=now()+interval '60s',
                          status='running', attempt=attempt+1
           WHERE id = (SELECT id FROM runs WHERE status='queued'
                       ORDER BY created_at
                       FOR UPDATE SKIP LOCKED LIMIT 1)
           RETURNING *;
heartbeat: extend lease_expires_at every 20s while executing
complete:  status='succeeded'|'failed', lease released
reaper:    lease_expires_at < now() AND status='running' → back to 'queued'

worker  → graph.astream(stream_mode=["messages","updates","custom"])
        → publish to Redis channel run:{run_id}
        → also append to run_events (seq monotonic)

API pod → subscribe to run:{run_id}, relay as SSE with id: {seq}
        → on reconnect with Last-Event-ID, replay from run_events,
          then subscribe

App. CTerminology

Reference only. Nothing in §1–§10 requires any of it.

TTFT

time to first token — how long before a response starts arriving

TPOT

time per output token — how fast it streams once started

KV cache

the key/value attention state a model server keeps per in-flight request; reusing it across requests with a shared prefix is the largest single win in serving throughput

MCP

Model Context Protocol — the standard interface between an agent and a tool server

HIL

human-in-the-loop — a run that pauses for a person to decide

RLS

PostgreSQL row-level security — per-row access policy enforced by the database

GUC

a Postgres runtime setting; RLS policies read the tenant from one

OTel / OTLP

OpenTelemetry and its wire protocol — the vendor-neutral tracing standard used throughout

OTel baggage

key/value context that rides along a trace across process boundaries, so descendant spans inherit it without being tagged by hand

SSE

server-sent events — the one-way HTTP streaming channel used to deliver tokens to a client

mTLS

mutual TLS — both sides of a connection present a certificate

SPIFFE / SPIRE

a standard and its implementation for issuing cryptographic workload identity to every pod

KEDA

Kubernetes event-driven autoscaling — scales workers on queue depth rather than CPU

PDB

pod disruption budget — the guardrail that stops a node drain from evicting everything at once

EPP

endpoint picker — the component that chooses which model server pod receives a request

TP / PP

tensor and pipeline parallelism — the two ways a model is split across GPUs

RAG

retrieval-augmented generation — fetching documents and putting them in the prompt

PAM

privileged access management — brokered, recorded administrative access

WAF

web application firewall

OIDC

OpenID Connect — the single-sign-on protocol that issues the identity claims the platform authorizes against