Enterprise AI Agent Frameworkv1.0

§ 1Executive summary

The agent can reason freely. It cannot act freely.

1.1At a glance

§ 2Why this architecture exists

Five problems appear when agents move from personal experiments into shared production. Each is answered by a structural choice rather than by policy or training, because none of them is the kind of failure a careful person avoids.

2.1Agents expand the security boundary

An agent chooses actions based partly on text it encounters at runtime. That text may come from a ticket, a document, an email, a web page, or a tool response, and some of it will eventually be controlled by someone who benefits from changing what the agent does next.

Prompt injection is therefore not merely a prompt-quality problem. The architecture assumes the model can be influenced, and keeps meaningful authorization outside it — in components the model has no way to instruct.

2.2Model-generated code is an untrusted workload

A capable agent will eventually write code. It may use Python to inspect a dataset, shell commands to transform files, or a browser runtime to complete a task.

The right security question is not can we make the model generate only safe code? It is what can the generated code reach when it is wrong? Isolation is what makes the second question answerable.

2.3Enterprise agents need accountability

When an agent changes a ticket, queries a warehouse, sends a message, modifies a repository, or spends model capacity, the organization has to know who initiated the work, which identity authorized it, what the agent attempted, which model and prompt version were involved, what systems were reached, where the run failed, and what it cost.

That is an audit requirement in some environments and an operations requirement in all of them — without it nobody can debug an agent, and nobody can defend one.

2.4Long-running work has to survive failure

Useful workflows do not necessarily fit inside one HTTP request. They may run for hours, delegate work, wait for external information, pause for a person, and resume after the software itself has been deployed or restarted.

LangGraph's persistence model is explicitly designed to checkpoint execution and resume stateful workflows, and its human-in-the-loop model persists state across an interrupt. Durability belongs in the runtime architecture rather than being reconstructed inside every agent.

SourceLangGraph persistence · LangGraph interrupts

2.5Shared agents create a real tenancy boundary

A single-user assistant can assume one person owns the process. An enterprise platform cannot.

Threads, memory, documents, tool permissions, model budgets, and audit records have to belong to a tenant and a principal — and the same fact has to remain true under retries, concurrency, and reconnects.

§ 3The architecture at a glance

Five conceptual areas, and one rule about how they connect. Everything else in this document is detail beneath this picture.

The rule is that the agent sits inside the platform and reaches the other four areas only through a control built for that purpose.

§ 4Four controls around every agent

These four controls are the architecture; everything else exists to make them enforceable. If you retain one thing from this document, it should be that an agent's authority is held by four brokers rather than by the agent.

4.1The concerns they answer

4.2The controls

AI Gateway — governs model access

Every model call is made through one service. The agent holds no local model endpoints and no external-provider credentials. The gateway decides which models the caller may use, whether the request may leave the estate, which provider or local fabric should serve it, what budget or quota applies, what redaction or data policy applies, and how usage is attributed.

Model access is at once a permissions question, a budget question, a data-leakage question, and a cost-attribution question. Answering those four in one place is tractable; answering them at every call site is not. A local Inference Fabric can sit behind this gateway exactly like any other model provider.

Tool Gateway — governs enterprise access

The agent asks for an operation. A trusted service decides whether the identity behind the run may perform it, and invokes the enterprise system using credentials held by that service. The agent gets capability without possession.

Where MCP is used it supplies a standard protocol for tool discovery and invocation. Current MCP authorization for HTTP transports is based on OAuth 2.1 and protected-resource discovery, but the protocol does not remove the enterprise's responsibility to decide which tools a caller should receive or what each operation is allowed to do. The policy boundary is therefore the Tool Gateway, whether or not every downstream integration speaks MCP.

SourceMCP authorization specification

Sandbox Broker — governs code execution

Model-generated code runs outside the trusted agent worker. Each execution receives only the inputs it needs, receives no ambient enterprise credentials, has no network unless a particular operation explicitly requires one, and is destroyed after use.

gVisor is one current implementation option, designed specifically to isolate untrusted code and containers from the host kernel while remaining compatible with container tooling and Kubernetes. The architecture does not require gVisor specifically. It requires the security boundary.

SourcegVisor architecture guide

Approval Broker — governs irreversible actions

Some actions require a person. Which ones is a risk decision, not an AI-framework decision. When the agent reaches a gated operation the run saves its state, the proposed action is presented to an authorized person, the execution releases the resources it was holding, the person approves, edits, or rejects, and the run resumes from persisted state.

LangChain's human-in-the-loop middleware implements this same interrupt / persist / resume pattern and supports approve, edit, and reject decisions. The approver's identity becomes part of the run record.

SourceLangChain human-in-the-loop

§ 5Engineering invariants

Six rules sit underneath the four controls. They are the load-bearing rules of the platform: every structural choice in the technical reference exists to make one of them enforceable, and every design review of a change should start by asking which of the six it touches.

§ 6The design decisions

Six decisions determine everything else. Each is stated with what it buys and what it costs, because a decision recorded without its price is a decision nobody can revisit.

§ 7One agent task, end to end

A single realistic example exercises almost the whole architecture. An engineer asks an agent to investigate a production issue, analyze the evidence, and update the corresponding ticket.

1. The request enters with identity. Single sign-on establishes the caller and the tenant. Those claims become explicit run context and travel with every step that follows — nothing downstream ever infers them.
2. The agent plans. It decides which information it needs, in what order to gather it, and whether to delegate work to subagents. This is genuinely autonomous, and the infrastructure makes no attempt to write the plan for it.
3. The agent calls a model. The request passes through the AI Gateway, which evaluates model policy and routes either to an authorized external provider or to the organization's local inference fabric, recording tokens and cost against the team.
4. The agent asks for enterprise data. It requests logs and ticket information through approved tools. The Tool Gateway evaluates the caller and the operation, and invokes each enterprise system with scoped service credentials. The agent sees the result; it never sees the credential.
5. The agent writes analysis code. The script goes to the Sandbox Broker. A fresh isolated environment receives the required input files, executes the script, returns its output, and is destroyed.
6. The agent proposes the ticket update. The organization has classified the write as a gated action, so execution pauses and state persists. The worker is released rather than held open.
7. A person decides. The engineer reviews the proposed change and approves it. Their identity and decision enter the run record.
8. The run resumes. The workflow continues from persisted state — possibly on a different machine, possibly after the platform was deployed in the interim — and executes the approved tool operation.
9. The complete run remains reconstructable. The trace contains the model calls, the tool decisions, the sandbox execution, the approval, the final action, latency, and cost attribution.

§ 8What the architecture guarantees

Stated as outcomes, with a pointer to the part of the technical reference that specifies each one.

§ 9What this costs to operate

Every architectural choice has a price. This section states the one attached to this design.

9.1What the enterprise has to own

Stated as capabilities rather than products — the architecture holds with any credible implementation of each row.

9.2The ledger

9.3Where the effort concentrates

Implementation effort is not spread evenly. Two areas hold most of it: the tenancy boundary, which is cheap to describe and unforgiving to implement; and the serving layer, which is the one component with no settled answer (§10). Everything else — the gateways, the sandbox broker, the approval flow, the observability stack — is integration and configuration of components that already exist. Costly in aggregate, but not uncertain.

This is infrastructure, not an application launch.

§ 10Open decisions

One decision gates delivery. The rest can be made after the platform is standing, and are listed so that nobody mistakes silence for consensus.

10.1The gating decision — the agent-serving implementation

10.2Decisions that can wait

§ 11When to build this — and when not to

This is a substantial platform. It is worth being explicit about the conditions under which it is the wrong answer, because most organizations reach those conditions gradually rather than starting at them.

The four controls can be adopted separately. An AI Gateway is useful without a sandbox. A Tool Gateway is useful without Deep Agents. Human approval is useful without self-hosted models. The complete framework is the point at which those controls become one coherent platform — a destination, not a prerequisite.

§ 12Trust zones and topology

A useful implementation separates five trust domains. The useful question about a zone boundary is not what sits inside it but why it is drawn at all — each of these separates two things that fail differently, scale differently, or deserve different amounts of trust.

Model services sit behind a separate logical boundary Z4 and are reached exclusively by the AI Gateway. That model service may be an external API or the organization's local Inference Fabric; the platform's topology is identical either way.

12.1What enforces each boundary

§ 13Multi-tenancy and identity

Identity is bound at ingress and propagated per request. At minimum, carry tenant_id, principal_id, run_id, and the roles, groups, or policy context the request will be evaluated against.

Do not authorize a request simply because it knows a thread identifier. Platform-owned records store tenant ownership explicitly, and database-level isolation reinforces application checks for the tables the platform controls.

The tenant boundary is a chain, and it breaks at its weakest link — so it is specified link by link rather than asserted once.

Two of those rows carry more weight than the rest. 404 rather than 403 on a cross-tenant read matters because a 403 confirms the identifier exists and turns the endpoint into an enumeration oracle. Row-level security as a backstop matters because it changes the consequence of an application-layer authorization bug from “another tenant's rows” to “zero rows”.

Tenant isolation should fail closed and visibly. An authorization bug that silently returns an empty record is safer than one that returns another tenant's state, but it is still an operational defect and should produce a traceable signal.

§ 14Agent harness and serving layer

14.1The harness

Deep Agents on the LangGraph runtime is the reference harness. It provides planning, subagents, filesystem-backed context management, skills, memory, and sandbox-aware execution, using LangGraph underneath for runtime capabilities.

SourceDeep Agents overview — described explicitly as an agent harness

The requirement that drove this choice is multi-tenancy. Single-user assistant runtimes assume one trusted operator per gateway; tenancy in those systems is simulated by running one full instance per user, and session identifiers route requests rather than authorize them. That is an operational dead end at organizational scale and a security boundary that does not exist.

Deep Agents gives the same open-ended capability surface on top of a runtime whose state model — threads, checkpointer, store — is ours to scope. Tenancy is enforced by our authorization code and our database, not by instance count.

14.2The serving layer

The serving layer is what turns a compiled graph into a service other teams can depend on. It supplies:

• Threads — durable execution and conversation context, owned by exactly one tenant.
• Runs — asynchronous submission, status, and cancellation; submission returns immediately and execution happens on a worker.
• Streaming — token-level and node-level events, reconnectable mid-run, from a pod that need not be the one running the graph.
• Background workers — leased execution, heartbeats, and a reaper that requeues work held by a dead worker.
• Persistence — checkpoints as the source of truth for resumption.
• Cancellation, resume, health, and scaling — queue-depth-driven, because these workers are I/O-bound and CPU-based autoscaling under-scales badly.

Which implementation provides these is the open decision in §10.1; the options are compared in Appendix B. The selection criterion is not feature count — it is whether the deployment mechanism preserves the invariants in §5.

§ 15Durable execution and state

LangGraph checkpoints graph state against a thread and persists execution progress so workflows can recover after interruption. The architecture separates four kinds of state, and confusing any two of them is the usual source of trouble.

Never confuse checkpoint durability with a distributed transaction across the business systems the agent touches. The first three can be replayed; the fourth cannot.

15.1Replay safety

Run execution is at-least-once, and the checkpoint record is the idempotency mechanism: a redelivered run re-enters the graph at its last checkpoint rather than at the start. At-least-once delivery is what lets a run survive a node reboot; the checkpoint is what stops that survival from doing the same work twice.

Tool side effects between the last checkpoint and the crash may still repeat. Irreversible tools must therefore be idempotent, use a transactional API, or sit behind the Approval Broker — a contract on tool authors, stated in the tool authoring guidelines, not an implementation detail of the queue.

EvidenceSustained non-zero lease expiries mean the graceful-termination window is shorter than node execution, and rolling deploys are manufacturing replays. That is an alert, not a debug signal.

§ 16AI Gateway and model access

The AI Gateway owns the enterprise model policy surface. It is the only component in the platform that knows where a model physically lives.

When local inference is selected, the gateway calls the sibling Inference Fabric. When an approved external model is selected, it calls that provider through controlled egress. The agent sees one model interface either way, and a change of provider is a gateway configuration change rather than an application change.

WhyCost is authoritative here and nowhere else, because every model call traverses this boundary. An unmetered path shows up as a reconciliation gap rather than as a surprise invoice.

§ 17Tool Gateway and MCP

MCP is a useful protocol boundary. It is not the authorization architecture by itself.

A production tool path performs seven things, in this order:

1. Tool discovery, filtered for the caller. An agent that is not authorized to perform an action should not receive that tool in the first place — an unavailable tool named in a prompt is an invitation to work around it.
2. Operation-level authorization against the principal behind the run, evaluated at the moment of the call rather than at session start.
3. Argument validation at the boundary, against a schema the tool service owns.
4. Credential acquisition by the trusted integration service — scoped, short-lived, and issued by the secrets system.
5. Execution against the enterprise system.
6. Response filtering where the result may contain more than the caller is entitled to see.
7. Audit emission, carrying the authorization decision alongside the call.

Where tools call external SaaS, credentials remain in the integration service or the secrets system. They never become agent state.

SourceMCP authorization — OAuth 2.1 and protected-resource discovery

§ 18Isolated code execution

The Sandbox Broker receives an execution request and creates an isolated environment. It is the only component permitted to address the sandbox zone, and the sandbox never calls back into the platform except to emit telemetry.

EvidenceDenied egress attempts and sandbox timeout or out-of-memory rates are alerting signals, not debug output. A spike in either means generated code is trying to do something the design says it cannot.

SourcegVisor architecture guide — one current implementation option

§ 19Human approval

Approval is represented as persistent workflow state, not as a synchronous request waiting for somebody to click a button. A run that holds a worker open while waiting for a person is a run that cannot survive a deployment.

Execution then releases compute. On resume, execute only the approved form of the action — an edited approval is a different action from the one proposed, and the record has to say which one ran.

Non-idempotent operations belong after the approval boundary. That is not a stylistic preference: it is what makes the replay behavior in §15.1 safe.

SourceLangChain human-in-the-loop — interrupt, persist, resume

§ 20Observability and evaluation

An operator has four questions, and the observability design exists to answer them rather than to collect signals.

WhyEach question maps to exactly one chokepoint. That is not a coincidence — it is the payoff for routing every capability through a broker in §4.

20.1Instrumentation strategy

A production agent trace should correlate the user request, the graph and prompt version, model calls, subagent delegation, tool calls, authorization decisions, sandbox executions, interrupts and approvals, the final outcome, latency, and token cost.

Instrument at infrastructure chokepoints plus the framework callback, and correlate on one trace identity:

• AI Gateway emits a span per model call — model, tokens, cost, latency, cache hit, tenant. Authoritative for cost, and complete regardless of caller.
• Serving layer and harness emit the reasoning tree — graph nodes, subagent delegation, planning steps, tool-call decisions, interrupts.
• Tool Gateway emits a span per tool call with the authorization decision attached.
• Sandbox Broker emits execution spans — exit code, duration, resource consumption, egress attempts.

Propagate trace attributes as telemetry baggage, not by manual per-span tagging. Tenant, principal, session, thread, and experiment set once at ingress will then appear on every descendant span across process boundaries.

WhyHand-tagging is where trace trees fall apart in practice: one un-instrumented hop and the tree becomes several disconnected fragments that no longer answer any of the four questions.

20.2Evaluation

A code-defined graph makes evaluation tractable in a way a self-modifying daemon does not: topology is code, so trace shape is stable across runs and span-level assertions are meaningful.

Reproducibility requirements

• Every trial runs as a fresh thread under a dedicated evaluation tenant — no checkpoint or memory state carries between trials.
• Pin model, model version, and sampling parameters explicitly. Zero temperature does not make an agent deterministic, but unpinned model routing makes results meaningless.
• Version the prompt and record the version on the run, so a score regression can be attributed to a prompt change rather than guessed at.
• The judge model runs on a pinned pool separate from the model under test.

Scoring must be outcome- and trajectory-weighted, not span-diffed. Even with fixed topology, an agent's loop count varies by input. The release gate should evaluate the behavior that matters rather than whether two stochastic traces look identical.

Useful longitudinal metrics: task success rate, cost per task, turns to completion, tool error rate, approval-policy violations, sandbox failure rate, and p95 task latency.

Evidence“Gated action attempted without approval” is an assertion in the trajectory suite, so a regression in the Approval Broker fails a pull request rather than surfacing in production.

20.3Operational metrics

The last row is the only one on this list with no acceptable non-zero rate. Everything above it describes a system under load; that one describes a system whose central invariant has failed.

§ 21Request paths

Five runtime paths carry essentially all traffic; the narrative walkthrough in §7 is F1, F2, F3, and F4 composed into one task. The agent worker has no alternative route around any of these paths.

App. ANetwork rules

The platform boundary is expressible almost entirely as a small allowlist. The denied flows are more informative than the permitted ones, so they are stated first.

Explicitly denied

• Agent worker → enterprise system — must traverse the Tool Gateway
• Agent worker → model server directly — must traverse the AI Gateway
• Agent worker → sandbox workload directly — only the broker protocol
• Sandbox → enterprise systems
• Sandbox → model infrastructure
• Sandbox → internet, by default
• Sandbox → platform, except its telemetry span
• Any workload → internet without the egress broker
• Any outbound license-verification or usage-telemetry call from the serving layer

Explicitly permitted

App. BAgent-serving deployment choices

The harness decision and the deployment decision are separate. Three paths are credible today; the architecture in Part I does not change whichever one wins.

The selection criterion is not feature count. It is whether the deployment mechanism preserves the architectural invariants in §5 — and specifically whether tenant isolation reaches state retrieval through documented extension points rather than through a patch to internal request handling.

What to validate before adopting a packaged server

• Tenant isolation, including the interaction with an independently pooled checkpointer (Appendix C).
• Failure recovery — kill workers under load, roll deploys mid-run, expire leases, and assert zero lost or duplicated runs.
• Upgrade behavior and migration ownership.
• Whether authorization can be integrated without patching internal request handling.
• Whether the serving layer makes any outbound call the estate does not permit.

If adopted

Fork and vendor immediately; pin to a commit in your own repository and build your own image through the internal mirror. Keep all graph and tool code free of imports from the serving project — it is a deployment target, not a dependency of your agents. Build to the Agent Protocol wire format so clients stay portable across serving layers, and keep a current rebuild-from-scratch estimate as fork insurance.

SourcesLangChain standalone server deployment · Aegra · Agent Protocol

App. CTenant-safe state rules

1. Every externally addressable thread or run is checked against tenant ownership.
2. Tenant context is request-scoped, never inferred from pooled connection or session state.
3. Platform-owned relational tables enforce tenant ownership again at the database boundary where practical.
4. Framework checkpointer access is reached only after platform authorization.
5. Evaluation uses a dedicated tenant and fresh state.
6. Cross-tenant access tests are release-blocking tests.
7. Any disagreement between ingress identity and execution identity is a page-worthy security event.

C.1The checkpointer interaction

The architectural rule is short: establish tenant ownership of a thread in platform-owned tables before that thread identifier reaches the framework's independently pooled checkpointer.

WhyThe checkpointer manages its own connection pool and executes its own SQL. It does not participate in the request-scoped session and will not carry the request's tenant context, so row-level security cannot be the thing protecting it. Authorize first, then delegate an already-authorized identifier.

The resolution has four parts: thread identifiers are unguessable UUIDs, so a leaked one is not enumerable; every request path resolves thread to tenant against the platform's own table under row-level security before delegating; workers revalidate the tenant at lease pickup rather than trusting the queue payload; and, where the threat model includes a compromised worker, a connection factory sets the tenant setting per checkout as defense in depth.

ALTER TABLE app.threads ENABLE ROW LEVEL SECURITY;
ALTER TABLE app.threads FORCE ROW LEVEL SECURITY;   -- policies bypass the owner by default

CREATE POLICY tenant_isolation ON app.threads
  USING (tenant_id = current_setting('app.tenant_id', true))
  WITH CHECK (tenant_id = current_setting('app.tenant_id', true));

async with SessionFactory() as session:
    async with session.begin():
        await session.execute(text("SET LOCAL ROLE tenant_user"))
        await session.execute(
            text("SELECT set_config('app.tenant_id', :tid, true)"),
            {"tid": principal.tenant_id},
        )
        yield session

App. DTerminology

Reference only. Nothing in §1–§11 requires any of it.

Agent harness

the code and runtime abstractions governing how an agent plans, reasons, delegates, and uses tools

Agent serving layer

infrastructure exposing threads, runs, streaming, queues, and durable execution

AI Gateway

the policy boundary for all model access — who may call which model, under what budget and data rules

Tool Gateway

the policy boundary for enterprise-system operations

MCP

Model Context Protocol — a standard for exposing model-context tools; useful inside the Tool Gateway architecture, but not itself the authorization system

Sandbox Broker

the trusted service that creates disposable environments for untrusted generated code

Approval Broker

the persistent workflow service governing human decisions on designated actions

Thread

durable execution and conversation context, owned by exactly one tenant

Run

one execution of an agent against a thread or a stateless input

Checkpoint

persisted graph state that allows a run to resume rather than restart

Interrupt

the primitive that suspends a graph, persists its state, and releases the worker

HIL

human-in-the-loop — a run that pauses for a person to decide

RLS

row-level security — per-row access policy enforced by the database

OTel / OTLP

OpenTelemetry and its wire protocol — the vendor-neutral tracing standard used throughout

OTel baggage

key/value context that rides along a trace across process boundaries, so descendant spans inherit it without being tagged by hand

SSE

server-sent events — the one-way HTTP streaming channel used to deliver tokens to a client

mTLS

mutual TLS — both sides of a connection present a certificate

OIDC

OpenID Connect — the single-sign-on protocol that issues the identity claims the platform authorizes against