LLM Gateway vs API Gateway: Follow One Model Request End to End

A conventional API gateway already authenticates, rate-limits, routes, proxies, and records status codes and latency. Why add a separate LLM gateway to an AI application?

The most accurate answer is not that an LLM gateway is “smarter.” It is this:

A conventional API gateway primarily understands HTTP. An LLM gateway also understands model semantics inside the request and response.

POST /v1/chat/completions is still HTTP, but its model, messages, tools, images, cache fields, SSE events, and usage data affect routing, retries, billing, and observability.

A conventional gateway can be extended to do all of this. Envoy, Kong, and AWS API Gateway expose filters, plugins, functions, and external integrations. Once those extensions implement model protocols, token accounting, and semantic stream parsing, however, that collection of components has become an LLM-aware gateway layer.

This article breaks one streamed model request into ten stages and checks the claims against Nbility's current code. The goal is an architectural boundary, not a marketing category.

Capability boundary at a glance

Capability	Conventional API gateway	Additional LLM semantics
API key, JWT, mTLS	Native strength	map user credentials to upstream provider credentials
Path and host routing	Native strength	route by model, capability, group, and channel
RPM limits	Native strength	add TPM, budget, model, and user dimensions
HTTP/SSE forwarding	Supported	parse provider stream events and completion semantics
Request transformation	Templates or plugins	transform messages, tools, multimodal inputs, and output constraints
Load balancing	status, latency, weight	add model availability, price, quota, and cache affinity
Retry	HTTP status and idempotency	know whether tokens or tool side effects already escaped
Metering	requests, bandwidth, plans	input, output, reasoning, cache-write, and cache-read tokens
Observability	QPS, 4xx/5xx, latency	TTFT, tokens/s, model, channel, cache, and cost
Safety	WAF, schemas, size limits	prompt/output policies, still coordinated with the application

This is not a binary support checklist. Generic gateways can implement the right-hand column with custom code. The LLM-gateway value proposition is making those semantics first-class instead of rebuilding them for every application.

Ten stages in one model request

We use this request as the reference:

{
  "model": "customer-support-fast",
  "stream": true,
  "messages": [
    {"role": "user", "content": "Summarize ticket T-1024"}
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_ticket",
        "parameters": {
          "type": "object",
          "properties": {"id": {"type": "string"}},
          "required": ["id"]
        }
      }
    }
  ]
}

customer-support-fast is an application alias, not a provider model ID. The request moves through:

1 authenticate
→ 2 limit/budget
→ 3 parse model request
→ 4 map model
→ 5 select channel
→ 6 translate protocol
→ 7 call upstream
→ 8 stream and retry safely
→ 9 settle usage
→ 10 log and observe

A small Python artifact accompanies the article. It emits the ten stages and labels the generic-gateway starting point and the model-aware requirement. This is not a network benchmark; it is code-backed architecture evidence that reconstructs the matrix instead of relying on an impressionistic diagram.

Stages 1–2: generic gateways already excel at authentication and RPM limits

AWS defines API Gateway as a service for creating, publishing, maintaining, monitoring, and securing REST, HTTP, and WebSocket APIs. Its REST API throttling works across account, stage, method, and API-key usage-plan levels. Envoy's HTTP Connection Manager handles routing tables, header manipulation, access logging, request IDs, tracing, and statistics.

These are not unique LLM-gateway features:

• TLS termination;
• API keys, JWT, OIDC, and mTLS;
• CORS and WAF;
• IP and body-size limits;
• RPM, concurrency, and bandwidth limits;
• path, host, and header routing;
• 4xx/5xx, total latency, and QPS.

Model traffic quickly exceeds the request-count dimension, though. Two HTTP requests can contain 300 and 300,000 input tokens. Their cost, accelerator time, and context footprint are radically different. An LLM gateway often needs limits across:

user / key / model / group / time window
× requests
× input tokens
× output tokens
× estimated budget

Nbility's real request chain includes TokenAuth(), ModelRequestRateLimit(), and Distribute(). Tokens can constrain models, groups, and remaining quota. This is the first transition from HTTP identity to model budget.

Stage 3: the gateway must read the body, not only the URL

A generic gateway sees:

POST /v1/chat/completions
Content-Type: application/json
Authorization: Bearer ...

An LLM-aware layer also needs to know:

• whether model is a public alias;
• whether streaming is enabled;
• whether input includes text, images, audio, or files;
• whether tools are defined;
• whether JSON Schema output is required;
• whether prompt-cache controls are present;
• whether the destination supports the requested output and reasoning controls.

A successful basic chat call does not prove compatibility with tools, images, structured output, or streamed usage. The earlier native multi-provider API comparison shows why changing only a Base URL is not enough.

Stage 4: model mapping is not path rewriting

Conventional API routing looks like:

/api/orders → orders-service
/api/users  → users-service

An LLM gateway may need to resolve:

customer-support-fast
→ evaluate the user's group and allowed capabilities
→ map to a provider model ID
→ confirm tools + streaming support
→ choose a credential and endpoint

The alias target may change because of region, price, provider retirement, or enterprise policy. The client should not need a code change.

Nbility's current channel configuration includes ModelMapping, model lists, groups, priority, and weight. Distribution places channel model mapping in request context and retains both the requested model and UpstreamModelName, allowing logs to distinguish what the caller requested from what the provider received.

Stage 5: model balancing needs four additional signal classes

Traditional load balancers consider node health, weight, connections, latency, and HTTP errors. Model routing also needs:

1. Capability — tools, vision, schema output, and context length;
2. Quota — provider key throttling and available budget;
3. Economics — input, output, cache, and reasoning prices;
4. Affinity — whether prompt caching benefits from stable routing.

This does not mean every request should invoke an opaque “AI router.” Production routing is easier to operate when rules are explainable: filter by capability and permission, then choose by priority, weight, health, and cost. Article #106 will focus on routing policy; here we only identify the signals missing from ordinary balancing.

Stage 6: protocol translation is far more than a header rewrite

Three common native APIs already disagree:

Meaning	OpenAI Responses	Claude Messages	Gemini GenerateContent
System instruction	instructions	top-level system	systemInstruction
Input	typed input items	message content blocks	contents[].parts[]
Tool request	function-call item + call_id	tool_use + tool_use_id	functionCall part
Tool result	function-call-output item	tool_result block	functionResponse part
Output cap	max_output_tokens	max_tokens	generationConfig.maxOutputTokens
Authentication	Bearer	x-api-key + version	x-goog-api-key

A correct adapter must:

1. parse the client protocol;
2. produce a canonical internal request;
3. validate destination capability;
4. generate a provider-native request;
5. translate provider errors, stream events, and usage back;
6. reject or explicitly downgrade fields that cannot be preserved.

Silently dropping an image, tool, or schema is the most dangerous form of “compatibility.” A Lambda, WASM filter, or plugin can implement this, but the transformer then needs model-protocol knowledge—the core LLM-gateway workload.

Stage 7: forwarding SSE is not the same as understanding a model stream

OpenAI Responses emits typed semantic events. Claude Messages SSE carries message, content-block, tool-use, and thinking deltas. A conventional reverse proxy can forward the bytes if buffering is disabled.

Byte forwarding alone cannot reliably:

• normalize client event formats;
• measure time to first visible token;
• assemble streamed tool arguments;
• distinguish text, reasoning, tools, and usage;
• settle the bill after a terminal event;
• detect provider errors delivered inside an HTTP 200 stream.

The transport layer also needs LLM-appropriate settings: response buffering must stay off, idle timeouts must not cut off long generations, larger multimodal bodies need deliberate limits, and client cancellation should propagate upstream so an abandoned generation does not continue consuming tokens.

Separate two responsibilities:

Transport streaming: keep the connection open, disable buffering, forward bytes
Semantic streaming: parse events, translate types, aggregate usage, detect completion

The first is generic proxy work. The second belongs to provider adapters.

Stage 8: retries can duplicate answers and side effects

A conventional gateway may retry 429, 502, 503, or connection failures. A model request may already have:

• delivered half an answer;
• emitted a tool call;
• submitted an asynchronous image or video job;
• incurred upstream cost;
• changed conversation state.

A useful baseline is:

Upstream connection failed before visible output
→ a policy may select another channel

The first visible token or tool call reached the client
→ do not transparently replay the whole request

This is not universal. Idempotency keys, resumable streams, and explicit task state machines allow finer policies. But “retry every 5xx” is unsafe for model streams.

Nbility's Relay loop iterates through RetryTimes, selects channels, and calls shouldRetry with the normalized error, remaining attempts, skip-retry flags, and status-code policy. Channel-affinity failures also have a separate skip path. This is not a claim of perfect exactly-once behavior; it shows why retry must be tied to model and response state.

Stage 9: the final price often appears after HTTP 200

Conventional gateway metering often uses requests, transfer, plan, and gateway processing time. Model billing is commonly reconstructed from response usage:

uncached input tokens
+ cache-write tokens × write rate
+ cache-read tokens × read rate
+ output tokens × output rate
+ reasoning / audio / image / tool surcharges

The previous prompt-caching billing replay demonstrates why multiplying all prompt tokens by one price misses cache writes and reads.

Streaming makes settlement harder because usage may arrive only in a terminal event or optional final chunk. The gateway needs consistent behavior across normal completion, client disconnect, and upstream failure, while retaining raw provider usage for audits.

Nbility's current billing code distinguishes ordinary input and output, OpenAI cache reads/writes, Claude creation/read tokens and 5m/1h writes, image/audio/reasoning/tool items, model and group ratios, tiered expressions, requested and upstream model, channel, and retries. Those are domain semantics a generic gateway does not know by default.

Stage 10: move from latency to TTFT, tokens, and cost

AWS API Gateway's documented CloudWatch metrics include Count, 4XXError, 5XXError, Latency, and IntegrationLatency. They remain useful, but they do not answer:

• How long before the user saw the first token?
• What was the generation rate?
• Which model, channel, and provider served the request?
• How many input, output, cache, and reasoning tokens were used?
• How much cost did a retry add?
• Is a public model alias drifting in quality or error rate?

A practical three-layer model is:

Layer	Metrics
HTTP	QPS, 4xx/5xx, total latency, connection errors, response bytes
Model	TTFT, generation time, tokens/s, stop reason, tool calls, cache hits
Business	request cost, budget burn, task success, retry recovery, feedback

An LLM gateway should not own every quality decision, but it should preserve the trace that connects an HTTP request to its model path and bill.

Code-backed stage trace

The companion script emits these ten stages:

Stage	Generic-gateway starting point	Why model semantics matter
authenticate	native	separate user and upstream credentials
rate_limit	native	TPM, model budget, and groups
parse_request	custom transform	messages, tools, multimodal inputs
map_model	custom routing	aliases, capabilities, provider model
select_channel	load balancing	quota, price, and cache affinity
translate	plugin	native protocols are not equivalent
stream	byte forwarding	events, tool arguments, and usage
retry	status retry	visible output and side-effect boundary
settle_usage	custom metering	token and cache price components
observe	latency/status metrics	TTFT, model, channel, and cost

The conclusion is not that every team must buy a new gateway. A team can build the LLM-aware components on its existing gateway or adopt an implementation. Responsibilities—not product naming—define the architectural boundary.

When a conventional API gateway is enough

Do not add another layer prematurely when:

• you call one provider and one model;
• clients use the provider's native protocol;
• there is no per-user token budget or resale billing;
• cross-provider translation, failover, and unified audits are unnecessary;
• the gateway only provides perimeter TLS, authentication, WAF, and coarse limits;
• an application service already handles usage, retries, and logs reliably.

The existing gateway is simpler in this situation.

When a dedicated LLM gateway starts paying off

Typical signals include:

• two or more providers or credential pools;
• clients need one stable protocol;
• application aliases must be decoupled from provider models;
• billing depends on tokens, caching, or multimodal outputs;
• model/channel health, budget, and audit are required;
• teams repeatedly build the same adapters;
• 429 and 5xx require controlled failover;
• one log must reconstruct the complete request cost.

The LLM gateway becomes a shared model-access layer, not a replacement for WAF, service mesh, or application logic.

Deployment is not an either-or decision

A common composition is:

Internet
→ CDN / WAF
→ conventional API gateway (identity, TLS, coarse limits)
→ LLM gateway (model protocols, routing, usage, billing)
→ OpenAI / Claude / Gemini / self-hosted models

An extensible gateway can also perform both roles if the team can own its plugins and upgrade boundary. Kong's official AI Proxy is an example of adding provider transformation, authentication, load balancing, streaming, logging, and usage-aware AI behavior to a general gateway. The categories are not mutually exclusive products.

Evaluation checklist

Do not stop at “how many models are supported.” Verify:

• exact client and upstream API surfaces
• faithful tools, images, audio, and JSON Schema behavior
• unbuffered SSE and provider error-event handling
• no unsafe transparent retry after visible output
• requested model and actual upstream model are both retained
• channel selection includes permission, capability, quota, and health
• raw usage is retained and can reconstruct a bill
• cache reads and writes are priced separately
• logs include TTFT, model, channel, retries, and cost
• unsupported fields fail explicitly instead of disappearing
• sensitive prompts are excluded from ordinary access logs by default
• gateway failure has a documented bypass or degradation policy

Conclusion

An LLM gateway is not merely Nginx pointed at an OpenAI-compatible URL, and a conventional API gateway is not obsolete infrastructure that “does not understand AI.” They share a large HTTP foundation. The real boundary is whether the gateway treats model protocols, model selection, stream events, token usage, and model cost as first-class semantics.

Following one request shows the split:

• authentication and coarse limits are already generic-gateway strengths;
• parsing, mapping, protocol translation, and stream events require model adapters;
• usage settlement and model observability determine whether the system is operable over time.

If your team already maintains those modules inside an existing gateway, you are effectively building an LLM gateway. If you prefer not to maintain multi-provider adapters repeatedly, Nbility provides a unified model endpoint, channel mapping, streaming, retries, usage logs, and component-level billing. A small testing credit can be requested in the ticket center.

Official sources

• What is Amazon API Gateway?
• API Gateway throttling
• API Gateway CloudWatch metrics
• Envoy HTTP connection management
• Kong AI Gateway
• Kong AI Proxy
• OpenAI streaming responses
• Claude streaming Messages