I wanted to learn LLM-as-a-judge properly — not “yeah, you use an LLM to grade another LLM’s output” properly, but actually understand where it came from, where it breaks, and what people smarter than me are doing about it. And since I’d just spent a week writing a reactive hot-take about someone else’s product launch and gotten (correctly) called out for having zero citations and zero original insight, I decided to do this one right: real research, real sources, and if I was going to claim something novel, it had to survive someone actively trying to kill it.

So I spun up a small army of research agents. Six of them went and surveyed the field — foundations, biases, calibration, mitigations, the 2025-2026 frontier, and how it’s actually used in production. Then a synthesis pass consolidated all of that into five candidate “gaps” in the current toolkit. Then I had agents develop each gap into a concrete, technically-grounded proposal. Then — and this is the part that actually taught me something — I had three independent skeptics per proposal try to refute each one: find the prior art, poke the mechanism, default to “this doesn’t survive” unless proven otherwise.

All five proposals died. Every single one. Not because the ideas were bad — the mechanisms were sound — but because someone had already published almost exactly the same thing, in some cases within the last twelve weeks. I’ll walk through the wreckage later in this post, because it turned out to be the most useful part of the whole exercise. But first, the actual field guide, because that’s what I came here to learn.

1. What Is LLM-as-a-Judge?

Picture a professor with 50,000 essays to grade by Friday. They can’t read all of them — nobody can. So they hire a teaching assistant: someone smart, fast, and (crucially) consistent, who reads a rubric once and then applies it identically to essay #1 and essay #49,999. The TA isn’t the professor. The TA can be wrong, tired, biased toward students who write in a font the TA likes, or unable to tell a genuinely brilliant unconventional answer from a well-formatted one that says nothing. But the TA is fast enough that grading actually finishes, and reliable enough — most of the time — that the grades mean something.

LLM-as-a-judge is that TA, and the “essays” are LLM outputs: chatbot responses, RLHF preference pairs, agent trajectories, summaries, code. Instead of a human rater or a fixed answer key, you prompt an LLM to score or compare outputs — pointwise (“rate this 1-10”), pairwise (“which of these two is better”), or via a rubric (“does this satisfy criteria A, B, and C”). The judge doesn’t need to be a different model than the one being graded, though — as we’ll get to — that turns out to matter a lot.

2. Why It Had to Exist

Two gaps forced this into existence, and both are obvious in hindsight.

Gap one: the old metrics don’t work for open-ended text. BLEU and ROUGE — the classic n-gram overlap metrics — were built for machine translation, where there’s a roughly “correct” answer to compare against. They fall apart on summarization, dialogue, or anything with legitimate diversity of good answers. G-Eval (Liu et al., Microsoft, EMNLP 2023) showed that GPT-4, prompted with chain-of-thought reasoning and a structured “form-filling” scoring template, correlated with human judgment far better than any n-gram metric — 0.514 Spearman correlation on summarization, a genuinely large jump.

Gap two: human evaluation doesn’t scale to how fast models ship. Getting crowdworkers to rate thousands of response pairs for every RLHF iteration or every weekly model release is slow and expensive. AlpacaFarm (Stanford, NeurIPS 2023) and its successor AlpacaEval showed that LLM auto-annotators could substitute for human preference labeling at roughly 50x lower cost, with high agreement — enough to actually run RLHF-style method development without a standing army of human raters.

Put those together and you get the pitch: an evaluator that’s fast enough to keep up with iteration speed, and correlated enough with humans to be trustworthy. Emphasis on “enough” — we’ll spend a lot of this post on the gap between “enough” and “always.”

3. The Papers That Built the Field

Notice the shape of this timeline: 2022-2023 is “does this work at all” (yes, roughly). 2024 onward is “okay, now fix the ways it doesn’t.” We’re still deep in the second phase, and it’s accelerating, not slowing down.

4. Where It’s Actually Used Today

LLM-as-a-judge isn’t a research curiosity — it’s load-bearing infrastructure in three distinct lanes.

Model release evaluation. Meta’s Llama 3 technical report documents model-graded scoring (correctness, informativeness) running alongside human eval during post-training. OpenAI’s open-source evals framework standardizes “model-graded” templates as a first-class evaluation primitive, and their grading guidance explicitly recommends using a different, typically stronger model as judge than the one being graded — a direct, practitioner-level acknowledgment of self-preference bias.

RLHF/RLAIF alignment pipelines. This is the highest-stakes lane: the judge’s verdict becomes the training signal. Anthropic’s Constitutional AI pipeline and Google’s RLAIF work both use LLM judges to generate the preference data that trains the reward model driving RL. If the judge is subtly wrong here, the model gets optimized toward the judge’s blind spots, not the user’s actual preferences — quietly, and at scale.

Production engineering pipelines. This is where the culture is most mature, and most honest about limitations. Anthropic runs a single-call rubric judge inside its multi-agent research system but keeps human testers around for the failure modes the judge misses. Google’s Vertex AI AutoSxS ships response-flipping and multi-sampling as built-in bias controls, plus an official notebook for checking the autorater against human preference data before trusting it. Scale AI’s SEAL leaderboards pair private human-curated gold sets (~1,000 examples) with LLM grading for scale. Databricks publishes its actual validation numbers — Krippendorff’s alpha of 0.565-0.698, Cohen’s kappa around 0.64-0.65 — rather than just claiming the judge “works.” And the widely-followed practitioner playbook from Hamel Husain and Braintrust is refreshingly unglamorous: label ~30 examples by hand, iterate the judge prompt against your own disagreements with it, prefer binary pass/fail over Likert scales, and never fully retire the human reviewer — just shrink their workload via sampling.

That last point is the tell. Nobody serious treats the judge as ground truth. They treat it as a fast, cheap, imperfect proxy that needs a human-labeled leash.

5. The Bias Zoo

Here’s where it gets genuinely funny, in a “I can’t believe this is the state of the art” way.

Position bias. Judges systematically favor whichever answer is shown first (or second — it’s judge-dependent). Wang et al. demonstrated you could make Vicuna-13B beat ChatGPT on 66 of 80 test queries purely by swapping which answer appeared first — no change to either model’s actual output. A later large-scale study across 15 judges and 150,000+ evaluations confirmed it’s real, judge- and task-dependent, and driven more by how close the two answers are in quality than by anything about length.

Verbosity bias. Longer answers get rated higher, independent of whether the extra length adds anything. This was bad enough that AlpacaEval needed a dedicated length-control regression just to stop measuring “which model rambles more” instead of “which model is better.” Interestingly, this one isn’t universal: recent work shows Gemini- and Llama-family judges prefer longer answers, Claude-family judges actually prefer shorter ones, and GPT-4o sits roughly neutral. Your bias-mitigation strategy needs to know which judge you’re running.

Self-preference bias. Judges rate their own model family’s outputs higher — Wataoka & Takahashi tie this to perplexity: the judge seems to conflate “text I find familiar/predictable” with “text that’s good,” which is a fairly damning thing to discover about your quality metric.

Format bias, and this is the punchline. A 2026 analysis quantified judges’ preference for markdown formatting — headers, bullets, code blocks — over plain prose with identical content, at an effect size of 0.76-0.92. For comparison, position bias in the same analysis measured ≤0.04. Format bias isn’t a minor confound sitting next to position bias — it dwarfs it by roughly 20x. And it’s not just incidental: a 2026 adversarial paper shows you can learn, via bandit search, exactly which formatting tweaks most reliably flip a judge’s verdict, independent of content quality. If your eval pipeline isn’t stripping markdown before comparing responses, you are, to a significant degree, benchmarking who writes the prettiest bullet points.

A broader CALM framework study catalogs 12 distinct bias categories — bandwagon effects, authority bias, sentiment bias, distraction bias, chain-of-thought bias, and more — and finds even the strongest judge models retain measurable bias on specific tasks. This is not a solved problem with a couple of known footguns. It’s an active taxonomy.

6. Where Judges Fall Off a Cliff

Bias is embarrassing. Domain collapse is worse, because it means the judge isn’t measuring the thing you think it’s measuring at all.

On factual consistency in summarization, GPT-3.5-class judges show only 0.3-0.6 correlation with human judgment (versus 0.8-0.9 for human experts), and miss 40-70% of factually inconsistent summaries outright — while maintaining high specificity, meaning they confidently pass bad summaries rather than obviously flailing.

On humor, it’s almost slapstick: LLM judges rated irrelevant, nonsensical responses as highly funny (mean scores 2.18-3.29 on a scale where humans rated the same content 0.681) — Spearman correlation with human funniness ratings sat at 0.169-0.266 across Claude Sonnet 4, GPT-4.1, and Gemini 2.5 Pro. To be fair, humans don’t agree with each other about what’s funny either (31.7% pairwise agreement) — but the judges tracked human consensus only ~52-58% of the time, which is barely better than a coin flip on top of an already-noisy target.

On safety judgment, the numbers get genuinely concerning. A study found LLM safety judges reaching near-zero or negative Krippendorff’s alpha on “operational misuse” harm categories — meaning the raw agreement numbers looked fine only because of label imbalance, not real signal — with identical queries labeled “safe” anywhere from 12% to 83% of the time depending purely on which model did the judging.

On clinical/global-health content, even the best-performing judge (Claude Opus-class) reached human-equivalent performance on only 4 of 11 evaluation criteria, and degraded further outside English.

And there’s a meta-point that ties all of this together: raw correlation or percent-agreement can overstate reliability, because it doesn’t correct for chance agreement on skewed labels. A large benchmark called “Judge’s Verdict”, using Cohen’s Kappa across 54 judge configurations, found only 27 of 54 actually showed genuinely human-like agreement patterns once chance was properly subtracted out. Half the field’s “reliable” judges are, statistically, a coin flip with good PR.

7. When Judges Get Attacked on Purpose

Everything above is what happens when nobody’s trying to break the judge. Things get worse fast once someone is.

One paper titled, with admirable bluntness, “One Token to Fool LLM-as-a-Judge,” showed that trivial adversarial tokens can fool even frontier reasoning-model judges like o1 and Claude-4-class reward models. Under self-play optimization pressure, reference-free judges reward-hack toward responses that are more convincing rather than more correct — the gap between judge-approval rate and actual accuracy widened to 0.74, and critically, this transfers across model families and survives a three-judge ensemble, which still accepted the hacked, wrong answers 55% of the time. Ensembling three judges sounds like a safety margin. It bought almost nothing.

Under adversarial/jailbreak distribution shift, safety judges collapse to AUROC ~0.48 — genuinely indistinguishable from random guessing. And there’s a subtler contamination problem called “preference leakage”: judges show systematic bias toward outputs from “student” models related to the judge’s own training lineage, which is exactly the kind of thing that quietly corrupts a benchmark leaderboard without anyone noticing until someone goes looking.

The throughline: everything in Section 5 (position, verbosity, self-preference, format bias) is a judge being imprecise. Everything in this section is a judge being actively exploitable — which matters enormously once you remember that RLAIF pipelines use judge verdicts as a training reward signal. An exploitable reward signal doesn’t just misjudge outputs. It teaches the next model exactly how to game it.

8. The Mitigation Toolkit, Ranked by How Solved It Actually Is

Mature and widely deployed:

• Chain-of-thought + form-filling (G-Eval): make the judge reason before scoring, not just spit out a number.
• Reference-guided grading: have the judge solve the problem itself first, then grade against its own solution. MT-Bench’s authors showed this cut math-grading failure from ~70% to ~15% — the single biggest before/after number in this entire post.
• Position-swap / order randomization: run the comparison both ways, only trust verdicts that agree. Standard in MT-Bench methodology and Vertex AI AutoSxS.
• Rubric decomposition: score independent sub-criteria instead of one holistic verdict — more interpretable, harder to game wholesale.

Popular, but with known blind spots:

• Panel of LLM evaluators (PoLL) / juries: diverse judges canceling out idiosyncratic bias. Cheaper than one giant judge, and reduces self-enhancement bias — if the panel is genuinely diverse.
• Multi-agent debate (ChatEval): persona-diverse agents argue before verdicting. The catch, straight from ChatEval’s own ablation: the gains vanish with homogeneous personas.
• The blind spot both share: a 2026 study titled “Nine Judges, Two Effective Votes” showed that correlated errors across same-family judges mean a 9-judge panel can carry the independent information content of only ~2 effective votes. Ensembling doesn’t buy you what it looks like it buys you unless you actually measure and select for independence.

Newest, least production-hardened:

• RL-trained “thinking” judges (JudgeLRM, J1): reasoning-trained judges outperforming much larger SFT-only judges, with meaningful multilingual accuracy gains.
• Calibration/uncertainty quantification: linear probes on judge hidden states (cheaper and better-calibrated than asking the judge to state its own confidence), TH-Score ensemble fusers, and trust-or-escalate cascades that fall back to a stronger judge (or a human) when confidence is low.
• Constitutional-lineage rubric judges: Anthropic’s Constitutional Classifiers, now Constitutional Classifiers++ (Jan 2026), pairing written principles with rubric-generation to bridge human policy and machine reward.

The de facto industry standard, and it isn’t a technique at all — it’s a validation habit: benchmark your judge against a small human-labeled gold set using chance-corrected agreement (Cohen’s kappa or Krippendorff’s alpha — not raw percent agreement, which lies to you on skewed labels), iterate the judge prompt against your own disagreements with it, prefer binary over Likert, and never fully retire the human reviewer. Every credible production pipeline in this post does some version of this. None of them skip it.

9. The Honest Interlude: I Tried to Find You Something Novel

Here’s the part I promised at the top. I had five grounded, technically-sound proposals, each targeting a genuine gap the survey surfaced. I had three independent skeptic agents try to kill each one — find prior art, poke the mechanism, default to “no” unless proven otherwise. Here’s the body count:

Zero for five. And I want to be direct about why that’s the interesting result rather than a disappointing one: every single “gap” my agents identified as under-explored had already been substantially closed by a real paper, and in two cases, closed within the previous few months. That’s not a story about my research process being bad. It’s a story about how fast this specific field is moving right now. If a competent, well-grounded ideation pass — searching the literature, checking mechanisms, being honest about limitations — gets scooped this reliably, the honest takeaway isn’t “invent something,” it’s “go read what already exists, because it’s better than what you’d invent, and it’s newer than you’d expect.”

10. Where the Frontier Actually Is

If you’re an engineer looking for where to actually contribute — not “impressive-sounding project” contribute, but real open-problem contribute — here’s what the graveyard above points at:

• Panel diversity that’s measured, not assumed. Nine Judges shows the problem; nobody has shipped a production-grade selection method that engineers panels for genuine statistical independence rather than nominal vendor diversity.
• Calibration under adversarial and cross-cultural pressure specifically, not just in-distribution accuracy. The existing calibration literature is real and works — on the easy regime. Nobody has proven it survives the regime where it actually matters.
• Defending against reward-hacking under self-play, where the judge is training the thing that’s learning to fool it. The 0.74 judge-truth gap surviving a three-judge ensemble is a five-alarm fire that the field has diagnosed but not extinguished.
• Cross-lingual/cross-cultural certification as an operational gate, not a research metric — turning “here’s our multilingual accuracy number” into “here’s a machine-checkable go/no-go before you ship this judge in a market you haven’t validated it for.”

None of these require you to out-invent the field. They require you to pick one of these open problems and go deeper than a survey paper — which, per Section 9, is a genuinely different and much harder bar than it sounds.

11. The War Room Reference Table

Conclusion: The Fundamentals Are Settled. The Frontier Is Not.

Here’s the honest shape of where things stand: the basic case for LLM-as-a-judge — chain-of-thought scoring, reference-guided grading, position-swap consistency checks, rubric decomposition, and validating everything against a small human-labeled gold set with chance-corrected agreement — is genuinely solved. It works, it’s deployed at scale by every serious lab and platform in this post, and if you’re building an eval pipeline today, that toolkit is your starting point, not a research project.

What’s not solved is everything that happens once the judge is under real pressure: adversarial optimization, safety-critical or culturally-specific content, and panels that look diverse but aren’t. That’s not a gap you close by reading one survey and having a clever idea — I tried, and the literature ate my lunch five times in a row. It’s a gap you close by actually reading the last two years of this field, which is exactly what six research agents and an afternoon of adversarial review just did for both of us.

If you’re an engineer trying to get on board with evaluation work, this is, genuinely, one of the best times to start. The fundamentals are stable enough to build on immediately. The open problems are documented, real, and moving fast enough that “novel” has a shelf life measured in months, not years. That’s not a discouraging fact. It means the frontier is close enough to touch.

References

Foundations: G-Eval · AlpacaFarm · AlpacaEval · MT-Bench & Chatbot Arena · Chatbot Arena platform paper · RewardBench · AgentBench · Constitutional AI · RLAIF · Llama 3 Herd of Models · Length-Controlled AlpacaEval

Biases: Large Language Models are not Fair Evaluators · Judging the Judges: Position Bias at Scale · Self-Preference Bias in LLM-as-a-Judge · LLM Evaluators Recognize and Favor Their Own Generations · Judging the Judges: Bias Mitigation Strategies · Your LLM Judge Has a Length, Position, and Format Bias · Turning Bias into Bugs · Justice or Prejudice? (CALM framework)

Calibration and domain failures: Judge’s Verdict · Human Evaluators vs. LLM-as-a-Judge in Global Health · HumorRank · LLM Judges Inconsistently Disagree Across Safety Criteria · Overconfidence in LLM-as-a-Judge · Calibrating LLM Judges: Linear Probes

Adversarial robustness: One Token to Fool LLM-as-a-Judge · More Convincing, Not More Correct · A Coin Flip for Safety · Preference Leakage

Mitigations: Replacing Judges with Juries (PoLL) · ChatEval · Nine Judges, Two Effective Votes · JudgeLRM · J1: Incentivizing Thinking in LLM-as-a-Judge · Trust or Escalate · Constitutional Classifiers++ · Rethinking Rubric Generation (RRD)

Production practice: Anthropic’s multi-agent research system · OpenAI Graders · OpenAI Evaluation Best Practices · Vertex AI AutoSxS · Scale SEAL Leaderboards · Databricks LLM Judges · Hamel Husain’s LLM Judge Guide · Braintrust: How to Eval

I’m an infrastructure engineer with 11+ years in distributed systems (D-Wave, Enbala, MasterCard, Cisco), currently going deep on LLM serving, evaluation, and agent infrastructure. This post is grounded in a real multi-agent research pipeline — six survey agents, a synthesis pass, five proposals, and fifteen adversarial reviews — not just my own reading. I write what I actually learned, including the parts where my own ideas lost.

Find me on GitHub: kraghavan

Find me on Linkedin: Karthika Raghavan

A layer-by-layer guide for DevOps, SRE, and infrastructure engineers building scalable, secure, production-grade LLM services — and wondering why the naive approach keeps falling over.

Lets hear a familliar story. An engineer, often a good one, spins up vLLM, drops it behind a round-robin load balancer, confirms it responds to curl, and ships it. The demo works. The first week of real traffic does not. TTFT climbs to 8 seconds. A GPU OOMs at 2am. Nobody can explain why because there’s nothing between the load balancer and the model worker that actually understands what’s happening.

The load balancer isn’t the problem. The mental model is.

Serving an LLM at scale is categorically different from serving a stateless REST API. An LLM response isn’t a database row lookup — it’s a two-phase, stateful compute process where the routing decision made before the request ever touches the GPU determines whether your p99 latency is 800ms or 8 seconds. Get the routing wrong and no amount of GPU capacity saves you.

This is the architecture that gets it close to production grade. We’ll cover each layer — what it does, why it exists, and what a real config looks like. If you want to understand what’s actually happening inside the GPU once a request lands, I’d start with my deep inference walkthrough first. This post picks up where that one ends.

One deliberate scope boundary worth stating upfront: this article covers the serving infrastructure — the layers between your user and your GPU. The application layer above it, including retrieval pipelines, vector databases, MCP tool integration, and orchestration frameworks, is a separate concern and not covered here. Those components plug in above Layer 3, but how they do so is its own topic.

Why You Can’t Just Load-Balance an LLM

The naive mental model — round-robin across replicas — breaks for four distinct reasons that compound on each other.

KV cache is stateful per-replica. When a request lands on a replica that hasn’t seen this prompt prefix before, it recomputes the entire KV cache from scratch. Route to the wrong replica and you pay full prefill cost every time. Route to the right one and prefill is nearly free. A round-robin load balancer has no idea this distinction exists. The vLLM prefix caching docs explain the underlying mechanism well if you want to go deeper on how blocks are hashed and matched.

Prefill and decode have wildly different compute profiles. Prefill is GPU compute-bound — it parallelizes across all input tokens at once. Decode is GPU memory-bandwidth-bound — it generates one token at a time against an ever-growing KV cache. Running both phases on the same pod pool means a single long-context prefill can stall your decode queue for every other in-flight request. These phases want different hardware and different scheduling treatment.

Token budgets are not request counts. Traditional RPS-based load balancing treats every request as equivalent. It’s completely blind to the difference between a 200-token chat turn and a 32k-token document summarization. A single 32k-context request consumes the GPU memory and compute that would otherwise serve 50 short requests. Your quota system needs to reason about tokens, not connections.

GPU memory is the real constraint. When your KV cache is full, new requests either queue and wait or trigger eviction of in-progress requests — which causes latency spikes as those requests recompute from scratch. A load balancer tracking CPU and RAM is measuring the wrong thing. The constraint is VRAM, and specifically KV cache block occupancy.

The punchline: LLM load balancing is a scheduling problem disguised as a networking problem. The stack below exists to solve the scheduling problem correctly at each layer.

The Stack at a Glance

Client
  └─ NGINX / Envoy              ← TLS termination, connection limits, WAF, basic rate limit
       └─ Kong Gateway           ← Auth, per-tenant quota, request logging, plugin chain
            └─ LiteLLM / L3 proxy  ← Model abstraction, fallbacks, cost attribution, caching
                 └─ llm-d        ← KV-cache-aware scheduling, prefill/decode disaggregation
                      └─ vLLM replicas  ← PagedAttention, continuous batching
                           └─ GPU

Before we get into each layer, one thing that trips people up early: NGINX at Layer 1 is not your LLM gateway. It’s your platform gateway. Your LLM service lives alongside your user API, your auth service, your billing endpoints. NGINX routes all of them. This distinction changes how you think about security, compliance, and observability across the entire stack.

Layer 1 — Edge Ingress (NGINX / Envoy / Traefik)

What it does: TLS termination, unified platform routing, WAF filtering, IP-based rate limiting. Everything entering your cluster north-south passes through here.

This layer routes your entire platform — not just LLM traffic. Your NGINX config has location blocks for your user API, your auth service, your product endpoints, and your LLM service. Security controls here — SQL injection filtering, request size limits, DDoS protection — apply uniformly across everything. That uniformity is the value. The moment you build a separate LLM-specific ingress, you’ve split your security posture in two and you’re maintaining both forever.

# nginx.conf — unified platform ingress
upstream app_backend {
    server app-service:8080;
}

upstream llm_gateway {
    server litellm:4000;
}

server {
    listen 443 ssl;
    server_name api.yourplatform.com;

    # Standard application traffic
    location /api/v1/users/ {
        proxy_pass http://app_backend;
        include /etc/nginx/modsecurity.conf;   # WAF: SQL injection, XSS
    }

    location /api/v1/auth/ {
        proxy_pass http://app_backend;
        limit_req zone=auth_limit burst=10 nodelay;
    }

    # LLM traffic — routed to LLM gateway (L3)
    location /api/v1/chat/ {
        proxy_pass http://llm_gateway;
        proxy_read_timeout 120s;          # Default 60s is not enough for long generations
        proxy_buffering off;              # Critical: SSE streaming breaks without this
        proxy_set_header X-Real-IP $remote_addr;
        limit_req zone=llm_limit burst=5 nodelay;
    }

    location /api/v1/embed/ {
        proxy_pass http://llm_gateway;
        proxy_read_timeout 30s;
    }
}

proxy_buffering off on the chat endpoint is not optional. Without it, NGINX holds the entire streaming response in its buffer before delivering it — your users see nothing until the full response is done. Verify this with a streaming test before you ship; it fails silently. The NGINX docs on proxying cover the interaction with SSE and chunked transfer encoding if you need the full picture.

Scalability: Completely stateless — add NGINX instances horizontally behind your cloud load balancer without coordination. Handles millions of concurrent connections.

Security: TLS offload, WAF rules, IP allowlisting, DDoS rate limiting. This is your perimeter — every request passes through before any application code sees it. ModSecurity is the standard WAF module for NGINX if you haven’t already integrated it.

Compliance: Full audit logging of inbound traffic. IP-based geo-blocking for data residency requirements. Request body size limits block oversized prompt payloads — raise the default 1MB limit to 10–50MB or legitimate LLM requests will be rejected.

Reliability: Health checks against all upstreams with automatic failover. If L3 goes down, NGINX returns a clean 502 rather than letting connections hang until they timeout.

Layer 2 — API Gateway (Kong / AWS API GW / Apigee)

What it does: Per-consumer identity, quota enforcement, centralized audit logging, compliance controls. Where Layer 1 knows what is coming in, Layer 2 knows who is sending it.

This is your consumer-facing contract layer. Every team, every external partner, every integration that calls your LLM service gets an API key managed here. Rate limits are enforced here — and critically, in token-per-minute terms, not requests-per-minute. When your security team asks “which consumers sent requests to the model between 9pm and midnight last Tuesday,” this is where that answer lives — not scattered across application logs.

Kong’s AI Gateway plugin adds prompt-level rate limiting and response caching as first-class plugins, which gives you a natural integration point between L2 and L3 concerns without duplicating configuration. AWS API Gateway is the natural choice if you’re already heavily on AWS and routing to Bedrock or SageMaker endpoints.

# Kong declarative config — AI traffic with per-consumer quota
services:
  - name: llm-service
    url: http://litellm:4000
    routes:
      - name: chat-route
        paths:
          - /api/v1/chat
    plugins:
      - name: key-auth                   # Identify the consumer by API key
      - name: rate-limiting
        config:
          minute: 60                     # 60 req/min per consumer key
          policy: redis                  # Shared limit across Kong replicas
          redis_host: redis
      - name: request-size-limiting
        config:
          allowed_payload_size: 10       # 10MB cap — prevents prompt stuffing
      - name: http-log                   # Immutable audit trail per request
        config:
          http_endpoint: http://audit-service/log

The ownership model here matters: platform teams own L2, ML teams own L3. Platform controls who gets a key, what their quota is, and what compliance requirements look like. ML teams control which models are available, how fallbacks work, and what cost attribution looks like. Different concerns, different change cadences — keeping them in separate layers means neither team blocks the other’s deploys.

Scalability: Centralized quota enforcement means no single consumer can saturate your GPU pool regardless of how aggressively they call. Token budgets per API key enforced before the request reaches inference.

Security: API key lifecycle management, scoped permissions, key rotation without application changes. Auth concern cleanly separated from inference logic.

Compliance: Immutable per-request audit log with consumer identity attached. This is what SOC2 and HIPAA auditors look at — you can’t reconstruct this after the fact from vLLM logs.

Reliability: Circuit breaker per consumer. A runaway batch job from one team hitting quota doesn’t degrade other consumers — they’re rate-limited independently.

Layer 3 — LLM-Aware Proxy (LiteLLM / PortKey)

What it does: Model name abstraction, fallback chains, cost attribution, semantic caching. This is the first layer in the stack that actually understands what an LLM request is.

LiteLLM gives your application a single OpenAI-compatible endpoint while managing all the complexity underneath — routing to different providers, retrying failed requests, tracking cost per model per team, and caching repeated prompts. Your application code never needs to know whether it’s talking to a self-hosted Llama cluster or Anthropic’s API. PortKey is a solid alternative if you need stronger guardrails and observability built in rather than composed through callbacks.

For teams that need to go further — classifying prompts by data sensitivity, enforcing that PII never routes to cloud providers — LiteLLM is a solid foundation that you’d extend with custom middleware at this layer. I’ve been exploring a similar pattern as a side project, though it’s still a long way from production-ready.

# litellm config.yaml
# Illustrative — verify field names and routing strategy values against
# your installed LiteLLM version before deploying
# Docs: https://docs.litellm.ai/docs/routing
model_list:
  # Primary: internal vLLM cluster (routed through llm-d at L4)
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_base: http://llmd-router:8080/v1
      api_key: "sk-internal"

  # Fallback: Anthropic (if internal cluster is saturated or unavailable)
  - model_name: gpt-4o
    litellm_params:
      model: anthropic/claude-sonnet-4-5
      api_key: os.environ/ANTHROPIC_API_KEY

  # Emergency fallback
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

router_settings:
  routing_strategy: least-busy
  num_retries: 2
  retry_after: 5

litellm_settings:
  success_callback: ["prometheus"]
  failure_callback: ["prometheus"]
  cache: true
  cache_params:
    type: redis
    host: redis
    port: 6379
    ttl: 3600

Semantic caching is worth enabling from day one. For any workload with repeated questions — knowledge base Q&A, customer support, internal tooling — cache hit rates of 30–40% are realistic. That’s 30–40% of inference traffic returning zero GPU compute. Enable it, set a sensible TTL for your workload, and check the hit rate in Grafana after your first week of real traffic.

One thing worth understanding clearly: L3 is the only layer in this entire stack that can stop a long prompt from traveling all the way to the GPU. Every other layer — NGINX, Kong, llm-d — reads the full request body, makes a decision, and forwards it wholesale. llm-d’s KV cache routing is often misread as meaning the prompt doesn’t travel to vLLM; it does, every time. What llm-d skips is the compute on the cached prefix, not the transmission of the prompt itself. The only genuine short-circuit is a semantic cache hit at L3, where LiteLLM returns the stored response from Redis and the prompt goes no further. For workloads with long system prompts — 4k, 32k, 128k tokens — this distinction has real implications for both latency and memory pressure across the stack. A cache miss on a 128k-token prompt means roughly 500KB of JSON moving through every layer, held in memory at each hop, before a single token is generated.

Provider API keys belong in a secrets manager, not in this config file. A leaked LiteLLM config that contains API keys in plaintext is an expensive incident.

Scalability: Run multiple LiteLLM instances behind a load balancer, all sharing a Redis cache. Fallback chains mean no single provider outage takes down your service.

Security: Cloud API keys centralized here — application code never holds provider credentials. Custom classification hooks live at this layer, not scattered across application logic.

Compliance: Per-model cost tracking for internal chargebacks. Audit log of which model handled which request. Policy enforcement on which teams can access which models.

Reliability: Automatic failover to fallback providers with configurable retry. Health checks per backend. If your internal cluster is unhealthy, traffic shifts to cloud automatically without any application change.

Layer 4 — Inference Router (llm-d / KServe)

What it does: KV cache-aware routing, disaggregated prefill/decode scheduling, GPU queue depth balancing. This is the layer that makes a fleet of GPU workers behave as a single coherent system rather than a collection of independent replicas.

This is the layer most infrastructure engineers haven’t seen before, and it makes the most difference at scale.

KV cache-aware routing is the core capability. llm-d maintains cluster-wide visibility into which KV cache blocks each pod currently holds. When a new request arrives carrying a system prompt that matches a cached prefix on a specific pod, llm-d routes it there. The first request with that prompt pays the full prefill cost. Every subsequent request with the same prefix costs almost nothing on prefill. Without this, every pod in your cluster recomputes the same system prompt KV vectors independently, for every request, forever.

Disaggregated prefill/decode is the architectural step beyond that. Because prefill is compute-bound and decode is memory-bandwidth-bound, running them on the same pods creates resource contention — long prefills starve decode queues at exactly the wrong moment. llm-d lets you run dedicated prefill pods (sized for compute) and dedicated decode pods (sized for memory bandwidth) as separate pools. The disaggregated serving RFC in the vLLM project is worth reading for the hardware-level reasoning behind why this separation matters.

# llm-d InferenceModel CRD
# Field names representative of llm-d v0.x — verify against the
# current operator spec at github.com/llm-d/llm-d before applying
apiVersion: inference.llm-d.io/v1alpha1
kind: InferenceModel
metadata:
  name: llama3-70b-instruct
  namespace: llm-serving
spec:
  modelName: meta-llama/Llama-3.1-70B-Instruct

  routing:
    strategy: kv-cache-aware

  prefillDecode:
    disaggregated: true
    prefillPool:
      replicas: 4
      resources:
        limits:
          nvidia.com/gpu: 4              # H100s — need FLOP throughput for prefill
    decodePool:
      replicas: 8
      resources:
        limits:
          nvidia.com/gpu: 2              # A10Gs — need HBM bandwidth for decode

  kvCache:
    prefixCachingEnabled: true
    blockSize: 16
    maxCachedPrefixes: 1000

  scheduling:
    queueDepthThreshold: 4

llm-d vs vLLM Standalone

This is the distinction most people are missing a mental model for.

vLLM is a superb inference engine. llm-d is the orchestration layer that makes a fleet of vLLM workers behave as a single, cache-aware system. You need both.

llm-d vs KServe — When to Use Which

KServe appears in L4 comparisons often enough that it deserves a direct answer. They solve adjacent but different problems.

Reach for KServe when you’re already on the Kubeflow ecosystem, you need multi-framework serving (not just vLLM — also TensorFlow, PyTorch Serve, Triton), or model versioning and canary rollout are first-class concerns. KServe has a mature InferenceService CRD with built-in traffic splitting and revision history that’s genuinely useful for teams managing many models across many versions.

Reach for llm-d when you’re optimizing specifically for LLM inference scheduling — KV cache locality, prefill/decode disaggregation, and GPU queue depth awareness. llm-d is built from the ground up around the resource dynamics of transformer inference. KServe treats the inference engine as a black box; llm-d is deeply aware of what’s happening inside it.

In practice: KServe is the right answer for general-purpose model serving platforms. llm-d is the right answer when LLM throughput and TTFT are your primary optimization targets.

Scalability: Scale prefill and decode capacity independently. Add prefill nodes when long-context load increases; add decode nodes when concurrent users increase. KV cache routing reduces redundant GPU work across the fleet.

Security: Namespace isolation per model and team. RBAC on InferenceModel CRDs. Pod-level network policy enforcement between pools.

Compliance: Model-level request audit trail. GPU resource quotas per team enforced at the scheduling layer. Policy on which models serve which namespaces lives here, not in application code.

Reliability: Pod-level health awareness — if a decode pod OOMs, llm-d drains it and reroutes without dropping in-flight requests. Priority scheduling for latency-critical traffic over batch workloads.

Layer 5 — GPU Worker (vLLM / TensorRT-LLM / TGI)

What it does: Tokenization, prefill, decode loop, KV cache management. This is where inference actually happens and tokens are generated.

The internals — PagedAttention, continuous batching, KV cache lifecycle, TTFT vs ITL, why decode is memory-bound — are covered in depth in my inference walkthrough. Anyscale’s post on continuous batching is also worth reading for the throughput argument. What matters here is what the GPU worker exposes upward to llm-d and the configuration decisions that affect production stability.

# vLLM Deployment — decode pool
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-decode-pool
  namespace: llm-serving
spec:
  replicas: 8
  selector:
    matchLabels:
      pool: decode
  template:
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          args:
            - "--model"
            - "meta-llama/Llama-3.1-70B-Instruct"
            - "--served-model-name"
            - "llama3-70b-instruct"
            - "--max-model-len"
            - "8192"
            - "--tensor-parallel-size"
            - "4"                # Split model weights across 4 GPUs on this node
            - "--gpu-memory-utilization"
            - "0.90"            # Never 1.0 — CUDA needs headroom for runtime allocations
            - "--enable-prefix-caching"
            - "--disable-log-requests"  # Log at L2/L3; avoid double-logging at high throughput
          resources:
            limits:
              nvidia.com/gpu: 4  # Must match --tensor-parallel-size exactly
          ports:
            - containerPort: 8000
          env:
            - name: VLLM_PROMETHEUS_ENABLED
              value: "1"        # These metrics are what llm-d reads for scheduling decisions
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 30
            periodSeconds: 10

--gpu-memory-utilization 0.90 matters more than it looks. vLLM uses this value to pre-allocate its KV cache block pool at startup. At 1.0 there’s no headroom left for CUDA runtime allocations — under load you get OOM crashes with no graceful degradation. I run 0.90 in production and haven’t had an unplanned OOM since making that change. The vLLM engine arguments reference has the full list of tunable parameters if you want to go further.

Tensor Parallelism and Pipeline Parallelism

Most production deployments of 70B+ parameter models require parallelism — the model simply doesn’t fit in a single GPU’s VRAM. vLLM supports two strategies and you’ll almost certainly need to reason about both.

Tensor parallelism (TP) splits a model’s weight matrices across multiple GPUs on the same node. Each GPU holds a shard of every layer, and they cooperate on every single forward pass using NCCL all-reduce operations. --tensor-parallel-size 4 means one logical replica is served by 4 GPUs working together. The key constraint: your nvidia.com/gpu pod resource request must equal your TP size. Mismatch these and vLLM either fails to start or silently under-uses the hardware.

Model: Llama-3.1-70B  (~140GB in bf16)
Single A100 80GB VRAM → doesn't fit
TP=2 across 2x A100 80GB → 70GB per GPU → fits with headroom
TP=4 across 4x A100 80GB → 35GB per GPU → fits comfortably, more KV cache room

The tradeoff with TP is that every decode step requires an all-reduce across all N GPUs. This adds ~0.5–2ms of inter-GPU communication latency per token, which is negligible on NVLink but noticeable over PCIe. Keep TP within a single node on fast interconnect whenever possible.

Pipeline parallelism (PP) splits the model’s layers across multiple nodes rather than splitting each layer across GPUs. Node 1 runs layers 1–40, node 2 runs layers 41–80. Requests flow through the pipeline sequentially. PP is the right tool when your model is too large for a single node even with TP, but it introduces pipeline bubbles — nodes idle waiting for the previous stage to finish. In practice, most teams reach for TP first and only add PP when they’re running models at the 405B+ scale.

How this interacts with llm-d: llm-d routes at the replica level. A replica with TP=4 looks like a single logical worker to llm-d — it routes a request to that replica, and the 4-GPU coordination happens transparently inside the pod. Your InferenceModel CRD’s nvidia.com/gpu: 4 in the decode pool spec should reflect the TP size you’ve chosen. If you later change TP, update both the vLLM args and the CRD resource limits together.

Inference engine selection:

Scalability: PagedAttention enables 2–4x more concurrent requests versus naive KV allocation. Continuous batching keeps GPU utilization high even when request lengths vary widely. TP lets you serve models that would otherwise be impossible to run, at the cost of tighter node topology requirements.

Reliability: KV cache eviction prevents OOM at the cost of latency spikes for the evicted requests — which is the right tradeoff. /health endpoint for readiness and liveness probes. Graceful shutdown drains in-flight requests before terminating.

What to Instrument at Each Layer

These are the metrics you watch in production to know the system is healthy and catch problems before users do — not a pre-launch checklist.

NGINX — connection saturation and upstream latency

nginx_connections_active           # Are you close to worker_connections limit?
nginx_upstream_response_time_ms    # p95/p99 latency from NGINX's perspective
nginx_upstream_connect_time_ms     # Spikes here mean L3 is slow to accept connections

Kong — quota headroom and 429 rate

kong_http_requests_total{code="429"}   # Rising 429s = consumers hitting quota
kong_upstream_latency_ms               # Time Kong adds before forwarding
quota_headroom_percent per consumer    # How close is each tenant to their limit?

LiteLLM — classification latency and routing distribution

litellm_request_duration_seconds    # End-to-end L3 latency
litellm_cache_hit_rate              # Below 0.20 on a Q&A workload is worth investigating
litellm_fallback_count_total        # Rising fallbacks = primary cluster has a problem

llm-d — cache hit rate, queue depth, prefill vs decode split

The cache hit rate is the single most important metric at this layer. Everything else follows from it.

llmd_kv_cache_hit_rate              # Target > 0.5 for workloads with shared system prompts
llmd_queue_depth_per_replica        # Alert if any replica queue > 10
llmd_prefill_latency_seconds        # Should be low if prefix caching is working
llmd_decode_latency_seconds         # Scales with KV cache size; memory-bound
llmd_routing_decisions{reason}      # cache_hit vs cold_start distribution

vLLM — GPU KV cache utilization, batch size, generation throughput

vllm:gpu_cache_usage_perc           # Alert at 0.85 — at 0.95 you're already evicting
vllm:num_requests_waiting           # Queue depth this replica can't yet serve
vllm:gpu_prefix_cache_hit_rate      # Local prefix cache effectiveness
vllm:time_to_first_token_seconds    # TTFT histogram — your primary latency SLO metric
vllm:time_per_output_token_seconds  # ITL histogram — decode throughput per user
vllm:generation_tokens_total        # Total tokens generated — maps directly to GPU cost

The relationship between layers matters for incident diagnosis. A spike in vllm:time_to_first_token_seconds with a simultaneous drop in llmd_kv_cache_hit_rate means you’ve lost prefix cache warmth — look for a recent deployment that cleared pod state. A spike in TTFT with a stable cache hit rate but rising vllm:gpu_cache_usage_perc means you’re running out of KV capacity and need more decode pods, not better routing.

Full Request Path, End to End

Walk a single request: POST /v1/chat/completions with a 4k-token system prompt and a 200-token user turn.

POST /api/v1/chat/completions
[4096-token system prompt + 200-token user turn]
     |
     v
[L1] NGINX
     - TCP accepted, TLS terminated
     - Connection limit checked — within bounds
     - WAF: request inspected, size limit checked (well under 10MB cap)
     - Routed to LLM gateway (L3)
     - proxy_buffering off: SSE will stream through cleanly
     |
     v
[L2] Kong
     - JWT / API key validated → consumer: "internal-team-a"
     - TPM quota checked: 47 of 500 req/min used — allowed
     - Trace ID stamped on request headers
     - Audit log entry written: consumer + timestamp + path
     |
     v
[L3] LiteLLM
     - Semantic cache lookup: has this exact prompt been seen? → miss
     - Routing decision: internal cluster healthy → route there
     - Request forwarded to llm-d with trace ID preserved
     |
     v
[L4] llm-d
     - System prompt hashed → looked up in KV cache index
     - Match found: replica R2 has the 4096-token prefix cached
     - Request routed to R2 specifically
     - Decode queue depth on R2: 2 active requests — within threshold
     |
     v
[L5] vLLM on replica R2 (TP=4 across 4x A100)
     - Tokenization: 200 new user tokens (system prompt already tokenized and cached)
     - Prefill: only the 200-token user turn computed — 4096 prefix tokens skipped
     - Each GPU shard processes its weight slice; NCCL all-reduce syncs after each layer
     - Decode loop begins: tokens generated one at a time, streamed back
     - KV blocks appended for the new 200 tokens
     - Response streams back token-by-token through the same chain
     |
     v (response path)
[L2] Kong
     - Token count recorded against consumer quota bucket
     |
[L1] NGINX
     - SSE frames streamed to client as they arrive
     - Access log entry written

Total routing overhead L1-L4: ~3-8ms
What your users are waiting on: GPU compute at L5
The system prompt's KV cache being warm on R2 saved ~400ms of prefill.

Decision Matrix: Which Layers Do You Actually Need?

The principal engineer’s test for each layer you add: if I removed this tomorrow, what specifically breaks? If you can’t answer that concretely, the layer isn’t earning its place yet.

Closing

None of this is over-engineering. Each layer exists because something specific breaks without it. NGINX without proxy_buffering off breaks streaming silently. A proxy without a fallback chain means a provider outage is your outage. llm-d without KV cache routing means you’re paying GPU costs for work that’s already been done on another pod.

Start with NGINX → LiteLLM → vLLM. That’s a real production stack that serves real traffic well. Add layers as the specific pressure they solve becomes concrete — not before. The stack I’ve laid out here is the shape of where you’ll end up as scale and compliance requirements grow.

The five layers above exist because the tools your team already knows — NGINX, Kong, Kubernetes — were built for networking problems. LLM serving is a scheduling problem wearing networking clothes, and each layer is one part of that disguise being pulled off.

The specific tools here — especially at L4 — will change faster than any other layer in this stack. llm-d is pre-v1 and the operator API is still settling. The shape of the stack won’t change. The components will.

The GPU is the expensive part. Everything above it exists to make sure that GPU spends its cycles on inference — not on routing errors, redundant compute, or requests that should have been served from cache.

Karthika Raghavan — infrastructure engineer, 11+ years in distributed systems across D-Wave, Enbala, MasterCard, and Cisco. Currently focused on LLM serving infrastructure and inference optimization. The experiments behind this post ran on everything from a Mac Mini M4 to Lambda Labs GH200 clusters.

Related reading:

My posts:

• What Is LLM Inference, Really? A Deep Technical Walkthrough — L5 internals: PagedAttention, KV cache, TTFT vs ITL

External references:

• vLLM — Automatic Prefix Caching
• vLLM Engine Arguments Reference
• PagedAttention paper (Kwon et al., 2023)
• Continuous Batching: A New Trick for LLM Throughput — Anyscale
• llm-d GitHub
• KServe documentation
• Kong AI Gateway plugin
• LiteLLM documentation
• ModSecurity / OWASP Core Rule Set
• NGINX proxy_buffering module reference
• NVIDIA NCCL — GPU communication library
• vLLM distributed inference and serving

The hypothesis: if you instrument a multi-agent system properly, you can observe it the same way you observe infrastructure. Conversations become traces. Token costs become metrics. Agent quality becomes an SLO.

This is the story of testing that hypothesis. The build, the dashboard, the moment it looked like it worked, the moment I realised it hadn’t — and what the gap reveals about where AI agent systems actually break. I’ll be honest about all of it.

The Context: Why This Matters

I’ve spent more than half a decade making distributed systems observable. Prometheus metrics, distributed traces, structured logs, SLOs, alerting pipelines. When a service degrades, I know where to look.

LLM agents have none of this. Until now, “observability” for an AI agent meant reading stdout and hoping the error messages were useful. Token costs were buried in billing dashboards three clicks deep. Quality was vibes-based — you knew something was off when your users told you. There was no equivalent of a trace showing “this is where the agent went wrong.”

Grafana AI Observability changes the mental model. Conversations become traces. Token counts become metrics. Agent quality becomes an SLO. The existing Grafana workflows — alerting, dashboards, explore, drill-down — work on AI data the same way they work on infrastructure data.

That’s the dream. This post is about testing it in the real world, four days after launch.

The Plan: An AI-Powered SRE War Room

The idea was simple: build a multi-agent system that mirrors how a real SRE team handles incidents, then instrument it completely with Grafana’s new Sigil SDK.

Five agents, each with a distinct role:

The model selection was deliberate. Triage and execution are fast, structured tasks — Haiku handles them in under 3 seconds. Runbook planning and post-mortem writing need reasoning across complex context — that’s Sonnet territory. The cost-cop agent using Haiku to audit the Sonnet agents is the most efficient use of irony I’ve found in systems design.

Constraints: Everything runs on an M4 Mac Mini. No GPU. No cloud Kubernetes. Total demo budget: under $2 in Anthropic API credits.

The Setup: Twenty Fake Incidents That Feel Real

The system needed incident volume. I built a library of 20 canned PagerDuty-style alert payloads covering the greatest hits of production failure:

{
  "incident_id": "INC-0001",
  "title": "payment-service: P1 OOMKilled pods - 3 replicas down",
  "severity_hint": "P1",
  "affected_services": ["payment-service", "payment-worker", "checkout-api"],
  "annotations": {
    "summary": "Pod OOMKilled in payments namespace",
    "description": "Container exceeded memory limit of 512Mi. 3/5 replicas down. Revenue impact estimated at $4,200/min."
  }
}

Each payload is realistic enough that the agents treat it as real. The triage agent doesn’t know it’s fake. It responds with genuine P1 urgency. That’s the point — the mocking is deliberate and specific, not random noise.

The incident library covers: OOM kills, connection pool exhaustion, Redis split-brain, TLS certificate expiry, etcd slow reads, CDN DDoS, CrashLoopBackOff with Kafka lag, Elasticsearch cluster RED, Istio sidecar memory leaks, GPU OOM on inference workloads.

An “incident blaster” fires these scenarios at configurable intervals:

python -m scenarios.blaster --count 10 --interval 15

The blaster in action. INC-0001 is a P1 CDN DDoS. INC-0002 is a Redis split-brain with dual master election. The ✅ Sigil line confirms telemetry is shipping.

The Orchestration: LangGraph State Machine

Each incident runs through a LangGraph state machine. One incident = one conversation = one Grafana thread.

Incident Alert → triage-agent → runbook-agent → executor-agent → postmortem-agent
                                                                        ↓
                                              LangGraph IncidentState (shared)

The conversation_id is the key insight. Every agent that touches an incident gets the same conversation_id. When Grafana receives four Sigil generations with the same ID, it threads them into a single conversation. You see the complete incident lifecycle — from raw alert to finished post-mortem — in one view.

with sigil_client.start_generation(
    GenerationStart(
        model=ModelRef(provider="anthropic", name="claude-haiku-4-5"),
        conversation_id=incident_id,   # same ID across all 4 agents
        agent_name="triage-agent",
        system_prompt=SYSTEM_PROMPT,
        tags={"incident_id": incident_id},
    )
) as rec:
    response = claude_client.messages.create(...)
    rec.set_result(
        input=[user_text_message(alert_text)],
        output=[assistant_text_message(response.content[0].text)],
    )

That’s the entire instrumentation for one agent. Fifteen lines. The SDK handles token counting, latency tracking, and cost calculation automatically.

A note on prompt design: Each agent’s system prompt is a strict output contract. The triage agent prompt, for example:

You are a senior SRE triage specialist. Output ONLY valid JSON:
{
  "severity": "P1" | "P2" | "P3" | "P4",
  "affected_services": ["service-a"],
  "blast_radius": "single-region" | "multi-region" | "global",
  "triage_summary": "one paragraph"
}
Severity guide: P1 = complete outage, revenue impact, >10k users...
Do not include markdown fences or any text outside the JSON object.

The “do not include markdown fences” instruction is violated by Claude Haiku consistently regardless. Hence strip_fences(). The larger issue is that structured output prompts work well for simple classification but break down when the output space is ambiguous — which is exactly what happens on the complex incidents that end up escalated.

The Dashboard: What Grafana Actually Shows

AI Observability Landing Page

The AI Observability landing page after running 24 incidents. 5 agents registered, 24 conversations tracked.

The landing page summarises the last 24 hours: total conversations, average calls per conversation, token totals, cost. After running 10+ incidents you start to see the shape of your agent workload at a glance.

Conversations View — The Main Event

24 conversations, 3.88 average calls per conversation (exactly right for a 4-agent pipeline). Each row is one incident. The activity chart shows the cadence of the blaster runs.

Each row is one incident run. The columns show conversation ID, duration, call count, which agents participated, and which models were used. The conversation activity chart maps exactly to when the blaster was running — you can literally see when I paused it to debug something.

Clicking into a conversation is where it gets genuinely useful.

Conversation Drilldown — Where the Real Value Lives

One incident, four agents. Triage: 2.08s (Haiku), runbook: 8.57s (Sonnet), executor: 7.58s (Haiku), postmortem: 21.61s (Sonnet). Total: 37.8 seconds from alert to post-mortem.

The per-agent breakdown on the left shows:

• triage-agent: 2.08s (Haiku, exactly what we want)
• runbook-agent: 8.57s (Sonnet reasoning through a complex runbook)
• executor-agent: 7.58s (Haiku simulating kubectl commands)
• postmortem-agent: 21.61s (Sonnet writing a full post-mortem)

The right panel shows the actual content — the triage agent’s JSON classification, the runbook it matched, the alert context it processed. This is the debugging interface for AI agents that we’ve never had before.

The Postgres P1 (connection pool exhausted — 512/512). The triage agent correctly identified P1 severity and matched the DB Connection Pool Exhaustion runbook. All visible in Grafana.

Agents View — Who’s Doing the Work

All five agents registered. The ‘Top by Generations’ chart shows near-equal usage across triage, runbook, executor, and postmortem (23 each). The cost-cop-agent ran twice.

The Agent Footprint section on the right is interesting: triage-agent is actually the most expensive by token volume despite using Haiku, because it runs on every incident. The postmortem-agent costs less in aggregate despite using Sonnet, because its output is dense and well-structured.

The Cost Cop: An AI Auditing Other AIs

Every 10 incidents, the cost-cop agent wakes up, queries Grafana’s API, and produces a FinOps report. That report — generated by a Haiku agent — gets shipped back to Grafana as its own conversation thread.

The cost-cop agent’s conversation in Grafana. It identified postmortem-agent consuming 60% of total spend and recommended swapping to Haiku for an 83% cost reduction. This analysis is itself a Grafana conversation.

The output is pointed:

postmortem-agent is your problem child. 1-hour cost: $1.81 (60% of total spend). Projected monthly: $1,294. Postmortem generation is templated, deterministic work — swap to Haiku immediately. Projected savings: ~$1,050/month. Risk: Low. Test on 5% traffic first.

The cost-cop is Haiku auditing Sonnet. A cheap model correctly identifying that expensive models are doing work they’re overqualified for. That’s the FinOps signal that used to require a dedicated billing dashboard. Now it’s a Grafana conversation.

At this point in the experiment, everything was working. Telemetry flowing. Dashboard populated. Costs tracked. Agents completing pipelines. The hypothesis looked confirmed.

Then I looked at whether the agents were actually right.

When the System Was Confidently Wrong

The auth-service JWT rotation incident (INC-0006 in the blaster output) is the one that made me sit up.

The alert: auth-service: 401 error rate spike — 23% of requests failing. The triage agent correctly classified this as P1 with global blast radius — JWT failures cascade into api-gateway and mobile-bff. Severity: correct. So far so good.

The runbook-agent matched a “JWT Key Rotation Authentication Failure Remediation” runbook and produced a reasonable-sounding 8-step plan. The executor simulated running those steps: checking pod logs, verifying secret mounts, rotating the signing key, rolling the pods. All steps returned status: ok. The executor declared resolution_status: "escalated" — which in this case was correct, but for the wrong reason. It escalated because one simulated step failed, not because it understood the actual blast radius of a JWT rotation gone wrong.

Here’s the problem: a real JWT rotation failure requires coordinating across every service that holds the public key. The runbook covered auth-service. It did not cover the downstream services that cached the old key. The executor had no way to know this, because the static runbook lookup doesn’t model service dependencies. The pipeline produced a plausible-looking post-mortem with actionable steps — and none of those steps would have actually fixed the problem.

This is the failure mode that matters. Not malformed JSON. Not truncated responses. An agent that’s confidently correct at the individual step level but wrong at the system level, producing documentation that would mislead an on-call engineer.

This is not a prompt problem. It’s a systems problem. Static runbooks fail because systems aren’t static.

The fix is architecture: the runbook-agent needs a dependency graph, not a keyword-match against a flat library of procedures.

What that graph looks like matters. A static service map (hardcoded in config) would have caught the JWT case — auth-service depends on api-gateway depends on mobile-bff, so any auth rotation plan must include key propagation across all three. A runtime-derived graph (built from OTel traces) would be more accurate but adds infrastructure complexity. A config-derived graph sitting in the platform layer is probably the right starting point: owned by the platform team, consumed by the runbook-agent as context injected into the prompt alongside the runbook data.

The ownership question is the harder one. If the dependency graph lives in the agent’s prompt context, it becomes a prompt engineering problem that degrades as services change. If it lives in the platform and is injected at runtime, it becomes an infrastructure contract. That’s a different kind of system to build and operate.

And this failure is invisible in the Grafana dashboard — the conversation shows 4 agents, all OK, resolution escalated. You’d have to read the actual post-mortem to catch it.

The Setup: What Was Hard

This is a four-day-old public preview. I am not going to pretend it was smooth. Here is the honest accounting:

The Endpoint Discovery Problem

The Sigil SDK defaults to localhost:8080. The Grafana setup wizard shows localhost:8080. Neither of those is where you actually send data in Grafana Cloud.

After trying:

• <my-grafana-username>.grafana.net/api/sigil/v1/generations:export → 404
• <my-grafana-username>.grafana.net/api/v1/generations:export → 404
• localhost:4317 (Alloy’s OTLP port) → UNIMPLEMENTED: unknown service sigil.v1.GenerationIngestService
• localhost:8080 → connection refused

The answer was buried in Administration → Plugins → AI Observability → Connection tab.

The actual endpoint: sigil-prod-ca-east-0.grafana.net. Not documented. Found by clicking through the plugin Configuration tab.

The correct endpoint: https://sigil-prod-ca-east-0.grafana.net
The correct auth: Basic auth with Instance ID as username + Cloud Access Policy token as password
The correct token scope: You need a Cloud Access Policy token with sigil write scope, not a service account token and not the OTLP token

Total time spent on authentication alone: approximately 90 minutes across five different token types, three endpoints, and two auth schemes.

For teams already running Grafana Cloud with established service accounts and Cloud Access Policies: this is a 10-minute task. For someone setting up from scratch on a 4-day-old SDK: it is a different experience.

Running Alloy

Rather than using Docker Compose, we ran the Grafana Alloy container directly — simpler, more transparent, and easier to debug:

docker run -d \
  --name war-room-alloy \
  -p 4317:4317 -p 4318:4318 -p 8080:8080 -p 12345:12345 \
  -v $(pwd)/alloy/config.alloy:/etc/alloy/config.alloy:ro \
  --env-file .env \
  grafana/alloy:latest \
  run /etc/alloy/config.alloy --server.http.listen-addr=0.0.0.0:12345

Alloy running — GRPC receiver on 4317, HTTP receiver on 4318. The Prometheus remote write errors are Alloy’s self-metrics hitting the wrong regional endpoint; they don’t affect agent telemetry.

Claude Returns Markdown-Fenced JSON

Despite being told “output ONLY valid JSON — no markdown fences, no explanation,” Claude Haiku wraps responses in ` json ` blocks. Every time. Without fail.

def strip_fences(text: str) -> str:
    """The fix that should have been in the system prompt."""
    text = text.strip()
    text = re.sub(r'^```(?:json)?\s*', '', text)
    text = re.sub(r'\s*```$', '', text)
    return text.strip()

One line of regex. Took longer to figure out than it should have. Now it’s in every agent.

The HTTP vs gRPC Transport Confusion

The sigil-sdk defaults to gRPC. Grafana Alloy doesn’t implement the sigil.v1.GenerationIngestService gRPC protocol — it’s an OTel collector, not a Sigil receiver. The fix was to use the HTTP exporter directly:

from sigil_sdk.exporters.http import HTTPGenerationExporter

http_exporter = HTTPGenerationExporter(
    endpoint="https://sigil-prod-ca-east-0.grafana.net",
    headers={"Authorization": f"Basic {base64_creds}"},
)
config = ClientConfig(generation_exporter=http_exporter)

This isn’t documented. I found it by reading the source code of the SDK package.

What Experienced Teams Get vs. What New Users Face

If your team already has Grafana Cloud with established Cloud Access Policies, Alloy collectors running, and a platform team who knows where the endpoint configuration lives: this takes 30 minutes. Install sigil-sdk, add the start_generation context manager to your LLM calls, point at your existing sigil endpoint, done.

If you’re setting this up from scratch: budget 3-4 hours for the first run. The SDK is solid. The dashboard is excellent. The undocumented corners are real but navigable. The endpoint discovery and auth story need better documentation before this can be called “zero-config.”

The tagline on the landing page is “Actually useful AI O11y.” That’s accurate, once you’re past setup.

The Dashboard Experience: Seamless Once You’re In

Here’s the part that genuinely impressed me. Once the telemetry was flowing, the Grafana AI Observability UI is fast, well-designed, and immediately useful.

The Grafana AI Assistant analysed the 24 conversations and flagged that token metrics showed 0 tokens recorded — a real instrumentation gap. It also recommended enabling feedback collection for quality assessment.

The AI Assistant panel on the right is a nice touch — it analyses your agent telemetry and surfaces actionable observations. It spotted that my token counts weren’t flowing through (a known gap with how the Sigil HTTP exporter handles token reporting in this early version) and suggested next steps. A Grafana dashboard that uses AI to help you observe your AI agents. We’ve reached full recursion.

The tabs — Analytics, Agents, Conversations, Tools, Evaluation — cover every dimension of LLM observability:

• Conversations: the thread view, for debugging individual incidents
• Agents: the fleet view, for understanding workload distribution
• Analytics: the metrics view, for SLO and cost tracking — sparse in this setup for reasons explained below
• Evaluation: the quality view, for LLM-as-judge scoring (Phase 4 for this project)

Why the Analytics tab is sparse in this setup

If you open Analytics and see mostly empty panels — that’s expected here, and worth explaining. The Analytics tab is powered by Prometheus-compatible metrics. This project instruments exclusively via the Sigil SDK’s HTTP exporter, which ships conversation telemetry directly to Grafana’s Sigil ingestion endpoint. There is no Prometheus scrape pipeline in play — we didn’t use Prometheus at all. The instrumentation path is: agent code → Sigil SDK → HTTP exporter → sigil-prod-ca-east-0.grafana.net. That path doesn’t feed the Prometheus data model that Analytics queries against.

What this means practically: token usage aggregates, latency histograms, and cost-over-time charts won’t populate from this setup. The Conversations tab carries everything meaningful — per-agent timing, call counts, model attribution, and the full prompt/response content. For the goal of this experiment (debugging agent behaviour across incident pipelines), Conversations is the right surface anyway.

Wiring Sigil telemetry into a Prometheus-compatible sink to fully populate Analytics is a Phase 4 item, alongside LLM-as-judge evaluation. It would require either a recording rule layer or a custom exporter that bridges Sigil’s generation data into the Prometheus data model — doable, but out of scope for a four-day experiment on a brand-new SDK.

The Numbers

After two runs of 10 incidents each:

The 11/20 resolved rate is the most interesting number here, and it deserves more than a footnote.

The 3 escalations were all high-ambiguity P1 scenarios — JWT key rotation failure across multiple services, etcd fsync degradation affecting the entire control plane, a Prometheus TSDB corruption that took out the monitoring stack itself. In each case, the runbook-agent returned a valid plan, but the executor-agent hit simulated steps that either failed (exit code 1) or produced ambiguous output. The postmortem-agent then faithfully documented an unresolved incident. The pipeline completed correctly; the incident just wasn’t resolved. That’s the right behaviour.

The 5 partials are more interesting. In several cases, the executor completed all 8 steps and declared resolution_status: "partial" — meaning it judged that the simulated outcomes weren’t conclusive enough to call resolved. This reflects a real gap: the executor agent has no feedback loop. It can’t verify whether the simulated kubectl rollout status actually showed healthy pods. It’s reasoning about resolution based on output strings alone.

The 1 error was a truncated JSON response from Claude on a CDN DDoS + multi-region blast radius scenario — a complex incident with a long triage output that pushed against the executor’s context window. The strip_fences() fix handles markdown wrapping; I need a similar safety net for truncated JSON. It’s on the backlog.

What’s Next

Phase 4: LLM-as-Judge Evaluation

The most compelling remaining capability is automated quality evaluation. The architecture is ready:

1. Configure a Grafana evaluator targeting postmortem-agent outputs
2. Rubric: “Does this post-mortem contain a specific root cause hypothesis?”
3. Alert: fire if quality score drops below 0.7
4. Test: intentionally degrade the postmortem system prompt and watch the alert fire

This closes the loop: instrument → observe → evaluate → alert. Agent quality becomes a production SLO. That’s the end state.

Extending to Real Projects

The patterns from this experiment map directly to production agent systems:

• Any LangGraph pipeline: add start_generation context managers, pass conversation_id through state
• Existing Grafana Cloud setups: five minutes to wire up once you have the endpoint
• Cost tracking: the cost-cop pattern works for any multi-agent system

Summary

Five agents processed 20 incident types without crashing. 11 resolved, 5 partial, 3 escalated, 1 error. Total pipeline time averaged 37 seconds. Total cost: under $2 in Anthropic API credits. Grafana tracked every conversation thread and the cost-cop identified real model swap opportunities. The endpoint setup required navigating undocumented configuration on a 4-day-old SDK, which took longer than it should.

That’s what was built and what it did.

Conclusion

The system did not consistently resolve incidents. That’s not a failure of the observability layer — it’s an accurate picture of what current LLM agents can and can’t do with incident data. The observability layer is doing its job: making that gap visible, measurable, and improvable.

Treating LLM agent conversations as first-class telemetry — alongside traces, metrics, and logs — is the right architecture for the agentic era. The patterns here — conversation threading, per-agent cost tracking, quality evaluation as an SLO — will matter at every company running AI agents in production.

But here is where it breaks specifically.

At 1000 agents, the conversation model gets expensive. Every generation is a round-trip to the Sigil endpoint. At 1000 agents × 4 calls × 1 incident/minute, that’s 4000 HTTP calls per minute before you’ve made a single LLM call. The batching and queue mechanics of the SDK need stress-testing at that volume.

Cross-conversation dependencies are invisible. Sigil tracks individual conversations. It doesn’t model the relationship between a JWT rotation conversation at 11:06pm and the cascading auth failures that appear in a different conversation five minutes later. What would “distributed tracing for agent behaviour” require? The primitive isn’t a trace ID — it’s a causal link between conversations. Normal distributed tracing models a single request flowing through services. Agent behaviour tracing would need to model a multi-hop causal chain across separate incidents, separate conversations, and potentially separate agent pipelines — none of which share a request context. The closest existing hook is parent_generation_id on GenerationStart, but nothing in the current tooling connects those links across conversation boundaries.

Evaluation is the hardest part, and I skipped it. I listed LLM-as-judge evaluation as Phase 4 because it requires defining what a good triage or good post-mortem actually looks like — a rubric problem that’s harder than the instrumentation problem. The dashboard has an Evaluation tab. It’s empty. That’s not Grafana’s failure. It’s mine.

As an SRE building toward LLM infrastructure roles, this experiment confirms the intersection I care about: SRE rigour applied to LLM systems. The tooling is here. The mental models transfer. The failure modes are real.

But here’s the shift in framing that matters most:

We don’t have an observability problem for agents anymore. We have a correctness problem — and we finally have the tools to see it.

The next question isn’t “can we observe what our agents are doing?” It’s “now that we can see it clearly, what do we do about it?” That’s a harder problem. It’s also a more honest one.

That’s the next post.

References

• Grafana AI Observability Blog Post
• Grafana AI Observability Docs
• Grafana Sigil SDK (GitHub)
• LangGraph Docs
• Anthropic Claude API

This post is part of my ongoing series on LLM inference infrastructure. Previous posts: What Is LLM Inference, Really? and Running LLM Inference on Apple Silicon.

The short version: the pods ran, the stack was healthy, and the NIXL KV cache transfer dashboard showed “No Data” for every single metric. Performance was worse than the aggregated setup from the previous post on every dimension — E2E latency 6.5× higher, TTFT 3.7× higher, ITL 3.5× higher.

On a single GPU, P/D disaggregation doesn’t degrade — it collapses back into aggregated serving with worse performance. That’s the thesis of this post, and the Grafana dashboards prove it.

This is not a failure of the implementation. It’s the expected outcome when the hardware preconditions for NIXL are not met. This post maps that failure mode precisely — what the architecture requires, what actually ran, and how to tell the difference from the dashboards.

Hardware: Lambda Labs GH200 480GB, instance xxx.xxx.xx.186
Model: Qwen/Qwen3-0.6B
Stack: llm-d v0.4.0, pd-disaggregation guide, NIXL sidecar v0.7.1
Load testing: Locust with tenant simulation (~50,000 requests)

What P/D Disaggregation Is Supposed to Do

In aggregated serving — what experiments 1–4 tested — a single pod handles both prefill and decode for every request. Prefill spikes for long prompts steal GPU time from ongoing decode sequences. The two phases compete for the same resources.

P/D disaggregation separates them. Dedicated prefill pods handle the compute-intensive, bursty work of processing input tokens. Dedicated decode pods handle the memory-bandwidth-intensive, steady work of generating output tokens. Each pool scales independently. Prefill spikes never interrupt decode.

The mechanism that makes this work is KV cache transfer. When a prefill pod finishes processing a request’s input tokens, it has a set of KV cache blocks — the computed key-value tensors for every input token. Those blocks need to move to the decode pod before decode can begin. The decode pod cannot start generating without them.

llm-d uses NIXL (NVIDIA Inference Xfer Library) for this transfer. NIXL is designed to move KV cache blocks between pods over high-bandwidth interconnects — specifically InfiniBand or RoCE RDMA (NIXL transport backends). The transfer happens over port 5600 via a routing sidecar injected into each pod.

The values.yaml for this guide makes the architecture explicit:

routing:
  proxy:
    image: ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.1
    connector: nixlv2        # ← NIXL for KV transfer
    secure: false

decode:
  containers:
    - name: "vllm"
      args:
        - "--kv-transfer-config"
        - '{"kv_connector":"NixlConnector", "kv_role":"kv_both"}'
        - "--gpu-memory-utilization"
        - "0.45"             # ← halved to leave room for both pods
      ports:
        - containerPort: 8200  # vllm
        - containerPort: 5600  # nixl transfer

prefill:
  create: true              # ← this is the key difference from exp 1-4
  containers:
    - name: "vllm"
      args:
        - "--kv-transfer-config"
        - '{"kv_connector":"NixlConnector", "kv_role":"kv_both"}'
        - "--gpu-memory-utilization"
        - "0.45"
      ports:
        - containerPort: 8000  # vllm
        - containerPort: 5600  # nixl transfer

Two notable changes from the inference-scheduling config: prefill: create: true (we now have a dedicated prefill pod), and gpu-memory-utilization: 0.45 per pod instead of 0.90. Each pod gets 45% of GPU memory — the assumption being that each runs on a dedicated GPU.

The Setup

Day 2 was faster. The gotchas from Post 3 were already solved — real Helm binary, kubeconfig chown’d, environment variables in .bashrc, Gateway API CRDs pre-installed. The stack came up in under an hour.

kubectl get pods -n llm-d

NAME                                                READY   STATUS    RESTARTS      AGE
gaie-pd-epp-6bd657c9f4-q882g                        1/1     Running   0             45m
infra-pd-inference-gateway-istio-666c6955bd-g2wnl   1/1     Running   0             45m
ms-pd-llm-d-modelservice-decode-696d86b5c5-tk8d9    1/1     Running   1 (25m ago)   26m
ms-pd-llm-d-modelservice-prefill-64d5f7bb7f-rxvwb   1/1     Running   0             26m

Four pods. EPP running, gateway running, both prefill and decode pods running at 1/1. The decode pod had one restart — normal during model download. HTTPRoute confirmed valid, gateway responding, model serving requests.

The system looked healthy at every layer except the one that actually mattered.

The observability stack confirmed the same:

Then the Locust test ran. And the numbers came back wrong.

The Locust Results

Same script as experiments 1–4. Same traffic shape: 4:1 tenant session to cold request ratio, same three tenant profiles with long system prompts.

Hardware:  GH200 480GB (prefill pod + decode pod, P/D disaggregation guide)
Model:     Qwen/Qwen3-0.6B
Duration:  sustained run, ~26 req/s

Task              Requests    Failures   avg      min      max
──────────────────────────────────────────────────────────────
cold_request      ~10,020     0 (0%)     1,047ms  969ms    1,094ms
tenant_session    ~39,976     0 (0%)     1,654ms  1,521ms  1,721ms
──────────────────────────────────────────────────────────────
Aggregated        ~49,996     0 (0%)     1,532ms  969ms    1,721ms

Note: terminal output only — no CSV saved for this run. Avg, min, max reported; p95/p99 not available.

Zero failures — the system was stable. But the latency numbers were dramatically worse than experiments 1–4:

Metric           Exp 1-4 (aggregated)   Exp 5-6 (P/D)      Change
────────────────────────────────────────────────────────────────────
cold_request avg   207ms                 1,047ms            5.1× worse
tenant_session avg 273ms                 1,654ms            6.1× worse

This was not what P/D disaggregation is supposed to deliver. The next step was Grafana.

What Grafana Showed

The Performance Dashboard

Three things immediately stand out comparing this to the experiments 1–4 dashboard:

TTFT went from 15ms to 53ms. More than 3× worse. In a properly functioning P/D disaggregated system, TTFT should improve because the decode pod never waits for prefill — prefill happens on a dedicated pod. Here it got worse.

ITL went from 5ms to 17.5ms. Inter-Token Latency — the gap between consecutive generated tokens during decode — is 3.5× higher. This is the clearest signal of a problem. ITL is purely a function of decode throughput. It shouldn’t change between setups unless the decode pod’s resources are constrained.

KV Cache Hit Rate is still 81.3%. The EPP routing is still working correctly. Tenant sessions are being routed to the pod holding their cached KV blocks. The routing layer is fine.

The P/D Coordinator Dashboard — The Smoking Gun

Diagnostic checklist: is your P/D setup actually disaggregated?

If your deployment shows all of the following, it is not disaggregated — it is aggregated serving with extra overhead:

All five conditions were present in these experiments. Check this dashboard before assuming P/D disaggregation is functioning.

The top row of the P/D Coordinator dashboard is designed to show the latency breakdown between prefill and decode. Three separate metrics: Avg vLLM TTFT (Prefill), Avg True TTFT (Prefill + Decode), and Avg vLLM TTFT (Decode).

All three show 54.9ms. Identical.

In a working P/D disaggregation system, these would differ. The True TTFT would be higher than either individual component because it includes the NIXL KV transfer time between pods. If all three are identical, the KV transfer is contributing zero measurable time — which means the transfer is not happening.

In this system, “No Data” is not a monitoring gap — it is the primary signal that KV transfer is not happening. Most engineers look past empty dashboard panels. Here, the empty panels are the answer.

No KV transfers happened. Zero. Not failed — just absent. NIXL didn’t move any data between the prefill and decode pods during the entire Locust run.

The EPP Pool — Only One Ready Pod

EPP Pool Health shows Ready Pods = 1. Despite two pods running and both showing 1/1 in kubectl get pods, the EPP only recognises one pod as ready for traffic. The prefill pod — despite being healthy — is not in the EPP’s routing pool.

The Diagnostic Drill-Down — Both Pods Present But Unequal

The P/D disaggregation section of the Diagnostic Drill-Down makes the situation clear. Prefill Worker Utilization is near zero throughout the test. The prefill pod is running, consuming memory, and doing nothing. The decode pod is handling all prefill and decode — effectively running as an aggregated server with 45% GPU memory instead of 90%.

Why NIXL Didn’t Work — The Hardware Constraint

NIXL is designed for KV cache transfer between pods running on separate physical GPUs connected by RDMA — InfiniBand or RoCE. This is not a configuration choice. It’s a transport layer requirement. NIXL uses ibv_reg_mr (InfiniBand verbs memory registration) to pin GPU memory for zero-copy transfer. Without an RDMA-capable network between the pods, the transport layer cannot initialise.

On a single GH200 with time-sliced GPU allocation:

What the architecture expects:
  Prefill Pod → [GPU 0] → RDMA/NVLink → [GPU 1] ← Decode Pod

What actually ran:
  Prefill Pod ↘                    ↗ Decode Pod
               [Single GH200 GPU]
               (time-sliced, shared)

Both pods share the same physical GPU. There is no RDMA path between them — they’re in the same physical memory space. NIXL cannot establish its transport. The sidecar starts, port 5600 opens, and then nothing moves.

This requirement is easy to overlook because the system still appears healthy at the pod and routing layers. Pods are Running. HTTPRoute is valid. The EPP is routing. Prometheus is scraping metrics. Everything except the KV transfer layer looks operational — and the KV transfer layer shows no errors, just silence.

This isn’t a bug. It’s the architecture working as documented. The llm-d pd-disaggregation guide states it requires multiple GPUs — the inference-scheduling guide’s README explicitly notes its default benchmark configuration targets 16 GPUs across 8 model servers. The GH200’s NVLink-C2C is designed for CPU-GPU communication within the package, not for inter-pod RDMA at the scale NIXL needs.

What NIXL actually needs is one of:

• Multiple physical GPUs on the same node connected by NVLink (H100 SXM with NVSwitch)
• Multiple nodes connected by InfiniBand or RoCE

A single GH200 satisfies neither condition.

What the Numbers Actually Mean

The performance degradation in experiments 5–6 has three causes:

1. Halved GPU memory per pod. gpu-memory-utilization: 0.45 instead of 0.90 means each pod gets roughly half the KV cache budget. The decode pod — which is doing all the work — has only 45% of GPU memory available for the KV cache. This directly limits batch size and increases the probability of KV evictions, which degrades both throughput and ITL.

2. GPU time-slicing competition. Both pods share one physical GPU. Every GPU operation from the prefill pod steals cycles from the decode pod and vice versa. Even though the prefill pod is idle, it’s not truly idle — it’s holding GPU memory allocations and context. This is why ITL went from 5ms to 17.5ms: the decode pod gets less consistent GPU access.

3. NIXL coordination overhead. The routing sidecar still runs, still listens on port 5600, still attempts to coordinate transfers. Even with no transfers completing, the coordination layer adds processing overhead per request.

The result is a system that has the complexity cost of disaggregation with none of its benefits.

The Complete Comparison

Setup         Exp 1-4 (aggregated)   Exp 5-6 (P/D attempted)   Notes
──────────────────────────────────────────────────────────────────────────────
Pods          1 decode               1 prefill + 1 decode       Both pods 1/1 Running
GPU mem/pod   90%                    45%                        Halved for time-slicing
NIXL active   N/A                    No (RDMA not available)    No data in dashboard
KV Hit Rate   80.6%                  81.3%                      EPP routing: unchanged
TTFT avg      15ms                   54.9ms                     3.7× worse
ITL p50       5ms                    17.5ms                     3.5× worse
E2E p50       260ms                  1,690ms                    6.5× worse
Failures      0                      0                          Both stable
Total reqs    26,826                 ~49,996                    More requests, longer run

What P/D Disaggregation Actually Needs

P/D disaggregation is architecturally sound and measurably beneficial — when run on appropriate hardware. The llm-d team’s published benchmark results on 16 H100 GPUs (8 vLLM pods, TP=2) show up to 99% lower TTFT and 109% higher throughput compared to a baseline Kubernetes service under shared-prefix workloads. A more detailed breakdown of the v0.4 scheduler benchmark on the same topology is in the v0.4 release post. Neither is something I measured — these are llm-d’s own published numbers on appropriate hardware.

But “appropriate hardware” is not optional. It means:

Minimum viable configuration:

• Two physical GPUs — one dedicated to prefill, one to decode
• High-bandwidth interconnect between them (NVLink SXM, InfiniBand, RoCE)
• RDMA drivers available in the container runtime

Production configuration:

• Dedicated prefill pod pool (multiple GPUs, auto-scaled to compute demand)
• Dedicated decode pool (multiple GPUs, auto-scaled to memory-bandwidth demand)
• RDMA fabric between pools
• The GH200’s NVLink-C2C is excellent for CPU-GPU communication but not for inter-node RDMA

The fundamental insight from these experiments: P/D disaggregation is a distributed systems problem, not a Kubernetes scheduling problem. You can schedule the pods correctly, set up the EPP, configure NIXL, and deploy the routing sidecar — and still have an aggregated server if the hardware doesn’t support RDMA-based KV transfer. The architecture is only as good as the interconnect it runs on.

What This Means for the Series

This series started with a theory: prefill and decode compete for the same resources in aggregated serving. Post 2 measured the consequences on an M4 Mac Mini. Post 3 deployed the infrastructure on a GH200. Post 4 showed EPP prefix cache routing working correctly on a single aggregated pod.

This post was supposed to show the next step — split prefill and decode, measure the improvement. Instead it showed why that split requires dedicated hardware.

This result is more actionable than a successful run would have been. Anyone planning to deploy P/D disaggregation needs to know:

• Single-GPU time-slicing will not work — NIXL needs RDMA
• Halving GPU memory per pod on shared hardware degrades performance significantly
• The “2 pods running” state is not sufficient validation — check the P/D Coordinator dashboard for actual KV transfer metrics before declaring success
• The correct hardware starting point is at minimum 2 physical GPUs with NVLink or InfiniBand

The next experiment worth running — when the GPU budget allows — is a 2-node setup with proper RDMA between prefill and decode pods. That’s what the architecture is designed for, and that’s where the TTFT numbers would actually improve rather than degrade.

What the EPP Got Right

One thing worth noting: despite everything else going wrong, the EPP routing continued to achieve 81.2% prefix cache hit rate. The routing intelligence — directing tenant sessions to pods with warm KV cache — worked correctly regardless of whether NIXL was functioning.

This is the right separation of concerns. The EPP’s job is routing. NIXL’s job is KV transfer. They’re independent layers. When NIXL fails, routing doesn’t fail with it. The system degrades gracefully — you get worse performance, not a broken system.

That’s principled architecture. It’s also why the llm-d design is worth understanding even if you’re not deploying at the scale that justifies dedicated prefill and decode pools. The routing layer’s correctness is independent of the transfer layer’s availability.

The Most Valuable Thing This Experiment Taught Me

I want to be direct about this, because it’s the kind of learning that only comes from actually running the thing rather than reading the architecture docs.

Before these experiments, I understood P/D disaggregation conceptually — prefill is compute-bound, decode is memory-bandwidth-bound, separate them onto dedicated hardware, scale independently. The theory was clear.

What I didn’t fully appreciate was how non-negotiable the hardware requirement is. It’s not a performance optimisation. It’s a correctness requirement. Without RDMA between pods, NIXL cannot transfer KV cache blocks. Without KV transfer, the decode pod cannot receive the prefill pod’s output. Without that handoff, you don’t have disaggregation — you have two pods running on a single shared GPU, each with half the memory budget, competing for the same compute cycles. Which is strictly worse than one pod using all of it.

The GH200 taught me this in a way that no benchmark paper could. I stood up the full architecture — correct values.yaml, correct pod configuration, correct NIXL sidecar, correct port mappings — and watched Grafana show “No Data” on every single KV transfer metric. The pods were healthy. The routing was working. The architecture was incomplete because the hardware was.

The lesson: when an architecture requires RDMA, it’s not a performance optimisation — it’s a correctness condition. The system will appear to run without it. The metrics will tell you otherwise.

The next experiment is a minimal 2-GPU setup with:

• Dedicated prefill and decode pods on separate physical GPUs
• NVLink or RDMA-backed interconnect between them
• Verified non-zero KV Transfer metrics in the P/D Coordinator dashboard
• Measurable separation between prefill TTFT and decode TTFT
• ITL improvement relative to the aggregated baseline from experiments 1–4

Success is defined: KV transfers showing actual data, prefill worker utilization above 0%, and E2E latency that improves on the 260ms p50 from experiments 1–4 — not degrades from it.

That comparison — aggregated (260ms p50) → P/D wrong hardware (1,690ms p50) → P/D proper RDMA (?ms p50) — will be the complete picture of what disaggregation delivers and what it costs.

Experiments run on Lambda Labs GH200 480GB, llm-d v0.4.0, pd-disaggregation guide, Qwen3-0.6B, NIXL v0.7.1. Platform engineer with 11+ years in distributed systems going deep on LLM serving infrastructure.

GitHub · LinkedIn

This post is the payoff — and more than that, it’s an attempt to say something useful beyond “look at these numbers.” The numbers are good. But the more interesting question is what they reveal about inference architecture that applies regardless of which GPU you’re running on or which serving framework you’ve chosen.

Hardware: Lambda Labs GH200 480GB, ARM64
Model: Qwen/Qwen3-0.6B
Stack: llm-d v0.4.0, vllm/vllm-openai:latest, Istio gateway, kube-prometheus-stack
Load testing: Locust with tenant simulation
Observability: Prometheus + Grafana (llm-d Performance Dashboard, llm-d vLLM Overview)

The Hardware Deserves More Than a Footnote

Most inference benchmarks treat hardware as a footnote — “tested on an A100” — without explaining why the hardware choice matters architecturally. The GH200 is worth understanding properly because it represents a design direction that the industry is converging on, and it changes some assumptions about what’s possible in inference.

One quick note before the specs: the GH200 is technically a unified memory architecture — just like the M4 Mac Mini from the previous post. CPU and GPU share the same address space without copying data between them. The Mac Mini has 16GB at roughly 200 GB/s. The GH200 has 576GB at up to 4,000 GB/s.

The GH200 is technically unified memory — just like the M4 Mac Mini. One costs $799. The other costs $2.29 per hour and will make you feel considerably better about your Mac Mini purchase. Same architectural principle, considerably different ambitions — and as the numbers in this post will show, considerably different outcomes.

Grace Hopper — One Package, Two Chips, One Memory Bus

The GH200 is not a GPU. It’s a Grace Hopper Superchip — NVIDIA’s Grace CPU (72 ARM Neoverse V2 cores) and an H100 Hopper GPU die, connected on the same package via NVLink-C2C (chip-to-chip).

The critical number is the NVLink-C2C bandwidth: 900 GB/s bidirectional. For context:

Interconnect              Bandwidth
──────────────────────────────────────────────
PCIe 5.0 x16 (discrete)   128 GB/s
NVLink-C2C (GH200)         900 GB/s    ← 7× faster than PCIe Gen5
HBM3e on-chip (GH200)    4,000 GB/s
LPDDR5X (Grace CPU)        512 GB/s

Source: NVIDIA GH200 Grace Hopper Superchip product page and GH200 architecture whitepaper. The 7× PCIe Gen5 figure is NVIDIA’s own stated specification — “NVLink-C2C delivers up to 900 GB/s total bandwidth. This is 7x higher bandwidth than x16 PCIe Gen5 lanes.”

On a conventional discrete GPU system — an A100 or H100 in a PCIe slot — the CPU and GPU have separate memory pools connected by a 128 GB/s bus. Moving data between them is expensive. The GPU can’t efficiently use CPU DRAM for KV cache overflow because the PCIe bandwidth is 31× slower than the GPU’s on-chip HBM bandwidth (4,000 ÷ 128 = 31.25×). Any spill to CPU memory becomes a bottleneck.

On the GH200, the Grace CPU’s 480GB of LPDDR5X memory is accessible to the Hopper GPU at 900 GB/s over NVLink-C2C. That’s not as fast as on-chip HBM3e, but it’s fast enough to be genuinely useful. The result is a unified addressable memory space — the GPU sees up to 576GB total (96GB HBM3e + 480GB LPDDR5X) — at bandwidths that make overflow to CPU DRAM a viable architectural choice rather than a last resort.

Why This Matters for LLM Inference

At scale, inference systems are typically memory-bandwidth bound, not compute bound. This isn’t universal — small models at low concurrency can be compute-bound during prefill, and some workloads shift the bottleneck elsewhere. But for production multi-tenant serving, the binding constraint is almost always memory: how fast the hardware can load model weights and KV cache tensors for each forward pass. Every decode step reads the full KV cache. On a 7B model with a 4K context window and 32 concurrent requests, that’s dozens of gigabytes moving through the memory subsystem per second — and the rate of that movement, not the number of CUDA cores, determines your throughput ceiling.

The GH200’s NVLink-C2C doesn’t solve this — HBM3e is still the primary memory bandwidth for active inference. But it changes the economics of KV cache management. Tiered KV storage (hot blocks in HBM, warm blocks in LPDDR5X, cold blocks evicted to NVMe) becomes viable in a way it isn’t on PCIe-connected systems. llm-d’s architecture diagram from the previous post already shows tiered KV cache offloading as a first-class feature — the GH200 is the hardware that makes that design practical.

For these experiments, we didn’t exercise KV cache tiering. The model is small and the cache pressure is low. But the architecture is there, and it’s why the GH200 was the right choice for running the full llm-d stack rather than a conventional discrete GPU setup.

One Mental Model Before the Numbers

Before looking at a single metric, here is the frame I’d encourage you to carry into any inference system analysis:

LLM inference is a memory scheduling problem that looks like a compute problem.

Every team I’ve seen optimise inference focuses first on compute — GPU utilization, batch size, model quantization. These matter. But the real leverage is in memory: how much of the KV cache is warm, how often prefill recomputation is avoided, how well the scheduler keeps the right data in the right tier. The system that wins at scale is the system that does the least unnecessary work — and unnecessary prefill recomputation is the biggest source of waste in multi-tenant inference.

This is what EPP prefix cache routing is solving. Not “make the GPU faster.” Make the system do less work.

What These Experiments Actually Test

Experiments 1–4 use the inference-scheduling guide from the llm-d repo. This deploys a single decode pod that handles both prefill and decode — aggregated serving. The values.yaml is unambiguous:

prefill:
  create: false   # one pod does everything

The architecture is:

Every request →  Istio Gateway
                      │
                      ▼
                 EPP (Endpoint Picker Plugin)
                 ┌────────────────────────────────┐
                 │  prefix-cache-scorer           │
                 │  queue-depth-scorer            │
                 │  kv-utilization-scorer         │
                 └────────────────────────────────┘
                      │
                      ▼
               Single vLLM decode pod
               (prefill + decode, one process)
               KV cache lives here

With one pod, there is no cross-pod routing decision to make. The EPP’s job here is narrower: route requests with matching prefix hashes back to this pod consistently, so its KV cache stays warm. Round-robin with one pod is also consistent routing — but a naive load balancer doesn’t know about prefix hashes, so it can’t make cache-aware decisions when you add a second pod later.

What this experiment answers: Does EPP prefix-cache-aware routing actually maintain high cache utilization under realistic multi-tenant load? Does the system hold TTFT stable as concurrency grows?

What it doesn’t answer yet: What happens when you separate prefill and decode onto dedicated pods. That’s the next post.

This single-pod result is the baseline. Everything in the P/D disaggregation post gets measured against what we establish here.

The Load Test — Simulating Multi-Tenant Traffic

The Locust script simulates two traffic types in a 4:1 ratio:

# llmd-locust.py
MODEL = "Qwen/Qwen3-0.6B"

# Three tenant personas — each with a distinct long system prompt
TENANTS = [
    "You are a financial analyst assistant specializing in market data. " * 5,
    "You are a DevOps engineer assistant specializing in Kubernetes and CI/CD. " * 5,
    "You are a data scientist assistant specializing in ML pipelines. " * 5,
]

class LLMDUser(HttpUser):
    wait_time = between(0.5, 2)

    def on_start(self):
        self.tenant_prompt = random.choice(TENANTS)  # sticky for session

    @task(4)
    def tenant_request(self):
        # Repeat system prompt → cache hit opportunity
        self.client.post("/v1/chat/completions", json={
            "model": MODEL,
            "messages": [
                {"role": "system", "content": self.tenant_prompt},
                {"role": "user", "content": random.choice([
                    "Summarize the key points.",
                    "What should I focus on?",
                    "Give me 3 recommendations.",
                ])}
            ],
            "max_tokens": 80,
        }, name="tenant_session")

    @task(1)
    def cold_request(self):
        # No system prompt → cold cache, no prefix benefit
        self.client.post("/v1/chat/completions", json={
            "model": MODEL,
            "messages": [{"role": "user",
                          "content": f"Question {random.randint(1,1000)}"}],
            "max_tokens": 50,
        }, name="cold_request")

The design is deliberate. Each simulated user picks a tenant persona at startup and sticks with it — the same ~200-token system prompt on every request. This is representative of real multi-tenant SaaS inference: each customer has a system prompt that defines their product’s persona, and it’s identical on every call. The cold_request tasks (1 in 5) have no system prompt and get no cache benefit — they’re the baseline comparison within the same run.

Requests travel the full path: Mac → SSH tunnel → kubectl port-forward → Istio gateway → EPP → vLLM pod. No shortcuts.

The Results

Locust — 26,826 Requests, Zero Failures

Hardware:  GH200 480GB (single decode pod, aggregated serving)
Model:     Qwen/Qwen3-0.6B
Rate:      32 req/s sustained

Task              Requests    Failures   p50    p95    p99    avg
───────────────────────────────────────────────────────────────────
cold_request      5,450       0 (0%)     200ms  270ms  280ms  207ms
tenant_session    21,376      0 (0%)     260ms  330ms  350ms  273ms
───────────────────────────────────────────────────────────────────
Aggregated        26,826      0 (0%)     260ms  330ms  350ms  260ms

Zero failures across 26,826 requests. p99 at 350ms. The system didn’t flinch.

The comparison that matters — same Locust script structure, same traffic intent, from the Mac Mini post:

Mac Mini vLLM (5 users, 3:1 short/long):
  short prompt p50:   1,900ms
  long prompt  p50:  31,000ms
  TTFT P99:          ~7,000ms under load

llm-d on GH200 (tenant simulation, 4:1 tenant/cold):
  tenant_session p50:   260ms    ← 7.3× faster at median
  cold_request   p50:   200ms
  TTFT P99:              34ms    ← ~200× better tail latency

The gap is real, but it deserves honest attribution. Part of it is raw hardware — a GH200 is not an M4 Mac Mini. Part of it is the serving stack — llm-d with EPP routing vs vanilla vLLM. And part of it is the traffic shape — these aren’t identical experiments. What the numbers establish is a clear direction: hardware matters, but the serving architecture amplifies or squanders what the hardware can do.

What Grafana Showed

The Performance Dashboard — Read This One First

Three numbers worth pausing on:

TTFT p50 at 15ms, flat under load. The Mac Mini showed TTFT climbing past 2 seconds at p50 and 7 seconds at p99 under comparable concurrency. Here it stays at 15ms throughout. This is not just hardware — it’s what happens when 80% of your requests skip prefill entirely because their KV blocks are already warm. The system is doing less work per request on average, which is why latency stays stable as concurrency grows.

A common mistake in inference optimisation is treating TTFT stability as a tuning parameter — something you achieve by adjusting batch sizes, memory utilization settings, or scheduler parameters. It isn’t. TTFT stability under load is an architectural property. It follows from having enough cache hit rate to keep the average prefill cost low. Once cache hit rate drops below ~50%, no amount of tuning recovers the tail latency. The right intervention is upstream: better routing, larger KV cache budgets, or separation of prefill and decode workloads.

KV Cache Hit Rate at 80.6%. This is the single most actionable metric in multi-tenant inference. It tells you how much work the system is not doing. At 80.6%, roughly 4 in 5 tenant requests reuse cached KV blocks and skip prefill recomputation. The inverse is the expensive number: 19.4% of requests are cold — those are full prefill operations. In a system with 100 GPU-hours of work per day, improving cache hit rate from 80% to 90% saves 10 GPU-hours. That’s not a performance metric. That’s a cost metric.

ITL at 5ms. Inter-Token Latency is the gap between consecutive tokens during decode — a direct measure of decode throughput. At 5ms per token, that’s 200 tokens per second per request. More importantly it’s flat — it doesn’t increase as the test runs, which confirms there’s no memory pressure or scheduler thrashing affecting the decode phase. When ITL climbs under load, it’s usually a sign that the KV cache is filling and evictions are occurring. Here it’s not.

Prefix Cache Hit Rate — The Number Behind the Number

The 81.1% gauge is the aggregate. The Per-Instance time series at 80% is the more meaningful signal — it shows the cache was warm within the first few minutes and held that level for the entire test duration. This is what stable prefix cache routing looks like: not a spike that decays, but a plateau that holds.

What this means at scale: At three tenant profiles with ~200-token system prompts, 81% cache hit rate is achievable on a single pod with a small model. As you scale to 50 tenants, 500, or 5000, the picture changes. The working set of system prompts grows beyond what a single pod’s KV cache can hold. Cache hit rate degrades. TTFT rises. This isn’t a failure of EPP routing — it’s a KV cache capacity problem, and the architectural response is either larger KV budgets per pod, more pods with affinity-based routing, or tiered KV offloading. The GH200’s NVLink-C2C is exactly the hardware that makes that third option viable. This experiment doesn’t exercise it. The architecture supports it.

E2E Latency and Scheduler State

The Scheduler State panel deserves attention. Num Running is the active batch size — how many sequences share a GPU forward pass. Num Waiting is the queue depth. Both staying low and flat means continuous batching is absorbing load without queuing. Compare this to the Mac Mini Locust test where Num Waiting spiked to 5 and TTFT degraded proportionally.

Cache Utilization at 0.15% is a function of model size. Qwen3-0.6B at this concurrency level barely touches the KV pool. The same test with Llama-3-8B would show a fundamentally different curve — and that’s where the GH200’s memory architecture starts mattering in ways that go beyond raw numbers.

Prefill vs Decode — The Two-Phase Separation Made Visible

This panel is the most architecturally revealing one in the dashboard. The yellow prefill line and the green decode line on the same chart, from the same pod, tell you everything about why P/D disaggregation exists.

Prefill is compute-bound and bursty. Its cost scales with prompt length. It spikes when a cold request arrives. Decode is memory-bandwidth-bound and steady. Its cost scales with the number of tokens being generated. On the same pod, every prefill spike steals GPU time from active decode sequences — those sequences stall mid-generation while the prefill runs.

In a disaggregated setup, the yellow line comes from a prefill pod pool and the green line from a decode pod pool. Prefill spikes are isolated. Decode runs uninterrupted. TTFT for long prompts no longer delays short prompts waiting in the decode queue.

Here those lines share a pod. The system works well at this scale and concurrency — the numbers prove it. But the architectural tension is visible in the chart. That’s what the next post is about.

TTFT P99 — The Warmup Signature

TTFT P99 dropping from 39ms to 32ms and then holding at 34ms is the cache warmup signature in the tail. The first requests from each tenant session are cold — full prefill. As sessions accumulate, KV blocks warm, and the P99 reflects that. The key signal is the plateau: P99 stops dropping once the cache is warm and doesn’t creep upward under sustained load. This is a system that has found equilibrium.

Model Throughput and Queue State

Model Throughput at 4,000–5,000 tokens per second is what a GH200 looks like under moderate load with a small model. The brief Queue Utilization spikes that drain immediately are continuous batching doing its job — burst arrivals get absorbed into the current decode step rather than queuing. This is the architectural behaviour that separates vLLM from naive sequential servers.

The Complete Benchmark Reference

Hardware:  Lambda Labs GH200 480GB (Grace Hopper)
Model:     Qwen/Qwen3-0.6B
Stack:     llm-d v0.4.0, single decode pod (prefill: create: false)
           EPP: prefix-cache-scorer + queue-scorer + kv-utilization-scorer

─── Locust ───────────────────────────────────────────────────────────
  Total requests:   26,826     Failures:    0 (0%)
  Sustained rate:   32 req/s

  Task             p50    p95    p99    avg    req/s
  tenant_session:  260ms  330ms  350ms  273ms  25.5
  cold_request:    200ms  270ms  280ms  207ms   6.5
  Aggregated:      260ms  330ms  350ms  260ms  32.0

─── Grafana — Performance Dashboard ──────────────────────────────────
  TTFT p50:                 15ms
  TTFT p95:                 19ms
  TTFT P99 (settled):       ~34ms
  ITL p50:                  5ms   (≈ 200 tok/s)
  ITL p95:                  9.5ms
  KV Cache Hit Rate:        80.6% (gauge) / 81.1% (drill-down)
  Request Throughput:       28.8 req/s
  Model Throughput:         4,000–5,000 tps peak
  Cache Utilization:        0.15%

─── vs Mac Mini (Post 2, same script structure) ──────────────────────
  tenant p50:  Mac Mini 1,900ms  →  GH200 260ms   (7.3× faster)
  TTFT P99:    Mac Mini ~7,000ms →  GH200 34ms    (~200× better)

What This Proves — and What It Doesn’t

What it proves:

EPP prefix cache routing achieves 81.1% cache hit rate under realistic multi-tenant load. That’s not a configuration artifact — it’s the result of the EPP scoring prefix hashes and routing tenant sessions to the pod holding their warm KV blocks. The system holds TTFT at 15ms p50 and 34ms p99 under 32 req/s sustained, with zero failures.

More broadly: in multi-tenant inference systems with repeated system prompts, cache hit rate is typically the highest-leverage optimisation variable. GPU utilization, batch size, and quantization all matter — but they reduce the cost of work the system is already doing. Cache hit rate determines how much work gets skipped entirely. At 81%, the system is doing roughly 5× less prefill work than a round-robin deployment that scatters the same tenant’s requests across pods with cold caches. That ratio is hard to match through hardware improvements alone.

What it doesn’t prove:

This is a single pod, a small model, and three tenant profiles. The working set fits comfortably in the KV cache — hence 0.15% utilization. Real multi-tenant deployments have hundreds or thousands of tenant profiles. As the working set grows, cache hit rate degrades. The EPP routing remains correct, but the pod’s KV cache can’t hold every tenant’s prefix simultaneously. The responses to that problem — larger KV allocations, more pods with affinity routing, tiered KV offloading to the GH200’s LPDDR5X — are architectural decisions that require knowing the working set size and access pattern distribution. This experiment establishes that the routing mechanism works. Capacity planning is a separate problem.

What Most Teams Get Wrong

Most inference optimisation work focuses on the supply side — faster hardware, more efficient models, better batching. This is necessary but insufficient. The demand side — how requests are shaped and routed before they reach the GPU — is where the real leverage lives at scale.

The dominant optimisation lever depends on where your system actually sits. Before reaching for more hardware, diagnose the bottleneck:

Most teams misdiagnose their bottleneck and optimise the wrong layer. A team with a 30% cache hit rate buying faster GPUs is solving the wrong problem — they’ll run twice as fast through twice as much unnecessary prefill work. The 81% cache hit rate in these experiments means the system is doing roughly 5× less prefill than a naive round-robin deployment on identical hardware. No GPU upgrade achieves that ratio. Routing does.

The practical entry point: instrument your cache hit rate first. If it’s below 50%, the fix is upstream — consistent system prompts, session-affinity routing, and a scheduler that knows about prefix hashes. Hardware comes after you’ve exhausted the routing lever.

A non-obvious failure mode: high aggregate cache hit rate masking per-tenant unfairness.

A subtle failure appears when cache hit rate is high but unevenly distributed across tenants. If a small number of tenants dominate traffic, their KV blocks stay hot while long-tail tenants constantly miss the cache. The system reports a healthy aggregate hit rate — 80%, 81% — but tail latency degrades because cold tenants always pay full prefill cost.

This leads to a misleading conclusion: “the cache is working.” In reality, the system is biased toward high-frequency tenants. The aggregate metric looks healthy precisely because the popular tenants pull it up — while every new or low-frequency tenant experiences the system as if caching doesn’t exist.

In these experiments, three tenant profiles at similar frequencies produced clean aggregate numbers. Production systems with hundreds of tenants and a power-law access distribution will not. Fixing uneven cache efficiency requires either per-tenant cache accounting to surface the distribution, or explicit isolation of long-tail traffic to prevent it from competing with high-frequency prefixes for the same KV blocks. Without this, cache efficiency improves averages while silently degrading fairness — which is the kind of problem that shows up in p99 SLA breaches, not in dashboard summaries.

Connecting the Dots Across This Series

This series started on an M4 Mac Mini with a 400MB model and a Python script measuring TTFT. It’s worth pausing to trace what the numbers across all three posts are actually saying — because they’re saying the same thing at different scales.

Post 1 established the theory: decode is memory-bandwidth bound. Prefill competes with decode for the same resources. KV cache management is the central architectural problem in inference. These aren’t vLLM-specific observations — they’re properties of the transformer architecture itself. They hold on any hardware, any framework.

Post 2 ran two experiments on identical hardware. Ollama vs vLLM on the same M4 chip, same unified memory, same model family. The result — 14,062ms vs 6,543ms p50, 2.15× faster — had nothing to do with the hardware. Ollama processes requests sequentially. vLLM uses continuous batching. Same silicon, 2× throughput difference from a software architecture decision. That result matters because it isolates the variable: serving architecture, not hardware.

Then under load, even vLLM on the Mac Mini hit the ceiling. Long prompts drove TTFT to 31 seconds at p50. The Mac Mini’s 16GB unified memory pool — shared between model weights, KV cache, and OS — ran out of headroom. The principle from Post 1 materialised as a real number: when the KV cache competes for the same memory as the model weights, concurrency suffers.

This post added two more variables: real GPU hardware and intelligent routing. The GH200 with NVLink-C2C at 900 GB/s changes the memory economics — the GPU can address 480GB at viable bandwidth, not just 16GB. EPP prefix cache routing adds the third variable: the system avoids prefill work entirely for 81% of requests by keeping KV blocks warm.

The result is 260ms p50 and 34ms P99 at 32 req/s. But attributing that entirely to the GH200 would be wrong — and that’s the point. The Ollama experiment on the Mac Mini already proved that hardware alone doesn’t determine the outcome. The GH200 sets a much higher ceiling. EPP routing determines how close you get to it.

The through-line: hardware sets the ceiling. Serving architecture determines how close you get to it. This has been true at every scale in this series — a $0 Mac Mini, a $2.29/hr GH200, and everything in between. The engineers who understand this spend their optimisation budget on routing and request shaping first, and on hardware second. The engineers who don’t buy more GPUs and wonder why the numbers don’t improve proportionally.

What Comes Next

The prefill vs decode time panel in this post showed two lines on the same chart — prefill varying with cache state, decode flat throughout. On this single pod they share a GPU. A prefill spike for a long-prompt request steals compute from ongoing decode sequences. The system handles it at this concurrency. It wouldn’t at 10×.

The next post deploys the P/D disaggregation guide on a second GH200 instance — separate prefill and decode pods, NIXL sidecar for KV cache transfer between them. The setup, the results, and the honest account of what P/D disaggregation actually requires in terms of hardware are all in one post.

The question it answers: given the baseline established here — 81.1% cache hit rate, 260ms p50, 34ms P99 — does separating prefill and decode onto dedicated pods move those numbers, or have we already captured most of the available gain on a single aggregated pod? The answer is more nuanced than either “yes it’s better” or “no it’s not worth it” — and the hardware constraint that makes it nuanced is one the GH200’s architecture makes visible in a way discrete GPU systems don’t.

Experiments run on Lambda Labs GH200 480GB, llm-d v0.4.0, Qwen3-0.6B, vllm/vllm-openai:latest. Platform engineer with 11+ years in distributed systems going deep on LLM serving infrastructure.

GitHub · LinkedIn

I had done the Mac Mini experiments. I had real benchmark numbers. I understood the theory. I knew what prefill and decode were, I could explain PagedAttention at a whiteboard, and I had wired up Prometheus and Grafana from scratch without breaking anything important.

Then I tried to deploy llm-d on a cloud GPU and spent the better part of a weekend staring at permission denied, No such file or directory, image pull failed, and other messages that are technically informative and emotionally frustrating. I’ll be candid — I walked away from the terminal twice while the instance meter kept running. GPU rental has a way of focusing the mind.

This post is the deployment war story. Every broken thing, in roughly the order it broke. I am not sugarcoating my experiences or providing you a polished “here are the steps”. Those promises exist in the official docs that assumes a level of environmental cooperation or prior experiences that was not there. What the docs don’t tell you is what I’m writing here.

If you’re trying to deploy llm-d yourself, this post will save you several hours and possibly your sanity. If you’re reading this for entertainment, welcome — the GH200 and I had quite a journey.

What Is llm-d? (One Diagram, Then We Move On)

llm-d is a Kubernetes-native inference scheduling layer that sits on top of vLLM. It doesn’t replace vLLM — vLLM still runs inside each pod doing exactly what it always does. What llm-d adds is an EPP (Endpoint Picker Plugin) — a custom Kubernetes scheduler that routes each incoming request to the right vLLM pod based on KV cache state, queue depth, and prefix cache hit probability.

The pitch: instead of dumb round-robin load balancing, llm-d routes your request to the decode pod that already has your system prompt cached. Lower TTFT, better GPU utilization, independently scalable prefill and decode pools.

The official architecture diagram shows the contrast clearly:

That’s what you’re deploying. Now let’s talk about what it takes to actually run it.

The Hardware

Lambda Labs GH200, single instance, Grace Hopper Superchip.

GPU:          NVIDIA GH200 480GB (unified CPU+GPU memory)
Architecture: ARM64 (aarch64)
OS:           Ubuntu 22.04 LTS
Storage:      1.4TB local SSD

The ARM64 part is important — it comes up multiple times in this post in ways that will make you want to scream. Lambda’s GH200 instances come pre-installed with K3s, GPU drivers, and a collection of tools that seem helpful until they quietly conflict with everything you’re trying to do.

Before You Even Start: Getting the Instance

There is a step zero that nobody writes about: actually getting the GPU.

Cloud GPU availability — especially for high-end hardware like the GH200 — is genuinely constrained. Lambda Labs operates on a first-come, first-served basis for on-demand instances. When you log in and open the Launch Instance dialog, you will see a list of available GPU types. Some will be available. Some will say “Out of capacity.” The GH200 in particular comes and goes.

Getting set up on Lambda Labs:

Creating an account takes about 5 minutes. Go to lambda.ai, sign up, add a payment method, and add your SSH public key under SSH Keys in the dashboard. That last step is easy to forget and will stop you from connecting to any instance you launch.

# Your SSH public key — paste this into Lambda Labs SSH Keys dashboard
cat ~/.ssh/id_<my-key>.pub

Once your account is ready, the practical strategy for getting a GH200 on-demand is:

• Check availability in the morning — instances turn over as other users terminate their sessions overnight
• Have your SSH key already added — you want to be able to launch immediately when a slot opens
• Don’t launch and walk away — you’re paying per hour, so have your setup commands ready to run
• Set a budget alert — Lambda Labs doesn’t do this automatically; track your usage manually

What these experiments actually cost:

I ran two separate sessions, on two separate GH200 instances. Being honest about the numbers:

The second session was slightly more expensive because the P/D disaggregation setup took longer to get right — more iteration time on a running instance. The first session was faster in wall-clock time but I was also slower at debugging, which explains the comparable cost.

For context: $35.78 for two full days of hands-on GPU infrastructure experiments is genuinely reasonable. A single A100 hour on AWS is $3.50+. Lambda Labs on-demand pricing is competitive precisely because availability isn’t guaranteed — you trade reliability for cost.

One important note for LLMOps Researchers: I’m running these experiments to gain “production like experience”, which means I’m deliberate about GPU spend. On-demand instances that you terminate when done are the right strategy here — avoid reserved instances or always-on setups until you have a specific recurring workload that justifies the commitment.

Gotcha 1: The Kubeconfig Belongs to Root

First thing you do when you SSH into a new Kubernetes node: check the cluster is running.

kubectl get namespaces

What you get instead:

WARN[0000] Unable to read /etc/rancher/k3s/k3s.yaml, please start server
with --write-kubeconfig-mode or --write-kubeconfig-group to modify kube
config permissions
error: error loading config file "/etc/rancher/k3s/k3s.yaml":
open /etc/rancher/k3s/k3s.yaml: permission denied

K3s installs its kubeconfig at /etc/rancher/k3s/k3s.yaml and owns it as root. Your Ubuntu user is not root. Nobody tells you this in the getting-started guide because it seems like a detail, and it is — right until it silently breaks every single kubectl and helm command you run for the next two hours.

The fix: copy the config to your home directory and set the environment variable.

mkdir -p $HOME/.kube
sudo cp /etc/rancher/k3s/k3s.yaml $HOME/.kube/config
sudo chown ubuntu:ubuntu $HOME/.kube/config
export KUBECONFIG=$HOME/.kube/config

# Make it permanent — critical for survival across sessions
echo 'export KUBECONFIG=$HOME/.kube/config' >> ~/.bashrc
source ~/.bashrc

Now kubectl get nodes works. You feel a small surge of optimism. Cherish it.

Gotcha 2: Lambda Ships Snap Helm and It Is Not Your Friend

which helm
# /snap/bin/helm

ls -lla /snap/bin/helm
# lrwxrwxrwx 1 root root 13 → /usr/bin/snap

Lambda Stack pre-installs Helm via snap. Snap packages run in a sandbox with PATH and permission constraints that quietly break plugin installations and kubeconfig resolution. The llm-d deployment requires specific Helm plugins (helm-diff) and specific versions. The snap Helm and plugin system do not get along cleanly on ARM64.

The fix: remove snap Helm and install the official arm64 binary.

# Remove snap helm first
sudo snap remove helm

# Install official arm64 binary
wget https://get.helm.sh/helm-v3.19.0-linux-arm64.tar.gz
tar xzf helm-v3.19.0-linux-arm64.tar.gz
sudo mv linux-arm64/helm /usr/local/bin/helm
rm -rf linux-arm64 helm-v3.19.0-linux-arm64.tar.gz

# Verify
helm version
# version.BuildInfo{Version:"v3.19.0"...}

# Export and persist
export HELM_BIN=/usr/local/bin/helm
echo 'export HELM_BIN=/usr/local/bin/helm' >> ~/.bashrc
source ~/.bashrc

Notice I added HELM_BIN to .bashrc. This is foreshadowing.

Gotcha 3: Environment Variables Die When You Close the Terminal

This one is deceptively simple and caused a disproportionate amount of grief.

llm-d’s helmfile commands use HELM_BIN to find the Helm binary, and take the namespace via -n $NAMESPACE. Both are environment variables. Both need to exist in every session.

SSH sessions do not carry your previous session’s exported variables. Every time you reconnect, those variables are gone — and the error messages when they’re missing are spectacularly unhelpful:

# What happens when NAMESPACE is empty:
helmfile apply -n ${NAMESPACE}
# flag needs an argument: 'n' in -n

That message doesn’t say “your environment variable is empty.” It sends you hunting through Helm documentation for a flag you’ve never seen, before you eventually notice the problem.

The fix: put everything in ~/.bashrc and verify at the start of every session.

echo 'export KUBECONFIG=$HOME/.kube/config' >> ~/.bashrc
echo 'export HELM_BIN=/usr/local/bin/helm' >> ~/.bashrc
echo 'export NAMESPACE=llm-d' >> ~/.bashrc
source ~/.bashrc

# First command of every session:
echo $KUBECONFIG $HELM_BIN $NAMESPACE
# /home/ubuntu/.kube/config /usr/local/bin/helm llm-d

If those three don’t print correctly, nothing downstream will work.

Gotcha 4: The Default values.yaml Will Crash Your GPU

When you clone the llm-d repo and open the inference-scheduling guide’s values.yaml, the default looks like this:

modelArtifacts:
  uri: "hf://Qwen/Qwen3-32B/tensor"
  name: "Qwen/Qwen3-32B"
  size: 80Gi

Qwen3-32B. Eighty gigabytes. On a single GPU.

Here is why this matters. GPU memory is not a bottomless pool — it gets divided between everything vLLM needs to run:

┌─────────────────────────────────────────────────────────┐
│              GH200 — 480GB Unified Memory               │
├──────────────────────┬──────────────────────────────────┤
│  Model Weights       │  Qwen3-32B FP16 ≈ 65GB           │
│  (static, loaded     │  Qwen3-0.6B 4-bit ≈ 0.4GB        │
│   once at startup)   │                                  │
├──────────────────────┼──────────────────────────────────┤
│  KV Cache            │  Grows per token, per request    │
│  (dynamic, grows     │  Fills whatever is left over     │
│   with context)      │                                  │
├──────────────────────┼──────────────────────────────────┤
│  CUDA Graphs,        │  vLLM pre-allocates ~10–15%      │
│  Activations,        │  for warmup and graph capture    │
│  Overhead            │                                  │
└──────────────────────┴──────────────────────────────────┘

With Qwen3-32B, model weights alone consume 65GB. vLLM then pre-allocates KV cache for the maximum sequence length on top. Under concurrent load you are pushing the GPU extremely hard before a single user request arrives — and you’ve also got 20+ minutes of model download and warmup before you can even test anything.

With Qwen3-0.6B (~400MB), the model loads in under 2 minutes and 99% of the GPU is available for KV cache and experiments. That’s the version to start with.

The fix: replace the values file entirely. The full working values.yaml is below — note two critical changes from the default: model is Qwen3-0.6B, and the image is changed (explained in the next gotcha).

# ms-inference-scheduling/values.yaml — working version
multinode: false

modelArtifacts:
  uri: "hf://Qwen/Qwen3-0.6B"
  name: "Qwen/Qwen3-0.6B"
  size: 2Gi
  authSecretName: "llm-d-hf-token"
  labels:
    llm-d.ai/inference-serving: "true"
    llm-d.ai/guide: "inference-scheduling"
    llm-d.ai/accelerator-variant: "gpu"
    llm-d.ai/accelerator-vendor: "nvidia"
    llm-d.ai/model: "Qwen3-0.6B"

routing:
  proxy:
    enabled: false
    targetPort: 8000

accelerator:
  type: nvidia

decode:
  create: true
  parallelism:
    tensor: 1
    data: 1
  replicas: 1
  monitoring:
    podmonitor:
      enabled: true
      portName: "vllm"
      path: "/metrics"
      interval: "30s"
  containers:
    - name: "vllm"
      image: vllm/vllm-openai:latest   # ← critical change, see next gotcha
      modelCommand: vllmServe
      args:
        - "--disable-uvicorn-access-log"
        - "--gpu-memory-utilization=0.90"
      ports:
        - containerPort: 8000
          name: vllm
          protocol: TCP
      resources:
        limits:
          cpu: '16'
          memory: 60Gi
        requests:
          cpu: '8'
          memory: 30Gi
      mountModelVolume: true
      volumeMounts:
        - name: metrics-volume
          mountPath: /.config
        - name: shm
          mountPath: /dev/shm
        - name: torch-compile-cache
          mountPath: /.cache
      startupProbe:
        httpGet:
          path: /v1/models
          port: vllm
        initialDelaySeconds: 15
        periodSeconds: 30
        timeoutSeconds: 5
        failureThreshold: 120
      livenessProbe:
        httpGet:
          path: /health
          port: vllm
        periodSeconds: 10
        timeoutSeconds: 5
        failureThreshold: 3
      readinessProbe:
        httpGet:
          path: /v1/models
          port: vllm
        periodSeconds: 5
        timeoutSeconds: 2
        failureThreshold: 3
  volumes:
    - name: metrics-volume
      emptyDir: {}
    - name: torch-compile-cache
      emptyDir: {}
    - name: shm
      emptyDir:
        medium: Memory
        sizeLimit: 20Gi

prefill:
  create: false

Start small. Prove the stack works. Scale the model up later.

Gotcha 5: llm-d-cuda Is x86-Only

The default values.yaml uses ghcr.io/llm-d/llm-d-cuda:v0.6.0 as the vLLM container image. This image was built for x86_64 (amd64). The GH200 is ARM64 (aarch64).

When Kubernetes tries to run an x86 image on an ARM64 node, it doesn’t fail with “wrong architecture.” It fails with a Triton compiler error deep inside the container startup, several minutes after the pod appears to be Running:

Triton compilation failed: RuntimeError: ...
CUDA error: device-side assert triggered

The pod crashes. CrashLoopBackOff. kubectl describe pod shows the image pulled successfully. The logs look like GPU memory issues. You spend time adjusting --gpu-memory-utilization and redeploying — none of which helps, because the problem is binary architecture, not memory configuration.

The fix: vllm/vllm-openai:latest is multi-arch and includes a proper ARM64 build.

# Change this:
image: ghcr.io/llm-d/llm-d-cuda:v0.6.0

# To this:
image: vllm/vllm-openai:latest

One line. Saves two hours.

Gotcha 6: Gateway API CRDs Don’t Come With the Cluster

llm-d uses the Kubernetes Gateway API — HTTPRoute and Gateway custom resources. These are not part of standard Kubernetes and do not come pre-installed with K3s.

When helmfile apply runs without these CRDs, it fails with unknown resource type errors. If you’re not familiar with the Gateway API, you’ll look at your Istio installation instead of at the missing CRDs.

The fix: install both sets of CRDs before anything else.

# Gateway API CRDs
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml

# llm-d InferencePool CRDs
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v0.3.0/manifests.yaml

# Verify both
kubectl get crd | grep -E "gateway|inference"

Ten seconds to apply. Saves a debugging session.

Gotcha 7: Use helm template | kubectl apply — K8s Debugging 101

helmfile apply is the documented deployment path. It’s also the one that, on K3s ARM64, would silently partially deploy — some resources applied, others skipped, exit code 0.

This is actually a universal Helm debugging pattern worth internalising regardless of llm-d: render the chart to YAML first, then apply. Helmfile adds an abstraction layer that can obscure what’s actually being sent to the API server. helm template removes that layer and gives you full visibility.

# Step 1: Render to YAML and inspect — no cluster changes
$HELM_BIN template ms-inference-scheduling \
  llm-d-modelservice/llm-d-modelservice \
  --namespace llm-d \
  --values ms-inference-scheduling/values.yaml \
  | less

# Step 2: Dry-run — validates against cluster API, shows what would change
$HELM_BIN template ms-inference-scheduling \
  llm-d-modelservice/llm-d-modelservice \
  --namespace llm-d \
  --values ms-inference-scheduling/values.yaml \
  | kubectl apply -n llm-d -f - --dry-run=client

# Step 3: Apply for real
$HELM_BIN template ms-inference-scheduling \
  llm-d-modelservice/llm-d-modelservice \
  --namespace llm-d \
  --values ms-inference-scheduling/values.yaml \
  | kubectl apply -n llm-d -f -

The --dry-run=client step is the one most people skip and most people regret skipping. It validates your YAML against the cluster’s API schemas, catches missing CRDs, and shows exactly what would be created or updated — before anything changes in the cluster. Whenever a Helm deployment behaves unexpectedly, this three-step render-validate-apply pattern is where to start debugging.

Gotcha 8: The HTTPRoute Is Not Applied by helmfile

After all pods are running, you port-forward the gateway and send a test request. You get nothing back.

The reason: the HTTPRoute — the rule that tells the gateway to forward traffic to the EPP — is not deployed by helmfile. It lives in a separate YAML file and must be applied manually.

# Check first:
kubectl get httproute -n llm-d
# No resources found in llm-d namespace.

# Apply:
kubectl apply -f ~/llm-d/llm-d/guides/inference-scheduling/httproute.yaml -n llm-d

# Verify:
kubectl get httproute -n llm-d
# NAME                         AGE
# llm-d-inference-scheduling   6s

Without this, the gateway has no routing rules. Every request returns empty or times out. No errors appear in pod logs because — from the gateway’s perspective — nothing is wrong. There is just no route configured.

Gotcha 9: Immutable Selector Labels Mean You Can’t Upgrade In-Place

If you change the model name in values.yaml after deploying, helm upgrade will fail:

Error: UPGRADE FAILED: cannot patch "ms-inference-scheduling-llm-d-modelservice-decode"
with kind Deployment: Deployment.apps is invalid:
spec.selector: Invalid value: ... field is immutable

Kubernetes Deployment selector labels are immutable. Changing the model name — which is part of the label selector — requires deleting the Deployment first.

kubectl delete deployment -n llm-d \
  ms-inference-scheduling-llm-d-modelservice-decode 2>/dev/null || true

$HELM_BIN template ms-inference-scheduling \
  llm-d-modelservice/llm-d-modelservice \
  --namespace llm-d \
  --values ms-inference-scheduling/values.yaml \
  | kubectl apply -n llm-d -f -

The || true prevents failure if the deployment doesn’t exist yet — useful for idempotent scripts.

Gotcha 10: Port-Forward Processes Die Silently and Don’t Tell You

Access to the llm-d gateway from your Mac goes through an SSH tunnel plus kubectl port-forward. When the port-forward dies — session timeout, network hiccup — the TCP port on your Mac stays bound. Next tunnel attempt:

Unable to listen on port 8080:
[unable to create listener: Error listen tcp4 127.0.0.1:8080: bind: address already in use]

The fix: kill the zombie first, then verify the fresh tunnel is alive before benchmarking.

# Kill whatever holds port 8080
lsof -ti :8080 | xargs kill -9 2>/dev/null || true

# Fresh tunnel
ssh -L 8080:localhost:8080 ubuntu@<LAMBDA_IP> \
  "KUBECONFIG=/home/ubuntu/.kube/config kubectl port-forward \
   -n llm-d svc/infra-inference-scheduling-inference-gateway-istio 8080:80"

# Verify before doing anything else
curl -s http://localhost:8080/v1/models | jq .data[0].id
# "Qwen/Qwen3-0.6B"  ← tunnel alive
# (nothing / timeout) ← restart the tunnel

Running Locust against a dead port-forward gives you 0 completed requests and makes you think the model is broken. Always verify first.

What Running Looks Like

After all of the above — correct kubeconfig, real Helm binary, environment variables in .bashrc, small model, right image, Gateway API CRDs installed, HTTPRoute applied, Deployment deleted and recreated when needed — this is what a healthy deployment looks like:

kubectl get pods -n llm-d

NAME                                                              READY   STATUS    RESTARTS   AGE
gaie-inference-scheduling-epp-584f797cc8-4gvw8                    1/1     Running   0          68m
infra-inference-scheduling-inference-gateway-istio-7c5546dr7kd2   1/1     Running   0          68m
ms-inference-scheduling-llm-d-modelservice-decode-57678587gqzlt   1/1     Running   0          10m

Three pods, all 1/1 Running. The decode pod may restart once while the model downloads — that’s normal. Zero restarts on EPP and gateway.

# Gateway is programmed and routing
kubectl get gateway -n llm-d
# infra-inference-scheduling-inference-gateway   istio   True   68m

# A request through the gateway returns a response
curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen3-0.6B",
       "messages":[{"role":"user","content":"What is KV cache?"}],
       "max_tokens":50}' | jq .choices[0].message.content
# "<think>\nOkay, so I need to figure out what KV cache is..."

That response, after everything above, feels like a small engineering miracle.

The Observability Stack

The kube-prometheus-stack installs Grafana with 33 dashboards. After a fresh deploy, confirm everything is healthy before running experiments:

If the llm-d dashboards show “No data”, check that the PodMonitor was created:

kubectl get podmonitor -n llm-d
# If missing, ensure values.yaml has monitoring.podmonitor.enabled: true

The Survival Checklist

First commands of every session — before touching anything else:

# 1. Environment variables present
echo $KUBECONFIG $HELM_BIN $NAMESPACE
# /home/ubuntu/.kube/config /usr/local/bin/helm llm-d

# 2. All pods running
kubectl get pods -n llm-d
# All 3 pods: 1/1 Running

# 3. HTTPRoute exists
kubectl get httproute -n llm-d

# 4. Port-forward alive
curl -s http://localhost:8080/v1/models | jq .data[0].id
# "Qwen/Qwen3-0.6B"

# 5. Grafana accessible
curl -s http://localhost:3000/api/health | jq .database
# "ok"

If any of these fail, fix it before proceeding.

The Full Deployment Sequence

For anyone who wants the complete working sequence without the narrative:

# ── Fix environment ─────────────────────────────────────────────────
mkdir -p $HOME/.kube
sudo cp /etc/rancher/k3s/k3s.yaml $HOME/.kube/config
sudo chown ubuntu:ubuntu $HOME/.kube/config
echo 'export KUBECONFIG=$HOME/.kube/config' >> ~/.bashrc
echo 'export HELM_BIN=/usr/local/bin/helm' >> ~/.bashrc
echo 'export NAMESPACE=llm-d' >> ~/.bashrc
source ~/.bashrc

# ── Install real Helm (ARM64) ───────────────────────────────────────
sudo snap remove helm
wget https://get.helm.sh/helm-v3.19.0-linux-arm64.tar.gz
tar xzf helm-v3.19.0-linux-arm64.tar.gz
sudo mv linux-arm64/helm /usr/local/bin/helm

# ── Install CRDs ────────────────────────────────────────────────────
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v0.3.0/manifests.yaml

# ── Repos and HuggingFace secret ────────────────────────────────────
git clone https://github.com/llm-d/llm-d.git
$HELM_BIN repo add llm-d-modelservice \
  https://llm-d-incubation.github.io/llm-d-modelservice/
$HELM_BIN repo update

kubectl create namespace $NAMESPACE
kubectl create secret generic llm-d-hf-token \
  --from-literal="HF_TOKEN=${HF_TOKEN}" \
  --namespace "${NAMESPACE}" \
  --dry-run=client -o yaml | kubectl apply -f -

# ── Deploy infra (EPP + gateway) ────────────────────────────────────
cd ~/llm-d/llm-d/guides/inference-scheduling
HELM_BIN=$HELM_BIN helmfile apply -n ${NAMESPACE}

# ── Deploy modelservice (reliable pattern) ──────────────────────────
$HELM_BIN template ms-inference-scheduling \
  llm-d-modelservice/llm-d-modelservice \
  --namespace llm-d \
  --values ms-inference-scheduling/values.yaml \
  | kubectl apply -n llm-d -f -

# ── HTTPRoute (always manual) ───────────────────────────────────────
kubectl apply -f httproute.yaml -n llm-d

# ── Verify ──────────────────────────────────────────────────────────
kubectl get pods -n llm-d
curl -s http://localhost:8080/v1/models | jq .

Every step is here because I needed it.

What This Unlocks

With the stack running, the EPP is making routing decisions on every request — consulting prefix cache, queue depth, and KV cache utilization scorers on each incoming call. You just can’t see it yet with a single decode pod and no load.

The next post in this series covers what happens when you actually put the system under load — EPP prefix cache routing in action, KV cache hit rate climbing to 81.1% in Grafana, TTFT stabilising at 15ms p50 under sustained concurrent traffic, and the Locust results from a system that’s routing intelligently rather than guessing.

The deployment pain was worth it. The numbers make that clear.

Deployed on Lambda Labs GH200 480GB, K3s, llm-d v0.4.0, Qwen3-0.6B, vllm/vllm-openai:latest. Scripts will be made available soon via github repository. Platform engineer with 11+ years in distributed systems going deep on LLM serving infrastructure.

GitHub · LinkedIn

This post is the hands-on companion to Post 1. Same mental model — but now with real hardware, real Grafana dashboards, and numbers you can reproduce yourself.

Hardware: M4 Mac Mini, 16GB unified memory
Model: mlx-community/Qwen3-0.6B-4bit (~400MB, 4-bit quantized)
Inference engine: vllm-metal 0.13.0 (official vllm-project Apple Silicon plugin)
Observability: Prometheus + Grafana via Docker Compose
Load testing: vegeta + Locust
Gateway experiment: kind cluster + nginx reverse proxy

Why Apple Silicon for This Experiment?

The M4 Mac Mini has 16GB of unified memory — shared between CPU and GPU. This is both a strength and a constraint for inference.

Strength: Unified memory means the model weights aren’t copied between CPU RAM and a discrete GPU’s VRAM. Zero-copy tensor operations. For a small quantized model, this is genuinely fast.

Constraint: 16GB is 16GB. The model, the KV cache, the OS, and every other process share the same pool. Under concurrent load, you will find the ceiling.

The more interesting reason: I wanted to validate that the mental model from Post 1 — prefill is compute-bound, decode is memory-bound, KV cache is the critical resource — holds on real hardware, not just in theory. It does. With caveats.

The Setup

Why vllm-metal, Not Ollama?

Before any benchmarks: I used both, and the choice matters more than I expected.

vllm-metal is the official vllm-project Apple Silicon plugin. It runs Metal GPU kernels via MLX, exposes a full OpenAI-compatible API, and critically — emits Prometheus metrics out of the box. Same codebase as cloud vLLM, same API surface, same metric names. What you learn here transfers directly to a GPU cluster.

Ollama is simpler to install and great for single-user local use. But it doesn’t expose Prometheus metrics natively, and — as the benchmark section will show — it doesn’t implement continuous batching. Under concurrent load, that difference is not subtle.

One gotcha worth flagging: vllm-mlx (different from vllm-metal) is a third-party wrapper that was broken with mlx-lm>=0.31.0 as of April 2026. Use vllm-metal — the official plugin — and avoid that detour.

A note on scope: this post covers vLLM and Ollama only. TGI, TensorRT-LLM, ExLlamaV2, and SGLang are all on the list — but running meaningful benchmarks against each requires dedicated GPU time, and GPU time costs money I’m not spending during a job search. I’ll get to them when the experiments justify the cost. For now, the vLLM vs Ollama comparison is grounded in real numbers on real hardware, and that’s the comparison worth making.

Observability Stack

Prometheus and Grafana running via Docker Compose, with host.docker.internal resolving to the Mac’s loopback so the containers can scrape vLLM’s metrics endpoint:

# docker-compose.yml
services:
  prometheus:
    image: prom/prometheus:latest
    extra_hosts:
      - "host.docker.internal:host-gateway"
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin

# prometheus.yml
global:
  scrape_interval: 5s

scrape_configs:
  - job_name: vllm
    static_configs:
      - targets:
          - 'host.docker.internal:8000'

Metric name gotcha: the official vLLM Grafana dashboard references gpu_cache_usage_perc but vllm-metal exposes vllm:kv_cache_usage_perc. If your KV cache panel shows “No data”, confirm the correct metric name directly:

curl localhost:8000/metrics | grep cache

The kind Cluster and nginx Gateway

Why this matters: In production with llm-d, requests flow through an Envoy gateway pod before reaching vLLM pods. I wanted to replicate that topology locally — understand the gateway layer before Week 2 introduced it with real routing logic.

On Apple Silicon, Metal GPU cannot be passed into Docker containers. So vllm-metal has to run natively on the Mac host. The resulting topology is deliberately artificial:

── LOCAL SETUP (Apple Silicon constraint) ───────────────────────────

curl localhost:9000
     │
     ▼  host port 9000 → kind NodePort 30000
nginx pod :80        ← simulates Envoy gateway position in llm-d
     │
     ▼  proxy_pass http://172.19.0.1:8000
vllm-metal           ← running natively on Mac host (Metal GPU)

── PRODUCTION (Week 2 — real GPU) ──────────────────────────────────

curl gateway:80
     │
     ▼
Envoy gateway pod    ← EPP does KV-cache-aware routing here
     │
     ▼
vLLM pod             ← FastAPI server + GPU inside the pod

Apple Silicon gotcha: host.docker.internal inside kind on M4 Mac resolves to IPv6, but vllm-metal only binds to IPv4. The nginx proxy_pass fails silently — you’ll see connect() failed (101: Network unreachable) in the logs. Fix: get the actual bridge gateway IP:

HOST_IP=$(docker inspect vllm-lab-control-plane \
  --format '')
echo $HOST_IP
# 172.19.0.1  ← use this in proxy_pass, not host.docker.internal

# kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    extraPortMappings:
      - containerPort: 30000
        hostPort: 9000

Experiment 1: Baseline — TTFT and TPOT Directly Measured

First question: what does a single uncontested request actually cost on this hardware?

I wrote a streaming latency script that timestamps the first chunk arrival (TTFT) then tracks per-token intervals (TPOT):

def measure_streaming(prompt, max_tokens=100):
    start = time.perf_counter()
    first_token_time = None
    token_times = []

    with client.stream("POST", "/v1/chat/completions", json=payload) as resp:
        for line in resp.iter_lines():
            if content := parse_token(line):
                now = time.perf_counter()
                if first_token_time is None:
                    first_token_time = now  # ← TTFT captured here
                token_times.append(now)

    # TPOT = average gap between consecutive tokens
    intervals = [token_times[i] - token_times[i-1]
                 for i in range(1, len(token_times))]
    avg_tpot = sum(intervals) / len(intervals)

Results:

The asymmetry is the point. TTFT increased 42% when the prompt got 25× longer — more tokens to process in parallel during prefill. But TPOT barely moved (3.4ms → 4.2ms) because decode generates one token at a time regardless of prompt length. Once the model starts generating, the input is already in the KV cache and irrelevant to decode speed.

TPOT is determined by model size and hardware bandwidth. TTFT is determined by how long your user made their prompt.

Experiment 2: Prefix Cache — 51% TTFT Reduction for Free

KV cache stores computed key-value tensors for each token so they don’t need recomputing on subsequent steps. Prefix caching extends this across requests: if two requests share the same system prompt, the second reuses the KV cache blocks from the first.

I tested this with a long shared system prompt (~50 repetitions, creating ~200 tokens of identical prefix):

SYSTEM_PROMPT = "You are a helpful assistant. " * 50  # long shared prefix

chat("What is 2+2?",   "First request (cold cache)")
chat("What is 3+3?",   "Second request (warm cache)")
chat("What is 4+4?",   "Third request (warmer cache)")

Results:

51% TTFT reduction. Zero model changes. No infrastructure changes. Just sending the same system prompt byte-for-byte across requests.

The operational implication: your production system prompt should be at the front of every request, identical every time. Anything that mutates it per-request — timestamp injection, per-user personalization in the system prompt, A/B testing different prompts — kills the cache hit rate and silently taxes every user’s TTFT.

This is also the per-request proof of what llm-d’s EPP prefix-cache scorer does at cluster scale: route requests to the decode pod that already holds relevant KV cache blocks. What I measured locally as a 51% reduction is what the EPP maximises across dozens of pods.

Experiment 3: KV Cache Pressure Under Concurrent Load

I fired 4 long concurrent requests simultaneously — each asking for 500 output tokens — to observe how continuous batching handled them:

# kv_pressure.py
def send_long_request(i):
    r = httpx.post("http://localhost:8000/v1/chat/completions",
        json={"model": MODEL,
              "messages": [{"role": "user",
                           "content": f"Write a very long essay about distributed systems topic {i}. Be extremely verbose and detailed."}],
              "max_tokens": 500},
        timeout=120)
    tokens = r.json()["usage"]["completion_tokens"]
    print(f"Request {i}: {tokens} tokens in {elapsed:.1f}s ({tokens/elapsed:.1f} tok/s)")

with concurrent.futures.ThreadPoolExecutor(max_workers=4) as ex:
    futures = [ex.submit(send_long_request, i) for i in range(4)]

Results:

Request 0: 500 tokens in 24.9s (20.1 tok/s)
Request 1: 500 tokens in 24.9s (20.1 tok/s)
Request 2: 500 tokens in 24.9s (20.1 tok/s)
Request 3: 500 tokens in 24.9s (20.1 tok/s)

All four completed simultaneously — continuous batching working correctly. In a naive sequential server, 4 requests of this size would take roughly 4× the single-request time. Instead, vLLM batched decode steps across all four active requests in a single GPU pass per iteration.

During this test, vllm:kv_cache_usage_perc climbed to 4% in Grafana and returned to baseline when all requests completed. On a 0.6B model with these short prompts, plenty of headroom. The same pattern on a larger model pushes KV cache toward the 85% danger threshold where evictions begin and latency spikes.

Experiment 4: Locust Mixed Traffic — Where the M4 Hits Its Ceiling

Real traffic isn’t uniform. I configured Locust to simulate a 3:1 mix of short chatbot-style prompts and long document-summarization prompts:

class InferenceUser(HttpUser):
    wait_time = between(0.5, 2)

    @task(3)
    def short_request(self):   # 75% of traffic — "What is 2+2?" etc, max_tokens=50

    @task(1)
    def long_request(self):    # 25% of traffic — "Explain transformer attention...", max_tokens=200

Results at 5 concurrent users:

The long prompt p50 of 31 seconds is not a bug — it’s the fundamental prefill/decode competition at scale. Long prompts trigger expensive prefill operations that block decode steps for all concurrent requests. Short-prompt users feel it as TTFT spikes. Long-prompt users wait in a growing queue.

Notice that Inter Token Latency stays flat throughout. Once a request gets GPU time for decode, it’s fine. The problem is always getting to the front of the queue. This is exactly the problem P/D disaggregation solves — dedicate separate GPUs to prefill so long prompts never preempt decode.

Experiment 5: vLLM vs Ollama — The Head-to-Head

I benchmarked vllm-metal against Ollama using vegeta at a sustained 3 req/s over 30 seconds. The primary comparison is Ollama qwen2.5:0.5b vs vllm-metal Qwen3-0.6B-4bit — same model family, same approximate parameter count, different serving engines. Mistral 7B is included as a reference only.

vLLM at p50: 2.15× faster. At p95: 1.83× faster.

The minimum latency of 974ms — sub-second — is the clearest signal: when the server isn’t saturated, continuous batching delivers a first response before Ollama has even started processing. Ollama’s 2,569ms minimum reflects its sequential model — each request waits for the previous one to complete before the GPU is available.

The qwen3.5 failure (2.2% success) is instructive. A larger model at 3 req/s causes Ollama’s queue to back up until clients time out. vllm-metal handles the same rate at 100% success because it batches multiple requests into each GPU forward pass rather than serialising them.

Experiment 6: The nginx K8s Gateway — What the Proxy Layer Actually Costs

With the kind cluster running and nginx proxying to vllm-metal, I ran inference requests through the full gateway path. I used max_tokens=20 — tiny inference — so the measured latency is mostly proxy traversal overhead:

vegeta attack -rate=3/s -duration=15s \
  -targets=<(echo "POST http://localhost:9000/v1/chat/completions
Content-Type: application/json
@/tmp/vllm_gateway.json") | vegeta report

Results (45 requests, 3 req/s, max_tokens=20):

The 163ms minimum reflects proxy traversal cost, not inference time — max_tokens=20 on a 0.6B model generates tokens in tens of milliseconds. The meaningful result is 100% success at sustained rate. The gateway adds overhead but is not a bottleneck and does not drop requests.

In production, Envoy adds single-digit milliseconds — it’s purpose-built for high-throughput proxying. The nginx simulation here adds more, but the structural lesson holds: a gateway layer in front of vLLM does not meaningfully affect inference latency. What matters in llm-d is the EPP’s routing intelligence — prefix cache scoring, queue depth scoring, KV utilisation scoring — not the proxy overhead itself.

What Grafana Showed — The Full Picture

The dashboard below captures the most informative view — TTFT latency percentiles, the prefill vs decode time split, finish reasons, and prompt length distribution:

Three takeaways from Grafana that weren’t obvious before running the experiments:

TPOT is a red herring under load. Inter Token Latency stayed flat throughout every experiment — even when E2E latency climbed to 31 seconds for long prompts. The per-token decode speed is stable. The problem is always pre-decode queuing.

The prefill/decode time split is visible and separable. Prefill varies with prompt length, decode stays constant. In a disaggregated setup each line would come from a different pool of pods, independently scalable.

KV cache utilisation is your early warning system. Peak of 4% during these experiments — plenty of headroom for a 0.6B model. On a larger model or busier system, the moment this crosses 85% is your fire alarm.

The Connection to Week 2

Everything measured this week points to the same structural problem: in aggregated serving, prefill and decode compete for the same resources. Long prompts delay short ones. High-concurrency workloads cause cascading TTFT degradation that no amount of hardware scaling can fully fix — because the problem is architectural, not computational.

── AGGREGATED SERVING (what Week 1 showed) ──────────────────────────

Request A (long prompt)  → [  prefill 487ms  ][  decode decode decode ...  ]
Request B (short prompt) → [WAITING...       ][  prefill 341ms  ][ decode  ]
Request C (short prompt) → [WAITING.....................][prefill ][ decode  ]

── P/D DISAGGREGATION (what Week 2 will fix) ────────────────────────

Prefill pool: [ A-prefill ][ B-prefill ][ C-prefill ]  ← compute-bound
Decode pool:  [ A-decode  ][ B-decode  ][ C-decode  ]  ← memory-bandwidth-bound

Decode pool never blocks on prefill. TTFT stays consistent under load.

The numbers from this week — TTFT 341ms vs 487ms, prefix cache 51% reduction, 31-second long-prompt p50 under load — are the baseline. Week 2 is the comparison.

The Complete Benchmark Reference

Hardware: M4 Mac Mini, 16GB unified memory
Model:    mlx-community/Qwen3-0.6B-4bit
Engine:   vllm-metal 0.13.0

─── Single request baseline ──────────────────────────────
  TTFT (short prompt, ~20 tokens):  341ms
  TTFT (long prompt, ~500 tokens):  487ms  (+42%)
  TPOT (short):                     3.4ms  (≈ 294 tok/s)
  TPOT (long):                      4.2ms  (barely changed)
  Throughput (single request):      145.8 tok/s

─── Prefix cache ─────────────────────────────────────────
  Cold TTFT (first request):        753ms
  Warm TTFT (same prefix):          367ms  (51% reduction)
  Session cache hit ratio:          17.7%

─── KV pressure (4 concurrent × 500 tokens) ─────────────
  Wall time (all 4 concurrent):     24.9s
  Per-request throughput:           20.1 tok/s
  Result: continuous batching confirmed

─── Mixed load (5 users, 3:1 short/long via Locust) ─────
  short_prompt: avg=9,471ms  min=710ms  p50=1,900ms  max=23,219ms
  long_prompt:  avg=16,830ms min=3,152ms p50=31,000ms max=30,508ms

─── Ollama vs vLLM (3 req/s, 30s, vegeta) ───────────────
  Ollama / mistral 7B p50:         15,979ms  (reference — 10× larger)
  Ollama / qwen2.5:0.5b p50:       14,062ms  (primary comparison)
  Ollama / qwen3.5 (large):        timeout   (2.2% success)
  vLLM  / Qwen3-0.6B-4bit p50:     6,543ms
  vLLM advantage at p50:            2.15× faster
  vLLM advantage at p95:            1.83× faster

─── nginx K8s gateway (max_tokens=20, 45 requests) ──────
  p50:  232ms  |  p95: 277ms  |  min: 163ms  |  Success: 100%
  Note: mostly proxy overhead — max_tokens=20 is tiny inference

What This Points To

The Mac Mini experiments answered the questions they were designed for. TTFT/TPOT/KV cache behave exactly as the theory predicts. Continuous batching is real and measurable. The gateway layer adds overhead but doesn’t drop requests. And Ollama’s lack of continuous batching is not a footnote — it’s the difference between a useful serving system and one that falls over at 3 requests per second with a moderately sized model.

What the Mac Mini can’t answer: what happens when you separate prefill and decode onto dedicated hardware? What does EPP prefix-cache-aware routing look like in practice? And eventually — how do TGI, TensorRT-LLM, and SGLang compare under the same load test conditions? Those experiments need cloud GPU budget earmarked for specific Week 2 and Week 3 labs. When they happen, they’ll get their own posts with real numbers.

Next up: Post 3 covers llm-d deployment on a Lambda Labs GH200 — the ten things nobody tells you before you try to run a Helm-based inference scheduler on K3s, including the NIXL/RDMA failure that explains why single-GPU P/D disaggregation doesn’t work the way you’d hope.

All experiments run on M4 Mac Mini 16GB, vllm-metal 0.13.0, Qwen3-0.6B-4bit. Scripts will be made available soon via github repository. I’m an platform engineer with 11+ years in distributed systems currently going deep on LLM serving. I write what I actually measured, including the parts that hit walls.

GitHub · LinkedIn

And yet the first time someone asked me what actually happens during inference, I said something like “the model reads the prompt and generates tokens.” Which is technically true the same way “a database reads your query and returns rows” is technically true — accurate, useless, and deeply embarrassing for someone drawing a principal engineer’s salary.

This post is what I wish I’d had on day one. We’re going to walk through the entire inference pipeline — from the moment your request arrives to the moment you see text on screen — with real examples, honest explanations of where the performance goes, and enough detail that you can actually reason about production problems.

No “and then the transformer does its thing.” No skipped steps. Strap in.

1. What Is Inference?

Training is where you take a massive dataset, run it through a model millions of times, and slowly adjust billions of numerical weights until the model gets good at predicting the next word. Training is done once (or occasionally). It costs millions of dollars in GPU-hours and requires a team of researchers.

Inference is what happens afterward, every time someone uses the model. It’s the model using those learned weights to respond to new input. No weights change. No learning happens. It’s pure forward-pass computation.

Think of it like this: training is baking the bread. Inference is slicing it and serving it to customers. The bread (weights) is done. The kitchen (inference engine) just has to plate it fast enough that the queue doesn’t back up to the street.

The inference engine is the runtime that takes the frozen model weights and executes them against an input. The same weights can run on Ollama, vLLM, TensorRT-LLM, or TGI — and get meaningfully different performance from each. The weights don’t change. The execution strategy does.

This distinction matters operationally: inference is not a solved problem. Serving a model efficiently at scale is a full engineering discipline.

2. The Artifact: What’s Actually in That 10GB Download?

When you run ollama pull mistral or grab a model from HuggingFace, you aren’t just downloading a “program.” You’re downloading a massive, frozen brain in a box. If you’ve ever wondered why a model that “just chats” takes up 10GB of your SSD, it’s because it is packed with billions of tiny numerical “preferences” the model learned during its training phase.

Think of the GGUF (or Safetensors) file as a giant Ikea flat-pack box. To build the working model, you need two things: the Instruction Manual and the Hardware.

What’s inside a 7B parameter model file:

GGUF file structure (simplified):
├── Header
│   ├── Model architecture (LlamaForCausalLM)
│   ├── Vocabulary (32000 tokens + their embeddings)
│   ├── Context length (4096, 8192, etc.)
│   └── Hyperparameters (n_layers, n_heads, etc.)
│
└── Weight tensors:
    ├── token_embeddings        [32000 × 4096]   ← the embedding matrix
    ├── layer.0.attention.q     [4096 × 4096]    ← Query projection weights
    ├── layer.0.attention.k     [4096 × 4096]    ← Key projection weights
    ├── layer.0.attention.v     [4096 × 4096]    ← Value projection weights
    ├── layer.0.attention.out   [4096 × 4096]    ← Output projection
    ├── layer.0.ffn.up          [4096 × 11008]   ← Feed-forward up
    ├── layer.0.ffn.down        [11008 × 4096]   ← Feed-forward down
    ├── ... × 32 layers
    └── output_norm + lm_head   [32000 × 4096]   ← Final projection to logits

The “Manual” (The Header)

This is the first few kilobytes of the file. It tells the inference engine (like Ollama or vLLM) how to put the brain together. It includes:

• The Architecture: Identifies the model type (e.g., LlamaForCausalLM) so the engine knows which math rules to apply.
• The Vocabulary: A dictionary of roughly 32,000 to 128,000 “tokens” (the syllables the model speaks).
• The Hyperparameters: Crucial settings like the number of layers (32 or 80) and the context length (how much it can remember).

The “Hardware” (The Tensors)

The rest of that file is just rows and rows of numbers called Weights. Every inference request is essentially looking up values from these matrices and multiplying them together—32 times over.

Quantization: Shrinking the Brain

You might notice some files are 15GB while others are 4GB for the same model. This is Quantization—the art of compression. We turn high-precision 16-bit floats into lower-precision integers (like 4-bit).

Why SREs love INT4: Lower precision = smaller tensors = faster memory transfers. Because decoding is memory-bound, an INT4 model often delivers 20-40% better TPOT (tokens per second) than the “full” version because the memory bus isn’t screaming as loud.

The takeaway: You aren’t executing code; you are loading a massive, math-heavy lookup table. GGUF is your single-file “box,” and quantization is how you fit that box into a smaller truck (your GPU).

3. The Three Phases: A Map Before the Territory

Every inference request goes through three broad phases. They are not equally expensive, not equally parallelizable, and not equally friendly to your p99 latency.

┌──────────────────────────────────────────────────────────────────┐
│  TOKENIZATION  │  Prefill (Prompt Processing)   │ Decode         │
│    (CPU)       │           (GPU, compute)       │ Loop(GPU, Mem) │
│                │                                │                │
│                │                                │ current →      │
│  Text → IDs    │  Embed → Position → Attention  │ next token     │
└──────────────────────────────────────────────────────────────────┘
     Fast          Scales with prompt length        Slow

• Tokenization: Split the text into token IDs the model understands. CPU-bound. Fast.
• Prefill: Process the entire prompt through the model. GPU compute-bound. Scales with prompt length.
• Decode: Generate output tokens one at a time. GPU memory-bound. Runs in a loop until done.

Each phase has its own bottleneck. Let’s go through them one by one.

4. Tokenization: Chopping Text Into Numbers

Before a single GPU operation happens, your text has to be converted into a format the model can work with: a sequence of integers called token IDs.

A token is not a character, and it’s not a word. It’s a chunk of text that appears frequently enough in the training corpus to deserve its own ID. There are several ways to build this vocabulary — WordPiece (used by BERT), Unigram (used by SentencePiece), and others — but the dominant approach in modern LLMs is Byte Pair Encoding (BPE): a compression algorithm that iteratively merges the most common pairs of characters or subwords into single tokens until it reaches a target vocabulary size.

The result is a vocabulary of roughly 32,000–128,000 tokens, each with a corresponding integer ID. The model never sees your text — it sees a list of numbers.

Take our example prompt: "The cat sat"

After tokenization, this becomes something like:

"The"  → 1026
" cat" → 5992
" sat" → 3290

Token IDs: [1026, 5992, 3290]

Note the space before “cat” and “sat” — it’s part of the token. Tokenizers care about whitespace because it affects meaning and frequency.

Is Tokenization CPU-Bound?

Yes. The tokenizer is usually written in Rust (HuggingFace’s tokenizers crate) or C++ for exactly this reason. For most requests it’s fast enough to be invisible — microseconds for a short prompt.

Where it bites you: very long documents fed to batch processing jobs. A 100,000-token context requires processing 100,000 token lookups. It’s still fast relative to GPU work, but it’s the one step in the pipeline running on CPU that you can’t just throw more GPU at.

How it’s improved: Parallelizing tokenization across CPU cores for batch workloads. Or — and this is the real fix — not re-tokenizing the same content repeatedly. If you have a shared system prompt you send to every request, tokenizing it once and caching the result is free latency.

5. Prefill: The Model Reads Your Prompt

Now we have token IDs. The model needs to turn those IDs into something it can reason about. This is prefill — the model processing the entire prompt in one shot.

Prefill has two sub-steps that are easy to conflate: embedding lookup and the actual transformer forward pass. Let’s take them in order.

The Embedding Matrix

Every token ID maps to a high-dimensional vector of floating-point numbers called an embedding. These vectors live in the model’s embedding matrix — a giant lookup table with one row per vocabulary token and one column per embedding dimension.

For a model with a 32,000-token vocabulary and 4,096 embedding dimensions, this matrix has shape [32000, 4096]. At 16-bit float precision, that’s about 256MB just for the embedding layer.

Our example [1026, 5992, 3290] becomes:

Token ID 1026 → embedding row 1026 → [0.12, -0.43, 0.81, ..., 0.07]  (4096 values)
Token ID 5992 → embedding row 5992 → [-0.34, 0.91, 0.12, ..., -0.22] (4096 values)
Token ID 3290 → embedding row 3290 → [0.67, 0.05, -0.88, ..., 0.44]  (4096 values)

I’m simplifying to 8 dimensions here so this fits on a page. In reality it’s 4,096 or 8,192 dimensions depending on the model.

Simplified (3D instead of 4096D), just to show the shape:

"The"  → [0.12, -0.43,  0.81]
" cat" → [-0.34,  0.91,  0.12]
" sat" → [0.67,  0.05, -0.88]

Shape: [3 tokens × 3 dims] = a matrix of floats

These vectors aren’t random. They’re the result of training — the model has learned that “cat” and “dog” live close together in this space, and “cat” and “quantum mechanics” are far apart. The geometry encodes semantic meaning.

How Do the Model Weights Help Here?

The embedding matrix IS the model weights, specifically. The 10GB (or 40GB, or 70GB) file you download — the GGUF or safetensors file — contains all the weight matrices the model learned during training. The embedding lookup is literally indexing into one of those weight matrices by row number.

When you run inference, you’re not computing anything creative. You’re doing matrix math against frozen numbers that were tuned over millions of training iterations.

6. Positional Embeddings: Teaching the Model About Order

Here’s a problem: the embedding lookup is a table lookup. It doesn’t care that “cat” is token 2 and “sat” is token 3. Two requests with the same tokens in different orders would produce identical embeddings.

But order matters enormously. “The cat sat on the dog” and “The dog sat on the cat” have the same tokens and very different meanings.

Positional embeddings solve this by adding a position-aware vector to each token’s embedding. The model learns that “token at position 1” feels different from “token at position 5,” even if the token ID is the same.

How Is It Calculated?

There are two main approaches:

Sinusoidal (original Transformers paper): Compute a fixed sine/cosine wave pattern based on position and dimension index. Deterministic, no learned parameters.

RoPE (Rotary Position Embedding): Used by Llama, Qwen, Mistral, and most modern models. Instead of adding a vector, it rotates the query and key vectors by an angle proportional to position. The result: the dot product between two token representations naturally captures their relative distance. Elegant, and generalizes better to longer contexts than the training data.

Continuing our example. After adding positional information:

"The"  at position 0: [0.12, -0.43, 0.81] + pos(0) → [0.15, -0.40, 0.84]
" cat" at position 1: [-0.34, 0.91, 0.12] + pos(1) → [-0.31, 0.88, 0.09]
" sat" at position 2: [0.67, 0.05, -0.88] + pos(2) → [0.65, 0.03, -0.86]

The position vectors are small adjustments. Their real value is that when attention is computed later, the model can tell whether two tokens are adjacent or 200 positions apart.

CPU Bottleneck in Prefill?

Embedding lookup and positional encoding are fast operations. The real CPU bottleneck in prefill is less about these steps and more about data movement: loading the right weight tensors from CPU RAM to GPU VRAM before the transformer forward pass can begin.

For very large models that don’t fully fit in VRAM, the CPU-GPU transfer becomes the bottleneck — you’re constantly paging weight blocks in. This is why model quantization matters: a 4-bit quantized model uses less VRAM, fits entirely on GPU, and eliminates this transfer overhead. More on that in a moment.

7. The Transformer Layers: Where the Real Work Happens

After embedding + positional encoding, we have a matrix of shape [sequence_length × embedding_dim]. This matrix now passes through N transformer layers — 32 layers for Llama-3.2-3B, 80 layers for Llama-3.1-70B.

Each layer applies:

1. Self-attention: every token looks at every other token and decides what’s relevant
2. Feed-forward network (FFN): each token’s representation is independently transformed

This is where the model’s “reasoning” happens — and where most of the GPU compute goes during prefill. All tokens are processed in parallel within a layer, making prefill compute-bound. More tokens = more compute = higher TTFT.

We’ll cover the attention mechanism in detail in section 9. First, let’s see what comes out.

8. Decoding: One Token at a Time, Forever

After prefill, the model produces its first output token. Then it produces another. Then another. Each token depends on all previous tokens. This is the decode loop.

Here’s what makes decode fundamentally different from prefill: you can’t parallelize it. Token N can’t be computed until token N-1 exists. It’s inherently sequential.

Let’s walk through two steps with our example. Our prompt was “The cat sat” and let’s say the model is going to output “on the mat.”

Decode Step 1: Predicting “on”

After prefill, we have KV cache entries for “The”, “ cat”, “ sat” (we’ll explain KV cache shortly). Now:

1. The model takes the last token’s representation (“ sat”) and runs it through the transformer layers
2. At each layer, attention is computed between “ sat” and all previous tokens via the KV cache
3. The final layer outputs a vector of size [vocabulary_size] — one score per possible next token. This is called the logits vector.
4. The logits are converted to probabilities via softmax
5. A token is sampled from this distribution (more below)
6. Result: token ID for “ on” → " on" is emitted as the first output token

KV cache: ["The", " cat", " sat"]
Current:  " sat" (last input token)
Attention: " sat" attends to "The", " cat", " sat"
Output logits: [0.001, 0.003, ..., 0.45 (" on"), ..., 0.002]
Sample: " on" ✓

Decode Step 2: Predicting “the”

Now “ on” has been generated. We add it to context:

1. Embed token “ on” → one new embedding vector (just one token, not the whole sequence)
2. Add positional embedding for position 4
3. Run through transformer layers, attending to KV cache for [“The”, “ cat”, “ sat”, “ on”]
4. Output logits → sample → “ the”

KV cache: ["The", " cat", " sat", " on"]  ← one entry added
Current:  " on" (new last token)
Attention: " on" attends to all four previous tokens
Output logits: [..., 0.67 (" the"), ...]
Sample: " the" ✓

And so it continues: “ mat” → “.” → <end> token → stop.

The Sampling Step (Where Creativity Lives)

The logits give you a probability distribution. How you sample from it is where temperature, top-k, and top-p come in:

• Greedy (temperature=0): always pick the highest probability token. Deterministic. Boring for creative tasks, good for code.
• Temperature > 1: flatten the distribution. More randomness, more surprising outputs, more hallucinations.
• Temperature < 1: sharpen the distribution. More conservative, more predictable.
• Top-k: only sample from the top K most probable tokens. Ignores the long tail.
• Top-p (nucleus sampling): only sample from the smallest set of tokens whose cumulative probability exceeds p. Adaptive — sometimes that’s 2 tokens, sometimes 50.

This step is trivially cheap computationally but has enormous impact on output quality. As an SRE, you don’t usually tune this — but you will get bug reports when someone’s temperature=2.0 config makes the model output Shakespeare from a JSON API endpoint.

9. Why Memory Is the Decode Bottleneck

Here’s the thing about decode that makes it hard to optimize: on every single decode step, the model needs to run attention against all previous tokens. Not a summary of them. All of them. Via the KV cache.

For a 1000-token conversation, every decode step reads 1000 rows of KV tensors from GPU memory. For a 32-layer model, that’s 32 reads of a large tensor. For a 7B model, the KV entry for a single token at a single layer is tens of kilobytes.

The GPU’s compute cores can execute these operations fast. But they’re waiting on HBM (High Bandwidth Memory) to deliver the data. The memory bus saturates before the compute does.

The GPU is memory-bound during decode, not compute-bound.

This is why adding more CUDA cores doesn’t help decode performance as much as adding more memory bandwidth. It’s why H100s are faster than A100s for serving despite similar compute specs — the memory bandwidth jump matters more for decode than the FLOP count.

A rough intuition: during prefill, GPU utilization is high and memory is barely stressed. During decode, GPU compute is mostly idle and the memory bus is screaming.

10. KV Cache: The Most Important Data Structure in Inference

We keep mentioning the KV cache. Let’s make it concrete.

During the attention step in each transformer layer, every token produces two vectors: a Key (K) and a Value (V). These are used in attention computation: other tokens use your Key to decide how much to attend to you, and then use your Value to extract information from you.

During decode, token N needs to compute attention against tokens 1 through N-1. If we recomputed K and V for all previous tokens on every step, we’d be doing O(N²) work per decode step. That’s catastrophic.

The KV cache solves this: after computing K and V for a token, we store them. On the next decode step, we only compute K and V for the new token and look up the rest from cache.

After prefill of "The cat sat":
KV cache = {
  layer_0: { K: [k_The, k_cat, k_sat], V: [v_The, v_cat, v_sat] }
  layer_1: { K: [...], V: [...] }
  ...  (32 layers total)
}

After generating " on":
KV cache = {
  layer_0: { K: [k_The, k_cat, k_sat, k_on], V: [v_The, v_cat, v_sat, v_on] }
  ...
}
← one new entry appended per layer per decode step

The cache grows with every token generated. When the KV cache fills GPU memory:

• New requests queue (they have nowhere to store their KV tensors)
• Long requests get partially evicted and have to recompute (latency spike)
• In the worst case: OOM crash

KV cache occupancy is the single most important resource to monitor in a serving system. It determines how many concurrent requests you can serve and how long those requests can be. When vllm:kv_cache_usage_perc starts approaching 0.9, you’re about to have a bad time.

Prefix Caching: Free Speedups

If two requests share the same system prompt, their KV cache entries for that prefix are identical. Prefix caching stores those entries once and reuses them.

In practice on my M4 Mac Mini:

Without prefix cache (cold):  TTFT 753ms
With prefix cache (warm):     TTFT 367ms
Savings:                       51% reduction in TTFT
Zero model changes required.

This is why your production system prompt should be at the beginning of every request, and why it should be identical byte-for-byte across requests. Drift in the system prompt = cache misses = higher TTFT = sad users.

11. Attention: The Mechanism That Makes It Work

Let’s go one level deeper into what happens at each transformer layer. Attention is the core operation. Everything else is bookkeeping.

Self-Attention in Plain English

For each token, attention computes: “which other tokens should I be paying attention to, and by how much?”

It does this via three learned projections of each token’s embedding:

• Query (Q): “what am I looking for?”
• Key (K): “what do I offer?”
• Value (V): “what information do I contain?”

For each token, you compute its dot product with every other token’s Key. High dot product = high attention score = attend more to that token. The scores are normalized via softmax, then used to weight-sum all the Value vectors.

Example with “The cat sat”:

Token: " sat"
Q_sat · K_The  = 0.8  → attend heavily to "The"
Q_sat · K_cat  = 0.9  → attend most to " cat" (makes sense)
Q_sat · K_sat  = 0.3  → attend a little to itself

Attention weights after softmax: [0.35, 0.55, 0.10]

Output = 0.35 × V_The + 0.55 × V_cat + 0.10 × V_sat

The output for “ sat” is now a blend of information from all tokens, weighted by relevance. After 32 such layers, the model has a rich, contextualized representation of every token in the sequence.

Paged Attention: Virtual Memory for KV Cache

Standard attention assumes the KV cache is one contiguous block of memory per request. This is wasteful: you have to pre-allocate the maximum possible sequence length upfront, and if the request ends early, that memory is wasted until the request completes.

PagedAttention (the key innovation in vLLM) borrows from OS virtual memory. Instead of one contiguous block, KV cache is stored in fixed-size pages (blocks) that can be non-contiguous in physical GPU memory. A page table maps logical token positions to physical blocks.

Without PagedAttention:
  Request A: [KKKKKKKKKK........] ← pre-allocated 20 slots, using 10, wasting 10
  Request B: [KKKKKK............] ← pre-allocated 20 slots, using 6, wasting 14

With PagedAttention:
  Block pool: [B1][B2][B3][B4][B5][B6][B7][B8]...
  Request A:  blocks B1, B3, B7 (non-contiguous, allocated on demand)
  Request B:  blocks B2, B4    (uses only what it needs)

The result: much higher memory utilization, more concurrent requests, less waste. Prefix cache blocks can be shared between requests with identical prefixes — only one copy of the system prompt’s KV entries needed, regardless of how many requests use it.

PagedAttention is why vLLM typically serves 2-4x more concurrent requests than a naive implementation on the same hardware.

12. Continuous Batching: The Throughput Unlock

Early inference servers were naive: accept a batch of requests, run them all through the model together, return all responses. Simple.

The problem: requests finish at different times. Short requests had to wait for long requests to complete before the next batch could start. GPU utilization looked like a sawtooth wave.

Continuous batching (also called iteration-level scheduling) fixes this. Instead of batching at the request level, the inference engine batches at the decode step level. Every iteration, it assembles the currently active tokens — some mid-generation, some just starting — into a single GPU operation.

When a request finishes, its slot is immediately available for a new request. When a new request arrives, it joins the active batch at the next iteration rather than waiting for the next batch boundary.

The result: GPU utilization stays high, latency for new requests is low, and throughput scales with the number of concurrent requests the KV cache can support — not with some fixed batch size parameter you tuned last Tuesday.

vLLM, TGI, and TensorRT-LLM all implement continuous batching. Ollama does not (as of early 2025). This is one of the primary reasons vLLM serves at 2x the throughput of Ollama under concurrent load.

13. The Metrics That Matter

Now that you know what’s happening, the metrics become obvious rather than mysterious.

TTFT — Time to First Token

Definition: Wall clock time from request submission to first output token.

What it captures: Prefill time + queuing time. If your GPU is busy, TTFT absorbs the wait.

PromQL:

histogram_quantile(0.95,
  sum(rate(vllm:time_to_first_token_seconds_bucket[2m])) by (le)
)

SLO guidance: < 500ms for interactive chat. < 2s is tolerable. > 5s and users think it’s broken.

Root causes when high:

• Long prompts (expected — prefill scales with length)
• GPU under heavy load, requests queuing
• Insufficient prefill capacity (in P/D disaggregated setups)

ITL — Inter-Token Latency (aka TPOT)

Definition: Average time between consecutive output tokens during decode. Inverse of token generation speed.

What it captures: Decode throughput per active request. Memory bandwidth is the primary lever.

PromQL:

histogram_quantile(0.95,
  sum(rate(vllm:time_per_output_token_seconds_bucket[2m])) by (le)
)

SLO guidance: < 30ms is fast, streaming feels smooth. > 100ms and you notice the typewriter effect.

Root causes when high:

• Too many concurrent requests (KV cache reads competing)
• Large model + small GPU memory bandwidth
• KV cache approaching capacity

KV Cache Hit Ratio

Definition: Fraction of prompt tokens whose KV vectors were served from prefix cache vs recomputed.

What it captures: Effectiveness of prefix caching. High hit ratio = lower TTFT for repeated system prompts.

PromQL:

vllm:gpu_prefix_cache_hit_rate

Target: > 0.5 for most chat workloads with consistent system prompts. Near 0 means your prompts are fully unique (batch processing, no shared prefix).

KV Cache Usage

Definition: Fraction of total KV cache capacity currently occupied.

PromQL:

vllm:gpu_cache_usage_perc

Alert threshold: > 0.85. At 0.9+, vLLM starts evicting in-progress requests, causing recomputation and latency spikes. At 1.0, new requests queue entirely.

Scaling Strategy by Traffic Shape

This is where the SRE work actually lives:

The strategic insight: short prompt, many concurrent users → decode is the bottleneck, optimize for memory bandwidth and parallelism across requests. Long context, few users → prefill is the bottleneck and KV cache pressure is the constraint; P/D disaggregation helps by giving prefill its own GPU.

14. Where Does the Time Actually Go?

After running these experiments on real hardware, here’s the honest answer:

Typical inference request (short prompt, moderate load):

Tokenization:          ~0.5ms   (CPU, negligible)
Embedding lookup:      ~1ms     (GPU memory read)
Prefill (32 layers):   ~40ms    (GPU compute, scales with prompt length)
First token decode:    ~3-5ms   (GPU memory read, KV cache write)
...each subsequent token: ~3-5ms

Total TTFT:            ~45ms under no load
Total TTFT at p95:     300-500ms under load (queuing dominates)

Where load makes it worse:

1. Queuing before prefill starts: your request sits behind other long prefills. TTFT absorbs the entire queue wait.
2. KV cache contention during decode: more concurrent requests = more KV cache reads per step = higher ITL for everyone.
3. Memory fragmentation: without PagedAttention, wasted KV cache blocks reduce effective concurrency.

What’s predicted to improve this:

• Speculative decoding: a small “draft” model generates 4-5 tokens speculatively; the large model verifies them in one forward pass. If accepted, 4 tokens for the price of ~1.5 forward passes. Reduces ITL dramatically under low concurrency, hurts at high concurrency (wasted draft compute).
• P/D disaggregation: dedicated prefill GPUs handle prompt processing, dedicated decode GPUs handle generation. Eliminates the resource contention between phases. Requires fast interconnect (NVLink or RDMA) for KV transfer.
• Flash Attention 3: kernel-level optimization that keeps attention computation in SRAM longer, reducing HBM reads. Already default in vLLM on H100.

15. The Inference Engines Worth Knowing

Not all inference engines are created equal, and the right tool depends on your constraints.

Ollama

• Origin: Open source, community
• Strengths: Dead simple to run, supports Apple Silicon natively, GGUF format
• Weaknesses: No continuous batching (as of early 2025), worse throughput under concurrent load
• When to use: Local development, single-user experiments, quick model testing
• When not to use: Serving more than one user concurrently
• GitHub: github.com/ollama/ollama

vLLM

• Origin: UC Berkeley, now a major open-source project with significant industry contributors
• Strengths: PagedAttention, continuous batching, Prometheus metrics out of the box, P/D disaggregation via llm-d
• Weaknesses: More complex setup, CUDA-first (Apple Silicon support via vllm-metal is experimental)
• When to use: Production serving, multi-user, research with real load
• When not to use: You just want to run one model locally and don’t want to think about it
• GitHub: github.com/vllm-project/vllm

TGI (Text Generation Inference)

• Origin: HuggingFace
• Strengths: First-class support for new HuggingFace models, FlashAttention, tensor parallelism
• Weaknesses: Slower to adopt innovations than vLLM, somewhat opinionated config
• When to use: You’re already in the HuggingFace ecosystem and want good defaults
• GitHub: github.com/huggingface/text-generation-inference

TensorRT-LLM

• Origin: NVIDIA
• Strengths: Best possible performance on NVIDIA hardware, optimized kernels, inference graph compilation
• Weaknesses: NVIDIA-only, complex setup, compiled engines are model+hardware-specific (can’t move them)
• When to use: You have a fixed model, fixed NVIDIA hardware, and need maximum performance
• When not to use: You want flexibility, you’re running experiments, or you don’t own your hardware
• GitHub: github.com/NVIDIA/TensorRT-LLM

The META / Research Options

• llama.cpp — github.com/ggerganov/llama.cpp: The CPU-first runtime. Runs quantized models on CPU, reasonably fast, the ancestor of Ollama.
• ExLlamaV2 — github.com/turboderp-org/exllamav2: Highly optimized for RTX GPUs specifically. EXL2 quantization format is more sophisticated than GPTQ or AWQ — per-layer bit allocation instead of uniform quantization.
• MLC-LLM — github.com/mlc-ai/mlc-llm: Cross-platform (compiles to CUDA, Metal, Vulkan). Good for deploying to diverse hardware.

16. The Languages Behind It All

If you open the source code of a modern inference engine, here’s what you’ll find:

Python: The top layer. API server, request handling, scheduling logic, metric collection. vLLM’s scheduler and OpenAI-compatible API are Python. This is also where most bugs live.

CUDA (C++): The performance layer. Attention kernels, memory management, GPU operations. Flash Attention is CUDA. PagedAttention’s physical block management is CUDA. If Python is the restaurant, CUDA is the kitchen.

Rust: The fast utilities layer. HuggingFace’s tokenizers library is Rust — because tokenizing millions of requests fast matters. NIXL (the KV cache transfer layer in llm-d) has Rust/C++ components. Growing presence in inference tooling.

Go: The orchestration layer. Kubernetes operators, control plane tooling, health checks, routing logic. If you’re writing infrastructure around inference — operators, routers, schedulers — Go is where the work happens.

C++ (non-CUDA): llama.cpp is pure C++ with optional CUDA/Metal backends. TensorRT-LLM has heavy C++ in the engine.

For SREs/DevOps: You live in Python (scripts, load tests), Go (operators, K8s controllers), and YAML (unfortunately). CUDA is worth being able to read — not write, just understand why a kernel fusion matters and what a grid/block size means. Rust fluency is a genuine differentiator if you want to contribute upstream.

17. What CPU vs. Memory Intensive Means, Summarized

After all of the above, here’s the clean summary of which hardware resource each step stresses:

The pattern: prefill is compute-bound, decode is memory-bound. This is the fundamental tension that all inference optimization — P/D disaggregation, speculative decoding, PagedAttention, quantization — is ultimately trying to resolve.

18. What an SRE Actually Needs to Know

You don’t need to write CUDA. You don’t need to derive the attention formula. But you do need a mental model that lets you answer these questions in production:

• Why is TTFT high? → Prefill bottleneck or queuing. Long prompts? GPU saturated?
• Why is ITL degrading? → KV cache pressure. Too many concurrent requests. Memory bandwidth saturating.
• Why did the GPU OOM? → KV cache exhausted. Too many long requests, no eviction headroom.
• Why is throughput low? → No continuous batching. Poor concurrency config. Batch size too small.
• Why does prefix caching not help? → System prompt is changing per-request. Fix the app layer.
• Which GPU should I buy? → For inference: memory bandwidth matters more than FLOP count. H100 > A100 for serving not because of compute but because of HBM3 bandwidth.
• Why is quantization worth it? → A 4-bit model serves faster (decode is memory-bound, smaller tensors = faster reads) and fits on cheaper hardware. Quality loss is usually acceptable.

The mental model in one sentence: LLM inference is split between a compute-hungry prefill phase and a memory-hungry decode phase, connected by a KV cache that is your most critical resource to manage.

Everything else — PagedAttention, continuous batching, P/D disaggregation, speculative decoding, quantization — is an optimization layered on top of that fundamental structure.

The Summary: A Mental Model for Production

If you’ve made it this far, you’ve realized that LLM inference isn’t magic; it’s a measurable, optimizable systems engineering challenge. The execution pipeline breaks down into three distinct phases with unique bottlenecks: Tokenization (CPU-bound), Prefill (GPU compute-bound), and Decode (GPU memory-bound).

To help you debug that 2:00 AM latency spike, here is the final synthesis of the mechanics we’ve covered:

The “War Room” Reference Table

The Final Principles (Your TL;DR)

• Weights are Static Data: The GGUF or Safetensors file is essentially a massive, math-heavy lookup table. You aren’t executing code; you are performing matrix math against frozen numbers.
• Quantization is a Free Lunch: Lowering precision (e.g., to INT4) reduces tensor size, which directly loosens the memory bottleneck during decoding, often improving throughput by 20-40%.
• The KV Cache is Your Most Precious Resource: It prevents $O(N^2)$ recomputation by storing token state. Managing this via PagedAttention and Prefix Caching is what separates a toy demo from a production-grade service.
• Attention is the Relationship Engine: It uses Queries, Keys, and Values to calculate which tokens matter to each other. It’s why the model understands context, but it’s also why memory pressure scales with your prompt length.
• Continuous Batching is the Efficiency Unlock: By batching at the “word” level rather than the “request” level, you keep the GPU busy even when individual users have wildly different response lengths.

The ultimate constraint in scaling model serving isn’t raw compute power, but memory bandwidth and the strict management of the KV Cache. By understanding the mechanical realities beneath the hood—like PagedAttention, continuous batching, and quantization—infrastructure engineers can move past guesswork and systematically optimize for the metrics that dictate user experience: TTFT and TPOT.

Conclusion: The Field Is Moving, The Fundamentals Aren’t

The implementation details of LLM inference in 2025/2026 are changing fast enough to give you whiplash, but the underlying physics of the problem haven’t moved an inch. Prefill will always be compute-hungry. Decode will always be memory-hungry. Attention will always scale quadratically with context length unless someone breaks the math. These aren’t framework quirks; they are the mechanical realities of how transformers work.

Transitioning to LLMOps requires a fundamental shift in how we manage system state. We are no longer scaling stateless pods; we are actively managing distributed GPU memory. The engineering headroom to optimize this is enormous, and the landscape is shifting rapidly:

• The model size curve is bending: Smaller, highly-optimized 7B models are now punching above the weight of older 70B giants, distributing the inference problem across a much wider variety of hardware.

• The memory bottleneck is softening: Bleeding-edge KV cache compression techniques are reducing per-token memory footprint from kilobytes down to a handful of bytes, loosening the strict constraints of the decode phase.

• The edge is becoming viable: As inference pushes to mobile NPUs and WebGPU, serving a 3B parameter model starts to look less like a Kubernetes workload and more like a firmware binary.

Yet, the operational reality remains unchanged. When a model is in production and serving real traffic, someone has to know why TTFT spiked at 2am, why the KV cache hit 95% utilization under Tuesday’s load, and why the p99 ITL is three times the p50.

Mastering the mechanics of inference separates a fragile AI prototype from a resilient production platform. The inference stack will keep evolving. The need for a rigorous mental model to debug it won’t.

Next up: In the next post, we’ll take these first principles and see how they dictate the architectural trade-offs behind the major inference engines: Ollama, vLLM, TGI, and TensorRT-LLM.

I’m an infrastructure engineer with 11+ years in distributed systems (D-Wave, Enbala, MasterCard, Cisco), currently going deep on LLM serving and inference optimization. This series is grounded in hands-on experiments — Mac Mini to Lambda Labs GH200 to RunPod A100 clusters. I write what I actually learned, including the parts that didn’t work.

Find me on GitHub: kraghavan

Find me on Linkedin: Karthika Raghavan

Part 1 of 2: Design decisions, trade-offs, and algorithms behind schema-travels

Why I Built This

Migrating from a relational SQL database to a NoSQL paradigm is notoriously difficult to automate. Every DBA has war stories: the naive migration that turned a 3-table JOIN into three round trips, or the “just flatten everything” approach that created 50GB documents.

The problem I chose: How do you automate a context-aware SQL-to-NoSQL schema migration without relying on raw, hallucination-prone LLM outputs?

The key insight: algorithms are excellent at graph clustering; LLMs are not. But algorithms lack business context. So I built a dual-engine system where deterministic algorithms do the math, and an LLM acts as a reviewing Principal Architect—bounded, structured, and unable to hallucinate schemas into existence.

This post walks through every architectural decision in schema-travels, including the trade-offs I considered and the bugs that taught me humility.

System Overview

Architecture Diagram (generated by NotebookLM)

The Schema Travels architecture: multi-provider AI review with specialized MongoDB and DynamoDB migration flows

Components

┌──────────────────────────────────────────────────────────────────────────────┐
│                              schema-travels                                  │
│                                                                              │
│  ┌────────────────────────────────────────────────────────────────────────┐  │
│  │                        Deterministic Engine                            │  │
│  │                                                                        │  │
│  │  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐                 │  │
│  │  │  SQL Log    │    │  Access     │    │  Target     │                 │  │
│  │  │  Parser     │───▶│  Pattern    │───▶│  Schema     │                 │  │
│  │  │             │    │  Analyzer   │    │  Designer   │                 │  │
│  │  │ • PostgreSQL│    │             │    │             │                 │  │
│  │  │ • MySQL     │    │ • HotJoins  │    │ • MongoDB   │                 │  │
│  │  │ • 10K+ qps  │    │ • Mutations │    │ • DynamoDB  │                 │  │
│  │  └─────────────┘    │ • Co-access │    │ • Union-Find│                 │  │
│  │                     └─────────────┘    └──────┬──────┘                 │  │
│  └───────────────────────────────────────────────┼────────────────────────┘  │
│                                                  │                           │
│                                    Algorithmic Draft (JSON)                  │
│                                                  │                           │
│  ┌───────────────────────────────────────────────▼────────────────────────┐  │
│  │                         LLM Review Layer                               │  │
│  │                                                                        │  │
│  │  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐                 │  │
│  │  │  Provider   │    │  Advisor    │    │  Pydantic   │                 │  │
│  │  │  Protocol   │───▶│  (Reviewer) │───▶│  Validator  │                 │  │
│  │  │             │    │             │    │             │                 │  │
│  │  │ • Claude    │    │ • Critique  │    │ • Bounded   │                 │  │
│  │  │ • GPT-4o    │    │ • Refine    │    │ • Typed     │                 │  │
│  │  │ • Gemini    │    │ • Explain   │    │ • No schema │                 │  │
│  │  │ • Ollama    │    │             │    │   invention │                 │  │
│  │  └─────────────┘    └─────────────┘    └──────┬──────┘                 │  │
│  └───────────────────────────────────────────────┼────────────────────────┘  │
│                                                  │                           │
│                                    Reviewed Design (JSON)                    │
│                                                  │                           │
│  ┌───────────────────────────────────────────────▼────────────────────────┐  │
│  │                         Output & Caching                               │  │
│  │                                                                        │  │
│  │  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐                 │  │
│  │  │  Cache      │    │  Terraform  │    │  NoSQL      │                 │  │
│  │  │  Manager    │    │  Generator  │    │  Workbench  │                 │  │
│  │  │             │    │             │    │  Export     │                 │  │
│  │  │ • Strict    │    │ • .tf files │    │             │                 │  │
│  │  │ • Relaxed   │    │ • GSI defs  │    │ • Import    │                 │  │
│  │  │ • ~3s hits  │    │ • Capacity  │    │   ready     │                 │  │
│  │  └─────────────┘    └─────────────┘    └─────────────┘                 │  │
│  └────────────────────────────────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────────────────────────────┘

Component 1: The Statistical Ground Truth

The Problem

You can design a relational database in a vacuum using normal forms. You cannot design a NoSQL database without knowing the access patterns.

Feeding an LLM just the CREATE TABLE statements produces generic, unoptimized schemas. It’s like asking an architect to design a house without knowing if it’s for a family of four or a fraternity.

My Solution: Log-Based Pattern Analysis

Before any AI is invoked, the pipeline ingests thousands of raw SQL queries (up to 10K in my benchmarks). Two analyzers extract the structural signals:

The output is a weighted co-access matrix—essentially a graph where edge weights represent how often two tables appear together in queries.

Design Decision: Static Schema vs. Log Perusal

Trade-off considered:

My choice: Log perusal is non-negotiable. A schema where users and user_preferences are separate tables tells you nothing about whether they’re accessed together. Query logs tell you everything.

Lesson learned: Early versions didn’t weight by query frequency. A table JOINed once in a rare admin query got the same weight as one JOINed 10,000 times per hour. Adding frequency weighting dramatically improved recommendations.

Component 2: MongoDB Document Designer

The Problem

MongoDB’s core design question: Embed or Reference?

Getting this wrong is expensive. Embed a million reviews inside a product document and MongoDB will hate you. Reference user addresses that are always fetched with the user and you’ve recreated SQL’s JOIN problem.

My Solution: Confidence-Weighted Decisions

The algorithm takes the co-access score from Component 1 and calculates a Confidence Score for each relationship:

Confidence = (co_access_weight × access_factor) + (bounded_growth × stability_factor)

• ≥ 0.85: Strong embed recommendation (green in output)
• 0.70-0.85: Moderate confidence (yellow)
• < 0.70: Reference or needs human review (red)

The color-coding isn’t cosmetic—it tells developers where to focus their review time.

Design Decision: Schema Only vs. Schema + Queries

Trade-off considered:

My choice: Generate both. For MongoDB, schema-travels outputs the exact aggregation pipelines or findOne queries that replace the original SQL JOINs.

Why this matters: A schema migration isn’t done when you have a new data model. It’s done when your application code works. Giving developers the target schema and the code to query it cuts migration time significantly.

Component 3: DynamoDB Paradigm Shift

The Problem

DynamoDB isn’t just “MongoDB but AWS.” It’s a fundamentally different paradigm:

The infamous “Single-Table Design” pattern—where multiple entity types share one table with carefully crafted partition and sort keys—is powerful but alien to SQL developers.

My Solution: Union-Find Access Clustering

This is where the computer science degree earns its keep. The DynamoDBDesigner uses a Union-Find (Disjoint Set Union) algorithm to cluster SQL tables based on their relationship weights.

How it works:

1. Each SQL table starts as its own cluster
2. For each edge (co-access relationship) above a threshold weight:
   • Find the cluster roots of both tables
   • Union them if they’re frequently accessed together
3. Result: clusters of tables that should share a DynamoDB table

Input: users, orders, order_items, products, reviews
       
Co-access weights:
  users ←→ orders:      0.92  (high)
  orders ←→ order_items: 0.95  (high)
  products ←→ reviews:   0.45  (low)
  
After Union-Find:
  Cluster 1: {users, orders, order_items}  → Single-table candidate
  Cluster 2: {products}                     → Separate table
  Cluster 3: {reviews}                      → Separate table

The algorithm then generates PK/SK patterns and GSI candidates for each cluster.

Design Decision: LLM Generation vs. Algorithmic Draft

Trade-off considered:

My choice: Algorithms generate the draft; LLMs review it.

Union-Find is deterministic—same input always produces the same clusters. LLMs are not. Asking an LLM to “design a DynamoDB schema” is asking for creative writing. Asking it to “review this algorithmic draft and flag issues” is asking for structured critique.

Lesson learned: Early versions let the LLM suggest entity renames during review. This broke the mapping back to the algorithmic draft, causing cryptic “Entity Not Found” errors. Now the validator explicitly forbids schema invention.

Component 4: The Multi-Provider LLM Layer

The Problem

I needed LLM review capabilities but didn’t want to be locked into one provider. Different providers have different strengths, costs, and availability.

My Solution: Protocol-Oriented Provider Abstraction

The LLMProvider protocol defines what any provider must support:

Any class implementing this protocol works with the Advisor:

Design Decision: Agentic Generation vs. Agentic Review

Trade-off considered:

My choice: The LLM is a reviewer, not a creator.

The prompt never says “design a schema.” It says “here is the algorithmic draft—critique it.” The LLM can flag issues:

“The algorithm suggested Single-Table Design here, but this Partition Key has low cardinality and will cause a hot partition. Consider adding a write-sharding suffix.”

But it cannot invent new tables or fundamentally restructure the design. Pydantic validators enforce this boundary.

Component 5: Caching & Infrastructure Output

The Problem

LLM calls are slow (2-30 seconds) and expensive. Re-analyzing the same schema during CI/CD is wasteful.

My Solution: Dual-Mode Hashing

Relaxed mode hashes only the structural elements. Minor metadata changes hit the cache, dropping a 25-second pipeline to ~3 seconds.

Strict mode requires cryptographic match of the entire input. Essential for reproducible E2E testing.

Closing the Loop: Infrastructure as Code

A schema design on paper is useless if it can’t be deployed. The pipeline terminates by generating actual IaC:

The Terraform output includes table definitions, GSI configurations, and capacity settings. Not a template—actual runnable infrastructure.

Lesson learned: My Terraform formatter had a subtle bug—it generated $${table_name} instead of "${table_name}" in some edge cases. The E2E test matrix caught this because it validated the .tf files could be parsed by Terraform’s HCL parser.

What I’d Do Differently

1. DynamoDB Query Translation from Day One

The MongoDB module translates SQL JOINs to aggregation pipelines. The DynamoDB module only outputs schemas—no Boto3 query code. This asymmetry bothers me.

Generating dynamodb.query(KeyConditionExpression=...) calls alongside the schema would make the DynamoDB path as developer-friendly as MongoDB.

2. Stricter Entity Name Validation

The LLM occasionally renames entities during review (“I’ll call this CustomerOrders instead of user_orders”). This breaks the mapping back to the algorithmic draft.

The fix was a Pydantic validator that rejects any entity name not in the original input. But I should have anticipated this—LLMs love to be “helpful” by renaming things.

3. Cache Key Design Up Front

I added the dynamodb_mode parameter to cache keys late. This meant cached results from auto mode were incorrectly served for single mode requests.

Cache key design should happen during architecture, not debugging.

The Proof: What Actually Happened

Let me show you what this architecture produces in practice. I ran the E2E test matrix across 4 providers × 2 targets on a 42-table e-commerce schema with 100 synthetic queries:

┌─────────────┬──────────────┬──────────────┐
│  Provider   │   MongoDB    │   DynamoDB   │
├─────────────┼──────────────┼──────────────┤
│ claude      │ ✓ PASS       │ ✓ PASS       │
│ openai      │ ✓ PASS       │ ✓ PASS       │
│ gemini      │ ✓ PASS       │ ✓ PASS       │
│ ollama      │ ✓ PASS       │ ✓ PASS       │
└─────────────┴──────────────┴──────────────┘
Total: 8 tests | Passed: 8 | Failed: 0

What the numbers tell us:

Here’s the insight that makes this architecture work: the algorithmic clusters are identical across all providers. Products had 82 accesses. Users had 54. Orders had 32. The Union-Find algorithm doesn’t care which LLM you’re using—it produces the same deterministic foundation every time.

But the interpretation differs. Three cloud providers looked at the same clusters and said “single-table design with GSI overloading.” The local model (gemma3:4b running on my Mac Mini) said “multi-table is fine here.”

Just like every architecture review I’ve ever been in—except this one finished in 22 seconds and nobody rage-quit to “work from home.”

Who’s right? Honestly, both approaches are defensible for this workload. The point isn’t that one answer is correct—it’s that the AI is reviewing a solid algorithmic foundation, not hallucinating schemas from scratch.

MongoDB showed similar patterns:

The Claude review of MongoDB produced 12 recommendations with confidence-scored decisions:

• orders → order_items: EMBED (95% confidence) — “Perfect co-access, always accessed together”
• products → reviews: REFERENCE (90% confidence) — “Unbounded growth, popular products can have thousands of reviews”
• users → addresses: EMBED (85% confidence) — “High co-access, addresses typically accessed with user profile”

That 90% confidence REFERENCE decision for reviews? That’s exactly the kind of nuanced judgment that makes NoSQL design hard. The algorithm detected high co-access, but the AI correctly identified the unbounded growth risk that would blow past MongoDB’s 16MB document limit.

More than 40 tables migrated, zero DBAs traumatized. Though if your schema has more foreign keys than a hotel concierge, maybe grab coffee first.

The Bigger Picture: What This Actually Solves

The Old Way: A senior engineer spends 2-3 weeks analyzing query logs, drawing ER diagrams on whiteboards, debating embed-vs-reference decisions in meetings, and manually translating that into DynamoDB access patterns. Then they write Terraform by hand and pray they didn’t miss a GSI.

The New Way: Feed the tool your PostgreSQL logs and schema. Get a reviewed, confidence-scored design with deployable Terraform in 22 seconds. Spend those 2-3 weeks on the parts that actually require human judgment—data migration strategy, application refactoring, rollback planning.

This isn’t about replacing engineers. It’s about replacing the tedious parts of engineering so we can focus on the interesting parts.

The Core Innovation: Bounded AI

The fundamental insight behind this architecture is that LLMs should critique, not create.

By forcing the AI into a reviewer role with Pydantic-enforced boundaries, we get the benefits of LLM intelligence (business context, heuristic reasoning, natural language explanations) without the chaos of unbounded generation.

This pattern—algorithmic draft + bounded LLM review—is applicable far beyond schema migration. It’s how I’d approach any problem where you need AI assistance but can’t afford hallucinations.

What’s Next: GraphRAG for Enterprise Scale

The current implementation handles schemas with 10-50 tables comfortably. But what happens when you’re migrating a legacy enterprise system with 500 tables? 1,000?

At that scale, the co-access matrix becomes unwieldy, the visualization becomes spaghetti, and even Union-Find starts to strain under the combinatorial explosion.

The roadmap: GraphRAG-powered schema analysis.

The idea is straightforward:

1. Store the schema graph in a proper graph database (NetworkX + SQLite for now, Neo4j for enterprise)
2. Embed table semantics using sentence transformers
3. Query with natural language: “Show me all tables related to order fulfillment” or “Which clusters would be affected if we split the users table?”

This transforms schema-travels from a migration tool into a schema intelligence platform. Instead of processing everything at once, you can explore the graph, ask questions, and migrate incrementally—cluster by cluster, bounded by what your team can absorb.

Coming Up: Part 2

In Part 2, we’ll dive deeper into the E2E test results:

• Latency breakdown: 22 seconds for first run → 3 seconds cached
• Provider comparison: Why Claude, OpenAI, and Gemini agreed on single-table while Ollama chose multi-table
• The cache drift bug: How the test matrix caught an inconsistency in DynamoDB Terraform output
• Cost analysis: Running the full matrix cost less than $0.50 in API calls

Final Thoughts

I’ve been building distributed systems for over a decade. In that time, I’ve seen plenty of “AI-powered” tools that are really just prompt wrappers—impressive demos that fall apart the moment you need reproducibility.

schema-travels is my answer to the question: How do you build AI-assisted tooling that a principal engineer would actually sign off on?

The answer, it turns out, is the same principle that makes distributed systems reliable: don’t trust any single component. Algorithms provide the deterministic foundation. LLMs provide the intelligence. Pydantic provides the guardrails. Caching provides the economics. And an 8-test E2E matrix provides the confidence that it all actually works.

The schema migration problem was just the vehicle. The real artifact is an architecture pattern for building AI tools that are auditable, reproducible, and won’t make your on-call engineer cry at 3am.

GitHub: github.com/kraghavan/schema-travels

Questions, feedback, or war stories from your own migrations? Connect with me on LinkedIn.

Part 2 of 2: Empirical evaluation of classification accuracy, routing performance, and cost attribution

Abstract

In Part 1, I described the architecture of inference-sentinel, a privacy-aware LLM routing gateway. This post presents empirical results from five experiments evaluating classification accuracy, routing latency, cost efficiency, controller effectiveness, and session stickiness.

Key findings:

• The hybrid classifier achieves 97.5% accuracy with 0.16ms mean latency, though a systematic failure mode in Tier 3 detection reveals the importance of enabling NER for healthcare identifiers
• Local inference introduces a 10× latency penalty compared to cloud backends, a fundamental trade-off for privacy preservation
• Routing 47.5% of traffic locally yields 68.6% cost savings, with an unexpected finding that Google’s Gemini is 44× cheaper per request than Anthropic’s Claude
• The closed-loop controller correctly withheld recommendations when local-cloud quality divergence exceeded thresholds

1. Experimental Setup

1.1 Hardware Configuration

1.2 Dataset

I constructed a synthetic evaluation dataset of 200 prompts with known ground-truth privacy labels, balanced across four tiers:

The balanced design enables per-class precision/recall analysis without class imbalance confounds.

1.3 Experimental Protocol

Each experiment was run independently with the gateway in a fresh state. Metrics were collected via Prometheus and exported to JSON for analysis. All experiments used the same dataset to enable cross-experiment comparison.

2. Experiment 1: Classification Accuracy

Research Question: How accurately does the hybrid classifier assign privacy tiers, and what are the failure modes?

2.1 Results

Figure 1: Per-tier precision, recall, and F1 scores. The dashed line indicates the 95% threshold. Tier 3 recall (90%) falls below threshold due to undetected health insurance identifiers.

2.2 Per-Tier Analysis

The asymmetry between Tier 0 and Tier 3 is notable: Tier 0 has 5 false positives (items incorrectly classified as PUBLIC), while Tier 3 has 5 false negatives (restricted items missed). These are the same 5 samples — Tier 3 prompts misclassified as Tier 0.

Figure 2: Confusion matrix showing classification performance. Off-diagonal entries appear only in the Tier 3→Tier 0 cell, indicating a systematic failure mode rather than random errors.

2.3 Error Analysis

All five misclassifications share a common pattern:

Root Cause Analysis:

NER was enabled during this benchmark, yet the PERSON_NAME entities were not detected. The failures stem from two factors:

1. Missing regex pattern: The health insurance ID format ([A-Z]{3}\d{9}) is not covered by existing Tier 3 patterns, which target SSNs (\d{3}-\d{2}-\d{4}), credit cards (Luhn-valid sequences), and MRN patterns (MRN:\s*\d+).

2. NER model limitation: The HuggingFace Transformers BERT model (dslim/bert-base-NER) failed to recognize the person names in medical record context. The phrase structure “Patient [Name], health insurance ID…” appears to confuse the model — likely because “Patient” is parsed as part of the name span, or the surrounding medical terminology disrupts entity boundary detection.

Examining the detection results:

{
  "expected_entities": ["PERSON_NAME", "PERSON_NAME"],
  "detected_entities": []
}

The NER model returned zero entities despite clear person names being present. This is a known limitation of lightweight NER models on domain-specific text — medical, legal, and financial documents often require fine-tuned models.

Implication: The 10% false negative rate on Tier 3 represents exactly the failure mode that matters most in a privacy system — restricted data being classified as public. This is not acceptable for production deployment without remediation.

2.4 Remediation

Three complementary approaches:

# Fix 1: Add regex pattern for health insurance IDs
PATTERNS["health_insurance"] = r'\b(?:health\s*insurance\s*id|member\s*id)[:\s]*[A-Z]{2,4}\d{8,12}\b'

# Fix 2: Add "Patient [Name]" pattern for medical contexts
PATTERNS["patient_name"] = r'\bPatient\s+[A-Z][a-z]+\s+[A-Z][a-z]+\b'

# Fix 3: Consider larger NER model for production
# "accurate" mode uses Jean-Baptiste/roberta-large-ner-english

The regex-first approach is particularly important here: rather than relying solely on NER for entity detection, adding domain-specific patterns provides a deterministic safety net for known sensitive formats.

3. Experiment 2: Routing Performance

Research Question: What latency overhead does the gateway introduce, and how does local inference compare to cloud?

3.1 Results

Figure 3: End-to-end latency distribution showing heavy right tail. The p99 latency (62.9s) is 26× higher than p50 (2.4s), indicating high variance primarily from local inference.

3.2 Latency Decomposition

Key Finding: The gateway overhead (classification + routing) is 1.69ms — effectively invisible relative to inference time. The privacy-aware routing layer does not meaningfully impact end-to-end latency.

3.3 Latency by Route

The latency trade-off is stark: Local inference is approximately 10× slower than cloud. This is the fundamental cost of privacy preservation with consumer-grade hardware.

3.4 Error Analysis

All 17 failures returned HTTP 503: No healthy local backends available. Examining the error distribution:

Failures occurred exclusively on local-routed traffic. The root cause is memory pressure: the Mac Mini M4 with 16GB unified memory struggles to serve concurrent requests across two loaded models (gemma3:4b ≈ 3GB, mistral ≈ 4GB).

Mitigation strategies:

1. Reduce to single local model (eliminates round-robin memory contention)
2. Implement request queuing with backpressure
3. Upgrade to 32GB+ RAM for concurrent model serving
4. Increase health check timeout to tolerate transient memory pressure

4. Experiment 3: Cost Attribution

Research Question: What are the realized cost savings from local routing, and how do cloud backends compare?

4.1 Results

Figure 4: Cost comparison showing actual spend vs. hypothetical all-cloud routing. Local routing of Tier 2-3 traffic yields 68.6% cost reduction.

4.2 Routing Distribution

Figure 5: Request distribution by route. 42.6% of requests routed to local inference (privacy-sensitive), 57.4% to cloud (non-sensitive).

4.3 Backend Cost Analysis

Figure 6: Cost distribution by backend. Anthropic accounts for 97.8% of cloud spend despite handling only 50.5% of cloud requests.

Unexpected Finding: Gemini is 44× cheaper per request than Claude ($0.0000355 vs $0.00156). This suggests a potential optimization: use Gemini as the primary cloud backend for cost-sensitive workloads, reserving Claude for quality-critical requests.

4.4 Cost by Privacy Tier

Tier 2 and Tier 3 traffic incurs zero marginal cost after hardware investment. For organizations with significant sensitive data volumes, the ROI calculation favors local inference.

4.5 Projected Annual Savings

Extrapolating from observed cost ratios:

Caveat: These projections assume similar traffic distribution (47.5% local-eligible) and do not account for hardware depreciation, electricity, or operational overhead.

5. Experiment 4: Controller Effectiveness

Research Question: Does the closed-loop controller generate actionable routing recommendations?

5.1 Results

5.2 Shadow Mode Metrics

5.3 Analysis

The controller generated zero recommendations because the quality match rate was 0%. This means local model responses (gemma3:4b, mistral) were semantically dissimilar enough from cloud responses (Claude, Gemini) that they never crossed the similarity threshold (default: 85%).

This is informative, not a failure. The controller correctly identified that:

1. Local models produce qualitatively different outputs than cloud models
2. Promoting Tier 0-1 traffic from cloud to local would degrade response quality
3. The conservative default (keep on cloud) is appropriate

5.4 Interpretation

The 0% quality match rate reflects the capability gap between 4B-parameter local models and frontier cloud models. For tasks where approximate answers suffice, lowering the similarity threshold would generate recommendations:

controller:
  quality_threshold: 0.70  # Down from 0.85

However, this requires explicit acceptance of quality trade-offs — a decision the controller correctly defers to human operators.

6. Experiment 5: Session Stickiness

Research Question: Does the one-way trapdoor correctly lock sessions after PII detection?

6.1 Results

6.2 Analysis

Zero sessions were locked despite 30% of requests containing PII and session tracking being enabled. Examining the test methodology reveals the issue:

The benchmark harness sent requests with simulated IPs (10.0.0.0 through 10.0.0.19), but the session ID computation may not have properly differentiated these synthetic sources. Additionally, the PII detection failures identified in Experiment 1 (the 5 health insurance records) would have prevented those sessions from locking — if PII isn’t detected, the trapdoor isn’t triggered.

Contributing factors:

1. Classification dependency: Session locking requires Tier 2+ classification. The 10% Tier 3 false negative rate means some PII-containing requests were classified as Tier 0, preventing session locks.

2. Test methodology: Requests originated from the same physical host with simulated client IPs. The session ID hashing (SHA-256(client_ip + daily_salt)) should differentiate these, but the harness may need validation.

3. PII probability vs. detection: The 30% PII probability applies to dataset generation, but if those PII patterns aren’t detected by the classifier, sessions won’t lock.

6.3 What We Can Validate

Despite no sessions locking, the core privacy invariant held:

• Trapdoor violations: 0 — No request with detected PII ever routed to cloud
• Per-request classification: Functional — Requests that were classified as sensitive routed locally

The gap is between “contains PII” (ground truth) and “detected as PII” (classifier output).

6.4 Required Follow-Up

A proper session stickiness evaluation requires:

1. Fix classification first: Address the Tier 3 detection gaps so PII is actually detected
2. Validate session ID generation: Ensure synthetic client IPs produce distinct session IDs
3. Use diverse source IPs: Run from multiple actual hosts or containers
4. Add session state logging: Instrument the session manager to log state transitions

# Re-run with verified distinct sessions
python -m benchmarks.harness --experiment session --sessions 20 --verify-session-ids

Expected behavior after fixes:

• Sessions should lock when Tier 2+ PII is detected
• Subsequent requests in that session should route to local regardless of content
• Locked session count should approximate sessions × pii_probability × detection_rate

7. Discussion

7.1 Principal Findings

7.2 Limitations

Dataset Size: 200 prompts is sufficient for detecting large effect sizes but underpowered for rare failure modes. A production evaluation should use 10,000+ samples.

Synthetic Data: The evaluation dataset was synthetically generated with known patterns. Real-world PII distributions may differ, particularly for domain-specific identifiers.

Single Hardware Configuration: Results reflect a specific hardware setup (M4 Mac Mini, 16GB). Performance characteristics will vary with different local inference hardware.

NER Model Limitations: The lightweight BERT NER model (dslim/bert-base-NER, “fast” mode) failed to detect person names in medical record contexts, revealing domain-specific NER gaps that require either fine-tuning or larger models like RoBERTa.

Session Test Methodology: The session stickiness experiment used simulated IPs from a single host, and classification failures prevented some PII-containing requests from triggering session locks.

7.3 Threats to Validity

Internal Validity: The 17 routing failures (8.5%) due to memory pressure may have biased latency statistics toward successful (potentially faster) requests.

External Validity: The balanced tier distribution (25% per tier) does not reflect production traffic, which is typically skewed toward Tier 0-1.

Construct Validity: Semantic similarity (cosine distance on embeddings) may not capture task-specific quality dimensions relevant to specific use cases.

8. Conclusion

8.1 Summary of Contributions

This work presents inference-sentinel, a privacy-aware LLM routing gateway that addresses a gap in the current MLOps landscape: the ability to enforce data residency policies at inference time without sacrificing developer experience.

What we built:

What the benchmarks revealed:

The evaluation across 200 synthetic prompts demonstrated that privacy-aware routing is feasible with sub-2ms overhead. The 68.6% cost savings from local routing validates the economic case, while the 10× latency penalty quantifies the privacy-performance trade-off. The systematic Tier 3 failures (health insurance IDs) highlight the importance of domain-specific pattern engineering and robust NER model selection — even with NER enabled, lightweight models may miss entities in specialized contexts.

This is not a production-ready system — it is a proof of architecture demonstrating that the building blocks exist and compose correctly.

8.2 Future Work

8.2.1 Scaling the Evaluation

The current benchmark uses 200 prompts — sufficient for detecting large effects but underpowered for tail behavior analysis. Future work should include:

Statistical power analysis suggests n=1,000+ is required to detect failure modes occurring at <1% frequency with 95% confidence.

8.2.2 Improving NER Accuracy

The current implementation uses HuggingFace Transformers with dslim/bert-base-NER (“fast” mode, ~400MB) for named entity recognition. Several directions merit exploration:

The optimal choice depends on latency budget. For sub-50ms classification, the “fast” BERT model with expanded regex patterns may outperform larger models. For offline batch classification, RoBERTa-based NER (“accurate” mode) is viable.

8.2.3 GPU-Accelerated Local Inference

The current setup runs local models on Apple Silicon (M4 Mac Mini) using Metal acceleration via Ollama. While sufficient for development and low-throughput production, this architecture has limitations:

GPU deployment options:

1. NVIDIA GPU server (RTX 4090, A100): Run vLLM or TGI for high-throughput local inference
2. Cloud GPU instances (but local network): AWS/GCP instances in private VPC, data never leaves controlled infrastructure
3. Apple M4 Max/Ultra: 128GB unified memory enables 70B models with acceptable latency

The shadow mode quality metrics would likely improve significantly with larger local models, potentially enabling automatic traffic promotion.

8.2.4 Kubernetes Deployment

The current Docker Compose stack is suitable for single-node deployment. Production deployment requires:

┌─────────────────────────────────────────────────────────────┐
│                    Kubernetes Cluster                       │
├─────────────────────────────────────────────────────────────┤
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐          │
│  │  Sentinel   │  │  Sentinel   │  │  Sentinel   │          │
│  │  Pod (HPA)  │  │  Pod (HPA)  │  │  Pod (HPA)  │          │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘          │
│         └────────────────┼────────────────┘                 │
│                          ▼                                  │
│                 ┌─────────────────┐                         │
│                 │ Redis (Session  │                         │
│                 │    State)       │                         │
│                 └─────────────────┘                         │
│                          │                                  │
│         ┌────────────────┼────────────────┐                 │
│         ▼                ▼                ▼                 │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐          │
│  │   Ollama    │  │   Ollama    │  │    vLLM     │          │
│  │  (gemma3)   │  │  (mistral)  │  │  (llama3)   │          │
│  │   Node 1    │  │   Node 2    │  │  GPU Node   │          │
│  └─────────────┘  └─────────────┘  └─────────────┘          │
└─────────────────────────────────────────────────────────────┘

K8s-specific requirements:

• Horizontal Pod Autoscaler (HPA) for gateway pods based on request rate
• Node affinity for GPU-accelerated inference pods
• PodDisruptionBudget ensuring availability during rollouts
• NetworkPolicy restricting egress to approved cloud endpoints
• ServiceMesh (Istio/Linkerd) for mTLS between components

8.2.5 Session State: In-Memory vs. Persistent Storage

The current architecture stores session state in an in-memory data structure with TTL-based eviction. This is a deliberate design choice:

Why in-memory is defensible:

1. Session data is ephemeral by design — 15-minute TTL means losing state on restart is acceptable for most use cases
2. Security through ephemerality — no persistent store means no data to exfiltrate, no backups to secure, no encryption-at-rest requirements
3. Operational simplicity — no Redis cluster to manage, no connection pooling, no failover logic
4. Latency — hash table lookup is O(1) with ~0.001ms; Redis adds network round-trip

When to move to Redis:

• Multi-pod deployment requiring shared session state
• Session TTL >1 hour (memory pressure)
• Audit requirements mandating session history
• Graceful restart without session loss

For the Redis migration path, the interface is already abstracted:

class SessionStore(Protocol):
    async def get(self, session_id: str) -> Optional[Session]: ...
    async def set(self, session_id: str, session: Session) -> None: ...
    async def delete(self, session_id: str) -> None: ...

# Swap implementations without changing business logic
store = RedisSessionStore(redis_url) if USE_REDIS else InMemorySessionStore()

8.3 Target Applications

inference-sentinel addresses use cases across multiple organizational functions:

Healthcare & Life Sciences

Regulatory context: HIPAA requires technical safeguards for PHI. A gateway that provably routes PHI to local-only inference provides auditable compliance.

Financial Services

Regulatory context: PCI-DSS, SOX, and GLBA impose data handling requirements that a privacy-aware gateway can enforce at the infrastructure layer.

Legal & Professional Services

Enterprise IT & Security

Human Resources

8.4 Broader Impact

The proliferation of LLM-powered applications creates a tension between capability and privacy. Cloud-hosted models offer superior performance but require transmitting potentially sensitive data to third parties. Local models preserve privacy but sacrifice quality and increase operational burden.

inference-sentinel demonstrates that this is not a binary choice. By making routing decisions at inference time based on content classification, organizations can:

1. Use cloud models for the 50%+ of traffic that contains no sensitive data — capturing the quality and cost benefits
2. Enforce local-only inference for genuinely sensitive content — preserving privacy guarantees
3. Measure the trade-off empirically — shadow mode quantifies exactly what quality you sacrifice for privacy

This is a fundamentally different approach from “all cloud” or “all local” architectures. It treats privacy as a first-class routing dimension, alongside latency and cost.

8.5 Closing Thoughts

Building inference-sentinel during my job search taught me more about LLM infrastructure than any tutorial could. Debugging why Ollama returns 404 (model name mismatch), why Grafana dashboards show “Value” instead of model names (missing metric labels), why round-robin wasn’t working (YAML override precedence) — these are the unglamorous details that separate working systems from prototypes.

The code is open source. The architecture is documented. The benchmarks are reproducible.

If you’re building LLM applications that handle sensitive data, I hope this work provides a useful reference — or at least saves you from repeating my mistakes.

Appendix A: Raw Data

A.1 Classification Misclassifications

All 5 errors follow the pattern:

• Input: Health record with insurance ID format [A-Z]{3}\d{9}
• Expected entities: PERSON_NAME (not detected by BERT NER in medical context)
• Detected entities: []

A.2 Routing Errors

All 17 errors: HTTP 503: No healthy local backends available

Distribution: 8 Tier 2, 9 Tier 3 (local-routed traffic only)

A.3 Entity Detection Summary

GitHub: github.com/kraghavan/inference-sentinel

Questions, feedback, or collaboration ideas? Connect on LinkedIn.

Updated Last: March 24, 2026