Workspace

gRPC, deadlines and why load balancing surprises people

3 day streakD

gRPC, deadlines and why load balancing surprises people

gRPC is typed RPC over HTTP/2 streams. Learn its message framing and trailers, how deadlines and cancellation should flow through a call tree, and why one long connection defeats an L4 balancer.

DepthLesson 4 of 620 min

gRPC defines services in Protobuf (Module 7) and generates client and server code. Each call is one HTTP/2 stream: a POST to /package.Service/Method, request messages in DATA frames, response messages back, and the call's status in trailers, headers sent after the body.

Call typeShapeExample
UnaryOne request, one responseGetUser
Server streamingOne request, many responsesGenerate tokens
Client streamingMany requests, one responseUpload audio chunks, get a transcript
BidirectionalMany each way, interleavedLive transcription with partial results

Each message on the stream has a 5-byte prefix: one byte saying whether it is compressed, then a 4-byte big-endian length. The same length-prefixed framing you wrote in Module 4. Status arrives as grpc-status and grpc-message trailers, which is why browsers, whose fetch API long hid trailers, need a proxy or the Connect protocol.

Deadlines and cancellation

gRPC calls carry a deadline: an absolute time after which the result is useless. It travels in the grpc-timeout header, and a well-behaved server passes the remaining time on to every call it makes. When the deadline passes or the client cancels, the cancellation propagates down the whole tree, so no server keeps working on an answer nobody will read.

Worked exampleDeadline propagation through three services
Gateway receives a request with
a 2,000 ms budget
Gateway spends 50 ms, calls the planner with
1,950 ms remaining
Planner spends 700 ms, calls a tool with
1,250 ms remaining
Tool takes 1,400 ms
Deadline exceeded at 1,250 ms; the tool's work is cancelled

Without propagation, the tool would run to completion and its result would be thrown away. With it, everyone stops at the same moment.

The load balancing surprise

Figure 4.1

One connection, many calls

With an L4 balancer, all 12 calls on one HTTP/2 connection reach backend 1. With an L7 balancer, the calls are spread 4, 4 and 4.L4: balances connectionsL7: balances requestsClientL4balancerClientL7balancerBackend 112 callsBackend 20 callsBackend 30 callsBackend 14 callsBackend 24 callsBackend 34 callsOne long HTTP/2 connection carries every call; an L4 balancer picked its backend once.
An L4 balancer chooses a backend per TCP connection. gRPC opens one long HTTP/2 connection and sends every call over it, so every call lands on the same backend. An L7 balancer, or client-side balancing across several connections, spreads individual calls.

A layer 4 balancer routes TCP connections and never looks inside. A layer 7 balancer understands HTTP/2 and routes each request. With HTTP/1.1, many short connections made L4 balancing good enough. With gRPC's single long connection, an L4 balancer sends all of one client's traffic to one backend, and new backends added during a scale-up receive nothing until clients reconnect.

Go deeperConnect RPC and gRPC-Web

gRPC-Web is a variant browsers can speak through a translating proxy. Connect goes further: the same Protobuf service definitions, served over plain HTTP/1.1 or HTTP/2 POST requests with JSON or binary bodies, readable with curl, and with streaming via length-prefixed bodies. One server can speak gRPC, gRPC-Web and Connect at once.

The trend is that RPC frameworks are converging on ordinary HTTP semantics so that every proxy, CDN and debugging tool in the path keeps working.

deadline.tsTypeScript
// Propagate the remaining budget to every downstream call
async function handle(req: Request, deadline: number) {
  const remaining = () => Math.max(0, deadline - Date.now());
  const signal = AbortSignal.timeout(remaining());

  const plan = await fetch(PLANNER_URL, { method: "POST", body: req.body, signal,
    headers: { "x-deadline-ms": String(remaining()) } });
  // …each hop subtracts its own time and passes the rest on
}

Send an absolute deadline or a remaining budget, never a fixed per-hop timeout. Fixed timeouts add up to more than the caller is willing to wait.

Where you will meet it

Service meshes

Envoy-based meshes balance gRPC per request at L7 for exactly this reason.

Kubernetes

A plain Kubernetes Service balances connections, not requests. gRPC clients there use headless services and client-side balancing, or a mesh.

Model serving

Inference servers expose gRPC streaming endpoints, and gateways in front of them must balance per request to keep GPUs evenly busy.

Words to know

Trailers
Headers sent after the body, used by gRPC for status.
Deadline
An absolute time after which a call's result is no longer useful.
L4 / L7 balancer
Balancing by TCP connection versus by HTTP request.
Connect
An RPC protocol using Protobuf services over plain HTTP.

Check yourself

Answer out loud before you open each one.

Why does gRPC load balancing surprise people?

One long HTTP/2 connection carries many calls, so an L4 balancer cannot spread them.

Where does gRPC put a call's status?

In HTTP/2 trailers: grpc-status and grpc-message.

Why propagate deadlines rather than set fixed per-hop timeouts?

So every service stops when the original caller stops caring, and nested timeouts never exceed the caller's budget.

Try it

Use grpcurl -plaintext localhost:50051 list against any gRPC server with reflection, then call a method with -max-time 0.01 and read the deadline error.

A good place to start

Explore a pattern

A little guidance

Start with a pattern, read through a problem, and mark it solved when the idea clicks. Save anything you want to revisit.

New tutorials are added one at a time. Your bookmarks and solved checkmarks stay in this browser.

Explore the roadmap