gRPC, deadlines and why load balancing surprises people
gRPC is typed RPC over HTTP/2 streams. Learn its message framing and trailers, how deadlines and cancellation should flow through a call tree, and why one long connection defeats an L4 balancer.
gRPC defines services in Protobuf (Module 7) and generates client and server code. Each call is one HTTP/2 stream: a POST to /package.Service/Method, request messages in DATA frames, response messages back, and the call's status in trailers, headers sent after the body.
| Call type | Shape | Example |
|---|---|---|
| Unary | One request, one response | GetUser |
| Server streaming | One request, many responses | Generate tokens |
| Client streaming | Many requests, one response | Upload audio chunks, get a transcript |
| Bidirectional | Many each way, interleaved | Live transcription with partial results |
Each message on the stream has a 5-byte prefix: one byte saying whether it is compressed, then a 4-byte big-endian length. The same length-prefixed framing you wrote in Module 4. Status arrives as grpc-status and grpc-message trailers, which is why browsers, whose fetch API long hid trailers, need a proxy or the Connect protocol.
Deadlines and cancellation
gRPC calls carry a deadline: an absolute time after which the result is useless. It travels in the grpc-timeout header, and a well-behaved server passes the remaining time on to every call it makes. When the deadline passes or the client cancels, the cancellation propagates down the whole tree, so no server keeps working on an answer nobody will read.
- Gateway receives a request with
- a 2,000 ms budget
- Gateway spends 50 ms, calls the planner with
- 1,950 ms remaining
- Planner spends 700 ms, calls a tool with
- 1,250 ms remaining
- Tool takes 1,400 ms
- Deadline exceeded at 1,250 ms; the tool's work is cancelled
Without propagation, the tool would run to completion and its result would be thrown away. With it, everyone stops at the same moment.
The load balancing surprise
One connection, many calls
A layer 4 balancer routes TCP connections and never looks inside. A layer 7 balancer understands HTTP/2 and routes each request. With HTTP/1.1, many short connections made L4 balancing good enough. With gRPC's single long connection, an L4 balancer sends all of one client's traffic to one backend, and new backends added during a scale-up receive nothing until clients reconnect.
Go deeperConnect RPC and gRPC-Web
gRPC-Web is a variant browsers can speak through a translating proxy. Connect goes further: the same Protobuf service definitions, served over plain HTTP/1.1 or HTTP/2 POST requests with JSON or binary bodies, readable with curl, and with streaming via length-prefixed bodies. One server can speak gRPC, gRPC-Web and Connect at once.
The trend is that RPC frameworks are converging on ordinary HTTP semantics so that every proxy, CDN and debugging tool in the path keeps working.
// Propagate the remaining budget to every downstream call
async function handle(req: Request, deadline: number) {
const remaining = () => Math.max(0, deadline - Date.now());
const signal = AbortSignal.timeout(remaining());
const plan = await fetch(PLANNER_URL, { method: "POST", body: req.body, signal,
headers: { "x-deadline-ms": String(remaining()) } });
// …each hop subtracts its own time and passes the rest on
}Send an absolute deadline or a remaining budget, never a fixed per-hop timeout. Fixed timeouts add up to more than the caller is willing to wait.
Where you will meet it
Envoy-based meshes balance gRPC per request at L7 for exactly this reason.
A plain Kubernetes Service balances connections, not requests. gRPC clients there use headless services and client-side balancing, or a mesh.
Inference servers expose gRPC streaming endpoints, and gateways in front of them must balance per request to keep GPUs evenly busy.
Words to know
- Trailers
- Headers sent after the body, used by gRPC for status.
- Deadline
- An absolute time after which a call's result is no longer useful.
- L4 / L7 balancer
- Balancing by TCP connection versus by HTTP request.
- Connect
- An RPC protocol using Protobuf services over plain HTTP.
Check yourself
Answer out loud before you open each one.
Why does gRPC load balancing surprise people?
One long HTTP/2 connection carries many calls, so an L4 balancer cannot spread them.
Where does gRPC put a call's status?
In HTTP/2 trailers: grpc-status and grpc-message.
Why propagate deadlines rather than set fixed per-hop timeouts?
So every service stops when the original caller stops caring, and nested timeouts never exceed the caller's budget.
Try it
Use grpcurl -plaintext localhost:50051 list against any gRPC server with reflection, then call a method with -max-time 0.01 and read the deadline error.