The handshake, sequence numbers and retransmission
How TCP turns lossy packets into a reliable stream: numbered bytes, acknowledgements, timers and selective repair. Then the leftovers: TIME_WAIT and running out of ports.
TCP numbers every byte. Each side picks a random initial sequence number, and every segment says which byte number it starts at. The receiver replies with an acknowledgement number: the next byte it expects. From those two numbers, both sides always know exactly what has been delivered.
The three-way handshake
Before data flows, the two sides agree on starting sequence numbers. The client sends SYN with its number. The server replies SYN-ACK with its own number and an acknowledgement of the client's. The client sends ACK. That costs one full round trip before the first request byte can be sent.
A connection from SYN to FIN
The client wants to send an HTTP request. First it must agree on sequence numbers with the server.
Detecting loss
TCP learns about loss in two ways. Timeouts: if an acknowledgement does not arrive within the retransmission timeout, computed from measured round-trip times, the sender resends. Timeouts are slow, at least 200 ms on Linux and often more. Duplicate ACKs: when segment 3 is lost but 4, 5 and 6 arrive, the receiver keeps acknowledging “I still need 3”. Three duplicate ACKs trigger fast retransmit without waiting for the timer.
Selective acknowledgement, SACK, lets the receiver say exactly which later ranges it already has, so the sender resends only the holes. Without SACK, a burst of losses can force the sender to resend data the receiver already holds.
- Sent
- Segments 1–10, each 1,448 bytes
- Lost
- Segment 3
- Arrived
- 1, 2, 4, 5, 6, 7, 8, 9, 10
- Delivered to the app
- Only 1 and 2. Segments 4–10 wait in the kernel's buffer.
- After fast retransmit (about 1 RTT later)
- 3 arrives; 3–10 are delivered at once
This is TCP head-of-line blocking. It is invisible in a single stream, and it becomes a big deal when HTTP/2 puts many streams on one connection (Module 5).
Closing, and TIME_WAIT
Each side closes its direction with FIN. The side that closes first enters TIME_WAIT and keeps the connection's 4-tuple reserved for twice the maximum segment lifetime, 60 seconds on Linux. This stops a delayed packet from an old connection being mistaken for part of a new one with the same ports.
- Default Linux ephemeral range
- 32768–60999, so 28,232 ports
- Each closed connection holds its port in TIME_WAIT for
- 60 seconds
- Sustainable new connections to one destination IP and port
- 28,232 ÷ 60 ≈ 470 per second
- A client making 1,000 fresh connections per second to one API
- Exhausts the range in under 30 seconds
Symptom: EADDRNOTAVAIL or connect timeouts under load. Fix: reuse connections with keep-alive pools rather than tuning kernel flags.
Go deeperSYN floods, SYN cookies and TCP Fast Open
A server must remember each half-open connection between SYN and the final ACK. An attacker sending millions of SYNs from spoofed addresses can fill that queue. SYN cookies let the server encode the needed state into its own initial sequence number and keep nothing until the final ACK proves the client is real.
TCP Fast Open lets a returning client put data in its SYN, saving a round trip. It saw limited deployment because middleboxes dropped or mangled SYNs carrying data, an example of ossification: a protocol becoming impossible to change because devices in the path assume its current shape. QUIC's encrypted headers were designed to avoid that fate.
# Connection states on this machine, counted
ss -tan | awk 'NR>1 {print $1}' | sort | uniq -c
# Per-connection internals: RTT, congestion window, retransmissions
ss -ti dst api.example.com
# Watch a handshake and close on the wire
sudo tcpdump -ni any 'host example.com and tcp port 443' -c 20ss -ti shows rtt, cwnd and retrans per connection, which turns guesses about a slow connection into numbers.
Where you will meet it
HTTP clients such as undici keep connections alive and reuse them, avoiding both handshake latency and TIME_WAIT buildup.
Naive load tests that open a new connection per request often measure their own port exhaustion instead of the server.
Wireshark marks retransmissions and duplicate ACKs in colour, which is often the fastest way to prove network loss.
Words to know
- SYN / ACK / FIN
- TCP flags: synchronise sequence numbers, acknowledge data, and finish a direction.
- Retransmission timeout
- How long the sender waits for an ACK before resending, derived from measured RTT.
- SACK
- Selective acknowledgement: reporting which later byte ranges have already arrived.
- TIME_WAIT
- The state that keeps a closed connection's 4-tuple reserved for a while.
Check yourself
Answer out loud before you open each one.
What does an ACK number mean?
The next byte the receiver expects. Everything before it has arrived.
Why does fast retransmit need three duplicate ACKs?
Duplicate ACKs can also come from reordering. Three is a heuristic that loss, not reordering, is likely.
Why does TIME_WAIT exist, and when does it hurt?
To stop old delayed packets confusing a new connection on the same 4-tuple. It hurts clients that open many short connections to one destination.
Try it
Capture curl -s https://example.com > /dev/null with tcpdump and open it in Wireshark. Find the SYN, SYN-ACK, ACK, the TLS handshake and the FINs. Note the relative sequence numbers.