Readiness, completion and io_uring
How one thread waits on 10,000 sockets. Learn thread-per-connection, epoll's readiness model and io_uring's completion model, and why io_uring is often disabled.
The oldest server design gives each connection its own thread that blocks in read. It is simple and works well for hundreds of connections. At 10,000 mostly idle connections, it means 10,000 stacks and a scheduler juggling 10,000 threads that are almost all asleep.
Three ways to wait for I/O
Thread per connection
- T1
read(fd 7)asleep - T2
read(fd 8)asleep - T3
read(fd 9)asleep - T6
read(fd 12)asleep - T9
read(fd 15)asleep - … 10,000 threads, 10,000 stacks
Simple code. Memory and scheduling cost grow with every idle connection.
Readiness · epoll, kqueue
epoll_wait()one thread sleeps on all sockets→ fd 7, fd 12 readykernel reports what can be read nowread(7) · read(12)your code does the reads↻ back to epoll_waitloop
One thread holds thousands of idle connections for a few KB each.
Completion · io_uring
read fd 7 → buf Aread fd 12 → buf Bwrite fd 3 ← buf Cfd 7: 512 bytes in buf Afd 3: 4096 bytes writtenBatches many operations per system call, and handles files too.
Readiness: select, poll, epoll, kqueue
select and poll take the full list of descriptors on every call and scan it, which is O(n) per wait. epoll on Linux and kqueue on BSD and macOS keep the interest list in the kernel and return only descriptors that are ready. The program then calls read itself. libuv, Go's netpoller, Tokio and nginx all sit on these.
Level-triggered notification repeats as long as data remains. Edge-triggered notification fires once when the state changes, so you must read until EAGAIN or you will never hear about the leftover bytes.
Completion: io_uring
io_uring shares two ring buffers between your process and the kernel. You place requests such as read this socket into this buffer in the submission queue. The kernel performs them and posts results in the completion queue. Many operations can go in with one system call or none, and it works for files as well as sockets, which epoll never handled well.
| Model | Answer to “what now?” | Who does the read | Examples |
|---|---|---|---|
| Blocking | Block until done | Your thread, sleeping | Thread-per-connection servers |
| Readiness | This descriptor can be read now | You, after the notification | epoll, kqueue, libuv, Tokio |
| Completion | This read is done; here is the result | The kernel | io_uring, Windows IOCP |
int ep = epoll_create1(0);
struct epoll_event ev = { .events = EPOLLIN, .data.fd = listen_fd };
epoll_ctl(ep, EPOLL_CTL_ADD, listen_fd, &ev);
for (;;) {
struct epoll_event ready[64];
int n = epoll_wait(ep, ready, 64, -1); // sleep until something is ready
for (int i = 0; i < n; i++) {
int fd = ready[i].data.fd;
if (fd == listen_fd) add_client(ep, accept4(listen_fd, NULL, NULL, SOCK_NONBLOCK));
else handle_readable(fd); // read until EAGAIN
}
}This is the heart of every event loop. libuv wraps it with timers, a thread pool and cross-platform backends.
Where you will meet it
A few worker processes, each with an epoll loop, serve hundreds of thousands of connections.
SSE and WebSocket connections are mostly idle, so readiness-based servers hold them for kilobytes each instead of a thread each.
Databases such as TigerBeetle and ScyllaDB can use io_uring for disk I/O on hosts they control, where the sandbox concern does not apply.
Check yourself
Answer out loud before you open each one.
What is the difference between readiness and completion I/O?
epoll says you can act now and you do the read. io_uring says the read is done and the data is in your buffer.
Why does edge-triggered epoll require reading until EAGAIN?
It notifies only on a change. Leftover data does not cause a new notification.
Why might a sandbox block io_uring?
seccomp cannot inspect operations submitted through the rings, so io_uring can perform actions the filter meant to forbid.
Try it
Build a TCP echo server three ways: thread per connection, epoll, and io_uring with liburing. Hold 10,000 idle connections against each and compare resident memory.