Tuning for a Million Concurrent Connections

In part 7, Relay learned to run on many nodes behind a load balancer, with broadcasts flowing through Redis. That solves the architecture. It does not solve the fact that a fresh Linux box with default settings will refuse connection number 1,025, and a Node process with default Socket.IO options wastes memory and CPU on every idle socket. A chat server spends most of its life holding connections that do nothing, so the cost of doing nothing is what we optimise here.

Measure memory per connection first

Every capacity number in this article depends on how much memory one idle connection costs in your process, with your auth data on socket.data, your rooms and your middleware. Published numbers vary wildly because they measure different things. As a ballpark only, expect a few kilobytes of kernel memory per idle TCP socket, roughly 10 to 20 KB of heap for a bare ws connection, and more for a Socket.IO connection once handshake data, rooms and user objects are attached. Measure instead of trusting any of these, including mine:

import { readFile } from 'node:fs/promises';
import { io } from 'socket.io-client';

const N = Number(process.env.N ?? 5_000);
const URL = process.env.URL ?? 'ws://localhost:3000';

const tokens: string[] = JSON.parse(await readFile('tokens.json', 'utf8'));
for (let i = 0; i < N; i++) {
  io(URL, { transports: ['websocket'], auth: { token: tokens[i % tokens.length] } });
  if (i % 500 === 0) await new Promise((r) => setTimeout(r, 200)); // don't stampede
}
console.log(`opened ${N}; now read the server's RSS and heap`);

setInterval(() => {
  const { rss, heapUsed } = process.memoryUsage();
  const n = io.engine.clientsCount;
  console.log({
    sockets: n,
    rssMB: Math.round(rss / 1e6),
    heapPerSocketKB: n ? Math.round(heapUsed / n / 1024) : 0,
  });
}, 10_000);

Record the baseline with zero sockets, open 5,000, 20,000 and 50,000, and plot the difference. The slope is your real per-connection cost. Run it again after every change in this article and after you add anything to socket.data, because a user object with a big permissions array can quietly double it.

File descriptors

Every socket is a file descriptor, and most distributions give a process a soft limit of 1,024. Once you hit it, accept() fails with EMFILE and new users simply cannot connect, while existing ones work fine, which makes it a confusing outage. Raise the limit where the process is started, not in your shell:

# What the running process actually got
cat /proc/$(pgrep -f "node dist/index.js" | head -1)/limits | grep "open files"

# systemd unit: /etc/systemd/system/relay.service
[Service]
LimitNOFILE=1048576

# Docker
docker run --ulimit nofile=1048576:1048576 relay-server

# Then reload
sudo systemctl daemon-reload && sudo systemctl restart relay

Kernel parameters

The per-process limit sits under system-wide limits and network queue sizes. These are the ones that matter for a connection-heavy server. Treat the values as a starting point, apply them, and verify under load:

# System-wide and per-process ceilings for open files
fs.file-max = 2097152
fs.nr_open = 2097152

# Accept queue: how many completed handshakes can wait for accept()
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535

# Outbound connections (proxies, load generators): widen the ephemeral range
net.ipv4.ip_local_port_range = 1024 65535

# Many idle sockets: keep default per-socket buffers small, let busy ones grow
net.ipv4.tcp_rmem = 4096 16384 4194304
net.ipv4.tcp_wmem = 4096 16384 4194304

# Total TCP memory in pages (min, pressure, max); size to your instance RAM
net.ipv4.tcp_mem = 786432 1048576 1572864

# apply: sudo sysctl --system

somaxconn only helps if the application asks for a large backlog too. Node defaults to 511, so pass it explicitly with httpServer.listen({ port, backlog: 65535 }). A reconnect storm of 50,000 clients arriving within seconds is exactly when the accept queue overflows and the kernel starts silently dropping SYNs.

Ephemeral port exhaustion

A TCP connection is identified by its 4-tuple: source IP, source port, destination IP, destination port. A server listening on one port can hold millions of connections because each client brings a different source IP and port. The problem appears on the other side of any hop where one machine opens many connections to one destination: a proxy connecting to an upstream, or a load generator connecting to your server.

If Nginx proxies every WebSocket to 10.0.1.11:3000, each of those upstream connections needs a unique source port from Nginx's single IP, and there are only about 64,000 of them. At connection 64,001, Nginx fails with Cannot assign requested address. The ways out all add variety to the 4-tuple:

• —More upstream ports. Run several Node processes per machine, each on its own port (3000, 3001, 3002...), and list each as a separate upstream server. Every destination port gets its own pool of 64k source ports.
• —More source IPs. Give the proxy several private IPs and spread connections across them with proxy_bind.
• —Fewer hops. An ALB or NLB routing straight to the Node processes avoids a self-managed proxy tier entirely.
• —Load generators hit the same wall, so plan on multiple machines or multiple source IPs when testing, which we do in part 9.

Compression: off unless you prove otherwise

The WebSocket permessage-deflate extension compresses each frame with zlib. Chat messages are small, so the savings are modest, while every compressed connection can hold its own zlib context costing tens to hundreds of kilobytes, and compression burns CPU on every send. Socket.IO has shipped with perMessageDeflate disabled by default since version 3. Keep it that way, and if you enable it for a bandwidth-sensitive mobile audience, set a threshold so only large frames are compressed and measure memory again.

export const io = new Server(httpServer, {
  transports: ['websocket'],
  perMessageDeflate: false,   // default since v3; stated for clarity
  pingInterval: 45_000,       // fewer pings per second across the fleet
  pingTimeout: 10_000,        // 45s + 10s stays below a 60s ALB idle timeout
  maxHttpBufferSize: 16_384,  // chat payloads are small; reject anything bigger
  connectionStateRecovery: { maxDisconnectionDuration: 2 * 60_000 },
});

The cost of keeping connections alive

Heartbeats are how both ends discover a dead connection, but they are not free. One million sockets pinged every 25 seconds is 40,000 pings and 40,000 pongs per second across the fleet, before anyone says a word. Raising pingInterval to 45 seconds cuts that almost in half. The constraint is every idle timeout on the path: pingInterval + pingTimeout must stay under the load balancer's idle timeout, or it will close quiet connections before the server even notices. Heartbeat intervals for presence from part 6 should also stay independent and relaxed, since a Redis write every 30 seconds per socket is its own form of load.

Faster WebSocket internals

Socket.IO uses the ws library underneath. ws has an optional native addon, bufferutil, which speeds up masking and unmasking of frame payloads; install it as an optional dependency and ws picks it up automatically. (utf-8-validate is no longer needed on current Node versions, which validate UTF-8 natively.)

npm install --save-optional bufferutil

For a bigger jump, Socket.IO can run on top of uWebSockets.js, a C++ HTTP and WebSocket server with a much smaller memory footprint per connection. It is installed from its GitHub releases rather than the npm registry, and it replaces the Node HTTP server, so any Express routes need to move or live on a separate port. The Socket.IO side is a small change:

import { App } from 'uWebSockets.js';
import { Server } from 'socket.io';

const app = App();
const io = new Server({ transports: ['websocket'] });
io.attachApp(app);

// Adapter, middleware and handlers stay exactly the same.
io.adapter(createAdapter(pubClient, subClient));

app.listen(Number(process.env.PORT ?? 3000), (token) => {
  if (!token) throw new Error('Port already in use');
});

Use every core

A Node process runs your JavaScript on one core. On an 8-core machine, a single process leaves most of the hardware idle while its event loop saturates. Run one process per core. Because each process already talks to the Redis adapter from part 7, there is nothing special about processes on the same machine: they are just more nodes. Giving each its own port (3000 to 3007) and listing them as separate upstreams also solves the proxy port problem above. With WebSocket-only transport, you do not need the Node cluster module's sticky helpers at all.

Surviving reconnect storms

The most dangerous moment for a chat backend is not peak traffic. It is the ten seconds after a node dies or a deploy restarts a process, when every one of its 50,000 clients reconnects at once. Each reconnect runs the TLS handshake, JWT verification, room joins from the database and a catch-up query from part 5. Multiply by 50,000 and the surviving nodes can be knocked over too, which creates the next storm.

The client-side defence is jittered exponential backoff. Socket.IO's client does this out of the box; tune it so clients spread out rather than retrying in lockstep:

const socket = io(API_URL, {
  transports: ['websocket'],
  reconnectionDelay: 1_000,     // first retry around 1s
  reconnectionDelayMax: 30_000, // cap: default is 5s, too aggressive at scale
  randomizationFactor: 0.5,     // each delay randomised by +/- 50%
  auth: (cb) => cb({ token: getAccessToken() }),
});

The server-side defence is admission control. Cap how many connections a node will accept per second, and bound the expensive work that follows each one. A rejected handshake returns a connect_error, and the client backs off and tries again, possibly landing on a less busy node:

const MAX_NEW_PER_SECOND = 2_000;
let admitted = 0;
setInterval(() => (admitted = 0), 1_000);

io.use((socket, next) => {
  if (++admitted > MAX_NEW_PER_SECOND) return next(new Error('server_busy'));
  next();
});

// Catch-up after a reconnect is GET /conversations/:id/messages?after=<id> (part 5).
// Allow at most 200 of those queries in flight per process; the rest wait their turn.
import pLimit from 'p-limit';
const catchUpLimit = pLimit(200);

export function catchUp(conversationId: string, afterId: string) {
  return catchUpLimit(() => since(conversationId, afterId));
}

Capacity planning, per node

Put the measurements together into a plan you can defend. Here is the shape of it with illustrative numbers: suppose your memory test showed about 40 KB per connection all-in, and load testing (part 9) showed one process stays healthy at 50,000 sockets with p99 message latency under 200 ms and event-loop lag under 20 ms.

The headroom is not optional. When one machine with eight processes dies, 400,000 clients reconnect to the rest. If the survivors were running at 95%, they are now over capacity in the middle of a storm. Plan so that losing your largest failure domain still leaves the fleet under about 70% of its tested limit.

Key takeaways

• —Measure memory per connection in your own process, with your real socket.data, before planning anything.
• —Raise file descriptor limits in systemd or Docker, and tune accept queues and TCP memory for many idle sockets.
• —A 4-tuple needs variety: add upstream ports or source IPs wherever one machine opens many connections to one destination.
• —Keep compression off, lengthen ping intervals within every idle timeout, and run one process per core.
• —Jittered backoff on clients plus admission control on servers turns a reconnect storm into a rolling wave.

Every number in this part came from a measurement you still have to run. Next up, part 9: Load Testing WebSockets with k6, where we speak the Socket.IO protocol directly and push Relay until it tells us its real limits.