Production Hardening: Backpressure, Deploys and Push

In part 9, we load tested Relay and found its real limits. A system that performs well in a load test can still fall over in production, because production has clients that misbehave, deploys that restart servers under live traffic, and users who close the app and still expect to hear about new messages. This final part closes those gaps, then steps back to look at the whole architecture we built across the series.

Rate limit every socket

A WebSocket is a firehose pointed at your server. Once connected, a client can send thousands of events a second, and HTTP rate limiting at the load balancer never sees them. Limits have to live inside the connection. A token bucket per socket is cheap, in memory, and allows natural bursts while capping sustained rates:

export function tokenBucket(capacity: number, refillPerSecond: number) {
  let tokens = capacity;
  let last = Date.now();
  return function take(): boolean {
    const now = Date.now();
    tokens = Math.min(capacity, tokens + ((now - last) / 1000) * refillPerSecond);
    last = now;
    if (tokens < 1) return false;
    tokens -= 1;
    return true;
  };
}

io.on('connection', (socket) => {
  const canSend = tokenBucket(10, 2);   // burst of 10, then 2 messages/s
  const canType = tokenBucket(5, 1);
  let strikes = 0;

  socket.use(([event], next) => {
    if (event.startsWith('typing:') && !canType()) return; // silently drop
    next();
  });

  socket.on('message:send', async (input, ack) => {
    if (!canSend()) {
      if (++strikes > 50) return socket.disconnect(true); // clearly abusive
      return ack({ ok: false, error: 'rate_limited', retryAfterMs: 1_000 });
    }
    // ...validate, persist and broadcast as in parts 4 and 5
  });
});

Returning the limit through the ack matters: the client outbox from part 5 keeps the message and retries after retryAfterMs, so a legitimate burst is delayed rather than lost. Per-socket limits do not stop one account opening fifty tabs, so add a per-user limit in Redis as well, using the same fixed-window or sliding-window counters as in the Scaling series' part on rate limiting.

Size limits come next. Socket.IO's maxHttpBufferSize defaults to 1 MB, far more than any chat message needs. Set it to something like 16 KB (as we did in part 8) so an oversized frame is rejected before it is parsed, and validate every payload's shape and body length in the handler itself. Attachments never travel over the socket; they go to object storage directly.

Backpressure and slow consumers

Rate limits protect you from clients that send too much. Backpressure protects you from clients that receive too slowly. A phone on a weak connection in a busy group cannot read frames as fast as the server writes them, so the unsent data accumulates in the server's memory. A few hundred of those in a large room and a node's memory climbs until the process is killed, taking every healthy connection with it.

• —Ephemeral events are volatile. Typing and presence updates use volatile, so Socket.IO drops them instead of buffering when the transport is not writable.
• —Watch the buffer. The underlying ws socket exposes bufferedAmount, the bytes queued but not yet sent. Reaching it goes through Socket.IO internals (socket.conn.transport), so wrap it in one helper and pin your Socket.IO version.
• —Cut off hopeless connections. If a socket keeps more than about 1 MB buffered for several seconds, disconnect it.

Disconnecting a user sounds harsh, but everything we built makes it safe. Messages are durable in Postgres, the client reconnects with backoff, and catch-up by lastSeenId from part 5 delivers everything it missed in a single paginated query, which is far cheaper than keeping megabytes of frames in memory.

const MAX_BUFFERED = 1_000_000;

// Internal API: isolated here so a Socket.IO upgrade touches one function.
function bufferedBytes(socket: Socket): number {
  const transport = socket.conn.transport as unknown as { socket?: { bufferedAmount?: number } };
  return transport.socket?.bufferedAmount ?? 0;
}

setInterval(() => {
  for (const socket of io.of('/').sockets.values()) {
    if (bufferedBytes(socket) > MAX_BUFFERED) {
      metrics.slowConsumerDisconnects.inc();
      socket.disconnect(true);
    }
  }
}, 5_000);

Deploying without dropping users

HTTP servers deploy gracefully by finishing in-flight requests. A WebSocket server has no in-flight requests to finish: its connections would last forever. Draining has to be active. On SIGTERM, a Relay node stops being a target for new connections, asks its clients to move in batches, and exits once they are gone or the deadline passes:

let ready = true;
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));

process.on('SIGTERM', async () => {
  ready = false;                 // readiness probe fails: LB stops sending new sockets
  await sleep(10_000);           // give the load balancer time to notice

  const sockets = [...io.of('/').sockets.values()];
  const BATCH = 1_000;
  for (let i = 0; i < sockets.length; i += BATCH) {
    for (const s of sockets.slice(i, i + BATCH)) s.emit('server:draining', { withinMs: 10_000 });
    await sleep(1_000);          // spread the reconnects over time
  }

  await sleep(15_000);           // clients reconnect elsewhere at their own jittered pace
  io.close();                    // closes any stragglers and the HTTP server
  await Promise.allSettled([pubClient.quit(), subClient.quit(), pool.end()]);
  process.exit(0);
});

socket.on('server:draining', ({ withinMs }) => {
  setTimeout(() => {
    socket.disconnect();
    socket.connect(); // lands on a healthy node; catch-up fills any gap
  }, Math.random() * withinMs);
});

Why not just call socket.disconnect(true) on the server? Because a server-initiated disconnect reports the reason io server disconnect, and the Socket.IO client deliberately does not reconnect automatically in that case, since it usually means the user was kicked. Letting the client disconnect itself, at a random moment inside the window, keeps reconnection under our control and turns a 50,000-socket restart into a gentle wave.

• —Load balancer: set the ALB target group deregistration delay to cover the whole drain window, around 60 seconds for the script above.
• —Orchestrator: give the container enough time to finish, with ECS stopTimeout or Kubernetes terminationGracePeriodSeconds longer than the drain.
• —Rollouts: replace a small percentage of nodes at a time, so the fleet absorbs each wave before the next one starts.

Health checks that mean something

// Liveness: is the process responsive at all? Restart it if not.
app.get('/healthz', (_req, res) => res.sendStatus(200));

// Readiness: should the load balancer send new connections here?
app.get('/readyz', (_req, res) => {
  const ok = ready && pubClient.status === 'ready' && io.engine.clientsCount < MAX_SOCKETS;
  res.sendStatus(ok ? 200 : 503);
});

Readiness fails during a drain, when Redis is unreachable (a node without the adapter would silently deliver to only its own sockets), and when the node is already at the socket limit you measured in part 9. Liveness stays trivially simple, so a slow dependency never triggers a restart loop.

Metrics that matter for chat

import client from 'prom-client';

client.collectDefaultMetrics(); // includes nodejs_eventloop_lag_seconds and heap metrics

export const metrics = {
  connected: new client.Gauge({
    name: 'relay_connected_sockets',
    help: 'Sockets connected to this process',
    collect() {
      this.set(io.engine.clientsCount);
    },
  }),
  messages: new client.Counter({
    name: 'relay_messages_total',
    help: 'message:send events by outcome',
    labelNames: ['result'] as const, // ok | duplicate | rate_limited | invalid | error
  }),
  ackLatency: new client.Histogram({
    name: 'relay_message_ack_seconds',
    help: 'Time from receiving message:send to acking it',
    buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1],
  }),
  slowConsumerDisconnects: new client.Counter({
    name: 'relay_slow_consumer_disconnects_total',
    help: 'Sockets dropped for exceeding the send buffer',
  }),
};

app.get('/metrics', async (_req, res) => {
  res.set('Content-Type', client.register.contentType);
  res.send(await client.register.metrics());
});

Alert on the symptoms users feel: p99 ack latency, a sudden drop in connected sockets across the fleet (a mass disconnect), event-loop lag above your tested limit, and a rising rate of error outcomes. Connection count alone is not an alert; it is a capacity signal to scale on.

Push notifications for offline users

A WebSocket only reaches people who are connected. For everyone else, and for mobile apps that are backgrounded, a new message has to become a push notification through FCM or APNs. That work is slow, can fail and must never delay the chat path, so it belongs on a queue, exactly like the async work in the Scaling series.

import { Queue } from 'bullmq';

const pushQueue = new Queue('push', { connection: { url: process.env.REDIS_URL } });

async function notifyOffline(message: Message, memberIds: string[]) {
  const recipients = memberIds.filter((id) => id !== message.senderId);
  const counts = await Promise.all(recipients.map((id) => liveConnections(redis, id)));
  const offline = recipients.filter((_, i) => counts[i] === 0);
  if (!offline.length) return;

  await pushQueue.addBulk(
    offline.map((userId) => ({
      name: 'new-message',
      data: { userId, conversationId: message.conversationId, messageId: message.id },
      // One pending push per user and conversation: a burst becomes one notification.
      opts: { jobId: `${userId}:${message.conversationId}`, delay: 3_000, removeOnComplete: true },
    })),
  );
}

• —Collapse bursts. The fixed job ID plus a short delay means ten quick messages produce one notification, not ten.
• —Re-check before sending. The worker checks presence again; if the user came online in those three seconds, the push is skipped.
• —Respect preferences. Muted conversations, do-not-disturb hours and blocked senders are enforced in the worker, not the client.
• —Keep bodies minimal. Send the conversation and sender name, and let privacy settings decide whether the message text appears on a lock screen.
• —Clean tokens. When FCM or APNs reports a token as invalid, delete it, or your error rate grows with every uninstall.

Security checklist

• —Trust nothing from the payload. The sender is always socket.data.user.id from part 3, never a field the client sends.
• —Authorise every event, not just the join. Check conversation membership on send, typing and read receipts, and leave rooms when membership is revoked.
• —Escape on render. Store message bodies as plain text and let React escape them. Never use dangerouslySetInnerHTML for message content, and build link previews on the server with an allowlist of protocols.
• —Check origins. Configure cors.origin to your real domains so a malicious page cannot open an authenticated socket using a victim's cookies.
• —Expire sessions mid-connection. When an access token expires or is revoked, disconnect the socket and make the client reconnect with a fresh token.
• —Build abuse tools early. Block and report actions, per-account send limits for new accounts, and a moderation queue for reported messages are much harder to add after spam arrives.

The whole system, part by part

Across ten parts, Relay grew from a single echo server into a production architecture. Each part added one layer:

1. 01WebSocket fundamentals: why persistent connections beat polling, and the protocol underneath everything else.
2. 02Socket.IO and typed events: rooms, acknowledgements and a React useChat hook.
3. 03Authentication: JWTs verified in the handshake, socket.data.user and personal rooms.
4. 04Persistence: Postgres with time-sortable IDs, idempotent inserts and cursor pagination.
5. 05Delivery guarantees: the client outbox, acks, retries, catch-up and read receipts.
6. 06Presence and typing: ephemeral state in Redis that heals itself.
7. 07Scaling out: the Redis adapter, load balancers and the cost of fan-out.
8. 08Tuning: file descriptors, kernel limits, ports and reconnect storms.
9. 09Load testing: k6 speaking Socket.IO, with end-to-end latency thresholds.
10. 10Hardening: rate limits, backpressure, graceful deploys, metrics, push and security.

A chat app is easy to demo and hard to run. The difference is everything that happens when connections drop, servers restart and users are not there.

Key takeaways

• —Rate limit inside the connection, and report limits through acks so the client outbox retries instead of losing messages.
• —Drop volatile events and disconnect hopeless slow consumers; durable storage and catch-up make that safe.
• —Drain on deploy: fail readiness, ask clients to move with jitter, and align LB and orchestrator timeouts.
• —Reach offline users through a queue with collapsed, preference-aware push notifications.
• —Authorise every event, escape every message and build abuse tooling before you need it.

That completes the series. If you want to go deeper on the infrastructure underneath, the Scaling to 1M Requests series covers caching, database scaling and observability in the same hands-on style. Or keep building: attachments uploaded straight to S3 with presigned URLs, message reactions and threads, and end-to-end encryption are all natural next features, and every one of them fits into the architecture Relay already has.