UDS socket reconnection via `udsGracefulErrorHandling` does not work on hot-shots ≥14

[KeenanLawrenceStitch]

Overview

UDS socket reconnection via udsGracefulErrorHandling does not work on hot-shots ≥14 because protocolErrorHandler() compares err.code against an array of string codes ('ENOTCONN', 'ECONNREFUSED', …), while unix-dgram (the UDS transport) sets err.code to a negative numeric errno (e.g. -107 on Linux, -54 on Darwin). The check never matches, so the socket is never replaced after Datadog Agent UDS path/inode churn.

This is the UDS counterpart of #301 / #302. #302 correctly fixed TCP by switching to string Node.js err.code values, but the same change was applied to UDS. TCP sockets from Node emit string codes; unix-dgram does not.

Environment

• hot-shots version: 15.0.0 (also affects ≥14.0.0, i.e. post-Fix issue #301 #302)
• unix-dgram version: 2.0.6
• Node.js version: v24.3.0 (also observed on Node 20+ in production)
• OS: macOS Darwin (local repro) / Linux (production Datadog Agent UDS)
• StatsD server: Datadog Agent DogStatsD over UDS (/var/run/datadog/dsd.socket)

Breakdown

In lib/statsd.js, send failures are passed to protocolErrorHandler with the raw transport error:

if ((this.protocol === PROTOCOL.TCP || this.protocol === PROTOCOL.UDS) && (callback || this.errorHandler)) {
  protocolErrorHandler(this, this.protocol, err); // err is from unix-dgram for UDS
}

Then:

} else if (protocol === PROTOCOL.UDS &&
    (!UDS_ERROR_CODES.includes(err.code) ||
     Date.now() - client.socket.createdAt < client.udsGracefulRestartRateLimit)) {
  return; // Always returns for unix-dgram numeric codes
}

UDS_ERROR_CODES comes from constants.udsErrors() (post-#302):

function udsErrors() {
  if (process.platform === 'linux') {
    return ['ENOTCONN', 'ECONNREFUSED'];
  }
  if (process.platform === 'darwin') {
    return ['EDESTADDRREQ', 'ECONNRESET'];
  }
  return [];
}

But unix-dgram builds errors like this (lib/unix_dgram.js):

function errnoException(errorno, syscall) {
  var e = new Error(syscall + ' ' + errorno);
  e.errno = e.code = errorno; // numeric, typically negative (-errno)
  e.syscall = syscall;
  return e;
}

So for a typical Agent-restart / stale-inode failure you get:

Field	Value (Linux)	Value (Darwin, live repro)
err.message	send -107	send -54
typeof err.code	number	number
err.code	-107 (-ENOTCONN)	-54 (-ECONNRESET)
UDS_ERROR_CODES.includes(err.code)	false	false

Pre-v14 this worked for UDS because the list was numeric os.constants.errno.* and the check was UDS_ERROR_CODES.includes(-err.code) (negating the negative errno back to the positive constant).

Sample log / live repro output

Production-style hot-shots formatted error (Linux Agent UDS):

Error sending hot-shots message: Error: send -107

Local live repro (Darwin, hot-shots@15.0.0 unpatched, unix-dgram@2.0.6):

platform darwin
udsErrors() [ 'EDESTADDRREQ', 'ECONNRESET' ]
SEND err {
  message: 'send -54',
  code: -54,
  typeofCode: 'number',
  errno: -54,
  includes: false
}

Matching simulation against unpatched constants.udsErrors():

linux ENOTCONN (-107):
  typeof(code)=number code=-107
  includes=false wouldRecreate=false
darwin ECONNRESET (-54):
  typeof(code)=number code=-54
  includes=false wouldRecreate=false
string 'ECONNRESET':
  typeof(code)=string code="ECONNRESET"
  includes=true wouldRecreate=true

Expected Behavior

When UDS send fails with ENOTCONN / ECONNREFUSED (Linux) or EDESTADDRREQ / ECONNRESET (Darwin), and udsGracefulErrorHandling: true is enabled, hot-shots should recreate the unix-dgram socket (subject to udsGracefulRestartRateLimit). After the Datadog Agent comes back and recreates /var/run/datadog/dsd.socket, metrics should resume without restarting the application process.

Actual Behavior

protocolErrorHandler never treats the unix-dgram error as retriable. The client keeps the stale connected socket (old inode). Every subsequent flush fails with send -107 / send -111 (or Darwin equivalents). Metrics are lost until the process is restarted / the StatsD client is manually replaced.

Reproduction

'use strict';
const fs = require('fs');
const os = require('os');
const path = require('path');
const StatsD = require('hot-shots');
const unixDgram = require('unix-dgram');

// --- 1) Show the matching bug in isolation (no Agent required) ---
const { udsErrors } = require('hot-shots/lib/constants'); // or copy udsErrors() from lib/constants.js
const UDS_ERROR_CODES = udsErrors();

function errnoException(errorno, syscall) {
  // Identical to unix-dgram's errnoException
  const e = new Error(syscall + ' ' + errorno);
  e.errno = e.code = errorno;
  e.syscall = syscall;
  return e;
}

const udsErr = errnoException(process.platform === 'linux' ? -107 : -54, 'send');
console.log({
  udsErrors: UDS_ERROR_CODES,
  message: udsErr.message,
  code: udsErr.code,
  typeofCode: typeof udsErr.code,
  includes: UDS_ERROR_CODES.includes(udsErr.code), // false on hot-shots >= 14
});

// --- 2) Live stale-inode UDS failure (unix-dgram) ---
const sockPath = path.join(os.tmpdir(), `hot-shots-uds-repro-${process.pid}.sock`);
try { fs.unlinkSync(sockPath); } catch (_) {}

const server = unixDgram.createSocket('unix_dgram');
server.bind(sockPath);

const clientSock = unixDgram.createSocket('unix_dgram');
clientSock.connect(sockPath);

setTimeout(() => {
  // Simulate Datadog Agent daemonset rollout: delete old socket, recreate path
  server.close();
  try { fs.unlinkSync(sockPath); } catch (_) {}

  const server2 = unixDgram.createSocket('unix_dgram');
  server2.bind(sockPath);

  clientSock.send(Buffer.from('test.metric:1|c\n'), (err) => {
    console.log('after agent restart simulation:', {
      message: err && err.message,
      code: err && err.code,
      typeofCode: err && typeof err.code,
      includes: err && UDS_ERROR_CODES.includes(err.code), // false
    });
    try { clientSock.close(); } catch (_) {}
    try { server2.close(); } catch (_) {}
    try { fs.unlinkSync(sockPath); } catch (_) {}
  });
}, 50);

// --- 3) hot-shots client never recovers (needs a real/missing dsd.socket) ---
const client = new StatsD({
  protocol: 'uds',
  path: '/var/run/datadog/dsd.socket', // or a test socket path
  udsGracefulErrorHandling: true,
  udsGracefulRestartRateLimit: 0,
  errorHandler: (err) => {
    console.log('hot-shots errorHandler:', {
      message: err.message,
      // Note: hot-shots wraps the message; raw unix-dgram code is what protocolErrorHandler sees
    });
  },
});

// With Agent running: metrics send.
// Restart Agent / replace UDS inode, then:
setInterval(() => client.increment('test.uds.recovery'), 1000);
// Expected: socket recreated, metrics resume
// Actual: perpetual "Error sending hot-shots message: Error: send -107" (Linux)

Suggested Fix

Keep string matching for TCP (Node string err.code), but make UDS matching accept both string codes and unix-dgram numeric (negative) errnos. Minimal approach for udsErrors():

function udsErrors() {
  if (process.platform === 'linux') {
    // unix-dgram reports negative numeric errnos, not string codes
    return ['ENOTCONN', 'ECONNREFUSED', -107, -111];
  }
  if (process.platform === 'darwin') {
    // unix-dgram reports negative numeric errnos, not string codes
    return ['EDESTADDRREQ', 'ECONNRESET', -39, -54];
  }
  return [];
}

A more defensive variant would map via os.constants.errno + Math.abs(err.code) so both signs and platforms stay correct without hardcoding.

Alternatively, restore the pre-v14 UDS check (includes(-err.code) with numeric constants) only for UDS, while leaving TCP on string codes.

Impact

Any process using protocol: 'uds' with udsGracefulErrorHandling: true on hot-shots ≥14 will never recover from Datadog Agent UDS socket replacement (daemonset rollout, agent restart, socket inode churn). Custom metrics stop until the application is restarted. This is especially painful in Kubernetes where Agent rollouts are routine and application pods are long-lived.

Issue #301 already suspected UDS might be affected; this confirms it, and shows the #302 string-code fix is correct for TCP but insufficient for unix-dgram.

[bdeitte]

Fixed in #323, patch release coming shortly