Skip to content

Flaky CI: post/test/socket_channels [TCP-Server] tests fail with NoMethodError on nil when server.accept times out #21720

Description

@Pushpenderrathore

Steps to reproduce

This is a CI flake in Meterpreter Acceptance / build / windows_meterpreter windows-2022, not something reproduced by hand.

  1. Push any branch, or watch master directly.
  2. Wait for Meterpreter Acceptance / build / windows_meterpreter windows-2022.
  3. Roughly a third of the time it fails in post/test/socket_channels on one of the [TCP-Server] cases.

It reproduces on master with no PR involved, so it is not specific to any contributor branch. Over the last 15 Meterpreter Acceptance runs on master: 7 success, 5 failure, 3 cancelled.

Expected behavior

The [TCP-Server] socket channel tests pass, or, when the underlying meterpreter channel genuinely does not come up, the failure names that cause.

Current behavior

The job fails with a NoMethodError on nil that does not describe the real problem. Two different messages have been seen, and they share a single root cause:

[-] [[TCP-Server] Propagates close events from the peer] Exception: NoMethodError: undefined method 'closed?' for nil
[-] [[TCP-Server] Has the correct peer information] Exception: NoMethodError: undefined method 'peerhost' for nil

Root cause

tcp_server_socket_trio in test/modules/post/test/socket_channels.rb polls server.accept until a deadline, and if the accept never lands it falls out of the loop and returns nil for server_client without raising:

server_client = nil
deadline = Time.now + timeout
while server_client.nil? && Time.now < deadline
  remaining = [deadline - Time.now, 5].min
  server_client = server.accept('Timeout' => remaining)
end

client_connector.stop

[client, server_client, server]   # server_client may be nil here

Every caller then uses server_client directly:

it '[TCP-Server] Propagates close events from the peer' do
  client, server_client = tcp_server_socket_pair
  client.close
  ret = retry_until_truthy(timeout: 5) { server_client.closed? }   # NoMethodError when nil

So the actual failure is "the meterpreter never relayed the channel-open notification within the 30s deadline", and whichever method the test calls first on nil is what gets reported. closed? and peerhost are the same timeout surfacing in two different test cases, not two separate bugs.

This also explains why it reads as random: it depends entirely on whether the Windows runner relays the channel open inside the deadline.

Observed occurrences

Same test, same error, on master and on unrelated PRs:

The master occurrence is the important one, since it rules out any contributor change as the trigger.

Prior work

@dwelch-r7 already narrowed this in ac78922 ("Retry server accept with short intervals to handle delayed channel-open notifications") and e22ba81 ("Correctly pass through the server accept timeout") on 2026-07-23. The retry loop clearly reduced the rate, but the failures above are all from after those commits, so it has not closed it out.

Suggested next steps

Two separable pieces, in case they are worth splitting:

  1. Diagnosability. Have tcp_server_socket_trio raise when the accept deadline expires, rather than returning nil, so the log says the accept timed out instead of reporting a NoMethodError on whatever method happened to be called next. This does not change pass or fail, only what the failure says.
  2. The actual flake. Work out why the Windows meterpreter sometimes does not relay the channel-open notification within 30s. That is the real fix and needs the channel code rather than the test.

Filing this rather than opening a PR since @dwelch-r7 is already on it and has the context. Happy to put up the change in point 1 if that is useful.

Metasploit version

Current master (reproduced as recently as 472bd832). CI runner: windows-2022.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status
    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions