Potential issue with virtual threads

[natan-abolafya]

Describe the bug
Sorry, this will be a bit vague as I'm having hard time understanding as well.
Related: #157
Version: 2.10.1

We are using org.newsclub.net.unix.AFUNIXSocketFactory. I have been investigating an issue with jdbc connection pools where the application would just hang for unclear reasons and jdbc pool would start throwing errors about pool being busy even though there was nothing going on. It looked like some sort of connection leak but I saw nothing on the thread dumps whenever this happened.

The problem solved itself as soon as I disabled Jetty virtual threads. I don't have a clear indication it was junixsocket that caused the problem but it felt like a good suspect. Unfortunately, it happens only on a specific environment and I can't reproduce locally. And on this specific environment, it is hard to change it so the application can use postgresql TCP socket instead of unix socket to rule it out.

So I've asked AI to create a minimal test. The result is not clear cut.

  @Test
  public void blockingRead_onVirtualThreads_completesForAllClients() throws Exception {
    int clients = Integer.getInteger("benchmark.junixsocket.clients", 64);
    int serverDelayMillis = Integer.getInteger("benchmark.junixsocket.server-delay-ms", 20);
    int timeoutSeconds = Integer.getInteger("benchmark.junixsocket.timeout-seconds", 60);
    byte payload = 0x2A;

    Path socketPath = Files.createTempFile("junixsocket-vt-", ".sock");
    Files.deleteIfExists(socketPath);

    var failures = new AtomicInteger(0);
    var completed = new CountDownLatch(clients);
    var address = AFUNIXSocketAddress.of(socketPath.toFile());

    try (var server = AFUNIXServerSocket.newInstance()) {
      server.bind(address);

      var acceptor = Executors.newSingleThreadExecutor();
      var handlers = Executors.newCachedThreadPool();

      try {
        acceptor.submit(() -> {
          for (int i = 0; i < clients; i++) {
            try {
              var clientSocket = server.accept();
              handlers.submit(() -> {
                try (clientSocket) {
                  Thread.sleep(serverDelayMillis);
                  clientSocket.getOutputStream().write(payload);
                  clientSocket.getOutputStream().flush();
                } catch (Throwable t) {
                  failures.incrementAndGet();
                }
              });
            } catch (IOException e) {
              failures.incrementAndGet();
            }
          }
        });

        long startNanos = System.nanoTime();
        try (var vexec = Executors.newVirtualThreadPerTaskExecutor()) {
          for (int i = 0; i < clients; i++) {
            vexec.submit(() -> {
              try (var socket = AFUNIXSocket.newInstance()) {
                socket.connect(address);
                int result = socket.getInputStream().read();
                if (result != payload) {
                  failures.incrementAndGet();
                }
              } catch (Throwable t) {
                failures.incrementAndGet();
              } finally {
                completed.countDown();
              }
            });
          }
        }

        boolean doneInTime = completed.await(timeoutSeconds, TimeUnit.SECONDS);
        long elapsedMs = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNanos);
        System.out.printf("%njunixsocket VT test: clients=%d, delayMs=%d, elapsedMs=%d, doneInTime=%s, failures=%d%n%n",
                          clients, serverDelayMillis, elapsedMs, doneInTime, failures.get());

        assertThat(doneInTime).isTrue();
        assertThat(failures.get()).isZero();
      } finally {
        acceptor.shutdownNow();
        handlers.shutdownNow();
        acceptor.awaitTermination(5, TimeUnit.SECONDS);
        handlers.awaitTermination(5, TimeUnit.SECONDS);
      }
    } finally {
      Files.deleteIfExists(socketPath);
    }
  }

This test hangs for 5 seconds (awaitTermination) and then gets a few failures.
If I change newVirtualThreadPerTaskExecutor to newCachedThreadPool, the test finishes immediately without any failure.
Can't say what that means exactly either but I was hoping perhaps you can? If this isn't helpful, feel free to close this issue as I lack the time to provide more right now and will go ahead with disabling virtual threads anyway.

[kohlschuetter]

Hi @natan-abolafya, thanks for reporting!

It looks like we're hitting a race between accept and connect: connect may fail with a "connection refused" error well within the specified connection timeout.

A reasonable fix will be to keep retrying to connect if running from a virtual thread is detected (fix is coming for 2.11.0).
In the meantime: either don't use virtual threads or simply catch the SocketException when calling connect, and keep retrying.

[natan-abolafya]

Thanks @kohlschuetter.

I am not actually using it directly, as it's tomcat's jdbc pool using the AFUNIXSocketFactory, so for now I have disabled virtual threads. That said, maybe this issue is not the same issue I'm hitting but I'll give it a try when there is a new version.

[kohlschuetter]

Please verify by using the latest 2.11.0-SNAPSHOT build, and report back here.

<dependencies>
  <dependency>
      <groupId>com.kohlschutter.junixsocket</groupId>
      <artifactId>junixsocket-common</artifactId>
      <version>2.11.0-20260330.132544-1</version>
  </dependency>
</dependencies>
<repositories>
  <repository>
      <id>snapshots-central</id>
      <url>https://central.sonatype.com/repository/maven-snapshots/</url>
      <releases>
          <enabled>false</enabled>
      </releases>
      <snapshots>
          <enabled>true</enabled>
      </snapshots>
  </repository>
</repositories>

Alternatively, add this selftest jar snapshot to your project, which should contain everything in one piece.