[BUG] Setting incorrect dynamic nodes_dn can break inter-node trust, making the cluster completely unavailable

[parislarkins]

What is the bug?
Using the PUT /_plugins/_security/api/nodesdn API incorrectly, to set the dynamic nodesdn for some remote cluster name but with the certificate DN of the local cluster completely breaks all inter-node trust and causes the cluster to become completely unavailable. In our case this was an accidental misconfiguration, but ideally would not cause an entire cluster outage. This makes it particularly hard to undo, as the same API can no longer be used to revert the change.

Once triggered, the cluster can't form due to inter-node trust issues. I can't think of a valid reason for why this would be the case, since the dynamic nodes_dn updated is for a different cluster name, and the opensearch.yml still correctly trusts the local certificate DN in the nodes_dn setting.

The logs on all nodes will be filled with the following error, and the cluster will be completely unavailable and unable to respond to further requests:

[WARN ][o.o.d.HandshakingTransportAddressConnector] [ip-10-1-148-128] handshake failed for [connectToRemoteMasterNode[10.1.180.182:9300]]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) ~[?:?]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1377) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:660) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:129) ~[?:?]
at org.opensearch.security.support.WildcardMatcher$MatcherCombiner.<init>(WildcardMatcher.java:525) ~[?:?]
at org.opensearch.security.transport.DefaultInterClusterRequestEvaluator.isInterClusterRequest(DefaultInterClusterRequestEvaluator.java:95) ~[?:?]
at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:140) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1519) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) ~[?:?]
at org.opensearch.security.OpenSearchSecurityPlugin$6$1.messageReceived(OpenSearchSecurityPlugin.java:873) ~[?:?]
at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) ~[opensearch-2.19.4.jar:2.19.4]
at org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) ~[opensearch-2.19.4.jar:2.19.4]
at org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) ~[opensearch-2.19.4.jar:2.19.4]
at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:768) ~[opensearch-2.19.4.jar:2.19.4]
at org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) ~[opensearch-2.19.4.jar:2.19.4]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:697) ~[?:?]
org.opensearch.transport.RemoteTransportException: [ip-10-1-180-182][10.1.180.182:9300][internal:transport/handshake]
Caused by: java.lang.IllegalArgumentException
at org.opensearch.security.support.WildcardMatcher.concat(WildcardMatcher.java:221) ~[?:?]
at org.opensearch.security.transport.DefaultInterClusterRequestEvaluator.getNodesDnToEvaluate(DefaultInterClusterRequestEvaluator.java:75) ~[?:?]
at org.opensearch.security.transport.SecurityRequestHandler.addAdditionalContextValues(SecurityRequestHandler.java:362) ~[?:?]
at org.opensearch.wlm.WorkloadManagementTransportInterceptor$RequestHandler.messageReceived(WorkloadManagementTransportInterceptor.java:63) ~[opensearch-2.19.4.jar:2.19.4]
at org.opensearch.transport.NativeMessageHandler.handleRequest(NativeMessageHandler.java:278) ~[opensearch-2.19.4.jar:2.19.4]
at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) ~[opensearch-2.19.4.jar:2.19.4]
at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) ~[opensearch-2.19.4.jar:2.19.4]
at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) ~[opensearch-2.19.4.jar:2.19.4]
at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
at java.lang.Thread.run(Thread.java:1583) [?:?]
at org.opensearch.security.transport.DefaultInterClusterRequestEvaluator.isInterClusterRequest(DefaultInterClusterRequestEvaluator.java:95) ~[?:?]
at org.opensearch.security.transport.SecurityRequestHandler.addAdditionalContextValues(SecurityRequestHandler.java:362) ~[?:?]
at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) ~[opensearch-2.19.4.jar:2.19.4]
at org.opensearch.transport.NativeMessageHandler.handleRequest(NativeMessageHandler.java:278) ~[opensearch-2.19.4.jar:2.19.4]
at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]

The only way I was able to fix this was to:

1. Disable the plugins.security.nodes_dn_dynamic_config_enabled setting in opensearch.yml for all nodes - this is the only way to get the cluster to start again.
2. Restart all nodes.
3. Manually edit the .opendistro_security/_doc/nodesdn dynamic nodesdn configuration document to remove the broken configuration. This is obviously not the intended way and quite error prone, particularly as the document content is a base64-encoded JSON document.
4. Re-enable the plugins.security.nodes_dn_dynamic_config_enabled setting in opensearch.yml for all nodes, and restart all nodes.

How can one reproduce the bug?
Steps to reproduce the behavior:

For example, with the following configuration in opensearch.yml:

plugins.security.nodes_dn_dynamic_config_enabled: true
plugins.security.nodes_dn:
 - CN=<local cluster's own certificate DN>

This request completely breaks the cluster:

PUT /_plugins/_security/api/nodesdn/<remote cluster name, I don't think this matters>
{
  "nodes_dn": [
    "CN=<local cluster's own certificate DN>"
  ]
}

What is the expected behavior?
I would expect:

• the request to be rejected if it is completely invalid, and/or,
• this configuration to not completely break inter-node cluster trust - maybe we could always fall back to the nodes_dn list in opensearch.yml even if the dynamic configuration is invalid?

What is your host/environment?

• OS: Debian 12
• Version: 2.19.4

[cwperks]

@parislarkins TY for the bug report. What validations could be put in place to prevent this? Agreed on the need for a simple way to rollback misconfigurations.

[parislarkins]

Thanks @cwperks!

The only validation I can think of that would have prevented the misconfiguration was if we had validation to prevent setting the dynamic nodesdn for a remote cluster to a value already present in the local cluster's plugins.security.nodes_dn list, i.e. enforce that:

Two connected clusters should always have distinct Distinguished Names

I'm not sure if this is already an assumed requirement, but I think it seems reasonable? Two distinct clusters sharing the same DN doesn't seem to align with the current certificate-based identity and trust model of the plugin. For example, if two clusters did share the same DN (present in both clusters' plugins.security.nodes_dn), would a node in cluster A be able to join cluster B?