xds: 'Unsupported configurations in UpstreamTlsContext!' error on creating a channel

[nikolays-fd]

What version of gRPC-Java are you using?

1.44.0, 1.43.2

What is your environment?

OS Name: Microsoft Windows 11 Pro
OS Version: 10.0.22000 N/A Build 22000
openjdk 11.0.8 2020-07-14

What did you expect to see?

I create a channel using the following function and then use it for communication with a gRPC service as a client.

fun createChannelUsingXds(
    name: String
): ManagedChannel {
    val credentials = XdsChannelCredentials.create(InsecureChannelCredentials.create())
    return Grpc.newChannelBuilder("xds:///$name", credentials).build()
}

What did you see instead?

I got an exception:

Caused by: java.lang.UnsupportedOperationException: Unsupported configurations in UpstreamTlsContext!

    io.grpc.StatusException: UNKNOWN: Channel Pipeline: [SdsProtocolNegotiators$BufferReadsHandler#0, SdsProtocolNegotiators$ClientSdsHandler#0, WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0]
        at io.grpc.Status.asException(Status.java:550)
        at io.grpc.kotlin.ClientCalls$rpcImpl$1$1$1.onClose(ClientCalls.kt:295)
        at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:463)
        at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:427)
        at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:460)
        at io.grpc.xds.XdsNameResolver$ConfigSelector$1ClusterSelectionInterceptor$1$1.onClose(XdsNameResolver.java:511)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
        at java.lang.Thread.run(Thread.java:920)
     Caused by: java.lang.UnsupportedOperationException: Unsupported configurations in UpstreamTlsContext!
        at io.grpc.xds.internal.sds.ClientSslContextProviderFactory.create(ClientSslContextProviderFactory.java:58)
        at io.grpc.xds.internal.sds.ClientSslContextProviderFactory.create(ClientSslContextProviderFactory.java:27)
        at io.grpc.xds.internal.sds.ReferenceCountingMap.getInternal(ReferenceCountingMap.java:81)
        at io.grpc.xds.internal.sds.ReferenceCountingMap.get(ReferenceCountingMap.java:56)
        at io.grpc.xds.internal.sds.TlsContextManagerImpl.findOrCreateClientSslContextProvider(TlsContextManagerImpl.java:76)
        at io.grpc.xds.internal.sds.SslContextProviderSupplier.getSslContextProvider(SslContextProviderSupplier.java:100)
        at io.grpc.xds.internal.sds.SslContextProviderSupplier.updateSslContext(SslContextProviderSupplier.java:60)
        at io.grpc.xds.internal.sds.SdsProtocolNegotiators$ClientSdsHandler.handlerAdded0(SdsProtocolNegotiators.java:206)
        at io.grpc.netty.shaded.io.grpc.netty.ProtocolNegotiators$ProtocolNegotiationHandler.handlerAdded(ProtocolNegotiators.java:1048)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:938)
        at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609)
        at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.addBefore(DefaultChannelPipeline.java:269)
        at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.addBefore(DefaultChannelPipeline.java:237)
        at io.grpc.netty.shaded.io.grpc.netty.WriteBufferingAndExceptionHandler.handlerAdded(WriteBufferingAndExceptionHandler.java:59)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:938)
        at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609)
        at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46)
        at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463)
        at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115)
        at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:514)
	    at io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486)
        at io.grpc.netty.shaded.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
        at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
        at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
        at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        	... 1 more

Steps to reproduce the bug

We have a golang service which uses gRPC and xDS. It creates an upstream context for a cluster as follows:

Go clients work well with this service. I've just started trying to call this service from Java using grpc-xds protocol.
When I got this error I made some research and found that before creating the SslContextProvider for the channel from the UpstreamContext, the context is checked for the presence of some fields:

grpc-java/xds/src/main/java/io/grpc/xds/internal/sds/ClientSslContextProviderFactory.java

Lines 51 to 57 in 9f644a0

	if (CommonTlsContextUtil.hasCertProviderInstance(
	upstreamTlsContext.getCommonTlsContext())) {
	return certProviderClientSslContextProviderFactory.getProvider(
	upstreamTlsContext,
	bootstrapInfo.node().toEnvoyProtoNode(),
	bootstrapInfo.certProviders());
	}

grpc-java/xds/src/main/java/io/grpc/xds/internal/sds/CommonTlsContextUtil.java

Lines 28 to 41 in 9f644a0

	static boolean hasCertProviderInstance(CommonTlsContext commonTlsContext) {
	return commonTlsContext != null
	&& (commonTlsContext.hasTlsCertificateCertificateProviderInstance()
	|| hasCertProviderValidationContext(commonTlsContext));
	}
	private static boolean hasCertProviderValidationContext(CommonTlsContext commonTlsContext) {
	if (commonTlsContext.hasCombinedValidationContext()) {
	CombinedCertificateValidationContext combinedCertificateValidationContext =
	commonTlsContext.getCombinedValidationContext();
	return combinedCertificateValidationContext.hasValidationContextCertificateProviderInstance();
	}
	return commonTlsContext.hasValidationContextCertificateProviderInstance();
	}

So, the UpstreamContext is checked for:

• The filed TlsCertificateCertificateProviderInstance has value or
• The field ValidationContextTypeCase is CommonTlsContext.CertificateProviderInstance or
• The field ValidationContextTypeCase is CommonTlsContext.CombinedCertificateValidationContext and the field ValidationContextCertificateProviderInstance has value.

None of these fields is in my UpstreamContext. But after that, some other fields are used for creating the SslContextProvider: CertificateValidationContext.CaCertificatedProviderInstance and TlsCertificateProviderInstance, which are in my context. So, I would be able to create the SslContextProvider if I passed the previous check.

I also noticed that all those fields are @deprecated in the proto and generated stubs, so I assumed that it could be a bug.

[dapengzhang0]

cc @sanjaypujare

[sanjaypujare]

Yes this is a bug and we will fix it in the next release.

[sanjaypujare]

@nikolays-fd if you are able to rebuild grpc and patch with #8892 could you verify the fix?

[nikolays-fd]

@nikolays-fd if you are able to rebuild grpc and patch with #8892 could you verify the fix?

Yes, sure. I was able to build the gPRC project locally with the patch and my client worked well. Thank you!

[sanjaypujare]

@nikolays-fd if you are able to rebuild grpc and patch with #8892 could you verify the fix?

Yes, sure. I was able to build the gPRC project locally with the patch and my client worked well. Thank you!

Thank you!