Netty Http2 instrumentation - Null checks in wrapped Http2Headers implementation

[jtduffy]

It looks like in certain (still unknown) configurations, the Http2Headers implementation that we wrap in our Netty 4.1.16 module isn't overriding the methods in AbstractHttp2Headers which throw UnsupportedOperationException

Specifically: .method() , .path(), .authority() and .contains()

To mitigate, a null check needs to be added in/around this code prior to de-referencing the wrapped Http2Headers.

Internal slack thread

[workato-integration]

https://new-relic.atlassian.net/browse/NR-306022

[kazablock]

Hi @jtduffy

We are encountering a similar issue while using the Java Jetcd library in conjunction with the latest New Relic Java agent. Would it be possible to expedite a hotfix at the earliest ?

[jasonjkeller]

@kazablock There's a custom build of the Java agent jar that you can download here that I believe should fix this issue: https://github.com/newrelic/newrelic-java-agent/actions/runs/10725431981

Would you be willing to give it a try and verify that it does in fact resolve the issue for you? If it does, we'll make sure to get it into our next release.

[kazablock]

Thank You @jasonjkeller , now the error rate for nettydispatcher has become zero which was previously 100%. But the request.uri is always unknown. Looks like the root cause is not fixed. I am using jetcd-core:0.7.5

NewRelic Finest Logs
2024-09-06T21:05:37,784+0900 [1 153] com.newrelic FINER: Unable to get Http2Headers method: null
at io.grpc.netty.AbstractHttp2Headers.method(AbstractHttp2Headers.java:510) ~[grpc-netty-1.51.0.jar:1.51.0]
2024-09-06T21:05:37,789+0900 [1 153] com.newrelic FINER: Unable to get Http2Headers path: null
at io.grpc.netty.AbstractHttp2Headers.path(AbstractHttp2Headers.java:540) ~[grpc-netty-1.51.0.jar:1.51.0]
2024-09-06T21:05:37,789+0900 [1 153] com.newrelic FINER: Unable to get Http2Headers authority: null
at io.grpc.netty.AbstractHttp2Headers.authority(AbstractHttp2Headers.java:530) ~[grpc-netty-1.51.0.jar:1.51.0]
2024-09-06T21:05:37,791+0900 [1 153] com.newrelic FINER: Unable to get Http2Headers header: AsciiString expected. Was: java.lang.String
at io.grpc.netty.GrpcHttp2HeadersUtils$GrpcHttp2InboundHeaders.requireAsciiString(GrpcHttp2HeadersUtils.java:252) ~[grpc-netty-1.51.0.jar:1.51.0]
at io.grpc.netty.GrpcHttp2HeadersUtils$GrpcHttp2ResponseHeaders.get(GrpcHttp2HeadersUtils.java:541) ~[grpc-netty-1.51.0.jar:1.51.0]
at io.grpc.netty.GrpcHttp2HeadersUtils$GrpcHttp2InboundHeaders.contains(GrpcHttp2HeadersUtils.java:150) ~[grpc-netty-1.51.0.jar:1.51.0]
at io.grpc.netty.GrpcHttp2HeadersUtils$GrpcHttp2InboundHeaders.contains(GrpcHttp2HeadersUtils.java:91) ~[grpc-netty-1.51.0.jar:1.51.0]
2024-09-06T21:05:37,856+0900 [1 153] com.newrelic FINER: Unable to get Http2Headers content-length: null
at io.grpc.netty.AbstractHttp2Headers.contains(AbstractHttp2Headers.java:260) ~[grpc-netty-1.51.0.jar:1.51.0]
at io.grpc.netty.AbstractHttp2Headers.contains(AbstractHttp2Headers.java:26) ~[grpc-netty-1.51.0.jar:1.51.0]
2024-09-06T21:05:37,857+0900 [1 157] com.newrelic FINER: Unable to get Http2Headers header: AsciiString expected. Was: java.lang.String
at io.grpc.netty.GrpcHttp2HeadersUtils$GrpcHttp2InboundHeaders.requireAsciiString(GrpcHttp2HeadersUtils.java:252) ~[?:?]
at io.grpc.netty.GrpcHttp2HeadersUtils$GrpcHttp2ResponseHeaders.get(GrpcHttp2HeadersUtils.java:541) ~[?:?]
at io.grpc.netty.GrpcHttp2HeadersUtils$GrpcHttp2InboundHeaders.contains(GrpcHttp2HeadersUtils.java:150) ~[?:?]
at io.grpc.netty.GrpcHttp2HeadersUtils$GrpcHttp2InboundHeaders.contains(GrpcHttp2HeadersUtils.java:91) ~[?:?]
2024-09-06T21:05:37,859+0900 [1 157] com.newrelic FINER: Unable to get Http2Headers content-type: null
at io.grpc.netty.AbstractHttp2Headers.contains(AbstractHttp2Headers.java:260) ~[?:?]
at io.grpc.netty.AbstractHttp2Headers.contains(AbstractHttp2Headers.java:26) ~[?:?]
2024-09-06T21:05:37,860+0900 [1 153] com.newrelic FINER: Unable to get Http2Headers method: null
at io.grpc.netty.AbstractHttp2Headers.method(AbstractHttp2Headers.java:510) ~[grpc-netty-1.51.0.jar:1.51.0]

[jasonjkeller]

@kazablock The current behavior is expected as we don't explicitly have support for grpc-netty and it looks like we're probably going to need to create additional instrumentation for that. I've gone ahead and filed a feature request to add support for grpc-netty: #2044

If you're able to provide a simple repro app using jetcd-core:0.7.5, it would help to expedite this work. Thanks.

[kazablock]

Thank you @jasonjkeller , i have attached the reproducer. Running it is pretty straight forward.
Steps

1. Run etcd. (docker compose up -d)
2. create jar (mvn clean package)
3. run the jar with newrelic agent .(i have used the jar mentioned in this issue which has try catch added https://github.com/newrelic/newrelic-java-agent/actions/runs/10725431981)
   java -javaagent:replaceme -Dnewrelic.config.app_name=replaceme -Dnewrelic.config.log_level=finest -Dnewrelic.config.log_file_name=STDOUT -Dnewrelic.config.license_key=replaceme -jar ./target/JetcdNewrelic-1.0-SNAPSHOT-jar-with-dependencies.jar
4. check the console logs for error

JetcdNewrelic.zip