[Lettuce] Redis timeouts in OpenTelemetry 2.30.0+ #19814
Copy link
Copy link
Open
Labels
bugSomething isn't workingSomething isn't workingneeds triageNew issue that requires triageNew issue that requires triage
Description
Describe the bug
Hello folks,
Following the upgrade of OpenTelemetry from 2.29.0 to 2.31.0, we've been experiencing a lot of errors in our service with the following stack trace:
org.springframework.dao.QueryTimeoutException: Redis command timed out
at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:84)
Suppressed: The stacktrace has been enhanced by Reactor, refer to additional information below:
Error has been observed at the following site(s):
...
Original Stack Trace:
at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:84)
at org.springframework.data.redis.connection.lettuce.LettuceReactiveRedisConnection.lambda$translateException$0(LettuceReactiveRedisConnection.java:245)
at reactor.core.publisher.Flux.lambda$onErrorMap$0(Flux.java:7315)
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:95)
at io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.onError(TracingSubscriber.java:89)
at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onError(MonoFlatMapMany.java:256)
at io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.onError(TracingSubscriber.java:89)
at io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.onError(TracingSubscriber.java:89)
at reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onError(FluxDefaultIfEmpty.java:156)
at io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.onError(TracingSubscriber.java:89)
at reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onError(FluxHide.java:142)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onError(FluxMapFuseable.java:143)
at io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.onError(TracingSubscriber.java:89)
at reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onError(FluxHide.java:142)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:241)
at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:316)
at io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.onError(TracingSubscriber.java:89)
at reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:94)
at reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:94)
at io.lettuce.core.RedisPublisher$ImmediateSubscriber.onError(RedisPublisher.java:901)
at io.lettuce.core.RedisPublisher$State.onError(RedisPublisher.java:722)
at io.lettuce.core.RedisPublisher$RedisSubscription.onError(RedisPublisher.java:362)
at io.lettuce.core.RedisPublisher$SubscriptionCommand.onError(RedisPublisher.java:807)
at io.lettuce.core.RedisPublisher$SubscriptionCommand.doOnError(RedisPublisher.java:803)
at io.lettuce.core.protocol.CommandWrapper.completeExceptionally(CommandWrapper.java:132)
at io.lettuce.core.protocol.CommandWrapper.completeExceptionally(CommandWrapper.java:131)
at io.lettuce.core.cluster.ClusterCommand.completeExceptionally(ClusterCommand.java:85)
at io.lettuce.core.protocol.MaintenanceAwareExpiryWriter.lambda$null$0(MaintenanceAwareExpiryWriter.java:119)
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:148)
at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:1204)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: io.lettuce.core.RedisCommandTimeoutException: GET. Command timed out after 1 minute(s)
at io.lettuce.core.internal.ExceptionFactory.createTimeoutException(ExceptionFactory.java:75)
at io.lettuce.core.protocol.MaintenanceAwareExpiryWriter.lambda$null$0(MaintenanceAwareExpiryWriter.java:119)
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:148)
at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:1204)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)
Steps to reproduce
No minimal reproduction scenario yet.
Expected behavior
Same behaviour as before, no errors :)
Actual behavior
Many errors :'(
Javaagent or library instrumentation version
2.31.0
Environment
- JDK: 21
- OS: Linux Ubuntu
- Lettuce: 7.7.0.RELEASE (recently upgraded from 7.6.0.RELEASE) ; TBH this could be related as well as I haven't tested to revert Lettuce and keep the new OpenTelemetry ; I only tested to revert OpenTelemetry which made the error go away.
- Spring Boot 4.1.1 (WebFlux)
Additional context
This happened under significant load (lot of queries), under normal load this didn't happen.
Claude is suggesting that it could be related to #19071 introduced in 2.30.0.
Tip
No response
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't workingneeds triageNew issue that requires triageNew issue that requires triage
Activity
laurit commentedon Aug 25, 2026
Since that PR applies to lettuce-4.0 instrumentation but the latest version is instrumented by lettuce-5.1 it is probably something else.
gaeljw commentedon Aug 25, 2026
Right.
FYI I'll try to come up with a minimal reproduction scenario using a test container a bit later but I won't be able to do it until a few days.
aditya1906 commentedon Sep 27, 2026
Is there any update on this issue. Our team once upgraded Opentelemtry-operator to 0.154.0, but it failed because of this bug #19373. and had to revert back to the old version.
Now when we upgraded to 0.159.0 (which is using the opentelemetry-java-instrumentation version 2.31.1), we are also getting the same redis timeout error, and having to again revert back to the older version. A fix would be appriciated, if you require some additional information, I can try to provide, but so far it is the same error stacktrace mentioned in the Github issue
jaydeluca commentedon Sep 28, 2026
if you can provide a minimal application that reproduces the issue, that would be helpful