Skip to content

[Lettuce] Redis timeouts in OpenTelemetry 2.30.0+ #19814

Open

Description

@gaeljw

Describe the bug

Hello folks,

Following the upgrade of OpenTelemetry from 2.29.0 to 2.31.0, we've been experiencing a lot of errors in our service with the following stack trace:

org.springframework.dao.QueryTimeoutException: Redis command timed out
	at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:84)
	Suppressed: The stacktrace has been enhanced by Reactor, refer to additional information below: 
Error has been observed at the following site(s):
...
Original Stack Trace:
		at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:84)
		at org.springframework.data.redis.connection.lettuce.LettuceReactiveRedisConnection.lambda$translateException$0(LettuceReactiveRedisConnection.java:245)
		at reactor.core.publisher.Flux.lambda$onErrorMap$0(Flux.java:7315)
		at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:95)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.onError(TracingSubscriber.java:89)
		at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onError(MonoFlatMapMany.java:256)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.onError(TracingSubscriber.java:89)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.onError(TracingSubscriber.java:89)
		at reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onError(FluxDefaultIfEmpty.java:156)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.onError(TracingSubscriber.java:89)
		at reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onError(FluxHide.java:142)
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onError(FluxMapFuseable.java:143)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.onError(TracingSubscriber.java:89)
		at reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onError(FluxHide.java:142)
		at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:241)
		at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:316)
		at io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.onError(TracingSubscriber.java:89)
		at reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:94)
		at reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:94)
		at io.lettuce.core.RedisPublisher$ImmediateSubscriber.onError(RedisPublisher.java:901)
		at io.lettuce.core.RedisPublisher$State.onError(RedisPublisher.java:722)
		at io.lettuce.core.RedisPublisher$RedisSubscription.onError(RedisPublisher.java:362)
		at io.lettuce.core.RedisPublisher$SubscriptionCommand.onError(RedisPublisher.java:807)
		at io.lettuce.core.RedisPublisher$SubscriptionCommand.doOnError(RedisPublisher.java:803)
		at io.lettuce.core.protocol.CommandWrapper.completeExceptionally(CommandWrapper.java:132)
		at io.lettuce.core.protocol.CommandWrapper.completeExceptionally(CommandWrapper.java:131)
		at io.lettuce.core.cluster.ClusterCommand.completeExceptionally(ClusterCommand.java:85)
		at io.lettuce.core.protocol.MaintenanceAwareExpiryWriter.lambda$null$0(MaintenanceAwareExpiryWriter.java:119)
		at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
		at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
		at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:148)
		at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66)
		at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:1204)
		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
		at java.base/java.lang.Thread.run(Unknown Source)
Caused by: io.lettuce.core.RedisCommandTimeoutException: GET. Command timed out after 1 minute(s)
	at io.lettuce.core.internal.ExceptionFactory.createTimeoutException(ExceptionFactory.java:75)
	at io.lettuce.core.protocol.MaintenanceAwareExpiryWriter.lambda$null$0(MaintenanceAwareExpiryWriter.java:119)
	at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
	at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:148)
	at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:1204)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Unknown Source)

Steps to reproduce

No minimal reproduction scenario yet.

Expected behavior

Same behaviour as before, no errors :)

Actual behavior

Many errors :'(

Javaagent or library instrumentation version

2.31.0

Environment

  • JDK: 21
  • OS: Linux Ubuntu
  • Lettuce: 7.7.0.RELEASE (recently upgraded from 7.6.0.RELEASE) ; TBH this could be related as well as I haven't tested to revert Lettuce and keep the new OpenTelemetry ; I only tested to revert OpenTelemetry which made the error go away.
  • Spring Boot 4.1.1 (WebFlux)

Additional context

This happened under significant load (lot of queries), under normal load this didn't happen.

Claude is suggesting that it could be related to #19071 introduced in 2.30.0.

Tip

No response

Activity

  1. added
    bugSomething isn't working
    needs triageNew issue that requires triage
    on Aug 25, 2026
  2. laurit commented on Aug 25, 2026

    @laurit
    Contributor

    Claude is suggesting that it could be related to #19071 introduced in 2.30.0.

    Since that PR applies to lettuce-4.0 instrumentation but the latest version is instrumented by lettuce-5.1 it is probably something else.

  3. gaeljw commented on Aug 25, 2026

    @gaeljw
    Author

    Right.

    FYI I'll try to come up with a minimal reproduction scenario using a test container a bit later but I won't be able to do it until a few days.

  4. aditya1906 commented on Sep 27, 2026

    @aditya1906

    Is there any update on this issue. Our team once upgraded Opentelemtry-operator to 0.154.0, but it failed because of this bug #19373. and had to revert back to the old version.

    Now when we upgraded to 0.159.0 (which is using the opentelemetry-java-instrumentation version 2.31.1), we are also getting the same redis timeout error, and having to again revert back to the older version. A fix would be appriciated, if you require some additional information, I can try to provide, but so far it is the same error stacktrace mentioned in the Github issue

  5. jaydeluca commented on Sep 28, 2026

    @jaydeluca
    Member

    if you require some additional information, I can try to provide

    if you can provide a minimal application that reproduces the issue, that would be helpful

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingneeds triageNew issue that requires triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions