x/build/cmd/watchflakes: periodically restarts

[cagedmantis]

Watchflakes is periodically being restarted.

Command kubectl get pods produces:
watchflakes-deployment-679f9fbf8c-zp9ld 1/1 Running 47 (26m ago) 27d

@golang/release

[gabyhelp]

Related Issues

• x/build/cmd/watchflakes: currently in a CrashLoopBackOff state #70091 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[dmitshur]

Watchflakes currently implements retries at the process level. That is, it has many code paths that handle errors with a log.Fatal (see here), regardless of whether that error is a temporary network problem that would be safe to retry, or something that cannot be retried safely.

To make progress on this issue, errors that are safe to retry within the process need to identified and replaced with something along the lines of a time.Sleep(...); continue.

[cagedmantis]

I found these log entries right before a pod crashed:

watchflakes: 2025/02/12 19:34:59 ListCommits go release-branch.go1.23
watchflakes: 2025/02/12 19:35:00 rpc error: code = Unavailable desc = service unavailable

[cagedmantis]

Watchflakes is restarting every couple of minutes:
watchflakes-deployment-7d5fb59cbf-6s5ph 1/1 Running 1070 (5m29s ago) 34d