Increased rate of transient failures in recent weeks

[jessebraham]

Over the last couples weeks I've noticed an increase in the number of transient failures of CI jobs using these workflow. See here for an example:
https://github.com/esp-rs/esp-hal/actions/runs/3676289852/jobs/6216757844

I'm not sure what has changed, but this needs investigating.

CC @SergioGasquez

[bjoernQ]

I have an idea what is going on

Apparently, the curl in

xtensa-toolchain/install.sh

Line 36 in b7ce3a9

curl -s https://api.github.com/repos/esp-rs/rust-build/releases \

doesn't get a valid response (might be a victim to rate limiting since we don't pass a GITHUB_TOKEN, or it is some other network failure) - then jq will error

No idea how to solve that in a Bash script - in general might be worth to rewrite the action in TS/JS or even better in Rust

[jessebraham]

Ahh good catch, thanks for the info! I've been wanting to re-write this action for awhile but just haven't gotten around to it, something I'll tackle in the new year I suppose.

[SergioGasquez]

We are working on this issue at the moment. For reference its also seen in esp-idf-svc: esp-rs/esp-idf-svc#209

[bjoernQ]

I did a couple of tests and found a few issues. You can view the log of my latest test here: https://github.com/bjoernQ/workflow-test/actions/runs/3881586716/jobs/6620756700 and inspect my code here: https://github.com/bjoernQ/workflow-test/

Issues I found:

• using ' around Authorization: $GITHUB_TOKEN in the script will result in not resolving the env-var - needs to be enclosed by "
• the correct header is "Authorization: Bearer $GITHUB_TOKEN"
• the GITHUB_TOKEN doesn't automatically materialize as an env-var - you need something like this https://github.com/bjoernQ/workflow-test/blob/44f1e220740d7717636f92b7e34f9b0f01bfe686/.github/workflows/test.yml#L14

With all of this you will get the expected 1000 invocations rate-limit.

However, it's still not good to query the latest version in the script - that should only be done by espup (e.g. passing the literal latest as toolchain-version) since there we can have retries (maybe in a naive way or a bit more advanced by checking the rate-limit and the x-ratelimit-reset headers) and also, we don't waste an additional invocation that way

I think most of the things are already addressed in espup but probably not all (or they don't apply to the espup Rust code)