Failures occur with OIDC Method During Parallel Requests

[jwenz723]

I have a repo in which I am testing out the new GitHub OIDC provider functionality. I have about 20 workflows in the repo. Typically these workflows are run individually since they are configured to be triggered by different events. However, in my particular scenario I have a PR open which is making changes to all of these workflows, thus, they are all triggering at the same time. When this occurs I see the following error in some, but not all of the workflows:

Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements

Simply clicking the Re-run all jobs button causes the error to resolve itself when the workflow runs a second time.

Are there any known limits with how many workflows can be run in parallel with the GitHub OIDC provider? Or is this a bug?

[jwenz723]

I'm wondering if it is possible that my workflows are getting rate limited and if maybe it would be a good idea to implement an exponential backoff within this action to handle this?

[stijndehaes]

I see the same thing

[jwenz723]

I've come up with a work around that is fairly reliable for this issue by adding conditional retry steps in my workflows. Ideally this is something that would be handled directly by this action.

Here is how I add the conditional retries. Between each invocation of the aws-actions/configure-aws-credentials action is a step which performs a sleep for a random amount of time between 15 and 65 seconds. The idea with the random sleep is this will hopefully cause the various workflows which are running in parallel to retry at different points in time.

- name: Configure AWS Credentials
  id: aws-auth
  continue-on-error: true
  uses: aws-actions/configure-aws-credentials@e21f7333e801ca751f058cc52de17f0ee6e1da6f
  with:
    aws-region: us-west-2
    mask-aws-account-id: false
    role-to-assume: arn:aws:iam::123456789012:role/myrole

- name: Sleep
  if: steps.aws-auth.outcome != 'success'
  run: sleep $[ ( $RANDOM % 50 )  + 15 ]s

- name: Configure AWS Credentials Retry
  id: aws-auth2
  continue-on-error: true
  if: steps.aws-auth.outcome != 'success'
  uses: aws-actions/configure-aws-credentials@e21f7333e801ca751f058cc52de17f0ee6e1da6f
  with:
    aws-region: us-west-2
    mask-aws-account-id: false
    role-to-assume: arn:aws:iam::123456789012:role/myrole

- name: Sleep
  if: steps.aws-auth.outcome != 'success' && steps.aws-auth2.outcome != 'success'
  run: sleep $[ ( $RANDOM % 50 )  + 15 ]s

- name: Configure AWS Credentials Retry
  id: aws-auth3
  if: steps.aws-auth.outcome != 'success' && steps.aws-auth2.outcome != 'success'
  uses: aws-actions/configure-aws-credentials@e21f7333e801ca751f058cc52de17f0ee6e1da6f
  with:
    aws-region: us-west-2
    mask-aws-account-id: false
    role-to-assume: arn:aws:iam::123456789012:role/myrole

This example only tries 3 times to authenticate. In theory any number of attempts could be done, but as you can see it gets quite verbose adding additional retries.

[abatilo]

Have you tried requesting the credentials once, storing it somewhere, and then using it in your matrix?

[jwenz723]

In my scenario, I am not using a matrix build. I have multiple workflows that are triggered at the same time. So there is no way to store the credentials for reuse across the various workflows.

[guitarrapc]

I met same issue and avoid via reuse aws auth in workflow. This is another approch to avoid retry attempt too long or failed.

Here is how I add the reuse. AWS Auth action run before any jobs. Then aws auth required job can run parallel.

https://gist.github.com/guitarrapc/bb279d0a0be2b229501a673980f96280

[jwenz723]

I met same issue and avoid via reuse aws auth in workflow. This is another approch to avoid retry attempt too long or failed.

Here is how I add the reuse. AWS Auth action run before any jobs. Then aws auth required job can run parallel.

https://gist.github.com/guitarrapc/bb279d0a0be2b229501a673980f96280

I like this solution for solving the scenario where multiple jobs are running in parallel and each job needs to authenticate. However, I don't know if it is considered good practice to store credentials in the github actions cache. There is a disclaimer here that states that sensitive data should NOT be stored in the cache for public repos. There is no recommendation for private repos.

[guitarrapc]

You are right. Although token will be disable in 900sec (minimum) or 1hour (default), caching senstive data is no recommended.
Looking into actions/runner#1466 (comment) and if it is JS SDK issue, I hope configure-aws-credentials will fix 10+ parallel requests limitation.

[zarnovican]

I tried to spread the OIDC login across 10x identical IAM roles. It did not help. It looks like the rate-limit is per AWS account and does not include the role.

This is the minimal reproducer:

name: test1

on:
  push:

permissions:
  id-token: write

jobs:

  test1:
    strategy:
      matrix:
        foo: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40]
      fail-fast: false

    runs-on: ubuntu-latest
    steps:

      - name: Login to AWS via Github OIDC
        uses: aws-actions/configure-aws-credentials@v1
        with:
          role-to-assume: <role-arn>
          aws-region: eu-west-1
          mask-aws-account-id: no

      - run: aws sts get-caller-identity

From the 40 jobs, the success rate is only ~10-20 jobs.

[mgmarino]

We see this issue when running jobs in parallel, either as separate jobs or with strategy.matrix. We have seen it happen with as few as 2 jobs running in parallel, but requesting tokens at the same time. In addition, we do not see any records in CloudTrail for AssumeRoleWithWebIdentity when this happens suggesting to me that this is all rather on the Github side before a call actually goes to the STS API.

A couple of questions:

• Has anyone raised/found an issue with Github?
• As anyone from AWS commented on this yet?

[zarnovican]

We have seen it happen with as few as 2 jobs running in parallel

We have a project which is running only once per hour that is sporadically failing as well. The problem is that all repos fall into the same rate-limit. You have to consider all workflows from all your repos combined (if the theory of rate-limit is correct).

..all rather on the Github side before a call actually goes to the STS API

I'm convinced that it is on AWS, or between AWS and Github. It is IMO not a problem of getting the JWT token from Github. Why ?

Before OIDC support was added to configure-aws-credentials, we have made a simple implementation https://github.com/zarnovican/aws-oidc-login-action that will only retrieve the JWT token from Github and write it to the file. This action is not making any Amazon call. When I update the reproducer to use this action instead of configure-aws-credentials, the problem moves to the next call (aws sts get-caller-identity). Jobs are not failing on Github api calls.

      - name: Login to AWS via Github OIDC
        uses: zarnovican/aws-oidc-login-action@v1
        with:
          role: <role-arn>
          aws-region: eu-west-1
          client-id: sigstore

      - run: aws sts get-caller-identity

In addition, we do not see any records in CloudTrail for AssumeRoleWithWebIdentity..

Could it be that api calls blocked by rate-limit did not get to CloudTrail ?

Has anyone raised/found an issue with Github?

I have opened the ticket today, but my hopes are very low. In the best case, I expect Amazon to confirm the rate-limit.

It still could be a problem of Amazon backend calling Github backend and failing. Unfortunatelly, we don't have a visibility into that, or do we ?

As anyone from AWS commented on this yet?

No, not on this issue. On a linked issue actions/runner#1466, there are couple of comments from someone from Github.

I think the best course of action is to implement retries in configure-aws-credentials action. I'm not sure if you can distinguish between "denied by rate-limit" from "denied by some other cause". Therefore, the action should fail on first error as it is now. And only when this retry is explicitly enabled by caller, it should retry failed calls.

[mgmarino]

I'm convinced that it is on AWS, or between AWS and Github. It is IMO not a problem of getting the JWT token from Github.

You are probably right, I am leaning towards between AWS and Github. IIRC the flow used here does have a server-to-server validation step. Adding the stack trace output (via setting the SHOW_STACK_TRACE: 'true' in the env variables) adds the additional information that the STS API returns a InvalidIdentityToken error (see e.g. here). In the issue you linked, there's mention of a timeout of 5 secs (presumably on Github's OIDC server response in the validation step), though I'm a little skeptical this is what's happening here, because then I would expect an error like IDPCommunicationError.

I think the best course of action is to implement retries in configure-aws-credentials action. I'm not sure if you can distinguish between "denied by rate-limit" from "denied by some other cause". Therefore, the action should fail on first error as it is now. And only when this retry is explicitly enabled by caller, it should retry failed calls.

According to the docs referenced above, a number of errors could be classified as retry-able. These could be used to decide to retry or not (perhaps with exponential backoff).

For us, though, the only viable path forward for now (other than dropping back to using tokens in GH Secrets) is to obtain the credentials once in a step, and export them using job outputs/needs, etc...

[zarnovican]

Has anyone raised/found an issue with Github?

Response from Amazon regarding rate-limiting (I misread your question before):

I can confirm that a cause of this error can be related to api throttling on the AWS server end. In terms of advising what the current rate limit is, this is not disclosed and AWS Support engineers also do not have information on this...

With relation to increasing rate limits, in rare circumstances this can be considered, but approval from the AWS service team is required. I understand inconvenience this throttling may cause but this throttling is in place to to protect...

Thus, your best bet to resolve throttling entirely is to spread your calls out over a longer period...

Even if Amazon confirmed that there is a rate-limit on their side, it does not prove that it is the root-cause of the issue. @mgmarino thanks for your insights. I'm now leaning towards server-to-server problem as well.

[mgmarino]

Just as another possibility of a workaround (that does not deal with the cross-account issue that we expect is happening), let me add this here. For a large job matrix, you can add a sleep based upon the index of the job, i.e.:

      - name: Sleep, Data, Sleep
        run: sleep $[ ( ${{ strategy.job-index }} + 1)*3 ]s
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
     ...

YMMV, but it may be an easy enough of a solution to get us through until this issue is dealt with.