Creation of Endpoints for Cloud Functions causes circular dependency between ESP and Endpoint services

[nikolay-bocharov]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

$ terraform -v
Terraform v0.12.20
+ provider.google v3.5.0

Affected Resource(s)

google_cloud_run_service
google_endpoints_service

Terraform Configuration Files

resource "google_cloud_run_service" "run" {
  name = var.name
  location = var.location

  template {
    spec {
      containers {
        image = "gcr.io/endpoints-release/endpoints-runtime-serverless:2"
        env {
          name = "ENDPOINTS_SERVICE_NAME"
          value = google_endpoints_service.openapi_service.service_name
        }
      }
    }
  }
}

resource "google_endpoints_service" "openapi_service" {
  service_name = replace(google_cloud_run_service.run.status[0].url,  "https://", "")
  project = var.project_id
  openapi_config = var.config
}

Debug Output

Panic Output

Expected Behavior

Please note that I do not expect this deployment with circular dependency to work.
But there must be a working scenario for creating Endpoints for Google Function possibly followed different approach or with different API.
Current API seem not to permit such cases.

Actual Behavior

The circular dependency error appears

Steps to Reproduce

1. terraform apply

Important Factoids

Google Cloud endpoint documentation from here - https://cloud.google.com/endpoints/docs/openapi/get-started-cloud-functions reads the following steps to create Endpoint for Google Function

1. Create ESP service with Google Run and remember the URL address allocated by Google for this service
2. Crete Endpoint service (Use that URL address obtained on previous step to build the endpoint openapi config)
3. Update the ESP service created on step 1 with the name of the Endpoint

But these steps make both resources dependent on each other and inevitably bring to circular dependency error in Terraform.
There must be a working scenario for creating Endpoints for Google Function.

References

• Google documentation for Endpoint for Google Function https://cloud.google.com/endpoints/docs/openapi/get-started-cloud-functions
• b/272363819

[nikolay-bocharov]

I've found a workaround for this using provisioners

resource "google_endpoints_service" "openapi_service" {
  service_name = replace(var.espService.url,  "https://", "")
  project = var.project_id
  openapi_config = var.config
  provisioner "local-exec" {
    command = "gcloud beta run services update ${var.espService.name} --set-env-vars ENDPOINTS_SERVICE_NAME=${self.service_name} --project ${var.project_id} --platform=managed --region=${var.region}"
  }
}

It uses provisioners block to update the ESP Service once Endpoint is created.

Unfortunately this won't work with Terraform Github Action as running the gcloud command from the provisioner block will start it on the docker container (where terraform step runs) but not on the github action runner machine.

[danawillow]

I think the best way to solve this would be to add a fine-grained resource for a single cloud run environment variable. That would also fix the issue of wanting to be able to say that some env vars exist, but not be authoritative about them, which IIRC has been brought up as a concern. @chrisst, does that align with some of the things you've been thinking about?

[chrisst]

Yes I have been thinking that the set up for the cloud run resource itself should be separated from the deploying of an actual revision. I think this scenario and those wanting to deploy from cloud build or other CI's but not have Terraform roll back a revision make it clear that managing both workflows in the same resource isn't ideal. Right now the API doesn't accept an "empty" request to set the cluster up without specifying the initial revision but I'll talk with the cloud run team and see if that's a viable thing to support. Otherwise we can possible set up a "fake" revision that could be overridden with a subsequent deploy.

[rbong]

It appears that setting ENDPOINTS_SERVICE_NAME is no longer the favored way of passing the OpenAPI config to ESPv2.
See this error.

If you wish to skip this step, please specify the name of the
service in the ENDPOINTS_SERVICE_NAME environment variable.
Note this deployment mode is not officially supported.

The recommended way of deploying is to customize and build the docker image with this script, and bake the OpenAPI config into the docker image, rather than having the image download the config from endpoints.

The new steps are linked in the originally linked guide for using endpoints with functions.

I think this thread is still relevant, because even using the new approach, you still have to build the docker image after the endpoint service is created.

[rbong]

I've got something working for the new instructions, the new workaround follows the same pattern but is more complicated:

resource "google_cloud_run_service" "api_function_proxy" {
  name     = "my-api-function-proxy"
  location = var.region

  template {
    spec {
      containers {
        # Default to the unmodified image on first deploy
        image = "gcr.io/endpoints-release/endpoints-runtime-serverless:2"

        env {
          name  = "ESPv2_ARGS"
          value = "--cors_preset=basic"
        }
      }
    }
  }

  lifecycle {
    # Image is set automatically by api_function_proxy_image resource
    ignore_changes = [
      template[0].spec[0].containers[0].image
    ]
  }
}

resource "null_resource" "api_function_proxy_image" {
  # Always trigger since we have no reliable way to check if the build process changed
  triggers = {
    time = timestamp()
  }

  provisioner "local-exec" {
    command = "make docker-deploy-espv2"

    environment = {
      GCP_PROJECT         = var.project
      GCP_REGION          = var.region
      ENDPOINTS_HOST      = google_endpoints_service.api.dns_address
      ENDPOINTS_CONFIG_ID = google_endpoints_service.api.config_id
      ESPV2_SERVICE_NAME  = google_cloud_run_service.api_function_proxy.name
    }
  }
}

Where make docker-deploy-espv2 submits the docker image just like the Google ESPv2 build script made for this purpose and also updates the cloud run image:

gcloud run deploy "${ESPV2_SERVICE_NAME}" \
  --project="${GCP_PROJECT}" \
  --platform=managed \
  --region="${GCP_REGION}" \
  --image="${ESPV2_IMAGE}"

You could probably make the trigger for the image a lot more intelligent, but I don't really want to risk not running it.

Definitely not ideal, the recommended approach now is arguably worse for Terraform because it can no longer be fixed by adding a new environment variable resource. Hopefully Google provides a better solution soon.

[neurolabs]

@rbong Thanks for sharing your approach. In that approach, can't we use google_endpoints_service.api.config_id as a trigger, because it will at least change when the config of the endpoints service changes? That won't cover updates to the base docker image, though.

[rbong]

@neurolabs that would work except for the docker image as you've noted.

[neurolabs]

Thanks, @rbong for the feedback.

In the hacks section of my head, I am pondering including another trigger part based on substr(timestamp()) that would force deployment every month and therefore at least reasonably current images ;).

[rbong]

@neurolabs my approach, which I didn't include because it's hacky and subjective, is to hash every file in the espv2 docker image build directory, use it for the tag name, and then check if the tag name exists before pushing it with gcloud build.

This way, if the image changes for any reason, or you update the base image, it redeploys.

API_ESPV2_IMAGE="gcr.io/${GCP_PROJECT}/endpoints-runtime-serverless:${ENDPOINTS_CONFIG_ID}-`cat build/* | md5sum | cut -d ' ' -f 1`"

// ...

if gcloud container images describe "${API_ESPV2_IMAGE}" &> /dev/null; then
  echo "Image ${API_ESPV2_IMAGE} already exists, not rebuilding"
else
  echo "Submitting ESPv2 image"
  gcloud builds submit --tag "${API_ESPV2_IMAGE}" --project "${GCP_PROJECT}" build/
fi

The gcloud run deploy command is executed regardless of if the image exists or not since the Run image is ignored by Terraform.