In this walkthrough we'll enable TLS encryption between two services in App Mesh using X.509 certificates packaged with your Envoy container. This walkthrough will be a simplified version of the Color App Example.

In App Mesh, traffic encryption works between Virtual Nodes and Virtual Gateways, and thus between Envoys in your service mesh. This means your application code is not responsible for negotiating a TLS-encrypted session, instead allowing the local proxy to negotiate and terminate TLS on your application's behalf.

In this guide, we will be configuring App Mesh to use the file based strategy of providing TLS certificates to Envoy.

1. Install Docker. It is needed to build the demo application images.
2. The unix command line utility jq. If you don't already have it, you can install it from here.

We'll start by setting up the basic infrastructure for our services. All commands will be provided as if run from the same directory as this README.

Next, we need to set a few environment variables before provisioning the infrastructure. Please change the value for AWS_ACCOUNT_ID and ENVOY_IMAGE below.

export AWS_ACCOUNT_ID=<your account id>
export AWS_DEFAULT_REGION="us-west-2"
export ENVIRONMENT_NAME="AppMeshTLSExample"
export MESH_NAME="ColorApp-TLS"
export ENVOY_IMAGE=<get the latest from https://docs.aws.amazon.com/app-mesh/latest/userguide/envoy.html>
export SERVICES_DOMAIN="default.svc.cluster.local"
export COLOR_TELLER_IMAGE_NAME="colorteller"
export COLOR_APP_ENVOY_IMAGE_NAME="colorapp-envoy"

You'll need a keypair stored in AWS to access a bastion host. You can create a keypair using the command below if you don't have one. See Amazon EC2 Key Pairs.

Export the name of the keypair:

export KEY_PAIR_NAME=<name of the key pair to use>

If you are creating a new keypair, run these commands:

aws ec2 create-key-pair --key-name $KEY_PAIR_NAME | jq -r .KeyMaterial > ~/.ssh/$KEY_PAIR_NAME.pem
chmod 400 ~/.ssh/$KEY_PAIR_NAME.pem

This command creates an Amazon EC2 Key Pair and saves the private key at ~/.ssh/${KEY_PAIR_NAME}.pem.

Now that you have your key pair set up, create the VPC.

./infrastructure/vpc.sh

Next, create the ECS cluster and ECR repositories.

./infrastructure/ecs-cluster.sh
./infrastructure/ecr-repositories.sh

Next, build and deploy the color app image.

./src/colorteller/deploy.sh

Note that the example apps use go modules. If you have trouble accessing https://proxy.golang.org during the deployment you can override the GOPROXY by setting GO_PROXY=direct

GO_PROXY=direct ./src/colorteller/deploy.sh

Before we can encrypt traffic between services in the mesh, we need to generate our certificates.

For this demo, we are going to set up two separate Certificate Authorities. The first one will be used to sign the certificate for the White Color Teller, the second will be used to sign the certificate for the Green Color Teller.

./src/tlsCertificates/certs.sh

This generates a few different files

• *_cert.pem: These files are the public side of the certificates
• *_key.pem: These files are the private key for the certificates
• *_cert_chain: These files are an ordered list of the public certificates used to sign a private key
• ca_1_ca_2_bundle.pem: This file contains the public certificates for both CAs.

You can verify that the White Color Teller certificate was signed by CA 1 using this command.

openssl verify -verbose -CAfile src/tlsCertificates/ca_1_cert.pem  src/tlsCertificates/colorteller_white_cert.pem

We are going to store these certificates in AWS Secrets Manager. This will allow us to securely retrieve them at a later time.

./src/tlsCertificates/deploy.sh

Finally, we can build and deploy our custom docker image. This container has a /keys directory and a custom startup script that will pull the necessary certificates from AWS Secrets Manager.

./src/customEnvoyImage/deploy.sh

Our service will contain three components, a Virtual Gateway that will split requests between a White Color Teller and a Green Color Teller. Initially, we will only serve traffic to the White Color Teller.

Let's create the mesh.

./mesh/mesh.sh up

The Virtual Node spec for the white colorteller looks like:

ColorTellerWhiteVirtualNode:
  Type: AWS::AppMesh::VirtualNode
  Properties:
    MeshName: !GetAtt Mesh.MeshName
    VirtualNodeName: ColorTellerWhite
    Spec:
      Listeners:
        - PortMapping:
            Port: 80
            Protocol: http
          HealthCheck:
            Protocol: http
            Path: /ping
            HealthyThreshold: 2
            UnhealthyThreshold: 3
            TimeoutMillis: 2000
            IntervalMillis: 5000
          TLS:
            Mode: STRICT
            Certificate:
              File:
                CertificateChain: "/keys/colorteller_white_cert_chain.pem"
                PrivateKey: "/keys/colorteller_white_key.pem"
      ServiceDiscovery:
        DNS:
          Hostname: !Sub "colorteller.${ServicesDomain}"

The tls block specifies a filepath to where the Envoy can find the certificates it expects. In order to encrypt the traffic, Envoy needs to have both the certificate chain and the private key.

Now with the mesh defined, we can deploy our service to ECS and test it out.

./infrastructure/ecs-service.sh

Let's issue a request to the color gateway.

COLORAPP_ENDPOINT=$(aws cloudformation describe-stacks \
    --stack-name $ENVIRONMENT_NAME-ecs-service \
    | jq -r '.Stacks[0].Outputs[] | select(.OutputKey=="ColorAppEndpoint") | .OutputValue')
curl "${COLORAPP_ENDPOINT}/color"

You should see a successful response with the color white.

Finally, let's log in to the bastion host and check the SSL handshake statistics.

BASTION_IP=$(aws cloudformation describe-stacks \
    --stack-name $ENVIRONMENT_NAME-ecs-cluster \
    | jq -r '.Stacks[0].Outputs[] | select(.OutputKey=="BastionIP") | .OutputValue')
ssh -i ~/.ssh/$KEY_PAIR_NAME.pem ec2-user@$BASTION_IP
curl -s http://colorteller.default.svc.cluster.local:9901/stats | grep ssl.handshake

You should see output similar to: listener.0.0.0.0_15000.ssl.handshake: 1, indicating a successful SSL handshake was achieved between gateway and color teller.

Check out the TLS Encryption documentation for more information on enabling encryption between services in App Mesh.

Enabling TLS communication from your virtual node is the first step to securing your traffic. In a zero trust system, the Color Gateway should also be responsible for defining what certificate authorities are trusted. App Mesh allows you to configure Envoy with information on what CAs you trust to vend certificates. We will demonstrate this by adding a new color teller to our service that has a TLS certificate vended from a different CA than the first.

We previously added two color tellers, the White Color Teller and the Green Color Teller. The TLS configuration for the Green Color Teller looks almost identical to the White, but now we are using the colorteller_green related certificates.

TLS:
  Mode: STRICT
  Certificate:
    File:
      CertificateChain: "/keys/colorteller_green_cert_chain.pem"
      PrivateKey: "/keys/colorteller_green_key.pem"

The route we have currently, only routes to the White Color Teller. We will also need to update the to serve traffic to both color tellers:

HttpRoute:
  Action:
    WeightedTargets:
      - VirtualNode: !GetAtt ColorTellerWhiteVirtualNode.VirtualNodeName
        Weight: 1
      - VirtualNode: !GetAtt ColorTellerGreenVirtualNode.VirtualNodeName
        Weight: 1

Let's update our mesh

./mesh/mesh.sh addGreen

After a couple seconds, when you hit the service, you should see both green and white returned. Note, you may have to call it a few times.

curl "${COLORAPP_ENDPOINT}/color"

As you just saw, we were able to add a new Virtual Node with TLS to our mesh and the Virtual Gateway was able to communicate with it no problem.

In the client/server relationship, if the server decides to turn on TLS, App Mesh configures the client Envoys to accept the certificate offered. However, clients should also validate that the certificate offered by the server is from a certificate authority they trust. App Mesh allows you to define a client policy for TLS validation to ensure that the certificate is valid and issued from a trustworthy source.

If you recall, the Green Color Teller certificates were signed by a different CA than the White Color Teller certificates. Perhaps this is not the intended behavior and we want to reject certificates from any CA that is not CA 1.

We are going to update the Virtual Gateway backend to have this configuration:

BackendDefaults:
  ClientPolicy:
    TLS:
      Enforce: !If [EnableClientValidation, True, False]
      Validation:
        Trust:
          File:
            CertificateChain: "/keys/ca_1_cert.pem"

In this situation, we add a backend default for the Client Policy that instructs Envoy to only allow certificates signed by CA 1 to be accepted.

./mesh/mesh.sh updateGateway

Now when call the service, you will see white is working properly, but you will start to see upstream connect error or disconnect/reset before headers. reset reason: connection failure from the Green Colorteller.

curl "${COLORAPP_ENDPOINT}/color"

We can restore communication by changing the certificateChain in the backend group to be ca_1_ca_2_bundle.pem. This contains both the public certificates for CA 1 and CA 2, which will instructs Envoy to accept certificates signed by both CA 1 and CA 2.

./mesh/mesh.sh updateGateway2

Now when you call the service, you will see both white and green again.

curl "${COLORAPP_ENDPOINT}/color"

If you want to keep the application running, you can do so, but this is the end of this walkthrough. Run the following commands to clean up and tear down the resources that we’ve created.

aws cloudformation delete-stack --stack-name $ENVIRONMENT_NAME-ecs-service
aws cloudformation delete-stack --stack-name $ENVIRONMENT_NAME-ecs-cluster
aws ecr delete-repository --force --repository-name $COLOR_TELLER_IMAGE_NAME
aws ecr delete-repository --force --repository-name $COLOR_APP_ENVOY_IMAGE_NAME
aws cloudformation delete-stack --stack-name $ENVIRONMENT_NAME-ecr-repositories
aws cloudformation delete-stack --stack-name $ENVIRONMENT_NAME-vpc

Delete the mesh.

aws cloudformation delete-stack --stack-name $ENVIRONMENT_NAME-mesh

And finally delete the certificates.

./src/tlsCertificates/cleanup.sh