Kubernetes on Azure

What is Kubernetes ?

Kubernetes, also known as K8s, is an

open-source system for automating
deployment, scaling, and management of
containerized applications.

Kubernetes builds upon 15 years of

experience of running production
workloads at Google, combined with
best-of-breed ideas and practices from
the community.
Kubernetes Architecture
Kubernetes Workflow
Preview Feature: Azure CNI Overlay
Preview Feature: Limitation of Azure CNI Overlay

● You can't use Application Gateway as an Ingress Controller (AGIC) for an Overlay cluster.
● Windows support is still in Preview
○ Windows Server 2019 node pools are not supported for Overlay
○ Traffic from host network pods is not able to reach Windows Overlay pods.
● Virtual Machine Availability Sets (VMAS) are not supported for Overlay
● Dualstack networking is not supported in Overlay
● You can't use DCsv2-series virtual machines in node pools. To meet Confidential Computing requirements,
consider using DCasv5 or DCadsv5-series confidential VMs instead.
Preview Feature: Backup AKS using Azure Backup
Preview Feature: Backup AKS using Azure Backup (Cont.)
Preview Feature: Limitation of Backup AKS using Azure Backup

● AKS backup supports Azure Disk-based persistent volumes (enabled by CSI driver) only
● AKS backup data is stored in your tenant only (The Backup vault and the AKS cluster should be in the same
region and subscription)
● AKS backup supports once-a-day backup only. It also supports more frequent backups (in every 4, 8, and 12
hours intervals) per day
● AKS backup uses the AKS cluster's managed system identity to perform backup operations. So, an AKS
backup doesn't support AKS clusters that use a service principal
Provisioning AKS using Terraform (IaC)

● Create an AAD AKS admin group named “AKSAdmin” and assign AKS admin users to the group
Provisioning AKS using Terraform (IaC) (Cont.)

● Copy the group object Id to be filled in Terraform script

Provisioning AKS using Terraform (IaC) (Cont.)

● Clone the Terraform script from Git with the following command:

$ git clone https: github.com/dmakeroam/aks-iac.git

● Fill in “object id” from previous step, adjust variables in dev.tfvars as needed, and run the following command
to create an AKS cluster

$ terraform apply -var-file=dev.tfvars

Provisioning AKS using Terraform (IaC) (Cont.)

● Assign AKS service role permissions “Azure Kubernetes Service Cluster Admin Role”, “Azure Kubernetes
Service Contributor Role” to the AAD AKS admin group to provide you access to the admin credentials of the
AKS cluster
Provisioning AKS using Terraform (IaC) (Cont.)

● Let’s connect to the AKS cluster

$ az aks get-credentials resource-group

cloudnc-cloudnative-develop-rg name cloudnc-cloudnative-develop-aks
–-admin

$ kubelogin convert-kubeconfig -l azurecli

Recap Kubernetes Concepts - Pods
Recap Kubernetes Concepts - Pods Design
Recap Kubernetes Concepts - Pods Configuration
Recap Kubernetes Concepts - Pods Configuration, .NET

Walkthrough .NET 6.0 Development

Recap Kubernetes Concepts - Pods Configuration, .NET (Cont.)

● Preparing a Dockerfile

FROM mcr.microsoft.com/dotnet/sdk:6.0-alpine-amd64 AS build-env

WORKDIR /app

# Copy csproj and restore as distinct layers

COPY *.csproj ./
RUN dotnet restore

# Copy everything else and build

COPY . ./
RUN dotnet publish -c Release -o out

# Build runtime image

FROM mcr.microsoft.com/dotnet/aspnet:6.0-alpine-amd64
WORKDIR /app
RUN apk no-cache add curl
COPY from=build-env /app/out .
EXPOSE 8090
ENTRYPOINT ["dotnet", "dotnet-core-web-api.dll"]
Recap Kubernetes Concepts - Pods Configuration, .NET (Cont.)

● Preparing an App Settings to be loaded into .NET pod via config map

generate app settings

config map
Recap Kubernetes Concepts - Pods Configuration, .NET (Cont.)

● Run “docker build” to construct a container image from the Dockerfile and push the image to your Docker Hub

$ docker build -t dmakeroam/webapp:1.0.0 .

$ docker push dmakeroam/webapp:1.0.0

Recap Kubernetes Concepts - Pods Configuration, .NET (Cont.)

● Preparing .NET Pod Configuration (pod.yaml)

config map volume for mouting app settings config map

specify container image

specify container environment variables

liveness probe make sure when the

app failed, the app can self restarted

readiness probe make sure when the

app ready to accept user requests

mount the config map volume to .NET pod

Recap Kubernetes Concepts - Pods Configuration, .NET (Cont.)

● Apply the .NET pod (deployment) and see the result

config map from kustomize

$ kubectl apply -f config.yaml -f pod.yaml -n [namespace]

$ kubectl logs -f webapp -n [namespace]

Recap Kubernetes Concepts - Pods Limitation

● It is a manual task to scale our app on demand

● We cannot make sure availability of our app
● Pods don’t support upgrading container images (app versions) with no downtime
Recap Kubernetes Concepts - Deployment

How many replicas for your app?

How to rolling update the deployment?

Pod Definition
Recap Kubernetes Concepts - Deployment Configuration

● Adjust and Apply the .NET deployment and see the result
config map from kustomize

$ kubectl apply -f config.yaml -f deployment.yaml -n [namespace]

$ kubectl logs -f -l app=dotnet-core-web-api -n [namespace]

Storing Application Credentials on Azure Key Vault
How Workload Identity Works?
Enabling Workload Identity to Existing AKS Cluster (IaC)

● Add the following code (line 14-15) and run “terraform apply” again
Adding Application Credentials to Azure Key Vault

● Create a key vault

Adding Application Credentials to Azure Key Vault (Cont.)

● Grant key vault admin permissions to the AAD AKS admin group
Adding Application Credentials to Azure Key Vault (Cont.)

● Add app credentials to the key vault

Setup Azure Key Vault Provider

● Upgrade the existing AKS Cluster with Azure Key Vault Provider Support
Configuring App Identity for Reading Secrets

● Create a managed identity to be used for reading secrets

$ az identity create –-name cloudnckv-user –-resource-group

cloudnc-cloudnative-secret-develop-rg

● Assign key vault reader permissions to the identity

Configuring App Identity for Reading Secrets (Cont.)

● Create an app service account for the app identity

$ cat <<EOF | kubectl apply -f -

apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
azure.workload.identity/client-id: ${USER_ASSIGNED_CLIENT_ID}
labels:
azure.workload.identity/use: "true"
name: cloudnckv-user
namespace: default

EOF
Configuring App Identity for Reading Secrets (Cont.)

● Federated between the app identity and the app service account

$ az identity federated-credential create --name cloudnckv-user-fed --identity-name

cloudnckv-user --resource-group $RESOURCE_GROUP --issuer ${AKS_OIDC_ISSUER} --subject
system:serviceaccount:dmakeroam:cloudnckv-user
Configuring Secret Provider Class for Syncing Secrets

● Create a secret provider class

$ cat EOF | kubectl apply -f -

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: cloudnckv-provider
namespace: dmakeroam
spec:
provider: azure
secretObjects:
- secretName: db-secret
type: Opaque
data:
- objectName: username
key: username
- objectName: password
key: password
parameters:
usePodIdentity: "false"
useVMManagedIdentity: "false"
clientID: "${USER_ASSIGNED_CLIENT_ID}"
keyvaultName: "cloudnckv"
objects: |
array:
- |
objectName: username
objectType: secret
- |
objectName: password
objectType: secret
tenantId: "3312c148-5eaf-4290-8127-df5190853b8d"
EOF
Configuring Secret Provider Class for Syncing Secrets (Cont.)

● Adjust the app deployment to use the secret provider and read environment variables

read db secrets and map to the app environment variables

tell the deployment to use the secret provider

Testing Syncing Secrets for the App
Recap Kubernetes Concepts - Service
Use of Application Gateway for the AKS Cluster
How App Gateway Ingress Controller (AGIC) works?
Enabling AGIC and App Gateway on Existing AKS Cluster

● Add the following code (line 21-24) and run “terraform apply” again
Configuring app service and Kubernetes Ingress

● Adjust and Apply the .NET service in manifests folder

$ kubectl apply -f service.yaml

● Adjust and Apply the .NET ingress in manifests folder

$ kubectl apply -f ingress.yaml

Testing the App Gateway to App Connection

● Open Browser and go to App Gateway Public IP

Demo: AKS GitOps using Argo CD
Q&A