ECSからEKSへの移行への移行事例の紹介

Transcript

1. &$4͔Β&,4΁ ͷҠߦࣄྫͷ঺հ !@NQPO
   +"846(
   ίϯςφࢧ෦
   

2. Masato Oshima github.com/mpon @_mpon Software Engineer

3. ίϯςφࢧ෦ͷൃද͸3ճ໨Ͱ͢ 2017/12 2018/12

4. ίϝϯτ΍࣭໰ੋඇ͓ئ͍͠·͢ ➤ ਖ਼௚ɺ΋ͬͱ͍͍ํ๏͋Δ͔΋ͱ͍͏ෆ҆ɾɾ ➤ ͳͷͰποίϛ΋Β͑Δͱخ͍͠Ͱ͢ʂ ➤ ͦΜͳ͜ͱ͠ͳͯ͘΋ࠓͳΒ͜ΕͰͰ͖Δͷʹɻͱ͔ ➤ ͬͪ͜ͷπʔϧ࢖͑͹Α͔ͬͨͷʹɻͱ͔ ➤

   ͳΜͰ͜͏͍͏ߏ੒ʹͨ͠ͷʁͱ͔ ➤ ͜͏͍͏έʔεͬͯରԠͰ͖ͯ·͔͢ʁͱ͔ ➤ ͕͜͜Α͘෼͔Βͳ͔ͬͨɻͱ͔

5. AGENDA ➤ ελσΟαϓϦENGLISHʹ͍ͭͯ ➤ ୈҰ෦: ECS͔ΒEKS΁Ҡߦͷഎܠ ➤ ՝୊1: ؀ڥ૿΍͢ͷ͕ਏ͔ͬͨ ➤

   ՝୊2: Ͳ͜ʹԿ͕σϓϩΠ͞ΕͯΔ͔೺Ѳ͠ʹ͔ͬͨ͘ ➤ ՝୊3: gRPCͷෛՙ෼ࢄ ➤ ୈೋ෦: EKSҠߦޙͷߏ੒ʹ͍ͭͯ ➤ Spot OceanΛ࢖ͬͨΫϥελʔ؅ཧ ➤ JenkinsͷJobΛCronJobʹҠߦʢFargateͷ࿩΋͋ΔΑʣ ➤ ·ͱΊɿίϯςφ͸͍͍ͧ

6. ελσΟαϓϦENGLISH ➤ ΦϯϥΠϯӳޠֶशαʔϏε ➤ ӳձ࿩Ի੠΍ը૾ɾಈըͳͲΛ഑৴ ➤ ΫΠζܗࣜͰֶशσʔλɺཤྺΛอଘ ➤ iOS, Android,

   WebͰར༻Մೳ

7. αʔϏεߏ੒ͷ֓ཁ

8. αʔϏεߏ੒ͷ֓ཁ ໿50αʔϏε ɾɾɾ ɾ ɾ ɾ dev1, dev2... ໿20؀ڥ

9. ୈҰ෦ ECS͔ΒEKS΁Ҡߦ͠ ͨഎܠʹ͍ͭͯ

10. ՝୊1 ؀ڥ૿΍͢ͷ͕ਏ ͔ͬͨ

11. ECSͷͱ͖ͷߏ੒؅ཧํ๏ ෼ྨ ίϯϙʔωϯτ ઃఆํ๏ Ծ૝Ϋϥελʔ ECS Cluster Terraform Πϯελϯε Autoscaling

    Group Terraform αʔϏεεέδϡʔϥ ECS Service Terraform αʔϏεσΟεΧόϦ Cloud Map Terraform ίϯςφఆٛ ECS Task Definition Terraform + ಠࣗπʔϧ ϩʔυόϥϯαʔ ALB Terraform αʔϏεͻ΋͚ͮ ALB Target Group Terraform ϧʔςΟϯά ALB Listenr Rule Terraform

12. ECSͰ؀ڥΛͨ͘͞Μ૿΍͢ʹ͸ʁ ➤ TerraformͰECS ClusterɺECS ServiceɺALBͳͲ৭ʑ࡞Δ ➤ Task Definitionʢಠࣗπʔϧͷ࢓༷ͷYAMLʣΛେྔʹίϐʔ ➤ ςϯϓϨʔτͷػೳ͕ͳ͘؀ڥ͝ͱʹ؀ڥݻ༗ͷ஋ʹॻ͖׵͑Δ

    ➤ ؀ڥ૿΍͢ͷʹTerraformͰΠϯελϯε΍ALBΛ૿΍͞ͳ͍ͱ͍͚ͳ͍ ➤ Terraform͸ModuleԽ͸͚ͨ͠Ͳɺ͋ͱ͔ΒมߋೖΕΔͱޓ׵ੑอͭͷ ʹେมͳ͜ͱ͕͋ͬͯؾܰʹmoduleΛมߋͰ͖ͳ͍ ➤ σϓϩΠͰมߋ͕ى͖Δ΋ͷΛTerraformͰ؅ཧͯ͠͠·͍ͬͯͨͷͰ ignore_changesͰແཧ΍Γແࢹ ➤ ϥΠϑαΠΫϧͷҧ͍ΛݟۃΊΒΕ͍ͯͳ͔ͬͨɾɾ

13. EKSͷߏ੒؅ཧํ๏ ෼ྨ ίϯϙʔωϯτ ઃఆํ๏ Ծ૝Ϋϥελʔ Namespace YAML(k8s) Πϯελϯε EC2Πϯελϯε Terraform

    + Spot Ocean αʔϏεεέδϡʔϥ Deployment YAML(k8s) αʔϏεσΟεΧόϦ Service YAML(k8s) ίϯςφఆٛ Pod YAML(k8s) ϩʔυόϥϯαʔ Ingress YAML(k8s) αʔϏεͻ΋͚ͮ Ingress YAML(k8s) ϧʔςΟϯά Ingress YAML(k8s)

14. EKSͰ؀ڥΛͨ͘͞Μ૿΍͢ʹ͸ʁ ➤ Terraformͷग़൪͸EKS ClusterΛ࠷ॳʹ࡞Δ͚ͩͰ؀ڥ૿΍͢ͱ ͖ʹ͸͍Βͳ͍ ➤ ؀ڥ૿΍͢৔߹͸NamespaceΛ૿΍͢ ➤ KustomizeΛ࢖ͬͯ؀ڥ͝ͱʹมΘΔ෦෼͚ͩΛ௥Ճ ➤

    ֎෦͔ΒͷϦΫΤετͷϚοϐϯά͸Ingress ➤ ಺෦௨৴͸ServiceΦϒδΣΫτͰ؆୯αʔϏεσΟεΧόϦɺ CoreDNS͕͋ΔͷͰRoute53ͳ͠Ͱ΋಺෦ͰDNS͕༻ҙ ➤ ΄ͱΜͲKubernetesͷΦϒδΣΫτͰ׬݁͢ΔͷͰએݴతʹ؅ཧ ͠΍͍͢

15. ΫϥελʔΛ࿦ཧతʹ෼ׂͰ͖Δ Namespace dev1 EC2 EC2 EC2 Cluster EC2 EC2 EC2

    Namespace dev2 Namespace dev3 ➤ Kubernetesʹ͸Namespace ͱ͍͏ΦϒδΣΫτ͕͋Δ ➤ Cluster͑͞࡞ͬͯ͠·͑͹ ͋ͱ͸NamespaceΛ૿΍͢ ͚ͩͰ؀ڥ͕૿΍ͤΔ ➤ Namespace͝ͱͷDNSΤϯ τϦ΋࡞੒͞ΕΔ ➤ ͜Μͳ୯७ͳYAMLͰOK

16. Kustomize, HelmͳͲσϑΝΫτͳߏ੒؅ཧ͕͋Δ

17. Kustomizeͷoverlays

18. ՝୊2 Ͳ͜ʹԿ͕σϓϩΠ͞Ε ͯΔ͔೺Ѳ͠ʹ͔ͬͨ͘

19. ECSͷͱ͖ʹײ͍ͯͨ͡՝୊ ➤ hoge؀ڥͷfugaαʔϏεͬͯࠓσϓϩΠ͞ΕͯΔͷ͸Ͳͷϒϥϯνʁ ➤ ίϯςφͷ؀ڥม਺͸Կ͕ઃఆ͞ΕͯΔʁ ➤ aws ecs list-servicesͱ͔ͰҰൃͰͲͷtag͕σϓϩΠ͞ΕͯΔ͔೺ѲͰ͖ ͳ͍ͷͰࣗ࡞πʔϧ࡞͙ͬͯ྇

    ➤ σϓϩΠͷઃఆʢJenkinsfileʣʹઃఆ͕ͲΜͲΜ૿͍͑ͯͬͯΧΦεʹ ͳ͖ͬͯͨ ➤ Α͘σϓϩΠ͢ΔAPIͳͲͷαʔόʔΞϓϦέʔγϣϯͱ୯ͳΔnginxͱ ͔ͷσϓϩΠ͸σϓϩΠํ๏͕ผʑʹ༻ҙͯ͠ϝϯς͍ͯͨ͠ ➤ gitϦϙδτϦΛݟΕ͹Կ͕σϓϩΠ͞ΕͯΔ͔෼͔ΔΑ͏ʹ͍ͨ͠ ➤ gitops͍ͨ͠

20. GITOPS͍ͨ͠

21. gitopsπʔϧͱͯ͠Argo CDͷಋೖ https://tech.recruit-mp.co.jp/infrastructure/gitops-cd-by-using-argo-cd-at-eks/

22. Argo CDͷArchitecture - k8sͷReconciliation Loop gitϦϙδτϦΛ ݟΕ͹Կ͕σϓϩΠ͞Ε ͍ͯΔ͔೺ѲͰ͖Δ gitϦϙδτϦͱ ࠩ෼Λఆظతʹ

    νΣοΫͯ͠ ࠩ෼͕͋Ε͹ σϓϩΠ

23. Dashboard͕͋ΔͷͰDeveloper Friendly GUI͕͋Δͱ։ൃऀʹ΋ ֓೦Λཧղͯ͠΋Β͍΍͍͢

24. Sync Phase and WavesͰσϓϩΠͷґଘؔ܎ΛදݱͰ͖Δ

25. Clusterʹඞཁͳresource΋gitopsͰ ➤ Ingress Controller΍ Datadog AgentͳͲ΋؅ཧ Ͱ͖Δ ➤ Helm Chart

    Repositoriesͷ ػೳ͕͋ΔͷͰArgo CDͰ Helm ChartΛιʔεʹ؅ཧ Մೳ ➤ ArgoCD ApplicationΛ࡞੒ ͢Ε͹ߏங׬ྃ

26. ެࣜπʔϧͷkubectlͰ͙͢ʹ೺ѲͰ͖Δ ➤ ྫ͑͹ͲͷImage͕σϓϩΠ͞ΕͯΔ͔֬ೝ͢Δͷʹ͜Ε͚ͩͰOK ➤ `kubectl get deploy -o wide`

27. ՝୊3 GRPCͷෛՙ෼ࢄ

28. ౰࣌ͷgRPCͷෛՙ෼ࢄ https://tech.recruit-mp.co.jp/infrastructure/post-17098/

29. ECSͷͱ͖ͷߏ੒

30. σϓϩΠ࣌ʹϦΫΤετ͕མͪΔ໰୊ Blue gRPC Green gRPC Blue gRPC Green gRPC blue.grpc.internal

    green.grpc.internal Cloud Map lb.grpc.internal ➤ grpcαʔόʔΛRolling Update͢Δ ͱSTOPࡁΈͷίϯςφͷIP͕ฦͬ ͖ͯͯϦΫΤετ͕མͪͯ͠·͏ ➤ JenkinsͷpipelineͱshΛ૊Έ߹Θ ͤͯBlue/GreenσϓϩΠ ➤ ECS Task Definitionͷ؀ڥม਺Λ sedͰBlue->Greenʹॻ͖׵͑ͯ envoyΛRolloing Update ➤ Sleep 30͔ͯ͠ΒBlueΛ࡟আ ➤ ϫʔΫΞϥ΢ϯυײຬࡌͳͷͰͳ Μͱ͔͔ͨͬͨ͠

31. App Meshͷ࠾༻ https://tech.recruit-mp.co.jp/infrastructure/post-20765/

32. App Meshͷ࠾༻ ➤ gRPC routingΛαϙʔτ͍ͯͨ͠ ➤ কདྷతʹαʔϏεϝογϡͱͯ͠׆༻Ͱ͖Δ ➤ ͲͪΒʹ͠Ζenvoyͷ؅ཧ͸୭͔͕΍Δඞཁ͸͋Δ ➤

    envoyͷretryPolicyɺkubernetesͷPod LifecycleΛ׆༻ͯ͠ϦΫ ΤετΛམͱͣ͞ʹσϓϩΠͰ͖Δ

33. App Meshͷߏ੒؅ཧ ➤ AWS App Mesh Controller for k8s͕༻ҙ͞Ε͍ͯΔ ➤

    APIͷίϯςφͷYAMLΛ͍ ͡Δ͜ͱͳ͘envoyΛInject ͯ͘͠ΕΔ ➤ k8sͷCRDͱͯ͠؅ཧͰ͖ ΔͷͰgitopsͰ͖Δ ➤ ։ൃ؀ڥΛ૿΍͢ͷ΋؆୯

34. σϓϩΠ࣌ʹϦΫΤετ͕མͪΔ໰୊ ➤ App Mesh best practicesͱ ͍͏υΩϡϝϯτ͕༻ҙ͞ Ε͍ͯͯͦͷ௨Γʹ RetryPolicyΛઃఆ

35. σϓϩΠ࣌ʹϦΫΤετ͕མͪΔ໰୊ ➤ PodͷpreStopͰsleepΛೖΕΔ͜ͱͰ Service͔Β੾Γ཭͞ΕΔͷΛ଴ͭ ➤ AWSαϙʔτͷํʹΞυόΠεΛ΋ Β͍ղܾ͠·ͨ͠ɻେײँ ➤ AWSαʔϏεͷ͜ͱ͔͠ฉ͍ͪΌ͍ ͚ͳ͍ͱࢥ͚ͬͯͨͲͦΜͳ͜ͱ͸ͳ

    ͔ͬͨ ➤ Kubernetes΍Envoyͷ͜ͱ΋αϙʔτ ͯ͘͠Εͨ ➤ Argo CDʹΑΔSync(Deploymentͷ Rolling Update)Ͱγϯϓϧʹσϓϩ ΠͰ͖ΔΑ͏ʹͳͬͨ

36. ༨ஊ: ਓྨ͕଴๬͍ͯͨ͠ALBͷgRPCαϙʔτʂʂ ΋ͬͱૣ͘ཉ͔ͬͨ͠

37. ୈೋ෦ EKSҠߦޙͷ ߏ੒ʹ͍ͭͯ

38. EKSҠߦޙͷߏ੒ ECS EKS

39. EKSҠߦޙͷߏ੒ ALB Ingress Controllerͱ Nginx Ingrss Controllerͷ ૊Έ߹Θͤ App MeshʹΑΔ

    EnvoyͷInject Cloud MapͰ Service Discovery

40. SPOT OCEANΛ࢖ͬͨ Ϋϥελʔ؅ཧ

41. ৄࡉ͸ͪ͜Βͷϒϩά https://tech.recruit-mp.co.jp/infrastructure/post-19364/

42. Spot Ocean ੲ͸Spotinstͱ͍͏αʔϏεͩͬͨ ࠓ͸NetAppʹങऩ͞ΕͯSpotͱ͍͏໊લʹ

43. Pod-Driven Scaling ➤ Pod͕UnscheduleʹͳΒͳ͍Α͏ʹ NodeΛΦʔτεέʔϦϯάͯ͘͠ ΕΔ ➤ PodʹׂΓ౰ͯΔrequests͚ͩΛߟ ͑Ε͹OK ➤

    Node͸ҙࣝ͠ͳͯ͘Α͍ͷͰ FargateͷΑ͏ͳ࢖͍উख ➤ Spot Instance͔Β࠷దͳΠϯελϯ εΛબΜͰ͘ΕΔ ➤ ΦϯσϚϯυͰಈ͍ͯཉ͍͠podʹ ͸annotation෇༩͢Δ͚ͩͰOK

44. Terraform΍Custom ControllerͰߏ੒؅ཧͰ͖Δ ➤ Terraformͷprovider͕༻ҙ ͞Ε͍ͯΔ ➤ Spot OceanͷCustom Controller͕Helm ChartͰ

    ༻ҙ͞Ε͍ͯΔͷͰ؆୯Π ϯετʔϧ

45. Headroom ➤ Headroom͸༨৒ͷϊʔυ Λ֬อ͓͍ͯͯ͘͠ΕΔػ ೳ ➤ ϦΫΤετͷεύΠΫ౳Ͱ PodͷεέʔϧΛૉૣ͘͢Δ

46. Cluster Roll ➤ Cluster Roll͸Nodeͷ҆શ ͳRolling UpdateΛͯ͘͠Ε Δ ➤ NodeͷೖΕସ͍͑ͨ͠ࣄ৘

    ͕ൃੜͨ͠৔߹ͳͲʹศར

47. JENKINSͷJOBΛ CRONJOBʹҠߦ

48. ৄࡉ͸ͪ͜Β https://tech.recruit-mp.co.jp/infrastructure/post-20631/ https://speakerdeck.com/yutachaos/the-story-of-moving-jenkins-job-to-cronjob

49. όον͸JenkinsͰ΍͍ͬͯͨ ➤ αʔόʔΞϓϦέʔγϣϯ ͱಉ͡ίϯςφImageΛ ࢖ͬͯλεΫΛىಈ ➤ JenkinsͷఆظτϦΨʔͰ࣮ ߦ ➤ ௨஌΋ϦτϥΠ΋ศརɺͳ

    ΜͰ΋Ͱ͖Δɻ͔͠͠ɾɾ

50. Jenkinsศར͗͢໰୊ ➤ ศརա͗ͯWebը໘্Ͱ ͡ΌΜ͡ΌΜδϣϒ͕࡞Β Ε͍ͯ͘ ➤ ҰମԿ͕͍ͭىಈͯ͠Δ͔ Α͘෼͔Βͳ͍ ➤ ාͯ͘updateͰ͖ͣʹԘ௮

    ͚ʹɾɾ

51. k8sͷCronJobʹ͢Δ͜ͱͰYAML؅ཧ ➤ CronJob͸ఆظతʹJobΛ࣮ߦ ͢Δk8sͷϦιʔε ➤ EKSҠߦͷӡ༻͸·ͣδϣϒҠ ߦ͔Βߦ͕ͬͨ͏·͍ͬͨ͘ ➤ YAMLͰ؅ཧͰ͖ΔͷͰgitops Ͱ͖Δ

    ➤ kubectlͰݟͯ΋͍͍͠ɺgitϨ ϙδτϦΛgrep͢Δ͚ͩͰԿ͕ ͍ͭͲ͜ͰJob͕ಈ͍ͯΔ͔͢ ͙ʹ෼͔Δ

52. όον࣮ߦͷͨͼʹNodeͷ૿ݮͯ͠͠·͏ ➤ ఆظతʹόον࣮ߦͷͨͼ ʹPodͷRequestsʹԠͯ͡ Node͕૿ݮ ➤ όον࣮ߦதʹNode͕མͪ ΔͱࠔΔ ➤ όονͷ࣮ߦʹΑͬͯαʔ

    ϏεʹӨڹ͸༩͑ͨ͘ͳ͍

53. EKS FargateͰCronJobΛ࣮ߦ͢Δ ➤ όον͸ίϯςφ͕ىಈ͠ ͯऴྃ͢Ε͹Α͍ͷͰ FargateʹͽͬͨΓ ➤ kubernetes͔Β͸Fargate͸ 1ͭͷnodeͷΑ͏ʹݟ͑Δ ➤

    FargateͰىಈͨ͠όον͕ طଘͷΫϥελʔͷϦιʔ εΛ࢖͏͜ͱ͸ͳ͍

54. Sidecarίϯςφͷఀࢭ͕೉͍͠ ➤ Datadog Agent΍FluentdͳͲΛ Sidecarʹ͓͔͘͠ͳ͍ ➤ JobͷϝΠϯͷίϯςφ͕ఀࢭͯ͠΋ sidecarίϯςφ͸ఀࢭ͠ͳ͍ͷͰJob ͕ऴྃ͠ͳ͍ʂʂ ➤

    ऴྃ࣌ʹsidecarίϯςφΛఀࢭͤ͞ ΔͳͲͷ޻෉͕ඞཁ ➤ ECSͰ͍͏essential:trueͷػೳ͕ͳ͍ ➤ Sidecar Containers͕Kubernetes v1.19ʹೖΔ༧ఆ͕ͩͬͨະఆʹͳͬ ͯ͠·ͬͨʁ

55. ·ͱΊͱࡶஊ ➤ ECS͔ΒEKSϔҠߦͨ͠࿩Λ͠·ͨ͠ ➤ ͋͘·Ͱ΋զʑͷ৔߹ͷ࿩ ➤ gitops΍KubernetesͷΤίγεςϜͰಠࣗ࢓༷Λͳͯ࣋͘͠ଓੑͷ͋ΔγεςϜ΁ ➤ App Mesh͸ຊ൪ར༻΋໰୊ͳ͠

    ➤ ECS͸ࠓͳΒFargate Spot΋͋Δ͠ALB͕gRPCαϙʔτͨ͠͠࠶ߟͯ͠Έ͍ͨ ➤ ecspressoΛ࢖͑͹ϥΠϑαΠΫϧͱςϯϓϨʔτͱ͔΋ͬͱ͏·͘Ͱ͖͔ͨ΋ ➤ ECSͰɺALBͱͷͻ΋͚ͮɺECS ServiceɺTask Definitionͱ͔Λ౷Ұతʹѻ͑Δπʔϧ ͕͋Ε͹͍͍ͷ͔΋ ➤ copilot͕ͦ͏ͳͷ͔ʁ ➤ ίϯςφ͸͍͍ͧ ➤ ͨͩ͠ɺRDSɺClodwatch LogsɺS3ͳͲϚωʔδυαʔϏε͕͋ͬͯͦ͜