{"id":79781,"date":"2022-10-11T09:00:00","date_gmt":"2022-10-11T16:00:00","guid":{"rendered":"https:\/\/2.zoppoz.workers.dev:443\/https\/www.cncf.io\/?post_type=lf_case_study&p=79781"},"modified":"2024-02-15T22:15:05","modified_gmt":"2024-02-16T06:15:05","slug":"datadog","status":"publish","type":"lf_case_study","link":"https:\/\/2.zoppoz.workers.dev:443\/https\/www.cncf.io\/case-studies\/datadog\/","title":{"rendered":"Datadog"},"content":{"rendered":"
\n\n
\n\t\n\t\"Bits\n\t<\/picture><\/figure>\n\n
\n\n\n\t<\/span>\n\n

Datadog<\/h1>\n\n<\/div>\n<\/section>\n\n\t\n\n\n\n\n\n\t\tBy the numbers<\/p>\n\t\n\t
\n\n\t\t\n\t\t
\n\n\t\t\t\t\t\t\n\t\t\t\t10 trillion+<\/p>\n\t\t\t\t\t\t\tData points every day<\/p>\n\t\t\t\n\t\t<\/div>\n\n\t\t\n\t\t
\n\n\t\t\t\t\t\t\n\t\t\t\t18,500+<\/p>\n\t\t\t\t\t\t\tCustomers using SaaS product<\/p>\n\t\t\t\n\t\t<\/div>\n\n\t\t\n\t\t
\n\n\t\t\t\t\t\t\n\t\t\t\t10,000+<\/p>\n\t\t\t\t\t\t\tNodes & Pods across multiple cloud providers<\/p>\n\t\t\t\n\t\t<\/div>\n\t<\/div>\n\n<\/section>\n\t\n\n\n

Problem: Limited scalability with iptables, IPVS, and kube-proxy<\/h3>\n\n\n\n

Datadog runs 10s of Kubernetes clusters, across multiple cloud providers. Some of them have up to 4,000 nodes, which makes for 10,000s of hosts in their infrastructure. Datadog\u2019s SaaS product serves over 18,500 customers with millions of hosts reporting in, resulting in 10s of trillions of data points per day. Such a large and growing infrastructure comes with challenges. <\/p>\n\n\n\n

For example, enabling cross-cluster communication in some environments isn’t simple, and ensuring that it is secure and encrypted complicates the process even further. There was also a need for routable pod IPs, which improves performance and enables direct cross-cluster communication often needed to reduce the management burden of data stores, such as Kafka or Cassandra. With many clusters and IPs routed between them, extra care must be taken to properly manage the IP address spaces and cross-cluster service discovery.<\/p>\n\n\n\n

Datadog\u2019s initial solution was to use cloud specific solutions, like Lyft\u2019s CNI plugin<\/a> for AWS. However, that meant finding additional solutions for other cloud providers. In addition, maintaining very different approaches and technologies across multiple clouds\u2014many without operators to automate tasks\u2014and managing provider and solution differences was time intensive. These solutions were not able to address the need for streamlined network policies nor did they meet the requirement for traffic encryption.<\/p>\n\n\n\n

Another challenge was service load balancing. The usual approach is to use iptables, but as the environment grows, the rule count also grows, making this difficult to scale. This can increase update time\u2014sometimes more than 10 seconds\u2014as you need to methodically review all of the rules that you are interested in (“matching time”). In the words of Laurent Bernaille, Staff Engineer at Datadog:<\/p>\n\n\n\n

\n

\u201cWhen you scale, and you have a large number of services and endpoints, iptables becomes challenging. We can use iptables for load balancing, but it was not designed for it.\u201d <\/em><\/p>\n Laurent Bernaille, Staff Engineer, Datadog<\/cite><\/blockquote>\n\n\n\n

As an alternative, they tried IPVS<\/a>, which was more powerful but brought another set of challenges, especially given IPVS was not designed for client-side load balancing. Connection tracking had to be done twice, once for IPVS and once for netfilter, and it lacked feature parity with iptables. <\/p>\n\n\n\n

The pain of using these solutions grew even more intense because they were difficult to debug, and it could take a lot of time between finding issues and getting the solution into production. As a result, the team asked itself: \u201cWhat if we could dynamically program these features?\u201d<\/p>\n\n\n\n

Solution: Cilium and eBPF powering the Kubernetes networking data plane<\/h3>\n\n\n\n

Datadog went to KubeCon + CloudNativeCon<\/a> and spoke with other teams in the hallway track to find out what solutions they were considering to solve their scaling problems, and many of them were considering Cilium. When they spoke with their development team, all the features that Datadog found interesting were already in the works or almost done. They started using some of the features in early beta, including putting v1.6 into production, and gave feedback to the team.<\/p>\n\n\n\n

\n

\u201cOf course we saw some small issues and edge cases, but our interaction with upstream has always been very good and we have engineers who hadn\u2019t contributed before that were quickly able to become contributors.\u201d <\/p>\n Laurent Bernaille, Staff Engineer, Datadog<\/cite><\/blockquote>\n\n\n\n

As part of the new architecture, Cilium was used to replace kube-proxy, addressing the growing pains with iptables and IPVS. Leveraging eBPF\u2019s efficiency also allowed Datadog to enforce network policies, which increased their security and was required for a certification and in regulated environments. Cilium has now become the default CNI for Kubernetes at Datadog and runs on almost every node of their SaaS offering. This allows them to abstract cloud providers and have a consistent data plane network. <\/p>\n\n\n\n

\n

\u201cAs the universal default Kubernetes CNI, Cilium can be used on multiple clouds, allowing network policies to be enforced across clouds where they are needed.\u201d <\/p>\n Laurent Bernaille, Staff Engineer at Datadog<\/cite><\/blockquote>\n\n\n\n

Beyond the SaaS offering, Datadog also has an integration with Cilium where their customers can send Cilium\u2019s metrics and logs into Datadog. The team is also always looking for ways to improve both their infrastructure and Cilium. They are currently working on a few improvements around supporting multiple CIDR ranges in Cilium and are testing Bandwidth Manager<\/a> to guarantee better throughput.<\/p>\n\n\n\n

\n

“eBPF and Cilium helped us to push the boundaries both within operations and also with product development. To do things safer, faster and more easily than what we could have with traditional techniques.” <\/p>\n Laurent Bernaille, Staff Engineer, Datadog<\/cite><\/blockquote>\n\n\n\n

Datadog has started using Envoy for more use cases and is very interested in the sidecarless approach developed in the context of Cilium Service Mesh<\/a>. They are also looking outside the cluster and considering using eBPF to create a smarter and more efficient network edge in the future. Using Cilium L4LB XDP<\/a> to create their own load balancers rather than relying upon a cloud provider would allow them to provide a consistent experience to end users.<\/p>\n\n\n\n

\n

\u201cThe overlay features in a single product, compatibility with multiple cloud providers, and ability to just run it. These three things are what made Cilium an obvious choice for us.\u201d <\/p>\n Laurent Bernaille, Staff Engineer, Datadog<\/cite><\/blockquote>\n\n\n\n

\n