Microservice architecture and serverless computing are two popular architectural styles for building cloud-native applications. Both of them support automatic scaling, manage instances in containers and decompose complex applications, making them able to be used to build scalable and resilient applications. However, while microservices let the developers manage the interaction between services, serverless computing takes over the infrastructure management totally and provides an event-driven architecture.
Microservices decompose the monolithic application into a set of loosely coupled services, while also introducing the network overhead of the communication between services. Different from function calls, the communication between services is usually done through the network. When one service calls another service, it needs to serialize the data, send it over the network, wait for the response, deserialize the data in the response, and continue the execution. What’s more, while waiting for the response, the process of callee service blocks execution and still maintains the state of execution in memory. The waiting limits the ability of the service to handle more concurrent requests.
In two cases, this waiting is unnecessary. The first one is that the callee service B returns the response data from service C to service A that calls service B, without any more processing, for example, the confirmation of DB writing. This means that sending the data to service B and deserializing and serializing the data by service B are unnecessary. The second one is that the callee service B only did some simple processes on the response data without using too much or any sensitive context, such as filtering some entries. This waiting is also not necessary if some more APIs are added to service C.
The event-driven architecture in serverless computing avoids the unnecessary overhead of waiting for responses and network communication. Cloud vendors provide various components as event sources and event consumers, provide event distribution and management services, and provide asynchronous and decoupled processing mechanisms. Serverless computing provides immediate results through API Gateway or asynchronous results through various event sources.
Take AWS as an example, event sources include S3, DynamoDB, API Gateway, EventBridge, Simple Queue Service (SQS), Simple Notification Service (SNS), etc. Event consumers include Lambda, Step Functions, ECS, etc. Event distribution and management services include EventBridge, SQS, SNS, etc. Asynchronous and decoupled processing mechanisms include Step Functions, SQS, SNS, EventBridge, etc.
However, the implementation of event-driven computing relies on the modification of application code and the developers’ understanding of the complex workflows. Some other disadvantages, including dependence on specific cloud platforms and the difficulty of migrating stateful applications, also prevent the adoption of serverless.
By the way, developers can benefit from event-driven architecture in microservice. Vert.x, a tool-kit for building reactive applications provided by Eclipse, also uses event-driven API to save resources and reach higher throughput.
]]>As interested in this important problem, I have surveyed it and tried to figure out some research opportunities. Unfortunately, many of them are done and must be done by companies or the cooperation of companies and academic institutes, because of the limitation of characterizing requests’ features, finding practical problems, and validating new ideas in laboratories. However, these works give valuable insights such as how to improve large-scale systems and handle indeterministic workload.
This article is aimed to summarize some research works and record some of my thoughts about this problem. If there are any errors in this article, feel free to contact me to correct them and I will appreciate it.
To abstract and define the problem of resource management, I would like to have a brief introduction of Cloud Computing first.
There are three main cloud computing models, that is Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). Public cloud providers provide multiple kinds of products in IaaS and PaaS, including:
| Product Type | Product Example | Computing Model |
|---|---|---|
| Bare-metal machines | AWS EC2 Bare Metal | IaaS |
| Virtual machines | AWS EC2 | IaaS |
| Kubernetes Engine | AWS EKS | PaaS |
| Serverless / Function as a Service | AWS Lambda | PaaS |
Given a data center with many resources, how do these different kinds of products run together in the same data center? For the convenience of management and development, the products (may) run hierarchically. Virtual machines are running on the top of physical machines. Containers and container schedulers are running on the top of virtual machines.
The process of deployment is composed of several steps, including resource allocation, instance placement and preemption & migration.
Users should specify the resource allocation strategy of the cloud products, such as the size of VMs or containers, and the autoscaling configurations of services.
As the resource pool is composed of multiple nodes, the cloud provider should decide the placement of each instance. The placement should consider both the resource efficiency and the additional requirements from users such as affinity configurations. The placement problem can be modeled as Bin-packing problem, where the nodes are bins and the instances are items.
On nodes with an oversubscription mechanism, part of the instances are able to be preempted, or evicted or migrated to release resources for the more important instances. Preemption and migration reduce the reliability of the affected instances. In IaaS, the preemptible instances are called spot VMs or harvest VMs and are cheaper than regular VMs.
Users have different targets for cloud projects for various kinds of workloads.
Referring to the three steps of the deployment of the projects, here are several methods to improve resource management.
Insight: Specifying the resource allocation strategies may be difficult for developers and operators, replacing the process with automatic frameworks can lead to better results.
Insight: As the utilizations of instances on the nodes are always below the highest level, the oversubscription (aka overcommitment) strategy is used to put more instances on the nodes to increase the utilization of the nodes. The simple oversubscription method is to set a fixed oversubscription ratio (> 1), and the scheduler can allocate resources to instances whose sum can exceed the total number of the resources of the node. However, oversubscription introduces the possibility of resource contention when utilizations of multiple instances go high. Some works focus on avoiding the problem through characterizing, profiling, predicting, etc.
Insight: The colocation of multiple instances in one nodes depends on the task scheduler of the node. In underlying schedulers, discovering fine-grained improvements can increase resource utilization.
Insight: The unified resource management system that manages all the machines in all datacenters and schedules all workloads of different priorities can lead to a more flexible and optimized resource utilization.
Insight: Resource disaggregation makes instance placement easier and increases the resource utilization. Currently, storage is managed separately, but CPU, memory and GPU resources are coupled in same nodes.
Insight: Preemptible VMs like spot VMs and harvest VMs are much cheaper but less reliable. Some works are aimed at reducing the damage to the applications caused by the stop of preemptible VMs.
vCPU
vCPU overcommitment
Cloud Computing Models: https://aws.amazon.com/types-of-cloud-computing/
Firecracker: Lightweight Virtualization for Serverless Applications: https://aws.amazon.com/cn/blogs/china/deep-analysis-aws-firecracker-principle-virtualization-container-runtime-technology/
]]>Microservice architecture splits complex monolithic applications into multiple smaller, independent services. Microservices are loosely coupled and can be developed, deployed, and scaled independently. They collaborate through calling well-defined APIs with RPC, HTTP, or message queues.
As microservices have multiple advantages, they also bring challenges, such as the complexity and overhead introduced by the communication between services. However, the decoupling of services enables us to perform fine-grained management of services and networks. The following are some research topics in microservices.
Kubernetes enables the autoscaling of the services. Vertical scaling changes the size of the resources of a single instance, such as the CPU and memory allocation of the container, while horizontal scaling changes the number of instances. The widely-used simple autoscaling algorithm needs the users to configure the range of vertical or horizontal scaling and refers to the CPU and memory usage of the services or other custom metrics to make decisions. The autoscaling algorithm helps the service adjust to dynamic or unpredictable workloads. When the workload is low, it can allocate fewer resources. When the workload increases, it can allocate more resources automatically to handle the request spikes.
However, the decisions of autoscaling are only aware of single services, rather than the SLO of the requests and the resource usage of the whole service. One whole service can be presented as a DAG. One request can be processed by the services in parallel or in serial. The latency of the requests is the sum of the latency of each service layer in the process.
What’s more, the configurations of the microservices are too complex for users. Users only know their SLO when deploying the services, and don’t know the SLO of each service, even the suitable resource allocation for each service.
The basic problem is how to save as many resources as possible while guaranteeing the SLO of the service. A series of research works have been done to solve the related problems.
The basic methodology includes three components. The first component involves exploring the observability of services, which provides insights into their current status. For example, leveraging new monitoring metrics to know the status more clearly, like using run queue length to guess the waiting task number (SHOWAR), and using gateway network traffic to guess the current request number (Nodens).
The second one is the profiling and modelling of the services to predict their performance in different conditions. This component bridges the gap between the concerned performance metrics that we want to optimize, the workloads we need to handle, and the resource allocation that we can control. For each service, we can collect the data points online or offline, and model them with simple functional relationships (Cilantro, Erms). A performance model is not always necessary, as it can be replaced by a feedback-based mechanism.
The third one is the scheduler to solve the optimization problem, that is reducing the resource usage in the constraints of guaranteeing SLO. As microservices give complex and huge action space, many works use neural networks, especially Deep Reinforcement Learning (FIRM, AWARE, Autothrottle). The DL methods have potential to solve complex strategy problems without artificial design. There are also non-DL algorithms to reach the goals (Cilantro, Erms, Derm). These methods introduce more expert knowledges of autoscaling.
The evaluation of the performance of these works is challenging. Due to the importance of the SLO and the potential side effects of adopting new methods, these studies typically experiment with dedicated experimental systems. The experimental systems always include microservice benchmarks and load generators simulating the real load traces, running on several hosts.
These works give us valuable insights into the resource management of microservices. They consider the QoS of services and propose various methods to manage resources efficiently.
However, some disadvantages still hinder the practical application of these methods. Here are some of them:
Microservice architecture introduces more network communication between services. Network communications introduce more overhead than function calls in monolith applications, including the inter-node and intra-node network overhead. If a sidecar mechanism is used, the network overhead will be greater. Thus the average overall latency of the service is higher.
Placing services with close relationships on the same host can reduce inter-node network traffic and latency. However, it may destroy the independence of the services, which is one of the principle of microservice architecture.
Reducing the host network overhead as much as possible is also a promising direction. Kernel bypass technologies have been applied to RPC.
The configuration space of microservice is huge, thus it is meaningful to build tools to explore the configuration automatically and easily.
Tracing of the request processing in the microservices is important for extracting dependencies, troubleshooting and performance analysis.
Current tracing systems like OpenTelemetry require modification of the application code to actively report the request information and timestamps.
This field focuses on using the relationships between services to locate the position of the performance or logic problem. The unique problem in microservices is that one fault in a service can spread in the call graph and cause multiple faults in related services, thus localizing the root causes is challenging.
Personally speaking, without experience in production microservices, current fault localization works don’t convince me of their motivation about the complexity of finding where the root causes are located. As the model of web services is relatively simple, the status of each service, such as workload overwhelming, resource contention and degration caused by other services, can be inferred through the metrics of the services, including request number, resource usage, response latency. Judging whether the performance problem is caused by the service itself or the other services seems not difficult.
Simulation is an important tool for understanding the performance of microservices. However, currently there are few actively maintained simulators for microservices.
本项目和Inclavare Container社区在阿里巴巴编程之夏项目中合作完成，项目地址。
Inclavare是enclave的拉丁语源，它是机密计算领域中一种为了保护用户数据，将用户的敏感工作负载与不受信任的、不可控的 基础设施隔离的技术。
Inclavare Container是一种从硬件辅助可信执行环境（TEE）中启动受保护容器的安全容器运行时。
确定性编译（Deterministic Compilation），或者说可复现构建（Reproducible Builds），是指从相同的源代码中编译出相同的二进制文件的技术。它在软件供应链的安全以及管理中有重要作用。
免费与开源软件的源代码可以供所有人来检查，但是大多数软件都是预编译分发的，很难确认编译出的二进制文件是否与源代码对应，而没有引入只隐藏于二进制文件的后门或漏洞。确定性编译技术目标是搭建一个与编译时间、机器、文件名等无关的编译环境，让第三方可以方便地验证二进制文件是否由指定版本的源代码编译而成。
同时在二进制文件的管理中，如果需要对二进制文件进行存储管理，那么最好不要让二进制文件中存储会随着环境改变的时间戳、文件路径等信息，以形成完全相同的二进制文件。而确定性编译技术可以免除编译器引入这些不确定因素。
检查两个二进制文件是否相同的一个简单的方法是消息摘要（message digest），包括MD5，SHA系列算法等。
已有的确定性编译方案通过对编译器进行配置、ptrace拦截系统调用实现。而本项目希望实现基于eBPF的确定性编译系统。eBPF是Linux内核提供的在内核中运行沙箱程序的技术，允许开发者在不改动内核代码的同时对Linux内核的安全、网络、监控等功能进行扩展。Kprobe是Linux内核提供的对内核内任意函数（包括系统调用）进行跟踪探测的机制，可以在指定的函数调用时、任意指令处与函数返回时执行额外的程序。Linux同样提供了很多tracepoint的tracing机制。eBPF可以把代码挂载到Kprobe机制上，相比于直接开发Kprobe模块拥有eBPF提供的安全保障。
使用eBPF来监控编译器会引入不确定因素的系统调用进行监控，并修改其返回值，我们可以开发出一个对编译器完全透明的确定性编译工具，只需要在后台运行eBPF程序，不需要在使用编译器时添加额外的配置与指令，也不会影响除了编译器与其他编译所需工具之外的其他进程。
本项目我们的主要目标是让工具可以实现Linux内核的确定性编译。
首先我们需要定位编译器引入的不确定因素，以及它们所对应的系统调用。经过调研与实验，编译器在运行中主要会受到时间、随机数、文件系统等影响，在字符串、符号名、可执行文件时间戳等包含不确定因素。
我们没有去阅读编译器代码，而是采用以下方法进行调查：一方面我们收集一些常见的不确定性因素（如宏定义等），进行实验并跟踪系统调用；一方面我们通过编译Linux内核查看编译出的二进制文件中存在哪些不确定性。
Linux内核的编译中，我们使用相同版本的编译器、相同的配置文件、相同的Linux内核源码版本，分别在不同的时间、不同的Linux系统、不同的文件路径上进行编译，检查不同情况下编译出的二进制文件vmlinux的区别。
strace可以用来监控编译器的系统调用。由于gcc编译器的预处理进行是在子进程中的，所以strace使用时需要使用--follow-forks或-f选项。
二进制文件的相关工具，用于比较二进制文件的可以使用xxd与diff（或colordiff、cmp）等配合，查看二进制文件内容的有strings、nm、readelf、objdump等，二进制文件的GUI编辑器有010editor等。
我们发现以下几种编译不确定性的来源：
gcc编译器提供了一些预先定义好的宏定义，在预处理阶段会进行宏定义展开与include展开等工作。其中__DATE__，__TIME__分别可以展开为编译时的日期与时间的字符串。__TIMESTAMP__与__FILE__可以展开为被编译的文件的最后一次修改时间戳和相对路径的字符串。
此外，在gcc中开启LTO优化时，会在.o文件中生成随机的符号名称，使用了系统的random相关系统调用，这会影响到.o文件的确定性。
在编译与比较Linux kernel中，发现vmlinux中有一个字符串是编译信息的记录，有编译此内核的Linux版本、用户名与机器名、gcc版本、时间戳、内核功能选项等。经调查，在内核安装后/proc/version会提供此版本信息；在编译时scripts/mk_compile_h会在compile.h中获取并在代码中生成各种信息，在编译机器上使用了date, whoami, uname等指令。
Linux version 5.18.11-051811-generic (kernel@gloin) (gcc (Ubuntu 11.3.0-4ubuntu1) 11.3.0, GNU ld (GNU Binutils for Ubuntu) 2.38.50.20220629) #202207121541 SMP PREEMPT_DYNAMIC Tue Jul 12 15:47:28 UTC 2022
我们还发现二进制文件中有5处随机数的差异。在用LD_PRELOAD方法简单拦截libc中相关时间函数后，发现时间戳与随机数都变成相同的值，说明内核编译中产生随机数与时间相关。
我们使用libbpf，使用C语言进行eBPF程序的开发。eBPF的开发工具还有Bpftrace，bcc等，但是尝试在这些框架下实现同样的修改系统调用功能时并不会获得预期的行为，具体原因还需要学习与调查。
与ptrace、seccomp等不同的是，eBPF并不允许直接修改函数的参数、返回值等。但是eBPF提供了一个bpf_probe_write_user函数，该函数可以传入一个用户空间地址（void*类型）dst和一个eBPF程序地址（void*类型）src以及一个u32类型的长度参数，它可以将src buffer的内容写入到用户空间，达到修改用户空间内存的目的。而一部分系统调用传入一个指向结构体的指针，调用者期待从结构体中获得函数返回的数据，这个地址也会被kprobe传入eBPF程序中，那这个结构体里的返回结果就可以被修改，如gettimeofday，以struct timeval *tv, struct timezone *tz两个参数返回具体的值，函数返回值只是0与-1的状态码。同样的，如果系统调用传入的参数是一个字符串，那么这个字符串也可以被修改成一个不比原来长的另一个字符串。很遗憾函数的返回值仍然不能被修改，但是能修改结构体中的结果对实现确定编译的目标已经足够了。而如果系统调用没有传入一个指针，那么eBPF程序对修改这个函数就无能为力了，比如time，直接以time_t类型作为返回值。
离可以修改系统调用只剩一步之遥了：eBPF的进入与退出hook点都只能获取到部分上下文信息，进入函数的hook点可以获得函数参数，包括结构体指针，但是不能在此时修改结构体，因为会被后面的函数逻辑再次修改；退出函数的hook点是修改结果的时机，但是只提供了函数返回值，没有提供结构体指针。这时就需要使用到eBPF的MAP（相当于eBPF程序各个地方都可以访问到的全局key value存储），实现进入函数向退出函数传递指针数据，我们使用进入与退出函数都可以获得的tid（对于每一次目标函数调用都是唯一的）作为key，传递的指针作为value，这样就使得退出函数也能获得结构体指针。
所以修改系统调用的流程是：
最终我们使用这种方法可以修改openat,newfstatat,read,getrandom,gettimeofday,clock_gettime，uname等系统调用，去除文件系统、时间、随机数、主机带来的不确定性。
在使用strace date等命令，对时间相关的系统调用进行分析时，会发现time，gettimeofday，clock_gettime并不会出现在strace的结果中，使用eBPF也没有办法被触发。这是因为Linux的vDSO机制，它把time相关的系统调用结果映射到内存中，在调用这些系统调用时直接从用户内存中读取结果，可以减少用户态与内核态的切换开销，做到快速系统调用。vDSO在Linux运行中是默认开启的，因此在使用工具前我们需要在启动选项中把它关闭。
由于eBPF程序无法修改time系统调用的返回值，我们需要使用另外的方法修改它。time系统调用被libc中的time函数使用，所以我们可以使用LD_PRELOAD的方法，在用户态把libc中的time函数替换掉。但是这样就需要在环境变量中指定LD_PRELOAD的so文件，或是在系统配置文件/etc/ld.so.preload中指定所有可执行文件的preload路径，无法做到自动拦截指定的编译相关进程的调用。而通过对LD_PRELOAD机制的系统调用进行分析，我们同样可以通过拦截系统调用的方式，让不同进程的读到的/etc/ld.so.preload不同，只对编译相关进程加载preload文件。
以date命令为例，使用modify_time可以使输出的时间为固定值（在编译前在一个头文件中配置）：
$ date # normal
Mon Oct 17 04:25:13 PM CST 2022
$ sudo ./modify_time &
$ date # modified
Mon Oct 17 04:26:25 PM CST 2022
以下为eBPF程序的log：
$ sudo cat /sys/kernel/debug/tracing/trace_pipe
<...>-816502 [006] d..31 5707624.022076: bpf_trace_printk: [sys_exit_clock_gettime] OVERWRITING struct timespec64 at 00000000c626f924 from (1665995173, 362091123) to (1659312000, 0)
<...>-816502 [006] d..31 5707624.022077: bpf_trace_printk: [sys_exit_clock_gettime] RESULT 0
为了在多个环境中测试编译确定性，我们基于shell脚本与docker实现了部分编译自动配置与编译的功能，如C语言宏定义的编译与kernel源码的配置与编译。
eBPF and syscall:
deterministic build:
]]>