06Conference: 30th ACM International Conference on Information and Knowledge Management; November 01–05, 2021; Gold Coast, Australia4Price: 15.00

Koji Matsuda email: matsuda.koji@ist.osaka-u.ac.jp Affiliation: Osaka University , Yuya Sasaki email: sasaki@ist.osaka-u.ac.jp Affiliation: Osaka University , Chuan Xiao email: chuanx@ist.osaka-u.ac.jp Affiliation: Osaka University and Makoto Onizuka email: onizuka@ist.osaka-u.ac.jp Affiliation: Osaka University

2021© , 2021;

Abstract.

Federated learning is a distributed machine learning method in which a single server and multiple clients collaboratively build machine learning models without sharing datasets on clients. Numerous methods have been proposed to cope with the data heterogeneity issue in federated learning. Existing solutions require a model architecture tuned by the central server, yet a major technical challenge is that it is difficult to tune the model architecture due to the absence of local data on the central server. In this paper, we propose Federated learning via Model exchange (FedMe), which personalizes models with automatic model architecture tuning during the learning process. The novelty of FedMe lies in its learning process: clients exchange their models for model architecture tuning and model training. First, to optimize the model architectures for local data, clients tune their own personalized models by comparing to exchanged models and picking the one that yields the best performance. Second, clients train both personalized models and exchanged models by using deep mutual learning, in spite of different model architectures across the clients. We perform experiments on three real datasets and show that FedMe outperforms state-of-the-art federated learning methods while tuning model architectures automatically.

Keywords:

federated learning, edge computing, IoT, deep learning, deep mutual learning

1. Introduction

With the growing popularity of mobile devices such as smartphones and tablets, an unprecedented amount of personal data has been generated. Such personal data are helpful to build machine learning models on a variety of applications such as action recognition (Anguita2013APD), next-word prediction (hard2018federated), and wake word detection (leroy2019federated). However, due to the concerns raised by data privacy and network bandwidth limitation, it is impractical to collect all local data from clients and train models in a centralized manner. To address the privacy concerns and network bandwidth bottleneck, federated learning has emerged as a decentralized learning paradigm to build a model without sharing local data on clients (mcmahan2017communication).

Federated learning builds a model with a single server and multiple clients in a collaborative manner. Its general procedure consists of two steps: ( $1$ ) client learning, in which clients train models on their local data and send their trained models to the server, and ( $2$ ) model aggregation, in which the server aggregates those models to build a global model and distributes the global model to the clients. These two steps are repeated until the global model converges. This procedure effectively uses clients’ local data by sharing their trained models.

A challenge of federated learning. One of the challenges in federated learning is on data heterogeneity: clients have local data that follow different distributions, i.e., they do not conform to the property of independent and identically distributed (IID) random variables. This causes difficulty in learning a single global model that is optimal for each client. Indeed, it has been reported that, in typical federated learning methods, model parameters of a global model are divergent when each client has non-IID local data (MLSYS2020_38af8613; li2019convergence). Personalized federated learning methods have been proposed to deal with data heterogeneity (mansour2020three; shen2020federated; NEURIPS2020_f4f1f13c; zhang2021personalized). These methods aim to build personalized models, which are optimized models for clients.

We have the following research questions for building optimal personalized models:

•

How to determine the model architectures of personalized models? In existing personalized federated learning methods, the server must tune model architectures in advance. Since the server is unaware of local data distributions on clients, the server needs to train multiple models with different architectures to tune model architectures remotely. However, this process requires high communication costs between the server and clients, making it impractical. Recently, an automatic architecture tuning method was proposed to automatically modify model architectures during learning process (FedNAS). It tunes the architecture of the single global model by the server. However, it is likely that the model architecture tuned by the server is not optimal for each client, and the server is unable to evaluate the accuracy of the tuned model by using the local data of the clients. Therefore, each client should individually tune its model architecture, which may differ across clients due to the non-IID data (see Table 3 in our experimental study).

To the best of our knowledge, there are no personalized federated learning methods that can automatically tune the model architecture during the learning process. Since each client is unaware of the local data on the other clients, we need means of leveraging other clients’ models to tune model architectures.
•

How does each client leverage other clients’ models with different architectures to improve its model accuracy? The server may not aggregate personalized models because their model architecture may differ across the clients. It is not effective to rely on the aggregation of models for leveraging other clients’ models. So, we need additional means of leveraging local data and models with different architectures.

Contributions. In this paper, we propose a novel federated learning method, federated learning via model exchange (FedMe for short). We propose a notion of exchanged models, i.e., each client can receive models sent from other clients. Then the clients are able to tune model architectures and train their models by utilizing the exchanged models. The learning process of FedMe addresses the aforementioned research questions. First, clients tune their model architectures based on the performance of exchanged models. To optimize the model architecture for local data, each client compares its own personalized model to the exchanged models and pick the one that yields the best performance. In this way, clients can automatically and autonomously modify their model architectures. Second, clients train both their own and exchanged models to improve both models, and the server aggregates the trained models of the same clients. We use two techniques for model training: deep mutual learning (8578552) and model clustering. Deep mutual learning is effective in simultaneously training two models by mimicking the outputs of the models regardless of model architecture. Model clustering selects similar personalized models as exchanged models for each client, which prevents models from overfitting the noise caused by deep mutual learning. In doing so, the aggregated models can reflect the local data on other clients because they are trained by using other clients’ local data and the exchanged models.

We evaluate the performance of FedMe by comparing with state-of-the-art methods on three real datasets. Our experiments show that FedMe achieves higher accuracy than state-of-the-art methods even if we manually tune these methods for their best model architecture. Another interesting takeaway of the evaluation is that traditional federated learning methods with fine-tuning can build highly accurate personalized models on clients, which is not evaluated fairly in existing studies.

Organization. The remainder of this paper is organized as follows. In Section 2, we review related work. In Section 3, we define the problem. We then present our proposed method, FedMe, in Section 4, and report our empirical evaluation results in Section 5. In Section 6, we summarize the paper and discuss future work.

2. Related Work

The research on federated learning has been actively studied since McMahan et al. introduced federated learning (mcmahan2017communication). Several survey papers summarize studies of federated learning (kairouz2019advances; lim2020federated; mothukuri2021survey).

Numerous federated learning methods have been proposed recently. Thus, we describe only typical methods due to the page limitation. We review federated learning methods from three points of view; (1) data heterogeneity, (2) personalization and (3) model architecture tuning. Methods for data heterogeneity aim to appropriately build models in environments that clients have non-IID local data. Methods for personalization aim to build optimal personalized models for each client. Personalization has two types, homogeneous and heterogeneous, in which model architectures of all personalized models are the same and different, respectively. Methods with model architecture aim to automatically tune the model architecture.

The most basic method on federated learning is FedAvg (mcmahan2017communication), which aggregates all trained models of clients by averaging their model parameters to build a single global model. Because the accuracy of FedAvg decreases in data heterogeneity, many methods have extended FedAvg to deal with data heterogeneity such as FedMA (Wang2020Federated) and HierFAVG (liu2019edge). Although these methods try to build a single global model by aggregating trained models, it is difficult to achieve high accuracy only by the single model.

Personalized federated learning methods have been proposed to build different models for each client (mansour2020three; NEURIPS2020_f4f1f13c; zhang2021personalized). These methods can increase the accuracy compared with methods that only build the single global model. We first reviews homogeneous personalization methods, which build personalized models with different parameters but their model architectures are the same. Mansour et al. proposed HypCluster and MAPPER (mansour2020three). In HypCluster, the server prepares several global models and distributes them to clients. Clients train only the model that has the highest accuracy and sends it back to the server. Then, the server aggregates each trained model as new global models. In MAPPER, clients compute the balancing weights of the global model and its trained model and then do a weighted sum of their parameters. T. Dinh et al. proposed pFedMe (NEURIPS2020_f4f1f13c), which builds global and local models by normalizing using the Moreau envelope function. These homogeneous personalization methods require the same model architecture for all personalized models, they cannot personalize their model architectures. We here note that these methods do not use fine-tuning (i.e., after finalizing the models on learning process, clients do not re-train the models by their local data), and also they do not compare simple methods with fine-tuning (e.g., FedAvg with fine-tuning). Our experiments show that most methods have lower accuracy than FedAvg with fine-tuning.

There are heterogeneous personalization methods that build personalized models with different parameters and architecture across clients (li2019fedmd; shen2020federated). Clients can choose arbitrary model architectures depending on the size of local data and their computation resources. Shen et al. proposed Federated Mutual Learning (FML) (shen2020federated). The server in FML distributes the global model and clients train both of the global and their personalized models by deep mutual learning. We use the similar idea of FML on client training, but FedMe does not build the global model. Li et al. proposed FedMD (li2019fedmd) which incorporates knowledge distillation into federated learning. FedMD needs public data, which is datasets similar to local data and can be used by all server and clients. These heterogeneous personalization methods can build personalized models with different architectures for each client. However, the server and clients need to determine model architectures before the learning process. In addition, in our experiments, these methods cannot achieve higher accuracy than non-personalized methods.

Model architecture search is a hot topic in deep learning fields, which searches the best model architecture among predefined search spaces (e.g., layer types and the maximum number of layers) (he2020milenas; NAS). FedNAS supports the network architecture search on federated learning (FedNAS). Although it automatically tunes model architecture, it aims to build a single global model. It does not aim to build personalized models.

In summary, our method FedMe is the first method that can satisfy all the data heterogeneity, heterogeneous personalization, and model architecture tuning.

3. Problem Definition

In this section, we describe our problem definition. The notation used in this paper is summarized in Table 1.

Given a classification task, a server and a set of clients collaboratively build personalized models of clients. Let $S$ denote the set of clients. The number of clients is denoted by $|S|$ . We use a subscript $i$ for the index of the $i$ -th client. For example, $D_{i}$ is the local data of client $i$ , and $n_{i}$ is the size of $D_{i}$ (i.e., the number of records). $N$ denotes the sum of $n_{i}$ for all clients. $x$ and $y$ are features and labels of records contained in the local data, respectively. We assume classification tasks, so $y$ is assigned with a single class among $M$ classes. $T$ and $E$ are the total numbers of global communication rounds and local training rounds, respectively. Global communication means communication between the server and clients during training. Local training means that clients train the model on its local data. $t$ denotes an index of global communication round. $w^{t}_{{p}_{i}}$ and $w^{t}_{{ex}_{i}}$ are the personalized and exchanged model of client $i$ in round $t$ , respectively. $Idx(w^{t}_{{ex}_{i}})$ is an index of the original client of $w^{t}_{{ex}_{i}}$ . For instance, $Idx(w^{t}_{{ex}_{i}})$ returns $j$ if $w^{t}_{{ex}_{i}}$ is a personalized model of client $j$ .

Table 1. Summary of notation used in this paper.


Symbol	Description
$S$	a set of clients
$i$	an index of clients
$D_{i}$	$i$ th client’s local data
$n_{i}$	the size of $D_{i}$
$x_{i}$ , $y_{i}$	a feature and label sampled from $D_{i}$ , resp.
$M$	the number of classes
$T$	the number of global communication rounds
$t$	an index of global communication rounds
$E$	the number of local training epochs
$w^{t}_{{p}_{i}}$	a personalized model of client $i$ at round $t$
$w^{t}_{{ex}_{i}}$	an exchanged model of client $i$ at round $t$
$Idx(w^{t}_{{ex}_{i}})$	an index of the original client of $w^{t}_{{ex}_{i}}$
$C_{k}$	models in cluster k
$K^{t}$	the number of clusters at round $t$

Figure 1. FedMe framework.

In FedMe, each client builds its personalized model instead of a single global model. We define the optimization problem as follows:

(1)

\displaystyle\{w_{p_{1}},\ldots,w_{p_{|S|}}\}=\text{argmin}\sum_{i\in|S|}\mathcal{T}_{i}(w_{p_{i}}).

$\mathcal{T}_{i}$ is the personalized objective for client $i$ , and is defined as follows:

(2)

\displaystyle\mathcal{T}_{i}=min\;\mathcal{L}(w_{p_{i}},D_{i}),

where $\mathcal{L}(w_{p_{i}},D_{i})$ : $\Theta\longmapsto\mathbb{R}$ is the loss function of client $i$ , corresponding to $D_{i}$ and $w_{p_{i}}$ . $\Theta$ is the space of models, and is not fixed. This optimization problem is similar to that of (zhang2021personalized). In (zhang2021personalized), since the model architecture of personalized model is fixed, the size of $w_{p_{i}}$ is predetermined and fixed. In contrast, the size of $w_{p_{i}}$ in our problem is also optimized, which indicates that our problem aims to optimize the model architectures of personalized models. By solving this optimization problem, we can build optimal personalized models for each client.

4. Methodology

In this section, we describe our proposed method, FedMe. We first explain the overall idea and the framework of FedMe. After that, we present an algorithm and its technical components in detail. Finally, we shows a concrete example of our algorithm.

4.1. Idea and Framework

FedMe is a heterogeneous personalized federated learning method with automatic model architecture tuning. Recall our research challenges; how we automatically tune optimal model architectures for clients and how clients use models with different architectures for improving their models. For solving these research challenges, our idea is simple; clients receive models of other clients for leveraging model architecture tuning and send their models to other clients for training models by local data on other clients. In other words, clients exchange their models for model architecture tuning and model training.

FedMe effectively leverages exchanged models by the following ways. First, clients tune their personalized models based on the performance of exchanged models. In more concretely, clients replace their personalized models with the exchanged models if the exchanged models have smaller loss on their local data than their personalized models. Each client can automatically tune its model architectures so that the accuracy on its local data improves. Second, clients train both their personalized and exchanged models and the server aggregates the trained models of the same client. This achieves model training that can train personalized models with different architectures. Third, clients simultaneously and effectively train the both personalized and exchanged models by using deep mutual learning and model clustering. Deep mutual learning simultaneously trains two models by mimicking outputs of models each other regardless of the model architecture. The output of the other models may become noise and may overfit the noise when models are trained by significantly different local data (chen2021fedbe). To prevent models from overfitting the noise, the server performs model clustering to select models with similar outputs. The model clustering groups model into subsets of models that have similar outputs by using Kmeans method (macqueen1967some).

Figure 1 shows a framework of FedMe. FedMe has five learning processes; (0) each client creates its personalized model with arbitrary model architectures, (1) clients send their personalized models to the server, (2) the server decides exchanged models for the clients based on model clustering and sends the exchanged models to clients, (3) clients train both their personalized and exchanged models by deep mutual learning and tune their personalized models based on the performance of the exchanged models, and (4) after clients send back the trained exchanged and personalized models, the server aggregates personalized and exchanged models for all clients and then send their aggregated personalized models to clients. FedMe repeats (1)–(4) until the number of global communication reaches a given threshold $T$ .

4.2. Algorithm

In this section, we describe the algorithm of learning procedures of FedMe. The pseudo-code of FedMe is shown in Algorithm 1. After clients initialize their models (line $1$ ), FedMe starts its learning process. First, clients send their personalized model to the server (line $4$ ). The server clusters the models using unlabeled data (line $6$ ), and each client receives a model that belongs to the same cluster as an exchanged model from the server (lines $8$ – $9$ ). Each client then trains the personalized and exchanged models (line $11$ ) and determines the index of its new personalized model (line $13$ ). Each client sends two trained models and $a^{t}_{i}$ to the server (line $14$ ). The server aggregates each model by averaging their parameters (line $16$ ) and sends them to each client based on $a^{t}_{i}$ (line $18$ ). These steps are repeated until the number of global communication rounds becomes $T$ .

We explain detailed procedures of initialization, model training, model tuning, model clustering, and model aggregation in the following.

Algorithm 1 Algorithm of FedMe.

Input: number of global communication rounds $T$ , number of local training epochs $E$ , set of clients $S$ and their local data $\{D_{i}\}^{|S|}_{1}$ , unlabeled data $U$ , numbers of cluster { $K^{1},K^{2},\cdots K^{T}$ }, learning rate $\eta$

Output: personalized models $\{w^{T}_{p_{i}}\}^{|S|}_{1}$

1: Initialize(

w^{0}_{p_{i}}

) on all client

i

2: for

t=1,\cdots,T

3: for

i\in S

4: Client

i

sends

w^{t-1}_{p_{i}}

to server

5: end for

\{C_{1},\ldots,C_{K^{t}}\}\leftarrow\bf{Model\;Clustering}

(

\{w^{t-1}_{p_{i}}\}_{i\in S_{t}}

U

K^{t}

)

7: for

i\in S

w^{t-1}_{{ex}_{i}}\leftarrow w\in C_{k}\;that\;includes\;w^{t-1}_{p_{i}}

9: Server sends

w^{t-1}_{{ex}_{i}}

to client

i

10: for

e=1,\cdots,E

11:

\bf{Model\;Training}

(

w^{t-1}_{p_{i}}

w^{t-1}_{{ex}_{i}}

D_{i}

)

12: end for

13:

a^{t}_{i}\leftarrow\bf{Model\;Tuning}

(

w^{t}_{p_{i}}

w^{t}_{{ex}_{i}}

D_{i}

)

14: Client

i

sends

w^{t}_{p_{i}},w^{t}_{{ex}_{i}},a^{t}_{i}

to server

15: end for

16:

\bf{Model\;aggregation}

(

\{w^{t}_{p_{i}}

w^{t}_{{ex}_{i}}\}_{i\in S}

)

17: for

i\in{S}

18: Server sends aggregated

w^{t}_{p_{a^{t}_{i}}}

to client

i

19: end for

20: end for

Initialization. FedMe first requires the initialization of model architectures to clients. Since clients can use arbitrary model architectures in FedMe, they can determine their model architecture depending on their local data. For example, clients build optimal models on their local data. Of course, the server can determine arbitrary models and distribute them to clients.

Model Training. In FedMe, each client exchanges its personalized model, and thus the model is trained on the local data of multiple clients, which enables training models even when clients have models with different architecture.

Each client trains personalized and exchanged models on its local data by deep mutual learning. Deep mutual learning between personalized and exchanged models improves accuracy compared to training them independently. Indeed, it is known that deep mutual learning effectively improves the inference performance of models when we use numerous models for training (8578552). Therefore, deep mutual learning has significant benefits on the learning process of FedMe.

We define loss functions $\mathcal{L}_{p}$ and $\mathcal{L}_{ex}$ of the personalized and exchanged models, respectively, as follows:

(3)		$\displaystyle\mathcal{L}_{p}{=}{-}\!\!\!\!\!\!\!\sum\limits_{(\!x,y\!)\in D_{i}}\!\sum\limits^{M}_{m=1}\!\!I(\!y,\!m\!)log(p^{m}_{p}(\!x\!)\!){+}\!\!\!\!\!\!\sum\limits_{(\!x,y\!)\in D_{i}}\!\sum\limits^{M}_{m=1}\!p^{m}_{ex}(\!x\!)log\frac{p^{m}_{ex}(\!x\!)}{p^{m}_{p}(\!x\!)},$
(4)		$\displaystyle\mathcal{L}_{ex}{=}{-}\!\!\!\!\!\!\!\sum\limits_{(\!x,y\!)\in D_{i}}\!\sum\limits^{M}_{m=1}\!\!I(\!y,\!m\!)log(p^{m}_{ex}(\!x\!)\!){+}\!\!\!\!\!\!\sum\limits_{(\!x,y\!)\in D_{i}}\!\sum\limits^{M}_{m=1}\!p^{m}_{p}(\!x\!)log\frac{p^{m}_{p}(\!x\!)}{p^{m}_{ex}(\!x\!)},$

where $p^{m}_{p}$ and $p^{m}_{ex}$ are the predictions of the personalized and exchanged models for class $m$ , respectively. The first and second terms of these equations are the cross-entropy error and the Kullback Leibler (KL) divergence, respectively. The function $I(y,x)$ returns 1 if $y=m$ and returns 0 otherwise.

To minimize the above loss functions, client $i$ updates the two models.

(5)		$\displaystyle w^{t}_{p_{i}}\leftarrow w^{t-1}_{p_{i}}-\eta\nabla\mathcal{L}_{p},$
(6)		$\displaystyle w^{t}_{{ex}_{i}}\leftarrow w^{t-1}_{{ex}_{i}}-\eta\nabla\mathcal{L}_{ex},$

where $\eta$ is learning rate, and $\nabla\mathcal{L}_{p}$ and $\nabla\mathcal{L}_{ex}$ is gradient of personalized and exchanged models, respectively.

Model Tuning. Clients can use models with any model architectures, which enabling clients to modify their models freely. In the learning process of FedMe, each client has many opportunities to optimally modify its model because it receives models of other clients as exchanged models at each global communication round. If the exchanged models achieve higher performance than the current models, they tune their models based on the exchanged models.

FedMe does not restrict means of model architecture tuning. In this paper, to validate the performance of design of FedMe, we use a simple tuning method which replaces their personalized models with the exchanged models. In more concretely, after each client trains models through deep mutual learning, it selects either its personalized or the exchanged models at round $t$ . FedMe computes $a^{t}_{i}$ , which represents an index of personalized model that client $i$ selects, as follows:

(7)

\displaystyle a^{t}_{i}=\begin{cases}i&\text{if }w^{t}_{p_{i}}\in\underset{w=w^{t}_{p_{i}},w^{t}_{{ex}_{i}}}{argmin}\mathcal{L}(w,D_{i}),\\ Idx(w^{t}_{{ex}_{i}})&\text{otherwise}.\end{cases}

In this equation, each client compares the loss of the personalized and exchanged models and then replaces the personalized model with the exchanged model if the exchanged model has smaller loss than the personalized model.

Of course, we can use other tuning methods instead of replacements, for example increasing the number of layers. Additionally, though we here consider the loss to tune the model architecture, each client can have its own criteria, such as the size of models and inference time. We remain optimal model tuning methods on FedMe as future work.

Model Clustering. Due to the data heterogeneity among clients, the outputs of the personalized models differ among clients. If clients perform deep mutual learning between models with significantly different outputs, the models may overfit the noise. In FedMe, models are clustered based on their outputs, and each client receives a model with similar output as an exchanged model from the server.

Model clustering reduces the difference between the output of the own model and that of the other model, thus preventing overfitting the noise (gao2017knowledge). On the other hand, continuous training of models with similar outputs may lead less generality. Therefore, in the early stages of training, we do not perform model clustering to increase the generality of the models. As training progresses, we increase the number of clusters in the model clustering. In this way, the model can be personalized without overfitting while maintaining its generality.

Since federated learning does not share local data, we cannot use local data for model clustering. Therefore, FedMe assumes that the server has access to unlabeled data, such as one-shot federated learning (guha2019one), and uses unlabeled data $U$ as input.

We use the Kmeans method (macqueen1967some) to cluster models. The server first computes the outputs of the models using unlabeled data. The server then uses kmeans with those outputs and divides models into $K^{t}$ clusters.

In the model exchange, each client receives a model of the same cluster as its own personalized model as exchanged models from the server at random. Here, if there is only one model in the cluster, the client receives a model from other clusters at random.

Model Aggregation. Client $i$ trains $w_{p_{i}}$ and $w_{{ex}_{i}}$ simultaneously. Therefore, it is necessary to aggregate all of them into a new model for each client. FedMe aggregates the models by averaging the model parameters as in FedAvg.

(8)

\displaystyle w^{t}_{p_{i}}\leftarrow\frac{1}{(s_{i}+1)}(w^{t}_{p_{i}}+\sum^{|S|}_{j=1}u^{t}_{i,j}w^{t}_{{ex}_{j}}),

where $s_{i}$ is the total number of clients that receive $w^{t}_{p_{i}}$ as the exchanged models. Also, $u_{i,j}$ represents which clients received the personalized model $w^{t}_{p_{i}}$ as an exchanged model and is defined by the following equation:

(9)

\displaystyle u^{t}_{i,j}=\begin{cases}1&\text{if }i=Idx(w^{t}_{{ex}_{j}}),\\ 0&\text{otherwise}.\end{cases}

The model parameters are averaged and aggregated for each client’s personalized model so that the aggregation is independent of the difference of model architecture.

4.3. Running Example

Figure 2. A running example of FedMe.

We explain the FedMe algorithm using concrete examples. We assume that the number of clients is five and the number of global communication rounds is two. The number of clusters $K$ is initially one and increases by one at each global communication round. Figure 2 illustrates the procedures of FedMe at the first and second global communication rounds.

Initialization: Clients initialized their personalized models. In this example, each client selects model architectures depending on their local data and sets up each of these models as an initial personalized model as $\{w^{0}_{{p}_{i}}\}_{i=1-5}$ .

First round: Clients send their personalized models to the server. In the first global communication round, since the number of clusters is one, the server randomly selects exchanged models for each client. In this example, clients $1$ – $5$ receive $w^{0}_{p_{3}}$ , $w^{0}_{p_{4}}$ , $w^{0}_{p_{1}}$ , $w^{0}_{p_{5}}$ , and $w^{0}_{p_{2}}$ from the server as their exchanged models, respectively.

Each client trains its personalized and exchanged models by deep mutual learning, and update $\{w^{1}_{{p}_{i}},\;w^{1}_{{ex}_{i}}\}_{i=1-5}$ , respectively. Next, each client compares the loss of the two models on its local data. Suppose that the personalized model of clients $2$ , $3$ , and $4$ have smaller losses than their exchanged models, while the exchanged models of other clients have smaller losses. Thus, $\{a^{1}_{i}\}_{i=1-5}$ is $3$ , $2$ , $3$ , $4$ , and $2$ , respectively.

All clients send the two trained models and $a^{1}_{i}$ to the server. The server then aggregates each personalized model. For example, the server aggregates $w^{1}_{p_{1}}$ and $w^{1}_{{ex}_{3}}$ , $w^{1}_{{p_{2}}}$ by averaging their parameters. The server sends back $w^{1}_{p_{3}}$ , $w^{1}_{p_{2}}$ , $w^{1}_{p_{3}}$ , $w^{1}_{p_{4}}$ , and $w^{1}_{p_{2}}$ to clients $1$ – $5$ , respectively, according to $a^{1}_{i}$ . Each client sets up each of these models as a new personalized model, $\{w^{1}_{{p}_{i}}\}_{i=1-5}$ .

Second round: We perform the second global communication round. Clients send their personalized models to the server. In the second global communication round, since the number of clusters is two, the server clusters the $5$ personalized models into two clusters using unlabeled data, and we assume that $\{w^{1}_{p_{1}},w^{1}_{p_{3}}\}$ , and $\{w^{1}_{p_{2}},w^{1}_{p_{4}},w^{1}_{p_{5}}\}$ belong to the same cluster, respectively. Clients $1$ – $5$ receive $w^{1}_{p_{3}}$ , $w^{1}_{p_{4}}$ , $w^{1}_{p_{1}}$ , $w^{1}_{p_{2}}$ , and $w^{1}_{p_{4}}$ from the server as their exchanged models according to model clustering, respectively.

Then, each client trains its personalized and exchanged models by deep mutual learning and updates to $\{w^{2}_{{p}_{i}},\;w^{2}_{{ex}_{i}}\}_{i=1-5}$ , respectively. We assume that $\{a^{2}_{i}\}_{i=1-5}$ is $1$ , $4$ , $3$ , $4$ , and $4$ in the second round.

All clients send the two trained models and $a^{2}_{i}$ to the server. The server then aggregates each personalized model. The server sends back $w^{2}_{p_{1}}$ , $w^{2}_{p_{4}}$ , $w^{2}_{p_{3}}$ , $w^{2}_{p_{4}}$ , and $w^{2}_{p_{4}}$ to clients $1$ – $5$ , respectively, based on $a^{2}_{i}$ , and each client sets up each of these models as its new personalized model. $\{w^{2}_{{p}_{i}}\}_{i=1-5}$ are the final personalized models.

5. Experiments

In this section, we test the accuracy of FedMe on three datasets with high degree of data heterogeneity. In our experiments, we aim to answer the following questions;

Q1.:: How accurate is the inference of FedMe compared with the state-of-the-art methods?
Q2.:: Does automatic model architecture tuning work well?
Q3.:: What techniques of FedMe impacts to the accuracy?
Q4.:: How fast is the learning process of FedMe compared with the state-of-the-art methods?
Q5.:: What is the impact of data heterogeneity and fine-tuning on FedMe?

To simplify the experiments, we use Pytorch (paszke2019pytorch) to create a virtual client and server on a single GPU machine.

5.1. Experimental Setup

5.1.1. Datasets, Tasks, and Models

In the experiment, we use three settings; FEMNIST, CIFAR-10, and Shakespeare. These datasets are frequently used in existing works (chen2021fedbe; li2019fedmd; MLSYS2020_38af8613; mansour2020three; mcmahan2017communication; Wang2020Federated).

•

FEMNIST: we use the Federated EMNIST-62 datset (caldas2018leaf), which includes images of handwritten characters with $62$ labels. This dataset is divided into 3,400 sub data based on writers. We conduct an image classification task.
•

CIFAR-10: We use CIFAR-10 dataset (cifar), which includes photo images with 10 labels. we divide the dataset into $20$ sub data using the Dirichlet distribution as in (Wang2020Federated). We set two parameters $\alpha_{label}$ and $\alpha_{size}$ to decide the degree of heterogeneity of the size of local data and labels, respectively. We use 0.5 and 10 as $\alpha_{label}$ and $\alpha_{size}$ , respectively. We conduct an image classification task.
•

Shakespeare: We use Shakespeare dataset (MLSYS2020_38af8613), which includes lines in “The Complete Works of William Shakespeare”. This dataset is divided into 143 sub data based on actors. We conduct a next-character prediction that infers next characters after given sentences.

Table 2 shows the statistics of the number of records on clients in datasets. We here note that we randomly divide CIFAR-10 in each test, so the statistics of CIFAR-10 is an example value.

We use different models for each setting following the existing works (reddi2020adaptive; Wang2020Federated). For FEMNIST and Shakespeare, we use CNN and LSTM, respectively (reddi2020adaptive). For CIFAR-10, we use VGG with the same modification reported in (Wang2020Federated). In each setting, we use four models varying the number of layers. For CNN and LSTM, we vary the number of convolution and LSTM layers from one to four, and the default value is two. For VGG, we use VGG $11$ , VGG $13$ , VGG $16$ , and VGG $19$ , and the default is VGG $13$ .

Table 2. Datasets Statistics.


Datasets	Total num	Mean	STD	Max	Min
FEMNIST	$671585$	$197.53$	$76.69$	$418$	$16$
CIFAR-10	$49000$	$2450$	$1024.66$	$5018$	$1131$
Shakespeare	$413629$	$2892.51$	$5445.89$	$33044$	$2$

5.1.2. Training and test

In our experiments, the number of clients is 20. In FEMNIST and Shakespeare, we select $20$ sub data for assigning local data on clients randomly. In CIFAR-10, we randomly divide the whole data into 20 local data. All clients participate in each global communication round following recent works (Wang2020Federated). We select 1,000 unlabeled data from each dataset. The unlabeled data was excluded from the train and test data. We divide each local data into training and test data by the ratio of 9:1, 8:2, and 5:1 for FEMNIST, CIFAR-10, and Shakespeare, respectively. Furthermore, we divide the training data into $7:3$ for FEMNIST and Shakespeare, and into $8:2$ for CIFAR-10, which are used as training and validation data, respectively.

We set the number of global communication rounds to $300$ , $500$ , and $100$ for FEMNIST, CIFAR-10, and Shakespeare respectively, and set the local epoch $E$ to $2$ for all setting. We conduct training and test five times and report mean and standard deviation (std) of accuracy over five times of experiments with different clients.

5.1.3. Baselines and hyperparameter tuning

We compare FedMe with three types of methods: ( $1$ ) non-personalized federated learning methods, ( $2$ ) personalized federated leaarning methods, and (3) non-federated learning methods. For ( $1$ ), we use FedAvg, and for ( $2$ ), we use HypCluster, MAPPER, FML, and pFedMe. For (3), we use Local Data Only, in which clients build their models on their model, and Centralized, in which a server collect local datasets from all clients (centralized can be considered as oracle). We use fine-tuning on each client for Centralized, FedAvg, HypCluster, and FedMe after building their models. In MAPPER and pFedMe, we do not use fine-tuning since their algorithms include the similar techniques to fine-tuning. We implement all methods except for pFedMe¹¹ 1 https://github.com/CharlieDinh/pFedMe because these codes are not available.

We explain hyperparameter tuning. The learning rate is optimized for each method by grid search using a grid of $\eta\in\{10^{-3},10^{-2.5},$ $10^{-2},\ldots,10^{0.5}\}$ . The optimization method is SGD (stochastic gradient descent) with momentum $0.9$ and weight decay $10^{-4}$ . The batch sizes of FEMNIST, CIFAR-10, and Shakespeare are $20$ , $40$ , and $10$ , respectively. In Hypcluster, we use two global models. In FedMe, we initialize model architectures of clients as the best accurate model on Local Data Only among model 1–4 (see Table 3). We set the range of number of clusters to $1$ – $4$ ; we initially use the number of clusters as $1$ and increase it by 1 at global communication round at $[150,225,275]$ , $[250,375,450]$ , and $[50,75,90]$ for FEMNIST, CIFAR-10, and Shakespeare, respectively.

Table 3. Average number of clients that select each model architecture based on their own local data.


	FEMNIST	CIFAR-10	Shakespeare
model1	$6.8\pm 2.5$	$5.6\pm 1.5$	$7.4\pm 1.7$
model2	$7.4\pm 0.5$	$9.2\pm 1.6$	$6.6\pm 1.8$
model3	$4.4\pm 2.8$	$4.0\pm 2.0$	$5.0\pm 1.6$
model4	$1.4\pm 1.1$	$1.2\pm 0.4$	$1.0\pm 1.7$

5.2. Experimental Results

Table 4. Test accuracy (mean

\pm

std).


	FEMNIST	CIFAR-10	Shakespeare
Local Data Only	$64.71\pm 2.94$	$73.17\pm 1.55$	$24.77\pm 1.95$
Centralized	$79.35\pm 2.29$	$90.80\pm 0.92$	$48.43\pm 3.32$
FedAvg	$77.25\pm 3.99$	$89.59\pm 0.94$	$42.53\pm 2.19$
HypCluster	$76.29\pm 3.15$	$88.54\pm 1.42$	$41.10\pm 3.29$
MAPPER	$60.95\pm 3.04$	$61.29\pm 4.19$	$36.77\pm 1.58$
FML	$67.91\pm 2.53$	$79.89\pm 1.44$	$28.73\pm 1.78$
pFedMe	$72.92\pm 3.54$	$79.46\pm 2.08$	$40.33\pm 2.27$
FedMe	$\bf{78.52}\pm 2.64$	$\bf{89.76}\pm 0.90$	$\bf{44.71}\pm 1.12$

(a) FEMNIST

(b) CIFAR-10

Figure 3. The validation accuracy over time of various methods.

We show experimental results to answer the five questions.

5.2.1. Q1. How accurate is the inference of FedMe compared with the state-of-the-art methods?

Table 4 and Figure 3 show the accuracy of FedMe and baselines. Table 4 shows average accuracy and standard deviation, and Figure 3 shows the validation accuracy at each global communication round.

From Table 4, we can see that FedMe achieves the highest accuracy among federated learning methods for all setting and its accuracy is very close to accuracy of Centralized. We here note that standard deviations of FEMNIST and Shakespeare are relatively large because clients differ in each test. FedMe achieves the lowerest (or the runner-up) standard deviation among federated learning methods for all settings, so we confirm that FedMe is the most robust among them. This result shows that its learning process is effective.

Comparing the baselines, it is interesting in that FedAvg, which is the most simple method with fine-tuning, achieves the highest accuracy among baselines. This result indicates that data heterogeneity can be solved by fine tuning. We show more experiments related to data heterogeneity and fine tuning in Section 5.2.5.

From Figure 3, we can see that FedAvg and FedMe generally have high accuracy at early rounds. This indicates that FedMe is early converge as the same as FedAvg.

5.2.2. Q2. Does automatic model architecture tuning work well?

We here show how well FedMe tunes optimal model architecture automatically. Table 5 shows the accuracy of FedMe with fixed model architectures. Model 1–4 are 1–4 CNN layers in FEMNIST, (VGG $11$ , VGG $13$ , VGG $16$ , and VGG $19$ ) in CIFAR-10, and 1–4 LSTM laysers in Shakespeare, respectively. For all setting, the accuracy of auto-tuning is middle among model 1–4. In particular, in FEMNIST, the accuracy of auto-tuning is comparable to the highest accuracy of the pre-determined model architecture. This result indicates that automatic model architecture tuning is effective without pre-defining the model architectures, so we can remove the cost to manually tune the model architecture.

Table 5. Impact of automatic model architecture tuning.


	FEMNIST	CIFAR-10	Shakespeare
model $1$	$74.80\pm 2.75$	$89.25\pm 0.74$	$45.31\pm 3.20$
model $2$	$78.06\pm 3.00$	$90.96\pm 0.84$	$45.83\pm 2.48$
model $3$	$77.85\pm 2.90$	$90.67\pm 0.47$	$46.01\pm 2.72$
model $4$	$78.54\pm 2.92$	$90.45\pm 0.54$	$42.55\pm 5.12$
auto-tuning	$78.52\pm 2.64$	$89.76\pm 0.90$	$44.71\pm 1.12$

Table 6. Comparison of test accuracy when removing each optimization technique of FedMe. MT, MC, and DML indicate model tuning, model clustering, and deep mutual learning, respectively.


MT	DML	MC	FEMNIST	CIFAR-10	Shakespeare
			75.85 $\pm$ 3.27	88.19 $\pm$ 0.53	37.27 $\pm$ 3.42
✓			76.46 $\pm$ 3.28	89.76 $\pm$ 1.21	45.59 $\pm$ 4.04
	✓		76.29 $\pm$ 4.22	86.92 $\pm$ 2.63	42.95 $\pm$ 5.05
		✓	75.87 $\pm$ 3.60	88.14 $\pm$ 0.93	36.86 $\pm$ 2.54
	✓	✓	76.13 $\pm$ 3.71	86.73 $\pm$ 2.54	43.30 $\pm$ 1.24
✓	✓		78.76 $\pm$ 2.26	89.67 $\pm$ 0.87	45.22 $\pm$ 3.70
✓		✓	77.64 $\pm$ 3.52	89.79 $\pm$ 1.23	46.32 $\pm$ 3.33
✓	✓	✓	78.52 $\pm$ 2.64	89.76 $\pm$ 0.90	44.71 $\pm$ 1.12

Table 7. Impact of data heterogeneity and fine-tuning.


	IID	$\alpha_{label}=5$	$\alpha_{label}=0.5$	$\alpha_{label}=0.1$
Centralized w/o fine-tuning	$86.50\pm 0.59$	$85.67\pm 0.67$	$85.96\pm 0.54$	$86.20\pm 0.95$
Centralized w/ fine-tuning	$86.45\pm 0.34$	$87.19\pm 0.56$	$90.80\pm 0.92$	$95.36\pm 1.16$
FedAvg w/o fine-tuning	$87.00\pm 0.30$	$86.64\pm 0.34$	$86.05\pm 0.48$	$80.86\pm 1.95$
FedAvg w/ fine-tuning	$85.91\pm 0.67$	$86.40\pm 0.84$	$89.59\pm 0.94$	$\textbf{94.60}\pm 1.12$
HypCluster w/o fine-tuning	$\textbf{87.10}\pm 0.30$	$85.21\pm 0.51$	$85.21\pm 1.22$	$82.43\pm 1.23$
HypCluster w/ fine-tuning	$86.03\pm 0.60$	$84.45\pm 0.48$	$88.54\pm 1.42$	$94.03\pm 1.42$
pFedMe	$33.14\pm 31.71$	$70.13\pm 0.87$	$79.46\pm 2.08$	$86.09\pm 3.38$
FedMe w/o fine-tuning	$85.78\pm 1.17$	$86.13\pm 1.07$	$88.15\pm 0.52$	$85.22\pm 2.92$
FedMe w/ fine-tuning	$86.04\pm 0.99$	$\textbf{86.94}\pm 1.15$	$\textbf{90.96}\pm 0.84$	$94.18\pm 1.24$

5.2.3. Q3. What techniques of FedMe impact to the inference accuracy?

We investigate the impact of optimization techniques of FedMe to the accuracy. FedMe uses the three optimization techniques; model tuning (MT), deep mutual learning (DML), and model clustering (MC). Table 6 shows the results that FedMe either partially or fully uses optimization techniques.

From Table 6, we can see that the accuracy of FedMe with all optimization techniques is higher than that without all techniques. We first see how much the accuracy improves when FedMe uses a single optimization technique. The model tuning has the most impact among the optimization techniques in all setting. Since they use accurate models more than initial models, the accuracy improves. Deep mutual learning also improves the accuracy except for CIFAR-10. The result indicates that deep mutual learning is effective in mutually learning personalized and exchanged models for leveraging predictions of models. Different from model tuning and deep mutual learning, model clustering does not improve the accuracy. We design the model clustering to combine deep mutual learning, so the model clustering itself is not effective.

Next, we investigate the combinations of optimization techniques. The accuracy of FedMe with two optimization techniques is generally higher than that of FedMe with a single optimization technique. The result indicates that each optimization technique has effective interaction to improve the accuracy. For example, in Shakespeare, FedMe with deep mutual learning and model clustering achieves higher accuracy than that with deep mutual learning though model clustering itself is not effective. While, some combinations decrease the accuracy, for example, DML+MC in FEMNIST and MT and DML in CIFAR-10. This deterioration is caused by ineffectiveness of model tuning and model clustering methods. We have research opportunities to improve the accuracy more, so we remain these tasks as our future work.

5.2.4. Q4.How fast is the learning process of FedMe compared with the state-of-the-art methods?

We evaluate run time on training phase in each method. Figure 4 shows the average run time of client and server process on global communication rounds. From this result, we can see that FedMe’s running time on clients is competitive with other methods, though FedMe trains two models. On the other hand, FedMe’s running time on server is larger than other methods because the server on FedMe uses model clustering after obtaining the outputs of all the personalized models. This is a time-consuming task compared with other methods. Note that we use the same hardware for the server and the clients in our experiments, while the server generally has more powerful computing resources in real-world scenarios. This means the computation cost on the server tends to be smaller in real-world applications. In addition, we can also control the computation cost of the server by changing the size of unlabeled data. Thus, seeing the accuracy gain of FedMe, we believe that the computation cost on the server is affordable.

(a) Client

(b) Server

Figure 4. Run time per global communication round.

5.2.5. Q5. What is the impact of data heterogeneity and fine-tuning on FedMe?

We finally investigate the impact of data heterogeneity. We conduct experiments using the CIFAR-10 dataset varying the degree of data heterogeneity controlled by $\alpha_{label}$ . The smaller $\alpha_{label}$ indicates greater data heterogeneity, and IID indicates that data distribution and the size of local data on clients are the same. We compare FedMe with Centralized and the top three most accurate existing methods in Table 4, FedAvg, HypCluster, and pFedMe. Table 7 shows the accuracy of each method with and without fine-tuning. First, FedAvg and HypCluster are equally accurate and have the highest accuracy in IID. The result indicates that when data is distributed in IID, it is enough to average model parameters of models on clients. We can also see that fine-tuning is not effective in IID. As $\alpha_{label}$ decreases (i.e., data heterogeneity becomes greater), the accuracy of methods without fine-tuning decreases, but that of methods with fine-tuning increases. This is because labels of local data have skews, training and test datasets have similar labels. The accuracy of pFedMe, which is the personalized federated learning method, also increases as $\alpha_{label}$ decreases. However, pFedMe always reports the worst performance (compared to other fine-tuning methods) even though it was designed as a solution to high degree of data heterogeneity and includes techniques similar to fine-tuning. This result was not observed in previous studies. For FedMe, when equipped with fine-tuning, it is the best (or the runner-up) method when local data is not IID. FedMe without fine-tuning is also the best (or the runner-up) method among the methods without fine-tuning. These results show that FedMe works well for high degree of heterogeneity and demonstrate the robustness of FedMe when fine-tuning is absent.

6. Conclusion and Future Work

In this paper, we presented FedMe, a novel federated learning method that builds personalized models with automatic model architecture tuning. In FedMe, clients exchange their models to tune and train their personalized models. FedMe can train models with different architectures by exchanging models and deep mutual learning. Our experiments showed that FedMe is more accurate than the state-of-the-art methods and can automatically tune the model architecture.

As our future work, we plan to extend model tuning and model clustering methods for tuning model architecture more flexibly. Although FedMe automatically tunes the model architecture, the candidates are only model architectures that clients design in advance. Thus, FedMe may not work well if optimal model architectures are not designed. We can improve model tuning methods for tuning models more flexibly, such as network architecture search. Additionally, model clustering is not effective much, so we can extend it to improve the accuracy.

References

[1] D. Anguita, A. Ghio, L. Oneto, X. Parra, and J. L. Reyes-Ortiz. A public domain dataset for human activity recognition using smartphones. In European Symposium on Artificial Neural Networks, volume 3, pages 437–442, 2013.
[2] S. Caldas, S. M. K. Duddu, P. Wu, T. Li, J. Konečnỳ, H. B. McMahan, V. Smith, and A. Talwalkar. Leaf: A benchmark for federated settings. arXiv preprint arXiv:1812.01097, 2018.
[3] H.-Y. Chen and W.-L. Chao. Fedbe: Making bayesian model ensemble applicable to federated learning. In International Conference on Learning Representations, 2021.
[4] J. Gao, Z. Li, R. Nevatia, et al. Knowledge concentration: Learning 100k object classifiers in a single cnn. arXiv preprint arXiv:1711.07607, 2017.
[5] N. Guha, A. Talwalkar, and V. Smith. One-shot federated learning. arXiv preprint arXiv:1902.11175, 2019.
[6] A. Hard, K. Rao, R. Mathews, S. Ramaswamy, F. Beaufays, S. Augenstein, H. Eichner, C. Kiddon, and D. Ramage. Federated learning for mobile keyboard prediction. arXiv preprint arXiv:1811.03604, 2018.
[7] C. He, M. Annavaram, and S. Avestimehr. Fednas: Federated deep learning via neural architecture search. In CVPR 2020 Workshop on Neural Architecture Search and Beyond for Representation Learning, 2020.
[8] C. He, H. Ye, L. Shen, and T. Zhang. Milenas: Efficient neural architecture search via mixed-level reformulation. In IEEE Conference on Computer Vision and Pattern Recognition, pages 11993–12002, 2020.
[9] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings, et al. Advances and open problems in federated learning. arXiv preprint arXiv:1912.04977, 2019.
[10] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. technical report, 2009.
[11] D. Leroy, A. Coucke, T. Lavril, T. Gisselbrecht, and J. Dureau. Federated learning for keyword spotting. In International Conference on Acoustics, Speech and Signal Processing, pages 6341–6345, 2019.
[12] D. Li and J. Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581, 2019.
[13] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith. Federated optimization in heterogeneous networks. In Machine Learning and Systems, volume 2, pages 429–450, 2020.
[14] X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang. On the convergence of fedavg on non-iid data. In International Conference on Learning Representations, 2019.
[15] W. Y. B. Lim, N. C. Luong, D. T. Hoang, Y. Jiao, Y.-C. Liang, Q. Yang, D. Niyato, and C. Miao. Federated learning in mobile edge networks: A comprehensive survey. IEEE Communications Surveys & Tutorials, 2020.
[16] L. Liu, J. Zhang, S. Song, and K. B. Letaief. Edge-assisted hierarchical federated learning with non-iid data. arXiv preprint arXiv:1905.06641, 2019.
[17] J. MacQueen et al. Some methods for classification and analysis of multivariate observations. In Berkeley symposium on mathematical statistics and probability, volume 1(14), pages 281–297, 1967.
[18] Y. Mansour, M. Mohri, J. Ro, and A. T. Suresh. Three approaches for personalization with applications to federated learning. arXiv preprint arXiv:2002.10619, 2020.
[19] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, pages 1273–1282, 2017.
[20] V. Mothukuri, R. M. Parizi, S. Pouriyeh, Y. Huang, A. Dehghantanha, and G. Srivastava. A survey on security and privacy of federated learning. Future Generation Computer Systems, 115:619–640, 2021.
[21] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, pages 8026–8037, 2019.
[22] S. Reddi, Z. Charles, M. Zaheer, Z. Garrett, K. Rush, J. Konečnỳ, S. Kumar, and H. B. McMahan. Adaptive federated optimization. arXiv preprint arXiv:2003.00295, 2020.
[23] T. Shen, J. Zhang, X. Jia, F. Zhang, G. Huang, P. Zhou, K. Kuang, F. Wu, and C. Wu. Federated mutual learning. arXiv preprint arXiv:2006.16765, 2020.
[24] C. T. Dinh, N. Tran, and J. Nguyen. Personalized federated learning with moreau envelopes. In Advances in Neural Information Processing Systems, pages 21394–21405, 2020.
[25] H. Wang, M. Yurochkin, Y. Sun, D. Papailiopoulos, and Y. Khazaeni. Federated learning with matched averaging. In International Conference on Learning Representations, 2020.
[26] M. Zhang, K. Sapra, S. Fidler, S. Yeung, and J. M. Alvarez. Personalized federated learning with first order model optimization. In International Conference on Learning Representations, 2021.
[27] Y. Zhang, T. Xiang, T. M. Hospedales, and H. Lu. Deep mutual learning. In Conference on Computer Vision and Pattern Recognition, pages 4320–4328, 2018.
[28] B. Zoph and Q. V. Le. Neural architecture search with reinforcement learning. In International Conference on Learning Representations, 2017.