Reducing Risk of Model Inversion Using Privacy-Guided Training

In this method, each feature is assigned a weight according to its sensitivity level: the higher the sensitivity, the higher the weight. Weight values are in the range of 0 to 1, 0 signifying a non-sensitive feature. During the training process, when searching for the split with minimum entropy (or Gini index) at each stage, we use the feature’s weight as a penalty. For a node $i$ that splits on feature $f$ with weight $w_{f}$, we replace the original entropy of the node with:

(The addition of 1 ensures that a feature with weight 0 will remain with its original entropy). This is analogous to the class weighting method described by Polo et al. (Polo et al. 2007). However, they assigned different weights to different classes in order to represent the relative importance of each class, whereas we assign different weights to different features. After performing this weighted entropy calculation, features with higher sensitivity (i.e., weight) will yield higher entropy values, thus penalizing splits on more sensitive features.

Building on an idea previously proposed as a countermeasure against tree inversion (Fredrikson et al. 2015), this method limits the levels at which splits can be made on a sensitive feature. In the original method proposed by Fredrikson et al., the sensitive feature is only considered for splitting after $l-1$ other features have already been selected and removed from consideration afterwards. However, we define a threshold for the highest level in the tree at which the sensitive features may be split. In other words, at any level higher than the threshold, these features are not considered for splitting; once the threshold is reached, they may be considered for any subsequent split. We consider the level as the distance from the root of the tree.

This method does not directly limit the number of splits that can be performed on a certain feature. That said, it does affect the feature’s importance by forcing it to be used in splits that affect fewer samples (the higher the split in the tree, the more samples it affects, and vice versa).

The threshold level for being considered for a split is customizable, and may differ between datasets and model types. Intuitively, the deeper the trees in the initial model, the higher this threshold will have to be in order to effectively limit the importance of sensitive features. The results presented in Section 4 were obtained by varying this parameter to yield different importance/accuracy trade-offs.

Since the importance of a feature is summed over all nodes that split on that feature, we decided to try directly limiting the number of splits on sensitive features. The number of splits for each feature is counted. Once the threshold is met, that feature is no longer considered for splitting. To avoid creating an imbalance in the tree, where a certain feature is used heavily on the left side of the tree and then no longer used on the right side, we implemented a breadth-first-search (BFS) version of tree training. Thus, a feature is considered for splitting in a balanced manner near the top of the tree, and is no longer considered towards the bottom, once the maximum number of splits is reached.

The number of allowed splits is customizable, and may differ between datasets and model types. Here, again, the larger the initial trees (i.e., the number of internal nodes in the tree), the higher this threshold will have to be. The results presented in Section 4 were obtained by varying this parameter to yield different importance/accuracy trade-offs.

We also implemented weighted variants of methods 3.2 and 3.3, a normalized variant of method 3.3 (where the split number is normalized by the number of samples in each split), as well as different combinations of all of the above methods. However, these did not significantly improve our results.

For each dataset, we first chose different random subsets of features to define as sensitive features. Next, we built a privacy-aware model that takes as input the sensitivity level of each feature. We recorded both the accuracy of the resulting model on our hold-out test dataset, as well as the new importance values of the selected features. We ran experiments on all combinations of dataset, model type and privacy-guided method (described in Section 3). Select results for a single feature are shown in Figures 1 and 2. For reproducibility purposes, the threshold/weight values used in each run are brought in the Appendix.

Not all of the methods for reducing the importance of sensitive features worked well in all cases. Some methods worked better for certain model types, datasets, or features that we tested. Table 1 presents these results. We marked a combination of method and model type as successful (with an X) if we were able to find at least one feature for which the method worked. In many cases, we were able to reduce the importance without harming accuracy too much. This was even the case for some of the most important features (initially), as in the example of the ‘age’ and ‘children’ features in the GSS dataset, which were consistently found to be the most important features across all models before applying privacy-guided training. In some cases it was even possible to bring them down to 0 importance without harming accuracy, as shown in Figure 2. Surprisingly, in some cases, the accuracy was even improved. This is another indication that the choice of features to split made during model training is not always optimal.

This does not mean, however, that these mothods necessarily work for any feature in the dataset. In fact, there were some features whose importance could not be reduced without significantly harming the model’s accuracy.

We also conducted experiments with multiple sensitive features. These experiments were performed with sets of randomly selected features. Results for the Nursery dataset, decision tree model, and method 3.1 (weights) are presented in Table 2. In addition to the model’s accuracy, we show the sum of importance losses over all sensitive features. The higher this number, the better since this represents the degree by which the importance of all of these features decreased collectively. Our results show that even with relatively high numbers of sensitive attributes (up to 6 out of 14), it is possible to gain a meaningful reduction in the importance of these features with a very low impact on the model’s accuracy. The last line in the table, displaying an accumulated importance loss of 0.264, corresponds to an overall initial importance of 0.507 summed over those 6 features; this thus represents a 50% combined reduction in the importance of these features.

Next, we chose one categorical feature in each dataset that had a relatively high importance, and could not simply be removed from the model without causing a significant decline in model accuracy. We then tried to infer the value of that feature using black-box (bb) and white-box (wb) model inversion attacks. The reasoning behind this choice was that if a feature can simply be removed without harming the accuracy of the model, there is no need for any special “privacy-aware” training process. It can simply be removed from the training. We found this to be the case, for example, for the ‘X-rated movies’ feature that was used in the inversion attacks described by Fredrikson et al. (Fredrikson et al. 2015).

For the GSS dataset we chose to attack the happiness feature, which had an initial importance of 0.315. For the Nursery dataset we chose the social feature, with an initial importance of 0.043. For simplicity of the attack implementation, we transformed each of these features to be Boolean, i.e., a feature that can only receive two values, 0 and 1. For the happiness feature, we assigned the value 1 to any record containing the initial value ‘Very happy’, and 0 for the rest, yielding a prior distribution of [0.6, 0.4] for the values 0 and 1, respectively. For the social feature, we assigned the value 1 to any record containing the initial value ‘problematic’ and 0 for the rest, yielding a prior distribution of [0.66, 0.34]. Here we also removed any rows missing values for the attacked feature.

We recorded the accuracy of each type of attack, once using the original model and once using the equivalent privacy-aware model. The description of each of the attacks appears in the Appendix. We also compared these with the results of what we consider a “baseline” attack, referred to as the ideal adversary in (Fredrikson et al. 2015). This attack consists of training an ML model on the original dataset to predict the sensitive attribute. In our case, we used a neural network model for this “ideal” attack. Those results are presented in Figure 3.

These results demonstrate that it is indeed possible to reduce the success rate of model inversion attacks by reducing the importance of the sensitive (attacked) feature in the underlying model, as suggested by previous work. The baseline ‘ideal’ attack, which does not depend on the model, is not directly affected by these changes and remains more or less constant, as we expected.

We performed two additional experiments with the aim of disproving the argument that the attacks’ accuracy may deteriorate due to a decline in the original model’s accuracy. In the first experiment, we used our privacy-guided training algorithm to generate models with decreased accuracy due to the reduced importance of other features (not the attacked feature). We show in Figure 4 that the success rate of the attacks on the sensitive feature is not affected by this reduction in model accuracy and remains more or less static. This demonstrates that decreasing model accuracy alone, while keeping the attacked feature’s importance at the same level, is not effective in countering attribute inference attacks.

In the second experiment, which was done on the GSS dataset, we were able to use our privacy-guided training algorithm to cause an increase in the attacked feature’s importance (by decreasing the importance of other features in the model). In this experiment, we observed that the attacks’ success increases with the increase of the feature’s importance relative to the original model, with almost constant model accuracy. The results are shown in Figure 5. As expected, the ideal attack is not affected by these changes as it does not depend on the model.

There have been several attempts at membership inference attacks, where one can deduce whether a specific individual was part of the training set or not. Examples include the work of Shokri et al. (Shokri et al. 2017) and Salem et al. (Salem et al. 2019). As a result, a great body of prior work has focused on protecting the privacy of the training dataset used to train machine learning models from such attacks. Approaches that add noise during the training process have been proposed to protect people participating in the training set and counter membership attacks (Zhang et al. 2018). The idea behind this approach is to reduce the effect of any single individual on the model’s outcome. Some have even applied the principle of differential privacy to the training process (Abadi et al. 2016; Holohan et al. 2019), which provides more robust privacy guarantees. Papernot et al. (Papernot et al. 2018) show how Private Aggregation of Teacher Ensembles (PATE) can scale to learning tasks with large numbers of output classes and uncurated, imbalanced, training data with errors. The PATE approach transfers to a “student” model the knowledge of an ensemble of “teacher” models, with intuitive privacy provided by training teachers on disjoint data and strong privacy guaranteed by noisy aggregation of teachers’ answers.

However, a recent survey of existing techniques based on differential privacy (Jayaraman & Evans 2019) found that current mechanisms for differentially private machine learning rarely offer acceptable utility-privacy trade-offs for complex learning tasks. Settings that provide limited accuracy-loss provide little effective privacy, and settings that provide strong privacy result in useless models. Moreover, it was recently shown that differentially private training mechanisms have a disproportionate effect on the accuracy of the model for underrepresented or complex subgroups, leading to an increased bias towards these populations (Bagdasaryan & Shmatikov 2019).

Moreover, most of these methods are aimed at preventing membership inference attacks and not attribute inference attacks. Fredrikson et al. (Fredrikson et al. 2014) specifically evaluated the effectiveness of differential privacy to prevent their attack and reached the conclusion that for privacy budgets that are effective at preventing the attack, the utility of the model is lethally damaged, potentially exposing patients to increased risk of stroke, bleeding events, and mortality.

Recently, a technique based on adversarial regularization was proposed to protect models against membership inference (Nasr et al. 2018). The researchers introduced a mechanism that can train models with membership privacy, while ensuring indistinguishability between the predictions of a model on its training data and other data points (from the same distribution). They achieved this by minimizing the accuracy of the best black-box membership inference attack against the model, using an adversarial training algorithm that minimizes the prediction loss of the model as well as the maximum gain of the inference attacks.

Ateniese et al. (Ateniese et al. 2015) devised a different type of attack on machine learning models, which was aimed at stealing trade secrets. Another type of attack proposed by He, Zhang, and Lee (He et al. 2019) compromises the privacy of inference data in collaborative deep learning systems. In this setting, when a deep neural network and the corresponding inference task are split and distributed to different participants, they show that one malicious participant can accurately recover an arbitrary input fed into the system by another participant.

Zhao et al. (Zhao et al. 2019) present a theoretical framework for privacy-preservation under the attack of attribute inference. They propose a minimax optimization formulation to protect the given attribute and analyze its privacy guarantees against arbitrary adversaries. They also prove an information-theoretic lower bound to precisely characterize the fundamental trade-off between utility and privacy. They then proceed to analyze different existing privacy preservation algorithms to see if they uphold the lower bound in practice, and characterize their trade-off between privacy and utility.

Attriguard (Jia & Gong 2018) uses evasion attacks from adversarial machine learning to defend against attribute inference attacks. However their setting is quite different from ours. They assume that the attacker uses a machine learning classifier to infer target private attributes based on public data. They further assume that the defender does not have access to this model or to the private attributes. Their solution is based on adding noise directly to users’ public data. In our work, we assume that the defender is the owner of the machine learning model, and has access to both public and private attributes of the users used to train the model. Our goal is to prevent an attacker from inferring private attribute values from the public data and model.

Huang et al. (Huang et al. 2017) present a novel context-aware privacy framework called generative adversarial privacy (GAP). Under GAP, learning the privacy mechanism is formulated as a constrained minimax game between two players: a privatizer that sanitizes the dataset in a way that limits the risk of inference attacks, and an adversary that tries to infer the private variables from the sanitized dataset. This solution is intended for settings where a data holder would like to publish a dataset in a privacy preserving fashion; it is not geared specifically for building privacy-preserving ML models.

MLPrivacyGuard (Alves et al. 2019) is a countermeasure against black-box confidence-based model inversion attacks. It is based on adding controlled noise to the output of the confidence function, causing numerical approximation of gradient ascent to be unable to converge. This method was shown to be effective against attacks that apply gradient ascent on the confidence returned by the model. Whereas this method may be capable of countering a certain class of attacks, our method does not assume any specific attack. We demonstrate its effectiveness on several types of attacks, both black-box and white-box.

Additional work aimed specifically at tree-based models includes protocols for privately evaluating decision trees and random forests (Wu et al. 2016). This work was carried out in a client-server setting, where the server has a decision tree (or random forest) model and the client holds an input to the model. They aimed at preserving the following security property: at the end of the protocol execution, the server would learn nothing about the input, and the client would learn nothing about the model. Also based on the concept of secure multiparty computation, Fritchman et al. (Fritchman et al. 2018) presented a first framework for privacy-preserving classification of tree ensembles with application in healthcare. Obviously, both of these works tackle a very different setting than ours, which is to protect the privacy of the training set against attribute inference.

Truex et al. (Truex et al. 2017) revisited the concepts and techniques of privacy-preserving decision tree learning. They reviewed and compared the privacy notions and properties of three orthogonal, yet complimentary, technical frameworks: randomization-based data obfuscation, differential privacy, and secure multiparty computation, analyzing their effectiveness.