On Privacy of Quantized Sensor Measurements through Additive Noise

During the past half-century, scientific and technological advances have greatly improved the performance of engineering systems. However, these new technologies have also led to vulnerabilities within critical infrastructure – e.g., power, water, transportation. Advances in communication and computing power have given rise to adversaries with enhanced and adaptive capabilities. Depending on adversary’s resources and system defenses, opponents may infer critical information about the operation of systems or even deteriorate their functionality. Therefore, designing efficient defence mechanisms is of importance for guaranteeing privacy, safety, and proper operation of critical systems. All these new challenges have attracted the attention of researchers from different fields (e.g., computer science, information theory, control theory) in the broad area of privacy and security of Cyber-Physical Systems (CPS) [1]-[16].

In most engineering applications, information about the state of systems is obtained through sensor measurements. Once this information is collected, it is usually quantized, encoded, and sent to a remote station for signal processing and decision-making purposes through communication networks. Examples of such systems are numerous: water and electricity consumption meters, traffic monitoring systems, industrial control systems, and so on. If the communication network is public or unsecured, adversaries might access and estimate the state of the system. To avoid an accurate state estimation, we add random noise to the quantized sensor measurements before transmission and send the sum to the remote station instead. This noise is designed to increase privacy of the transmitted data. Note, however, that it is not desired to overly distort the original sensor data by injecting noise. We might change the data excessively for practical purposes. Hence, when designing the additive noise, we need to take into account the trade-off between privacy and distortion.

In this manuscript, we follow an information-theoretic approach. We propose to use mutual information between quantized-sensor-data and quantized-sensor-data plus privacy noise as privacy metric, and the mean square error between them as distortion metric. The design of the discrete additive noise is posed as a convex optimization problem. In particular, the distribution of the noise is designed to minimize the mutual information for a desired level of maximal distortion.

The use of additive noise to increase privacy is common practice. In the context of privacy of databases, a popular approach is differential privacy [12]-[17], where noise is added to the response of queries so that private information stored in the database cannot be inferred. In differential privacy, because it provides certain privacy guarantees, Laplace noise is usually used [18]. However, when maximal privacy with minimal distortion is desired, Laplace noise is generally not the optimal solution. This raises the fundamental question: for a given allowable distortion level, what is the noise distribution achieving maximal privacy? This question has many possible answers depending on the particular privacy and distortion metrics being considered and the system configuration [19]-[22]. There are also results addressing this question from an informationtheoretic perspective, where information metrics – e.g., mutual information, entropy, Kullback-Leibler divergence, and Fisher information – are used to quantify privacy [1]-[4],[23]-[25].

In general, if the data to be kept private follows continuous distributions, the problem of finding the optimal additive noise to maximize privacy (even without considering distortion) is hard to solve. If a close-form solution for the distribution is desired, the problem amounts to solving a set of nonlinear partial differential equations which, in general, might not have a solution, and even if they do have a solution, it is hard to find [1]. This problem has been addressed by imposing some particular structure on the considered distributions or assuming the data to be kept private is deterministic [1],[20],[21].

The authors in [20],[21] consider deterministic input data sets and treat optimal distributions as distributions that concentrate probability around zero as much as possible while ensuring differential privacy. Under this framework, they obtain a family of piecewise constant density functions that achieve minimal distortion for a given level of privacy. In [1], the authors consider the problem of preserving the privacy of deterministic databases using constrained additive noise. They use the Fisher information and the Cramer-Rao bound to construct a privacy metric between noise-free data and the one with the additive noise and find the probability density function that minimizes it. Moreover, they prove that, in the unconstrained case, the optimal noise distribution minimizing the Fisher information is Gaussian.

Most of the aforementioned papers propose optimal continuous distributions assuming deterministic data. However, in a networked context, unavoidable sensor noise leads to stochastic data and thus existing tools do not fit this setting. Here, we identify two possibilities for addressing our problem: 1) we might inject continuous noise to sensor measurements, then quantize the sum, and send it over the unsecured network; or 2), the one considered here, quantize sensor measurements, add noise with discrete distribution, and send the sum over the network. As motivated above, to address the first option, even assuming deterministic sensor data, we have to impose some particular structure on the distributions of the additive noise; and, if sensor data is stochastic, the problem becomes hard to solve (sometimes even untractable). As we prove in this manuscript, if we select the second alternative, under some mild assumptions on the alphabet of the injected noise, we can cast the problem of finding the optimal noise as a constrained convex optimization. To the best of the authors knowledge, this problem has not been considered before as it is posed it here.

Let $X\in{\mathds{R}}^{n}$ be the state of some deterministic process that must be kept private. Information about the state is obtained through $m$ sensors of the form:

$$Y=CX+W,$$

(1)

with sensor measurements $Y\in{\mathds{R}}^{m}$, matrix $C\in{\mathds{R}}^{m\times n}$, and sensor noise $W\in{\mathds{R}}^{m}$, $E[W]=\mathbf{0}$, $\Sigma_{W}:=E[WW^{T}]$, $\Sigma_{W}>0$. The entries of the noise are uncorrelated, i.e., $\Sigma_{W}=\text{diag}[\sigma_{1}^{2},\ldots,\sigma_{m}^{2}]$. Then, $E[Y]=CX$, the covariance $\Sigma_{Y}:=E[(Y-CX)(Y-CX)^{T}]=\Sigma_{W}$, and the entries of $Y$ are uncorrelated. We assume that the probability distribution of $Y$ is known. This is not an strong assumption since it is often possible to obtain a number of realization of $Y$ to estimate its distribution. Let $Y=(Y_{1},\ldots,Y_{m})^{T}$. Each sensor measurement $Y_{i}$, $i=1,\ldots,m$ is quantized using a uniform quantizer on a finite range $Q_{i}(Y_{i},y_{i}^{1},\Delta_{i},N_{i})$:

$$Q_{i}(Y_{i},y_{i}^{1},\Delta_{i},N_{i}):=\small\left\{\begin{array}[]{l}y^{1}_{i}$ if $Y_{i}\in(-\infty,y^{1}_{i}+\frac{\Delta_{i}}{2}],\\[5.69054pt] y^{2}_{i}$ if $Y_{i}\in(y^{1}_{i}+\frac{\Delta_{i}}{2},y^{2}_{i}+\frac{\Delta_{i}}{2}],\\[5.69054pt] y^{3}_{i}$ if $Y_{i}\in(y^{2}_{i}+\frac{\Delta_{i}}{2},y^{3}_{i}+\frac{\Delta_{i}}{2}],\\[5.69054pt] \hskip 71.13188pt\vdots\\[4.2679pt] y^{N_{i}}_{i}$ if $Y_{i}\in(y^{N_{i}-1}_{i}+\frac{\Delta_{i}}{2},\infty),\end{array}\right.$$

(2)

where $y^{j}_{i}=y^{1}_{i}+(j-1)\Delta_{i}$, $j=1,\ldots,N_{i}$. Thus, for each sensor, the $N_{i}$ quantization levels are given by

$$\mathcal{Y}_{i}^{Q}:=\{y^{1}_{i},y^{1}_{i}+\Delta_{i},\ldots,y^{1}_{i}+(N_{i}-1)\Delta_{i}\}.$$

It follows that the vector of quantized sensor measurements $Y^{Q}:=(Y^{Q}_{1},\ldots,Y^{Q}_{m})^{T}$, $Y^{Q}_{i}:=Q_{i}(Y_{i},y_{i}^{1},\Delta_{i},N_{i})$ is determined by the initial quantization level $y^{1}_{i}\in{\mathds{R}}$, the quantization step $\Delta_{i}\in{\mathds{R}}_{>0}$, and the number of intervals $N_{i}\in{\mathds{N}}$, $i=1,\ldots,m$. Note that, because we know the distribution of $Y$ and the quantizer, we can always obtain the probability mass function $p(y^{Q})$ of $Y^{Q}$ (and thus also $p(y^{Q}_{i})$ of $Y^{Q}_{i}$). Moreover, the alphabet of the discrete random variable $Y^{Q}_{i}$ is the set of quantization levels $\mathcal{Y}^{Q}_{i}$.

After $Y$ is quantized, a random vector $Z$ is added to $Y^{Q}$ to obtain $V:=Z+Y^{Q}$. The vector $V$ is transmitted over an unsecured communication network to a remote station, see Fig. 1. Notice that, if we do not add $Z$ to $Y^{Q}$ before transmission, information about the state is directly accessible through the unsecured network. To minimize this information leakage, we send the sum $V=Z+Y^{Q}$ to the remote station instead of directly sending $Y^{Q}$. Note, however, that we do not want to make $Y^{Q}$ and $Y^{Q}+Z$ overly different either. By adding $Z$, we might distort $Y^{Q}$ excessively for any practical purposes. Hence, when designing the distribution of $Z$, we need to consider the trade-off between privacy and distortion. In this manuscript, we propose to use the mutual information between $V=Z+Y^{Q}$ and $Y^{Q}$, $I[V;Y^{Q}]$, as privacy metric, and the mean square error, $E[(V-Y^{Q})^{2}]$, as distortion metric. Thus, we aim at minimizing $I[V;Y^{Q}]$ using the probability mass function of $Z$, $p(z)$, as optimization variable subject to $E[(V-Y^{Q})^{2}]=E[Z^{2}]\leq\epsilon$, for a desired level of distortion $\epsilon\in{\mathds{R}}_{>0}$. In what follows, we formally present the optimization problem we seek to address.

To delimit the solution of Problem 1, we restrict the class of probability mass functions of $Z$. First, we fix the alphabet $\mathcal{Z}_{i}$ of $Z_{i}$ – the $i$-th component of $Z$ – to be equal to the alphabet $\mathcal{Y}^{Q}_{i}$ of $Y^{Q}_{i}$, i.e., equal to the quantization levels. This imposes a tractable convex structure on the objective and restrictions, and reduces the optimization variables to the probabilities of each element of the alphabet. The case with arbitrary alphabet leads to a combinatorial optimization problem where the objective of (3) changes its structure for different combinations. In this manuscript, we do not address this case; it is left as a future work.

Next, note that, because $X$ is deterministic and the covariance matrix $\Sigma_{W}$ is diagonal, the elements of the vector $Y^{Q}$ are mutually independent. Then, if we let $Z$ to have independent components, the objective function $I[Y^{Q}+Z;Y^{Q}]$ in (3) can be written as follows.

Proof: Proposition 1 follows from Lemma 4.

To impose a decoupled structure in the optimization problem, as pointed out in Remark 2, we consider individual constraints for the distortion, i.e., $E[Z_{i}^{2}]\leq\epsilon_{i}$, $\epsilon_{i}\in{\mathds{R}}_{\geq 0}$, $i=1,\ldots,m$. Then, we can replace (3) by the following $m$ decoupled optimization problems:

$$\left\{\begin{aligned} &\min_{p(z_{i})}\ H[V_{i}]-H[Z_{i}],\\ &\hskip 2.84526pt\text{\emph{s.t. }}E[Z_{i}^{2}]\leq\epsilon_{i},\hskip 5.69054pti=1,\ldots,m,\end{aligned}\right.$$

(4)

where $p(z_{i})$ denotes the probability mass function of $Z_{i}$ and $\epsilon_{i}$ is the desired distortion level associated with the mean square error $E[(V_{i}-Y_{i}^{Q})^{2}]$.

In what follows, we focus on the solution of (4) assuming independence of $Z$ and restricting the alphabet $\mathcal{Z}_{i}$ of $Z_{i}$ to be equal to $\mathcal{Y}^{Q}_{i}$.

Next, we write $I[Y_{i}^{Q}+Z_{i};Y_{i}^{Q}]=H[V_{i}]-H[Z_{i}]$ in (4) in terms of $p(y^{Q}_{i})$ and $p(z_{i})$. Denote the probabilities of $Y_{i}^{Q}$ and $Z_{i}$ as follows:

	$\displaystyle p_{i,j}^{Y}$	$\displaystyle:=\text{Pr}[Y_{i}^{Q}=y_{i}^{j}],$		(5)
	$\displaystyle p_{i,j}^{Z}$	$\displaystyle:=\text{Pr}[Z_{i}=y_{i}^{j}],$		(6)

with $j=1,\ldots,N_{i}$. Then, the entropy $H[Z_{i}]$ is given by $H[Z_{i}]=-\sum_{j=1}^{N_{i}}p_{i,j}^{Z}\log p_{i,j}^{Z}$ and $E[Z_{i}^{2}]=\sum_{j=1}^{N_{i}}(y_{i}^{j})^{2}p_{i,j}^{Z}$. Moreover, since $y_{i}^{j}=y^{1}_{i}+(j-1)\Delta_{i}$, then, in terms of the quantizer parameters, $E[Z_{i}^{2}]=\sum_{j=1}^{N_{i}}(y^{1}_{i}+(j-1)\Delta_{i})^{2}p_{i,j}^{Z}$. To get an expression for $H[V_{i}]$, we need the probability mass function $p(v_{i})$ of $V_{i}$. We compute all the possible elements of the alphabet of $V_{i}=Y_{i}^{Q}+Z_{i}$ and their corresponding probabilities in terms of the elements of the alphabet $\mathcal{Y}^{Q}_{i}$, $y_{i}^{j}=y^{1}_{i}+(j-1)\Delta_{i}$. Thus, the random variable $V_{i}$ has an alphabet with $2N_{i}-1$ elements and the corresponding probabilities are the sums of the probabilities of equal elements. The probability mass function $p(v_{i})$ of $V_{i}$ is given in Table I.

Now, we can write an explicit expression for the objective function in (4):

	$\displaystyle I[Y_{i}^{Q}+Z_{i};Y_{i}^{Q}]$	$\displaystyle=H[V_{i}]-H[Z_{i}],$		(7)
		$\displaystyle=-\sum_{j=1}^{2N_{i}-1}p_{i,j}^{V}\log p_{i,j}^{V}+\sum_{j=1}^{N_{i}}p_{i,j}^{Z}\log p_{i,j}^{Z},$

where

	$\displaystyle v_{i}^{j}$	$\displaystyle:=2y_{i}^{1}+(j-1)\Delta_{i},\hskip 2.84526ptj=1,\ldots,2N_{i}-1,$		(8)
	$\displaystyle p_{i,j}^{V}$	$\displaystyle:=\text{Pr}[V_{i}=v_{i}^{j}]$		(9)
		$\displaystyle=\small\left\{\begin{array}[]{l}\sum\limits_{k=1}^{j}p_{i,j+1-k}^{Y}p_{i,k}^{Z},\hskip 2.84526ptj=1,\ldots,N_{i},\\[2.84526pt] \sum\limits_{k=j+1-N_{i}}^{N_{i}}p_{i,j+1-k}^{Y}p_{i,k}^{Z},\hskip 2.84526ptj=N_{i}+1,\ldots,2N_{i}-1.\end{array}\normalsize\right.$

The expressions in (7)-(9) give a complete characterization of the objective $I[Y_{i}^{Q}+Z_{i};Y_{i}^{Q}]$ in terms of the known probabilities of the quantized sensors $p_{i,j}^{Y}$, $j=1,\ldots,N_{i}$, and the optimization variables, the probabilities of the injected noise $p_{i,j}^{Z}$, $j=1,\ldots,N_{i}$. Moreover, the distortion constraint $E[Z_{i}^{2}]=\sum_{j=1}^{N_{i}}(y^{1}_{i}+(j-1)\Delta_{i})^{2}p_{i,j}^{Z}\leq\epsilon_{i}$ is linear in $p_{i,j}^{Z}$. Therefore, if the objective is convex, we could, in principle, efficiently solve (4) numerically. However, since $I[Y_{i}^{Q}+Z_{i};Y_{i}^{Q}]=H[V_{i}]-H[Z_{i}]$, $H[V_{i}]$ is concave in $p_{i,j}^{V}$, and $H[Z_{i}]$ is concave in $p_{i,j}^{Z}$ [26], it is not clear whether $H[V_{i}]-H[Z_{i}]$ is convex in $p_{i,j}^{Z}$ or not.

Proof: Define the sum:

$\displaystyle f^{V}_{i}:$

$\displaystyle=-\sum_{j=2}^{2N_{i}-2}p_{i,j}^{V}\log p_{i,j}^{V}.$

The entropy of $V_{i}$ can be written in terms of $f^{V}_{i}$ as

	$\displaystyle H[V_{i}]$	$\displaystyle=f^{V}_{i}-p_{i,1}^{V}\log p_{i,1}^{V}-p_{i,2N_{i}-1}^{V}\log p_{i,2N_{i}-1}^{V},$		(12)
		$\displaystyle=f^{V}_{i}-p_{i,1}^{Y}p_{i,1}^{Z}\log p_{i,1}^{Y}-p_{i,1}^{Y}p_{i,1}^{Z}\log p_{i,1}^{Z}$
		$\displaystyle\hskip 11.38109pt-p_{i,N_{i}}^{Y}p_{i,N_{i}}^{Z}\log p_{i,N_{i}}^{Y}-p_{i,N_{i}}^{Y}p_{i,N_{i}}^{Z}\log p_{i,N_{i}}^{Z},$

where the last equality follows from (9). Write the entropies $H[Z_{i}]$ and $H[Y_{i}^{Q}]$ as

	$\displaystyle H[Z_{i}]$	$\displaystyle=-\underbrace{\big(p_{i,1}^{Y}+\ldots+p_{i,N_{i}}^{Y}\big)}_{=1}\sum_{j=1}^{N_{i}}p_{i,j}^{Z}\log p_{i,j}^{Z},$		(13)
		$\displaystyle=:f^{Z}_{i}-p_{i,1}^{Y}p_{i,1}^{Z}\log p_{i,1}^{Z}-p_{i,N_{i}}^{Y}p_{i,N_{i}}^{Z}\log p_{i,N_{i}}^{Z},$
	$\displaystyle H[Y_{i}^{Q}]$	$\displaystyle=-\underbrace{\big(p_{i,1}^{Z}+\ldots+p_{i,N_{i}}^{Z}\big)}_{=1}\sum_{j=1}^{N_{i}}p_{i,j}^{Y}\log p_{i,j}^{Y},$		(14)
		$\displaystyle=:f^{Y}_{i}-p_{i,1}^{Y}p_{i,1}^{Z}\log p_{i,1}^{Y}-p_{i,N_{i}}^{Y}p_{i,N_{i}}^{Z}\log p_{i,N_{i}}^{Y}.$

Combining (12)-(14), we can write

$$H[Z_{i}]+H[Y_{i}^{Q}]=f^{Z}_{i}+f^{Y}_{i}+H[V_{i}]-f^{V}_{i},$$

which implies $H[V_{i}]=H[Z_{i}]+H[Y_{i}^{Q}]+f^{V}_{i}-f^{Z}_{i}-f^{Y}_{i}$ and thus $I[Y_{i}^{Q}+Z_{i};Y_{i}^{Q}]=H[Y_{i}^{Q}]+f^{V}_{i}-f^{Z}_{i}-f^{Y}_{i}$. The entropy $H[Y_{i}^{Q}]$ is constant; then, $I[Y_{i}^{Q}+Z_{i};Y_{i}^{Q}]$ is convex if and only if $f_{i}(p_{i,1}^{Z},\ldots,p_{i,N_{i}}^{Z}):=f^{V}_{i}-f^{Z}_{i}-f^{Y}_{i}$ is convex. Next, collecting the $p_{i,j}^{Z}$ terms and using properties of logarithmic functions, we can write $f_{i}(p_{i,1}^{Z},\ldots,p_{i,N_{i}}^{Z})$ as follows

$$f_{i}=\left\{\begin{array}[]{l}\sum\limits_{k=1}^{j}p_{i,k}^{Y}p_{i,j+1-k}^{Z}\log\left(\dfrac{p_{i,k}^{Y}p_{i,j+1-k}^{Z}}{\sum_{l=1}^{j}p_{i,l}^{Y}p_{i,j+1-l}^{Z}}\right),\\[17.07164pt] $\hskip 0.28453pt for $j=2,\ldots,N_{i},\\[11.38109pt] \sum\limits_{k=j-1-N_{i}}^{N_{i}}p_{i,k}^{Y}p_{i,j+1-k}^{Z}\log\left(\dfrac{p_{i,k}^{Y}p_{i,j+1-k}^{Z}}{\sum_{l=1}^{j}p_{i,l}^{Y}p_{i,j+1-l}^{Z}}\right),\\[17.07164pt] $\hskip 0.28453pt for $j=N_{i}+1,\ldots,2N_{i}-2.\end{array}\right.$$

Note that every element of $f_{i}(p_{i,1}^{Z},\ldots,p_{i,N_{i}}^{Z})$ above is a function of the form $g(a,b,c,\ldots,r)=a\log\big(\frac{a}{a+b+c+\cdots+r}\big)$, $a,b,c\ldots,r\in[0,1]$. The function $g(a,b,c,\ldots,r)$ can be proved to be convex using Theorem 2.7.1 in [26] – the log sum inequality. Hence, $f_{i}(p_{i,1}^{Z},\ldots,p_{i,N_{i}}^{Z})$ is the sum of convex functions and thus convex as well. $\blacksquare$

Note that the ultimate goal is to make it hard for adversaries to infer $X$ from $V=Y^{Q}+Z$. That is, if someone estimates $X$ using the available data at the public network $V$, the estimation $\hat{X}(V)$ should carry less information about $X$ than an estimate $\hat{X}(Y^{Q})$ obtained using $Y^{Q}$ directly. In other words, we want to make $I[\hat{X}(V);\hat{X}(Y^{Q})]$ small.

Proof: The assertion follows from property (P₃) in Section II – the data processing inequality [26].

Consider system (1) with $X=(\pi^{2},\pi^{2}/4)^{T}$, $C=I_{2}$, and sensor noise $W=(W_{1},W_{2})^{T}$, $W_{1}\sim\mathcal{N}(0,\sigma_{1}^{2})$, $\sigma_{1}^{2}=\pi$, $W_{2}\sim\mathcal{U}(-a,a)$, $a=\pi^{2}/40$, $\sigma_{2}^{2}=(1/3)a^{2}$. Each sensor measurement $Y_{i}$, $i=1,2$, is quantized using the uniform quantizer (2) with $y_{1}^{1}=\pi^{2}-3\sigma_{1}$, $\Delta_{1}=6\sigma_{1}/N_{1}$, $N_{1}=11$,and $y_{2}^{1}=9.09a$, $\Delta_{2}=2a/N_{2}$, $N_{2}=11$. In Figure 2, we show the optimal distribution $p(z_{1})$ of $Z_{1}$ solution of (4), first without the distortion constraint, and then for the distortion levels $\epsilon_{1}=60,40$. The distortion level for the unconstrained case is $E[Z_{1}^{2}]=105.03$. For comparison, we also show the distributions $p(y_{1}^{Q})$ of $Y_{1}^{Q}$ and the one of the sum $V_{1}=Y_{1}^{Q}+Z_{1}$, $p(v_{1})$. In Figure 3, we show the corresponding results for sensor 2: the optimal distributions for the unconstrained case, which yields $E[Z_{2}^{2}]=6.10$, and then for the distortion levels $\epsilon_{2}=5.6,5.1$.