Solving Non-identifiable Latent Feature Models

In this paper, as is also seen in [14, 7], we consider identifiability in LFMs in terms of uniqueness of the solution. Before presenting our theoretical analyses, let us briefly overview (non-)identifiability in LFMs with an example.

In LFMs, non-identifiability difficulties are mainly due to discrete nature of $Z$. Since the optimization problem (3) for $W$ with fixed $Z$ is convex, the MAP solution $\hat{W}$ corresponding to $\hat{Z}$ (which is also an expectation $\mathbb{E}[W|\hat{Z},X]$) is uniquely calculated in closed form [1, 7]. The uniqueness of $Z$ is more complicated. Figure 1 (a, b) shows every possible combinations of $Z$ in two different $X$’s. The axes of the figures correspond to the terms of (3). In the identifiable case (a), only the ground truth has zero-residual $\|X-ZW\|_{F}=0$. However, in the non-identifiable case (b), there are many solutions having zero-residual, and inference methods minimizing the residual, such as [7, 11], may fall into incorrect zero-residual solution rather than ground truth. Moreover, zero-residual solutions (Fig. 1c) are distant from one another at Hamming distances, and this results in a multi-modal nature in a cost function preventing iterative algorithms, such as Markov-chain Monte Carlo (MCMC)[1, 12] and variational Bayesian (VB)[18] methods, from converging to a global optimum.

Let us next consider conditions for (non-)identifiability and properties of equivalent solutions. Since LFMs are a class of matrix factorizations, we start from identifiability in general matrix factorization [19, 7, 14] to get an overall picture.

We can consider wide-ranging classes of matrix factorization by choosing $\mathcal{Z}$ and $\mathcal{W}$. The LFM is the case of $\mathcal{Z}=\{0,1\}^{N\times K}$ and $\mathcal{W}=\mathbb{R}^{K\times D}$. We can further say that a matrix factorization of $X\in\mathbb{R}^{N\times D}$ is identifiable if one of the minimizers of a residual $\|X-ZW\|_{F}^{2}$ (i.e., one of the ML solutions) is identifiable. In noiseless settings ($\varepsilon\!=\!0$), the identifiability of $X$ is consistent with that of the ground truth $(Z^{*},W^{*})$.

It is known that most equivalent solutions of matrix factorization $Z^{\prime}W^{\prime}=ZW$ have a specific form. For a context of non-negative matrix factorization (NMF), where $\mathcal{Z}=\mathbb{R}_{+}^{N\times K}$ and $\mathcal{W}=\mathbb{R}_{+}^{K\times D}$, Laurberg, et al. [19] showed that all equivalent NMF solutions $Z^{\prime}W^{\prime}=ZW$ have a form $Z^{\prime}=ZU,W^{\prime}=U^{-1}W$ if $\mathrm{rank}(ZW)=K$. This assertion is also true in the general matrix factorization in Definition 1 since their proof only used properties of linear space.

Let us consider a set of equivalent solutions of an LFM to quantify (non-)identifiability. Let $[(Z,W)]$ be an equivalence class of an equivalence relation "$\sim$" defined as $(Z,W)\sim(Z^{\prime},W^{\prime})\,\Leftrightarrow\,ZW=Z^{\prime}W^{\prime}$. Namely, $[(Z,W)]=\left\{(Z^{\prime},W^{\prime})\in\mathcal{Z}\times\mathcal{W}\middle|ZW=Z^{\prime}W^{\prime}\right\}$.

Assuming $\mathrm{rank}(ZW)=K$, Theorem 1 guarantees that all elements of $[(Z,W)]$ will be represented as $({Z}U,U^{-1}{W})$. Then we can consider LFM ($\mathcal{Z}\!=\!\{0,1\}^{N\times K},\mathcal{W}\!=\!\mathbb{R}^{K\times D}$), the equivalence class to be expressible as $[(Z,W)]=\left\{({Z}U,U^{-1}{W})\middle|U\in H({Z})\right\}$, where

Non-identifiability in the LFM is characterized by distinct elements $U\in H(Z)$ other than permutation matrices $U\not\in\mathcal{S}_{K}$, where $\mathcal{S}_{K}$ is symmetric group of degree $K$. To reduce the degree of freedom in permutation, we assume a quotient set¹¹ 1 In a precise sense, $H(Z)/\mathcal{S}_{K}$ stand s for $H(Z)/\mathop{\sim_{\mathcal{S}_{K}}}$, the quotient of $H(Z)$ by an equivalence relation “$\sim_{\mathcal{S}_{K}}$” such that $U\mathop{\sim}_{\mathcal{S}_{K}}U^{\prime}\Leftrightarrow{}^{\exists}\sigma\in\mathcal{S}_{K},U^{\prime}=U\sigma$. $H(Z)/\mathcal{S}_{K}$, which is a set of sets, in which each set consists of transform matrices $U\in H(Z)$ having the same column entries but in different orders.

By using the notation above, non-identifiability can be quantified by a cardinality $|H(Z)/\mathcal{S}_{K}|$, which represents the distinct number of equivalent solutions avoiding duplication in permutations. Additionally, we get $|H(Z)/\mathcal{S}_{K}|=1$ as a necessary and sufficient condition for the identifiability under the assumption of $\mathrm{rank}(ZW)\!=\!K$. We consider $H(Z)/\mathcal{S}_{K}$ to be trivial if $H(Z)/\mathcal{S}_{K}$ has only one element . While the assumption $\mathrm{rank}(ZW)\!=\!K$ denotes both $\mathrm{rank}(Z)\!=\!K$ and $\mathrm{rank}(W)\!=\!K$, we further prove that the former is unnecessary, and we get a following theorem providing a necessary and sufficient condition for the identifiability:

Note that Theorem 2 is a stronger result than the “identifiability condition” mentioned in [7], which supplies only a sufficient condition for identifiability in the case of $\mathrm{rank}(ZW)\!=\!K$. While the assumption $\mathrm{rank}(W)\!=\!K$ in Theorem 2 is not necessarily required for the identifiability²² 2 We can show an example in which identifiability holds despite $\mathrm{rank}(W)\!<\!K$. See Supplemental Materials. , we assume $\mathrm{rank}(W)=K$ in the following discussion in consideration for $D\gg K$ in many applications.

Finally, in Figure 2, we summarize the relationship between the conditions we have derived: (Full-Z) $\mathrm{rank}(Z)=K$, (ID) the identifiability, and (Trivial) $|H(Z)/\mathcal{S}_{K}|=1$.

Non-identifiability in LFMs is significantly related to dependency between features. We illustrate here two sufficient conditions, both for identifiability and non-identifiability (i.i.d. and PDC in Figure 2), related to independence and dependence of features, respectively.

The first condition, the sufficient condition for identifiability, is the statistical independence of the features. [7] have shown that an LFM is identifiable with high probability if entries of $Z$ are i.i.d. Bernoulli $p\!=\!0.5$. More generally, identifiability holds with probability one in $N\!\to\!\infty$ for any $0\!<\!p\!<\!1$. We show this in the following theorem:

If $Z$ is i.i.d. Bernoulli with $0<p<1$, probabilities for every combination of each row$P(Z_{n,:})$ will be non-zero. Therefore, in the limit of $N\to\infty$, all the combinations $\{0,1\}^{K}$ may appear in rows of $Z$ with probability one, and this results in identifiability via Theorem 3. In other words, from a contraposition of Theorem 3, it can be said that possible non-identifiability is due to a lack of observed combinations. In real-world applications observing all $2^{K}$ combinations is rarely possible, and, as we will see later, some combinations may never appear in rows of $Z$ even if $N\to\infty$ because of hidden dependency between features.

The second condition is in regard to non-identifiability. We propose three pairwise dependency conditions (PDCs) sufficient for non-identifiability,

The PDCs in Theorem 4 often appear in real-world applications³³ 3 We can show the commonality of PDCs from our survey on actual datasets. See Supplemental Materials. , and, unfortunately, they are unknown in most cases. Inference methods may then suffer from non-identifiability whenever there exists at least a pair of features PDC holds. For instance, a typical case of PDC1 would be disjoint features. Assuming, for example, that these features correspond to characteristics of cats, then a pair of features $(i,j)=("\textrm{male}","\textrm{black}")$ may appear at the same time, but $(i,j)=("\textrm{male}","\textrm{female}")$ will not appear concurrently. In the case of PDC2 and 3, typical cases would be latent hierarchical structures. For example, in considering features $(i,j)=("\textrm{cats}","\textrm{mammals}")$, the feature "mammals" is always active whenever "cats" is active since cats are mammals.

Another example of non-identifiability is the existence of a bias term. An LFM with bias term is equivalent to an unbiased LFM with an extra feature that is always active [2, 3]. The existence of such a bias feature is followed by PDC2 because the feature is always active regardless of other features. Let $(Z^{\prime},W^{\prime})$ be an equivalent solution corresponding to the transform matrix $U=I+\bm{e}_{i}\bm{e}_{\mathrm{bias}}^{\mathrm{T}}-2\bm{e}_{i}\bm{e}_{i}^{\mathrm{T}}$, where the index "bias" refer to the bias feature, absence and the presence of $i$-th feature $\bm{z}_{i}^{\prime}$ is inverted from $\bm{z}_{i}$ and a sign of the $i$-th feature is flipped $\bm{w}_{i}^{\prime}=-\bm{w}_{i}$ in swap of raising the level of the bias $\bm{w}_{\mathrm{bias}}^{\prime}=\bm{w}_{\mathrm{bias}}+\bm{w}_{i}$. Such an “inverted” solution has been obtained by some algorithms, including Gibbs sampler [20], K-features [4], and possibly other LFM algorithms. In the case of a biased LFM, we can prove that there is a lower bound to the number of equivalent solutions $|H(Z)/\mathcal{S}_{K}|\geq(K+1)\times 2^{K-1}$ (see Supplemental Materials).

Let $\bm{u}_{k}$ be the $k$-th column of $U$, and $\Lambda$ be a $K\times K$ matrix such that $\Lambda\Lambda^{\mathrm{T}}=(Z^{\mathrm{T}}Z)^{-1}$, which is calculated by e.g., singular value decomposition (SVD) of $Z$, $\bm{s}_{k}=\Lambda^{-1}\bm{u}_{k}$, and $\bm{\mu}=\frac{1}{2}\Lambda^{\mathrm{T}}Z^{\mathrm{T}}\bm{1}$, where $\bm{1}=(1,\cdots,1)^{\mathrm{T}}$. then $f(ZU)$ for $k$-th column can be evaluated as:

If $U\in\tilde{H}(Z)$, (5) is evaluated to be zero, then $\bm{s}_{k}$ will be on the ($K\!-\!1$)-sphere of center $\bm{\mu}$ and radius $\|\bm{\mu}\|$. A possible $\bm{u}_{k}$ can then be obtained by sampling $\bm{s}$ uniformly from the ($K\!-\!1$)-sphere, taking the nearest integer $\bm{u}\!=\!\mathrm{round}(\Lambda\bm{s})$, and accept $\bm{u}_{k}=\bm{u}$   if   $\bm{u}\neq\bm{0}\wedge f(Z\bm{u})\!=\!0$.

The strict method mentioned above (which samples $U$ strictly from $\tilde{H}({Z})$) may, unfortunately, fail in some cases because ${Z}$, estimated with an arbitrary algorithm, may have randomness and its flipped component may spoil equivalent solutions by breaking PDCs in Theorem 4. To handle such a randomness, we employ a tolerance to the equivalent conditions. If the transformed matrix $Z^{\prime}=ZU$ includes an integer other than 0 or 1, it will be rejected from equivalent solutions. However, if there exists another binary matrix $\tilde{Z}^{\prime}$ which is close to $Z^{\prime}$, then $\tilde{Z}^{\prime}$ will be a nearly equivalent solution to $Z$. We measure this closeness by $f(ZU)$ since it is an approximation for the number of non-binary components as mentioned above, and it can be calculated by (5) without scanning all $N$ rows of $ZU$. In the Algorithm 1 shows this tolerant method. We first sample $f^{*}$ from some distribution, and then sample $\bm{u}_{k}$ such that $f(Z\bm{u}_{k})=f^{*}$ by sampling $\bm{s}$ uniformly from the ($K\!-\!1$)-sphere of radius $\sqrt{\|\bm{\mu}\|^{2}+{2f^{*}}^{2}}$ instead of $\|\bm{\mu}\|$. By choosing the distribution of $f^{*}$, we can tune the tolerance for the protrusion of ${Z}U$ from binary matrices. We employ discrete exponential distribution with parameter $\lambda$ for this role. The strict case is a limit of $\lambda\to\infty$.

Once $\mathcal{U}$ is sampled by Algorithm 1, equivalent solutions are obtained by selecting $K$ columns of $U$ from $\mathcal{U}$ so that $U$ is a regular matrix. Figure 3 shows the number of equivalent solutions found from $\tilde{H}(Z)/\mathcal{S}_{K}$ by using the strict version ($\lambda\to\infty$) of Algorithm 1. In the i.i.d. case (Fig. 3a), the number of equivalent solutions diminishes rapidly to one with an increasing $N$. This is consistent with results mentioned in [7]. Figure 3 (b) to (d) show cases of $Z$ sampled with a bias feature and PDCs (Theorem 4). In these three cases, solutions remains multiple even in $N=200$, keeping the problem non-identifiable. In the case of a bias feature (Fig. 3b), the number of solutions converges to the theoretical lower bound $(K+1)\times 2^{K-2}=112$, and when a pair of features holds a PDC (Fig. 3c), it converges to 3, the lower bound shown in Theorem 4. When three pairs of features have PDCs (Fig. 3d), the number of solutions fluctuate up to $10^{3}$ while a combination of 3 independent PDCs will result in $3^{3}=27$ solutions. This implies that multiple PDCs sharing a same feature give rise to another PDC, e.g., PDC2 for $(i,j)=(1,2)$ and $(2,3)$ implies PDC2 for $(1,3)$.

Now we consider obtaining appropriate solution among equivalent solutions. We select $U\in\tilde{U}(\hat{Z})$ so that obtained solution $(Z^{\prime},W^{\prime})=(\hat{Z}U,U^{-1}\hat{W})$ is more appropriate, i.e., having higher prior probability without degrading likelihood. So we employ the following cost function:

The first two terms of (6) correspond to log-priors in (1). The second term with parameter $\gamma$ plays a role keeping $({Z}^{\prime},{W}^{\prime})$ nearly equivalent to $(\hat{Z},\hat{W})$ and maintaining little change in the likelihood.

We can solve the optimization problem by MCMC-based sampling method (Algorithm 2), sampling $U$ by updating its columns successively according to a Boltzmann distribution with parameter $\beta$:

In the limit of $\beta\to\infty$, Algorithm 2 becomes a greedy local-search algorithm, which has a lower-bound in its optimality for typical linear-Gaussian LFMs (see Supplemental Materials). However, sampling method ($\beta<\infty$) works well in many cases since the feasible space is small enough.