Joint Embedding of Meta-Path and Meta-Graph for
Heterogeneous Information Networks

The basic mathematical object of multilinear algebra is the tensor, a higher order generalization of vectors (first order tensors) and matrices (second order tensors) to multiple indices. The order of a tensor is the number of dimensions, also known as modes or ways. An $N$-th order tensor is represented as $\mathcal{X}\in\mathbb{R}^{I_{1}\times I_{2}\times\cdots\times I_{N}}$, where $I_{n}$ is the cardinality of its $n$-th mode, $n\in\{1,2,\cdots,N\}$. An element of a vector $\mathbf{x}$, a matrix $\mathbf{X}$, or a tensor $\mathcal{X}$ is denoted by $x_{i}$, $x_{i,j}$, $x_{i,j,k}$, etc., depending on the number of modes. All vectors are column vectors unless otherwise specified. For an arbitrary matrix $\mathbf{X}\in\mathbb{R}^{I\times J}$, its $i$-th row and $j$-th column vector are denoted by $\mathbf{x}^{i}$ and $\mathbf{x}_{j}$, respectively.

Definitions of outer product, partial symmetric tensor, mode-$n$ matricization, and CP factorization are given below, which will be applied to present our approach.

To obtain the CP factorization $\llbracket\mathbf{X}^{(1)},\cdots,\mathbf{X}^{(N)}\rrbracket$, the objective is to minimize the following estimation error:

However, $\mathcal{L}$ is not jointly convex w.r.t. $\mathbf{X}^{(1)},\cdots,\mathbf{X}^{(N)}$. A widely used optimization technique is the Alternating Least Squares (ALS) algorithm, which alternatively minimize $\mathcal{L}$ for each variable while fixing the other, that is,

where $\odot_{i\neq n}^{N}\mathbf{X}^{(i)}=\mathbf{X}^{(N)}\odot\cdots\mathbf{X}^{(n-1)}\odot\mathbf{X}^{(n+1)}\cdots\odot\mathbf{X}^{(1)}$.

Since a meta-graph only has one source node and one target node, not all sub-graphs of HINs can be meta-graph.

where $g_{v_{s}\rightarrow v_{t}}$ is a meta-graph instance of $\mathcal{G}$, and $s(v_{s},v_{t}\,|\,g_{v_{s}\rightarrow v_{t}})$ is the relevance score between $v_{s}$ and $v_{t}$, which will be determined by the number of meta-graph instances connecting them.

Prior works provide different meta-graph-based relevance measures, such as StructCount, SCSE and BSCSE [15].

We study the problem of meta-graph-based node embedding in the HIN. Given a HIN $G=(V,E)$, we have two goals in this study. First, we want to explore a customized meta-graph-based relevance measure which can more efficiently capture the complex structural information. Second, we aim at finding an effective node embedding that can better preserve the closeness between nodes in a HIN based on a meta-graph and its embeded meta-paths analysis. Specifically, we integrate all the similarity information of a meta-graph and its embedded meta-paths into a symmetric matrix and a partial symmetric tensor, and perform multilinear analysis of the coupled partial symmetric tensor and symmetric matrix to find the node embedding.

First, we want to propose a new meta-graph-based similarity measure called GraphSim. In previous work, Huang et al. [15] proposed three meta-graph-based similarity measures: StructCount, SCSE, and BSCSE which is a mixed measure based on previous two measures. GraphSim can be viewed as a normalized version of StructCount.

StructCount [15] is a straightforward meta-graph-based similarity measure in HIN, which counts the number of meta-graph instances in the graph $G$ with an $n_{s}$ as source and an $n_{t}$ as target object.

Comparing to StructCount, GraphSim is normalized version of StructCount. $s_{GraphSim}(v_{t},v_{s})$ is determined by two parts: First, the number of meta-graph instance between $v_{t},v_{s}\in V$ by following $\mathcal{G}$; Second, the balance of their visibility, where the visibility is defined as the number of meta-graph instances between themselves. Normalized relevance score can present better relation between different nodes. For example, an author $A_{1}$ published all his four papers with $A_{2}$. $A_{3}$ published five papers with $A_{2}$, and $A_{3}$ published other five papers with other authors. Without normalized process, the relation between $A_{2}$ and $A_{3}$ is closer than $A_{2}$ and $A_{1}$. However, for common sense, we should agree $A_{2}$ and $A_{1}$ have closer relation, which indicates GraphSim is a better measure.

Comparing to three measures in [15], $s(v_{t},v_{s})$ of GraphSim is between 0 and 1 like SCSE. However, SCSE measures the random walk probability that $v_{s}$ expands a meta-graph instance to $v_{t}$. In our application, we find GraphSim shows better performance than all those three meta-graph measures [15].

In this section, we show how to jointly consider similarity information of a meta-graph and its embedded meta-paths to learn a node embedding. The basic idea is the integration of similarity matrices and coupled embedding by joint factorization. Specifically, we first compute a meta-graph similarity matrix using the proposed GraphSim, denoted as $\mathbf{Y}\in\mathbb{R}^{M\times M}$, and for each meta-path $\mathcal{P}$, compute an embedded meta-path similarity matrix using the PathSim, denoted as $\mathbf{M}\in\mathbb{R}^{M\times M}$. Next, we concatenate the embedded meta-path similarity matrices of different embedded meta-paths to form a third-order tensor comprising three modes: nodes, nodes, and paths, denoted as $\mathcal{X}=[\mathbf{M}_{1},\cdots,\mathbf{M}_{N}]$ $\in\mathbb{R}^{M\times M\times N}$. Then, we introduce a novel coupled tensor-matrix decomposition (CTMD) method to find common latent features between $\mathcal{X}$ and $\mathbf{Y}$. Last, we use the latent features to measure the similarity between different nodes in the HIN.

In the following, we detail the CTMD method, which can be seen as a special case of the coupled tensor-matrix decomposition [1] with input partial symmetric tensor $\mathcal{X}$ and symmetric matrix $\mathbf{Y}$. Notice that since similarity matrix is symmetric, the resulting $\mathcal{X}$ is a partial symmetric tensor, and $\mathbf{Y}$ is a symmetric matrix.

Tensors (including matrix) provide a natural and efficient representation for a meta-graph data, but there is no guarantee that such representation will be good for subsequent learning, since learning will only be successful if the regularities that underlie the data can be discerned by the model. Tensor factorization is a powerful tool to analyze tensors. In previous work, it was found that CP factorization (which is a higher order generalization of SVD) is particularly effective to acknowledge the connections and find valuable features among tensor data [32]. Motivated by these observations, we exploit the benefits of CP and SVD factorizations to find an effective embedding in the sense of meta-path-based similarity tensor $\mathcal{X}$ and meta-graph similarity matrix $\mathbf{Y}$.

Based on above analysis, we design our CTMD objective function as below:

where $\mathbf{P}\in\mathbb{R}^{M\times R}$ and $\mathbf{T}\in\mathbb{R}^{N\times R}$ are latent matrices. Specifically, $\mathbf{P}$ is jointly learned from both meta-graph and meta-path similarity information.

The objective function in Eq. (7) is non-convex with respect to $\mathbf{P}$ and $\mathbf{T}$ together, thus there is no closed-form solution. We introduce an effective iteration method to solve this problem. The main idea is to decouple the parameters using an Alternating Direction Method of Multipliers (ADMM) approach [4], by alternatively optimizing the objective with respect to one variable, while fixing others.

Update $\mathbf{P}$: First, we optimize $\mathbf{P}$ while fixing $\mathbf{T}$. Notice that the objective function in Eq. (7) involves a fourth-order term with respect to $\mathbf{P}$ which is difficult to optimize directly. To obviate this problem, we use a variable substitution technique and minimize the following objective function

where $\mathbf{Q}\in\mathbb{R}^{M\times R}$ is an auxiliary variable.

The augmented Lagrangian function of Eq. (8) is

where $\mathbf{U}\in\mathbb{R}^{M\times R}$ is the Lagrange multiplier, and $\lambda$ is the penalty parameter which can be adjusted efficiently according to [18].

To compute $\mathbf{P}$, Eq. (9) can be transformed as

where $\mathbf{X}_{(1)}\in\mathbb{R}^{M\times(MN)}$ is the mode-1 matricization of $\mathcal{X}$, and $\mathbf{F}=\mathbf{T}\odot\mathbf{Q}\in\mathbb{R}^{(MN)\times R}$.

By setting the derivative of Eq. (10) with respect to $\mathbf{P}$ to zero, we obtain the closed-form solution

To efficiently compute $\mathbf{F}^{\mathrm{T}}\mathbf{F}$, we consider the following property of the Khatri-Rao product of two matrices

Then the auxiliary matrix $\mathbf{Q}$ can be optimized successively in a similar way, and the solution is

where $\mathbf{X}_{(2)}$ is the mode-2 matricization of $\mathcal{X}$, and $\mathbf{G}=\mathbf{T}\odot\mathbf{P}$.

Moreover, we optimize the Lagrange multiplier $\mathbf{U}$ using the gradient descent method by

Update $\mathbf{T}$: Next, we optimize $\mathbf{T}$ while fixing $\mathbf{P}$ and $\mathbf{S}$. We need to optimize the following objective function

where $\mathbf{X}_{(3)}$ is the mode-3 matricization of $\mathcal{X}$, and $\mathbf{H}=\mathbf{Q}\odot\mathbf{P}$.

By setting the derivative of Eq. (15) with respect to $\mathbf{T}$ to zero, we obtain the closed-form solution as

The overall algorithm is summarized in Algorithm 1.

Each iteration in Algorithm 1 consists of simple matrix operations. Therefore, rough estimates of its computational complexity can be easily derived based on ADMM  [17].

The estimate for the update of $\mathbf{P}$ according to Eq. (11) is as follows: $O(M^{2}NR)$ for the computation of the term $2\mathbf{X}_{(1)}\mathbf{F}+2\alpha\mathbf{Y}\mathbf{Q}+\lambda\mathbf{Q}-\mathbf{U}$; $O((M+N)R^{2})$ for the computation of the term $2\mathbf{F}^{\mathrm{T}}\mathbf{F}+2\alpha\mathbf{Q}^{\mathrm{T}}\mathbf{Q}+\lambda\mathbf{I}$ due to Eq. (12) and $O(R^{3})$ for its Cholesky decomposition; $O(MK^{2})$ for the computation of the system solution that gives the updated value of $\mathbf{P}$. An analogous estimate can be derived for the update of $\mathbf{Q}$.

Overall, the updates of model parameters $\mathbf{P}$ and $\mathbf{Q}$, require O($R^{3}+(M+N)R^{2}+M^{2}NR$) arithmetic operations in total.

We first conduct a clustering task to evaluate the performance of the compared methods on DBLP and YAGO Movie datasets. For DBLP, we use the areas of authors as ground-truth label for clustering authors (A2A), and use the areas of venues as labels for clustering venues (V2V). For YAGO Movie, we use the genres of movies as labels (M2M). To be specific, we use $k$-means on the derived meta-graph-based relevance matrices for the clustering task. To evaluate the results, we use NMI and purity as evaluation metrics.

Clustering results of the three tasks are shown in Table II. Comparing to previous meta-graph-based relevance measures, the proposed GraphSim always shows the best performance of all. We observe at least 19.94% improvement in NMI of GraphSim method when compared with the previous meta-graph-based relevance measure on clustering the venues and authors in DBLP, respectively. The clustering results can be sensitive to initialization of centroid seeds, so we set 100 times of random initializations. All methods show worse performance on YAGO Movie than DBLP, but the proposed methods, especially MEGA++, show the best performance comparing to prior works..

We then conduct a classification task. Comparing to the clustering task, in DBLP we do not evaluate the results of classifying the venues, as the total number of venues is only 18. We first apply previous methods and our works to generate the similarity matrices or embedding space of the original network. Then, we randomly partition the samples, and set 80% samples as training set and the rest as testing set. Last, we apply $k$ nearest neighbor (k-NN) classifier with $k=5$ to evaluate the methods with training and testing dataset [29, 15]. To prevent the special case of random partition, we repeat and use different random partition 10 times in total. For multi-label classification task, we use the average Macro-F1 score and Micro-F1 score as the evaluation metrics.

GraphSim outperforms the existing relevance measures (e.g StructCount, SCSE and BSCSE) because it represents a better relations between objects in the HINs by normalizing the presence of meta-graph structures. MEGA++ outperforms all the baselines because it captures both lower-order (i.e. meta-path) and higher-order (i.e. meta-graph) structural information by facilitating the use of coupled tensor-matrix decomposition method to obtain a joint embedding for nodes in HINs.

In this section, we first analyze the parameter sensitivity of our methods as shown in Figure 4. We use two evaluation metrics, Normalized Mutual Information(NMI), and Purity (both the larger, the better), to evaluate the performances of our methods for clustering task. In Figure 4 (a)-(b), the penalty parameters $\lambda$ of MEGA is used for minimizing the Frobenius Norm of embedding space $\mathbf{P}$ and $\mathbf{Q}$ in Eq. (8), and the best performance is achieved when $\lambda$ is set as 3.2768e-04. From 4 (b)-(c), setting the embedding dimensions as 5 shows the best performance for both MEGA and MEGA++. MEGA++ outperforms MEGA with the same number of embedding dimensions. The Figure 4 (e)-(f) show the two penalty parameters $\lambda$ and $\alpha$ of MEGA++. The penalty parameter $\lambda$ of MEGA++ is the same as that in MEGA. The penalty parameter $\alpha$ of MEGA++ is used for minimizing the Frobenius Norm of meta-graph similarity matrix $\mathbf{Y}$ and its embedding space in Eq. (7). We find that $\lambda=0.6711$ and $\alpha=1.6$ produce the best performance of clustering task.

In this section, we evaluate the execution time of MEGA++. In Table IV, it shows the execution time is linear with respect to the embedding dimensions. Based on the time complexity of MEGA++, when we have a fixed size of dataset, the embedding dimensions $R$, and the number of views $N$ are linear with respect to the execution time. Sometimes, MEGA++ can be early stopped when it is already converge, so as to the same time consuming of DBLP (A2A) with $R=10$ and $R=15$. The same results are shown in the real testing on three tasks, which show the efficiency of MEGA++.

Network embedding want to learn a low-dimensional representations from a network. Previous traditional works [3] usually construct the affinity graph using the feature vectors of the vertexes and then compute the eigenvectors of the affinity graph. Some other groups use matrix factorization to represent graph as adjacency matrix [2].

Recently, DeepWalk [22] and LINE [31] are proposed for learning the network embedding. Besides these two most popular node embedding methods, many other network embedding are proposed recent years [6, 10, 33, 7]. [6, 33] learn the node embedding by deep learning encoder methods. However, none previous node embedding methods consider the meta-graph and its embedded meta-paths information.

Just like deep learning, tensor learning becomes very hot and popular topic in recent years due to the stronger computing capability and lower computation cost  [14, 19, 11, 20, 24, 5, 12]. Coupled tensor matrix embedding tries to fuse multiple information sources where matrices and tensors sharing some common modes are jointly embedding [8]. A gradient-based optimization approach for joint tensor-matrix analysis is proposed by Acar et al. [1].

Multi-view learning is a hot idea to think one object with different views [26, 27, 23, 13]. In this paper, we think the HIN with different views such as meta-paths and meta-graph, and fuse the different information for node embedding. However, none of these frameworks can be directly applicable to learn jointly embedding with a partial symmetric tensor and a symmetric matrix, and also do not leverage meta-path and meta-structure information for similarity search in HINs.