GB-KMV: An Augmented KMV Sketch for Approximate Containment Similarity Search

In this paper, the element universe is $\mathcal{E}=\{e_{1},e_{2},...,e_{n}\}$. Let $\mathcal{S}$ be a collection of records (sets) $\{X_{1},X_{2},...,X_{m}\}$’ where $X_{i}$ ( $1\leq i\leq m$) is a set of elements from $\mathcal{E}$.

Before giving the definition of containment similarity, we first introduce the Jaccard similarity.

Similar to the Jaccard similarity, the containment similarity (a.k.a Jaccard containment similarity) is defined as follows.

Note that by replacing the union size $|X\cup Y|$ in Equation 1 with size $|X|$, we get the containment similarity. It is easy to see that Jaccard similarity is symmetric while containment similarity is asymmetric.

In this paper, we focus on the problem of containment similarity search which is to look up a set of records whose containment similarity towards a given query record is not smaller than a given threshold. The formal definition is as follows.

Next, we give an example to show the problem of containment similarity search.

Minwise Hashing is proposed by Broder in [13, 14] for estimating the Jaccard similarity of two records $X$ and $Y$. Let $h$ be a hash function that maps the elements of $X$ and $Y$ to distinct integers, and define $h_{min}(X)$ and $h_{min}(Y)$ to be the minimum hash value of a record $X$ and $Y$, respectively. Assuming no hash collision, Broder[13] showed that the Jaccard similarity of $X$ and $Y$ is the probability of two minimum hash values being equal: $Pr[h_{min}(X)=h_{min}(Y)]=J(X,Y)$. Applying such $k$ different independent hash functions $h_{1},h_{2},...,h_{k}$ to a record $X$($Y$, resp.), the MinHash signature of $X$($Y$, resp.) is to keep $k$ values of $h^{i}_{min}(X)$( $h_{min}(Y)$, resp.) for $k$ functions. Let $\mathbf{n}_{i},i=1,2,...,k$ be the indicator function such that

then the Jaccard similarity between record $X$ and $Y$ can be estimated as

Let $s=J(X,Y)$ be the Jaccard similarity of set $X$ and $Y$, then the expectation of $\hat{J}$ is

and the variance of $\hat{s}$ is

The $k$ minimum values(KMV) technique introduced by Bayer et. al in [11] is to estimate the number of distinct elements in a large dataset. Given a no-collision hash function $h$ which maps elements to range $[0,1]$, a KMV synopses of a record $X$, denoted by $\mathcal{L}_{X}$, is to keep $k$ minimum hash values of $X$. Then the number of distinct elements $|X|$ can be estimated by $\widehat{|X|}=\frac{k-1}{U_{(k)}}$ where $U_{(k)}$ is $k$-th smallest hash value. By $h(X)$, we denote hash values of all elements in the record $X$.

In [11], Bayer et. al also methodically analyse the problem of distinct element estimation under multi-set operation. As for union operation, consider two records $X$ and $Y$ with corresponding KMV synopses $\mathcal{L}_{X}$ and $\mathcal{L}_{Y}$ of size $k_{X}$ and $k_{Y}$, respectively. In [11], $\mathcal{L}_{X}\oplus\mathcal{L}_{Y}$ represents the set consisting of the $k$ smallest hash values in $\mathcal{L}_{X}\cup\mathcal{L}_{Y}$ where

Then the KMV synopses of $X\cup Y$ is $\mathcal{L}=\mathcal{L}_{X}\oplus\mathcal{L}_{Y}$. An unbiased estimator for the number of distinct elements in $X\cup Y$, denoted by $D_{\cup}=|X\cup Y|$ is as follows.

For intersection operation, the KMV synopses is $\mathcal{L}=\mathcal{L}_{X}\oplus\mathcal{L}_{Y}$ where $k=min(k_{X},k_{Y})$. Let $K_{\cap}=|\{v\in\mathcal{L}:v\in\mathcal{L}_{X}\cap\mathcal{L}_{Y}\}|$, i.e., $K_{\cap}$ is the number of common distinct hash values of $\mathcal{L}_{X}$ and $\mathcal{L}_{Y}$ within $\mathcal{L}$. Then the number of distinct elements in $X\cap Y$, denoted by $D_{\cap}$, can be estimated as follows.

The variance of $\hat{D}_{\cap}$, as shown in[11], is

LSH Ensemble technique, LSH-E for short, is proposed by Zhu et. al in [44] to tackle the problem of approximate containment similarity search. The key idea is : (1) transform the containment similarity search to the well-studied Jaccard similarity search; and (2) partition the data by length and then apply the LSH forest [9] technique for each individual partition.

Given the containment similarity search threshold as $t^{*}$ for the query $q$, we may come up with its corresponding Jaccard similarity threshold $s^{*}$ by Equation 12. A straightforward solution is to apply the existing approximate Jaccard similarity search technique for each individual record $X\in\mathcal{D}$ with the Jaccard similarity threshold $s^{*}$ (e.g., compute Jaccard similarity between the query $Q$ and a set $X$ based on their MinHash signatures). In order to take advantages of the efficient indexing techniques (e.g., LSH forest [9]), LSH-E will partition the dataset $\mathcal{S}$.

One of the LSH-E’s advantages is that it converts the containment similarity problem to Jaccard similarity search problem which can be solved by the mature and efficient MinHash LSH method. Also, LSH-E carefully considers the record size distribution and partitions the records by record size. In this sense, we say LSH-E is a data-dependent method and it is reported that LSH-E significantly outperforms existing asymmetric LSH based solutions [34, 35] (i.e., data-independent methods) as LSH-E can exploit the information of data distribution by partitioning the dataset. However, this benefit is offset by the fact that the the upper bound will bring extra false positives, in addition to the error from the MinHash technique.

Below we theoretically analyse the performance of LSH-E by studying the expectation and variance of its estimator.

Using the notations same as above, let $s=J(Q,X)$ be the Jaccard similarity between query $Q$ and set $X$ and $t=C(Q,X)$ be the containment similarity of $Q$ in $X$. By Equation 5, given the MinHash signature of query $Q$ and $X$ respectively, an unbiased estimator $\hat{s}$ of Jaccard similarity $s=J(Q,X)$ is the ratio of collisions in the signature, and the variance of $\hat{s}$ is $Var[\hat{s}]=\frac{s(1-s)}{k}$ where $k$ is signature size of each record. Then by transformation Equation 12, the estimator $\hat{t}$ of containment similarity $t=C(Q,X)$ by MinHash LSH is

where $q=|Q|$ and $x=|X|$. The estimator $\hat{t}^{\prime}$ of containment similarity $t=C(Q,X)$ by LSH-E is

where $q=|Q|$ and $u$ is the upper bound of $|X|$.

Next, we use Taylor expansions to approximate the expectation and variance of a function with one random variable [26]. We first give a lemma.

According to Equation 14, let $\hat{t}=f(\hat{s})=\alpha\frac{\hat{s}}{1+\hat{s}}$ where $\alpha=\frac{x}{q}+1$. We can see that the estimator $\hat{t}$ is a function of $\hat{s}$, and $f^{\prime}(\hat{s})=\alpha\frac{1}{(1+\hat{s})}$ and $f^{\prime\prime}(\hat{s})=-2\alpha\frac{1}{(1+\hat{s})}$. Then based on Lemma 1, the expectation and variance of $\hat{t}$ are approximated as

Similarly, the expectation and variance of LSH-E estimator $\hat{t}^{\prime}$ can be approximated as

The computation details are in technique report [41]. Since $u$ is the upper bound of $x$, the variance of LSH-E estimator $Var[\hat{t}^{\prime}]$ is larger than that of MinHash LSH estimator. Also, by Equation 18 and Equation 20, we can see that both estimators are biased and LSH-E method is quite sensitive to the setting of the upper bound $u$ by Equation 20. Because the presence of upper bound $u$ will enlarge the estimator off true value, LSH-E method favours recall while the precision will be deteriorated. The larger the upper bound $u$ is, the worse the precision will be. Our empirical study shows that LSH-E cannot achieve a good trade-off between accuracy and space, compared with our proposed method.

The key idea of our method is to propose a data-dependent indexing technique such that we can exploit the distribution of the data (i.e., record size distribution and element frequency distribution) for better performance of containment similarity search. We augment the existing KMV technique by introducing a global threshold for sample size allocation and a buffer for frequent elements, namely GB-KMV, to achieve better trade-off between synopses size and accuracy. Then we apply the existing set similarity join/search indexing technique to speed up the containment similarity search.

Below we outline the motivation of the key techniques used in this paper. Detailed algorithms and theoretical analysis will be introduced in Section IV-B and IV-C, respectively.

Given a query $Q$ and a threshold $t^{*}$ on containment similarity, the goal is to find record $X$ from dataset $\mathcal{S}$ such that

Applying some simple transformation to Equation 22, we get

Let $\theta=t^{*}|Q|$, then the containment similarity search problem is converted into finding record $X$ whose intersection size with the query $Q$ is not smaller than $\theta$, i.e., $|Q\cap X|\geq\theta$.

Therefore, we can directly apply the KMV method introduced in Section II-C. Given KMV signatures of a record $X$ and a query $Q$, we can estimate their intersection size ($|Q\cap X|$) according to Equation 10. Then the containment similarity of $Q$ in $X$ is immediately available given the query size $|Q|$. Below, we show an example on how to apply KMV method to containment similarity search.

The above analysis on optimal KMV sketch suggests an equal size allocation strategy, that is, each record is associated with the same size signature. Intuitively we should assign more resources (i.e., signature size) to the records with large size because they are more likely to appear in the results. However, the estimate accuracy of KMV for two sets size intersection is determined by the sketch with smaller size since we choose $k=\min(k_{1},k_{2})$ for KMV signatures of $X_{1}$ and $X_{2}$ for $D_{\cup}$ and $D_{\cap}$ in Equation 9, thus it is useless to give more resource to one of the records. We further explain the reason behind with the following example.

Before we introduce the global threshold to KMV sketch, consider the KMV sketch shown in the Fig. 2.

Inspired by the above observation, we can carefully choose a global threshold $\tau$ (e.g., $0.6$ in the above example) for a given space budget $b$, and ensure all hash values smaller than $\tau$ will be kept for KMV sketch of the records. By imposing a global threshold, we can identify a better (i.e., larger) $k$ value used for estimation, compared with Equation 8.

Given a record $X$ and a global threshold $\tau$, the sketch of a record $X$ is obtained as $\mathcal{L}_{X}=\{h(e):h(e)\leq\tau,e\in X\}$ where $h$ is the hash function. The sketch of $Q$ ($\mathcal{L}_{Q}$) is defined in the same way. In this paper, we say a KMV sketch is a G-KMV sketch if we impose a global threshold to generate KMV sketch. Then we set $k$ value of the KMV estimation as follows.

Meanwhile, we have $K_{\cap}=|\mathcal{L}_{Q}\cap\mathcal{L}_{X}|$. Let $U_{(k)}$ be the $k$-th minimal hash value in $\mathcal{L}_{Q}\cup\mathcal{L}_{X}$, then the overlap size of $Q$ and $X$ can be estimated as

Then the containment similarity of $Q$ in $X$ is

where $q$ is the query size. We remark that, as a by-product, the global threshold favours the record with large size because all elements with hash value smaller than $\tau$ are kept for each record.

Below is an example on how to compute the containment similarity based on G-KMV sketch.

In addition to the skewness of the record size, it is also worthwhile to exploit the skewness of the element frequency. Intuitively, more resource should be assigned to high-frequency elements because they are more likely to appear in the records. However, due to the nature of the hash function used by KMV sketch, the hash value of an element is independent to its frequency; that is, all elements have the same opportunity contributing to the KMV sketch.

One possible solution is to divide the elements into multiple disjoint groups according to their frequency (e.g., low-frequency and high-frequency ones), and then apply KMV sketch for each individual group. The intersection size between two records $Q$ and $X$ can be computed within each group and then sum up together. However, our initial experiments suggest that this will lead to poor accuracy because of the summation of the intersection size estimations. In Theorem 4 (Section IV-C5), our theoretical analysis suggests that the combination of estimated results are very likely to make the overall accuracy worse.

To avoid combining multiple estimation results, we use a bitmap buffer with size $r$ for each record to exactly keep track of the $r$ most frequent elements, denoted by $\mathcal{E}_{H}$. Then we apply G-KMV technique to the remaining elements, resulting in a new augmented sketch, namely GB-KMV. Now we can estimate $|Q\cap X|$ by combining the intersection of their bitmap buffers (exact solution) and KMV sketches (estimated solution).

As shown in Fig. 4, suppose we have $\mathcal{E}_{H}=\{e_{1},e_{2}\}$ and the global threshold for hash value is $\tau=0.5$, then the sketch of each record consists of two parts $\mathcal{L}_{H}$ and $\mathcal{L}_{GKMV}$; that is, for each record we use bitmap to keep the elements corresponding to high-frequency elements $\mathcal{E}_{H}=\{e_{1},e_{2}\}$, then we store the left elements with hash value less than $\tau=0.5$.

In this section, we introduce the technique details of our proposed GB-KMV method. We first show how to build GB-KMV sketch on the dataset $\mathcal{S}$ and then present the containment similarity search algorithm.

Algorithm 1 illustrates the construction of GB-KMV sketch. Let the element universe be $\mathcal{E}=\{e_{1},e_{2},...,e_{n}\}$ and each element is associated with its frequency in dataset $\mathcal{S}$. Line 1 calculates a buffer size $r$ for all records based on the skewness of record size and elements as well as the space budget $b$ in terms of elements. Details will be introduced in Section IV-C6. We use $\mathcal{E}_{H}$ to denote the set of top-$r$ most frequent elements (Line 1), and they will be kept in the buffer of each record. Let $\mathcal{E}_{K}$ denote the remaining elements. Line 1 identifies maximal possible global threshold $\tau$ for elements in $\mathcal{E}_{K}$ such that the total size of GB-KMV sketch meets the space budget $b$. For each record $X$, let $n_{X}$ denote the number of elements in $\mathcal{E}_{K}$ with hash values less than $\tau$, we have $\sum_{X\in\mathcal{S}}(\frac{r}{32}+n_{X})\leq b$. Then Lines 1-1 build the buffer $\mathcal{H}_{X}$ and G-KMV sketch $\mathcal{L}_{X}$ for every record $X\in\mathcal{S}$. In section 2, we will show the correctness of our sketch in Theorem 2.

Note that $|\mathcal{H}_{Q}\cap\mathcal{H}_{X}|$ is the number of common elements of $Q$ and $X$ in $\mathcal{E}_{H}$.

In this section, we provide theoretical underpinnings of the claims and observations in this paper.