Linear-time Hierarchical Community DetectionCCS: Mathematics of computing Graph algorithmsCCS: Mathematics of computing Approximation algorithmsCCS: Mathematics of computing Graph theoryCCS: Information systems Data miningCCS: Theory of computation Graph algorithms analysisCCS: Networks Network types

Ryan A. Rossi OrcID: 1234-5678-9012-3456 Affiliation: Adobe Research , Nesreen K. Ahmed Affiliation: Intel Labs , Eunyee Koh Affiliation: Adobe Research and Sungchul Kim Affiliation: Adobe Research email:

Abstract.

Community detection in graphs has many important and fundamental applications including in distributed systems, compression, image segmentation, divide-and-conquer graph algorithms such as nested dissection, document and word clustering, circuit design, among many others. Finding these densely connected regions of graphs remains an important and challenging problem. Most work has focused on scaling up existing methods to handle large graphs. These methods often partition the graph into two or more communities. In this work, we focus on the problem of hierarchical community detection (i.e., finding a hierarchy of dense community structures going from the lowest granularity to the largest) and describe an approach that runs in linear time with respect to the number of edges and thus fast and efficient for large-scale networks. The experiments demonstrate the effectiveness of the approach quantitatively. Finally, we show an application of it for visualizing large networks with hundreds of thousands of nodes/links.

Keywords:

Community detection, hierarchical communities, linear-time algorithms, label propagation, graph clustering, graph mining

1. Introduction

Communities of a graph are sets of nodes that are densely connected and close to one another in the graph (Fortunato 2010). Communities are important for understanding complex systems modeled as graphs (Schaeffer 2007; Fortunato 2010). In our modern age of big data, it has become increasingly important to study and understand complex systems that arise from large data of diversely interconnected entities such as biological networks (Alon 2003), social networks (Girvan and Newman 2002), citation networks (Giles 2006), among many others. Community detection in graphs has been one of the most fundamental tools for analyzing and understanding the components of complex networks and has been used for many real-world applications. It has been used extensively in applications to distributed systems (Hendrickson and Leland 1995; Simon 1991; Van Driessche and Roose 1995), compression (Rossi et al. 2015; Buehrer and Chellapilla 2008), image segmentation (Shi and Malik 2000; Felzenszwalb and Huttenlocher 2004), document and word clustering (Dhillon 2001), among others.

Communities are sets of vertices $C_{1},\ldots,C_{k}$ such that each set $C_{k}$ has with more connections inside the set than outside (Fortunato 2010). While there are many different methods for finding communities (Schaeffer 2007; Fortunato 2010), it is generally agreed that a community $C_{k}\subseteq V$ is “good" if the induced subgraph is dense (e.g., many edges between the vertices in $C_{k}$ ) and there are relatively few edges from $C_{k}$ to other vertices $\bar{C_{k}}=V\setminus C_{k}$ (Schaeffer 2007). Let $E(C_{k})$ denote the set of edges between vertices in $C_{k}$ (internal edges) and $E(C_{k},\bar{C}_{k})$ be the set of all edges between $C_{k}$ and $\bar{C}_{k}$ (external edges). Another desired property of a community $C_{k}$ is that vertices in $C_{k}$ are all close to one another, i.e., the distance between any two vertices $v,w\in C_{k}$ denoted as $\mathtt{dist}(v,w)$ is as small as possible (small proximity, distance). Community detection aims to cut a graph into two or more sparsely interconnected dense subgraphs (Fortunato 2010). Semantically, these subgraphs may represent a tightly-knit group of friends, a household or organization, web pages of the same general topic, or a group of researchers that frequently publish together. In this work, we address the following problem:

Definition 1 (Hierarchical Community Detection).

Given an (un)directed graph $G=(V,E)$ , the problem of hierarchical community detection is to find

(i)

a hierarchy of communities denoted as $\mathbb{H}=\{\mathbfcal{C}^{1},\ldots,\mathbfcal{C}^{L}\}$ where $\mathbfcal{C}^{t}=\{C^{t}_{1},\ldots,C^{t}_{k}\}$ are the communities at level $t$ in the hierarchy

(1)

\displaystyle V_{t}=\bigcup_{k}C^{t}_{k}\;\;\text{ and }\;\;|\mathbfcal{C}^{1}|<\cdots<|\mathbfcal{C}^{t}|<\cdots<|\mathbfcal{C}^{L}|

(ii)

a hierarchy of community (super) graphs $G_{1},\ldots,G_{t},\ldots,G_{L}$ where $G_{t}=(V_{t},E_{t})$ succinctly captures the relationships between the communities (nodes in $G_{t}$ ) at a lower $t-1$ level in the hierarchy. The hierarchy of community (super) graphs indicate how the functional units (communities) of the graph interact at each level and how they combine to form larger communities.

Refer to caption — (a) $\mathbfcal{C}^{1}$ (40 communities)

While there have been a lot of work on community detection (Schaeffer 2007; Fortunato 2010), most research (i) does not address the hierarchical community detection problem (Definition 1) or are (ii) inefficient for large networks with a worst-case time (and space) complexity that is not linear in the number of edges. In this work, we describe an approach called hLP that addresses both these limitations. In particular, hLP solves the hierarchical community detection problem by detecting a hierarchy of communities (going from the lowest to highest granularity) along with a hierarchy of community (super) graphs that reveal the higher-order organization and components at each level and how these components interact with one another to form larger components at a higher-level in the hierarchy. Most importantly, hLP is fast and efficient for large networks with a worst-case time complexity that is linear in the number of edges whereas the space complexity of hLP is linear in the number of nodes.

Algorithm 1 Hierarchical Community Detection (hLP)

1 a graph

G=(V,E)

2 hierarchical communities

\mathbb{H}=\{\mathbfcal{C}^{1},\ldots,\mathbfcal{C}^{L}\}

3 Set

G_{0}\leftarrow G

to be the initial graph and

t\leftarrow 1

4 repeat

\mathbfcal{C}^{t}\leftarrow

LabelProp(

G_{t-1}

)

G_{t}=(V_{t},E_{t})\leftarrow\,

CreateSuperGraph(

G_{t-1}

\mathbfcal{C}^{t}

) via Eq. 3

t\leftarrow t+1

8 until

|V_{t}|<2

\triangleright

Stop when no nodes to combine

Algorithm 2 Create Super Graph

1 a graph

G_{t-1}=(V_{t-1},E_{t-1})

, communities

\mathbfcal{C}^{t}

from

G_{t-1}

2 community (super) graph

G_{t}=(V_{t},E_{t})

for layer

t

V_{t}\leftarrow\mathbfcal{C}^{t-1}

where

\mathbfcal{C}^{t-1}=\{C_{1},\ldots,C_{k}\}

\triangleright

Super node set

E_{t}\leftarrow\emptyset

\triangleright

Super edge set

5 Let

\boldsymbol{\mathrm{c}}

be the community assignment vector where

c_{i}=k

v_{i}\in C_{k}

6 parallel for

i\in V_{t-1}

7 for

j\in\Gamma_{i}

\triangleright

Neighbor of vertex

i

8 if

c_{i}\not=c_{j}

and

(c_{i},c_{j})\not\in E_{t}

then

E_{t}\leftarrow E_{t}\cup(c_{i},c_{j})

10 end parallel

2. Approach

This section describes our fast linear-time approach for revealing hierarchical communities in large graphs. Given $G$ , the algorithm outputs a hierarchy of communities $\mathbb{H}=\{\mathbfcal{C}^{1},\ldots,\mathbfcal{C}^{L}\}$ where $L$ is the number of layers (i.e., levels in the community hierarchy $\mathbb{H}$ ). A summary of the approach is shown in Algorithm 1. There are two general steps: Label Propagation (Section 2.1) and Super Graph Construction (Section 2.2).

2.1. Label Propagation

Note $\Gamma(v_{i})=\{j\in V\,|\,(i,j)\in E\}$ is the set of neighbors of node $i$ . The first step performs label propagation. In particular, the approach begins with each node belonging to its own community. For each node $v_{i}\in V$ (or edge), we assign it to the community $C_{k}\in\mathbfcal{C}$ that has the maximum number of neighbors $\Gamma(v_{i})$ in it. More formally,

(2)

\argmax_{C_{k}\in\,\mathbfcal{C}}\sum_{v_{j}\in\,\Gamma(v_{i})}\mathbb{I}\big[\,v_{j}\in C_{k}\,\big]

where for any predicate $p$ the indicator function $\mathbb{I}[p]=1$ iff $p$ holds and $0$ otherwise. Hence, $\mathbb{I}\big[\,v_{j}\in C_{k}\,\big]=1$ iff $v_{j}\in C_{k}$ , and 0 otherwise. In other words, every node $v_{i}\in V$ is assigned the label that appears the most frequent in the 1-hop neighborhood of the node Eq. 2 can be easily replaced/modified to take into account other important aspects. The algorithm converges when an iteration results in no further changes (i.e., no new assignments are made) or if the max number of iterations is reached which can be interactively tuned by the user. Upon each iteration, we compute a random permutation and use this ordering to assign nodes (or edges) to communities. To further speedup the approach, we leverage the number of previous iterations that the community assignment of a node (or edge) remained unchanged (i.e., the community of $v_{i}$ remained stable over the last $t$ iterations). In particular, let $\delta$ denote a hyperparameter that controls the number of previous iterations that the community assignment of a node or edge must remain unchanged before it is declared as final. Thus, each iteration of the approach can be defined over the set $S$ of graph elements (nodes/edges) that are still active, i.e., $T_{i}<\delta$ where $T_{i}$ denotes the number of subsequent iterations that $v_{i}$ has remained unchanged (w.r.t. community assignment). Fast and efficient localized updates are performed when new nodes/edges arrive.

2.2. Super Graph Construction

Given a graph $G_{t-1}$ and $\mathbfcal{C}^{t}=\{C_{1}^{t},\ldots,C_{k}^{t}\}$ , Algorithm 2 computes the community (super) graph $G_{t}=(V_{t},E_{t})$ for layer $t$ in the community hierarchy where $V_{t}\leftarrow\mathbfcal{C}^{t}$ and thus the number of nodes in $G_{t}$ is $n_{t}=|\mathbfcal{C}^{t}|$ , i.e., the number of communities detected in the previous graph $G_{t-1}$ (or level in the community hierarchy). Similarly, an edge $(i,j)\in E_{t}$ iff there is an edge between $C_{i}^{t}$ and $C_{j}^{t}$ in $G_{t-1}$ , i.e., a link exists between a node $r\in V_{t-1}$ assigned to community $C_{i}^{t}$ and another node $s\in V_{t-1}$ assigned to community $C_{j}^{t}$ . More formally,

(3)

E_{t}=\big\{(i,j)\,:\,r\in C_{i}^{t},\,s\in C_{j}^{t}\wedge\;(r,s)\in E_{t-1}\,\wedge\;i\not=j\big\}

Property 1.

Let $E_{t}(C_{i},C_{j})$ be the set of edges between $C_{i}$ and $C_{j}$ (cut set), then the number of edges $|E_{t+1}|$ in the next level $t+1$ is:

(4)

\displaystyle|E_{t+1}|=\sum_{C_{i}\in\mathbfcal{C}^{t}}\;\;\sum_{C_{j}\in\mathbfcal{C}^{t}}|E_{t}(C_{i},C_{j})|\;\;\text{ s.t. }\;i<j

Note $|E_{t}(C_{i},C_{j})|$ does not include multi-edges.

Algorithm 2 returns $G_{t}=(V_{t},E_{t})$ for layer $t$ in the hierarchy. The approach terminates when $|V_{t}|<2$ as shown in Algorithm 1. Hence, hLP terminates when there are no nodes remaining to combine.

Property 2.

Let $|E(G_{t})|$ and $|V(G_{t})|$ be the number of edges and nodes in $G_{t}$ and $G_{0}\leftarrow G$ , then

(5)

\displaystyle|E(G_{0})|\geq\cdots\geq|E(G_{L})|\;\;\text{ and }\;\;|V(G_{0})|\geq\cdots\geq|V(G_{L})|

Property 2 has a number of important and useful implications that are leveraged in Section 3.

3. Analysis

This section shows the worst-case time and space complexity of the proposed approach. Let $L$ denote the number of layers (hierarchies) and let $T$ denote the maximum number of iterations at any given layer. Both $L$ and $T$ are small. Further, let $N=|V|$ denote the number of nodes and let $M=|E|$ denote the number of edges in $G$ .

3.1. Time Complexity

Lemma 1.

The worst-case time complexity of hierarchical label propagation is

(6)

\mathcal{O}(LTM)=\mathcal{O}(M)

where $L$ and $T$ are small constants. Therefore, the time complexity is linear in the number of edges $M$ in the graph.

Supergraph construction: The worst-case time complexity of Algorithm 2 is $\mathcal{O}(|E_{t-1}|)$ . This is bounded above by the number of edges denoted as $|E|$ in the input graph $G$ .

3.2. Space Complexity

Lemma 2.

The space complexity of hierarchical label propagation is

(7)

\mathcal{O}(NL)

where $L$ is a small constant. Therefore, the space complexity is linear in the number of nodes in $G$ .

Lemma 2 assumes the node community assignments at each layer are stored and given as output to the user. However, this information can be significantly compressed by storing only the community assignments at the first layer, and then storing only how these communities are merged at each subsequent layer.

Supergraph construction: The worst-case space complexity of Algorithm 2 is $\mathcal{O}(|E_{t-1}|)$ . Similar to time complexity, this is bounded above by the number of edges denoted as $|E|$ in the input graph $G$ .

4. Experiments

The experiments in this section are designed to investigate the quality of the communities revealed by hLP and the utility of the hierarchical communities for a visualization application. For comparison, we use a wide variety of graphs from different application domains including social networks (soc), biological/protein networks (bio), infrastructure networks (inf), web graphs (web), road networks (road), and collaboration networks (ca). Due to space constraints, network statistics were removed but can be accessed online at http://networkrepository.com along with the data (Rossi and Ahmed 2016).

4.1. Comparison

4.1.1. Baseline methods.

For fair comparison, we use baselines that are fast with linear-time complexity (with the exception of Louvain):

$\bullet$

Densest Subgraph (DS) (Khuller and Saha 2009): This method finds an approximation of the densest subgraph in $G$ using degeneracy ordering, and removes this subgraph. This is repeated until all nodes have been assigned.
$\bullet$

KCore Communities (KCore) (Rossi et al. 2015; Shin et al. 2016): Many have observed that the largest k-core subgraphs of a real-world network are highly dense subgraphs that often contain the max clique (Rossi et al. 2015). The KCore baseline simply uses the maximum k-core subgraph as $S$ and $\bar{S}=V\setminus S$ .
$\bullet$

Label Propagation (LP) (Raghavan et al. 2007): Label propagation takes a labeling of the graph, then for each node, the label is updated according to the label that occurs the most among its neighbors. This is repeated until convergence.
$\bullet$

Louvain (Louv) (Blondel et al. 2008): Louvain performs a greedy optimization of modularity by forming small, locally optimal communities then grouping each community into one node. This two-phase process is repeated until modularity cannot be maximized locally.
$\bullet$

Spectral Clustering (Spec) (Chung 1997): This baseline uses spectral clustering on the normalized Laplacian of the adjacency matrix to greedily build the sweeping cluster that minimizes conductance.

Table 1. Quantitative evaluation of the methods (modularity). The best result from each graph is bold. Note hLP is the proposed method.

	DS	KCore	LP	Louv	Spec	hLP
soc-yahoo-msg	0.0003	0.0004	0.0479	0.0394	0.0005	0.0569
bio-gene	0.0195	0.0217	0.0315	0.0408	-0.0208	0.0846
ca-cora	0.0089	0.0304	0.0444	0.0608	0.0164	0.1026
soc-terror	0.0888	0.0892	0.0967	0.0967	0.0999	0.1243
inf-US-powerGrid	0.0027	0.0027	0.0061	0.0212	0.1127	0.1242
web-google	0.0272	0.0275	0.0429	0.0471	0.1010	0.1122
ca-CSphd	0.0224	0.0224	0.0234	0.0198	0.0131	0.1201
ca-netscience	0.0164	0.0168	0.1063	0.0561	0.1229	0.1233
road-luxem.	0.0629	0.0629	0.0077	0.0046	-0.1170	0.1141
bio-DD21	0.0865	0.0866	0.0106	0.0202	0.1241	0.1247

4.1.2. Quantitative evaluation.

We quantitatively evaluate the communities using modularity (Newman 2001). Modularity is defined as:

(8)

\mathbb{E(\boldsymbol{\mathrm{c}})}=\frac{1}{2m}\sum_{ij}\Bigg[A_{ij}-\frac{d_{i}d_{j}}{2m}\Bigg]\delta(c_{i},c_{j})

where $M$ is the number of edges, $\boldsymbol{\mathrm{A}}$ is the adjacency matrix with $A_{ij}=1$ if $(i,j)\in E$ and 0 otherwise; $d_{i}$ and $d_{j}$ is the degree of node $i$ and $j$ ; $c_{i}$ and $c_{j}$ are the community assignments of node $i$ and node $j$ ; and $\delta$ is an indicator function such that $\delta(c_{i},c_{j})=1$ if $c_{i}=c_{j}$ and $0$ otherwise. We report the best result from any layer/level in the community hierarchy. Results are provided in Table 1. Notably, hLP outperforms all the other baseline methods across all graphs as shown in Table 1. hLP reveals better high quality communities across a wide variety graphs from different application domains (social, biological, infrastructure, among others) as shown in Table 1. Overall, hLP typically achieves at least an order of magnitude improvement over the other baseline methods.

Now we investigate the communities found by hLP by overlaying the community assignments on top of the network structure (node-link diagram). The communities given by hLP at different levels in the hierarchy are shown in Figure 1 for the network science co-authorship network. Communities in 1(a) represent small groups of researchers that frequently publish together whereas communities in 1(b) represent different research areas and so on.

4.1.3. Runtime Performance

Figure 2 visualizes the important components (functional modules) of a large road network from luxembourg at multiple scales (layers). Note that using a serial python implementation of the proposed method takes only 10.2 seconds to derive the initial 9,452 communities visualized in Figure 2(a). However, the next layer is orders of magnitude faster taking less than a second (0.611 sec.) and the runtime steadily decreases as a function of the supergraph size (number of supernodes, superedges) and the number of iterations to converge in the preceding layers. Furthermore, the number of iterations until convergence also steadily decreases as the number of layers increases.

4.2. Visualizing Large Networks

One important application of hLP is visualization of large networks. In Figure 2, we use hLP to compute a hierarchy of communities for a large real-world network consisting of 114,600 nodes and 239,332 edges. While it is impractical and often impossible to visualize such a large network, we can use hLP to summarize the graph structure at multiple levels as shown in Figure 2. Instead of visualizing the graph at the level of intersections (nodes in the original road network), we can instead visualize the graph at a higher-level where nodes represent something more meaningful, e.g., instead of intersections, nodes at layer 2 shown in Figure 2(b) might represent neighborhoods and edges represent routes from one neighborhood to another. Thus, hLP uncovers the hierarchical higher-order organization of complex networks.

References

(1)
Alon (2003) Uri Alon. 2003. Biological networks: the tinkerer as an engineer. Science 301, 5641 (2003), 1866–1867.
Blondel et al. (2008) Vincent D Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. 2008. Fast unfolding of communities in large networks. JSTAT 10 (2008).
Buehrer and Chellapilla (2008) Gregory Buehrer and Kumar Chellapilla. 2008. A scalable pattern mining approach to web graph compression with communities. In WSDM. 95–106.
Chung (1997) Fan RK Chung. 1997. Spectral graph theory. AMS.
Dhillon (2001) Inderjit S Dhillon. 2001. Co-clustering documents and words using bipartite spectral graph partitioning. In SIGKDD.
Felzenszwalb and Huttenlocher (2004) Pedro F Felzenszwalb and Daniel P Huttenlocher. 2004. Efficient graph-based image segmentation. IJCV 59, 2 (2004).
Fortunato (2010) Santo Fortunato. 2010. Community detection in graphs. Phy. Rep. 3 (2010).
Giles (2006) C Lee Giles. 2006. The future of citeseer: citeseer x. In ECML. Springer, 2–2.
Girvan and Newman (2002) Michelle Girvan and Mark EJ Newman. 2002. Community structure in social and biological networks. PNAS 99, 12 (2002), 7821–7826.
Hendrickson and Leland (1995) Bruce Hendrickson and Robert Leland. 1995. An improved spectral graph partitioning algorithm for mapping parallel computations. SIAM SISC 16, 2 (1995).
Khuller and Saha (2009) Samir Khuller and Barna Saha. 2009. On finding dense subgraphs. In ICALP.
Newman (2001) M.E.J. Newman. 2001. The structure of scientific collaboration networks. PNAS 98, 2 (2001), 404.
Raghavan et al. (2007) Usha Nandini Raghavan, Réka Albert, and Soundar Kumara. 2007. Near linear time algorithm to detect community structures in large-scale networks. Physical Review E 76, 3 (2007), 036106.
Rossi and Ahmed (2016) Ryan A. Rossi and Nesreen K. Ahmed. 2016. An Interactive Data Repository with Visual Analytics. SIGKDD Exp. (2016). http://networkrepository.com
Rossi et al. (2015) Ryan A. Rossi, David Gleich, and Assefaw Gebremedhin. 2015. Parallel Maximum Clique Algorithms with Applications to Network Analysis. SISC (2015).
Schaeffer (2007) Satu Elisa Schaeffer. 2007. Graph clustering. Comp. sci. rev. 1, 1 (2007), 27–64.
Shi and Malik (2000) Jianbo Shi and Jitendra Malik. 2000. Normalized cuts and image segmentation. TPAMI 22, 8 (2000), 888–905.
Shin et al. (2016) Kijung Shin, Tina Eliassi-Rad, and Christos Faloutsos. 2016. CoreScope: Graph Mining Using k-Core Analysis–Patterns, Anomalies and Algorithms. In ICDM.
Simon (1991) Horst D Simon. 1991. Partitioning of unstructured problems for parallel processing. Comp. Sys. in Eng. 2, 2 (1991).
Van Driessche and Roose (1995) Rafael Van Driessche and Dirk Roose. 1995. An improved spectral bisection algorithm and its application to dynamic load balancing. Parallel comp. (1995).