CI-6226
Lecture 5 part 1.
Index Construction & Compressing
Information Retrieval and Analysis
Vasily Sidorov
1
Let’s Recall
Term TermID
friend 1
roman 2
countryman 3
Can also be positional
lend 4
i 5 TermID Freq. Postings List (DocIDs)
you 6 → 1 → 5 → 6 → 12
1 4
ear 7
5 2 → 1 → 8
Dictionary 7 6 → 1 → 2 → 6 → 8 → 12 → 13
Inverted Index
3
Sec. 4.1
Hardware basics
• Many design decisions in information retrieval are
based on the characteristics of hardware
• We begin by reviewing hardware basics
5
Hardware basics
6
Sec. 4.1
Hardware basics
• Access to data in memory (RAM) is much faster
than access to data on disk.
• Disk seeks: No data is transferred from disk while
the disk head is being positioned.
• Therefore: Transferring one large chunk of data
from disk to memory is faster than transferring
many small chunks.
• Disk I/O is block-based: Reading and writing of
entire blocks (as opposed to smaller chunks).
• Block sizes: 8KB to 256 KB.
7
Hardware basics
• Solid State Drives (SSD) are mitigating some of the
problems:
—~100 times faster access time than HDDs
—1-2 order of magnitude faster I/O than HDDs
—Possesses a property of Random Access — (almost)
no “disk seek” delay
• Still much slower than RAM
• Still reads/writes in blocks
• Still too expensive compared to HDD
8
Sec. 4.1
Hardware basics
• Servers used in IR systems now typically have
hundreds of GB of main memory, sometimes
several TB
• Available disk space is several (2–3) orders of
magnitude larger. 128 GB of RAM → 5–10 TB of disk
• Fault tolerance is very expensive: It’s much cheaper
to use many regular machines rather than one fault
tolerant machine.
9
Sec. 4.1
Hardware assumptions for this
lecture
symbol statistic value
s average seek time 5 ms = 5 x 10−3 s
b transfer time per byte 0.02 μs = 2 x 10−8 s
processor’s clock rate 109 s−1
p low-level operation 0.01 μs = 10−8 s
(e.g., compare & swap a word)
size of main memory several GB
size of disk space 1 TB or more
10
Sec. 4.2
RCV1: Our collection for this lecture
• Shakespeare’s collected works definitely aren’t large
enough for demonstrating many of the points in
this course.
• The collection we’ll use isn’t really large enough
either, but it’s publicly available and is at least a
more plausible example.
• As an example for applying scalable index
construction algorithms, we will use the Reuters
RCV1 collection.
• This is one year of Reuters newswire (part of 1995
and 1996)
11
Sec. 4.2
A Reuters RCV1 document
12
Sec. 4.2
Reuters RCV1 statistics
symbol statistic value
N documents 800,000
L avg. # tokens per doc 200
M terms (= word types) 400,000
avg. # bytes per token 6
(incl. spaces/punct.)
avg. # bytes per token 4.5
(without spaces/punct.)
avg. # bytes per term 7.5
non-positional postings 100,000,000
4.5 bytes per token vs. 7.5 bytes per term: why? 13
Sec. 4.2
Recall IIR 1 index construction Term
I
Doc #
1
did 1
enact 1
julius 1
• Documents are parsed to extract words and these caesar
I
1
1
are saved with the Document ID. was 1
killed 1
i' 1
the 1
capitol 1
brutus 1
killed 1
me 1
Doc 1 Doc 2 so 2
let 2
it 2
be 2
I did enact Julius So let it be with with 2
Caesar I was killed Caesar. The noble
caesar
the
2
2
i' the Capitol; Brutus hath told you noble
brutus
2
2
Brutus killed me. Caesar was ambitious hath 2
told 2
you 2
caesar 2
was 14 2
ambitious 2
Sec. 4.2
Key step Term
I
Doc #
1
Term
ambitious
Doc #
2
did 1 be 2
enact 1 brutus 1
julius 1 brutus 2
• After all documents have been caesar
I
1
1
capitol
caesar
1
1
parsed, the inverted file is sorted was 1 caesar 2
caesar 2
by terms. killed
i'
1
1 did 1
the 1 enact 1
capitol 1 hath 1
brutus 1 I 1
killed 1 I 1
We focus on this sort step. me
so
1
2
i'
it
1
2
We have 100M items to sort. let
it
2
2
julius
killed
1
1
be 2 killed 1
with 2 let 2
caesar 2 me 1
the 2 noble 2
noble 2 so 2
brutus 2 the 1
hath 2 the 2
told 2 told 2
you 2 you 2
caesar 2 was 1
was 2 was 2
ambitious 2 with 2
15
Sec. 4.2
Scaling index construction
• In-memory index construction does not scale
—Can’t stuff entire collection into memory, sort, then
write back
• How can we construct an index for very large
collections?
• Taking into account the hardware constraints we
just learned about . . .
• Memory, disk, speed, etc.
16
Sec. 4.2
Sort-based index construction
• As we build the index, we parse docs one at a time.
—While building the index, we cannot easily exploit
compression tricks (you can, but much more complex)
• The final postings for any term are incomplete until the end.
• At 12 bytes per non-positional postings entry (term, doc,
freq), demands a lot of space for large collections.
• T = 100,000,000 in the case of RCV1
—So… we can do this in memory in 2021, but typical
collections are much larger. E.g., the New York Times
provides an index of >150 years of newswire
• Thus: We need to store intermediate results on disk.
17
Sec. 4.2
Sort using disk as “memory”?
• Can we use the same index construction algorithm
for larger collections, but by using disk instead of
memory?
• No: Sorting T = 100,000,000 records on disk is too
slow – too many disk seeks.
• We need an external sorting algorithm.
18
Sec. 4.2
Bottleneck
• Parse and build postings entries one doc at a time
• Now sort postings entries by term (then by doc
within each term)
• Doing this with random disk seeks would be too
slow – must sort T=100M records
If every comparison took 2 disk seeks, and N items could be
sorted with N log2N comparisons, how long would this take?
19
Sec. 4.2
BSBI: Blocked Sort-Based Indexing
(sorting with fewer disk seeks)
• 12-byte (4+4+4) records (term, doc, freq)
• These are generated as we parse docs
• Must now sort 100M such 12-byte records by term.
• Define a Block ~ 10M such records
—Can easily fit a couple into memory.
—Will have 10 such blocks to start with.
• Basic idea of algorithm:
—Accumulate postings for each block, sort, write to
disk.
—Then merge the blocks into one long sorted order.
20
21
Sec. 4.2
Sorting 10 blocks of 10M records
▪First, read each block and sort within:
▪Quicksort takes 2N ln N expected steps
▪In our case 2 x (10M ln 10M) steps
▪Exercise: estimate total time to read each block
from disk and quicksort it.
▪10 times this estimate – gives us 10 sorted runs of
10M records each.
▪Done straightforwardly, need 2 copies of data on
disk
▪But can optimize this
22
Sec. 4.2
23
Sec. 4.2
How to merge the sorted runs?
• Can do binary merges, with a merge tree of log210 = 4 layers.
• During each layer, read into memory runs in blocks of 10M,
merge, write back.
1
1 2
2 Merged run.
3 4
3
4
Runs being
merged.
Disk
24
Sec. 4.2
How to merge the sorted runs?
• But it is more efficient to do a multi-way merge, where you
are reading from all blocks simultaneously
• Providing you read decent-sized chunks of each block into
memory and then write out a decent-sized output chunk,
then you’re not killed by disk seeks
25
Sec. 4.3
Remaining problem with sort-based
algorithm
• Our assumption was: we can keep the dictionary in
memory.
• We need the dictionary (which grows dynamically)
in order to implement a term to termID mapping.
• Actually, we could work with term,docID postings
instead of termID,docID postings . . .
• . . . but then intermediate files become very large.
(We would end up with a scalable, but very slow
index construction method.)
26
Sec. 4.3
SPIMI:
Single-Pass In-Memory Indexing
• Key idea 1: Generate separate dictionaries for each
block – no need to maintain term-termID mapping
across blocks.
• Key idea 2: Don’t sort. Accumulate postings in
postings lists as they occur.
• With these two ideas we can generate a complete
inverted index for each block.
• These separate indexes can then be merged into
one big index.
27
Sec. 4.3
SPIMI-Invert
Merging of blocks is analogous to BSBI. 28
Sec. 4.3
SPIMI: Compression
• Compression makes SPIMI even more efficient.
—Compression of terms
—Compression of postings
• We’ll discuss later today
29
Sec. 4.4
Distributed indexing
• For web-scale indexing (don’t try this at home!):
must use a distributed computing cluster
• Individual machines are fault-prone
—Can unpredictably slow down or fail
• How do we exploit such a pool of machines?
30
Sec. 4.4
Web search engine data centers
• Web search data centers (Google, Bing, Baidu)
mainly contain commodity machines.
https://www.pcworld.com/article/112891/article.html
• Data centers are distributed around the world
—15 locations in the world
◦ There’s one in Jurong West Avenue 2
• Estimate: Google uses ~900K servers (2011 report)
http://www.datacenterknowledge.com/archives/2011/08/01/report-google-
uses-about-900000-servers
31
Google Data Center in Jurong West
32
Sec. 4.4
Massive data centers
• If in a non-fault-tolerant system with 1000 nodes,
each node has 99.9% uptime, what is the uptime of
the system?
• Answer: 37%
• Exercise: Calculate the number of servers failing per
minute for an installation of 1 million servers.
• Colloquially, SLA is counted in “nines”, e.g., an SLA
of four “nines”: 99.99% uptime.
33
Sec. 4.4
Distributed indexing
• Maintain a master machine directing the indexing
job – considered “safe”.
• Break up indexing into sets of (parallel) tasks.
• Master machine assigns each task to an idle
machine from a pool.
34
Sec. 4.4
Parallel tasks
• We will use two sets of parallel tasks
—Parsers
—Inverters
• Break the input document collection into splits
• Each split is a subset of documents (corresponding
to blocks in BSBI/SPIMI)
35
Sec. 4.4
Parsers
• Master assigns a split to an idle parser machine
• Parser reads one document at a time and emits
(term, doc) pairs
• Parser writes pairs into j partitions
• Each partition is for a range of terms’ first letters
—(e.g., a-f, g-p, q-z) – here j = 3.
• Now to complete the index inversion
36
Sec. 4.4
Inverters
• An inverter collects all (term,doc) pairs (= postings)
for one term-partition.
• Sorts and writes to postings lists
37
Sec. 4.4
Data flow
assign Master assign
Postings
Parser a-f g-p q-z Inverter a-f
Parser a-f g-p q-z
Inverter g-p
splits Inverter q-z
Parser a-f g-p q-z
Map Reduce
Segment files
phase phase 38
Sec. 4.4
MapReduce
• The index construction algorithm we just described
is an instance of MapReduce.
• MapReduce (Dean & Ghemawat 2004) is a robust
and conceptually simple framework for distributed
computing …
• … without having to write code for the distribution
part.
• They describe the Google indexing system (ca.
2002) as consisting of a number of phases, each
implemented in MapReduce.
39
Sec. 4.4
MapReduce
• Index construction was just one phase.
• Another phase: transforming a term-partitioned
index into a document-partitioned index.
—Term-partitioned: one machine handles a subrange
of terms
—Document-partitioned: one machine handles a
subrange of documents
• As we’ll discuss in the web part of the course, most
search engines use a document-partitioned index
for better load balancing, etc.
40
Sec. 4.5
Dynamic indexing
• Up to now, we have assumed that collections are
static.
• They rarely are:
—Documents come in over time and need to be
inserted.
—Documents are deleted and modified.
• This means that the dictionary and postings lists
have to be modified:
—Postings updates for terms already in dictionary
—New terms added to dictionary
◦ #MoonbyulxPunch
43
Sec. 4.5
Simplest approach
• Maintain “big” main index
• New docs go into “small” auxiliary index
• Search across both, merge results
• Deletions
—Invalidation bit-vector for deleted docs
—Filter docs output on a search result by this
invalidation bit-vector
• Periodically, re-index into one main index
44
Issues with multiple indexes
• Collection-wide statistics are hard to maintain
—E.g., when we spoke of spell-correction: which of
several corrected alternatives do we present to the
user?
—We said, pick the one with the most hits
• How do we maintain the top ones with multiple
indexes and invalidation bit vectors?
—One possibility: ignore everything but the main
index for such ordering
• Will see more such statistics used in results ranking
49
Dynamic indexing at search engines
• All the large search engines now do dynamic
indexing
• Their indices have frequent incremental changes
—News, blogs, twitter, new topical web pages
• But (sometimes/typically) they also periodically
reconstruct the index from scratch
—Query processing is then switched to the new index,
and the old index is deleted
50
