ADVANCED TOPICS
IN INFORMATION RETRIEVAL
AND WEB SEARCH
Lecture 1:
Introduction
S. M. Vahidipour
[email protected]
Outline
□ Introduction to the Course
□ Overview of the Semester
2
Text Books
Search Engines:
Information Retrieval in Practice
W. Bruce Croft, Donald Metzler, Trevor Strohman
Pearson Education, 2010
3
Text Books
Modern Information Retrieval:
The Concepts and Technology behind Search
(2nd Edition)
Ricardo Baeza-Yates, Berthier Ribeiro-Neto
ACM Press Books, 2010
4
Text Books
Introduction to Information Retrieval
C. Manning, P. Raghavan, and H. Schütze
Cambridge University Press, 2008
5
Search and Information Retrieval
Search on the Web is a daily activity for many people
throughout the world
□ Google: 40,000 searches per second (3.5 billion per
day; 1.2 trillion per year)
□ Yahoo: 3,200 searches per second (280 million per day;
8.4 billion per month)
□ Bing: 927 searches per second ( 80 million per day;
2.4 billion per month)
106: Million, 109: billion, 1012: Trillion, 1015: Quadrillion, 1018: Quintillion, …
6
Search and Information Retrieval
□ Search and communication are most popular uses of the computer.
□ Applications involving search are everywhere.
□ The field of computer science that is most involved with R&D for search
is information retrieval (IR).
7
Information Retrieval
“Information retrieval is a field concerned with the structure, analysis,
organization, storage, searching, and retrieval of information.”
(Salton, 1968)
□ General definition that can be applied to many types of information
and search applications
□ Still appropriate after 40 years.
□ Primary focus of IR since the 50s has been on text and documents
8
Data/Information
□ Storage
□ Search
9
Data/Information
□ Structured
□ Unstructured
10
Structured vs. Unstructured Data
11
What is a Document?
Examples:
Web pages, email, books, news stories, scholarly papers, text
messages, Word™, Powerpoint™, PDF, forum postings, patents, IM
(Instant Messages) sessions, etc.
Common properties
Significant text content
Some structure (≈ attributes in DB)
□ Papers: title, author, date
□ Email: subject, sender, destination, date
12
Comparing Text
Comparing the query text to the document text and determining what is
a good match is the core issue of information retrieval.
Exact matching of words is not enough
Many different ways to write the same thing in a “natural language” like
English
Does a news story containing the text “karl benz built the first automobile in 1886” match
the query “car inverter”?
Defining the meaning of a word, a sentence, a paragraph, or a story is
more difficult than defining the meaning of a database field.
13
Dimensions of IR
IR is more than just text, and more than just web search
although these are central
People doing IR work with different media, different types of search
applications, and different tasks
Three dimensions of IR
□ Content
□ Applications
□ Tasks
20
The Content Dimension
Textual data, but…
New applications increasingly involve new media
□ Video, photos, music, speech
□ Scanned documents (for legal purposes)
Like text, content is difficult to describe and compare
□ Text may be used to represent them (e.g., tags)
IR approaches to search and evaluation are appropriate
15
The Application D imension
Web search Desktop search
□ Personal enterprise search
□ Most common
□ See above plus recent web pages
Vertical search
P2P search
□ Restricted domain/topic
□ No centralized control
□ Books, movies, suppliers □ File sharing, shared locality
Enterprise search Literature search
□ Corporate intranet
□ Databases, emails, web pages, Forum search
documentation, code, wikis, tags,
directories, presentations, spreadsheets …
16
The Task Dimension
User queries / ad-hoc search
□ Range of query enormous, not pre-specified
Filtering
□ Given a profile (interests), notify about interesting news stories
□ Identify relevant user profiles for a new document
Classification / categorization
□ Automatically assign text to one or more classes of a given set
□ Identify relevant labels for documents
Question answering
□ Similar to search
□ Automatically answer a question posed in natural language
□ Provide concrete answer, not list of documents.
17
Main Issues in IR
Relevance
□ A relevant document contains the information a user was looking for when
he/she submitted the query
Evaluation
□ How well does the ranking meet the expectation of the user
Users and information needs
□ Users of a search engine are the ultimate judges of quality
18
IR and Search Engines
A search engine is the practical application of information retrieval
techniques to large scale text collections
Big issues include main IR issues but also some others…
Information Retrieval Search Engines
● Relevance: Effective ranking ● Performance: Efficient search and indexing
● Evaluation: Testing and measuring ● Incorporating new data: Coverage and freshness
● Information needs: User interaction ● Scalability: Growing with data and users
● Adaptability: Tuning for applications
● Specific problems: e.g., Spam
Additional
19
Outline
□ Introduction to the Course
□ Overview of the Semester
20
Search Engine
Basic architecture
Main issues
Indexing
Text acquisition
Text
transformation
Index creation
Querying
User interaction
Ranking
Evaluation
21
Overview of Traditional Retrieval Models
Boolean retrieval
Vector space model
Probabilistic models
22
Overview of Evaluation Metrics
Effectiveness metrics
Efficiency metrics
Training, testing, and statistics
23
Advanced Retrieval Models
Language model-based retrieval
Learning to rank
30
Word Mismatch Problem
Language model-based approaches
□ Translation model
□ Topic model
□ Word cluster model
□ Wordnet
□ Dependency model
Query expansion approaches
25
Advanced/Specific IR Tasks
Query log and query suggestion
Personalized search
Information extraction
Cross-language IR
Question answering
Recommendation systems
Enterprise search
Digital library
Structured text retrieval
Multimedia retrieval
26
Query Log and Query Suggestion
27
Personalized Search
28
Information Extraction
29
Cross- language Retrieval
30
Question Answering
31
Recommendation Systems
32
Enterprise Search
33
Digital Library
40
Structured Text Retrieval
35
Multimedia Retrieval
36
Questions?
37
