Advances in Database Querying

Comp. Sci. and Engg. Dept. I.I.T., Bombay [email protected] http://www.cse.iitb.ernet.in/~sudarsha

S. Sudarshan

Acknowledgments

A. Balachandran Anand Deshpande Sunita Sarawagi S. Seshadri

2

Overview
Part Part Part Part 1: 2: 3: 4: Data Warehouses OLAP Data Mining Query Processing and Optimization

Part 1: Data Warehouses

Data, Data everywhere yet ...

I cant find the data I need
data is scattered over the network many versions, subtle differences

I cant get the data I need

need an expert to get the data

I cant understand the data I found

available data poorly documented

I cant use the data I found

results are unexpected data needs to be transformed from one form to other

What is a Data Warehouse?

A single, complete and consistent store of data obtained from a variety of different sources made available to end users in a what they can understand and use in a business context. [Barry Devlin]
6

Why Data Warehousing?

Which are our Which are our lowest/highest margin lowest/highest margin customers ?? customers What is the most What is the most effective distribution effective distribution channel? channel? Who are my customers Who are my customers and what products and what products are they buying? are they buying?

What product promWhat product prom-otions have the biggest -otions have the biggest impact on revenue? impact on revenue? What impact will What impact will new products/services new products/services have on revenue have on revenue and margins? and margins?

Which customers Which customers are most likely to go are most likely to go to the competition ?? to the competition

Decision Support
Used to manage and control business Data is historical or point-in-time Optimized for inquiry rather than update Use of the system is loosely defined and can be ad-hoc Used by managers and end-users to understand the business and make judgements
8

Evolution of Decision Support

60s: Batch reports
hard to find and analyze information inflexible and expensive, reprogram every request

70s: Terminal based DSS and EIS 80s: Desktop data access and analysis tools
query tools, spreadsheets, GUIs easy to use, but access only operational db

90s: Data warehousing with integrated OLAP engines and tools

9

What are the users saying...

Data should be integrated across the enterprise Summary data had a real value to the organization Historical data held the key to understanding data over time What-if capabilities are required
10

Data Warehousing -It is a process

Technique for assembling and managing data from various sources for the purpose of answering business questions. Thus making decisions that were not previous possible A decision support database maintained separately from the organizations operational database
11

Traditional RDBMS used for OLTP

Database Systems have been used traditionally for OLTP
clerical data processing tasks detailed, up to date data structured repetitive tasks read/update a few records isolation, recovery and integrity are critical

Will call these operational systems

12

OLTP vs Data Warehouse

OLTP
Application Oriented Used to run business Clerical User Detailed data Current up to date Isolated Data Repetitive access by small transactions Read/Update access

Warehouse (DSS)
Subject Oriented Used to analyze business Manager/Analyst Summarized and refined Snapshot data Integrated Data Ad-hoc access using large queries Mostly read access (batch update) 13

Data Warehouse Architecture

Relational Databases
Optimized Loader

Extraction Cleansing Data Warehouse Engine Analyze Query

Legacy Data

Purchased Data

Metadata Repository
14

From the Data Warehouse to Data Marts

Information
Individually Structured Departmentally Structured

Less History Normalized Detailed More

Organizationally Structured

Data Warehouse

Data
15

Users have different views of Data

OLAP Tourists: Browse information harvested by farmers Farmers: Harvest information from known access paths Explorers: Seek out the unknown and previously unsuspected rewards hiding in the detailed data
16

Organizationally structured

Wal*Mart Case Study

Founded by Sam Walton One the largest Super Market Chains in the US Wal*Mart: 2000+ Retail Stores SAM's Clubs 100+Wholesalers Stores
This case study is from Felipe Carinos (NCR Teradata) presentation made at Stanford Database Seminar
17

Old Retail Paradigm

Wal*Mart
Inventory Management Merchandise Accounts Payable Purchasing Supplier Promotions: National, Region, Store Level

Suppliers
Accept Orders Promote Products Provide special Incentives Monitor and Track The Incentives Bill and Collect Receivables Estimate Retailer Demands

18

New (Just-In-Time) Retail Paradigm

No more deals Shelf-Pass Through (POS Application)
One Unit Price
Suppliers paid once a week on ACTUAL items sold

Wal*Mart Manager
Daily Inventory Restock Suppliers (sometimes SameDay) ship to Wal*Mart

Warehouse-Pass Through
Stock some Large Items
Delivery may come from supplier

Distribution Center
Suppliers merchandise unloaded directly onto Wal*Mart Trucks
19

Information as a Strategic Weapon

Daily Summary of all Sales Information Regional Analysis of all Stores in a logical area Specific Product Sales Specific Supplies Sales Trend Analysis, etc. Wal*Mart uses information when negotiating with
Suppliers Advertisers etc.
20

Schema Design
Database organization
must look like business must be recognizable by business user approachable by business user Must be simple

Schema Types
Star Schema Fact Constellation Schema Snowflake schema
21

Star Schema
A single fact table and for each dimension one dimension table Does not capture hierarchies directly
T i
m date, custno, prodno, cityname, sales

e c u s t

f a c t

p r o d c i t y

22

Dimension Tables
Dimension tables
Define business in terms already familiar to users Wide rows with lots of descriptive text Small tables (about a million rows) Joined to fact table by a foreign key heavily indexed typical dimensions
time periods, geographic region (markets, cities), products, customers, salesperson, etc.
23

Fact Table
Central table
Typical example: individual sales records mostly raw numeric items narrow rows, a few columns at most large number of rows (millions to a billion) Access via dimensions

24

Snowflake schema
Represent dimensional hierarchy directly by normalizing tables. Easy to maintain and saves storage
T i
m
date, custno, prodno, cityname, ...

e c u s t

f a c t

p r o d c i t y
r e g i o 25 n

Fact Constellation
Fact Constellation
Multiple fact tables that share many dimension tables Booking and Checkout may share many dimension tables in the hotel industry
Hotels

Booking Checkout

Promotion

Travel Agents

Room Type Customer

26

Data Granularity in Warehouse

Summarized data stored
reduce storage costs reduce cpu usage increases performance since smaller number of records to be processed design around traditional high level reporting needs tradeoff with volume of data to be stored and detailed usage of data
27

Granularity in Warehouse
Solution is to have dual level of granularity
Store summary data on disks
95% of DSS processing done against this data

Store detail on tapes

5% of DSS processing against this data

28

Levels of Granularity
Banking Example
Operational
account activity date amount teller location account bal account month # trans withdrawals monthly account deposits register -- up to average bal

10 years 60 days of activity

Not all fields need be archived

amount activity date amount account bal

29

Data Integration Across Sources

Savings Loans Trust Credit card

Same data different name

Different data Same name

Data found here nowhere else

Different keys same data

30

Data Transformation
Operational/ Source Data Data Transformation Sequential Legacy Relational External

Accessing Capturing Extracting Reconciling Conditioning Loading

Householding Filtering Validating Scoring

Data transformation is the foundation for achieving single version of the truth Major concern for IT Data warehouse can fail if appropriate data transformation strategy is not developed

31

Data Transformation Example

Data Warehouse
encoding unit field
appl A - m,f appl B - 1,0 appl C - x,y appl D - male, female appl A - pipeline - cm appl B - pipeline - in appl C - pipeline - feet appl D - pipeline - yds appl A - balance appl B - bal appl C - currbal appl D - balcurr
32

Data Integrity Problems

Same person, different spellings Agarwal, Agrawal, Aggarwal etc... Multiple ways to denote company name Persistent Systems, PSPL, Persistent Pvt. LTD. Use of different names mumbai, bombay Different account numbers generated by different applications for the same customer Required fields left blank Invalid product codes collected at point of sale manual entry leads to mistakes in case of a problem use 9999999
33

Data Transformation Terms

Extracting Conditioning Scrubbing Merging Householding Enrichment Scoring Loading Validating Delta Updating

34

Data Transformation Terms

Householding
Identifying all members of a household (living at the same address) Ensures only one mail is sent to a household Can result in substantial savings: 1 million catalogues at Rs. 50 each costs Rs. 50 million . A 2% savings would save Rs. 1 million

35

Refresh
Propagate updates on source data to the warehouse Issues:
when to refresh how to refresh -- incremental refresh techniques

36

When to Refresh?
periodically (e.g., every night, every week) or after significant events on every update: not warranted unless warehouse data require current data (up to the minute stock quotes) refresh policy set by administrator based on user needs and traffic possibly different policies for different sources 37

Refresh techniques
Incremental techniques
detect changes on base tables: replication servers (e.g., Sybase, Oracle, IBM Data Propagator)
snapshots (Oracle) transaction shipping (Sybase)

compute changes to derived and summary tables maintain transactional correctness for incremental load

38

How To Detect Changes

Create a snapshot log table to record ids of updated rows of source data and timestamp Detect changes by:
Defining after row triggers to update snapshot log when source table changes Using regular transaction log to detect changes to source data
39

Querying Data Warehouses

SQL Extensions Multidimensional modeling of data
OLAP More on OLAP later

40

SQL Extensions
Extended family of aggregate functions
rank (top 10 customers) percentile (top 30% of customers) median, mode Object Relational Systems allow addition of new aggregate functions

Reporting features
running total, cumulative totals
41

Reporting Tools
Andyne Computing -- GQL Brio -- BrioQuery Business Objects -- Business Objects Cognos -- Impromptu Information Builders Inc. -- Focus for Windows Oracle -- Discoverer2000 Platinum Technology -- SQL*Assist, ProReports PowerSoft -- InfoMaker SAS Institute -- SAS/Assist Software AG -- Esperant Sterling Software -- VISION:Data
42

Decision support tools

Direct Query Reporting tools
Crystal reports

OLAP
Essbase

Mining tools
Intelligent Miner

Merge Clean Summarize Detailed transactional data

Data warehouse

Relational DBMS+ e.g. Redbrick GIS data

Operational data

Bombay branch Delhi branch Oracle

Calcutta branch IMS

Census data SAS

43

Deploying Data Warehouses

What business information keeps you in business today? What business information can put you out of business tomorrow? What business information should be a mouse click away? What business conditions are the driving the need for business information?
44

Cultural Considerations
Not just a technology project New way of using information to support daily activities and decision making Care must be taken to prepare organization for change Must have organizational backing and support
45

User Training
Users must have a higher level of IT proficiency than for operational systems Training to help users analyze data in the warehouse effectively

46

Warehouse Products
Computer Associates -- CA-Ingres Hewlett-Packard -- Allbase/SQL Informix -- Informix, Informix XPS Microsoft -- SQL Server Oracle -- Oracle7, Oracle Parallel Server Red Brick -- Red Brick Warehouse SAS Institute -- SAS Software AG -- ADABAS -- SQL Server, IQ, MPP
47

Sybase

Part 2: OLAP

48

Nature of OLAP Analysis

Aggregation -- (total sales, percent-tototal) Comparison -- Budget vs. Expenses Ranking -- Top 10, quartile analysis Access to detailed and aggregate data Complex criteria specification Visualization
Need interactive response to aggregate queries
49

Multi-dimensional Data
Measure - sales (actual, plan, variance)
W S N Juice Cola Milk Cream Toothpaste Soap 1 2 34 5 6 7

Re gi on

Dimensions: Product, Region, Time Hierarchical summarization paths

Product Industry Region Country Time Year

Product

Category

Region

Quarter

Product

City Office

Month Day

week
50

Month

Conceptual Model for OLAP

Numeric measures to be analyzed
e.g. Sales (Rs), sales (volume), budget, revenue, inventory

Dimensions
other attributes of data, define the space e.g., store, product, date-of-sale hierarchies on dimensions
e.g. branch -> city -> state
51

Operations
Rollup: summarize data
e.g., given sales data, summarize sales for last year by product category and region

Drill down: get more details

e.g., given summarized sales as above, find breakup of sales by city within each region, or within the Andhra region

52

More Cube Operations

Slice and dice: select and project
e.g.: Sales of soft-drinks in Andhra over the last quarter

Pivot: change the view of data

L 2 2 2 2 2 2 2 2 S Total 5 2 5 2 Q1 Q2 Total
2 2 5 5 22 2

Red 2 2 2 2 Blue 2 2 2 2 Total 2 2 2 2

S Total
22 22 22 2
53

More OLAP Operations

Hypothesis driven search: E.g. factors affecting defaulters
view defaulting rate on age aggregated over other dimensions for particular age segment detail along profession

Need interactive response to aggregate queries => precompute various aggregates

54

MOLAP vs ROLAP
MOLAP: Multidimensional array OLAP ROLAP: Relational OLAP
Type
Shirt Shirt Shirt Shirt Shirt Shirt Shirt ALL

Size
S L ALL S L ALL ALL ALL

Colour Amount
Blue Blue Blue Red Red Red ALL ALL 22 22 22 2 2 22 22 55 55

55

SQL Extensions
Cube operator
group by on all subsets of a set of attributes (month,city) redundant scan and sorting of data can be avoided

Various other non-standard SQL extensions by vendors

56

OLAP: 3 Tier DSS

Data Warehouse OLAP Engine Decision Support Client

Database Layer

Application Logic Layer

Presentation Layer

Store atomic data in industry standard Data Warehouse.

Generate SQL execution plans in the OLAP engine to obtain OLAP functionality.

Obtain multidimensional reports from the DSS Client.

57

Strengths of OLAP
It is a powerful visualization tool It provides fast, interactive response times It is good for analyzing time series It can be useful to find some clusters and outliners Many vendors offer OLAP tools
58

Brief History
Express and System W DSS Online Analytical Processing - coined by EF Codd in 1994 - white paper by Arbor Software Generally synonymous with earlier terms such as Decisions Support, Business Intelligence, Executive Information System MOLAP: Multidimensional OLAP (Hyperion (Arbor Essbase), Oracle Express) ROLAP: Relational OLAP (Informix MetaCube, Microstrategy DSS Agent)
59

OLAP and Executive Information Systems

Andyne Computing -- Pablo Arbor Software -- Essbase Cognos -- PowerPlay Comshare -- Commander OLAP Holistic Systems -- Holos Information Advantage -AXSYS, WebOLAP Informix -- Metacube Microstrategies --DSS/Agent Oracle -- Express Pilot -- LightShip Planning Sciences -Gentium Platinum Technology -ProdeaBeacon, Forest & Trees SAS Institute -- SAS/EIS, OLAP++ Speedware -- Media

60

Microsoft OLAP strategy

Plato: OLAP server: powerful, integrating various operational sources OLE-DB for OLAP: emerging industry standard based on MDX --> extension of SQL for OLAP Pivot-table services: integrate with Office 2000
Every desktop will have OLAP capability.

Client side caching and calculations Partitioned and virtual cube Hybrid relational and multidimensional storage

61

Part 3: Data Mining

62

Why Data Mining

Credit ratings/targeted marketing:
Given a database of 100,000 names, which persons are the least likely to default on their credit cards? Identify likely responders to sales promotions

Fraud detection
Which types of transactions are likely to be fraudulent, given the demographics and transactional history of a particular customer?

Customer relationship management:

Which of my customers are likely to be the most loyal, and which are most likely to leave for a competitor? :

Data Mining helps extract such information

63

Data mining
Process of semi-automatically analyzing large databases to find interesting and useful patterns Overlaps with machine learning, statistics, artificial intelligence and databases but
more scalable in number of features and instances more automated to handle heterogeneous data

64

Some basic operations

Predictive:
Regression Classification

Descriptive:
Clustering / similarity matching Association rules and variants Deviation detection

65

Classification
Given old data about customers and payments, predict new applicants loan eligibility.
Previous customers Age Salary Profession Location Customer type Classifier
Salary > 5 L
Prof. = Exec

Decision rules Good/ bad

New applicants data

66

Classification methods
Goal: Predict class Ci = f(x1, x2, .. Xn) Regression: (linear or any other polynomial)
a*x1 + b*x2 + c = Ci.

Nearest neighour Decision tree classifier: divide decision space into piecewise constant regions. Probabilistic/generative models Neural networks: partition by non-linear boundaries

67

Decision trees
Tree where internal nodes are simple decision rules on one or more attributes and leaf nodes are predicted class labels.
Salary < 1 M Prof = teacher Good Bad Bad Age < 30 Good

68

Pros and Cons of decision trees

Pros + Reasonable training time + Fast application + Easy to interpret + Easy to implement + Can handle large number of features Cons Cannot handle complicated relationship between features simple decision boundaries problems with lots of missing data

More information: http://www.stat.wisc.edu/~limt/treeprogs.html

69

Neural network
Set of nodes connected by directed weighted edges
Basic NN unit
x1 x2 x3 w1 w2 w3

A more typical NN
x1 x2 x3 Output nodes Hidden nodes

o = ( wi xi )
i =2

2 ( y) = 2 e y +

70

Pros and Cons of Neural Network

Pros + Can learn more complicated class boundaries + Fast application + Can handle large number of features Cons Slow training time Hard to interpret Hard to implement: trial and error for choosing number of nodes

Conclusion: Use neural nets only if decision trees/NN fail.

71

Bayesian learning
Assume a probability model on generation of data. p(d | c j ) p(c j ) predicted class : c = max p ( c j | d ) = max c c p(d ) Apply bayes theorem to find most likely class as:
j j

c = max
cj

p(c j )

Nave bayes: Assume attributes conditionally

independent given class value
72

p(a p(d )
i =2

| cj)

Clustering
Unsupervised learning when old data with class labels not available e.g. when introducing a new product. Group/cluster existing customers based on time series of payment history such that similar customers in same cluster. Key requirement: Need a good measure of similarity between instances. Identify micro-markets and develop policies for each 73

Association rules
Given set T of groups of items Example: set of item sets purchased Tea, rice, bread Goal: find all rules on itemsets of the form a-->b such that
support of a and b > user threshold s conditional probability (confidence) of b given a > user threshold c T Milk, cereal Tea, milk

Example: Milk --> bread Purchase of product A --> service B

cereal
74

Variants
High confidence may not imply high correlation Use correlations. Find expected support and large departures from that interesting..
see statistical literature on contingency tables.

Still too many rules, need to prune...

75

Prevalent Interesting
Analysts already know about prevalent rules Interesting rules are those that deviate from prior expectation Minings payoff is in finding surprising phenomena
1995 Milk and cereal sell together!

Zzzz...

1998

Milk and cereal sell together!

76

What makes a rule surprising?

Does not match prior expectation
Correlation between milk and cereal remains roughly constant over time

Cannot be trivially derived from simpler rules

Milk 10%, cereal 10% Milk and cereal 10% surprising Eggs 10% Milk, cereal and eggs 0.1% surprising! Expected 1%
77

Application Areas
Industry Finance Insurance Telecommunication Transport Consumer goods Data Service providers Utilities Application Credit Card Analysis Claims, Fraud Analysis Call record analysis Logistics management promotion analysis Value added data Power usage analysis
78

Data Mining in Use

The US Government uses Data Mining to track fraud A Supermarket becomes an information broker Basketball teams use it to track game strategy Cross Selling Target Marketing Holding on to Good Customers Weeding out Bad Customers
79

Why Now?
Data is being produced Data is being warehoused The computing power is available The computing power is affordable The competitive pressures are strong Commercial products are available

80

Data Mining works with Warehouse Data

Data Warehousing provides the Enterprise with a memory

Data Mining provides the Enterprise with intelligence

81

Mining market
Around 20 to 30 mining tool vendors Major players:
Clementine, IBMs Intelligent Miner, SGIs MineSet, SASs Enterprise Miner.

All pretty much the same set of tools Many embedded products: fraud detection,
electronic commerce applications
82

OLAP Mining integration

OLAP (On Line Analytical Processing)
Fast interactive exploration of multidim. aggregates. Heavy reliance on manual operations for analysis: Tedious and error-prone on large multidimensional data Ideal platform for vertical integration of mining but needs to be interactive instead of batch.
83

State of art in mining OLAP integration

Decision trees [Information discovery, Cognos]
find factors influencing high profits

Clustering [Pilot software]

segment customers to define hierarchy on that dimension

Time series analysis: [Seagates Holos]

Query for various shapes along time: eg. spikes, outliers etc

Multi-level Associations [Han et al.]

find association between members of dimensions
84

Vertical integration:
the web

Mining on

Web log analysis for site design: what are popular pages, what links are hard to find. Electronic stores sales enhancements: recommendations, advertisement: Collaborative filtering: Net perception, Wisewire Inventory control: what was a shopper looking for and could not find..
85

Part 4: Speeding up Query Processing

86