MySQL 8.0 Optimizer Guide

Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
MySQL 8.0 Optimizer Guide
Morgan Tocker
MySQL Product Manager (Server)
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended
for information purposes only, and may not be incorporated into any contract. It
is not a commitment to deliver any material, code, or functionality, and should
not be relied upon in making purchasing decisions. The development, release,
and timing of any features or functionality described for Oracle’s products
remains at the sole discretion of Oracle.
3

Agenda
1. Introduction
2. Server Architecture
3. B+trees
4. EXPLAIN
5. Optimizer Trace
6. Logical
Transformations
7. Cost Based
Optimization
8. Hints and Switches
9. Comparing Plans
10.Composite Indexes
11.Covering Indexes
12.Visual Explain
13.Transient Plans
14.Subqueries
15.CTEs and Views
16.Joins
17.Aggregation
18.Descending Indexes
19.Sorting
20.Partitioning
21.Query Rewrite
22.Invisible Indexes
23.Profiling
24.JSON
25.Character Sets
4

Introduction
• SQL is declarative
• You state “what you want” not “how you want”
• Can’t usually sight check queries to understand execution
efficiency
• Database management system is like a GPS navigation system. It
finds the “best”route.
5

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 6
GPS…

MySQL Optimizer

Diagnostic Commands
• EXPLAIN (all versions)
• EXPLAIN FORMAT=JSON (5.6+)
– Supported by Workbench in Visual format
• Optimizer Trace (5.6+)
8

Examples from “The World Schema”
• Contains Cities, Countries, Language statistics
• Download from:
– https://dev.mysql.com/doc/index-other.html
• Very small data set
– Good for learning
– Not good for explaining performance differences
9

Primary Table we are using
CREATE TABLE `Country` (
`Code` char(3) NOT NULL DEFAULT '',
`Name` char(52) NOT NULL DEFAULT '',
`Continent` enum('Asia','Europe','North America','Africa','Oceania','Antarctica','South America') NOT NULL DEFAULT
'Asia',
`Region` char(26) NOT NULL DEFAULT '',
`SurfaceArea` float(10,2) NOT NULL DEFAULT '0.00',
`IndepYear` smallint(6) DEFAULT NULL,
`Population` int(11) NOT NULL DEFAULT '0',
`LifeExpectancy` float(3,1) DEFAULT NULL,
`GNP` float(10,2) DEFAULT NULL,
`GNPOld` float(10,2) DEFAULT NULL,
`LocalName` char(45) NOT NULL DEFAULT '',
`GovernmentForm` char(45) NOT NULL DEFAULT '',
`HeadOfState` char(60) DEFAULT NULL,
`Capital` int(11) DEFAULT NULL,
`Code2` char(2) NOT NULL DEFAULT '',
PRIMARY KEY (`Code`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
1 row in set (0.00 sec)
10

Companion Website
• Content from “The Unofficial MySQL 8.0 Optimizer Guide”
• http://www.unofficialmysqlguide.com/
• More detailed text for many of the examples here…
• Most still applies to 5.6+
• EXPLAIN FORMAT=JSON in 5.6 does not show cost
• Costs will be different
• Output from Optimizer Trace may differ
• Some features will be missing
11

Danger: Code on slides!
• Some examples may appear small
• Please feel free to download this deck from:
• https://www.slideshare.net/morgo/mysql-80-optimizer-guide
• Follow along on your laptop
12

Agenda
1. Introduction
3. B+trees
4. EXPLAIN
5. Optimizer Trace
6. Logical
Transformations
7. Cost Based
Optimization
9. Comparing Plans
11.Covering Indexes
12.Visual Explain
13.Transient Plans
14.Subqueries
15.CTEs and Views
16.Joins
17.Aggregation
19.Sorting
20.Partitioning
21.Query Rewrite
23.Profiling
24.JSON
25.Character Sets
13

Server Architecture

Just the Important Parts
• Comprised of the Server and Storage Engines
• Query Optimization happens at the Server Level
• Semantically there are four stages of Query Optimization
• Followed by Query Execution
15

Agenda
1. Introduction
3. B+trees
4. EXPLAIN
5. Optimizer Trace
6. Logical
Transformations
7. Cost Based
Optimization
9. Comparing Plans
11.Covering Indexes
12.Visual Explain
13.Transient Plans
14.Subqueries
15.CTEs and Views
16.Joins
17.Aggregation
19.Sorting
20.Partitioning
21.Query Rewrite
23.Profiling
24.JSON
25.Character Sets
16

B+trees
• When we mean “add an index” we usually mean “add a B+tree
index”:
– Includes PRIMARY, UNIQUE, INDEX type indexes.
• Understanding the basic structure of B+trees helps with
optimization
17

Binary Tree
• Not the same as a B+tree
• Understand Binary Tree first then compare and contrast
18
Locate 829813 in a
(balanced) binary tree of
1MM ~= 20 hops.
is this good?

B+tree
• Amortizes disk accesses by clustering into pages:
• Can achieve same outcome in two hops:
19
CREATE TABLE users ( 
id INT NOT NULL auto_increment, 
username VARCHAR(32) NOT NULL, 
payload TEXT, 
PRIMARY KEY (id) 
);

B+tree
• Amortizes disk  
accesses by clustering 
into pages
• Can achieve same  
outcome in two hops:
20

B-trees are wide not deep
• From the root page: values >= 800788 but < 829908 are on page
16386.
• From page 16386: values >= 829804 but < 829830 are on leaf page
32012.
• Large fan out factor; 1000+ keys/page which point to another index
page with 1000+ keys/page
21

InnoDB uses a Clustered Index
• In InnoDB the data rows are also stored in a B+tree, organized by
the primary key
• Secondary key indexes always include the value of the primary key
22

Agenda
1. Introduction
3. B+trees
4. EXPLAIN
5. Optimizer Trace
6. Logical
Transformations
7. Cost Based
Optimization
9. Comparing Plans
11.Covering Indexes
12.Visual Explain
13.Transient Plans
14.Subqueries
15.CTEs and Views
16.Joins
17.Aggregation
19.Sorting
20.Partitioning
21.Query Rewrite
23.Profiling
24.JSON
25.Character Sets
23

EXPLAIN
• Pre-execution view of how MySQL intends to execute a query
• Prints what MySQL considers the best plan after a process of
considering potentially thousands of choices
24

EXPLAIN FORMAT=JSON
SELECT * FROM Country WHERE continent='Asia' and population > 5000000;
{
"query_block": {
"select_id": 1,
"cost_info": {
"query_cost": "25.40"
},
"table": {
"table_name": "country",
"access_type": "ALL",
"rows_examined_per_scan": 239,
"rows_produced_per_join": 11,
"filtered": "6.46",
..
"attached_condition": "((`world`.`country`.`Continent` = 'Asia')
and (`world`.`country`.`Population` > 5000000))"
}
}
}

What indexes will make this query faster?
• Some Suggestions:
– Index on p (population)
– Index on c (continent)
– Index on p_c (population, continent)
– Index on c_p (continent, population)
26

ALTER TABLE Country ADD INDEX p (population);
EXPLAIN FORMAT=JSON
{
"query_block": {
"select_id": 1,
"cost_info": {
},
"table": {
"table_name": "Country",
"possible_keys": [
"p"
],
"filtered": "6.46",
..
"attached_condition": "((`world`.`country`.`Continent` = 'Asia') and
(`world`.`country`.`Population` > 5000000))"
..

Why would an index not be used?

Agenda
1. Introduction
3. B+trees
4. EXPLAIN
5. Optimizer Trace
6. Logical
Transformations
7. Cost Based
Optimization
9. Comparing Plans
11.Covering Indexes
12.Visual Explain
13.Transient Plans
14.Subqueries
15.CTEs and Views
16.Joins
17.Aggregation
19.Sorting
20.Partitioning
21.Query Rewrite
23.Profiling
24.JSON
25.Character Sets
29

Optimizer Trace
• What other choices did EXPLAIN not show?
• Why was that choice made?
• Output is quite verbose
30

ALTER TABLE Country ADD INDEX p (population);
EXPLAIN FORMAT=JSON
{
"query_block": {
"select_id": 1,
"cost_info": {
},
"table": {
"possible_keys": [
"p"
],
"filtered": "6.46",
"cost_info": {
"read_cost": "23.86",
"eval_cost": "1.54",
"prefix_cost": "25.40",
"data_read_per_join": "3K"
},
..
"attached_condition": "((`world`.`country`.`Continent` = 'Asia') and
(`world`.`country`.`Population` > 5000000))"
..
It’s available but not
used. Why?

SET optimizer_trace="enabled=on";
SELECT * FROM information_schema.optimizer_trace;
{
"steps": [
{
"join_preparation": {
"select#": 1,
"steps": [
{
"expanded_query": "/* select#1 */ select `country`.`Code` AS `Code`,`country`.`Name` AS
`Name`,`country`.`Continent` AS `Continent`,`country`.`Region` AS `Region`,`country`.`SurfaceArea` AS
`SurfaceArea`,`country`.`IndepYear` AS `IndepYear`,`country`.`Population` AS
`Population`,`country`.`LifeExpectancy` AS `LifeExpectancy`,`country`.`GNP` AS
`GNP`,`country`.`GNPOld` AS `GNPOld`,`country`.`LocalName` AS `LocalName`,`country`.`GovernmentForm`
AS `GovernmentForm`,`country`.`HeadOfState` AS `HeadOfState`,`country`.`Capital` AS
`Capital`,`country`.`Code2` AS `Code2` from `country` where ((`country`.`Continent` = 'Asia') and
(`country`.`Population` > 5000000))"
}
]
}
},
{
"join_optimization": {
"select#": 1,
"steps": [
{
"condition_processing": {
"condition": "WHERE",
Page 1 of 6

"original_condition": "((`country`.`Continent` = 'Asia') and (`country`.`Population` > 5000000))",
"steps": [
{
"transformation": "equality_propagation",
"resulting_condition": "((`country`.`Population` > 5000000) and multiple equal('Asia',
`country`.`Continent`))"
},
{
"transformation": "constant_propagation",
},
{
"transformation": "trivial_condition_removal",
}
]
}
},
{
"substitute_generated_columns": {
}
},
{
"table_dependencies": [
{
"table": "`country`",
"row_may_be_null": false,
"map_bit": 0,
Page 2 of 6

"depends_on_map_bits": [
]
}
]
},
{
"ref_optimizer_key_uses": [
]
},
{
"rows_estimation": [
{
"range_analysis": {
"table_scan": {
"rows": 239,
"cost": 27.5
},
"potential_range_indexes": [
{
"index": "PRIMARY",
"usable": false,
"cause": "not_applicable"
},
{
"index": "p",
"usable": true,
"key_parts": [
"Population",
"Code"
]
Page 3 of 6

}
],
"setup_range_conditions": [
],
"group_index_range": {
"chosen": false,
"cause": "not_group_by_or_distinct"
},
"analyzing_range_alternatives": {
"range_scan_alternatives": [
{
"index": "p",
"ranges": [
"5000000 < Population"
],
"index_dives_for_eq_ranges": true,
"rowid_ordered": false,
"using_mrr": false,
"index_only": false,
"rows": 108,
"cost": 38.06,
"chosen": false,
"cause": "cost"
}
],
"analyzing_roworder_intersect": {
"usable": false,
"cause": "too_few_roworder_scans"
}
}
}
Aha! It was too
expensive.
Page 4 of 6

}
]
},
{
"considered_execution_plans": [
{
"plan_prefix": [
],
"best_access_path": {
"considered_access_paths": [
{
"rows_to_scan": 239,
"access_type": "scan",
"resulting_rows": 239,
"cost": 25.4,
"chosen": true
}
]
},
"condition_filtering_pct": 100,
"rows_for_plan": 239,
"cost_for_plan": 25.4,
"chosen": true
}
]
},
{
"attaching_conditions_to_tables": {
"original_condition": "((`country`.`Continent` =  
'Asia') and (`country`.`Population` > 5000000))",
Prefer to table scan
instead
Page 5 of 6

"attached_conditions_computation": [
],
"attached_conditions_summary": [
{
"attached": "((`country`.`Continent` = 'Asia') and (`country`.`Population` > 5000000))"
}
]
}
},
{
"refine_plan": [
{
"table": "`country`"
}
]
}
]
}
},
{
"join_execution": {
"select#": 1,
"steps": [
]
}
}
]
}
Page 6 of 6

Why would an index not be used?
"analyzing_range_alternatives": {
"range_scan_alternatives": [
{
"index": "p",
"ranges": [
"5000000 < Population"
],
"index_dives_for_eq_ranges": true,
"rowid_ordered": false,
"using_mrr": false,
"index_only": false,
"rows": 108,
"cost": 38.06,
"chosen": false,
"cause": "cost"
}
],
OPTIMIZER TRACE:
..
"query_block": {
"select_id": 1,
"cost_info": {
},
"table": {
"access_type": "range",
"possible_keys": [
"p"
],
"key": "p",
..
FORCE INDEX (p):

Reason again…

Agenda
1. Introduction
3. B+trees
4. EXPLAIN
5. Optimizer Trace
6. Logical
Transformations
7. Cost Based
Optimization
9. Comparing Plans
11.Covering Indexes
12.Visual Explain
13.Transient Plans
14.Subqueries
15.CTEs and Views
16.Joins
17.Aggregation
19.Sorting
20.Partitioning
21.Query Rewrite
23.Profiling
24.JSON
25.Character Sets
40

Logical Transformations
• First part of optimization 
is eliminating  
unnecessary work
41

Why eliminate unnecessary work?
• Short-cut/reduce number of execution plans that need to be
evaluated
• Transform parts of queries to take advantage of better execution
strategies
• Think of a how a compiler transforms code to be more efficient
• MySQL does similar at runtime
42

Example:
SELECT * FROM Country 
WHERE population > 5000000 AND continent='Asia'  
AND 1=1;
43

SHOW WARNINGS says:
EXPLAIN FORMAT=JSON SELECT * FROM Country WHERE population > 5000000 AND 1=1;
SHOW WARNINGS;
/* select#1 */ select 
`world`.`Country`.`Code` AS `Code`, 
`world`.`Country`.`Name` AS `Name`, 
`world`.`Country`.`Continent` AS `Continent`, 
`world`.`Country`.`Region` AS `Region`, 
`world`.`Country`.`SurfaceArea` AS `SurfaceArea`, 
`world`.`Country`.`IndepYear` AS `IndepYear`, 
`world`.`Country`.`Population` AS `Population`, 
`world`.`Country`.`LifeExpectancy` AS `LifeExpectancy`, 
`world`.`Country`.`GNP` AS `GNP`, 
`world`.`Country`.`GNPOld` AS `GNPOld`, 
`world`.`Country`.`LocalName` AS `LocalName`, 
`world`.`Country`.`GovernmentForm` AS `GovernmentForm`, 
`world`.`Country`.`HeadOfState` AS `HeadOfState`, 
`world`.`Country`.`Capital` AS `Capital`, 
`world`.`Country`.`Code2` AS `Code2` 
from `world`.`Country` 
where (`world`.`Country`.`Population` > 5000000)
44

..
"steps": [
{
"condition_processing": {
"condition": "WHERE",
"original_condition": "((`Country`.`Population` > 5000000) and (1 = 1))",
"steps": [
{
"transformation": "equality_propagation",
"resulting_condition": "((`Country`.`Population` > 5000000) and (1 = 1))"
},
{
"transformation": "constant_propagation",
"resulting_condition": "((`Country`.`Population` > 5000000) and (1 = 1))"
},
{
"transformation": "trivial_condition_removal",
"resulting_condition": "(`Country`.`Population` > 5000000)"
..
OPTIMIZER TRACE says:
45

What sort of transformations can occur?
• Merging views back with definition of base tables
• Derived table in FROM clause merged back into base tables
• Unique subqueries converted directly to INNER JOIN statements
• Primary key lookup converted to constant values.
– Shortcut plans that will need to be evaluated.
46

Primary Key Lookup
SELECT * FROM Country WHERE code='CAN'
/* select#1 */ select
'CAN' AS `Code`,
'Canada' AS `Name`,
'North America' AS `Continent`,
'North America' AS `Region`,
'9970610.00' AS `SurfaceArea`,
'1867' AS `IndepYear`,
'31147000' AS `Population`,
'79.4' AS `LifeExpectancy`,
'598862.00' AS `GNP`,
'625626.00' AS `GNPOld`,
'Canada' AS `LocalName`,
'Constitutional Monarchy, Federation' AS `GovernmentForm`,
'Elisabeth II' AS `HeadOfState`,
'1822' AS `Capital`,
'CA' AS `Code2`
from `world`.`Country` where 1
47

Primary key does not exist
SELECT * FROM Country WHERE code='XYZ'
/* select#1 */ select NULL AS `Code`,NULL AS
`Name`,NULL AS `Continent`,NULL AS `Region`, NULL AS
`SurfaceArea`,NULL AS `IndepYear`,NULL AS
`Population`,NULL AS `LifeExpectancy`,NULL AS `GNP`,
NULL AS `GNPOld`,NULL AS `LocalName`,NULL AS
`GovernmentForm`,NULL AS `HeadOfState`,NULL AS
`Capital`, NULL AS `Code2` from `world`.`Country`
where multiple equal('XYZ', NULL)
48

Impossible WHERE
SELECT * FROM Country WHERE code='CAN' AND 1=0
/* select#1 */ select `world`.`Country`.`Code` AS
`Code`,`world`.`Country`.`Name` AS `Name`,
`world`.`Country`.`Continent` AS `Continent`,`world`.`Country`.`Region`
AS `Region`, `world`.`Country`.`SurfaceArea` AS
`SurfaceArea`,`world`.`Country`.`IndepYear` AS `IndepYear`,
`world`.`Country`.`Population` AS
`Population`,`world`.`Country`.`LifeExpectancy` AS `LifeExpectancy`,
`world`.`Country`.`GNP` AS `GNP`,`world`.`Country`.`GNPOld` AS
`GNPOld`, `world`.`Country`.`LocalName` AS
`LocalName`,`world`.`Country`.`GovernmentForm` AS `GovernmentForm`,
`world`.`Country`.`HeadOfState` AS
`HeadOfState`,`world`.`Country`.`Capital` AS `Capital`,
`world`.`Country`.`Code2` AS `Code2` from `world`.`Country` where 0
49

Are transformations always safe?
• Yes they should be
• New transformations (and execution strategies) may return non
deterministic queries in a different order
• Some illegal statements as a result of derived_merge
transformation
50

Agenda
1. Introduction
3. B+trees
4. EXPLAIN
5. Optimizer Trace
6. Logical
Transformations
7. Cost Based
Optimization
9. Comparing Plans
11.Covering Indexes
12.Visual Explain
13.Transient Plans
14.Subqueries
15.CTEs and Views
16.Joins
17.Aggregation
19.Sorting
20.Partitioning
21.Query Rewrite
23.Profiling
24.JSON
25.Character Sets
52

Query Optimizer Strategy
• Model each of the possible execution plans (using support from
statistics and meta data)
• Pick the plan with the lowest cost
53

Model you say?
1. Assign a cost to each operation
2. Evaluate how many operations each possible plan would take
3. Sum up the total
4. Choose the plan with the lowest overall cost
54

How are statistics calculated?
• Dictionary Information
• Cardinality Statistics
• Records In Range Dynamic Sampling
• Table Size
55

Example Model: Table Scan
IO Cost:
# pages in table * (IO_BLOCK_READ_COST | MEMORY_BLOCK_READ_COST)
CPU Cost:
# records * ROW_EVALUATE_COST
Defaults: 
IO_BLOCK_READ_COST = 1 
MEMORY_BLOCK_READ_COST = 0.25 
ROW_EVALUATE_COST=0.1
Values:
# pages in table = 6
# records = 239
100% on Disk:  
= (6 * 1) + (0.1 * 239)
= 29.9
EXPLAIN said
cost was 25.40
New! MySQL 8.0 estimates how
many of the pages will be in
memory.
SELECT clust_index_size from
INNODB_SYS_TABLESTATS WHERE
name='world/country'
100% in Memory: 
= (6 * 0.25) + (0.1 * 239)
= 25.4

Example Model: Range Scan
IO Cost:
# records_in_range * (IO_BLOCK_READ_COST | MEMORY_BLOCK_READ_COST) 
CPU Cost: 
# records_in_range * ROW_EVALUATE_COST 
+ # records_in_range * ROW_EVALUATE_COST  
 
 
 
 
= (108 * 0.25) + ( (108 * 0.1) + (108 * 0.1) )
= 48.6
Evaluate range condition
Evaluate WHERE condition
Compares to "query_cost":
“48.86" in EXPLAIN.
100% in memory.
On disk = 129.6

{
"query_block": {
"select_id": 1,
"cost_info": {
},
"table": {
"possible_keys": [
"p"
],
..
"cost_info": {
"read_cost": "23.86",
},
..
CPU Cost
IO Cost
Total Cost

Cost Constant Refinement
UPDATE mysql.server_cost SET cost_value=1 WHERE cost_name=‘row_evaluate_cost';
UPDATE mysql.engine_cost set cost_value = 1;
FLUSH OPTIMIZER_COSTS;
EXPLAIN FORMAT=JSON SELECT * FROM Country WHERE continent='Asia' and population > 5000000;
{
"query_block": {
"select_id": 1,
"cost_info": {
},
"table": {
..
60
Increase row evaluate cost from
0.1 to 1. Make memory and IO
block read cost the same.
New Table Scan Cost:
= (6 * 1) + (1 * 239)
= 245

Are plans exhaustively evaluated?
• Short cuts are taken to not spend too much time in planning:
– Some parts of queries may be transformed to limit plans
evaluated
– The optimizer will by default limit the search depth of bad plans: 
optimizer_search_depth=64 
optimizer_prune_level=1
61

Agenda
1. Introduction
3. B+trees
4. EXPLAIN
5. Optimizer Trace
6. Logical
Transformations
7. Cost Based
Optimization
9. Comparing Plans
11.Covering Indexes
12.Visual Explain
13.Transient Plans
14.Subqueries
15.CTEs and Views
16.Joins
17.Aggregation
19.Sorting
20.Partitioning
21.Query Rewrite
23.Profiling
24.JSON
25.Character Sets
62

How often is the query optimizer wrong?
• Yes it happens
• Similar to GPS; you may not have traffic data available for all
streets
• The model may be incomplete or imperfect
• There exist method(s) to overwrite it
63

Hints and Switches
• Typically a better level of override to modifying cost constants
• Come in three varieties:
– Old Style Hints
– New Comment-Style Hints
– Switches
64

Old Style Hints
• Have SQL and Hint intermingled
• Cause errors when indexes don’t exist
65
SELECT * FROM Country FORCE INDEX (p) WHERE population >
5000000;
SELECT * FROM Country IGNORE INDEX (p) WHERE population >
5000000;
SELECT * FROM Country USE INDEX (p) WHERE population > 5000000;
SELECT STRAIGHT_JOIN ..;
SELECT * FROM Country STRAIGHT_JOIN ..;

New Comment-Style Hints
• Can be added by a system that doesn’t understand SQL
• Clearer defined semantics as a hint not a directive
• Fine granularity
66
SELECT
/*+ NO_RANGE_OPTIMIZATION (Country) */
* FROM Country
WHERE Population > 1000000000 AND Continent=‘Asia';

Switches
• As new optimizations are added, some cause regressions
• Allow the specific optimization to be disabled (SESSION or GLOBAL)
67
SELECT @@optimizer_switch;
 
index_merge=on,index_merge_union=on,index_merge_sort_union=on, 
index_merge_intersection=on,engine_condition_pushdown=on, 
index_condition_pushdown=on,mrr=on,mrr_cost_based=on, 
block_nested_loop=on,batched_key_access=off,materialization=on, 
semijoin=on,loosescan=on,firstmatch=on,duplicateweedout=on, 
subquery_materialization_cost_based=on,use_index_extensions=on, 
condition_fanout_filter=on,derived_merge=on

How to consider hints and switches
• They provide immediate pain relief to production problems at the
cost of maintenance
• They add technical debt to your applications
68

Agenda
1. Introduction
3. B+trees
4. EXPLAIN
5. Optimizer Trace
6. Logical
Transformations
7. Cost Based
Optimization
9. Comparing Plans
11.Covering Indexes
12.Visual Explain
13.Transient Plans
14.Subqueries
15.CTEs and Views
16.Joins
17.Aggregation
19.Sorting
20.Partitioning
21.Query Rewrite
23.Profiling
24.JSON
25.Character Sets
69

Our simple query with n candidate indexes
• Indexes exist on p(population) and c(continent):
70
SELECT * FROM Country 
WHERE population > 50000000 AND continent=‘Asia';
>50M, how many
are less?
How many countries in
Asia vs total world?
Does order of
predicates matter? No.

Role of the Optimizer
• Given these many choices, which is the best choice?
• A good GPS navigator finds the fastest route!
• We can expect a good query optimizer to do similar
71

ALTER TABLE Country ADD INDEX c (continent);
EXPLAIN FORMAT=JSON # 50M 
SELECT * FROM Country WHERE population > 50000000 AND continent=‘Asia';
{
"query_block": {
"select_id": 1,
"cost_info": {
},
"table": {
"access_type": "ref",
"possible_keys": [
"p",
"c"
],
"key": "c",
"used_key_parts": [
"Continent"
],
"key_length": "1",
"ref": [
"const"
],
..
"attached_condition": "(`world`.`country`.`Population` > 50000000)"
..
Continent is
determined to be lower
cost.

EXPLAIN FORMAT=JSON # 500M
{
"query_block": {
"select_id": 1,
"cost_info": {
},
"table": {
"possible_keys": [
"p",
"c"
],
"key": "p",
"used_key_parts": [
"Population"
],
"key_length": "4",
..
"attached_condition": "(`world`.`country`.`Continent` = ‘Asia')"
..
Change the predicate,
the query plan changes.

Query Plan Evaluation
• Evaluated for each query, and thus each set of predicates
• Currently not cached*
• For prepared statements, permanent transformations are cached
74
* Cardinality statistics are cached. Don’t get confused.

Cost Estimates
p>5M c=’Asia’ p>50M, c=’Asia’ p>500M, c=’Asia’
p 48.86 11.06 1.16
c 9.60 9.60 9.60
ALL 25.40 25.40 25.40
p

Agenda
1. Introduction
3. B+trees
4. EXPLAIN
5. Optimizer Trace
6. Logical
Transformations
7. Cost Based
Optimization
9. Comparing Plans
11.Covering Indexes
12.Visual Explain
13.Transient Plans
14.Subqueries
15.CTEs and Views
16.Joins
17.Aggregation
19.Sorting
20.Partitioning
21.Query Rewrite
23.Profiling
24.JSON
25.Character Sets
76

The role of composite indexes
• Useful when two or more predicates combined improves filtering
effect. i.e. 
 
Not all countries with a population > 5M are in Asia
77

Composite Indexes
• p_c (population, continent)
• c_p (continent, population)
78

ALTER TABLE Country ADD INDEX p_c (Population, Continent);
EXPLAIN FORMAT=JSON
SELECT * FROM Country FORCE INDEX (p_c) WHERE continent='Asia' and population >
5000000;
{
"query_block": {
"select_id": 1,
"cost_info": {
},
"table": {
"possible_keys": [
"p_c"
],
"key": "p_c",
"used_key_parts": [
"Population"
],
"key_length": "4",
..
Only part of the key is
used!

Rule of Thumb
• Index on (const, range) instead of (range, const)
• Applies to all databases
80

ALTER TABLE Country ADD INDEX c_p (Continent, Population); 
EXPLAIN FORMAT=JSON 
SELECT * FROM Country WHERE continent='Asia' and population > 5000000; 
{
"query_block": {
"select_id": 1,
"cost_info": {
},
"table": {
"possible_keys": [
"p",
"c",
"p_c",
"c_p"
],
"key": "c_p",
"used_key_parts": [
"Continent",
"Population"
],
"key_length": “5”, 
..
All of the key is
used

Composite Left-most Rule
• An index on (Continent, Population) can also be used as an index on
(Continent)
• It can not be used as an index on (Population)
82

EXPLAIN FORMAT=JSON  
SELECT * FROM Country FORCE INDEX (c_p) WHERE population >
500000000;
{
"query_block": {
"select_id": 1,
"cost_info": {
},
"table": {
"filtered": "33.33",
..
"attached_condition": "(`world`.`country`.`Population` >
500000000)"
..

Agenda
1. Introduction
3. B+trees
4. EXPLAIN
5. Optimizer Trace
6. Logical
Transformations
7. Cost Based
Optimization
9. Comparing Plans
11.Covering Indexes
12.Visual Explain
13.Transient Plans
14.Subqueries
15.CTEs and Views
16.Joins
17.Aggregation
19.Sorting
20.Partitioning
21.Query Rewrite
23.Profiling
24.JSON
25.Character Sets
84

Covering Indexes
• A special kind of composite index
• All information returned just by accessing the index
85

ALTER TABLE Country ADD INDEX c_p_n (Continent,Population,Name);
EXPLAIN FORMAT=JSON
SELECT Name FROM Country WHERE continent='Asia' and population > 5000000;
{
"query_block": {
"select_id": 1,
"cost_info": {
},
"table": {
"possible_keys": [
..
"c_p_n"
],
"key": "c_p_n",
"used_key_parts": [
"Continent",
"Population"
],
"key_length": "5",
..
"filtered": "100.00",
"using_index": true,
..
Cost is reduced by
53%
Using index means
"covering index"

Use cases
• Can be used as in this example
• Also beneficial in join conditions (join through covering index on
intermediate table)
• Useful in aggregate queries
87

Agenda
1. Introduction
3. B+trees
4. EXPLAIN
5. Optimizer Trace
6. Logical
Transformations
7. Cost Based
Optimization
9. Comparing Plans
11.Covering Indexes
12.Visual Explain
13.Transient Plans
14.Subqueries
15.CTEs and Views
16.Joins
17.Aggregation
19.Sorting
20.Partitioning
21.Query Rewrite
23.Profiling
24.JSON
25.Character Sets
88

Visual Explain
• For complex queries, it is useful to see visual representation
• Visualizations in this deck are produced by MySQL Workbench.
89

Agenda
1. Introduction
3. B+trees
4. EXPLAIN
5. Optimizer Trace
6. Logical
Transformations
7. Cost Based
Optimization
9. Comparing Plans
11.Covering Indexes
12.Visual Explain
13.Transient Plans
14.Subqueries
15.CTEs and Views
16.Joins
17.Aggregation
19.Sorting
20.Partitioning
21.Query Rewrite
23.Profiling
24.JSON
25.Character Sets
90

A quick recap:
• So far we’ve talked about 4 candidate indexes:
– p (population)
– c (continent)
– p_c (population, continent)
– c_p (continent, population)
• We’ve always used c=‘Asia’ and p > 5M
91

Cost Estimates
p>5M c=’Asia’ p>5M c=’Antarctica’ p>50M, c=’Asia’
p>50M
c=’Antarctica’
p>500M, c=’Asia’
p>500M
c=’Antarctica’
p 48.86 48.86 11.06 11.06 1.16 1.16
c 9.60 1.75 9.60 1.75 9.60 1.75
c_p 7.91 0.71 5.21 0.71 1.16 0.71
p_c 48.86 48.86 11.06 11.06 1.16 1.16
ALL 25.40 25.40 25.40 25.40 25.40 25.40

Cost Estimates

Actual Execution Time

Agenda
1. Introduction
3. B+trees
4. EXPLAIN
5. Optimizer Trace
6. Logical
Transformations
7. Cost Based
Optimization
9. Comparing Plans
11.Covering Indexes
12.Visual Explain
13.Transient Plans
14.Subqueries
15.CTEs and Views
16.Joins
17.Aggregation
19.Sorting
20.Partitioning
21.Query Rewrite
23.Profiling
24.JSON
25.Character Sets
95

Subquery (Scalar)
• Can optimize away the inner part first and then cache it.
• This avoids re-executing the inner part for-each-row
96
SELECT * FROM Country WHERE 
Code = (SELECT CountryCode FROM City WHERE
name=‘Toronto’);

EXPLAIN FORMAT=JSON
SELECT * FROM Country WHERE Code = (SELECT CountryCode FROM City WHERE name=‘Toronto’);
{
"query_block": {
"select_id": 1,
"cost_info": {
},
"table": {
"access_type": "const",
..
"key": "PRIMARY",
..
},
"optimized_away_subqueries": [
{
"dependent": false,
"cacheable": true,
"query_block": {
"select_id": 2,
"cost_info": {
},
"table": {
"table_name": "City",
..
(misleading
visualization)
First query + its
cost
Second query +
its cost

ALTER TABLE city ADD INDEX n (name);
EXPLAIN FORMAT=JSON
SELECT * FROM Country WHERE Code = (SELECT CountryCode FROM City WHERE name=‘Toronto’);
{
"query_block": {
"select_id": 1,
"cost_info": {
},
"table": {
..
"key": "PRIMARY",
..
},
"optimized_away_subqueries": [
{
"dependent": false,
"cacheable": true,
"query_block": {
"select_id": 2,
"cost_info": {
},
"table": {
"possible_keys": [
"n"
],
"key": “n",
..
First query + its
cost
Second query +
its cost
(misleading
visualization)

Subquery (IN list)
• When the result inner subquery returns unique results it can safely
be transformed to an inner join:
99
EXPLAIN FORMAT=JSON SELECT * FROM City WHERE CountryCode IN 
(SELECT Code FROM Country WHERE Continent = 'Asia');
show warnings;
/* select#1 */ select `world`.`city`.`ID` AS `ID`,`world`.`city`.`Name` AS
`Name`,`world`.`city`.`CountryCode` AS `CountryCode`,`world`.`city`.`District`
AS `District`,`world`.`city`.`Population` AS `Population` from
`world`.`country` join `world`.`city` where ((`world`.`city`.`CountryCode` =
`world`.`country`.`Code`) and (`world`.`country`.`Continent` = 'Asia'))

EXPLAIN FORMAT=JSON
SELECT * FROM City WHERE CountryCode IN 
(SELECT Code FROM Country WHERE Continent = 'Asia');
{
"query_block": {
"select_id": 1,
"cost_info": {
},
"nested_loop": [
{
"table": {
..
"key": "c",
..
..
"used_columns": [
"Code",
"Continent"
..
{
"table": {
"possible_keys": [
"CountryCode"
],
"key": "CountryCode",
..
"ref": [
"world.Country.Code"
],
..

SELECT * FROM Country WHERE Code IN
(SELECT CountryCode FROM CountryLanguage  
WHERE isOfficial=1);
Subquery (cont.)
• When non-unique the optimizer needs to pick a semi-join strategy
• Multiple options: FirstMatch, MaterializeLookup,
DuplicatesWeedout
101

ALTER TABLE CountryLanguage ADD INDEX i (isOfficial);
EXPLAIN FORMAT=JSON SELECT * FROM Country WHERE Code IN
(SELECT CountryCode FROM CountryLanguage WHERE isOfficial=1);
{
"query_block": {
"select_id": 1,
"cost_info": {
},
"nested_loop": [
{
"table": {
"possible_keys": [
"PRIMARY"
],
..
"filtered": "100.00",
..
"table": {
"table_name": "<subquery2>",
"access_type": "eq_ref",
"key": "<auto_key>",
"key_length": "3",
"ref": [
"world.Country.Code"
],
"materialized_from_subquery": {
"using_temporary_table": true,
"query_block": {
"table": {
"table_name": "CountryLanguage",
..
"key": "i",
..
..

Agenda
1. Introduction
3. B+trees
4. EXPLAIN
5. Optimizer Trace
6. Logical
Transformations
7. Cost Based
Optimization
9. Comparing Plans
11.Covering Indexes
12.Visual Explain
13.Transient Plans
14.Subqueries
15.CTEs and Views
16.Joins
17.Aggregation
19.Sorting
20.Partitioning
21.Query Rewrite
23.Profiling
24.JSON
25.Character Sets
103

Views
• A way of saving a SELECT statement as a table
• Allows for simplified queries
• Processed using one of two methods internally:
– Merge - transform the view to be combined with the query.
– Materialize - save the contents of the view in a temporary table,
then begin querying
104

ALTER TABLE country ADD INDEX c_n (continent, name);
CREATE VIEW vCountry_Asia AS SELECT * FROM Country WHERE Continent='Asia';
EXPLAIN FORMAT=JSON
SELECT * FROM vCountry_Asia WHERE Name='China';
{
"query_block": {
"select_id": 1,
"cost_info": {
},
"table": {
"possible_keys": [
..
"c_n"
],
"key": "c_n",
"used_key_parts": [
"Continent",
"Name"
],
"key_length": "53",
"ref": [
"const",
"const"
],
..
This is the base table
Predicates from the view
deﬁnition and query
combined

SHOW WARNINGS;
`world`.`Country`.`Code` AS `Code`,
`world`.`Country`.`Name` AS `Name`,
`world`.`Country`.`Continent` AS `Continent`,
`world`.`Country`.`Region` AS `Region`,
`world`.`Country`.`SurfaceArea` AS `SurfaceArea`,
`world`.`Country`.`IndepYear` AS `IndepYear`,
`world`.`Country`.`Population` AS `Population`,
`world`.`Country`.`LifeExpectancy` AS `LifeExpectancy`,
`world`.`Country`.`GNP` AS `GNP`,
`world`.`Country`.`GNPOld` AS `GNPOld`,
`world`.`Country`.`LocalName` AS `LocalName`,
`world`.`Country`.`GovernmentForm` AS `GovernmentForm`,
`world`.`Country`.`HeadOfState` AS `HeadOfState`,
`world`.`Country`.`Capital` AS `Capital`,
`world`.`Country`.`Code2` AS `Code2`
from `world`.`Country`
where
((`world`.`Country`.`Continent` = 'Asia')
and (`world`.`Country`.`Name` = 'China'))

CREATE VIEW vCountrys_Per_Continent AS
SELECT Continent, COUNT(*) as Count FROM Country
GROUP BY Continent;
EXPLAIN FORMAT=JSON
SELECT * FROM vCountrys_Per_Continent WHERE Continent='Asia';
{
"query_block": {
"select_id": 1,
"cost_info": {
},
"table": {
"table_name": "vCountrys_Per_Continent",
"possible_keys": [
"<auto_key0>"
],
"key": "<auto_key0>",
"used_key_parts": [
"Continent"
],
"key_length": "1",
"ref": [
"const"
],
..
"used_columns": [
"Continent",
"Count"
],
..
This is the view name
..
"dependent": false,
"cacheable": true,
"query_block": {
"select_id": 2,
"cost_info": {
},
This is only the cost of
accessing the materialized table
This step happens ﬁrst.

SHOW WARNINGS;
`vCountrys_Per_Continent`.`Continent` AS `Continent`,
`vCountrys_Per_Continent`.`Count` AS `Count`
from `world`.`vCountrys_Per_Continent`
where (`vCountrys_Per_Continent`.`Continent` = 'Asia')

WITH (CTE)
• A view for query-only duration
• Same optimizations available as views:
– Merge - transform the CTE to be combined with the query.
– Materialize - save the contents of the CTE in a temporary table,
then begin querying
109

# Identical Queries - CTE and VIEW
WITH vCountry_Asia AS (SELECT * FROM Country WHERE
Continent='Asia')
CREATE VIEW vCountry_Asia AS SELECT * FROM Country WHERE
Continent='Asia';

CTEs are new!
• May provide performance enhancements over legacy code using temporary
tables - which never merge.
• Derived tables may need to materialize more than once. A CTE does not! i.e.
111
SELECT * FROM my_table, (SELECT ... ) as t1 ...
UNION ALL
SELECT * FROM my_table, (SELECT ... ) as t1 ...

WITH RECURSIVE - new!
WITH RECURSIVE my_cte AS (
SELECT 1 AS n
UNION ALL
SELECT 1+n FROM my_cte WHERE n<10
)
SELECT * FROM my_cte;
+------+
| n |
+------+
| 1 |
| 2 |
..
| 9 |
| 10 |
+------+
112

{
"query_block": {
"select_id": 1,
"cost_info": {
},
"table": {
"table_name": "my_cte",
..
"used_columns": [
"n"
],
"dependent": false,
"cacheable": true,
"query_block": {
"union_result": {
"using_temporary_table": false,
..
..
{
"dependent": false,
"cacheable": true,
"query_block": {
"select_id": 3,
"recursive": true,
"cost_info": {
},
"table": {
"table_name": "my_cte",
..
"used_columns": [
"n"
],
"attached_condition":  
"(`my_cte`.`n` < 10)"
}
..
Requires a
temporary table for
intermediate results
Cost per iteration

Agenda
1. Introduction
3. B+trees
4. EXPLAIN
5. Optimizer Trace
6. Logical
Transformations
7. Cost Based
Optimization
9. Comparing Plans
11.Covering Indexes
12.Visual Explain
13.Transient Plans
14.Subqueries
15.CTEs and Views
16.Joins
17.Aggregation
19.Sorting
20.Partitioning
21.Query Rewrite
23.Profiling
24.JSON
25.Character Sets
114

SELECT
Country.Name as Country, City.Name as Capital,
Language
FROM
City
INNER JOIN Country ON Country.Capital=City.id
INNER JOIN CountryLanguage ON
CountryLanguage.CountryCode=Country.code
WHERE
Country.Continent='Asia' and
CountryLanguage.IsOfficial='T';

Join Strategy (Nested Loop Join)
1. Pick Driving Table (Country)
2. For each row in Country 
step through to City table
3. For each row in City table 
step through to 
CountryLanguage table
4. Repeat
116

Join efficiency
• Important to eliminate work before accessing other tables (WHERE
clause should have lots of predicates that filter driving table)
• Indexes are required on the columns that connect between driving
table, and subsequent tables:
117
ON Country.Capital=City.id

INNER JOIN vs LEFT JOIN
• LEFT JOIN semantically says “right row is optional”.
– Forces JOIN order to be left side first.
– Reduces possible ways to join tables
118

Join Order Hints
• One of the most frequent types of hints to apply
• New join order hints in 8.0:
– JOIN_FIXED_ORDER
– JOIN_ORDER
– JOIN_PREFIX
– JOIN_SUFFIX
119

Agenda
1. Introduction
3. B+trees
4. EXPLAIN
5. Optimizer Trace
6. Logical
Transformations
7. Cost Based
Optimization
9. Comparing Plans
11.Covering Indexes
12.Visual Explain
13.Transient Plans
14.Subqueries
15.CTEs and Views
16.Joins
17.Aggregation
19.Sorting
20.Partitioning
21.Query Rewrite
23.Profiling
24.JSON
25.Character Sets
120

Group By - Loose Index Scan
• Scan the index from start to finish without buffering. Results are
pipelined to client:
121
SELECT count(*) as c, continent FROM Country  
GROUP BY continent;

Group By - Index Filtering Rows
• Use the index to eliminate as much work as possible
• Store rows in intermediate temporary file and then sort
122

Group By - Index Filtering + Guaranteed Order
• Use the index to eliminate as much work as possible
• The index also maintains order
123

UNION
• Requires an intermediate temporary table to weed out duplicate
rows
• The optimizer does not really have any optimizations for UNION
(such as a merge with views)
124

EXPLAIN FORMAT=JSON
SELECT * FROM City WHERE CountryCode = 'CAN'
UNION
SELECT * FROM City WHERE CountryCode = 'USA'
{
"union_result": {
"table_name": "<union1,2>",
"query_specifications": [
{
"dependent": false,
"cacheable": true,
"query_block": {
"select_id": 1,
"cost_info": {
},
"table": {
..
..
Temporary table to
de-duplicate
{
"dependent": false,
"cacheable": true,
"query_block": {
"select_id": 2,
"cost_info": {
},
"table": {
"possible_keys": [
"CountryCode"
],
"used_key_parts": [
"CountryCode"
],
"key_length": "3",
"ref": [
"const"
],
..

UNION ALL
• Results may contain duplicate rows
• Does not require an intermediate temporary table in simple use
cases. i.e. no result ordering.
• Otherwise similar to UNION
126

EXPLAIN FORMAT=JSON
SELECT * FROM City WHERE CountryCode = 'CAN'
UNION ALL
SELECT * FROM City WHERE CountryCode = 'USA'
{
"query_block": {
"union_result": {
"using_temporary_table": false,
"query_specifications": [
{
"dependent": false,
"cacheable": true,
"query_block": {
"select_id": 1,
"cost_info": {
},
"table": {
..
..
{
"dependent": false,
"cacheable": true,
"query_block": {
"select_id": 2,
"cost_info": {
},
"table": {
"possible_keys": [
"CountryCode"
],
"used_key_parts": [
"CountryCode"
],
"key_length": "3",
"ref": [
"const"
],
..
No temporary table

Agenda
1. Introduction
3. B+trees
4. EXPLAIN
5. Optimizer Trace
6. Logical
Transformations
7. Cost Based
Optimization
9. Comparing Plans
11.Covering Indexes
12.Visual Explain
13.Transient Plans
14.Subqueries
15.CTEs and Views
16.Joins
17.Aggregation
19.Sorting
20.Partitioning
21.Query Rewrite
23.Profiling
24.JSON
25.Character Sets
128

Descending Indexes
• B+tree indexes are ordered
• In 8.0 you can specify the order
• Use cases:
– Faster to scan in order
– Can’t change direction in a composite index
129

EXPLAIN FORMAT=JSON
SELECT * FROM Country WHERE continent='Asia' AND population > 5000000
ORDER BY population DESC;
{
"query_block": {
"select_id": 1,
"cost_info": {
},
"ordering_operation": {
"using_filesort": false,
"table": {
..
"key": "c_p",
..
"backward_index_scan": true,
..
Still uses the index, but
about 15% slower

EXPLAIN FORMAT=JSON
SELECT * FROM Country WHERE continent IN ('Asia', 'Oceania') AND population > 5000000
ORDER BY continent ASC, population DESC
{
"query_block": {
"select_id": 1,
"cost_info": {
},
"using_filesort": true,
"cost_info": {
"sort_cost": "33.00"
},
"table": {
"key": "c_p",
..
"filtered": "100.00",
..
Must sort values of
population in reverse

ALTER TABLE Country DROP INDEX c_p, DROP INDEX c_p_n,
ADD INDEX c_p_desc (continent ASC, population DESC);
EXPLAIN FORMAT=JSON
SELECT * FROM Country WHERE continent IN ('Asia', 'Oceania') AND population > 5000000
ORDER BY continent ASC, population DESC;
{
"query_block": {
"select_id": 1,
"cost_info": {
},
"table": {
..
"key": "c_p_desc",
"used_key_parts": [
"Continent",
"Population"
],
"key_length": "5",
..
TIP: The optimizer does not consider sort cost in
evaluating plans. You may need to FORCE INDEX or
DROP similar ascending indexes to use it.

Agenda
1. Introduction
3. B+trees
4. EXPLAIN
5. Optimizer Trace
6. Logical
Transformations
7. Cost Based
Optimization
9. Comparing Plans
11.Covering Indexes
12.Visual Explain
13.Transient Plans
14.Subqueries
15.CTEs and Views
16.Joins
17.Aggregation
19.Sorting
20.Partitioning
21.Query Rewrite
23.Profiling
24.JSON
25.Character Sets
133

How is ORDER BY optimized?
1. Via an Index
2. Top N Buffer (“priority queue”)
3. Using temporary files
134

Via an Index
• B+tree indexes are ordered
• Some ORDER BY queries do not require sorting at all
135
EXPLAIN FORMAT=JSON
SELECT * FROM Country WHERE continent='Asia' ORDER BY population;
{
"query_block": {
"select_id": 1,
"cost_info": {
},
..
"key": "c_p",
The order is provided by
c_p

Via a Priority Queue
• Special ORDER BY + small limit optimization
• Keeps top N records in an in memory buffer
• Usage is NOT shown in EXPLAIN
136
SELECT * FROM Country IGNORE INDEX (p, p_c)
ORDER BY population LIMIT 10;

"select#": 1,
"steps": [
{
"filesort_information": [
{
"direction": "asc",
"table": "`country` IGNORE INDEX (`p_c`) IGNORE INDEX (`p`)",
"field": "Population"
}
],
"filesort_priority_queue_optimization": {
"limit": 10,
"chosen": true
},
"filesort_execution": [
],
"filesort_summary": {
"memory_available": 262144,
"key_size": 4,
"row_size": 272,
"max_rows_per_buffer": 11,
"num_rows_estimate": 587,
"num_rows_found": 11,
"num_examined_rows": 239,
"num_tmp_files": 0,
"sort_buffer_size": 3080,
"sort_algorithm": "std::sort",
"unpacked_addon_fields": "using_priority_queue",
"sort_mode": "<fixed_sort_key, additional_fields>”
..
OPTIMIZER TRACE
showing Priority Queue for
sort

Using Temporary Files
• Either “Alternative Sort Algorithm” (no blobs present) or “Original
Sort Algorithm”
138
SELECT * FROM Country IGNORE INDEX (p, p_c)
ORDER BY population;

"select#": 1,
"steps": [
{
"filesort_information": [
{
"direction": "asc",
"table": "`country` IGNORE INDEX (`p_c`) IGNORE INDEX (`p`)",
"field": "Population"
}
],
"filesort_priority_queue_optimization": {
"usable": false,
"cause": "not applicable (no LIMIT)"
},
"filesort_execution": [
],
"filesort_summary": {
"memory_available": 262144,
"key_size": 4,
"row_size": 274,
"max_rows_per_buffer": 587,
"num_rows_estimate": 587,
"num_rows_found": 239,
"num_examined_rows": 239,
"num_tmp_files": 0,
"sort_buffer_size": 165536,
"sort_algorithm": "std::stable_sort",
"sort_mode": "<fixed_sort_key, packed_additional_fields>"
..
Not Using Priority Sort

Agenda
1. Introduction
3. B+trees
4. EXPLAIN
5. Optimizer Trace
6. Logical
Transformations
7. Cost Based
Optimization
9. Comparing Plans
11.Covering Indexes
12.Visual Explain
13.Transient Plans
14.Subqueries
15.CTEs and Views
16.Joins
17.Aggregation
19.Sorting
20.Partitioning
21.Query Rewrite
23.Profiling
24.JSON
25.Character Sets
140

Partitioning
• Split a table physically into smaller tables
• At the user-level make it still appear as one table
141

Use Cases
• Can be a better fit low cardinality columns than indexing
• Useful for time series data with retention scheme
• i.e. drop data older than 3 months
• Data where queries always have some locality
• i.e. store_id, region
142

Partition Pruning
• Optimizer looks at query and identifies which partitions need to be
accessed
143
ALTER TABLE CountryLanguage MODIFY IsOfficial CHAR(1) NOT NULL DEFAULT 'F', DROP
PRIMARY KEY, ADD PRIMARY KEY(CountryCode, Language, IsOfficial); 
ALTER TABLE CountryLanguage PARTITION BY LIST COLUMNS (IsOfficial) (
PARTITION pUnofficial VALUES IN ('F'),
PARTITION pOfficial VALUES IN ('T')
);

EXPLAIN FORMAT=JSON
SELECT * FROM CountryLanguage WHERE isOfficial='T' AND
CountryCode='CAN';
{
"query_block": {
"select_id": 1,
"cost_info": {
},
"table": {
"table_name": "CountryLanguage",
"partitions": [
"pOfficial"
],
..
"key": "PRIMARY",
..
Only accesses one
partition

Explicit Partition Selection
• Also possible to “target” a partition
• Consider this similar to query hints
145
SELECT * FROM CountryLanguage PARTITION (pOfficial)
WHERE CountryCode='CAN';

Agenda
1. Introduction
3. B+trees
4. EXPLAIN
5. Optimizer Trace
6. Logical
Transformations
7. Cost Based
Optimization
9. Comparing Plans
11.Covering Indexes
12.Visual Explain
13.Transient Plans
14.Subqueries
15.CTEs and Views
16.Joins
17.Aggregation
19.Sorting
20.Partitioning
21.Query Rewrite
23.Profiling
24.JSON
25.Character Sets
146

Query Rewrite
• MySQL allows you to change queries before they are executed
• Insert a hint, or remove a join that is not required
147
mysql -u root -p < install_rewriter.sql
INSERT INTO query_rewrite.rewrite_rules(pattern_database, pattern,
replacement) VALUES (
"world",
"SELECT * FROM Country WHERE population > ? AND continent=?",
"SELECT * FROM Country WHERE population > ? AND continent=? LIMIT 1"
);
CALL query_rewrite.flush_rewrite_rules();

SELECT * FROM Country WHERE population > 5000000 AND
continent='Asia';
SHOW WARNINGS;
*********************** 1. row ***********************
Level: Note
Code: 1105
Message: Query 'SELECT * FROM Country WHERE population >
5000000 AND continent='Asia'' rewritten to 'SELECT *
FROM Country WHERE population > 5000000 AND
continent='Asia' LIMIT 1' by a query rewrite plugin

SELECT * FROM query_rewrite.rewrite_rulesG
********************** 1. row **********************
id: 1
pattern: SELECT * FROM Country WHERE
population > ? AND continent=?
pattern_database: world
replacement: SELECT * FROM Country WHERE
population > ? AND continent=? LIMIT 1
enabled: YES
message: NULL
pattern_digest: 88876bbb502cef6efddcc661cce77deb
normalized_pattern: select `*` from `world`.`country`
where ((`population` > ?) and (`continent` = ?))

Agenda
1. Introduction
3. B+trees
4. EXPLAIN
5. Optimizer Trace
6. Logical
Transformations
7. Cost Based
Optimization
9. Comparing Plans
11.Covering Indexes
12.Visual Explain
13.Transient Plans
14.Subqueries
15.CTEs and Views
16.Joins
17.Aggregation
19.Sorting
20.Partitioning
21.Query Rewrite
23.Profiling
24.JSON
25.Character Sets
150

Changing Indexes is a Destructive Operation
• Removing an index can make some queries much slower
• Adding can cause some existing query plans to change
• Old-style hints will generate errors if indexes are removed
151

Invisible Indexes, the “Recycle Bin”
• Hide the indexes from the optimizer
• Will no longer be considered as part of query execution plans
• Still kept up to date and are maintained by insert/update/delete
statements
152

Invisible Indexes: Soft Delete
ALTER TABLE Country ALTER INDEX c INVISIBLE;
SELECT * FROM information_schema.statistics WHERE is_visible='NO';
*************************** 1. row ***************************
TABLE_CATALOG: def
TABLE_SCHEMA: world
TABLE_NAME: Country
NON_UNIQUE: 1
INDEX_SCHEMA: world
INDEX_NAME: c
SEQ_IN_INDEX: 1
COLUMN_NAME: Continent
COLLATION: A
CARDINALITY: 7
SUB_PART: NULL
PACKED: NULL
NULLABLE:
INDEX_TYPE: BTREE
COMMENT: disabled
INDEX_COMMENT:
IS_VISIBLE: NO
153

Invisible Indexes: Staged Rollout
ALTER TABLE Country ADD INDEX c (Continent)
INVISIBLE;
# after some time
ALTER TABLE Country ALTER INDEX c VISIBLE;
154

Do indexes hurt reads or writes?
• They can have some impact on both:
– On writes, indexes need to space, and to be maintained
– On reads, lets use an example…
156

Indexes Hurting Reads
CREATE TABLE t1 (
id INT NOT NULL primary key auto_increment,
a VARCHAR(255) NOT NULL,
b VARCHAR(255) NOT NULL,
c TEXT,
d TEXT,
INDEX a (a),
INDEX ab (a,b));
# Sample Query
SELECT * FROM t1 WHERE a = 'abc' AND b = 'bcd';
157
Both indexes are candidates.
Both will be examined.

A use case for invisible indexes!
CREATE TABLE t1 (
id INT NOT NULL primary key auto_increment,
a VARCHAR(255) NOT NULL,
b VARCHAR(255) NOT NULL,
c TEXT,
d TEXT,
INDEX a (a),
INDEX ab (a,b));
# Consider:
SELECT count(*) FROM t1 FORCE INDEX (a) 
WHERE a='1234' AND id=1234;
158
Index (a) is made redundant by
(a,b). Can we drop it?

No, due to clustered Index!
FORCE INDEX (a) WHERE a=‘1234' AND id=1234;
{
"query_block": {
"select_id": 1,
"cost_info": {
},
"table": {
"table_name": "t1",
"possible_keys": [
"a"
],
"key": "a",
"used_key_parts": [
"a",
"id"
],
..
FORCE INDEX (ab) WHERE a='1234' AND id=1234;
{
"query_block": {
"select_id": 1,
"cost_info": {
},
"table": {
"table_name": "t1",
"possible_keys": [
"ab"
],
"key": "ab",
"used_key_parts": [
"a"
],
..

Agenda
1. Introduction
3. B+trees
4. EXPLAIN
5. Optimizer Trace
6. Logical
Transformations
7. Cost Based
Optimization
9. Comparing Plans
11.Covering Indexes
12.Visual Explain
13.Transient Plans
14.Subqueries
15.CTEs and Views
16.Joins
17.Aggregation
19.Sorting
20.Partitioning
21.Query Rewrite
23.Profiling
24.JSON
25.Character Sets
160

Profiling
• Optimizer only shows estimates from pre-execution view
• Can be useful to know actual time spent
• Support for profiling is only very basic
161
wget http://www.tocker.ca/files/ps-show-profiles.sql
mysql -u root -p < ps-show-profiles.sql

CALL sys.enable_profiling();
CALL sys.show_profiles;
*************************** 1. row ***************************
Event_ID: 22
Duration: 495.02 us
Query: SELECT * FROM Country WHERE co ... Asia' and population > 5000000
CALL sys.show_profile_for_event_id(22);
+----------------------+-----------+
| Status | Duration |
+----------------------+-----------+
| starting | 64.82 us |
| checking permissions | 4.10 us |
| Opening tables | 11.87 us |
| init | 29.74 us |
| System lock | 5.63 us |
| optimizing | 8.74 us |
| statistics | 139.38 us |
| preparing | 11.94 us |
| executing | 348.00 ns |
| Sending data | 192.59 us |
| end | 1.17 us |
| query end | 4.60 us |
| closing tables | 4.07 us |
| freeing items | 13.60 us |
| cleaning up | 734.00 ns |
+----------------------+-----------+

SELECT * FROM Country WHERE Continent='Antarctica' and SLEEP(5);
CALL sys.show_profiles();
CALL sys.show_profile_for_event_id(<event_id>);
+----------------------+-----------+
+----------------------+-----------+
| init | 45.75 us |
| User sleep | 5.00 s |
| end | 2.05 us |
| cleaning up | 1.20 us |
+----------------------+-----------+
Sleeps for each row after
index used on (c)

SELECT region, count(*) as c FROM Country GROUP BY region;
CALL sys.show_profiles();
CALL sys.show_profile_for_event_id(<event_id>);
+----------------------+-----------+
+----------------------+-----------+
| init | 25.81 us |
| Creating tmp table | 35.57 us |
| Sorting result | 2.38 us |
| Creating sort index | 49.45 us |
| end | 1.71 us |
| removing tmp table | 4.71 us |
| cleaning up | 1.00 us |
+----------------------+-----------+

SELECT * FROM performance_schema.events_statements_history_long
WHERE event_id=<event_id>G
*********************** 1. row ***********************
THREAD_ID: 3062
EVENT_ID: 1566
END_EVENT_ID: 1585
EVENT_NAME: statement/sql/select
SOURCE: init_net_server_extension.cc:80
TIMER_START: 588883869566277000
TIMER_END: 588883870317683000
TIMER_WAIT: 751406000
LOCK_TIME: 132000000
SQL_TEXT: SELECT region,  
count(*) as c FROM Country GROUP BY region
DIGEST: d3a04b346fe48da4f1f5c2e06628a245
DIGEST_TEXT: SELECT `region` , 
COUNT ( * ) AS `c` FROM `Country`  
GROUP BY `region`
CURRENT_SCHEMA: world
OBJECT_TYPE: NULL
OBJECT_SCHEMA: NULL
OBJECT_NAME: NULL
OBJECT_INSTANCE_BEGIN: NULL
MYSQL_ERRNO: 0
RETURNED_SQLSTATE: NULL
MESSAGE_TEXT: NULL
ERRORS: 0
WARNINGS: 0
..
..
ROWS_AFFECTED: 0
ROWS_SENT: 25
ROWS_EXAMINED: 289
CREATED_TMP_DISK_TABLES: 0
CREATED_TMP_TABLES: 1
SELECT_FULL_JOIN: 0
SELECT_FULL_RANGE_JOIN: 0
SELECT_RANGE: 0
SELECT_RANGE_CHECK: 0
SELECT_SCAN: 1
SORT_MERGE_PASSES: 0
SORT_RANGE: 0
SORT_ROWS: 25
SORT_SCAN: 1
NO_INDEX_USED: 1
NO_GOOD_INDEX_USED: 0
NESTING_EVENT_ID: NULL
NESTING_EVENT_TYPE: NULL
NESTING_EVENT_LEVEL: 0
For non-aggregate queries rows sent
vs. rows examined helps indicate index
effectiveness.

Agenda
1. Introduction
3. B+trees
4. EXPLAIN
5. Optimizer Trace
6. Logical
Transformations
7. Cost Based
Optimization
9. Comparing Plans
11.Covering Indexes
12.Visual Explain
13.Transient Plans
14.Subqueries
15.CTEs and Views
16.Joins
17.Aggregation
19.Sorting
20.Partitioning
21.Query Rewrite
23.Profiling
24.JSON
25.Character Sets
166

JSON
• Optimizer has native support for JSON with indexes on generated
columns used for matching JSON path expressions
167
CREATE TABLE CountryJson (Code char(3) not null primary key, doc JSON NOT NULL);
INSERT INTO CountryJSON SELECT code,
JSON_OBJECT(
'Name', Name,
'Continent', Continent,
..
'HeadOfState',HeadOfState,
'Capital', Capital,
'Code2', Code2
) FROM Country;

EXPLAIN FORMAT=JSON 
SELECT * FROM CountryJSON where doc->>"$.Name" = ‘Canada';
{
"query_block": {
"select_id": 1,
"cost_info": {
},
"table": {
"table_name": "CountryJSON",
"filtered": "100.00",
"cost_info": {
"read_cost": "1.00",
"eval_cost": "47.80",
},
..

ALTER TABLE CountryJSON ADD Name char(52) AS (doc->>"$.Name"),
ADD INDEX n (Name);
EXPLAIN FORMAT=JSON
SELECT * FROM CountryJSON where doc->>"$.Name" = ‘Canada';
{
"query_block": {
"select_id": 1,
"cost_info": {
},
"table": {
"table_name": "CountryJSON",
..
"key": "n",
..
"key_length": "53",
"ref": [
"const"
],
..
Key from virtual column
Matches expression from
indexed virtual column

JSON Comparator
• JSON types compare to MySQL types
170
SELECT CountryJSON.* FROM CountryJSON  
INNER JOIN Country ON CountryJSON.doc->>"$.Name" = Country.Name WHERE
Country.Name=‘Canada';
********************** 1. row **********************
Code: CAN
doc: {"GNP": 598862, "Name": "Canada", "Code2": "CA", "GNPOld": 625626, "Region":
"North America", "Capital": 1822, "Continent": "North America", "IndepYear": 1867,
"LocalName": "Canada", "Population": 31147000, "HeadOfState": "Elisabeth II",
"SurfaceArea": 9970610, "GovernmentForm": "Constitutional Monarchy, Federation",
"LifeExpectancy": 79.4000015258789}
Name: Canada

Agenda
1. Introduction
3. B+trees
4. EXPLAIN
5. Optimizer Trace
6. Logical
Transformations
7. Cost Based
Optimization
9. Comparing Plans
11.Covering Indexes
12.Visual Explain
13.Transient Plans
14.Subqueries
15.CTEs and Views
16.Joins
17.Aggregation
19.Sorting
20.Partitioning
21.Query Rewrite
23.Profiling
24.JSON
25.Character Sets
171

Character Sets
• The default character set in MySQL 8.0 is utf8mb4
• Utf8mb4 is variable length (1-4 bytes)
• InnoDB will always store as variable size for both CHAR and
VARCHAR
• Some buffers inside MySQL may require the fixed length (4 bytes)
172

Character Sets (cont.)
• CHAR(n) or VARCHAR(n) refers to n characters - x4 for maximum length
• EXPLAIN will always show the maximum length
• Mysqldump will preserve character set
173
 
ALTER TABLE City DROP FOREIGN KEY city_ibfk_1;
ALTER TABLE CountryLanguage DROP FOREIGN KEY
countryLanguage_ibfk_1;
ALTER TABLE Country CONVERT TO CHARACTER SET
utf8mb4;

{
"query_block": {
"select_id": 1,
"cost_info": {
},
"table": {
"possible_keys": [
"n"
],
"key": "n",
"used_key_parts": [
"Name"
],
"key_length": "52",
..
"filtered": "100.00",
"cost_info": {
"data_read_per_join": "264"
},
..
{
"query_block": {
"select_id": 1,
"cost_info": {
},
"table": {
"possible_keys": [
"n"
],
"key": "n",
"used_key_parts": [
"Name"
],
"key_length": "208",
..
"filtered": "100.00",
"cost_info": {
"data_read_per_join": "968"
},
..
Key length as latin1 Key length as utf8

Conclusion
• Thank you for coming!
• This presentation is available as a website: 
www.unofficialmysqlguide.com
175

MySQL 8.0 Optimizer Guide

Recommended

More Related Content

What's hot (20)

Similar to MySQL 8.0 Optimizer Guide (20)

More from Morgan Tocker (20)

Recently uploaded (20)

MySQL 8.0 Optimizer Guide