What this lesson is about...

Up to this point you have learned a lot about working with data using SQL. This lesson will focus on
three topics:

1. Subqueries
2. Table Expressions
3. Persistent Derived Tables

Both subqueries and table expressions are methods for being able to write a query that creates a
table, and then write a query that interacts with this newly created table. Sometimes the question you
are trying to answer doesn't have an answer when working directly with existing tables in database.
However, if we were able to create new tables from the existing tables, we know we could query
these new tables to answer our question. This is where the queries of this lesson come to the rescue.

If you can't yet think of a question that might require such a query, don't worry because you are about
to see a whole bunch of them!

Whenever we need to use existing tables to create a new table that we then want to query again, this
is an indication that we will need to use some sort of subquery. In the next couple of concepts, we
will walk through an example together. Then you will get some practice tackling some additional
problems on your own.

Your First Subquery

The first time you write a subquery it might seem really complex. Let's try breaking
it down into its different parts.

If you get stuck look again at the video above. We want to find the average number
of events for each day for each channel. The first table will provide us the number
of events for each day and channel, and then we will need to average these values
together using a second query.

You try solving this yourself

Solutions to Your First Subquery

1. First, we needed to group by the day and channel. Then ordering by the number of events
(the third column) gave us a quick way to answer the first question.
2. SELECT DATE_TRUNC('day',occurred_at) AS day,
3. channel, COUNT(*) as events
4. FROM web_events
5. GROUP BY 1,2
6. ORDER BY 3 DESC;
7. Here you can see that to get the entire table in question 1 back, we included an  *  in
our SELECT statement. You will need to be sure to alias your table.
8. SELECT *
9. FROM (SELECT DATE_TRUNC('day',occurred_at) AS day,
10. channel, COUNT(*) as events
11. FROM web_events
12. GROUP BY 1,2
13. ORDER BY 3 DESC) sub;
14. Finally, here we are able to get a table that shows the average number of events a day for
each channel.
15. SELECT channel, AVG(events) AS average_events
16. FROM (SELECT DATE_TRUNC('day',occurred_at) AS day,
17. channel, COUNT(*) as events
18. FROM web_events
19. GROUP BY 1,2) sub
20. GROUP BY channel
ORDER BY 2 DESC;

Subquery Formatting
When writing Subqueries, it is easy for your query to look incredibly complex. In order to assist
your reader, which is often just yourself at a future date, formatting SQL will help with
understanding your code.
The important thing to remember when using subqueries is to provide some way for the reader to
easily determine which parts of the query will be executed together. Most people do this by indenting
the subquery in some way - you saw this with the solution blocks in the previous concept.

The examples in this class are indented quite far—all the way to the parentheses. This isn’t practical
if you nest many subqueries, but in general, be thinking about how to write your queries in a readable
way. Examples of the same query written multiple different ways is provided below. You will see
that some are much easier to read than others.

Badly Formatted Queries

Though these poorly formatted examples will execute the same way as the well formatted examples,
they just aren't very friendly for understanding what is happening!

Here is the first, where it is impossible to decipher what is going on:

SELECT * FROM (SELECT DATE_TRUNC('day',occurred_at) AS day, channel, COUNT(*)

as events FROM web_events GROUP BY 1,2 ORDER BY 3 DESC) sub;
This second version, which includes some helpful line breaks, is easier to read than that previous
version, but it is still not as easy to read as the queries in the Well Formatted Query section.
SELECT *
FROM (
SELECT DATE_TRUNC('day',occurred_at) AS day,
channel, COUNT(*) as events
FROM web_events
GROUP BY 1,2
ORDER BY 3 DESC) sub;

Well Formatted Query

Now for a well formatted example, you can see the table we are pulling from much easier than in the
previous queries.

SELECT *
FROM (SELECT DATE_TRUNC('day',occurred_at) AS day,
channel, COUNT(*) as events
FROM web_events
GROUP BY 1,2
ORDER BY 3 DESC) sub;
Additionally, if we have a GROUP BY, ORDER BY, WHERE, HAVING, or any other statement
following our subquery, we would then indent it at the same level as our outer query.
The query below is similar to the above, but it is applying additional statements to the outer query, so
you can see there are GROUP BY and ORDER BY statements used on the output are not tabbed.
The inner query GROUP BY and ORDER BY statements are indented to match the inner table.
SELECT *
FROM (SELECT DATE_TRUNC('day',occurred_at) AS day,
channel, COUNT(*) as events
FROM web_events
GROUP BY 1,2
ORDER BY 3 DESC) sub
GROUP BY day, channel, events
ORDER BY 2 DESC;
These final two queries are so much easier to read!

Subqueries Part II
In the first subquery you wrote, you created a table that you could then query again in
the FROM statement. However, if you are only returning a single value, you might use that value in
a logical statement like WHERE, HAVING, or even SELECT - the value could be nested within
a CASE statement.
On the next concept, we will work through this example, and then you will get some practice on
answering some questions on your own.

Expert Tip
Note that you should not include an alias when you write a subquery in a conditional statement. This
is because the subquery is treated as an individual value (or set of values in the IN case) rather than
as a table.
 Also, notice the query here compared a single value. If we returned an entire column IN would need
to be used to perform a logical argument. If we are returning an entire table, then we must use
an ALIAS for the table, and perform additional logic on the entire table.

Queries Needed to Find the Solutions to the Previous Quiz

1. Here is the necessary quiz to pull the first month/year combo from the orders table.
2. SELECT DATE_TRUNC('month', MIN(occurred_at))
3. FROM orders;
4. Then to pull the average for each, we could do this all in one query, but for readability, I
provided two queries below to perform each separately.
SELECT AVG(standard_qty) avg_std, AVG(gloss_qty) avg_gls, AVG(poster_qty)
avg_pst
FROM orders
WHERE DATE_TRUNC('month', occurred_at) =
(SELECT DATE_TRUNC('month', MIN(occurred_at)) FROM orders);

SELECT SUM(total_amt_usd)
FROM orders
WHERE DATE_TRUNC('month', occurred_at) =
(SELECT DATE_TRUNC('month', MIN(occurred_at)) FROM orders);

Notice the tables referenced in the video below are those that were shown on the previous
page. Sorry they are not shown at the beginning of the video. Skipping to 1:30 will move
past the information on the previous page.

More Subqueries Quizzes

Above is the ERD for the database again - it might come in handy as you tackle the quizzes below.
You should write your solution as a subquery or subqueries, not by finding one solution and copying
the output. The importance of this is that it allows your query to be dynamic in answering the
question - even if the data changes, you still arrive at the right answer.

1. Provide the name of the sales_rep in each region with the largest amount

of total_amt_usd sales.

2. For the region with the largest (sum) of sales total_amt_usd, how many total (count) orders
were placed?

3. How many accounts had more total purchases than the account name which has bought the
most standard_qty paper throughout their lifetime as a customer?

4. For the customer that spent the most (in total over their lifetime as a
customer) total_amt_usd, how many web_events did they have for each channel?
5. What is the lifetime average amount spent in terms of total_amt_usd for the top 10 total
spending accounts?

6. What is the lifetime average amount spent in terms of total_amt_usd, including only the
companies that spent more per order, on average, than the average of all orders.

Solution: Subquery Mania

1. Provide the name of the sales_rep in each region with the largest amount
of total_amt_usd sales.

First, I wanted to find the total_amt_usd totals associated with each sales rep, and I also wanted
the region in which they were located. The query below provided this information.
2. SELECT s.name rep_name, r.name region_name, SUM(o.total_amt_usd) total_amt
3. FROM sales_reps s
4. JOIN accounts a
5. ON a.sales_rep_id = s.id
6. JOIN orders o
7. ON o.account_id = a.id
8. JOIN region r
9. ON r.id = s.region_id
10. GROUP BY 1,2
11. ORDER BY 3 DESC;
Next, I pulled the max for each region, and then we can use this to pull those rows in our final
result.
SELECT region_name, MAX(total_amt) total_amt
FROM(SELECT s.name rep_name, r.name region_name, SUM(o.total_amt_usd)
total_amt
FROM sales_reps s
JOIN accounts a
ON a.sales_rep_id = s.id
JOIN orders o
ON o.account_id = a.id
JOIN region r
ON r.id = s.region_id
GROUP BY 1, 2) t1
GROUP BY 1;
Essentially, this is a JOIN of these two tables, where the region and amount match.
SELECT t3.rep_name, t3.region_name, t3.total_amt
FROM(SELECT region_name, MAX(total_amt) total_amt
FROM(SELECT s.name rep_name, r.name region_name, SUM(o.total_amt_usd)
total_amt
FROM sales_reps s
JOIN accounts a
ON a.sales_rep_id = s.id
JOIN orders o
ON o.account_id = a.id
JOIN region r
ON r.id = s.region_id
 GROUP BY 1, 2) t1
GROUP BY 1) t2
JOIN (SELECT s.name rep_name, r.name region_name, SUM(o.total_amt_usd) total_amt
FROM sales_reps s
JOIN accounts a
ON a.sales_rep_id = s.id
JOIN orders o
ON o.account_id = a.id
JOIN region r
ON r.id = s.region_id
GROUP BY 1,2
ORDER BY 3 DESC) t3
ON t3.region_name = t2.region_name AND t3.total_amt = t2.total_amt;
12. For the region with the largest sales total_amt_usd, how many total orders were placed?

The first query I wrote was to pull the total_amt_usd for each region.

13. SELECT r.name region_name, SUM(o.total_amt_usd) total_amt
14. FROM sales_reps s
15. JOIN accounts a
16. ON a.sales_rep_id = s.id
17. JOIN orders o
18. ON o.account_id = a.id
19. JOIN region r
20. ON r.id = s.region_id
21. GROUP BY r.name;
Then we just want the region with the max amount from this table. There are two ways I
considered getting this amount. One was to pull the max using a subquery. Another way is to
order descending and just pull the top value.
SELECT MAX(total_amt)
FROM (SELECT r.name region_name, SUM(o.total_amt_usd) total_amt
FROM sales_reps s
JOIN accounts a
ON a.sales_rep_id = s.id
JOIN orders o
ON o.account_id = a.id
JOIN region r
ON r.id = s.region_id
GROUP BY r.name) sub;
Finally, we want to pull the total orders for the region with this amount:
SELECT r.name, COUNT(o.total) total_orders
FROM sales_reps s
JOIN accounts a
ON a.sales_rep_id = s.id
JOIN orders o
ON o.account_id = a.id
JOIN region r
ON r.id = s.region_id
GROUP BY r.name
HAVING SUM(o.total_amt_usd) = (
SELECT MAX(total_amt)
FROM (SELECT r.name region_name, SUM(o.total_amt_usd) total_amt
FROM sales_reps s
 JOIN accounts a
ON a.sales_rep_id = s.id
JOIN orders o
ON o.account_id = a.id
JOIN region r
ON r.id = s.region_id
GROUP BY r.name) sub);
This provides the Northeast with 2357 orders.

22. How many accounts had more total purchases than the account name which has bought
the most standard_qty paper throughout their lifetime as a customer?

First, we want to find the account that had the most standard_qty paper. The query here pulls
that account, as well as the total amount:
23. SELECT a.name account_name, SUM(o.standard_qty) total_std, SUM(o.total)
total
24. FROM accounts a
25. JOIN orders o
26. ON o.account_id = a.id
27. GROUP BY 1
28. ORDER BY 2 DESC
29. LIMIT 1;
Now, I want to use this to pull all the accounts with more total sales:
SELECT a.name
FROM orders o
JOIN accounts a
ON a.id = o.account_id
GROUP BY 1
HAVING SUM(o.total) > (SELECT total
FROM (SELECT a.name act_name, SUM(o.standard_qty) tot_std,
SUM(o.total) total
FROM accounts a
JOIN orders o
ON o.account_id = a.id
GROUP BY 1
ORDER BY 2 DESC
LIMIT 1) sub);
This is now a list of all the accounts with more total orders. We can get the count with just
another simple subquery.

SELECT COUNT(*)
FROM (SELECT a.name
FROM orders o
JOIN accounts a
ON a.id = o.account_id
GROUP BY 1
HAVING SUM(o.total) > (SELECT total
FROM (SELECT a.name act_name, SUM(o.standard_qty) tot_std,
SUM(o.total) total
FROM accounts a
JOIN orders o
 ON o.account_id = a.id
GROUP BY 1
ORDER BY 2 DESC
LIMIT 1) inner_tab)
) counter_tab;
30. For the customer that spent the most (in total over their lifetime as a
customer) total_amt_usd, how many web_events did they have for each channel?

Here, we first want to pull the customer with the most spent in lifetime value.
31. SELECT a.id, a.name, SUM(o.total_amt_usd) tot_spent
32. FROM orders o
33. JOIN accounts a
34. ON a.id = o.account_id
35. GROUP BY a.id, a.name
36. ORDER BY 3 DESC
37. LIMIT 1;
Now, we want to look at the number of events on each channel this company had, which we can
match with just the id.
SELECT a.name, w.channel, COUNT(*)
FROM accounts a
JOIN web_events w
ON a.id = w.account_id AND a.id = (SELECT id
FROM (SELECT a.id, a.name, SUM(o.total_amt_usd) tot_spent
FROM orders o
JOIN accounts a
ON a.id = o.account_id
GROUP BY a.id, a.name
ORDER BY 3 DESC
LIMIT 1) inner_table)
GROUP BY 1, 2
ORDER BY 3 DESC;
I added an ORDER BY for no real reason, and the account name to assure I was only pulling
from one account.

38. What is the lifetime average amount spent in terms of total_amt_usd for the top 10 total
spending accounts?

First, we just want to find the top 10 accounts in terms of highest total_amt_usd.

39. SELECT a.id, a.name, SUM(o.total_amt_usd) tot_spent
40. FROM orders o
41. JOIN accounts a
42. ON a.id = o.account_id
43. GROUP BY a.id, a.name
44. ORDER BY 3 DESC
45. LIMIT 10;
Now, we just want the average of these 10 amounts.
SELECT AVG(tot_spent)
FROM (SELECT a.id, a.name, SUM(o.total_amt_usd) tot_spent
FROM orders o
JOIN accounts a
ON a.id = o.account_id
 GROUP BY a.id, a.name
ORDER BY 3 DESC
LIMIT 10) temp;

46. What is the lifetime average amount spent in terms of total_amt_usd, including only the
companies that spent more per order, on average, than the average of all orders.

First, we want to pull the average of all accounts in terms of total_amt_usd:

47. SELECT AVG(o.total_amt_usd) avg_all
48. FROM orders o
Then, we want to only pull the accounts with more than this average amount.
SELECT o.account_id, AVG(o.total_amt_usd)
FROM orders o
GROUP BY 1
HAVING AVG(o.total_amt_usd) > (SELECT AVG(o.total_amt_usd) avg_all
FROM orders o);
Finally, we just want the average of these values.
SELECT AVG(avg_amt)
FROM (SELECT o.account_id, AVG(o.total_amt_usd) avg_amt
FROM orders o
GROUP BY 1
HAVING AVG(o.total_amt_usd) > (SELECT AVG(o.total_amt_usd) avg_all
FROM orders o)) temp_table;

The WITH statement is often called a Common Table Expression or CTE. Though

these expressions serve the exact same purpose as subqueries, they are more
common in practice, as they tend to be cleaner for a future reader to follow the
logic.
In the next concept, we will walk through this example a bit more slowly to make
sure you have all the similarities between subqueries and these expressions down
for you to use in practice! If you are already feeling comfortable skip ahead to
practice the quiz section.

Your First WITH (CTE)

The same question as you saw in  your first subquery  is provided here along with
the solution.
QUESTION: You need to find the average number of events for each channel per
day.
SOLUTION:
SELECT channel, AVG(events) AS average_events
FROM (SELECT DATE_TRUNC('day',occurred_at) AS day,
channel, COUNT(*) as events
 FROM web_events
GROUP BY 1,2) sub
GROUP BY channel
ORDER BY 2 DESC;

Let's try this again using a WITH statement.

Notice, you can pull the inner query:

SELECT DATE_TRUNC('day',occurred_at) AS day,

channel, COUNT(*) as events
FROM web_events
GROUP BY 1,2
This is the part we put in the WITH statement. Notice, we are aliasing the table
as  events  below:
WITH events AS (
SELECT DATE_TRUNC('day',occurred_at) AS day,
channel, COUNT(*) as events
FROM web_events
GROUP BY 1,2)
Now, we can use this newly created  events  table as if it is any other table in our
database:
WITH events AS (
SELECT DATE_TRUNC('day',occurred_at) AS day,
channel, COUNT(*) as events
FROM web_events
GROUP BY 1,2)

SELECT channel, AVG(events) AS average_events

FROM events
GROUP BY channel
ORDER BY 2 DESC;

For the above example, we don't need anymore than the one additional table, but
imagine we needed to create a second table to pull from. We can create an
additional table to pull from in the following way:

WITH table1 AS (
SELECT *
FROM web_events),

table2 AS (
SELECT *
FROM accounts)

SELECT *
FROM table1
JOIN table2
 ON table1.account_id = table2.id;
You can add more and more tables using the WITH statement in the same way.
The quiz at the bottom will assure you are catching all of the necessary components
of these new queries.
Feel free to explore how this works with the environment below.

WITH Quizzes
Essentially a WITH statement performs the same task as a Subquery. Therefore,
you can write any of the queries we worked with in the "Subquery Mania" using
a WITH. That's what you'll do here. Try to perform each of the earlier queries again,
but using a WITH instead of a subquery.

Above is the ERD for the database again - it might come in handy as you tackle the
quizzes below. You should write your solution as using a WITH statement, not by
finding one solution and copying the output. The importance of this is that it allows
your query to be dynamic in answering the question - even if the data changes, you
still arrive at the right answer.
1. Provide the name of the sales_rep in each region with the largest amount
of total_amt_usd sales.

2. For the region with the largest sales total_amt_usd, how many total orders were placed?

3. How many accounts had more total purchases than the account name which has bought

the most standard_qty paper throughout their lifetime as a customer?

4. For the customer that spent the most (in total over their lifetime as a
customer) total_amt_usd, how many web_events did they have for each channel?

5. What is the lifetime average amount spent in terms of total_amt_usd for the top 10 total
spending accounts?

6. What is the lifetime average amount spent in terms of total_amt_usd, including only the
companies that spent more per order, on average, than the average of all orders.

WITH Solutions
Below, you will see each of the previous solutions restructured using
the WITH clause. This is often an easier way to read a query.
1. Provide the name of the sales_rep in each region with the largest amount
of total_amt_usd sales.

2. WITH t1 AS (
3. SELECT s.name rep_name, r.name region_name, SUM(o.total_amt_usd)
total_amt
4. FROM sales_reps s
5. JOIN accounts a
6. ON a.sales_rep_id = s.id
7. JOIN orders o
8. ON o.account_id = a.id
9. JOIN region r
10. ON r.id = s.region_id
11. GROUP BY 1,2
12. ORDER BY 3 DESC),
13. t2 AS (
14. SELECT region_name, MAX(total_amt) total_amt
15. FROM t1
16. GROUP BY 1)
17. SELECT t1.rep_name, t1.region_name, t1.total_amt
18. FROM t1
19. JOIN t2
20. ON t1.region_name = t2.region_name AND t1.total_amt = t2.total_amt;
21. For the region with the largest sales total_amt_usd, how many total orders were placed?

22. WITH t1 AS (
23. SELECT r.name region_name, SUM(o.total_amt_usd) total_amt
24. FROM sales_reps s
25. JOIN accounts a
26. ON a.sales_rep_id = s.id
27. JOIN orders o
28. ON o.account_id = a.id
29. JOIN region r
30. ON r.id = s.region_id
31. GROUP BY r.name),
32. t2 AS (
33. SELECT MAX(total_amt)
34. FROM t1)
35. SELECT r.name, COUNT(o.total) total_orders
36. FROM sales_reps s
37. JOIN accounts a
38. ON a.sales_rep_id = s.id
39. JOIN orders o
40. ON o.account_id = a.id
41. JOIN region r
42. ON r.id = s.region_id
43. GROUP BY r.name
44. HAVING SUM(o.total_amt_usd) = (SELECT * FROM t2);
45. For the account that purchased the most (in total over their lifetime as a
customer) standard_qty paper, how many accounts still had more in total purchases?
46. WITH t1 AS (
47. SELECT a.name account_name, SUM(o.standard_qty) total_std, SUM(o.total)
total
48. FROM accounts a
49. JOIN orders o
50. ON o.account_id = a.id
51. GROUP BY 1
52. ORDER BY 2 DESC
53. LIMIT 1),
54. t2 AS (
55. SELECT a.name
56. FROM orders o
57. JOIN accounts a
58. ON a.id = o.account_id
59. GROUP BY 1
60. HAVING SUM(o.total) > (SELECT total FROM t1))
61. SELECT COUNT(*)
62. FROM t2;
63. For the customer that spent the most (in total over their lifetime as a
customer) total_amt_usd, how many web_events did they have for each channel?

64. WITH t1 AS (
65. SELECT a.id, a.name, SUM(o.total_amt_usd) tot_spent
66. FROM orders o
67. JOIN accounts a
68. ON a.id = o.account_id
69. GROUP BY a.id, a.name
70. ORDER BY 3 DESC
71. LIMIT 1)
72. SELECT a.name, w.channel, COUNT(*)
73. FROM accounts a
74. JOIN web_events w
75. ON a.id = w.account_id AND a.id = (SELECT id FROM t1)
76. GROUP BY 1, 2
77. ORDER BY 3 DESC;
78. What is the lifetime average amount spent in terms of total_amt_usd for the top 10 total
spending accounts?

79. WITH t1 AS (
80. SELECT a.id, a.name, SUM(o.total_amt_usd) tot_spent
81. FROM orders o
82. JOIN accounts a
83. ON a.id = o.account_id
84. GROUP BY a.id, a.name
85. ORDER BY 3 DESC
86. LIMIT 10)
87. SELECT AVG(tot_spent)
88. FROM t1;
89. What is the lifetime average amount spent in terms of total_amt_usd, including only the
companies that spent more per order, on average, than the average of all orders.

90. WITH t1 AS (
91. SELECT AVG(o.total_amt_usd) avg_all
92. FROM orders o
93. JOIN accounts a
94. ON a.id = o.account_id),
95. t2 AS (
96. SELECT o.account_id, AVG(o.total_amt_usd) avg_amt
97. FROM orders o
98. GROUP BY 1
99. HAVING AVG(o.total_amt_usd) > (SELECT * FROM t1))
100. SELECT AVG(avg_amt)
FROM t2;

Recap
This lesson was the first of the more advanced sequence in writing SQL. Arguably,
the advanced features of Subqueries and CTEs are the most widely used in an
analytics role within a company. Being able to break a problem down into the
necessary tables and finding a solution using the resulting table is very useful in
practice.
If you didn't get the solutions to these queries on the first pass, don't be afraid to
come back another time and give them another try. Additionally, you might try
coming up with some questions of your own to see if you can find the solution.

The remaining portions of this course may be key to certain analytics roles, but you
have now covered all of the main SQL topics you are likely to use on a day to day
basis.