Python for Everybody

Exploring Data Using Python 3

Charles R. Severance
9.3. LOOPING AND DICTIONARIES 113

Because the inner loop executes all of its iterations each time the outer loop makes
a single iteration, we think of the inner loop as iterating “more quickly” and the
outer loop as iterating more slowly.
The combination of the two nested loops ensures that we will count every word on
every line of the input file.

fname = input('Enter the file name: ')

try:
fhand = open(fname)
except:
print('File cannot be opened:', fname)
exit()

counts = dict()
for line in fhand:
words = line.split()
for word in words:
if word not in counts:
counts[word] = 1
else:
counts[word] += 1

print(counts)

# Code: http://www.pythonlearn.com/code3/count1.py

When we run the program, we see a raw dump of all of the counts in unsorted hash
order. (the romeo.txt file is available at www.pythonlearn.com/code3/romeo.txt)

python count1.py
Enter the file name: romeo.txt
{'and': 3, 'envious': 1, 'already': 1, 'fair': 1,
'is': 3, 'through': 1, 'pale': 1, 'yonder': 1,
'what': 1, 'sun': 2, 'Who': 1, 'But': 1, 'moon': 1,
'window': 1, 'sick': 1, 'east': 1, 'breaks': 1,
'grief': 1, 'with': 1, 'light': 1, 'It': 1, 'Arise': 1,
'kill': 1, 'the': 3, 'soft': 1, 'Juliet': 1}

It is a bit inconvenient to look through the dictionary to find the most common
words and their counts, so we need to add some more Python code to get us the
output that will be more helpful.

9.3 Looping and dictionaries

If you use a dictionary as the sequence in a for statement, it traverses the keys of
the dictionary. This loop prints each key and the corresponding value:

counts = { 'chuck' : 1 , 'annie' : 42, 'jan': 100}

for key in counts:
print(key, counts[key])
114 CHAPTER 9. DICTIONARIES

Here’s what the output looks like:

jan 100
chuck 1
annie 42

Again, the keys are in no particular order.

We can use this pattern to implement the various loop idioms that we have de-
scribed earlier. For example if we wanted to find all the entries in a dictionary
with a value above ten, we could write the following code:

counts = { 'chuck' : 1 , 'annie' : 42, 'jan': 100}

for key in counts:
if counts[key] > 10 :
print(key, counts[key])

The for loop iterates through the keys of the dictionary, so we must use the index
operator to retrieve the corresponding value for each key. Here’s what the output
looks like:

jan 100
annie 42

We see only the entries with a value above 10.

If you want to print the keys in alphabetical order, you first make a list of the keys
in the dictionary using the keys method available in dictionary objects, and then
sort that list and loop through the sorted list, looking up each key and printing
out key-value pairs in sorted order as follows:

counts = { 'chuck' : 1 , 'annie' : 42, 'jan': 100}

lst = list(counts.keys())
print(lst)
lst.sort()
for key in lst:
print(key, counts[key])

Here’s what the output looks like:

['jan', 'chuck', 'annie']

annie 42
chuck 1
jan 100

First you see the list of keys in unsorted order that we get from the keys method.
Then we see the key-value pairs in order from the for loop.
9.4. ADVANCED TEXT PARSING 115

9.4 Advanced text parsing

In the above example using the file romeo.txt, we made the file as simple as possi-
ble by removing all punctuation by hand. The actual text has lots of punctuation,
as shown below.

But, soft! what light through yonder window breaks?

It is the east, and Juliet is the sun.
Arise, fair sun, and kill the envious moon,
Who is already sick and pale with grief,

Since the Python split function looks for spaces and treats words as tokens sep-
arated by spaces, we would treat the words “soft!” and “soft” as different words
and create a separate dictionary entry for each word.
Also since the file has capitalization, we would treat “who” and “Who” as different
words with different counts.
We can solve both these problems by using the string methods lower, punctuation,
and translate. The translate is the most subtle of the methods. Here is the
documentation for translate:
string.translate(s, table[, deletechars])
Delete all characters from s that are in deletechars (if present), and then translate
the characters using table, which must be a 256-character string giving the trans-
lation for each character value, indexed by its ordinal. If table is None, then only
the character deletion step is performed.
We will not specify the table but we will use the deletechars parameter to delete
all of the punctuation. We will even let Python tell us the list of characters that
it considers “punctuation”:

>>> import string

>>> string.punctuation
'!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~'

We make the following modifications to our program:

import string

fname = input('Enter the file name: ')

try:
fhand = open(fname)
except:
print('File cannot be opened:', fname)
exit()

counts = dict()
for line in fhand:
line = line.rstrip()
line = line.translate(line.maketrans('', '', string.punctuation))
116 CHAPTER 9. DICTIONARIES

line = line.lower()
words = line.split()
for word in words:
if word not in counts:
counts[word] = 1
else:
counts[word] += 1

print(counts)

# Code: http://www.pythonlearn.com/code3/count2.py

We use translate to remove all punctuation and lower to force the line to lower-
case. Otherwise the program is unchanged. Note that for Python 2.5 and earlier,
translate does not accept None as the first parameter so use this code instead for
the translate call:

print a.translate(string.maketrans(' ',' '), string.punctuation

Part of learning the “Art of Python” or “Thinking Pythonically” is realizing that

Python often has built-in capabilities for many common data analysis problems.
Over time, you will see enough example code and read enough of the documentation
to know where to look to see if someone has already written something that makes
your job much easier.
The following is an abbreviated version of the output:

Enter the file name: romeo-full.txt

{'swearst': 1, 'all': 6, 'afeard': 1, 'leave': 2, 'these': 2,
'kinsmen': 2, 'what': 11, 'thinkst': 1, 'love': 24, 'cloak': 1,
a': 24, 'orchard': 2, 'light': 5, 'lovers': 2, 'romeo': 40,
'maiden': 1, 'whiteupturned': 1, 'juliet': 32, 'gentleman': 1,
'it': 22, 'leans': 1, 'canst': 1, 'having': 1, ...}

Looking through this output is still unwieldy and we can use Python to give us
exactly what we are looking for, but to do so, we need to learn about Python tuples.
We will pick up this example once we learn about tuples.

9.5 Debugging
As you work with bigger datasets it can become unwieldy to debug by printing and
checking data by hand. Here are some suggestions for debugging large datasets:

Scale down the input If possible, reduce the size of the dataset. For example
if the program reads a text file, start with just the first 10 lines, or with the
smallest example you can find. You can either edit the files themselves, or
(better) modify the program so it reads only the first n lines.
If there is an error, you can reduce n to the smallest value that manifests the
error, and then increase it gradually as you find and correct errors.
9.6. GLOSSARY 117

Check summaries and types Instead of printing and checking the entire
dataset, consider printing summaries of the data: for example, the number
of items in a dictionary or the total of a list of numbers.
A common cause of runtime errors is a value that is not the right type. For
debugging this kind of error, it is often enough to print the type of a value.
Write self-checks Sometimes you can write code to check for errors automati-
cally. For example, if you are computing the average of a list of numbers, you
could check that the result is not greater than the largest element in the list
or less than the smallest. This is called a “sanity check” because it detects
results that are “completely illogical”.

Another kind of check compares the results of two different

computations to see if they are consistent. This is called a
"consistency check".

Pretty print the output Formatting debugging output can make it easier to
spot an error.

Again, time you spend building scaffolding can reduce the time you spend debug-
ging.

9.6 Glossary
dictionary A mapping from a set of keys to their corresponding values.

hashtable The algorithm used to implement Python dictionaries.

hash function A function used by a hashtable to compute the location for a key.

histogram A set of counters.

implementation A way of performing a computation.

item Another name for a key-value pair.

key An object that appears in a dictionary as the first part of a key-value pair.

key-value pair The representation of the mapping from a key to a value.

lookup A dictionary operation that takes a key and finds the corresponding value.

nested loops When there are one or more loops “inside” of another loop. The
inner loop runs to completion each time the outer loop runs once.

value An object that appears in a dictionary as the second part of a key-value

pair. This is more specific than our previous use of the word “value”.
118 CHAPTER 9. DICTIONARIES

9.7 Exercises
Exercise 2: Write a program that categorizes each mail message by which day of
the week the commit was done. To do this look for lines that start with “From”,
then look for the third word and keep a running count of each of the days of the
week. At the end of the program print out the contents of your dictionary (order
does not matter).
Sample Line:

From [email protected] Sat Jan 5 09:14:16 2008

Sample Execution:

python dow.py
Enter a file name: mbox-short.txt
{'Fri': 20, 'Thu': 6, 'Sat': 1}

Exercise 3: Write a program to read through a mail log, build a histogram using
a dictionary to count how many messages have come from each email address, and
print the dictionary.

Enter file name: mbox-short.txt

{'[email protected]': 1, '[email protected]': 3,
'[email protected]': 5, '[email protected]': 1,
'[email protected]': 2, '[email protected]': 3,
'[email protected]': 4, '[email protected]': 1,
'[email protected]': 4, '[email protected]': 2,
'[email protected]': 1}

Exercise 4: Add code to the above program to figure out who has the most
messages in the file.
After all the data has been read and the dictionary has been created, look through
the dictionary using a maximum loop (see Section [maximumloop]) to find who
has the most messages and print how many messages the person has.

Enter a file name: mbox-short.txt

[email protected] 5

Enter a file name: mbox.txt

[email protected] 195

Exercise 5: This program records the domain name (instead of the address) where
the message was sent from instead of who the mail came from (i.e., the whole email
address). At the end of the program, print out the contents of your dictionary.

python schoolcount.py
Enter a file name: mbox-short.txt
{'media.berkeley.edu': 4, 'uct.ac.za': 6, 'umich.edu': 7,
'gmail.com': 1, 'caret.cam.ac.uk': 1, 'iupui.edu': 8}
Chapter 10

Tuples

10.1 Tuples are immutable

A tuple1 is a sequence of values much like a list. The values stored in a tuple can
be any type, and they are indexed by integers. The important difference is that
tuples are immutable. Tuples are also comparable and hashable so we can sort lists
of them and use tuples as key values in Python dictionaries.
Syntactically, a tuple is a comma-separated list of values:

>>> t = 'a', 'b', 'c', 'd', 'e'

Although it is not necessary, it is common to enclose tuples in parentheses to help

us quickly identify tuples when we look at Python code:

>>> t = ('a', 'b', 'c', 'd', 'e')

To create a tuple with a single element, you have to include the final comma:

>>> t1 = ('a',)
>>> type(t1)
<type 'tuple'>

Without the comma Python treats (’a’) as an expression with a string in paren-
theses that evaluates to a string:

>>> t2 = ('a')
>>> type(t2)
<type 'str'>

Another way to construct a tuple is the built-in function tuple. With no argument,
it creates an empty tuple:
1 Fun fact: The word “tuple” comes from the names given to sequences of numbers of varying

lengths: single, double, triple, quadruple, quituple, sextuple, septuple, etc.

119
120 CHAPTER 10. TUPLES

>>> t = tuple()
>>> print(t)
()

If the argument is a sequence (string, list, or tuple), the result of the call to tuple
is a tuple with the elements of the sequence:

>>> t = tuple('lupins')
>>> print(t)
('l', 'u', 'p', 'i', 'n', 's')

Because tuple is the name of a constructor, you should avoid using it as a variable
name.
Most list operators also work on tuples. The bracket operator indexes an element:

>>> t = ('a', 'b', 'c', 'd', 'e')

>>> print(t[0])
'a'

And the slice operator selects a range of elements.

>>> print(t[1:3])
('b', 'c')

But if you try to modify one of the elements of the tuple, you get an error:

>>> t[0] = 'A'

TypeError: object doesn't support item assignment

You can’t modify the elements of a tuple, but you can replace one tuple with
another:

>>> t = ('A',) + t[1:]

>>> print(t)
('A', 'b', 'c', 'd', 'e')

10.2 Comparing tuples

The comparison operators work with tuples and other sequences. Python starts by
comparing the first element from each sequence. If they are equal, it goes on to the
next element, and so on, until it finds elements that differ. Subsequent elements
are not considered (even if they are really big).

>>> (0, 1, 2) < (0, 3, 4)

True
>>> (0, 1, 2000000) < (0, 3, 4)
True
10.2. COMPARING TUPLES 121

The sort function works the same way. It sorts primarily by first element, but in
the case of a tie, it sorts by second element, and so on.
This feature lends itself to a pattern called DSU for

Decorate a sequence by building a list of tuples with one or more sort keys
preceding the elements from the sequence,

Sort the list of tuples using the Python built-in sort, and

Undecorate by extracting the sorted elements of the sequence.

[DSU]
For example, suppose you have a list of words and you want to sort them from
longest to shortest:

txt = 'but soft what light in yonder window breaks'

words = txt.split()
t = list()
for word in words:
t.append((len(word), word))

t.sort(reverse=True)

res = list()
for length, word in t:
res.append(word)

print(res)

# Code: http://www.pythonlearn.com/code3/soft.py

The first loop builds a list of tuples, where each tuple is a word preceded by its
length.
sort compares the first element, length, first, and only considers the second el-
ement to break ties. The keyword argument reverse=True tells sort to go in
decreasing order.
The second loop traverses the list of tuples and builds a list of words in descending
order of length. The four-character words are sorted in reverse alphabetical order,
so “what” appears before “soft” in the following list.
The output of the program is as follows:

['yonder', 'window', 'breaks', 'light', 'what',

'soft', 'but', 'in']

Of course the line loses much of its poetic impact when turned into a Python list
and sorted in descending word length order.
122 CHAPTER 10. TUPLES

10.3 Tuple assignment

One of the unique syntactic features of the Python language is the ability to have
a tuple on the left side of an assignment statement. This allows you to assign more
than one variable at a time when the left side is a sequence.
In this example we have a two-element list (which is a sequence) and assign the first
and second elements of the sequence to the variables x and y in a single statement.

>>> m = [ 'have', 'fun' ]

>>> x, y = m
>>> x
'have'
>>> y
'fun'
>>>

It is not magic, Python roughly translates the tuple assignment syntax to be the
following:2

>>> m = [ 'have', 'fun' ]

>>> x = m[0]
>>> y = m[1]
>>> x
'have'
>>> y
'fun'
>>>

Stylistically when we use a tuple on the left side of the assignment statement, we
omit the parentheses, but the following is an equally valid syntax:

>>> m = [ 'have', 'fun' ]

>>> (x, y) = m
>>> x
'have'
>>> y
'fun'
>>>

A particularly clever application of tuple assignment allows us to swap the values

of two variables in a single statement:

>>> a, b = b, a
2 Python does not translate the syntax literally. For example, if you try this with a dictionary,

it will not work as might expect.

10.4. DICTIONARIES AND TUPLES 123

Both sides of this statement are tuples, but the left side is a tuple of variables;
the right side is a tuple of expressions. Each value on the right side is assigned
to its respective variable on the left side. All the expressions on the right side are
evaluated before any of the assignments.
The number of variables on the left and the number of values on the right must be
the same:

>>> a, b = 1, 2, 3
ValueError: too many values to unpack

More generally, the right side can be any kind of sequence (string, list, or tuple).
For example, to split an email address into a user name and a domain, you could
write:

>>> addr = '[email protected]'

>>> uname, domain = addr.split('@')

The return value from split is a list with two elements; the first element is assigned
to uname, the second to domain.

>>> print(uname)
monty
>>> print(domain)
python.org

10.4 Dictionaries and tuples

Dictionaries have a method called items that returns a list of tuples, where each
tuple is a key-value pair:

>>> d = {'a':10, 'b':1, 'c':22}

>>> t = list(d.items())
>>> print(t)
[('b', 1), ('a', 10), ('c', 22)]

As you should expect from a dictionary, the items are in no particular order.
However, since the list of tuples is a list, and tuples are comparable, we can now
sort the list of tuples. Converting a dictionary to a list of tuples is a way for us to
output the contents of a dictionary sorted by key:

>>> d = {'a':10, 'b':1, 'c':22}

>>> t = list(d.items())
>>> t
[('b', 1), ('a', 10), ('c', 22)]
>>> t.sort()
>>> t
[('a', 10), ('b', 1), ('c', 22)]

The new list is sorted in ascending alphabetical order by the key value.
124 CHAPTER 10. TUPLES

10.5 Multiple assignment with dictionaries

Combining items, tuple assignment, and for, you can see a nice code pattern for
traversing the keys and values of a dictionary in a single loop:

for key, val in list(d.items()):

print(val, key)

This loop has two iteration variables because items returns a list of tuples and key,
val is a tuple assignment that successively iterates through each of the key-value
pairs in the dictionary.

For each iteration through the loop, both key and value are advanced to the next
key-value pair in the dictionary (still in hash order).

The output of this loop is:

10 a
22 c
1 b

Again, it is in hash key order (i.e., no particular order).

If we combine these two techniques, we can print out the contents of a dictionary
sorted by the value stored in each key-value pair.

To do this, we first make a list of tuples where each tuple is (value, key). The
items method would give us a list of (key, value) tuples, but this time we want
to sort by value, not key. Once we have constructed the list with the value-key
tuples, it is a simple matter to sort the list in reverse order and print out the new,
sorted list.

>>> d = {'a':10, 'b':1, 'c':22}

>>> l = list()
>>> for key, val in d.items() :
... l.append( (val, key) )
...
>>> l
[(10, 'a'), (22, 'c'), (1, 'b')]
>>> l.sort(reverse=True)
>>> l
[(22, 'c'), (10, 'a'), (1, 'b')]
>>>

By carefully constructing the list of tuples to have the value as the first element
of each tuple, we can sort the list of tuples and get our dictionary contents sorted
by value.