Download More ebooks [PDF]. Format PDF ebook download PDF KINDLE.

Full download ebooks at ebookmass.com

Advanced Guide to Python 3 Programming,

2nd 2nd Edition John Hunt

For dowload this book click BUTTON or LINK below

https://ebookmass.com/product/advanced-guide-to-
python-3-programming-2nd-2nd-edition-john-hunt/

OR CLICK BUTTON

DOWLOAD NOW

Download More ebooks from https://ebookmass.com

More products digital (pdf, epub, mobi) instant
download maybe you interests ...

A Beginners Guide to Python 3 Programming 2nd Edition

John Hunt

https://ebookmass.com/product/a-beginners-guide-to-
python-3-programming-2nd-edition-john-hunt/

Python Programming: 3 Books in 1: Ultimate

Beginneru2019s, Intermediate & Advanced Guide to Learn
Python Step-by-Step

https://ebookmass.com/product/python-programming-3-books-
in-1-ultimate-beginners-intermediate-advanced-guide-to-learn-
python-step-by-step/

Learning Scientific Programming with Python 2nd Edition

Christian Hill

https://ebookmass.com/product/learning-scientific-programming-
with-python-2nd-edition-christian-hill-2/

Learning Scientific Programming With Python 2nd Edition

Christian Hill

https://ebookmass.com/product/learning-scientific-programming-
with-python-2nd-edition-christian-hill/
Python Mini Reference 2022: A Quick Guide to the Modern
Python Programming Language for Busy Coders (A
Hitchhiker's Guide to the Modern Programming Languages
Book 3) Harry Yoon
https://ebookmass.com/product/python-mini-reference-2022-a-quick-
guide-to-the-modern-python-programming-language-for-busy-coders-
a-hitchhikers-guide-to-the-modern-programming-languages-
book-3-harry-yoon/

Python for Beginners: Master Python Programming from

Basics to Advanced Level Tim Simon

https://ebookmass.com/product/python-for-beginners-master-python-
programming-from-basics-to-advanced-level-tim-simon/

MicroPython for the Internet of Things: A Beginner’s

Guide to Programming with Python on Microcontrollers
2nd Edition Bell

https://ebookmass.com/product/micropython-for-the-internet-of-
things-a-beginners-guide-to-programming-with-python-on-
microcontrollers-2nd-edition-bell/

Introduction to Computation and Programming Using

Python, Third Edition John V. Guttag

https://ebookmass.com/product/introduction-to-computation-and-
programming-using-python-third-edition-john-v-guttag/

A Guide to Assessments That Work 2nd Edition John

Hunsley (Editor)

https://ebookmass.com/product/a-guide-to-assessments-that-
work-2nd-edition-john-hunsley-editor/
Undergraduate Topics in Computer Science

John Hunt

Advanced Guide
to Python 3
Programming
Second Edition
Undergraduate Topics in Computer Science

Series Editor
Ian Mackie, University of Sussex, Brighton, UK

Advisory Editors
Samson Abramsky , Department of Computer Science, University of Oxford,
Oxford, UK
Chris Hankin , Department of Computing, Imperial College London, London,
UK
Mike Hinchey , Lero – The Irish Software Research Centre, University of
Limerick, Limerick, Ireland
Dexter C. Kozen, Department of Computer Science, Cornell University, Ithaca,
NY, USA
Andrew Pitts , Department of Computer Science and Technology, University of
Cambridge, Cambridge, UK
Hanne Riis Nielson , Department of Applied Mathematics and Computer
Science, Technical University of Denmark, Kongens Lyngby, Denmark
Steven S. Skiena, Department of Computer Science, Stony Brook University,
Stony Brook, NY, USA
Iain Stewart , Department of Computer Science, Durham University, Durham,
UK
Joseph Migga Kizza, College of Engineering and Computer Science, The
University of Tennessee-Chattanooga, Chattanooga, TN, USA
‘Undergraduate Topics in Computer Science’ (UTiCS) delivers high-quality instruc-
tional content for undergraduates studying in all areas of computing and information
science. From core foundational and theoretical material to final-year topics and
applications, UTiCS books take a fresh, concise, and modern approach and are ideal
for self-study or for a one- or two-semester course. The texts are all authored by
established experts in their fields, reviewed by an international advisory board, and
contain numerous examples and problems, many of which include fully worked
solutions.
The UTiCS concept relies on high-quality, concise books in softback format, and
generally a maximum of 275–300 pages. For undergraduate textbooks that are likely
to be longer, more expository, Springer continues to offer the highly regarded Texts
in Computer Science series, to which we refer potential authors.
John Hunt

Advanced Guide to Python 3

Programming
Second Edition
John Hunt
Midmarsh Technology Ltd.
Chippenham, Wiltshire, UK

ISSN 1863-7310 ISSN 2197-1781 (electronic)

Undergraduate Topics in Computer Science
ISBN 978-3-031-40335-4 ISBN 978-3-031-40336-1 (eBook)
https://doi.org/10.1007/978-3-031-40336-1

© Springer Nature Switzerland AG 2019, 2023

This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of
the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microfilms or in any other physical way, and transmission or information
storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology
now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors, and the editors are safe to assume that the advice and information in this book
are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or
the editors give a warranty, expressed or implied, with respect to the material contained herein or for any
errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional
claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AG
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
For Denise, my wife, my soulmate, my best
friend.
Preface to the Second Edition

This second edition represents a significant expansion of the material in the first
edition, as well as an update of that book from Python 3.7 to 3.12.
This book includes whole new sections on advanced language features, Reactive
Programming in Python and data analysts. New chapters on working with Tkinter,
on event handling with Tkinter and a simple drawing application using Tkinter have
been added. A new chapter on performance monitoring and profiling has also been
added. A chapter on pip and conda is included at the end of the book.
In all there are 18 completely new chapters that take you far further on your Python
journey. Enjoy the book and I hope you find it useful.

Chippenham, UK John Hunt

vii
Preface to the First Edition

Some of the key aspects of this book are

1. It assumes knowledge of Python 3 and of concepts such as functions, classes,
protocols, abstract base classes, decorators, iterables and collection types (such
as list and tuple).
2. However, the book assumes very little knowledge or experience of the topics
presented.
3. The book is divided into eleven topic areas: advanced language features,
Computer Graphics, games, testing, file input/output, database access, logging,
concurrency and parallelism, Reactive Programming, network programming and
data analytics.
4. Each topic in the book has an introductory chapter followed by chapters that
delve into that topic.
5. The book includes exercises at the end of most chapters.
6. All code examples (and exercise solutions) are provided on line in a GitHub
repository.

What You Need

You can of course just read this book; however following the examples in this book
will ensure that you get as much as possible out of the content. For this you will need
a computer.
Python is a cross-platform programming language, and as such you can use Python
on a Windows PC, a Linux box, an Apple Mac, etc. So you are not tied to a particular
type of operating system; you can use whatever you have available.
However you will need to install some software on that computer. At a minimum
you will need Python. The focus of this book is Python 3 so that is the version that
is assumed for all examples and exercises. As Python is available for a wide range

ix
x Preface to the First Edition

of platforms from Windows, to Mac OS and Linux, you will need to ensure that you
download the version for your operating system.
Python can be downloaded from the main Python website which can be found at
http://www.python.org/.

You will also need some form of editor to write your programs. There are numerous
generic programming editors available for different operating systems with VIM on
Linux, Notepad++ on Windows and Sublime Text on windows and Macs being
popular choices.
However, using an Integrated Development Environment (IDE) editor such as
PyCharm, Visual Studio Code or Spyder can make writing and running your programs
much easier.
However, this book does not assume any particular editor, IDE or environment
(other than Python 3 itself).

Conventions

Throughout this book you will find a number of conventions used for text styles.
These text styles distinguish between different kinds of information. Code words,
variable and Python values, used within the main body of the text, are shown using
a Courier font. A block of Python code is set out as shown here:
Preface to the First Edition xi

def draw_koch(size, depth):

if depth > 0:
for angle in ANGLES:
draw_koch(size / 3, depth - 1)
turtle.left(angle)
else:
turtle.forward(size)
# Draw three sides of snowflake
for_in range(3):
draw_koch(SIZE_OF_SNOWFLAKE, depth)
turtle.right(120)

Note that keywords and points of interest are shown in bold font.
Any command line or user input is shown in standard font as shown below, for
example:
Hello, world
Enter your name: John
Hello John

Example Code and Sample Solutions

The examples used in this book (along with sample solutions for the exercises at the
end of most chapters) are available in a GitHub repository. GitHub provides a web
interface to Git, as well as a server environment hosting Git.
Git is a version control system typically used to manage source code files (such
as those used to create systems in programming languages such as Python but also
Java, C#, C++ and Scala). Systems such as Git are very useful for collaborative
development as they allow multiple people to work on an implementation and to
merge their work together. They also provide a useful historical view of the code
(which also allows developers to roll back changes if modifications prove to be
unsuitable).
The GitHub repository for this book can be found at:
• https://github.com/johnehunt/advancedpython3_2nd
If you already have Git installed on your computer, then you can clone (obtain a
copy of) the repository locally using:
git clone https://github.com/johnehunt/advancedpyth
on3_2nd.git
If you do not have Git, then you can obtain a zip file of the examples using
https://github.com/johnehunt/advancedpython3_2nd/arc
hive/refs/heads/main.zip
xii Preface to the First Edition

You can of course install Git yourself if you wish. To do this, see https://git-scm.
com/downloads. Versions of the Git client for Mac OS, Windows and Linux/Unix
are available here.
However, many IDEs such as PyCharm come with Git support and so offer another
approach to obtaining a Git repository.
For more information on Git see http://git-scm.com/doc. This Git guide
provides a very good primer and is highly recommended.

Acknowledgement I would like to thank Phoebe Hunt for creating the pixel images used for the
Starship Meteors game in Chap. 22.
Contents

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Useful Python Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Part I Advanced Language Features

2 Python Type Hints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 Pythons Type System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.3 The Challenge for Python Developers . . . . . . . . . . . . . . . . . . . . . . 8
2.4 Static Typing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.5 Python Type Hints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.6 Type Hint Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.7 Type Hints for Multiple Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.8 The Self Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.9 The Benefits of Type Hints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.11 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3 Class Slots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.2 The Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.3 Slots to the Rescue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.4 Performance Benefits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.5 Why Not Use Slots? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.6 Online Resource . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4 Weak References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.2 How Garbage Collection Works: Reference Counting . . . . . . . . 23
4.3 Weak References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.4 When to Use Weak References . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

xiii
xiv Contents

4.5 The Weakref Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

4.6 Creating Weak References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.7 Retrieving Objects from Weak References . . . . . . . . . . . . . . . . . . 28
4.8 The WeakValueDicitonary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
4.9 WeakKeyDictionary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.10 Proxy Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.11 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
5 Data Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
5.2 A Traditional Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
5.3 Defining Data Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
5.4 Defining Additional Behaviour . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
5.5 The Dataclass Decorator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
5.6 Custom Factory for Fields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
5.7 Immutable Dataclasses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
5.8 Data Classes and Inheritance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
5.9 Post Initialisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
5.10 Initialisation Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5.11 Positional Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5.12 Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
6 Structural Pattern Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
6.2 The Match Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
6.3 Matching Classes with Positional Attributes . . . . . . . . . . . . . . . . . 50
6.4 Matching Against Standard Classes . . . . . . . . . . . . . . . . . . . . . . . . 51
6.5 Online Resource . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
7 Working with pprint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
7.2 The pprint Data Printer Module . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
7.3 Basic pprint Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
7.4 Changing the Width . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
7.5 Changing the Depth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
7.6 Managing the Indentation Level . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
7.7 Reducing Line Breaks Using Compact . . . . . . . . . . . . . . . . . . . . . 59
7.8 The pformat Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
7.9 The saferepr() Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
7.10 Using the PrettyPrinter Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
7.11 Online Resource . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
8 Shallow v Deep Copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
8.2 Copying a List of Lists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
8.3 The Problem with Copying . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
Contents xv

8.4 The Copy Module to the Rescue . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

8.5 Using the deepcopy() Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
8.6 Online Resource . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
9 The __init__ Versus __new__ and __call . . . . . . . . . . . . . . . . . . . . . . . . . 71
9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
9.2 The __new__ and __init__ Methods . . . . . . . . . . . . . . . . . . . . . . . 71
9.3 The __new__ Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
9.4 When to Use the __new__ Method . . . . . . . . . . . . . . . . . . . . . . . . . 73
9.5 Using __new__ to Create a Singleton Object . . . . . . . . . . . . . . . . 74
9.6 The __init__ Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
9.7 Can __new__ and __init__ Be Used Together? . . . . . . . . . . . . . . 76
9.8 The __call__ Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
9.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
10 Python Metaclasses and Meta Programming . . . . . . . . . . . . . . . . . . . . . 79
10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
10.2 Metaprogramming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
10.3 Decorators as a Form of Metaprogramming . . . . . . . . . . . . . . . . . 81
10.4 Metaclasses for Metaprogramming . . . . . . . . . . . . . . . . . . . . . . . . 83
10.4.1 Singleton Metaclass . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
10.5 Exec and Eval for Metaprogramming . . . . . . . . . . . . . . . . . . . . . . 85
10.5.1 The exec() Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
10.5.2 The eval() Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
10.5.3 eval Versus exec() . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

Part II Computer Graphics and GUIs

11 Introduction to Computer Graphics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
11.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
11.3 The Graphical Computer Era . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
11.4 Interactive and Non Interactive Graphics . . . . . . . . . . . . . . . . . . . . 93
11.5 Pixels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
11.6 Bit Map Versus Vector Graphics . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
11.7 Buffering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
11.8 Python and Computer Graphics . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
11.9 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
11.10 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
12 Python Turtle Graphics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
12.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
12.2 The Turtle Graphics Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
12.2.1 The Turtle Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
12.2.2 Basic Turtle Graphics . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
12.2.3 Drawing Shapes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
12.2.4 Filling Shapes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
xvi Contents

12.3 Other Graphics Libraries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

12.4 3D Graphics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
12.4.1 PyOpenGL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
12.5 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
12.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
13 Computer Generated Art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
13.1 Creating Computer Art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
13.2 A Computer Art Generator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
13.3 Fractals in Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
13.4 The Koch Snowflake . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
13.5 Mandelbrot Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
13.6 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
13.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
14 Introduction to Matplotlib . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
14.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
14.2 Matplotlib . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
14.3 Plot Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
14.4 Matplotlib Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
14.4.1 Backend Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
14.4.2 The Artist Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
14.4.3 The Scripting Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
14.5 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
15 Graphing with Matplotlib Pyplot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
15.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
15.2 The pyplot API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
15.3 Line Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
15.3.1 Coded Format Strings . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
15.4 Scatter Graph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
15.4.1 When to Use Scatter Graphs . . . . . . . . . . . . . . . . . . . . . 134
15.5 Pie Charts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
15.5.1 Expanding Segments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
15.5.2 When to Use Pie Charts . . . . . . . . . . . . . . . . . . . . . . . . . 138
15.6 Bar Charts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
15.6.1 Horizontal Bar Charts . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
15.6.2 Coloured Bars . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
15.6.3 Stacked Bar Charts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
15.6.4 Grouped Bar Charts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
15.7 Figures and Subplots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
15.8 3D Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
15.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
Contents xvii

16 Graphical User Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151

16.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
16.2 GUIs and WIMPS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
16.3 Windowing Frameworks for Python . . . . . . . . . . . . . . . . . . . . . . . . 153
16.3.1 Platform-Independent GUI Libraries . . . . . . . . . . . . . . 154
16.3.2 Platform-Specific GUI Libraries . . . . . . . . . . . . . . . . . . 154
16.4 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
17 Tkinter GUI Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
17.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
17.2 Tkinter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
17.3 Windows as Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
17.4 Key Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
17.4.1 The Tk Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
17.4.2 TK Widgets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
17.4.3 The TopLevel Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
17.4.4 The Frame Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
17.4.5 Dialogs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
17.4.6 The Canvas Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
17.5 The Class Inheritance Hierarchy . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
17.5.1 Layout Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
17.6 A Simple Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
17.7 Tkinter Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
17.7.1 Mac Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
17.7.2 Windows Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
17.8 GUI Builders for Tkinter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
17.9 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
17.10 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
18 Events in Tkinter User Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
18.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
18.2 Event Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
18.3 What is Event Handling? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
18.4 What Are Event Handlers? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
18.5 Event Binders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
18.6 Virtual Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
18.7 Event Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
18.8 What Types of Event Are There? . . . . . . . . . . . . . . . . . . . . . . . . . . 174
18.9 Binding an Event to an Event Handler . . . . . . . . . . . . . . . . . . . . . . 175
18.10 Implementing Event Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
18.11 An Interactive GUI Application . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
18.12 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
18.13 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
xviii Contents

19 PyDraw Tkinter Example Application . . . . . . . . . . . . . . . . . . . . . . . . . . 185

19.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
19.2 The PyDraw Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
19.3 The Structure of the Application . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
19.3.1 Model, View and Controller Architecture . . . . . . . . . . 188
19.3.2 PyDraw MVC Architecture . . . . . . . . . . . . . . . . . . . . . . 189
19.3.3 Additional Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
19.3.4 Object Relationships . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
19.4 The Interactions Between Objects . . . . . . . . . . . . . . . . . . . . . . . . . 191
19.4.1 The PyDrawApp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
19.5 The PyDrawView Constructor . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
19.5.1 Changing the Application Mode . . . . . . . . . . . . . . . . . . 193
19.5.2 Adding a Graphic Object . . . . . . . . . . . . . . . . . . . . . . . . 193
19.6 The Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
19.6.1 The PyDrawConstants Class . . . . . . . . . . . . . . . . . . . . . 194
19.6.2 The PyDrawView Class . . . . . . . . . . . . . . . . . . . . . . . . . 195
19.6.3 The PyDrawMenuBar Class . . . . . . . . . . . . . . . . . . . . . . 196
19.6.4 The PyDrawController Class . . . . . . . . . . . . . . . . . . . . . 196
19.6.5 The DrawingModel Class . . . . . . . . . . . . . . . . . . . . . . . . 197
19.6.6 The DrawingView Class . . . . . . . . . . . . . . . . . . . . . . . . . 198
19.6.7 The DrawingController Class . . . . . . . . . . . . . . . . . . . . . 198
19.6.8 The Figure Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
19.6.9 The Square Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
19.6.10 The Circle Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
19.6.11 The Line Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
19.6.12 The Text Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
19.7 Reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
19.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201

Part III Computer Games

20 Introduction to Games Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
20.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
20.2 Games Frameworks and Libraries . . . . . . . . . . . . . . . . . . . . . . . . . 205
20.3 Python Games Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
20.4 Using Pygame . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
20.5 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
21 Building Games with Pygame . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
21.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
21.2 The Display Surface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
21.3 Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
21.3.1 Event Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
21.3.2 Event Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
21.3.3 The Event Queue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
Contents xix

21.4 A First pygame Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214

21.5 Further Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
21.6 A More Interactive pygame Application . . . . . . . . . . . . . . . . . . . . 219
21.7 Alternative Approach to Processing Input Devices . . . . . . . . . . . 221
21.8 pygame Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
21.9 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
22 StarshipMeteors Pygame . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
22.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
22.2 Creating a Spaceship Game . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
22.3 The Main Game Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
22.4 The GameObject Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
22.5 Displaying the Starship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
22.6 Moving the Spaceship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
22.7 Adding a Meteor Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
22.8 Moving the Meteors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
22.9 Identifying a Collision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
22.10 Identifying a Win . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
22.11 Increasing the Number of Meteors . . . . . . . . . . . . . . . . . . . . . . . . . 238
22.12 Pausing the Game . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
22.13 Displaying the Game Over Message . . . . . . . . . . . . . . . . . . . . . . . 239
22.14 The StarshipMeteors Game . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
22.15 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
22.16 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244

Part IV Testing
23 Introduction to Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
23.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
23.2 Types of Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
23.3 What Should Be Tested? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
23.4 Types of Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
23.4.1 Unit Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250
23.4.2 Integration Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
23.4.3 System Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
23.4.4 Installation/Upgrade Testing . . . . . . . . . . . . . . . . . . . . . 252
23.4.5 Smoke Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
23.5 Automating Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
23.6 Test-Driven Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
23.6.1 The TDD Cycle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
23.6.2 Test Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
23.6.3 Refactoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
23.7 Design for Testability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
23.7.1 Testability Rules of Thumb . . . . . . . . . . . . . . . . . . . . . . 255
23.8 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
23.9 Book Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
xx Contents

24 PyTest Testing Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257

24.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
24.2 What is PyTest? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
24.3 Setting up PyTest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
24.4 A Simple PyTest Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
24.5 Working with PyTest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
24.6 Parameterised Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
24.7 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
24.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
25 Mocking for Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
25.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
25.2 Why Mock? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
25.3 What is Mocking? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
25.4 Common Mocking Framework Concepts . . . . . . . . . . . . . . . . . . . 273
25.5 Mocking Frameworks for Python . . . . . . . . . . . . . . . . . . . . . . . . . . 274
25.6 The Unittest.Mock Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
25.6.1 Mock and Magic Mock Classes . . . . . . . . . . . . . . . . . . . 275
25.6.2 The Patchers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
25.6.3 Mocking Returned Objects . . . . . . . . . . . . . . . . . . . . . . . 277
25.6.4 Validating Mocks Have Been Called . . . . . . . . . . . . . . . 278
25.7 Mock and MagicMock Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
25.7.1 Naming Your Mocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
25.7.2 Mock Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
25.7.3 Attributes on Mock Classes . . . . . . . . . . . . . . . . . . . . . . 279
25.7.4 Mocking Constants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
25.7.5 Mocking Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
25.7.6 Raising Exceptions with Mocks . . . . . . . . . . . . . . . . . . . 281
25.7.7 Applying Patch to Every Test Method . . . . . . . . . . . . . 281
25.7.8 Using Patch as a Context Manager . . . . . . . . . . . . . . . . 281
25.8 Mock Where You Use It . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
25.9 Patch Order Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
25.10 How Many Mocks? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
25.11 Mocking Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
25.12 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
25.13 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284

Part V File Input/Output

26 Introduction to Files, Paths and IO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
26.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
26.2 File Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292
26.3 Paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293
26.4 File Input/Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295
Contents xxi

26.5 Sequential Access versus Random Access . . . . . . . . . . . . . . . . . . 296

26.6 Files and I/O in Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297
26.7 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297
27 Reading and Writing Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
27.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
27.2 Obtaining References to Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
27.3 Reading Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301
27.4 File Contents Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302
27.5 Writing Data to Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302
27.6 Using Files and with Statements . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
27.7 The Fileinput Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
27.8 Renaming Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
27.9 Deleting Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
27.10 Random Access Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
27.11 Directories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306
27.12 Temporary Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307
27.13 Working with Paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309
27.14 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312
27.15 Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313
28 Stream IO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
28.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
28.2 What is a Stream? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
28.3 Python Streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316
28.4 IOBase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317
28.5 Raw IO/UnBuffered IO Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
28.6 Binary IO/Buffered IO Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
28.7 Text Stream Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320
28.8 Stream Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321
28.9 Closing Streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322
28.10 Returning to the Open() Function . . . . . . . . . . . . . . . . . . . . . . . . . . 322
28.11 Online Resource . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
28.12 Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
29 Working with CSV Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
29.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
29.2 CSV Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
29.2.1 The CSV Writer Class . . . . . . . . . . . . . . . . . . . . . . . . . . . 326
29.2.2 The CSV Reader Class . . . . . . . . . . . . . . . . . . . . . . . . . . 327
29.2.3 The CSV DictWriter Class . . . . . . . . . . . . . . . . . . . . . . . 328
29.2.4 The CSV DictReader Class . . . . . . . . . . . . . . . . . . . . . . 329
29.3 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330
29.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330
xxii Contents

30 Working with Excel Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333

30.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333
30.2 Excel Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333
30.3 The Openpyxl. Workbook Class . . . . . . . . . . . . . . . . . . . . . . . . . . . 334
30.4 The Openpyxl. WorkSheet Objects . . . . . . . . . . . . . . . . . . . . . . . . . 334
30.5 Working with Cells . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334
30.6 Sample Excel File Creation Application . . . . . . . . . . . . . . . . . . . . 335
30.7 Loading a Workbook from an Excel File . . . . . . . . . . . . . . . . . . . . 336
30.8 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338
30.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338
31 Regular Expressions in Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341
31.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341
31.2 What Are Regular Expressions? . . . . . . . . . . . . . . . . . . . . . . . . . . . 341
31.3 Regular Expression Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342
31.3.1 Pattern Metacharacters . . . . . . . . . . . . . . . . . . . . . . . . . . 342
31.3.2 Special Sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343
31.3.3 Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344
31.4 The Python re Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345
31.5 Working with Python Regular Expressions . . . . . . . . . . . . . . . . . . 345
31.5.1 Using Raw Strings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345
31.5.2 Simple Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346
31.5.3 The Match Object . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346
31.5.4 The search() Function . . . . . . . . . . . . . . . . . . . . . . . . . . . 347
31.5.5 The match() Function . . . . . . . . . . . . . . . . . . . . . . . . . . . 348
31.5.6 The Difference Between Matching
and Searching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349
31.5.7 The finadall() Function . . . . . . . . . . . . . . . . . . . . . . . . . . 349
31.5.8 The finditer() Function . . . . . . . . . . . . . . . . . . . . . . . . . . 349
31.5.9 The split() Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350
31.5.10 The sub() Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350
31.5.11 The compile() Function . . . . . . . . . . . . . . . . . . . . . . . . . . 351
31.6 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353
31.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 354

Part VI Database Access

32 Introduction to Databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359
32.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359
32.2 What Is a Database? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359
32.2.1 Data Relationships . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 360
32.2.2 The Database Schema . . . . . . . . . . . . . . . . . . . . . . . . . . . 361
32.3 SQL and Databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363
32.4 Data Manipulation Language . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364
32.5 Transactions in Databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365
32.6 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366
Contents xxiii

33 Python DB-API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369

33.1 Accessing a Database from Python . . . . . . . . . . . . . . . . . . . . . . . . 369
33.2 The DB-API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369
33.2.1 The Connect Function . . . . . . . . . . . . . . . . . . . . . . . . . . . 370
33.2.2 The Connection Object . . . . . . . . . . . . . . . . . . . . . . . . . . 370
33.2.3 The Cursor Object . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371
33.2.4 Mappings from Database Types to Python Types . . . . 372
33.2.5 Generating Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372
33.2.6 Row Descriptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373
33.3 Transactions in PyMySQL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374
33.4 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375
34 PyMySQL Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377
34.1 The PyMySQL Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377
34.2 Working with the PyMySQL Module . . . . . . . . . . . . . . . . . . . . . . 377
34.2.1 Importing the Module . . . . . . . . . . . . . . . . . . . . . . . . . . . 378
34.2.2 Connect to the Database . . . . . . . . . . . . . . . . . . . . . . . . . 378
34.2.3 Obtaining the Cursor Object . . . . . . . . . . . . . . . . . . . . . . 379
34.2.4 Using the Cursor Object . . . . . . . . . . . . . . . . . . . . . . . . . 379
34.2.5 Obtaining Information About the Results . . . . . . . . . . . 380
34.2.6 Fetching Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 380
34.2.7 Close the Connection . . . . . . . . . . . . . . . . . . . . . . . . . . . 381
34.3 Complete PyMySQL Query Example . . . . . . . . . . . . . . . . . . . . . . 381
34.4 Inserting Data to the Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382
34.5 Updating Data in the Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384
34.6 Deleting Data in the Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 385
34.7 Creating Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386
34.8 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386
34.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387

Part VII Logging

35 Introduction to Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391
35.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391
35.2 Why Log? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391
35.3 What is the Purpose of Logging? . . . . . . . . . . . . . . . . . . . . . . . . . . 392
35.4 What Should You Log? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392
35.5 What not to Log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393
35.6 Why not Just Use Print? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394
35.7 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395
36 Logging in Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397
36.1 The Logging Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397
36.2 The Logger . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398
36.3 Controlling the Amount of Information Logged . . . . . . . . . . . . . . 399
36.4 Logger Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 401
xxiv Contents

36.5 Default Logger . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402

36.6 Module Level Loggers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403
36.7 Logger Hierarchy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404
36.8 Formatters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405
36.8.1 Formatting Log Messages . . . . . . . . . . . . . . . . . . . . . . . . 406
36.8.2 Formatting Log Output . . . . . . . . . . . . . . . . . . . . . . . . . . 406
36.9 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 408
36.10 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 408
37 Advanced Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409
37.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409
37.2 Handlers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409
37.2.1 Setting the Root Output Handler . . . . . . . . . . . . . . . . . . 411
37.2.2 Programmatically Setting the Handler . . . . . . . . . . . . . 412
37.2.3 Multiple Handlers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413
37.3 Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415
37.4 Logger Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 416
37.5 Performance Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 418
37.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419

Part VIII Concurrency and Parallelism

38 Introduction to Concurrency and Parallelism . . . . . . . . . . . . . . . . . . . . 423
38.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423
38.2 Concurrency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423
38.3 Parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425
38.4 Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 426
38.5 Grid Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427
38.6 Concurrency and Synchronisation . . . . . . . . . . . . . . . . . . . . . . . . . 428
38.7 Object Orientation and Concurrency . . . . . . . . . . . . . . . . . . . . . . . 428
38.8 Threads V Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 429
38.9 Some Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 430
38.10 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 430
39 Threading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433
39.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433
39.2 Threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433
39.2.1 Thread States . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433
39.2.2 Creating a Thread . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434
39.2.3 Instantiating the Thread Class . . . . . . . . . . . . . . . . . . . . 435
39.3 The Thread Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 436
39.4 The Threading Module Functions . . . . . . . . . . . . . . . . . . . . . . . . . . 438
39.5 Passing Arguments to a Thread . . . . . . . . . . . . . . . . . . . . . . . . . . . . 438
39.6 Extending the Thread Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 440
39.7 Daemon Threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441
Contents xxv

39.8 Naming Threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 442

39.9 Thread Local Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 442
39.10 Timers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444
39.11 The Global Interpreter Lock . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 445
39.12 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 446
39.13 Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 446
40 MultiProcessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 449
40.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 449
40.2 The Process Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 449
40.3 Working with the Process Class . . . . . . . . . . . . . . . . . . . . . . . . . . . 451
40.4 Alternative Ways to Start a Process . . . . . . . . . . . . . . . . . . . . . . . . 453
40.5 Using a Pool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 454
40.6 Exchanging Data Between Processes . . . . . . . . . . . . . . . . . . . . . . . 458
40.7 Sharing State Between Processes . . . . . . . . . . . . . . . . . . . . . . . . . . 460
40.7.1 Process Shared Memory . . . . . . . . . . . . . . . . . . . . . . . . . 460
40.8 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 461
40.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 462
41 Inter Thread/Process Synchronisation . . . . . . . . . . . . . . . . . . . . . . . . . . . 463
41.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463
41.2 Using a Barrier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463
41.3 Event Signalling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 467
41.4 Synchronising Concurrent Code . . . . . . . . . . . . . . . . . . . . . . . . . . . 469
41.5 Python Locks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 470
41.6 Python Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 472
41.7 Python Semaphores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 475
41.8 The Concurrent Queue Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 476
41.9 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 478
41.10 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 478
42 Futures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 481
42.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 481
42.2 The Need for a Future . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 481
42.3 Futures in Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 482
42.3.1 Future Creation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483
42.3.2 Simple Example Future . . . . . . . . . . . . . . . . . . . . . . . . . . 483
42.4 Running Multiple Futures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 485
42.4.1 Waiting for All Futures to Complete . . . . . . . . . . . . . . . 486
42.4.2 Processing Results as Completed . . . . . . . . . . . . . . . . . 488
42.5 Processing Future Results Using a Callback . . . . . . . . . . . . . . . . . 489
42.6 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 490
42.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 490
xxvi Contents

43 Concurrency with AsyncIO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 493

43.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 493
43.2 Asynchronous IO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 493
43.3 Async IO Event Loop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 494
43.4 The Async and Await Keywords . . . . . . . . . . . . . . . . . . . . . . . . . . . 495
43.4.1 Using Async and Await . . . . . . . . . . . . . . . . . . . . . . . . . . 496
43.5 Async IO Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 498
43.6 Running Multiple Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 500
43.6.1 Collating Results from Multiple Tasks . . . . . . . . . . . . . 500
43.6.2 Handling Task Results as They Are Made
Available . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 501
43.7 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 502
43.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 502
44 Performance Monitoring and Profiling . . . . . . . . . . . . . . . . . . . . . . . . . . 505
44.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 505
44.2 Why Monitor Performance and Memory? . . . . . . . . . . . . . . . . . . . 505
44.3 Performance Monitoring and Profiling . . . . . . . . . . . . . . . . . . . . . 506
44.4 Performance Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 507
44.4.1 The Time Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 507
44.4.2 The Timeit Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 508
44.4.3 The Psutil Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 510
44.5 Python Profiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 511
44.5.1 The cProfile Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . 511
44.5.2 The Line_Profiler Module . . . . . . . . . . . . . . . . . . . . . . . 511
44.5.3 The Memory_Profiler Module . . . . . . . . . . . . . . . . . . . . 512
44.5.4 Additional Third-Party Libraries . . . . . . . . . . . . . . . . . . 512
44.6 Profiling with cProfile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 512
44.7 Memory Profiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 515
44.8 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 517

Part IX Reactive Programming

45 Reactive Programming Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521
45.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521
45.2 What Is a Reactive Application? . . . . . . . . . . . . . . . . . . . . . . . . . . . 521
45.3 The ReactiveX Project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 522
45.4 The Observer Pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 522
45.5 Hot and Cold Observables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 523
45.6 Differences Between Event Driven Programming
and Reactive Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 524
45.7 Advantages of Reactive Programming . . . . . . . . . . . . . . . . . . . . . . 525
45.8 Disadvantages of Reactive Programming . . . . . . . . . . . . . . . . . . . 525
45.9 The RxPy Reactive Programming Framework . . . . . . . . . . . . . . . 526
45.10 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 526
Contents xxvii

46 RxPy Observables, Observers and Subjects . . . . . . . . . . . . . . . . . . . . . . 527

46.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 527
46.2 RxPy Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 527
46.3 Observables in RxPy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 527
46.4 Observers in RxPy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 528
46.5 Multiple Subscribers/Observers . . . . . . . . . . . . . . . . . . . . . . . . . . . 530
46.6 Subjects in RxPy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 531
46.7 Observer Concurrency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 534
46.7.1 Available Schedulers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 535
46.8 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 536
46.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 536
47 RxPy Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 537
47.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 537
47.2 Reactive Programming Operators . . . . . . . . . . . . . . . . . . . . . . . . . . 537
47.3 Piping Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 538
47.4 Creational Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 539
47.5 Transformational Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 539
47.6 Combinatorial Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 541
47.7 Filtering Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 542
47.8 Mathematical Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 543
47.9 Chaining Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 544
47.10 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 546
47.11 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 546

Part X Network Programming

48 Introduction to Sockets and Web Services . . . . . . . . . . . . . . . . . . . . . . . 551
48.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 551
48.2 Sockets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 551
48.3 Web Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 552
48.4 Addressing Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 552
48.5 Localhost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 553
48.6 Port Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 554
48.7 IPv4 Versus IPv6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 555
48.8 Sockets and Web Services in Python . . . . . . . . . . . . . . . . . . . . . . . 555
48.9 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 556
49 Sockets in Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 557
49.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 557
49.2 Socket to Socket Communication . . . . . . . . . . . . . . . . . . . . . . . . . . 557
49.3 Setting up a Connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 558
49.4 An Example Client Server Application . . . . . . . . . . . . . . . . . . . . . 558
49.4.1 The System Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . 558
49.4.2 Implementing the Server Application . . . . . . . . . . . . . . 559
xxviii Contents

49.4.3 Socket Types and Domains . . . . . . . . . . . . . . . . . . . . . . . 560

49.4.4 Implementing the Client Application . . . . . . . . . . . . . . 561
49.5 The Socketserver Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 562
49.6 Http Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 565
49.7 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 567
49.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 568
50 Web Services in Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 571
50.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 571
50.2 RESTful Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 571
50.3 A RESTful API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 572
50.4 Python Web Frameworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 573
50.5 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 574
51 Flask Web Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575
51.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575
51.2 Flask . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575
51.3 Hello World in Flask . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 576
51.3.1 Using JSON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 576
51.4 Implementing a Flask Web Service . . . . . . . . . . . . . . . . . . . . . . . . 577
51.4.1 A Simple Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 577
51.4.2 Providing Routing Information . . . . . . . . . . . . . . . . . . . 577
51.5 Running the Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 578
51.6 Invoking the RESTFul Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . 579
51.6.1 The Final Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 580
51.7 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 580
52 Flask Bookshop Web Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 583
52.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 583
52.2 Building a Flask Bookshop Service . . . . . . . . . . . . . . . . . . . . . . . . 583
52.3 The Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 583
52.4 The Domain Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 585
52.5 Encoding Books into JSON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 586
52.6 Setting Up the GET Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 588
52.7 Deleting a Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 589
52.8 Adding a New Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 590
52.9 Updating a Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 592
52.10 What Happens if We Get It Wrong? . . . . . . . . . . . . . . . . . . . . . . . . 593
52.11 Bookshop Services Listing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 594
52.12 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 596

Part XI Data Science: Data Analytics and Machine Learning

53 Introduction to Data Science . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 601
53.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 601
53.2 Data Science . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 601
53.3 Data Science Tools and Techniques . . . . . . . . . . . . . . . . . . . . . . . . 602
Contents xxix

53.4 Data Analytics Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 604

53.5 Python and Data Science . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 606
53.6 Machine Learning for Data Science . . . . . . . . . . . . . . . . . . . . . . . . 607
53.7 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 608
54 Pandas and Data Analytics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 611
54.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 611
54.2 The Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 611
54.2.1 The UK Government COVID Data Set . . . . . . . . . . . . . 611
54.2.2 The Google Mobility Data Set . . . . . . . . . . . . . . . . . . . . 613
54.3 Python Pandas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 614
54.3.1 Pandas Series and DataFrames . . . . . . . . . . . . . . . . . . . . 615
54.4 Loading and Analysing UK COVID Data Set . . . . . . . . . . . . . . . . 616
54.5 Loading the Google Mobility Data Set . . . . . . . . . . . . . . . . . . . . . 621
54.6 Merging Two DataFrames . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 622
54.7 Analysing the Combined Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 623
54.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 627
55 Alternatives to Pandas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 629
55.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 629
55.2 Comparing Pandas 2.0.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 629
55.3 Pandas 1.x v 2.x . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 629
55.4 Pandas Versus Other Libraries and Tools . . . . . . . . . . . . . . . . . . . . 630
55.5 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 632
56 Machine Learning in Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 633
56.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 633
56.2 The Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 633
56.3 SciKitLearn . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 634
56.4 The Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 635
56.5 Using Regression Supervised Learning Systems . . . . . . . . . . . . . 636
56.6 K-Nearest Neighbour Regressor . . . . . . . . . . . . . . . . . . . . . . . . . . . 636
56.7 Decision Tree Regressor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 638
56.8 Random Forest Regressor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 639
56.9 Summary of Metrics Obtained . . . . . . . . . . . . . . . . . . . . . . . . . . . . 640
56.10 Creating the Regressor Object . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 640
56.11 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 642
57 Pip and Conda Virtual Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . 643
57.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 643
57.2 Virtual Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 643
57.3 Working with Pip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 644
57.3.1 Activating a Pip Environment . . . . . . . . . . . . . . . . . . . . 644
57.3.2 Installing Modules Using Pip . . . . . . . . . . . . . . . . . . . . . 645
57.3.3 Deactivating a Pip Environment . . . . . . . . . . . . . . . . . . 646
57.3.4 Check Version of Pip . . . . . . . . . . . . . . . . . . . . . . . . . . . . 646
57.3.5 Installing Modules into a Pip Environment . . . . . . . . . 646
xxx Contents

57.3.6 Freezing Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 647

57.4 Conda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 648
57.5 Anaconda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 649
57.5.1 Installing Anaconda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 649
57.6 Working with Anaconda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 651
57.6.1 Checking the Conda Version . . . . . . . . . . . . . . . . . . . . . 651
57.6.2 Updating Conda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 651
57.6.3 Creating a Conda Environment . . . . . . . . . . . . . . . . . . . 652
57.6.4 Listing Available Conda Environments . . . . . . . . . . . . 653
57.6.5 Activating a Conda Environment . . . . . . . . . . . . . . . . . . 654
57.6.6 Deactivating a Conda Environment . . . . . . . . . . . . . . . . 655
57.6.7 Listing the Modules Loaded into a Conda
Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 655
57.6.8 Removing an Anaconda Environment . . . . . . . . . . . . . 656
57.6.9 Installing a Module into a Conda Environment . . . . . . 656
57.7 Anaconda in PyCharm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 658
57.8 Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 658
Chapter 1
Introduction

1.1 Introduction

I have heard many people over the years say that Python is an easy language to learn
and that Python is also a simple language.
To some extent both of these statements are true; but only to some extent.
While the core of the Python language is easy to lean and relatively simple (in
part thanks to its consistency), the sheer richness of the language constructs and
flexibility available can be overwhelming. In addition the Python environment, its
eco system, the range of libraries available, the often competing options available,
etc., can make moving to the next-level daunting.
Once you have learned the core elements of the language such as how classes
and inheritance work, how functions work, what are protocols and Abstract Base
Classes, etc. where do you go next?
The aim of this book is to delve into those next steps. The book is organised into
eleven different topics:
1. Advanced Language Features. The first section in the book covers topics
that are often missed out from introductory Python books such as slots, weak
references __init__() versus __new__() and metaclasses.
2. Computer Graphics. The book covers Computer Graphics and Computer
Generated Art in Python as well as graphical user interfaces and graphing/
charting via Matplotlib.
3. Games Programming. This topic is covered using the pygame library.
4. Testing and Mocking. Testing is an important aspect of any software develop-
ment; this book introduces testing in general and the PyTest module in detail.
It also considers mocking within testing including what and when to mock.

© Springer Nature Switzerland AG 2023 1

J. Hunt, Advanced Guide to Python 3 Programming,
Undergraduate Topics in Computer Science,
https://doi.org/10.1007/978-3-031-40336-1_1
2 1 Introduction

5. File Input/Output. The book covers text file reading and writing as well as
reading and writing CSV and Excel files. Although not strictly related to file
input, regulator expressions are included in this section as they can be used to
process textual data held in files.
6. Database Access. The book introduces databases and relational database in
particular. It then presents the Python DB-API database access standard and
one implementation of this standard, the PyMySQL module used to access a
MySQL database.
7. Logging. An often missed topic is that of logging. The book therefore introduces
logging the need for logging, what to log and what not to log as well as the Python
logging module.
8. Concurrency and Parallelism. The book provides extensive coverage of
concurrency topics including threads, processes and inter-thread or process
synchronisation. It also presents futures and AsyncIO.
9. Reactive Programming. This section of the book introduces Reactive Program-
ming using the PyRx Reactive Programming library.
10. Network Programming. The book introduces socket and web service commu-
nications in Python. It looks at both the Flask and the Django web service
libraries.
11. Data Analytics. A very hot topic for any potential Python programmer is data
analytics (and the related use of machine learning). The book concludes by
introducing these topics and there Pandas and scikit-learn (or SK-learn as it is
sometimes known) libraries.
Each section is introduced by a chapter providing the background and key concepts
of that topic. Subsequent chapters then cover various aspects of the topic.
For example, the second topic covered is on Computer Graphics. This section
has an introductory chapter on Computer Graphics in general. It then introduces the
Turtle Graphics Python library which can be used to generate a graphical display.
The following chapter considers the subject of Computer Generated Art and
uses the Turtle Graphics library to illustrate these ideas. Thus several examples
are presented that might be considered art. The chapter concludes by presenting the
well-known Koch Snowflake and the Mandelbrot Fractal set.
This is followed by a chapter presenting the Matplotlib library used for generating
2D and 3D charts and graphs (such as a line chart, bar chart or scatter graph).
The section concludes with a chapter on graphical user interfaces (or GUIs) using
the wxpython library. This chapter explores what we mean by a GUI and some of
the alternatives available in Python for creating a GUI.
Other topics follow a similar pattern.
Each programming or library-oriented chapter also includes numerous sample
programs that can be downloaded from the GitHub repository and executed. These
chapters also include one or more end of chapter exercises (with sample solutions
also in the GitHub repository).
1.2 Useful Python Resources 3

The topics within the book can be read mostly independently of each other. This
allows the reader to dip into subject areas as and when required. For example, the
File Input/Output section and the Database Access section can read independently
of each other (although in this case assessing both technologies may be useful in
selecting an appropriate approach to adopt for the long-term persistent storage of
data in a particular system).
Within each section there are usually dependencies; for example, it is neces-
sary to understand pygame library from the ‘Building Games with pygame’ intro-
ductory chapter, before exploring the worked case study presented by the chapter
on the StarshipMeteors game. Similarly it is necessary to have read the threading
and multiprocessing chapters before reading the inter-thread/process synchronisation
chapter.

1.2 Useful Python Resources

There are a wide range of resources on the web for Python; we will highlight a few
here that you should bookmark. We will not keep referring to these to avoid repetition
but you can refer back to this section whenever you need to:
• https://en.wikipedia.org/wiki/Python_Software_Foundation Python Software
Foundation.
• https://docs.python.org/3/ The main Python 3 documentation site. It contains
tutorials, library references, set up and installation guides as well as Python
how-tos.
• https://docs.python.org/3/library/index.html A list of all the built-in features for
the Python language—this is where you can find online documentation for the
various class and functions that we will be using throughout this book.
• https://pymotw.com/3/ the Python 3 Module of the week site. This site contains
many, many Python modules with short examples and explanations of what the
modules do. A Python module is a library of features that build on and expand
the core Python language. For example, if you are interested in building games
using Python then pygame is a module specifically designed to make this easier.
• https://www.fullstackpython.com/email.html is a monthly newsletter that
focusses on a single Python topic each month, such as a new library or module.
• http://www.pythonweekly.com/ is a free weekly summary of the latest Python
articles, projects, videos and upcoming events.
Each section of the book will provide additional online references relevant to the
topic being discussed.
 Part I
Advanced Language Features
Chapter 2
Python Type Hints

2.1 Introduction

Python is a dynamically typed language right—well yes it is however there is a

feature known as Type Hints that allows typing information to be provided when
functions and methods are defined. These Type Hints are extremely useful and can
help a developer understand what types are expected by a function or a method and
indeed what types are likely to be returned.
Recent versions of Python, including 3.10 and 3.11, have increased the support
for Type Hints so that they are now quite usable. These can be used by analysis tools
and IDEs to help developers create more stable and reliable applications.

2.2 Pythons Type System

Many people consider Python to be an untyped programming language. However,

that is not quite true. The type system in Python can be referred to as representing
a dynamically typed programming language. That is a variable holds a value and
the type of that value is known and understood by the language. At runtime Python
checks that what you are trying to do is valid given the types involved. For example,
you can use the type() function to find out what type of thing a variable holds at
any point in time:
a_variable = 42
print(f'a_variable type = {type(a_variable)} = {a_variable}')
a_variable = 1.345
print(f'a_variable type = {type(a_variable)} = {a_variable}')
a_variable = "Hello"
print(f'a_variable type = {type(a_variable)} = {a_variable}')
a_variable = True
print(f'a_variable type = {type(a_variable)} = {a_variable}')

© Springer Nature Switzerland AG 2023 7

J. Hunt, Advanced Guide to Python 3 Programming,
Undergraduate Topics in Computer Science,
https://doi.org/10.1007/978-3-031-40336-1_2
8 2 Python Type Hints

This produces as output:

a_variable type = <class 'int'> = 42
a_variable type = <class 'float'> = 1.345
a_variable type = <class 'str'> = Hello
a_variable type = <class 'bool'> = True

Of course, as the above shows, a variable in Python can hold different types of
things at different types, hence the term dynamically typed.
As Python knows what types variables hold it can check at runtime that your
programs are valid/correct given the types involved for example in a particular oper-
ation. Thus, it is valid to add two integers together and indeed two strings together
(as this is string concatenation) but attempting to add an integer to a string will result
in a TypeError:
print(1 + 1)
print(1.2 + 3.4)
print("Hello" + "world")
print("Hello" + 1)

This code produced the following output including the TypeError:

2
4.6
Helloworld
Traceback (most recent call last):
File "/Users/jeh/temp/pythonProjects/course/main.py", line 15,
in <module>
print("Hello" + 1)
TypeError: can only concatenate str (not "int") to str

2.3 The Challenge for Python Developers

The challenge for Python developers comes when they need to understand what types
are required by, or work with, some API. As a very simple example, consider the
following function:
def add(x, y):
return x + y

What types can be used with this function?

In essence any type can be used for the parameter x that supports the plus operator
(+) with the type in y. From the above we know that integers and strings can be used,
but we can also use floating point numbers, for example:
print(add(1, 2))
print(add(1.2, 3.4))
2.4 Static Typing 9

print(add(1, 3.4))
print(add(5.5, 1))
print(add("Hi", "There"))

All of the above are valid parameters, and the output produced from the above
code is:
3
4.6
4.4
6.5
HiThere

Even custom types can be used if they implement the special __add__(self,
other) operator method, for example:
class Quantity:
def __init__(self, amount):
self.amount = amount

def __add__(self, other):

return Quantity(self.amount + other.amount)

def __str__(self):
return f"Quantity({self.amount})"

q1 = Quantity(5)
q2 = Quantity(4)
print(add(q1, q2))

The __add__() method allows the custom type (class) being defined to be used
with the add operator (‘+’). Thus this program generates the following output:
Quantity(9)

However, what was the intent of the designer of this add() function? What did
they expect you to add together? The only option in traditional Python code is for
the developer to provide some form of documentation, for example in the form of a
docstring:
def add(x, y):
"""adds two integers together and
returns the resulting integer."""
return x + y

2.4 Static Typing

Languages such as Java, C# and C are statically typed languages. That is when a
variable, object attribute, parameter or return type is defined then the type of that
element is specified statically a compile time.
10 2 Python Type Hints

Thus, an add() method on a Java class Calculator might be written as

follows:
package com.jjh;

public class Calculator {

public int add(int x, int y) {

return x + y;
}

This makes it clear to a Java programmer and to the Java compiler that the add()
method will only handle integers and will return as a result an integer type. Thus,
there is no possibility that a developer might try to add a number to a Boolean value,
etc. In fact the compiler will not even allow it!
The Java Calculator class can be used as shown below, note that this code
will not even compile if the developer tries to add two strings together. In this case
we are adding two integers together, so all is fine:
package com.jjh;

public class App {

public static void main( String[] args ) {
System.out.println( "Starting" );
Calculator calc = new Calculator();
System.out.println(calc.add(4, 5));
System.out.println("Done");
}
}

As the above program uses valid integer types with the add() method, the output
from the compiled and executed program is:
Starting
9
Done

2.5 Python Type Hints

Python’s Type Hints are more like a half-way house between traditional Python’s
lack of typing information at all and the very strict string static typing approach of
languages such as Java.
A Type Hint is additional type information that can be used with a function
definition to indicate what types parameters should be and what type is returned.
This is illustrated below:
def add(x: int, y: int) -> int:
2.6 Type Hint Layout 11

return x + y

In this case it makes it clear that both x and y should be of type int (integer
types) and the returned result will be an int. However, adding Type Hints as shown
above has no effect on the runtime execution of the program; they are only hints and
are not enforced by Python per se. For example, it is still possible to pass a string
into the add() function as far as Python is concerned.
However, static analysis tools (such as MyPy) can be applied to the code to check
for such misuse. Some editors, such as the widely used PyCharm, already have such
tools integrated into their code checking behaviour.
If you want to use a tool such as mypy instead, or in addition to that available in
your IDE, then you can install it using
pip install mypy

Or if you want to use conda/Anaconda by using

conda install mypy

You can now analyse your code by applying MyPy to a Python file, for example:
% mypy main.py
main.py:3: error: Incompatible types in assignment (expression has
type "float", variable has type "int")
main.py:5: error: Incompatible types in assignment (expression has
type "str", variable has type "int")
main.py:24: error: Argument 1 to "add" has incompatible type "str";
expected "int"
main.py:24: error: Argument 2 to "add" has incompatible type "str";
expected "int"
main.py:44: error: Argument 1 to "add" has incompatible type
"Quantity"; expected "int"
main.py:44: error: Argument 2 to "add" has incompatible type
"Quantity"; expected "int"
Found 6 errors in 1 file (checked 1 source file)

2.6 Type Hint Layout

The Python Style Guide defined by Python Enhancement Proposal 8 (PEP 8) provides
some guidance for using Type Hints, for example:
• Use normal rules for colons, that is, no space before and one space after a colon:
text: str.
• Use spaces around the = sign when combining an argument annotation with a
default value: align: bool = True.
• Use spaces around the -> arrow: def headline(…) -> str.
12 2 Python Type Hints

2.7 Type Hints for Multiple Types

Of course our add() function could work with floating point numbers as well as it
works with integers. It would therefore be useful to be able to state this in terms of the
Type Hints. Prior to Python 3.10 this could be done using a Union type, for example
Union[int, float] which while it worked was a little unwieldy. Since Python
3.10 we can use the style syntax bar ‘|’ for example int | float as shown below:
def add(x: int | float, y: int | float) -> int:
return x + y

2.8 The Self Type

Python 3.11 introduced the Self type which is defined in PEP 673. This can be used
to indicate that a method returns a reference to itself, for example:
from typing import Self

class Shape:

def __init__(self):
self.scale = 0.0

def set_scale(self, scale: float) -> Self:

self.scale = scale
return self

2.9 The Benefits of Type Hints

There are a range of benefits to using Type Hints in Python, for example:
• They help catch some errors within programs. Obviously, the biggest benefit
is that Type Hints can help developers catch certain types of problems in their
code (assuming that some form of type checker is used).
• They provide documentation. Type Hints can also act as a level of document
that editors such as IDEs can pick up and display to other developers.
• They can be work with IDEs. They can help with code generation and IDE
auto-complete functionality.
• They can make developers stop and think. They can help ensure that developers
think about their code and what types should be supported.
• They can improve understanding of libraries. Although Type Hints may offer
little advantage in a single use script, or throw away program, they can be of
significant benefit when a library is being created. Such libraries will be used
2.11 Online Resources 13

by a range of different developers, and some may be released into the wild, for
example via PyPI, the Python Package Index. The use of Type Hints can greatly
enhance others understanding of the APIs provided by these libraries.

2.10 Summary

If you are just starting out with Python, or you are writing scripts that will only be used
once, then Type Hints may not be particularly useful. However, if you are creating
libraries or developing larger more complex applications with teams of developers,
then they can be very useful indeed.

2.11 Online Resources

• https://docs.python.org/3.10/ Python 3.10 documentation.

• https://docs.python.org/3.11/ Python 3.11 documentation.
• https://en.wikipedia.org/wiki/Type_system Wikipedia Type System page.
• https://en.wikipedia.org/wiki/Dynamic_programming_language Dynamically
typed languages.
• https://docs.python.org/3/library/exceptions.html Exception handling in Python.
• https://www.pythontutorial.net/python-basics/python-type-hints/ Tutorial on
Python Type Hints.
• https://pypi.org/project/mypy/ MyPy static type hint analysis tool.
• https://mypy.readthedocs.io/en/stable/ MyPy documentation.
• https://www.jetbrains.com/pycharm/ PyCharm IDE tool.
• https://peps.python.org/pep-0008/ Python (PEP 8) Style Guide including guid-
ance of how to layout Type Hints.
• https://peps.python.org/pep-0673/ Information on the Self type.
• https://pypi.org/ The Python package Index PyPi.
Another random document with
no related content on Scribd:
“And now,” he said, “I must be up and doing, although it is past two
o’clock. I must bid my mother good-bye, and I foresee there will be
no time to do it when once I have reported, which I promised to do
within twenty-four hours. By starting now I can reach Ferry Farm to-
morrow morning, spend an hour with her, and return here at night; so
if you, captain, will have my horses brought, I will wake up my boy
Billy”—for, although Billy was quite George’s age, he remained ever
his “boy.”
Next morning at Ferry Farm, about ten o’clock, Betty, happening to
open the parlor door, ran directly into George’s arms, whom she
supposed to be forty-five miles off. Betty was speechless with
amazement.
“Don’t look as if you had seen a rattlesnake, Betty,” cried George,
giving her a very cruel pinch, “but run, like a good child as you are,
though flighty, and tell our mother that I am here.”
Before Betty could move a step in marched Madam Washington,
stately and beautiful as ever. And there were the three boys, all
handsome youths, but handsomer when they were not contrasted
with the elder brother; and then, quite gayly and as if he were a mere
lad, George plunged into his story, telling his mother that he was to
make the campaign with General Braddock as first aide-de-camp,
and trying to tell her about the officers’ letter, which he took from his
pocket, but, blushing very much, was going to return it, had not Betty
seized it as with an eagle’s claw.
“Betty,” cried George, stamping his foot, “give me back that letter!”
“No, indeed, George,” answered Betty, with calm disdain. “Do not put
on any of your grand airs with me. I have heard of this letter, and I
mean to read it aloud to our mother. And you may storm and stamp
and fume all you like—’tis not of the slightest consequence.”
So George, scowling and yet forced to laugh a little, had to listen to
all the compliments paid him read out in Betty’s rich, ringing young
voice, while his mother sat and glowed with pride, and his younger
brothers hurrahed after the manner of boys; and when Betty had got
through the letter her laughing face suddenly changed to a very
serious one, and she ran to George and kissed him all over his
cheeks, saying:
“Dear George, it makes me so happy that I both want to laugh and
cry—dear, dear brother!”
And George, with tender eyes, kissed Betty in return, so that she
knew how much he loved her.
When Madam Washington spoke it was in a voice strangely different
from her usually calm, musical tones. She had just got the idol of her
heart back from all his dangers, and she was loath to let him go
again, and told him so.
“But, mother,” answered George, after listening to her respectfully,
“when I started upon my campaign last year you told me that you
placed me in God’s keeping. The God to whom you commended me
then defended me from all harm, and I trust He will do so now. Do
not you?”
Madam Washington paused, and the rare tears stole down her
cheeks.
“You are right, my son,” she answered, presently. “I will not say
another word to detain you, but will once more give you into the
hands of the good God to take care of for me.”
That night, before twelve o’clock, George reported at Alexandria to
General Braddock as his aide.
On the 20th of April, near the time that George had set out the year
before, General Braddock began his march from Alexandria in
Virginia to the mountains of Pennsylvania, where the reduction of
Fort Duquesne was his first object. There were two magnificent
regiments of crack British troops and ten companies of Virginia
troops, hardy and seasoned, and in the highest spirits at the
prospect of their young commander being with them. They cheered
him vociferously when he appeared riding with General Braddock,
and made him blush furiously. But his face grew very long and
solemn when he saw the immense train of wagons to carry baggage
and stores which he knew were unnecessary, and the general at that
very moment was storming because there were not more.
“These,” he said, “were furnished by Mr. Franklin, Postmaster-
General of Pennsylvania, and he sends me only a hundred and fifty
at that.”
“A hundred too many,” was George’s thought.
The march was inconceivably slow. Never since George could
remember had he so much difficulty in restraining his temper as on
that celebrated march. As he said afterwards, “Every mole-hill had to
be levelled, and bridges built across every brook.” General Braddock
wished to march across the trackless wilderness of the Alleghanies
as he did across the flat plains of Flanders, and he spent his time in
constructing a great military road when he should have been pushing
ahead. So slow was their progress that in reaching Winchester
George was enabled to make a détour and go to Greenway Court for
a few hours. The delight of Lord Fairfax and Lance was extreme, but
in a burst of confidence George told them the actual state of affairs.
“What you tell me,” said the earl, gravely, “determines me to go to
the low country, for if this expedition results disastrously I can be of
more use at Williamsburg than here. But, my dear George, I am
concerned for you, because you look ill. You are positively gaunt,
and you look as if you had not eaten for a week.”
“Ill!” cried George, beginning to walk up and down the library, and
clinching and unclinching his lists nervously. “My lord, it is my heart
and soul that are ill. Can you think what it is to watch a general,
brave but obstinate, and blind to the last degree, rushing upon
disaster? Upon my soul, sir, those English officers think, I verily
believe, that the Indians are formed into regiments and battalions,
with a general staff and a commissary, and God knows what!” And
George raved a while longer before he left to ride back to
Winchester, with Billy riding after him. This outbreak was so unlike
George, he looked so strange, his once ruddy face was so pallid at
one moment and so violently flushed at another that the earl and
Lance each felt an unspoken dread that his strong body might give
way under the strain upon it.
George galloped back into Winchester that night. Both his horse and
Billy’s were dripping wet, and as he pulled his horse almost up on his
haunches Billy said, in a queer voice:
“Hi, Marse George, d’yar blood on yo’ bridle. You rid dat boss hard,
sho’ nough!”
“Hold your tongue!” shouted George, in a tone that Billy had never
heard from him before; and then, in the next minute, he said,
confusedly, “I did not mean to speak so, but my head is in a whirl; I
think I must be ill.”
And as he spoke he reeled in his saddle, and would have fallen had
not Billy run forward and caught him. He staggered into the house
where he had lodgings, and got into his bed, and by midnight he was
raving with fever.
Billy had sense enough to go for Dr. Craik, George’s old
acquaintance, who had volunteered as surgeon to General
Braddock’s staff. He was a bright-eyed, determined-looking man, still
young, but skilled in his profession. By morning the fever was
reduced, and Dr. Craik was giving orders about the treatment as he
sat by George’s bedside, for the army was to resume its march that
day.
“Your attack is sharp,” said the doctor, “but you have an iron
constitution, and with ordinary care you will soon be well.”
George, pale and haggard, but without fever, listened to the doctor’s
directions with a half-smile. The troops were already on the move;
outside could be heard the steady tramp of feet, the thunder of
horses’ hoofs, the roll of artillery-wagons, and the commotion of an
army on the move. In a few moments the doctor left him, saying:
“I think you will shortly be able to rejoin the army, Colonel
Washington.”
“I think so, too,” answered George.
As soon as the doctor was out of the room George turned to Billy
and said:
“Help me on with my clothes, and as soon as the troops are well out
of the town fetch the horses.”
When the soldiers halted at noon, General Braddock, sitting under a
tree by the road-side, was asking Dr. Craik’s opinion of the time that
Colonel Washington could rejoin, when around the corner of a huge
bowlder rode George with Billy behind him. He was very pale, but he
could sit his horse. He could not but laugh at the doctor’s angry face,
but said deprecatingly to him:
“I would have fretted myself more ill had I remained at Winchester,
for I am not by nature patient, and I have been ill so little that I do not
know how to be ill.”
“I see you don’t,” was the doctor’s dry reply.
For four days George kept up with the army, and managed, in spite
of burning fevers, of a horrible weakness and weariness, of
sleepless nights racked with pain, to ride his horse. On the fifth he
was compelled to take to a covered wagon. There, on a rough bed,
with Billy holding his burning head, he was jolted along for ten days
more, each day more agonizing than the one before. In that terrible
time master and man seemed to have changed places. It was
George who was fretful and unreasonable and wildly irritable, while
Billy, the useless, the lazy, the incorrigible, nursed him with a
patience, a tenderness, a strange intelligence that amazed all who
saw it, and was even dimly felt by George. The black boy seemed
able to do altogether without sleep. At every hour of the day and
night he was awake and alert, ready to do anything for the poor
sufferer. As the days passed on, and George grew steadily worse,
the doctor began to look troubled. In his master’s presence Billy
showed no sign of fear, but he would every day follow Dr. Craik when
he left, and ask him, with an ashy face:
“Marse doctor, is Marse George gwi’ die?”
“I hope not. He is young and strong, and God is good.”
“Ef he die, an’ I go home, what I gwi’ say when mistis come out and
say, ‘Billy, wh’yar yo’ Marse George?’” And at that Billy would throw
himself on the ground in a paroxysm of grief that was piteous to see.
The doctor carefully concealed from the soldiers George’s real
condition. But after four or five days of agony a change set in, and
within the week George was able to sit up and even ride a little. The
wagons had kept with the rear division of the army, but George knew
that General Braddock, with twelve hundred picked men, had gone
ahead and must be near Fort Duquesne. On the fourteenth day, in
the evening, when the doctor came he found his patient walking
about. He was frightfully thin and pale, but youth and strength and
good habits were beginning to assert themselves. He was getting
well.
“Doctor,” said he, “this place is about fifteen miles from Fort
Duquesne. I know it well, and from this hour I emancipate myself
from you. This day I shall report for duty.”
The next morning, the 9th of July, 1755, dawned beautifully, and the
first long lances of light revealed a splendid sight on the banks of the
Monongahela. On one side flowed the great river in majestic beauty.
Following the shores was a kind of natural esplanade, while a little
way off the rich woods, within which dwelt forever a purple twilight,
overhung this charming open space. And along this open space
marched, in exquisite precision, two thousand of England’s crack
troops—cavalry, infantry, and artillery—and a thousand bronzed
Virginia soldiery to the music of the fife and drum. Often in after-
years George Washington was heard to say that the most beautiful
sight his eyes ever rested on was the sight of Braddock’s army at
sunrise on that day of blood. Officers and men were in the highest
spirits; they expected within a few hours to be in sight of Fort
Duquesne, where glory, as they thought, awaited their coming. Even
George’s apprehensions of the imprudence of this mode of attack
were soothed. He rode by General Braddock’s side, and was by far
the most conspicuous figure there for grace and nobility. His illness
seemed to have departed in a night, and he was the same erect,
soldierly form, fairly dwarfing every one contrasted with him. As the
general and his first aide rode together, General Braddock said,
confidently:
“Colonel Washington, in spite of your warning, see how far we have
come upon our way without disaster. The danger of an attack by
Indians is now passed, and we have but to march a few miles more
and glory is ours.”
Scarcely were the words out of his mouth when there was one sharp
crack of a gun, followed by a fierce volley, and fifty men dropped in
their tracks. But there was no enemy visible. The shots were like a
bolt of lightning from a clear sky.
“The Indians,” said George, in a perfectly composed voice, reining
up his horse.
“I see no Indians,” cried General Braddock, excitedly. “There is
disorder in the ranks; have them closed up at once, and march in
double time. We will soon find the enemy.”
But the firing from the invisible foe again broke forth, this time fiercer
and more murderous than before. General Braddock, riding to the
head of the first regiment, which had begun to waver, shouted the
order for them to reform and fire. The veteran troops, as coolly as if
on parade, closed up their ranks, and gave a volley, but it was as if
fired in the air. They saw no enemy to fire at. Meanwhile the Virginia
troops, after the first staggering effect of a terrific musketry fire
poured into them by an unseen enemy, suddenly broke ranks, and,
each man running for a tree, they took possession of the skirts of the
woods. On seeing this General Braddock galloped up to George.
“Colonel Washington,” he cried, violently, “your Virginia troops are
insubordinate! They have scattered themselves through the woods,
and I desire them formed again in columns of fours to advance.”
“Sir,” answered George, in agony, “the ravines are full of Indians—
many hundreds of them. They can slaughter us at their pleasure if
we form in the open. The Virginians know how to fight them.”
“You are an inexperienced soldier, sir, and therefore I excuse you.
But look at my English veterans—see how they behave—and I
desire the Virginians to do the same.”
At that moment George’s horse fell upon his knees, and the next he
rolled over, shot through the heart. The English regiments had
closed up manfully, after receiving several destructive volleys,
returning the fire of their assailants without seeing them and without
producing the smallest effect. But suddenly the spectacle of half their
men dead or wounded on the ground, the galloping about of
riderless horses, the shrieks of agony that filled the air, seemed to
unman them. They broke and ran in every direction. In vain General
Braddock rode up to them, actually riding over them, waving his
sword and calling to them to halt.
The men who had faced the legions of Europe were panic-stricken
by this dreadful unseen foe, and fled, only to be shot down in their
tracks. To make it more terrible, the officers were singled out for
slaughter, and out of eighty-six officers in a very little while twenty-six
were killed and thirty-seven wounded. General Braddock himself had
his horse shot under him, and as he rolled on the ground a cry of
pain was wrung from him by two musket-balls that pierced his body.
Of his three aides, two lay weltering in their blood, and George alone
was at his side helping him to rise.
Rash and obstinate as General Braddock might be, he did not lack
for courage, and in that awful time he was determined to fight to the
last.
“Get me another horse,” he said, with difficulty, to George. “Are you,
too, wounded?”
“No, general, but I have had two horses shot under me. Here is my
third one—mount!” And by the exertion of an almost superhuman
strength he raised General Braddock’s bulky figure from the ground
and placed him in the saddle.
“I am badly wounded,” said General Braddock, as he reeled slightly;
“but I can sit my horse yet. Your Virginians are doing nobly, but they
should form in columns.”
Besotted to the end, but seeing that the Virginians alone were
standing their ground, General Braddock did not give a positive
order, and George did not feel obliged to obey this murderous
mistake. But General Braddock, after a gasp or two, turned a livid
face towards George.
“Colonel Washington, the command is yours. I am more seriously
wounded than I thought.” He swayed forward, and but for George
would have fallen from his horse.
The panic was now at its height. Men, horses, wagons, all piled
together in a terrible mêlée, made for the rear; but there, again, they
were met by a hail of bullets. Staggered, they rushed back, only to
be again mowed down by the hidden enemy. The few officers left
commanded, begged, and entreated the men to stand firm; but they,
who had faced death upon a hundred fields, were now so mad with
fear that they were incapable of obedience. George, who had
managed to have General Braddock carried to the rear with the aid
of Dr. Craik, had got another horse, and riding from one end of the
bloody field to the other, did all that mortal man could do to rally the
panic-stricken men, but it was in vain. His clothes were riddled with
bullets, but in the midst of the carnage around him he was
unharmed, and rode over the field like the embodied spirit of battle.
The Virginians alone, cool and determined, fought steadily and sold
their lives dearly, although picked off one by one. At last, after hours
of panic, flight, and slaughter, George succeeded in bringing off the
remnant of the Virginians, and, overtaking the fleeing mob of regular
troops some miles from the scene of the conflict, got them across the
ford of the Monongahela. They were safe from pursuit, for the
handful of Frenchmen could not persuade their Indian allies to leave
the plunder of the battle-field for the pursuit of the enemy. The first
thing that George did was to send immediately for wagons, which
had been left behind, to transport the wounded. General Braddock,
still alive but suffering agonies from his wounds, was carried on
horseback, then in a cart, and at last, the jolting being intolerable, on
a litter upon the shoulders of four sturdy backwoodsmen. But he was
marked for death. On the third day of this terrible retreat, towards
sunset, he sank into a lethargy. George, who had spent every
moment possible by his side, turned to Dr. Craik, who shook his
head significantly—there was no hope. As George dismounted and
walked by the side of the litter, the better to hear any words the dying
soldier might utter, General Braddock roused a little.
“Colonel Washington,” he said, in a feeble voice, “I am satisfied with
your conduct. We have had bad fortune—very bad fortune; but”—
here his mind began to wander—“yonder is the smoke rising from
the chimneys; we shall soon be home and at rest. Good-night,
Colonel Washington—”

“GEORGE DID ALL THAT MORTAL MAN COULD DO”

The men with the litter stopped. George, with an overburdened
heart, watched the last gasp of a rash but brave and honorable
soldier, and presently gently closed his eyes. That night the body of
General Braddock, wrapped in his military cloak, was buried under a
great oak-tree in the woods by the side of the highway, and before
daylight the mournful march was resumed.
The news of the disaster had preceded them, and when George,
attended only by Captain Vanbraam and a few of his Virginia officers,
rode into Williamsburg on an August evening, it was with the
heaviest heart he had ever carried in his bosom. But by one of those
strange paradoxes, ever existing in the careers of men of destiny,
the events that had brought ruin to others only served to exalt him
personally. His gallant conduct in battle, his miraculous escape, his
bringing off of the survivors, especially among the Virginia troops,
and the knowledge which had come about that had his advice been
heeded the terrible disaster would not have taken place—all
conspired to make him still more of a popular hero. Not only his own
men adored him, and pointed out that he had saved all that could be
saved on that dreadful day, but the British troops as well saw that the
glory was his, and the return march was one long ovation to the one
officer who came out of the fight with a greater reputation than when
he entered it. Everywhere crowds met him with acclamations and
with tears. The streets of the quaint little town of Williamsburg were
filled with people on this summer evening, who followed the party of
officers, with George at their head, to the palace. George responded
to the shouts for him by bowing gracefully from side to side.
Arrived at the palace he dismounted, and, just as the sentry at the
main door presented arms to him, he saw a party coming out, and
they were the persons he most desired to see in the world and least
expected. First came the Earl of Fairfax with Madam Washington,
whom he was about to hand down the steps and into his coach,
which had not yet driven up. Behind them demurely walked Betty,
and behind Betty came Lance, carrying the mantles of the two
ladies.
The earl and Madam Washington, engaged in earnest conversation,
did not catch sight of George until Betty had rushed forward, and
crying out in rapture, “George, dear George!” they saw the brother
and sister clasped in each other’s arms.
Madam Washington stood quite still, dumfounded with joy, until
George, kissing her hand tenderly, made her realize that it was
indeed he, her best beloved, saved from almost universal destruction
and standing before her. She, the calmest, the stateliest of women,
trembled, and had to lean upon him for support; the earl grasped his
hand; Lance was in waiting. George was as overcome with joy as
they were.
“But I must ask at once to see the governor,” said he, after the first
rapture of meeting was over. “You, my lord, must go with me, for I
want friends near me when I tell the story of sorrow and disaster.”
Four days afterwards, the House of Burgesses being in session,
Colonel Washington was summoned by the Speaker to read his
report of the campaign before it, and to be formally designated as
commander-in-chief of the forces. The facts were already known, but
it was thought well, in order to arouse the people to the sense of
their danger, and to provide means for carrying on the war in
defence of their frontiers, that Colonel Washington should make a
public report, and should publicly receive the appointment of
commander-in-chief of the next expedition. The House of Burgesses,
then as ever, proudly insistent of its rights, had given the governor to
understand that they would give him neither money nor supplies
unless their favorite soldier should have the command in the next
campaign—and, indeed, the attitude of the officers and soldiery
made this absolutely necessary. Even the governor had realized this,
and, disheartened by the failure of his much-heralded regulars, was
in a submissive mood, and these haughty colonial legislators, of
whose republican principles Governor Dinwiddie already complained
much, took this opportunity to prove that without their co-operation
but little could be done.
The large hall of the House of Burgesses, but dimly lighted by small
diamond-paned windows, was filled with the leading men of the
colony, including Lord Fairfax. Ladies had been admitted to the floor,
and among them sat in majestic beauty Madam Washington, and
next to her sat Betty, palpitating, trembling, blushing, and with proud,
bright eyes awaited the entrance of her “darling George,” as she
called him, although often reproved for her extravagance by her
mother.
At last George entered this august assembly. His handsome head
was uncovered, showing his fair hair. He wore a glittering uniform,
and his sword, given him by Lord Fairfax, hung at his side. He
carried himself with that splendid and noble air that was always his
characteristic, and, quietly seating himself, awaited the interrogatory
of the president. When this was made he rose respectfully and
began to read from manuscript the sad story of Braddock’s
campaign. It was brief, but every sentence thrilled all who heard it.
When he said, in describing the terrible story of the 9th of July, “The
officers in general behaved with incomparable bravery, for which
they suffered, upwards of sixty being killed or wounded,” a kind of
groan ran through the great assemblage; and when he added, in a
voice shaken with emotion, “The Virginia companies behaved like
men and died like soldiers; for, I believe, out of three companies on
the ground that day scarce thirty men were left alive,” sobs were
heard, and many persons, both men and women, burst into tears.
His report being ended, the president of the House of Burgesses
arose and addressed him:
“Colonel Washington: We have listened to your account of the late
campaign with feelings of the deepest and most poignant sorrow, but
without abandoning in any way our intention to maintain our lawful
frontiers against our enemies. It has been determined to raise
sixteen companies in this colony for offensive and defensive warfare,
and by the appointment of his excellency the governor, in deference
to the will of the people and the desire of the soldiers, you are
hereby appointed, by this commission from his excellency the
governor, which I hold in my hand, commander-in-chief of all the
forces now raised or to be raised by this colony, reposing special
confidence in your patriotism, valor, conduct, and fidelity. And you
are hereby invested with power and authority to act as you shall
think for the good of the service.
“And we hereby strictly charge all officers and soldiers under your
command to be obedient to your orders and diligent in the exercise
of their several duties.
“And we also enjoin and require you to be careful in executing the
great trust reposed in you, by causing strict discipline and order to be
observed in the army, and that the soldiers be duly exercised and
provided with all necessaries.
“And you are to regulate your conduct in every respect by the rules
and discipline of war, and punctually to observe and follow such
orders and directions as you shall receive from his excellency the
governor and this or other House of Burgesses, or committee of the
House of Burgesses.”
A storm of applause broke forth, and George stood silent, trembling
and abashed, with a noble diffidence. He raised one hand in
deprecation, and it was taken to mean that he would speak, and a
solemn hush fell upon the assembly. But in the perfect silence he felt
himself unable to utter a word, or even to lift his eyes from the floor.
The president sat in a listening attitude for a whole minute, then he
said:
“Sit down, Colonel Washington. Your modesty is equal to your valor,
and both are above comparison. Your life would not have been
spared, as if by a miracle, had not the All-wise Ruler of the heavens
and the earth designed that you should fulfil your great destiny; and
one day, believe me, you shall be called the prop, the stay, and the
glory of your country.”
THE END
 FOOTNOTES:
[A] In Washington’s will he mentions “my man
William, calling himself William Lee,” and gives
him his freedom, along with the other slaves, and
an annuity besides: “and this I give him as a
testimony of my sense of his attachment to me,
and for his faithful services during the
Revolutionary War.”
[B] Washington, in his journal, speaks of the
Indian firing at him at short range, but says
nothing of his preventing his companion from
killing the would-be murderer. But his companion
expressly says that he would have killed the
Indian on the spot had not Washington forbidden
him. The Indians became very superstitious
about Washington’s immunity from bullets,
especially after Braddock’s defeat. In that battle
he was the target for the best marksmen among
them, and not only escaped without a scratch,
although two horses were killed under him and
his clothes riddled with bullets, but he was the
only officer of Braddock’s military family who
survived.
[C] This letter, which is printed in full in Marshall’s
Life of Washington, was among the highest
personal compliments ever paid Washington. The
signers were seasoned soldiers, addressing a
young man of twenty-three, under whom they
had made a campaign of frightful hardship ending
in disaster. They were to be ordered to resume
operations in the spring, and it was to this young
man that these officers appealed, believing him to
be essential to the proper conduct of the
campaign.
 By JAMES BARNES
A LOYAL TRAITOR. A Story of the War of 1812.
Illustrated. Post 8vo, Cloth, Ornamental, $1 50.
A vigorous romance it is, full of life and adventure.—
N. Y. Herald.
A stirring story.... The several characters are finely
balanced and well drawn, and are admirably
interwoven to give reasonableness and
completeness in every chapter.—Chicago Inter-
Ocean.
We turn with a feeling of relief to Mr. Barnes’s
breezy and Marryat-like story of the American
privateersman.... This is all genuine romance, with a
wide horizon and many changes of scene.... Mr.
Barnes has done his work well.—N. Y. Times.
An excellent and intensely interesting story of
romance and adventure.—Brooklyn Standard-Union.
The book has freshness, animation, and strong
story-interest.—Outlook, N. Y.
A quick-moving, picturesque story. The daring
Yankee sailor of that naval war, his life above deck
and below, in battle and in cruise, his dialect and
yarns, are all quaintly reproduced in a series of vivid
scenes.—Philadelphia Record.

FOR KING OR COUNTRY. A Story of the American

Revolution. Illustrated. Post 8vo, Cloth, Ornamental, $1
50.

NAVAL ACTIONS OF THE WAR OF 1812. Illustrated. 8vo,

Cloth, Ornamental, $4 50.
 NEW YORK AND LONDON
HARPER & BROTHERS, Publishers
☞ The above works are for sale by all booksellers, or will
be sent by mail by the publishers, postage prepaid, on
receipt of the price.
 TRANSCRIBER’S NOTES:
Obvious typographical errors have been corrected.
Inconsistencies in hyphenation have been
standardized.
Archaic or variant spelling has been retained.
New original cover art included with this eBook is
granted to the public domain.
*** END OF THE PROJECT GUTENBERG EBOOK A VIRGINIA
CAVALIER ***

Updated editions will replace the previous one—the old editions

will be renamed.

Creating the works from print editions not protected by U.S.

copyright law means that no one owns a United States copyright
in these works, so the Foundation (and you!) can copy and
distribute it in the United States without permission and without
paying copyright royalties. Special rules, set forth in the General
Terms of Use part of this license, apply to copying and
distributing Project Gutenberg™ electronic works to protect the
PROJECT GUTENBERG™ concept and trademark. Project
Gutenberg is a registered trademark, and may not be used if
you charge for an eBook, except by following the terms of the
trademark license, including paying royalties for use of the
Project Gutenberg trademark. If you do not charge anything for
copies of this eBook, complying with the trademark license is
very easy. You may use this eBook for nearly any purpose such
as creation of derivative works, reports, performances and
research. Project Gutenberg eBooks may be modified and
printed and given away—you may do practically ANYTHING in
the United States with eBooks not protected by U.S. copyright
law. Redistribution is subject to the trademark license, especially
commercial redistribution.

START: FULL LICENSE

THE FULL PROJECT GUTENBERG LICENSE