Reinforcement Learning: With Open AI, TensorFlow

and Keras Using Python 1st Edition Abhishek

Nandy download

https://textbookfull.com/product/reinforcement-learning-with-
open-ai-tensorflow-and-keras-using-python-1st-edition-abhishek-
nandy/

Download more ebook from https://textbookfull.com

We believe these products will be a great fit for you. Click
the link to download now, or visit textbookfull.com
to discover even more!

Applied Reinforcement Learning with Python: With OpenAI

Gym, Tensorflow, and Keras Beysolow Ii

https://textbookfull.com/product/applied-reinforcement-learning-
with-python-with-openai-gym-tensorflow-and-keras-beysolow-ii/

Deep Learning with Python Develop Deep Learning Models

on Theano and TensorFLow Using Keras Jason Brownlee

https://textbookfull.com/product/deep-learning-with-python-
develop-deep-learning-models-on-theano-and-tensorflow-using-
keras-jason-brownlee/

Deep Learning Projects Using TensorFlow 2: Neural

Network Development with Python and Keras 1st Edition
Vinita Silaparasetty

https://textbookfull.com/product/deep-learning-projects-using-
tensorflow-2-neural-network-development-with-python-and-
keras-1st-edition-vinita-silaparasetty/

Deep Learning with Applications Using Python Chatbots

and Face, Object, and Speech Recognition With
TensorFlow and Keras Springerlink (Online Service)

https://textbookfull.com/product/deep-learning-with-applications-
using-python-chatbots-and-face-object-and-speech-recognition-
with-tensorflow-and-keras-springerlink-online-service/
Beginning Anomaly Detection Using Python-Based Deep
Learning: With Keras and PyTorch Sridhar Alla

https://textbookfull.com/product/beginning-anomaly-detection-
using-python-based-deep-learning-with-keras-and-pytorch-sridhar-
alla/

Computer Vision Using Deep Learning Neural Network

Architectures with Python and Keras 1st Edition Vaibhav
Verdhan

https://textbookfull.com/product/computer-vision-using-deep-
learning-neural-network-architectures-with-python-and-keras-1st-
edition-vaibhav-verdhan/

Machine Learning Concepts with Python and the Jupyter

Notebook Environment: Using Tensorflow 2.0 Nikita
Silaparasetty

https://textbookfull.com/product/machine-learning-concepts-with-
python-and-the-jupyter-notebook-environment-using-
tensorflow-2-0-nikita-silaparasetty/

Natural language processing with TensorFlow Teach

language to machines using Python s deep learning
library 1st Edition Thushan Ganegedara

https://textbookfull.com/product/natural-language-processing-
with-tensorflow-teach-language-to-machines-using-python-s-deep-
learning-library-1st-edition-thushan-ganegedara/

Building an Enterprise Chatbot: Work with Protected

Enterprise Data Using Open Source Frameworks Abhishek
Singh

https://textbookfull.com/product/building-an-enterprise-chatbot-
work-with-protected-enterprise-data-using-open-source-frameworks-
abhishek-singh/
Reinforcement
Learning
With Open AI, TensorFlow and
Keras Using Python
—
Abhishek Nandy
Manisha Biswas
 Reinforcement
Learning
With Open AI, TensorFlow and
Keras Using Python

Abhishek Nandy
Manisha Biswas
Reinforcement Learning
Abhishek Nandy Manisha Biswas
Kolkata, West Bengal, India North 24 Parganas, West Bengal, India
ISBN-13 (pbk): 978-1-4842-3284-2 ISBN-13 (electronic): 978-1-4842-3285-9
https://doi.org/10.1007/978-1-4842-3285-9
Library of Congress Control Number: 2017962867
Copyright © 2018 by Abhishek Nandy and Manisha Biswas
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole
or part of the material is concerned, specifically the rights of translation, reprinting, reuse of
illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical
way, and transmission or information storage and retrieval, electronic adaptation, computer
software, or by similar or dissimilar methodology now known or hereafter developed.
Trademarked names, logos, and images may appear in this book. Rather than use a trademark
symbol with every occurrence of a trademarked name, logo, or image we use the names, logos,
and images only in an editorial fashion and to the benefit of the trademark owner, with no
intention of infringement of the trademark.
The use in this publication of trade names, trademarks, service marks, and similar terms, even if
they are not identified as such, is not to be taken as an expression of opinion as to whether or not
they are subject to proprietary rights.
While the advice and information in this book are believed to be true and accurate at the
date of publication, neither the authors nor the editors nor the publisher can accept any legal
responsibility for any errors or omissions that may be made. The publisher makes no warranty,
express or implied, with respect to the material contained herein.
Cover image by Freepik (www.freepik.com)
Managing Director: Welmoed Spahr
Editorial Director: Todd Green
Acquisitions Editor: Celestin Suresh John
Development Editor: Matthew Moodie
Technical Reviewer: Avirup Basu
Coordinating Editor: Sanchita Mandal
Copy Editor: Kezia Endsley
Compositor: SPi Global
Indexer: SPi Global
Artist: SPi Global
Distributed to the book trade worldwide by Springer Science+Business Media New York,
233 Spring Street, 6th Floor, New York, NY 10013. Phone 1-800-SPRINGER, fax (201) 348-4505,
e-mail [email protected], or visit www.springeronline.com. Apress Media,
LLC is a California LLC and the sole member (owner) is Springer Science + Business Media
Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation.
For information on translations, please e-mail [email protected], or visit
http://www.apress.com/rights-permissions.
Apress titles may be purchased in bulk for academic, corporate, or promotional use. eBook
versions and licenses are also available for most titles. For more information, reference our
Print and eBook Bulk Sales web page at http://www.apress.com/bulk-sales.
Any source code or other supplementary material referenced by the author in this book is
available to readers on GitHub via the book’s product page, located at www.apress.com/
978-1-4842-3284-2. For more detailed information, please visit http://www.apress.com/
source-code.
Printed on acid-free paper
Contents

About the Authors���������������������������������������������������������������������������� vii

About the Technical Reviewer���������������������������������������������������������� ix
Acknowledgments���������������������������������������������������������������������������� xi
Introduction������������������������������������������������������������������������������������ xiii

■
■Chapter 1: Reinforcement Learning Basics������������������������������������ 1
What Is Reinforcement Learning?����������������������������������������������������������� 1
Faces of Reinforcement Learning����������������������������������������������������������� 6
The Flow of Reinforcement Learning������������������������������������������������������ 7
Different Terms in Reinforcement Learning�������������������������������������������� 9
Gamma������������������������������������������������������������������������������������������������������������������� 10
Lambda������������������������������������������������������������������������������������������������������������������� 10

Interactions with Reinforcement Learning�������������������������������������������� 10

RL Characteristics�������������������������������������������������������������������������������������������������� 11
How Reward Works������������������������������������������������������������������������������������������������ 12
Agents�������������������������������������������������������������������������������������������������������������������� 13
RL Environments����������������������������������������������������������������������������������������������������� 14

Conclusion��������������������������������������������������������������������������������������������� 18
■
■Chapter 2: RL Theory and Algorithms������������������������������������������� 19
Theoretical Basis of Reinforcement Learning��������������������������������������� 19
Where Reinforcement Learning Is Used������������������������������������������������ 21
Manufacturing�������������������������������������������������������������������������������������������������������� 22
Inventory Management������������������������������������������������������������������������������������������� 22

iii
■ Contents

Delivery Management��������������������������������������������������������������������������������������������� 22
Finance Sector�������������������������������������������������������������������������������������������������������� 23

Why Is Reinforcement Learning Difficult?��������������������������������������������� 23

Preparing the Machine�������������������������������������������������������������������������� 24
Installing Docker����������������������������������������������������������������������������������� 36
An Example of Reinforcement Learning with Python���������������������������� 39
What Are Hyperparameters?���������������������������������������������������������������������������������� 41
Writing the Code����������������������������������������������������������������������������������������������������� 41

What Is MDP?���������������������������������������������������������������������������������������� 47
The Markov Property���������������������������������������������������������������������������������������������� 48
The Markov Chain��������������������������������������������������������������������������������������������������� 49
MDPs���������������������������������������������������������������������������������������������������������������������� 53

SARSA��������������������������������������������������������������������������������������������������� 54
Temporal Difference Learning�������������������������������������������������������������������������������� 54
How SARSA Works�������������������������������������������������������������������������������������������������� 56

Q Learning��������������������������������������������������������������������������������������������� 56
What Is Q?�������������������������������������������������������������������������������������������������������������� 57
How to Use Q���������������������������������������������������������������������������������������������������������� 57
SARSA Implementation in Python��������������������������������������������������������������������������� 58
The Entire Reinforcement Logic in Python������������������������������������������������������������� 64

Dynamic Programming in Reinforcement Learning������������������������������ 68

Conclusion��������������������������������������������������������������������������������������������� 69
■
■Chapter 3: OpenAI Basics������������������������������������������������������������� 71
Getting to Know OpenAI������������������������������������������������������������������������ 71
Installing OpenAI Gym and OpenAI Universe����������������������������������������� 73
Working with OpenAI Gym and OpenAI������������������������������������������������� 75
More Simulations���������������������������������������������������������������������������������� 81

iv
 ■ Contents

OpenAI Universe������������������������������������������������������������������������������������ 84
Conclusion��������������������������������������������������������������������������������������������� 87
■
■Chapter 4: Applying Python to Reinforcement Learning�������������� 89
Q Learning with Python������������������������������������������������������������������������� 89
The Maze Environment Python File������������������������������������������������������������������������ 91
The RL_Brain Python File��������������������������������������������������������������������������������������� 94
Updating the Function�������������������������������������������������������������������������������������������� 95

Using the MDP Toolbox in Python���������������������������������������������������������� 97

Understanding Swarm Intelligence����������������������������������������������������� 109
Applications of Swarm Intelligence���������������������������������������������������������������������� 109
Swarm Grammars������������������������������������������������������������������������������������������������� 111
The Rastrigin Function������������������������������������������������������������������������������������������ 111
Swarm Intelligence in Python������������������������������������������������������������������������������� 116

Building a Game AI������������������������������������������������������������������������������ 119

The Entire TFLearn Code��������������������������������������������������������������������������������������� 124

Conclusion������������������������������������������������������������������������������������������� 128
■■Chapter 5: Reinforcement Learning with Keras,
TensorFlow, and ChainerRL�������������������������������������������������������� 129
What Is Keras?������������������������������������������������������������������������������������ 129
Using Keras for Reinforcement Learning�������������������������������������������� 130
Using ChainerRL���������������������������������������������������������������������������������� 134
Installing ChainerRL���������������������������������������������������������������������������������������������� 134
Pipeline for Using ChainerRL�������������������������������������������������������������������������������� 137

Deep Q Learning: Using Keras and TensorFlow����������������������������������� 145

Installing Keras-rl������������������������������������������������������������������������������������������������� 146
Training with Keras-rl������������������������������������������������������������������������������������������� 148

Conclusion������������������������������������������������������������������������������������������� 153

v
■ Contents

■■Chapter 6: Google’s DeepMind and the Future of

Reinforcement Learning������������������������������������������������������������� 155
Google DeepMind�������������������������������������������������������������������������������� 155
Google AlphaGo����������������������������������������������������������������������������������� 156
What Is AlphaGo?�������������������������������������������������������������������������������������������������� 157
Monte Carlo Search���������������������������������������������������������������������������������������������� 159
Man vs. Machines������������������������������������������������������������������������������� 161
Positive Aspects of AI������������������������������������������������������������������������������������������� 161
Negative Aspects of AI������������������������������������������������������������������������������������������ 161

Conclusion������������������������������������������������������������������������������������������� 163

Index���������������������������������������������������������������������������������������������� 165

vi
About the Authors

Abhishek Nandy has a B.Tech. in information

technology and considers himself a constant learner.
He is a Microsoft MVP in the Windows platform, an
Intel Black Belt Developer, as well as an Intel software
innovator. Abhishek has a keen interest in artificial
intelligence, IoT, and game development. He is
currently serving as an application architect at an IT
firm and consults in AI and IoT, as well does projects
in AI, Machine Learning, and deep learning. He is also
an AI trainer and drives the technical part of Intel AI
student developer program. He was involved in the first
Make in India initiative, where he was among the top
50 innovators and was trained in IIMA.

Manisha Biswas has a B.Tech. in information

technology and currently works as a software developer
at InSync Tech-Fin Solutions Ltd in Kolkata, India. She
is involved in several areas of technology, including
web development, IoT, soft computing, and artificial
intelligence. She is an Intel Software innovator and was
awarded the Shri Dewang Mehta IT Awards 2016 by
NASSCOM, a certificate of excellence for top academic
scores. She very recently formed a “Women in
Technology” community in Kolkata, India to empower
women to learn and explore new technologies. She
likes to invent things, create something new, and
invent a new look for the old things. When not in front
of her terminal, she is an explorer, a foodie, a doodler,
and a dreamer. She is always very passionate to share
her knowledge and ideas with others. She is following
her passion currently by sharing her experiences with the community so that others can
learn, which lead her to become Google Women Techmakers, Kolkata Chapter Lead.

vii
About the Technical
Reviewer

Avirup Basu is an IoT application developer at

Prescriber360 Solutions. He is a researcher in robotics
and has published papers through the IEEE.

ix
Acknowledgments

I want to dedicate this book to my parents.

—Abhishek Nandy

I want to dedicate this book to my mom and dad. Thank you to my teachers and my
co-author, Abhishek Nandy. Thanks also to Abhishek Sur, who mentors me at work
and helps me adapt to new technologies. I would also like to dedicate this book to my
company, InSync Tech-Fin Solutions Ltd., where I started my career and have grown
professionally.

—Manisha Biswas

xi
Introduction

This book is primarily based on a Machine Learning subset known as Reinforcement

Learning. We cover the basics of Reinforcement Learning with the help of the Python
programming language and touch on several aspects, such as Q learning, MDP, RL with
Keras, and OpenAI Gym and OpenAI Environment, and also cover algorithms related
to RL.
Users need a basic understanding of programming in Python to benefit from this
book.
The book is meant for people who want to get into Machine Learning and learn more
about Reinforcement Learning.

xiii
CHAPTER 1

Reinforcement Learning
Basics

This chapter is a brief introduction to Reinforcement Learning (RL) and includes some
key concepts associated with it.
In this chapter, we talk about Reinforcement Learning as a core concept and then
define it further. We show a complete flow of how Reinforcement Learning works. We
discuss exactly where Reinforcement Learning fits into artificial intelligence (AI). After
that we define key terms related to Reinforcement Learning. We start with agents and
then touch on environments and then finally talk about the connection between agents
and environments.

What Is Reinforcement Learning?

We use Machine Learning to constantly improve the performance of machines or
programs over time. The simplified way of implementing a process that improves
machine performance with time is using Reinforcement Learning (RL). Reinforcement
Learning is an approach through which intelligent programs, known as agents, work
in a known or unknown environment to constantly adapt and learn based on giving
points. The feedback might be positive, also known as rewards, or negative, also
called punishments. Considering the agents and the environment interaction, we then
determine which action to take.
In a nutshell, Reinforcement Learning is based on rewards and punishments.
Some important points about Reinforcement Learning:
• It differs from normal Machine Learning, as we do not look at
training datasets.
• Interaction happens not with data but with environments,
through which we depict real-world scenarios.

© Abhishek Nandy and Manisha Biswas 2018 1

A. Nandy and M. Biswas, Reinforcement Learning,
https://doi.org/10.1007/978-1-4842-3285-9_1
Chapter 1 ■ Reinforcement Learning Basics

• As Reinforcement Learning is based on environments, many

parameters come in to play. It takes lots of information to learn
and act accordingly.
• Environments in Reinforcement Learning are real-world
scenarios that might be 2D or 3D simulated worlds or game-
based scenarios.
• Reinforcement Learning is broader in a sense because the
environments can be large in scale and there might be a lot of
factors associated with them.
• The objective of Reinforcement Learning is to reach a goal.
• Rewards in Reinforcement Learning are obtained from the
environment.
The Reinforcement Learning cycle is depicted in Figure 1-1 with the help of a robot.

Figure 1-1. Reinforcement Learning cycle

2
 Chapter 1 ■ Reinforcement Learning Basics

A maze is a good example that can be studied using Reinforcement Learning, in

order to determine the exact right moves to complete the maze (see Figure 1-2).

Figure 1-2. Reinforcement Learning can be applied to mazes

In Figure 1-3, we are applying Reinforcement Learning and we call it the

Reinforcement Learning box because within its vicinity the process of RL works. RL starts
with an intelligent program, known as agents, and when they interact with environments,
there are rewards and punishments associated. An environment can be either known
or unknown to the agents. The agents take actions to move to the next state in order to
maximize rewards.

3
Chapter 1 ■ Reinforcement Learning Basics

Figure 1-3. Reinforcement Learning flow

In the maze, the centralized concept is to keep moving. The goal is to clear the maze
and reach the end as quickly as possible.
The following concepts of Reinforcement Learning and the working scenario are
discussed later this chapter.
• The agent is the intelligent program
• The environment is the maze
• The state is the place in the maze where the agent is
• The action is the move we take to move to the next state
• The reward is the points associated with reaching a particular
state. It can be positive, negative, or zero
We use the maze example to apply concepts of Reinforcement Learning. We will be
describing the following steps:

1. The concept of the maze is given to the agent.

2. There is a task associated with the agent and Reinforcement
Learning is applied to it.
3. The agent receives (a-1) reinforcement for every move it
makes from one state to other.
4. There is a reward system in place for the agent when it moves
from one state to another.

4
 Chapter 1 ■ Reinforcement Learning Basics

The rewards predictions are made iteratively, where we update the value of each
state in a maze based on the value of the best subsequent state and the immediate reward
obtained. This is called the update rule.
The constant movement of the Reinforcement Learning process is based on
decision-making.
Reinforcement Learning works on a trial-and-error basis because it is very difficult to
predict which action to take when it is in one state. From the maze problem itself, you can
see that in order get the optimal path for the next move, you have to weigh a lot of factors.
It is always on the basis of state action and rewards. For the maze, we have to compute
and account for probability to take the step.
The maze also does not consider the reward of the previous step; it is specifically
considering the move to the next state. The concept is the same for all Reinforcement
Learning processes.
Here are the steps of this process:
1. We have a problem.
2. We have to apply Reinforcement Learning.
3. We consider applying Reinforcement Learning as a
Reinforcement Learning box.
4. The Reinforcement Learning box contains all essential
components needed for applying the Reinforcement Learning
process.
5. The Reinforcement Learning box contains agents,
environments, rewards, punishments, and actions.
Reinforcement Learning works well with intelligent program agents that give rewards
and punishments when interacting with an environment.
The interaction happens between the agents and the environments, as shown in
Figure 1-4.

Figure 1-4. Interaction between agents and environments

From Figure 1-4, you can see that there is a direct interaction between the agents and
its environments. This interaction is very important because through these exchanges,
the agent adapts to the environments. When a Machine Learning program, robot, or
Reinforcement Learning program starts working, the agents are exposed to known or
unknown environments and the Reinforcement Learning technique allows the agents to
interact and adapt according to the environment’s features.
Accordingly, the agents work and the Reinforcement Learning robot learns. In order
to get to a desired position, we assign rewards and punishments.

5
Chapter 1 ■ Reinforcement Learning Basics

Now, the program has to work around the optimal path to get maximum rewards if
it fails (that is, it takes punishments or receives negative points). In order to reach a new
position, which also is known as a state, it must perform what we call an action.
To perform an action, we implement a function, also known as a policy. A policy is
therefore a function that does some work.

Faces of Reinforcement Learning

As you see from the Venn diagram in Figure 1-5, Reinforcement Learning sits at the
intersection of many different fields of science.

Figure 1-5. All the faces of Reinforcement Learning

6
 Chapter 1 ■ Reinforcement Learning Basics

The intersection points reveal a very strong feature of Reinforcement Learning—it

shows the science of decision-making. If we have two paths and have to decide which
path to take so that some point is met, a scientific decision-making process can be
designed.
Reinforcement Learning is the fundamental science of optimal decision-making.
If we focus on the computer science part of the Venn diagram in Figure 1-5, we
see that if we want to learn, it falls under the category of Machine Learning, which is
specifically mapped to Reinforcement Learning.
Reinforcement Learning can be applied to many different fields of science. In
engineering, we have devices that focus mostly on optimal control. In neuroscience, we
are concerned with how the brain works as a stimulant for making decisions and study
the reward system that works on the brain (the dopamine system).
Psychologists can apply Reinforcement Learning to determine how animals make
decisions. In mathematics, we have a lot of data applying Reinforcement Learning in
operations research.

The Flow of Reinforcement Learning

Figure 1-6 connects agents and environments.

Figure 1-6. RL structure

The interaction happens from one state to another. The exact connection starts
between an agent and the environment. Rewards are happening on a regular basis.
We take appropriate actions to move from one state to another.
The key points of consideration after going through the details are the following:
• The Reinforcement Learning cycle works in an interconnected
manner.
• There is distinct communication between the agent and the
environment.
• The distinct communication happens with rewards in mind.
• The object or robot moves from one state to another.
• An action is taken to move from one state to another

7
Chapter 1 ■ Reinforcement Learning Basics

Figure 1-7 simplifies the interaction process.

Figure 1-7. The entire interaction process

An agent is always learning and finally makes a decision. An agent is a learner, which
means there might be different paths. When the agent starts training, it starts to adapt and
intelligently learns from its surroundings.
The agent is also a decision maker because it tries to take an action that will get it the
maximum reward.
When the agent starts interacting with the environment, it can choose an action and
respond accordingly.
From then on, new scenes are created. When the agent changes from one place to
another in an environment, every change results in some kind of modification. These
changes are depicted as scenes. The transition that happens in each step helps the agent
solve the Reinforcement Learning problem more effectively.

8
 Chapter 1 ■ Reinforcement Learning Basics

Let’s look at another scenario of state transitioning, as shown in Figures 1-8 and 1-9.

Figure 1-8. Scenario of state changes

Figure 1-9. The state transition process

Learn to choose actions that maximize the following:

r0 +γr1 +γ2r2 +............... where 0< γ<1

At each state transition, the reward is a different value, hence we describe reward
with varying values in each step, such as r0, r1, r2, etc. Gamma (γ) is called a discount
factor and it determines what future reward types we get:
• A gamma value of 0 means the reward is associated with the
current state only
• A gamma value of 1 means that the reward is long-term

Different Terms in Reinforcement Learning

Now we cover some common terms associated with Reinforcement Learning.
There are two constants that are important in this case—gamma (γ) and lambda (λ),
as shown in Figure 1-10.

9
Chapter 1 ■ Reinforcement Learning Basics

Figure 1-10. Showing values of constants

Gamma is common in Reinforcement Learning problems but lambda is used

generally in terms of temporal difference problems.

Gamma
Gamma is used in each state transition and is a constant value at each state change.
Gamma allows you to give information about the type of reward you will be getting in
every state. Generally, the values determine whether we are looking for reward values in
each state only (in which case, it’s 0) or if we are looking for long-term reward values (in
which case it’s 1).

Lambda
Lambda is generally used when we are dealing with temporal difference problems. It is
more involved with predictions in successive states.
Increasing values of lambda in each state shows that our algorithm is learning fast.
The faster algorithm yields better results when using Reinforcement Learning techniques.
As you’ll learn later, temporal differences can be generalized to what we call
TD(Lambda). We discuss it in greater depth later.

Interactions with Reinforcement Learning

Let’s now talk about Reinforcement Learning and its interactions. As shown in
Figure 1-11, the interactions between the agent and the environment occur with a reward.
We need to take an action to move from one state to another.

10
 Chapter 1 ■ Reinforcement Learning Basics

Figure 1-11. Reinforcement Learning interactions

Reinforcement Learning is a way of implementing how to map situations to actions

so as to maximize and find a way to get the highest rewards.
The machine or robot is not told which actions to take, as with other forms of
Machine Learning, but instead the machine must discover which actions yield the
maximum reward by trying them.
In the most interesting and challenging cases, actions affect not only the immediate
reward but also the next situation and all subsequent rewards.

RL Characteristics
We talk about characteristics next. The characteristics are generally what the agent does
to move to the next state. The agent considers which approach works best to make the
next move.
The two characteristics are
• Trial and error search.
• Delayed reward.
As you probably have gathered, Reinforcement Learning works on three things
combined:

(S,A,R)

Where S represents state, A represents action, and R represents reward.

If you are in a state S, you perform an action A so that you get a reward R at time
frame t+1. Now, the most important part is when you move to the next state. In this case,
we do not use the reward we just earned to decide where to move next. Each transition
has a unique reward and no reward from any previous state is used to determine the next
move. See Figure 1-12.

11
Chapter 1 ■ Reinforcement Learning Basics

Figure 1-12. State change with time

The T change (the time frame) is important in terms of Reinforcement Learning.

Every occurrence of what we do is always a combination of what we perform in terms
of states, actions, and rewards. See Figure 1-13.

Figure 1-13. Another way of representing the state transition

How Reward Works

A reward is some motivator we receive when we transition from one state to another. It
can be points, as in a video game. The more we train, the more accurate we become, and
the greater our reward.

12
 Chapter 1 ■ Reinforcement Learning Basics

Agents
In terms of Reinforcement Learning, agents are the software programs that make
intelligent decisions. Agents should be able to perceive what is happening in the
environment. Here are the basic steps of the agents:
1. When the agent can perceive the environment, it can make
better decisions.
2. The decision the agents take results in an action.
3. The action that the agents perform must be the best, the
optimal, one.
Software agents might be autonomous or they might work together with other agents
or with people. Figure 1-14 shows how the agent works.

Figure 1-14. The flow of the environment

13
Chapter 1 ■ Reinforcement Learning Basics

RL Environments
The environments in the Reinforcement Learning space are comprised of certain factors
that determine the impact on the Reinforcement Learning agent. The agent must adapt
accordingly to the environment. These environments can be 2D worlds or grids or even a
3D world.
Here are some important features of environments:
• Deterministic
• Observable
• Discrete or continuous
• Single or multiagent.

Deterministic
If we can infer and predict what will happen with a certain scenario in the future, we say
the scenario is deterministic.
It is easier for RL problems to be deterministic because we don’t rely on the
decision-making process to change state. It’s an immediate effect that happens with state
transitions when we are moving from one state to another. The life of a Reinforcement
Learning problem becomes easier.
When we are dealing with RL, the state model we get will be either deterministic or
non-deterministic. That means we need to understand the mechanisms behind how DFA
and NDFA work.

DFA (Deterministic Finite Automata)

DFA goes through a finite number of steps. It can only perform one action for a state. See
Figure 1-15.

Figure 1-15. Showing DFA

14
 Chapter 1 ■ Reinforcement Learning Basics

We are showing a state transition from a start state to a final state with the help of
a diagram. It is a simple depiction where we can say that, with some input value that is
assumed as 1 and 0, the state transition occurs. The self-loop is created when it gets a
value and stays in the same state.

NDFA (Nondeterministic Finite Automaton)

If we are working in a scenario where we don’t know exactly which state a machine will
move into, this is a case of NDFA. See Figure 1-16.

Figure 1-16. NDFA

The working principle of the state diagram in Figure 1-16 can be explained as
follows. In NDFA the issue is when we are transitioning from one state to another, there is
more than one option available, as we can see in Figure 1-16. From State S0 after getting
an input such as 0, it can stay in state S0 or move to state S1. There is decision-making
involved here, so it becomes difficult to know which action to take.

Observable
If we can say that the environment around us is fully observable, we have a perfect
scenario for implementing Reinforcement Learning.
An example of perfect observability is a chess game. An example of partial
observability is a poker game, where some of the cards are unknown to any one player.

15
Chapter 1 ■ Reinforcement Learning Basics

Discrete or Continuous
If there is more than one choice for transitioning to the next state, that is a continuous
scenario. When there are a limited number of choices, that’s called a discrete scenario.

Single Agent and Multiagent Environments

Solutions in Reinforcement Learning can be of single agent types or multiagent types.
Let’s take a look at multiagent Reinforcement Learning first.
When we are dealing with complex problems, we use multiagent Reinforcement
Learning. Complex problems might have different environments where the agent is doing
different jobs to get involved in RL and the agent also wants to interact. This introduces
different complications in determining transitions in states.
Multiagent solutions are based on the non-deterministic approach.
They are non-deterministic because when the multiagents interact, there might be
more than one option to change or move to the next state and we have to make decisions
based on that ambiguity.
In multiagent solutions, the agent interactions between different environments are
enormous. They are enormous because the amount of activity involved in references to
environments is very large. This is because the environments might be different types and
the multiagents might have different tasks to do in each state transition.
The difference between single-agent and multiagent solutions are as follows:
• Single-agent scenarios involve intelligent software in which the
interaction happens in one environment only. If there is another
environment simultaneously, it cannot interact with the first
environment.
• When there is little bit of convergence in Reinforcement
Learning. Convergence is when the agent needs to interact far
more often in different environments to make a decision. This
scenario is tackled by multiagents, as single agents cannot tackle
convergence. Single agents cannot tackle convergence because
it connects to other environments when there might be different
scenarios involving simultaneous decision-making.
• Multiagents have dynamic environments compared to
single agents. Dynamic environments can involve changing
environments in the places to interact with.

16
 Chapter 1 ■ Reinforcement Learning Basics

Figure 1-17 shows the single-agent scenario.

Figure 1-17. Single agent

Figure 1-18 shows how multiagents work. There is an interaction between two agents
in order to make the decision.

17
Chapter 1 ■ Reinforcement Learning Basics

Figure 1-18. Multiagent scenario

Conclusion
This chapter touched on the basics of Reinforcement Learning and covered some key
concepts. We covered states and environments and how the structure of Reinforcement
Learning looks.
We also touched on the different kinds of interactions and learned about single-
agent and multiagent solutions.
The next chapter covers algorithms and discusses the building blocks of
Reinforcement Learning.

18
Exploring the Variety of Random
Documents with Different Content
The Project Gutenberg eBook of Notes and
Queries, Number 173, February 19, 1853
 This ebook is for the use of anyone anywhere in the United
States and most other parts of the world at no cost and with
almost no restrictions whatsoever. You may copy it, give it away
or re-use it under the terms of the Project Gutenberg License
included with this ebook or online at www.gutenberg.org. If you
are not located in the United States, you will have to check the
laws of the country where you are located before using this
eBook.

Title: Notes and Queries, Number 173, February 19, 1853

Author: Various

Editor: George Bell

Release date: May 24, 2013 [eBook #42788]

Most recently updated: October 23, 2024

Language: English

Credits: Produced by Charlene Taylor, Jonathan Ingram, Keith

Edkins
and the Online Distributed Proofreading Team at
http://www.pgdp.net (This file was produced from
images
generously made available by The Internet
Archive/Canadian
Libraries)

*** START OF THE PROJECT GUTENBERG EBOOK NOTES AND

QUERIES, NUMBER 173, FEBRUARY 19, 1853 ***
NOTES AND QUERIES:
 A MEDIUM OF INTER-
COMMUNICATION FOR LITERARY
MEN, ARTISTS, ANTIQUARIES,
GENEALOGISTS, ETC.
"When found, make a note of."—Captain Cuttle.

Price
Fourpence.
No. 173. Saturday, February 19. 1853.
Stamped
Edition 5d.
 CONTENTS.
Notes:— Page
Predictions of the Fire and Plague of London,
No. II., by Vincent T. Sternberg 173
Examples of the French Sizain, by W. Pinkerton 174
Epigrams 174
"Goe, soule, the bodies guest," by George
Daniel 175
Petitions from the County of Nottingham 175
Folk Lore:—Lancashire Fairy Tale—Teeth,
Superstition respecting—New Moon Divination
—The Hyena an Ingredient in Love Potions—
The Elder Tree 177
Minor Notes:—The Word "Party"—Epitaphs—
Campbell's "Pleasures of Hope"—Palindromical
Lines—"Derrick" and "Ship's Painter"—Lord
Reay's Country 177
Queries:—
Unanswered Queries 178
Mr. John Munro, by Dan. Wilson 179
Minor Queries:—Song in Praise of the Marquess 179
of Granby—Venda—The Georgiad—R. S.
Townshend of Manchester—"Mala malæ
malo"—"Dimidium Scientiæ"—Portrait Painters
—"An Impartial Inquiry," &c.—"As poor as Job's
Turkey"—Fuss—Suicide encouraged in
Marseilles—Fabulous Bird—Segantiorum Portus
—Stamping on Current Coinage—Rhymes:
Dryden—The Cadenham Oak—St. Mary's
Church, Beverley—The Rev. Joshua Marsden—
 Bentley's Examination—Derivation of
"Lowbell"—Meaning of Assassin—Punishment
for exercising the Roman Catholic Religion—
Hogarth's Pictures—Lines in a Snuff-box—Rosa
Mystica—Old-Shoe throwing at Weddings—
Herbé's Costumes Français
Minor Queries with Answers:—Humphry Smith—
Meaning and Etymology of "Conyngers" or
"Connigries"—Letters U, V, W, and St. Ives 182
Replies:—
The Orkney Islands in Pawn 183
The Passage in King Henry VIII., Act. III. Sc. 2,
by S. W. Singer 183
Miniature Ring of Charles I., by C. Ley 184
Chantry Chapels 185
Photographic Notes and Queries:—The Collodion
Process—Mr. Weld Taylor's Iodizing Process—Sir
William Newton's Process: Further Explanations 185
Replies to Minor Queries:—Lady Nevell's Music-
book—Tuch—Eva, Princess of Leinster—
Whipping Post—The Dodo—"Then comes the
reckoning," &c.—Sir J. Covert, not Govett—
Chatterton—Tennyson—Llandudno on the Great
Orme's Head—Oldham, Bishop of Exeter—Arms
of Bristol—The Cross and the Crucifix—Sir
Kenelm Digby—Martin Drunk—The Church
Catechism—Sham Epitaphs and Quotations—
Door-head Inscription—Potguns—"Pompey the
Little"—Eagles supporting Lecterns—Lady Day
in Harvest—Inscriptions in Churches—
Macaulay's Young Levite, &c. 187
Miscellaneous:—
Books and Odd Volumes wanted 194
Notices to Correspondents 194
Advertisements 195
 Notes.
PREDICTIONS OF THE FIRE AND PLAGUE OF
LONDON, NO. II.
One of the most striking predictions occurs in Daniel Baker's Certaine
Warning for a Naked Heart, Lond. 1659. After much invective against
the evil ways of the metropolis, he proceeds:

"A fire, a consuming fire, shall be kindled in the bowels of the

earth, which will scorch with burning heat all hypocrites,
unstable, double-minded workers of iniquity.... A great and large
slaughter shall be throughout the land of darkness where the
unrighteous decrees and laws have been founded. Yea, a great
effusion of blood, fire, and smoke shall encrease up in the dark
habitations of cruelty; howling and great wailing shall be on
every hand in all her streets."

Thomas Ellwood disposes of the city in a very summary manner:

"For this shall be judgment of Babylon (saith the Lord); in one

day shall her plagues come upon her, death, and mourning, and
famine, and she shall be utterly burnt with fire; for great is the
Lord who judgeth her."—Alarm to the Priests, Lond. 1662.

George Fox also claims to have had a distinct prevision of the fire
(See Journal, p. 386., ed 1765.) He also relates the story of a Quaker
who was moved to come out of Huntingdonshire a little before the
fire, and to—

"Scatter his money up and down the streets, turn his horse
loose, untie the knees of his breeches, and let his stockings fall
 down, and to tell the people 'so they should run up and down
scattering their money and goods, half undressed, like mad
people, as he was a sign to them,' which they did when the city
was burning."

Lilly's celebrated book of Hieroglyphicks, which procured the author

the dubious honour of an examination before the committee
appointed to inquire into the origin of the fire, is well known. In one
of the plates, a large city, understood to denote London, is
enveloped in flames; and another rude woodcut, containing a large
amount of graves and corpses, was afterwards interpreted to bear
reference to the Plague. Aubrey seems to be a little jealous of the
renown which Lilly acquired by these productions for he asserts that
—

"Mr. Thomas Flatman (poet) did affirm that he had seen those
Hieroglyphicks in an old parchment manuscript, writ in the time
of the monks."—Misc., p. 125. ed. 1721.

Nostradamus also, more than a century before, is said to have

foretold the very year of the burning. In the edition, or reputed
edition, of 1577, cent. ii. quatrain 51., is the following:

"Le sang du jusse à Londres fera faute

Bruslez par foudres de vingt trois les six
La dame anticque cherra de place haute
De mesme secte plusieurs seront occis."

Those of your readers who incline to dubiety on this subject, I refer

to the copy from whence it was taken, in the Museum Library, press-
mark 718. a 14. If it is a forgery (and such I take it to be), it is
decidedly the best I ever met with. Some time ago the Queries of
your correspondent Speriend elicited some interesting particulars
relative to Nostradamus and his prophecies; but I do not think the
question of his claim to having predicted the death of Charles I. was
finally decided.
I should be glad if any of your correspondents could tell me whether
the quatrain above, or anything like it, occurs in any of the genuine
early editions. Dugdale, by the way, evidently believed in its
authenticity, and has inserted a version in his History of St. Paul's.

Such a promising theme as the destruction of London was, of

course, too good a thing to escape the chap-book makers. During
the period of the Civil Wars, we find many allusions to it. In a little
quarto brochure, published in 1648, entitled Twelve Strange
Prophecies, the following is placed in the mouth of the much
maligned and caricatured Mrs. Ann Shipton. The characteristic
termination I consider a fine stroke of the art vaticinatory.

"A ship shall come sayling up the Thames till it come to London,
and the master of the ship shall weep, and the mariners shall
ask him why he weepeth, and he shall say, 'Ah, what a goodly
city was this! none in the world comparable to it! and now there
is scarce left any house that can let us have drinke for our
money.'"

This string of notes, turned up at different times, and while in search

of more important matter, can no doubt be materially increased from
the collections of your correspondents. If my researches prove
interesting, I may trouble you with another paper: at present I leave
the facts brought together above to the candid investigation of your
readers.
Vincent T. Sternberg.

EXAMPLES OF THE FRENCH SIZAIN.

The epigram (if it may with propriety receive that appellation)
printed in Vol. vi., p. 603., reminded me of some similar pieces of
composition stored in my note-book; and as they are not devoid of a
certain degree of curious interest, I now forward them pro bono
publico.
On Luther, Calvin, and Henry VIII., the leaders of the Reformation:

"Vous, dont le sens est encore sain,

Fuyez Luther, Henri, Calvin.
Vous, dont le cœur n'est point flétri,
Fuyez Calvin, Luther, Henri.
Vous, à qui le salut est cher,
Fuyez Henri, Calvin, Luther."

On the death of Francis II.:

"Par l'œil, par l'oreille, et l'épaule,

Trois rois sont morts naguère en Gaule;
Par l'épaule, l'oreille, et l'œil,
Trois rois son entrés au cercueil;
Par l'épaule, l'œil, et l'oreille,
Dieu a montré grande merveille."

By Beaumarchais:

"Connaissez-vous rien de plus sot

Que Merlin, Bazire, et Chabot?
Non, certes, il n'est rien de pire
Que Chabot, Merlin, et Bazire;
Et nul ne vit-on plus coquin
Que Chabot, Bazire, et Merlin."

A more modern one still, date 1842:

"L'Etat est fort mal attelé

Avec Thiers, Guizot, ou Molé;
L'Etat marche tout de travers,
Avec Molé, Guizot, ou Thiers;
Vers l'abîme il court à galop,
Avec Molé, Thiers, ou Guizot."

The prophecy in the last two lines has been unfortunately fulfilled.
 W. Pinkerton.
Ham.

EPIGRAMS.
The two epigrams which follow were communicated to me many
years ago by the Rev. George Loggin, M.A., of Hertford College, long
one of the masters of Rugby School. He died July 15, 1824, at the
age of forty; and this reminiscence of their old tutor's name will be
welcomed by many a Rugbæan. They were represented to have
proceeded from the pen of Thomas Dunbar of Brasenose, who, from
1815 to 1822, was keeper of the Ashmolean Museum. I have never
seen them in print, or even in writing. They were recited memoriter,
and from memory I write them down; and hence, no doubt, there
will be some deviations from the true text. But they seem too good
to be lost; and I am not without hope that a correct copy may
eventually be elicited from some of your correspondents.

With regard to the first, whether the lines were really made on the
occasion stated, or the occasion was invented (as I am inclined to
suspect) to suit the lines, is perhaps not very material:

"Reply to Miss Charlotte Ness, who inquired the meaning

of the logical terms Abstract and Concrete.

"'Say what is Abstract, what Concrete?

Their difference define.'
'They both in one fair person meet,
And that, dear maid, is thine.'

'How so? The riddle pray undo.'

'I thus your wish express;
For when I lovely Charlotte view,
I then view loveli-Ness.'"
On a certain D.D. (who, from a peculiarity in his walk, had acquired
the sobriquet of Dr. Toe) being jilted by Miss H——, who eloped with
her father's footman:

"'Twixt Footman Sam and Doctor Toe

A controversy fell,
Which should prevail against his foe,
And bear away the belle.
The lady chose the footman's heart.
Say, who can wonder? no man:
The whole prevail'd above the part,
'Twas Foot-man versus Toe-man."

I should like to ascertain the author of the following:

The Parson versus Physician.

"How D.D. swaggers—M.D. rolls!

I dub them both a brace of noddies:—
Old D.D. takes the cure of souls,
And M.D. takes the care of bodies.
Between them both what treatment rare
Our souls and bodies must endure!
One takes the cure without the care,
T'other the care without the cure."
Balliolensis.

GOE, SOULE, THE BODIES GUEST."

I have a cotemporaneous MS. of this wonderfully-fine poem, that
came into my possession with a certain rare bunch of black-letter
ballads, printed between the years 1559 and 1597, and all of them
unique (of the said bunch, Mr. Editor, more hereafter), which
contains two additional verses not to be found in A Poetical
Rhapsodie, compiled by Francis Davison, and "printed by William
Stansby for Roger Jackson, dwelling in Fleet Street, neere the great
Conduit, 1611;" nor in Poems by Sir Henry Wotton, Sir Walter
Raleigh, and others, carefully edited by the Rev. John Hannah, M.A.,
and published by my friend William Pickering in 1845. They are
prefaced by the word "Additions." They are written on the same leaf,
and in the same quaint hand, and are as follow:

Tell London of their stewes,

Tell marchants of their usury;
And, though it be no newes,
Tell courtyers of theyr lechery;
And if they will reply,
They best deserve the lye.

Let cuckolds be remembred,

I will not dye theyr debtor;
Theire heads beying armed,
Theyl beare the brunt the better;
And if they chaunce reply,
Theyr wives know best they lye.

Having compared this MS. with the poem as it is printed in the

above-mentioned volumes (both of which are in my library), I find it
contains several variations, not however very important. Though
these "Additions," in good taste, expression, and power, do not
equal the noble verses that precede them, they are interesting and
curious, and well worthy of preservation. After much inspection and
inquiry, I have not discovered that they have ever yet appeared in
print. The cabinet in which they slept, and the company they kept
(undisturbed, it would appear) for more than two centuries, assure
me that they have not been published.

If you, Mr. Editor, or any of your many friends desire to see this MS.,
say so, and you and they shall be welcome. It has been in my
possession (unseen) twenty years.
George Daniel.
 Canonbury.

PETITIONS FROM THE COUNTY OF

NOTTINGHAM.
The documents, copies of which I inclose, are written on the blank
leaves in a copy of Willett's Hexapla, edit. 1611. I should be glad to
know if the petitions, of which they are drafts, or rather copies, were
presented, and when? There is no date to the petitions; but the copy
of a letter, on another blank page, which seems to be in the same
handwriting (signed "William Middleton"), is dated February 5th,
1658. Any information regarding the parties whose names are
appended to the petitions would be acceptable.

"To his Highness the Lord Protector of the

Commonwealth of England, Scotland, and Ireland,
and the dominions and territories thereunto
belonging, the humble Address and Petition of
diuers Justices of the Peace, Gentlemen, Ministers
of the Gospell, and others, wel-affected persons,
inhabitants in the County of Nottingham.

"Upon consideration of the signall and glorious appearances of

God on the behalfe of his people and interest, wherein he hath
pleased to make great use of your Highness, we account
ourselues deeply engaged to acknowledge the wonderfull
power, wisdome, and goodness of God, and to ascribe the glory
to him alone, yet would we not be found ingratefull to your
Highness, as an eminent instrument under God of the peace
and liberty we have injoyed, with a continued series of manifold
mercies from the Lord, under your Highness' gouernment
(notwithstanding all our declensions and unworthynesses),
together with the influence it hath had upon the nations abroad
to the promoteing of the Protestant interest, we judge it alsoe
exceedingly remarkable that the Lord hath so signally blasted
the pernicious designes of the common enemy against your
Highness' person and gouernment, and against the common
interest of the people of God and of these nations, for which we
desire unfeignedly to bless the Lord.

"These things premised, we humbly pray,

"That the Lord would please to stir up the heart and strengthen
the hands of your Highness, in carrying on what yet remains for
the reforming of these nations (according to the word of God)
and the secureing of the interest of godlyness and
righteousness for the future, that such as are found in the faith
and of holy conversation may live peaceably, and receive
encouragement to persevere in that upon which the Lord may
delight to doe your Highness and these nations good; in order
whereunto we humbly propose these following particulars to
your Highness' consideration:

"1. First, that a stop may be put to the spreading infection of

damnable errors and heresies, by a lively and due suppressing
of them according to the mind of the Lord.

"2. That an effectuall course may be taken for the curbeing of

all profaneness and libertineisme by the sword of justice, which
the Lord hath put into your magistrates' hands.

"3. That your Highness would haue an eye upon the designes of
the common enemy in generall, and particularly on this (vid.),
their traininge up a young generation in the old destructive
principles, as also on the designes of any persons whatsoeuer
that indeauour to disturb your Highness' gouernment and the
peace of these nations.

"4. That the lawes of the nation may be reuised, that for what
in them is agreeable to the rules of righteousness may be
continued and executed, and whatever corruption is crept into,
or may grow up in, courts of judicature may be duly purged
away.

"5. That in your Highness' lifetime such prouision be made for

the future gouernment of the commonwealth, as may secure
the interest of good people of these nations for succeeding
generations, that they may call you blessed.

"And in the prosecution of such ends we shall be ready, as the

Lord shall help us, with all that is dear to us, to defend your
Highness' person and gouernment, with the true interest of
religion and the lawes, and shall ever pray, &c.

"—— Ansley.
Chrystopher Sanderson, Minister of Annesley.
Will. Lee. John Dan.
Geo. Brittain.
Abraham" [Torn off].

"To the honourable the Parliament of England.

"The humble Petition of diuers Gentlemen,

Ministers of the Gospell, and others, inhabiteing in
the County of Nottingham,

"Sheweth,

"That your petitioners, haueing seriously considered how much

of a thorough reformation of religion and pure administration of
the ordinances of Christianity would tend to the honour of God,
the good of soules, and the abundant satisfaction of the truly
godly in this nation, who have long waited for these mercies as
the return of their prayers, and the fruit of their expense both of
blood and treasure, and being alsoe very sensible that the duty
we owe to God, the eminent and signall mercies of God towards
this nation, and our own solemn engagements, doe strongly
oblidge us euery one in our places, to the utmost of our power,
to indeauour the promoteing and aduancement of pure gospell
worship, we are humbly bold to address ourselues to your
honors.

"We are not undmindfull of, nor would we be unthankfull for,

what hath been indeauoured this way by former Parliaments,
yet we cannot but sadly resent the many obstructions this work
hath hitherto met withall, and how much it hath been retarded,
chiefly, we confess, by our own sins, and the sins of these
nations, partly through the malice of Satan, the diuisions of
brethren, the secret and subtile practices of Romish emissaries,
fomenting errors and heresies, and not a little, as we humbly
conceive, through the want of church gouernment, settled and
established by the ciuil authority, whereby those unto whom the
exercise of church power is committed by Christ may be
impoured to keep back ignorant and prophane persons from
polluting the ordinances of God, as alsoe by reason of some
ancient lawes, alledged and urged by diuers as yet in force,
injoyning ministers to dispense the Sacrament of the Lord's
Supper, without affording them (as we conceiue) sufficient
power regularly to keep back such as are not duly qualified for
the same, by reason whereof ministers are liable to prosecution
att law (of which we have had a late instance in this county).

"We therefore, your petitioners, in faithfulness to the interest of

God and his glory, Christ and his gospell, our own and other
men's soules, and from our sincere desires of the aduancement
of the kingdome of Christ in these nations, in the promoting
whereof the interest and welfare of states and nations is uery
much concerned, we neither could nor durst be longer silent,
but being persuaded of your willingness to act for Christ, and
hopeing that God hath raised you up to carry on the work of
reformation already begun amongst us, and to be repairers of
our breaches and restorers of pathes to dwel in, we are
incouraged humbly to pray,
"1. That such ancient lawes as may be yet in force relating to
the Sacrament of the Lord's Supper, so far as they are or may
prove burdensome to truly godly and conscientious ministers
and people, may be duly regulated.

"2. That so far as you in your wisdomes shall think fitt,

ordinances of Parliament that have been made after aduice had
with the late Assembly of Diuines in order to Church settlement,
may be returned upon, and begun reformation carried on.

"3. That in regard a thorough settlement of Church affaires may

be long under debate, in the mean time some speedy and
effectuall course may be taken, where by ignorant and
scandalous persons may be kept from the Lord's Supper.

"And your petitioners shall ever pray.

"Charles Jackson.
Lancelot Coates.
Will. Coup.
Francis Brunt.
Will...llow [obliterated].
John Hoyland.
Tho. Shaw.
Hen. Clark.
Will. Farnworth.
Chrystopher Clark.
Will. Saunder.
George Flint.
Dauid Taylor.
Charles Shepheard.
Es. Brettun."
T. S.
Leeds.
 FOLK LORE.
Lancashire Fairy Tale.—The nursery rhymes in one of your late
Numbers remind me of a story I used to be told in the nursery. It
was, that two men went poaching, and having placed nets, or rather
sacks, over what they supposed to be rabbit-holes, but which were
in reality fairies' houses, the fairies rushed into the sacks, and the
poachers, content with their prey, marched home again. A fairy
missing another in the sack, called out (the story was told in broad
Lancashire dialect) "Dick (dignified name for a fairy), where art
thou?" To which fairy Dick replied,

"In a sack,
On a back,
Riding up Barley Brow."

The story has a good moral ending, for the poachers were so
frightened that they never poached again.
T. G. C.

Teeth, Superstition respecting (Vol. vi., p. 601.).—A similar (perhaps

the same) piece of childish superstition respecting the teeth is, that
when the upper incisors are large, it is a sign that you will live to be
rich.
Furvus.

New Moon Divination.—Being lately on a visit in Yorkshire, I was

amused one evening to find the servants of the house excusing
themselves for being out of the way when the bell rang, on the plea
that they had been "hailing the first new moon of the new year."
This mysterious salutation was effected, I believe, by means of a
looking-glass, in which the first sight of the moon was to be had,
and the object to be gained was the important secret as to how
many years would elapse before the marriage of the observers. If
one moon was seen in the glass, one year; if two, two years; and so
on. In the case in question, the maid and the boy saw only one
moon a-piece. Whether the superstition would, in this instance, be
suggestive to their minds of anything to be deduced from the
coincidence, I do not know; but as they were both very old-
fashioned folks, I suppose the custom may not be unknown to those
learned in Folk Lore.

What is the orthodox mode of conducting this kind of divination?

Oxoniensis.

The Hyena an Ingredient in Love Potions.—In Busbequius's Letters

(Elzevir, 1633) I note that the Turks consider the hyena useful in love
potions. I extract the passage:

"In amatoriis ei vim magnam Turcæ, ut etiam veteres, tribuunt,

cumque essent duæ eo tempore Constantinopoli, mihi tamen
vendere gravabantur, quod se Sultanæ, hoc est, principis uxori,
eas reservare dicerent, quippe quas philtris et magicis artibus
animum mariti retinere, recepta in vulgus (ut dixi) opinio est."—
P. 84.

Allow me to add a Query: What ancient authors allude to this old

specimen of Folk Lore?
S. A. S.
Bridgewater.

The Elder Tree.—I was visiting a poor parishioner the other day,
when the following question was put to me.

"Pray, Sir, can you tell me whether there is any doubt of what kind of
wood our Lord's cross was made? I have always heard that it was
made of elder, and we look carefully into the faggots before we burn
them, for fear that there should be any of this wood in them."

My Query is, Whether this is a common superstition?

Rubi.
 Minor Notes.
The Word "Party."—Our facetious friend Punch has recently made
merry with the modern use of the word "party," as applied to any
absent person concerned in any pending negotiation. It was used
thus, however, by William Salmon, professor of physic, in his Family
Dictionary, 1705:

"Let the party, if it can be agreeable, rub frequently his teeth

with the ashes that remain in a pipe after it is smoaked."—P.
315.

"Having cooled it, rub the party's mouth with a little of it," &c.—
P. 321.
E. D.

Epitaphs.—Churchyard literature presents to us some curious

specimens of metaphor; and it is interesting to observe how an old
idea is sometimes unintentionally reproduced. The following lines
may be seen on a gravestone in the churchyard at Kinver,
Staffordshire:

"Tired with wand'ring thro' a world of sin,

Hither we came to Nature's common Inn,
To rest our wearied bodys for a night,
In hopes to rise that Christ may give us light."

The writer was probably not aware that Spenser says, in his Faerie
Queen, iii. 3. 30.:

"And if he then with victorie can lin,

He shall his days with peace bring to his earthly In."
And again, Faerie Queen, ii. 1. 59.:

"Palmer, quoth he, death is an equall doome

To good and bad, the common In of rest."

A Leicestershire poet has recorded, in the churchyard of Melton

Mowbray, a very different conception of our "earthly Inn." He says:

"This world's an Inn, and I her guest:

I've eat and drank and took my rest
With her awhile, and now I pay
Her lavish bill, and go my way."

You may, perhaps, consider this hardly worthy of a place in your

paper; but I act upon the principle which you inculcate in your
motto.
Erica.

Campbell's "Pleasures of Hope."—It has often occurred to me that in

two lines of the most celebrated passage in this poem,—

"O'er Prague's proud arch the fires of ruin glow,

Her blood-red waters murmuring far below,"

the author has confounded Prague, the capital of Bohemia, with

Praga, the suburb of Warsaw. The bridge over the Moldau, at the
former place, is a stone one of European celebrity; and to it
Campbell must have referred when using terms not at all applicable
to that over the Vistula, which is of much humbler form and
material.

In Campbell's "Ode to the Highland Society on 21st March," he

describes the 42nd Regiment as having been at Vimiera, which it
assuredly was not; and no Highland regiment was in the battle
except the 71st. I suspect he confounded the "Black Watch" with the
distinguished corps next to it on the army list,—an error into which
the author of Charles O'Malley also must have fallen, as he makes
Highlanders form a part of the Light Division, which consisted of the
43rd, 52nd, and 95th.
J. S. Warden.

Palindromical Lines.—In addition to the verses given by your

correspondent H. H. Breen (Vol. vi., p. 449.), I send you the
following, as perhaps the most remarkable of its kind in existence. It
is mentioned by Jeremy Taylor as the inscription somewhere on a
font. Letter by letter it reads the same, whether taken backward or
forwards:

"ΝΙΨΟΝ ΑΝΟΜΗΜΑ ΜΗ ΜΟΝΑΝ ΟΨΙΝ."

"Wash my guilt, and not my face only."

Agricola de Monte.

"Derrick" and "Ship's Painter."—The following Note may perhaps

interest some of your readers:—The ancient British word derrick, or
some such word, still exists in our marine. It is used in sea phrase to
define a crane for temporary purposes, and is not unusually
represented by a single spar, which is stepped near a hatchway,
provided with a tackle or purchase, in order to the removal of goods
from the hold of a vessel. The use of Derry, both as a termination in
the names of places, and in the old ballad chorus of Down derry
down, is familiar to every one.

Some other of our sea terms might receive apt illustration in "N. &
Q.;" and I should beg to suggest "unde derivatur" a boat's painter,—
the name of the rope which confines a ship's boat to the vessel,
when at sea.

Turner gave a world-wide interest to the phrase when he called, in

his eccentric manner, one of his finest marine pictures "Now for the
painter."
J. C. G.
Tavistock Square.
Lord Reay's Country.—Formerly the parish of Durness comprehended
the whole of the district known as "Lord Reay's country," or, as it is
called in Gaelic, "Duthaic Mhic Aoi," i. e. the land of the Mackays,
extending from the river of Borgie, near Strathnaver, to the Kyle of
Assynt, and comprehending a space of about 800 square miles!
Since 1734 it has been divided into three parishes, viz. Eddrachillis,
Durness, and Tongue, with the parish of Farr: it was disjoined from
the presbytery of Caithness, and by an act of the Assembly attached
to the presbytery of Tongue.
Kirkwallensis.
 Queries.
UNANSWERED QUERIES.
I think it may be permitted to Querists, who may fail in obtaining
answers, to recur to their questions after the lapse of a reasonable
time, in order to awaken attention. I asked a question at page 270.,
Vol. vi., in which I was, and still am, much interested. Perhaps Mr.
Collier will do me the favour to answer it, particularly as his
annotated folio is remarkably rich in "stage directions."

Before taking the liberty of putting the question so directly to Mr.

Collier, I awaited an examination of his recently-published volume of
selected corrections, in which, however, the point upon which I seek
information is not alluded to.

In glancing over that volume, I perceive that Mr. Collier, in his notes
at the end (p. 508.), does "N. & Q." the honour to refer to it, by
alluding to an emendation "proposed by Mr. Cornish" ("N. & Q.," Vol.
vi., p. 312.).

When that emendation appeared I recognised it at once as having

been proposed by Warburton and applauded by Dr. Johnson. I did
not, however, then think it of sufficient importance to trouble the
editor of "N. & Q.," by correcting a claim which, although apparent,
might not perhaps be intentional.

But now, since the ownership (quantum valeat) has deceived even
Mr. Collier, and is endorsed by him, it is time to notice it.
A. E. B.
Leeds.
P.S.—I may add that, with respect to these words "happy low lie
down," from my habit of looking for solutions of difficulties in
parallels and antitheses, I have arrived at a different conclusion from
any that has yet been suggested. Finding "uneasy" used adverbially
in the last line, I see no reason why "happy" should not also be
taken adverbially in the preceding line: we should then have the
same verb, "lie" and "lies," repeated antithetically in the same mood
and tense.

The article the before "low" has probably been omitted in the press,
and may be either actually restored or elliptically understood:

"Then happy [the] low lie down;

Uneasy lies the head that wears a crown."

MR. JOHN MUNRO.

Between the years 1803 and 1830, a gentleman resident of London,
under the signature A. Z., presented from time to time to the Society
of Antiquaries of Scotland, a collection of works respecting the
Orkney and Shetland islands, copiously illustrated with manuscript
notes and inserted prints, maps, &c. The internal evidence leaves no
room to doubt that the donor of this valuable collection was a native
of Kirkwall; and recent investigations lead to the conclusion that he
was a Mr. John Munro, originally in the office of Mr. John Heddle,
Town Clerk of Kirkwall. He appears to have gone to London about
1789, and to have passed the rest of his life there, down to May,
1830, when his last communication was made to the Scottish
Antiquaries. A list of his donations is printed in the Archæologia
Scotica, vol. iii. pp. 267-274. His copious manuscript notes, written
in a very neat and legible hand, indicate not only a man of
intelligence and research, but also of an exceedingly amiable and
kindly disposition, and strongly influenced by the amor patriæ, which
gave to his donations their exclusive character.
I am anxious to ascertain what was Mr. Munro's occupation in
London, the date of his death, and any interesting or characteristic
notes concerning him. Judging from his tastes, it seems highly
probable that he may have been known to more than one of your
metropolitan correspondents.

Perhaps you will not think such Queries undeserving of a corner in

your useful vehicle of literary intercommunication, nor A. Z.'s
anonimity unworthy of an effort to rede the riddle.
Dan. Wilson.
Edinburgh.
 Minor Queries.
Song in Praise of the Marquess of Granby.—Can any of your
correspondents furnish me with the words of a song written in praise
of the Marquess of Granby, who was so distinguished as a general
officer in the middle of the last century?

I think the first verse ended with—

"But the jewel of Grantham is Granby."

It was sung to the tune of "Over the Water to Charlie."

F. W. S.

Venda.—Can any of your correspondents tell me what is the origin

and use of this word, as a prefix to names of places in Portugal; as it
occurs, for instance, in Venda da Agua, Venda da Pia, Venda das
Monachos, &c., places not far from Torres Vedras?
C. E. F.

The Georgiad.—About 1814, at Cambridge, some lines under this

title were commonly attributed to the late Rev. E. Smedley (Seaton
prizeman). Can any reader supply a copy? Two stanzas run thus:

"George B——[1] has turn'd a saint, they say:

But who believes the tale?
George D—— [2] might as soon turn gay!
George C——'s[3] flirting fail!

"George D——[4] set the Thames on fire!

George R—— his reign renew!
George R—— imitate his sire,
And to his friends be true!"
 Aitch.

Footnote 1:(return)

"G. A. B.," Fellow of Trinity, a lively companion.

Footnote 2:(return)

Editor of the Bible.

Footnote 3:(return)

Lay Fellow and Tutor of Jes. Coll.; used to read Theocritus Græcè
in the stage-coach.

Footnote 4:(return)

Author of History of London, or some topographical quarto. The

next may be guessed.

R. S. Townshend of Manchester.—I know that you have several

intelligent correspondents in the neighbourhood of Manchester, and
it is probable that they may be able to give me some information
respecting a Mr. R. S. Townshend, a person of literary taste and
pursuits, who resided in that town about the year 1730. His
Common-place Book, or Diary, which has fallen into my hands,
contains numerous allusions to the leading gentry and clergy of the
neighbourhood; and more than once it mentions the well-known Dr.
Byrom, under the title of "Il Gran Maestro de Tachigraphia." Dr.
Deacon, a distinguished person among the Nonjurors, is also
mentioned. The acting of Cato by the scholars of the grammar-
school on Dec. 20, 1732, is also mentioned, with some critiques
upon the performers. The elections at the collegiate church are
constantly referred to as subjects of all-absorbing interest; there
being a strong party, as well in the town as in the church, of
Jacobites, and these elections being regarded as a trial of party
strength.
O. G.
"Mala malæ malo."—Will any of your correspondents be good
enough to complete the distich of which the following is the first
line?—

"Mala malæ malo mala pertulit omnia in orbem,"

or something like it. And, as a further favour, finish the hexameter in

this epigram?

"Roma amor è retro perlecto nomine....

Tendit enim retro Roma in amore Dei."

This is in the style of Audoenus. The former I have heard attributed

to Porson.
Balliolensis.

"Dimidium Scientiæ."—I should be glad if some one of your

Baconian annotators would direct me to that famous maxim which
Coleridge ascribes to the great philosopher, "Dimidium scientiæ,
prudens quæstio," in the original.
B. B. Woodward.

Portrait Painters.—I am in possession of some good paintings,

portraits, &c., which were taken at the end of the last, and early in
the present century. Some were painted at Bath, and others at
Derby: and I should feel obliged if, in your Notes, I could obtain
information as to what artists of celebrity were known in those
places from fifty to seventy years ago. I have heard that White of
Derby was an artist of high repute.
J. Knight.
Aylestone.

"An Impartial Inquiry," &c.—Who was author of—

"An Impartial Inquiry into the true Nature of the Faith which is
required in the Gospel as necessary to Salvation. In which is
 briefly shown upon how righteous Terms Unbelievers may
become true Christians: and the case of the Deists is reduced to
a short Issue, by Philalethes Cestriensis. 8vo., Lond. 1746."
Y. B. N. J.

"As poor as Job's Turkey."—This proverbial expression is used in the

United States, sometimes with an addition showing how poor he
was, thus: "As poor as Job's turkey, that had but one feather in his
tail;" "As poor as Job's turkey, that had to lean against a fence to
gobble."
Uneda.

Fuss.—Perhaps some of your correspondents can favour the public

with the etymology and date of the word fuss.
W. W.

Suicide encouraged in Marseilles.—In the Lancet of Nov. 30, 1839, it

is stated by De Stone that anciently, in Marseilles, persons having
satisfactory reasons for committing suicide were supplied with
poison at the public expense. What authority is there for this? I
should also like to be informed what was the occasion on which a
suicidal propensity in the Milesian ladies was corrected by an appeal
to their posthumous modesty?
Elsno.

Fabulous Bird.—Among the many quaint and beautiful conceits in

Fuller, there is one preeminently fine: in which he likens the life-long
remorse of a man who has slain another in a duel to the condition of
"a bird I have read of, which hath a face like, and yet will prey upon,
a man; who, coming to the water to drink, and finding there, by
reflection, that he had killed one like himself, pineth away by
degrees, and never afterwards enjoyeth itself."

Where did Fuller read this story? I do not recollect it in Pliny.

V. T. Sternberg.
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and

personal growth!

textbookfull.com