IE7295: Applied Reinforcement Learning

Prof. Mohammad Dehghani

"Sell your cleverness and buy bewilderment. Cleverness is mere opinion, bewilderment is intuition"
- Rumi
 2

Course Description
Reinforcement Learning (RL) is a machine learning approach that aims to formulate
an optimal policy from the trial-and-error learning process that is needed in many
real-world applications in which no explicit instructive information is available. It
has roots in operations research, behavioral psychology and artificial intelligence.
The goal of the course is to introduce the basic mathematical foundations of rein-
forcement learning, as well as highlight some of the recent directions of research.
The application of Reinforcement Learning in different domains including: manu-
facturing, robotics, healthcare, game playing, etc, will be discussed.

Applied Reinforcement Course Objectives

Learning This course is primarily designed to introduce reinforcement learning to graduate
level students. Students will learn different methods that are used for learning
IE7295 based on feedback. On completion of the course, students should be able to:
 Identify basic concepts, terminology, theories, models and methods in rein-
forcement learning
Instructor Info  Develop and systematically test a number of basic methods in reinforcement
learning
g Mohammad Dehghani  Evaluate different learning algorithms experimentally and interpret and doc-
ument results of experimental studies
U Office Hrs: Thu 2:00pm-3:30pm  account for basic methods and limitations in reinforcement learning
½ Zoom (make an appointment)  Apply reinforcement learning on real-life problems in order to be familiar with
its basic possibilities and limitations and thereby be able to assess which prob-
Website lems can be solved with these technologies
 Be able to implement, analyze and evaluate simple systems based on rein-
@ [email protected]
forcement learning

Reference Books
Course Info Main Textbook

 Prerequisite: 1. Richard S. Sutton and Andrew G. Barto, Reinforcementt Learning: An

* IE 6200 - Engineering Proba- Introduction’ MIT Press, Second edition , 2018. PDF.
bility and Statistics. Additional References (Optional)
* A previous exposure to the ba-
sic paradigms of algorithm de- 1. Csaba Szepesvari, ’Algorithms for Reinforcement Learning’, Morgan and
sign and analysis and familiarity Claypool, 2010. Link
with Python are essential. 2. Lucian Busoniu, Robert Babuska, Bart De Schutter, Damien Ernst
’Reinforcement Learning and Dynamic Programming Using Function
 Mon Approximators’, CRC Press, 2017. Link
3. Aurlien Gron, ’Hands-On Machine Learning with Scikit-Learn and TensorFlow:
U 6:00pm - 9:30pm
Concepts, Tools, and Techniques to Build Intelligent Systems’, O’Reilly Media,
½ Behrakis Health Sciences Cntr First edition , 2017. Link
Room 007
Course Outline
The following is a tentative schedule of the course.
TA Info  Introduction to Reinforcement learning
 Part 1: Tabular Solution Methods
g Yash Nema ◦ Multi-armed Bandits (MABs)
U Office Hrs: Thu & Fri 6:00pm- ◦ Markov Decision Processes (MDPs)
8:00pm ◦ Dynamic Programming (DP)
◦ Monte Carlo Methods (MCM) and Bootstrapping
½ Zoom (make an appointment) ◦ Reinforcement learning with Temporal Difference (TD), i.e.Sarsa and
Q-learning
@ [email protected]
◦ Planning and Learning with Tabular Methods (Dyna, Dyna-Q)
 Part 2: Prediction with Approximate Solution Methods
◦ Value-function Approximation
◦ Artificial Neural Networks (Deep Reinforcement Learning)
◦ Semi-gradient n-step Sarsa
◦ Control with Approximate Solution Methods
Software Tools
You need to have access to a python development environment for this course.
 Google Colab
◦ The environment is already set up. There is no installation of packages required. You can run any python
notebook on this environment. You need an internet connection to run the code. Link
 Jupyter-notebook-Local Development Environment
◦ Please download a conda manager, either anaconda or miniconda. You can go ahead and download the
environment.yml file and then run the following command on the terminal. In Windows 10, it is cmd or powershell.

$conda env create -f environment.yml

This created the conda environment with the name IE7295 with the base packages required for this course.

Grading Scheme
Assignments
Labs
Quizzes
15%
5% 25%

30% 20%

5%
Exams Project

Presentation

 Assignments (Individual)
◦ Students will answer theoretical questions based on the coursework.
 Labs (Individual)
◦ Students will be asked to customize or extend the algorithms explained in the labs for different needs.
 Quizzes (Individual)
◦ Students will be required to complete these take-home quizzes on Canvas. Questions will be from the to be taught
chapter(s).
 Exams: Midterm and Final Exam (Individual)
◦ Both exams are equally weighted.
 Project (Individual or Group based)
◦ Students will work in groups (2-3 memebrs) to apply RL algorithms to solve IE-related or real-life problems. Each
group can include 2-3 members.Groups will present their project in the class in two-stages: (1) Project proposal
presentation, (2) Final Presentation
◦ The project timeline will be announced as the course progresses.
 Presentation (Group based)
◦ Groups will be asked to have a short presentation related to the coursework in class. The topic for the
presentation will be provided.

Grading
Your grades will be posted on CANVAS. You can check your running grade at any time, so there are no surprises. In fact, you
should audit the correct entry of your grades. No grade corrections will be made after the last week of the semester. Final
Grades and Course Letter Grades are not curved and are assigned as follows:

Letter Grade Points Letter Grade Points Letter Grade Points

A 93-100 B 83-86 C 73-76

A- 90-92 B- 80-82 C- 70-72

B+ 87-89 C+ 77-79 F <70

3
Emailing Us?
Please use the following format for the subject line of your email to avoid miss communication. This subject line will reduce
the confusion.
IE7295 - Sec # - Subject Line
Make-up Policy
◦ No makeup opportunity will be given for any homeworks/lab assignment/projects.
◦ Students unable to attend classes are responsible for obtaining information regarding announcements and lecture
content

Class Policy
◦ Students are expected to attend all class sessions and are responsible for all materials presented in class. Past
experience has shown students who attend regularly and interact with instructors/classmates tend to be better overall.
◦ Laptops: You may need to use your laptop during the class upon the instructor’s notes
◦ Phones/tablets/smart watches: Electronic devices may not be used during the class. If you have class work, social
networking or sports-score browsing, that is more important to do, then please skip class. Life is full of such choices.
◦ Discussion between students for homeworks and projects is permissible. However the ideas / solutions / codes must
not be copied. Each individual submission must be an outcome of individual effort

Recording of Classes
Classes will be recorded to enable all students to review material covered in synchronous classes. Please contact me if you
have any concerns.

Academic Integrity
A commitment to the principles of academic integrity is essential to the mission of Northeastern University. The promotion
of independent and original scholarship ensures that students derive the most from their educational experience and their
pursuit of knowledge. Academic dishonesty violates the most fundamental values of an intellectual community and
undermines the achievements of the entire University.
As members of the academic community, students must become familiar with their rights and responsibilities. In each
course, they are responsible for knowing the requirements and restrictions regarding research and writing, examinations of
whatever kind, collaborative work, the use of study aids, the appropriateness of assistance, and other issues. Students are
responsible for learning the conventions of documentation and acknowledgment of sources in their fields. Northeastern
University expects students to complete all examinations, tests, papers, creative projects, and assignments of any kind
according to the highest ethical standards, as set forth either explicitly or implicitly in this Code or by the direction of
instructors.
Go to Academic Integrity website to access the full academic integrity policy.

Student accommodations
Northeastern University and the Disability Resource Center (DRC) are committed to providing disability services that enable
students who qualify under Section 504 of the Rehabilitation Act and the Americans with Disabilities Act Amendments Act
(ADAAA) to participate fully in the activities of the university. To receive accommodations through the DRC, students must
provide appropriate documentation that demonstrates a current substantially limiting disability.
For more information, visit Disability Resource Center website

Northeastern University Copyright Statement

This course material is copyrighted, and all rights are reserved by Northeastern University. No part of this course material
may be reproduced, transmitted, transcribed, stored in a retrieval system, or translated into any language or computer
language, in any form or by any means, electronic, mechanical, magnetic, optical, chemical, manual, or otherwise, with the
express prior written permission of the University.

Diversity and Inclusivity Statement

Northeastern University is committed to equal opportunity, affirmative action, diversity and social justice while building a
climate of inclusion on and beyond campus. In the classroom, members of the University community work to cultivate an
inclusive environment that denounces discrimination through innovation, collaboration and an awareness of global
4
perspectives on social justice. It is my intention that students from all backgrounds and perspectives will be well served by
this course, and that the diversity that students bring to this class will be viewed as an asset. I welcome individuals of all
ages, backgrounds, beliefs, ethnicities, genders, gender identities, gender expressions, national origins, religious affiliations,
sexual orientations, socioeconomic background, family education level, ability - and other visible and nonvisible differences.
All members of this class are expected to contribute to a respectful, welcoming and inclusive environment for every other
member of the class. Your suggestions are encouraged and appreciated.
Please visit the website for complete information on Diversity and Inclusion.

TITLE IX
Title IX of the Education Amendments of 1972 protects individuals from sex or gender-based discrimination, including
discrimination based on gender-identity, in educational programs and activities that receive federal financial assistance.
Northeastern's Title IX Policy prohibits Prohibited Offenses, which are defined as sexual harassment, sexual assault,
relationship or domestic violence, and stalking. The Title IX Policy applies to the entire community, including male, female,
transgender students, faculty and staff.
In case of an emergency, please call 911.
Please visit the website for a complete list of reporting options and resources both on- and off-campus