ENPM809B: Building a Manufacturing

Robot Software System

Dr. Craig Schlenoff

Dr. Zeid Kootbally
 About Your Professors
Craig Schlenoff, PhD
National Institute of Standards and Technology (NIST)
• Group Leader, Cognition and Collaboration Systems Group
• Acting Group Leader, Sensing and Perception Systems Group
• Associate Program Manager, Robotic Systems for Smart Manufacturing
Program
• Project Leader, Agility Performance for Robotic Systems Project
IEEE
• Chair, IEEE Ontologies for Robotics and Automation Working Group
• Co-Chair, IEEE Robot Task Representation Working Group
Contact:
• Email: [email protected]
 About Your Professors
Zeid Kootbally, PhD
National Institute of Standards and Technology (NIST)
• Guest Researcher
• Agility Performance of Robotic Systems Teams Member
USC
• Senior Research Associate, Department of Aerospace and Mechanical
Engineering
Contact:
• Email: [email protected]
 Class Introductions
• Who are you?
• What experience, if any, do you have with robotics?
• Why are you interested in the class?
• One interesting fact about yourself.
 Class Goals
• The course will look at the components of manufacturing robots,
including architectures, knowledge representation, planning, control,
safety, standards, and human-robot interaction. Students will explore
the work that is being performed around the world in each of these
areas, and will perform small hands-on exercises in class to gain a
deeper understanding of how a selected set of these technologies can
be applied to real-world challenges. The final project will be to
develop a control system for a simulated robot as part of the ARIAC
Competition.
• This course will have invited presentations from experts in the field.
 Class Syllabus
Class Number Date Topic
1 1/29/2018 Overview, Syllabus, Class Goals, Agility, ARIAC Competition
2 2/5/2018 ROS Interfaces
3 2/12/2018 Robot Programming and ARIAC Interfaces
4 2/19/2018 State of the Art of Robotics
5 2/26/2018 Knowledge Representation
6 3/5/2018 Architectures
7 3/12/2018 High Level Planning
3/19/2018 Spring Break
 Class Syllabus
Class Number Date Topic
8 3/26/2018 Low Level Planning
9 4/2/2018 Simulation
10 4/9/2018 Measurement Science
11 4/16/2018 Safety and Human Robot Interaction
12 4/23/2018 Open Session – Time to Develop Control System
13 4/30/2018 Trial Runs in ARIAC Competition
14 5/7/2018 Team Presentations
 Grading
Category Description Percentage
Quizzes • to be held at the beginning of 20%
most classes
• will cover material presented at
the last class
Homework • Usually continuation of what we 30%
started in class or small
independent research project
Project: Status Checks 10%
Project: Code/Implementation *10 points will be added to score of 10%
winning team*
Project: Presentation 15%
Project: Report 15%
 Software We Will Use
• ROS
• Gazebo
• Protégé (https://protege.stanford.edu/)
• PDDL (available from our CANVAS web site in the Files Section)
• DIA (http://dia-installer.de/)

• Bring your laptop to every class!

 Textbooks
• (required) “Programming Robots with ROS” by Quigley, Gerkey, and
Smart, 1st Edition, 978-1449323899, O’Reilly Media, 2015.
• (optional) “Automated Planning, theory and practice” by Ghallab,
Nau, and Traverso, 1st Edition, 978-1558608566, Elsevier, 2004.
• (optional) “Intelligent Systems: Architecture, Design, and Control” by
Meystel and Albus. 1st Edition, 978-0471193746, Wiley 2001.
 A Typical Class
• Nobody wants to hear us lecture for three hours
• The robotics department wants this to be a hand-on class
• Individual class structure
• Questions and discussion about previous lecture
• Quiz on previous lecture
• Report out from homework
• Lecture
• Hands-on exercise
• This is the first year that this course is being offered so we will adjust
as we go along
 NIST Smart Manufacturing Programs

Measurement Science for Additive Manufacturing

• Develop and deploy measurement science that will enable rapid design-to-product transformation through
advances in; material characterization; in-process process sensing, monitoring, and model-based optimal
control; performance qualification of materials, processes and parts; and end-to-end digital implementation
of Additive Manufacturing processes and systems

Robotic Systems for Smart Manufacturing

• Develop and deploy advances in measurement science that enhance U.S. innovation and industrial
competitiveness by improving robotic system performance, collaboration, agility, and ease of integration into
the enterprise to achieve dynamic production for assembly-centric manufacturing.

Smart Manufacturing Systems Design and Analysis

• Deliver measurement science, standards and protocols, and tools needed to design and analyze SMS based
on a cyber-physical infrastructure for digital and manufacturing systems.

Smart Manufacturing Systems Operations Planning and Control

• Develop and deploy advances in measurement science that enable performance, quality, interoperability,
wireless and cybersecurity standards for real-time prognostics and health monitoring, control, and
optimization of smart manufacturing systems.
 Robotics @ NIST

ARL/DARPA Demos I-III, Robotics CTA, …

RIA/ISO Collaborative Robot Safety

Standards

DOT Integrated Vehicle-Based

Safety Systems Crash Warning ARL/FCS Autonomous
Navigation System TRL-6
A Rich Legacy Evaluation
 Robotics @ NIST
Emergency Response Manufacturing

DARPA Robotics Challenge

Robotic Systems for Smart Manufacturing

DHS Standard Test Methods for Response Robots
http://www.nist.gov/el/isd/ms/rssm.cfm
http://www.nist.gov/el/isd/ms/robottestmethods.cfm
 Robots @ NIST
• Robots
• KUKA LWR 4+ Sensors
• Rethink Robotics Baxter RR
• 2X Universal Robots UR10
• Universal Robots UR5 and UR3
• ABB IRB 14000 Universal
• FANUC LR Mate Arm
• Yaskawa Motoman
• Dexterous robot hands AIM AGV
• Schunk Dexterous Hand
• Robotiq 3-fingered adaptive robot gripper
• Allegro 4-fingered anthropomorphic hand
• Mobile Robotics Testbed
• Mobile platforms
• Custom mobile platform
• AIM AGV forklift
• Adept mobile robot
• Sensor suites
• 8-Camera Vicon Bonita 10
• 12-Camera OptiTrack system
• OptiTrack stereo camera
• Cognex 2.5D capture system
• Biomimetic pressure/force sensors

contact 6 Axis contact

6 Axis
force Load Cell force
Load Cell
 Fulfilling Robotics’ Potential

Current Practice
• Strong, fast, repeatable, tireless
• Ad hoc, expensive custom solutions for
installation, fixturing, tooling, integration
• Lengthy programming & tweaking for
each new task
• Limited or no ability to deal with
variations or errors
• Separated from humans for safety;
cannot collaborate/assist
Fulfilling Robotics’ Potential
Future Vision: Robots that are sensing-
and knowledge-enabled to
• Execute tasks with minimal upfront
programming, without expensive end-
of-arm tooling
• Adapt to variations in part position,
size and other changes in their
workspace, removing the need for
custom fixturing
• Intelligently navigate around the
factory
• Assist humans by working safely and
collaboratively
• Easily installed and integrated with
subsystems and rest of enterprise
 Program Thrusts

Develop measurement science to assure and advance robotics for

manufacturing
• Foundational competences: perception, mobility, and grasping
dexterity
• Safe collaboration with humans and other robots
• Agility (ease of tasking and re-tasking)
• Ease of integration
• Ease of adoption by small manufacturers
 Characterizing Performance of Sensing,
Grasping, and Mobility
• To achieve future vision, robots need to
• have situational awareness: detecting,
identifying, tracking objects and humans in
their surroundings
• be able to grasp, manipulate, and place
objects, using more general purpose grippers
that have diverse forms of sensing
• move around a cluttered and dynamic work
environment safely, not just detecting
obstacles and stopping, but driving around
them
• These basic capabilities must be well-
characterized and robust to reduce risk of
adoption.
• Examples of NIST actions to address these
needs:
• Shepherded 2 test methods for 6D pose
measurement systems in ASTM E57
• Formed robotic grasping metrics working
group under IEEE Robotics & Automation
Society
• Founded ASTM Committee on performance
standards for industrial vehicles F45 in 2015.
 Collaborative Robot Systems

Robots are starting to work alongside

humans. How is their performance teaming
with humans and with other robots
described, implemented, and assessed?
How is their safety assured?

• NIST experiments and analyses

provided technical input to new
international industrial robots
collaborative robots safety technical
specification ISO TS 15066,
specifically for “Speed and
Separation Monitoring” and “Power
and Force Limiting”
• NIST analyses of incompatibilities
and gaps between AGV and
industrial arm safety standards led
to the formation of a new Robotic
Industries Association effort on mobile
manipulator safety standards
Agility
• Manufacturers need robots that are easier to re-task and are more robust to
execution failure during operations.
• Support for small lot productions
• Able to detect and recover from failures to avoid down time, scrap
• NIST is developing definitions of agility, means of testing agility of a system,
and underlying supporting technologies
• Examples
• NIST led the development of new IEEE
standard for knowledge representation
for robot systems.
• NIST is now working with IEEE to develop
a standard industrial robot ontology .
• Per invitation by IEEE, NIST is organizing
an Agility competition. Partnering with
industry (Siemens, Dassault, ABB);
to be held in the cloud.
 Interoperability & Integration
• Integration of robotic systems within manufacturing facilities is impeded by the
inability of robots to easily communicate with the rest of the enterprise, as well
as components, such as sensors or hands.
• NIST is developing the information exchange
methods and tools to automate the integration of
robot systems
• Examples of NIST contributions
• Dissemination of Canonical Robot Command
Language to Robotics community; CRCL is a
formal hardware-agnostic set of movement
commands for the assembly domain.
• MTConnect integration to make a CRCL robot
look like an MTConnect device
(MTConnect is “the communication standard
of choice for manufacturing”)
Robots for SMEs
• Installing and keeping robotic systems operational is challenging and beyond
the technical know-how of many small manufacturers. The calibration of
robots and registration of multiple sensors requires expertise that small- and
medium-sized manufacturers may not have.
• NIST is developing easy to use tools that facilitate the deployment of robots by
small manufacturers.
• Examples
• NIST published an introduction to robotic
system calibration and registration
concepts to educate potential end users.
• EL partnered with MEP to host a
collaborative robotics workshop to reach
SMEs and educate them about robotics.
• EL has developed a simple, 3-point
calibration that can be used to avoid
reprogramming robot positions when a
robot is moved or bumped.
Agility Performance
of Robotic Systems
How Robotics Are Used Now
Kia Sportage Factory Production Line

https://www.youtube.com/watch?v=sjAZGUcjrP8
Class Discussion
• What do you consider robot agility?
• What agility challenges do you think that industry is facing?
• What can be done to make robots more agile?
Exercise
• You need to develop a control system that will make a robot more
agile.
• What are the necessary parts of the control system?
• How do they fit together?
• What information get passed amongst the components?

• Your job: Draw a block diagram, with blocks representing the main
components of the control system, arrows representing the
connections between the components, and text on the arrows
representing the data that is passed.
 What is Robot Agility?
• Hardware agility
– How can different hardware
configurations affect a robot’s ability
to accomplish a variety of tasks?
• Software agility
– How well can a robot adapt/respond
to task failures?
– How well can a robot re-plan when a
new goal is provided to it?
– How can we allow for
interchangeability of robots without
the need for reprogramming?
– How well can a robot respond to
changing environmental conditions
(e.g., non-fixtured tray moves)?
 Big Picture
updates Canned Plans
provides B
Robot robot
capabilities Canonical Robot instructs Robot
Description predicate(s) predicate(s) predicate(s)
Command Language Controller
Model false - true - false -
3 fix continue replan
monitors
truth
interprets value
System Predicate Predicate
Monitor Evaluation Results
Plan Instance
File generates

provides provides
provides
location Metrics location tasks Knowledge representations
available generates info
info
robot
actions A Robot capabilities
Planning

updates
Domain Independent Planner MySQL
Sensor Planning approaches
DB Processing

defines
Test methods
PDDL Domain PDDL Problem
File File
Agility Control Loops
generates
converts converts
A Real-Time World State Evaluation
Ontology (OWL/XML)
B Real-Time Re-Planning
1
Environment Initial Conditions

Color Key Tools and GUIs

2 Actions, Predicates Goal Conditions

Database XML Instances To OWL Init/Goal

1
Decision
converts XML Schema to OWL Kitting Ontology
Files 2

Use Case Loops Predicate Generation GUI

Scenarios 3
Processses

Tools and GUIs

 Core Ontologies for Robotics and
Automation Standard
• IEEE 1872 -Core Ontology for Robotics and Automation
(CORA)
• “… allows for the representation of, reasoning about, and
communication of knowledge in the robotics and automation
domain.”
• http://standards.ieee.org/findstds/standard/1872-2015.html
• First ontology-based IEEE RAS standard
• Articles written about it in Engadget and Scientific Computing
Magazine
• IEEE Ontologies for Robotics and Automation Standards
Working Group
• November 2011 – Became a working group
• July 2014 – Initial standard applied to robots at NIST and
Georgia Tech
• April 2015 – CORA Becomes a Standard (unanimous approval
from ballot group)
• 175 members representing 23 countries
• ~50% educational institutions, ~25% industry, ~25% government
• ~50% US, ~50% non-US
• Strong representation from Europe / Industrie 4.0
C. I. Schlenoff, "Let’s Talk, Robots" Scientific Computing Magazine,
Scientific Computing, 100 Enterprise Dr. Suite 600, Rockaway, NJ, 07866,
United States, (21-Nov-2016)
 IEEE Robot Task Representation
Standards Working Group
• IEEE 1872.11
• “the representation of, reasoning about,
and communication of task knowledge
in the robotics and automation domain.”
• “provides a common set of terms and
definitions structured in a logical theory,
allowing for unambiguous knowledge
transfer among groups of human,
robots, and other artificial systems.”
• October 2015 – Study Group Formed
• February 2017 – Working Group Formed
• Chairs: Craig Schlenoff (NIST), Stephen
Balakirsky (Georgia Tech Research
Institute)
• Letters of support from:
• Caterpillar
• Siemens
• ABB
• Niche Frontiere (Malaysia)
• SkyDrone (Brazil)
Canonical Robot Command Language (CRCL)
• A messaging language for sending
commands to, and receiving
status from a robot and end
effector.
• Provides basic commands that
are independent of the
kinematics of the robot that
executes the commands.
• Ability to utilize set of commands
on different vendor’s robots with
same results
• Implemented using XML Schema
for the information model and
XML for the instance document
F. M. Proctor, S. B. Balakirsky, Z. Kootbally, T. R. Kramer, C. I. Schlenoff, W. P. Shackleford, "The Canonical Robot
Command Language (CRCL)" Industrial Robot-An International Journal, Vol. 43, No. 5, pp. 494-502, (01-Aug-2016)
 Robot Capability Model

• Robot capabilities are usually only found in the

robot expert’s heads
• We are trying to implant this knowledge in the
robot so it can:
• Understand what it can and can not do
• Provide this information for automated high-
level workcell planning
• When failures happen, understand if it is able to
address the failure
• Explore alternative sequences of actions to
accomplish the goal

Courtesy: GTRI

M. O. Shneier, E. R. Messina, C. I. Schlenoff, F. M. Proctor, T. R. Kramer, J. A. Falco, "Measuring and Representing the
Performance of Manufacturing Assembly Robots" NIST Interagency/Internal Report (NISTIR) 8090, (10-Dec-2015)
Allowing for Robot Interoperability Using CORA
Amazon Picking Challenge
 Recognition and Use
• Core Ontology for Robotics and Automation (CORA) Standard
• Won the IEEE-SA Emerging Technology Award (2015) - given once a year to a working group which “advanced, initiated, or
progressed a new technology within the IEEE-SA open consensus process.”
• “These standards and software modules will allow increased competition in the industrial robot marketplace as well as
provide openings to small- and mid-sized companies that have previously been reticent to adopt robot technologies.” Erik
Nieves (Yaskawa Motoman -> PlusOne)
• “The 1872-2015 standard has and will continue to serve as a great resource to developers, users, and systems integrators in
the robotics field. The resulting standard has shown value and has the support of many in the field in robotics, including
representatives in the industrial, service, and healthcare fields.” Gurvinder Virk, ISO TC184/SC2/WG7 convener
• Prominently featured in two journal special issues
• Robotic and Computer Integrated Manufacturing Journal entitled “Knowledge Driven Robotics”
• Industrial Robots journal entitled “Industrial Robot Agility”
• Featured in articles in Engadget and Scientific Computing
• Applied to research efforts in numerous institutions such as Georgia Tech and Lund University
• Currently being applied at Denso North America, a leading supplier of advanced automotive technology,
systems, and components for major automakers, to address the agility limitations they are facing with their
robotic systems in dealing with the wide variety of parts they need to handle.
• Caterpillar started an internal effort to apply the technology to address their robot agility challenge, which is
allowing them to collect and structure manufacturing source data to better target their automation
development initiatives.
36
Class Discussion/Exercise
• How to you measure how agile a robot is?
• What are the appropriate metrics?
• How can you verify that those metrics are correct?
 Measuring Robot Agility
• Developing a set of test methods, to
assess the agility performance of
manufacturing robotic assembly
systems.
• Includes:
• Representative scenarios
• Initially focusing on kitting
• Agility challenges:
• Dropped parts, in-process kit changes,
fixtureless assembly, mechanical
breakdowns
• Metrics
• Time, distance travelled, completion (both
quantitative and qualitative)
• But how do we validate the test
methods?

A. J. Downs, W. S. Harrison, C. I. Schlenoff, "Test Methods for Robot Agility in Manufacturing" Industrial Robot
Journal 8134, Vol. 43, pp. 563-572, (02-Jun-2016)
 Agile Robotics for Industrial
Automation Competition (ARIAC)
• Goal: To test the agility of industrial robot systems, with the
goal of enabling industrial robots on the
shop floors to be more productive, more autonomous, and
to require less time from shop floor workers.
• Simulation-based competition held in the June 2017
timeframe (Gazebo and ROS)
• Teaming closely with the Open Source Robotics Foundation
(OSRF)
• Participation
• 65 teams registered
• 10 teams qualified for finals
• 4 team competed in finals
• Held an ARIAC Workshop at IROS 2017 (September)
• Recent outcomes:
• Portugal team wants to start a local competition building off of
ARIAC
• Invitation from RoboCup Logistics League -Planning and Execution
Competition for Robotics in Simulation to team together
(http://www.robocup-logistics.org/sim-comp)
 How It Works
• Teams are developing robot control systems to handle agility
challenges
• Teams can place sensors in the environment to track objects (for a
cost)
• Teams use ROS interfaces to control actuators, read sensor
information and send/receive notifications
• Retrieving Orders
• rostopic echo /ariac/orders
• Querying Storage Locations of Parts
• rosservice call /ariac/material_locations “material_type: piston_rod_part”
• Querying Sensors
• Beam Break / Proximity Sensor – Is there something in the proximity of the sensor?
(min/max range)
• Laser Sensor – range data
• Camera – models detected
• Controlling the arm (trajectory messages)
• rostopic pub /ariac/arm/command trajectory_msgs/JointTrajectory “
[joint_names], points: [time_from_start in seconds], positions: [x,y,z, roll, pitch,
yaw] …
• Control vacuum gripper
• rosservice call /ariac/gripper/control “enable: true”
• https://bitbucket.org/osrf/ariac/wiki/competition_
interface_documentation

4/7/2017 41
 Industrial Scenarios
• Conducted over 30 interviews with
representatives of industry and academia.
• Consulted previous robotics competition
organizers.
• Amazon Picking Challenge
• Virtual DARPA Robotics Challenge
• Robot Perception Challenge
• Focus on Kitting: the process of gathering
required components for assembly
 Scenario Setup
• Parts arrive in different ways with increasing levels
of difficulty
• The conveyor belt is a 1 m wide plane that
transports objects across the work environment at a
fixed speed of roughly 0.2 m/s. Parts continuously
appear on the belt for the duration of the a trial.
When parts reach the end of the conveyor belt they
are automatically removed. Teams can control the
conveyor belt during development, but not during
the final competition.
• There are eight part bins that may be used for
building kits. Parts in these bins will not be replaced
once used.
• There is a robot arm mounted on a linear actuator
that operates parallel to the conveyor belt. The
linear actuator measures 4 m.
• Two automated guided vehicles (AGV) are located at
either end of the linear actuator. Kits are built on
top of these AGVs. A team will programmatically
signal the AGVs when the kits are ready to be taken
away. The signaled AGV will depart for a short
period and then return with an empty tray.
 The Robot Arm
• The robot arm used in each
trial will be a Universal
Robots UR10.
• The robot arm's position is
controlled through the linear
actuator on which it is
mounted.
• The end of the arm is
equipped with a vacuum
gripper. The vacuum gripper
is controlled in a binary
manner and reports whether
or not it is successfully
gripping an object.
 Sensors
• A team can place sensors around
the environment. Each sensor has
a cost that factors into the final
score.
• Available sensors are:
• Break beam: reports when a beam is
broken by an object. It does not
provide distance information.
• Laser scanner: provides an array of
distances to a sensed object.
• Cognex logical camera: provides
information about the pose and type
of all models within its field of view.
• Proximity: detects the range to an
object.
 Orders and Faulty Parts
• Orders
• An order is an instruction containing kits for the
robot system to complete.
• Each order will specify the kit to be assembled, i.e.
the list of parts to be put in the kit.
• Each specified part has the following structure:
• The type of part.
• The position and orientation of the part on the tray.
• Faulty parts
• Above each AGV is a quality control sensor that
detects faulty parts. If faulty parts are detected
while teams are filling trays, those parts should be
removed from the tray and replaced with another
part of the same type. Faulty parts are considered
unwanted parts: they will not count for any points
when the kit is submitted, and they will cost teams
the all-parts bonus if left in trays (see Scoring,
below).
 Agility Challenges
• Identified 36 agility challenges based on
conversations with industry and literature
reviews.
• Ranked each challenge based upon three
criteria:
• Industry’s perception of how critical the challenge
is
• Difficulty of challenge, based upon perceived team
capabilities
• Difficulty to implement in Gazebo
• Looked for optimal challenges among the three
criteria
• Qualifier 1 focused on parts dropping from the
gripper
• Qualifier 2 focused on in-process kit changes
and more complicated drop scenarios
• Qualifier 3 focused on faulty parts and needing
to flip parts
ARIAC Competition Highlights

48
Class Discussion/Exercise
• How can you put the appropriate metrics together into an equation
that can provide a “number” as to how agile a robot is?
 Metrics
• Time-based
• How long did it take to complete
the task?
• Motion time vs. thinking time?
• Accuracy-based
• Number of parts in the kit tray
• Parts in right position/orientation?
• Cost-based
• How much did it cost for the
infrastructure in the environment?

4/7/2017 50
 Competition Process
• Each trial will consist of the following steps:
• The robot programmatically signals that it is able to begin accepting orders.
• The first Order (Order 1) is sent to the robot.
• A fixed amount of time is allowed to complete the order.
• In the case of the Dropped Part testing method, up to three parts will be forcibly
dropped from the gripper.
• In the case of the In-Process Kit Change testing method, a new Order (Order 2) will
be issued that is of higher priority than the previously issued Order 1. When Order 2
is complete, building of Order 1 is to resume.
• The robot signals programmatically when a kit is complete and ready for quality
control.
• The robot system will be notified that the trial is over. The trial is over when time
runs out or all Orders have been fulfilled.
 Qualifiers
• Three Qualifiers:
• First Qualifier – Jan 27, 2017 – Feb 24, 2017
• Second Qualifier – March 6, 2017 – April 3, 2017
• Third Qualifier – April 10, 2017 – May 8, 2017
• Why Three?
• Gives teams a lot of opportunities to get involved
• Allows us to show teams the direction we are going with the
challenges
• Allows us to understand teams’ capabilities so we can make
the competition at the right level of difficulty
• Qualifier 1 Observations
• 11-12 attempts, 7 submissions
• Bar purposely set low
• All teams that responded qualified
• Scores varied considerably

4/7/2017 52
 About the Participants
• About 50% educational institutions and 50% industry (almost all small companies)
• Educational institutions (subset):
• Case Western Reserve University
• Cranfield University
• Idaho State University
• NC State
• Northeastern University
• Simon Fraser University
• University of Southern California

• Industry (subset):
• High West Labs
• Patriot Robotics
• Robomakery
• Space Weavers
• Stanley Robotics
• TRACLabs

• Two teams work with high school STEM students to teach them robotics
• At least one college class being taught around the competition (Case Western)

53
 Components of a Control System
ROS and ROS Interfaces
• Robot Operating Systems (ROS)
Architecture
• ROS master, nodes, topics
• Catkin workspace and build systems
• Launch files
• Gazebo simulator

• Hands-On: Develop a small ROS program

 Components of a Control System
Robot Programming and ARIAC Interfaces
• Building and running ARIAC packages
• Placing sensors in the environment
• Turning conveyor belts on and off
• Turning the suction on and off on the
gripper
• Moving the arm in the environment
to grasp parts and put them in the
AGV
• Hands-On: Control a robot in the
ARIAC simulation environment
 Components of a Control System
State of the Art of Robotics
• Guest Lecturer: Fred Proctor
• Dexterous Manipulation
• Mobility
• Sensing and Perception
• Collaborative Robots
• Simulation
• Planning

• Will also involve a visit to the Robotics Realization Lab at UMD

 Components of a Control System
Knowledge Representation
• To understand the various types
of knowledge representation
that are used in robotics
• To see an example of the
knowledge acquisition process Prediction Equations

• To see examples of how various 1

CONDUCT TACTICAL ROAD MARCH TO ASSEMBLY AREA

NewCommand S1 DetermineMarchColumnOrganization:
Databases
types of knowledge 2

3
S1

S2
S3
MarchOrganizationDetermined

TentativePlans_Done
S2
S3

S4
all_FormTacticalRoadMarchOrganization
MakeTentativePlan: Determine_Route,
_FireSupport, _MovementFactors, _AA
sp3_PrepareForRouteReconnaissance

representation can be applied

4
RoadMarchOrganizationInPlace qp_PrepareToOrganizeAssemblyArea

5 S4 sp3_ReadyToConductRouteRecon S5 sp3_ConductRouteReconnaissance

6 S5 qp_ReadyToOrganizeAA S6 qp_FollowReconPlatoonToAssemblyArea

Perception
• To perform hands-on exercises
7 S6 qp_ClearOfStartPoint S7 mb_tp_PrepareForRoadMarch

8 S7 mb_tp_ReconToStartPoint_Done S8 PrepareDetailedMovementPlans:
Co
Pla st B io n
lat
9 S8 sp3_RouteRecon_Done S9 sp3_EstablishAssemblyAreaSecurity
nn ase
mu

to see how to develop an

i ng d
10 S9 qp_AtReleasePoint S10 qp_ConductAreaReconnaissanceOfAA
Si
11 S10 qp_AreaReconOfAADone S11 qp_OrganizeAssemblyArea

ontology
12 S11 qp_Status_AssemblyAreaSuitable S12 1st_mb_Unit_MoveIntoRoadMarchFormation

S12 mb_Unit_ExecuteTacticalRoadMarch
13 S12 mb_Unit_AtStartPoint
next_mb_unit_MoveIntoRoadMarchFormation State
S13 mb_Unit_ExecuteTacticalRoadMarch
14 S12 mb_LastUnit_AtStartPoint
tp_MoveIntoRoadMarchFormation Machine Ontology Use Cases
Planning

• Hands-On: Develop an ontology

15 S13 tp_TrailPartyAtStartPoint S14 tp_SupportMarchColumnMovement

16 S14 HaltCriteriaMet S15 mb_tp_ExecuteScheduledHalt

M
Ob ovi n
17 S15 HaltEnded_ReadyToContinueMarch S14 mb_tp_ResumeExecutionOfTacticalRoadMarch

lue Pr ject g

using OWL and Protégé

18 DisabledVehicleUnscheduledHalt S15 mb_tp_ExecuteUnscheduledHalt

S14 mb_Unit_AtReleasePoint Va ent ed

ict
19 S16 1st_mb_Unit_OccupyAssemblyArea m i on
dg
qp_AssemblyAreaReady

20 S16 mb_Unit_AtReleasePoint S16 mb_Unit_OccupyAssemblyArea Ju

21 S16 tp_AtReleasePoint S17 tp_OccupyAssemblyArea
Autonomous Vehicle Ontologies
22 S17 mb_tp_OccupyingAssemblyArea S18 all_FormStandardTroopOrganization

23 S18 AllUnitsSecureInAssemblyArea S0 TacticalRoadMarchToAssemblyArea_Done

24 NewSituationReport UpdateDetailedMovementPlans

25 NewHigherLevelInformation UpdateDetailedMovementPlans

State Tables
 Components of a Control System
Architecture Perception Planning and Control

• To understand the benefits of VALUE

JUDGMENT
COMMANDED
TASK (GOAL)

different types of robot OPERATOR

EVALUATION
PERCEIVED EV PL

SITUATION
OBJECTS & AL AN INTERFACE

RESULTS
architectures
UA
EVENTS

PLAN
TI
O
N

PLAN

• To understand what a node in

UPDATE
S ENS ORY WORLD BEHAVIOR
PROCES S ING MODELING GENERATION
PREDICTED STATE
INPUT

an architecture looks like and OBSERVED

INPUT
KNOWLEDGE
DATABAS E
COMMANDED
ACTIONS (SUBGOALS)

how it works
• To explore hierarchical
architectures to see how Battalion Formation SP
WM BG
SURROGAT E BAT T ALION Plans for next 24 hours

information is represented at Platoon Formation

Section Formation
SP

SP
WM BG SURROGAT E PLAT OON Plans for next 2 hours

different levels of abstraction

WM BG
SURROGAT E SECT ION Plans for next 10 minutes
Tasks relative to nearby objects
Objects of attention SP WM BG VEHICLE Plans for next 50 seconds
Task to be done on objects of attention

OPERATOR INTERFACE
• To go through examples of how Surfaces Attention Communication M ission Package Locomotion SUBSYST EM

architectures are populated SP WM BG SP WM BG SP WM BG SP WM BG

5 second plans
Subtask on object surface
Obstacle-free paths

with information Lines

SP WM BG SP WM BG SP WM BG SP WM BG
PRIMIT IVE
0.5 second plans

• Hands-On: Design an
Steering,
velocity
Points
SERVO

architecture for the ARIAC SP WM BG SP WM BG SP WM BG SP WM BG SP WM BG SP WM BG SP WM BG SP WM BG

0.05 second plans
Actuator output

competition
SENSORS AND ACTUATORS
 Components of a Control System
High Level Planning
• Defining of AI planning and its
applications in the real world
• Understanding how planning is used in
robotics through the description of
conceptual models
• Understanding the different
ingredients of AI planning
• Developing PDDL files for the kitting
domain
• Hands-On: Design and develop a PDDL
plan
 Components of a Control System
Low Level Planning
• Using MoveIt! ROS packages
• Exploring the concept of
motion planning and search
space
• Getting robots to automatically
determine how to move while
avoiding collisions with
obstacles
• Collision-free trajectories
• Reaching goal location as fast as
possible
• Hands-on: Use MoveIt! To
perform motion control
Components of a Control System
Simulation

• Gazebo • V-REP
 Components of a Control System
Measurement Science
• What is performance evaluation and
why is it important?
• What are some of the approaches to
evaluating the performance of
intelligent systems?
• Look at some performance evaluation
examples
• Force-based control and
grasping/manipulation
• Intelligent vehicles
• Mobile manipulation
• Robot agility
• Response robots
• Hands-On: Determine an equation to
measure the performance of an ARIAC
robot
 Components of a Control System
Safety / HRI
• Collaborative robots, definitions and
capabilities;
• Collaborative robot safety and the risk
assessment process;
• State of the art of human-machine interfaces;
• Performance requirements for collaborative
human-robot interaction; and
• Performance metrics for human-robot teaming.

• Hands-On:
• Perform a risk assessment of a human-robot
collaborative environment
• Determine relevant information exchange between
a robot and a human
 What To Do Now
• Read up on ARIAC:
• www.nist.gov/ariac
• http://gazebosim.org/ariac

• Install Software:
• Linux – you will need a dual boot machine.
• Useful YouTube Video: https://www.youtube.com/watch?v=1uVcsFhv2Vo
• Ubuntu Distribution: http://www.ubuntu.com/download/desktop
• USB Installer: http://www.pendrivelinux.com/universal-usb-installer-easy-as-1-2-3/
• Bit Torrent (to download Ubuntu): http://www.bittorrent.com/
• ROS
• Gazebo
• Protégé - http://webprotege.stanford.edu/ (no need to install – can access from the web)
• PDDL (available from our CANVAS web site in the Files Section)
• DIA ((http://dia-installer.de/)

• Work through ROS Tutorial at http://wiki.ros.org/ROS/Tutorials

 Next Week
• Quiz on ARIAC Competition and Concepts Discussed in Lecture
• Lecture on ROS