Institute of Automotive Technology
Faculty for Mechanical Engineering
Technical University of Munich
Autonomous Driving Software Engineering
Prof. Dr.-Ing. Markus Lienkamp
Phillip Karle, M. Sc.
Lecture – 90min Practice – 45min
1 Introduction: Autonomous Driving 1 Practice
Karle Karle
2 Perception I: Mapping 2 Practice
Sauerbeck Sauerbeck
3 Perception II: Localization 3 Practice
Sauerbeck Sauerbeck
4 Perception III: Detection 4 Practice
Huch Huch
5 Prediction 5 Practice
Karle Karle
6 Planning I: Global Planning 6 Practice
Trauth Trauth
7 Planning II: Local Planning 7 Practice
Ögretmen Ögretmen
8 Control 8 Practice
Wischnewski Wischnewski
9 Safety Assessment 9 Practice
Stahl Stahl
10 Teleoperated Driving 10 Practice
Feiler Feiler
11 End-to-End 11 Practice
Betz Betz
12 From Driver to Passenger 12 Practice
Fank Karle
8-2
Objectives for Lecture 8: Control
Depth of understanding
After the lecture you are able to…
Remember Understand Apply Analyze Evaluate Develop
.. understand the necessity of stabilizing control actions
and the responsibilities of the control algorithm within the
autonomous driving software stack
.. understand the most important control concepts for
autonomous driving (geometric path following controllers,
state-space controllers and model predictive control)
.. design and tune geometric path following controllers and
state-space lateral path tracking controllers
.. design and tune a velocity controller for basic cruise
control in highway scenarios
8-3
Autonomous Driving Software Engineering
Prof. Dr. Markus Lienkamp
Alexander Wischnewski, M. Sc.
Agenda
1. Introduction
2. Basic control (Geometric & PID)
3. Model-based control
4. Model predictive control
5. Summary
Lecture Overview
12
Human-Machine-Interface
11
End-to-End
Modular Software Stack
5 8
Sensors Perception Prediction Planning Control Actuators
2
Mapping Global
3 6
Localization Behavior
4
Detection 7 Local
9 10
Safety Teleoperated Driving
X = Lectures
8-5
Role of control in the AV software stack
Target trajectory Steering
Vehicle position Throttle &
Brakes
IMU,
Wheelspeeds…
Control module
▪ Control is focussed on the short-term behavior (roughly 2 seconds)
and updates frequently (at minimum with 20 Hertz)
▪ Real-time operation is crucial
▪ Control needs maximum knowledge about the vehicle
8-6
Why do we actually need feedback control?
How engineers like to think What it‘s actually like:
about the world:
▪ The real world is full of uncertainties and disturbances
▪ Control is about taking into account those uncertainties and
manage them in a structured way to achieve the desired task
8-7
The evolution of autonomous vehicle control
Geometric controllers
Stanley – Stanford (2005) Talos – MIT (2007)
Model-based
feedforward and
feedback controllers
Shelley – Stanford (2015) Roborace – TUM (2018)
Model predictive
controllers
1:43 Racing - ETH (2015) Indy Challenge – TUM (2021)
8-8
Additional Slides
The evolution of autonomous vehicle control
The evoluation of autonomous vehicle control can be divided into three major development steps:
• Inspired from the robotics community, the first control algorithms have been inspired by geometric
concepts such as planning a circular movement towards the target path or aligning the steering
wheels with the path. These approaches show good performance for low to medium speeds.
• When increasing the speed and accelerations of the autonomous driving prototypes, more
thorough analysis methods from control theory and system dynamics became popular within the
research community. They allow to design explicit state-feedback controllers and consider more
detailed effects such as vehicle yaw dynamics, steering actuator dynamics and others.
• In recent years, the interest gathers around model predictive control algorithms. These use a
dynamic model of the system to predict the future behavior for a few seconds and determine
optimal control actions. In contrast to explicit state-feedback controllers, they can use and exploit
knowledge about constraints and uncertainties. Furthermore, they can be extended to more
complex (nonlinear) vehicle dynamics.
Image sources:
https://stanford.edu/~cpiech/cs221/img/stanley1.jpg
https://www.researchgate.net/profile/Emilio_Frazzoli/publication/228794498/figure/fig1/AS:30069332
9072128@1448702284145/Talos-Team-MITs-race-vehicle-at-the-DARPA-Urban-Challenge.png
https://news-media.stanford.edu/wp-content/uploads/2019/03/25110734/friction_shelly9-555x370.jpg
https://control.ee.ethz.ch/research/team-projects/autonomous-rc-car-
racing/_jcr_content/par/textimage_1984079938/image.imageformat.lightbox.1404291081.jpg
1- 9
Control theory basics – System dynamics I
Input Output
System
Examples:
▪ Spring-mass-damper
systems
▪ Inverse pendulum
▪ Autonomous vehicles
8 - 10
Control theory basics – System dynamics II
System matrix State Input matrix
Input
Output
Output matrix Feedthrough matrix
Imaginary axis
Poles of the
System dynamics are given system
from the spectrum of A: Real axis
8 - 11
Additional Slides
Control theory basics – System dynamics I and II
• A dynamic system is a mathematical description of the relation between an input u and an output
y signal. This description is usually given in form of an ODE (Ordinary Differential Equation).
• The system has so-called states x. These are variables which allow us to formulate the system
behavior in form of a set of first-order ODE for each of this variables.
• The standard system description consists of the system dynamics and the output equation. The
former describes the timely behavior of the states as a reaction to the inputs and the initial state.
The latter describes the relation between the state and the output.
• An important special case are linear time-invariant systems (LTI-Systems). They can be
described using matrix algebra only and can be analyzed using Laplace-Transformation theory.
• The system dynamics are specified via the so-called characteristic equation
• The roots of this equation are called the „poles“ and they are related to the stability and
convergence properties of the dynamic system
• If all poles have negative real parts the system is stable
• The absolute value of the negative real part of the poles determines the convergence speed
• The imaginary parts of the poles is related to it‘s oscillation characteristics
1- 12
Control theory basics – Standard control loop I
Control error Disturbances
Setpoint Input Output
Controller System
Examples for control loops in autonomous vehicles:
Vehicle motion Steering angle Brake pressure
8 - 13
Additional Slides
Control theory basics – Standard control loop I
The basic control loop is divided into the main components: sensor, dynamic system under control
(plant), actuator (can also be considered part of the plant) and the controller.
• All parts of this pipeline are packed with uncertainties:
• Sensor calibration errors
• Sensor noise and faulty measurements
• Complex (and unmodelled) actuator dynamics
• External disturbances such as wind gusts, inclination and rolling resistance
• Parametric uncertainties in the plant (e.g. number and weight of passengers)
• In an idealized world a pure feedforward controller would be sufficient → plan the sequence of
actuation commands based on the model of the plant and execute this plan until the task is done
• Uncertainties require corrective actions and a stable feedback loop
1- 14
Control theory basics – PID controller design I
Block diagram with transfer functions:
Control
error
Control Control
value gains
▪ Three main parts: Proportional / Integral / Derivative
▪ Standard approach for feedback control in industrial applications
8 - 15
Control theory basics – PID controller design II
▪ Tuning often done empirically during system operation
▪ Works well for systems with low complexity (e.g. mass-spring-
damper system)
8 - 16
Control theory basics – PID controller design III
▪ Zero steady-state in the presence of uncertainty or external
disturbances requires integral action
▪ Alternative: high feedback gains to suppress disturbances but
this might lead to oscillations in real world scenarios 8 - 17
Control theory basics – State-space design
• Algorithmic method based on
designing a state-feedback matrix
• Analyze characteristic equation of
closed-loop dynamics
• Possibilities to determine controller
• Pole-placement via coefficient
comparison
• LQR controller design
8 - 18
Autonomous Driving Software Engineering
Prof. Dr. Markus Lienkamp
Alexander Wischnewski, M. Sc.
Agenda
1. Introduction
2. Basic control (Geometric & PID)
3. Model-based control
4. Model predictive control
5. Summary
Decoupling of longitudinal and lateral dynamics
Joint trajectory Decoupled longitudinal
tracking control and lateral control
Real vehicles have
✓ coupled dynamics ✓ Controllers are easy to
design and tune
✗ Difficult to handle due to
nonlinearities
✗ Less accurate for high
acceleration and dynamic
maneuvers
✗
Advanced control
algorithms necessary
8 - 20
Stanley controller
Lane
▪ Determine cross-track error and centerline
heading error at the front axle
▪ Align front wheels with path and
add a correction term
▪ The control gain adjusts the
convergence speed
8 - 21
Additional Slides
The stanley controller is a geometric controller. This name originates from the geometric interpration for
the control laws. The PID control concept is more sophisticated but can still be designed without any
knowledge about the vehicle dynamics. In contrast to the geometric controllers, it adds an integral term
which helps to mitigate external disturbances such as cross-wind or banking, but can also lead to
difficulties such as integrator wind-up.
All of the presented concepts are suitable for low-medium speed and acceleration driving (urban
scenarios and easy high-way situations), depending on the tuning efforts spent and the quality of the
vehicle hardware.
Literature:
• S. F. Campbell, “Steering control of an autonomous ground vehicle with application to the DARPA
urban challenge,” University of Notre Dame, 2005
• Snider, Automatic Steering Methods for Autonomous Automobile Path Tracking
1- 22
Velocity PI controller
Disturbances such as
wind, inclination, …
Setpoint Speed Longitudinal Force Speed
PI- Longitudinal
Controller Dynamics
▪ Longitudinal dynamics are a single integrator dynamic, therefore
no derivative part is needed
▪ Integrator required to achieve zero steady-state error due to
external disturbance
8 - 23
Additional Slides
The velocity control task receives a lot less interest within the control engineering community, however,
its a great example for the success of easy to design and tune control concepts.
• The system dynamics consists of a single integrator (the vehicle mass) and a low-pass modelling the
actuation delay within the brake system and the engine. The latter can often be neglected as in our
example presented in the lecture.
• The influence of modelling error (driving resistances such as aerodynamic drag or tire roll resistance)
and the presence of wind gusts and road inclination makes it necessary to add an integral part to the
control law. A derivative part can be used but is optional as only one integrator is present in the open-
loop system.
Even though the velocity control task is simple from a control engineering point of view, it has two
important consequences for the overall performance of the autonomous driving software:
• It contributes heavily to ride comfort. Jerky behavior or noise corrupted sensors leading to oscillating
force requests have to be prevented.
• The performance of the lateral controller can depend significantly on the longitudinal controller. If the
actual speed deviates a few percent from the target speed this might have significant consequences
on the resulting lateral accelerations. This is due to the quadratic influence of velocity on the lateral
acceleration.
1- 24
Autonomous Driving Software Engineering
Prof. Dr. Markus Lienkamp
Alexander Wischnewski, M. Sc.
Agenda
1. Introduction
2. Basic control (Geometric & PID)
3. Model-based control
4. Model predictive control
5. Summary
Improved velocity control with model knowledge
Setpoint stabilization with PI-controller:
Disturbance 3000N
8 - 26
Improved velocity control with model knowledge
Setpoint tracking (e.g. changing speed limit) with PI-controller:
Disturbance 3000N
8 - 27
Improved velocity control with model knowledge
Feedforward
Controller
Control Disturbances
value
Measured
Setpoint
value
Feedback
System
Controller
▪ Two degree of freedom structure allows to design separate
dynamics for setpoint changes and disturbance influences
▪ One of the most common structures in industrial applications
8 - 28
Improved velocity control with model knowledge
Vehicle longitudinal
dynamics model
Force
request Velocity
Vehicle mass Integrator
System inverse for feedforward control
Differentiator Vehicle mass
Velocity Setpoint Feedforward
setpoint acceleration force request
Low-pass
8 - 29
Improved velocity control with model knowledge
Setpoint tracking (e.g. changing speed limit) with PI-controller and
feedforward control:
Disturbance 3000N
8 - 30
Additional Slides
• The two-degree-of-freedom (2-DOF) control structure allows for seperate design of setpoint and
disturbance dynamics. It is well-known for its high performance for trajectory tracking
applications. The general idea is to do „everything we know about the system“ already before a
control error through a setpoint change occurs.
• This concept allows to track trajectories with fast dynamics why keeping smooth and cautious
feedback compensation via the feedback controller.
• Feedforward designs are oriented around the inverse of a plant. The inverse generates a
suitable input sequence such that the plant output matches the desired setpoint trajectory.
• There are many ways to calculate such a feedforward control law:
• Directly obtain the required input sequences or derivatives from the trajectory planning
algorithm/method, e.g. via splines or other basis functions.
• Approximate it via an inverse transfer function model. The exact inverse is usually not
realizable in this form due to the resulting non-causal transfer functions (numerator
polynomial order is higher than denominator polynomial order). This reflects the fact that
numerical inversion takes some time to work correctly.
• Use a model-predictive-control law (see chapter 5 for more details on this concept).
1- 31
Kinematic single track model
▪ Dynamic equations derived from kinematic
constraints only → no tire slip and forces
position heading
Steering Vehicle speed
angle
8 - 32
Additional Slides
Vehicle dynamics models in motion controller design
• There are three main models used for motion controller design:
• Kinematic vehicle models
• Linear single track models
• Nonlinear single track models
• It is only very rarely that more complex models are used for controller design. The testing of the
designed controller should however still be done with more complex models such as a nonlinear dual
track model, considering effects of suspension as well as combined tire effects.
• The kinematic vehicle model is considered sufficient for slow driving at low lateral accelerations. It
represents the key relation between vehicle movement and position dynamics and is usually a good
choice to get started.
• If the speeds or accelerations are in a medium range (high-way driving or accelerations around 3-
5mps2), changing to a linear single track model should be considered. The key advantage here is that
it models the self-steering behavior and gives a coarse estimate of the vehicles lateral dynamics.
• The nonlinear single track model is often used for high-performance applications such as racing. It
models the tire behavior at the handling limits better than its linear counterpart, however, this also
poses difficulties during the control design process.
1- 33
Important relations in vehicle dynamics
Steady-state acceleration GG-Diagram
Steady-state yaw rate
Steady-state curvature
▪ Max. combined accelerations
for overall vehicle
▪ Different forms are possible
depending on the vehicle
8 - 34
Frenet coordinate frame
Progress
along the path
Lateral
deviation
8 - 35
Trajectory specification in Frenet coordinates
Target
8 - 36
Additional Slides
Coordinate choice for the vehicle control problem
• Many simulations models, also the ones we have presented earlier, are naturally formulated in
cartesian coordinates. This space is the most natural to formulate a mechanical problem and
therefore widely used by practioniers and researchers alike.
• One of the key issues here is the strong nonlinearity in with respect to the trajectory. A stabilizing
mechanism has to be adapted to the heading all the time, this renders the problem nonlinear and
difficult to analyze in the theoretical settings we have learned about earlier.
• This issue might be overcome by the choice of a so-called curvilinear coordinate system. It transforms
the eucledian coordinates into a coordinate system which is located along the path. The first
component is the progress along the path, the second component is the lateral deviation with respect
to that path. The coordinate system itself can be defined via a series of points on the path or via a
curvature profile for each point on the path. The actual path can then be obtained from double
integration.
• This curvilinear system will be the base for many of the control designs we see later on. It is a natural
formulation of the split between lateral and longitudinal tracking and therefore well suited for the
analysis of the respective control systems.
• There is a key difficulty: the mapping between cartesian coordinates and curvilinear coordinates might
not be well-defined in very sharp turns or if the path crosses itself. However, it should be kept in mind
that these cases also pose significant difficulty to deal with in control design of cartesian coordinates
and should therefore be considered challenging in general.
1- 37
Improved lateral control with model knowledge
▪ Reformulation of the lateral path tracking
problem in Frenet-coordinates
▪ Differentiating twice leads to double-integrator
dynamics in the curvilinear coordinate frame
corrective acceleration
in path coordinates
▪ For small errors this is linear w.r.t. the
acceleration in vehicle coordinates
Lateral acceleration required Corrective lateral
to stay on the path acceleration (feedback)
(feedforward)
8 - 38
Improved lateral control with model knowledge
▪ Control law has the structure of a 2-DOF
gain-scheduling PD controller
▪ Calculate steering angle from kinematic
bicycle model
▪ Calculate lateral error derivative from
Velocity heading error
8 - 39
Advanced motion control – I
Tire side slipe angle Linear tire model Nonlinear tire model
Tire lateral force
Cornering
stiffness
▪ Advanced vehicle models require detailed modelling of the tire
▪ Kinematic rolling condition is replaced with tire slip and tire models
8 - 40
Advanced motion control – II
Yaw rate
Neutral steering /
Over- Kinematic bicycle model
steering
Understeering
Steering angle
▪ Self-steering characteristic is most important factor
▪ Depends heavily on tires, suspension setup and vehicle speed
▪ Different reasons for understeer (e.g. front-rear tire balance,
approaching the limits of the tire, …)
8 - 41
Advanced motion control – III
Steering angle
Nonlinear Curvature
Vehicle velocity vehicle
dynamics
Steering angle
Steady-state Curvature
Vehicle velocity self-steering
characteristic
While operating in the stable driving region it is sufficient to know
about the steady-state self-steering characteristics to obtain a
reasonable good feedforward control law!
8 - 42
Advanced motion control – IV
Inverse self-
steering
characteristic
Target Steering
Lateral path curvature Curvature angle
tracking controller + Vehicle
controller P-controller
▪ Nonlinear feedforward controller based on self-steering
characteristic
▪ Curvature feedback can be done using a simple P-controller
8 - 43
Application to autonomous racing @ Roborace 2019
✓ Successful application at
Roborace competition 2019
✓ Close to human driver
performance on qualifying lap
✓ Moderate implementation
efforts required Source: https://www.youtube.com/watch?v=-vqQBuTQhQw
More details in:
✗
Minimum curvature trajectory planning and control for
Depends on curvature and an autonomous race car
acceleration signal quality (https://doi.org/10.1080/00423114.2019.1631455)
✗ A software architecture for the dynamic path planning of
an autonomous racecar at the limits of handling
Difficulties with dynamic (https://doi.org/10.1109/ICCVE45908.2019.8965238)
trajectory planning 8 - 44
Autonomous Driving Software Engineering
Prof. Dr. Markus Lienkamp
Alexander Wischnewski, M. Sc.
Agenda
1. Introduction
2. Basic control (Geometric & PID)
3. Model-based control
4. Model predictive control
5. Summary
Basics of model predictive control – I
Source: https://de.wikipedia.org/wiki/Model_Predictive_Control
8 - 46
Basics of model predictive control – II
Steering angle
Vehicle motion
prediction
Time Vehicle
dynamics
model
Longitudinal force
Time
Target
Optimize based on comparison of trajectory
prediction and target trajectory
8 - 47
Basics of model predictive control – III
▪ Solution to this optimization problem is computational expensive
▪ Typical problem sizes are around 20-50 discretization steps
Quadratic Nonlinear
Programming Programming
▪ Quadratic cost function, ▪ Arbitrary cost functions and
linear constraints constraints possible
▪ Works well for linear ▪ Nonlinear dynamics
dynamics ▪ Solvers are based on
▪ Efficient and reliable solvers linearization schemes
available with <10ms ▪ Solution times range from a
computation time few milliseconds to multiple
seconds 8 - 48
Basics of model predictive control – IV
Considers constraints and is suited for multi-input and multi-
✓ output systems
✓ Nonlinear dynamics and complex cost functions
✗ Implementation and solution of numerical problem is a
complicated challenge
✗ Computation times get large for many active constraints or
nonlinear dynamics
✗ Solver might not converge → backup strategy needed
8 - 49
Additional Slides
• Model predictive control is a promising concept – it allows to handle general nonlinear systems, multi-
input/multi-output systems as well as constrained control problems. These advantages make the
concept attractive to researchers and practitioniers within the community. However, this comes at a
price which is related to the requirement to implement the optimization as a tractable (finite
dimension) and finite-time horizon optimization task.
• One challenge within constrained problems is that a controller might tend to be overly optimistic
towards the end of the optimization horizon. This can render the solution of the next timestep
optimization infeasible. There is a great amount of literature on this problem called „recursive
feasibility“. It aims at deriving certain modifications to the optimization problem which guarantee that a
solution to the next optimization problem will be available under certain assumptions.
• The same issue might arise due to inaccuracies within the model or external disturbances, both
effects which are likely to occur. This might be tackled by „Robust MPC“ or Soft-Constraints.
• Despite these control engineering challenges, it remains to find a computational efficient version of
the desired optimization problem. Especially nonlinear problems are time consuming to solve and
usually need a lot of hyperparameter tuning for the optimization algorithms which makes them difficult
to use for online operation in safety critical systems. Many practitioniers therefore add a backup
controller which can take over in case the MPC fails.
• While this control concept seems to be promising, it is still an active research topic with many
contributions to be made until it can be considered a „standard-tool“ which can be deployed easily. In
the case of autonomous vehicle control there are many easier control concepts which allow to reach
reasonable performance levels of about 50-80% of the vehicle capabilities.
1- 50
Autonomous Driving Software Engineering
Prof. Dr. Markus Lienkamp
Alexander Wischnewski, M. Sc.
Agenda
1. Introduction
2. Basic control (Geometric & PID)
3. Model-based control
4. Model predictive control
5. Summary
Summary – What did we learn today
• Responsibilities of the controller within the autonomous driving stack
• Recap of control theory and vehicle dynamics modeling
Model predictive control
Model-based feedforward
and feedback control
Performance
Stanley &
Velocity PI
Complexity
8 - 52
Helpful literature for this lecture
Required control engineering basics:
▪ Lecture „Regelungstechnik“ – Prof. Dr.-Ing. Boris Lohmann – Chapter 6 & 7
(Reglerentwurf & Erweiterte Regelungsstrukturen und Zustandsregelung)
Further control engineering materials:
▪ Otto Föllinger, Regelungstechnik, VDE Verlag GmbH
▪ Heinz Unbehauen, Regelungstechnik I, Springer Vieweg
▪ John Doyle, Bruce Francis, Allen Tannenbaum, Feedback Control Theory,
Macmillan Publishing Co.
Further vehicle dynamics materials:
▪ Lecture „Dynamik der Straßenfahrzeuge“ – Prof. Dr.-Ing. Markus Lienkamp
▪ William F. Milliken and Douglas L. Milliken – Race Car Vehicle Dynamics
8 - 53
Related literature
A list of papers for the controllers presented within the literature and more
interesting concepts:
Brian Paden, Michal Cap, Sze Zheng Yong, Dmitry Yershov, and Emilio Frazzoli, A Survey of Motion
Planning and Control Techniques for Self-Driving Urban Vehicles, 2016
Gabriel M. Hoffmann, Claire J. Tomlin, Michael Montemerlo, and Sebastian Thrun, Autonomous
Automobile Trajectory Tracking for Off-Road Driving: Controller Design, Experimental Validation and
Racing, 2009
Moritz Werling, Ein neues Konzept für die Trajektoriengenerierung und -stabilisierung in zeitkritischen
Verkehrsszenarien, 2011
Nitin R. Kapania and J. Christian Gerdes, Design of a feedback-feedforward steering controller for
accurate path tracking and stability at the limits of handling, 2015
Alexander Heilmeier, Alexander Wischnewski, Leonhard Hermansdorfer, Johannes Betz, Markus
Lienkamp, and Boris Lohmann, Minimum curvature trajectory planning and control for an autonomous
race car, 2019
Alexander Liniger, Alexander Domahidi, and Manfred Morari, Optimization‐based autonomous racing of
1: 43 scale RC cars, 2015
8 - 54
