Automated Docking of an Off-

shore Gangway
A Predictive Control Approach

M. A. Ranneh
Master of Science Thesis

Delft Center for Systems and Control

 Automated Docking of an Offshore
Gangway
A Predictive Control Approach

Master of Science Thesis

For the degree of Master of Science in Systems and Control at Delft

University of Technology

M. A. Ranneh

June 29, 2019

Faculty of Mechanical, Maritime and Materials Engineering (3mE) · Delft University of

Technology
The work in this thesis was supported by Bosch Rexroth. Their cooperation is hereby grate-
fully acknowledged.

Copyright © Delft Center for Systems and Control (DCSC)

All rights reserved.
 Delft University of Technology
Department of
Delft Center for Systems and Control (DCSC)

The undersigned hereby certify that they have read and recommend to the Faculty of
Mechanical, Maritime and Materials Engineering (3mE) for acceptance a thesis
entitled
Automated Docking of an Offshore Gangway
by
M. A. Ranneh
in partial fulfillment of the requirements for the degree of
Master of Science Systems and Control

Dated: June 29, 2019

Supervisor(s):
Dr.ir. E. Steur

Ir. M.J. van de Molengraft

Reader(s):
Prof.dr.ir. B. De Schutter

Dr.ir. K. Batselier
 Abstract

Objects floating on or near the water surface (e.g. vessels, and floating wind-turbines) suffer
from motions induced by waves of varying height, direction, and frequency. This not only
causes unpleasantness for passengers and crew of ships but also it limits the accessibility to
the offshore platforms. Bosch Rexroth with their partner Barge Master have developed the
so-called Motion Compensated Gangway that provides a safe passage for cargo and personnel
to offshore structures.
In order to maintain a motionless connection with the offshore structure, once the tip of the
gangway is pushed against the offshore structure. The gangway system actively compensates
for the sea-induced motion that acts on the vessel.
However, the docking procedure is still manually attained, where accidents may occur due to
human error (i.e. insufficient training, loss of concentration). One way to improve the current
control scheme is to enable an automated docking scheme.
Accordingly, the main of this project focuses on eliminating the human factor from the con-
trol loop, so the overall process is accomplished automatically and more efficiently in terms
of safety and performance.
Inspired by how the operator estimates the relative motion between the Gangway and the tar-
get (i.e. the offshore platform). In this thesis, a measurement system is proposed to measure
this relative motion. This measurement system comprises a vision sensor, force tip measure-
ments, and Motion Reference Unit (MRU). In this thesis, the proposed automated docking
scheme is developed around a nonlinear MPC scheme. For the simulation environment and for
the MPC scheme employs, a nonlinear model of the gangway system is derived. This model
embeds an approximation of the joint-level control loop of the Gangway system. Also, this
model comprises the open-chain kinematic model of the Gangway system and the proposed
measurement system including a perspective projection model of the vision sensor. Due to
modelling the vision sensor as such and the MPC’s capability in handling various constraints,
the proposed control scheme enjoys a singularity-free solution.
The proposed control scheme detects and tracks the target in the 2D image plane. To safe-
guard against visual measurements discontinuity (i.e. cluttering, target outside the field of
view), a linear Kalman filter is designed to predict the target position in the image plane.
To gain higher performance, the disturbance anticipatory property in MPC is enabled by

Master of Science Thesis M. A. Ranneh

ii

forecasting the sea-induced motion. Where a neural network with the NARX topology was
designed and trained to acquire a multi-step-ahead prediction model of the induced motion.
Several numerical experiments were carried to evaluate the performance of the proposed
control scheme for automated docking. Where for the nominal case scenarios all the control
requirements are fulfilled. Also, more extreme scenarios are performed to evaluates the overall
performance under plant model mismatch and against various sea-induced motion conditions.
Evidently, the proposed control scheme is prone to camera calibrations error.
In terms of the efficiency, the proposed automated docking scheme performs the docking in 4
to 10 seconds (based on the initial conditions and sea state). Whereas the time it takes the
operator to perform the docking is up to 3 minutes which depends on his/her experience.

M. A. Ranneh Master of Science Thesis

 Table of Contents

Acknowledgements xi

1 Introduction 1
1-1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1-2 Problem statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1-3 Research Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1-4 Preliminary assumptions and considerations . . . . . . . . . . . . . . . . . . . . 5
1-5 Thesis contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1-6 Thesis outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2 System Architecture 9
2-1 Reference frames . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2-2 Mechanical structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2-3 Actuators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2-4 Disturbances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2-5 Measurement system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2-5-1 Joint-level sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2-5-2 Motion Reference Unit . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2-5-3 Tip force Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2-6 Control scheme and operator input . . . . . . . . . . . . . . . . . . . . . . . . . 15
2-7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3 Relative Motion Measurements 17

3-1 The measurement system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3-2 Sensor architecture and sensor placement . . . . . . . . . . . . . . . . . . . . . 18
3-3 Camera mathematical model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3-4 Camera calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3-5 Representing the offshore platform . . . . . . . . . . . . . . . . . . . . . . . . . 22
3-6 Synthetic data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3-7 The motion model of the target . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3-8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

Master of Science Thesis M. A. Ranneh

iv Table of Contents

4 Modelling & Control Design 29

4-1 The concept of vision-based control . . . . . . . . . . . . . . . . . . . . . . . . 29
4-1-1 Pose-Based Visual Servoing (PBVS) . . . . . . . . . . . . . . . . . . . . 32
4-1-2 Image-Based Visual Servoing (IBVS) . . . . . . . . . . . . . . . . . . . . 32
4-1-3 Visual Predictive Control (VPC) . . . . . . . . . . . . . . . . . . . . . . 32
4-2 The proposed control scheme . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4-3 Modelling considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4-3-1 Kinematics modelling . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
4-3-2 Incorporating the vision sensor within the model . . . . . . . . . . . . . . 36
4-3-3 Force modelling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4-3-4 The nonlinear state space model . . . . . . . . . . . . . . . . . . . . . . 38
4-4 Nonlinear Model Predictive Control (MPC) . . . . . . . . . . . . . . . . . . . . 39
4-5 MPC parameters tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
4-5-1 Reducing the output variables . . . . . . . . . . . . . . . . . . . . . . . 41
4-5-2 Sampling time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
4-5-3 Prediction and control horizons . . . . . . . . . . . . . . . . . . . . . . . 43
4-5-4 Input and output weights . . . . . . . . . . . . . . . . . . . . . . . . . . 43
4-6 Perturbations anticipation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4-7 Closed-loop stability analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
4-8 Summary and discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

5 Results 49
5-1 Nominal Case Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
5-1-1 No disturbances case . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
5-1-2 Adding disturbances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
5-1-3 Disturbance anticipation . . . . . . . . . . . . . . . . . . . . . . . . . . 52
5-2 Selecting prediction & control horizons values . . . . . . . . . . . . . . . . . . . 55
5-3 Model mismatch tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
5-3-1 Output parameters mismatch . . . . . . . . . . . . . . . . . . . . . . . . 57
5-3-2 Plant dynamics mismatch . . . . . . . . . . . . . . . . . . . . . . . . . . 58
5-4 Extreme disturbances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
5-5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

6 Conclusions and Recommendations 63

6-1 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

Bibliography 65

Glossary 71
List of Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

M. A. Ranneh Master of Science Thesis

 List of Figures

1-1 Global annual installed capacity and operating capacity of offshore wind between
2001-2015. From [1]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1-2 (a) A crane with a suspended basket. (b) A boat bushing onto a monopile. (C) A
helicopter lowering personnel on a wind turbine. From [2–4]. . . . . . . . . . . . 2
1-3 The vessel’s axes signs and naming conventions. From [5]. . . . . . . . . . . . . 3
1-4 The working principle of Bosch’s gangway system. From [6]. . . . . . . . . . . . 3
1-5 A high-level scheme of the control system of Bosch gangway. . . . . . . . . . . . 4

2-1 Definition of the relevant reference frames gangway model. . . . . . . . . . . . . 10

2-2 The workspace and Degrees of freedom in which the gangway operates. From [7]. 11
2-3 Two hydraulic rotational motors with gearbox and pinion acting on a slewing ring.
From [7]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2-4 Luffing cylinder. From [7]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2-5 The telescope winch and the telescope extension using a cable system. From [5]. 12
2-6 Ship movements in six degrees of freedom in a time window of three minutes. . . 14
2-7 To the left, the definition of forces in the tip. To the right, the docking interface
at an offshore wind turbine. From [5]. . . . . . . . . . . . . . . . . . . . . . . . 15

3-1 Classification of range sensing technologies . . . . . . . . . . . . . . . . . . . . . 18

3-2 (a) eye-in-hand. (b) eye-to-hand. From [8]. . . . . . . . . . . . . . . . . . . . 19
3-3 Perspective projection in the pin-hole camera model. From [8] . . . . . . . . . . 20
3-4 The transformation between target and camera reference frames. (Modified) From [8] 21
3-5 An image of a checkerboard captured with an ideal lens, a lens with positive
distortion, and a lens with negative distortion, respectively. From [9]. . . . . . . 21
3-6 Captured images for camera calibration . . . . . . . . . . . . . . . . . . . . . . 22
3-7 Aruco marker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3-8 The process of generating the synthesised images . . . . . . . . . . . . . . . . . 24

Master of Science Thesis M. A. Ranneh

vi List of Figures

3-9 The target (the marker) movement is 3D observed from the camera reference frame
and the corresponding 2D mapping in the image plane. . . . . . . . . . . . . . 25
3-10 3D representation of the a Gangway-like structure with the camera attached on
the tip. The offshore platform is represented by the square grey marker. . . . . . 26
3-11 The process of generating the synthetic data . . . . . . . . . . . . . . . . . . . . 27

4-1 The general control scheme of visual servoing. (Modified) From [10] . . . . . . . 30
4-2 The different visual servoing schemes. (a) PBVS general control scheme the set-
point signal is the desired relative pose between the tip and the target. (b) IBVS
general control scheme where the setpoint is the desired field of view of the target
in the image plane, this solve the pose estimation problem implicitly when the
tracking error converges to zero. From [8]. . . . . . . . . . . . . . . . . . . . . . 31
4-3 The general control scheme of VPC. This scheme, similarly to IBVS, takes the
desired field of view as a setpoint s∗ (k). The Model block is the nominal model
that is utilised in the internal model control structure. From [11] . . . . . . . . . 33
4-4 High-level control scheme of the proposed method. . . . . . . . . . . . . . . . . 35
4-5 The 3D visualisation of the Gangway kinematic model. (Modified) From [12].
Where θ1 , θ2 , and θ3 are the Gangway system actuated joints represented by slew-
ing, luffing, and telescoping actuators, respectively. ψb and ψtip are the reference
frame of the Gangway base and the tip, respectively. L1 , L2 , and L3 are the
constant displacements of the corresponding joints. . . . . . . . . . . . . . . . 36
4-6 The mass-spring analogy of the contact force model. . . . . . . . . . . . . . . . 37
4-7 A zoomed-in view of the corresponding block diagram of the open-loop Gangway
system as described in (4-13) to (4-14). . . . . . . . . . . . . . . . . . . . . . . 39
4-8 To the right, the predicted marker images over the prediction horizon (e.g. Np = 3
for visualisation purposes). To the left, the plant actual response over multiple time
steps {t, t + 1, t + 2} to the left. (Reproduced) From [11]. . . . . . . . . . . . . 41
4-9 The nonlinear MPC scheme of for the task of automated docking. . . . . . . . . 42
4-10 The designed architecture of the employed NARX neural network. . . . . . . . . 46
4-11 The neural network forecasting error of the sea-induced motion (velocity). . . . . 46
4-12 The cost function value. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

5-1 Here, the target is stationary with a relative position, to the Gangway base, of
Ptarget = [0.5, 23, 1]T in case of (a) and (b), and Ptarget = [4, 25, 2]T in case
of (c) and (d). (a) and (c) illustrate time-lapse footage of the target (marker)
detected features in the field of view over 20 seconds. (b) and (d) show the
corresponding tracking error of the target features in the 2D image plane. . . . . 50
5-2 The corresponding contact force response of the first and the second initial condi-
tions set denoted by x00 and x01 , respectively. . . . . . . . . . . . . . . . . . . . 51
5-3 This set of figures corresponds to the case where the induced motion is present on
the system, however, MPC has no access to these data. Correspondingly, MPC
shows poor performance in tracking the setpoints. Therefore, the target visual
information coming for the camera is not enough and MRU readings are needed.
(a) and (b) illustrate time-lapse footage of the target (marker) detected features in
the field of view over 25 seconds and the corresponding tracking error, respectively.
The corresponding contact force response and the target pose is depicted in (c)
and (d), respectively. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

M. A. Ranneh Master of Science Thesis

List of Figures vii

5-4 This figures corresponds to the case where the induced motion is present on the
system and provided accordingly to MPC at each time step (instant feedback). The
contact force response for this case is shown in (a). The corresponding target’s
features tracking error is shown in (b). . . . . . . . . . . . . . . . . . . . . . . . 54
5-5 The contact force response in case of; no disturbance anticipation (nominal case),
disturbance anticipation with the induced motion data set, and disturbance antic-
ipation with the trained neural network. . . . . . . . . . . . . . . . . . . . . . . 54
5-6 The contact force response for different Np values (a), and different Nc values (b).
Note, the scale of the vertical axis is enlarged in (a) compared to the one depicted
in (b). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
5-7 The corresponding target pose. . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
5-8 The corresponding contact force response for different Nc values in case of stiffness
coefficient Kb mismatch. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
5-9 (a) and (b) illustrate time-lapse footage of the target (marker) detected features in
the field of view over 50 seconds and the corresponding tracking error, respectively.
The corresponding contact force response for different Nc values and the error
between the tip and the target is depicted in (c) and (d), respectively. . . . . . . 59
5-10 The contact force response in case of plant dynamics mismatch. . . . . . . . . . 60
5-11 The target pose for different induced motion cases. . . . . . . . . . . . . . . . . 61
5-12 The contact force response for different induced motion cases. . . . . . . . . . . 61

Master of Science Thesis M. A. Ranneh

viii List of Figures

M. A. Ranneh Master of Science Thesis

 List of Tables

2-1 The operational limits of the hydraulic actuators . . . . . . . . . . . . . . . . . . 12

2-2 Desired tip forces and acceptable forces along the tip principal axes [6], as depicted
in Figure 2-7. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

4-1 Selecting a proper time step . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

Master of Science Thesis M. A. Ranneh

x List of Tables

M. A. Ranneh Master of Science Thesis

 Acknowledgements

First and foremost, I would like to praise God for his blessings and for giving me this won-
derful opportunity - to be part of this master program.

I would like to thank my supervisor at TU Delft, Dr. Erik Steur, for his adequate supervision
throughout my master’s thesis project, and for teaching me to be critical and organised in
my work.
I would also like to thank my supervisor at Bosch, Ir. Maik van de Molengraft, for his excel-
lent guidance and support on both technical and personal levels.

Last but not least, I want to express my gratitude for my parents who dedicated their lives
to bring the best out of me, while never stopped offering their unwavering support.

Delft, University of Technology M. A. Ranneh

June 29, 2019

Master of Science Thesis M. A. Ranneh

xii Acknowledgements

M. A. Ranneh Master of Science Thesis

“A year spent in artificial intelligence is enough to make one believe in God.”
— Alan Perlis
 Chapter 1

Introduction

1-1 Background

According to the World Wind Energy Association, the total capacity of wind energy has
been significantly increased over the last decade, especially, the offshore sector [1], see Figure
1-1. Offshore wind farms are becoming more favourable since it has many advantages over
the onshore wind energy (e.g. less turbulent, more space for larger rotors diameter, higher
capacity due to more sustained wind) [1].
However, the enormous costs of construction and maintenance of offshore wind farms intro-
duce a striking increase to the overall cost. Consequently, implementing innovative solutions
and mechanisms in the affiliated industrial fields will result in reducing the technical risk and
finance costs which will attract more investments.

Figure 1-1: Global annual installed capacity and operating capacity of offshore wind between
2001-2015. From [1].

Master of Science Thesis M. A. Ranneh

2 Introduction

Former methods of personnel and cargo transfer to offshore structures are generally limited
to; a helicopter lowering personnel to the structure, cranes with a suspended basket, and
a crew transfer boats that push against a landing zone, See Figure 1-2. these methods are
rather inefficient in terms of cost, safety, and the operating window [13].
In 2008, the first motion compensated gangway was introduced to the market by Ampel-
mann [14], which is a gangway attached to the top of a hydraulically actuated Stewart. This
was a game changer in the offshore logistics sector. Accordingly, many companies started de-
veloping their own motion compensated gangway becoming a highly competitive field which
resulted in the lack of related published work in the literature.

(a) (b) (c)

Figure 1-2: (a) A crane with a suspended basket. (b) A boat bushing onto a monopile. (C) A
helicopter lowering personnel on a wind turbine. From [2–4].

Waves of varying height, direction and frequency are constantly present, objects floating on
or near the water surface (e.g. ships, vessels, and floating wind-turbines) suffer from motions
induced by these waves, this can cause unpleasantness for passengers and crew of ships but
can also carry serious risks for both people and equipment in any type of offshore operation.
Offshore gangway systems are usually installed on a vessel. The vessel responds to the induced
motions via sea state in six Degrees of Freedom Degree of Freedom (DoF); three translations
(surge, sway, heave) and three rotations (roll, pitch, yaw). Figure 1-3 illustrates the axes
conventions of a vessel.
This type of vessels is supplied with a Dynamic Positioning (DP) system. With the DP sys-
tem, the vessel can stay in place at a short distance from the offshore structure with relatively
small deviations by actively controlling the propellers, rudders and, thrusters. This can only
compensate for the occurred movements in the horizontal plane. The gangway system is
activated only when the vessel’s DP system is enabled.
Consequently, a gangway system has to account mainly for heave motions. However, due to
a possible drift error in the DP system, the gangway system also needs to compensate for
yaw and sway. It is worth mentioning that roll and pitch movements can be neglected due
to the length of and the huge mass of the vessel which can damp out the perturbations on
the aforementioned axes. Therefore, with a predefined operating window, along with the DP
system, it is safe to argue that a three-DoF structure for an offshore gangway is sufficient.
Based on this concept, Bosch Rexroth with their partner Barge Master have developed a
Motion Compensated Gangway Motion Compensated Gangway (MCG) that provides a safe
passage for cargo and personnel transfer to offshore structures, see Figure 1-4. In this thesis,
the configuration and design of Bosch’s gangway system will be used for modelling and control
design.

M. A. Ranneh Master of Science Thesis

1-2 Problem statement 3

Figure 1-3: The vessel’s axes signs and naming conventions. From [5].

Figure 1-4: The working principle of Bosch’s gangway system. From [6].

The word "Gangway" in sentence case will be used throughout this report to refer to Bosch’s
gangway (Gangway) system.

1-2 Problem statement

At the present time, all existing gangway systems in the market are obliged to employ an
operator in their control loop for legal and safety-related issues [15].
Figure 1-5 depicts the high-level scheme of the control system of Bosch’s gangway system.
The system, through an inertial measurement unit, can automatically counteract the sea-
induced motion in a feedforward fashion. The operator function is limited to steer the motion
compensated tip of the gangway. The control interface in the gangway system is merely based
on three joysticks where each joystick generates the position setpoint for each actuator.
Moreover, the operator’s cabin is attached on top of the Gangway base. This means that

Master of Science Thesis M. A. Ranneh

4 Introduction

he/she perceives the vessel surrounding environment in a non-inertial frame. Because of this,
the operator’s task in docking the Gangway to the offshore platform becomes challenging
and nontrivial. Consequently, the operator must follow excessive training (similar to a crane
operating training) in order to be qualified as an offshore gangway operator [15].
Since the docking procedure is manually attained, accidents may occur due to human error
(i.e. insufficient training, loss of concentration). One way to improve the current control
scheme is to enable an automated docking scheme, which employs range sensing technologies
within its measurement system (e.g. radar, lidar, camera, etc.).
Accordingly, the outcome of this project focuses on eliminating the human factor from the
control loop, so the overall process is accomplished automatically and more efficiently in terms
of safety and performance.

Figure 1-5: A high-level scheme of the control system of Bosch gangway.

1-3 Research Objectives

Enabling the automated docking scheme for the Gangway system corresponds to eliminating
the human presence (i.e. the operator) in the control loop.
Thus, the main research objective is to propose a control scheme for docking the Gangway
tip onto the offshore platform autonomously while actively compensating for the sea-induced
motion.
At the beginning of this project, a literature study was conducted, [16], to create a base of
knowledge to assist in the completion of the thesis-project:

"Automated Docking of an Offshore Gangway"

Correspondingly, the research questions were formulated as:

Q1. What is the architecture and characteristics of the current system? What are the control
requirement of the Gangway system?

M. A. Ranneh Master of Science Thesis

1-4 Preliminary assumptions and considerations 5

Q2. How to measure the relative motion between the Gangway and the docking area?

Q3. What kind of sensing technologies are to be included in the measurement system for the
automated docking?

Q4. How can a vision sensor be incorporated within the control loop for automated docking?

Q5. Is there a control scheme that combines vision and force control while handling con-
straints?

Q6. Is it possible to forecast the sea-induced motion? Would the proposed control method
benefit from anticipating the external perturbation ahead?

1-4 Preliminary assumptions and considerations

In order to answer the research questions, a set of assumptions and operating constraints are
set for the system modelling and control design. The following assumptions are made:

• Actuators dynamics are not considered explicitly but rather implicitly. An approxima-
tion is provided by Bosch to describe the low-level (joint-level) control loop as a second
order system.

• For the sake of simplicity; no deformations occur in both the vertical and torsional axes
of the Gangway.

• Relative motion hypothesis: The residual motion of the vessel, which is caused by the
sea-induced motion, is slow sinusoidal-based motion. From the operator’s point of view
(with ignoring his vestibular senses) the sea-induced motion that acts on the vessel
causes the target (i.e. offshore platform) to move instead. This assumption is made to
describe the Gangway kinematics and dynamics in an inertial frame. This should result
in a simpler model which is required by the proposed control strategy.

• Bosch has defined an operation window for which the Gangway system function within:

– Maximum wind speed: 20 m/sec. The wind speed below this limit does not affect
the overall performance of the Gangway system [6].
– Operating in both daylight and nighttime with sufficient illumination on the gang-
way tip that is defined by DNV-GL standards [15].
– The wave period is between 4 and 12 seconds with Hsig (the significant wave height)
is no more than 3.0 m [6].

1-5 Thesis contributions

In Chapter 3, a measurement system is proposed to measure the relative motion between the
Gangway and the offshore platform. This measurement system comprises a vision sensor,
force tip measurements, and Motion Reference Unit (MRU). In Chapter 4, a nonlinear Model
Predictive Control (MPC) is designed to solve the automated docking problem by utilising

Master of Science Thesis M. A. Ranneh

6 Introduction

the aforementioned measurement system. Hence, the main contributions of this thesis-project
lie in the outcome of Chapter 4. The proposed control scheme enjoys the following:

• Fast docking time (up to 10 seconds) compared to the time it takes the operator to
perform the docking (up to 3 minutes) which is dependent on his/her experience [7].

• This control scheme utilises the vision, the induced motion, and tip force measurements
in one control scheme to steer and dock the Gangway tip onto the offshore platform.
Hence, no need for a switching control scheme.

• This control scheme detects and tracks the target (i.e. the offshore platform) in the
2D image plane. Therefore, the control scheme is prone to camera calibrations error.
Additionally, in Chapter 3, a linear Kalman filter is designed to predict the target
position in the image plane when if the target leaves the field of view. This can safeguard
against visual measurements discontinuity, which might be problematic from a control
perspective.

• The proposed scheme is based on nonlinear MPC. Hence, system constraints and its
physical limitations can naturally be satisfied.

• To gain higher performance, the disturbance anticipatory property in MPC is enabled

by forecasting the sea-induced motion. A neural network with the NARX topology
was designed and trained to acquire a multi-step-ahead prediction model of the induced
motion.

1-6 Thesis outline

In this chapter, the problem statement and the research objectives of this project are presented
along with preliminary research assumptions and considerations. The remainder of this thesis
structured as follows:

• Chapter 2: System Architecture. This chapter address the research questions Q1 by

introducing the Gangway system architecture as designed by Bosch. In terms of its
mechanical structure, actuator, and measurement system. Furthermore, the physical
limitation and control task requirements of the system are presented.

• Chapter 3: Relative Motion Measurements. This chapter is concerned with answering

the research questions Q2 and Q3. The chapter presents the proposed measurement
system with a detailed model of the employed vision sensor. Additionally, the processes
of creating synthetic data are outlined.

• Chapter 4: Modelling & Control Design. This chapter is concerned with answering the
research questions Q4, Q5, and Q6. The chapter presents a background over the common
vision-based control approaches, then the proposed control scheme is formulated. with
the mathematical model of the Gangway system. Additionally, the different parameters
of the MPC are discussed. and the design of the employed neural network for disturbance
anticipation is outlined.

M. A. Ranneh Master of Science Thesis

1-6 Thesis outline 7

• Chapter 5: Results. This chapter evaluates the performance of the proposed solution
for automating the docking procedure. The controller is first tested for the nominal
case with multiple scenarios (i.e. with and without disturbance anticipation, different
MPC settings). Then, tests are preformed against the plant model mismatch. Lastly,
the performance is evaluated for a more extreme case in terms of the amplitude of the
sea-induced motion.

• Chapter 6: Conclusions and Recommendations. This chapter discusses the project

outcome and the potential space for enhancement and future work.

Master of Science Thesis M. A. Ranneh

8 Introduction

M. A. Ranneh Master of Science Thesis

 Chapter 2

System Architecture

This chapter introduces the Gangway system architecture as designed by Bosch. In terms
of its mechanical structure, actuator, and measurement system. Furthermore, the physical
limitation and control task requirements of the system are presented. Then, the current
control loop is outlined with the operator function description.

2-1 Reference frames

The vessel, which the Gangway system is attached to, is subject to different disturbances
(e.g., sea state, wind gust). The induced motion via sea state causes undesired motion at the
Gangway tip.
As stated in Chapter 1, the problem of motion compensation can be recast as a relative mo-
tion tracking problem. Hence, instead of describing the Gangway system within a non-inertial
frame, we can assume that the fixed-world frame of reference is attached to the Gangway base.
This means that the offshore platform is moving and the Gangway base is fixed in the 3D
space. Certainly, this hypothesis will simplify the system mathematical model and the control
design.
Accordingly, we need to define a set of reference frames for the various rigid bodies in the
system (i.e. the base, the offshore platform).
Figure 2-1, depicts the different reference frames for the Gangway system. Where Ψt repre-
sents the reference frame of the docking area on the offshore platform and Ψb is the reference
frame of the Gangway base which is attached to the fixed-world coordinates as the aforemen-
tioned assumptions states. The reference frame Ψc represent the vision sensor coordinates.
Note that the homogeneous transformation is fixed between the vision sensor frame Ψc and
the Gangway tip reference frame Ψtip assuming the Gangway is a rigid body and no defor-
mations occur.
Furthermore, three hydraulic actuators facilitate movement between the base reference frame
Ψb and the tip reference frame Ψtip . The actuators form an open-chain kinematic with a
three-DoF. The slewing and luffing actuators are rotational joints and lead to rotations in the

Master of Science Thesis M. A. Ranneh

10 System Architecture

Figure 2-1: Definition of the relevant reference frames gangway model.

horizontal and vertical planes around yaw and pitch axes, respectively. The telescoping is a
prismatic joint along the Sway axis, see Figure 1-3 for axes conventions.
This kinematics chain will be utilised within the Gangway mathematical model in the control
design in Chapter 4. Where the joints displacement is represented by the state variable qn
with qn = [qslew , qluf f , qtele ]T .
Figure 2-2 depicts the Gangway system workspace. This workspace represents the set of the
reachable points by the Gangway tip in 3D space.

2-2 Mechanical structure

Besides some safety and redundant components, the system structure mainly consists of a
rigid mechanical structure, actuators, and sensors.
For the structural design, the following was taken into account by Bosch during the design
phase of the Gangway system:

• The lowest vertical eigenfrequency of the Gangway is designed to be at least 3Hz to

avoid excitation by walking personnel (human walking excitation frequency is around
2Hz) [17].

• The Gangway is designed to be stiff to contribute to the safe feeling of personnel in

transfer including load (e.g. equipment, cargo, etc). Both bending and torsional stiffness
are considered.

• Hydraulic actuators were considered because of the high rigidity and high torques/forces
these actuators can provide.

M. A. Ranneh Master of Science Thesis

2-3 Actuators 11

Figure 2-2: The workspace and Degrees of freedom in which the gangway operates. From [7].

2-3 Actuators

Each actuator is controlled by a specific Programmable Logic Controller (PLC) referred to

as a Hydraulics-capable Numerical Control (HNC) computer. HNCs accept a position and a
velocity setpoint. The HNC controls the voltage signals sent to the actuator valves.
Accordingly, the control on the joint-level control is already designed by Bosch and the High-
level control is the main contribution in this project. In addition, Gangway’s actuators are
configured as follows.

• The slewing drive, to rotate the Gangway.

The slewing actuator consists of two rotational drives working on a large slewing ring,
as shown in Figure 2-3.

Figure 2-3: Two hydraulic rotational motors with gearbox and pinion acting on a slewing ring.
From [7].

• The luffing cylinder, to raise and lower the gangway.

For the luffing cylinder, a double driven cylinder is used. Which means that both

Master of Science Thesis M. A. Ranneh

12 System Architecture

chambers in the cylinder are pressurised and connected to the valves. With the oil flow
controlled to both chambers, double driven actuators can exert pushing and pulling
forces on the environment. See Figure 2-4.

• The telescoping winch, to extend and retract the moving part of the gangway.
The moving part is driven by a direct drive hydraulic motor. The motor drives a winch
drum which connects to the moving part with a cable. See Figure 2-5 to the left: The
telescope winch. The winch drum translates the rotational motion of the motor to a
linear motion of the cable and the telescope. As shown to the right in Figure 2-5.

Relevant operational constraints are shown in Table 2-1 [5, 6].

Figure 2-4: Luffing cylinder. From [7].

Figure 2-5: The telescope winch and the telescope extension using a cable system. From [5].

Table 2-1: The operational limits of the hydraulic actuators

Slewing drive Luffing cylinder Telescoping winch

Work range 0 to 360 deg -23 to 23 deg 17 to 27 m
Maximum operational speed 8 deg/s 0.13 m/s 1.6 m/s
Maximum operational force/torque 284 kNm 927 kN 27 kN

M. A. Ranneh Master of Science Thesis

2-4 Disturbances 13

2-4 Disturbances

The primary source of disturbances which the Gangway system is predominately subject to
is the vessel’s induced motion via sea state.
It is stated in Chapter 1, the problem of motion compensation problem can be recast, through
the relative motion hypothesis, as a dynamic target tracking (i.e. the offshore platform). Nev-
ertheless, understanding the nature of the target motion (i.e. the vessel’s induced motion)
is still of great interest for prediction and control design as it will be further discussed in
Chapter 3 and 4.
Regarding the sea state, an accurate approximation can be made by a representation of waves
that consists of a spectrum of harmonic waves. Well-known spectra are the Pierson-Moskowitz
and the JONSWAP spectra [18]. The latter is based on measurements carried out in the North
Sea [19].
When the Dynamic Positioning (DP) system of the vessel is enabled (i.e. prior to docking the
Gangway). The DP system actively controls the propellers, rudders, and thrusters so that
the induced motion is attenuated on the horizontal plane. However, the measurements and
control signals are not perfect. This results in small deviations (0.5 to 2 m), relatively to the
size of the ship, along the surge and sway axes.
The response of the vessel to a wave spectrum is represented by a set of transfer functions
along the principal axes. These transfer functions are known as Response Amplitude Oper-
ators (RAO). The RAOs contain a magnitude and phase response for the wave frequencies
and angles from which the waves hit the vessel. Using the wave spectrum and the RAOs,
vessel motions can be estimated [20]. However, RAOs are highly dependent on the vessel
characteristics (i.e. shape, mass distribution).
This approach required accurate information about the vessel and its DP system characteris-
tics. This is usually not accessible since most of the manufacturers are very conservative over
sharing such information. Therefore, this approach is not considered in this project.
Alternatively, Bosch provided a data set for the vessel residual motion in DP mode while the
Gangway is docked onto the offshore platform. In Chapter 4, this data set is utilised to train
a neural network type NARX to anticipate the incoming disturbances in a multi-step-ahead
fashion. Intuitively, the performance of the proposed control method benefits significantly
from anticipating the external perturbation (the wave-induced motion).
Figure 2-6, depicts the vessel response to the sea state which results in sinusoidal motion on
six axes that have the following characteristics:

• The wave period should not exceed 12 seconds and 4 seconds for the shortest wave
period with a significant wave height Hsig no more than 3.0 m [6].
• Motion on roll, pitch and yaw axes deviate between ±4 degrees and that is mainly due
to the length and the huge mass of the vessel which can damp out the perturbations on
the aforementioned axes.
• Maximum wind speed: 20 m/sec.

These three points define the operating window which the Gangway system is designed to
function within. Furthermore, other external perturbations (e.g. walking personnel, moving
cargo) have a less significant impact on the Gangway system are therefore out of the scope of
this project.

Master of Science Thesis M. A. Ranneh

14 System Architecture

Figure 2-6: Ship movements in six degrees of freedom in a time window of three minutes.

2-5 Measurement system

The Gangway system consists of the following set of sensors.

2-5-1 Joint-level sensor

Position and pressure sensors on the joint-level to regulate the position and the hydraulic flow
of the system actuators. These low-level control loops are realised with the HNCs.

2-5-2 Motion Reference Unit

The Gangway system counteracts the induced motion on the vessel by utilising a Motion Ref-
erence Unite (MRU). This sensing unit is capable of detecting the vessel motion in six Degree
of Freedom (DoF). MRU is also referred to as an Inertial Measurement Unit (IMU). Where
it utilises accelerometers and gyroscopes to measure its location and orientation. Usually,
the MRUs manufacturers, employ a set of filtering and signal processing techniques within
to provide good readings [21, 22]. However, due to the double integrator that is applied to
the filtered MRU data, A small bias in the acceleration signal can result over time in a large
position error.
Moreover, the Gangway system is employed with three MRUs at the Gangway base for accu-
racy and redundancy. Readings from these MRUs are averaged and considered as one MRU.
Figure 2-1, the reference frame of the MRU is depicted in the base of the system Ψb .

2-5-3 Tip force Measurements

When the control task involves the robot interacting with the environment, a pure motion
control scheme becomes insufficient to execute the manipulation task. Alternatively, a control

M. A. Ranneh Master of Science Thesis

2-6 Control scheme and operator input 15

Figure 2-7: To the left, the definition of forces in the tip. To the right, the docking interface at
an offshore wind turbine. From [5].

scheme with force/torque sensors can provide adequate haptic feedback to the robot.
In our case, to guarantee a safe connection with the offshore structure, the Gangway should
exert a specific amount of force onto the docking area. Also, to avoid damaging the offshore
platform, this exerted force should be actively regulated so that it does not exceed a certain
limit.
Table 2-2 shows the required force and the force limits that the Gangway system can exert.
In order to measure the forces at a robot tip, force/torque sensors are placed precisely between
the tip of the robot and the tool-point. In the Gangway system, three force sensors are placed
along the principal axes at the end of the telescoping part behind a rubber bumper which
was designed specifically to have a predetermined stiffness [6, 7], see Figure 2-7.
The contact force will be translated to the force sensors through the rubber bumper. When the
bumper is in contact with the structure, the friction force helps to avoid stick-slip motion. The
stiffness of the bumper will be used to model the force sensor in the simulation environment.

Table 2-2: Desired tip forces and acceptable forces along the tip principal axes [6], as depicted
in Figure 2-7.

Wind turbine monoplie [N] Design force [N] MCG Limit [N]
Fx 0 - 1000 4000 55000
Fy 4000 - 8000 20000 65000
Fz 0 - 1000 4000 45000

2-6 Control scheme and operator input

The low-level control of the Gangway system is realised via regulating the position and velocity
of each hydraulic actuator separately with the HNCs. Where the operator uses three joysticks
input to influence the velocity and positions setpoints of the actuators. The Gangway system
with the actuators dynamics were considered to behave as three decoupled second-order sys-

Master of Science Thesis M. A. Ranneh

16 System Architecture

tems with a bandwidth of wn = 6 Hz and a damping ratio of ζ = 0.8. This assumption was
made by Bosch to substitute the plant dynamics with the aforementioned characteristics.
Moreover, as seen in Figure 1-5, the high-level control scheme of the Gangway system employs
an MRU in a feedforward fashion to actively compensate for the induced motion.
The position control of the Gangway tip depends on the joints encoders and the MRUs to
compensate for the induced motion.
In order to maintain the connection between the Gangway tip and the offshore platform, we
cannot depend purely on position control due to the MRU drift error and a force control
scheme becomes vital for maintaining contact with the offshore platform [7].
Accordingly, one-dimension force control loop is on parallel with the position control loop and
the force control is enabled once the contact with offshore structure is initiated.
Bosch defines the control task requirements as:

• To guarantee safe passage and to avoid damaging the offshore platform. The contact
force along the contact perpendicular axis should be within the limits as defined in
Table 2-2.

• The time it takes the operator to perform the docking is 1 to 3 minutes which depends
on his/her experience [7]. Thus, to increase the efficiency of the system, it is expected
from the proposed system a shorter docking time.

• The current setpoint signals of the actuators is delivered via the joysticks in velocity and
position terms. Hence, to match the compatibility with the employed HNC structure,
the proposed control scheme should provide both velocity and position setpoints to the
actuators.

2-7 Conclusion

This chapter outlined the Gangway system design and critical component of the control system
as designed by Bosch. In addition, the current control approach is outlined with the control
task requirements. With that, the first research question in Section 2-4 has been addressed.
Based on that, a kinematic model of the Gangway is derived. Additionally, Bosch provides an
approximation of the joint-level control loop of the Gangway system. This model is employed
in the simulation environment and within the proposed control methods in Chapter 4.

M. A. Ranneh Master of Science Thesis

 Chapter 3

Relative Motion Measurements

This chapter is concerned with answering the key research questions:

• How to measure the relative motion between the Gangway and the docking area?

• What kind of sensing technologies are included in the measurement system for the au-
tomated docking?

3-1 The measurement system

Enabling the automated docking of the Gangway system can be translated into eliminating
the human factors from the control loop. This is done by substituting the operator’s function
with a (high-level) control strategy that employs a suitable measurement system.
Correspondingly, as a starting point in this research, the preliminary choice for the proposed
measurement system is based on the fact that the operator uses mainly his/her visual sen-
sory information while steering the Gangway. Moreover, one cannot depend purely on MRU
data for estimating the relative motion. This is mainly due to the drift error.The problem of
mimicking the operator’s perception can be recast as a relative motion estimation problem.
Which is a common problem in robotics and well-studied in literature.
During the literature study phase of this project, various sensing technologies were initially
considered and reviewed as classified in Figure 3-1. To choose a proper measurement system,
the proposed measurement system should not only be able to estimate the relative motion
but also to detect the location docking area on the offshore platform. This is possible by
detecting the features of the target.
Mainly, there are two types of features; geometric (cylindrical, angle, etc.) and textural
(fiducial marker, retroreflective tape, etc.). For example, in the space industry, a spacecraft
docks onto the target, a process referred to as a space rendezvous. The spacecraft acquires
information over the target pose and velocity in order to minimise the impact force. Using
a cutting-edge vision system referred to as 3D Flash LiDAR Camera the relative motion is
estimated by first capturing the geometrical and textural features of the target, [23, 24].

Master of Science Thesis M. A. Ranneh

18 Relative Motion Measurements

Figure 3-1: Classification of range sensing technologies

Similarly, to achieve the automated docking of the Gangway, the relative motion between
the Gangway and the offshore structure should be estimated and the exact location of the
docking area should be detected. So, it is a matter of tracking a dynamic target (i.e. the
docking area) then docking onto the target. In [25–27], a robotic manipulator is employed to
track and grasp free-flying objects in the 3D space while depending purely on a vision-based
measurement system.
Correspondingly, the automated docking task has a strong point of resemblance to the pro-
cess of grasping a flying object. Together with the aforementioned criteria, a vision-based
measurement system would be a good fit for the application of automated docking.
However, for the force control task (i.e. regulating the contact force), as reported in Chapter
2 that; due to MRU drift error, tip force measurements are essential elements to maintain
continuous contact and to avoid damaging the offshore structure.

3-2 Sensor architecture and sensor placement

In literature, there are various architectures of vision-based measurement systems. The most
common ones namely are a monocular camera, a stereo camera, and multiple monocular
cameras.
To maintain simplicity in system and control design, a monocular camera was chosen.
Mainly, there are two different configurations when it comes to placing a vision sensor within a
robotic manipulator system; eye-in-hand or eye-to-hand. In the first configuration, the vision
sensor(s) attached on the robot end- effector. Whereas in the second, the sensor(s) is fixed in
the 3D space with a field of view focused on the workspace of the control task, see Figure 3-2.

M. A. Ranneh Master of Science Thesis

3-3 Camera mathematical model 19

From a control point of view, it can be classified as a collocated system or a non-collocated

one, respectively. Additionally, for an object-tracking task, eye-in-hand is more appealing to
ensure that the tracked features stay in the field of view [28]. Hence, this configuration is
employed in the proposed control scheme in Chapter 4.

Figure 3-2: (a) eye-in-hand. (b) eye-to-hand. From [8].

3-3 Camera mathematical model

One advantage of employing a vision sensor in the measurement system is that a mathematical
model can be defined. Hence, this analytic expression of a vision sensor can enable us to design
and testing a vision-based control strategy in a simulation environment. This model is based
on the so-called pin-hole camera model which employs the perspective projection (i.e. 3D to
2D mapping), see Figure 3-3. This model is widely employed in literature [9].
Hence, the function of this model is mapping the target coordinates from the Cartesian space
into the image plane, which allow us to generate synthetic data. For a point P expressed in
the world reference frame in 3D space, this mapping is realised through the following steps

1. Transforming the observed feature point P , represented in euclidean coordinates P =

[Px , Py , Pz ]T , to homogeneous coordinates P̄ = [Px , Py , Pz , λ]T , where λ is a scaling
factor with default value λ = 1.

2. The homogeneous point P̄ is, then, described in the camera reference frame via the
homogeneous matrix H0c , which is the inverse transformation matrix between camera
and the world reference frames Hc0 . This transformation results in the homogeneous
matrix HP̄c .

3. A 3D-to-2D mapping is, then, applied to the resulted homogeneous matrix HP̄c .

4. The outcome of this process, a projected point p̄ = [ū, v̄, w̄]T represented in homogeneous
coordinates.

5. The projected point can be retrieved to the euclidean image plane as p = [ū/w̄, v̄/w̄]T .

Master of Science Thesis M. A. Ranneh

20 Relative Motion Measurements

Figure 3-3: Perspective projection in the pin-hole camera model. From [8]

The corresponding mathematical formulation, of the aforementioned steps, is given as

 
P
u0 h i  x 
   
ū fx 0
P 
 v̄  =  0 fy v0  H0c  y  , (3-1)
    
 Pz 
w̄ 0 0 1
1
with " #−1
h i−1 Rc0 t0c
H0c = Hc0 = , (3-2)
01×3 1

Here, fx and fy are the focal lengths on the horizontal and vertical axes respectively described
in pixel units. The point c = (u0 , v0 ) is referred to as the principle point and ideally it is
located in the centre of the image plane, Figure 3-3. These parameters are referred to as the
intrinsic parameters.
The homogeneous matrix Hc0 describes the transformation between the fixed-world reference
frame and the camera reference frame. Accordingly, Rc0 , t0c represent the rotational and the
translational parts of Hc0 , respectively. This transformation matrix is known as the extrinsic
parameters of the camera. Figure 3-4 depicts the transformation between the camera and the
fixed-world reference frames.
The pin-hole camera model assumes an ideal thin lens, however, this is usually not the case.
Commercial lenses distorts the light array prior reaching the vision sensor. The lens distortion
mainly consists of radial and tangential distortion components and it can be modelled as an
additive change to the projected point (x, y) in the image plane

xdistorted = x (1 + k1 r2 + k2 r4 + k3 r6 )
, (3-3)
ydistorted = y (1 + k1 r2 + k2 r4 + k3 r6 )

M. A. Ranneh Master of Science Thesis

3-4 Camera calibration 21

Figure 3-4: The transformation between target and camera reference frames. (Modified) From [8]

Figure 3-5: An image of a checkerboard captured with an ideal lens, a lens with positive distortion,
and a lens with negative distortion, respectively. From [9].

with
r 2 = x2 + y 2 , (3-4)
where k1 , k2 and k3 are the distortion coefficient. Figure 3-5 illustrates the different effect
on the image when distortion exits with positive (k1 , k2 , k3 > 0) or negative (k1 , k2 , k3 < 0)
coefficients. The process of estimating the camera intrinsic and extrinsic parameters and lens
distortion is referred to as camera calibration.

3-4 Camera calibration

In order to simulate the camera model, camera parameters should be estimated. Calibration
requires acquiring a set of images from different angles and various distances of a predefined
planar object (e.g. a checkerboard), see Figure 3-6. Then, the observed features in the images

Master of Science Thesis M. A. Ranneh

22 Relative Motion Measurements

Figure 3-6: Captured images for camera calibration

of the checkerboard (i.e. corners and edges) are compared with the ground truth measurement
of this planar object. Afterwards, the calibration problem is recast as a nonlinear least-squares
optimisation problem for estimating the camera’s intrinsic and extrinsic parameters [29, 30].
This procedure can be realised seamlessly in the Camera Calibration Toolbox for MATLAB.
In robotics applications that involve a vision-based control scheme, only calibrating the vision
sensor is often not sufficient. Therefore, additional calibration is required between the robot
and the vision sensor. This is done by estimating the transformation matrices between the
sensor(s) and robot reference frames either within an eye-in-hand or an eye-to-hand scheme
[31].
Fortunately, the proposed control method in Chapter 4 solves the relative motion problem
implicitly in the image plane. Where it has proven to be robust against various calibration
errors and no on-site precise calibrations between the camera and the Gangway tip reference
frames is required but rather a coarse measure will suffice.

3-5 Representing the offshore platform

Instead of detecting the docking area on the offshore platform, a solution is proposed to detect
a fiducial marker the so-called Aruco marker, see Figure 3-7.
Hence, in the simulation environment, this target is considered to be the marker. Whereas, in
reality, it can be attached next to the docking area. Aruco markers can be reliably detected
and localised in both the Cartesian space and the image plane [32]. Therefore, Aruco markers

M. A. Ranneh Master of Science Thesis

3-6 Synthetic data 23

are widely employed in robotics and Augmented Reality (AR) as a source for the ground
truth data in localisation applications.
From a critical point of view, One can argue that attaching a marker to each offshore platform
sounds impractical. For example, over time, the marker surface might accumulate some dirt
(i.e. dust, birds faeces), and the invisibility of the marker during the night and other bad
weather conditions. This can cause a potential error in detecting the marker.
Some ideas were proposed during the literature study to tackle the aforementioned problems.
For instance, employing a self-cleaning mechanism onto the marker stand and adding a NIR
(Near Infra Red) illumination layer to the marker with a NIR camera on the Gangway, as
proposed in [33].
On the other hand, the choice for this marker in this project is for the sake of simplicity
and one can argue that, nowadays, many state-of-the-art algorithms are being developed in
the area of object detection and localisation and the need to add visual marker to identify a
specific object is not required anymore [34, 35].

Figure 3-7: Aruco marker

3-6 Synthetic data

As stated above, the outcome of the literature study suggests a vision-based control scheme
for solving the automated docking problem. However, there are no images recorded from the
gangway tip. Therefore, synthetic data were made from the MRU data of a vessel with the
Dynamic Positioning mode switched on. Figure 3-8 depicts the pipeline of generating the
synthesised images.
First, to simulate a moving target, the MRU data is fed to the four dots to move in six-
DoF. These dots represent the marker corners attached to the offshore platform. Then, the
perspective projection model of the camera projects these dots to the image plane. After
that, an image of the marker is wrapped into projected points. This process, also known as
homography [8], is concerned in finding the transformation between two objects in a plane.
Then, additive white Gaussian noise is applied to the generated image to make it more
realistic. Finally, the image is passed to the marker detection script to detect the marker
location in the image plane.

Master of Science Thesis M. A. Ranneh

24 Relative Motion Measurements

Figure 3-9 shows the target motion in 3D (MRU data) observed from the camera reference
frame and the corresponding 2D mapping of this motion in the 2D image plane. Figure 3-10
is the 3D representation of the target relative pose to the camera that is attached at the tip
of the Gangway.
Additionally, this scheme is embedded within the model-based control method in Chapter
4 to make use of the MRU data in predicting the target motion in the image plane in a
multi-step-ahead prediction scheme.

Figure 3-8: The process of generating the synthesised images

3-7 The motion model of the target

In Chapter 4, the proposed vision-based control method utilises the visual features explicitly
in the control loop as an output to the system. In other words, the corners of the detected
marker are considered to be the output signal of the measurement system.
In many control systems, the output signal is rather contaminated with noise and subject to
disturbances. Likewise, the measurement of the detected location of the marker in the image
plane should be corrected.
In the literature of dynamic object tracking, model-based tracking techniques are quite pop-
ular [31, 36–38]. The motion model of the object is not only utilised to estimate and correct
the readily observed measurements, but it can also ensure the continuity of the output signal
in case the object leaves the field of view, this is important to avoid instability in the control
loop.
In [39], a survey is presented over the existing motion models and filtering techniques for
dynamic object tracking. In [40], the so-called constant velocity model is employed with a
linear Kalman filter as a state-observer to track a moving ship assuming the sea-induced ship
motion is relatively slow.

M. A. Ranneh Master of Science Thesis

3-7 The motion model of the target 25

Marker motion in 3D space

Marker displacement (m)

6

Sway
0 Surge
Heave

0 500 1000 1500 2000 2500 3000

Iteration interval
Marker motion in 2D image plane
400
Marker dispalcement (pixel)

350

300

250

200
Horizontal image axis
150 Vertical image axis

0 500 1000 1500 2000 2500 3000

Iteration interval

Figure 3-9: The target (the marker) movement is 3D observed from the camera reference frame
and the corresponding 2D mapping in the image plane.

Similarly, in this project, the same approach was followed to track the moving marker. It
consists of using a Kalman filter to estimate the corners and the centre of the marker.
This approach can also estimate, for a short time window, how the marker moves in the image
plane when the detection is a not available anymore (i.e. when the marker leaves the field of
view).
Figure 3-11 shows the real-time tracking data of the Aruco marker in the image plane, the
images were taken from a standard web-cam with an update rate of 30fps. A sinusoidal-based
motion that resembles the sea-induced motion was applied to the marker. Notice, when the
marker leaves the field of view it becomes undetectable (for visualisation purpose the value
filled with -1), Kalman filter still provides an estimate of how the target is moving. This esti-
mation depends merely on the employed model and it is essential to avoid any discontinuity
in the output signal which the tracking control depends on.
The so-called constant velocity model is the first order approximation of Taylor series, for a
point p = (px , py ) in a 2D plane it can be given as
x(k) = A4x4 x(k − 1), (3-5)
with the state vector x = [px , py , p˙x , p˙y ]T is the displacement and the velocity components of
the horizontal and vertical axes respectively, and
 
1 0 Ts 0
0 1 0 T s
A4x4  . (3-6)
 
0 0 1 0
0 0 0 1
The marker is tracked by tracking the its centre and the four corners as an interest points.
Hence,the previous model can be trivially extended to five point. The standard linear Kalman

Master of Science Thesis M. A. Ranneh

26 Relative Motion Measurements

Figure 3-10: 3D representation of the a Gangway-like structure with the camera attached on
the tip. The offshore platform is represented by the square grey marker.

filter is employed with the mathematical formulation that is given in [41].

3-8 Conclusion

A measurement system that is comprised of a monocular camera, MRU, and force sensors
at the tip is required to enable the automated docking of the Gangway. Where eye-in-hand
configuration (i.e. a camera is attached on the tip) is employed to ensure that the tracked
features stay in the field of view. Next chapter presents the proposed control approach that
accommodates this measurement system. Where, it was learned that vision sensor becomes
insufficient for the force control task. Hence, a monocular camera alone cannot solve the
docking problem and a multi-sensor solution is required.

M. A. Ranneh Master of Science Thesis

3-8 Conclusion 27

Horizontal axis of 2d Image (Pixel)

Marker detected location along the Horizontal axis vs. estimated location by kalaman filer

Marker detecteted location in image plane

600 Marker estimated location by kalaman filer

400

200

0
100 200 300 400 500 600 700
Iteration interval
Marker detected location along the vertical axis vs. estimated location by kalaman filer
Vertical axis of 2d Image (Pixel)

600
Marker detecteted location in image plane
Marker estimated location by kalaman filer
400

200

0 100 200 300 400 500 600 700

Iteration interval

Figure 3-11: The process of generating the synthetic data

Master of Science Thesis M. A. Ranneh

28 Relative Motion Measurements

M. A. Ranneh Master of Science Thesis

 Chapter 4

Modelling & Control Design

This chapter is concerned with answering the key research questions:

• How can a vision sensor be incorporated within the control loop for automated docking?

• Is there a control scheme that combines vision and force control while handling con-
straints?

• Is it possible to forecast the sea-induced motion? Would the proposed control method
benefit from anticipating the external perturbation ahead?

Thus, this chapter presents the modelling and control concept of the Gangway system.
Where, first, a brief background over the vision-based control concept in robotics is given.
Then, for the purpose of automated docking, a nonlinear Model Predictive Control (MPC)
method is proposed. MPC is a model-based control approach, thus, a mathematical model of
the Gangway system is derived. Subsequently, the MPC problem is formulated with a discus-
sion over its parameters selection process. To enable the disturbance anticipatory property in
MPC, a neural network with NARX topology is desired to forecast the sea-induced motion.
Lastly, a brief analysis of closed-loop stability for MPC is shown.

4-1 The concept of vision-based control

In Chapter 3, it was concluded that a vision-based measurement system fits best for the
relative motion estimation between the Gangway and the offshore structure. As a result, a
measurement system with a vision sensor (i.e. monocular camera) will be employed in the
control loop.
In robotics, integrating a vision sensor into the control loop is often described in the relevant
literature in two different fashions: either in the open-loop [31, 33, 42] or closed-loop [43–45].
The former employs the term ’Look-and-move’ which basically means the vision sensor gives
the setpoint to the system each iteration without incorporating the visual feedback within

Master of Science Thesis M. A. Ranneh

30 Modelling & Control Design

the control loop. The latter, also referred to as ’Visual Servoing’, employs the vision sensor
into the control loop as a feedback signal [46].
Usually, vision-based measurement systems have a relatively low bandwidth compared to
the joint encoders of a robotic manipulator, and that is mainly due to the latency in image
acquisition and processing [8]. Therefore, when it comes to employing a vision sensor within
a control loop, it is reasonable to consider the control system of a robotic manipulator with a
vision sensor as two cascaded loops; an inner loop for joint velocity control using the encoders
feedback, and an outer loop that incorporates the visual feedback within the high-level control
strategy, where the nature of the reference signal is chosen based on of the employed visual
servoing approach, as seen in Figure 4-1.

Figure 4-1: The general control scheme of visual servoing. (Modified) From [10]

Moreover, the mathematical formulation of a classical visual servoing scheme is given as

e(t) = s(t) − s(t)∗ , (4-1)

where s∗ (t) is the desired setpoint which its nature follows the same as s(t) and usually is
chosen as a constant [47]. For the sake of simplicity, we choose s∗ ∗ as constant. s, one the
other hand, can consist of a set of features that are readily available in the image data (image-
based), or it consists of a set of three-DoF or six-DoF parameters, which must be estimated
from image measurements (pose-based). In both cases, s(t) can be related to the six-DoF
velocity twist of the vision sensor by
ṡ = Ls Vc , (4-2)
where Vc represents the spatial twist of the camera, and the so-called interaction matrix Ls ,
also known as the image Jacobian matrix, is the matrix which maps the velocity vector of the
feature to the velocity of the camera. The image Jacobian is either an explicit or an implicit
function of the target pose in 3D space depending on the employed VS concept pose-based
or image-based, respectively.
The output of the visual seroving control is basically the desired spatial velocity of the camera.
From the previous relationship
Vc = L†s ṡ, (4-3)
where the pseudo inverse, denoted by †, is usually employed here to avoid inverting an ill-
conditioned Jacobian [47]. With a constant reference signal s∗ , the previous relationship can
be written in terms of ė
Vc = L†s ė. (4-4)

M. A. Ranneh Master of Science Thesis

4-1 The concept of vision-based control 31

Since it is desired to have an exponential decrease of the error signal, we write

ė = −λe. (4-5)

Then, the final expression becomes

Vc = −λL†s e. (4-6)
In case of a monocular camera attached at end-effector of a robotic manipulator, the desired
special velocity of the camera can be translated into the desired joints velocity of the robot q̇
with the robot Jacobian Jr
q̇ = −λJr† L†s e. (4-7)
One of the main challenges in visual seroving schemes is acquiring a full rank well-conditioned
interaction matrix. Additionally, there are different schemes of visual servoing to design the
interaction matrix Ls and the nature of the observed feature s. Where Ls can be an explicit
function of the target pose in 3D space with respect to the camera in case of Pose-Based
Visual Seroving (PBVS). Alternatively, Ls can be an implicit function of the target pose
in case of Image-Based Visual Seroving (IBVS). Where it depends on the target measured
features s(t) and the camera intrinsic and extrinsic parameters.
The advantages and drawbacks of the different common schemes of visual servoing are dis-
cussed in the following subsections.

Figure 4-2: The different visual servoing schemes. (a) PBVS general control scheme the setpoint
signal is the desired relative pose between the tip and the target. (b) IBVS general control scheme
where the setpoint is the desired field of view of the target in the image plane, this solve the pose
estimation problem implicitly when the tracking error converges to zero. From [8].

Master of Science Thesis M. A. Ranneh

32 Modelling & Control Design

4-1-1 Pose-Based Visual Servoing (PBVS)

In the PBVS scheme, the control task is achieved within the 3D Cartesian space. This
is achieved by estimating the relative pose between the target and the vision sensor. The
reference signal here is given in Cartesian space where the error signal holds information
about the current pose of the camera with respect to the target. This is realised using
photogrammetry and sensor fusion techniques [48, 49]. Figure 4-2 depicts the general control
scheme where the setpoint signal is the desired relative pose between the tip and the target.
PBVS usually required multiple vision sensors (e.g. stereo camera) with precise calibration.
Also, the relative pose estimation process is highly dependent on the accuracy of the camera’s
intrinsic and extrinsic parameters. This sensitivity might lead to a ’bad’ convergence in the
closed-loop. However, if the pose estimation is flawlessly attained and the camera is well
calibrated then global asymptotic stability of the closed-loop system can be ensured.
Moreover, in the PBVS scheme, the robot end effector might leave the field of view which
might result in a discontinuity in the measurement signal [8, 47]. This is considered a major
drawback because this might results in discontinuation in the output signal which might lead
the controller to unstable behaviour.

4-1-2 Image-Based Visual Servoing (IBVS)

Unlike the PBVS scheme, in IBVS, the task function represents the error in the 2D image
space, as seen in Figure 4-2. Basically, the feedback signal here is the readily available location
of the tracked features in the image plane, and the setpoint represents the desired location
for these features in the image plane [8, 47]. With this, the relative pose problem is solved
implicitly without the needs for multiple vision sensors nor for an excessive calibration of the
vision sensor(s) parameters.
IBVS relies on basic image processing techniques and requires a minimum of three observed
features to ensure well-condition image jacobian [47]. Therefore, IBVS is more computation-
ally efficient and more immune to calibration error and noise [50]. Furthermore, in IBVS the
observed features are kept in the field of view. However, due to the nature of the feedback
signal, the 3D Cartesian trajectory is not considered. This might lead the robot to a singu-
larity configuration (e.g. joint limits) [47].
Moreover, when it comes to stability analysis, IBVS can only ensure local asymptotic stability
of the closed-loop and it might be attracted to a local minimum when the desired locations
of the observed features are somewhat distant from the current locations [8, 47].

4-1-3 Visual Predictive Control (VPC)

IBVS proved to achieve adequate and robust performance against the camera and robot
calibration errors and image measurement errors while assuring local asymptotic stability [51].
Nevertheless, classical IBVS schemes cannot deal with constraints such as joint limits and the
camera field of view.
Visual Predictive Control (VPC) was first reported and formulated in the work of Allibert et
al [11]. This control method considers the so-called Internal Model Control (IMC) structure.
The process block contains the robotic system and the camera, as seen in Figure 4-3. The
input U is the robotic control variable and the output s is the current value of the visual

M. A. Ranneh Master of Science Thesis

4-2 The proposed control scheme 33

features. For IBVS, the reference s∗ is expressed in the 2D image plane, as the visual features,
and can be static or dynamic [52].
The scheme of VPC offers not only constraints handling but also modelling and calibrations
errors if the utilised internal model in the IMC block is the nominal model of the plant. This
model can be acquired based on the image Jacobean matrix which can suffer from singularity
condition (as it is the case with classical IBVS), [47].

4-2 The proposed control scheme

To enable the automated docking in a hierarchical fashion, the control task, during the design
phase of this project, was categorised into two main parts; tracking the target (i.e. the offshore
platform) and docking into the target.
This method is based on the standard nonlinear Model Predictive Control (MPC). Figure
4-4, depicts a high-level control scheme of the proposed method. This proposed scheme highly
mainly resembles the general control scheme of VPC in the work of Allibert et al [11], see
Figure 4-3. In a sense that, both are MPC-based approaches and in the nature of the reference
(desired field of view) and the output signal of the target (target’s features in the image plane).
Similarly to the VPC scheme, the proposed method can handle the different constraints of
the Gangway system such as workspace constraints, field of view constraints, and contact
force constraints, see Table 2-1 and 2-2. However, the proposed scheme differs from the VPC
scheme in numerous aspects. The following points highlight the main differences between the
proposed scheme and the VPC scheme.

• Different plant model considerations

In the VPC scheme in [11], the plant model is merely a free-flying camera that can
move in six-DoF in the Cartesian space. Each DoF axis is represented by an integrator
which means instant convergence to the setpoint. Clearly, this is highly abstract and
simplified settings for a visual servoing problem. On the other hand, the derived model
of the Gangway system is more concrete in describing realistic settings, more details in
Section 4-3.

Figure 4-3: The general control scheme of VPC. This scheme, similarly to IBVS, takes the
desired field of view as a setpoint s∗ (k). The Model block is the nominal model that is utilised
in the internal model control structure. From [11]

Master of Science Thesis M. A. Ranneh

34 Modelling & Control Design

• No need for the IMC structure

In [11], it is stated that the internal model structure in the VPC scheme offers robustness
against parametric uncertainties. The main function of this scheme is basically mea-
suring the difference between the output of the real plant and the nominal plant model
and adjusting the setpoint signal accordingly to compensate for the model mismatch.
However, in reality, the image plane is discretised in pixels unit and the resolution of the
image plane is rather insufficient to for minor setpoint adjustments which are needed in
the case of occurring model mismatch while the target is close to the camera in 3D space
(i.e. when the Gangway tip is in contact with the platform). Accordingly, during the
extermination phase, the proposed control method showed robustness against different
model mismatch cases (e.g. camera calibration, actuators dynamics). Thus, from a
practical point of view, there is no need here for the internal model structure.
• No need for the image Jacobian
In VPC and other visual seroving schemes, the image Jacobian matrix is employed in the
model. However, the proposed method depends on the pinhole perspective projection
model, as described in (3-1). Generally, it is difficult to ensure a well-conditioned image
Jacobian. This is an inadmissible property, especially, when the camera has restricted
movement, in case the camera is attached to a three-DoF structure (e.g. the Gangway
system) instead of a six-DoF one (e.g. free-flying camera). In addition, the pinhole
model maps the displacement of the target in 3D space to the 2D image plane rather
than mapping the spatial velocity of the target as the image Jacobian do. For the case of
automated docking, this is permissible since the target moves in sinusoidal-based motion
(the sea-induced motion) with a slow velocity with a period of 4 to 12 sec. Hence, the
proposed control method eliminates this potential source of singularities caused by an
ill-conditioned Jacobian matrix.
• Continuity of the measurement signal
In case of a sudden cluttering or invisibility of the target, the proposed can ensure the
continuity of the target visual signal by employing the Kalman filter scheme described
in 3-7. The Kalman filter estimates the target location in the field of view in case of
sudden invisibility.
• Different control task objectives
Although, both the proposed scheme and the VPC scheme are concerned with matching
the current field of view with the desired one. However, to enable the automated docking
of the Gangway system, regulating the contact force at the tip is essential to maintain a
motionless contact while tracking the target motion. To effectively keep the contact force
within the allowed limits, the proposed scheme also utilises the MRU measurements,
more details in Section 5-1-2. Thus, from a control point of view, the proposed control
scheme is concerned with a more challenging problem while comprises a more complex
measurement system than the one the VPC scheme in [11].

4-3 Modelling considerations

Normally, standard nonlinear MPC is a model-based method. Hence, for the purpose of
control design and simulation, a mathematical model that describes the Gangway system

M. A. Ranneh Master of Science Thesis

4-3 Modelling considerations 35

Figure 4-4: High-level control scheme of the proposed method.

with its measurement system should be derived.

4-3-1 Kinematics modelling

As stated in Section 4-1, due to the latency in image acquisition, and processing the visual
servo control scheme is usually designed on a top of a low-level control which is employed
on the joints level. Additionally, in Chapter 2, it is stated that each hydraulic actuator in
the Gangway system is supplied with a position feedback loop, feedforward velocity control,
and pressure feedback control. Based on that, Bosch suggests that the current closed-loop
Gangway system with joint-level control is considered to behave as three decoupled second-
order systems. This assumption was made by Bosch in order to substitute the low-level
control loop dynamics of the Gangway system with the linear system as follows
 
g(s) 0 0
G(s) =  0 g(s) 0 , (4-8)
 
0 0 g(s)

where g(s) is a second-order transfer function which is an approximation of the dynamics

behaviour of the low-level control loop along each actuated joint with a bandwidth of wn = 6
Hz and a damping ratio of ζ = 0.8. Hence, g(s) can be written as
wn2 1421.29
g(s) = = 2 . (4-9)
s2 + 2ζwn s + wn2 s + 60.32s + 1421.29
The nonlinear MPC requires the derived model in a state space representation. Thus, the
state space realisation of the subsystem g(s) can be written as
" # " #
−60.32 −1421.29 1421.29
ẋ(t) = x(t) + u(t)
1 0 0 (4-10)
h i
y(t) = 0 1 x(t)

where x(t) is the state variables vector of the closed-loop actuator dynamics(e.g. slewing).
u(t) is the setpoint signal of the actuator, and y is the corresponding actuator response.

Master of Science Thesis M. A. Ranneh

36 Modelling & Control Design

Correspondingly, the state space realisation of the low-level Gangway system G(s) is denoted
by Gss . Which takes the actuators setpoint as inputs variables and the responses of the
actuators as output variables.
Accordingly, the open-chain kinematics model of the Gangway system was derived in an
inertial reference frame assuming that, as stated in Chapter 1, it is a matter of relative
motion. Hence, the Gangway base is attached to the fixed-world reference frame and the
offshore platform is moving, see Appendix 1 for more details on the kinematics model. Figure
4-5, depicts a 3D visualisation the of open-chain kinematics of a Gangway-like mechanism
(RRP) with two revolute joints (i.e. slewing and luffing joints), and one prismatic joint (i.e.
the telescoping joint).

Figure 4-5: The 3D visualisation of the Gangway kinematic model. (Modified) From [12].
Where θ1 , θ2 , and θ3 are the Gangway system actuated joints represented by slewing, luffing, and
telescoping actuators, respectively. ψb and ψtip are the reference frame of the Gangway base and
the tip, respectively. L1 , L2 , and L3 are the constant displacements of the corresponding joints.

4-3-2 Incorporating the vision sensor within the model

This derived model should also incorporate a model for the vision sensor that is attached on
the Gangway tip as Chapter 3 concludes. Now, despite the fact that visual servoing schemes
usually employ the image Jacobian as the mapping matrix between the target spatial velocity
and the corresponding velocity vector in the 2D image plane. Instead, the pinhole camera
mathematical model, as described in (3-1), is employed for the following reasons:

• In visual servoing schemes, it is assumed that the vision sensor has six-DoF, whereas,
in our case, the camera attached on the Gangway tip which has only three-DoF.

• Generally, it is difficult to ensure a well-conditioned image Jacobian, especially, with

the restricted movement of the camera (three-DoF instead of six-DoF).

M. A. Ranneh Master of Science Thesis

4-3 Modelling considerations 37

Figure 4-6: The mass-spring analogy of the contact force model.

• The pinhole camera model, reported in Chapter 3, describes the 3D-to-2D mapping in
terms of displacement. This is convenient in the application of automated docking since
the target moves slowly in the image plane with a period between 4 to 12 sec, which is
originally the period of the wave-induced motion, as described in Section 2-4.

4-3-3 Force modelling

The contact force measurement is a vital element for the docking control task, thus, the
derived model should include the contact force model.
Usually, in robotic manipulator systems, force control is combined with motion control [10].
When it comes to any manipulation process (e.g. docking onto a dynamic target), combining
the feedback of vision and force sensors becomes essential in order to fulfil the control task
requirements.
When the Gangway tip is in contact with the offshore platform, the contact force can be
modelled as a mass-spring system as depicted in Figure 4-6 and shown in the expression
below
fc = Kb (ytip − yplatf orm ). (4-11)
Here, Kb is the stiffness of the bumper at the tip of the Gangway. A few tests were carried out
by Bosch, to identify the bumper stiffness, show that the bumper stiffness behaves linearly
in the operating range [6]. The term (ytip − yplatf orm ) is the difference between the Gangway
tip and the offshore platform along the perpendicular axis respectively.
Ideally, the force feedback is only enabled when the robot is in direct contact with the tar-
get/environment, hence to ensure that fc ≥ 0 in the derived model, the following applies

fc = max(0, Kb (ytip − yplatf orm )). (4-12)

Master of Science Thesis M. A. Ranneh

38 Modelling & Control Design

4-3-4 The nonlinear state space model

During the design phase of the proposed control method, it was noticed that in order to fulfil
the force control task requirements, the target 3D motion should be incorporated within the
derived model, more details in Section 5-1-2.
Now, considering the aforementioned assumption with the Gangway is fixed and the target
is moving, and given that in the real system the MRU can provide the spatial velocity mea-
surements of Gangway base, the nonlinear model is extended to include the MRU readings
as external perturbations that can be mapped into the target spatial motion.
Consequently, the continuous-time nonlinear state space representation of the Gangway with
the measurement system can be given as follows

ẋ(t) = f (x(t), u(t), d(t))

(4-13)
y(t) = h(x(t)) .

Here, the state vector x(t) represents the Gangway joints displacement, and the target posi-
tion in 3D space, respectively. The input vector u(t) is the tip/camera spatial 3D velocity,
and the sea-induced spatial velocity d(t) acts as external perturbations to the system. The
output vector y is the marker readily observed features coordinates in the 2D image plane
(four corners and the centre of the marker), and the contact force response along the perpen-
dicular axis with target, respectively.
Moreover, the nonlinear state transition function f (x, u) is merely the inverse Gangway Jaco-
bian matrix J −1 (x) and the mapping matrix AdH M RU between the external perturbation and
target
the target spatial velocity, respectively. Lastly, the output function h(x) is the perspective
3D-to-2D projection pinhole camera model, as described in (3-1). This results in the target’s
detected features px,y (marker’s centre and corners). The second part of h(x) is the contact
force model along the contact perpendicular axis.
A more detailed description of the aforementioned variables and function is given below



 x = [qn , qg , Ptarget ]T ; x ∈ X ⊆ R12




qn = [qslew , qluf f , qtele ]T








qg = Gss qn








= [Px , Py , Pz ]T




 Ptarget



= [T ipvel ]; u ∈ U ⊆ R3

u


, (4-14)
T ipvel = [ẋtip , ẏtip , żtip ]








= [vesselvel ]; d ∈ D ⊆ R3




 d







 vesselvel = [surgevel , swayvel , heavevel ]




y = [image, f orcey ]T ; y ∈ Y ⊆ R11








image = vec(px,y ); px,y ∈ P ⊆ N2×5


M. A. Ranneh Master of Science Thesis

4-4 Nonlinear Model Predictive Control (MPC) 39

with   

 Gss x(t)

  
  −1
= J (x(t)) u(t)

f (x, u, d)

 

 

  
AdH M RU d(t)

target . (4-15)

 " #

P inhole(x(t))




 h(x(t)) =
fc (x(t))





Note that, qg here is the joints displacement response of the low-level dynamics Gss , described
in (4-8) and (4-10), and qn is the joints displacement setpoint for the low-level control loop.
Accordingly, the open-loop block diagram that describes this model is depicted in Figure 4-7.

Figure 4-7: A zoomed-in view of the corresponding block diagram of the open-loop Gangway
system as described in (4-13) to (4-14).

4-4 Nonlinear Model Predictive Control (MPC)

With the increase of computational power capabilities, MPC became a very active area of
research, especially, because of its ability of to deal with Multiple-Input Multiple-Output
(MIMO) systems while handling terminal constraints and systems limitations.
MPC problems are solved numerically since the analytical solution is mostly nontrivial. Hence,
MPC requires a discrete-time mathematical model of the plant.
The derived model for the Gangway systems described in (4-14) is a nonlinear continuous-time
model. The forward Euler method, for solving first-order differential equations, is considered
here as the discretisation method, since it is intuitive and computationally efficient. Euler

Master of Science Thesis M. A. Ranneh

40 Modelling & Control Design

method is given as
x(k + 1) − x(k)
ẋ = , (4-16)
Ts
and the discrete-time state space equations becomes

x(k + 1) = x(k) + Ts f (x(k), u(k), d(k)), (4-17)

where k is the sampling time instance and Ts is the discretisation time step.
In the case of employing a linear dynamic model in an MPC framework, the MPC optimisa-
tion problem can be regarded as a convex problem and global optimum might be acquired.
However, when the plant model is nonlinear, the optimisation problem becomes a nonlinear
non-convex problem. Hence, solving the problem for the global optimum becomes computa-
tionally expensive and the global optimum is not guaranteed.
In case of the absence of constraints in an MPC problem, standard linear MPC. in essence, is
merely a finite-horizon formulation of the Linear Quadratic Regulator (LQR) [53], in a sense
that the control objective is to minimise a quadratic cost function along with a given time
window. The length of the time window, in MPC, is referred to as the prediction horizon
over which MPC predicts the system response upon a set of control input sequence.
Likewise, the nonlinear MPC follows the same logic of linear MPC, however, with nonlinear
system equations and the formulation of the employed nonlinear MPC can be read as

min J(y(k), ∆u(k)),

∆U
where
Np c −1
NX
X
J(y(k), ∆u(k)) = ||Q (y(k + i|k) − r(k))||22 + ||R ∆u(k + i|k)||22 ,
i=1 i=0



 x(k + 1) = x(k) + Ts f (x(k), u(k), d(k)) (4-18)

y(k) = h(x(k))





u(k) = u(k − 1) + ∆u(k)


subject to .


 umin ≤ u(k) ≤ umax

∆umin ≤ ∆u(k) ≤ ∆umax







ymin ≤ y(k) ≤ ymax

The objective here is to find manipulated variables ∆U that minimises the quadratic cost
function J(k) the most over the prediction horizon Np at each time step. The cost function
J(k) is subject to a set of constraints including the Gangway system equations, boundaries
on the input u(k), the input increment ∆u(k), state vector x(k), and on the system out-
put y(k). The optimisation manipulated variables ∆U are the input increment sequence
{∆u(k|k), · · · , ∆u(k + Nc − 1|k)} over the control horizon Nc .
The optimisation runs at each time step k and tries to find the ’best’ solution ∆U that min-
imises J(k) the most and it is feasible with respect to constraints. This is achieved recursively
where each generated solution is tested out by MPC. MPC predicts, using the plant model,
the system response {y(k + 1|k), · · · , u(k + Np |k)} against the generated admissible set of the
control input sequence {u(k + 1), · · · , u(k + Nc )} over the prediction horizon Np . Then, the
corresponding cost function’s value for the generated solution is computed.
The weighting matrices Q, R, described in (4-17), are the error tracking weights matrix and

M. A. Ranneh Master of Science Thesis

4-5 MPC parameters tuning 41

Figure 4-8: To the right, the predicted marker images over the prediction horizon (e.g. Np =
3 for visualisation purposes). To the left, the plant actual response over multiple time steps
{t, t + 1, t + 2} to the left. (Reproduced) From [11].

the input effort weights matrix, respectively. The setpoint signal r(k) is constant vector along
the prediction horizon where it represents the desired location of the marker’s corner within
the desired field of view and the setpoint for the contact force, respectively.
Figure 4-8, depicts the predicted response over the prediction horizon against the actual plant
response over multiple time steps. The system response is represented with which is repre-
sented by the observed marker features(i.e. the red dots) in the image plane. The marker
appears initially in an arbitrary field of view. Then, evolves with MPC control commands to
the desired field of view that corresponds to error-free of the Gangway tip 3D pose with re-
spect to the target. Correspondingly, Figure 4-9 depicts the closed-loop of the MPC proposed
method.

4-5 MPC parameters tuning

The main difficulty in MPC design lies in the number of parameters and settings that should
be tuned to fulfil the control task requirements. Some of these parameters are often found by
trial-and-error approaches some follows a general rule of thumb and others are merely based
on the MPC designer’s intuition.

4-5-1 Reducing the output variables

As it is described in the derived nonlinear model in (4-13), the output variables are the
readily observed marker’s features and the contact force measurements. The marker is the
Aruco marker, given in Figure 3-7, which has its detected features represented by its centre

Master of Science Thesis M. A. Ranneh

42 Modelling & Control Design

Figure 4-9: The nonlinear MPC scheme of for the task of automated docking.

and its four corners. During the design phase of MPC, it was noticed that choosing a set of
three non-collinear points of the target would be beneficial in reducing MPC computational
burden while demonstrating the same level of performance in case all target features are
considered. Thus, the MPC-model output is reduced where only, besides the contact force
measurements, the centre of the marker and its lower corners were considered.

4-5-2 Sampling time

Choosing a large time step Ts might not be enough to capture the system dynamics and having
a very small time step will increase computational power demands in the MPC scheme. To
select a proper time step value h. A general rule of thumb is to have 4 to 10 samples per rise
time of the step response of the discretised system [54].
The approximation of the low-level closed-loop dynamics of the gangway system Gss , as
described in (4-8) and (4-10), is the slow part of the derived model dynamics in (4-14).
Additionally, having a vision sensor in the control loop imposes an extra limitation of choosing
a small value for Ts . Camera latency for detecting and tracking the target (i.e. the marker)
can be as low as (20 to 30 ms). Therefore, the time step was chosen as Ts = 0.02 sec according
to Table 4-1.
Table 4-1: Selecting a proper time step

Sampling time Ts (sec) Number of samples per rise time (-)

0.02 6
0.025 4
0.03 3

M. A. Ranneh Master of Science Thesis

4-5 MPC parameters tuning 43

4-5-3 Prediction and control horizons

The prediction horizon Np is basically the length of the time window over which MPC predicts
the plant model response. The higher the value for Np the more computational power needed
to solve the MPC optimisation problem. Also, a high Np value leads to less aggressive system
response. Because when prediction horizon Np is large, MPC has more observations on
the system response hence the optimisation problem might find a better solution. However,
increasing Np to certain value no improvement in the overall performance was observed.
Correspondingly, low Np value can reduce the required computational effort however, it leads
the system to behave more aggressively.
Furthermore, high value for the control horizon Nc determines the length of the manipulated
variables vector ∆U along the prediction horizon, which implies Nc ≤ Np .
Increasing the control horizon Nc can result in less aggressive control input and more relaxed
with respect to the saturation limit. Also, Nc plays a crucial role in increasing the optimisation
problem complexity, thus, increasing the required computational effort drastically. On the
other hand, setting a low value to Nc can reduce the computational effort but might lead to
a more aggressive response.
Based on that, via a trial-and-error approach, Np and Nc values were tuned accordingly. More
details about the numerical experiments of this tuning process can be found in Chapter 5.

4-5-4 Input and output weights

The weighting matrix Q, described in (4-17), is a diagonal matrix where each item corresponds
to how much emphasis MPC should put on one particular output of the system. For instance,
to enable automated docking in a hierarchical fashion, the control task was categorised into
two main parts; tracking the target (i.e. the offshore platform) and docking into the target.
Hence, more emphasis is put on the centre of the marker because, according to the design
objectives, minimising the error in the image plane of between the desired field of view and
the detected marker’s features results in minimising the relative spatial velocity between the
gangway tip and the target.
Correspondingly, when the docking is enabled the contact between the Gangway tip and the
structure is initiated. Since the exerted force by the Gangway should not exceed certain lim-
its to avoid damaging the structure and maintaining the contract with the offshore platform.
The contact force associated weight is the most important one, see the last item in Q in (4-19)
Furthermore, the weighting matrix R is also a diagonal matrix that penalties the input in-
crement vector ∆u(k). Let us recall, that u(k) is the control input vector to the system
which represents the Gangway tip velocity in 3D space. High associated value to a particular
input will result in a ’sluggish’ response on the output channels that are associated with that
particular input.
However, setting the associated weight of this particular input to a low value will result in
a more sensitivity on the associated output channel which might potentially affect the ro-
bustness. For instance, while the tip is in contact with the platform it is important to keep
tracking the contact force while tracking the target motion (i.e. compensating the sea-induced
motion). This means if the target moves along the sway axis the telescoping joints should
track this motion while assuring that the exerted force at the tip still with permissible limits.
Moreover, the ratio between the output and input weights Q/R determines the aggressiveness

Master of Science Thesis M. A. Ranneh

44 Modelling & Control Design

and robustness of the controller performance (e.g. in disturbance rejection) [53].

Accordingly, Q and R values were found via a trial-and-error process based on judging the
corresponding performance of the contact force response (in terms of fluctuation).
 
2 0 0 0 0 0 0
0 0.5 0 0 0 0 0 
 
 
0 0 2 0 0 0 0  0.5 0 0
 
 
0 0 0 0.5 0 0 0  , R =  0 0.2 0  .
Q= (4-19)
  
0 0 0 0 5 0 0  0 0 0.5
 
0 0 0 0 0 5 0
 
 
0 0 0 0 0 0 20

Recall that, the output vector y(k) in the derived model in (4-15) is the corners, the centre of
the marker, and the contact force response, respectively. Where each marker feature has two
coordinates (i.e. location in the image plane). In Section 4-5-1, however, the output vector is
reduced to 7 channels where only the lower concerns, the centre of the maker, and the force
response were considered in the MPC-embedded model. Thus we see, in the last three items
in the principal diagonal of Q, that the weight values were selected to be highest since they
are associated the centre of the marker and the force, respectively.
Note that, the second weight in the R matrix is associated with second input to the system,
namely the Gangway tip velocity along y-axis. Which is translated into a velocity command
for the telescoping joint through the inverse Jacobian matrix, as described in (4-14).

4-6 Perturbations anticipation

As described in the derived model (4-13) and depicted accordingly in Figure 4-7, the sea-
induced motion that acts on the vessel is mapped out to the target reference frame and it is
incorporated within the nonlinear model where it can be regarded as external perturbations
that act on the system.
In Figure 4-9, the MPC block takes in these perturbations, denoted by ’induced motion’ block,
as a measured disturbance signal. The formulation of the MPC problem, described in (4-17)
lets the MPC scheme enjoys the anticipatory property in case the measured disturbance signal
is available for more than one-time instance k along the prediction horizon Np .
Although, in the simulation environment, it might be ’natural’ to feed the MPC block with
multi-step-ahead of the measured target motion (sea-induced motion) with the available data
set. This is not the case however in reality, where MRU is providing a single reading at each
time step.
In Chapter 2, it is stated that predicting the incoming waves is possible using wave spectra and
historical readings. However, when the vessel reaches the offshore platform the vessel’s bridge
turns on the Dynamic Positioning (DP) system which tries to maintain a fixed distance to
the offshore platform while counteracting the sea-induced motion. This motion compensation
system is not perfect and results in residual motion mainly along the horizontal plane. Thus,
the incoming induced motion is distorted because of the DP system and the radial forces
between the vessel and the offshore platform. Thus, forecasting the incoming waves using
wave-spectra techniques is pointless.

M. A. Ranneh Master of Science Thesis

4-6 Perturbations anticipation 45

Accordingly, in this thesis, a data driven approach for forecasting the induced motion is
designed. This multi-step-ahead forecasting scheme is based using historical data of the
MRU measurements.
This is achieved, first, by training a neural network with NARX topology (Nonlinear Auto-
regressive model with Exogenous inputs). Then, the trained NARX function is employed
in the control loop. Where it provides the MPC block with a prediction sequence of the
disturbance variable d(k) along the prediction horizon Np .
NARX is a multi-layer of feedforward neural network with an architecture of Recurrent Neural
Network (RNN). NARX is a popular technique for estimating the dynamics of a nonlinear
system (i.e. black-box identification) [55]. Where it employs tapped delays of the input
variable to resemble a memory of the system dynamics. For a multi-step-ahead prediction
of a time series, it is recommended to feed the previous output of the neural network as an
input channel [55]. NARX can deliver a good performance for multi-step-ahead prediction in
case of univariate forecasting problem. Where for multivariate problem poor performance is
reported [56].
The sea-induced motion affects the system mainly along the translational axes. Hence, three
variables need to be forecasted. To gain higher performance, three NARX neural networks
were trained separately to perform the prediction for each axis of motion. MPC utilises
these prediction to substitute d(k) values along its prediction horizon Np . Therefore, the
prediction horizon for the induced motion Nd should not exceed Np . The design of each
neural network along each axis of motion is identical. Where the input for the neural network
is the MRU velocity readings at time instance k. With each neural network is designed
preforms three-step-ahead prediction. Hence, the supplied disturbance sequence to the MPC
block is {d(k|k), · · · , d(k + 3|k)}, which corresponds to Nd = 4.
A data set of the MRU measurements were used to train these neural networks. The data
set were divided into three batches; 70% for training, 15% for cross-validation, and 15%
for testing. This is a common practice in training the neural network to avoid overfitting
regularisation [55]. The training was performed with MATLAB Deep Learning Toolbox.
The performance metric for training and validation is chosen as Root Mean Square Error
(RMSE) which is a gradient defined function (smooth function), RMSE is given in terms of
the prediction error as s
1 n  2
RM SE = Σ e (4-20)
n i=1
Each neural network preforms three-step-ahead prediction with three output channels (an
output channel for each prediction step). Figure 4-10 is a MATLAB-generated diagram of
the designed NARX neural network topology. Where the hidden layer utilises 30 neurons
with one input variable x(t) (the forecated variable) and the output feedback vector {y(t +
1), y(t + 2), y(t + 3)} (three-step-ahead prediction).
High performance (i.e. low RMSE value) is recorded, when the tapped delays of the forcaseted
input variable x(t) are set to; {x(t), x(t − 1), · · · , x(t − 10)}. For feedback coming from
the prediction output vector y(t) one tapped delay is chosen, which results in shifting the
prediction vector one step back {y(t), y(t + 1), y(t + 2)}. Where t refers to the time instance
of the MRU measurements.
Once the neural network showed a proper performance, the designed neural networks were
validated on another data set to avoid confirmation biased. Figure 4-11 shows the neural
network forecasting error of the sea-induced motion (velocity) along the surge, sway, and

Master of Science Thesis M. A. Ranneh

46 Modelling & Control Design

heave axes. With the corresponding RMSE value along each axis, respectively.
h i
RM SE = 0.0030318 0.010249 0.0066153

After applying this scheme, a noticeable enhancement in the performance in the contact force
task was observed. Chapter 5 discusses more in-depth the effect of the anticipatory action on
the performance.

Figure 4-10: The designed architecture of the employed NARX neural network.

The neural network forecasting error of the sea-induced motion (velocity)

Error along Surge (m/s)

0.04

0.02

-0.02
0 50 100 150 200 250 300
Time (s)
Error along Sway (m/s)

0.1

0.05

0 50 100 150 200 250 300

Time (s)
Error along Heave (m/s)

0.1

-0.1
0 50 100 150 200 250 300
Time (s)

Figure 4-11: The neural network forecasting error of the sea-induced motion (velocity).

M. A. Ranneh Master of Science Thesis

4-7 Closed-loop stability analysis 47

4-7 Closed-loop stability analysis

The cost function J(k) in the MPC block, described in (4-17) is a quadratic cost function
which has, by implication, a positive value. In [53, 57], M. Lazar et al, provide intuitive proof
for the endpoint convergence in the quadratic MPC closed-loop scheme. By assuming, a
Lyapunov cost function that is equivalent to the MPC’s cost function J(k).
We recall, if a function is positive definite and it is a monotonically decreasing function, then,
the cost function is considered to be a Lyapunov function candidate [58].
In Figure 4-12, one can see that the value of cost function, decreases asymptotically over the
experiment time. This may indicate that the cost function is a Lyapunov function candidate
since it is also positive definite by its nature. Consequently, this would have proved global
stability of the closed-loop system in case of the cost function is not a subject to constraints.
Thus, one cannot conclude the global stability but rather the endpoint convergence of the
closed-loop system.

Cost Function Value

18000

16000

14000

12000

10000

8000

6000

4000

2000 X 16.92
Y 0.9358
0

0 2 4 6 8 10 12 14 16 18
Time (seconds)

Figure 4-12: The cost function value.

4-8 Summary and discussion

In conclusion, this chapter answers the research questions regarding incorporating the vision
sensor in a high-level control scheme to enable the automated docking scheme.
After reviewing the current development and the classical approaches in the literature, a
’high-level’ control scheme was proposed. This control scheme is mainly based on the standard
nonlinear MPC as a control strategy.
The novelty in this work lies in deriving a mathematical model of the Gangway system that

Master of Science Thesis M. A. Ranneh

48 Modelling & Control Design

is suitable for the MPC scheme with fulfilling the control task requirements. Moreover, a
neural network with NARX architecture was designed and trained to perform multi-step-
ahead forecasting of the MRU readings. This can enable the anticipatory property within the
MPC scheme which enhances the contact force response.
Overall, it may be said that this method offers a singularity-free solution for a vision-based
control task (i.e. the automated docking problem) while handling system constraints (e.g. the
workspace) and the control objective constraints (i.e. contact force response). However, the
proposed control method is computationally expensive and further research is required, hence,
for now, this is regarded as a major drawback, especially, from the industry perspective.

M. A. Ranneh Master of Science Thesis

 Chapter 5

Results

This chapter evaluates the performance of the proposed MPC solution for automating the
docking procedure of Bosch’s Gangway system. The controller is first tested for the nom-
inal case with multiple scenarios (i.e. with and without disturbance anticipation, different
prediction and control horizons Np and Nc values). Then, tests are performed against plant
model mismatch. Lastly, the performance is evaluated for a more extreme case in terms of
the amplitude of the sea-induced motion.

5-1 Nominal Case Scenarios

This section discusses the performance of the employed nonlinear MPC scheme in Chapter 4.
These scenarios are evaluated with the nominal plant model (i.e. no model mismatch).

5-1-1 No disturbances case

First, the nominal case scenario evaluates the proposed control scheme performance in case
no disturbances are applied to the system. This means the amplitude of the sea-induced
motion is set to zero. In the real world scenario, this case does not exist. Nevertheless, this
trial is beneficial since it provides the baseline evaluation of the proposed controller in the
simulation environment.
This test was attained in two trials with different initial conditions (i.e. various relative poses
between the Gangway tip and the target). In this project, it is supposed that the initial
conditions of the Gangway are fixed to certain values. In practice, this holds since the vessel’s
bridge has a communication channel with the Gangway system. Hence, coordination between
the vessel’s Dynamic Positioning (DP) system and the Gangway’s joint-level control system is
possible. In the current system, this coordination happens between the vessel’s bridge and the
Gangway’s operator [7]. Accordingly, in order to enable the automated docking, this project
proposes direct coordination between the Gangway joint-level control loop and the vessel’s
bridge such that the target is within the camera field of view once the automated docking is

Master of Science Thesis M. A. Ranneh

50 Results

initiated.
Therefore, the initial value of the joint displacement vector qg for all the tests in this Chapter
is fixed to the value qg = [qslew , qluf f , qtele ] = [90◦ , 90◦ , 17m]. The case of no disturbances (i.e.

Target tracking error on image plane

Timelapse of the marker features motion in image plane 200
0

50 150

100
100

Tracking features error(pixel)

150
Vertical axis (pixel)

50
200

0
250
Right corner err
300 -50 x
Right corner err y
350 Left corner err
-100 x
Left corner err y
400 Desired field of view feature2 trajectory
Initial field of view feature3 trajectory Marker center err
-150 x
450 feature1 trajectory feature4 trajectory Marker center err y

0 100 200 300 400 500 600 -200

0 2 4 6 8 10 12 14 16 18 20
Horizontal axis (pixel)
Time (s)

(a) (b)

Target tracking error on image plane

Timelapse of the marker features motion in image plane
0 350 Right corner err x
50 300 Right corner err y
Left corner err x
100 250
Left corner err y
Tracking features error(pixel)

150 200 Marker center err x

Vertical axis (pixel)

150 Marker center err y

200
100
250
50
300
0
350
-50
400 Desired field of view feature2 trajectory
-100
Initial field of view feature3 trajectory
450 feature1 trajectory feature4 trajectory -150

0 100 200 300 400 500 600

0 2 4 6 8 10 12 14 16 18 20
Horizontal axis (pixel)
Time (s)

(c) (d)

Figure 5-1: Here, the target is stationary with a relative position, to the Gangway base, of
Ptarget = [0.5, 23, 1]T in case of (a) and (b), and Ptarget = [4, 25, 2]T in case of (c) and (d). (a)
and (c) illustrate time-lapse footage of the target (marker) detected features in the field of view
over 20 seconds. (b) and (d) show the corresponding tracking error of the target features in the
2D image plane.

no induced motion) means that the target is stationary with respect to the Gangway base.
The first set of initial conditions, denoted by x00 in Figure 5-2, is the target pose Ptarget , as
described in (4-14), in 3D space with respect to the Gangway base reference frame is set to
Ptarget = [0.5, 23, 1]T . Figure 5-1a, shows time-lapse footage of the target (marker) detected
features in the field of view. We recall that in the proposed control scheme the camera is
placed on the Gangway tip, see Figure 3-10. Consequently, we can see, in Figure 5-1a that
in the initial field of view, the target (red dashed-square) appears in a ’small’ size. This size
corresponds to the distance between the target and the camera optical axis, see Figure 3-3 in

M. A. Ranneh Master of Science Thesis

5-1 Nominal Case Scenarios 51

Contact force along Sway axis

8000

7000

6000

Contact force (N)

5000

4000

3000

2000

Contact force setpoint

1000 Contact force response for x00
Contact force response for x01
0
0 5 10 15 20 25 30 35 40 45 50
Time (s)

Figure 5-2: The corresponding contact force response of the first and the second initial conditions
set denoted by x00 and x01 , respectively.

Section 3-3.
In the MPC scheme, higher weight is put on the centre of the marker than the corners
associated weights, as described in Section 4-5. The MPC scheme considers a higher priority
to the task of tracking the marker centre. Hence, MPC aims to align the marker in the
centre of the desired field of view, by manipulating the rotational joints of the Gangway.
Once the marker centre reaches the desired location in the image plane, MPC manipulates
the telescoping joint which has the zoom-in effect on the field of view since the camera is
attached to the tip. Moreover, the desired field of view represents the constant setpoint
signal r(k) and it is chosen such that the 3D displacement error between the tip and the
target is zero (i.e. the case of docking).
The corresponding tracking error of the target features in the image plane is shown in Figure
5-1b.
The second set of initial conditions, denoted by x01 in Figure 5-2, the target pose Ptarget in
3D space with respect to the Gangway base reference frame is set to Ptarget = [2, 25, 2]T . In
this case, the initial target location is even more distant than in the first case. Hence, the
marker appears even smaller in the initial field of view, see Figure 5-1c. The corresponding
tracking error of the target features in the image plane is shown in Figure 5-1d.
Correspondingly, the contact force response for both the first and the second set of the initial
conditions is depicted in Figure 5-2. Notice that, in the second set, the target is more distant
to the Gangway tip. Therefore, the automated docking task takes more time than in the first
case. as shown in Table 2-2, to guarantee a connection between the Gangway tip and the
offshore platform, and to avoid damaging the structure, the contact force response should be
between (4000-8000 N). Therefore, the force setpoint is chosen in the middle of the allowed
range as (6000 N) to safeguard against response fluctuations.

Master of Science Thesis M. A. Ranneh

52 Results

5-1-2 Adding disturbances

After demonstrating the baseline performance of the proposed control, we need to test the
performance in the case of induced motion which better represents the real-world scenario.
As Section 4-3-4 states, it is expected that providing the MPC with the induced motion
measurements as a measured disturbance channel is essential to achieve the control task of
contact force regulation. This is demonstrated and confirmed indeed in this trial, see Figure
5-3.
Figure 5-3a illustrates time-lapse footage of the target (marker) detected features in the field of
view over 25 seconds. Where the target 3D displacement is shown in Figure 5-3d. The target
motion is caused by the sea-induced motion. However, this induced motion was not provided
to MPC. Therefore, the target poorly tracked to the desired field of view and control force
requirements is not fulfilled, as shown in Figure 5-1c. Notice that, in Figure 5-3b, during the
first five seconds the performance of the tracking control is similar to the case of a stationary
target. This is mainly due to the perspective projection effect along the optical axis. Where
the motion of a distant object from the camera results in a much smaller deviation in the
image plane.
Thus, based on the results of the previous test, the induced motion measurements are provided
to MPC as a measured disturbances channel d(k), as described in (4-17) Where every time
instant k, the MRU readings is provided to MPC. However, MPC predicts the system response
over the prediction horizon window. Thus, the provided value for d(k) is held constant in the
predicted model along the prediction horizon and only updated in the next time step with a
new MRU reading.
Figure 5-4a shows the contact force response when the sea-induced motion is present in the
system and provided accordingly to MPC via MRU readings at each time instance. The force
response is fluctuating, however, within the allowed range. Correspondingly, in Figure 5-4b
the tracking error in the image plane is shown. Hence, by comparing the system responses in
Figure 5-3 and 5-4, we can say that providing the MPC with the MRU readings is crucial to
obtain the control task requirements while tracking the target motion.

5-1-3 Disturbance anticipation

As stated in Chapter 4, enabling the disturbance anticipatory action in the MPC scheme
should enhance the overall preference. Accordingly, a neural network with NARX topology
was designed and trained for the purpose of anticipating ahead the induced motion.
Indeed as expected, by supplying MPC with the induced motion predictions, MPC force
response is improved. Figure 5-5 shows the contact force response in case of no disturbance
anticipation, disturbance anticipation with the induced motion data set provided by Bosch,
and the force response with the trained neural network. Clearly, in case of anticipating ahead
the induced motion, the fluctuation in the force response is attenuated compared to the case
of no anticipation. Moreover, the force response using the neural network is similar to the
force response using the data set. However, these two responses are not identical due to the
prediction error in the neural network as seen in Figure 4-11.

M. A. Ranneh Master of Science Thesis

5-1 Nominal Case Scenarios 53

Target tracking error on image plane

Timelapse of the marker features motion in image plane 200
0

50 150

100
100

Tracking features error(pixel)

150
Vertical axis (pixel)

50
200

0
250

Right corner err x

300 -50
Right corner err y
350 Left corner err x
-100
Left corner err y
400 Desired field of view feature2 trajectory
Initial field of view feature3 trajectory -150 Marker center err x
450 feature1 trajectory feature4 trajectory Marker center err y

0 100 200 300 400 500 600 -200

0 5 10 15 20 25
Horizontal axis (pixel)
Time (s)

(a) (b)

Target 3D displacement
1
Surge displacement (m)

Contact force along Sway axis

8000 0

7000 -1
0 5 10 15 20 25
Time (s)
6000 25
Sway displacement (m)
Contact force (N)

24
5000

23
4000
22
0 5 10 15 20 25
3000 Time (s)
Heave displacement (m)

2
2000

1
1000
Contact force setpoint
Contact force response
0
0 5 10 15 20 25
0 5 10 15 20 25 Time (s)
Time (s)

(c) (d)

Figure 5-3: This set of figures corresponds to the case where the induced motion is present
on the system, however, MPC has no access to these data. Correspondingly, MPC shows poor
performance in tracking the setpoints. Therefore, the target visual information coming for the
camera is not enough and MRU readings are needed. (a) and (b) illustrate time-lapse footage of
the target (marker) detected features in the field of view over 25 seconds and the corresponding
tracking error, respectively. The corresponding contact force response and the target pose is
depicted in (c) and (d), respectively.

Master of Science Thesis M. A. Ranneh

54 Results

Target tracking error on image plane

200
Contact force along Sway axis
9000
150
Force setpoint
8000 Contact force response
100

Tracking features error(pixel)

7000
50
6000
Contact force (N)

5000 0

4000 Right corner err

x
-50
Right corner err
y
3000 Left corner err
-100 x
Left corner err y
2000
Marker center err
-150 x
1000 Marker center err y

0 -200
0 5 10 15 20 25 30 35 40 45 50 0 5 10 15 20 25 30 35 40 45 50
Time (s) Time (s)

(a) (b)

Figure 5-4: This figures corresponds to the case where the induced motion is present on the
system and provided accordingly to MPC at each time step (instant feedback). The contact force
response for this case is shown in (a). The corresponding target’s features tracking error is shown
in (b).

Contact force along Sway axis

9000

8000

7000

6000
Contact force (N)

5000

4000

3000

2000 Force setpoint

Force response without disturbance anticipation
1000 Force response disturbance anticipation via neural net
Force response disturbance anticipation via dataset
0
0 5 10 15 20 25 30 35 40 45 50
Time (s)

Figure 5-5: The contact force response in case of; no disturbance anticipation (nominal case),
disturbance anticipation with the induced motion data set, and disturbance anticipation with the
trained neural network.

M. A. Ranneh Master of Science Thesis

5-2 Selecting prediction & control horizons values 55

5-2 Selecting prediction & control horizons values

One of the challenging points in implementing an MPC scheme is the degrees of freedom
MPC has in terms of the tuning knobs (e.g. prediction and control horizons, input and
output weights).
In this project, the selection of the MPC parameters is based on an iterative process of trial-
and-error and general guidelines for MPC tuning in [54, 59].
Thus, several trials were performed to evaluate the proposed MPC scheme against different
prediction and control horizons Np and Nc , respectively. The results of these trials indicate
the impact of increasing/decreasing Np , Nc values on the overall performance.
Choosing values for these parameters affects not only the controller performance but also the
computational complexity of the MPC algorithm which solves an online optimisation problem
at each time step.
In the work of Garriga and Soroush [54], a literature review is presented over the techniques
and the general guidelines for tuning MPC parameters. A general rule of thumb is to set the
prediction horizon as (Tset ≤ Np Ts ). Where Ts is the chosen sampling time, and Tset is the
settling time of the open-loop step response. Recall that, the low-level plant dynamics G(s),
described in (4-8), is the slow dominant part of the Gangway system dynamics. The settling
time of a second order system can be given as, [60],
4
Tset = = 0.065
ζωn
Thus, in this set of trials the initial value for
Tset
= 5 ≤ Np
Ts
The control horizon is the set of future admissible control actions leading to this predicted
plant output. Usually, the control horizon is set as Nc < Np . Where the rest of the inputs
are held constant along the prediction horizon.
For this set of trials, MPC shows nearly identical performance in tracking the target in the
image plane while differing in the calculation time. However, the impact of choosing different
values for Np , Nc can be significantly observed in the contact force response.
Figure 5-6, depicts the contact force response for different values of Np , Nc . The corresponding
target pose for this set of trials is depicted in Figure 5-7. Figure 5-6a, evaluates the contact
force response for different Np values and a constant Nc value. Note, the scale of the vertical
axis is enlarged compared to the one depicted in Figure 5-6b. When choosing a proper value
Nc (i.e. Nc = 3) the performance gain of the different Np values is insignificant. Whereas,
Nc value has higher impact on the force response, see Figure 5-6b. Setting a small value
for Nc (i.e. Nc = 1) leads MPC to perform worse against incoming disturbances. Thus, we
see a fluctuation in the force response when Nc = 1, 2. However, the impact on the force
response of choosing any value of Nc > 3 is insignificant and only increases the optimisation
problem complexity. In this set of trials, the disturbance anticipatory property in the MPC
was enabled using the provided data set.

Master of Science Thesis M. A. Ranneh

56 Results

Contact force along Sway axis Contact force along Sway axis

6100
9000

6050 8000

7000
6000
Contact force (N)

Contact force (N)

6000
5950
5000

5900 Contact force setpoint 4000

Contact force response N =14, N =3
p c Contact force setpoint
3000
Contact force response N p =12, Nc=3 Contact force response N p =10, Nc=1
5850
Contact force response N =10, N =3 2000 Contact force response N =10, N =2
p c p c
Contact force response N p =8, N c=3 Contact force response N p =10, Nc=3
5800
1000
Contact force response N =6, N =3 Contact force response N =10, N =4
p c p c

0
0 5 10 15 20 25 30 35 40 45 50 5 10 15 20 25 30 35 40 45 50
Time (s) Time (s)

(a) (b)

Figure 5-6: The contact force response for different Np values (a), and different Nc values (b).
Note, the scale of the vertical axis is enlarged in (a) compared to the one depicted in (b).

Target 3D displacement
Surge displacement (m)

3.5

2.5

1.5

0 5 10 15 20 25 30 35 40 45 50
Time (s)
Sway displacement (m)

26

25

24

23
0 5 10 15 20 25 30 35 40 45 50
Time (s)
Heave displacement (m)

0 5 10 15 20 25 30 35 40 45 50
Time (s)

Figure 5-7: The corresponding target pose.

5-3 Model mismatch tests

The previous test scenarios were performed to evaluate the proposed control method in terms
of fulfilling the control task objectives. Likewise, in this section, the robustness of the proposed
MPC scheme is evaluated through a set of trials that involves a model mismatch between the
MPC embedded model and the plant model. In this set of trials, the disturbance anticipatory
property in the MPC was enabled using the provided data set. The prediction horizon was
set to Np = 8, and different control horizon values were test Nc = 2, 3.

M. A. Ranneh Master of Science Thesis

5-3 Model mismatch tests 57

5-3-1 Output parameters mismatch

The measurement system of the proposed control scheme employs a monocular camera and
a force sensor at the tip. The output of these sensors might vary compared to the MPC
embedded model. This variation is caused by operating conditions such as vibrations, fatigue,
and sensor misalignment.

• Bumper stiffness mismatch test

The rubber bumper at the tip has a linear stiffness coefficient estimated by Bosch around
the operating window. However, fatigue and other operating conditions may cause a
variation in the stiffness coefficient Kb , introduced in (4-10), of the rubber bumper
which leads to a variation in the contact force response.
Accordingly, a set of trials were run to evaluate the impact of the stiffness coefficient
variation on the contact force response. As part of the iterative tuning processes, these
trials were performed for different control horizon Nc values. Figure 5-8 depicts the
corresponding contact force response when the stiffness coefficient Kb has a mismatch
of ±%5 from the nominal model. This trial was performed for Nc = 2, 3, as part of the
iterative tuning process of MPC, and beyond these values for Nc no gain in performance
was observed.
Clearly, the impact of the stiffness mismatch on the force responses is linear and pro-
portional to the mismatch percentage. When Nc = 3 MPC yields better disturbance
rejection. Where the fluctuation in the force response, caused by the target motion, is
attenuated.

• Camera misalignment test

As Chapter 3 concludes, the camera is attached nearly at the tip of the Gangway. Often,
in vision-based robotic manipulation tasks, a precise calibration between the camera and
the robot end-effector is needed to eliminate the error in the target pose estimation.
However, in the proposed control scheme the target is detected and tracked within the
2D image plane. Thus, the relative pose estimation problem is solved implicitly by
matching the target image with the desired field of view. This concept is enabled in
the scheme of Image-Based Visual Seroving (IBVS), as stated in Section 4-1. As a
consequence, IBVS schemes enjoy robustness properties against the camera calibration
error [8, 47]. The proposed control scheme also tracks the target in the 2D image plane.
Hence, correspondingly, the proposed scheme is expected to be pron against camera
calibration error. This is demonstrated and confirmed indeed in this trial, see Figure
5-9.
In this trial, the camera position is misaligned by ±10 cm on along the principal axes.
Figure 5-9 shows a time-lapse over 50 seconds of the target in the image plane and
the corresponding features tracking error in 5-9a and 5-9b, respectively. Notice, even
a small misalignment (i.e. ±10 cm) in placing the camera at the tip can lead to big
shift for the final marker location in the field of view. This is caused by the perspective
projection in the employed vision sensor.
On the other hand, in this trial, the contact force response followed the force setpoint
without any offset, see Figure 5-9c. This is mainly due to the way the output weights
matrix Q was constructed in MPC. As stated in Section 4-5, a higher weight value
is chosen to put more emphasis on the force output channel since it is more vital for

Master of Science Thesis M. A. Ranneh

58 Results

maintaining the contact with the target. Moreover, the corresponding error along the
principal axes between the Gangway tip and the target is depicted in Figure 5-9d.

Contact force along Sway axis

9000

8000

7000
Contact force (N)

6000

5000

4000

Contact force setpoint

3000
Contact force response N p =8, N c=2, +5% stiffness
2000 Contact force response N p =8, N c=2, -5% stiffness
Contact force response N p =8, N c=3, +5% stiffness
1000
Contact force response N p =8, N c=3, -5% stiffness

0
5 10 15 20 25 30 35 40 45 50
Time (s)

Figure 5-8: The corresponding contact force response for different Nc values in case of stiffness
coefficient Kb mismatch.

5-3-2 Plant dynamics mismatch

In this thesis, the derived model parameters of the Gangway system was provided by Bosch
according to their design (e.g. the Gangway geometrical dimensions). Bosch also provided an
approximation of the closed-loop dynamics of each controlled joint in the Gangway system.
However, this approximation is based on their intuition of their system performance (the
numerical experiments may be carried in future work). Hence, testing the proposed scheme
with this approximation mismatch is an interesting case scenario. The approximation of the
low-level closed-loop equivalent system is a second-order transfer function with a bandwidth
of ωn = 6 Hz, as described in (4-9). An interesting test scenario is to evaluate the performance
of the proposed MPC in case of plant dynamics mismatch.
Figure 5-10 depicts the contact force response for different dynamics mismatch cases.
As stated in Section 4-3-4, MPC can incorporate in its scheme the different constraints the
system is subject to. However, setting all the system constraints are set as hard constraints
may lead the optimisation solver to an unfeasible solution. Accordingly, the system input
associated constraints are set as hard constraints (i.e. to avoid introducing instability) and
the output constraint are relaxed into soft constraints by introducing a slack variable  ≥ 0.
Where introducing soft constraints is a common practice in MPC design to avoid unfeasible
solutions [53, 54].
Notice that, in Figure 5-10, the force constraints are violated in the case of the plant dynamics

M. A. Ranneh Master of Science Thesis

5-4 Extreme disturbances 59

Target tracking error on image plane

Timelapse of the marker features motion in image plane 200
0

50 150

100

Tracking features error(pixel)

100
150
Vertical axis (pixel)

50
200

0
250

300 -50

350
-100
Desired field of view feature2 trajectory Right corner err Left corner err
400 x y
Initial field of view feature3 trajectory Right corner erry Marker center err x
-150
Final field of view feature4 trajectory
450 Left corner err Marker center err
feature1 trajectory x y

0 100 200 300 400 500 600 -200

0 5 10 15 20 25 30
Horizontal axis (pixel)
Time (s)

(a) (b)

Gangway tip dispalcement error w.r.t target

2
Tip x-error (m)

Contact force along Sway axis

0
0 5 10 15 20 25 30 35 40 45 50
9000 Time (s)

8000 8
Tip y-error (m)

6
7000 4
Contact force (N)

2
6000
0
5000
0 5 10 15 20 25 30 35 40 45
Time (s)
4000
Tip z-error (m)

3000
1

2000 Contact force setpoint 0.5

Contact force response N p =8, N c=2, camera misalignment
1000
Contact force response N p =8, N c=3, camera misalignment 0
0 5 10 15 20 25 30 35 40 45 50
0
5 10 15 20 25 30 35 40 45 50 Time (s)

Time (s)

(c) (d)

Figure 5-9: (a) and (b) illustrate time-lapse footage of the target (marker) detected features
in the field of view over 50 seconds and the corresponding tracking error, respectively. The
corresponding contact force response for different Nc values and the error between the tip and
the target is depicted in (c) and (d), respectively.

mismatch. However, the performance in the case of ωn = 6.5 is acceptable since it is nearly
within the contact force boundaries (i.e. 4000 to 8000 N).

5-4 Extreme disturbances

Essentially, the target motion is solely the sea-induced motion mapped into the target relative
position to the Gangway. The sea-induced motion is a sinusoidal-based motion that varies
in the period and amplitude. Thus, in order to guarantee the performance consistency of

Master of Science Thesis M. A. Ranneh

60 Results

Contact force along Sway axis

10000

8000
Contact force (N)

6000

4000

Contact force setpoint

Contact force response for w n = 5.5 Hz
2000 Contact force response for w n = 6 Hz (nominal model)
Contact force response for w n = 6.5 Hz
Contact force response for w n = 7 Hz
0
5 10 15 20 25 30 35 40 45 50
Time (s)

Figure 5-10: The contact force response in case of plant dynamics mismatch.

the proposed control scheme in tracking the target motion, this set of trials is performed. In
this set of trials, the disturbance anticipatory property in the MPC was enabled using the
provided data set. The prediction horizon was set to Np = 8, and different control horizon
values were test Nc = 3.
This test is performed for different sea-induced motion conditions, the target pose depicted
in Figure 5-7 is considered the nominal case scenario. The induced motion act as a velocity
source for the target motion. In Figure 5-11, the second and the third trials employ the
same data set of the induced motion amplified by a factor of %150 and %200, receptively.
Additionally, the performance is evaluated for a different batch of the induced motion data
set.
Figure 5-12, depicts the corresponding contact force response of the aforementioned scenarios.
The force response for the nominal case and the fourth case (the different data set case) is
nearly identical. However, for the third case, the upper bound (8000 N) for the force response
in MPC is violated. Note, this is an extreme case scenario and to ensure the feasibility of
the optimisation problem solution the upper bound is this trial was constructed as a soft
constraint which allows a near-boundary solution.

5-5 Summary

The performance of the proposed MPC solution for automating the docking procedure is
evaluated in this chapter. The controller is first tested for the nominal case with multiple sce-
narios. Then, tests are preformed against the plant model mismatch. Lastly, the performance
is evaluated for a more extreme case in terms of the amplitude of the sea-induced motion.

M. A. Ranneh Master of Science Thesis

5-5 Summary 61

Surge displacement (m)

Target 3D displacement

4
3
2
1

0 5 10 15 20 25 30 35 40 45 50
Time (s)
Sway displacement (m)

28

26

24

0 5 10 15 20 25 30 35 40 45 50
Time (s)
Heave displacement (m)

2
0 5 10 15 20 25 30 35 40 45 50
Time (s)
nominal case induced motion amplified by %150 induced motion amplified by %200 different dataset

Figure 5-11: The target pose for different induced motion cases.

Contact force along Sway axis

9000

8000

7000

6000
Contact force (N)

5000

4000

3000
Contact force setpoint
2000 Force response - nominal case
Force respons - induced motion amplified by %150
1000 Force respons - induced motion amplified by %200
Force response - different dataset
0
0 5 10 15 20 25 30 35 40 45 50
Time (s)

Figure 5-12: The contact force response for different induced motion cases.

Master of Science Thesis M. A. Ranneh

62 Results

In Section 2-6, the control task requirements for the automated docking are defined. Based on
the empirical analysis in this chapter, all the control requirements are fulfilled for the nominal
case scenarios and in case of output model mismatch.

M. A. Ranneh Master of Science Thesis

 Chapter 6

Conclusions and Recommendations

The research objective of this thesis is to develop an automated docking scheme for the gang-
way system of Bosch. The main purpose is eliminating the operator from the control loop to
enhance the safety and efficiency of the system.
According to Bosch, in the current system, the operator relies on his visual perception and
vestibular senses to steer the Gangway actuators while the motion of the tip is automati-
cally compensated. Inspired by how the operator estimates the relative motion between the
Gangway and the target (i.e. the offshore platform). In Chapter 3, a measurement system
is proposed to measure this relative motion. This measurement system comprises a vision
sensor, force tip measurements, and Motion Reference Unit (MRU).
The automated docking scheme in this thesis is developed around a nonlinear MPC scheme.
The nonlinear MPC employs a nonlinear mathematical model of the Gangway system with
the proposed measurement system. This mathematical model is derived for the purpose of
simulation and control design. This model embeds an approximation, provided by Bosch, of
the joint-level control loop of the Gangway system. Additionally, this model comprises the
open-chain kinematic model of the Gangway system and a perspective projection model of
the vision sensor.
In this thesis, the proposed control scheme detects and tracks the target in the 2D image
plane. Therefore, the control scheme is prone to camera calibrations error. Contrarily to
Image-Based Visual Servoing (IBVS) approaches, this proposed scheme enjoys a singularity-
free solution in terms of the singularities that are based on image mapping and workspace
limitations. Thanks to the employed pinhole camera model and the MPC capability in han-
dling various constraints. Additionally, in Chapter 3, a linear Kalman filter is designed to
predict the target position in the image plane in case of a sudden invisibility of the target
occurs (i.e. cluttering, outside the field of view). This can safeguard against visual measure-
ments discontinuity which might be problematic from a control perspective.
To gain higher performance, the disturbance anticipatory property in MPC is enabled by
forecasting the sea-induced motion. Where a neural network with the NARX topology was
designed and trained to acquire a multi-step-ahead prediction model of the induced motion.
In terms of the efficiency, the proposed automated docking scheme performs the docking in 4
to 10 seconds (based on the initial conditions and sea state). Whereas the time it takes the

Master of Science Thesis M. A. Ranneh

64 Conclusions and Recommendations

operator to perform the docking is up to 3 minutes which depends on his/her experience [7].
In Section 2-6, the control task requirements for the automated docking are defined. Based
on the empirical analysis in Chapter 5, all the control requirements for the nominal case
scenarios are fulfilled. Also, more extreme scenarios are performed to evaluates the overall
performance under plant model mismatch and against various sea-induced motion conditions.
Overall, in this thesis, the research questions defined in section 1-3, are addressed and an-
swered correspondingly.
Lastly, the work of this thesis is considered as a proof of concept for employing a visions
sensor and an MPC scheme to solve the problem of automated docking.

6-1 Future work

The proposed automated docking scheme shows good performance and fulfils the control
task requirements. However, there is still a room for improvements and further research.
In order to implement the proposed scheme in practice numerous aspects should be further
investigated:

• It was assumed that, for the sake of simplicity; no deformations occur in both the
vertical and torsional axes of the Gangway. However, according to Bosch, deformations
might occur with an observed effect at the tip. One scheme can be proposed is a
multi-camera scheme. Where one camera is attached at the tip and another camera
is attached on top of the Gangway base with both cameras observe the same target.
Similar to the proposed control scheme, the target can be tracked in both camera’s
field of view, to counteract for tip deformations. This scheme refers to in literature as
eye-in-hand/eye-to-hand visual servoing, as presented in [61].

• In Section 1-4, the relative motion hypothesis is made. Where it is assumed that the
sea-induced motion that acts on the vessel causes the target (i.e. offshore platform) to
move instead. This assumption is made so the Gangway kinematics and dynamics are
described in an inertial frame. This hypothesis is yet to be tested by implanting the
proposed control scheme on the real system or on a scaled version.

• The employed nonlinear MPC is computationally demanding. While the real-time im-
plementation requirement is out of scope in this thesis. The calculation time for each
time step of the control loop takes between 0.05 to 0.3 seconds. This time depends
on the optimisation problem complexity (i.e. length of Np and Nc , sampling time Ts ).
With a sampling time of Ts = 0.02 seconds, the proposed MPC scheme becomes unfea-
sible to be implemented on the real system. One solution may be proposed is the fast
nonlinear MPC scheme in the work of Schindele and Ascheman in [62].

M. A. Ranneh Master of Science Thesis

 Bibliography

[1] IRENA, “Innovation outlook: Offshore wind,” tech. rep., International Renewable En-
ergy Agency, Abu Dhabi, 2016.

[2] “Lewek Falcon,” Jun 2019. [Online; accessed 21. Jun. 2019].

[3] “Welcome to Windcat Workboats,” Jun 2019. [Online; accessed 21. Jun. 2019].

[4] “Home | German Offshore Wind Energy Foundation,” Jun 2019. [Online; accessed 21.
Jun. 2019].

[5] B. Walgaard, “Hybrid force-position control of the motion compensated gangway,” mas-
ter thesis, Delft University of Technology, Delft, 2017.

[6] Rexroth, “Motion compensated gangway - system description,” tech. rep., Bosch
Rexroth, Boxtel, 2017.

[7] Rexroth, “Mcg - basis of design and control,” tech. rep., Bosch Rexroth, Boxtel, 2017.

[8] P. Corke, Robotics, Vision and Control: Fundamental Algorithms In MATLAB® Second,
Completely Revised, vol. 118. Springer, 2017.

[9] R. Szeliski, Computer Vision: Algorithms and Applications. Springer Science & Business
Media, 2010.

[10] G. J. Garcia, J. A. Corrales, J. Pomares, and F. Torres, “Survey of visual and force/tactile
control of robots for physical interaction in spain,” Sensors, vol. 9, no. 12, pp. 9689–9733,
2009.

[11] G. Allibert, E. Courtial, and F. Chaumette, “Predictive control for constrained image-
based visual servoing,” IEEE Transactions on Robotics, vol. 26, no. 5, pp. 933–939, 2010.

[12] K. M. Lynch and F. C. Park, Modern Robotics. Cambridge University Press, 2017.

Master of Science Thesis M. A. Ranneh

66 Bibliography

[13] W. Bierbooms and G. van Bussel, “The impact of different means of transport on the
operation and maintenance strategy for offshore wind farms,” Section Wind Energy,
Faculty Civil Engineering and Geosciences; Delft University of Technology: Delft, The
Netherlands, 2002.

[14] “Story - ampelmann - motion compensated gangways.” https://www.ampelmann.nl/

story, note=Accessed: 30-05-2019.

[15] D. GL, “Offshore gangways dnvgl-st-0358,” tech. rep., DNV GL, 2017.

[16] M. A. Ranneh, “Automating the docking procedure of an offshore gangway, a literature

survey,” Master’s thesis, Delft University of Technology, Delft, 2017.

[17] D. Newl, “Pedestrian excitation of bridges,” Proceedings of the Institution of Mechanical

Engineers, Part C: Journal of Mechanical Engineering Science, vol. 218, no. 5, pp. 477–
492, 2004.

[18] J. Journée, W. Massie, and R. Huijsmans, Offshore Hydromechanics: Course OE4630.

Delft University of Technology, 2000.

[19] L. H. Holthuijsen, Waves in oceanic and coastal waters. Cambridge university press,
2010.

[20] C. S. D. Julio, Ampelmann: development of the access system for offshore wind turbines.
BOXPress, 2010.

[21] Ø. F. Auestad, J. T. Gravdahl, and T. I. Fossen, “Heave motion estimation on a craft

using a strapdown inertial measurement unit,” IFAC Proceedings Volumes, vol. 46, no. 33,
pp. 298–303, 2013.

[22] “Kongsberg Maritime,” Jun 2019. [Online; accessed 21. Jun. 2019].

[23] R. Stettner, “Asc workshop presentation,” (Aspen, Colorado), NASA, 2008.

[24] J. Gladwin, R. Umamahaeswaran, K. Sivan, and P. Mathias, “Performance evaluation

of a vision sensor in 3d virtual environment for rendezvous and docking application,” in
Sensing Technology (ICST), 2012 Sixth International Conference on, pp. 201–206, IEEE,
2012.

[25] S. Kim and A. Billard, “Estimating the non-linear dynamics of free-flying objects,”
Robotics and Autonomous Systems, vol. 60, pp. 1108–1122, 2012.

[26] S. Kim, A. Shukla, and A. Billard, “Catching objects in flight,” IEEE Transactions on
Robotics, vol. 30, no. EPFL-ARTICLE-198748, 2014.

[27] J. Robert, Autonomous capture of a free-floating object using a predictive approach. PhD
thesis, McGill University, 2008.

[28] S. R. Bista, Indoor navigation of mobile robots based on visual memory and image-based
visual servoing. PhD thesis, Universite de Rennes 1; Inria Rennes Bretagne Atlantique,
2016.

M. A. Ranneh Master of Science Thesis

 67

[29] J. Heikkila, O. Silven, et al., “A four-step camera calibration procedure with implicit
image correction,” in cvpr, vol. 97, p. 1106, Citeseer, 1997.

[30] Z. Zhang, “A flexible new technique for camera calibration,” IEEE Transactions on
pattern analysis and machine intelligence, vol. 22, 2000.

[31] S. S. Tørdal and G. Hovland, “Relative vessel motion tracking using sensor fusion, aruco
markers, and mru sensors,” 2017.

[32] S. Garrido-Jurado, R. Muñoz-Salinas, F. J. Madrid-Cuevas, and M. J. Marín-Jiménez,

“Automatic generation and detection of highly reliable fiducial markers under occlusion,”
Pattern Recognition, vol. 47, no. 6, pp. 2280–2292, 2014.

[33] M. Faessler, E. Mueggler, K. Schwabe, and D. Scaramuzza, “A monocular pose estima-

tion system based on infrared leds,” in Robotics and Automation (ICRA), 2014 IEEE
International Conference on, pp. 907–913, IEEE, 2014.

[34] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-
time object detection,” in Proceedings of the IEEE conference on computer vision and
pattern recognition, pp. 779–788, 2016.

[35] D. Gordon, A. Farhadi, and D. Fox, “Re3: Real-time recurrent regression networks for
visual tracking of generic objects,” IEEE Robotics and Automation Letters, vol. 3, no. 2,
pp. 788–795, 2018.

[36] H. C. Kam, Y. K. Yu, and K. H. Wong, “An improvement on aruco marker for pose
tracking using kalman filter,” in 2018 19th IEEE/ACIS International Conference on
Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Com-
puting (SNPD), pp. 65–69, IEEE, 2018.

[37] K. Halvorsen and F. Olsson, “Pose estimation of cyclic movement using inertial sensor
data,” in 2016 IEEE Statistical Signal Processing Workshop (SSP), pp. 1–5, IEEE, 2016.

[38] T. Kautz, B. H. Groh, and B. M. Eskofier, “Augmented motion models for constrained
position tracking with kalman filters,” in 2016 19th International Conference on Infor-
mation Fusion (FUSION), pp. 849–854, IEEE, 2016.

[39] X. R. Li and V. P. Jilkov, “Survey of maneuvering target tracking. part i. dynamic

models,” IEEE Transactions on aerospace and electronic systems, vol. 39, no. 4, pp. 1333–
1364, 2003.

[40] H. H. Helgesen, F. S. Leira, T. I. Fossen, and T. A. Johansen, “Tracking of ocean surface

objects from unmanned aerial vehicles with a pan/tilt unit using a thermal camera,”
Journal of Intelligent & Robotic Systems, vol. 91, no. 3-4, pp. 775–793, 2018.

[41] M. Verhaegen and V. Verdult, Filtering and system identification: a least squares ap-
proach. Cambridge university press, 2007.

[42] S. Sanden, W. Pawlus, G. Hovland, et al., “Real-time 6-dof vessel-to-vessel motion com-
pensation using laser tracker,” in OCEANS 2017-Aberdeen, pp. 1–9, IEEE, 2017.

Master of Science Thesis M. A. Ranneh

68 Bibliography

[43] C. Liu, J. Xu, J. Zhao, H. Chen, N. Xi, and K. Chen, “Non-vector space visual servoing
for multiple pin-in-hole assembly by robot,” in Advanced Robotics and its Social Impacts
(ARSO), 2016 IEEE Workshop on, pp. 134–140, IEEE, 2016.

[44] T. Dallej, N. Andreff, Y. Mezouar, and P. Martinet, “3d pose visual servoing relieves
parallel robot control from joint sensing.,” in IROS, pp. 4291–4296, 2006.

[45] À. Santamaria-Navarro and J. Andrade-Cetto, “Uncalibrated image-based visual servo-

ing,” Institute of Electrical and Electronics Engineers, 2013.

[46] P. I. Corke, “Visual control of robot manipulators-a review,” in Visual Servoing: Real-
Time Control of Robot Manipulators Based on Visual Sensory Feedback, pp. 1–31, World
Scientific, 1993.

[47] F. Chaumette and S. Hutchinson, “Visual servo control. i. basic approaches,” IEEE
Robotics & Automation Magazine, vol. 13, no. 4, pp. 82–90, 2006.

[48] U. Klank, D. Pangercic, R. B. Rusu, and M. Beetz, “Real-time cad model matching for
mobile manipulation and grasping,” in Humanoid Robots, 2009. Humanoids 2009. 9th
IEEE-RAS International Conference on, pp. 290–296, IEEE, 2009.

[49] M. Ulrich, C. Wiedemann, and C. Steger, “Cad-based recognition of 3d objects in monoc-

ular images.,” in ICRA, vol. 9, pp. 1191–1198, 2009.

[50] F. Chaumette and S. Hutchinson, “Visual servo control. ii. advanced approaches [tuto-
rial],” IEEE Robotics & Automation Magazine, vol. 14, no. 1, pp. 109–118, 2007.

[51] F. Chaumette and S. Hutchinson, “Visual servoing and visual tracking,” in Handbook of
Robotics (B. Siciliano and O. Khatib, eds.), pp. 563–583, Springer, 2008.

[52] G. Allibert, E. Courtial, and F. Chaumette, “Visual servoing via nonlinear predictive
control,” in Visual Servoing via Advanced Numerical Methods, pp. 375–393, Springer,
2010.

[53] F. Borrelli, A. Bemporad, and M. Morari, “Predictive control,” Online at

http://www.mpc.berkeley.edu/mpc-course-material, 2012.

[54] J. L. Garriga and M. Soroush, “Model predictive control tuning methods: A review,”
Industrial & Engineering Chemistry Research, vol. 49, no. 8, pp. 3505–3515, 2010.

[55] J. M. P. Menezes Jr and G. A. Barreto, “Long-term time series prediction with the narx
network: An empirical evaluation,” Neurocomputing, vol. 71, no. 16-18, pp. 3335–3343,
2008.

[56] T.-N. Lin, C. L. Giles, B. G. Horne, and S.-Y. Kung, “A delay damage model selection
algorithm for narx neural networks,” IEEE Transactions on Signal Processing, vol. 45,
no. 11, pp. 2719–2730, 1997.

[57] M. Lazar, W. Heemels, S. Weiland, and A. Bemporad, “Stabilizing model predictive

control of hybrid systems,” IEEE Transactions on Automatic Control, vol. 51, no. 11,
pp. 1813–1818, 2006.

M. A. Ranneh Master of Science Thesis

 69

[58] H. K. Khalil and J. W. Grizzle, Nonlinear systems, vol. 3. Prentice hall Upper Saddle
River, NJ, 2002.

[59] A. Bemporad, M. Morari, and N. Ricker, “The mpc simulink library,” Automatic Control
Laboratory Physikstrasse, vol. 3, p. 8092, 2000.

[60] K. Ogata and Y. Yang, Modern control engineering, vol. 4. Prentice-Hall, 2002.

[61] G. Flandin, F. Chaumette, and E. Marchand, “Eye-in-hand/eye-to-hand cooperation for

visual servoing,” in Proceedings 2000 ICRA. Millennium Conference. IEEE International
Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065),
vol. 3, pp. 2741–2746, IEEE, 2000.

[62] D. Schindele and H. Aschemann, “Fast nonlinear mpc for an overhead travelling crane,”
IFAC proceedings volumes, vol. 44, no. 1, pp. 7963–7968, 2011.

Master of Science Thesis M. A. Ranneh

70 Bibliography

M. A. Ranneh Master of Science Thesis

 Glossary

List of Acronyms

Gangway Bosch’s gangway

DoF Degree of Freedom

MCG Motion Compensated Gangway

PLC Programmable Logic Controller

HNC Hydraulics-capable Numerical Control

RAO Response Amplitude Operators

DP Dynamic Positioning

MPC Model Predictive Control

MRU Motion Reference Unite

IMU Inertial Measurement Unit

PBVS Pose-Based Visual Seroving

IBVS Image-Based Visual Seroving

VPC Visual Predictive Control

IMC Internal Model Control

LQR Linear Quadratic Regulator

RNN Recurrent Neural Network

RMSE Root Mean Square Error

Master of Science Thesis M. A. Ranneh

72 Glossary

M. A. Ranneh Master of Science Thesis