# Vehicle Teleoperation: Successive Reference-Pose Tracking to Improve Path
Tracking and to Reduce Time-Delay Induced Instability

Jai Prakash, Michele Vignati, Edoardo Sabbioni, and Federico Cheli The authors
belong to the Department of Mechanical Engineering, Politecnico Di Milano,
Italy (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Vehicle teleoperation is an interesting feature in many fields. A typical
problem of teleoperation is communication time delay which, together with
actuator saturation and environmental disturbance, can cause a vehicle
deviation from the target trajectory imposed by the human operator who imposes
to the vehicle a steering wheel angle reference and a speed/acceleration
reference.

With predictive techniques, time-delay can be accounted at sufficient extent.
But, in presence of disturbances, due to the absence of instantaneous haptic
and visual feedback, human-operator steering command transmitted to the the
vehicle is unaccounted with disturbances observed by the vehicle. To improve
reference tracking without losing promptness in driving control, reference
trajectory in the form of successive reference poses can be transmitted
instead of steering commands to the vehicle.

We introduce this new concept, namely, the “successive reference-pose tracking
(SRPT)” to improve path tracking in vehicle teleoperation. This paper
discusses feasibility and advantages of this new method, compare to the smith
predictor control approach.

Simulations are performed in SIMULINK environment, where a 14-dof vehicle
model is being controlled with Smith and SRPT controllers in presence of
variable network delay. Scenarios for performance comparison are low adhesion
ground, strong lateral wind and steer-rate demanding maneuvers. Simulation
result shows significant improvement in reference tracking with SRPT approach.

###### Index Terms:

Latency, time-delay, vehicle teleoperation, smith predictor, NMPC, successive
reference-pose tracking.

##  I Introduction

Vehicle teleoperation refers to driving a vehicle by transmitting driving
commands to the vehicle’s control system from a control station that is
stationary and, in general, far away from the vehicle. A human operator who
remains in the control station generates the driving commands. The data
communication protocol, interestingly, can be wired or wireless. Wireless data
communication protocol is required due to the potential applications of
vehicle teleoperation, which include last-mile delivery of rental/shared
vehicles, avoiding driver presence in dangerous areas, human remote assistance
in the event of autonomous vehicle failure, and valet parking, among others.
Due to its widespread availability around the world, 4G LTE wireless broadband
connection is the greatest candidate for communication. Fully autonomous
vehicles may perform well in the aforementioned scenarios, but they may still
fail in key traffic situations that a human could easily handle, such as
parking lots, pedestrian crossing areas, or construction roads. Vehicle
teleoperation could potentially help with the transition from human-driven to
completely autonomous vehicles.

Vehicle teleoperation has a number of challenges, including control loop
delays, complete connection loss, limited situation awareness, etc. Human
performance in virtual environments has shown that people can notice latency
as low as 10–20 ms. [1]. According to control theory, time delays can cause
the vehicle’s real-time control to become unstable [2]. Frank L. H. [3], in a
simulated driving task, observed that delays of 170 milliseconds had a
significant impact on driving performance. Humans can adapt to time delays in
control systems, but this adaptation is contingent on the human operator’s
capacity to foresee the outcome of his control actions, and the level of this
adaptation is determined by the time delay’s features (e.g., magnitude and
variability)[4]. Human operators, on the other hand, have a tendency to
overcorrect steer when delays are large, resulting in oscillations that can
significantly reduce teleoperation performance and even destabilize the
closed-loop system [5].

Literature of interest can be classified broadly in two major categories:
Predictive display and Trajectory based vehicle teleoperation. Predictive
displays, have been proven effective to compensate delays and improve vehicle
mobility in human-in-the-loop experiments [6, 7, 8, 9, 10, 11, 12]. It
considers the delay together with the control signals from the operator to
estimate the vehicle position. This predicted position is displayed to the
operator as a ”third-person view” [10]. The prediction model can either be
model-based [9], model-free [11], or a blend of both [12]. In model-based
predictors, a vehicle model is required to predict the vehicle response, and
the prediction accuracy depends on the accuracy of the vehicle model. In the
presence of unknown disturbances in driving scenario such as low adhesion road
or cross-wind, prediction accuracy deteriorates. In model-free approach [11],
prediction is made by taking into account the delayed state dynamics received
from the vehicle. It suffers from convergence time in the state prediction
which results in delayed prediction. Blending of both results improved
operation, but not much significantly. In summary, predictive displays tries
to bypass the time delay in loop, by predicting states taking input the
delayed states. This is helpful for human-in-the-loop teleoperation, as it
allows the human operator to not to wait for the feedback and provides the
sense of controlling the vehicle in real-time. The limitation is, if the
prediction accuracy decreases, chances of asynchrony increases.

In trajectory-based driving (shared-control approach), vehicle control is
based on automated driving along predefined paths, using trajectories as
command variables, which consist of parameterized curves overlaid by reference
vehicle speeds. This eliminates network time-delay out of control loop.
Control-station transmits the mission plan, and vehicle autonomy performs the
maneuvers. Michael Fennel[13], proposed an offline path follower where the
operator “draws” a desired 2D path by walking in a large-scale haptic
interface while a guiding force is exerted, which ensures that the generated
path can later be accurately followed by a path tracking controller running
offline on a remote robot. The limitation of trajectory-based driving is, the
operator is not actively controlling the task.

Figure 1: Successive reference-pose tracking (SRPT) in vehicle teleoperation

This paper extends the trajectory-based driving approach, according to fig
(1), by transmitting successive reference-pose to the vehicle at $30$ fps
instead of transmitting the whole mission plan. The successive poses
corresponds to the target to reached in around $1$s. A Non-linear Model
Predictive Control (NMPC) inside the vehicle optimizes steer and vehicle speed
considering vehicle states, steer actuator constraints, and vehicle
acceleration limits adopting a conservative friction value according to the
reference pose sent by the control-station. The contribution of this paper is
the introduction of the approach of Successive Reference-Pose Tracking (SRPT),
in which control-station transmits target 2D-poses instead of direct steer and
speed commands. To assess its usefulness, result of this approach is compared
with state prediction (Smith predictor) approach in Simulink simulation
environment. In simulation environment, variable network delays and 14-dof
vehicle model for the main vehicle are considered.

The paper is organized as follows. Section II - Method, A) exhibits delay
trend in vehicle teleoperation over 4G. B) discuss the Smith predictor
approach, C) explains the SRPT approach. Section III compares the two
approaches over their simulation results. Section IV concludes the paper with
key findings.

##  II Method

###  II-A Uplink delay and variable downlink delay

Taking control-station as reference point, time-delay in vehicle teleoperation
can be divided into two parts. One is downlink-delay ($\tau_{2}$), associated
to the availability of streamed images to the human operator at the control
station. Other is uplink-delay ($\tau_{1}$), associated to the delay between
generation of driving commands at the control station and actuation of those
at the vehicle. $\tau_{2}$ can be considered as the sum of camera exposure
delay, image encoding time consumption, network delay in transmitting the
images towards the control station and image decoding time consumption. These
contributions can be lumped, because time point of image capture is used in
image time-stamping. $\tau_{1}$ can be considered as the sum of network delay
in transmitting the driving commands towards the vehicle, and vehicle
actuation delay. In case of wireless communication using 4G, variability is
associated both downlink and uplink delays. Fig(2) presents both sided delays
with corresponding utilized bandwidth. Here the data corresponds to 5000 image
frames and corresponding driving commands. This test is performed in typical
urban environment where the vehicle is connected to 4G mobile communication
and control-station is connected to internet using wired LAN.

Figure 2: Delays observed in data transmission over 4G

From the control-station perspective, uplink-delay is unknown and the
downlink-delay is known. Being smaller in amplitude and relatively lower
variability of uplink-delays, a constant high stochastic value (an approach
adopted also in [9]) of 60ms is considered in teleoperation simulations of
this paper. The downlink delay is known, as the data packets are timestamped.
_Generalized extreme value distribution_ , $GEV(\xi=0.29,\mu=200,\sigma=9)$
fits accurately on downlink-delay experimental data. Also, authors from
articles [12, 9] found GEV distribution apt for representing distribution of
time-delays in mobile communication. Here, $\xi$ is shape parameter, $\mu$ is
location parameter and $\sigma>0$ is scale parameter. Positive $\xi$ means,
the distribution has a lower bound $(\mu-\frac{\sigma}{\xi})\approx
169ms\>(>0)$ and a continuous right tail, based on the extreme value theory.
During a stable 4G connection (fig 2), data packets received are found to be
in FIFO order. Which indicates no data loss and FIFO queue behaviour of the
communication.

###  II-B Smith predictor in vehicle teleoperation

Figure 3: Smith predictor scheme

Among the predictive control approaches in bilateral teleoperation, Smith
predictor approach is quite popular. It is a model based prediction approach
introduced by O.J. Smith in 1957 [14]. A schematic of the Smith Predictor in
vehicle teleoperation control loop is shown in fig (3) for system with known
time-delays. The control input ($\delta$) is passed through a local predictor
model ($P^{\prime}$) of the vehicle, which then passes through
$(1-e^{-\left(\tau_{1}+\tau_{2}\right)s})$, where a time-delayed version of
the output is subtracted from the real-time version. With this schematic,
feedback given to the human operator ($X_{p}$) is

$X_{p}=P^{\prime}u\left(1-e^{-\left(\tau_{1}+\tau_{2}\right)s}\right)+\delta
e^{-\tau_{1}s}Pe^{-\tau_{2}s}$ (1)

Which in turn if the predictor model ($P^{\prime}$) is equal to vehicle model
($P$) becomes

$X_{p}=P\delta$ (2)

It provides the human operator the sense of controlling the vehicle in real-
time. Although the Smith Predictor was originally aimed at controller design,
the same scheme can be used to help the human better observe the feedback
state by eliminating the backward delay. The transfer function of the closed
loop delayed system is

$\frac{X}{X_{REF}}=\frac{HPe^{-\tau_{1}s}}{1+HP^{\prime}-HP^{\prime}e^{-\left(\tau_{1}+\tau_{2}\right)s}+HPe^{-\left(\tau_{1}+\tau_{2}\right)s}}$
(3)

If $P^{\prime}=P$, it becomes

$\frac{X}{X_{REF}}=\frac{HP}{1+HP}e^{-\tau_{1}s}$ (4)

Smith Predictor results bypassing the delay in the observation and transform
the system into a pure forward delay to the vehicle. The backward delay is
eliminated, and the forward delay is moved aft of the control loop. This
allows the human operator to command the vehicle without being hindered by the
time delay. The system still tracks the input with a constant forward delay
offset; however, this will not affect the controllability of the vehicle.
Driving inputs in normal driving are steering, throttle and braking. In order
to reduce mental fatigue of human operator and eliminate coupling of decision
parameters (steering and throttle), cruise speed control can be implemented
inside the vehicle. Which inherently eliminates delay out of the control loop
for longitudinal dynamics. This cruise control node receives reference speed
from the human operator at the start of the teleoperation task or at regular
time-interval. Consequently, the control input ($\delta$) is only the steering
angle. In case of constant time-delays, the schematic given in fig (3), can be
simulated in continuous time domain, without maintaining any history of the
states. But, to simulate variable downlink delays linked with output ($X$), as
$(1-e^{-\left(\tau_{1}+\tau_{2}\right)s})$ suggests current output of
$P^{\prime}$ has to be subtracted by its output at
$(t-\left(\tau_{1}+\tau_{2}\right))$. Time history of output of $P^{\prime}$
needs to be managed to be available when the delayed output is required.

For vehicle teleoperation simulation, $P^{\prime}$ is a dynamic single-track
model, whose equations of motion are given by first seven terms of eq 13, with
null acceleration ($a=0$). P is the main vehicle with 14-dof vehicle model
using MATLAB vehicle dynamics blockset [15].

###  II-C Successive reference-pose tracking using NMPC

In vehicle teleoperation, model-based prediction approach as discussed in
section above is effective, as dynamic single track vehicle model is a decent
estimation in linear tire-dynamics range when lateral acceleration $<4m/s^{2}$
[16] on normal roads and in normal conditions. But disturbances such as strong
winds, very low adherence roads, bumps etc., can change the vehicle dynamics
for small as well as for big duration. Parameter estimation techniques
discussed in articles [17, 18] are useful for medium to high duration change
in dynamics. As this techniques are based on sliding window batch estimation,
estimations has a lead time associated to it. Momentary disturbances can
impact the vehicle output significantly before the new plant dynamics is
estimated at the control station and corrective action taken by the human
operator.

In SRPT approach, instead of the control station transmitting the steering
inputs, it transmits successive reference 2D-poses to be followed by the
teleoperator vehicle. The schematic of SRPT approach in vehicle teleoperation
control loop is shown in fig (4).

Figure 4: Successive reference-pose tracking (SRPT) scheme

The NMPC block receives the reference pose which belongs to the (approximate)
end of its prediction horizon. Important fact is that it receives feedback
(vehicle states, $X(t)$) without any network delay from the vehicle. By
analyzing reference pose, vehicle current states, actuator and environmental
constraints, it generates optimize steer and acceleration inputs. On the
control-station side, the human operator receives delayed vehicle pose.
Knowing the whole reference trajectory to be followed by the vehicle, downlink
delay, uplink delay and prediction horizon, the operator transmits the
reference pose for the end of the prediction horizon. Here the downlink delay
is considered varying but known, uplink delay is considered a known constant
and prediction time horizon is also a known constant.

####  II-C1 NMPC vehicle model

Dynamic single track vehicle model, being a good trade-off between simplicity
for real-time implementation and accuracy for high-performance vehicle
control, is used as prediction model in NMPC. It is illustrated in fig (5)
with states

$\mathbf{x}=\left[\>\beta;\>\dot{\psi};\>\psi;\>F_{y,F};\>F_{y,R};\>x;\>y;\>\delta;\>V_{x}\right],$
(5)

$\beta$ is side-slip angle, $[F_{y,F},F_{y,R}]$ are lateral forces at axles in
wheel reference plane, $V_{x}$ is vehicle longitudinal velocity, $\delta$ is
steering angle. Before each optimization routine, starting vehicle 2D-pose
resets $[x=0;\>y=0;\>\psi=0]$. This is because vehicle receives target
reference pose in global reference frame, but before feeding it to NMPC, it is
converted into vehicle reference frame.

Figure 5: Single track vehicle model

Balancing the lateral forces gives

$mA_{x}=m\left(\dot{V}_{y}+\dot{\psi}V_{x}\right)=F_{y,F}\cos\delta+F_{x,F}\sin\delta+F_{y,R}$
(6)

Considering small $\beta$ angle

$\displaystyle V_{x}=V\cos\beta\simeq V$ (7) $\displaystyle
V_{y}=V\sin\beta\simeq V\beta$

eq (6) becomes

$\dot{\beta}=\frac{1}{mV}\left(F_{y,F}\cos\delta+F_{x,F}\sin\delta+F_{y,R}\right)-\frac{\beta\cdot
a}{V}-\dot{\psi}$ (8)

Balancing the yaw moment around the center of mass gives

$\ddot{\psi}=\frac{1}{I_{Z}}\left(\left(F_{y,F}\cos\delta+F_{x,F}\sin\delta\right)l_{F}-F_{y,R}l_{R}\right).$
(9)

For simplicity, relaxation length phenomenon is ignored in longitudinal force
development and thus given by

$\displaystyle
F_{x,F}=\begin{cases}m\,a+f_{V}\,m_{R}\,g+C_{Aero}\,V^{2},&\text{if }a\geq
0\\\
\>\>\>\>\>\>\>\>\>\gamma\,(m\,a+f_{V}\,m\,g+C_{Aero}\,V^{2}),&\text{otherwise}\end{cases}$
(10) $\displaystyle
F_{x,R}=\begin{cases}\>\>\>\>\>\>\>\>\>-f_{V}\,m_{R}\,g,&\text{if }a\geq 0\\\
(1-\gamma)\,(m\,a+f_{V}\,m\,g+C_{Aero}\,V^{2}),&\text{otherwise}\end{cases}$

$\begin{array}[]{l}\zeta_{F}=\sqrt{1-\left(\frac{F_{x,F}}{\mu_{cons}\>m_{F}\>g}\right)^{2}}\\\
\zeta_{R}=\sqrt{1-\left(\frac{F_{x,R}}{\mu_{cons}\>m_{R}\>g}\right)^{2}}\end{array}$
(11)

$\begin{array}[]{l}\alpha_{F}\simeq\tan^{-1}\left(\tan\delta-\beta-\dot{\psi}\frac{l_{F}}{V}\right)\\\
\alpha_{R}\simeq\tan^{-1}\left(\>\>\>\>\>\>\>\>\>\>\>-\beta+\dot{\psi}\frac{l_{R}}{V}\right)\end{array}$
(12)

$[C_{\alpha,F},C_{\alpha,R}]$ are the lumped cornering stiffness’s of front
and rear axles respectively; $[\zeta_{F},\zeta_{R}]$ are the reduction factor
for cornering stiffness considering longitudinally forces;
$[\alpha_{F},\alpha_{R}]$ are the slip angles of tires. [$m_{f},m_{R}$] are
distribution of vehicle mass on front and rear axle based of [$l_{F},l_{R}$].

Altogether, the NMPC prediction model can be expressed as

$\dot{\mathbf{x}}=\left[\begin{array}[]{c}\frac{1}{mV}\left(F_{y,F}\cos\delta+F_{x,F}\sin\delta+F_{y,R}\right)-\frac{\beta\cdot
a}{V}-\dot{\psi}\\\
\frac{1}{I_{Z}}\left[\left(F_{y,F}\cos\delta+F_{x,F}\sin\delta\right)l_{F}-F_{y,R}\>l_{R}\right]\\\
\dot{\psi}\\\
\frac{V}{\lambda}\left[\zeta_{F}\,C_{\alpha,F}\>\alpha_{F}-F_{y,F}\right]\\\
\frac{V}{\lambda}\left[\zeta_{R}\,C_{\alpha,R}\>\alpha_{R}-F_{y,R}\right]\\\
V\cos(\psi+\beta)\\\ V\sin(\psi+\beta)\\\ \dot{\delta}\\\
a\end{array}\right].$ (13)

Here, $a$ is the input longitudinal acceleration, $\dot{\delta}$ is input
steer velocity. To make it applicable for zero vehicle speed, wherever the $V$
is in denominator, is substituted by $max(0.01,V)$.

TABLE I: Vehicle parameters for the single-track model Parameter | Value
---|---
$m$ | 1180 kg
$I_{Z}$ | 2066 kg s2
$l_{F}$ | 1.515 m
$l_{R}$ | 1.504 m
$C_{\alpha,F}$ | 46 kN/rad
$C_{\alpha,R}$ | 46 kN/rad
$\lambda$ (Relaxation length) | 0.3 m
$\gamma$ (Braking bias) | 0.6
$C_{Aero}$ (Aerodynamic drag) | 0.4 N / (m2/s2)
$f_{V}$ (Rolling resistance coeff) | 0.025

####  II-C2 NMPC constraints

Input constraints are steering velocity, vehicle acceleration and
deceleration.

$\begin{array}[]{c}\left[\dot{\delta}_{min}=-10^{\circ}/s\right]\leq\dot{\delta}\leq\left[\dot{\delta}_{max}=+10^{\circ}/s\right]\\\
\>\>\>\>\left[a_{min}=-\mu_{cons}\cdot 4m/s^{2}\right]\leq
a\leq\left[a_{max}=\mu_{cons}\cdot 1m/s^{2}\right]\par\end{array}$ (14)

$[a_{min},a_{max}]$ are down-scaled by $\mu_{cons}$, to prohibit friction
force saturation on low adhesion surface. Output constraints are max steering
angle, vehicle speed, friction utilization at the tires. They can be
summarized as

$\begin{array}[]{c}\left[\delta_{min}=-25^{\circ}\right]\leq\delta\leq\left[\delta_{max}=+25^{\circ}\right]\\\
0\leq V\\\
\frac{\left\|\left(\zeta_{F}C_{\alpha,F}\,\alpha_{F},\>\>F_{x,F}\right)\right\|_{2}}{m_{F}\,g}\leq\mu_{cons}\\\
\frac{\left\|\left(\zeta_{R}C_{\alpha,R}\,\alpha_{R},\>\>F_{x,R}\right)\right\|_{2}}{m_{R}\,g}\leq\mu_{cons}.\end{array}$
(15)

The fact that human is in the loop, can be exploited in assigning a
conservative friction coefficient $\mu_{cons}$. E.g., if the upcoming road is
dry asphalt or snowy, the human operator assigns a conservative value of
$\mu_{cons}=0.90$ or $\mu_{cons}=0.25$ to NMPC respectively.

####  II-C3 NMPC cost function

The primary aim is to align the target reference pose with optimized vehicle
trajectory. Clothoid path is a preferred path in vehicle motion, because it
resembles natural driving where steer changes linearly. Eliou and Kaliabetsos
[19], suggests cubic spline can be a first approximation for clothoid curve.
For each prediction NMPC node estimates a cubic spline (eq 16-17) from vehicle
CG to reference pose in vehicle reference frame.

$y=Ax^{3}+Bx^{2}+Cx+D$ (16)

Figure 6: Cubic spline generation; inputs are in blue.

Target reference pose in vehicle reference frame is
$[x_{Ref};\,y_{Ref};\,\psi_{Ref}]$, the cubic coefficients are given by

${\left[\begin{array}[]{l}A\\\ B\\\ C\\\
D\end{array}\right]=\left[\begin{array}[]{cccc}0&0&0&1\\\
x_{Ref}^{3}&x_{Ref}^{2}&x_{Ref}&1\\\ 0&0&1&0\\\
3x_{Ref}^{2}&2x_{Ref}&1&0\end{array}\right]^{-1}\left[\begin{array}[]{c}0\\\
y_{Ref}\\\ \tan\beta\\\ \tan\psi_{Ref}\end{array}\right]}$ (17)

which maintains $G1$ continuity at both ends, i.e., starting tail with $\beta$
and end tail of spline with $\psi_{Ref}$, as shown in fig 6.

Cost function of NMPC formulation is given by

$\displaystyle\min$
$\displaystyle\sum_{i=0}^{N-1}{U_{i}^{t}\,R\,U_{i}}+\sum_{i=0}^{N-1}{X_{i}^{t}\,Q\,X_{i}}+X_{N}^{t}\,P\,X_{N}$
(18)

Where,

$\displaystyle U_{i}$ $\displaystyle=\left[\begin{array}[]{l}\dot{\delta}\\\
a\end{array}\right]\quad\forall i\in[0,N-1]$ (21) $\displaystyle X_{i}$
$\displaystyle=\left\\{\begin{array}[]{l}{\left[V_{Ref}-V_{i}\right]\quad\forall
i\in[0,N-1]}\\\ \\\
{\left[\begin{array}[]{l}Ax_{i}^{3}+Bx_{i}^{2}+Cx_{i}+D\>\>\>\>\>\>-\>y_{i}\\\
\tan^{-1}\left(3Ax_{i}^{2}+2Bx_{i}+C\right)-\psi_{i}\end{array}\right]\quad\forall
i=N}\end{array}\right.$ (27)

The stage cost ($\forall i\in[0,N-1]$) tries to minimize inputs ($U_{i}$) and
also tries to be as near as possible to the reference vehicle speed
($V_{Ref}$) asked by the human operator. The first term in terminal cost aims
to coincide end point of the NMPC trajectory with the reference spline and
second term match the heading of end point with spline tangent.
$[x_{i},y_{i}]$ are trajectory points in vehicle reference frame. Penalties
assigned in the cost function is summarized in the Appendix. Prediction
horizon ($\Delta t_{Horizon}$) is of $1\,sec$. The optimal control problem is
discretized in $N=50$ intervals using discrete multiple shooting and solved by
sequential quadratic programming using the real-time NMPC solver ACADOS [20,
21].

##  III Simulation Results

Figure 7: Trajectory tracking comparison between smith predictor approach and
SRPT approach at VRef=20km/h

To compare the smith predictor approach and SRPT approach, Reference
trajectory shown in figure 7 is chosen. It has 5 sections, namely A-F. A-B is
cornering, C is double lane change, D is cornering on low adhesion road, E is
cornering in strong diagonal wind and F is slalom. For simulation, vehicle
dynamics blockset [15] of MATLAB is utilized. It consists of a 14-dof
passenger FWD vehicle model which accepts steering, acceleration and brake
commands. Longitudinal dynamics control is kept same for the comparison, which
is a PI control with tracking windup and feed-forward gains to maintain cruise
speed of $20km/h$. Artificial bidirectional delay is introduced to simulate
teleoperation network delays, constant uplink-delay ($\tau_{1}$) of $60ms$ and
variable downlink-delay ($\tau_{2}$) as discussed in section<EMAIL_ADDRESS>
Human ($H$) in smith predictor approach (fig 3) is simulated by kinematic
Stanley controller [22].

$\delta(t)=\begin{cases}\psi(t)+\tan^{-1}\frac{ke(t)}{V(t)}&\text{ if
}\left|\psi(t)+\tan^{-1}\frac{ke(t)}{V(t)}\right|<\delta_{\max}\\\
\>\>\>\delta_{\max}&\text{ if
}\psi(t)+\tan^{-1}\frac{ke(t)}{V(t)}\geq\delta_{\max}\\\ -\delta_{\max}&\text{
if }\psi(t)+\tan^{-1}\frac{ke(t)}{V(t)}\leq-\delta_{\max}\end{cases}$ (28)

Here, $\psi(t)$ is the yaw angle (heading) of the vehicle with respect to the
closest trajectory segment. $e(t)$ is cross-track error from the front axle
center. $V(t)$ is the vehicle speed in $m/s$ and $k$ is tuneable gain which is
set to be equal to $0.7$ in this case.

Whereas the human ($H$) in SRPT approach (fig 4) is a simple look-ahead pose
selector, which receives delayed pose and vehicle speed. Knowing the target
reference trajectory, vehicle delayed pose and speed, it selects and transmits
the look ahead pose corresponds to [$\tau_{2}+\tau_{1}+\Delta t_{Horizon}$]
seconds ahead of the delayed pose. The NMPC controller runs well below 20 ms
($mean=6ms,max=15ms$) on an Intel Core i7-11800H 4.6 Ghz 8 Core CPU with 16 GB
of RAM in parallel with Simulink environment and hence it can be expected to
perform real-time @50Hz on a dedicated hardware of the experimental vehicle.

Trajectories traversed by the two approaches smith predictor and SRPT is
presented in figure 7. For better comparison, trajectory traversed in case of
no delay and delay with no teleoperation control is also presented. Due to
poor path following performance in case of delay with no teleoperation
control, only for this particular mode, low vehicle speed of $12km/h$ is
chosen. The zoom plots in fig 8, emphasise better reference tracking behaviour
with the SRPT approach in all sections of the trajectory. In this figure,
region A and B are ignored, due to insignificant improvement found.

Figure 8: Zoomed regions

Figure 9a quantifies the performance difference by comparing the RMS of
lateral deflection from the reference trajectory for each section of the
trajectory.

RMS deflection is computed as given by

$\displaystyle\Delta
Y_{\mathrm{RMS}}=\sqrt{\frac{1}{D_{i+1}-D_{i}}\int_{D_{i}}^{D_{i+1}}[\Delta
Y(D)]^{2}\>dD}$ (29)

where,
$\Delta Y(D)$ \- is the lateral deflection of vehicle CG from the reference
trajectory
$D$ \- distance travelled along the reference trajectory.
$i,i+1$ \- indicates the start and end of the particular region (fig 9a).

Compare to the deflections observed with smith predictor approach, deflections
observed with SRPT are lesser in all the trajectory sections except for region
B, where difference is insignificant as shown in fig 9a and fig 10. Figure
9(b-c-d) shows steer, steer rate, vehicle speed, and lateral acceleration
respectively vs the distance travelled.

Figure 9: Cross-track error, steer, steer-rate, vehicle speed, and lateral
acceleration comparison

Result is analysed below for each trajectory section:

### Trajectory section - A, and B

Insignificant differences are found in all the teleoperation modes due to less
aggressive steer requirement due to low curvature.

### Trajectory section - C, and F

Successive pose tracking approach showed lesser deviations because wherever
the steer-rate requirement exceeds actuator limitation
($\>|\dot{\delta}|\leq\;\;10deg/s\>$), NMPC moderates the vehicle speed (fig
9d) to allow more time for steering. It can be clearly observed in C and F
trajectory sections, where steer (fig 9b) is actuated slightly before in
distance as compare to smith predictor approach.

### Trajectory section - D

Here the road persists low adhesion property ($\mu=0.3$), which corresponds to
snowy environment condition. Exploiting the fact that human is in the control
loop, human can convey a conservative friction ($\mu=0.25$ in this case) value
to the NMPC controller at the section entrance. According to which, NMPC
moderated the vehicle speed and started steering in advance (fig 9c) to be on
the reference track. On the contrary smith predictor estimation accuracy
suffered due to the presence of disturbance in the plant and eventually
resulted noisy steering command.

### Trajectory section - E

Here in the presence of strong cross winds ($V_{Wind}=100\;km/h$), again
disturbance in the plant causes noisy steering commands (fig 9c) with smith
predictor approach. Whereas being unaware of cross wind disturbance but aware
of reference poses, NMPC steering commands are smooth. Which leads to less
deviations.

Figure 10, summarizes the cross-track error in all regions with all modes.
Shortness of blue bars with SRPT mode, reflects significant less lateral error
in vehicle teleoperation.

Figure 10: Cross-track error comparison in individual regions with different
teleoperation modes

##  IV Conclusion

The classical Smith predictor approach to mitigate detrimental time-delay
effects in closed-loop operation is compared with proposed SRPT approach for
vehicle teleoperation in presence of disturbances such as low adhesion road,
cross-wind and aggressive maneuvers such as slalom & double-lane change. The
simulation framework consists of Simulink environment with 14-dof vehicle
model equipped with PI cruise control feature. Smith predictor approach
transmits directly the steer commands to the vehicle, whereas SRPT approach
transmits reference poses to the vehicle based on look-ahead time and
thereupon NMPC controller modulates steer as well as reference vehicle speed.
The optimal routine uses ACADOS frameowrk, which shows the capability to run
at $50Hz$ as mean computation time observed is $6ms$.

The optimization routine penalizes cross-track error and eventually decelerate
the vehicle to dilate the time-window available for steering to account
saturation of steering actuator. To analyse the performance benefit, RMS of
cross-track error is compared for different critical trajectory sections shown
in figure 7. Comparison indicates significant improvement in terms of
reduction in cross-track error (fig 10) with SRPT approach.

Futurework \- This paper presents a control approach where successive
reference poses are transmitted to the vehicle at close time intervals
(@30fps) as compare to transmitting steering commands directly to the vehicle.
Successive work would propose a novel method where the issue of real-time
generation of successive reference poses is attempted to solve with joystick
steering wheel to provide a realistic driving experience to the human
operator. A human in the loop vehicle simulator environment would be utilized
to assess performance benefits with the proposed SRPT approach.

## Appendix

NMPC cost penalties are summarized in the following table.

TABLE II: NMPC cost penalties R | Q | P
---|---|---
$diag([1,0.1])$ | $1$ | $diag([50,3])$

## References

  * [1] S. R. Ellis, K. Mania, B. D. Adelstein, and M. I. Hill, “Generalizeability of Latency Detection in a Variety of Virtual Environments,” _Proceedings of the Human Factors and Ergonomics Society Annual Meeting_ , 2004.
  * [2] H. Pongrac, “Gestaltung und evaluation von virtuellen und telepräsenzsystemen an hand von aufgabenleistung und präsenzempfinden,” 2011.
  * [3] L. H. Frank, J. G. Casali, and W. W. Wierwille, “Effects of visual display and motion system delays on operator performance and uneasiness in a driving simulator,” _Human Factors_ , 1988.
  * [4] J. Davis, C. Smyth, and K. McDowell, “The effects of time lag on driving performance and a possible mitigation,” _IEEE Transactions on Robotics_ , 2010.
  * [5] T. B. Sheridan, “Space Teleoperation Through Time Delay: Review and Prognosis,” _IEEE Transactions on Robotics and Automation_ , 1993.
  * [6] F. Chucholowski, “Evaluation of display methods for teleoperation of road vehicles,” _Journal of Unmanned System Technology_ , vol. 3, pp. 80–85, 02 2016.
  * [7] A. Hosseini, F. Richthammer, and M. Lienkamp, “Predictive haptic feedback for safe lateral control of teleoperated road vehicles in urban areas,” in _2016 IEEE 83rd Vehicular Technology Conference (VTC Spring)_ , 2016, pp. 1–7.
  * [8] M. J. Brudnak, “Predictive displays for high latency teleoperation,” 2016.
  * [9] J. Prakash, M. Vignati, D. Vignarca, E. Sabbioni, and F. Cheli, “Predictive display with perspective projection of surroundings in vehicle teleoperation to account time-delays,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 11, no. 21, 2022.
  * [10] T. Chen, “Methods for improving the control of teleoperated vehicles,” Ph.D. dissertation, Technical University of Munich, 4 2015.
  * [11] Y. Zheng, M. J. Brudnak, P. Jayakumar, J. Stein, and T. Ersal, “A delay compensation framework for predicting heading in teleoperated ground vehicles,” _IEEE/ASME Transactions on Mechatronics_ , vol. 24, pp. 2365–2376, 2019.
  * [12] Y. Zheng, M. J. Brudnak, P. Jayakumar, J. L. Stein, and T. Ersal, “Evaluation of a predictor-based framework in high-speed teleoperated military ugvs,” _IEEE Transactions on Human-Machine Systems_ , vol. 50, no. 6, pp. 561–572, 2020.
  * [13] M. Fennel, A. Zea, and U. D. Hanebeck, “Haptic-guided path generation for remote car-like vehicles,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 2, pp. 4087–4094, 2021.
  * [14] J. M. Smith, “Closer Control of Loops with Dead Time,” _Chemical Engineering Progress_ , vol. 53, pp. 217–219, 1957.
  * [15] MATLAB. Double Lane Change Reference Application. [Online]. Available: https://www.mathworks.com/help/vdynblks/ug/double-lane-change-reference-application.html
  * [16] S. Reicherts, B. S. Hesse, and D. Schramm, “Use of naturalistic driving studies for identification of vehicle dynamics,” _IEEE Open Journal of Intelligent Transportation Systems_ , vol. 2, pp. 195–206, 2021.
  * [17] T. Teng and P. R. Grant, _Adaptive Smith Predictor for Teleoperation of UAVs using Parameter Estimation Techniques_.
  * [18] T. Christopher, M. Sun Yi, Shava, and S. Ryan, “Adaptive smith predictor for teleoperation of uavs with time-varying internet delay,” _International Journal of Control, Automation and Systems_ , vol. 18, no. 6, pp. 1465–1473, 2020\.
  * [19] N. Eliou and G. Kaliabetsos, “A new, simple and accurate transition curve type, for use in road and railway alignment design,” _European Transport Research Review_ , vol. 6, pp. 171–179, 2014.
  * [20] R. Verschueren, G. Frison, D. Kouzoupis, J. Frey, N. van Duijkeren, A. Zanelli, B. Novoselnik, T. Albin, R. Quirynen, and M. Diehl, “acados: a modular open-source framework for fast embedded optimal control,” 2020.
  * [21] R. Verschueren, G. Frison, D. Kouzoupis, N. van Duijkeren, A. Zanelli, R. Quirynen, and M. Diehl, “Towards a modular software package for embedded optimization,” in _Proceedings of the IFAC Conference on Nonlinear Model Predictive Control (NMPC)_ , 2018.
  * [22] G. M. Hoffmann, C. J. Tomlin, M. Montemerlo, and S. Thrun, “Autonomous automobile trajectory tracking for off-road driving: Controller design, experimental validation and racing,” in _2007 American Control Conference_ , 2007, pp. 2296–2301.