Erkan Kayacan, Senseable City Laboratory and Computer Science & Artificial
Intelligence Laboratory,
Massachusetts Institute of Technology, MA 02139, USA.

# Sliding Mode Learning Control of Uncertain Nonlinear Systems with Lyapunov
Stability Analysis

Erkan Kayacan  Senseable City Laboratory and Computer Science & Artificial
Intelligence Laboratory, Massachusetts Institute of Technology<EMAIL_ADDRESS>
###### Abstract

This paper addresses to Sliding Mode Learning Control (SMLC) of uncertain
nonlinear systems with Lyapunov stability analysis. In the control scheme, a
conventional control term is used to provide the system stability in compact
space while a Type-2 Neuro-Fuzzy Controller (T2NFC) learns system behavior so
that the T2NFC takes the overall control of the system completely in a very
short time period. The stability of the sliding mode learning algorithm was
proven in literature; however, it is so restrictive for systems without the
overall system stability. To address this shortcoming, a novel control
structure with a novel sliding surface is proposed in this paper and the
stability of the overall system is proven for nth-order uncertain nonlinear
systems. To investigate the capability and effectiveness of the proposed
learning and control algorithms, the simulation studies have been achieved
under noisy conditions. The simulation results confirm that the developed SMLC
algorithm can learn the system behavior in the absence of any mathematical
model knowledge and exhibit robust control performance against external
disturbances.

###### keywords:

Adaptive control, neuro-fuzzy control, uncertain nonlinear systems, online
learning algorithm, sliding mode control theory.

##  1 Introduction

In modern control engineering, control of uncertain nonlinear systems is one
of the most crucial topics Chen et al. (2016); Kayacan et al. (2012, 2013b,
2017); Kayacan and Peschel (2016); Kayacan and Fossen (2019). Robust
controllers intend to ensure the best control performance in the presence of
the uncertainties, and high controller gain is the common method to overcome
uncertainity problem in nonlinear control theory Sun et al. (2016). However,
these methods bring about large control actions, and demand of very powerful
actuators Ren et al. (2017). Moreover, the robust control performance against
uncertainties is mostly obtained at a price of sacrificing the nominal control
performance of the system Liu et al. (2017a). Therefore, a control method is
required to exhibit robust control performance in the presence of
uncertainties while either maintaining or improving the nominal control
performance Huang et al. (2015). As a model-free method, fuzzy logic control
is an alternative solution to the model-based control approaches in the
presence of uncertainties and has been applied to different real-time systems,
e.g., mobile robots Castillo et al. (2012), spherical rolling robot Kayacan et
al. (2013a) and agricultural ground vehicles Kayacan et al. (2015b, 2018).
Even though type-1 fuzzy logic controllers are the most well known and widely
used types of fuzzy logic controllers, researchers have recently focused on
type-2 fuzzy logic controllers, which can only deal with high levels of
uncertainty in real-time applications Maldonado et al. (2013); Castillo et al.
(2014); Lee et al. (2015); Muhuri and Shukla (2017); Liu et al. (2017c);
Rubio-Solis and Panoutsos (2015); Zaheer et al. (2015).

As a learning controller, feedback-error learning control structure was
proposed in Gomi and Kawato (1993) and firstly implemented for the control of
robot manipulators that was relied on the parallel structure of a conventional
controller and a neuro-fuzzy based controller. The generated control signal by
the conventional controller is benefited as a learning error signal to train
learning algorithms. The latter learns the system behavior online and thus
eliminates the former from the control of the system Kalanovic et al. (2000);
Ruan et al. (2007). Since the gradient-based methods as training algorithm are
complex and computationally more expensive, sliding-mode control (SMC) theory-
based learning algorithms were proposed for feedback-error learning control
structures Efe et al. (2000). They can ensure faster convergence rather than
the conventional learning techniques for online adjustment in neural networks,
type-1 and type-2 neuro-fuzzy structures Kayacan et al. (2015a). Moreover, the
stability and robustness of NFCs based on SMC theory-based learning algorithms
can also be examined Topalov et al. (2007a). The novelty of SMC theory-based
learning algorithms is that the parameters are adjusted by the designed
algorithm in order to fulfill a stable equation by imposing the error instead
of minimizing an error function. On the other hand, SMC is inherently robust
to uncertainties Yin et al. (2017a, b).

Despite their practicality and easiness to design, model-free control methods
– such as fuzzy control and neuro-fuzzy control – are mostly criticized by
model-based control community due to the fact that there is generally no
convincing analytical proof of the system stability Feng (2006). These
criticisms are reasonable since trial-and-error methods cannot be tolerated
for sophisticated and expensive robotic systems, such as unmanned air vehicles
and spacecraft systems. Therefore, analytical stability analysis is an
overriding requirement so that researchers tend to design model-based
controllers Mayne et al. (2000); Kayacan et al. (2014, 2016); Lam (2018). In
order to accomplish this limitation of model-free controllers, the overall
system stability proven in this paper lightens the major disadvantage of
model-free control methods. In the previous studies in literature, since the
overall system stability of the sliding mode learning algorithm could not be
proven, the stability analysis was accomplished, and a proportional-derivative
controller was used to ensure the stability in compact space Topalov et al.
(2007b). Recently, it has been shown that if the system is a second-order
system, it is possible to guarantee the stability of the overall system by
adding a robust term to the control scheme Khanesar et al. (2015). However,
this stability analysis is too restrictive.

The contributions of this paper are as follows.

  1. 1.

The stability of the overall system is proven for an nth-order uncertain
nonlinear system. The overall system stability analysis removes the most
significant shortcoming of model-free control methods.

  2. 2.

A novel sliding surface is proposed to train the sliding mode learning
algorithm for the type-2 neuro-fuzzy control (T2NFC) algorithm.

  3. 3.

The learning rate of the online sliding mode learning algorithm of T2NFC and
the controller gain in the conventional control term are adaptive. Therefore,
it is possible to control the system without foreknowing the upper bounds of
the system states and their derivatives.

  4. 4.

The developed algorithm adjusts the proportion of the lower and upper MFs in
the T2NFC that allows us to handle non-uniform uncertainties in T2FLSs.

The paper consists of four sections: The SMLC structure consisting of the
T2NFC and the conventional control term, and the sliding mode control theory-
based learning algorithm are given in Section 2. The overall system stability
is given in Section 3. The simulation results are presented to discuss control
performance and noise analysis in Section 4. Finally, some conclusions are
drawn from this study in Section 5.

##  2 Sliding Model Learning Control Structure

###  2.1 Control Scheme

Figure 1: Block diagram of the control scheme

The whole control structure consists of a conventional controller and a type-2
neuro-fuzzy controller (T2NFC) as illustrated in Fig. 1. The T2NFC controls
the system by learning the system behavior owing to the fact its parameters
are updated by a sliding mode learning algorithm. The conventional control
action is added to guarantee the stability of overall system in compact space
while T2NFC learns the system behavior. In this control structure, the purpose
is that the output of the conventional controller converges to zero in finite
time while T2NFC takes the overall control signal. The total control action
applied to the system $u$ is determined as follows:

$u=u_{c}+u_{n}$ (1)

where $u_{c}$ and $u_{n}$ denote respectively the conventional control signal
and the control signal generated by the T2NFC. The conventional control law is
formulated as follows:

$u_{c}=k\textrm{sgn}\left(s\right)$ (2)

where $s$ and $k$ denote the sliding surface and the controller gain,
respectively. The controller gain $k$ is positive, i.e., $k>0$, adaptive and
updated with the following rule

$\dot{k}=\frac{\gamma_{k}\mid s\mid}{2}$ (3)

where $\gamma_{k}$ is the coefficient of the adaption for the controller gain
and positive, i.e., $\gamma_{k}>0$.

###  2.2 Type-2 Neuro-Fuzzy Controller

An interval type-2 Takagi-Sugeno-Kang fuzzy _if-then_ rule $R_{ij}$ is
employed in this study Wang et al. (2017); Minjie et al. . The premise parts
are type-2 fuzzy functions while the consequent parts are crisp numbers. The
rule $R_{ij}$ is defined as below:

$R_{ij}:\;\;\textrm{If}\;e\;\textrm{is}\;\;\widetilde{1}_{i}\;\;\textrm{and}\;\dot{e}\;\textrm{is}\;\;\widetilde{2}_{j},\;\;\textrm{then}\;f_{ij}=d_{ij}$
(4)

where $e$ and $\dot{e}$ denote respectively the error and the error rate while
$\widetilde{1}_{i}$ and $\widetilde{2}_{j}$ denote respectively type-2 fuzzy
sets for the input 1 and input 2. The function $f_{ij}$ is the output of the
rules and the total number of rules is equal to $K=I\times J$ in which $I$ and
$J$ denote respectively the numbers of membership functions used for the input
1 and input 2. The detailed of the proposed T2NFS is shown in Fig. 2.

Figure 2: Structure of the proposed T2NFS for two inputs

The lower and upper Gaussian MFs for T2FLSs are written as follows:

$\underline{\mu}_{1i}(e)=\exp\Bigg{(}-\bigg{(}\frac{e-\underline{c}_{1i}}{\underline{\sigma}_{1i}}\bigg{)}^{2}\Bigg{)}$
(5)
$\overline{\mu}_{1i}(e)=\exp\Bigg{(}-\bigg{(}\frac{e-\overline{c}_{1i}}{\overline{\sigma}_{1i}}\bigg{)}^{2}\Bigg{)}$
(6)
$\underline{\mu}_{2j}(\dot{e})=\exp\Bigg{(}-\bigg{(}\frac{\dot{e}-\underline{c}_{2j}}{\underline{\sigma}_{2j}}\bigg{)}^{2}\Bigg{)}$
(7)
$\overline{\mu}_{2j}(\dot{e})=\exp\Bigg{(}-\bigg{(}\frac{\dot{e}-\overline{c}_{2j}}{\overline{\sigma}_{2j}}\bigg{)}^{2}\Bigg{)}$
(8)

where $\underline{c}$ and $\overline{c}$ denote respectively the lower and
upper means the membership functions, and $\underline{\sigma}$ and
$\overline{\sigma}$ the lower and upper standard deviations of the membership
functions. They are adjusted for the T2NFC method throughout the simulations.

The lower and upper memberships $\underline{\mu}$ and $\overline{\mu}$ of
fuzzy system are written for each input signal being fed to the system. The
firing strengths of rules are formulated as follows:

$\underline{w}_{ij}=\underline{\mu}_{1i}(e)\underline{\mu}_{2j}(\dot{e})\quad\textrm{and}\quad\overline{w}_{ij}=\overline{\mu}_{1i}(e)\overline{\mu}_{2j}(\dot{e})$
(9)

The control signal generated by the network is formulated below Biglarbegian
et al. (2010):

$u_{n}=q\sum_{i=1}^{I}\sum_{j=1}^{J}f_{ij}\widetilde{\underline{w}}_{ij}+(1-q)\sum_{i=1}^{I}\sum_{j=1}^{J}f_{ij}\widetilde{\overline{w}}_{ij}$
(10)

where $\widetilde{\underline{{w}}}_{ij}$ and $\widetilde{\overline{{w}}}_{ij}$
are the normalized firing strengths of the lower and upper output signals of
the neuron $ij$ and written as follows:

$\displaystyle\widetilde{\underline{w}}_{ij}=\frac{\underline{w}_{ij}}{\sum_{i=1}^{I}\sum_{j=1}^{J}\underline{w}_{ij}}\quad\textrm{and}\quad\widetilde{\overline{w}}_{ij}=\frac{\overline{w}_{ij}}{\sum_{i=1}^{I}\sum_{j=1}^{J}\overline{w}_{ij}}$
(11)

The parameter $q$ determines the contributions of the lower and upper firing
levels to the control signal and the adaptation rule for the parameter $q$ is
given Section 2.3.

###  2.3 Sliding Mode Learning Algorithm

The sliding surface utilized the fundamental basis of SMC theory Utkin (1992)
is written as follows:

$s=\Big{(}\frac{d}{dt}+\lambda\Big{)}^{n-1}e$ (12)

where $\lambda$, $e$ and $n$ denote respectively the slope of the sliding
surface, the system error and the order of the system to be controlled. The
slope of the sliding surface is a strictly positive constant, i.e.,
$\lambda>0$. The error is the difference between a desired setpoint $x_{d}$
and a measured state $x$, i.e., $e=x_{d}-x$. The sliding surface is utilized
to train the sliding mode learning algorithm. The T2NFC parameters are updated
with the following equations:

$\dot{\underline{c}}_{1i}=\dot{e}+(e-\underline{c}_{1i})\alpha\textrm{sgn}\left(s\right)$
(13)
$\dot{\overline{c}}_{1i}=\dot{e}+(e-\overline{c}_{1i})\alpha\textrm{sgn}\left(s\right)$
(14)
$\dot{\underline{c}}_{2j}=\ddot{e}+(\dot{e}-\underline{c}_{2j})\alpha\textrm{sgn}\left(s\right)$
(15)
$\dot{\overline{c}}_{2j}=\ddot{e}+(\dot{e}-\overline{c}_{2j})\alpha\textrm{sgn}\left(s\right)$
(16)
$\dot{\underline{\sigma}}_{1i}=-\underline{\sigma}_{1i}\bigg{(}1+\Big{(}\frac{\underline{\sigma}_{1i}}{e-\underline{c}_{1i}}\Big{)}^{2}\bigg{)}\alpha\textrm{sgn}\left(s\right)$
(17)
$\dot{\overline{\sigma}}_{1i}=-\overline{\sigma}_{1i}\bigg{(}1+\Big{(}\frac{\overline{\sigma}_{1i}}{e-\overline{c}_{1i}}\Big{)}^{2}\bigg{)}\alpha\textrm{sgn}\left(s\right)$
(18)
$\dot{\underline{\sigma}}_{2j}=-\underline{\sigma}_{2j}\bigg{(}1+\Big{(}\frac{\underline{\sigma}_{2j}}{\dot{e}-\underline{c}_{2j}}\Big{)}^{2}\bigg{)}\alpha\textrm{sgn}\left(s\right)$
(19)
$\dot{\overline{\sigma}}_{2j}=-\overline{\sigma}_{2j}\bigg{(}1+\Big{(}\frac{\overline{\sigma}_{2j}}{\dot{e}-\overline{c}_{2j}}\Big{)}^{2}\bigg{)}\alpha\textrm{sgn}\left(s\right)$
(20)
$\dot{f}_{ij}=\frac{q\widetilde{\underline{w}}_{ij}+(1-q)\widetilde{\overline{w}}_{ij}}{(q\widetilde{\underline{W}}+(1-q)\widetilde{\overline{W}})^{T}(q\widetilde{\underline{W}}+(1-q)\widetilde{\overline{W}})}\alpha\textrm{sgn}\left(s\right)$
(21)
$\dot{q}=\frac{1}{F(\widetilde{\underline{W}}-\widetilde{\overline{W}})^{T}}\alpha\textrm{sgn}\left(s\right)$
(22) $\dot{\alpha}=\gamma_{\alpha}\mid s\mid$ (23)

where $\alpha$ is the learning rate while $\gamma_{\alpha}$ is the coefficient
to update the learning rate, and they are positive, i.e.,
$\alpha,\gamma_{\alpha}>0$. Normalized firing strengths vectors
$\widetilde{\underline{W}}(t)$ and $\widetilde{\overline{W}}(t)$, and the
consequent part vector $F$ are defined as follows:

$\displaystyle\widetilde{\underline{W}}$ $\displaystyle=$
$\displaystyle[\widetilde{\underline{w}}_{11}\;\widetilde{\underline{w}}_{12}\dots\widetilde{\underline{w}}_{21}\dots\widetilde{\underline{w}}_{ij}\dots\widetilde{\underline{w}}_{IJ}]^{T}$
$\displaystyle\widetilde{\overline{W}}$ $\displaystyle=$
$\displaystyle[\widetilde{\overline{w}}_{11}\;\widetilde{\overline{w}}_{12}\dots\widetilde{\overline{w}}_{21}\dots\widetilde{\overline{w}}_{ij}\dots\widetilde{\overline{w}}_{IJ}]^{T}$
$\displaystyle F$ $\displaystyle=$ $\displaystyle[f_{11}\;f_{12}\dots
f_{21}\dots f_{ij}\dots f_{IJ}]$

where $0<\widetilde{\underline{w}}_{ij}\leq 1$ and
$0<\widetilde{\overline{w}}_{ij}\leq 1$. Moreover
$\sum_{i=1}^{I}\sum_{j=1}^{J}\widetilde{\underline{w}}_{ij}=1$ and
$\sum_{i=1}^{I}\sum_{j=1}^{J}\widetilde{\overline{w}}_{ij}=1$.

######  Assumption 1.

The input signal and its time derivative are assumed to be bounded in a
compact set:

$\mid u\mid<B_{u},\qquad\mid\dot{u}\mid<B_{\dot{u}}$ (24)

where $B_{u}$ and $B_{\dot{u}}$ are considered as positive constants.

######  Theorem 1 (Stability of sliding mode learning algorithm).

If adaptations rules are proposed as in (13)-(23), the final value of the
learning rate $\alpha^{*}$ is large enough, i.e., $\alpha^{*}>B_{\dot{u}}$,
and the term, which is the product of the controller gain $k$ and its final
value $k^{*}$, is larger than the absolute value of the sliding surface $\mid
s\mid$, i.e., $k^{*}k>\mid s\mid$, this ensures that the output of the
conventional controller $u_{c}$ will converge to zero in finite time.

###### Proof.

The stability of the learning algorithm is checked by using the following
Lyapunov candidate function:

$V=\frac{1}{2k^{*}}u_{c}^{2}+\frac{(k^{*})^{2}}{2\gamma_{\alpha}}\Big{(}\frac{\alpha}{(k^{*})^{2}}-\alpha^{*}\Big{)}^{2}$
(25)

The time-derivative of the aforementioned Lyapunov candidate function is
calculated as follows:

$\dot{V}=\frac{u_{c}\dot{u}_{c}}{k^{*}}+\frac{\dot{\alpha}}{\gamma_{\alpha}}\Big{(}\frac{\alpha}{(k^{*})^{2}}-\alpha^{*}\Big{)}$
(26)

If (1) and (23) are inserted into (26), it is obtained as:

$\dot{V}=\frac{u_{c}}{k^{*}}(-\dot{u}_{n}+\dot{u})+\mid
s\mid\Big{(}\frac{\alpha}{(k^{*})^{2}}-\alpha^{*}\Big{)}$ (27)

$\dot{u}_{n}$ in (57) is inserted into (27), the time-derivative of the
Lyapunov function is written as:

$\dot{V}=\frac{u_{c}}{k^{*}}\Big{(}-2\alpha\textrm{sgn}\left(s\right)+\dot{u}\Big{)}+\mid
s\mid\Big{(}\frac{\alpha}{(k^{*})^{2}}-\alpha^{*}\Big{)}$ (28)

As it is stated in Assumption 1, the total input rate $\dot{u}$ is upper
bounded by $B_{\dot{u}}$, and the conventional control action in (2) is
inserted into (28)

$\displaystyle\dot{V}$ $\displaystyle<$ $\displaystyle\frac{-2\alpha
k}{k^{*}}+\frac{k}{k^{*}}\mid s\mid B_{\dot{u}}+\frac{\alpha\mid
s\mid}{(k^{*})^{2}}-\alpha^{*}\mid s\mid$ $\displaystyle\dot{V}$
$\displaystyle<$ $\displaystyle\alpha\underbrace{\Big{(}\frac{-2kk^{*}+\mid
s\mid}{(k^{*})^{2}}\Big{)}}_{<0}+\mid
s\mid\underbrace{\Big{(}\frac{k}{k^{*}}B_{\dot{u}}-\alpha^{*}\Big{)}}_{<0}$
(29)

where $\alpha^{*}$ and $k^{*}$ are the final values of the learning rate and
controller gain, and considered to be unknown parameters that are determined
throughout the adaptations of learning rate and controller gain. If the final
value of the learning rate $\alpha^{*}$ is large enough, i.e.,
$\alpha^{*}>B_{\dot{u}}$, and the term, which is the product of the controller
gain $k$ and its final value $k^{*}$, is larger than the absolute value of the
sliding surface $\mid s\mid$, i.e., $k^{*}k>\mid s\mid$, then the time-
derivative of the Lyapunov candidate function is negative, i.e., $\dot{V}<0$.
This purports that the sliding mode learning algorithm is stable. In other
words, the output of the conventional controller $u_{c}$ converges to zero
after the T2NFC algorithm learns the system. ∎

######  Remark 1.

As can be seen in (3), the controller gain only increases throughout
simulations. Therefore, the final value of the controller $k^{*}$ is equal or
bigger than $k$. This results in $\frac{k}{k^{*}}\leq 1$.

######  Remark 2.

The adaptation rules for the conventional controller gain in (3) and learning
rate in (23) are enforced; therefore, the final values of the controller gain
and learning rate are unknown and are determined throughout the adaptations of
the controller gain and learning rate, and they can reach at large values to
make system and sliding mode learning algorithm stable.

##  3 Stability Analysis of the Overall System

An nth-order uncertain nonlinear system is described in the following form:

$\displaystyle\dot{x}_{1}$ $\displaystyle=$ $\displaystyle x_{2}$
$\displaystyle\vdots$ $\displaystyle\vdots$ $\displaystyle\vdots$
$\displaystyle\dot{x}_{n-1}$ $\displaystyle=$ $\displaystyle x_{n}$
$\displaystyle\dot{x}_{n}$ $\displaystyle=$ $\displaystyle f(\textbf{x})+gu+d$
(30)

where $f(\textbf{x})$ is the nonlinear function, $g$ is the coefficient of the
control input and positive, $u$ is the control input, $d$ is the disturbance,
and $\textbf{x}=[x_{1},x_{2},\ldots,x_{n}]^{T}$ is the state vector. The error
$e$ is defined as $e=x_{d}-x_{1}$ where $x_{d}$ is the desired state.

######  Assumption 2.

The nth-order time-derivative of the desired state, the nonlinear function,
the disturbance, the output of the T2NFC and the term $E$, which is equal to
$\dot{s}-\frac{d^{n}e}{dt^{n}}$, are assumed to be bounded:

$\mid\frac{d^{n}x_{d}}{dt^{n}}\mid+\mid f(\textbf{x})\mid+\mid d\mid+g\mid
u_{n}\mid+\mid E\mid<B$ (31)

where $B$ is considered as a positive constant.

######  Theorem 2 (Stability of the overall system).

The system to be controlled is the form of (3) and the controller in (1) is
applied to the system. Then, if the final value of the controller gain $k^{*}$
is large enough, i.e., $k^{*}>2B$, the sliding surface $s$ converges to zero
for an arbitrary initial condition.

###### Proof.

The stability of the overall system is checked for an nth-order uncertain
nonlinear system by following Lyapunov candidate function:

$V=\frac{1}{2}s^{2}+\frac{g}{2\gamma_{k}}(k-k^{*})^{2}$ (32)

The time-derivative of $V$ is calculated as:

$\dot{V}=s\dot{s}+\frac{g\dot{k}}{\gamma_{k}}(k-k^{*})$ (33)

The sliding surface rate $\dot{s}$ is obtained by taking time-derivative of
the sliding surface defined in (12). Then, it is inserted into (33) as
follows:

$\dot{V}=s\Big{(}\frac{d^{n}e}{dt^{n}}\underbrace{+\ldots+\lambda^{n-1}\dot{e}\Big{)}}_{E}+\frac{g\dot{k}}{\gamma_{k}}(k-k^{*})$
(34)

Since $E=\dot{s}-\frac{d^{n}e}{dt^{n}}$ as stated in Assumption 2 and
$\frac{d^{n}e}{dt^{n}}=\frac{d^{n}x_{d}}{dt^{n}}-\frac{d^{n}x}{dt^{n}}$ , it
is obtained as

$\displaystyle\dot{V}=s\Big{(}\frac{d^{n}x_{d}}{dt^{n}}-\frac{d^{n}x}{dt^{n}}+E\Big{)}+\frac{g\dot{k}}{\gamma_{k}}(k-k^{*})$
(35)

Then, it is re-written by taking nonlinear system in (3) into consideration as
follows:

$\displaystyle\dot{V}=s\Big{(}\frac{d^{n}x_{d}}{dt^{n}}-f(\textbf{x})-d-gu+E\Big{)}+\frac{g\dot{k}}{\gamma_{k}}(k-k^{*})$
(36)

The control law in (1) and the adaptation for the controller gain in (3) are
inserted into (36).

$\dot{V}=s\Big{(}\frac{d^{n}x_{d}}{dt^{n}}-f(\textbf{x})-d-gu_{n}-gu_{c}+E\Big{)}+\frac{g\mid
s\mid}{2}(k-k^{*})$ (37)

The conventional control action in (2) is inserted into (37)

$\dot{V}=s\Big{(}\frac{d^{n}x_{d}}{dt^{n}}-f(\textbf{x})-d-gu_{n}+E-gk\textrm{sgn}\left(s\right)\Big{)}+\frac{g\mid
s\mid}{2}(k-k^{*})$ (38)

As stated in Assumption 2, the term $E$, the nth-order time-derivative of the
reference $\frac{d^{n}x_{d}}{dt^{n}}$, the function $f(\textbf{x})$, the
disturbance $d$ and the output of the T2NFC $u_{n}$ are assumed to be upper
bounded, the aforementioned equation (38) is obtained as follows:

$\dot{V}<\mid s\mid(B-gk)+\frac{g\mid s\mid}{2}(k-k^{*})$ (39)

where $k^{*}$ is the final value of the controller gain and considered to be
an unknown parameter that is determined throughout the adaptation of the
controller gain. If the final value of the controller gain $k^{*}$ is large
enough, i.e., $k^{*}>2B$, $\dot{V}$ yields

$\dot{V}<-\frac{gk\mid s\mid}{2}$ (40)

Since $g,k>0$, this guarantees the stability of the overall system and
purports that the sliding surface converges to zero in finite time. ∎

######  Remark 3.

Since the controller gain can reach at a large value to make the system
stable, it is not required to know the upper bound. This is one of the
advantages of the SMLC algorithm.

######  Remark 4.

A sliding motion appears on the sliding manifold $s=0$ after finite time
$t_{h}$ if the condition $s\dot{s}<0$ is satisfied for all $t$ in some
nontrivial semi-open sub-interval of time of the form
$[t,t_{h})\subset(-\infty,t_{h})$. Therefore, a sliding motion will be
maintained on $s=0$ for all $t>t_{h}$.

##  4 Simulation Results

The performance of the developed SMLC algorithm against an external
disturbance is firstly shown in Section 4.1 and then analyzed under noisy
conditions in Section 4.2. Throughout simulation studies, the number of
membership functions of the T2NFC are set to 3, i.e., $I=J=3$ and the sampling
time is set to $0.01$ second, Since SMC theory suffers from high-frequency
oscillations, the following function in (41) is used as the sign function to
remove the chattering effect:

$\textrm{sgn}\left(s\right):=\frac{s}{|s|+\chi}$ (41)

with $\chi=0.05$.

To avoid division by zero in the adaptation laws of (13)-(22), an instruction
is included in the algorithm to make the denominator equal to 0.001 when its
calculated value is smaller than this threshold. Moreover, the controller gain
of the conventional controller and learning rate of the learning algorithm
must not have infinite values under noisy conditions; therefore, a dead-zone
is employed. If the sliding surface is within the bounds of the dead-zone
parameter $\epsilon=0.001$, i.e., $\mid s\mid<\epsilon$, then these parameters
are not updated. Furthermore, the reference must be changing smoothly and
there must exist no discontinuous disturbance to avoid infinite values for the
controller gain and learning rate.

###  4.1 Scenerio 1: Control Performance Against External Disturbance

To evaluate control performance against disturbances, the developed SMLC
algorithm is applied to an adaptive cruise control system, which is
illustrated in Fig. 3 and the control structure is shown in Fig. 4. The
equations of motion for the longitudinal vehicle dynamics are written as
follows Stankovic et al. (2000); Kayacan (2017)

$\displaystyle\dot{x}_{1}$ $\displaystyle=$ $\displaystyle x_{2}$
$\displaystyle\dot{x}_{2}$ $\displaystyle=$ $\displaystyle x_{3}$
$\displaystyle\dot{x}_{3}$ $\displaystyle=$
$\displaystyle-2\frac{k_{a}}{m}x_{2}x_{3}-\frac{1}{\tau}\Big{[}x_{3}+\frac{k_{a}}{m}x_{2}^{2}\Big{]}+\frac{u}{m\tau}+d$
(42)

where $x_{1}$, $x_{2}$, $x_{3}$, m, $k_{a}$, $\tau$, u and d represent
respectively the position, velocity and acceleration, the mass of the vehicle,
the aerodynamic drag coefficient, the engine time-lag, the throttle command
input and the external disturbance. The numerical values for the parameters in
this study are $m=9$ $kg$, $k_{a}=0.26$, $\tau=0.1$ Mehra et al. (2015).

Figure 3: Schematic diagram of the adaptive cruise control system
Figure 4: Control structure of the adaptive cruise control system

The time-headway spacing policy to follow the preceding vehicle with the
desired relative distance is formulated as follows:

$z=hx_{2}$ (43)

where z is the desired relative inter-vehicle distance and h is the desired
time headway. The spacing policy is called the constant time-headway time
policy which aims a constant inter-vehicle time gap. The position reference in
this study is considered as the position of the preceding vehicle. The
relative spacing error is defined by taking the spacing policy in (43) into
account as follows:

$e=x_{d}-x_{1}-hx_{2}$ (44)

The sliding surface for the adaptive cruise control system is second-order
since the adaptive cruise control systems is a third-order system. The slope
of the sliding surface $\lambda$ to update the T2NFC is set to $1$; therefore,
the sliding surface becomes as $s=\ddot{e}+2\dot{e}+e$ as formulated in (12).
The coefficients $\gamma_{k}$ and $\gamma_{\alpha}$ for the adaptions of the
controller gain in (3) and the learning rate in (23) are set to $0.1$. The
initial conditions on the states of the system are considered as
$[x_{1},x_{2},,x_{3}]=[0,0,0]^{T}$. An external disturbance is imposed in the
vehicle dynamics as follows:

$d(t)=1+0.25\sin{(t)}$ (45)

The test trajectory is defined by the position reference $x_{d}(t)$ consists
of ramp signals as follows:

$x_{d}(t)=\Bigg{\\{}\begin{array}[]{rl}0\leq t<20&x_{d}=t\\\ 20\leq
t<40&x_{d}=t+\frac{0.05}{2}(t-20)^{2}\\\ 40\leq
t<40&x_{d}=t+\frac{0.05}{2}(t-20)^{2}-\frac{0.05}{2}(t-40)^{2}\end{array}$
(46)

The position, speed and acceleration responses of the vehicle are respectively
shown in Figs. 5(a)-5(c). The developed SMLC algorithm can learn the system
behavior online in finite time so that it exhibits robust control performance
against the external disturbance. As the error is shown in Fig. 5(d), it
controls the system without any steady-state error.

The control signals are shown in Fig. 5(e). The generated control signal by
the conventional control action $u_{c}$ converges zero while the generated
control signal by the T2NFC $u_{n}$ takes overall control signal. Thus, the
T2NFC becomes the leading controller after a short time period. The output of
the conventional controller $u_{c}$ becomes non-zero only at the beginning of
the simulations while the T2NFC is learning system behaviour.

The adaptations of the controller gain $k$, the learning rate $\alpha$ and the
parameter $q$ are presented in Figs. 5(f)-5(h). The parameter $q$ is generally
set to $0.5$ in T2FLCs while the controller gain and learning rate are chosen
arbitrarily in this study. Therefore, the initial conditions of these
parameters are set to $k(0)=1$, $\alpha(0)=3$ and $q(0)=0.5$. Thanks to these
adaptations, the controller gain, the learning rate and the proportions of the
upper and lower membership functions are updated throughout the simulations.
The adaptations of the controller gain $k$ and learning rate $\alpha$ converge
to a certain value since (i) the reference is changing smoothly, (ii) there is
no discontinuous disturbance on the system and (iii) the dead-zone parameter
is not lower than noise-level in real-time applications.

In literature, disturbance observers-based control approaches have been
generally implemented for uncertain systems Liu et al. (2017b). In these
approaches, unmodeled dynamics are considered as an external disturbance, and
a control law including disturbance estimate is derived. In our work, there is
no need of such an observer due to the fact that the proposed algorithm learns
system behavior. This is one of the superiorities of the developed algorithm
in the paper.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure 5: Control performance on an adaptive cruise control system: (a)
Position (b) Speed (c) Acceleration (d) Error (e) Control signals (f)
Adaptation of the controller gain (g) Adaptation of the learning rate (g)
Adaptation of the parameter q

###  4.2 Scenerio 2: Noise Analysis

To evaluate control performance under noisy condition, the developed SMLC
algorithm is applied to the following numerical system Yang et al. (2013)

$\displaystyle\dot{x}_{1}$ $\displaystyle=$ $\displaystyle x_{2}$
$\displaystyle\dot{x}_{2}$ $\displaystyle=$
$\displaystyle-2x_{1}-x_{2}+e^{x_{1}}+u$ (47)

The initial conditions on the system states are set to $x(0)=[1,-1]^{T}$. The
slope of the sliding surface $\lambda$ is set to $2$. The coefficients
$\gamma_{k}$ and $\gamma_{\alpha}$ for the adaptions of the controller gain in
(3) and the learning rate in (23) are respectively set to $1$ and $0.1$.

In order to test the robustness of this control approach, the states $x_{1}$
and $x_{2}$ are measured with a noise level signal-to-noise ratio (SNR) = $50$
$dB$. The states’ responses are shown in Figs. 6(a) and 6(b). The T2NFC can
learn system behavior online and thus regulates the system without any steady-
state error. Thanks to the proposed T2NFC structure, our control algorithm
ensures robust control performance against noisy measurements.

The control signals are shown in Fig. 6(c). As seen in this figure, the
generated control signal by the robust control action $u_{c}$ converges zero
while the generated control signal by the T2NFC $u_{n}$ is different from
zero. It is concluded that the T2NFC can take the overall control of the
system in a very short time period.

The adaptations of the controller gain $k$, the learning rate $\alpha$ and the
parameter $q$ are presented in Figs. 6(d)-6(f). The initial conditions of
these parameters are considered as $k(0)=1$, $\alpha(0)=0.03$ and $q(0)=0.5$.
Thanks to these adaptations, the controller gain, the learning rate and the
proportions of the upper and lower membership functions are updated throughout
the simulations. Moreover, the controller gain converges to a certain value
under noisy conditions owing to the dead-zone.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 6: Control performance under noisy measurements: (a) State $x_{1}$ (b)
State $x_{2}$ (c) Control signals (d) Adaptation of the controller gain (e)
Adaptation of the learning rate (f) Adaptation of the parameter q

##  5 Conclusion

The developed SMLC algorithm for uncertain nonlinear systems has been
investigated in this paper. In addition to the stability of the training
algorithm, the analytical proof of the overall system stability has been
achieved for nth-order uncertain nonlinear systems. The simulation results
show that the developed control structure exhibits robust control performance
in the presence of external disturbances and noisy measurements. The
parameters of the T2NFC controller are automatically regulated through the
sliding mode learning algorithm so that T2NFC can learn the system behavior
and take the overall control of systems in a very short time period. The
proposed sliding mode learning algorithm is valid for only single-input
systems. As a future work, the extension of the proposed learning algorithm
for multi-input systems is an interesting topic to be investigated.

##  Appendix A The time-derivative of the output of the T2NFC $\dot{u}_{n}$

The following equations are obtained by taking the time derivative of (5)-(8):

$\displaystyle\dot{\underline{\mu}}_{1i}(x_{1})=-2\underline{N}_{1i}\dot{\underline{N}}_{1i}\underline{\mu}_{1i}(x_{1}),\quad\dot{\overline{\mu}}_{1i}(x_{1})=-2\overline{N}_{1i}\dot{\overline{N}}_{1i}\overline{\mu}_{1i}(x_{1})$
(48)
$\displaystyle\dot{\underline{\mu}}_{2j}(x_{2})=-2\underline{N}_{2j}\dot{\underline{N}}_{2j}\underline{\mu}_{2j}(x_{2}),\quad\dot{\overline{\mu}}_{2j}(x_{2})=-2\overline{N}_{2j}\dot{\overline{N}}_{2j}\overline{\mu}_{2j}(x_{2})$
(49)

where
$\underline{N}_{1i}=\Big{(}\frac{x_{1}-\underline{c}_{1i}}{\underline{\sigma}_{1i}}\Big{)}$,
$\overline{N}_{1i}=\Big{(}\frac{x_{1}-\overline{c}_{1i}}{\overline{\sigma}_{1i}}\Big{)}$,
$\underline{N}_{2j}=\Big{(}\frac{x_{2}-\underline{c}_{2j}}{\underline{\sigma}_{2j}}\Big{)}$,
and
$\overline{N}_{2j}=\Big{(}\frac{x_{2}-\overline{c}_{2j}}{\overline{\sigma}_{2j}}\Big{)}$.

By taking the time derivative of (11), the following equations are obtained as
follows:

$\displaystyle\dot{\widetilde{\underline{w}}}_{ij}$ $\displaystyle=$
$\displaystyle\frac{\Big{(}\underline{\mu}_{1i}(x_{1})\underline{\mu}_{2j}(x_{2})\Big{)}^{\prime}\Big{(}\sum_{i=1}^{I}\sum_{j=1}^{J}\underline{w}_{ij}\Big{)}}{(\sum_{i=1}^{I}\sum_{j=1}^{J}\underline{w}_{ij})^{2}}-\frac{\big{(}\underline{w}_{ij}\big{)}\Big{(}\sum_{i=1}^{I}\sum_{j=1}^{J}\underline{\mu}_{1i}(x_{1})\underline{\mu}_{2j}(x_{2})\Big{)}^{\prime}}{(\sum_{i=1}^{I}\sum_{j=1}^{J}\underline{w}_{ij})^{2}}$
$\displaystyle\dot{\widetilde{\overline{w}}}_{ij}$ $\displaystyle=$
$\displaystyle\frac{\Big{(}\overline{\mu}_{1i}(x_{1})\overline{\mu}_{2j}(x_{2})\Big{)}^{\prime}\Big{(}\sum_{i=1}^{I}\sum_{j=1}^{J}\overline{w}_{ij}\Big{)}}{(\sum_{i=1}^{I}\sum_{j=1}^{J}\overline{w}_{ij})^{2}}-\frac{\big{(}\overline{w}_{ij}\big{)}\Big{(}\sum_{i=1}^{I}\sum_{j=1}^{J}\overline{\mu}_{1i}(x_{1})\overline{\mu}_{2j}(x_{2})\Big{)}^{\prime}}{(\sum_{i=1}^{I}\sum_{j=1}^{J}\overline{w}_{ij})^{2}}$
(50)

Since
$\widetilde{\underline{w}}_{ij}=\frac{\underline{w}_{ij}}{\sum_{i=1}^{I}\sum_{j=1}^{J}\underline{w}_{ij}}$,

$\displaystyle\dot{\widetilde{\underline{w}}}_{ij}$ $\displaystyle=$
$\displaystyle\frac{\Big{(}\dot{\underline{\mu}}_{1i}(x_{1})\underline{\mu}_{2j}(x_{2})+\underline{\mu}_{1i}(x_{1})\dot{\underline{\mu}}_{2j}(x_{2})\Big{)}}{(\sum_{i=1}^{I}\sum_{j=1}^{J}\underline{w}_{ij})}-\frac{(\widetilde{\underline{w}}_{ij})\bigg{(}\sum_{i=1}^{I}\sum_{j=1}^{J}\Big{(}\dot{\underline{\mu}}_{1i}(x_{1})\underline{\mu}_{2j}(x_{2})+\underline{\mu}_{1i}(x_{1})\dot{\underline{\mu}}_{2j}(x_{2})\Big{)}\bigg{)}}{(\sum_{i=1}^{I}\sum_{j=1}^{J}\underline{w}_{ij})}$
(51) $\displaystyle=$
$\displaystyle\frac{\Big{(}-2\underline{N}_{1i}\dot{\underline{N}}_{1i}\underline{\mu}_{1i}(x_{1})\underline{\mu}_{2j}(x_{2})-2\underline{N}_{2j}\dot{\underline{N}}_{2j}\underline{\mu}_{1i}(x_{1})\underline{\mu}_{2j}(x_{2})\Big{)}}{(\sum_{i=1}^{I}\sum_{j=1}^{J}\underline{w}_{ij})}-\frac{(\widetilde{\underline{w}}_{ij})\sum_{i=1}^{I}\sum_{j=1}^{J}\Big{(}-2\underline{N}_{1i}\dot{\underline{N}}_{1i}\underline{\mu}_{1i}(x_{1})\underline{\mu}_{2j}(x_{2})\Big{)}}{(\sum_{i=1}^{I}\sum_{j=1}^{J}\underline{w}_{ij})}$
$\displaystyle-\frac{(\widetilde{\underline{w}}_{ij})\sum_{i=1}^{I}\sum_{j=1}^{J}\Big{(}-2\underline{N}_{2j}\dot{\underline{N}}_{2j}\underline{\mu}_{1i}(x_{1})\underline{\mu}_{2j}(x_{2})\Big{)}}{(\sum_{i=1}^{I}\sum_{j=1}^{J}\underline{w}_{ij})}$
$\displaystyle=$
$\displaystyle-\widetilde{\underline{w}}_{ij}\dot{\underline{K}}_{ij}+\widetilde{\underline{w}}_{ij}\sum_{i=1}^{I}\sum_{j=1}^{J}\widetilde{\underline{w}}_{ij}\dot{\underline{K}}_{ij}$

where

$\dot{\underline{K}}_{ij}=2\Big{(}\underline{N}_{1i}\dot{\underline{N}}_{1i}+\underline{N}_{2j}\dot{\underline{N}_{2j}}\Big{)}$

In a similar way, $\dot{\widetilde{\overline{w}}}_{ij}$ obtained:

$\dot{\widetilde{\overline{w}}}_{ij}=-\widetilde{\overline{w}}_{ij}\dot{\overline{K}}_{ij}+\widetilde{\overline{w}}_{ij}\sum_{i=1}^{I}\sum_{j=1}^{J}\widetilde{\overline{w}}_{ij}\dot{\overline{K}}_{ij}$
(52)

where

$\dot{\overline{K}}_{ij}=2\Big{(}\overline{N}_{1i}\dot{\overline{N}}_{1i}+\overline{N}_{2j}\dot{\overline{N}}_{2j}\Big{)}$

The time derivative of the output of the T2NFC algorithm is obtained as:

$\displaystyle\dot{u}_{n}$ $\displaystyle=$ $\displaystyle
q\sum_{i=1}^{I}\sum_{j=1}^{J}(\dot{f}_{ij}\widetilde{\underline{w}}_{ij}+f_{ij}\dot{\widetilde{\underline{w}}}_{ij})+(1-q)\sum_{i=1}^{I}\sum_{j=1}^{J}(\dot{f}_{ij}\widetilde{\overline{w}}_{ij}+f_{ij}\dot{\widetilde{\overline{w}}}_{ij})+\dot{q}\sum_{i=1}^{I}\sum_{j=1}^{J}f_{ij}\widetilde{\underline{w}}_{ij}-\dot{q}\sum_{i=1}^{I}\sum_{j=1}^{J}f_{ij}\widetilde{\overline{w}}_{ij}$
(53)

If (51) and (52) are inserted to the aforementioned equation:

$\displaystyle\dot{u}_{n}$ $\displaystyle=$ $\displaystyle
q\sum_{i=1}^{I}\sum_{j=1}^{J}\bigg{(}\Big{(}-\widetilde{\underline{w}}_{ij}\dot{\underline{K}}_{ij}+\widetilde{\underline{w}}_{ij}\sum_{i=1}^{I}\sum_{j=1}^{J}\widetilde{\underline{w}}_{ij}\dot{\underline{K}}_{ij}\Big{)}f_{ij}+\widetilde{\underline{w}}_{ij}\dot{f}_{ij}\bigg{)}$
(54)
$\displaystyle+(1-q)\sum_{i=1}^{I}\sum_{j=1}^{J}\bigg{(}\Big{(}-\widetilde{\overline{w}}_{ij}\dot{\overline{K}}_{ij}+\widetilde{\overline{w}}_{ij}\sum_{i=1}^{I}\sum_{j=1}^{J}\widetilde{\overline{w}}_{ij}\dot{\overline{K}}_{ij}\Big{)}f_{ij}+\widetilde{\overline{w}}_{ij}\dot{f}_{ij}\bigg{)}+\dot{q}\sum_{i=1}^{I}\sum_{j=1}^{J}f_{ij}(\widetilde{\underline{w}}_{ij}-\widetilde{\overline{w}}_{ij})$
$\displaystyle=$ $\displaystyle
q\sum_{i=1}^{I}\sum_{j=1}^{J}\bigg{(}\Big{(}-\widetilde{\underline{w}}_{ij}\big{(}2\underline{N}_{1i}\dot{\underline{N}}_{1i}+2\underline{N}_{2j}\dot{\underline{N}}_{2j}\big{)}+\widetilde{\underline{w}}_{ij}\sum_{i=1}^{I}\sum_{j=1}^{J}\widetilde{\underline{w}}_{ij}\big{(}2\underline{N}_{1i}\dot{\underline{N}}_{1i}+2\underline{N}_{2j}\dot{\underline{N}}_{2j}\big{)}\Big{)}f_{ij}+\widetilde{\underline{w}}_{ij}\dot{f}_{ij}\bigg{)}$
$\displaystyle+(1-q)\sum_{i=1}^{I}\sum_{j=1}^{J}\bigg{(}\Big{(}-\widetilde{\overline{w}}_{ij}\big{(}2\overline{N}_{1i}\dot{\overline{N}}_{1i}+2\overline{N}_{2j}\dot{\overline{N}}_{2j}\big{)}+\widetilde{\overline{w}}_{ij}\sum_{i=1}^{I}\sum_{j=1}^{J}\widetilde{\overline{w}}_{ij}\big{(}2\overline{N}_{1i}\dot{\overline{N}}_{1i}+2\overline{N}_{2j}\dot{\overline{N}}_{2j}\big{)}\Big{)}f_{ij}+\widetilde{\overline{w}}_{ij}\dot{f}_{ij}\bigg{)}$
$\displaystyle+\dot{q}\sum_{i=1}^{I}\sum_{j=1}^{J}f_{ij}(\widetilde{\underline{w}}_{ij}-\widetilde{\overline{w}}_{ij})$

where

$\displaystyle\dot{\underline{N}}_{1i}=\frac{(\dot{x_{1}}-\dot{\underline{c}}_{{1i}})\underline{\sigma}_{1i}-(x_{1}-\underline{c}_{1i})\dot{\underline{\sigma}}_{1i}}{\underline{\sigma}^{2}_{1i}},\quad\dot{\underline{N}}_{2i}=\frac{(\dot{x_{2}}-\dot{\underline{c}}_{{2i}})\underline{\sigma}_{2i}-(x_{2}-\underline{c}_{2i})\dot{\underline{\sigma}}_{2i}}{\underline{\sigma}^{2}_{2i}}$
$\displaystyle\dot{\overline{N}}_{1i}=\frac{(\dot{x_{1}}-\dot{\overline{c}}_{{1i}})\overline{\sigma}_{1i}-(x_{1}-\overline{c}_{1i})\dot{\overline{\sigma}}_{1i}}{\overline{\sigma}^{2}_{1i}},\quad\dot{\overline{N}}_{2i}=\frac{(\dot{x_{2}}-\dot{\overline{c}}_{{2i}})\overline{\sigma}_{2i}-(x_{2}-\overline{c}_{2i})\dot{\overline{\sigma}}_{2i}}{\overline{\sigma}^{2}_{2i}}$

(55) can be obtained as follows:

$\underline{N}_{1i}\dot{\underline{N}}_{1i}=\underline{N}_{2i}\dot{\underline{N}}_{2i}=\overline{N}_{1i}\dot{\overline{N}}_{1i}=\overline{N}_{2i}\dot{\overline{N}}_{2i}=\alpha
sgn{(s)}$ (55)

(55) is inserted to (54):

$\displaystyle\dot{u}_{n}$ $\displaystyle=$ $\displaystyle
q\sum_{i=1}^{I}\sum_{j=1}^{J}\bigg{(}2\Big{(}-\widetilde{\underline{w}}_{ij}f_{ij}2\alpha
sgn{(s)}+\widetilde{\underline{w}}_{ij}f_{ij}\sum_{i=1}^{I}\sum_{j=1}^{J}\widetilde{\underline{w}}_{ij}2\alpha
sgn{(s)}\Big{)}+\widetilde{\underline{w}}_{ij}\dot{f}_{ij}\bigg{)}$ (56)
$\displaystyle+(1-q)\sum_{i=1}^{I}\sum_{j=1}^{J}\bigg{(}2\Big{(}-\widetilde{\overline{w}}_{ij}f_{ij}2\alpha
sgn{(s)}+\widetilde{\overline{w}}_{ij}f_{ij}\sum_{i=1}^{I}\sum_{j=1}^{J}\widetilde{\overline{w}}_{ij}2\alpha
sgn{(s)}\Big{)}\widetilde{\overline{w}}_{ij}\dot{f}_{ij}\bigg{)}+\dot{q}\sum_{i=1}^{I}\sum_{j=1}^{J}f_{ij}(\widetilde{\underline{w}}_{ij}-\widetilde{\overline{w}}_{ij})$
$\displaystyle=$ $\displaystyle-4q\alpha
sgn{(s)}\sum_{i=1}^{I}\sum_{j=1}^{J}\Big{(}\widetilde{\underline{w}}_{ij}f_{ij}-\widetilde{\underline{w}}_{ij}f_{ij}\sum_{i=1}^{I}\sum_{j=1}^{J}\widetilde{\underline{w}}_{ij}\Big{)}-4(1-q)\alpha
sgn{(s)}\sum_{i=1}^{I}\sum_{j=1}^{J}\Big{(}\widetilde{\overline{w}}_{ij}f_{ij}-\widetilde{\overline{w}}_{ij}f_{ij}\sum_{i=1}^{I}\sum_{j=1}^{J}\widetilde{\overline{w}}_{ij}\Big{)}$
$\displaystyle+q\sum_{i=1}^{I}\sum_{j=1}^{J}\widetilde{\underline{w}}_{ij}\dot{f}_{ij}+(1-q)\sum_{i=1}^{I}\sum_{j=1}^{J}\widetilde{\overline{w}}_{ij}\dot{f}_{ij}+\dot{q}\sum_{i=1}^{I}\sum_{j=1}^{J}f_{ij}(\widetilde{\underline{w}}_{ij}-\widetilde{\overline{w}}_{ij})$

Since $\sum_{i=1}^{I}\sum_{j=1}^{J}\widetilde{\overline{w}}_{ij}=1$ and
$\sum_{i=1}^{I}\sum_{j=1}^{J}\widetilde{\underline{w}}_{ij}=1$, $\dot{u}_{n}$
becomes by using (21) and (22) as below:

$\displaystyle\dot{u}_{n}$ $\displaystyle=$ $\displaystyle
q\sum_{i=1}^{I}\sum_{j=1}^{J}\widetilde{\underline{w}}_{ij}\dot{f}_{ij}+(1-q)\sum_{i=1}^{I}\sum_{j=1}^{J}\widetilde{\overline{w}}_{ij}\dot{f}_{ij}+\dot{q}\sum_{i=1}^{I}\sum_{j=1}^{J}f_{ij}(\widetilde{\underline{w}}_{ij}-\widetilde{\overline{w}}_{ij})$
(57) $\displaystyle=$ $\displaystyle 2\alpha sgn{(s)}$

## References

  * Biglarbegian et al. (2010) Biglarbegian M, Melek WW and Mendel JM (2010) On the stability of interval type-2 tsk fuzzy logic control systems.  _IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)_ 40(3): 798–818.
  * Castillo et al. (2014) Castillo O, Castro JR, Melin P and Rodriguez-Diaz A (2014) Application of interval type-2 fuzzy neural networks in non-linear identification and time series prediction.  _Soft Computing_ 18(6): 1213–1224.
  * Castillo et al. (2012) Castillo O, Martínez-Marroquín R, Melin P, Valdez F and Soria J (2012) Comparative study of bio-inspired algorithms applied to the optimization of type-1 and type-2 fuzzy controllers for an autonomous mobile robot.  _Information Sciences_ 192: 19 – 38.
  * Chen et al. (2016) Chen WH, Yang J, Guo L and Li S (2016) Disturbance-observer-based control and related methods: An overview.  _IEEE Transactions on Industrial Electronics_ 63(2): 1083–1095.
  * Efe et al. (2000) Efe MO, Kaynak O and Yu X (2000) Sliding mode control of a three degrees of freedom anthropoid robot by driving the controller parameters to an equivalent regime.  _ASME Journal of Dynamic Systems, Measurement, and Control_ 122(4): 632 –640.
  * Feng (2006) Feng G (2006) A survey on analysis and design of model-based fuzzy control systems.  _IEEE Transactions on Fuzzy Systems_ 14(5): 676–697.
  * Gomi and Kawato (1993) Gomi H and Kawato M (1993) Neural network control for a closed-loop system using feedback-error-learning.  _Neural Networks_ 6(7): 933 – 946.
  * Huang et al. (2015) Huang J, Ri S, Liu L, Wang Y, Kim J and Pak G (2015) Nonlinear disturbance observer-based dynamic surface control of mobile wheeled inverted pendulum.  _IEEE Transactions on Control Systems Technology_ 23(6): 2400–2407.
  * Kalanovic et al. (2000) Kalanovic VD, Popovic D and Skaug NT (2000) Feedback error learning neural network for trans-femoral prosthesis.  _IEEE Transactions on Rehabilitation Engineering_ 8(1): 71–80.
  * Kayacan (2017) Kayacan E (2017) Multiobjective $h_{\infty}$ control for string stability of cooperative adaptive cruise control systems.  _IEEE Transactions on Intelligent Vehicles_ 2(1): 52–61.
  * Kayacan and Fossen (2019) Kayacan E and Fossen TI (2019) Feedback linearization control for systems with mismatched uncertainties via disturbance observers.  _Asian Journal of Control_ 21(4): 1–13.  10.1002/asjc.1802.
  * Kayacan et al. (2018) Kayacan E, Kayacan E, Chen IM, Ramon H and Saeys W (2018) _On the Comparison of Model-Based and Model-Free Controllers in Guidance, Navigation and Control of Agricultural Vehicles_.  Cham: Springer International Publishing, pp. 49–73.
  * Kayacan et al. (2015a) Kayacan E, Kayacan E and Khanesar M (2015a) Identification of nonlinear dynamic systems using type-2 fuzzy neural networks - a novel learning algorithm and a comparative study.  _IEEE Transactions on Industrial Electronics_ 62(3): 1716–1724.
  * Kayacan et al. (2015b) Kayacan E, Kayacan E, Ramon H, Kaynak O and Saeys W (2015b) Towards agrobots: Trajectory control of an autonomous tractor using type-2 fuzzy logic controllers.  _Mechatronics, IEEE/ASME Transactions on_ 20(1): 287–298.
  * Kayacan et al. (2012) Kayacan E, Kayacan E, Ramon H and Saeys W (2012) Velocity control of a spherical rolling robot using a grey-pid type fuzzy controller with an adaptive step size.  _IFAC Proceedings Volumes_ 45(22): 863 – 868.  10th IFAC Symposium on Robot Control.
  * Kayacan et al. (2013a) Kayacan E, Kayacan E, Ramon H and Saeys W (2013a) Adaptive neuro-fuzzy control of a spherical rolling robot using sliding-mode-control-theory-based online learning algorithm.  _IEEE Transactions on Cybernetics_ 43(1): 170–179.
  * Kayacan et al. (2013b) Kayacan E, Kayacan E, Ramon H and Saeys W (2013b) Modeling and identification of the yaw dynamics of an autonomous tractor.  In: _2013 9th Asian Control Conference (ASCC)_. pp. 1–6.
  * Kayacan et al. (2014) Kayacan E, Kayacan E, Ramon H and Saeys W (2014) Nonlinear modeling and identification of an autonomous tractor-trailer system.  _Computers and Electronics in Agriculture_ 106(Supplement C): 1 – 10.
  * Kayacan and Peschel (2016) Kayacan E and Peschel J (2016) Robust model predictive control of systems by modeling mismatched uncertainty.  _IFAC-PapersOnLine_ 49(18): 265 – 269.  10th IFAC Symposium on Nonlinear Control Systems NOLCOS 2016.
  * Kayacan et al. (2017) Kayacan E, Peschel JM and Chowdhary G (2017) A self-learning disturbance observer for nonlinear systems in feedback-error learning scheme.  _Engineering Applications of Artificial Intelligence_ 62(Supplement C): 276 – 285.
  * Kayacan et al. (2016) Kayacan E, Peschel JM and Kayacan E (2016) Centralized, decentralized and distributed nonlinear model predictive control of a tractor-trailer system: A comparative study.  In: _2016 American Control Conference (ACC)_. pp. 4403–4408.
  * Khanesar et al. (2015) Khanesar M, Kayacan E, Reyhanoglu M and Kaynak O (2015) Feedback error learning control of magnetic satellites using type-2 fuzzy neural networks with elliptic membership functions.  _IEEE Transactions on Cybernetics_ 45(4): 858–868.
  * Lam (2018) Lam H (2018) A review on stability analysis of continuous-time fuzzy-model-based control systems: From membership-function-independent to membership-function-dependent analysis.  _Engineering Applications of Artificial Intelligence_ 67(Supplement C): 390 – 408.
  * Lee et al. (2015) Lee CS, Wang MH, Wu MJ, Teytaud O and Yen SJ (2015) T2fs-based adaptive linguistic assessment system for semantic analysis and human performance evaluation on game of go.  _IEEE Transactions on Fuzzy Systems_ 23(2): 400–420.
  * Liu et al. (2017a) Liu H, Zhao W, Zuo Z and Zhong Y (2017a) Robust control for quadrotors with multiple time-varying uncertainties and delays.  _IEEE Transactions on Industrial Electronics_ 64(2): 1303–1312.
  * Liu et al. (2017b) Liu J, Vazquez S, Wu L, Marquez A, Gao H and Franquelo LG (2017b) Extended state observer-based sliding-mode control for three-phase power converters.  _IEEE Transactions on Industrial Electronics_ 64(1): 22–31.
  * Liu et al. (2017c) Liu J, Wu C, Wang Z and Wu L (2017c) Reliable filter design for sensor networks using type-2 fuzzy framework.  _IEEE Transactions on Industrial Informatics_ 13(4): 1742–1752.
  * Maldonado et al. (2013) Maldonado Y, Castillo O and Melin P (2013) Particle swarm optimization of interval type-2 fuzzy systems for FPGA applications.  _Applied Soft Computing_ 13(1): 496 – 508.
  * Mayne et al. (2000) Mayne D, Rawlings J, Rao C and Scokaert P (2000) Constrained model predictive control: Stability and optimality.  _Automatica_ 36(6): 789 – 814.
  * Mehra et al. (2015) Mehra A, Ma WL, Berg F, Tabuada P, Grizzle JW and Ames AD (2015) Adaptive cruise control: Experimental validation of advanced controllers on scale-model cars.  In: _2015 American Control Conference (ACC)_. pp. 1411–1418.
  * Minjie et al. (0) Minjie Z, Yujie Z, Shenhua Y and Lina L (0) Sampled-data control for nonlinear singular systems based on a takagi–sugeno fuzzy model.  _Transactions of the Institute of Measurement and Control_ 0(0): 0142331217740420\.
  * Muhuri and Shukla (2017) Muhuri PK and Shukla AK (2017) Semi-elliptic membership function: Representation, generation, operations, defuzzification, ranking and its application to the real-time task scheduling problem.  _Engineering Applications of Artificial Intelligence_ 60(Supplement C): 71 – 82.
  * Ren et al. (2017) Ren B, Zhong QC and Dai J (2017) Asymptotic reference tracking and disturbance rejection of ude-based robust control.  _IEEE Transactions on Industrial Electronics_ 64(4): 3166–3176.
  * Ruan et al. (2007) Ruan X, Ding M, Gong D and Qiao J (2007) On-line adaptive control for inverted pendulum balancing based on feedback-error-learning.  _Neurocomputing_ 70(4–6): 770 – 776.
  * Rubio-Solis and Panoutsos (2015) Rubio-Solis A and Panoutsos G (2015) Interval type-2 radial basis function neural network: A modeling framework.  _IEEE Transactions on Fuzzy Systems_ 23(2): 457–473.
  * Stankovic et al. (2000) Stankovic SS, Stanojevic MJ and Siljak DD (2000) Decentralized overlapping control of a platoon of vehicles.  _IEEE Transactions on Control Systems Technology_ 8(5): 816–832.
  * Sun et al. (2016) Sun Z, Zheng J, Man Z and Wang H (2016) Robust control of a vehicle steer-by-wire system using adaptive sliding mode.  _IEEE Transactions on Industrial Electronics_ 63(4): 2251–2262.
  * Topalov et al. (2007a) Topalov AV, Cascella GL, Giordano V, Cupertino F and Kaynak O (2007a) Sliding mode neuro-adaptive control of electric drives.  _IEEE Transactions on Industrial Electronics_ 54(1): 671–679.
  * Topalov et al. (2007b) Topalov AV, Kaynak O and Aydin G (2007b) Neuro-adaptive sliding-mode tracking control of robot manipulators.  _International Journal of Adaptive Control and Signal Processing_ 21(8-9): 674–691.
  * Utkin (1992) Utkin VI (1992) _Sliding Modes in Control Optimization_.  Springer-Verlag.
  * Wang et al. (2017) Wang W, Chen H, Zhang M, Liu X, Zhang Z and Sun Y (2017) Application of takagi–sugeno fuzzy model optimized with an improved free search algorithm to industrial polypropylene melt index prediction.  _Transactions of the Institute of Measurement and Control_ 39(11): 1613–1622.
  * Yang et al. (2013) Yang J, Li S and Yu X (2013) Sliding-mode control for systems with mismatched uncertainties via a disturbance observer.  _IEEE Transactions on Industrial Electronics_ 60(1): 160–169.
  * Yin et al. (2017a) Yin S, Gao H, Qiu J and Kaynak O (2017a) Descriptor reduced-order sliding mode observers design for switched systems with sensor and actuator faults.  _Automatica_ 76: 282 – 292.
  * Yin et al. (2017b) Yin S, Yang H and Kaynak O (2017b) Sliding mode observer-based ftc for markovian jump systems with actuator and sensor faults.  _IEEE Transactions on Automatic Control_ 62(7): 3551–3558.
  * Zaheer et al. (2015) Zaheer SA, Choi SH, Jung CY and Kim JH (2015) A modular implementation scheme for nonsingleton type-2 fuzzy logic systems with input uncertainties.  _IEEE/ASME Transactions on Mechatronics_ 20(6): 3182–3193.


# Multi-Modal Deep Learning for Multi-Temporal Urban Mapping with a Partly
Missing Optical Modality

###### Abstract

This paper proposes a novel multi-temporal urban mapping approach using multi-
modal satellite data from the Sentinel-1 Synthetic Aperture Radar (SAR) and
Sentinel-2 MultiSpectral Instrument (MSI) missions. In particular, it focuses
on the problem of a partly missing optical modality due to clouds. The
proposed model utilizes two networks to extract features from each modality
separately. In addition, a reconstruction network is utilized to approximate
the optical features based on the SAR data in case of a missing optical
modality. Our experiments on a multi-temporal urban mapping dataset with
Sentinel-1 SAR and Sentinel-2 MSI data demonstrate that the proposed method
outperforms a multi-modal approach that uses zero values as a replacement for
missing optical data, as well as a uni-modal SAR-based approach. Therefore,
the proposed method is effective in exploiting multi-modal data, if available,
but it also retains its effectiveness in case the optical modality is missing.

Index Terms—  Sentinel-1 SAR, Sentinel-2 MSI, data fusion, missing modality,
urban

##  1 INTRODUCTION

Multi-modal deep learning offers new opportunities for timely and accurate
urban mapping and change detection by exploiting the complementary information
acquired by Synthetic Aperture Radar (SAR) and optical sensors. In particular,
the Copernicus Program’s Sentinel-1 (S1) SAR and Sentinel-2 (S2) MultiSpectral
Instrument (MSI) missions are playing a key role in multi-modal remote sensing
research. For example, our previous work demonstrated that the complementary
information in S1 SAR and S2 MSI data can be utilized to improve the
transferability of deep learning models for urban extraction at a global scale
[1].

However, while the availability of multi-modal satellite data from S1 and S2
is a valid assumption for uni-temporal mapping, for multi-temporal urban
mapping and change detection, the optical modality may not always be available
due to cloud cover or other atmospheric conditions. Surprisingly, multi-modal
research in the remote sensing domain has paid little attention to the missing
modality problem, with few exceptions. For example, Saha et al. [2] considered
the case of urban change detection with both optical and SAR images available
at $t_{1}$, but only SAR images at $t_{2}$. The proposed Siamese network,
accounting for the missing optical modality at $t_{2}$, outperformed a network
using only SAR data. On the other hand, Zheng et al. [3] proposed a prototype
network that learns a meta-sensory representation for all-weather building
mapping. The prototype network is capable of dynamically generating sensor-
specific networks that can be trained on multi-modal data. For SAR and optical
data, the respective sensor-specific network generated by the prototype
network outperformed networks that were directly trained on either modality.
Recently, Li et al. [4] proposed an efficient dense adaptive grouping
distillation network to effectively tackle multi-modal deep learning when one
of the modalities is completely missing for inference.

Although these methods proved to be effective in tackling the missing modality
problem, they are focusing on the case when a modality is completely missing
for inference. In practice, however, a modality is often only partly missing.
Therefore, it is desirable to develop methods that are capable of exploiting
multi-modal data for inference, if both modalities are available. Bischke et
al. [5] addressed this by generating missing depth images from optical images
using Generative Adversarial Networks (GANs). However, there exist major
challenges in the training of GANs, i.e., mode collapse, non-convergence, and
instability. Moreover, SAR-to-optical image translation may be limited by the
scene complexity and textural information [6].

In this paper, we propose a multi-temporal urban mapping model with the
capability to exploit the complementary information in SAR and optical data,
while also being able to exploit uni-modal SAR data, in case the optical
modality is missing. The model consists of two networks to extract features
from the SAR and optical input separately, in addition to a third network that
reconstructs the optical features from the SAR input in order to cope with a
missing optical modality. The effectiveness of the proposed model is
demonstrated on the multi-temporal urban mapping problem posed by the
SpaceNet7 dataset [7] using S1 SAR and S2 MSI data.

##  2 Methodology

###  2.1 Satellite Data Preparation

The SpaceNet 7 dataset [7] was leveraged to generate a multi-temporal urban
mapping dataset featuring S1 SAR and S2 MSI images. It contains monthly time
series of Planet imagery acquired between 2017 and 2020 and corresponding
manually annotated building footprints for 60 distinct sites across the globe.
S1 SAR images (VV + VH polarization at 20 m spatial resolution) and S2 MSI
images (blue, green, red, and near-infrared at 10 m spatial resolution) were
acquired for each of the approximately 24 timestamps per study site. The
monthly S1 SAR image and S2 MSI image correspond to the temporal mean of all
acquired S1 scenes within that month and the least cloudy S2 scene,
respectively. For an in-depth description of the data preprocessing, we refer
to [8], Finally, the images were resampled to the Ground Sampling Distance of
the respective Planet images (GDS of approx. 4 m). Fig. 1 exemplifies the
prepared data triplets for a section of timestamps in a time series. It should
be noted that the optical modality of the second timestamp ($t_{2}$) is
considered missing due to clouds. The prepared data is available on
Zenodo111https://doi.org/10.5281/zenodo.7794693.

Fig. 1: Data triples consisting of satellite data from S1 SAR and S2 MSI in
addition to building labels ($\\{x_{\rm S1},x_{\rm S2},y\\}$) for a selection
of timestamps $t$ in a time series of length $n$.

###  2.2 Proposed Approach

The proposed approach is illustrated in Fig. 2. The model consists of two
networks with identical architectures to separately extract multi-channel
feature maps from the S1 SAR image ($F_{\rm S1}$) and the S2 MSI image
($F_{\rm S2}$). A prediction is then obtained from the concatenated feature
maps in a late fusion manner. The popular encoder-decoder network U-Net [9] is
used as underlying architecture for the two networks. To cope with missing S2
data, a feature reconstruction network is introduced. The reconstruction
network uses the S1 image to approximate the multi-channel feature map
extracted from the S2 image. The architecture of the reconstruction network is
also based on U-Net.

Fig. 2: Overview of the proposed approach for multi-modal urban extraction
with a partly missing optical modality.

The loss function differentiates between two cases: (1) the multi-modal case
where both the S1 image and S2 image are available, and (2) the missing
modality case where only the S1 image is available. In the multi-modal case, a
prediction $p_{\rm S1S2}$ is obtained from the fused feature maps extracted
from the S1 image and S2 image. The prediction is then used to compute a
supervised loss term using label $y$. A second prediction $p_{\rm S1}$ is
obtained from simulating a missing optical modality by replacing the feature
map extracted from the S2 image ($F_{\rm S2}$) with its approximation
($\hat{F}_{\rm S2}$) produced by the reconstruction network from the S1 image.
Accordingly, a second supervised loss term is computed based on $p_{\rm S1}$
and $y$. The third loss term is used to train the reconstruction network by
minimizing the difference between the feature map obtained from the S2 image
$F_{\rm S2}$ and its approximation $\hat{F}_{\rm S2}$ obtained from the
reconstruction network. In the missing modality case, the loss function
consists of a single supervised term. Having only the S1 image available, a
prediction $p_{S1}$ is obtained from the features extracted by the S1 network
and the approximated features obtained from the reconstruction network and
compared to the corresponding label.

A Jaccard-like loss function, namely Power Jaccard loss, was used for
supervised terms. In comparison to Jaccard loss, Power Jaccard accounts for
continuous values by replacing the intersection and union in Intersection over
Union (IoU) with product and sum, respectively, and it increases the weight of
wrong predictions by introducing exponents in the denominator. On the other
hand, L2 loss was used for the similarity term. It should be noted that the
impact of the similarity term on the loss is regulated by hyper-parameter
$\varphi$ in the multi-modal case.

During training, we use mini-batch gradient descent where a mini-batch can
consist of multi-modal samples and uni-modal samples with a missing optical
modality. Consequently, the cost for a mini-batch is computed by determining
the loss for each sample in the mini-batch separately, before adding them
together.

###  2.3 Experimental Setup

The 60 sites were split into a training, validation, and test set containing
41, 15, and 14 sites, respectively. The training set consists of 591 samples,
each corresponding to a unique timestamp of the 31 training sites. For each
epoch, training samples were generated dynamically by randomly cropping
patches of size 64 x 64 pixels from the images and labels. Furthermore, flips
and rotations were used as data augmentation to increase the diversity of the
training set. AdamW was employed as an optimizer with an initial learning rate
of $10^{-5}$. Hyper-parameter $\varphi$ was set to $10^{-2}$. Models were
trained with a batch size of 16 for 100 epochs, but early stopping (patience
10) was employed to avoid overfitting. Everything was implemented using
Facebook’s deep learning framework PyTorch.

The proposed model was compared to two baselines: 1) U-Net [9] trained on S1
data and 2) a Dual Stream (DS) U-Net that fuses extracted features from a
separate S1 and S2 network branch at the decision level, as proposed in [10]
for multi-modal urban change detection. Since the DS U-Net always requires two
input modalities, an array of all zeros was used as a replacement for the
optical modality in case only the SAR modality is available.

We used two common accuracy metrics for the quantitative accuracy assessment:
F1 score and IoU. Formulas for the metrics are given in Eq. 1, where TP, FP
and FN denote true positives, false positives, and false negatives,
respectively.

$F1=\frac{TP}{TP+\frac{1}{2}(FP+FN)}\;IoU=\frac{TP}{TP+FP+FN}$ (1)

##  3 Results

Results were obtained for each model by training it five times with different
seeds. Table 1 lists F1 score and IoU ($\mu\pm\sigma$) for the three models.
Column 1 summarizes the results for all of the test samples, Column 2 for only
the test samples with S1 and S2 images available, and Column 3 for only the
test samples with a missing modality. It should be noted that about 12 % of
the timestamps in the test set were affected by a missing optical modality.
Higher mean values for both accuracy metrics were obtained by DS U-Net which
uses multi-modal data if available compared to U-Net which exclusively uses S1
data. However, a closer look reveals that while the DS U-Net approach achieved
improvements over the U-Net approach on the multi-modal samples (+ 0.062 F1
and + 0.049 IoU), its performance on the samples with a missing optical
modality is considerably worse than that of the U-Net approach (- 0.115 F1 and
- 0.078 IoU). In contrast to the DS U-Net approach, the proposed approach
achieved better performance on the missing modality samples, although falling
slightly short of the U-Net approach (- 0.021 F1 and - 0.015 IoU). Moreover,
it clearly outperformed the DS U-Net and U-Net approaches on the multi-modal
samples. Therefore, the highest overall accuracy values — taking into account
multi-modal and missing modality samples — were obtained using the proposed
approach.

Table 1: Quantitative test result. Values represent $\mu\pm\sigma$ of 5 runs. The highest mean values are boldfaced. Method | All samples | Multi-modal samples | Missing modality samples
---|---|---|---
| F1 $\uparrow$ | IoU $\uparrow$ | F1 $\uparrow$ | IoU $\uparrow$ | F1 $\uparrow$ | IoU $\uparrow$
U-Net S1 | 0.362 $\pm$ 0.005 | 0.221 $\pm$ 0.003 | 0.364 $\pm$ 0.005 | 0.222 $\pm$ 0.004 |  0.348 $\pm$ 0.004 |  0.210 $\pm$ 0.003
DS U-Net | 0.411 $\pm$ 0.008 | 0.259 $\pm$ 0.006 | 0.426 $\pm$ 0.008 | 0.271 $\pm$ 0.006 | 0.233 $\pm$ 0.033 | 0.132 $\pm$ 0.021
Proposed |  0.423 $\pm$ 0.006 |  0.269 $\pm$ 0.005 |  0.435 $\pm$ 0.006 |  0.278 $\pm$ 0.005 | 0.327 $\pm$ 0.008 | 0.195 $\pm$ 0.006

Finally, it should be noted that the obtained accuracy values in Table 1 are
relatively low, especially compared to state-of-the-art urban mapping
approaches using multi-modal S1 SAR and S2 MSI data (e.g. [1]). However,
unlike in most land cover mapping studies including [1], model predictions
were assessed at a finer spatial resolution (i.e., approx. 4 m) than that of
the input imagery (i.e., 10 m and 20 m for S1 and S2 imagery, respectively).
This decision was made to maintain correspondence with the Planet imagery.
Another reason for the low accuracy values is that the SpaceNet-7 sites are
very challenging since many of them are located in arid and semi-arid areas
that are undergoing rapid urbanization.

##  4 Discussion

The quantitative results (Table 1) highlight that a multi-modal approach
dealing with a partly missing modality by replacing it with zero values
outperforms a uni-modal approach on an urban mapping dataset consisting of
predominately multi-modal samples (approx. 90 %). However, the results also
reveal that this way of dealing with missing modality samples leads to bad
performance on uni-modal samples. The proposed approach successfully mitigates
this effect by employing a separate network to reconstruct S2 features from S1
input images. While the proposed approach was deemed effective to deal with a
partly missing modality in this study, it should also be emphasized that its
performance on uni-modal samples is lower than that of a uni-modal network.

##  5 Conclusion

This paper addressed the problem of a partly missing optical modality during
inference time for multi-modal urban mapping from S1 and S2 data. To that end,
a model that utilizes a reconstruction network to approximate the features of
the optical modality when its missing was proposed. The model achieved
improved performance over a uni-modal approach trained on S1 SAR data and a
multi-modal approach using zero values in case of a missing optical modality.
Despite these improvements, our findings indicate that further research is
needed to improve the performance of missing modality samples.

## References

  * [1] Hafner, S. et al,  “Unsupervised domain adaptation for global urban extraction using Sentinel-1 SAR and Sentinel-2 MSI data,”  Remote Sensing of Environment, vol. 280, pp. 113192, 2022.
  * [2] Saha, S. et al,  “Supervised change detection using prechange optical-SAR and postchange SAR data,”  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 15, pp. 8170–8178, 2022.
  * [3] Zheng, Z. et al,  “Deep multisensor learning for missing-modality all-weather mapping,”  ISPRS Journal of Photogrammetry and Remote Sensing, vol. 174, pp. 254–264, 2021.
  * [4] Li, X. et al,  “Dense adaptive grouping distillation network for multimodal land cover classification with privileged modality,”  IEEE Transactions on Geoscience and Remote Sensing, 2022.
  * [5] Bischke, B. et al,  “Overcoming missing and incomplete modalities with generative adversarial networks for building footprint segmentation,”  in 2018 International Conference on Content-Based Multimedia Indexing (CBMI). IEEE, 2018, pp. 1–6.
  * [6] Zhao, Y. et al,  “A comparative analysis of GAN-based methods for SAR-to-optical image translation,”  IEEE Geoscience and Remote Sensing Letters, 2022.
  * [7] Van Etten, A. et al,  “The multi-temporal urban development SpaceNet dataset,”  arXiv preprint arXiv:2102.04420, 2021.
  * [8] Hafner, S.,  Multi-Modal Deep Learning with Sentinel-1 and Sentinel-2 Data for Urban Mapping and Change Detection,  Ph.D. thesis, KTH Royal Institute of Technology, 2022.
  * [9] Ronneberger, O. et al,  “U-net: Convolutional networks for biomedical image segmentation,”  in International Conference on Medical image computing and computer-assisted intervention. Springer, 2015, pp. 234–241.
  * [10] Hafner, S. et al,  “Sentinel-1 and Sentinel-2 data fusion for urban change detection using a dual stream U-Net,”  IEEE Geoscience and Remote Sensing Letters, vol. 19, pp. 1–5, 2022\.