# A Multi-task Learning Framework for Drone State Identification and
Trajectory Prediction

Antreas Palamas, Nicolas Souli, Tania Panayiotou, Panayiotis Kolios, Georgios
Ellinas KIOS Research and Innovation Center of Excellence (KIOS CoE) and
Department of Electrical and Computer Engineering, University of Cyprus.
E-mail:{palamas.antreas, nsouli02, panayiotou.tania, kolios.panayiotis,
gellinas} @ucy.ac.cy

###### Abstract

The rise of unmanned aerial vehicle (UAV) operations, as well as the
vulnerability of the UAVs’ sensors, has led to the need for proper monitoring
systems for detecting any abnormal behavior of the UAV. This work addresses
this problem by proposing an innovative multi-task learning framework (MLF-ST)
for UAV state identification and trajectory prediction, that aims to optimize
the performance of both tasks simultaneously. A deep neural network with
shared layers to extract features from the input data is employed, utilizing
drone sensor measurements and historical trajectory information. Moreover, a
novel loss function is proposed that combines the two objectives, encouraging
the network to jointly learn the features that are most useful for both tasks.
The proposed MLF-ST framework is evaluated on a large dataset of UAV flights,
illustrating that it is able to outperform various state-of-the-art baseline
techniques in terms of both state identification and trajectory prediction.
The evaluation of the proposed framework, using real-world data, demonstrates
that it can enable applications such as UAV-based surveillance and monitoring,
while also improving the safety and efficiency of UAV operations.

###### Index Terms:

Machine learning; multi-task learning; trajectory prediction; state
identification; UAV applications.

##  I Introduction

Unmanned aerial vehicles (UAVs) have gained immense popularity in recent
years, attracting attention from both the research community and the
industrial sector. This has resulted in the development of numerous hardware
and software modules specifically designed to meet the diverse needs of the
market. UAVs’ customizable features make them suitable for a wide range of
applications, including remote surveillance, search and rescue operations, and
autonomous deliveries. However, the numerous applications of UAVs (drones)
necessitate strict safety and security measures during missions. As with any
other machines, drones’ build-in hardware and software components are
susceptible to faults and malicious cyber-attacks that may disrupt their
normal operation. Thus, proper monitoring mechanisms must be in place to
detect any abnormal behavior.

One of the key tasks involved in the safety assessment and flight monitoring
of autonomous UAVs is to ensure that they follow a predetermined trajectory
defined in the flight/mission plan within the expected time [1]. Real-time
monitoring utilizes historical trajectory as a-priori knowledge to estimate
the drone’s current state, as well as to predict the remaining trajectory. The
historical trajectory is updated by GPS measurements once the UAV operation is
activated (i.e., the drone is in flight). The monitoring system is then
responsible to compute the safety index based on the distance from obstacles,
which relies heavily on the accuracy of predicting the remaining trajectory.
Hence, accurately and quickly obtaining the future flight trajectory of
autonomous UAVs is of great significance in reducing accidents and improving
UAV operation safety levels.

Clearly, both tasks are important for different types of drone applications.
For example, trajectory prediction is vital for applications such as search
and rescue, surveillance, and delivery, where accurate and reliable trajectory
prediction is essential. Additionally, in terms of current state
identification, this task provides the drone operator with the ability to
identify the drone’s state at any moment and especially in an environment
where there is no visibility. Moreover, this task is useful in the event that
the drone’s operator perceives that the flight is being performed normally,
while the drone is actually performing different movements/actions (e.g., due
to a malicious attack).

At present, there exists a plethora of systems that attempt to identify the
current state of a drone and predict its trajectory. The majority of these
systems employ classification algorithms to recognize the drone’s current
state, while deep neural networks are utilized for trajectory prediction. For
instance, certain studies have concentrated on solving multilabel
classification problems to accurately identify the drone’s current state [2].
Alternatively, other research efforts have used deep learning models, that
primarily rely on long short-term memory (LSTM) networks constructed by
recurrent neural networks (RNNs), to predict the drone’s trajectory for a
specific number of future time-steps [3].

In accordance, this work proposes a novel multi-task learning framework (MLF-
ST), where the two aforementioned tasks can be performed by training and
executing a single model. The proposed MLF-ST framework is trained with a
common input and has the ability to output the two-fold objective of state
identification and trajectory prediction. Specifically, the contributions of
this work are as follows:

  * •

A multi-task learning model is proposed that utilizes LSTM to integrate the
identification of a drone’s current state and the prediction of its future
trajectory for a given number of time-steps. The proposed model achieves this
by leveraging annotated time series data, collected from various sensors for
multiple drone flights in diverse outdoor environments and conditions. This
approach offers a robust solution to the complex challenge of simultaneously
performing multiple tasks during the operation of the drone.

  * •

A sliding window technique is implemented to segment the input data into
smaller subsets, providing the model with more contextual information about
the past. The approach aims to improve the accuracy of the model’s predictions
for the future, resulting in a robust and effective solution for data
processing.

In the rest of the paper, Section II outlines the state of the art, while the
methodology and the algorithm implementation of this work are described in
Sections III and IV, respectively. The description of the datasets and the
various experiments performed and results obtained are presented in Section V,
while the main conclusions along with future research directions are discussed
in Section VI.

##  II Related Work

In this work, a real-world dataset is utilized [4] to train and evaluate the
proposed multi-task learning framework for drone state identification and
trajectory prediction. The dataset includes readings from multiple sensors
mounted on a DJI Matrice 100 quadcopter, which were collected during various
missions with similar prescribed actions. This dataset was also employed in
previous studies [5, 6, 7] to train models for predicting UAV battery
consumption. For example, in [5], temporal convolutional networks were
employed to establish an energy consumption model that predicted the energy
consumption of a moving drone, in an effort to evaluate the probability of the
drone depleting its energy during a pre-planned mission. Moreover, work in [6]
developed a machine-learning algorithm that evaluated energy usage during
takeoff, cruise, and landing, utilizing distinct models for each one of the
three regimes in order to predict the drone’s total energy consumption during
a mission. Moreover, in [7], a comprehensive and precise model for energy
consumption prediction was established with the use of ensemble learning, and
combining the random forest and extreme gradient boosting machine learning
algorithms, demonstrating a mean absolute percentage error of approximately
$10\%$.

In another comparative study of two state-of-the-art supervised approaches
[8], long short-term memory (LSTM) and convolutional LSTM (ConvLSTM) were used
to identify potentially defective sensors while analyzing streamed data from a
UAV. The first approach involved using multiple LSTM networks to detect
anomalies in the continuous values of each attribute, while the second one
depended on a multi-output ConvLSTM network to identify abnormalities in the
values of the concerning attribute by considering the effect of all other
attributes. In addition, the authors proposed a method to address the
redundancy issue that arises from using multiple LSTM networks to analyze
multivariate and correlated data. A real-life dataset of four flights
employing a fixed-wing aircraft was used to conduct experiments, and both
approaches were able to detect different types of faults. However, it was
demonstrated that the multi-output ConvLSTM is faster and achieves better
results in most of the cases examined.

In terms of trajectory prediction, which is an active research topic in
various fields (i.e., transportation, security, and robotics), a brief
overview of related works that utilize multi-task learning models is provided
below. For example, a recent work [9], proposed a multi-task learning
framework for trajectory prediction, which jointly learned to predict the
future locations and directions of moving objects, utilizing both spatial and
temporal features to improve prediction accuracy. Moreover, the work in [10]
proposed an RNN-based model for trajectory prediction (i.e., for capturing the
temporal dependencies of trajectories) and used an LSTM network to model the
spatial features. Also, works in [11, 12] implemented two different models for
trajectory prediction; the first model was based on multimodal and multi-task
learning, utilizing various weather parameters to improve the prediction of
typhoon trajectories, while the second model was based on deep feature
representation, which extracted features from historical trajectories using a
deep neural network. Furthermore, using historical flight data [3] deployed an
LSTM-based model to predict the future locations of UAVs. Finally, in [13], a
deep multi-task learning framework for joint localization, perception, and
prediction of moving objects was presented. Specifically, a deep neural
network was utilized to learn these tasks jointly, while also improving the
accuracy of trajectory prediction.

This work complements the aforementioned research efforts by implementing a
novel multi-task learning framework that succeeds in simultaneously performing
state classification and trajectory prediction of a UAV agent utilizing
various data modalities and under different environmental conditions, aiming
to create an accurate system that can be successfully deployed in emergency
response situations. Further, contrary to other research efforts, the end goal
of MLF-ST is to extract an accurate result for both trajectory prediction and
current state identification without utilizing two different models for the
two tasks.

##  III Methodology

In this section, the methodology of the proposed framework is described in
detail. At first, the framework overview provides a comprehensive description
of the proposed approach, highlighting its key components and their
interactions. This is followed by a brief description of LSTM and the multi-
task learning techniques. Moreover, the multi-task learning section discusses
the rationale behind the use of this technique in the proposed model and how
it can improve the overall system performance.

###  III-A Framework Overview

The proposed MLF-ST framework (in Fig. 1) consists of a shared LSTM layer
followed by separate output layers for each task. The input to the model is a
window of historical drone sensor data with a fixed size ($WS$), which
includes measurements such as GPS coordinates, altitude, wind speed, wind
angle, velocity components of the ground speed, angular velocity components,
ground acceleration, and orientation. The shared LSTM layer has $256$ units
with ReLU activation, which enables the model to capture the temporal
dependencies within the input data. The output of the shared LSTM layer is
then fed into a separate LSTM layer with $128$ units and ReLU activation,
which is used for both tasks.

For the multilabel classification task, a batch normalization layer is applied
to the output of the LSTM layer, followed by a fully connected layer with $64$
units and no activation function. The final output layer for the
classification task is a sigmoid activation function that outputs a vector of
probabilities for each of the possible drone states such as HOVER-IDLE,
ASCENT, TURN, horizontal movement on a straight line (HMSL), and DESCENT.

For the trajectory prediction task, a time-distributed dense layer with $64$
units is applied to the output of the biLSTM layer, which generates a sequence
of predicted coordinates for each time-step in the forecast horizon (i.e., a
fixed number of time-steps into the future - typically ranging from $1$ to $3$
seconds). The final output layer for the trajectory prediction task is a
linear activation function, which outputs a vector of three values
representing the drone’s predicted $(x,y,z)$ coordinates at each time-step in
the forecast horizon.

During the training step, a weighted combination of the mean squared error
loss for the trajectory prediction task and the binary cross-entropy loss for
the classification task is minimized. The Adam optimizer with a learning rate
of $0.0001$ and a batch size of $64$ is employed and the proposed model is
trained for $100$ epochs. Furthermore, to circumvent overfitting, early
stopping criteria is implemented, ceasing the training process when no
discernible improvement is observed on the validation set. Finally, it is
worth to note that exhaustive experimentation, encompassing various hidden
layers and batch sizes has been conducted, with the selected hyperparameters
exhibiting superior performance in terms of minimizing prediction errors.

Figure 1: An overview of the proposed multi-task learning framework (MLF-ST)
methodology.

###  III-B Background on LSTM

LSTM is an example of an RNN architecture which is proficient in capturing
long-term dependencies in sequential data by managing the flow of information
through memory cells and gates [14]. The main building block of an LSTM
network is the LSTM cell which consists of three gates (input, output, and
forget) and a memory cell. In particular, the input gate controls the flow of
new information into the memory cell, the output gate is responsible for the
output of information from the memory cell, and the forget gate controls the
retention or deletion of information from the memory cell. The gates are
implemented using sigmoid functions which output values between $0$ and $1$,
and the memory cell uses a hyperbolic tangent function which outputs values
between $-1$ and $1$.

The equations for the LSTM cell can be expressed as:

$\displaystyle f_{t}$ $\displaystyle=\sigma(W_{f}x_{t}+U_{f}h_{t-1}+b_{f})$
$\displaystyle i_{t}$ $\displaystyle=\sigma(W_{i}x_{t}+U_{i}h_{t-1}+b_{i})$
$\displaystyle\tilde{C}t$ $\displaystyle=\tanh(W_{c}x_{t}+U_{c}h{t-1}+b_{c})$
$\displaystyle C_{t}$ $\displaystyle=f_{t}\odot C_{t-1}+i_{t}\odot\tilde{C}t$
$\displaystyle o_{t}$ $\displaystyle=\sigma(W_{o}x_{t}+U_{o}h{t-1}+b_{o})$
$\displaystyle h_{t}$ $\displaystyle=o_{t}\odot\tanh(C_{t})$

where $x_{t}$ is the input at time-step $t$, $h_{t-1}$ is the output from the
previous time-step, $\sigma$ denotes the sigmoid function,
$W_{f},W_{i},W_{c},W_{o}$ are weight matrices for the forget gate, input gate,
memory cell, and output gate, respectively, $U_{f},U_{i},U_{c},U_{o}$ are
weight matrices for the corresponding gates and memory cell from the previous
time-step, and $b_{f},b_{i},b_{o},b_{c}$ are bias vectors for the forget gate,
input gate, output gate, and memory cell, respectively. Further,
$f_{t},i_{t},\tilde{C}_{t}$ are defined as the values of the forget gate,
input gate, and candidate cell state at time-step $t$, $C_{t}$ is the cell
state at time-step $t$, $o_{t}$ is the value of the output gate at time-step
$t$, and $h_{t}$ is the output at time-step $t$. Finally, $\odot$ represents
element-wise multiplication.

LSTM networks have in general achieved considerable success in diverse fields
such as natural language processing [15], speech recognition [16], and image
captioning [17], demonstrating to be highly effective in modeling long-term
dependencies in sequential data, and outperforming traditional RNNs in several
tasks. However, LSTM networks tend to be computationally expensive and require
meticulous tuning of hyperparameters such as the number of LSTM cells and the
learning rate [18].

###  III-C Background on Multi-task Learning

Multi-task learning (MTL) is a technique that aims to improve the performance
of a model on multiple related tasks by jointly learning those tasks [19]. In
general, MTL has become increasingly popular due to its ability to improve
generalization and reduce overfitting [20].

MTL can be implemented using various techniques, including hard parameter
sharing, soft parameter sharing, and task-specific attention [20]. Hard
parameter sharing involves sharing the same set of parameters across all tasks
[19], while soft parameter sharing allows for task-specific parameters that
are encouraged to be similar to each other [20]. Task-specific attention on
the other hand is a technique that allows the model to selectively attend to
different parts of the input for each task [21].

MTL has been applied to different domains, including natural language
processing [22], computer vision [23], speech recognition [24], and drug
discovery [25]. In natural language processing, MTL has been employed to
jointly learn multiple tasks, such as named entity recognition and part-of-
speech tagging [22]. In computer vision, MTL is used for object detection,
segmentation, and classification [23], while in speech recognition, is
deployed to jointly learn multiple tasks, such as acoustic modeling and
language modeling [24]. In drug discovery, MTL has been utilized to predict
multiple properties of a molecule (e.g., toxicity, solubility, etc.) [25].

##  IV MLF-ST Algorithm

This section provides a detailed description of the MLF-ST algorithm (Alg. 1).
The MLF-ST algorithm is comprised of several stages, in an effort to enhance
its functionality: (i) Stage-1: data partitioning, where a sliding window
technique is employed to divide the data into subsets; (ii) Stage-2: a pre-
processing phase which includes standard normalization to prepare the data for
subsequent processing; (iii) Stage-3: data splitting which separates the data
into distinct sets for training, testing, and validation purposes; (iv)
Stage-4: the construction phase that builds the multi-task learning mode; (v)
Stage-5: a compilation phase, where the model is fine-tuned for optimal
efficacy; (vi) Stage-6: the fitting phase, that involves the training of the
model with the use of the previously generated data subsets; (vii) Stage-7:
the evaluation phase (that follows once training is finished), for assessing
the accuracy and performance of the model; (viii) Stage-8: the results
processing phase; this phase includes applying inverse transformations to the
scaled data, obtaining original data, redefining the testing set, and making
predictions. It also reshapes trajectory predictions, creates a confusion
matrix for current state identification, and constructs data frames containing
the actual and predicted positions ($(x,y,z)$ coordinates). Further, to refine
the results, the Haversine formula is incorporated to calculate the distance
between the actual and predicted positions ($x$ and $y$ coordinates),
providing distance measurements in meters. The Euclidean distance formula is
then employed to determine the error between the actual and predicted
positions by taking into account the differences in the $z$ coordinates.

Finally, the proposed algorithm’s output is visualized, showcasing the results
of both trajectory prediction and current state identification. This in-depth
analysis of the MLF-ST algorithm offers valuable insights into its functioning
and performance, enabling further optimization and application in various
multi-task learning scenarios.

The reader should note that Alg. 1 can also be seen as comprised of several
key procedures (as shown in the pseudocode), namely the sliding window
(Stage-1), model (Stages2-7), and results (Stage-8) procedures.

In the following section, experimental results are presented, demonstrating
the effectiveness of the proposed multi-task framework for drone state
identification and trajectory prediction.

Algorithm 1 MLF-ST Algorithm

Input: Acquired flight data

1:Define input size (IS), window size (WS) and horizon size (HS)

2:procedure (Sliding Window)

3: Define L1, L2, L3 as empty lists

4: for $i<IS-WS-HS+1$ do

5: append to L1 array Input from $i$ to WS$+i$

6: append to L2 array Input from $i+$WS to WS$+i+$HS

7: append to L3 array Input from $i+$WS to WS$+i+$HS

8: end for

9:end procedure

10:Reshape arrays L1 and L2 from 3D to 2D

11:Apply standard scaling on input data

12:Reshape arrays L1 and L2 from 2D to 3D

13:Split data into training, validation, and testing sets

14:procedure (Model)

15: Build the multi-task learning model

16: Define input and output of the model

17: Define target sets for training, testing, and validation

18: Define batch size, number of epochs, learning rate

19: Compilation of the model. Define the loss function for each task and
metrics.

20: Train the model

21: Evaluate the model

22:end procedure

23:procedure (Results)

24: Apply inverse transform to the scaled data

25: Get the original data

26: Define again testing set

27: Make predictions

28: Reshape trajectory predictions from 3D to 2D

29: Apply inverse transform to scaled predictions

30: Get the original values of predictions

31: Reshape again trajectory predictions from 2D to 3D

32: Make Confusion matrix for current state identification

33: Make data frames which contain the actual and predicted positions
$(x,y,z)$

34: Apply Haversine formula between actual and predicted positions

35: Get the distance of actual and predicted positions $(x,y)$ in meters

36: Apply Euclidean distance formula to get the error between actual and
predicted positions

37:end procedure

Output: Trajectory prediction and current state identification results

##  V Performance Evaluation

###  V-A Hardware and Software Configuration

For the performance evaluation of the proposed MLF-ST framework a UAV is
deployed in real-world experiments. Specifically, the autonomous drone is
equipped with an onboard processing unit, advanced sensors, flight control
systems, a solid-state weather station (TriSonica), and the NVIDIA Jetson
Xavier NX Developer Kit that is responsible for saving high-precision flight
data. Also, it is important to note that the data and commands are transmitted
through a 4G communication architecture (i.e., a 4G dongle is applied on the
moving agent). For the software implementation, the Tensorflow 2’s Keras API
[26] is used to build the multi-task learning model, while the scikit-learn
library [27] is employed to preprocess the time series data. Also, the NumPy
and Pandas Python libraries are used to analyze and visualize the results of
this work.

###  V-B Dataset Collection

This section details the sources, characteristics, and the way the data used
in the experiments was annotated, including the pre-processing steps taken
into account in order to ensure data quality and consistency. Specifically, in
this work two different datasets are utilized, i.e., the real-world datasets
provided by [4] (Dataset-1) and [28] (Dataset-2).

####  V-B1 Dataset-1: Data Collected with Package Delivery Quadcopter Drone

In particular, the first dataset utilized (Dataset-1) is obtained from [4],
which includes sensor data collected from a multirotor UAV during a variety of
flights. Specifically, the DJI Matrice 100 (M100) UAV was used to perform
autonomous takeoffs, transport payloads along a triangular flight pattern, and
land. The flights were conducted at different altitudes such as $25$m, $50$m,
$75$m, and $100$m, with speeds ranging from $4$m/s to $12$m/s, and payloads
ranging from $0$g, $250$g, $500$g. All specific flights were repeated at least
three times, for a total of $195$ flights. Specifically, this work focuses on
the $182$ flights that followed the R1 route as specified in [4], as these
flights demonstrated similar flight modes at each time-step. The sensor data
is synchronized at a frequency of $10$Hz and the specifics of each sensor are
outlined in Table I.

####  V-B2 Dataset-2: Dataset Obtained from Multi-modal Sensor Onboard the
Drone.

This dataset contains time-series data from numerous drone flights that took
place at the University of Cyprus (UCY) campus. Each flight record has a
unique identifier (ID) and a timestamp indicating when the flight occurred. In
particular, the drone’s position is represented by the coordinates $(x,y,z)$
and altitude and the orientation of the drone is described by the quaternion
$(o_{x},o_{y},o_{z},o_{w})$ (i.e., orientation x,y,z,w). The drone’s linear
and angular velocity are represented by $(v_{x},v_{y},v_{z})$ and
$(av_{x},av_{y},av_{z})$, respectively. Also, the linear acceleration of the
drone is defined as $(a_{x},a_{y},a_{z})$.

Furthermore, the dataset also contains information about the battery voltage
(BV), battery current (BC), and the payload attached (i.e., the devices
onboard the drone, such as the Nvidia Jetson, various sensors, weather
station, etc.).

Also, the dataset includes annotations for the current state of the drone,
including IDLE-HOVER, ASCEND, TURN, HMSL, and DESCEND. These states can be
used for classification to identify the current state of the drone. Thus, the
labeled dataset is used for predicting the trajectory of the drone using the
proposed MLF-ST framework.

For the annotation procedure, the change in position $(x,y,z)$ and yaw is
inspected. Specifically, if the position $(x,y)$ changes, it means that the
drone moves in a horizontal straight line, and if position $z$ changes, it
means that the drone performs an ascending or descending operation (depending
on whether it increases or decreases). Also, if the yaw changes, it means that
the drone performs a turn, and finally, if none of the above features changes,
then the drone is in idle or hover mode.

In addition to the features already mentioned, the dataset also includes
information from various sensors including a weather station and an inertial
measurement unit (IMU). The weather station provides information about the
weather conditions during the flight which includes wind speed and wind angle.
These weather variables are important factors that influence the flight of the
drone and battery consumption. Further, the IMU provides information on the
drone’s acceleration, angular velocity, and magnetic field. Specifically, the
accelerometer provides information about the drone’s linear acceleration, the
gyroscope provides information about the drone’s angular velocity, while the
magnetometer measures the earth’s magnetic field, which can be subsequently
used to determine the drone’s orientation.

Field experiments are performed in order to collect empirical data using a
specific type of UAV (DJI Matrice 300 (M300)). The M300 is equipped with
advanced sensors and flight control systems, providing high-precision flight
data. The flights are designed to cover a range of flight patterns, which
include triangular, square, polygonal, and random flight patterns.

Table I: Specifics of sensors used in Dataset-1. Features | Summary |  |  |
---|---|---|---|---
wind speed |  | airspeed provided by the
---
anemometer (in m/s)
|  |
wind angle |  | angle of the air flowing through
---
the anemometer with respect to
the north (in deg)
|  |
battery voltage (BV) |  | system voltage measured right
---
after the battery (in V)
|  |
battery current (BC) |  | system current measured right
---
after the battery (in A)
|  |
position $(x,y)$ |  | longitude & latitude of
---
the UAV (in deg)
|  |
position $z$ |  | altitude of the UAV with
---
respect to the sea-level (in m)
|  |
orientation $(o_{x},o_{y},o_{z},o_{w})$ | UAV orientation (in quartenions) |  |  |
velocity $(v_{x},v_{y},v_{z})$ |  | velocity components of ground
---
speed (in m/s)
|  |
angular velocity $(av_{x},av_{y},av_{z})$ |  | angular velocity components
---
(in rad/s)
|  |
linear acceleration $(a_{x},a_{y},a_{z})$ |  | ground acceleration in (m/s^2)
---
|  |
payload |  | mass of the payload
---
attached to the UAV (in g)
|  |

###  V-C Experimental Results

This section showcases the experimental results for the proposed MLF-ST
framework by comparing the localization performance for each estimated
trajectory with the ground truth (i.e., measurements acquired via GPS+IMU),
while also demonstrating the current state identification by performing
multilabel classification for each aforementioned dataset.

To investigate the average performance of the proposed MLF-ST framework
concerning Dataset-2, $10$ real outdoor experiments are conducted in the area
of the UCY campus. For the purposes of the experiments, the M300 drone is
utilized (specifications about the UAV agent utilized can be found in
https://www.dji.com/matrice-300).

Table II: Average Euclidean prediction error for Dataset-1. All testing flights | Prediction time [s] | Euclidean distance error [m]
---|---|---
t+1 | 2.5521
t+2 | 2.6637
t+3 | 3.4412

####  V-C1 Trajectory Prediction - Dataset-1

Table II tabulates the location estimation accuracy of the proposed MLF-ST
framework for Dataset-1. Even though the estimated location performance
decreases, as the time-step ($t$) increases, the proposed system is able to
provide accurate location estimation that is comparable to the ground truth
values (with Euclidean distance error lower than $3.5$m).

Further, Fig. 2 illustrates the cumulative distribution function (CDF) of the
forecast for time intervals $t+1$, $t+2$, and $t+3$. The analysis of the CDF
for prediction errors highlights that both the mean prediction errors and the
overall dispersion of errors are indeed considerably small.

Figure 2: Average error CDF for predicted trajectories at $t+1$, $t+2$, $t+3$
(Dataset-1).

Specifically, $90\%$ of the $1$-second look-ahead predictions manifest an
error magnitude less than $2.45$ meters, while the $2$-second look-ahead
accounts for $90\%$ of errors falling below $2.5$ meters. Also, the $3$-second
horizon encompasses $90\%$ of errors not exceeding $3.56$ meters.

####  V-C2 Current State Identification Results (Dataset-1)

In this work, regarding the second main task (current state identification),
the performance analysis of the multilabel classification for the drone’s
movement is presented with the use of a confusion matrix. Dataset-1 is
comprised of five classes, namely IDLE-HOVER, ASCEND, TURN, HMSL, and DESCEND.
The confusion matrix, as shown in Fig. 3, represents the model’s performance
in classifying these movements.

Figure 3: Multilabel classification confusion matrix.

To better understand the performance of the proposed framework’s multilabel
classification, the Precision is computed. Moreover, the Recall, and F1-score
for each class are presented in Table III. To calculate the Precision, Recall,
and F1-score metrics, the following equations can be used:

$Precision=\frac{TP}{TP+FP}$ (1)

$Recall=\frac{TP}{TP+FN}$ (2)

$F1-score=\frac{2\cdot(Precision\cdot Recall)}{Precision+Recall}$ (3)

The results indicate that the model performs considerably well for the IDLE-
HOVER, ASCEND, and HMSL classes, with F1-scores greater than $0.93$. However,
the model struggles with the TURN and DESCEND classes, with F1-scores of
$0.5898$ and $0.8836$, respectively.

Table III: Class-wise metrics. Class | Precision | Recall | F1-score
---|---|---|---
IDLE-HOVER | 0.9671 | 0.9333 | 0.9499
ASCEND | 0.9561 | 0.9186 | 0.937
TURN | 0.795 | 0.4683 | 0.5898
HMSL | 0.9842 | 0.9826 | 0.9834
DESCEND | 0.8079 | 0.975 | 0.8836

Also, for the objective of conducting a multilabel classification analysis,
overall pertinent micro-average and macro-average metrics are meticulously
computed and evaluated for Precision, Recall, and F1-score. The micro-average
metrics are higher than the macro-average metrics as shown in Table IV,
indicating high overall model performance. However, the macro-average metrics
present that there is still room for improvement in the classification
performance, particularly for the TURN and DESCEND classes.

Table IV: Average metrics. Metric | Micro-Average | Macro-Average
---|---|---
Precision | 0.9877 | 0.9021
Recall | 0.9825 | 0.8556
F1-score | 0.9851 | 0.8687

In summary, the multilabel classification task is investigated, exhibiting
considerable efficacy in discerning various drone movement categories.
However, it is imperative to address the sub-optimal performance observed in
the TURN and DESCEND classifications. Potential strategies for refinement may
encompass the incorporation of supplementary training data, the application of
feature engineering techniques, or the exploration of alternative algorithmic
approaches. Subsequent research endeavors will concentrate on these facets
with the intent of augmenting the model’s performance, thereby contributing to
the evolution of robust and precise drone movement classification systems.

####  V-C3 Trajectory Prediction Results (Dataset-2)

Table V presents the spatial estimation precision of the proposed MLF-ST
framework when Dataset-2 is utilized. Despite the decline in the estimated
location performance as the temporal interval ($t$) expands, the proposed
framework delivers precise spatial approximations that are analogous to the
ground truth measurements (exhibiting Euclidean distance error less than $8.1$
meters).

Table V: Average Euclidean prediction error (Dataset-2). All testing flights | Prediction time [s] | Euclidean distance error [m]
---|---|---
t+1 | 6.8166
t+2 | 7.1228
t+3 | 8.0021

Figure 4 depicts the CDF of the trajectory predictions for the time intervals
$t+1$, $t+2$, and $t+3$. It must be noted that the empirical data employed for
the extraction of the distinct CDFs encompassed an array of motion patterns,
including triangular, square, circular, and stochastic trajectories. In
particular, for the $1$-second look-ahead predictions, $90\%$ exhibit an error
magnitude below $3.4$ meters. Also, the $2$-second look-ahead accounts for
$90\%$ of errors that are under $3.59$ meters, while for the $3$-second time
horizon, $90\%$ of the errors do not surpass $4.2$ meters.

Figure 4: Average error CDF for predicted trajectories at $t+1$, $t+2$, $t+3$
(Dataset-2).

It is worth mentioning that Dataset-1 comprises of $227,552$ entries, whereas
Dataset-2 contains only $95,840$ entries, leading to a lower trajectory
prediction performance. The small number of entries for Dataset-2 is also the
reason for not utilizing it for state identification purposes. Nevertheless,
it is important for the reader to note that the expansion of Dataset-2 can
potentially lead to a better or analogous system performance to the results
obtained with Dataset-1. This is currently in progress, as Dataset-2 is
continuously being augmented with new data points obtained through various
drone flights.

In this study, MLF-ST framework resulted in average Euclidean distance errors
of 2.5521m, 2.6637m, and 3.4412m for the first three seconds using a sliding
window technique with WS equals 30 and HS equals 30, respectively. In
comparison, another study [3] reported an overall mean error of 1.5m. It is
essential to consider potential differences in methodologies and datasets when
comparing these results. It is worth mentioning that, in study [3], the mean
error was extracted after four flights, compared to the specific work that the
results were extracted using a dataset equipped from 195 flights.

Regarding the multilabel classification task, the MLF-ST framework in this
study achieved an accuracy of $98.51\%$, closely matching the compared paper’s
$98.5\%$ accuracy [2]. This demonstrates competitive performance in
classification accuracy.

##  VI Conclusion

A novel multi-task learning framework (MLF-ST) is presented, leveraging deep
neural networks to simultaneously optimize the desired two-fold objective of
drone state identification and trajectory prediction. The proposed framework,
that utilizes shared layers and various metrics to extract information from a
set of input data, can efficiently learn the useful features for both tasks.
Further, by incorporating LSTM networks and sliding window techniques along
with the drone’s historical data, MLF-ST is able to accurately identify the
drone’s current state and predict its future trajectory. An extensive
evaluation of the presented framework showcases its ability to contribute to
the overall safety and efficiency of drone operations. Also, the extensive
testing conducted on two different datasets, collected under various flight
conditions, further validates the robustness and adaptability of the MLF-ST
framework.

Future research avenues include the extension of the proposed MLF-ST framework
by incorporating additional tasks, different deep-learning techniques, and the
employment of various sensor modalities to enhance the capabilities of drone
monitoring systems.

## Acknowledgements

This work was supported by the Secure and Safe Multi-Robot Systems (SESAME)
H2020 Project under Grant Agreement 101017258. It was also supported by the
European Union’s Horizon 2020 research and innovation program under grant
agreement No 739551 (KIOS CoE - TEAMING) and from the Republic of Cyprus
through the Deputy Ministry of Research, Innovation and Digital Policy.

## References

  * [1] Matteo Corbetta et al.  “Real-time UAV trajectory prediction for safety monitoring in low-altitude airspace”  In _Proc. AIAA Aviation Forum_ , 2019
  * [2] Nikolaos Georgiou and Panayiotis Kolios  “Accurate real-time UAV flight-mode classification”  In _Proc. IFIP/IEEE 30th International Conference on Very Large Scale Integration (VLSI-SoC)_ , 2022, pp. 1–6
  * [3] Peng Shu et al.  “Trajectory prediction of UAV Based on LSTM”  In _Proc. 2nd International Conference on Big Data & Artificial Intelligence & Software Engineering (ICBASE)_, 2021, pp. 448–451
  * [4] Thiago A Rodrigues et al.  “In-flight positional and energy use data set of a DJI Matrice 100 quadcopter for small package delivery”  In _Scientific Data_ 8.155 Nature Publishing Group UK London, 2021, pp. 1–8
  * [5] Arnav Choudhry et al.  “CVaR-based flight energy risk assessment for multirotor UAVs using a deep energy model”  In _Proc. IEEE International Conference on Robotics and Automation (ICRA)_ , 2021, pp. 262–268
  * [6] Thiago A Rodrigues et al.  “Drone flight data reveal energy and greenhouse gas emissions savings for small package delivery”  In _arXiv preprint arXiv:2111.11463_ , 2021
  * [7] Wei Dai, Mingcheng Zhang and Kin Huat Low  “Data-Efficient Modeling for Precise Power Consumption Estimation of Quadrotor Operations Using Ensemble Learning”  In _arXiv preprint arXiv:2205.10997_ , 2022
  * [8] Ahmad Alos and Zouhair Dahrouj  “Using MLSTM and multioutput convolutional LSTM algorithms for detecting anomalous patterns in streamed data of unmanned aerial vehicles”  In _IEEE Aerospace and Electronic Systems Magazine_ 37.6, 2021, pp. 6–15
  * [9] Kaijun Liu et al.  “Modeling Trajectories with Multi-task Learning”  In _Proc. 23rd IEEE International Conference on Mobile Data Management (MDM)_ , 2022, pp. 208–213
  * [10] Hao Wu et al.  “Modeling trajectories with recurrent neural networks”  In _Proc. 26th International Joint Conference on Artificial Intelligence (IJCAI)_ , 2017, pp. 3083–3090
  * [11] Wanting Qin, Jun Tang, Cong Lu and Songyang Lao  “A typhoon trajectory prediction model based on multimodal and multitask learning”  In _Applied Soft Computing_ 122 Elsevier, 2022, pp. 108–804
  * [12] Wanting Qin, Jun Tang and Songyang Lao  “DeepFR: A trajectory prediction model based on deep feature representation”  In _Information Sciences_ 604 Elsevier, 2022, pp. 226–248
  * [13] John Phillips et al.  “Deep multi-task learning for joint localization, perception, and prediction”  In _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 4679–4689
  * [14] Sepp Hochreiter and Jürgen Schmidhuber  “Long short-term memory”  In _Neural Computation_ 9.8 MIT press, 1997, pp. 1735–1780
  * [15] Martin Sundermeyer, Ralf Schlüter and Hermann Ney  “LSTM neural networks for language modeling”  In _Proc. Thirteenth Annual Conference of the International Speech Communication Association_ , 2012, pp. 194–197
  * [16] Alex Graves, Abdel-rahman Mohamed and Geoffrey Hinton  “Speech recognition with deep recurrent neural networks”  In _Proc. IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2013, pp. 6645–6649
  * [17] Kelvin Xu et al.  “Show, attend and tell: Neural image caption generation with visual attention”  In _Proc. International Conference on Machine Learning (ICML)_ , 2015, pp. 2048–2057
  * [18] Samy Bengio, Oriol Vinyals, Navdeep Jaitly and Noam Shazeer  “Scheduled sampling for sequence prediction with recurrent neural networks”  In _Advances in Neural Information Processing Systems_ 28, 2015, pp. 1171–1179
  * [19] Rich Caruana  “Multitask Learning”  Springer, 1998
  * [20] Sebastian Ruder  “An overview of multi-task learning in deep neural networks”  In _arXiv preprint arXiv:1706.05098_ , 2017
  * [21] Xiaodong Liu, Pengcheng He, Weizhu Chen and Jianfeng Gao  “Multi-task deep neural networks for natural language understanding”  In _arXiv preprint arXiv:1901.11504_ , 2019
  * [22] Ronan Collobert and Jason Weston  “A unified architecture for natural language processing: Deep neural networks with multitask learning”  In _Proc. 25th International Conference on Machine Learning (ICML)_ , 2008, pp. 160–167
  * [23] Ross Girshick  “Fast R-CNN”  In _Proc. IEEE International Conference on Computer Vision_ , 2015, pp. 1440–1448
  * [24] Li Deng, Geoffrey Hinton and Brian Kingsbury  “New types of deep neural network learning for speech recognition and related applications: An overview”  In _Proc. IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2013, pp. 8599–8603
  * [25] Bharath Ramsundar et al.  “Massively multitask networks for drug discovery”  In _arXiv preprint arXiv:1502.02072_ , 2015
  * [26] Xin Yang, Partha Roop, Hammond Pearce and Jin Woo Ro  “A compositional approach using Keras for neural networks in real-time systems”  In _Proc. IEEE Design, Automation & Test in Europe Conference & Exhibition (DATE)_, 2020, pp. 1109–1114
  * [27] “scikit-learn library”, 2023  URL: https://scikit-learn.org/stable/
  * [28] Antreas Palamas and Panayiotis Kolios  “Dataset-2: Dataset obtained from multi-modal sensor onboard the drone”, 2023  URL: https://doi.org/10.5281/zenodo.7643456


# A hybrid CNN-RNN approach for survival analysis in a Lung Cancer Screening
study

Yaozhi Lu, Shahab Aslani, An Zhao, Ahmed Shahin, David Barber, Mark Emberton,
Daniel C. Alexander, and Joseph Jacob Corresponding author: Y. Lu, E-mail:
<EMAIL_ADDRESS>
Y. Lu, S. Aslani, A. Zhao, A. Shahin, D. Alexander, and J. Jacob are
affiliated with the Centre for Medical Image Computing, University College
London, UK. Y. Lu, A. Zhao, A. Shahin, D. Barber, and D. Alexander are
affiliated with the Department of Computer Science, University College London,
UK. S. Aslani and J. Jacob are affiliated with the Department of Respiratory
Medicine, University College London, UK. D. Barber is affiliated with the
Centre for Artificial Intelligence, University College London, UK. M. Emberton
is affiliated with the Division of Surgery and Interventional Science,
University College London, UK. Manuscript received March X, 2023; revised XX
XX, 2023.

###### Abstract

In this study, we present a hybrid CNN-RNN approach to investigate long-term
survival of subjects in a lung cancer screening study. Subjects who died of
cardiovascular and respiratory causes were identified whereby the CNN model
was used to capture imaging features in the CT scans and the RNN model was
used to investigate time series and thus global information. The models were
trained on subjects who underwent cardiovascular and respiratory deaths and a
control cohort matched to participant age, gender, and smoking history. The
combined model can achieve an AUC of 0.76 which outperforms humans at
cardiovascular mortality prediction. The corresponding F1 and Matthews
Correlation Coefficient are 0.63 and 0.42 respectively. The generalisability
of the model is further validated on an ’external’ cohort. The same models
were applied to survival analysis with the Cox Proportional Hazard model. It
was demonstrated that incorporating the follow-up history can lead to
improvement in survival prediction. The Cox neural network can achieve an IPCW
C-index of 0.75 on the internal dataset and 0.69 on an external dataset.
Delineating imaging features associated with long-term survival can help focus
preventative interventions appropriately, particularly for under-recognised
pathologies thereby potentially reducing patient morbidity.

###### Index Terms:

Computed tomography, lung, deep learning, computer vision, saliency map,
longitudinal data, cox regression

##  1 Introduction

###  1.1 Overview

Cardiac and respiratory illnesses are the leading causes of mortality globally
[1, 2], especially amongst older age groups. Compounded with the ageing global
population [3], this translates to an increased number of patients with
multimorbid conditions and causes spiralling pressure on healthcare services.
Responding to such growing healthcare needs requires cost-effective approaches
to early disease detection. Early detection allows timely intervention before
diseases become irreversible.

Various large-scale Lung Cancer Screening (LCS) studies, such as NELSON
(Dutch-Belgian)[4], SUMMIT (UK)[5], the National Lung Screening Trial (NLST,
US)[6, 7], have been set up to improve the detection of early lung cancer in
high-risk populations. Screening involves annual computed tomography (CT)
imaging and various clinical measurements (e.g. lung spirometry). LCS with CT
imaging has been demonstrated to reduce lung cancer mortality in the NLST
effectively [6, 7, 8] and NELSON studies[4].

A critical and currently underutilised benefit of imaging acquired as part of
LCS is the potential to detect underdiagnosed illnesses in LCS populations.
For example, Cardiovascular Disease (CVD) shares the same risk factors as lung
cancer. Accordingly, cardiovascular risk detection could be improved by a
detailed interrogation of LCS imaging. Similarly, detecting early respiratory
pathologies that influence survival could also be improved. Currently, lung
cancer detection is typically the radiologist’s overarching aim, but improving
the detection of cardio-respiratory morbidity could enhance LCS’s cost-
effectiveness and overall health benefits.

Rather than analysing images in isolation, this study proposes a hybrid CNN-
RNN approach to fully utilise both the imaging and temporal information to
predict long-term survival outcomes in a LCS cohort. In turn, this may allow
early interventions that may prevent or delay the occurrence of adverse
cardio-respiratory health events, thereby prolonging a patient’s life
expectancy.

###  1.2 Literature Review

Lung cancer and Cardiovascular Diseases (CVD) share several similar risk
factors, including smoking (both active and passive) and exposure to fine
particulates from air pollution [9]. Though the pathophysiological mechanisms
differ, it has been shown that smoking leads to increased mortality risk from
both lung cancer and CVD [10, 9]. It is, therefore, logical that patient
cohorts enriched with heavy smokers, such as LCS studies [6, 4, 5] can be used
to develop prediction models of CVD-related mortality.

Recently, attempts have been made to predict CVD-related mortality in the NLST
cohort [11, 12]. As demonstrated in van Velzen et al. 2019[11], a
Convolutional Autoencoder (CAE) was trained to derive abstract image features.
Then, the extracted features were fed into three separate classifiers to
predict CVD-related mortality. The CAE encoded the automatically extracted 3D
LDCT volume around the heart and exported the image features to the subsequent
classifiers. The support vector machine classifier achieved performance in the
Area under ROC curve (AUC) of 0.72. Though the study recognised the value of
using clinical information for prediction, including handcrafted variables
such as the Coronary Artery Calcium (CAC) score [13, 14], which is a known
predictor of CVD, such information was not utilised in making predictions.
Instead, the study demonstrated that it is possible to predict CVD-related
mortality from LDCT scans alone.

Predicting CVD-related mortality was further improved by Guo et al. 2020 [12].
A multimodal approach was adopted in this study, where models incorporated
both LDCT imaging information and handcrafted features to make mortality
predictions. When the contributions of the imaging features and clinical data
were optimised, this approach improved the AUC performance to 0.82. Both
methods show improvement over human performance in this regard. In fact, as
reported by Guo et al., visual inspection of the coronary artery calcium
measured by a radiologist could only achieve performance with an AUC of 0.64.

Instead of examining medical images in isolation without their global context,
a few recent studies considered adopting a hybrid model architecture to
analyse the imaging and temporal features in a patient’s data. Thus, hybrid
models, which have a CNN portion to investigate the medical imaging features
while a RNN portion to keep track of the temporal information, had been
adopted to examine retinal video [15], spectrogram [16], chest radiograph
[17], and computed tomography [18] etc. Thus, global temporal information
embedded in the follow-up scans can be fully utilised. In the above-mentioned
cases, the hybrid models outperformed their CNN-only counterparts, which
naturally limited the contextual information from the patient’s follow-up
history. Thus, the findings illustrate the superior performance of using time-
series data over a single snapshot in making the diagnosis.

Given the unavoidable heterogeneous follow-up times in clinical practices,
various studies [19, 17, 18] had attempted to incorporate the interval
variation in their analysis. Instead of adopting the implicit homogeneous
interval assumption in the LSTM model, various approaches [19, 17, 18] of
adding time-weighted terms to accentuate the temporal irregularities are
proposed. For example, Santeramo et al. 2018 [17] proposed additional interval
time-related weights to the LSTM gates calculation, while Gao et al. 2019 [18]
proposed using the Temporal Emphasis Modules (TEM) to emphasise the more
recent scans.

An alternative and more informative approach to evaluating survival would be
the Cox Proportional Hazard (PH) model [20] which examines the effect of
covariates on the patient’s time to death. Katzman et al. 2018[21] proposed,
DeepSurv, a Cox Proportional Hazard neural network to examine clinical data to
help provide a personalised approach to predict a patient’s survival time. A
few studies [22, 23] have applied a similar approach to survival analysis
using medical images. In particular, Shahin et al. 2022[23] performed survival
analysis in an Idiopathic Pulmonary Fibrosis cohort with a modified Residual
Network and an equivalent Cox PH loss function.

In our previous work, Lu et al. 2021 [24], 3D-CNN-based models were deployed
to inspect the latest screening CT scan and correlate the imaging features to
long-term survival status. A multimodal approach utilising the clinical
information was also attempted. However, as discussed in the literature
review, such approaches under-utilise the global information in the LCS study
and discard an individual’s speed of disease progression. Therefore, in this
study, we adopted a hybrid CNN-RNN approach to treat the patients’ follow-up
scans as time series data and combined the local and global information for
diagnosis. As can be demonstrated, the hybrid approach can enhance model
performance and, in particular, improve their generalisability. When applied
to the prediction of time to death, the application of such a method may help
direct timely early medical or therapeutic interventions that could prevent or
lessen adverse cardio-respiratory events, thus improving morbidity and
mortality.

The key contributions of the study are as follows:

  * •

A hybrid 4D CRNN approach to predict long-term survival using the patients’
longitudinal LCS imaging history.

  * •

Predict a patient’s risk, thereby suggesting timely personalised interventions
to improve patient morbidity and mortality.

##  2 Methodology

###  2.1 Dataset selection and split by screening centres

The National Lung Screening Trial was a lung cancer screening study conducted
with 33 US screening centres from 2002 to 2007 [6, 7, 8]. 53,454 heavy smokers
aged 55-74 at high risk for developing lung cancer were recruited. For our
study, we were provided with a subset of 15,000 patients’ imaging data
comprising three annual screening CT attendances, denoted as T0-T2. The
participants’ survival status and the cause of death were ascertained through
the evaluation of death certificate ICD-10 (International Classification of
Diseases, 10th edition) codes. In the LDCT branch, the leading cause of death
(as of the end of 2009) was cardiovascular disease (26.1%), followed by lung
cancer (22.9%) and then other types of cancer (22.3%) [7].

In our study, the participant’s survival status in 2015 was chosen as the
ground-truth label, and the follow-up times were adjusted with respect to the
latest scan (i.e. T2). The targeted causes of death were cardiac111ICD-10
codes: I10-I52 and respiratory diseases222ICD-10 codes: J00-J99. Only cases
with all three screening CTs (i.e. T0-2) were included to avoid bias
introduced by patients who left the trial early for unknown reasons. Only CT
scan thickness in the axial plane of less than 2.5mm was analysed. A
radiologist reviewed the corresponding CT images to exclude cases with imaging
artefacts and anatomical biases (i.e. severe forms of thoracic spinal
scoliosis). The patients with cardiac and respiratory deaths were age, gender,
and smoking history matched in a 1:2 ratio with a control population of
survivors.

2154 patients from 32 NLST screening centres meet the eligibility criteria
mentioned above. CT scans from 26 of the 32 centres were used as the
’internal’ training-validating-testing dataset (n=1,869). Scans from the
remaining 6 centres were kept aside as the ’external’ dataset (n=285) for
later evaluation of the generalisability of the models on heterogeneous
imaging protocols and equipment (i.e. out-of-distribution data). The
composition of the datasets is tabulated in Table I. The current combination
of the centres in the subsets allows the maximum number of scans to be used.

Dataset | Survivor | Non-survivor | Total
---|---|---|---
|  |  | Cardiac | Respiratory |
Internal | CV fold 1 | 208 | 47 | 57 | 312
| CV fold 2 | 208 | 56 | 48 | 312
| CV fold 3 | 208 | 59 | 45 | 312
| CV fold 4 | 208 | 62 | 42 | 312
| CV fold 5 | 208 | 50 | 54 | 312
| Test | 206 | 53 | 50 | 309
External | Test | 190 | 47 | 48 | 285
Total |  | 1436 | 374 | 344 | 2154
TABLE I: Composition of the datasets.

As shown in Table I, after a random split, 1560 cases from the internal
dataset (n=1,869) were used for a 5-fold cross-validation while the remaining
309 cases were used as an internal test set to assess model performance during
training. The composition of the subsets was kept the same throughout the
study. The trained models were evaluated on 285 patients’ scans in the
external test. Additionally, the relative proportions of the two causes of
mortality, which are also presented in Table I, were maintained across the
internal and external datasets. The time to death of the non-survivors from
the date of the last screening point is illustrated in Figure 1.

Figure 1: Distribution of the non-survivors’ follow-up time.

###  2.2 Lung CT-volume preprocessing

Similar to the previous study [24], the primary imaging processing pipeline
consisted of a modified pre-processing approach from Liao et al. [25]. The CT
image was filtered with a Gaussian filter for each axial slice, and a -600
Hounsfiled Unit (HU) threshold was applied to create the binarised slice. The
absolute size and eccentricity of connected components were then used to
filter out small components and imaging noise. The resulting 3D volumes were
then filtered by their size (0.68 - 7.5 L) and distance to the centre of the
scan. The results were joined to create the approximate lung mask.
Morphological transformations, i.e. erosion, dilation, and convex-hull
calculation, were performed to further separate the result into the left and
right lung masks. An additional convex-hull analysis was performed on the
joint lung masks to include the cardiac region in the final results. The
Hounsfield Unit (HU) range was clipped to an interval between -1200 HU to 600
HU. The range was linearly normalised to between 0 and 255. The regions
outside the masks corresponding to the surrounding tissue were filled with an
average value of 170. The pre-processing pipeline was applied over all axial
layers to extract the thorax region in 3D. An example case is illustrated in
Figure 2 where Figure 2(a) is a CT scan in DICOM format while Figure 2(b) is
the pre-processed input to the neural networks.

(a) Raw axial slice at 512 by 512 voxels per slice.

(b) Slice processed and scaled to 256 by 256 voxels per slice.

Figure 2: CT volume pre-processing.

The imaging pre-processing pipeline described above does not explicitly
perform image registration. However, it does centre the image based on the
joint mask. To assess whether image registration can lead to performance
improvement, particularly for the time-series models, 3D affine registration
was attempted for one of the sub-studies. In particular, the scans from the
two earlier screening points, T0 and T1, were registered to the most recent T2
scan through the DIPY333The total processing time for 2154 patients, when
fully parallelised on an Intel i7-10700K CPU, is approximately 20 hours while
running on a SSD. (v1.5.0) library [26]. The example results of the 3D affine
registration are illustrated in Figure 3. Comparing Figure 3(c) to Figure
3(d), it can be seen that the additional processing has minimised the
difference in position and orientation between the T0 scan to the target T2
scan.

(a) T0 scan.

(b) T2 scan.

(c) Overlay of T2 and pre-registration T0 scans.

(d) Overlay of T2 and registered T0 scans.

Figure 3: 3D affine registration demonstrated on 2D slices.

###  2.3 Models

####  2.3.1 CNN models

The deep-learning-based models in this study were based on a 3D implementation
of the ResNet [27][28]. A 10-layer implementation of the 3D ResNet was used as
the backbone for both the CNN models and the subsequent CRNN models in this
study. The choice of the CNN network depth reflects the trade-off between
performance and model size. The pre-trained weights from Chen et al.[28],
initially optimised for medical imaging (CT images) segmentation tasks, were
used as initial weights during training. The output from the 3D ResNet
backbone was converted to a 1D tensor after passing through a global adaptive
average pooling layer. It was then passed through two subsequent fully-
connected layers (including dropout with $p=0.5$ and ReLU activation) with 512
and 32 neurons, respectively.

The CNN models were trained with the latest scan, i.e. NLST T2 scan. The 3D CT
volumes were interpolated into 256 by 256 by 128 tensors and fed into the
networks. The 2015 survival outcome was used as the ground truth label for the
classification task between survivors and non-survivors.

####  2.3.2 CRNN models

The two sides of the CRNN model, as illustrated in Figure 4, were trained
independently from each other. The model parameters from the trained CNN model
were inherited by the CNN side of the model and were not optimised during the
CRNN training. In contrast, the RNN (i.e. LSTM) side, which feeds in the
spatial /imaging feature from the CNN side, was optimised during CRNN
training. In other words, the CNN backbone was used to capture the local
imaging features from each CT scan, while the LSTM model was utilised to
capture the global time-series information embedded in the patient’s entire
follow-up history. The output from the LSTM model was passed through one
fully-connected layer (including dropout with $p=0.3$ and ReLU activation).
The number of neurons in the fully-connect layer matched the LSTM’s hidden
state (hx) size, a tunable hyperparameter.

For each patient, the interpolated 3D CT volumes were stacked into a 4D
tensor, with the temporal dimension (i.e. T0-2) as the additional dimension.,
All 3 NLST screenings were analysed by the hybrid CRNN model, which can
process a variable number of scans in a patient’s follow-up history. Thus, in
this study, the input to the CRNN network was in the form of 256 by 256 by 128
by three tensors.

Figure 4: Schematic diagram of the CRNN model.

It is important to note that the vanilla LSTM implementation implicitly
assumes the inputs are equally spaced. Given the inevitable irregular time
intervals between screening timepoints that occur in medical studies, two
additional LSTM variants, Time-Aware LSTM (TALSTM)[19] and time-modulated LSTM
(tLSTM)[17], which explicitly take irregular time intervals between inputs
into account were tested. To account for the influence of non-uniform time
intervals, the former model[19] attempts to separate the memory component in
LSTM into its long and short-term parts, with the latter’s effect disregarded.
In contrast, the time-modulated LSTM variant [19] explicitly introduces
temporal weights into the LSTM gates update. For both variants, as illustrated
by the dashed lines in Figure 4, the temporal information was passed directly
into the RNN portion of the model. Pre-processing of the temporal data was
implemented in the same manner recommended by the respective studies.

####  2.3.3 Cox Proportional Hazard Network

In addition to performing classification on the long-term survival outcome, a
survival outcome prediction based on the Cox Proportional Hazard model[20] was
attempted using the same CNN and CRNN networks. In this study, the survival
time is right-censored and adjusted with respect to the latest screening point
(T2).

In survival analysis, a hazard function measures the instantaneous risk of
event /death ($E=1$) at time $t$ for an individual that has survived beyond
time $t$:

$h(t)=\displaystyle{\lim_{\triangle t\to 0}}\frac{p(t\leq T<t+\triangle
t|T\geq t)}{\triangle t}$ (1)

The Cox PH model attempts to model the hazard function given a patient’s
baseline data $x$:

$h(t|x)=h_{0}(t)\cdot e^{h(x)}$ (2)

In Eq. 2, $h_{0}(t)$ is the baseline hazard function and represents the hazard
when all the covariates are zero. $h(x)$ in this study is the output of the
neural network. To optimise such a network, the negative partial log-
likelihood function [21, 23] is implemented:

$L=-\frac{1}{N_{E=1}}\sum\limits_{i:E_{i}=1}\left(h(x_{i})-\log\sum\limits_{j\in
R(T_{i})}e^{h(x_{j})}\right)$ (3)

where $N_{E=1}$ is the total number of patients with an event and $R(T_{i})$
is the set of patients that have not died before patient $i$ at time $T_{i}$.
The loss function $L$ is optimised using stochastic gradient descent.

####  2.3.4 Experiments

A grid search approach was adopted to tune the hyperparameters in the neural
network models. The hyperparameter value that produced the best AUC value was
chosen.

Similar to the previous work[24], to counter the class imbalance in the 1:2
matched dataset, a weighted random sampler assigns a sampling probability
inversely proportional to the class size was utilised. This approach attempts
to create balanced batches during training. Cross entropy loss was adopted as
the loss function for the classification model while the Cox Loss in Eq. 3 was
adopted for the regression models. The models were trained with Sharpness
Aware Minimisation (SAM)[29] with Stochastic Gradient Descent (SGD) as the
base optimiser. SAM simultaneously minimises the value and sharpness in the
training loss landscape. Thus, it aims to optimise for model parameters within
the neighbourhood of low loss and hence improves the generalisability of the
model. Adaptive implementation of this optimisation algorithm was also tested,
yet it did not improve performance.

As a substudy, an additional set of networks was trained to further
differentiate the non-survivor classes into the respective mortality groups,
i.e. cardiac-related and respiratory-related. This additional insight provides
the clinician with further information as to which areas on the CT need
specific consideration. A two-step process is proposed to utilise this model
in a clinical setting. Firstly, the main model examines a patient’s scans to
produce a mortality probability. Secondly, the scans are examined by the
additional cause-specific model should the previous output exceed a predefined
threshold which is 0.5 in this study. The cause-specific model will then
attempt to categorise the site of concern further, and the clinician can
perform further checks accordingly. Thus, additional models were trained with
only the two non-survivor classes shown in Table I and the composition of each
subset was kept the same as the main model. The primary CRNN model’s training
approach was adopted for this substudy.

The classification models were optimised using an initial learning rate of
1E-3, a cyclic learning rate schedule, momentum of 0.9, and weight decay (L2
regularisation) of 5E-4. The regression models differ regarding the initial
learning rate, which was set as 5E-3. Each neural network was trained with a
batch size of 24 on a single Nvidia A100 GPU (40GB HBM2) on the UCL CS CMIC
cluster. The CNN models were trained with approximately 800 epochs, while the
CRNN models were trained with approximately 150 epochs. The deep learning
models were developed in Python (v3.9.5) and PyTorch (v1.9.1).

##  3 Results

###  3.1 Classifying Cardiorespiratory Mortality

To compare the performance with related studies [11, 12], the models were
mainly assessed with the Area Under the Curve (AUC) metric. The average (and
standard deviation) of the AUC metrics from the 5-fold cross-validation were
used to assess the internal performance of the model. As tabulated in Table
II, the same was done for the F1 score and Matthews Correlation Coefficient
(MCC)[30]. To evaluate the model’s generalisation ability, the average (and
standard deviation) inference performance of the five models on the external
dataset was also recorded through the three metrics in Table II.

In the baseline Study A, where the CT scans are analysed in isolation from the
rest of the sequence of CT time points, the model reached an average AUC of
0.759, an average F1 of 0.626, and an average MCC of 0.407 on the internal
dataset. As expected, the performance in all three metrics deteriorated when
evaluated on the external dataset. In comparison, when the global and temporal
information from the patient’s follow-up was considered in Study B, all three
performance metrics on the internal dataset improved over those in Study A. In
particular, the Matthews Correlation Coefficient improved by 3.4% from 0.407
to 0.421. The extent of the performance improvement introduced by the hybrid
approach is on par with related studies [18][17]. More importantly, the
generalisation ability of the model, indicated by performance on the external
dataset, improved, as illustrated by the increase in AUC performance from
0.714 to 0.731 in Table II. The corresponding ROC curves for Study B are shown
in Figure 5.

Study C, where a 3D registered dataset was used to train the combined CRNN
model, showed no improvement in performance on internal and external datasets.
It also showed that the additional noise introduced by the affine registration
did not outweigh the benefits gained. To further investigate the utility of
the 3D registration, GradCAMs (Gradient-weighted Class Activation Map)[31]
from a Chronic Obstructive Pulmonary Disease (COPD) non-survivor’s follow-ups
scans are shown. As illustrated by the GradCAM overlays in Figure 6, the model
is relatively consistent with lesion position over time, without the explicit
registration, thereby negating the need for the additional 3D registration.

Study | Model Component(s) | LSTM | 5-Fold CV Performance | External Performance
---|---|---|---|---
| CNN | RNN | hx | AUC | F1 | MCC | AUC | F1 | MCC
A. CNN | ResNet | - | - | 0.759 (0.019) | 0.626 (0.026) | 0.407 (0.047) | 0.714 (0.011) | 0.578 (0.010) | 0.315 (0.021)
B. LSTM | ResNet | LSTM | 32 | 0.763 (0.020) | 0.629 (0.031) | 0.421 (0.047) | 0.731 (0.015) | 0.571 (0.021) | 0.319 (0.037)
C. LSTM (3d reg) | ResNet | LSTM | 32 | 0.760 (0.017) | 0.605 (0.048) | 0.404 (0.042) | 0.729 (0.014) | 0.565 (0.030) | 0.324 (0.034)
D. TALSTM | ResNet | TALSTM | 32 | 0.763 (0.019) | 0.628 (0.034) | 0.423 (0.048) | 0.730 (0.015) | 0.573 (0.026) | 0.326 (0.035)
E. tLSTM | ResNet | tLSTM | 64 | 0.763 (0.018) | 0.630 (0.028) | 0.419 (0.046) | 0.731 (0.014) | 0.575 (0.018) | 0.320 (0.039)
TABLE II: Performance metrics on long term survival prediction.

(a) ROC curves from the 5 cross-validation folds (i.e. internal dataset).

(b) ROC curves from inference on the external dataset.

Figure 5: CRNN model (Study B) performance illustrated by Receiver Operating
Characteristic (ROC) curves.

(a) The first screening point (T0).

(b) The second screening point (T1).

(c) The third screening point (T2).

Figure 6: GradCAM overlays of the 3 screening points for a COPD non-survivor.

In Study D and E, where the irregularity in follow-up intervals are explicitly
addressed, the performance is comparable to Study B, which implicitly assumes
constant intervals between scans. This implies the follow-up interval
variation, which has an inter-quartile range of 40 days, provides limited
information to the long-term survival prediction task.

###  3.2 Classifying Cause-Specific Cardiorespiratory Mortality

The CRNN network in Study B was applied to patients who died from cardiac and
respiratory deaths, as shown in Table III, where deaths were classified into
three bands: 0-3 years from the last CT date (i.e. T2); 0-7 years from the
last CT date; within 11 years of the last CT date. The model had initially
been trained with labels corresponding to known patient deaths 10 years after
the first CT, but had not been provided with information about the exact cause
of death (i.e. whether cardiac or respiratory related). Any patient predicted
by the ensemble model from Study B to have a less than 0.5 mortality
probability was classified as a survivor. Any cases passing this probability
threshold were separated into the two causes of death by the CRNN model in
Study F. The final prediction from this two-step decoupled process was then
evaluated against their ground truth label of dead/alive within the relevant
time band. Accordingly, the ground truth survival status varied according to
the follow-up time band.

As shown in Table III, respiratory mortality within 3 years of the CT scan was
predicted with good sensitivity (0.737) and specificity (0.765), with cardiac
mortality prediction also showing good specificity (0.771) but poor
sensitivity (0.476). Respiratory death prediction showed improved specificity
and acceptable sensitivity at time intervals increasingly distant from the
last available imaging that could be analysed. Cardiac death prediction
however showed poor sensitivity and good specificity across all follow-up time
bands.

Class | Band a (death <3.0 yrs) | Band b (death <7.0 yrs) | Band c (death <11.0 yrs)
---|---|---|---
| Cases | Sensitivity | Specificity | Cases | Sensitivity | Specificity | Cases | Sensitivity | Specificity
Survivor | 2034 | 0.626 | 0.744 | 1776 | 0.688 | 0.746 | 1436 | 0.791 | 0.719
NS(Cardiac) | 82 | 0.476 | 0.771 | 202 | 0.510 | 0.806 | 374 | 0.513 | 0.853
NS(Respiratory) | 38 | 0.737 | 0.765 | 176 | 0.665 | 0.808 | 344 | 0.634 | 0.861
TABLE III: Performance by time range and cause of mortality (Study F).

###  3.3 Predicting Patients’ Risk through the Cox model

Our final analysis considered discriminating the NLST cohort based on their
cardio-respiratory mortality risk. The performance of the Cox-regression-based
survival analysis was measured across internal and external cohorts and
evaluated using the Inverse Probability of Censored Weights (IPCW) Concordance
Index [32] which is modified from the original Concordance Index [33]. The
tabulated metrics in Table IV illustrate that adopting the CRNN model which
examines all available follow-up imaging information outperforms the CNN
approach which investigates the latest CT scan in isolation. In particular,
the IPCW C-index is improved by 4.3% from 0.719 to 0.750 between the two
models. Though less pronounced, an improvement of 2.5% in IPCW C-index from
0.673 to 0.690 can be observed in the external dataset. This reaffirms our
hypothesis that the hybrid CNN-RNN approach improves model performance by
capturing additional global time-series CT features that relate to cardio-
respiratory disease progression.

CoxPH | Model | LSTM | IPCW C-Index
---|---|---|---
Study |  | hx | 5-fold CV | External
G. CNN | ResNet | - | 0.719 (0.040) | 0.673 (0.009)
H. CRNN | ResNet+LSTM | 32 | 0.750 (0.050) | 0.690 (0.019)
TABLE IV: Regression model performance in terms of average IPCW C-index (and
standard deviation).

##  4 Discussion

Our study examined the ability of time-series imaging data to predict
mortality in subjects participating in a lung cancer screening study. We
demonstrate that time series analysis improves the ability of single timepoint
data in mortality prediction. We demonstrate the high sensitivity and
specificity of our model in estimating the likelihood of respiratory deaths at
3 years, and the high specificity for the detection of cardiac-related deaths
across follow-up timepoints. We also demonstrate the improved ability of our
time-series model to predict the time to cardiorespiratory death using a
modified Cox model when compared to single timepoint data.

###  4.1 Survival prediction using both spatial and temporal data

By comparing the performance between Study A and B, in Table II, it is evident
that the inclusion of longitudinal information improves classification
performance. When compared to examining CT images in isolation, (Study A where
a CNN model is adopted to view the latest CT scan), considering the entire
follow-up imaging history of a patient provides a greater global context of
the patient’s health and/or disease progression. Specifically, the CRNN model
improves the internal MCC by 3.4% from 0.407 to 0.421 and the external AUC by
2.4% from 0.714 to 0.731. The extent of the improvement, through the inclusion
of temporal information, is consistent with that observed in related studies
[18, 17].

Despite the marginal improvement in external F1 and MCC, the irregular time
interval model examined in Study D and E showed similar performance to the
model evaluated in Study B. This is in contrast to the performance improvement
noted in the related studies [19, 18, 17]. It can be argued that this study’s
classification task, which focuses on long-term survival, is more tolerant to
the time interval variation than counterparts, such as the study by Santeramo
et al. 2018 [17]. Yet the most likely cause for the comparable performance
between models that did and did not consider time intervals is the relatively
constant time interval (inter-quartile range of 40 days between CTs) in the
NLST study which was a clinical trial. Models trained to examine irregular
time intervals on longitudinal data are far more likely to show utility in
real-world data where imaging intervals are more varied than would be seen in
protocolised clinical trials.

###  4.2 Cause-specific survival outcomes

Our results demonstrate how differently our model treated respiratory versus
cardiac deaths in the screening population. The high sensitivity and
specificity for 3-year respiratory mortality suggest the model has potential
utility in identifying screened patients who might benefit from targeted
respiratory interventions over a meaningful time frame for clinical trials.

The low sensitivity of the model for cardiac death prediction may relate to
the inherently unpredictable nature of many acute cardiac events. A central
challenge of cardiology lies in risk prediction and prevention for populations
at risk of adverse cardiovascular events. Whilst calcification can be
identified in coronary arteries on non-contrast-enhanced CTs, calcified
plaques are the main surrogates for the presence of non-calcified plaques
which are more likely to rupture. Yet coronary calcification and non-calcified
coronary damage may be very challenging to identify comprehensively on low-
dose, non-gated, non-contrast CT scans. The high specificity for cardiac death
prediction identified in our study suggests some events can be predicted using
imaging and the results are reassuring in the context of a screening
population, where false positive identification of disease should be avoided.
Analysis of the time-series CTs in a screening cohort with a model such as
ours could help rule-in subjects who would warrant a more definitive
evaluation of cardiac disease with for example a contrast-enhanced, gated
coronary CT scan.

The improved performance of the Cox regression model when using time-series
data confirms our hypothesis that capturing change in imaging features and
therefore disease progression, is important when estimating mortality and has
clear advantages over single timepoint data. The standard workflow for a
radiologist analysing imaging data relies on the assessment of historic
imaging to assess the extent of disease progression. Models utilising the
additional information available through repeated imaging therefore should be
better at estimating disease progression.

##  5 Conclusion and Future Works

The results shown in our study lead us to the following conclusions and
directions for future studies:

  1. 1.

The comparison between the baseline CNN model and the main CRNN model
indicates that capturing both local and global information in a patient’s
medical history can lead to better model performance and improved
generalisation ability. Thus, unsurprisingly, medical images should not be
viewed in isolation. Instead, longitudinal information should be captured to
monitor disease progression. The next step would be deploying the current
hybrid model on contemporaneous lung cancer screening studies. The SUMMIT
study[5] provides such an opportunity where the ability to predict cardio-
respiratory adverse events could be examined prospectively.

  2. 2.

It is clear from Study D and E that the limited variation in follow-up
intervals across the NLST screening time points did not unduly influence long
term survival prediction. It would be important to deploy models that consider
irregular time interval acquisitions on non-protocolised real-world data to
gauge the influence of irregular scanning intervals on estimations of disease
progression.

  3. 3.

An alternative to the proposed CRNN model would be a Vision Transformer (ViT)
[34]. A transformer-based architecture was proposed by Sarasua et al. 2021
[35] to model spatio-temporal neuroanatomical changes in a patient’s left
hippocampus. To account for missing follow-up imaging timepoints, padding was
applied to the sequences. We would therefore like to explore a time-dependent
variant to our model to account for temporal heterogeneity.

## Acknowledgments

This work was supported by the International Alliance for Cancer Early
Detection, an alliance between Cancer Research UK [C23017/A27935], Canary
Center at Stanford University, the University of Cambridge, OHSU Knight Cancer
Institute, University College London, and the University of Manchester. The
authors would also like to thank the National Institute of Health Research
(NIHR) UCLH Biomedical Research Centre for the funding.

The authors thank the National Cancer Institute for access to NCI’s data
collected by the National Lung Screening Trial (NLST). The statements
contained herein are solely those of the authors and do not represent or imply
concurrence or endorsement by NCI.

The GPU computing resource of this work is jointly provided by the following
funders (and grants): CRUK International Alliance for Cancer Early Detection
(ACED) (C23017/A27935); CRUK-EPSRC (NS/A000069/1); MRC-JPND (MR/T046473/1);
Open Source Imaging Consortium; Royal Academy of Engineering
(RF\201718\17140); UKRI /EPSRC /MRC (EP/R006032/1, EP/R014019/1, EP/T026693/1,
EP/T029404/1, EP/V034537/1, EP/W00805X/1, MR/S03546X/1, MR/T019050/1);
Wellcome Trust (209553/Z/17/Z, 221915/Z/20/Z).

The first author would like to thank Mr T Clark and Mr E Martin from the UCL
CS Technical Support Group for their assistance with the CMIC GPU cluster.

## References

  * [1] G. A. Roth, D. Abate, K. H. Abate, S. M. Abay, C. Abbafati, N. Abbasi, H. Abbastabar, F. Abd-Allah, J. Abdela, A. Abdelalim _et al._ , “Global, regional, and national age-sex-specific mortality for 282 causes of death in 195 countries and territories, 1980-2017: a systematic analysis for the Global Burden of Disease Study 2017,” _The Lancet_ , vol. 392, no. 10159, pp. 1736–1788, 2018.
  * [2] T. Vos, S. S. Lim, C. Abbafati, K. M. Abbas, M. Abbasi, M. Abbasifard, M. Abbasi-Kangevari, H. Abbastabar, F. Abd-Allah, A. Abdelalim _et al._ , “Global burden of 369 diseases and injuries in 204 countries and territories, 1990-2019: a systematic analysis for the Global Burden of Disease Study 2019,” _The Lancet_ , vol. 396, no. 10258, pp. 1204–1222, 2020.
  * [3] United Nations Department of Economic and Social Affairs Population Division, “World Population Prospects 2022: Summary of Result,” Tech. Rep. UN DESA/POP/2022/TR/NO. 3, 2022.
  * [4] H. J. de Koning, C. M. van der Aalst, P. A. de Jong, E. T. Scholten, K. Nackaerts, M. A. Heuvelmans, J.-W. J. Lammers, C. Weenink, U. Yousaf-Khan, N. Horeweg, S. van ’t Westeinde, M. Prokop, W. P. Mali, F. A. M. Hoesein, P. M. van Ooijen, J. G. Aerts, M. A. den Bakker, E. Thunnissen, J. Verschakelen, R. Vliegenthart, J. E. Walter, K. ten Haaf, H. J. Groen, and M. Oudkerk, “Reduced lung-cancer mortality with volume CT screening in a randomized trial,” _New England journal of medicine_ , vol. 382, no. 6, pp. 503–513, 2020, doi: 10.1056/NEJMoa1911793.
  * [5] C. Horst, J. L. Dickson, S. Tisi, M. Ruparel, A. Nair, A. Devaraj, and S. M. Janes, “Delivering low-dose CT screening for lung cancer: a pragmatic approach,” _Thorax_ , vol. 75, no. 10, pp. 831–832, 2020, doi: 10.1136/thoraxjnl-2020-215131.
  * [6] National Lung Screening Trial Research Team, “The National Lung Screening Trial: Overview and Study Design,” _Radiology_ , vol. 258, no. 1, pp. 243–253, 2011, doi: 10.1148/radiol.10091808.
  * [7] National Lung Screening Trial Research Team, “Reduced lung-cancer mortality with low-dose computed tomographic screening,” _New England Journal of Medicine_ , vol. 365, no. 5, pp. 395–409, 2011, doi: 10.1056/NEJMoa1102873.
  * [8] D. R. Aberle, S. DeMello, C. D. Berg, W. C. Black, B. Brewer, T. R. Church, K. L. Clingan, F. Duan, R. M. Fagerstrom, I. F. Gareen _et al._ , “Results of the two incidence screenings in the National Lung Screening Trial,” _New England Journal of Medicine_ , vol. 369, no. 10, pp. 920–931, 2013, doi: 10.1056/NEJMoa1208962.
  * [9] Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, and Office on Smoking and Health, “How tobacco smoke causes disease: The biology and behavioral basis for smoking-attributable disease: A report of the surgeon general,” 2010.
  * [10] C. A. Pope III, R. T. Burnett, M. C. Turner, A. Cohen, D. Krewski, M. Jerrett, S. M. Gapstur, and M. J. Thun, “Lung cancer and cardiovascular disease mortality associated with ambient air pollution and cigarette smoke: shape of the exposure-response relationships,” _Environmental health perspectives_ , vol. 119, no. 11, pp. 1616–1621, 2011, doi: 10.1289/ehp.1103639.
  * [11] S. G. van Velzen, M. Zreik, N. Lessmann, M. A. Viergever, P. A. de Jong, H. M. Verkooijen, and I. Išgum, “Direct prediction of cardiovascular mortality from low-dose chest CT using deep learning,” in _Medical Imaging 2019: Image Processing_ , vol. 10949. International Society for Optics and Photonics, 2019, p. 109490X, doi: 10.1117/12.2512400.
  * [12] H. Guo, U. Kruger, G. Wang, M. K. Kalra, and P. Yan, “Knowledge-based analysis for mortality prediction from CT images,” _IEEE journal of biomedical and health informatics_ , vol. 24, no. 2, pp. 457–464, 2019, doi: 10.1109/JBHI.2019.2946066.
  * [13] P. C. Jacobs, M. J. Gondrie, Y. van der Graaf, H. J. de Koning, I. Isgum, B. van Ginneken, and W. P. Mali, “Coronary artery calcium can predict all-cause mortality and cardiovascular events on low-dose CT screening for lung cancer,” _American Journal of Roentgenology_ , vol. 198, no. 3, pp. 505–511, 2012, doi: 10.2214/AJR.10.5577.
  * [14] C. Chiles, F. Duan, G. W. Gladish, J. G. Ravenel, S. G. Baginski, B. S. Snyder, S. DeMello, S. S. Desjardins, R. F. Munden, and N. S. Team, “Association of coronary artery calcification and mortality in the National Lung Screening Trial: a comparison of three scoring methods,” _Radiology_ , vol. 276, no. 1, pp. 82–90, 2015, doi: 10.1148/radiol.15142062.
  * [15] S. Gheisari, S. Shariflou, J. Phu, P. J. Kennedy, A. Agar, M. Kalloniatis, and S. M. Golzan, “A combined convolutional and recurrent neural network for enhanced glaucoma detection,” _Scientific reports_ , vol. 11, no. 1, pp. 1–11, 2021, doi: 10.1038/s41598-021-81554-4.
  * [16] G. Petmezas, G. Cheimariotis, L. Stefanopoulos, B. Rocha, R. P. Paiva, A. K. Katsaggelos, and N. Maglaveras, “Automated lung sound classification using a hybrid CNN-LSTM network and focal loss function,” _Sensors_ , vol. 22, no. 3, p. 1232, 2022, doi: 10.3390/s22031232.
  * [17] R. Santeramo, S. Withey, and G. Montana, “Longitudinal detection of radiological abnormalities with time-modulated LSTM,” in _Deep learning in medical image analysis and multimodal learning for clinical decision support_. Springer, 2018, pp. 326–333, doi: 10.48550/arXiv.1807.06144.
  * [18] R. Gao, Y. Huo, S. Bao, Y. Tang, S. L. Antic, E. S. Epstein, A. B. Balar, S. Deppen, A. B. Paulson, K. L. Sandler, P. P. Massion, and B. A. Landman, “Distanced LSTM: time-distanced gates in long short-term memory models for lung cancer detection,” in _International Workshop on Machine Learning in Medical Imaging_. Springer, 2019, pp. 310–318, doi: 10.1007/978-3-030-32692-0_36.
  * [19] I. M. Baytas, C. Xiao, X. Zhang, F. Wang, A. K. Jain, and J. Zhou, “Patient subtyping via time-aware LSTM networks,” in _Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining_ , 2017, pp. 65–74, doi: 10.1145/3097983.3097997.
  * [20] D. R. Cox, “Regression Models and Life-Tables,” _Journal of the Royal Statistical Society: Series B (Methodological)_ , vol. 34, no. 2, pp. 187–202, 1972, doi: 10.1111/j.2517-6161.1972.tb00899.x.
  * [21] J. L. Katzman, U. Shaham, A. Cloninger, J. Bates, T. Jiang, and Y. Kluger, “DeepSurv: personalized treatment recommender system using a Cox proportional hazards deep neural network,” _BMC medical research methodology_ , vol. 18, no. 1, pp. 1–12, 2018, doi: 10.48550/arXiv.2203.11391.
  * [22] P. Mobadersany, S. Yousefi, M. Amgad, D. A. Gutman, J. S. Barnholtz-Sloan, J. E. Velázquez Vega, D. J. Brat, and L. A. Cooper, “Predicting cancer outcomes from histology and genomics using convolutional networks,” _Proceedings of the National Academy of Sciences_ , vol. 115, no. 13, pp. E2970–E2979, 2018, doi: 10.1073/pnas.171713911.
  * [23] A. H. Shahin, J. Jacob, D. C. Alexander, and D. Barber, “Survival analysis for idiopathic pulmonary fibrosis using ct images and incomplete clinical data,” _arXiv preprint arXiv:2203.11391_ , 2022, doi: 10.1186/s12874-018-0482-1.
  * [24] Y. Lu, S. Aslani, M. Emberton, D. C. Alexander, and J. Jacob, “Deep learning-based long term mortality prediction in the National Lung Screening Trial,” _IEEE Access_ , vol. 10, pp. 34 369–34 378, 2022, doi: 10.1109/ACCESS.2022.3161954.
  * [25] F. Liao, M. Liang, Z. Li, X. Hu, and S. Song, “Evaluate the malignancy of pulmonary nodules using the 3D deep leaky noisy-or network,” _IEEE transactions on neural networks and learning systems_ , vol. 30, no. 11, pp. 3484–3495, 2019, doi: 10.1109/tnnls.2019.2892409.
  * [26] E. Garyfallidis, M. Brett, B. Amirbekian, A. Rokem, S. Van Der Walt, M. Descoteaux, I. Nimmo-Smith, and D. Contributors, “DIPY, a library for the analysis of diffusion MRI data,” _Frontiers in neuroinformatics_ , vol. 8, p. 8, 2014, doi: 10.3389/fninf.2014.00008.
  * [27] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 770–778, doi: 10.1109/CVPR.2016.90.
  * [28] S. Chen, K. Ma, and Y. Zheng, “Med3d: Transfer learning for 3D medical image analysis,” _arXiv preprint arXiv:1904.00625_ , 2019.
  * [29] P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur, “Sharpness-aware minimization for efficiently improving generalization,” _arXiv preprint arXiv:2010.01412_ , 2020, doi: 10.48550/arXiv.2010.01412.
  * [30] D. Chicco and G. Jurman, “The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation,” _BMC genomics_ , vol. 21, no. 1, pp. 1–13, 2020, doi: 10.1186/s12864-019-6413-7.
  * [31] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-CAM: Visual explanations from deep networks via gradient-based localization,” in _2017 IEEE International Conference on Computer Vision (ICCV)_ , 2017, pp. 618–626, doi: 10.1109/ICCV.2017.74.
  * [32] H. Uno, T. Cai, M. J. Pencina, R. B. D’Agostino, and L.-J. Wei, “On the C-statistics for evaluating overall adequacy of risk prediction procedures with censored survival data,” _Statistics in medicine_ , vol. 30, no. 10, pp. 1105–1117, 2011, doi: 10.1002/sim.4154.
  * [33] F. E. Harrell Jr, K. L. Lee, and D. B. Mark, “Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors,” _Statistics in medicine_ , vol. 15, no. 4, pp. 361–387, 1996, doi: 10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4.
  * [34] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020, doi: 110.48550/arXiv.2010.11929.
  * [35] I. Sarasua, S. Pölsterl, and C. Wachinger, “Transformesh: A transformer network for longitudinal modeling of anatomical meshes,” in _Machine Learning in Medical Imaging: 12th International Workshop, MLMI 2021, Held in Conjunction with MICCAI 2021, Strasbourg, France, September 27, 2021, Proceedings 12_. Springer, 2021, pp. 209–218, doi: 10.1007/978-3-030-87589-3_22.