Spatiotemporal Event Graphs for Dynamic Scene Understanding

Salman Khan

A Thesis Submitted in Partial Fulfilment

of the Requirements for the Degree of

Doctor of Philosophy

in

Computer Science and Mathematics

Visual Artificial Intelligence Laboratory (VAIL)

School of Engineering, Computing and Mathematics

Oxford Brookes University

Supervised by :

Prof. Fabio Cuzzolin and Dr. Tjeerd Olde Scheper

January 2023

_The author holds the _copyright_ of this thesis. Any person(s) intending to
use a part or whole of the materials in the thesis in a proposed publication
must seek copyright release from the author._

Thesis/Assessment Committee

Prof. Dima Damen (External Examiner)

Professor of Computer Vision

Lead of Machine Learning and Computer Vision Group

Department of Computer Science

University of Bristol

Dr. Matthias Rolf (Internal Examiner)

Reader
School of Engineering, Computing and Mathematics

Oxford Brookes University

Preface

This dissertation is submitted for the degree of Doctor of Philosophy at
Oxford Brookes University, United Kingdom. The research presented herein was
undertaken under the kind supervision of Prof. Fabio Cuzzolin and Dr. Tjeerd
Olde Scheper between February 2020 and February 2023. To the best of my
knowledge, this work is original, excluding those previous works for which
acknowledgment and reference have been made. Neither this nor any
substantially similar dissertation has been or is being submitted for any
other degree, diploma, or qualification at any other university. Part of this
work has been published as follows:

  * •

Salman Khan, Izzeddin Teeti, Andrew Bradley, Mohamed Elhoseiny, and Fabio
Cuzzolin, “A Hybrid Graph Network for Complex Activity Detection in Video”,
Pre-print arXiv, (2023).

  * •

Gurkirt Singh, …, Salman Khan, …, and Fabio Cuzzolin, “ROAD: The ROad event
Awareness Dataset for autonomous Driving”, in IEEE Transactions on Pattern
Analysis and Machine Intelligence (IEEE TPAMI) 45, pp. 1036-1054, January
2023.

  * •

E. Giunchiglia, Salman Khan, …, F. Cuzzolin and T. Lukasiewicz, “ROAD-R: the
Autonomous Driving Dataset for Learning with Requirements”, Machine Learning,
Springer, September 2022 (accepted with minor revision)

  * •

A. Shahbaz, Salman Khan, … and F. Cuzzolin, “International Workshop on
Continual Semi-Supervised Learning: Introduction, Benchmarks and Baselines”,
in CSSL 2021, Lecture Notes in Artificial Intelligence Vol. 13418, pages 1-14,
Springer, 2022

  * •

Salman Khan and Fabio Cuzzolin, “Spatiotemporal Deformable Scene Graphs for
Complex Activity Detection”, in Proceedings of The 32nd British Machine Vision
Conference (BMVC), 2021.

  * •

Eleonora Giunchiglia, Mihaela Catalina Stoian, Salman Khan, Fabio Cuzzolin and
Thomas Lukasiewicz, “ROAD-R: The Autonomous Driving Dataset with Logical
Requirements”, in Proceedings of IJCAI 2022 Workshop on Artificial
Intelligence for Autonomous Driving (AI4AD 2022) and IJCLR 2022.

  * •

Ajmal Shahbaz, Salman Khan, Mohammad Asiful Hossain, Vincenzo Lomonaco, Kevin
Cannons, and Fabio Cuzzolin, International Workshop on Continual Semi-
Supervised Learning: Introduction, Benchmarks and Baselines, in Proceedings of
IJCAI Workshop, International Workshop on Continual Semi-Supervised Learning
(2021).

  * •

Fabio Cuzzolin, Salman Khan, Ajmal Shahbaz “Continual Semi-Supervised Learning
for Classification via a Time-Varying Hidden Markov Model”, Pre-print arXiv
2023.

Dedication

In memory of my father

Late _Mushtaq Ahmad_

With love and eternal appreciation

&

I would like to dedicate this work to my parents, siblings, and wife. Without
whom none of my success would be possible

_“The best things that a man can leave behind are three: A righteous son who
will pray for him, ongoing charity whose reward will reach him, and knowledge
which is acted upon after his death.”_

[The Messenger of Allah: Prophet Muhammad SAW

\- Sunan Ibn Majah Vol. 1, Book 1, Hadith 241 ]

Acknowledgements

First and foremost, I would like to praise Allah the Almighty, the Most
Gracious, and the Most Merciful for His blessing given to me during my study
and in completing this thesis. May Allah’s blessing goes to His final Prophet
Muhammad (peace be upon him), his family, and his companions.

I would like to express my heartfelt thanks to my PhD supervisor Prof. Fabio
Cuzzolin, and co-supervisor Dr. Tjeerd Olde Scheper for providing such an
exceptional opportunity for me to pursue a doctorate in such an interesting
topic as Computer Vision! I also thank my supervisors for their invaluable
time, guidance and support. In particular, Fabio’s strive for excellence,
openness, and positive criticism always inspired me to push myself beyond my
limits and allowed me to gradually improve my scientific, technical and
academic skills. Moreover, human values like elegance and generosity that I
found in Prof. Cuzzolin have inspired me to work in the Visual Artificial
Intelligence Lab (VAIL), School of Engineering, Computing and Mathematics,
Oxford Brookes University.

I would also like to thank my co-authors and collaborators Dr. Andrew Bradley,
Dr. Gurkirt Singh, Dr. Eleonora Giunchiglia, and Izzeddin Teeti for their time
and effort spent during our long technical discussions in our various
projects.

I want to thank: Tjeerd Olde Scheper and Matthias Rolf for their invaluable
guidance and support as research tutors; Jill Organ, Catherine Joyejob, and
Claire Ferone for helping me with all administrative tasks; all the members of
VAIL with whom I had a chance to work, especially Inna Skarga-Bandurova, Vivek
Singh, Neha Bhargava, Dinesh Jackson Samuel, Mohamed Mohamed, Bogdan Cirstea,
Ajmal Shahbaz, Shireen Kudukkil, Aytac Kanaci, and Amirul Islam.

I am grateful to all external collaborators I worked with: Reza Javanmard,
Kevin Cannons, Naeemullah Khan, Mihaela Catalina Stoian, and Valentina Musat.

My special thanks go to Prof. Mohamed H. Elhoseiny for offering me an
internship at his lab Vision-CAIR, King Abdullah University of Science and
Technology (KAUST), Saudi Arabia. I feel fortunate to have had the chance to
work in a highly stimulating environment and thankful to the Vision-CAIR
members for their invaluable time and guidance during my stay at KAUST.

Due acknowledgement goes to my MS supervisors Prof. Sung Wook Baik and Prof.
Khan Muhammad for their time, advice and support during my MS dissertation. I
would also like to thank my BS supervisor Prof. Muhammad Sajjad for his
guidance and support.

I am thankful to my friends, who have greatly supported me for many years, Dr.
Amin Ullah, Dr. Ijaz Ul Haq, Dr. Tanveer Husaain, Dr. Fath U Min Ullah,
Muhammad Irfan, Imran Ullah, Hayat Ullah, Abid Ali, Hamza Ikram, Yahya Shah,
Malik Yasir, and Uzair Shah.

Last but not least, I am thankful for my _parents_ and my _family_ in Pakistan
for their endless patience, encouragement, and generosity. Especially, my late
Father _Mushtaq Ahmad_ who did not only raise and mature me but also struggled
over the years for my education to make me what I am today, and my elder
brother _Dr. Ishtiaq Ahmad_ to be my mentor since my childhood. Without their
perseverance, I would not be writing this today. My heartfelt thanks go to my
wife _Malaika_ for her unbounded patience and pushing me to be my best self.

Abstract

Dynamic scene understanding is the ability of a computer system to interpret
and make sense of the visual information present in a video of a real-world
scene. In this thesis, we present a series of frameworks for dynamic scene
understanding starting from road event detection from an autonomous driving
perspective to complex video activity detection, followed by continual
learning approaches for the life-long learning of the models.

Firstly, we introduce the ROad event Awareness Dataset (ROAD) for Autonomous
Driving, to our knowledge the first of its kind. ROAD is designed to test an
autonomous vehicle’s ability to detect road events, defined as triplets
composed by an active agent, the action(s) it performs and the corresponding
scene locations. Due to the lack of datasets equipped with formally specified
logical requirements, we also introduce the ROad event Awareness Dataset with
logical Requirements (ROAD-R), the first publicly available dataset for
autonomous driving with requirements expressed as logical constraints, as a
tool for driving neurosymbolic research in the area.

Next, we extend event detection to holistic scene understanding by proposing
two complex activity detection methods. In the first method, we present a
deformable, spatiotemporal scene graph approach, consisting of three main
building blocks: action tube detection, a 3D deformable RoI pooling layer
designed for learning the flexible, deformable geometry of the constituent
action tubes, and a scene graph constructed by considering all parts as nodes
and connecting them based on different semantics. In a second approach
evolving from the first, we propose a hybrid graph neural network that
combines attention applied to a graph encoding of the local (short-term)
dynamic scene with a temporal graph modelling the overall long-duration
activity. Our contribution is threefold: i) a feature extraction technique;
ii) a method for constructing a local scene graph followed by graph attention,
and iii) a graph for temporally connecting all the local dynamic scene graphs.

Finally, the last part of the thesis is about presenting a new continual semi-
supervised learning (CSSL) paradigm, proposed to the attention of the machine
learning community. We also propose to formulate the continual semi-supervised
learning problem as a latent-variable.

###### Contents

  1. 1 Introduction
    1. 1.1 Video Action Recognition and Detection
    2. 1.2 Event Detection
      1. 1.2.1 Road Event Detection
    3. 1.3 Temporal Action Localisation
    4. 1.4 Complex Activity Detection
    5. 1.5 Scene Graphs
    6. 1.6 Continual Learning
    7. 1.7 Contributions of the Thesis
    8. 1.8 Organisation of the Thesis
    9. 1.9 Publications
    10. 1.10 List of Packages and Demos
  2. 2 Related Work
    1. 2.1 Autonomous Driving Datasets
    2. 2.2 Temporal Action Localisation
    3. 2.3 Complex Activity Detection
      1. 2.3.1 One-Stage vs Two-Stage Approaches
    4. 2.4 Graph Networks
      1. 2.4.1 GCN-Based Temporal Activity Localisation
    5. 2.5 Continual Learning
      1. 2.5.1 Continual Unsupervised Learning
      2. 2.5.2 Time-Varying Hidden Markov Models
    6. 2.6 Benchmark Datasets
      1. 2.6.1 ROAD
      2. 2.6.2 SARAS-ESAD
      3. 2.6.3 Thumos-14
      4. 2.6.4 ActivityNet-1.3
      5. 2.6.5 50-Salad
    7. 2.7 Evaluation Metrics
      1. 2.7.1 Classification Accuracy
      2. 2.7.2 Precision
      3. 2.7.3 Recall
      4. 2.7.4 F1-score
      5. 2.7.5 Mean Average Precision
      6. 2.7.6 Mean Absolute Error
  3. 3 The ROad event Awareness Dataset for Autonomous Driving
    1. 3.1 Introduction
      1. 3.1.1 Motivation
      2. 3.1.2 ROAD: a multi-label, multi-task dataset
      3. 3.1.3 ROAD-R
      4. 3.1.4 Main Contributions
    2. 3.2 The Dataset
      1. 3.2.1 A Multi-Label Benchmark
      2. 3.2.2 Data Collection
      3. 3.2.3 Annotation Process
      4. 3.2.4 Tasks
      5. 3.2.5 Quantitative Summary
    3. 3.3 Baseline and Challenge
      1. 3.3.1 3D-RetinaNet Architecture
      2. 3.3.2 Online Tube Generation via Agentness Score
      3. 3.3.3 The ROAD Challenge
    4. 3.4 ROAD-R
    5. 3.5 Experimental Validation
      1. 3.5.1 Implementation Details
      2. 3.5.2 Experimental Results on ROAD
      3. 3.5.3 Qualitative Results
      4. 3.5.4 Challenge Results
      5. 3.5.5 Experimental Results of ROAD-R
    6. 3.6 Further Extensions
    7. 3.7 Summary of the Chapter
  4. 4 Spatiotemporal Deformable Scene Graphs for Complex Activity Detection
    1. 4.1 Introduction
      1. 4.1.1 Motivation
      2. 4.1.2 Objective
      3. 4.1.3 Main Contributions
    2. 4.2 Proposed Method
      1. 4.2.1 Action Tube Detection
      2. 4.2.2 3D Part-Based Deformable Layer
      3. 4.2.3 Graph Convolutional Network
      4. 4.2.4 Complete Framework
    3. 4.3 Experiments
      1. 4.3.1 Datasets and Evaluation Metrics
      2. 4.3.2 Action Tube Detection
      3. 4.3.3 Complex Activity Recognition
      4. 4.3.4 Temporal Activity Detection - Comparison with State-of-the-Art
      5. 4.3.5 Ablation Studies
    4. 4.4 Summary of the chapter
      1. 4.4.1 Limitations and Future Work
      2. 4.4.2 Conclusion
  5. 5 A Hybrid Graph Network for Complex Activity Detection in Video
    1. 5.1 Introduction
      1. 5.1.1 Motivation
      2. 5.1.2 Objectives
      3. 5.1.3 Contributions
    2. 5.2 Proposed Method
      1. 5.2.1 Tube Detection and Feature Extraction
      2. 5.2.2 Scene Graph Attention
      3. 5.2.3 Temporal Graph Localisation
      4. 5.2.4 Loss Functions
    3. 5.3 Experiments
      1. 5.3.1 Datasets
      2. 5.3.2 Implementation Details
      3. 5.3.3 Comparison with State-of-the-Art
      4. 5.3.4 Ablation Studies
    4. 5.4 Summary of the chapter
      1. 5.4.1 Future Work
      2. 5.4.2 Conclusion
  6. 6 Continual Semi-Supervised Learning
    1. 6.1 Introduction
      1. 6.1.1 Motivation
      2. 6.1.2 Overview of the CSSL Challenge
      3. 6.1.3 Overview of Hidden Markov Model as Latent Variable
      4. 6.1.4 Main Contributions
    2. 6.2 Continual Semi-Supervised Learning Workshop and Challenge
      1. 6.2.1 Learning
      2. 6.2.2 Testing
      3. 6.2.3 Benchmarks
      4. 6.2.4 Challenges
      5. 6.2.5 Tasks
      6. 6.2.6 Baselines
    3. 6.3 HMM Latent Variable Model for Unsupervised Data-Streams
      1. 6.3.1 Principles of HMM Latent Variable Model
      2. 6.3.2 A Two-Stage Learning Process
      3. 6.3.3 Joint Distribution of the Data Stream
      4. 6.3.4 Discrete-State T-HMM for Continuous Classification
      5. 6.3.5 Random Sampling based Memory Replay
      6. 6.3.6 Alternative to HMM as Latent Variable Model
    4. 6.4 Experimental Analysis of Challenge Baselines
      1. 6.4.1 Results on Activity Recognition
      2. 6.4.2 Results on Crowd Counting
    5. 6.5 Experimental Analysis of HMM latent Variable
      1. 6.5.1 Datasets and Setups
      2. 6.5.2 Quantitative Analysis
    6. 6.6 Summary of the Chapter
  7. 7 Conclusions and Future Research Directions
    1. 7.1 Summary of the Thesis
      1. 7.1.1 ROad event Awareness Dataset for Autonomous Driving
      2. 7.1.2 Complex Activity Detection
      3. 7.1.3 Continual Semi-Supervised Learning
    2. 7.2 Future Research Directions
      1. 7.2.1 Extension of ROAD
      2. 7.2.2 Holistic Graph Approach for Complex Activity Detection and Prediction
      3. 7.2.3 Extension to Spatiotemporal Sentence Grounding
      4. 7.2.4 Continual Learning Extension to Regression

###### List of Figures

  1. 1.1 Our complex activity recognition framework workflow for two videos from the ROAD (autonomous driving) and SARAS (surgical robotics) datasets. The overall concept of detecting complex activities via a part-based deformation mechanism is the same in both cases. However, in ROAD background frames exist which do not contain any complex road activity. In contrast, the SARAS complex activities are actually the phases of a surgical procedure, which are contiguous without the need for a background label.
  2. 3.1 Use of labels in ROAD to describe typical road scenarios. (a) A green car is in front of the AV while changing lanes, as depicted by the arrow symbol. The associated event will then carry the following labels: in vehicle lane (location), moving left (action). Once the event is completed, the location label will change to: in outgoing lane. (b) Autonomous vehicle turning left from lane 6 into lane 4: lane 4 will be the outgoing lane as the traffic is moving in the same direction as the AV. However, if the AV turns right from lane 6 into lane 4 (a wrong turn), then lane 4 will become the incoming lane as the vehicle will be moving into the incoming traffic. The overall philosophy of ROAD is to use suitable combinations of multiple label types to fully describe a road situation, and allow a machine learning algorithm to learn from this information.
  3. 3.2 Number of instances of each class of individual label-types, in logarithmic scale.
  4. 3.3 Proposed 3D-RetinaNet architecture for online video processing.
  5. 3.4 Success cases in which our model detects the actions and locations correctly, and only for those agents which are active. Ground truth bounding boxes and labels are shown in green, while the predictions of our model are shown in red.
  6. 3.5 Some of the failure modes of our model, shown detecting inactive agents and/or assigning incorrect action and location labels. Ground truth bounding boxes and labels are shown in green, while the predictions of our model are shown in red.
  7. 3.6 Examples of violations of $\\{\neg{\text{RedTL}},\neg{\text{GreenTL}}\\}$.
  8. (a) I3D
  9. (b) C2D
  10. (c) RCGRU
  11. (d) RCLSTM
  12. (e) RCN
  13. (f) SlowFast
  14. 3.7 ROAD-R and SOTA models. In the $x$-axis, there is the threshold $\theta\in[0.1,0.9]$, step 0.1.
  15. (a) Percentage of predictions violating at least one constraint.
  16. (b) Average number of violations committed per prediction.
  17. (c) Percentage of constraints violated at least once.
  18. 4.1 Overall pipeline of our long-term complex activity detection framework. (i) The input video is first divided into snippets. (ii) Snippets are passed to an action tube detector module one by one. (iii) A part-based deformation module receives 3D VGG features and action tube locations and returns features for the salient (non-background) parts of each tube instance. (iv) A GCN module represents activity parts (action tubes) as nodes with the features generated by ROI pooling and builds edges with different semantics to construct a spatiotemporal graph representation. Finally (v), the graph representation features produced by GCN inference on the consolidated graph are used for temporal activity detection.
  19. 4.2 (a) AMTNet [1] generates micro-action tubes by predicting bounding boxes for the start and the end frame of a snippet. Predictions for a fixed number of intermediate frames are constructed by bilinear interpolation. (b) Complete action tubes are created by temporally linking micro action tubes by dynamic programming.
  20. 4.3 Our 3D deformable RoI pooling layer takes feature maps and action tube locations as input and arranges them into a fixed-size grid of components (here illustrated for size $3\times 3$). For each grid component an offset is generated and multiplied by the original tube feature to produce the final component features.
  21. 4.4 Our graph module takes as input the features generated for each tube by the ROI pooling layer $T_{1,2,3...K}$ and builds edges between them according to different semantics. ( order of appearance, spatial similarity, label). The overall graph is processed by a GCN to deliver a fixed-size graph representation.
  22. 4.5 Complex activity classification performance over ROAD dataset.
  23. 4.6 Complex activity classification performance on SARAS-ESAD dataset.
  24. 4.7 The performance of different components of our proposed method over ROAD and SARAS-ESAD dataset. The mAP at a threshold of 0.5 is reported for the test set of both datasets.
  25. 4.8 The performance comparison of various edge types in our GCN modules. The mAP at a threshold of 0.5 is reported for the test set of both datasets.
  26. 5.1 An overview of our Complex Activity Detection (CompAD) framework. The input video is divided into fixed-size snippets $S_{1},\ldots,S_{N}$ (top); each snippet is then processed in three major steps (bottom). A) Firstly, scene objects (agents) are detected and tracked throughout the snippet to form agent tubes. 3D features are then extracted from all the cropped agent tubes ($f_{a^{i}}$) as well as the local scene ($f_{s_{i}}$). B) Next, a fully-connected scene graph is constructed where agent nodes (in gray) are connected to each other and to the snippet node (in light blue). The local scene graph is processed using a graph attention network (GAT), resulting in intermediate scene features ($f^{\prime\prime}_{s_{i}}$). C) Finally, all local scene features associated with individual snippets are temporally connected and processed using a global temporal graph to identify the boundaries of the activity using anchor proposals (shown in different colors). Each of the anchors represents a binary graph mask where snippets which belong to the activity receive label $y_{{br}_{i}}=1$, 0 otherwise.
  27. 5.2  Comparing TAL datasets (ActivityNet-1.3 and Thumos-14) with a CompAD dataset (ROAD) in terms of average activity and video durations and mean number of agents per snippet.
  28. 5.3 Visualisation of our agent detection and tracking stage using a bird’s-eye view of the spatiotemporal volume corresponding to a video segment of _ActivityNet-1.3_ dataset. The first half of the graph shows the plotting of agent tubes while the second half illustrates the scene graph of the detection agents. In this figure, we show examples of activities performed by multiple actors (agent tubes).
  29. 5.4 Visualisation of our agent detection and tracking stage using a bird’s-eye view of the spatiotemporal volume corresponding to a video segment of _Thumos-14_ dataset. The first half of the graph shows the plotting of agent tubes while the second half illustrates the scene graph of the detection agents. Both of the examples show the activities performed by multiple agents (recognising sports activities).
  30. 5.5 Visualisation of our agent detection and tracking stage using a bird’s-eye view of the spatiotemporal volume corresponding to a video segment of the _ROAD_ dataset. The upper section shows an example of activity performed by only three agents. Below is the example of a night video where the activity is performed by multiple agents. The first half of the graph shows the plotting of agent tubes while the second half illustrates the scene graph of the detection agents.
  31. 5.6 Our scene graph attention layer (Stage B of our approach, Fig. 5.1) takes the node features generated in the feature extraction Stage A as an input, and updates each node’s features using node attention with respect to its all connected neighbours. Multi-head attention with $H$ heads is applied to each node to further robustify the representation – averaging yields the final node features. To obtain a fixed-size overall representation for a specific local scene (snippet), the features of all its nodes are aggregated using a learnable weight matrix $W_{2}$.
  32. 5.7 Class-wise average precision (%) with IoU threshold of 0.5 for ROAD dataset.
  33. 5.8 Class-wise average precision (%) with IoU threshold of 0.5 for Thumos-14 dataset.
  34. 5.9 Performance comparison with the state-of-the-art over all the three datasets considered, as a function of average activity duration and average no. of agents per snippet.
  35. 5.10 Average mAP of variants of our method with: attended scene graph (’Agents with GAT’), scene graph (’Agents without GAT’) and ’Scene features only’, over the all three datasets.
  36. 5.11 The qualitative results of our proposed method for all three datasets. The green rectangles covering the snippets (local scenes) are the ground truth while the yellow boxes show the prediction of our model.
  37. 5.12 The qualitative results of our proposed method for all three datasets. The green rectangles covering the snippets (local scenes) are the ground truth while the yellow boxes show the prediction of our model.
  38. 6.1 Overview of the proposed latent variable pipeline.
  39. 6.2 Overall pipeline of our CAR baseline. 1) Firstly, a pre-trained EfficientNet model is fine-tuned over the supervised fold of the dataset (jointly over the 15 sequences). 2) The unlabeled validation and test folds are divided into subfolds. In each session video frames from each sub-fold are used in a self-training cycle in which a pseudo label $P_{L}$ with prediction probability score $P_{s}$ is generated for each frame, but only the pseudolabels with $P_{s}$ above a fixed threshold (namely, 0.4) are considered as ground truth for retraining the model. Frames with lower $P_{s}$ are ignored. The model updated by self-training in session $n$ is used as initial model in session $n+1$.
  40. 6.3 Time-varying HMM (dashed bars) vs. traditional HMM (colored bars).

###### List of Tables

  1. 2.1  Comparison of ROAD with similar datasets for perception in autonomous driving in terms of diversity of labels. The comparison is based on the number of classes portrayed and the availability of action annotations and tube tracks for both pedestrians and vehicles, as well as location information. Most competitor datasets do not provide action annotation for either pedestrians or vehicles.
  2. 3.1 ROAD labels.
  3. 3.2 ROAD tasks and attributes.
  4. 3.3 Constraint statistics. $\Pi_{n}$ is the set of constraints $r$ in $\Pi$ with $|r|=n$, i.e., with $n$ positive and negative labels. $\overline{\mathcal{C}}=\\{\neg{A}:A\in\mathcal{C}\\}$. Each row shows the number of rules $r$ with $|r|=n$, and the average number of negative and positive labels in such rules.
  5. 3.4 Splits of training, validation and test sets for the ROAD dataset with respect to weather conditions. The table shows the number of videos in each set or split. For splits, the first figure is the number of training videos, the second number that of validation videos.
  6. 3.5 Frame-level results (mAP $\%$) averaged across the three splits of ROAD. The considered models differ in terms of backbone network (2D, I3D, and Slowfast) and clip length (08 vs 32). The performance of YOLOv5 on agent detection is also reported. Detection threshold $\delta=0.5$. Both validation and test performance are reported for each entry.
  7. 3.6 Video-level results (mAP $\%$) averaged across the three ROAD splits. The models differ in terms of backbone network (2D, I3D and Slowfast) and test time clip length (08 vs 32). The performance of YOLOv5 on agent detection is also reported. Both validation and test performance are reported for each entry.
  8. 3.7 Number of video- and frame-level instances for each label (individual or composite), left. Corresponding frame-/video-level results (mAP@$\%$) for each of the three ROAD splits (right). Val-$n$ denotes the validation set for Split $n$. Results produced by an I3D backbone.
  9. 3.8 AV temporal action detection results (frame mAP$\%$) averaged across all three splits.
  10. 3.9 Results (in video-mAP) of the winning entries to the ICCV 2021 ROAD challenge compared with the Slowfast and YOLOv5 baselines, at a detection threshold of $0.2$.
  11. 3.10<EMAIL_ADDRESS>and<EMAIL_ADDRESS>for the current SOTA models.
  12. 4.1 List of ROAD complex activities with number of train and test snippets splits and brief description.
  13. 4.2 SARAS video phases with number of train and test snippets and a brief description.
  14. 4.3 Action tube detection performance on both the ROAD and SARAS-ESAD datasets. Both Frame-mAP and Video-mAP at different IoU thresholds are reported for evaluation
  15. 4.4 Comparative analysis of temporal activity localisation performance on ROAD and SARAS-ESAD, reporting mAP (%) at five different IoU thresholds.
  16. 4.5 ROAD activity localisation performance (mAP, %) for each complex road activity, at a standard IoU threshold of 0.5.
  17. 4.6 SARAS-ESAD activity localisation performance (mAP, %) for each activity at a fixed IoU threshold of 0.5.
  18. 4.7 The performance comparison of different sequence lengths over both ROAD and SARAS-ESAD dataset, reporting mAP (%) at two different IoU thresholds.
  19. 5.1 Comparing our approach with the state-of-the-art methods for CompAD available on the ROAD dataset. The mAP at the various standard thresholds is reported. Best results are in bold and second best underlined.
  20. 5.2 Activity detection performance comparison with state-of-the-art methods on Thumos-14. mAP values at different IoU thresholds are reported for the test Thumos-14. Best results are in bold and second best underlined.
  21. 5.3 Activity detection performance comparison with state-of-the-art methods on ActivityNet-1.3. mAP values at different IoU thresholds are reported for validation sets of ActivityNet-1.3. Best results are in bold and second best underlined.
  22. 5.4 Effect of local scene graph topologies and feature aggregation strategies on the performance of our proposal. The average mAP for the relevant IoU thresholds is reported for each dataset.
  23. 5.5 Performance of our method as a function of different snippet lengths over the three datasets. The average mAP for the relevant IoU thresholds is reported for each dataset.
  24. 5.6 Performance of our method as a function of different temporal graph lengths over the three datasets. The average mAP for the relevant IoU thresholds is reported for each dataset.
  25. 6.1 Performance of the initial supervised model versus that of the incrementally updated models (separately on $V$ and $T$ or in combination), using three standard evaluation metrics. For each metric we report both class average (C) and weighted average (W).
  26. 6.2 Evaluation on the CCC validation ($V$) and test ($T$) folds for different experimental setups. Here $sup-ft$, $upd-V$, and $upd-V+T$ refer to the supervised model fine-tuned on $S$, the model incrementally updated via self-training in sessions on the validation fold $V$, and the model incrementally updated on the combined validation $V$ and test $T$ folds, respectively. The average test MAE is reported for each model.
  27. 6.3 Time-varying HMM (TV HMM) vs. traditional HMM (HMM) based on one-off training, where P, R, and F1 are precision, recall, and F1-score, respectively.
  28. 6.4 Time-varying HMM (TV HMM) vs. traditional HMM (HMM) based on continual retraining, where P, R, and F1 are precision, recall, and F1-score, respectively.
  29. 6.5 Time-varying HMM (TV HMM) vs. traditional HMM (HMM) based on incremental update, where P, R, and F1 are precision, recall, and F1-score, respectively.

## Nomenclature

  * Acronyms
  * C3D:

Convolutional 3D Network

  * I3D:

Inflated Convolutional 3D Network

  * mAP:

mean Average Precision

  * RGB:

Red Green Blue, 3-Channel Camera output

  * TAL:

Temporal Action Localisation

  * AUC:

Area Under the Curve

  * AV:

Autonomous Vehicle

  * BB:

Bounding Box

  * CompAD:

Complex Activity Detection

  * CL:

Constrained Loss

  * CO:

Constrained Output

  * CAR:

Continual Activity Recognition

  * CCC:

Continual Crowd Counting

  * CSSL:

Continual Semi-Supervised Leanring

  * CNN:

Convolutional Neural Network

  * EM:

Expectation Maximisation

  * FPN:

Feature Pyramid Network

  * fps:

frames per second

  * GAT:

Graph Attention Network

  * GCN:

Graph Convolutional Network

  * HMM:

Hidden Markov Model

  * HMC:

Hierarchical Multi-label Classification

  * IoU:

Intersection over Union

  * LSTM:

Long Short-Term Memory recurrent neural network

  * MAE:

Mean Absolute Error

  * MC:

Multi-label Classification

  * 1D:

One Dimensional

  * RNN:

Recurrent Neural Network

  * RE:

ROAD Event

  * SSL:

Semi-Supervised Learning

  * ESAD:

the Endoscopic Surgeon Action Detection Dataset

  * ROAD:

The ROad Awareness Dataset for Autonomous Driving

  * ROAD-R:

The ROad Awareness Dataset with logical Requirements

  * 3D:

Three Dimensional (spatiotemporal for video)

  * T-HMM:

Time-varying Hidden Markov Model

  * 2D:

Two Dimensional (spatially for image)

##  Chapter 1 Introduction

Scene understanding from videos has a significant role in the computer vision
community due to its vast real-world applications such as autonomous driving
[2, 3], surveillance [4], medical robotics [5] or team sports analysis [6].
The scene understanding problem can be addressed by a variety of techniques
starting from simple video action recognition to complex activity detection.
In this thesis, we address the problem in a comprehensive fashion, starting
from individual event detection to holistic complex scene analysis (complex
activity detection).

###  1.1 Video Action Recognition and Detection

_Video action recognition_ is the task of identifying and classifying human
actions in video sequences. Video _action detection_ , on the other hand, is
the process of identifying and locating the actions of an individual within a
video, in both space and time. In the current literature, a number of
convolutional neural networks (CNNs) frameworks are used to learn spatial
features from individual video frames, which are then fed to recurrent neural
networks (RNNs) to model the temporal dynamics across frames [7]. Another
approach is to use 3D CNNs, which can learn spatiotemporal features directly
from the video data [8].

One of the key challenges in video action recognition and detection is the
need to deal with the large variability in the appearance and motion of the
actions across different videos. To address this, researchers often use
techniques such as data augmentation and transfer learning to improve the
robustness of the models [9, 10]. Additionally, some researchers have proposed
using attention mechanisms to allow the model to focus on the most relevant
parts of the video when detecting and recognising actions [11]. Video action
recognition and detection have applications to a wide range of areas such as
surveillance, human-computer interaction and sports analytics, to name a few.

###  1.2 Event Detection

_Video event detection_ refers to the task of identifying and localising
specific events or activities in a video. Such activities may be performed by
a single agent (object) or by multiple agents at the same time. Examples
include activities such as sports events [12], traffic incidents [13],
disaster management [14, 15] or criminal activities. The most popular deep
learning-based method for video event detection is the two-stream CNN, which
uses two streams of CNNs, one for processing the RGB frames and the other for
processing the optical flow frames [1]. Another popular method is the 3D CNN,
which uses 3D convolutional kernels that can learn spatiotemporal features
from the video. These features can then be used to detect events over time,
making 3D CNNs useful for tasks such as road events for decision-making in
autonomous driving.

####  1.2.1 Road Event Detection

In this thesis, we define _road events_ to be the collection of three labels
(agent, action, and location) associated with a single “tube” (the
spatiotemporal representation of a series of bounding boxes). The agent is an
active road user (object); the event triple also contains the action(s) the
latter performs (possibly more than one at the same time), and the location(s)
in which this takes place (which may vary from the start to the end of the
event itself), as seen from the point of the view of an autonomous vehicle.

###  1.3 Temporal Action Localisation

The ability to recognise and localise activities temporally from untrimmed
videos is a research problem which has been attracting significant attention,
due in part to the rapid growth of online video generation. The current state-
of-the-art in action detection is _Temporal Action Localisation_ (TAL) - whose
approaches not only recognise the action label(s) present in a video, but also
identify the start and end time of each action instance occurring.

TAL can be instrumental in generating sports highlights [16], understanding
road scenes in autonomous driving [17], video summarisation in surveillance
videos [18] and video captioning [19]. A number of TAL methods [20, 21, 22,
23, 24, 25, 26, 27, 28, 29] have recently been proposed. Whereas various new
datasets have been recently proposed, the two most common relevant benchmarks
remain ActivityNet 1.3 [30], and Thumos-14 [31]. State-of-the-art performance
on Thumos-14 has improved in four years by some 19% [32].

Almost all TAL approaches contemplate two major aspects: _features/scene
representation_ and _temporal localisation_. In the scene representation
stage, snippets (continuous sequences of frames) are processed to understand
the local scene in the video. Methods [27, 28] typically employ pre-extracted
features obtained using a sequential learning model (e.g., I3D [33]), often
pre-trained on the Kinetics [34] dataset. Such features are then processed,
e.g., via a temporal or semantic graph neural network, by applying appropriate
feature encoding techniques or by generating temporal proposals, in an object
detection style [35]. In the second step of TAL, actions need to be temporally
localised. Existing methods vary in the way this is done, e.g. through
temporal graphs [25, 21], boundary regression and proposals generation [20,
22, 28] or encoder-decoder approaches [32]. All such methods, however, are
only applicable to short- or mid-duration actions that last for a few seconds
(e.g., a person jumping or pitching a baseball).

Figure 1.1: Our complex activity recognition framework workflow for two videos
from the ROAD (autonomous driving) and SARAS (surgical robotics) datasets. The
overall concept of detecting complex activities via a part-based deformation
mechanism is the same in both cases. However, in ROAD background frames exist
which do not contain any complex road activity. In contrast, the SARAS complex
activities are actually the phases of a surgical procedure, which are
contiguous without the need for a background label.

###  1.4 Complex Activity Detection

_Complex activities_ , in opposition, can be defined as longer-term events,
typically comprising a series of shorter actions. For example, an autonomous
vehicle (AV) “negotiating a pedestrian crossing” presents a complex activity -
first the AV drives along the road, then the traffic lights switches to red,
the vehicle stops and several pedestrians cross the road. Eventually, the
lights change back to green, and the AV drives off again. Whereas a number of
studies have attempted to address the localisation of longer-term activities
[36, 37, 38], these approaches mostly focus on activities performed by single
actor. To the author’s knowledge, [36] is the only existing study which
attempts to model truly complex activities, performed by multiple actors over
an extended period of time. The work, however, relies upon the availability of
heavily-annotated datasets which include granular labels for the individual
actions which make up a complex activity, and the corresponding frame-level
bounding boxes. A general overview of a complex activity framework is
visualised in Figure 1.1.

###  1.5 Scene Graphs

A _scene graph_ is a graph-based representation of a scene, where the nodes of
the graph represent entities in the scene (such as objects, people, and
locations) and the edges represent relationships between those entities (such
as proximity, or interaction) [39]. Scene graphs are used in computer vision
[40] and natural language processing [41] to represent the semantic structure
of a scene and to facilitate tasks such as object detection, scene
understanding, and natural language generation.

Scene graphs are typically constructed by analysing images or videos, using
techniques such as object detection, instance segmentation, and scene
understanding [42]. The nodes of the graph are typically labeled with object
categories, and the edges are labeled with the type of relationship between
the entities.

Due to the vast importance of scene graphs in scene understanding, in this
thesis, we also adopt a scene graph representation for modelling complex
activities.

###  1.6 Continual Learning

_Continual learning_ refers to the ability of a machine learning model to
learn and adapt to new information over time, without forgetting previously
acquired knowledge [43]. In other words, it allows a model to continuously
improve its performance by learning from new data, even after it has been
deployed in the real world. This is an important aspect of developing AI
systems that can function in dynamic, real-world environments and adapt to
changing conditions [44]. Continual learning is an active area of research in
machine learning, with the goal of developing models that can learn and adapt
throughout their lifetime. A typical example is that of an object detector
which needs to be extended to include classes not originally in its list
(e.g., ‘donkey’) while retaining its ability to correctly detect, say, a
‘horse’. The hidden assumption there is that we are quite satisfied with the
model we have, and we merely wish to extend its capabilities to new settings
and classes.

In the last part of the thesis, we also proposed two continual learning
paradigms in Chapter 6 for updating the machine learning model continually in
a semi-supervised manner.

###  1.7 Contributions of the Thesis

The main contributions of this thesis are the following:

  1. 1.

A ROad event Awareness Dataset for Autonomous Driving (ROAD), the first of its
kind, designed to support a paradigm shift toward the detection of
semantically meaningful road events, and allow the testing of a range of tasks
related to situation awareness for autonomous driving: agent and/or action
detection, event detection, ego-action classification.

  2. 2.

The ROad event Awareness Dataset with logical Requirements (ROAD-R), the first
publicly available dataset whose commonsense requirements are expressed in
full propositional logic, equipped with baseline models.

  3. 3.

A spatiotemporal scene graph generation and processing mechanism for complex
activity detection able to cope with a variable number of parts while learning
the overall semantics of an activity class.

  4. 4.

An extension of 2D deformable RoI pooling to 3D for flexibly pooling features
from the various components of the detected tubes, to create an overall
representation for activity parts.

  5. 5.

A hybrid graph network approach for general complex activity detection,
comprising a local scene graph as well as a global temporal graph, capable of
localising both long and short-term activities.

  6. 6.

A feature extraction strategy for fine-grained temporal activity localisation
without the need for additional annotation, consisting in detecting and
tracking the relevant scene agents to construct tubes, followed by feature
extraction from both the whole scene and each agent tube individually, as
opposed to existing methods which only rely on snippet-level features.

  7. 7.

A formal definition of the continual semi-supervised learning (CSSL) problem,
and the associated training and testing protocols, with a suitable baseline.
The first two benchmark datasets for the validation of semi-supervised
continual learning approaches, one for classification (continual activity
recognition) and one for regression (continual crowd counting), which we
propose as the foundations for the first challenge in this new domain area.

  8. 8.

A principled two-stage approach to the CSSL problem, based on a latent model
formulation which reduces the task to continual supervised learning.

###  1.8 Organisation of the Thesis

Firstly, we review all the relevant related work, including the benchmark
datasets and evaluation metrics, in Chapter 2. Next, the main body of the
thesis (Chapters 3 to 7) is divided into three parts; Part I, Part II, and
Part III. Finally, the thesis is concluded in Chapter 7 by a summary and a
discussion of possible future extensions.

The theme of Part I 2.7.6 is ROAD (The ROad event Awareness Dataset for
Autonomous Driving) which includes two chapters 3. In Chapter 3, we present
the baseline ROAD dataset, which is designed to test an autonomous vehicle’s
ability to detect road events, defined as triplets composed by an active
agent, the action(s) it performs, and the corresponding scene locations. We
also introduce the ROad event Awareness Dataset with logical Requirements
(ROAD-R) in Part I, the first publicly available dataset for autonomous
driving with requirements expressed as logical constraints, as a neurosymbolic
extension to ROAD.

Next, in Part II 3.7 we present the concept of Complex Activity Detection
(CompAD), over two Chapters (Chapter 4 and Chapter 5). Chapter 4 presents a
novel deformable, spatiotemporal scene graph approach for CompAD, consisting
of three main building blocks: (i) action tube detection, (ii) the modelling
of the deformable geometry of parts, and (iii) a graph convolutional network.
The latter is further improved in Chapter 5 by addressing the CompAD problem
using a hybrid graph neural network that combines attention applied to a graph
encoding the local (short-term) dynamic scene with a temporal graph modelling
the overall long-duration activity.

Finally, Part III 5.4.2 is about continual learning and is composed of a
single Chapter6. There, we present a new continual semi-supervised learning
(CSSL) paradigm, which we proposed to the attention of the machine learning
community via the IJCAI 2021 International Workshop on Continual Semi-
Supervised Learning. In addition to the baseline made available for the
challenge, we also propose to formulate the continual semi-supervised learning
problem as a latent-variable one, in which the labels of a series of
unlabelled, streaming data points play the role of the hidden states, while
the instances play that of the observations.

###  1.9 Publications

###### Part of thesis:

the following is the list of publications that form the core of this PhD
thesis.

  1. 1.

Salman Khan, Izzeddin Teeti, Andrew Bradley, Mohamed Elhoseiny, and Fabio
Cuzzolin, “A Hybrid Graph Network for Complex Activity Detection in Video”,
Pre-print arXiv, (2023).

  2. 2.

Gurkirt Singh, …, Salman Khan, …, and Fabio Cuzzolin, “ROAD: The ROad event
Awareness Dataset for autonomous Driving”, in IEEE Transactions on Pattern
Analysis and Machine Intelligence (IEEE TPAMI) 45, pp. 1036-1054, January
2023.

  3. 3.

E. Giunchiglia, Salman Khan, …, F. Cuzzolin and T. Lukasiewicz, “ROAD-R: the
Autonomous Driving Dataset for Learning with Requirements”, Machine Learning,
Springer, September 2022 (accepted with minor revision)

  4. 4.

A. Shahbaz, Salman Khan, … and F. Cuzzolin, “International Workshop on
Continual Semi-Supervised Learning: Introduction, Benchmarks and Baselines”,
in CSSL 2021, Lecture Notes in Artificial Intelligence Vol. 13418, pages 1-14,
Springer, 2022

  5. 5.

Salman Khan and Fabio Cuzzolin, “Spatiotemporal Deformable Scene Graphs for
Complex Activity Detection”, in Proceedings of The 32nd British Machine Vision
Conference (BMVC), 2021.

  6. 6.

Eleonora Giunchiglia, Mihaela Catalina Stoian, Salman Khan, Fabio Cuzzolin and
Thomas Lukasiewicz, “ROAD-R: The Autonomous Driving Dataset with Logical
Requirements”, in Proceedings of IJCAI 2022 Workshop on Artificial
Intelligence for Autonomous Driving (AI4AD 2022) and IJCLR 2022.

  7. 7.

Ajmal Shahbaz, Salman Khan, Mohammad Asiful Hossain, Vincenzo Lomonaco, Kevin
Cannons, and Fabio Cuzzolin, International Workshop on Continual Semi-
Supervised Learning: Introduction, Benchmarks and Baselines, in Proceedings of
IJCAI Workshop, International Workshop on Continual Semi-Supervised Learning
(2021).

  8. 8.

Fabio Cuzzolin, Salman Khan, Ajmal Shahbaz “Continual Semi-Supervised Learning
for Classification via a Time-Varying Hidden Markov Model”, Pre-print arXiv
2023.

###### Other works:

Below is the list of other works which I published in the course of my PhD
term, but are not directly connected to the topic of this thesis.

  1. 1.

Muhammad, Khan, Hayat Ullah, Salman Khan, Mohammad Hijji, and Jaime Lloret.
”Efficient Fire Segmentation for Internet-of-Things-Assisted Intelligent
Transportation Systems.” IEEE Transactions on Intelligent Transportation
Systems, September 2022.

  2. 2.

Salman Khan, Khan Muhammad, Tanveer Hussain, Javier Del Ser, Fabio Cuzzolin,
Siddhartha Bhattacharyya, Zahid Akhtar, and Victor Hugo C. de Albuquerque.
”Deepsmoke: Deep learning model for smoke detection and segmentation in
outdoor environments.” Expert Systems with Applications 182 (2021): 115125.

  3. 3.

Teeti, Izzeddin, Salman Khan, Ajmal Shahbaz, Andrew Bradley, and Fabio
Cuzzolin. ”Vision-based Intention and Trajectory Prediction in Autonomous
Vehicles: A Survey.” Proceedings of IJCAI 2022 (Survey track), pages
5630-5637.

  4. 4.

Teeti, Izzeddin, Valentina Musat, Salman Khan, Alexander Rast, Fabio Cuzzolin,
and Andrew Bradley. ”Vision in adverse weather: Augmentation using CycleGANs
with various object detectors for robust perception in autonomous racing.”
ICML 2022 - Workshop on Safe Learning for Autonomous Driving (SL4AD 2022).

  5. 5.

Onyeogulu, Tochukwu, Amirul Islam, Salman Khan, Izzeddin Teeti, and Fabio
Cuzzolin. ”Situation Awareness for Automated Surgical Check-listing in AI-
Assisted Operating Room.” arXiv preprint arXiv:2209.05056 (2022).

  6. 6.

Jin, Kaizhe, Adrian Rubio-Solis, Ravi Naik, Tochukwu Onyeogulu, Amirul Islam,
Salman Khan, Izzeddin Teeti et al. ”Identification of Cognitive Workload
during Surgical Tasks with Multimodal Deep Learning.” arXiv preprint
arXiv:2209.06208 (2022).

###  1.10 List of Packages and Demos

  * •

All resources of our ROAD project (IEEE TPAMI 2022 [45] and The ROAD Workshop
and Challenge: Event Detection for Situation Awareness in Autonomous Driving,
ICCV 2021) are publicly available at:

_Workshop and Challenge link_ :
https://sites.google.com/view/roadchallangeiccv2021/

_ROAD dataset_ : https://github.com/gurkirt/road-dataset

_3D RetinaNet baseline_ : https://github.com/gurkirt/3D-RetinaNet

_Inference demo_ : https://www.youtube.com/watch?v=CmxPjHhiarA

_Challenge evalAI_ https://eval.ai/web/challenges/challenge-
page/1059/overview.

  * •

The resources of our ROAD-R project [46] are publicly available at:

_code_ : https://github.com/EGiunchiglia/ROAD-R

_video_ : https://www.youtube.com/watch?v=_-Ll5d1VQXY.

  * •

The source code of our BMVC 2021 [17] is publicly available at:
https://github.com/salmank255/SDSG_Complex_Activity.

  * •

The source code of our CompAD paper Pre-print is publicly available at:
https://anonymous.4open.science/r/HG_CompAD-7697/README.md.

  * •

All resources of the continual learning project (Continual Semi-Supervised
Learning Workshop and Challenge at IJCAI 2021 [47]) are publicly available at:

_Workshop link_ : https://sites.google.com/view/sscl-workshop-ijcai-2021/

_Code_ : https://github.com/salmank255/cssl

_CAR challenge baseline_ : https://github.com/salmank255/IJCAI-2021-Continual-
Activity-Recognition-Challenge

_CAR evalAI_ : https://eval.ai/web/challenges/challenge-page/984/overview

_CCC challenge baseline_ :
https://github.com/Ajmal70/IJCAI_2021_Continual_Crowd_Counting_Challenge

_CCC evalAI_ : https://eval.ai/web/challenges/challenge-page/986/overview.

##  Chapter 2 Related Work

This chapter presents a literature review of the most relevant methods in the
areas (discussed in Chapter 1) which most contribute to scene understanding.
We start from a comparison of the existing major autonomous driving datasets
(Section 2.1) with our ROAD dataset, which is presented in Chapter 3. Next, we
provide a literature review of temporal action localisation, complex activity
detection, graph convolutional networks, and continual learning in Sections
2.2, 2.3, 2.4, and 2.5, respectively. We also discuss the benchmark datasets
used for the evaluation of our proposed methods in Section 2.6. Lastly, we
introduce all the evaluation metrics (Section 2.7) used for the evaluation of
the different tasks targeted in this thesis.

###  2.1 Autonomous Driving Datasets

In recent years a multitude of AV datasets have been released, mostly focusing
on object detection, scene segmentation, trajectory prediction, and action
detection. We can categorise them into two main bins: (1) RGB without range
data (single modality) which is relatively less time-consuming and expensive
than building multimodal datasets including range data from LiDAR or radar and
(2) RGB with range data (multimodal) that contain multiple types of data, such
as lidar, camera, and radar data.

To compare ROAD (proposed in Chapter 3) with autonomous driving perception
datasets, we provide a comparative analysis of the recent state-of-the-art
datasets in Table 2.1. As it can be noted in the table, the unique feature of
ROAD is its diversity in terms of the types of actions and events portrayed,
by all types of road agents in the scene. With 12 agent classes, 30 action
classes and 15 location classes ROAD provides (through a combination of these
three elements) a much more refined description of road scenes.

Table 2.1:  Comparison of ROAD with similar datasets for perception in
autonomous driving in terms of diversity of labels. The comparison is based on
the number of classes portrayed and the availability of action annotations and
tube tracks for both pedestrians and vehicles, as well as location
information. Most competitor datasets do not provide action annotation for
either pedestrians or vehicles.

Dataset | Class Num. | Ann. frames | Countries | Loc. lab. | Avg. obj./f | Action Ann | Tube Ann
---|---|---|---|---|---|---|---
Ped. | Veh. | Ped. | Veh.
SYNTHIA[48] | 13 | 13400 | 1 | pixelwise ann. | - | - | - | - | -
Cityscapes [49] | 30 | 5000 | 2 | pixel level sem. | - | - | - | - | -
Waymo [50] | 4 | 1,200,000 | 1 | - | 20 | - | - | ✓ | ✓
Apolloscape [51] | 25 | 140,000 | 1 | pixel level sem. | - | - | - | ✓ | ✓
PIE [52] | 6 | 293,437 | 1 | - | 2.5 | ✓ | - | ✓ | -
TITAN [53] | 50 | 75,262 | 1 | - | 5.2 | ✓ | ✓ | ✓ | ✓
KITTI360 [54] | 19 | 83,000 | 1 | sem. ann. | - | - | - | - | -
A*3D [55] | 7 | 39,000 | 1 | - | - | 4.8 | - | - | -
H3D [56] | 8 | 27,721 | 1 | - | 16 | - | - | ✓ | ✓
Argoverse [57] | 15 | 70,557 | 1 | - | 18 | - | - | ✓ | ✓
NuScense [2] | 23 | 40,000 | 2 | 3D sem. seg. | 20 | - | - | ✓ | ✓
DriveSeg [58] | 12 | 5,000 | 1 | sem. ann. | - | - | - | - | -
Spatiotemporal action detection datasets |  |  |  |  |
UCF24 [59] | 24 | 80175 | - | - | 1.5 | ✓ | - | ✓ | -
AVA [60] | 80 | 1,600,000 | - | - | 3 | ✓ | - | ✓ | -
Multisports [61] | 66 | 37701 | - | - | 2.5 | ✓ | - | ✓ | -
ROAD [45] | $43$ | 122,000 | 1 | ✓ | 5 | ✓ | ✓ | ✓ | ✓
Ped. Pedestrian, Veh. Vehicle, ann. annotation, sem. seg. semantic segmentation |  |  |

###  2.2 Temporal Action Localisation

Significant progress has been made in recent years toward temporal action
localisation (TAL) in untrimmed videos. The main goal of TAL is to remove the
irrelevant background from the input videos and identify the start and end
instants of the action of interest. Temporal localisation methods can be
divided into supervised [62, 63] versus weakly/unsupervised ones [64, 65, 66].
While the former requires temporal annotations to learn a model able to
identify the start and end of an action, in weakly-supervised settings the
model is learned from partial annotation.

In Part II of the thesis, our goal is the same as temporal activity
localisation, with the difference that we wish to target _complex, long
duration events articulated into a possibly large number of parts_ (complex
activity detection), whose internal structure cannot be leveraged by naive
temporal segmentation methods.

###  2.3 Complex Activity Detection

Most recent work on complex activity recognition concerns scalar sensors [67,
68, 69] or a combination of both scalar and vision sensors [70, 71]. Recently,
though, several vision-based complex activity recognition methods have been
proposed [63, 72, 73, 74, 75, 40] with the goal of understanding an overall
scene by recognising and segmenting atomic actions. These methods can be
further divided into (i) sliding windows approaches [76, 38], in which an
activity classifier is applied to each snippet, and (ii) boundaries analyses
[77, 78], in which a model is trained to identify the start and end time of
each action. Overall, current activity recognition methods are geared to
recognise short-term activities via a combination of small atomic actions.

Unlike existing approaches, the objective of our complex activity model
(proposed in Chapter 4) is to understand _long-term_ activities in dynamic
scenes, such as the _phases_ a surgical procedure is broken into, whose
detection is crucial to inform the decision making of automated robotic
assistants.

####  2.3.1 One-Stage vs Two-Stage Approaches

Complex activity methods can been broadly divided into _one-stage_ and _two-
stage_ approaches. The former [79, 80, 23, 81] detect actions/activities in a
single shot, and can be easily trained in an end-to-end manner. For example,
Wang et al. [82] detect actions using transformers which, unlike RNNs, do not
suffer from nonparallelism and vanishing gradients. Most one-stage methods,
however, only perform action classification, rather than spatiotemporal
localisation. In contrast, Lin et al. [81] have recently proposed an anchor-
free, one-stage light model which generates proposals, locates actions within
them, and classifies them end-to-end.

The latter group of methods [20, 22, 62, 83, 84], instead, consists of two
stages, similarly to region-proposal object detectors. The first stage
generates suitable proposals for predicting the start and end time of an
activity, while the second stage extracts features and processes the proposals
before passing them to both a classification head and a regression head (for
temporal localisation). Some work, including [22], focus on the first stage to
improve the quality of the proposals, while others focus on the processing or
refining of the proposals. [84], for instance, uses an off-the-shelf method
for proposal generation. The second stage consists of two networks, a
‘disentanglement’ network to separate the classification and regression
representations, and a ‘context aggregation’ network to add them together.
Such methods are not trainable end-to-end and limited to short-or mid-duration
actions. In contrast, we propose a _complex activity detection_ method in
Chapter 5, which exhibits the advantages of both classes of methods, thanks to
our hybrid graph approach capable of recognising and localising both short-
and long-duration activities.

###  2.4 Graph Networks

Recently, Graph Convolutional Networks (GCNs) have been widely used for action
and activity detection and recognition, building on their success in different
areas of computer vision such as point cloud segmentation [85, 86] and 3D
object detection [87]. Relevant GCN approaches have been broadly focussing on
either action recognition [88, 89, 90] or TAL [21, 25]. In the former, videos
are represented in different spatiotemporal formats such as 3D point clouds
and time-space region graphs, and methods focus on recognising atomic actions
only. In contrast, Zeng et al. [21] use GCN for temporal activity localisation
by considering action proposals as nodes and a relation between two proposals
as an edge. In opposition, in our model nodes are action tubes and their
connections are based on an array of semantics. In another recent study, Xu et
al. [25] generate graphs by considering temporal snippets as nodes and drawing
connections between them based on temporal appearance and semantic similarity.

Most graph-based activity detection methods [25, 91, 21] construct a graph for
a whole video by taking snippets as nodes and their temporal linkage as edges,
not paying much attention to the constituent atomic actions within each
snippet, and are typically limited to shorter videos or memory dependent in
case of larger videos due to adjacency matrices. For a large number of
snippets (nodes) in the video required memory can be substantial. The
adjacency matrix of the graph with n nodes (snippets) requires $O(n^{2})$
memory, which can be impractical for large videos.

In contrast, our proposed framework presented in Chapter 4 is designed to
construct a scene graph for each snippet rather than the whole video which
reflects the structure of a dynamic scene in terms of atomic action tubes
(nodes) and the different types of relationships between them.

####  2.4.1 GCN-Based Temporal Activity Localisation

GCNs have been extensively investigated for TAL [21, 62, 25, 92, 93]. GCN-
based TAL methods can also be further divided into two-stage and one-stage
methods. The former, once again, perform localisation after generating
suitable proposals. For instance, in [21] two different types of boundary
proposals are generated and then individually passed to the same graph,
resulting in both an action label and a temporal boundary.

One-stage GCN-based TAL methods, instead, solve the detection problem without
proposals in one go by learning spatiotemporal features in an end-to-end
manner. For examples, in [25] a graph is first generated by connecting the
snippets both temporally and by virtue of meaningful semantics. This graph is
then divided into sub-graphs (anchors), where each anchor represents the
activity in an untrimmed video. In contrast, Khan et al. [17] proposed a
spatiotemporal scene graph-based long-term TAL method where each of the
snippets is considered as a separate graph, which is heavily dependent on the
particular actions present in the scene, and is only applicable to datasets
providing (label and bounding box) annotations for each individual action.

In Chapter 5, we propose a new framework leveraging GCNs, by incorporating
them in an overall hybrid graph capable of modelling both the local scenes,
via a Graph Attention Network (GAT) [94], and the overall global activity via
a temporal graph. GATs build on the transformer concept by applying attention
to graphs, and were originally proposed in [94] for node classification. The
idea is to update the representation of the current node with respect to its
neighbours by applying attention to learn the importance of the various
connections.

In Chapter 5, GAT is used at the local scene level to learn the features of
each node (active agent tube), to generate a more robust local scene
representation.

###  2.5 Continual Learning

Current continual learning methods can be categorised into three major
families based on how the information of previous data is stored and used.
_Prior-focused_ methods [95] use a penalty term to regularise the parameters
rather than a hard constraint. _Parameter isolation_ methods [96] dedicate
different parameters for different tasks to prevent interference. Finally,
_replay-based_ approaches store the examples’ information in a ‘replay buffer’
or a generative model, which is used for rehearsal/retraining or to provide
constraints on the current learning step [97].

####  2.5.1 Continual Unsupervised Learning

Continual unsupervised learning is a relatively unexplored territory due to
the heavy dependency of existing works on the availability of ground truth
information. Another challenge lies in the clear distinction of class
boundaries for controlling model updates [98]. Clustering approaches that
provide structure to the unlabelled stream are a popular theme of existing
continual unsupervised works [98, 99, 100]. A self-adaptive clustering
approach [98] is proposed which does not rely on labelled data for model
updates. The labelled data is initially leveraged to learn the cluster-to-
class associations. Similarly, in [99] the pseudo labels for new data are
obtained by a K-means global clustering approach, whereas the updated model
without fully connected layers is employed as a feature extractor.

####  2.5.2 Time-Varying Hidden Markov Models

A number of competing formulations exist for HMMs whose parameters depend on
time. They have been employed recently for useful life prediction in
engineering [101], analysing time-varying networks in EEG data [102], and
incorporating periodic variability in animal movement [103]. Hidden _semi-
Markov_ models [104], in particular, have been proposed, which augment HMMs
with time-varying parameters, so that the transition probability to a new
state varies in accordance with the time spent in the current state. In
opposition, the TVHMM model formulated in [105] encodes non-stationarity into
a finite sequence of time varying transition densities and can simulate state
duration of infinite length. A common approach [106, 103] is to model the
transition probabilities as functions of a covariate process $\\{z_{t}\\}$,
which is observed in parallel to the main observation process [107]. As the
covariates $z_{t}$ are simultaneously correlated with the hidden variables
$x_{t}$, it may be preferable to use lagged informative variables or
artificial variables. In fact, in [108] a time-varying hidden Markov model
with latent information was proposed in which an additional hidden variable
(‘signal’) with AR(1) dynamics drives the transition probabilities in the main
hidden variable, $x_{t}$.

In [109], the authors observed that the states respond more to their own past
values than any other variable. They thus proposed a model in which the state
depends on the sign of an autoregressive latent variable, which induces time-
varying probabilities in a non-Markovian process, as the probability of a
certain state at time $t$ depends on the lagged value of the latent variable
and not the state at the previous time.

###  2.6 Benchmark Datasets

In the literature, several datasets have been proposed to evaluate the
performance of scene understanding from simple action detection to event
detection and complex activity recognition. Here, in this chapter, we describe
all the datasets used in this thesis.

####  2.6.1 ROAD

ROAD (The ROad event Awareness Dataset for Autonomous Driving) [45] is a
multi-labeled dataset proposed for road agent, action, and location detection.
The combination of these three labels is referred to in [45] as a ‘road
event’. The ROAD dataset consists of a total of 22 videos with an average
duration of 8 minutes, captured by the Oxford RobotCar [110] under diverse
lighting and weather conditions. The dataset was further extended as a testbed
for complex activity detection in [17]. Complex activities in the ROAD dataset
belong to six different classes, including: negotiating an intersection,
negotiating a pedestrian crossing, waiting in a queue, merging into the (ego)
vehicle lane, sudden appearance (of other vehicles/agents), and (people)
walking in the middle of the road. Activities can span up to two minutes and
involve a large number of road agents. We introduce ROAD in Chapter 3 and
evaluate our 3D-RetinaNet baseline in the same chapter. We also use ROAD
extension for complex activity detection in Chapter 4 and Chapter 5

####  2.6.2 SARAS-ESAD

ESAD (the Endoscopic Surgeon Action Detection Dataset) [111] is a benchmark
devised for surgeon action detection in real-world endoscopic surgery videos.
The goal of ESAD is contributing to increasing the potential and dependability
of surgical assistant robots by realistically testing their awareness of the
actions performed by a surgeon. In ESAD, surgeon actions are classed into 21
different categories and annotated with the help of professional surgeons.
This dataset is used in Chapter 4 for complex activity detection.

####  2.6.3 Thumos-14

Thumos-14 [31] is one of the benchmark datasets for temporal action
localisation. Thumos-14 contains 413 untrimmed temporally annotated videos
categorised into 20 actions. Videos are characterised by a large variance in
duration, from one second to 26 minutes. On average, each video contains 16
action instances. To compare our performance with the state-of-the-art, we
also adopt the standard practice of using the validation set (200 videos) for
training while evaluating our model on the test set (213 videos). We use this
dataset for the evaluation of our complex activity framework in Chapter 5.

####  2.6.4 ActivityNet-1.3

ActivityNet-1.3 [30] is one of the largest action localisation datasets with
around 20K untrimmed videos comprising 200 action categories. The videos are
divided into training, validation, and testing folds according to a ratio of
2:1:1, respectively. The number of action instances per video is 1.65, which
is quite low compared to Thumos-14. Following the previous art, we train our
model on the training set and test it on the validation set. We use this
dataset in Chapter 4 for the evaluation of our complex activity framework.

####  2.6.5 50-Salad

50-Salad [112] is an activity segmentation dataset with the aim of stimulate
research on recognizing manipulative gestures. It consists of 50 videos of 25
cooks (actors) preparing salads, where each cook recorded two videos of
themselves preparing salads by following different sequence of steps. The
dataset contains two levels of annotations: “high-level” labels only span
three classes, whereas “low-level” annotation involves a total of 51 classes.
50-Salad is a multimodel dataset contains other modality besides RGB videos
including RGB-D (depth information), 3-axis accelerometer devices attached to
a knife, a mixing spoon, a small spoon, a peeler, a glass, an oil bottle, and
a pepper dispenser. This dataset is used in Chapter 6 for the evaluation of
our latent model.

###  2.7 Evaluation Metrics

We also provide the standard evaluation metrics used for the evaluation of our
methods proposed in this thesis.

####  2.7.1 Classification Accuracy

Classification accuracy is a measure of how well a classification model is
able to correctly predict the class labels of a given dataset. It is defined
as the proportion of correctly classified instances to the total number of
instances in the dataset. Classification accuracy is used in Chapter 4 for
recognition of activities.

####  2.7.2 Precision

Precision is a measure of the accuracy of a classification model’s positive
predictions. It is defined as the proportion of true positive predictions
(correctly predicted positive instances) to the total number of positive
predictions (true positives plus false positives).

A high precision value means that the model has a low number of false
positives, or that it is good at not classifying negative instances as
positive. In other words, it means that the model is good at identifying only
the relevant instances among the positive predictions. Precision is used in
Chapter 4 and Chapter 6 for recognition of activities.

####  2.7.3 Recall

Recall, also known as Sensitivity or the True Positive Rate (TPR), is a
measure of a classification model’s ability to correctly identify all relevant
instances. It is defined as the proportion of true positive predictions
(correctly predicted positive instances) to the total number of actual
positive instances (true positives plus false negatives).

A high recall value means that the model has a low number of false negatives,
or that it is good at identifying all relevant instances. In other words, it
means that the model is good at detecting all the positive instances among all
the actual positive instances. Recall is used in Chapter 4 and Chapter 6 for
recognition of activities.

####  2.7.4 F1-score

The F1 Score is a measure of a classification model’s accuracy that combines
precision and recall. It is defined as the harmonic mean of precision and
recall.

The F1 Score is a single value that represents the balance between precision
and recall. A higher F1 Score indicates that the model has a better balance
between precision and recall.

It is particularly useful when the classes are imbalanced, or when the cost of
false positives and false negatives is different. In these cases, it is
possible to have a high accuracy score but a low F1 score if precision and
recall are not in balance. F1 score is used in Chapter 4 and Chapter 6 for
recognition of activities.

####  2.7.5 Mean Average Precision

Mean Average Precision (mAP) is a common evaluation metric used in action
detection, which is a subfield of computer vision that deals with recognizing
and locating instances of specific actions within video sequences. MAP
calculates the average precision across all classes of actions in a video
dataset, by taking into account both the precision and recall of the action
detection algorithm.

Precision measures the proportion of true positive detections (correctly
detected actions) to the total number of positive detections (both true
positives and false positives). Recall measures the proportion of true
positive detections to the total number of instances of the action that should
have been detected.

mAP is calculated by first computing the average precision for each action
class, and then taking the mean of those values. A higher MAP value indicates
a better performance of the action detection algorithm. This evaluation metric
is used for two purposes; firstly for the spatiotemporal detection of events
in Chapter 3. Secondly, used for the temporal evaluation of complex activities
in Chapter 4 and Chapter 5.

####  2.7.6 Mean Absolute Error

Mean Absolute Error (MAE) is a measure of the average magnitude of the errors
in a set of predictions, without considering their direction. It measures the
difference between the predicted values and the true values.

MAE is a commonly used regression evaluation metric and it is relatively easy
to understand and interpret. It is expressed in the same unit as the target
variable and it gives an idea of the magnitude of the error, regardless of the
direction of the error. MAE is used in Chapter 6 for the evaluation of
continual crowd counting (regression problem).

Part I : ROAD (The ROad event Awareness Dataset for Autonomous Driving)

##  Chapter 3 The ROad event Awareness Dataset for Autonomous Driving

###  3.1 Introduction

As anticipated, the first part of this thesis concerns individual event
detection, specifically the detection of road events from an autonomous
driving perspective. Injecting capabilities such as the understanding of
dynamic road events and their evolution into autonomous vehicles has the
potential to take situational awareness and decision-making closer to human-
level performance. To this purpose, in this chapter, we introduce the ROad
event Awareness Dataset (ROAD) for Autonomous Driving. ROAD is designed to
test an autonomous vehicle’s ability to detect road events. To the best of our
knowledge, ROAD is the first multi-label multi-action autonomous driving
dataset where each of the bounding box is associated with three labels (agent,
action, and location). This unique labeling schema makes ROAD more demanding
and challenging in comparison to its competitors. ROAD comprises videos
originally from the Oxford RobotCar Dataset, annotated with bounding boxes
showing the location in the image plane of each road event. We benchmark
various detection tasks, proposing as a baseline a new incremental algorithm
for online road event awareness termed 3D-RetinaNet. We also report the
performance on the ROAD tasks of Slowfast and YOLOv5 detectors, as well as
that of the winners of the ICCV2021 ROAD challenge, which highlights the
challenges faced by situation awareness in autonomous driving. ROAD is
designed to allow scholars to investigate exciting tasks such as complex
(road) activity detection, future event anticipation and continual learning.

Further down the line, we also introduce logical constraints to ROAD. The main
intuition behind these logical constraints is that neural networks have proven
to be very powerful at computer vision tasks. However, they often exhibit
unexpected behaviours, violating known requirements and expressing background
knowledge. This calls for models (i) able to learn from the requirements, and
(ii) be guaranteed to be compliant with the requirements themselves.
Unfortunately, the development of such models is hampered by the lack of
datasets equipped with formally specified requirements. Therefore, we
introduce the ROad event Awareness Dataset with logical Requirements (ROAD-R),
the first publicly available dataset for autonomous driving with requirements
expressed as logical constraints.

####  3.1.1 Motivation

The history of autonomous driving has witnessed significant milestones and
advancements since the Defense Advanced Research Projects Agency
(DARPA111https://www.darpa.mil/) Grand Challenge 2004, which marked a pivotal
moment in the field. The DARPA Grand Challenge was a groundbreaking
competition organized by the U.S. Department of Defense’s DARPA agency. Its
aim was to accelerate the development of autonomous vehicle technology. The
challenge required participating vehicles to navigate a designated route in a
desert environment, relying solely on their autonomous capabilities. The main
purpose of the Challenge was to push the boundaries of autonomous vehicle
technology. The goal was to foster innovation and encourage advancements that
could potentially benefit military applications, such as unmanned ground
vehicles for reconnaissance or supply missions [113]. The DARPA Grand
Challenge helped establish autonomous driving as a viable research area and
laid the foundation for subsequent advancements in the field.

In recent years, autonomous driving has emerged as a fast-growing research
area. The race towards fully autonomous vehicles pushed many large companies,
such as Google, Toyota, and Ford, to develop their own concept of _robot-car_
[114, 115, 116]. While self-driving cars are widely considered to be a major
development and testing ground for the real-world application of artificial
intelligence, major reasons for concern remain in terms of safety, ethics,
cost, and reliability [117]. From a safety standpoint, in particular, smart
cars need to robustly interpret the behaviour of the humans (drivers,
pedestrians or cyclists) they share the environment with, in order to cope
with their decisions. _Situation awareness_ and the ability to understand the
behaviour of other road users are thus crucial for the safe deployment of
autonomous vehicles (AVs). Therefore, in this chapter, we only consider the
context of _vision-based_ autonomous driving [118] from video sequences
captured by cameras mounted on the vehicle in a streaming, online fashion.

While detector networks [119] are routinely trained to facilitate object and
actor recognition in road scenes, this simply allows the vehicle to ’see’ what
is around it. The philosophy of this work is that robust self-driving
capabilities require a deeper, more human-like understanding of dynamic road
environments (and of the evolving behaviour of other road users over time) in
the form of semantically meaningful concepts, as a stepping stone for
intention prediction and automated decision making. One advantage of this
approach is that it allows the autonomous vehicle to focus on a much smaller
amount of relevant information when learning how to make its decisions, in a
way arguably closer to how decision-making takes place in humans.

Figure 3.1: Use of labels in ROAD to describe typical road scenarios. (a) A
green car is in front of the AV while changing lanes, as depicted by the arrow
symbol. The associated event will then carry the following labels: in vehicle
lane (location), moving left (action). Once the event is completed, the
location label will change to: in outgoing lane. (b) Autonomous vehicle
turning left from lane 6 into lane 4: lane 4 will be the outgoing lane as the
traffic is moving in the same direction as the AV. However, if the AV turns
right from lane 6 into lane 4 (a wrong turn), then lane 4 will become the
incoming lane as the vehicle will be moving into the incoming traffic. The
overall philosophy of ROAD is to use suitable combinations of multiple label
types to fully describe a road situation, and allow a machine learning
algorithm to learn from this information.

On the opposite side of the spectrum lies end-to-end reinforcement learning.
There, the behaviour of a human driver in response to road situations is used
to train, in an imitation learning setting [120], an autonomous car to respond
in a more ‘human-like’ manner to road scenarios. This, however, requires an
astonishing amount of data from a myriad of road situations. For highway
driving only, a relatively simple task when compared to city driving, Fridman
et al. in [121] had to use a whole fleet of vehicles to collect 45 million
frames. Perhaps more importantly, in this approach, the network learns a
mapping from the scene to control inputs, without attempting to model the
significant facts taking place in the scene or the reasoning of the agents
therein. As discussed in [122], many authors [123, 124] have recently
highlighted the insufficiency of models which directly map observations to
actions [125], specifically in the scenario of the self-driving car.

####  3.1.2 ROAD: a multi-label, multi-task dataset

Concept. This work aims to propose a new framework for situation awareness and
perception, departing from the disorganised collection of object detection,
semantic segmentation or pedestrian intention tasks which is the focus of much
current work. We propose to do so in a “holistic”, multi-label approach in
which agents, actions and their locations are all ingredients in the
fundamental concept of _road event_ (RE). Road events are defined as triplets
$E=(Ag,Ac,Loc)$ composed by an active road agent $Ag$, the action(s) $Ac$ it
performs (possibly more than one at the same time), and the location(s)
$Loc$222Note: The label location $Loc$ in our case does not refer to the
spatiotemporal coordinates (x, y) of the bounding box. in which this takes
place (which may vary from the start to the end of the event itself), as seen
from the point of the view of an autonomous vehicle. The complete list of all
three label types is provided in Table 3.1. This takes the problem to a higher
conceptual level, in which AVs are tested on their _understanding of what is
going on_ in a dynamic scene rather than their ability to describe what the
scene _looks like_ , putting them in a position to use that information to
make decisions and a plot course of action. Modelling dynamic road scenes in
terms of road events can also allow us to model the causal relationships
between what happens; these causality links can then be exploited to predict
further future consequences.

To transfer this conceptual paradigm into practice, this chapter introduces
ROAD, the first _ROad event Awareness in Autonomous Driving Dataset_ , as an
entirely new type of dataset designed to allow researchers in autonomous
vehicles to test the situation awareness capabilities of their stacks in a
manner impossible until now. Unlike all existing benchmarks, ROAD provides
ground truth for the action performed by all road agents, not just humans. In
this sense ROAD is unique in the richness and sophistication of its
annotation, designed to support the proposed conceptual shift. We are
confident this contribution will be very useful moving forward for both the
autonomous driving and the computer vision community. However, at the same
time, it is also worth mentioning that the quantisation of the action and
activities in the context of autonomous driving with several limitation and
challenges including computational complexity, robustness to uncertainty, and
control and decision-making.

Features. ROAD is built upon (a fraction of) the Oxford RobotCar Dataset
[126], by carefully annotating 22 carefully selected, relatively long-duration
videos. Road events are represented as ’tubes’, i.e., time series of frame-
wise bounding box detections. ROAD is a dataset of significant size, most
notably in terms of the richness and complexity of its annotation rather than
the raw number of video frames. A total of $122K$ video frames are labelled
for a total of $560K$ detection bounding boxes in turn associated with $1.7M$
unique individual labels, broken down into $560K$ agent labels, $640K$ action
labels and $499K$ location labels.

The dataset was designed according to the following principles.

  * •

A _multi-label_ benchmark: each road event is composed by the label of the
(moving) agent responsible, the label(s) of the type of action(s) being
performed, and labels describing where the action is located.

  * •

Each event can be assigned _multiple instances_ of the same label type
whenever relevant (e.g., an RE can be an instance of both moving away and
turning left).

  * •

The labelling is done _from the point of view of the AV_ : the final goal is
for the autonomous vehicle to use this information to make the appropriate
decisions.

  * •

The meta-data is intended to contain all the information required to fully
describe a road scenario: an illustration of this concept is given in Figure
3.1. After closing one’s eyes, the set of labels associated with the current
video frame should be sufficient to recreate the road situation in one’s head
(or, equivalently, sufficient for the AV to be able to make a decision).

In an effort to take action detection into the real world, ROAD moves away
from human body actions almost entirely, to consider (besides pedestrian
behaviour) actions performed by humans as drivers of various types of
vehicles, shifting the paradigm from _actions performed by human bodies_ to
_events caused by agents_. As shown in our experiments, ROAD is more
challenging than current action detection benchmarks due to the complexity of
road events happening in real, non-choreographed driving conditions, the
number of active agents present and the variety of weather conditions
encompassed.

Tasks. ROAD allows one to validate manifold tasks associated with situation
awareness for self-driving, each associated with a label type (agent, action,
location) or a combination thereof:

  1. 1.

_Agent detection_ , the output is in the form of ’agent tubes’ collecting the
BBs associated with an active road agent in consecutive frames.

  2. 2.

_Action detection_ , the output is in the form of ’action tubes’ (series of
bounding box (BB) detections with an attached action label).

  3. 3.

_Location detection_ , the output is in the form of ’agent-location’ tubes
collecting the BBs associated with an agent and its location in consecutive
frames.

  4. 4.

_Agent-action detection_ , the detector is trained on pairs of agent-action
labels. The output is again in the form of a tube.

  5. 5.

_Road event detection_ , as well as the AV’s decisions make use of all the
three types of information provided by ROAD, detecting RE triplets is crucial
for autonomous driving applications.

  6. 6.

_Temporal AV actions detection_, the output of this task is the class of
action the AV performs in each video frame, from a list suitably compiled.

For each task, one can assess both _frame-level_ detection, which outputs
independently for each video frame the bounding box(es) of the instances there
present and the relevant class labels, and _video-level_ detection, which
consists in regressing the whole series of temporally-linked bounding boxes
(i.e., in current terminology, a ’tube’) associated with an instance, together
with the relevant class label. In this chapter, we conduct tests on both. All
tasks come with both the necessary annotation and a shared baseline, which is
described in Section 3.3.

Table 3.1: ROAD labels. Agents | Actions | Locations
---|---|---
Pedestrian | Move away | AV lane
Car | Move towards | Outgoing lane
Cyclist | Move | Outgoing cycle lane
Motorbike | Brake | Incoming lane
Medium vehicle | Stop | Incoming cycle lane
Large vehicle | Indicating left | Pavement
Bus | Indicating right | Left pavement
Emergency vehicle | Hazards lights on | Right pavement
AV traffic light | Turn left | Junction
Other traffic light | Turn right | Crossing location
| Overtake | Bus stop
| Wait to cross | Parking
| Cross from left |
| Cross from right |
| Crossing |
| Push object |
| Red traffic light |
| Amber traffic light |
| Green traffic light |

####  3.1.3 ROAD-R

In this chapter, we also generalize hierarchical multi-label classification
(HMC) problems by introducing multi-label classification problems with (full)
propositional logic requirements. Thus, given a multi-label classification
problem with labels $A$, $B$, and $C$, we can, for example, write the
requirement:

$(\neg A\wedge B)\vee C,$

stating that for each data point in the dataset either the label $C$ is
predicted, or $B$ but not $A$ are predicted.

Then, we present the ROad event Awareness Dataset with logical Requirements
(ROAD-R), the first publicly available dataset for autonomous driving with
requirements expressed as logical constraints. ROAD-R extends the ROAD dataset
by manually annotated ROAD-R with 243 constraints, each verified to hold for
each bounding box. A typical constraint is thus “a traffic light cannot be red
and green at the same time”, while there are no constraints like “pedestrians
should cross at crossings”, which should always be satisfied in theory, but
which might not be in real-world scenarios.

Given ROAD-R, we considered 6 current state-of-the-art (SOTA) models, and we
showed that they are not able to learn the requirements just from the data
points, as more than 90% of the times, they produce predictions that violate
the constraints.

####  3.1.4 Main Contributions

The major contributions of this work are thus the following.

  * •

A conceptual shift in situation awareness centred on a formal definition of
the notion of road event, as a triplet composed by a road agent, the action(s)
it performs and the location(s) of the event, seen from the point of view of
the AV.

  * •

A new ROad event Awareness Dataset for Autonomous Driving (ROAD), the first of
its kind, designed to support this paradigm shift and allow the testing of a
range of tasks related to situation awareness for autonomous driving: agent
and/or action detection, event detection, ego-action classification.

  * •

A multi-label classification problem with propositional logic requirements.

  * •

Introduction of ROAD-R, which is the first publicly available dataset whose
requirements are expressed in full propositional logic.

  * •

Considered 6 SOTA models and show that on ROAD-R, they produce predictions
violating the requirements more than $\sim$90% of the times,

Instrumental to the introduction of ROAD as the benchmark of choice for
semantic situation awareness, we propose a robust baseline for online
action/agent/event detection (termed _3D-RetinaNet_) which combines state-of-
the-art single-stage object detector technology with an online tube
construction method [127], with the aim of linking detections over time to
create _event tubes_ [128, 129]. Results for two additional baselines based on
a Slowfast detector architecture [130] and
YOLOv5333https://github.com/ultralytics/yolov5. (for agent detection only) are
also reported and critically assessed.

###### Related publications:

The work of this chapter appeared in IEEE transactions on pattern analysis and
machine intelligence (IEEE TPAMI 2022) [45] and The ROAD Workshop and
Challenge: Event Detection for Situation Awareness in Autonomous Driving at
ICCV 2021444https://sites.google.com/view/roadchallangeiccv2021/. The
dissertation’s author was the co-author of the TPAMI paper and co-organiser of
the workshop and challenge. The main contributions of the dissertation’s
author to this project are: adding a new backbone (SlowFast) to the 3D
RetinaNet architecture (baseline for ROAD), implementing a YOLOv5 for agents
detection, and setting up the evaluation site for the challenge. All the
resources of this project are publically available and are accessed from _ROAD
dataset_ 555https://github.com/gurkirt/road-dataset, _3D RetinaNet baseline_
666https://github.com/gurkirt/3D-RetinaNet, _Inference demo_
777https://www.youtube.com/watch?v=CmxPjHhiarA, and _Challenge evalAI_
888https://eval.ai/web/challenges/challenge-page/1059/overview.

The work of ROAD-R appeared in IJCAI 2022 Workshop on Artificial Intelligence
for Autonomous Driving (AI4AD 2022)999https://learn-to-race.org/workshop-
ai4ad-ijcai2022/ and IJCLR 2022 the International Joint Conference on Learning
and Reasoning [46]. This work received the best paper award at IJCAI 2022
AI4AD and won the best student paper award at IJCLR 2022. The dissertation’s
author was the co-author in both papers. The main contribution of the
dissertation’s author to this project is providing the vision baselines for on
ROAD dataset for logical constraint. All the resources of this project are
publically available and are accessed from _code_
101010https://github.com/EGiunchiglia/ROAD-R,
video111111https://www.youtube.com/watch?v=_-Ll5d1VQXY.

###### Outline:

The remainder of the chapter is organised as follows. Section 3.2 presents our
ROAD dataset in full detail, including: its multi-label nature (Sec. 3.2.1),
data collection (Sec. 3.2.2), annotation (Sec. 3.2.3), the tasks it is
designed to validate (Sec. 3.2.4), and a quantitative summary (Sec. 3.2.5).
Section 3.3 presents an overview of the proposed 3D-RetinaNet baseline, and
recalls the ROAD challenge organised by some of us at ICCV 2021 to disseminate
this new approach to situation awareness within the autonomous driving and
computer vision communities, using ROAD as the benchmark. Section 3.4
describes ROAD-R. Experiments are described in Section 3.5, where a number of
ablation studies are reported and critically analysed in detail, together with
the results of the ROAD challenge’s top participants. Section 3.6 outlines
additional exciting tasks the dataset can be used as a benchmark for in the
near future, such as future event anticipation, decision making and machine
theory of mind [122]. Conclusions and future work are outlined in Section 3.7.

###  3.2 The Dataset

####  3.2.1 A Multi-Label Benchmark

The ROAD dataset is specially designed from the perspective of self-driving
cars, and thus includes actions performed not just by humans but by all road
agents in specific locations, to form _road events_ (REs). REs are annotated
by drawing a bounding box around each active road agent present in the scene,
and linking these bounding boxes over time to form ’tubes’. As explained, to
this purpose three different types of labels are introduced, namely: (i) the
category of _road agent_ involved (e.g. Pedestrian, Car, Bus, Cyclist); (ii)
the _type of action_ being performed by the agent (e.g. Moving away, Moving
towards, Crossing and so on), and (iii) the _location_ of the road user
relative the autonomous vehicle perceiving the scene (e.g. In vehicle lane, On
right pavement, In incoming lane). In addition, ROAD labels the actions
performed by the vehicle itself. Multiple agents might be present at any given
time, and each of them may perform multiple actions simultaneously (e.g. a Car
may be Indicating right while Turning right). Each agent is always associated
with at least one action label. The complete list of all three labels types
are given in Table 3.1.

_Agent labels_. Within a road scene, the objects or people able to perform
actions which can influence the decision made by the autonomous vehicle are
termed agents. We only annotate _active_ agents (i.e., a parked vehicle or a
bike or a person visible to the AV but located away from the road are not
considered to be ’active’ agents). Three types of agent are considered to be
of interest, in the sense defined above, to the autonomous vehicle: people,
vehicles and traffic lights. For simplicity, the AV itself is considered just
like another agent: this is done by labelling the vehicle’s bonnet. People are
further subdivided into two sub-classes: pedestrians and cyclists. The vehicle
category is subdivided into six sub–classes: car, small–size motorised
vehicle, medium–size motorised vehicle, large–size motorised vehicle, bus,
motorbike, emergency vehicle. Finally, the ‘traffic lights’ category is
divided into two sub–classes: Vehicle traffic light (if they apply to the AV)
and Other traffic light (if they apply to other road users). Only one agent
label can be assigned to each active agent present in the scene at any given
time.

_Action labels_. Each agent can perform one or more actions at any given time
instant. For example, a traffic light can only carry out a single action: it
can be either red, amber, green or ‘black’. A car, instead, can be associated
with two action labels simultaneously, e.g., Turning right and Indicating
right. Although some road agents are inherently multitasking, some action
combinations can be suitably described by a single label: for example, pushing
an object (e.g. a pushchair or a trolley-bag) while walking can be simply
labelled as Pushing object. The latter was our choice.

_AV own actions_. Each video frame is also labelled with the action label
associated with what the AV is doing. To this end, a bounding box is drawn on
the bonnet of the AV. The AV can be assigned one of the following seven action
labels: AV-move, AV-stop, AV-turn-left, AV-turn-right, AV-overtake, AV-move-
left and AV-move-right.

_Location labels_. Agent location is crucial for deciding what action the AV
should take next. As the final, long-term objective of this project is to
assist autonomous decision making, we propose to label the location of each
agent from the perspective of the autonomous vehicle. For example, a
pedestrian can be found on the right or the left pavement, in the vehicle’s
own lane, while crossing or at a bus stop. The same applies to other agents
and vehicles as well. There is no location label for the traffic lights as
they are not movable objects, but agents of a static nature and well-defined
location. To understand this concept, Fig. 3.1 illustrates two scenarios in
which the location of the other vehicles sharing the road is depicted from the
point of view of the AV. _Traffic light_ is the only agent type missing
location labels, all the other agent classes are associated with at least one
location label.

####  3.2.2 Data Collection

ROAD is composed of 22 videos from the publicly available Oxford Robot Car
Dataset [126] (OxRD) released in 2017 by the Oxford Robotics
Institute121212http://robotcar-dataset.robots.ox.ac.uk/, covering diverse road
scenes under various weather conditions. The OxRD dataset, collected from the
narrow streets of the historic city of Oxford, was selected because it
presents challenging scenarios for an autonomous vehicle due to the diversity
and density of various road users and road events.

Note, however, that our labelling process (described below) is not limited to
OxRD. In principle, other autonomous vehicle datasets (e.g. [131, 132]) may be
labelled in the same manner to further enrich the ROAD benchmark, we plan to
do exactly so in the near future.

_Video selection_. Within OxRD, videos were selected with the objective of
ensuring diversity in terms of weather conditions, times of the day and types
of scenes recorded. Specifically, the 22 videos have been recorded both during
the day (in strong sunshine, rain or overcast conditions, sometimes with snow
present on the surface) and at night.

_Preprocessing_. First, the original sets of video frames were downloaded and
demosaiced, in order to convert them to red, green, and blue (RGB) image
sequences. Then, they were encoded into proper video sequences using
ffmpeg131313https://www.ffmpeg.org/ at the rate of 12 frames per second (fps).
Although the original frame rate in the considered frame sequences varies from
11 fps to 16 fps, we uniformised it to keep the annotation process consistent.

####  3.2.3 Annotation Process

_Annotation tool_. Annotating tens of thousands of frames rich in content is a
very intensive process; therefore, a tool is required which can make this
process both fast and intuitive. For this work, we adopted Microsoft’s
VoTT141414https://github.com/Microsoft/VoTT/. The most useful feature of this
annotation tool is that it can copy annotations (bounding boxes and their
labels) from one frame to the next, while maintaining a unique identification
for each box, so that boxes across frames are automatically linked together.
Moreover, VoTT also allows for multiple labels, thus lending itself well to
ROAD’s multi-label annotation concept.

_Annotation protocol_. All salient objects and actors within the frame were
labelled, with the exception of inactive participants (mostly parked cars) and
objects/actors at large distances from the ego vehicle, as the latter were
judged to be irrelevant to the AV’s decision-making. This can be seen in the
attached 30-minute video151515https://www.youtube.com/watch?v=CmxPjHhiarA.
portraying ground truth and predictions. As a result, pedestrians, cyclists,
and traffic lights were always labelled. Vehicles, on the other hand, were
only labelled when active (i.e., moving, indicating, being stopped at lights
or stopping with hazard lights on the side of the road). As mentioned, only
parked vehicles were not considered active (as they do not arguably influence
the AV’s decision-making) and were thus not labelled.

_Event label generation_. Using the annotations manually generated for actions
and agents in the multi-label scenario as discussed above it is possible to
generate _event-level_ labels about agents, e.g. Pedestrian / Moving towards
the AV On right pavement or Cyclist / Overtaking / In vehicle lane. Any
combinations of location, action and agent labels are admissible. If location
labels are ignored, the resulting event labels become location-invariant.
Namely, given an agent tube and the continuous temporal sequence of action
labels attached to its constituent bounding box detections, we can generate
action tubes by looking for changes in the action label series associated with
each agent tube. For instance, a Car appearing in a video might be first
Moving away before Turning left. The agent tube for the car will then be
formed by two contiguous agent-action tubes: a first tube with label pair Car
/ Moving away and a second one with pair Car / Turning left.

Table 3.2: ROAD tasks and attributes. Task type | Problem type | Output | Multiple labels
---|---|---|---
Active agent | Detection | Box&Tube | No
Action | Detection | Box&Tube | Yes
Location | Detection | Box&Tube | Yes
Event | Detection | Box&Tube | Yes
AV-action | Temporal detection | Start/End | No

####  3.2.4 Tasks

ROAD is designed as a sandbox for validating the six tasks relevant to
situation awareness in autonomous driving outlined in Sec. 3.1.2. Five of
these tasks are detection tasks, while the last one is a frame-level action
recognition task sometimes referred to as ’temporal action segmentation’
[133], Table 3.2 shows the main attributes of these tasks.

All detection tasks are evaluated both at frame-level and at video-
(tube-)level. _Frame-level detection_ refers to the problem of identifying in
each video frame the bounding box(es) of the instances there present, together
with the relevant class labels. _Video-level detection_ consists in regressing
a whole series of temporally-linked bounding boxes (i.e., in current
terminology, a ’tube’) together with the relevant class label. In our case,
the bounding boxes will mark a specific active agent in the road scene. The
labels may issue (depending on the specific task) either from one of the
individual label types described above (i.e., agent, action or location) or
from one of the meaningful combinations described in 3.2.3 (i.e., event).

Below we list all the tasks for which we currently provide a baseline, with a
short description.

  1. 1.

_Active agent detection_ (or _agent detection_) aims at localising an active
agent using a bounding box (frame-level) or a tube (video-level) and assigning
a class label to it.

  2. 2.

_Action detection_ seeks to localise an active agent occupied in performing a
specific action from the list of action classes.

  3. 3.

In _agent location detection_ (or _location detection_) a label from the
relevant list of locations (as seen from the AV) is sought and attached to the
relevant bounding box or tube.

  4. 4.

_Road event detection_ (or _event detection_) consists in assigning to each
box or tube a triplet of class labels.

  5. 5.

_Autonomous vehicle temporal action segmentation_ is a frame-level action
classification task in which each video frame is assigned a label from the
list of possible AV own actions. We refer to this task as ’AV-action
segmentation’, similarly to [133].

Figure 3.2: Number of instances of each class of individual label-types, in
logarithmic scale.

####  3.2.5 Quantitative Summary

Overall, $122K$ frames extracted from 22 videos were labelled, in terms of
both AV own actions (attached to the entire frame) and bounding boxes with
attached one or more labels of each of the three types: agent, action,
location. In total, ROAD includes $560K$ bounding boxes with $1.7M$ instances
of individual labels. The latter figure can be broken down into $560K$
instances of agent labels, $640K$ instances of action labels, and $499K$
instances of location labels. Based on the manually assigned individual
labels, we could identify $454K$ instances of triplets (event labels).

The number of instances for each individual class from the three lists is
shown in Fig. 3.2 (frame-level, in orange). The $560K$ bounding boxes make up
$7,029$, $9,815$, $8,040$, and $8,394$ tubes for the label types agent,
action, location, and event, respectively. Figure 3.2 also shows the number of
tube instances for each class of individual label types as number of video-
level instances (in blue). The figure also highlights the presence of an
imbalance in the dataset distribution among the classes. Specifically, certain
classes exhibit a significant disparity in sample sizes, for example, ”car”
has a considerably larger number of samples compared to ”emergency vehicle.”

###  3.3 Baseline and Challenge

Inspired by the success of recent 3D CNN architectures [33] for video
recognition and of feature-pyramid networks (FPN) [134] with focal loss [135],
we propose a simple yet effective 3D feature pyramid network (3D-FPN) with
focal loss as a baseline method for ROAD’s detection tasks. We call this
architecture _3D-RetinaNet_.

Figure 3.3: Proposed 3D-RetinaNet architecture for online video processing.

####  3.3.1 3D-RetinaNet Architecture

The data flow of 3D-RetinaNet is shown in Figure 3.3. The input is a sequence
of $T$ video frames. As in classical FPNs [134], the initial block of
3D-RetinaNet consists of a backbone network outputting a series of forward
feature pyramid maps, and of lateral layers producing the final feature
pyramid composed by $T$ feature maps. The second block is composed by two sub-
networks which process these features maps to produce both bounding boxes ($4$
coordinates) and $C$ classification scores for each anchor location (over $A$
possible locations). In the case of ROAD, the integer $C$ is the sum of the
numbers of agent, action, location, and agent-action-location (event) classes,
plus one reserved for an _agentness_ score. The extra class agentness is used
to describe the presence or absence of an active agent. As in FPN [134], we
adopt ResNet50 [136] as the backbone network.

_2D versus 3D backbones_. In our experiments we show results obtained using
three different backbones: frame-based ResNet50 (2D), inflated 3D (I3D) [33]
and Slowfast [130], in the manner also explained in [137, 130]. Choosing a 2D
backbone makes the detector completely online [127], with a delay of a single
frame. Choosing an I3D or a Slowfast backbone, instead, causes a 4-frame delay
at detection time. Note that, as Slowfast and I3D networks makes use of a max-
pool layer with stride 2, the initial feature pyramid in the second case
contains $T/2$ feature maps. Nevertheless, in this case we can simply linearly
upscale the output to $T$ feature maps.

_AV action prediction heads_. In order for the method to also address the
prediction of the AV’s own actions (e.g. whether the AV is stopping, moving,
turning left etc.), we branch out the last feature map of the pyramid (see
Fig. 3.3, bottom) and apply spatial average pooling, followed by a temporal
convolution layer. The output is a score for each of the $C_{a}$ classes of AV
actions, for each of the $T$ input frames.

_Loss function_. As for the choice of the loss function, we adopt a binary
cross-entropy-based focal loss [135]. We chose a binary cross entropy because
our dataset is multi-label in nature  for which we only train a single binary
cross entropy function. The rationale behind selecting a focal-type loss
function stems from the anticipation that it could potentially help the neural
network in addressing the challenge of long tail distribution of data (see
Figure 3.2), where some classes have significantly more samples than others.
The focal-type loss accomplishes this by giving more weight to samples from
minority classes during training, enabling the model to better learn their
distinctive characteristics.

####  3.3.2 Online Tube Generation via Agentness Score

The autonomous driving scenario requires any suitable method for agent, action
or event tube generation to work in an _online_ fashion, by incrementally
updating the existing tubes as soon as a new video frame is captured. For this
reason, this work adopts a recent algorithm proposed by Singh et al.[127],
which incrementally builds action tubes in an online fashion and at real-time
speed. To be best of our knowledge, [127] was the first online multiple action
detection approach to appear in the literature, and was later adopted by
almost all subsequent works [138, 139, 140] on action tube detection.

_Linking of detections_. We now briefly review the tube-linking method of
Singh et al. [127], and show how it can be adapted to build agent tubes based
on an ’agentness’ score, rather than build a tube separately for each class as
proposed in the original paper. This makes the whole detection process faster,
since the total number of classes is much larger than in the original work
[127]. The proposed 3D-RetinaNet is used to regress and classify detection
boxes in each video frame potentially containing an active agent of interest.
Subsequently, detections whose score is lower than $0.025$ are removed and
non-maximal suppression is applied based on the agentness score.

At video start, each detection initialises an agentness tube. From that moment
on, at any time instance $t$ the highest scoring tubes in terms of mean
agentness score up to $t-1$ are linked to the detections with the highest
agentness score in frame $t$ which display an Intersection-over-Union (IoU)
overlap with the latest detection in the tube above a minimum threshold
$\lambda$. The chosen detection is then removed from the pool of frame-$t$
detections. This continues until the tubes are either assigned or not assigned
a detection from current frame. Remaining detections at time $t$ are used to
initiate new tubes. A tube is terminated after no suitable detection is found
for $n$ consecutive frames. As the linking process takes place, each tube
carries scores for all the classes of interest for the task at hand (e.g.,
action detection rather than event detection), as produced by the
classification subnet of 3D-RetinaNet. We can then label each agentness tube
using the $k$ classes that show the highest mean score over the duration of
the tube.

####  3.3.3 The ROAD Challenge

To introduce the concept of road event, our new approach to situation
awareness and the ROAD dataset to the computer vision and AV communities, some
of us have organised in October 2021 the workshop ”The ROAD challenge: Event
Detection for Situation Awareness in Autonomous
Driving”161616https://sites.google.com/view/roadchallangeiccv2021/.. For the
challenge, we selected (among the tasks described in Sec. 3.2.4) only three
tasks: agent detection, action detection and event detection, which we
identified as the most relevant to autonomous driving.

As standard in action detection, evaluation was done in terms of video mean
average precision (video-mAP). 3D-RetinaNet was proposed as the baseline for
all three tasks. Challenge participants had 18 videos available for training
and validation. The remaining 4 videos were to be used to test the final
performance of their model. This split was applied to all the three challenges
(split 3 of the ROAD evaluation protocol, see Section 3.5.2).

The challenge opened for registration on April 1 2021, with the training and
validation folds released on April 30, the test fold released on July 20 and
the deadline for submission of results set to September 25. For each stage and
each Task the maximum number of submissions was capped at 50, with an
additional constraint of 5 submissions per day. The workshop, co-located with
ICCV 2021, took place on October 16 2021.

In the validation phase we had between three and five teams submit between 15
and 17 entries to each of three challenges. In the test phase, which took
place after the summer, we noticed a much higher participation with 138
submissions from 9 teams to the agent challenge, 98 submissions from 8 teams
to the action challenge, and 93 submission from 6 teams to the event detection
challenge.

The methods proposed by the winners of each challenge are briefly recalled in
Section 3.5.4.

_Benchmark maintenance_. After the conclusion of the ROAD @ ICCV 2021
workshop, the challenge has been re-activated to allow for submissions
indefinitely. The ROAD benchmark will be maintained by withholding the test
set from the public on the eval.ai
platform171717https://eval.ai/web/challenges/challenge-page/1059/overview,
where teams can submit their predictions for evaluation. Training and
validation sets can be downloaded from https://github.com/gurkirt/road-
dataset.

###  3.4 ROAD-R

ROAD-R extends the ROAD dataset by introducing a set $\Pi$ of 243 constraints
that specify the space of admissible outputs. In order to improve the
usability of our dataset, we write the constraints in a way that allows us to
easily express $\Pi$ as a single formula in conjunctive normal form (CNF). The
above can be done without any loss in generality, as any propositional formula
can be expressed in CNF, and is important because many solvers expects
formulas in CNF as input. Thus, each requirement in $\Pi$ has form:

$l_{1}\vee l_{2}\vee\cdots\vee l_{n}\,,$ (3.1)

where $n\geq 1$, and each $l_{i}$ is either a negative label $\neg A$ or a
positive label $A$. The requirements have been manually specified following
three steps:

  1. 1.

an initial set of constraints $\Pi_{1}$ was manually created,

  2. 2.

a subset $\Pi_{2}\subset\Pi_{1}$ was retained by eliminating all those
constraints that were entailed by the others,

  3. 3.

the final subset $\Pi\subset\Pi_{2}$ was retained by keeping only those
requirements that were always satisfied by the ground-truth labels of the
entire ROAD-R dataset.

Table 3.3: Constraint statistics. $\Pi_{n}$ is the set of constraints $r$ in $\Pi$ with $|r|=n$, i.e., with $n$ positive and negative labels. $\overline{\mathcal{C}}=\\{\neg{A}:A\in\mathcal{C}\\}$. Each row shows the number of rules $r$ with $|r|=n$, and the average number of negative and positive labels in such rules. $n$ | $|\Pi_{n}|$ | $\text{avg}_{r\in\Pi_{n}}(|r\cap\overline{\mathcal{C}}|)$ | $\text{avg}_{r\in\Pi_{n}}(|r\cap\mathcal{C}|)$
---|---|---|---
2 | 215 | 1.995 | 0.005
3 | 5 | 1 | 2
7 | 1 | 1 | 6
8 | 6 | 1 | 7
9 | 6 | 1 | 8
10 | 1 | 0 | 10
12 | 1 | 1 | 11
14 | 1 | 0 | 14
15 | 7 | 1 | 14
Total | 243 | 1.87 | 0.96

Finally, redundancy in the constraints has been automatically checked with
relsat181818https://github.com/roberto-bayardo/relsat/. Note that our process
of gathering and further selecting the logical requirements follows more
closely the software engineering paradigm rather than the machine learning
view. To this end, we ensured that the constraints were consistent with the
provided labels from the ROAD dataset in the sense that they were acting as
strict conditions to be absolutely satisfied by the ground-truth labels, as
emphasized in the third step of the annotation pipeline above. Table 3.3 gives
a high-level description of the properties of the set $\Pi$ of constraints.
Notice that, with a slight abuse of notation, in the tables we use a set based
notation for the requirements. Each requirement of form (3.1) thus becomes

$\\{l_{1},l_{2},\ldots,l_{n}\\}.$

Such notation allows us to express the properties of the requirements in a
more succinct way. In addition to the information in the tables, we report
that of the 243 constraints, there are two in which all the labels are
positive (expressing that there must be at least one agent and that every
agent but traffic lights has at least one location), and 214 in which all the
labels are negative (expressing mutual exclusion between two labels). All the
constraints with more than two labels have at most one negative label, as they
express a one-to-many relation between actions and agents (like in our dataset
“if something is crossing, then it is a pedestrian or a cyclist”). Constraints
like “pedestrians should cross at crossings”, which might not be satisfied in
practice, are not included. Additionally embedding such logical constraints
would require, e.g., using modal operators and, while it would be an
interesting study to see the impact on the model’s predictions when adding
more complex layers to the expressivity of our logic, we opted for using a
simpler logic in this first instance. This also provides more transparency to
the wider research community, as the full propositional logic covers a vast
range of applications that do not require extra logical operators. Notice that
the 243 requirements restrict the number of admissible prediction to
$4985868\sim 5\times 10^{6}$, thus ruling out $(2^{41}-4985868)\sim 10^{12}$
non-admissible predictions.191919The number of admissible predictions has been
computed with relsat: https://github.com/roberto-bayardo/relsat/. In
principle, the set of admissible predictions can be further reduced by adding
other constraints. Indeed, the 243 requirements are not guaranteed to be
complete from every possible point of view: as standard in the software
development cycle, the requirement specification process deeply involves the
stakeholders of the system (see, e.g., [141]). For example, we decided not to
include constraints like “it is not possible to both move towards and move
away”, which were not satisfied by all the data points because of errors in
the ground truth labels. In these cases, we decided to dismiss the constraint
in order to maintain (i) consistency between the knowledge provided by the
constraints and by the data points, and (ii) backward compatibility.

As an additional point, we underline that, even though the annotation of the
requirements introduces some overhead in the annotation process, it is also
the case that the effort of manually writing 243 constraints (i) is negligible
when compared to the effort of manually annotating the 22 videos, and (ii) can
improve such last process, e.g., allowing to prevent errors in the annotation
of the data points.

###  3.5 Experimental Validation

In this section we present results on the various tasks the ROAD dataset is
designed to benchmark (see Sec. 3.2.4).

####  3.5.1 Implementation Details

The results are evaluated in terms of both frame-level bounding box detection
and tube detection. In the first case, the evaluation measure of choice is
_frame mean average precision_ (f-mAP). We set the Intersection over Union
(IoU) detection threshold to $0.5$ (signifying a 50% overlap between the
predicted and true bounding box). For the second set of results, we use _video
mean average precision_ (video-mAP), as information on how the ground-truth
BBs are temporally connected is available. These evaluation metrics are
standard in action detection [142, 143, 138, 127, 144].
We also evaluate actions performed by AV, as described in 3.2.1. Since this is
a temporal segmentation problem, we adopt the mean average precision metric
computed at frame-level, as standard on the Charades [133] dataset.

Table 3.4: Splits of training, validation and test sets for the ROAD dataset
with respect to weather conditions. The table shows the number of videos in
each set or split. For splits, the first figure is the number of training
videos, the second number that of validation videos.

Condition | sunny | overcast | snow | night
---|---|---|---|---
Training and validation | 7 | 7 | 1 | 3
Split-1 | 7/0 | 4/3 | 1/0 | 3/0
Split-2 | 7/0 | 7/0 | 1/0 | 0/3
Split-3 | 4/3 | 7/0 | 1/0 | 3/0
Testing | 1 | 1 | 1 | 1

We use sequences of $T=8$ frames as input to 3D-RetinaNet. Input image size is
set to $512\times 682$. This choice of $T$ is the result of GPU memory
constraints; however, at test time, we unroll our convolutional 3D-RetinaNet
for sequences of 32 frames, showing that it can be deployed in a streaming
fashion. We initialise the backbone network with weights pretrained on
Kinetics [34]. For training we use an SGD optimiser with step learning rate.
The initial learning rate is set to $0.01$ and drops by a factor of $10$ after
$18$ and $25$ epochs, up to an overall $30$ epochs.

The parameters of the tube-building algorithm (Sec. 3.3.2) are set by cross
validation. For ROAD we obtain $\lambda=0.5$ and $k=4$.

####  3.5.2 Experimental Results on ROAD

##### Three splits: modelling weather variability

For the benchmarking of the ROAD tasks, we divided the dataset into two sets.
The first set contains 18 videos for training and validation purposes, while
the second set contains 4 videos for testing, equally representing the four
types of weather conditions encountered.

The group of training and validation videos is further subdivided into three
different ways (’splits’). In each split, 15 videos are selected for training
and 3 for validation. Details on the number of videos for each set and split
are shown in Table 3.4. All 3 validation videos for Split-1 are overcast; 4
overcast videos are also present in the training set. As such, Split-1 is
designed to assess the effect of different overcast conditions. Split-2 has
all 3 night videos in the validation subset, and none in the training set. It
is thus designed to test model robustness to day/night variations. Finally,
Split-3 contains 4 training and 3 validation videos for sunny weather: it is
thus designed to evaluate the effect of different sunny conditions, as camera
glare can be an issue when the vehicle is turning or facing the sun directly.

Note that there is no split to simulate a bias towards snowy conditions, as
the dataset only contains one video of that kind. The test set (bottom row) is
more uniform, as it contains one video from each environmental condition.

##### Results on the various tasks

Results are reported for the tasks discussed in Section 3.2.4.

_Frame-level results across the five detection tasks_ are summarised in Table
3.5 using the frame-mAP (f-mAp) metric, for a detection threshold of
$\delta=0.5$. The reported figures are averaged across the three splits
described above, in order to assess the overall robustness of the detectors to
domain variations. Performance within each split is evaluated on both the
corresponding validation subset and test set. Each row in the Table shows the
result of a particular combination of backbone network (2D, I3D, or Slowfast)
and test-time sequence length (in a number of frames, 8 and 32). Frame-level
results vary between 16.8% (events) and 65.4% (agentness) for I3D, and between
23.9% and 69.2% for Slowfast.

_Video-level results_ are reported in terms of video-mAP in Table 3.6. As for
the frame-level results, tube detection performance (see Sec. 3.3.2) is
averaged across the three splits. One can appreciate the similarities between
frame- and video-level results, which follow a similar trend albeit at a much
lower absolute level. Again, results are reported for different backbone
networks and sequence lengths.

Table 3.5: Frame-level results (mAP $\%$) averaged across the three splits of
ROAD. The considered models differ in terms of backbone network (2D, I3D, and
Slowfast) and clip length (08 vs 32). The performance of YOLOv5 on agent
detection is also reported. Detection threshold $\delta=0.5$. Both validation
and test performance are reported for each entry.

Model | Agentness | Agents | Actions | Locations | Events
---|---|---|---|---|---
2D-08 | 51.8/63.4 | 30.9/39.5 | 15.9/22.0 | 23.2/30.8 | 10.6/12.8
2D-32 | 52.4/64.2 | 31.5/39.8 | 16.3/22.6 | 23.6/31.4 | 10.8/13.0
I3D-08 | 52.3/65.1 | 32.2/39.5 | 19.3/25.4 | 24.5/34.9 | 12.3/16.5
I3D-32 | 52.7/65.4 | 32.3/39.2 | 19.7/25.9 | 24.7/35.3 | 12.6/16.8
Slowfast-08 |  68.8/69.2 | 41.9/47.5 | 26.9/31.1 |  34.6/37.3 |  18.1/23.7
Slowfast-32 |  69.3/68.7 | 42.6/43.7 |  27.3/31.7 |  34.8/36.4 |  18.0/23.9
YOLOv5 | - |  57.9/56.9 | - | - | -

Table 3.6: Video-level results (mAP $\%$) averaged across the three ROAD
splits. The models differ in terms of backbone network (2D, I3D and Slowfast)
and test time clip length (08 vs 32). The performance of YOLOv5 on agent
detection is also reported. Both validation and test performance are reported
for each entry.

Model | Agents | Actions | Locations | Events
---|---|---|---|---
Detection threshold $\delta=0.2$
2D-08 | 22.2/25.1 | 10.3/13.9 | 18.2/24.8 | 12.8/14.7
2D-32 | 22.6/25.0 | 11.2/14.5 | 18.5/25.9 | 13.0/15.3
I3D-08 | 23.2/26.5 | 14.1/15.8 | 20.8/25.8 | 14.9/17.4
I3D-32 | 24.4/26.9 | 14.3/17.5 | 21.3/27.1 | 15.9/17.9
Slowfast-08 | 24.1/29.0 |  16.0/20.5 |  28.3/33.0 | 18.9/22.4
Slowfast-32 | 24.2/28.6 |  16.0/19.55 |  29.0/29.7 |  19.1/22.5
YOLOv5 |  38.8/43.3 | - | - | -
Detection threshold $\delta=0.5$
2D-08 | 8.9/7.5 | 2.3/3.0 | 5.2/6.1 | 5.1/5.3
2D-32 | 8.3/8.0 | 2.7/3.3 | 5.6/7.1 | 5.0/5.7
I3D-08 | 9.2/9.6 |  4.0/4.3 | 5.8/6.9 | 4.6/5.4
I3D-32 | 9.7/10.2 |  4.0/4.6 | 6.4/7.7 | 4.8/6.1
Slowfast-08 | 7.1/8.9 |  3.9/4.7 |  7.1/11.0 |  6.5/6.6
Slowfast-32 | 8.3/9.8 | 3.7/4.4 |  8.4/10.0 |  5.3/7.3
YOLOv5 |  18.7/13.9 | - | - | -

Table 3.7: Number of video- and frame-level instances for each label
(individual or composite), left. Corresponding frame-/video-level results
(mAP@$\%$) for each of the three ROAD splits (right). Val-$n$ denotes the
validation set for Split $n$. Results produced by an I3D backbone.

| Number of instance |  Frame-mAP<EMAIL_ADDRESS>
---|---|---
Train subset | #Boxes/#Tubes | Train-1 | Train-2 | Train-3
Eval subset | All | Val-1 | Val-2 | Val-3 | Test | Val-1 | Test | Val-2 | Test | Val-3 | Test
Agent | 559142/7029 | 60103/781 | 79119/761 | 83750/809 | 82465/1138 | 44.5/30.1 | 34.0/25.7 | 17.2/16.0 | 40.9/27.4 | 35.3/27.1 | 42.6/27.5
Action | 639740/9815 | 69523/1054 | 89142/1065 | 95760/1111 | 94669/1548 | 26.2/17.0 | 26.6/17.4 | 11.7/11.4 | 25.3/17.3 | 21.2/14.6 | 25.7/17.9
Location | 498566/8040 | 56594/851 | 67116/864 | 77084/914 | 70473/1295 | 34.9/28.6 | 35.2/26.4 | 13.7/12.1 | 33.9/26.3 | 25.4/23.2 | 36.7/28.6
Event | 453626/8394 | 43569/883 | 65965/963 | 72152/967 | 64545/1301 | 17.7/18.6 | 15.9/15.8 | 6.4/11.8 | 16.4/18.9 | 13.7/17.2 | 18.1/18.9
Number of instances | Frame-AP
AV-action | 122154/490 | 17929/67 | 18001/56 | 16700/85 | 20374/82 | 57.9 | 45.7 | 33.5 | 43.6 | 43.7 | 48.2

_Streaming deployment_. Increasing test sequence length from 8 to 32 does not