1.0images/aveas-bold-wallpaper.pdf

# An Approach to Systematic Data Acquisition and Data-Driven Simulation for
the Safety Testing of Automated Driving Functions∗

L. Eisemann1,7, M. Fehling-Kaschek${}^{2\text{A}}$, H. Gommel3, D. Hermann4,7,
M. Klemp5, M. Lauer5, B. Lickert${}^{2\text{A}}$,
F. Luettner${}^{2\text{A}}$, R. Moss${}^{2\text{A}}$, N. Neis5,7, M.
Pohle${}^{2\text{B}}$, S. Romanski6, D. Stadler${}^{2\text{C},5}$, A.
Stolz${}^{2\text{A}}$, J. Ziehn${}^{2\text{C}}$, J. Zhou5,7 *This publication
was written in the context of the AVEAS research project (www.aveas.org),
funded by the German Federal Ministry for Economic Affairs and Climate Action
(BMWK) within the program “New Vehicle and System Technologies”.1Stuttgart
Media University, 70569, Stuttgart, Germany2Fraunhofer-Gesellschaft e.V.,
80007, Munich, Germany ${}^{\text{A}}$Fraunhofer EMI, 79588 Efringen-Kirchen,
Germany ${}^{\text{B}}$Fraunhofer IVI, 01069 Dresden, Germany
${}^{\text{C}}$Fraunhofer IOSB, 76131 Karlsruhe, Germany{mirjam.fehling-
kaschek@emi, florian.luettner@emi, maria.pohle@ivi,
<EMAIL_ADDRESS>GmbH, 71287 Weissach, Germany4Technische
Universität München, 85748 Garching b. München<EMAIL_ADDRESS>Institute of Technology, 76131, Karlsruhe, Germany6understandAI GmbH, 76227,
Karlsruhe, Germany7Porsche Engineering Group GmbH, 71287, Weissach, Germany

###### Abstract

With growing complexity and criticality of automated driving functions in road
traffic and their operational design domains (ODD), there is increasing demand
for covering significant proportions of development, validation, and
verification in virtual environments and through simulation models.

If, however, simulations are meant not only to augment real-world experiments,
but to replace them, quantitative approaches are required that measure to what
degree and under which preconditions simulation models adequately represent
reality, and thus, using their results accordingly. Especially in R&D areas
related to the safety impact of the “open world”, there is a significant
shortage of real-world data to parameterize and/or validate
simulations—especially with respect to the behavior of human traffic
participants, whom automated driving functions will meet in mixed traffic.

We present an approach to systematically acquire data in public traffic by
heterogeneous means, transform it into a unified representation, and use it to
automatically parameterize traffic behavior models for use in data-driven
virtual validation of automated driving functions.

##  I INTRODUCTION

The introduction of automated driving functions achieving SAE Level 3 and
above presents a significant shift in the requirements for validation and
testing approaches, due to a substantial increase in complexity. This
complexity is a combination of several related factors: The increased scope of
the operational design domain (ODD) that needs to be covered, including
diverse urban scenarios; the increased responsibility of the automated driving
functions for a wider range of aspects within the dynamic driving task (DDT);
the increased level of safety that must be achieved without relying on the
human driver as a fallback; and the increased complexity of the
hardware/software system capable of operating under these conditions. The
latter typically involves a wide array of heterogeneous sensors, different
high-performance hardware platforms interoperating in realtime, and a
combination of classically-developed software and modern machine learning (ML)
models for various tasks that require new development and testing paradigms.

The increase in complexity is mirrored in the evolutions of automotive safety
standards from functional safety (ISO 26262) over safety of the intended
functionality (ISO 21448, SOTIF) to safety and artificial intelligence (ISO
8800). This evolution marks a transition from the consideration of individual
components under general conditions towards the consideration of the overall
system (including, e.g., training data for ML) in its realistic environment
(including, e.g., the behavior of human traffic participants [1] and
passengers of automated vehicles [2]). At the same time it highlights the
necessity of addressing not only “known” limitations and failure modes, but
also defining processes to identify system limits that were not known or
specified during development time. The combination of these effects manifested
notably in the fatal 2016 accident between a Tesla Model S and a tractor-
semitrailer truck in Florida [3].

###  I-A The Role of Data in Validation

For validation approaches of automated driving functions that aim at
establishing quantitative estimates for the safety impact of operation, the
availability of statistically significant real world data is crucial: Any
tested scenario, especially a scenario where the _system under test_ (SUT)
fails, must be quantified in a stochastic sense to determine the impact of
this failure on the safety level. Failure scenarios that have a high exposure
(e.g., failing to detect a leading vehicle) will be considered a priority in
development, while scenarios that are (based on available data) extremely
unlikely (e.g., failing to detect a zebra on a zebra crossing) may be
considered low priority or even be left as residual risks. Specifically, the
perspective would be that if the SUT ever has an accident, it will be
considered “well designed” if the accident occurs within the stochastic rate
predicted during development and under conditions that were known during
development. Achieving this degree of accuracy by means of virtual validation
requires the provision of real-world data to parameterize state of the art
behavior models (cf. also [4]).

TABLE I: Overview of different employed acquisition methods and distinctive properties to be considered in the unified format. Platform / Source |  Environment Sensor(s) |  Ground FOV |  Interior Observation |  Platform Speed |  Max. Rec. Duration |  Mapping Principle |  Notable _missing_ features |  Notable Acq. Limitations
---|---|---|---|---|---|---|---|---
Road vehicle [5] |  LiDAR, RGB camera |  $\approx 50~{}\text{m}^{2}$ |  — |  0–200 km/h and above |  5 h |  LiDAR-based |  — |  Strongly adverse driving conditions
Infrastructure |  LiDAR |  $\approx 100~{}\text{m}^{2}$ |  — |  — |  weeks |  landmark-based ref. |  lights, colors |  Strong precipitation
Infrastructure |  LWIR camera |  $\approx 100~{}\text{m}^{2}$ |  — |  — |  weeks |  landmark-based ref. |  lights, colors |  Temp. below freezing
Aircraft |  RGB camera |  $\approx 400~{}\text{m}^{2}$ |  — |  70–200 km/h |  2 h |  orthophoto-based |  lights, small VRUs |  Strong precipitation, night, fog
GIDAS data |  (accident reconst.) |  — |  (accident reconst.) |  — |  30 s prior to accident |  manual |  accurate trajectories |  Only selected accidents
Driving sim. studies |  sim. ground truth |  — |  mult. driving/human param. |  — |  1 h |  Sim. ground truth |  — |  Only one instructed test person per run

###  I-B Scope of the Paper

In this paper, we present a pipeline from data acquisition to the optimized
parameterization of behavior models and realistic traffic simulations of
relevant use cases. First, we describe the steps which need to be taken to
process data recordings to obtain trajectories with relevant parameters of all
recorded traffic participants, accompanied by OpenDRIVE-based road
representations [6]. Second, we introduce an approach to derive optimized
behavior models for PTV Vissim [7] based on the trajectories obtained from the
recorded data. We present an exemplary optimization of the desired speed
distributions of cars and trucks based on data collected via aerial image
acquisition as described before. We also discuss how the optimization can be
extended in the future. Finally, we describe the usage of the derived
optimized behavior models for traffic simulations of relevant use cases. Here,
PTV Vissim is coupled with a Carla-based [8] simulator to generate a digital
twin of the SUT [5].

##  II MEANS OF DATA ACQUISITION

Within the scope of our studies, various methods of traffic data acquisition
are employed, each with individual strengths and weaknesses, namely road
vehicle-based, infrastructure-based, and aircraft-based platforms featuring
either RGB cameras, LWIR cameras, LiDAR, radar, or a combination thereof (cf.
Tab. I). Compatible data are also acquired from driving simulator studies and
accident reconstructions, leading to a wide range of different data
granularities and available fields (see Tab. I for an overview). Each method
includes a highly individual processing chain based on available sensor data
and platform parameters, whose output, however, is a unified format of _map-
referenced trajectories_. This format comprises a minimal set of mandatory
parameters that can be determined across any acquisition method, sufficient to
formulate, parameterize and/or evaluate driving behavior models. These
parameters include:

  * •

a map representation based on OpenDRIVE [6], including road geometry, relevant
traffic signs, etc.

  * •

dynamic traffic participants, containing at least:

    * –

a coarse classification of the participant (car, truck bicycle, pedestrian,
…);

    * –

its physical size (at least length and width);

    * –

its trajectory (geographic _and_ road map coordinates over time) at a rate of
several Hz (pref.: $>$ 10 Hz);

    * –

uncertainty specifications for all measured and derived values, limited to
Gaussians for real-valued data and probability mass functions for finite,
discrete properties (classes, binary states, etc.);

  * •

unobserved areas due to occlusions or sensor limits;

  * •

metadata such as timestamp, geographic location, etc., and information whether
the scenario is entirely “natural”, or partly or completely “staged” or
“simulated”;

  * •

derived information from post-processing such as time to collision (TTC) [9]
and post encroachment time (PET) [10]; accelerations per participant over
time; traffic density.

A wider range of optional parameters cannot be extracted by all methods, but
may be valuable for specific models or hypotheses, and is in some cases added
by fusion with external sources, including weather and lighting conditions,
vehicle light states, road conditions, high-resolution lateral positions (cf.
[11]) and (ego) vehicle interior properties such as driver pose and attention
direction, which is used, e.g., in the SCM behavior simulation model [12].
Within the driving simulation studies, multiple objective (technical/human)
and subjective measurements will be derived using a virtual reality-based
advanced driving simulation system as depicted in [13] using different
interior setups, one of them based on the vehicle used for data acquisition.

###  II-A Scope and Assumptions for the Acquisition

Regions of interest for data acquisition range from highways over urban
environments to parking areas. The scope of the project is the acquisition of
_critical_ scenarios for the validation of near-future automated driving
functions in mixed traffic, based on the perspective of ISO 21448, and
considers relevant factors to comprise…

  * •

criticality introduced by the behavior of human traffic participants as
statistically observable today;

  * •

criticality introduced by characteristics of automated vehicles’
sense–plan–act pipelines through which their behavior deviates from human
behavior while in automated driving mode (such as sensor characteristics, ML
perception and fusion [14], planning methods, …);

  * •

criticality from the transition between manual and automated control of the
DDT, either by handover from the system to the human driver, or from
intervention of the human driver into the system’s actions.

Therefore, accident databases and police records were filtered by relevant
accident types for automated driving functions, to determine specific
acquisition sites with a relatively high exposure of homogeneous such accident
types. At these sites, the goal is to acquire not the relatively rare actual
accidents, but a larger proportion of near misses. Here, we define _near
misses_ as “scenarios in which an accident was averted predominantly due to
stochastic effects, not (exclusively) due to the controlled behavior of the
involved parties”. This definition enables the stochastic verification for
derived models: A variation of a “near miss” scenario under stochastic
parameters determined from the data-driven approach should approximate the
actual accident statistics.

###  II-B Map-Referencing

To enable meaningful behavior analyses on the detected traffic participants,
their trajectories must be viewed in relation to the underlying road network
geometry and topology. Whether, for example, two cars are on the same lane or
on neighboring lanes is decisive in established behavior models like the one
used in PTV Vissim [7], but can only be determined reliably by considering the
road geometry. Similarly, behaviors such as lane changes depend on available
options (available lanes) and demands (such as upcoming highway exits or speed
limits).

Hence, the observed traffic participants and their trajectories must be
referenced with detailed map data, and in particular, this map data must above
all assure that the dynamic objects (traffic participants) and the map data
are consistent with each other. The geographic accuracy of the maps is, in
contrast, an expendable feature. This consideration is of highly practical
importance: Mapping and object detection may be performed by different
methods, on different data and—possibly—at different times using different
platforms. Furthermore, the ASAM OpenDRIVE format (as the most widely adopted
map standard in virtual testing, see [15]) uses Cartesian coordinates
generally disregarding earth’s curvature. Hence, in addition to measurement
errors, surface projection errors can be significant. This is not a relevant
limitation either for pure simulation scenarios or automated driving where the
vehicle performs active ego localization within the map. When merging data
from different sources, however, offsets from mapping uncertainty,
reprojection uncertainty and localization/detection uncertainty (generally
independent) will easily accumulate to orders of magnitude comparable to
typical lane widths.

Therefore, to primarily assure that extracted map data and extracted traffic
data are consistent, our approach requires each acquisition method to provide
elementary mapping directly related to the raw sensor data (opposed to
requiring purely a detection of dynamic objects which are overlaid with map
data from a different source), as described in [16].

(a) Computed orthophoto and elevation map

(b) Detection of driving and parked cars (blue) and trucks (green)

(c) Detected and tracked vehicles with OpenDRIVE lanes (light blue)

Figure 1: Samples of the aerial image traffic data acquisition method, based
on images recorded by a Flight Design CTSW ultralight aircraft between the
cities of Karlsruhe and Rastatt.

###  II-C Object Detection and Tracking

We describe the process of object detection and tracking on the example of the
aerial image acquisition method shown in Fig. 1 and used subsequently in Sec.
III, as well as for the LiDAR / infrastructure-based acquisition shown in Fig.
2 and briefly for the road vehicle-based acquisition.

####  II-C1 Aerial Data Acquisition

The data are acquired by a Flight Design CTSW ultralight aircraft, carrying a
sensor unit combining a calibrated 4K camera with IMU and GNSS. Images are
acquired at 20 FPS and a ground sampling distance of approximately 12.5 cm.

In the first processing step, a digital elevation model (DEM) is computed
through bundle adjustment, using raw images and the IMU/GNSS data (Fig. 1(a)).
This step assures a robust registration of the raw images into an _orthophoto_
, a projection of the acquired data onto reference ground plane coordinates
(in particular eliminating geometric displacement effects from elevated
structures), and hence an intrinsically consistent geo-referencing within the
raw data footage as outlined in Sec. II-B. The orthophoto enables an easy
extraction of the road geometry (in the OpenDRIVE format). For the shown
dataset, this information was annotated manually; for large-scale acquisition
of different regions, an automatic implementation is currently being
developed.

Furthermore, object detection is performed on the raw data stream using the
CNN-based rotation-invariant detection architecture ReDet [17], available in
MMRotate [18], trained on the iSAID [19] dataset and fine-tuned on 1324
manually annotated images from the target domain. The iSAID instance
segmentation dataset provides a wide range of classes over 2806 high-
resolution images, of which only the distinction between cars (iSAID class
“small vehicle”) and trucks (iSAID class “large vehicle”) is relevant for the
use case. The detection already yields object-aligned bounding boxes for each
vehicle, providing an indication of the expected direction of travel (Fig.
1(b)). Detection results per vehicle class are shown in Tab. II using 510
manually annotated test images (mean average precision (mAP) is obtained at an
IoU threshold of 0.5), comparing pure iSAID training with fine-tuning on the
target domain. Both runs use the ReDet detector with a ReResNet50 [17]
backbone.

Subsequently, detected objects are tracked between the geo-referenced frames
to achieve map-referenced trajectories w.r.t. the extracted OpenDRIVE maps
(Fig. 1(c)), using a DeepSORT-based tracker [20]. The original CNN for
appearance feature extraction is replaced with OSNet [17] which is trained on
a custom built re-identification dataset based on cropped image patches from
the VisDrone [21] and UAVDT [22] multi-object-tracking datasets. The custom
re-identification dataset comprises a total of 31 591 images containing 1 372
various vehicles.

TABLE II: Quantitative results over frames and vehicle classes of the aerial acquisition using ReDet & ReResNet50. Train datasets | AP${}_{\text{car}}$ | AP${}_{\text{truck}}$ | mAP
---|---|---|---
iSAID [19] | 89.6 | 69.9 | 79.7
iSAID + fine-tuning on target domain | 90.4 | 85.5 | 87.9
Figure 2: A point cloud from the infrastructure recording after applying the
processing pipeline. Background points are colored in blue and points
belonging to an object have a different color based on the tracked object.

####  II-C2 Infrastructure LiDAR Data Acquisition

For the LiDAR-based infrastructure recording, the LiDAR is mounted on a pole
in approx. 3 m height next to intersections selected according to Sec. II-A.
As the processing is done on-site, it requires efficient algorithms. In the
infrastructure-based recording, we can consider the background as static, with
only traffic participants moving in consecutive point clouds, enabling a
background subtraction approach, based on the general idea of [23], in which
the background is constructed based on the range of individual LiDAR beams.
Accordingly, in each point cloud, we remove the background and cluster the
remaining points. These detected clusters represent the traffic participants.
To track them, we formulate tracking as a min-cost flow problem and build up a
graph of detections over time by connecting detections in consecutive point
clouds with their Euclidean distance as cost. We use the muSSP algorithm [24]
to efficiently solve this min-cost flow problem. Fig. 2 shows the result after
tracking individual traffic participants.

####  II-C3 Vehicle-Based Data Acquisition

The vehicle-based data acquisition follows a similar principle. The recording
vehicles make use of camera and LiDAR data to use their complementary
strengths. LiDARs provide accurate 3D information but lack information density
in far distances in comparison to cameras. As the recording vehicle is
typically moving in critical scenarios, the background is harder to detect and
remove in vehicle-based data acquisition. Hence, only the ground plane is
algorithmically removed from the point cloud to simplify the dynamic object
detection task. The camera signal is used to detect and track dynamic objects
with an ML-driven pipeline. The corresponding 3D shapes and movements are
obtained by separate ML models. As the data processing is mostly performed
offline, human quality reviews can elevate the data quality to ground truth
accuracy where needed. Through the use of accurate GNSS/IMU systems, the 3D
objects are referenced onto the individual OpenDRIVE maps.

##  III MODEL PARAMETERIZATION

Under the premise of evaluating automated driving functions against human
traffic participants in simulations, we aim at developing an optimization
framework to parameterize driving behavior models based on the previously
showcased data. These models can then be used in agent-based simulations, in
which the interaction between a vehicle with automated driving functions (the
SUT) and realistically behaving opponents can be observed and assessed from a
safety perspective. For data driven validation of SUTs, it is important to do
so not just by replaying recorded trajectories of opponents identically, but
rather reproducing their characteristics from a stochastic point of view. As
such, as part of the optimization framework, we determine the log-likelihood
of recorded observations given the simulated distributions of interest, such
as the vehicle velocities, as shown below. Subsequently, the parameters of the
underlying behavior models are iteratively improved such that the likelihood
to observe the data, under the hypothesis that the model parameters of the
simulation are true, is maximized. Having determined an optimized model
parameterization, we can then create arbitrary variations of realistic traffic
dynamics, in which automated driving functions can be tested interactively or
which can be used to create alternative scenarios.

###  III-A Simulation Model Framework

All of the simulations of traffic dynamics shown hereafter were performed
using PTV Vissim [7]. Here, the Wiedemann99 model [25] was used to simulate
the vehicle following dynamics of the traffic participants. Each of the
model’s ten parameters describes a different aspect of the agent dynamics when
following a preceding vehicle. Further, a modified version of the model by
Sparmann [26] was used to characterize lane change dynamics of the vehicles.
As part of an early investigation, a simulation setup was created based on the
initially recorded aerial data showcased in Sec. II / Fig. 1. Therefore, a
3-lane 4 km long interstate highway (_autobahn_) section with a traffic volume
of 1680 cars and 320 trucks per hour was created within PTV Vissim. The
distribution of the _desired velocity_ for each vehicle type was then set
based on a Gaussian distribution, with a given mean $\mu$ and standard
deviation $\sigma$. To ensure the simulated traffic dynamics have reached a
stable state, an additional inflow track with the length of 1 500 m was added
to the previously defined highway (but disregarded in the later analysis of
the simulation results).

###  III-B Optimization Framework

To evaluate the applicability of the concept, we optimized the four parameters
($\mu_{\text{vel,car}}$, $\sigma_{\text{vel,car}}$, $\mu_{\text{vel,truck}}$,
$\sigma_{\text{vel,truck}}$) of the _desired velocity_ distributions needed to
recreate the mean recorded velocities from Sec. II. This includes the data
extracted from the trajectory of 178 cars and 33 trucks, recorded within a
time frame of approximately 2 min. For the optimization, as previously
mentioned, we defined an objective function to be minimized via the log-
likelihood ($\ln(\mathcal{L})$) based on simulated and recorded mean
velocities of the respective observed vehicle types,

$\begin{split}\\!\\!-\ln(\mathcal{L})=&-\\!\\!\sum_{k}^{N_{\text{car}}}\\!\ln(\phi_{\text{vel,car}}(v_{\text{car,k}}))-\\!\\!\sum_{l}^{N_{\text{truck}}}\\!\ln(\phi_{\text{vel,truck}}(v_{\text{truck,l}}))\,,\\!\end{split}$
(1)

where $\phi_{\text{vel,car}}$ and $\phi_{\text{vel, truck}}$ are the simulated
probability densities of car and truck velocities, $v_{\text{car,k}}$ and
$v_{\text{truck,l}}$ are the recorded velocities of the $k^{\text{th}}$ car
and $l^{\text{th}}$ truck, and $N_{\text{car}}$ and $N_{\text{truck}}$ denote
the total number of recorded cars and trucks. The simulated probability
densities $\phi_{\text{vel,car}}$ and $\phi_{\text{vel,truck}}$ were estimated
using Gaussian kernel density estimators as implemented in the Python scikit-
learn package [27]. The objective function was minimized using the Nelder–Mead
algorithm [28] implemented in the Python SciPy package [29]. To confirm
robustness of the optimization method, the parameter estimation was repeated
100 times from different initial parameter values sampled using a uniform
distribution. The estimated _desired velocity_ parameters corresponding to the
best objective function value of the 100 estimation runs were
$\mu_{\text{vel,car}}=131.05~{}\text{km/h}$ and
$\sigma_{\text{vel,car}}=17.48~{}\text{km/h}$ for the _car_ type vehicles and
$\mu_{\text{vel,truck}}=89.22~{}\textrm{km/h}$ and
$\sigma_{\text{vel,truck}}=6.20~{}\text{km/h}$ for the _truck_ type vehicles.
A histogram of the recorded velocity data for each vehicle type in comparison
to the density estimations calculated from the simulation results using the
optimized set of parameters is shown in Fig. 3. Generally, the same minimum
was found for the majority of the estimation runs for all four parameters,
confirming the robustness of our optimization framework, see Fig. 4. Some
exceptions could be observed in terms of $\sigma_{\text{vel,car}}$. Here, our
optimization framework, depending on the starting values, sometimes
overestimated $\sigma_{\text{vel,car}}$, due to the underlying measured data
including some cars with a significantly higher mean velocity ($>$160 km/h).
This could potentially be rectified in the future by including a third
behavior type, beside the already included _truck_ and _car_ vehicles.

The identification of the number of minimum necessary agent classes for the
optimal simulative mapping of real traffic can be integrated with the
optimization algorithm in the future. After an initial clustering of the data
to identify possible agent classes, the amount would be iteratively increased
for several optimization runs. The quotient of the resulting different
optimization results to the optimum can serve as a measure for significance.
If the significance for additional agent classes falls below an acceptance
threshold to be defined, the optimal number of agent classes is reached. Yet,
as the data we have been using as part of the optimization is only a very
small subset of the expected future available data, such an optimization thus
far lacks a sufficient data basis and would be prone to overfitting.

Figure 3: Comparison between the histogram of the extracted mean velocities
from our recorded data (trucks: green, cars: blue) and the respective density
estimations determined from the simulated mean velocities (trucks: orange,
cars: black). Figure 4: Comparison of the respective 100 resulting $\mu$ and
$\sigma$, of each individual estimation run. The parameters for the truck type
are depicted in green and car type in blue. Further, initial guesses for each
estimation run are shown in gray.

###  III-C Extension of Optimization

Up to now, the main focus of the simulations has solely been on adjusting the
four parameters describing the distributions of the _desired velocity_ via
optimization. Yet, in order to recreate a more complete picture of the traffic
dynamics, our framework needs to be expanded to include the other parameters
of the behavior model as well. Thus, distributions of metrics, which correlate
with specific parameters of the Wiedemann99 model, need to be defined. These
distributions can then be added to the definition of the objective function
(1), to broaden the scope of the optimization framework. As part of this
project, a further focus will be set on metrics that ensure the inclusion of
realistic critical traffic situations within the simulated dynamics.

To illustrate this for one such potential metric, Fig. 5 presents a
sensitivity analysis regarding the influence of Wiedemann99 parameters on the
aforementioned _time to collision_ (TTC) metric, i.e., the ratio of the
distance between consecutive cars and their velocity difference [9]. The data
inspected here consist of numerous simulations with varying Wiedemann99
parameters [25] applied to a subset of simulated cars. Each of the ten
Wiedemann99 parameters (cc0–cc9) was varied individually within the value
ranges listed in Tab. III, while fixing the other parameters to PTV Vissim
default values. Every single simulation has a duration of 4 hours and the
traffic composition is similar to the data considered for the optimization
(Fig. 3), i.e., consisting of roughly $80~{}\%$ cars and $20~{}\%$ trucks.

TABLE III: Variation of Wiedemann99 parameters. Parameter | Start Value | End Value | Default Value [25]
---|---|---|---
cc0 |  $0.25$ m |  $2.5$ m |  $1.5$ m
cc1 |  $0.1$ s |  $1.0$ s |  $0.9$ s
cc2 |  $1.0$ m |  $5.5$ m |  $4.0$ m
cc3 |  $-2.0$ s |  $-11.0$ s |  $-8.0$ s
cc4 |  $-0.1$ m/s |  $-0.55$ m/s |  $-0.35$ m/s
cc5 |  $0.1$ m/s |  $0.55$ m/s |  $0.35$ m/s
cc6 |  $8.44$ (m$\cdot$s)-1 |  $12.94$ (m$\cdot$s)-1 |  $11.44$ (m$\cdot$s)-1
cc7 |  $0.1$ m/s2 |  $0.55$ m/s2 |  $0.25$ m/s2
cc8 |  $1.0$ m/s2 |  $5.5$ m/s2 |  $3.5$ m/s2
cc9 |  $0.5$ m/s2 |  $5.0$ m/s2 |  $1.5$ m/s2

Fig. 5 depicts the minimal observed mean TTC and the minimal observed minimum
TTC extracted from the simulations for the cars with altered Wiedemann
parameters. Here, the minimal mean TTC refers to the single altered vehicle
with the smallest mean TTC per simulation, while the smallest minimal TTC
refers to the smallest TTC that was observed in any timestep of the respective
simulation. It can be seen that smaller values of cc1, which represents the
preferred gap time of a car to the preceding vehicle, lead to a smaller TTC
(Fig. 5, orange curves), indicating that the simulated traffic has become more
critical. Additionally we can observe that parameter cc3, which relates to the
time where the agent starts to decelerate when approaching a slower vehicle,
also has an influence (Fig. 5, red curves). Consequentially, in the future,
the potential inclusion of the distribution of the TTC would allow to expand
our previously illustrated framework to incorporate the optimization of cc1
and cc3.

Figure 5: Variation of the minimal observed mean (top) and minimal observed
minimum (bottom) _time to collision_ (TTC) values when varying Wiedemann99
parameters for a subset of car agents (different colors encode different
parameter variations). Only the parameters cc1 (orange) and cc3 (red) show
significant influences.

##  IV USE CASES

For the productive usage of the developed behavior models and OpenDRIVE scene
descriptions, PTV Vissim will be coupled with a Carla-based simulator. Carla
is an open-source driving simulator that offers a detailed and realistic
virtual environment based on the Unreal Engine [30] for testing automated
driving systems as SUTs. It supports a wide range of sensors and offers
customization options, making it a versatile tool for testing and development.

While PTV Vissim will take over the simulation of the surrounding traffic, the
Carla-based simulator will be responsible for simulating the vehicle with the
SUT deployed. It features a digital twin of the recording vehicle described in
[5]. The digital twin allows the SUT to be integrated into the simulation loop
by incorporating an accurate vehicle dynamics model, sensor model and vehicle
interfaces. Based on this, the twin will be used not only for the resimulation
of the recorded scenarios, but also for the evaluation of the SUT in different
scenarios and scenes based on the behavioral models in PTV Vissim established
according to Sec. III. Here, we will use a prototypical adaptive cruise
control (ACC) advanced driver assistance system and a driver-assisted parking
function as exemplary SUTs for the evaluation of the virtual validation
environment.

The full pipeline can then be applied to various use cases in the field of
automated driving function development and validation. With this setup, the
SUT becomes testable before the first prototype of the vehicle platform is
available. Moreover, the number of costly real-world tests, requiring for
example exclusive access to a test track (where the degree of realism again is
questionable for a different set of reasons), can be reduced, as a pre-
calibration of the parameters can be performed in the virtual environment. In
addition to that, testing of the SUT is easily scalable and an exact
repetition of tests is enabled, e.g., to ensure consistency between different
versions. Due to the variety of acquisition domains, also a variety of
automated driving functions can be tested, ranging from ACC functions
operating on highways to parking functions.

A key benefit of using behavioral models in combination with simulation is the
capability to test scenarios across a variety of environments. The
transferability of these models allows the evaluation of SUTs in different
scenes, such as different variations of lane guidance on highways.
Furthermore, this approach offers the potential to account for regional
disparities in driving behavior. For instance, driving behavior in Europe may
vary from that in China [31], making a European behavior model more precise
when simulating traffic scenarios and ultimately assessing SUTs in Europe.
This ability to incorporate regional variations in driving behavior can
produce more accurate evaluations of SUTs in diverse locations.

##  V CONCLUSIONS AND OUTLOOK

We have presented an approach to systematically acquire real-world data of
driving behavior under critical conditions, and utilizing this data to
parameterize established driving behavior models, outlining both systematic
acquisition goals and assumptions, as well as technical methods for extracting
and tracking traffic participants with sufficient accuracy and optimizing the
behavior models based on the established Wiedemann99 model framework.

The real-world data are acquired at accident hotspots with homogeneous
accident types, using a variety of acquisition means with individual strengths
and limitations. Individual processing chains transform the data into a common
format that can be processed consistently without the need to consider
differences in acquisition methods. Method-dependent data are preserved,
however, through optional parameters—for example the state of vehicle lights,
especially turn indicators, or interior/passenger data.

The analyses indicate that a defined minimal set of parameters, which can be
provided by all considered acquisition methods (including existing datasets
based on accident reconstruction), is sufficient for parametrizing microscopic
behavior models towards a more realistic behavior under critical conditions,
and can serve to provide a more realistic and data-based foundation for the
virtual evaluation of automated driving functions.

### Outlook

As part of the AVEAS research project, aiming at the systematic acquisition of
behavior data for use in the virtual validation of automated driving
functions, the presented results only provide a proof of concept. The amount
of data required to satisfy the requirements for safety assurance of SAE Level
4 or 5 functions in complex environments can only be provided through a long-
term and international acquisition effort. To achieve this, the raw data
processing methods must significantly reduce the effort for manual post-
processing and provide means of determining _quantitatively_ the statistical
significance of the acquired data.

To support international cooperation on this challenge, a recommended format
definition based on OpenSCENARIO, OpenDRIVE and OpenLABEL is underway in
cooperation with ASAM e.V. that can be used across related project efforts for
road safety data acquisition.

The significance of extracted behavior models based on this data still remains
to be shown in the aforementioned quantitative sense, including the comparison
of the extracted models to larger quantities of data, a systematic
identification of distinct behavior models, and an analysis as to whether the
“near miss” definition can be supported stochastically by approximating the
statistical accident count over time. Based on the paradigm of data-driven,
stochastic validation, only a quantitative measure of quality for the
extracted virtual validation pipeline can enable simulations to be used _in
lieu_ of real-world test drives.

## ACKNOWLEDGEMENT

Dedicated to Alexander Muckenhirn (★1978, ✝2023), our long-time pilot for the
aerial image acquisition, without whom we would not have started this project
together.

## References

  * [1] B. Schlag, “Automatisiertes Fahren im Straßenverkehr–Offene Fragen aus Sicht der Psychologie,” _Zeitschrift für Verkehrssicherheit_ , vol. 2, no. 2016, pp. 94–98, 2016.
  * [2] W. König, “Nutzergerechte Entwicklung der Mensch-Maschine-Interaktion von Fahrerassistenzsystemen,” in _Handbuch Fahrerassistenzsysteme_. Springer, 2015, pp. 621–632.
  * [3] N. T. S. B. (NTSB), “Collision between a Car Operating with Automated Vehicle Control Systems and a Tractor-Semitrailor Truck near Williston, Florida, 7 May 2016; NTSB/HAR-17/02-PB2017-102600,” Sept. 2017.
  * [4] N. Neis and J. Beyerer, “Literature review on maneuver-based scenario description for automated driving simulations,” in _2023 IEEE Intelligent Vehicles Symposium_ , 2023, pp. 1–8.
  * [5] J. Haselberger, M. Pelzer, B. Schick, and S. Müller, “JUPITER – ROS based Vehicle Platform for Autonomous Driving Research,” in _2022 IEEE International Symposium on Robotic and Sensors Environments (ROSE)_ , 2022, pp. 1–8.
  * [6] Association for Standardization of Automation and Measuring Systems e.V. ASAM OpenDRIVE. [Online]. Available: https://www.asam.net/standards/detail/opendrive/
  * [7] PTV, “PTV Vissim and PTV Viswalk Help,” https://cgi.ptvgroup.com/vision-help/VISSIM_2023_ENG/Content/0_TitelCopyright/Index.htm, 2023, accessed: 2023-05-03.
  * [8] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “CARLA: An open urban driving simulator,” in _Proceedings of the 1st Annual Conference on Robot Learning_ , ser. Proceedings of Machine Learning Research, S. Levine, V. Vanhoucke, and K. Goldberg, Eds., vol. 78. PMLR, 13–15 Nov 2017, pp. 1–16. [Online]. Available: https://proceedings.mlr.press/v78/dosovitskiy17a.html
  * [9] L. Westhofen, C. Neurohr, T. Koopmann, M. Butz, B. Schütt, F. Utesch, B. Neurohr, C. Gutenkunst, and E. Böde, “Criticality metrics for automated driving: A review and suitability analysis of the state of the art,” _Archives of Computational Methods in Engineering_ , vol. 30, no. 1, pp. 1–35, 2023.
  * [10] L. N. Peesapati, M. P. Hunter, and M. O. Rodgers, “Can post encroachment time substitute intersection characteristics in crash prediction models?” _Journal of safety research_ , vol. 66, pp. 205–211, 2018.
  * [11] N. Neis and J. Beyerer, “A two-level stochastic model for the lateral movement of vehicles within their lane under homogeneous traffic conditions,” in _26th IEEE International Conference on Intelligent Transportation Systems ITSC 2023_ , 2023.
  * [12] A. Fries, F. Fahrenkrog, K. Donauer, M. Mai, and F. Raisch, “Driver behavior model for the safety assessment of automated driving,” in _2022 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2022, pp. 1669–1674.
  * [13] P. Bouchner, “Interactive driving simulators-history, design and their utilization in area of hmi research,” _International journal of systems applications, engineering & development_, vol. 10, no. 2016, pp. 179–188, 2016\.
  * [14] J. Zhou, N. Wandelburg, and J. Beyerer, “Unknown-aware hierarchical object detection in the context of automated driving,” in _26th IEEE International Conference on Intelligent Transportation Systems ITSC 2023_ , 2023\.
  * [15] K.-W. Chiang, H.-Y. Pai, J.-C. Zeng, M.-L. Tsai, and N. El-Sheimy, “Automated Modeling of Road Networks for High-Definition Maps in OpenDRIVE Format Using Mobile Mapping Measurements,” _Geomatics_ , vol. 2, no. 2, pp. 221–235, June 2022. [Online]. Available: https://www.mdpi.com/2673-7418/2/2/13
  * [16] L. Eisemann and J. Maucher, “Automatic odometry-less opendrive generation from sparse point clouds,” in _26th IEEE International Conference on Intelligent Transportation Systems ITSC 2023_ , 2023.
  * [17] J. Han, J. Ding, N. Xue, and G.-S. Xia, “Redet: A rotation-equivariant detector for aerial object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 2786–2795.
  * [18] Y. Zhou, X. Yang, G. Zhang, J. Wang, Y. Liu, L. Hou, X. Jiang, X. Liu, J. Yan, C. Lyu, W. Zhang, and K. Chen, “Mmrotate: A rotated object detection benchmark using pytorch,” in _Proceedings of the 30th ACM International Conference on Multimedia_ , 2022.
  * [19] S. Waqas Zamir, A. Arora, A. Gupta, S. Khan, G. Sun, F. Shahbaz Khan, F. Zhu, L. Shao, G.-S. Xia, and X. Bai, “isaid: A large-scale dataset for instance segmentation in aerial images,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2019, pp. 28–37.
  * [20] N. Wojke, A. Bewley, and D. Paulus, “Simple online and realtime tracking with a deep association metric,” in _2017 IEEE international conference on image processing (ICIP)_. IEEE, 2017, pp. 3645–3649.
  * [21] P. Zhu, L. Wen, D. Du, X. Bian, H. Fan, Q. Hu, and H. Ling, “Detection and tracking meet drones challenge,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , pp. 1–1, 2021.
  * [22] D. Du, Y. Qi, H. Yu, Y. Yang, K. Duan, G. Li, W. Zhang, Q. Huang, and Q. Tian, “The unmanned aerial vehicle benchmark: Object detection and tracking,” 2018\.
  * [23] Z. Zhang, J. Zheng, H. Xu, X. Wang, X. Fan, and R. Chen, “Automatic background construction and object detection based on roadside lidar,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 21, no. 10, pp. 4086–4097, 2020.
  * [24] C. Wang, Y. Wang, Y. Wang, C.-T. Wu, and G. Yu, “mussp: Efficient min-cost flow algorithm for multi-object tracking,” in _Advances in Neural Information Processing Systems_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds., vol. 32. Curran Associates, Inc., 2019\.
  * [25] PTV, “PTV Vissim and PTV Viswalk Help: Description Wiedemann99,” https://cgi.ptvgroup.com/vision-help/VISSIM_2023_ENG/Content/4_BasisdatenSim/FahrverhaltensparameterFolgeverh_Wied99.htm, 2023, accessed: 2023-05-03.
  * [26] U. Sparmann, _Spurwechselvorgänge auf zweispurigen BAB-Richtungsfahrbahnen. Hrsg. vom Bundesminister für Verkehr, Abteilung Strassenbau_ , ser. Forschung Strassenbau und Strassenverkehrstechnik. Forschungsgesellschaft für Straßen- und Verkehrswesen (FGSV), 1978.
  * [27] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine learning in Python,” _Journal of Machine Learning Research_ , vol. 12, pp. 2825–2830, 2011.
  * [28] F. Gao and L. Han, “Implementing the nelder-mead simplex algorithm with adaptive parameters,” _Computational Optimization and Applications_ , vol. 51, pp. 259–277, 05 2012.
  * [29] P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, İ. Polat, Y. Feng, E. W. Moore, J. VanderPlas, D. Laxalde, J. Perktold, R. Cimrman, I. Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, and SciPy 1.0 Contributors, “SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python,” _Nature Methods_ , vol. 17, pp. 261–272, 2020.
  * [30] Epic Games, “Unreal engine.” [Online]. Available: https://www.unrealengine.com
  * [31] Y. Üzümcüoğlu, T. Özkan, C. Wu, and H. Zhang, “Traffic climate and driver behaviors: The moderating role of driving skills in turkey and china,” _Journal of Safety Research_ , vol. 75, pp. 87–98, 2020. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0022437520300931