# Deep Perspective Transformation Based Vehicle Localization on Bird’s Eye
View

1st Abtin Mahyar  Department of Computer and Data Sciences,
Shahid Beheshti University,
Tehran, Iran
<EMAIL_ADDRESS>2nd Hossein Motamednia  High Performance Computing
Laboratory,
School of Computer Science,
Institute for Research in Fundamental Sciences,
Tehran, Iran
<EMAIL_ADDRESS>2nd Dara Rahmati  Faculty of Computer Science and
engineering,
Shahid Beheshti University,
Tehran, Iran
<EMAIL_ADDRESS>
###### Abstract

An accurate understanding of a self-driving vehicle’s surrounding environment
is crucial for its navigation system. To enhance the effectiveness of existing
algorithms and facilitate further research, it is essential to provide
comprehensive data to the routing system. Traditional approaches rely on
installing multiple sensors to simulate the environment, leading to high costs
and complexity. In this paper, we propose an alternative solution by
generating a top-down representation of the scene, enabling the extraction of
distances and directions of other cars relative to the ego vehicle. We
introduce a new synthesized dataset that offers extensive information about
the ego vehicle and its environment in each frame, providing valuable
resources for similar downstream tasks. Additionally, we present an
architecture that transforms perspective view RGB images into bird’s-eye-view
maps with segmented surrounding vehicles. This approach offers an efficient
and cost-effective method for capturing crucial environmental information for
self-driving cars. 111Code and dataset are available at
https://github.com/IPM-HPC/Perspective-BEV-Transformer.

##  I Introduction

Self-driving vehicles rely on multiple sensors, such as Lidar, Radar, and
distance sensors, to obtain a comprehensive representation of the surrounding
environment [1]. RGB cameras are also commonly used to detect traffic signs,
traffic lights, and street lanes. These cameras offer a cost-effective
alternative to other sensors for extracting features for navigation algorithms
[2]. However, the use of multiple sensors in this field presents challenges
due to the high cost and complexity involved. Recent studies have attempted to
extract a similar representation of the environment using only RGB cameras. To
extract characteristics from RGB photos, both Deep Learning (DL) and
traditional image processing techniques have been applied.

One valuable piece of information extracted from RGB images is the detection
of object locations and distances between vehicles in the captured images and
the ego vehicle. Additionally, the direction of surrounding vehicles can be
determined, enabling predictions of their future movements [3]. These types of
information can be effectively aggregated into a single top-down
representation, also called as a bird’s-eye-view (BEV) image, which provides
valuable insights for assessing the navigation system of self-driving vehicles
[4].

Classical image processing methods have utilized homography matrices to
transform RGB images into a 2D top-down representation [5]. These methods
involve calculating the transfer matrix values by adjusting the camera,
allowing each pixel of the RGB image to be mapped to the top-down
representation. However, one challenge of these methods lies in the need to
update the transfer matrix values when the camera’s location is changed to
ensure proper transformation. DL approaches have also been employed for this
task, achieving higher accuracy and efficiency compared to classical methods.
Deep neural networks have the advantage of being able to extract important
information from images, which reduces their sensitivity to changes in the
direction and location of the camera with respect to the objects. [4].

To train a deep neural network to transform perspective view (PV) images
captured by an RGB camera into a bird’s-eye-view (BEV) map, a large and
diverse dataset is required to enhance the model’s generalization and
performance under various conditions. Collecting such data from real-world
environments can be challenging and expensive, particularly as it necessitates
obtaining corresponding top-down representations of the PV images. Therefore,
we present a new approach for data collection tailored to our specific needs,
providing a comprehensive dataset that can be utilized for similar research in
the field of self-driving vehicles. Thus, the following can be summed up as
our primary contributions:

  * •

Created a dataset using a simulator that produces PV images with vehicle
bounding box location information, along with their corresponding BEV map
which can also be applied to similar tasks in the field.

  * •

Proposed an end-to-end deep learning architecture designed and tuned to
transform PV images into accurate BEV maps.

##  II Related Work

###  II-A Mapping from PV to BEV

The task of transforming PV to BEV was mentioned in a couple of studies in the
literature. A wide variety of approaches are employed to fulfill this
objective. In the past, before the emergence and gigantic surge to DL, [5]
tried to solve the problem by doing the mapping using a homography matrix. A
computationally efficient and the earliest work that raises today’s
improvements in this field, yet having some flaws including the rigid flat-
world assumption. Thanks to the recent enhancements in DL, researchers gain
much more accuracy and efficiency in their methods, getting an extraordinary
interest both from academics and industries. However, homography is not yet
fully substituted by new methods and still are being used in recent studies
[6].

Table I provides an overview of relevant studies relevant to this assignment.
The main focus of our work is dedicated to those studies that empower DL
techniques and generate 2D BEV maps of the ego vehicle’s surrounding in which
objects, especially other cars, are identified.

These studies can be divided into four groups based on the model proposed. In
the first group, researchers harness a simple, yet effective, convolutional
neural network (CNN) or propose a brand-new architecture to fulfill the
objective. Given the fact that PV is completely located in a different space
compared to the BEV, studies tried to tackle this variation problem using
generative adversarial network (GAN), auto-encoder (AE), and empowering
transformers along with CNNs, in the next three groups respectively to perform
the transformation. Considering GANs, they can be considerably efficient in
generating BEV maps corresponding to their ability to adversarially learn
producing these maps, ensuring high accuracy by their discriminator. Related
studies, also, change some features in this kind of Neural Network by adding
an extra discriminator [7] or applying geometry [8] as an additional attribute
to the network, or proposing novel methods to adopt the difficulties and
challenges presented in this task, resulting in a better performance.

In the next group of studies, transformation is performed leveraging AE [9],
variational AE (VAE) [10], and an ensemble of these models [11]. In this
regard, given the characteristics of these architectures to encode the input
images to a lower-dimensional latent space, providing a valuable set of
information, they can decently generate corresponding BEV map representations.
Within the final group, there are studies that leverage transformers (with
cross-attention) to carry out the mapping objective by constructing queries
and searching through the corresponding BEV image in order to gain beneficial
features with an attention mechanism.

This is also worth mentioning that there are some studies that established
multiple cameras around the ego vehicle to generate a 360∘ BEV map by
combining features extracted in different angles [12, 13], providing a high-
value scene understanding of the ego vehicle’s surroundings, relaxing the
downstream tasks much more than before. The proposed method in this paper does
not support this since we only installed a single camera parallel to the
ground and facing frontward to capture PV images; However, this method is
flexible and simple enough that can be easily modified to generate 360∘ BEV
maps using previous techniques. Moreover, there are some studies that work
with a sequence of images or videos instead of just a single image each time
to take the best advantage of the information in previous or future frames,
resulting in better accuracy and performance.

TABLE I: Related studies on BEV generation from PV. Ref. | Model | Modality | Accuracy | Year | #Frames/Sequences | Dataset
---|---|---|---|---|---|---
[14] | CNN (SDPN) | image | mIoU = 37.0%, mDistance = 78m | 2017 | - | Synthesised (SVA)
[12] | CNN | image | mIoU=80.9% | 2020 | 33,000 | Synthesised
[8] | BridgeGAN | image | LPIPS=0.242 | 2019 | 40,000 | SVA
[15] | Conditional GAN | image | mDistance=5.91m | 2020 | 7,481 | KITTI, nuScenes
[7] | GAN | image | LPIPS=0.264 | 2020 | - | SVA
[16] | GAN | sequence | RMSE=30.1 | 2021 | - | SVA
[10] | VAE | image | mIoU=59.5% | 2018 | 2,975 | Cityspcases, KITTI
[9] | AE | sequence | mIoU=0.61% | 2019 | 59,500 | Cityspcases, KITTI
[11] | Ensemble of AE | sequence | mIoU=44.3% | 2020 |  $\sim$2,000,000 | Synthesised, NuScenes, Lyft
[17] | CNN + Transformer | image | mIoU=19.1% | 2020 | - | NuScenes, Argoverse
[18] | CNN + Transformer | image | mIoU=14.9% | 2021 | - | NuScenes
[19] | CNN + Transformer | image | mIoU=25.7% | 2022 | - | NuScenes, Argoverse, Lyft
[13] | CNN + Transformer | sequence | mIoU=33.65% | 2022 | 702 | KITTI-360, NuScenes

###  II-B Related Datasets

In recent years, few datasets containing images from street spaces,
pedestrians, and vehicles were publicly published mainly because of the ease
of capturing RGB images and its low cost. Also, some of them provide other
annotations such as bounding boxes for different objects and semantic
segmented maps in various types of environmental and weather conditions as
additional information. [20] is a popular dataset in this context which in
addition to the RGB images that were captured using cameras placed in 6
different directions around the ego vehicle, provides Lidar and Radar images
along with a 360∘ field of view. This dataset comprises 1000 scenes, Each of
the 1000 scenes in this dataset has a duration of 20 seconds and is completely
annotated with 3D bounding boxes for 23 classes and 8 attributes. Also, [21]
is a similar dataset, providing Lidar images along with RGB images; However,
it does not generate a 360∘ field of view and just contains PV images. In the
next version of the dataset [22], 360∘ FoV is also added.

There are also some studies including [14], that synthesized a brand-new
dataset using a virtual city space simulator that replicates real-world
objects and events, reducing the cost of acquisition, in this case, making the
impossible task of taking BEV image along with corresponding PV image
feasible. To the best of our knowledge, there is no specific real dataset that
can directly address this task and researchers need to construct a pre-
processing step before using the dataset [13, 16], increasing the chance of
data loss and decelerate the process. Because DL models are data-hungry, they
require a large volume of data in order to make more accurate predictions. In
this study, we also proposed a huge dataset using a simulator to resolve the
mentioned issue while generating more realistic images.

##  III Proposed Model

We now introduce our suggested approach to the issue of creating a top-down
representation of the environment for ego vehicles utilizing PV pictures and
the locations of the bounding boxes of the vehicles within them. The fact that
certain vehicles are partially unseen to the ego vehicle presents a problem
while executing this transformation; as a result, it is not suitable to use
classical or geometrical techniques in these situations due to the high
probability of mistakes. Since a DL network can learn such situations and
attempt to reduce the error, we can lessen their impact by applying DL
techniques.

Our model comprises two main sections, with the first section referred to as
the input segment. This segment is responsible for gathering inputs,
extracting their key characteristics, and converting them into a different
feature space for subsequent utilization. The input segment is further divided
into two branches. The initial branch employs a pre-trained backbone model to
extract crucial features, such as spatial relative positions among objects in
the input image. These features are then transformed into a feature vector,
representing encoded attributes for further processing. Notably, the backbone
model choice offers flexibility, enabling the utilization of different pre-
trained models for the task at hand. Since the extracted features
significantly influence the final outputs, employing state-of-the-art CNNs
would enhance the overall system performance and accuracy. We elaborate on the
impact of the backbone model’s efficiency in the experimental section.

The second branch of our model is responsible for processing the location
information of the bounding box of the selected vehicle in the front view
image. This branch comprises multiple fully connected dense layers that encode
these positions into another feature vector. Subsequently, these vectors are
aggregated to establish the relationship between the features extracted from
the first and second branches. This aggregation enables the model to locate
selected vehicles in a top-down representation, thereby enhancing its ability
in recognizing their spatial relations.

The second segment of our model comprises two branches dedicated to
determining the location of the bounding box of the selected vehicle along the
horizontal and vertical axes. This design allows the first branch to focus on
the input image and its corresponding extracted features in the horizontal
direction, specifically considering rows of the input image. Similarly, the
second branch focuses on the vertical direction, taking into account the
columns of the input image. To construct these branches, we utilize multiple
fully connected dense layers, where the extracted features from the previous
segment are fed into both branches.

Through our experimental investigations and statistical analysis of the
dataset, we have observed that the majority of vehicles in the proposed
dataset are vertically positioned. This observation aligns with real-world
scenarios. Consequently, we have allocated more dense layers to the second
branch of the second segment, which is responsible for generating the vertical
locations of the bounding box.The decision was motivated by the notably
greater standard deviation noted in the vertical value distribution as opposed
to the horizontal value distribution.

Finally, the localization of the bounding box is accomplished by an aggregator
script, which combines the generated horizontal and vertical locations to form
the final bounding box. To optimize the model, we utilize the mean squared
error (MSE) as our loss function. The MSE compares the predicted values in
both the horizontal and vertical directions and, through the use of an
optimizer, we aim to minimize this loss. In our final architecture, the ReLU
activation function is employed for all layers, except for the output layers.
The output layers utilize linear activation to facilitate the prediction of
bounding boxes in pixel coordinates. A scheme of our proposed architecture is
illustrated in Figure 1.

Figure 1: The proposed architecture of deep neural network for predicting the
bird-view.

##  IV Proposed dataset

The aforementioned challenges discussed in previous sections, including the
lack of availability of a dataset that contains BEV maps along with
corresponding PV images in real-world without any need for further pre-
processing and interpretation, except for some synthesized datasets generated
via simulation engines which were far away from realistic scenes, lead us to
come up with new dataset, a more realistic one, providing a wide variety of
different situation of street spaces, catalyzing researches in this field. We
used an open-source simulation environment called CARLA [23] which provides
various city maps and efficient tools for simulating interactions of
autonomous vehicles and developing different self-driving projects. Moreover,
we take advantage of [24, 25] to develop our data generation pipeline,
structure our code and stored data, and modify it to suit our task.

###  IV-A Ego Vehicle Setup

In order to prepare the dataset using the simulator, we spawned various ego
vehicles in random spawning locations of the map as the origin of the output
images, and we repeat this process in each epoch of data capturing so that the
resulting dataset would have different records from divergent distributions
and environments. Ego vehicle types are different in each epoch and it is
chosen randomly from various options that the simulator offers. In each record
in the final dataset, various valuable information about the ego car such as
name, id, and speed is stored in a structured format.

###  IV-B Sensors and synchronization

We installed multiple sensors on the ego vehicle in order to provide various
and valuable information in one go of the data acquisition process. First, an
RGB camera, capturing PV images, is established parallel to the ground, facing
frontward in the middle of the vehicle. Its attributes such as FOV, output
image size, and relative location with respect to the ego vehicle were tuned
in a way that produces PV images with desired aspect ratio synced with other
sensors while covering the vehicle bonnet. Next, the depth camera sensor which
provides a view over the scene codifying the distance of each pixel to the
camera with the same attributes and location was installed; so the outputs of
the RGB camera and depth sensor completely match each other. Some attributes
of these sensors are described in detail in Table II.

Moreover, we establish a BEV generator pipeline, producing semantic segmented
top-down representation respected to ego vehicle with synchronized aspect
ratio with PV generated images, illustrated 8 various objects with different
colors, is described further in Table II. Also, output maps are rotated to
locate the ego car at the center and bottom of the BEV, which is always facing
frontward, even when it is turning around. Finally, all objects in the
simulated world are frizzed while data acquisition is proceeding in order to
maintain the synchronization of the aforementioned output flows.

TABLE II: Details and attributes associated with installed sensors and BEV generator pipeline. Sensor |  Details
---|---
RGB |  1024 × 768 resolution, 110∘ horizontal FOV, $\sim$800KB, iso = 100, gamma = 2.2, shutter speed = 200 per second, auto exposure, PNG
Depth |  1024 × 768 resolution, 110∘ horizontal FOV, $\sim$100KB, PNG, depth range = [0, 16777215], normalized values
BEV |  200 × 200 resolution, 4 pixels per meter ratio, covering 50 meters in horizontal and vertical aspects respected to ego vehicle, $\sim$10KB, segmented objects: ego vehicle, other vehicles, pedestrians, traffic lights and their state, roads, center lines, margin lines, navigation roads

###  IV-C Maps

We used 6 city maps, the one’s simulator provides in its main release, 3D
models of different towns, and their road definitions. these maps contain
several types of roads (e.g. 2, 3, 4-lane roads), junctions, buildings,
characteristics (e.g. urban, or rural), surface (e.g. flat or uneven), etc.
Also, we define 5 different custom weather conditions based on daytime which
are described in detail in Table III. We also provide different climate
categories such as rainy, sunny, and cloudy. These attributes make the
generated dataset a generalized set, containing a wide variety of situations,
giving us the ability to make our simulation as realistic as possible. The
final dataset evenly contains images from these 6 maps.

TABLE III: Various weather conditions implemented for the simulator.

Type | Cloudiness | Precipitation | Wind Intensity | Sun Altitude Angle
---|---|---|---|---
Morning | Low | High | Medium | 30∘
Midday | Medium | Low | Medium | 80∘
Afternoon | High | Low | Medium | 320∘
Night | Medium | Medium | Medium | 300∘
Default | Low | Low | Low | 0∘

###  IV-D Data annotation

After collecting the raw data in each iteration of the data acquisition
process from the aforementioned sensors, we annotate each vehicle bounding box
both in PV and BEV by using the tools the simulator provides. Outputs only
contain vehicles that have an equal or lower distance of 50 meters respected
to the ego vehicle. Bounding boxes of vehicles were invisible vehicle PV, but
are located at mentioned distance are also issued. The location of the top-
left and bottom-right corners of each vehicle in PV and their corresponding
locations in the BEV map were stored. Distances of other cars with respect to
ego vehicles were calculated using the location of vehicles in the map of
which simulator provided.

###  IV-E Data acquisition

The data acquisition process was repeated 6 times with 6 different maps. For
each process, 100 vehicles were randomly selected from various available cars
which the simulator offers and spawned in different locations. Each vehicle
has an autopilot feature, providing a low-level intelligence for it to make
the number of collisions as small as possible while making random decisions
for navigation in the city. In each process, 20 different ego vehicles were
selected consecutively. Each ego vehicle captured data for 20 continuous
frames for 5 different weather conditions mentioned before. After a frame is
stored, 5 frames will be skipped in order to prevent storing similar images,
ensuring the generalization of situations with different possible
combinations.

Finally, all the acquired data including images, maps, and extra information
were stored in a structured HDF5 file, an open-source file format that
supports large, complex, heterogeneous data that suits our task as well. An
additional script is provided in order to convert stored records in this
format into a raw and usable form for training the model. Table IV provides
statistical information about the proposed dataset, whereas Table V presents a
sample scheme of a single record in a dataset for each car spotted by ego
vehicle.

Figure 2: Data generating pipelines TABLE IV: statistical details of the proposed dataset. Attribute | Number
---|---
Total number of images | 12,000
Total number of unique bounding boxes | 26,070
FPS | 30
Size | 108 GB
Generated vehicles in each map | 100
Different types of vehicles | 26
TABLE V: Attributes in a single record in the proposed dataset. Attribute |  Details
---|---
ID |  Identical number representing each vehicle in the map.
Name |  Producer company along with its model.
Distance |  Distance from ego vehicle in meters.
Timestamp |  Time of issuing.
RGB |  The PV output in the form of an RGB array captured by the front view camera.
Depth |  A normalized depth array captured by the depth sensor.
Bounding box - RGB |  Location of top-left and bottom-right pixels representing the selected vehicle in PV.
Bounding box - BEV |  Location of top-right and bottom-left pixels representing the selected vehicle in BEV map.
Speed |  Speed of ego vehicle during capturing the record.

##  V Experiments

###  V-A Training protocol

In order to assess our model, we divided our dataset into two separate and
independent sets: a test set, which made up the remaining 10% of the created
dataset, and a training set, which made up 90% of the total. In terms of data
augmentation, no specific techniques were required as our proposed dataset
inherently provided a diverse range of input images with varying compositions.
However, a normalization process was applied to the PV images to ensure
consistent and standardized inputs throughout the training process.

Weights pre-trained on the ImageNet dataset were used to initialize the
backbone models in the first branch of our model[26], while the remaining
layers were randomly initialized. To optimize the network, we employed the
Adam optimizer [27] and trained the model for a total of 100 epochs on the
training set. A batch size of 32 was used, along with a momentum of 0.9 and an
initial learning rate of 0.001. During each training iteration, a single
bounding box was selected from all the bounding boxes in a given PV image.
This selected bounding box was then fitted into the model along with the
corresponding PV image.

###  V-B Evaluations

In order to assess the efficacy of our suggested methodology, we performed a
comparison examination with prior research, implementing essential adjustments
to align it with the framework of our dataset. Specifically, we trained the
modified version of the prior work from scratch to ensure a fair comparison.
We compared the performance of our model with two versions of SDPN, employing
linear and tanh activation functions in the last layer. Furthermore, we
evaluated the performance of our proposed model using various backbone
architectures. The details of these backbone architectures are presented in
Table VI. For evaluation, we rely on four metrics, as described in [14]:

#### Intersection over Union (IoU)

The IoU metric quantifies the accuracy of the predicted bounding box $BB_{p}$
in relation to the target bounding box $BB_{t}$:

$IoU(BB_{p},BB_{t})=\frac{Area(BB_{p}\cap BB_{t})}{Area(BB_{p}\cup BB_{t})}$
(1)

where $Area(BB)$ denotes the surface area of the bounding box $BB$.

#### Centroid Distance (CD)

CD is the Euclidean distance, measured in pixels, between the centers of the
bounding boxes. It serves as an indicator of localization precision.222It’s
important to note that the target images have a resolution of 200 × 200, with
each meter represented by 4 pixels.

#### Height, Width Error (hE, wE)

hE and wE represent the average discrepancies in height and width of the
bounding boxes respectively, expressed as a percentage relative to the ground
truth dimensions.

#### Aspect Ratio Mean Error (arE)

arE measures the absolute difference in aspect ratio between the predicted and
ground truth bounding boxes:

$arE=\Bigg{|}\frac{BB_{p}\cdot width}{BB_{p}\cdot height}-\frac{BB_{t}\cdot
width}{BB_{t}\cdot height}\Bigg{|}$ (2)

The results of our evaluation indicate that our proposed model demonstrates
superior performance compared to the previous study across various metrics and
configurations. Notably, when considering each metric and configuration, our
model consistently outperformed the previous study. Specifically, we observed
that the model utilizing a ResNet-50 [28] backbone achieved more precise
predictions of the height, width, and aspect ratio of bounding boxes compared
to other configurations. This highlights the significant impact of the choice
of backbone architecture on the predictive capabilities of our proposed
method.

Alternatively, VGG [29] backbone models exhibited greater accuracy in
predicting the location of the center of bounding boxes compared to ResNet
backbone models. However, it is important to note that the difference in
centroid distances between the two was not significantly high. Nonetheless,
the proposed model with ResNet backbone demonstrated higher IoU values,
indicating better overall performance. These findings emphasize the influence
of the backbone architecture on the prediction accuracy of our proposed
method. Specifically, the ResNet backbone architecture yields superior
performance in terms of predicting bounding box characteristics, resulting in
higher IoU values.

TABLE VI: Results of the performance of the proposed model against the baseline. Model | IoU | CD | hE | wE | arE
---|---|---|---|---|---
SDPN-tanh | 0.1582 | - | 2.8726 | 1.2225 | 3.8878
SDPN-linear | 0.1318 | 4.3232 | 0.9657 | 0.5134 | 1.6710
ProposedMethod-VGG16 | 0.1190 | 3.5033 | 0.0571 | 0.1045 | 0.4298
ProposedMethod-VGG19 | 0.0137 | 4.9355 | 0.2974 | 0.1356 | 0.7286
ProposedMethod-Resnet50 | 0.1897 | 4.4275 | 0.0089 | 0.0744 | 0.5506

##  VI Conclusion

In this paper, we propose an end-to-end deep learning architecture that
generates Bird’s Eye View (BEV) maps from PV images and bounding box location
information. We introduce a novel data generator pipeline that provides PV
images and corresponding BEV maps, eliminating the need for extensive pre-
processing. Through evaluations of our dataset, we demonstrate that our model
outperforms existing baselines. Future work includes constructing 3D BEV maps,
incorporating transformers for improved mapping, expanding the dataset, and
integrating multiple cameras for a comprehensive view. Our approach
contributes to advancing research in this field by providing a realistic
dataset and showcasing the effectiveness of our model in generating accurate
BEV maps.

## References

  * [1] O. Saoudi, I. Singh, and H. Mahyar, “Autonomous vehicles: Open-source technologies, considerations, and development,” _arXiv preprint arXiv:2202.03148_ , 2022.
  * [2] C. Badue, R. Guidolini, R. V. Carneiro, P. Azevedo, V. B. Cardoso, A. Forechi, L. Jesus, R. Berriel, T. M. Paixao, F. Mutz _et al._ , “Self-driving cars: A survey,” _Expert Systems with Applications_ , vol. 165, p. 113816, 2021.
  * [3] Y. Yao, M. Xu, C. Choi, D. J. Crandall, E. M. Atkins, and B. Dariush, “Egocentric vision-based future vehicle localization for intelligent driving assistance systems,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 9711–9717.
  * [4] Y. Ma, T. Wang, X. Bai, H. Yang, Y. Hou, Y. Wang, Y. Qiao, R. Yang, D. Manocha, and X. Zhu, “Vision-centric bev perception: A survey,” _arXiv preprint arXiv:2208.02797_ , 2022.
  * [5] H. A. Mallot, H. H. Bülthoff, J. Little, and S. Bohrer, “Inverse perspective mapping simplifies optical flow computation and obstacle detection,” _Biological cybernetics_ , vol. 64, no. 3, pp. 177–185, 1991\.
  * [6] J. Gu, B. Wu, L. Fan, J. Huang, S. Cao, Z. Xiang, and X.-S. Hua, “Homography loss for monocular 3d object detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 1080–1089.
  * [7] T. Zhou, D. He, and C.-H. Lee, “Pixel-level bird view image generation from front view by using a generative adversarial network,” in _2020 6th international conference on control, automation and robotics (ICCAR)_. IEEE, 2020, pp. 683–689.
  * [8] X. Zhu, Z. Yin, J. Shi, H. Li, and D. Lin, “Generative adversarial frontal view to bird view synthesis,” in _2018 International conference on 3D Vision (3DV)_. IEEE, 2018, pp. 454–463.
  * [9] L. Hoyer, P. Kesper, A. Khoreva, and V. Fischer, “Short-term prediction and multi-camera fusion on semantic grids,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops_ , 2019, pp. 0–0.
  * [10] C. Lu, M. J. G. van de Molengraft, and G. Dubbelman, “Monocular semantic occupancy grid mapping with convolutional variational encoder–decoder networks,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 2, pp. 445–452, 2019.
  * [11] N. Hendy, C. Sloan, F. Tian, P. Duan, N. Charchut, Y. Xie, C. Wang, and J. Philbin, “Fishing net: Future inference of semantic heatmaps in grids,” _arXiv preprint arXiv:2006.09917_ , 2020.
  * [12] L. Reiher, B. Lampe, and L. Eckstein, “A sim2real deep learning approach for the transformation of images from multiple vehicle-mounted cameras to a semantically segmented image in bird’s eye view,” in _2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC)_. IEEE, 2020, pp. 1–7.
  * [13] N. Gosala and A. Valada, “Bird’s-eye-view panoptic segmentation using monocular frontal view images,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 2, pp. 1968–1975, 2022.
  * [14] A. Palazzi, G. Borghi, D. Abati, S. Calderara, and R. Cucchiara, “Learning to map vehicles into bird’s eye view,” in _International Conference on Image Analysis and Processing_. Springer, 2017, pp. 233–243.
  * [15] H. Fraser and S. Wang, “Deepbev: A conditional adversarial network for bird’s eye view generation,” in _2020 25th International Conference on Pattern Recognition (ICPR)_. IEEE, 2021, pp. 5581–5586.
  * [16] V. Jain, Q. Wu, S. Grover, K. Sidana, G. Chaudhary, S. H. Myint, and Q. Hua, “Generating bird’s eye view from egocentric rgb videos,” _Wireless Communications and Mobile Computing_ , vol. 2021, 2021.
  * [17] T. Roddick and R. Cipolla, “Predicting semantic map representations from images using pyramid occupancy networks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 11 138–11 147.
  * [18] Y. B. Can, A. Liniger, D. P. Paudel, and L. Van Gool, “Structured bird’s-eye-view traffic scene understanding from onboard images,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 15 661–15 670.
  * [19] A. Saha, O. Mendez, C. Russell, and R. Bowden, “Translating images into maps,” in _2022 International Conference on Robotics and Automation (ICRA)_. IEEE, 2022, pp. 9200–9206.
  * [20] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, “nuscenes: A multimodal dataset for autonomous driving,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 11 621–11 631.
  * [21] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: The kitti dataset,” _The International Journal of Robotics Research_ , vol. 32, no. 11, pp. 1231–1237, 2013.
  * [22] Y. Liao, J. Xie, and A. Geiger, “KITTI-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d,” _Pattern Analysis and Machine Intelligence (PAMI)_ , 2022.
  * [23] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “Carla: An open urban driving simulator,” in _Conference on robot learning_. PMLR, 2017, pp. 1–16.
  * [24] A. Naoto, “carla dataset runner,” https://github.com/AlanNaoto/carla-dataset-runner, 2019.
  * [25] deepsense ai and M. Martyniak, “Bird-eye’s view for carla,” https://github.com/deepsense-ai/carla-birdeye-view, 2020.
  * [26] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein _et al._ , “Imagenet large scale visual recognition challenge,” _International journal of computer vision_ , vol. 115, no. 3, pp. 211–252, 2015.
  * [27] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [28] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [29] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.


# Improving Zero-Shot Event Extraction via Sentence Simplification

Sneha Mehta
Twitter Cortex
<EMAIL_ADDRESS>
Huzefa Rangwala
George Mason University
<EMAIL_ADDRESS>
Naren Ramakrishnan
Virginia Tech
<EMAIL_ADDRESS>

###### Abstract

The success of sites such as ACLED and Our World in Data have demonstrated the
massive utility of extracting events in structured formats from large volumes
of textual data in the form of news, social media, blogs and discussion
forums. Event extraction can provide a window into ongoing geopolitical crises
and yield actionable intelligence. With the proliferation of large pretrained
language models, Machine Reading Comprehension (MRC) has emerged as a new
paradigm for event extraction in recent times. In this approach, event
argument extraction is framed as an extractive question-answering task. One of
the key advantages of the MRC-based approach is its ability to perform zero-
shot extraction. However, the problem of long-range dependencies, i.e., large
lexical distance between trigger and argument words and the difficulty of
processing syntactically complex sentences plague MRC-based approaches. In
this paper, we present a general approach to improve the performance of MRC-
based event extraction by performing unsupervised sentence simplification
guided by the MRC model itself. We evaluate our approach on the ICEWS
geopolitical event extraction dataset, with specific attention to ‘Actor’ and
‘Target’ argument roles. We show how such context simplification can improve
the performance of MRC-based event extraction by more than 5% for actor
extraction and more than 10% for target extraction.

##  1 Introduction

With the proliferation of social media, microblogs and online news, we are
able to gain a real-time understanding of events happening around the world.
By ingesting large unstructured datasets and converting them into structured
formats such as (actor, event, target) tuples we can make rapid progress in
systems for event forecasting (Ramakrishnan et al., 2014), real-time event
coding (Saraf and Ramakrishnan, 2016) or other applications that can grant
organizations a strategic advantage. Historically, this has been enabled by
efforts such as ICEWS111https://dataverse.harvard.edu/dataverse/icews &
GDELT222https://www.gdeltproject.org/. These systems rely on event extraction
technology to populate their knowledge bases. Fig. 1 gives an example of an
event ‘Bring lawsuit against’ from the ICEWS dataset. Extraction involves
identifying entities (businessman, employees) corresponding to argument roles
‘Actor’ and ‘Target’. The _event_ is triggered by the predicate ‘sued’ in the
figure. However, the extraction technology employed by these

Figure 1: An example of an event of the type ‘Bring lawsuit against’ from the
ICEWS dataset.

systems relies on pattern-based approaches that use handcrafted patterns
designed to extract entities and events (Boschee et al., 2013). Even though
pattern-based methods have high precision, they fail to work on unseen event
types and with new event categories. Hence, there is a need to explore
extraction methods that can extend beyond a fixed domain. Modern approaches
for event extraction (Chen et al., 2015; Nguyen et al., 2016; Wadden et al.,
2019) rely on fine-grained annotations and suffer from data scarcity issues
and error propagation due to pipeline systems.

With the success of large scale pretrained language models on machine reading
comprehension (MRC) tasks (Devlin et al., 2019a; Liu et al., 2019; Huang et
al., 2018), a new paradigm for event extraction based on MRC has surfaced (Du
and Cardie, 2020; Liu et al., 2020). In this approach, event argument
extraction is posed as a span extraction problem from a context conditioned on
a question for each argument. This approach is promising because it mitigates
some of the issues faced by traditional approaches, such as relying on
upstream systems to extract entities/triggers and hence sidestepping the error
propagation problem in pipeline systems. It also gives rise to the possibility
of zero-shot event extraction and hence the ability to extend to new domains
which is traditionally hard due to difficulties in collecting high-quality
labeled training data. However, MRC models struggle with long-range
dependencies and syntactic complexities. For instance, Liu et al. (2020)
observe that one typical error from their MRC-based extraction system is
related to long-range dependency between an argument and a trigger, accounting
for 23.4% errors on the ACE-2005 event dataset Doddington et al. (2004) (here
“long-range” denotes that the distance between a trigger and an argument is
greater than or equal to 10 words). Du and Cardie (2020) observe that one of
the failure modes of their extraction system is sentences with complex
sentence structures containing multiple clauses, each with trigger and
arguments. These observations make a promising case for complexity reduction
or context simplification for MRC systems.

To mitigate the above problem and to reduce the syntactic complexity we
propose an unsupervised approach that is guided by a scoring function that
incorporates syntactic fluency, simplicity and the confidence of an MRC
model(§ 2) Our key contributions are:

  1. 1.

The exploration of sentence and context simplification to help mitigate
thelong-range dependency problem for MRC based zero-shot event extraction (§
4).

  2. 2.

Experimental results on the ICEWS political event datasets including a
detailed analysis of areas of selective superiority of our approach.

  3. 3.

Followup analysis to demonstrate how simplification can be controlled based on
desired factors such as preference for high performance for a certain argument
role.

Figure 2: The RUSS sentence simplification approach.

##  2 Methodology

Reading comprehension models can be brittle to subtle changes in context. They
can be thrown-off by syntactic complexity, especially when the questions are
not specific and do not include words overlapping with the context. Moreover,
long range dependencies between the trigger/predicate and the argument can
also throw the model off. For this purpose, we propose an MRC-guided
Unsupervised Sentence Simplification algorithm (RUSS), that iteratively
performs deletions and extractions from the context in search for a higher-
scoring candidate. The score function incorporates components that ensure
sentence fluency, information preservation and the confidence of the target
MRC model. Fig. 2 gives an overview of the proposed approach.

Concretely, given that an event has been detected in a sentence, the task is
to identify the arguments of the detected event. For instance, in Fig. 1 the
task is to identify the arguments ‘Actor’ and ‘Target’ of the event ‘Bring
lawsuit against’. The algorithm takes the QA pairs corresponding to each
argument role as input. The QA generation procedure for the dataset used in
this paper for evaluation is outlined in Appendix B. Table 1 shows the
generated QA-pair for the arguments Actor and Target for the event shown in
Fig. 1.

Table 1: An example of a generated QA record for an event shown in Fig. 1 from the ICEWS dataset. The highlighted words are answers to the generated questions. Sentence |  A businessman detained for his links to disgraced army general Xu Caihou has been sued by his former employees.
---|---
Q-Actor |  Who sued someone?
Q-Target |  Who was sued by someone?

###  2.1 Sentence Simplification Algorithm

Given an input sentence $s$ and a list of questions $\\{q_{1},...,q_{n}\\}$
corresponding to different arguments, our algorithm iteratively performs two
operations on the sentence – deletion and extraction, in search for a higher-
scoring sentence and outputs a candidate simplification $c$. For generating
candidates, the algorithm first obtains the constituency parse tree of the
context using a span-based constituency parser (Joshi et al., 2018). It then
sequentially performs two operations on the parse tree – deletion and
extraction.

#### Deletion

In this operation, the algorithm sequentially drops subtrees from the parse
tree corresponding to different phrases. Note that the subtrees with the NP
(Noun-Phrase) label are omitted because it is expected that many entities that
form event arguments will be noun phrases and deleting them from the sentence
would result in significant information loss.

#### Extraction

This operation simply extracts a phrase, specifically corresponding to the the
S and SBAR labels as the candidate sentence. This allows us to select
different clauses in a sentence and remove remaining peripheral information.

These operations generate multiple candidates. Candidates with fewer than a
threshold of $t$ words are filtered out. We heuristically determine $t=5$.
From the remaining candidates, a highest-scoring candidate is chosen based on
the score function described in the next section(§ 2.2). The algorithm
terminates if the maximum score assigned to a candidate in the current
iteration does not exceed the previous maximum score. The simplification
algorithm RUSS is outlined as Algorithm 1 and the candidate generation
algorithm is outlined as Algorithm 2.

Input: $sentence:=s$, $questions=\\{q_{1},..q_{n}\\}$

Output: $simplification:=c$

Function RUSS($s$):

$maxIter\leftarrow M$

for  _$iter\in maxIter$_ do

$candidates\leftarrow\texttt{generateCandidates}(c)$

$scores\leftarrow\emptyset$

$maxScore\leftarrow 0$

for  _$cand\in candidates$_ do

$scores\leftarrow
scores\cup\nu_{lm}^{a}*\nu_{entity}^{b}*\nu_{pred}^{c}*\prod\nu_{rc_{role_{i}}^{r_{i}}}$

end for

$currMax\leftarrow max(scores)$

if  _$currMax >maxScore$_ then

$maxScore\leftarrow currMax$

$c\leftarrow candidates[argmax(scores)]$

end if

end for

return $c$

Algorithm 1 Sentence Simplification Algorithm – RUSS

Input: $sentence:=s$

Output: $candidates$

Function generateCandidates($s$):

$parseTree\leftarrow getParseTree(s)$

$toRemove\leftarrow\emptyset$

$extractions\leftarrow\emptyset$

$candidates\leftarrow\emptyset$

$phraseTags\leftarrow getValidPhraseTags()$

for  _$pos\in parseTree.positions$_ do

if  _$parseTree[pos]\in phraseTags$_ then

$toRemove\leftarrow toRemove\cup parseTree[pos].leaves$

end if

if  _$pos.label\in[S,SBAR]$_ then

$extractions\leftarrow extractions\cup parseTree[pos].leaves$

end if

end for

for  _$phrase\in toRemove$_ do

$candidate\leftarrow s.replace(phrase,\emptyset)$

if  _$candidate.length >t$_ then

$candidates\leftarrow candidates\cup candidate$

end if

end for

for  _$phrase\in extractions$_ do

if  _$phrase.length >t$_ then

$candidates\leftarrow candidates\cup candidate$

end if

end for

return $candidates$

Algorithm 2 Candidate Generation Algorithm

###  2.2 Scoring Function

We score a candidate as a product of different scores corresponding to
fluency, simplicity and its amenability to the downstream MRC model.

#### LM Score ($\nu_{lm})$

This score is designed to measure the language fluency and structural
simplicity of a candidate sentence. Instead of using LM-perplexity we use the
syntactic log-odds ratio (SLOR) (Pauls and Klein, 2012; Carroll et al., 1999)
score to measure the fluency. SLOR was also shown to be effective in
simplification to enhance text readability (Kann et al., 2018; Kumar et al.,
2020). Given a trained language model (LM) and a sentence $s$, SLOR is defined
as

$SLOR(s)=\frac{1}{|s|}(ln(P_{LM}(s))-ln(P_{U}(s))$ (1)

where $P_{LM}$ is the sentence probability given by the language model,
$P_{U}(s)=\prod_{w\in s}P(w)$ is the product of the unigram probability of a
word $w$ in the sentence, and $|s|$ is the sentence length. SLOR essentially
penalizes a plain LM’s probability by unigram likelihood and the length. It
ensures that the fluency score of a sentence is not penalized by the presence
of rare words. A probabilistic language model (LM) is often used as an
estimate of sentence fluency. In our work, instead of using a plain LM we use
a syntax-aware LM, i.e., in addition to words, we use part-of-speech (POS) and
dependency tags as inputs to the LM Zhao et al. (2018). For a word $w_{i}$ ,
the input to the syntax-aware LM is $[e(w_{i});p(w_{i});d(w_{i})]$, where
$e(w_{i})$ is the word embedding, $p(w_{i})$ is the POS tag embedding, and
$d(w_{i})$ is the dependency tag embedding. Note that our LM is trained on the
original train corpus. Thus, the syntax-aware LM helps to identify candidates
that are structurally ungrammatical.

#### Entity Score ($\nu_{entity})$

Entities help identify the key information of a sentence and therefore are
also useful in measuring meaning preservation. The desired argument roles are
also entities. Thus, if any entity detected in the original sentence is
omitted from a candidate the entity score for that candidate is 0, else it is
set to 1.

#### Predicate Score ($\nu_{pred})$

This score preserves the event predicates in a candidate. It checks if a
candidate contains any predicate of interest corresponding to the event
detected (Table 5). If it does not then $\nu_{pred}$ is set to 0, else it is
set to 1.

#### MRC Score ($\nu_{rc})$

Transformer-based MRC models can be brittle to subtle changes in context. To
make the context robust to the MRC model this score allows us to control the
complexity of context with respect to the confidence of the MRC model. It is
computed separately for each role. Each argument of an event is a span in the
context. $\nu_{rc_{role_{i}}^{r_{i}}}$ is the score of the best span in the
context for the argument role $i$, where the score of a candidate span is
defined as $S·T_{x}+E·T_{y}$ where $S\in R^{H}$ is a start vector and $E\in
R^{H}$ is an end vector as defined in Devlin et al. (2019b). $T_{x}$ and
$T_{y}$ are the final layer representations from the BERT model of the
$x^{th}$ and $y^{th}$ tokens in the context. Note that for a valid span,
$y>x$. This score is computed separately for each argument role (Actor and
Target in Example 1). The importance of the $i^{th}$ role can be controlled by
the exponent $r_{i}$. The total contribution of each role is computed as the
product of score corresponding to each role, given by
$\prod\nu^{r_{i}}_{rc_{role_{i}}}$.

The final score of a candidate $c$ is computed as follows:

$\nu(c)=\nu_{lm}(c)^{a}*\nu_{entity}^{b}(c)*\nu_{pred}^{c}(c)*\prod\nu^{r_{i}}_{rc_{role_{i}}}(c)$
(2)

Note that $b,c$ can be either $1$ or $0$ since $\nu_{entity}$ and $\nu_{pred}$
are binary. In later sections, we evaluate how the simplification can be
controlled by varying the constants $r_{i}$’s.

##  3 Datasets and Metrics

We evaluate RUSS on the ICEWS event
dataset333https://dataverse.harvard.edu/dataverse/icews from years 2013 to
2015. In this dataset, event data consists of coded interactions between
socio-political actors (i.e., cooperative or hostile actions between
individuals, groups, sectors and nation states) mapped to the CAMEO
444https://parusanalytics.com/eventdata/data.dir/cameo.html ontology. These
events are in the form of triples consisting of a source actor, an event type
(according to the CAMEO taxonomy of events), and a target actor. For
evaluation, we aim to extract the Actor and Target roles for event types shown
in Table 5. We use the data from years 2013-2015 for training and 2016 for
testing/evaluation in section 4.4. See Appendix for distribution of event
types.

###  3.1 Evaluation

We evaluate the performance of an MRC system before and after simplification
in the cross-domain zero-shot setting. In this setting, we emulate a no-
resource scenario, i.e. using the MRC system out-of-the-box in a target
domain. We do not finetune a pretrained MRC model with the generated QA
dataset. Rather, the aim is to assess the model performance in a zero-shot
setting, without using any training data from the target domain whatsoever. We
used the pretrained BERT model finetuned on the SQUAD 2.0 dataset (Rajpurkar
et al., 2016) and use the predictor API provided here
555https://docs.allennlp.org/models/v2.4.0/models/rc/predictors/transformer_qa/.
We further conduct follow-up analysis to study the controllability of
simplification by performing ablation analysis and assessing model performance
for different values of score component coefficients. For evaluation, we
extracted the best span(s) and computed an exact match F1 score Seo et al.
(2017) matching the span against the ground truth answer.

##  4 Results & Discussion

Table 2: Results of zero-shot event extraction on the ICEWS dataset. $\nu_{lm}$ coefficient $a=1.5$ and $\nu_{entity}$ coefficient $b=1$ for all settings in which simplification is performed. $\Delta$ $+ve$ indicates the % of records for which F1 improves after simplification, $\Delta$ $-ve$ indicates the % of records for which F1 becomes worse after simplification and $\Delta$ same indicates the % of records for which F1 remains unchanged. |  | Actor | Target
---|---|---|---
| Method | F1 |  $\Delta$ $+ve$ |  $\Delta$ $-ve$ |  $\Delta$ same | F1 |  $\Delta$ $+ve$ |  $\Delta$ $-ve$ |  $\Delta$ same
1 | No simplification | 0.412 | - | - | - | 0.354 | - | - | -
2 | $c=0,r_{1}=1,r_{2}=1$ | 0.431 |  10.99% | 6.54 % | 82.45% | 0.391 |  17.35% | 7.9% | 74.9%
3 | $c=1,r_{1}=0,r_{2}=0$ | 0.429 | 10.81% | 6.57 % | 82.61% | 0.390 | 16.54% | 7.53% | 75.93%
4 | $c=1,r_{1}=1,r_{2}=1$ | 0.424 | 10.5% | 6.3 % | 83.1% | 0.387 | 16.29% | 7.64% | 76.05 %
5 | $c=1,r_{1}=3,r_{2}=0$ | 0.435 | 9.72% |  5.67% |  84.6% | 0.391 | 16.89% | 7.97% | 75.12%
6 | $c=1,r_{1}=0,r_{2}=3$ | 0.427 | 10.54% | 6.95% | 82.5% | 0.391 | 16.12% |  7.29% |  76.59%

The results of zero-shot extraction on the ICEWS dataset are outlined in Table
2. In the baselines used, simplification is performed with score function
exponents for $\nu_{lm}$ as $a=1.5$ and $\nu_{entity}$ as $b=1$ held constant
while varying $c$ for $\nu_{pred}$, $r_{1}$ for $\nu_{actor}$ and $r_{2}$ for
$\nu_{target}$. With no simplification we get F1 scores of 0.412 and 0.354 for
actor and target roles respectively. For the most basic setting for
simplification with $c=0$, $r_{1}=1$ and $r_{2}=1$ scores improve by 4.6% for
actor prediction to 0.431 and by 10.4% to 0.391 for target prediction
respectively which shows that simplifying context can further improve a
powerful model like BERT in a cross-domain zero-shot setting. For actor
prediction, out of 37,894 records we find that for 10.99% records, F1 score
improves after simplification, for 6.54% records F1 decreased after
simplification and for the rest the score remained unchanged. For target
prediction, for 17.4% records scores improve where as for 7.9% records the
scores decreased and for the rest of the records, the scores remained
unchanged. After introducing the predicate score ($c=1$) we see that these
improvements drop slightly. This is counter-intuitive, because one would
expect model performance to improve when relevant predicates are present in
the context. We attribute this behavior to the MRC model leveraging the
language priors in the training data to predict the answers. For instance, the
model could predict the subject of the predicate as an answer for ‘Who’ type
of questions.

Next, we increase the coefficients of Actor and Target roles from 1 to 3. The
reason why we choose an odd number for this exponent is because sometimes for
bad candidates the RC scores can be negative and since all the scores are
combined in a multiplicative way, raising a negative score to an even power
would reverse the desired effect. Observing the results in rows 5 & 6 of Table
2 we can see that percentage of sentences with similar scores before and after
simplification have increased. We can also observe that percentage of
sentences for which scores decrease after simplification have also decreased.
We can conclude that by raising the coefficients of role specific scores we
can make the simplification models more robust to inaccurate simplifications
for those roles. We also observe, when $r_{1}=3$, we get the highest F1 for
actor prediction, an improvement of 5.6% over no simplification and for
$r_{2}=3$ we can an F1 on-par with the highest obtained in row 2. Our results
clearly indicate the benefit of simplification over no simplification and also
the gradual improvement in scores when the argument coefficients $r_{1},r_{2}$
are varied from 0 to 3.

###  4.1 Long Range Dependencies

Mean length of the original sentences is 32 words where as mean length of the
sentences after simplification is 22 words (row 2 setting). This indicates
that simplification doesn’t make sentences too short as is intuitive because
cutting relevant information would harm the performance.

We proceeded to investigate if simplification has addressed the long-range
dependency problem. We look at statistics concerning the distance between the
predicate and its arguments (Actor and Target) for the setting
$c=0,r_{1}=1,r_{2}=1$, that is, when the predicate score($\nu_{pred}$) is not
taken into account. As Table 2 indicates for $11\%$ of the records performance
increases after simplification. We find that for those records the average
distance between the predicate and its argument Actor is about 13 words and
the average distance between the predicate and target in the simplified
context is about 10 words. For the argument Target the average distance
between the predicate and target is about 8 words for original and about 6
words for the simplified context.

We see that RUSS cuts about 3 words for Actor prediction and 2 words for
Target prediction on average. We conclude that a certain percentage of
improvement comes from cutting down the distance between the predicates and
arguments hence mitigating the long-range dependency problem.

Moreover, we observe that actor prediction performance is better than that of
target prediction. This could be attributed to the fact that for sentences
that are in active construction, the subject of the predicate is a candidate
for the actor and in active constructions the distance between the predicate
and its subject will be small. However, for a complex sentence a clause or a
phrase can occur between the subject and the predicate. In this case, the
distance between them increases and hence we expect the performance to go
down. We quantitatively verify this as follows – the average length between
the actor and predicate in the candidate simplification for which the scores
improve is about 7 words and for which the scores decreases is about 8 words.
Hence, whenever the distance between the predicate and actor is large the
performance tends to become worse confirming our hypothesis.

###  4.2 Qualitative Analysis

Table 3 lists some cases in which simplification helps MRC system perform
better. In the first example, the proposed method deleted the word
‘personally’ from the original sentence (Sentence) to obtain the simplified
sentence (Simplified) as shown in the Table. The question posed to RC model
was “Who is being apologized to by someone” and the ground truth answer is
“the opposition”. For the original context the model extracts “Nawaz Sharif”
as the answer which is the wrong, whereas after removing the adverb
“personally”, it gets the correct answer. Note, that this decreases the
distance between the predicate apologized from its argument Nawaz Sharif. In
the second example, RC model extracts the closest noun phrase “Xu Caihou” as
answer which is incorrect. Simplification deletes the prepositional phrase “to
disgraced army general Xu Caihou” aiding the RC model in extracting the
correct answer. Note, that in this case it was especially important to delete
the above phrase due to the inherent ambiguity of construction. This case also
highlights the limitations of the current RC systems as the system was not
able to successfully associate employees with businessman and predicted the
noun-phrase closest to the predicate sued. In the third example, there was
segmentation error in the ICEWS dataset and two sentences were strung together
as seen in the Table. RUSS successfully deleted the unrelated sentence aiding
the RC system in extracting the correct answer.

Table 3: Qualitative examples of zero-shot performance of RC model before and after simplifying the context using the proposed algorithm. Underlined words are ground truth answers, emphasized words are predicates(triggers) and strikethrough indicates that words were removed by the algorithm. Question |  Who is being apologized to by someone?
---|---
Sentence |  Islamabad prime minister Nawaz Sharif personally apologized to the opposition today for what he called unfortunate comments made against PPP’s Aitzaz Ahsan
Answer |  Nawaz Sharif
Simplified |  Islamabad prime minister Nawaz Sharif personally apologized to the opposition today for what he called unfortunate comments made against PPP’s Aitzaz Ahsan
Answer |  the opposition
Question |  Who is being sued by someone?
Sentence |  Scmp a businessman detained for his links to disgraced army general Xu Caihou has been _sued_ by his former employees
Answer |  Xu Caihou
Simplified |  Scmp a businessman detained for his links to disgraced army general Xu Caihou has been _sued_ by his former employee
Answer |  businessman
Question |  Who is being accused of something?
Sentence |  Thus after having attacked the two elected to his party ump Brice Hortefeux and Claude Goasguen it was accused of pressure and insults. Rachida Dati has _accused_ Claude Goasguen to take to her because she had refused to sleep with him and this during an altercation proved by the Canard Enchan.
Answer |  Rachida Dati
Simplified |  Thus after having attacked the two elected to his party ump Brice Hortefeux and Claude Goasguen it was accused of pressure and insults. Rachida Dati has _accused_ Claude Goasguen to take to her because she had refused to sleep with him and this during an altercation proved by the Canard Enchan.
Answer |  Claude Goasguen

###  4.3 Error Analysis

From 6.54% records for which the score decreased after simplification (row 2
of Table 2) for 39.5% records from those the prediction from the original
context is a substring of the prediction from the simplified context. This
means that for some cases, both the original and the simplified context
facilitate the correct answer, rather it is the case that the answer from the
simplified context contains extra information for which it is penalized during
F1 score computation. For example consider the context “baghdad security
source said unknown gunmen assassinated an employee working in the secretariat
of baghdad near her home in ur district northeast of baghdad" which after
running the simplification algorithm is shortened to “in baghdad security
source said unknown gunmen assassinated an employee working in the secretariat
of baghdad near her home in ur district northeast of baghdad". (The
strikethrough text represents the text deleted by the proposed algorithm.) For
the question; “Who was assassinated by someone?" when presented with the
original context the RC model extracts “an employee" whereas after removing
the strikethrough text, RC model extracts “an employee working in the
secretariat". The ground truth answer for this is “employee". As can be seen
both answers are correct but the simplified contex is penalized for extra
words. Interestingly, such cases make up 48% of records for cases for which
performance improves after simplification. This is intuitive, because since
context becomes shorter and more precise after simplification and hence one
expects RC models to extract more precise answers. The fact that this happens
in 39.5% cases in the reverse scenario is surprising.

###  4.4 In-Domain Training

In sections 4.1- 4.3 we saw how RUSS improved zero-shot event extraction
performance in the cross-domain setting. In this section, we consider the
scenario when we have labeled in-domain training data available and we wish to
investigate if simplification can help improve performance when the MRC system
has been finetuned on in-domain data. We use the BERT-base-cased model Devlin
et al. (2019b) as our base model and finetune it on the ICEWS train dataset.
We finetune all layers as opposed to just the classification layer as we
observe large improvement in the former case as compared to the latter. We use
an initial learning rate of 3e-5 and use early stopping with $patience=5$ to
find the best model. For training we use the ICEWS dataset from years
2013-2015 and the year 2016 for testing. The QA generation procedure is
described in section B. There are total 75,788 (37,894$\times$2) examples for
training and 5,906 (2,953$\times$2) for test. BERT-RC in Table 4 indicates the
performance of the model on the original test set. We use the RUSS algorithm
to obtain simplifications of the test set. BERT-RC-Simple indicates the
performance of the model on this simplified test set. It can be observed that
simplification brings about an improvement(1.4%) even on a model that’s
finetuned on in-domain data.

Table 4: Table shows the performance of a BERT-base-uncased model finetuned on in-domain dataset. It can be seen that even after finetuning, RUSS approach improves model performance (BERT-RC-Simple). | Model
---
F1
BERT-RC | 0.776
BERT-RC-Simple | 0.787

##  5 Related Work

Event extraction(EE) has been an active area of research in the past decade.
In EE, supervised approaches usually rely on manually labeled training
datasets and handcrafted ontologies. Li et al. (2013) utilize the annotated
arguments and specific keyword triggers in text to develop an extractor.
Supervised approaches have also been studied using dependency parsing by
analyzing the event-argument relations and discourse of event interactions
(McClosky et al., 2011). These approaches are usually limited by the
availability of the fine-grained labeled data and required elaborately
designed features. Recent work formulates event argument extraction as an MRC
task. A major challenge with this approach is generating a dataset of QA
pairs. Liu et al. (2020) propose a method combining template based and
unsupervised machine translation for question generation. Du and Cardie (2020)
follow a template approach and show that more natural the constructed
questions better the event extraction performance. However, none of these
methods directly aim to address the long-range dependency problem using
simplification.

Automatic text simplification (ATS) systems aim to transform original texts
into their lexically and syntactically simpler variants. The motivation for
building the first ATS systems was to improve the performance of machine
translation systems and other text processing tasks, e.g. parsing, information
retrieval, and summarization Chandrasekar et al. (1996). In the context of
extraction, Zhang et. al. Zhang et al. (2018) show that pruning dependency
trees to remove irrelevant structures can improve relation extraction
performance. Efforts have been made to incorporate syntactic dependencies into
models in an effort to mitigate this problem 2016; 2018; 2020. Recently, Mehta
et al. (2020) have used sentence simplification as a preprocessing step for
improving machine translation. Edit-based simplification has been investigated
to a great degree to improve the readability of the text (Kumar et al., 2020;
Dong et al., 2019; Alva-Manchego et al., 2017). To the best of our knowledge
this is the first work that studies sentence simplification for improving MRC-
based event extraction.

##  6 Conclusion & Future Work

In this work, we motivated the need for MRC-based event extraction paradigm
especially for zero-shot scenarios(§ 1). Next, we discussed the long-range
dependency problem ubiquitously faced by event extraction systems. We proposed
a context simplification algorithm to reduce the syntactic complexity of the
context aided by MRC-system feedback to address the problem(§ 2). Our results
indicate that simplification can not only aid MRC systems in a zero-shot
setting(§ 4.1- 4.3) but also when they’re finetuned on in-domain data(§ 4.4).

Although, the proposed method can be useful to boost extraction performance
when offline computations can be afforded, it may be difficult to scale in
real-time use cases. Reasons include – 1) RC system inference time while
running the algorithm; 2) call latency if using APIs and 3) search algorithms
can be computationally expensive. A promising way to improve efficiency would
be to generate parallel training data for simplification using the RUSS method
offline and guide the MRC model during training using the generated data. This
can be done by using attention masks over the deleted words obtained using
RUSE. We leave this approach for future work.

Reproducibility: Appendix is provided as supplementary material.

## References

  * Alva-Manchego et al. (2017) Fernando Alva-Manchego, Joachim Bingel, Gustavo Paetzold, Carolina Scarton, and Lucia Specia. 2017.  Learning how to simplify from explicit labeling of complex-simplified text pairs.  In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 295–305, Taipei, Taiwan. Asian Federation of Natural Language Processing.
  * Boschee et al. (2013) Elizabeth Boschee, Premkumar Natarajan, and Ralph Weischedel. 2013.  _Automatic Extraction of Events from Open Source Text for Predictive Forecasting_, pages 51–67. Springer New York, New York, NY.
  * Carroll et al. (1999) John Carroll, Guido Minnen, Darren Pearce, Yvonne Canning, Siobhan Devlin, and John Tait. 1999.  Simplifying text for language-impaired readers.  In _Ninth Conference of the European Chapter of the Association for Computational Linguistics_ , pages 269–270, Bergen, Norway. Association for Computational Linguistics.
  * Chandrasekar et al. (1996) Raman Chandrasekar, Christine Doran, and Bangalore Srinivas. 1996.  Motivations and methods for text simplification.  In _Proceedings of the 16th conference on Computational linguistics-Volume 2_ , pages 1041–1044. Association for Computational Linguistics.
  * Chen et al. (2015) Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng, and Jun Zhao. 2015.  Event extraction via dynamic multi-pooling convolutional neural networks.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics_ , volume 1, pages 167–176.
  * Devlin et al. (2019a) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019a.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL-HLT_.
  * Devlin et al. (2019b) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019b.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Doddington et al. (2004) George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004.  The automatic content extraction (ACE) program – tasks, data, and evaluation.  In _LREC’04_ , Lisbon, Portugal. European Language Resources Association (ELRA).
  * Dong et al. (2019) Yue Dong, Zichao Li, Mehdi Rezagholizadeh, and Jackie Chi Kit Cheung. 2019.  EditNTS: An neural programmer-interpreter model for sentence simplification through explicit editing.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3393–3402, Florence, Italy. Association for Computational Linguistics.
  * Du and Cardie (2020) Xinya Du and Claire Cardie. 2020.  Event extraction by answering (almost) natural questions.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 671–683, Online. Association for Computational Linguistics.
  * Huang et al. (2018) Lifu Huang, Heng Ji, Kyunghyun Cho, Ido Dagan, Sebastian Riedel, and Clare Voss. 2018.  Zero-shot transfer learning for event extraction.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2160–2170, Melbourne, Australia. Association for Computational Linguistics.
  * Joshi et al. (2018) Vidur Joshi, Matthew Peters, and Mark Hopkins. 2018.  Extending a parser to distant domains using a few dozen partially annotated examples.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1190–1199, Melbourne, Australia. Association for Computational Linguistics.
  * Kann et al. (2018) Katharina Kann, Sascha Rothe, and Katja Filippova. 2018.  Sentence-level fluency evaluation: References help, but can be spared! In _Proceedings of the 22nd Conference on Computational Natural Language Learning_ , pages 313–323, Brussels, Belgium. Association for Computational Linguistics.
  * Kumar et al. (2020) Dhruv Kumar, Lili Mou, Lukasz Golab, and Olga Vechtomova. 2020.  Iterative edit-based unsupervised sentence simplification.  _arXiv preprint arXiv:2006.09639_.
  * Li et al. (2013) Qi Li, Heng Ji, and Liang Huang. 2013.  Joint event extraction via structured prediction with global features.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , volume 1, pages 73–82.
  * Liu et al. (2020) Jian Liu, Yubo Chen, Kang Liu, Wei Bi, and Xiaojiang Liu. 2020.  Event extraction as machine reading comprehension.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1641–1651, Online. Association for Computational Linguistics.
  * Liu et al. (2018) Xiao Liu, Zhunchen Luo, and Heyan Huang. 2018.  Jointly multiple events extraction via attention-based graph information aggregation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1247–1256, Brussels, Belgium. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, et al. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Ma et al. (2020) Jie Ma, Shuai Wang, Rishita Anubhai, Miguel Ballesteros, and Yaser Al-Onaizan. 2020\.  Resource-enhanced neural model for event argument extraction.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3554–3559, Online. Association for Computational Linguistics.
  * McClosky et al. (2011) David McClosky, Mihai Surdeanu, and Christopher D Manning. 2011.  Event extraction as dependency parsing.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics_ , pages 1626–1635.
  * Mehta et al. (2020) Sneha Mehta, Bahareh Azarnoush, Boris Chen, Avneesh Saluja, Vinith Misra, Ballav Bihani, and Ritwik Kumar. 2020.  Simplify-then-translate: Automatic preprocessing for black-box machine translation.
  * Nguyen et al. (2016) Thien Huu Nguyen, Kyunghyun Cho, and Ralph Grishman. 2016.  Joint event extraction via recurrent neural networks.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 300–309, San Diego, California. Association for Computational Linguistics.
  * Pauls and Klein (2012) Adam Pauls and Dan Klein. 2012.  Large-scale syntactic language modeling with treelets.  In _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 959–968, Jeju Island, Korea. Association for Computational Linguistics.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Ramakrishnan et al. (2014) Naren Ramakrishnan, Patrick Butler, Sathappan Muthiah, Nathan Self, Rupinder Khandpur, Parang Saraf, Wei Wang, Jose Cadena, Anil Vullikanti, Gizem Korkmaz, Chris Kuhlman, Achla Marathe, Liang Zhao, Ting Hua, Feng Chen, Chang Tien Lu, Bert Huang, Aravind Srinivasan, Khoa Trinh, Lise Getoor, Graham Katz, Andy Doyle, Chris Ackermann, Ilya Zavorin, Jim Ford, Kristen Summers, Youssef Fayed, Jaime Arredondo, Dipak Gupta, and David Mares. 2014.  ’beating the news’ with embers: Forecasting civil unrest using open source indicators.  In _Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’14, page 1799–1808, New York, NY, USA. Association for Computing Machinery.
  * Saraf and Ramakrishnan (2016) Parang Saraf and Naren Ramakrishnan. 2016.  Embers autogsr: Automated coding of civil unrest events.  In _Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , pages 599–608.
  * Seo et al. (2017) Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017.  Bidirectional attention flow for machine comprehension.  _ArXiv_ , abs/1611.01603.
  * Sha et al. (2016) Lei Sha, Jing Liu, Chin-Yew Lin, Sujian Li, Baobao Chang, and Zhifang Sui. 2016\.  RBPB: Regularization-based pattern balancing method for event extraction.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1224–1234, Berlin, Germany. Association for Computational Linguistics.
  * Wadden et al. (2019) David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. 2019.  Entity, relation, and event extraction with contextualized span representations.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5784–5789, Hong Kong, China. Association for Computational Linguistics.
  * Zhang et al. (2018) Yuhao Zhang, Peng Qi, and Christopher D. Manning. 2018.  Graph convolution over pruned dependency trees improves relation extraction.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2205–2215, Brussels, Belgium. Association for Computational Linguistics.
  * Zhao et al. (2018) Yang Zhao, Zhiyuan Luo, and Akiko Aizawa. 2018.  A language model based evaluator for sentence compression.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 170–175.

##  Appendix A Appendix

For training the RUSS algorithm we used the TransformerQA model made available
through the allennlp library predictors API
666https://github.com/allenai/allennlp-
models/blob/main/allennlp_models/rc/models/transformer_qa.py. Running the
algorithm takes 5 hours on 1 CPU core and 1 GPU. However when parallelizing
the computation across 5 cores that time can be brought down to 1 hour.

##  Appendix B QA Dataset Generation

For the ICEWS dataset, we create one question template per predicate per event
type for each slot. For each event type we identified a list of most common
predicates (triggers) for that event type since trigger labels are not
available in the ICEWS dataset. For example, for ‘Demonstrate or rally’ event
type the predicates identified are ‘condemn’, ‘protest’, ‘demonstrate’ and for
‘Accuse’ event type the predicates are ‘blame’, ‘blaming’, ‘accused’,
‘alleged’, ‘accusing’. Table 5 enumerates all of the ICEWS event types used
and their identified predicates. For each of the predicates identified for
each event type we use one question template for each of the two argument
roles Actor and Target. For the Actor role, the template used an active
construction ‘Who $predicate$ someone? and for the same event for the Target
role the templated used a passive construction – ‘Who was $predicate$ by
someone?’. This results in 37,894 records with a sentence and two questions
one each for the Actor and Target roles respectively. The list of entities for
the ICEWS dataset comes from the ICEWS actors and agents
dictionaries777https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/28118.

Table 5: Table lists the ICEWS event types used and their corresponding predicates that were identified for generating question templates. Event Type |  Predicates
---|---
Abduct, hijack, or take hostage |  kidnapped, abducting, abducted, captured
Accuse |  blame, blaming, accused, alleged, accusing
Apologize |  apologize, apology
Assassinate |  carried out assassination of, assassinate
Bring lawsuit against |  is suing someone, sued, has sued, filed a suit against
Demonstrate or rally |  condemn, protest, demonstrate
Arrest, detain, or charge with legal action |  arrested, sentenced, detained, nabbed, captured, arresting, capture, jailed, routinely arrested, prosecuted, convicted
Use conventional military force |  killed, shelled, combating, shells, strikes, strike, kill

##  Appendix C Dataset Statistics

Table 6 outlines the distribution of different event types used in the ICEWS
dataset used.

Table 6: Table shows the distribution of event types in the ICEWS Train and Test datasets used. Event Type | #Records Train | #Records Test
---|---|---
Abduct, hijack, or take hostage | 3473 | 193
Accuse | 8856 | 651
Apologize | 181 | 11
Arrest, detain, or charge with legal action | 9933 | 782
Assassinate | 146 | 12
Bring lawsuit against | 206 | 18
Demonstrate or rally | 2890 | 175
Use conventional military force | 12209 | 1111