# Semi-supervised and Deep learning Frameworks for Video Classification and
Key-frame Identification

Sohini Roychowdhury,
Adjunct Faculty, Computer Engineering
Santa Clara University, email<EMAIL_ADDRESS>
###### Abstract

Automating video-based data and machine learning pipelines poses several
challenges including metadata generation for efficient storage and retrieval
and isolation of key-frames for scene understanding tasks. In this work, we
present two semi-supervised approaches that automate this process of manual
frame sifting in video streams by automatically classifying scenes for content
and filtering frames for fine-tuning scene understanding tasks. The first
rule-based method starts from a pre-trained object detector and it assigns
scene type, uncertainty and lighting categories to each frame based on
probability distributions of foreground objects. Next, frames with the highest
uncertainty and structural dissimilarity are isolated as key-frames. The
second method relies on the simCLR model for frame encoding followed by label-
spreading from 20% of frame samples to label the remaining frames for scene
and lighting categories. Also, clustering the video frames in the encoded
feature space further isolates key-frames at cluster boundaries. The proposed
methods achieve 64-93% accuracy for automated scene categorization for outdoor
image videos from public domain datasets of JAAD and KITTI. Also, less than
10% of all input frames can be filtered as key-frames that can then be sent
for annotation and fine tuning of machine vision algorithms. Thus, the
proposed framework can be scaled to additional video data streams for
automated training of perception-driven systems with minimal training images.

###### Index Terms:

semi-supervised, key frame, classification, label spreading, active learning

##  I Introduction

Computer Vision and scene understanding applications rely on processing large
volumes of video data to learn the patterns related to the objects/regions of
interest (ROIs) [1]. Therefore, automated video sequence processing systems
for efficient frame tagging, storage and retrieval become key components for
building such automated machine vision systems. Also, there is a constant need
to isolate key-frames with high structural variance and good quality from
video sequences that can then be used to train machine learning models [2]. In
this work we present two methods that rely on minimal annotated data to
automatically classify and tag individual frames from video sequences based on
the scene type and lighting conditions. Also, both methods can filter key-
frames with varying processing speed and reliability across datasets to
identify minimal training datasets for subsequent training of machine vision
algorithms. The proposed end-to-end system is shown in Fig. 1.

Figure 1: Proposed framework for automated scene classification per frame and
for key-frame filtering. Raw video sequences are supplied to either one of the
two proposed semi-supervised algorithms such that each video frame gets tagged
for lighting conditions (day, night, poor lighting, shadows, low standing sun
etc.) and content (city, freeway, pedestrians, parked-cars). Also frame IDs
corresponding to highly uncertain frames are filtered as key-frames.

Till date, several image-specific metrics have been applied for key-frame
selection based on image quality such as: peak signal to noise ratio (PSNR),
contrast to noise ratio (CNR, that is sensitive to compression formats), SIFT
features [3] and mean squared error. However, these image metric-based methods
can only eliminate poor quality image frames with blurriness and poor light.
Besides, deep learning-based feature extraction methods in [1] [4] have been
applied to narrow field-of-view image sequences for encoding and video content
summarizing tasks. These deep learning methods have shown to encode actions
from video sequences only after learning from large volumes of annotated data,
which is a costly process altogether. Since annotation costs for video-based
applications are highly dependent on training data volumes, there is a need to
streamline the process of key-frame identification, and to curate sequences
with appropriate metadata. Additionally, the recent work in [5] used self-
supervised deep-learning representations to isolate key frames from targeted
human action-specific tasks only. Motivated by these existing works, we
implement a novel two-stage end-to-end process to first automatically classify
each frame from video sequences for frame content, uncertainty and lighting
conditions for multiple targets across wide-fields of view videos. Next, key-
frames are selected as ones with the highest degree of structural uncertainty,
dissimilarity and high image quality.

This paper makes two major contributions. First, we present two methods that
require less than 20% of available video frames to be annotated for automated
classification and tagging of outdoor scenes and lighting conditions per
frame. The rule-based method has minimal parameters and is more robust to
training data variations when compared to the novel deep-learning based
method. However, the rule-based method has ten-times more computation time
complexity and captures less variety in structural content such as snow
accumulations and weather-related variations for key-frames when compared to
the deep learning-based method. Second, we present novel key-frame filtering
methods that isolate less than 10% of all frames from video sequences that
represent structural variety and high quality for improved scene understanding
tasks.

##  II Data and Methods

The details of the two proposed methods for per-frame scene classification and
key-frame filtering are presented below. The first rule-based baseline method
utilizes sequential image content information to identify the uncertain key-
frames. The second novel deep-learning representation based method requires a
critical mass of annotated frames to automatically perform scene
classification based on clustering trends per frame. We assess both methods
for speed, accuracy of per-frame classification and generalizability across
outdoor datasets.

###  II-A Data

We implement scene classification and key-frame filtering by combining
individual video sequences from public datasets of JAAD [6] and KITTI [7]. The
manual annotations per sequence will be publicly released for benchmarking
purposes. The goal is to identify scene changes and to isolate frames with
previously unseen content that can further enhance scene understanding tasks.
For example, object detection tasks in night-time sequences are specifically
difficult and standard object detection algorithms trained on day-time images
do not scale well in such situations. Identifying key-frames where objects
such as pedestrians, bicyclists etc. first appear into a scene, or frames
where key objects are missed due to poor lighting is crucial to further fine-
tuning such object detectors. Other scene understanding tasks that can be
further fine tuned by key-frames can be object trajectory prediction,
pedestrian intention prediction and landmark detection tasks as shown in [2].
Each video sequence is manually annotated for the following 3 parameters:

  * •

Time of Day: {night(1), day(0)}

  * •

Lighting conditions: {poor(1), good (0)}

  * •

Scene Class: {city (0), pedestrians (1), freeway (2), parked-cars (3)}

The first dataset is the sub-sampled sequences from the JAAD data set [6] with
54 video sequences of outdoor scenes with sequence names [0022-0024,
00273-0324]. These video sequences are selected since they represent varying
lighting and weather conditions, such that each video sequence contains an
average of 237 frames per sequence. The JAAD dataset is acquired at 30 frames-
per-second and each image has dimensions [1080 x 1920] pixels per frame.
Combining all video frames together results in 6101 total frames with
annotated pedestrians only. Each JAAD scene is manually annotated to belong to
one of the five following categories: [{day, good light, city}, {day, good
light, pedestrians}, {day, poor light, pedestrians}, {day, poor light, city},
{night, good light, city}] with representations [10, 7.6, 2.16 , 77.93, 4.92,
7.38]% for each category/class, respectively. We observe a major scene
category corresponding to daytime city scenes with shadows and poor lighting.
To isolate a minimal training dataset with similar distributions as test data,
we select all frames from the video sequences [0022, 0023, 0273, 0286, 0290,
0292, 0295, 0310, 0303, 0307, 0312] as training data and all remaining frames
become test data. Examples of scene categories are shown in Fig. 2.

(a) Image with {night, good light, city} from sequence 0024, JAAD

(b) Image with {day, good light, city} from sequence 0283, JAAD

(c) Image with {day, poor light, city} from sequence 0280, JAAD

(d) Image with {day, poor light, city} from sequence 0279, JAAD

(e) Image with {day, poor light, pedestrians}, sequence 0292, JAAD

(f) Image with {day, good light, pedestrians}, sequence 0295, JAAD

(g) Image with {day, poor light, freeway}, sequence 0005, KITTI

(h) Image with {day, good light, freeway} from sequence 0008, KITTI

Figure 2: Examples of Scene categories from the JAAD and KITTI datasets.

The second dataset is the KITTI tracking dataset for 2D objects [7] that has a
variety of video-sequences of outdoor scenes of varying lengths. These video
sequences are acquired at a frame-rate of 10 frames-per-second and the images
are cropped frames with [1382 x 512] pixels per frame. Here, we use the 21
video sequences with names [0000-0020] from the training folder with 8008
total frames. Each video sequence belongs to one or more of the following five
categories: [{day, good light, freeway}, {day, poor light, parked-cars}, {day,
poor light, city}, {day, poor light, freeway}, {day, poor light, pedestrians}]
with representations [8.24, 35.35, 32, 20.2, 4.42]% per category,
respectively. Here, the category parked-cars refers to videos with several
parked cars, either in parking lots or street parking. The minimal training
dataset with similar distribution of scenes as test data, is selected as all
frames from the sequences [0000, 0001, 0005, 0006, 0017]. All frames from the
remaining sequences are combined to create the test video sequence.

###  II-B Frame Selection and Quality Detection Modules

Scene understanding tasks typically require learning from a variety of
training samples. Thus, selection of frames that are significantly dissimilar
in structural content and high in quality is necessary. One key building block
for our scene categorization models is a frame structural selector and quality
selector module that eliminates consecutive frames from a sequence that are
structurally similar to others and that may be blurry.

In the algorithm 1, we implement such a frame selection module which starts
iterating frames ($j$) from the first frame ($I_{0}$) as a reference frame and
compares subsequent frames to the reference frame based on the structural
similarity index metric (SSIM, $s$) [3]. As soon as a frame with
$SSIM<\alpha$, where $\alpha\in[0,1]$ is encountered, this new frame becomes
the new reference frame ($I_{ref}$), gets added to selected frame set
($F_{S}$) and the process continues till the last frame ($I_{l}$). Since SSIM
within a frame neighborhood are compared, the computational complexity remains
relatively linear.

Output: Selected Frames $F_{S}=\\{I_{k}\\}$

Input: Video frames:$\\{I_{l}\\}$, $F_{S}=\\{\phi\\}$

Initialization;

$I_{ref}\leftarrow I_{0}$, step$\leftarrow$ 2, s$\leftarrow$1, j$\leftarrow$0,
$F_{S}\leftarrow\\{I_{0}\\}$

while  _$j <l$_ do

while  _s $>\alpha$_ do

j$\leftarrow$ j+step

s$\leftarrow ssim(I_{ref},I_{j})$

end while

$I_{ref}\leftarrow I_{j}$

$F_{S}\leftarrow F_{S}\cup I_{ref}$

end while

Algorithm 1 Frame Structural Selection

The next task is to further filter the structurally dissimilar frames based on
image quality to ensure that blurry images are discarded. For this purpose we
implement the OpenCV Laplacian of variance function based on existing works in
[8]. Here, the 2D Laplacian kernel highlights foreground edges, thereby
yielding a higher score for pixel variances in well-focused and sharp images.
We apply the Laplacian function to grayscale images converted from the video
sequences and retain only high quality images with high coefficients. This
structural and quality-based frame selection module is used for isolating
samples for training the self-supervised deep learning model and for further
filtering key-frames from the rule-based method. This module is capable of
reducing the number of retained frames from 40-90% of the initial volume by
varying the threshold $\alpha$ between [0.4-0.8] and normalized Laplacian of
variance in range [0.4-0.7], respectively.

###  II-C Rule-based Frame Classification: Baseline

As a baseline frame classification model with minimal training data
requirements, we implement a rule-based method that relies on pre-trained
structural detectors to estimate reliable content and relative unreliability
or noise per frame. This baseline model that requires minimal training data is
shown in Fig. 3. First, a pre-trained general purpose object detector or
semantic segmentation model is used to extract the reliable foreground objects
of interest per image. For outdoor images, this can be the number of
pedestrians, bicyclists, vehicles, trains etc. Next, based on the number of
foreground objects and the distribution in the probability of objectness per
frame, each image can be classified for lighting and scene-related metadata.
The unreliability/uncertainty of object bounding boxes per frame can then be
utilized to screen for key-frames.

Figure 3: The proposed rule-based model to classify video frames starts with a
pre-trained general purpose object detector. Next, the number of foreground
objects and probability distributions of objects is used to classify each
frame. The most uncertain frames are selected using a peak detector as key-
frames.

Here, we implement a Faster-RCNN model [9] pre-trained on the MS-COCO dataset
to extract foreground object bounding boxes and their objectness probabilities
($r$) for all objects with probabilities greater than 0.2. We retain only the
relevant foreground objects in terms of the number of pedestrians and
bicyclists ($P$), vehicles such as cars, trucks, buses, airplanes and
motorcycles ($V$) and background lighting i.e. night vs. day ($B$). Next, the
uncertainty in each frame is estimated in terms of the coefficient of
variation ($U=\frac{\sigma_{r}}{r_{med}}$), where, $\sigma_{r}$ represents the
standard deviation in all relevant foreground object detection probabilities
and $r_{med}$ represents the median of the foreground object probabilities. A
higher $U$ would imply high variations in foreground object probabilities and
therefore captures uncertainties in frames corresponding to poor lighting,
shadows, low standing sun etc. Other known methods to estimate uncertainty per
frame can be estimating intersection over union for multiple detections with
dropout applied at test time. Examples of variations in content ($P,V,B,U$)
for varying scenes are shown in Fig. 4.

Figure 4: Examples of images with persons (P, blue), vehicles (V, red),
uncertainty (U, red dots) and background light (B, black) for frames from JAAD
datasets sequences 0023, 0280, 0289, and 0290, respectively. City images show
non zero combination of $\\{P,V\\}$, and images shown here with poor lighting
conditions have high uncertainty ($U>0.5$).

Once the content of each frame is estimated, the impact of neighboring frames
in sequence is computed by averaging the uncertainty for 10 frames within a
sliding window. So, for each frame, the mean of 5 previous and 5 subsequent
frame uncertainties is computed as the mean uncertainty ($\bar{U}$). Finally,
the following frame classifications are made based on the content.

  * •

{$P>V,V<3,B=1$}, class $\rightarrow$ {pedestrians, good light, night}

  * •

{$P>V,V<3,B=0,\bar{U}<=\beta$}, class $\rightarrow$ {pedestrians, good light,
day}

  * •

{$P>V,V<3,B=0,\bar{U}>\beta$}, class $\rightarrow$ {pedestrians, poor light,
day}

  * •

{$P=0,V>0,B=1$}, class $\rightarrow$ {freeway, good light, night}

  * •

{$P=0,V>0,B=0,U>\delta,\bar{U}<=\beta$}, class $\rightarrow$ {freeway, good
light, day}

  * •

{$P=0,V>0,B=0,U>\delta,\bar{U}>\beta$}, class $\rightarrow$ {freeway, poor
light, day}

  * •

{$P=0,V>0,B=0,U<\delta,\bar{U}<\beta$}, class $\rightarrow$ {parked cars, poor
light, day}

  * •

{$P>=0,V>=0,B=1$}, class $\rightarrow$ {city, good light, night}

  * •

{$P>=0,V>=0,B=0,\bar{U}<=\beta$}, class $\rightarrow$ {city, good light, day}

  * •

{$P>0,V>=0,B=0,\bar{U}>\beta$}, class $\rightarrow$ {city, poor light, day}

For any other combination of $P,V,B,\bar{U}$, the Other category is assigned,
signifying an unknown combination that can be due to indoor parking areas,
tunnels etc. We implement the [1024x1024] pixel Faster-RCNN model from
Tensorflow hub and observe that the test time per frame ranges between
[0.15-0.17] seconds. The parameters $\beta$ and $\delta$ are estimated using
5-fold cross validation to identify the parameter in range [0,1] that
maximizes weighted accuracy of per-frame classification.

Finally, key frames are identified as frames that cause peaks in relative
objectness uncertainties. Here, a peak detector is applied to the $U$ metric
across frames, to isolate frames that have sudden variations in object
probabilities with respect to its neighborhood frames and that have
uncertainty greater than $\eta$. Here, $\eta$ can be varied in range [0.5-1]
to control for the number of filtered key frames with high uncertainty. Once
the key-frames are identified, they are further subjected to the frame and
quality selector module to ensure minimal structural similarity and high
quality for the finally filtered key-frames.

###  II-D Label Propagation with simCLR Representations: Semi-supervised

As an active learning framework, we propose a semi-supervised method to
cluster scene frames based on their deep learning representations. The goal is
to assign labels to previously unlabelled frames and to identify
samples/frames that lie around cluster boundaries, which are difficult to
assign cluster labels, as key-frames. The proposed framework is shown in Fig.
5.

Figure 5: The proposed simCLR-based scene classification and key-frame
filtering method starts with a subset of selected training frames for fine-
tuning a self-supervised deep learning encoder model. Once the best
representation feature space is identified, a semi-supervised approach of
label spreading is used to label all test frames. Frames that lie on decision
boundaries shown in the red circles are filtered as key-frames.

We begin by implementing a self-supervised simCLR [10] model with a ResNet
backbone that identifies cropped and modified versions of an image from other
images. The simCLR model uses augmented versions of each frame for learning
weights as opposed to frame-labels, thereby reducing its dependence on frame
annotation. The images needed to train the simCLR model should ideally
represent significant structural variations. For this reason, we apply the
frame structural selector module from Section II-B on the combined training
video sequences from each dataset and extract a training subset to train the
simCLR model. At the end of training, all frames from the training sequences
get encoded into their deep learning representations. The representations from
the last 3 layers prior to classification, corresponding to 128, 256 and 2048
neurons, respectively, are analyzed for the best high-dimensional
representation as follows.

All frames from the training video sequences are encoded in terms of the 128,
256 and 2048 neurons, followed by high-dimensional embedding using PCA, t-SNE
and UMAP. Next, the labels from the training subset are propagated (by label
spreading) through a best fit rbf-kernel to label the remaining training
sequence frames. Empirically, we observe that the third from last deep
learning layer with 2048 neurons subjected to PCA with 10 dimensions results
in the best high-dimensional encoding with highest classification weighted
per-frame accuracy on validation data. This process enables encoding each
video frame $I$ to an encoded sample $\mathcal{F}(I)$. Next, using encoded
training frames ($\mathcal{F}(I_{j})$) with the annotated scene class
($Y_{j}$), we incorporate label-spreading to assign a scene class to each test
frame ($\mathcal{F}(I_{k})$) using equations (1)-(3) based on [11].

$\displaystyle\bf{w_{j,k}}=e^{-\gamma||\mathcal{F}(I_{j})-\mathcal{F}(I_{k})||^{2}},$
(1) $\displaystyle S=D^{-\frac{1}{2}}{\bf W}D^{-\frac{1}{2}},$ (2)
$\displaystyle\text{Given, }Y(0)=[y_{j},[-1]^{[k]}],$ (3) $\displaystyle
Y(t+1)=\nu SY(t)+(1-\nu)Y(0).$

In (1), an rbf-kernel-based distance matrix $\bf{W}$ is computed for each
combination of train ($j$) and test images ($k$). Next, a symmetric normalized
Laplacian matrix $S$ is estimated using diagonal matrix $D$, where the $i$th
diagonal entry is the sum of the $i$th row of $\bf{W}$ in (2). The initial
label vector for the training and test data combined is represented by $Y(0)$
with $j+k$ entries that contains the training labels (with $j$ entries) padded
with -1 placeholder labels for the $k$ test samples. Next, the iterative label
spreading method continues using (3) till convergence is reached or maximum 30
time steps (t) have been completed. In each new iteration, the label matrix
$Y(t+1)$ is updated based on the neighboring training samples. A higher value
of $\nu$ in the range [0,1] indicates a higher update in each iteration versus
the initial label set $Y(0)$. It is noteworthy that label-spreading using the
rbf-kernel has high time complexity for large numbers of training and test
samples. However, this process can be applied efficiently for small data
batches (up to 10,000 images per train/test batch size). Also, we observe that
the simCLR-based frame labeling approach, though more training data dependent
when compared to the rule-based method, is significantly faster since
classifying all test sequences takes upto 4 minutes on a Windows system with
8GB RAM, 8GB Nvidia 2070 GPU. Additionally, the k-nearest neighbor method can
be substituted in (1-2) to reduce the computational time complexity at a cost
of up to 5% reduction in classification accuracies for the data sets here.

Finally, to extract key-frames from test sequences, we implement rbf-label-
spreading with varying kernel variances $\gamma$ and learning parameters $\nu$
for 20 runs. We detect the frames/samples that are assigned more than one
unique label across the 20 runs. These samples can be considered to lie around
cluster boundaries since they get labeled differently across the runs. These
cluster boundary samples can then be filtered as key-frames and the volume of
filtered frames can be varied by controlling the $\gamma$ and $\nu$
parameters.

##  III Experiments and Results

We evaluate the performances of the rule-based baseline model and the novel
simCLR-based model for test scene classification per frame, for dependence on
training data and for key-frame filtering through three experiments. First, we
train both models using training frames from JAAD and KITTI data set and test
the classification performances per test frame for video sequences from the
same dataset. Second, we vary the training datasets for both methods and
analyze robustness in the per-frame scene classification performance. Third,
we analyze the number of key-frames filtered from each dataset by both methods
to assess their agreeability and to qualitatively explain the contents of the
filtered frames. The scene classification performances are evaluated in terms
of averaged macro and weighted precision, recall, f-score and accuracy across
all the classes.

###  III-A Per-frame Scene Classification

Scene classification performances per frame for the JAAD and KITTI data sets
are given in Table I.

TABLE I: Performance of scene classification using the proposed methods.

Method Accuracy Macro Precision Macro Recall Macro f-score Weighted Precision
Weighted Recall Weighted f-score Dataset Train: JAAD Test: JAAD Rule-based
88.5 43 46.8 44.2 89.3 88.5 88.8 simCLR-based 93.25 37.67 39.88 38.71 87.5
93.25 90.3 Dataset Train: KITTI Test: KITTI Rule-based 72.2 52.6 47.3 46.6
69.5 72.2 69.1 simCLR-based 66.4 45.0 54.8 48.8 52.3 66.4 57.7

Here, we observe that for the JAAD dataset, if all the scene categories were
equally likely, the macro classification performances of the rule-based method
would be better than the simCLR-based method. However, considering consistent
scene category distributions from the train to test scenes, the simCLR-based
method has superior weighted accuracy for test frame classification. It is
noteworthy that for the JAAD dataset, there exists a class imbalance with a
large number of frames belonging to {day, poor light, city} category. Thus,
over prediction for this class results in higher weighted classification
performance.

On the other hand, for the KITTI dataset, the scene category distribution is
relatively uniform. The video sequences in this dataset are longer than JAAD
and some sequences contain parked cars in residential areas followed by
merging to a freeway in the same sequence. Also, there are sequences that
contain parked cars, and city scenes together. Therefore, manual scene
classification per sequence may not align with automated per-frame
classifications. For the KITTI dataset, in Table I, we observe that the rule-
based method has better weighted classification while the simCLR method has
better macro classification performances. This occurs because the rule-based
method relies on a task specific object detector for foreground and
uncertainty detection and is therefore able to classify city scenes from
parked cars and pedestrians more efficiently. The simCLR-based method however
results in significant mis-classifications between parked-cars, city and
freeway scenes since there are significant structural similarities between the
frames of all these classes. Also, from Table I we observe that for the KITTI
dataset, the longest test video sequence 0020 corresponding to a freeway with
837 frames is misclassified as city due to the foliage along the freeway. This
misclassification significantly reduces the overall classification accuracy
for the KITTI dataset when compared to JAAD.

###  III-B Robustness Analysis to Training Data

In this experiment we analyze the generalizability of the two scene
classification methods to new use-cases. Here, we vary the training dataset
and analyze the scene classification performances per frame. It is noteworthy
that the rule-based method is intuitively more generalizable for outdoor scene
classification since it relies on a pre-trained foreground object detector and
requires minimal parameterization for $\gamma$ and $\delta$ only. Next, we
train the semi-supervised simCLR-based label-spreading model with labeled
frames from the other dataset. For this analysis, the combined training and
test datasets from JAAD and KITTI comprise 8 unique scene categories and only
2 overlapping categories corresponding to {day, poor light, city} and {day,
poor light, pedestrians}. Thus, lack of actual scene labels in the training
samples significantly reduces classification performances for the simCLR based
model.

In Table II we observe that using the rbf-kernel based simCLR-based model and
training on KITTI but testing on JAAD, all the frames get classified as the
same major common class between both datasets {day, poor light, city}. Thus,
the macro classification performances are low, however the weighted
performances are higher owing to the class imbalance. Similarly for the same
method training on JAAD while testing on KITTI, the test frames get
categorized into the 2 overlapping categories representing lighting variations
for city and pedestrians in daytime with shadows. Thus, the rule-based method
however, shows consistent generalizability across varying training datasets.

TABLE II: Performance of scene classification by varying training datasets.

Method Accuracy Macro Precision Macro Recall Macro f-score Weighted Precision
Weighted Recall Weighted f-score Dataset Train: KITTI Test: JAAD Rule-based
69.2 45.7 48.7 43.2 92.6 69.2 78.6 simCLR-based 84.4 16.9 20 18.3 71.2 84.4
77.3 Dataset Train: JAAD Test: KITTI Rule-based 64.0 58.6 42.8 40.7 65.0 64.0
61.3 simCLR-based 36.0 22.5 33.3 25.3 14.7 36.0 20.2

This analysis demonstrates that though simCLR is faster and more reliable when
a critical mass of frames are annotated, it does not generalize well for new
test datasets with new scene classes. Thus, for new use-cases and fields of
view, the rule-based method is a good starting point followed by simCLR
approach for speed up and accuracy.

###  III-C Filtering key frames

Once each frame in video sequences is classified, the next task is to isolate
frames with the highest uncertainty or previously unseen structural content
that can then be annotated to re-train or fine-tune scene understanding
algorithms. The fraction of all the test frames (in percentage) filtered by
the two methods (training and testing on images from the same dataset) and the
video sequences they belong to are shown in Table III. Additionally, we
analyze the key-frames that are filtered by both methods to assess for
structural agree-ability between the two methods. The frame filtering
parameters for both methods are controlled to ensure similar fractions of
filtered frames (less than 10%).

TABLE III: Key frame filtering performance from video sequences.

Dataset: Test JAAD Method % Filtered frames Video Sequences Rule-based 301
(6.4) 0024,0274,0275,0276,0277,0278,0279,0280,0281 simCLR-based 328 (7.0)
0024,0279,0280,0289,0304,0324 Rule-based $\cap$ simCLR 23 (0.5) 0024, 0279,
0280 Dataset: Test KITTI Rule-based 489 (7.6) 0002, 0003, 0007, 0008, 0009
simCLR-based 533 (8.35) 0003,0007, 0008, 0011, 0014, 0018, 0020 Rule-based
$\cap$ simCLR 117 (1.7) 0003, 0007, 0008

From Table III we observe that the rule-based method mostly isolates frames
from video sequences with rain in the JAAD dataset. Running windshield wipers
and droplets on the windscreen structurally alter the scene across frames.
Some examples of key-frames filtered by this method are shown in Fig. 6. Here,
Fig. 6(a) represents a blurry image that is eliminated by the frame quality
detector/selection module. Also, the variations across boundary box
probabilities in each of the poor lighting sequences are shown here.

(a) Image with {night, good light, city} from sequence 0024, JAAD

(b) Image with {day, poor light, city} from sequence 0273, JAAD

(c) Image with {day, poor light, city} from sequence 022, JAAD

(d) Image with {day, poor light, city} from sequence 0276, JAAD

(e) Image with {day, good light, city}, sequence 0000, KITTI

(f) Image with {day, good light, parked-cars}, sequence 0009, KITTI

(g) Image with {day, poor light, city}, sequence 0019, KITTI

(h) Image with {day, poor light, freeway} from sequence 0020, KITTI

Figure 6: Examples of images filtered by the rule-based baseline method.

Additionally, from Table III we observe that the simCLR-based method filters
frames from video sequences with shadows from trees and buildings in city
regions in the JAAD dataset. Thus, all outdoor images with poor lighting
combined with weather-related scenarios including snow and rain are found to
contain structural uncertainty and to lie at cluster boundaries. For the KITTI
dataset, most filtered key-frames correspond to scenes from parked cars in
city residential areas, merging roads to freeways and freeway images with
several cars and foliage. The most challenging decision boundary in the KITTI
dataset therefore is to classify a freeway image with foliage from a city
image with only parked cars in sight. Some examples of images filtered by this
method are shown in Fig. 7. Here, Fig 7(a) represents a night-time frame where
a pedestrian first appears. Filtering such frames and fine-tuning for such
outlier situations will improve object detection tasks such as pedestrian
detection at night.

(a) Night image (JAAD), where a pedestrian first appears (red box).

(b) Image from sequence 0279 in JAAD with several shadows.

(c) Image from sequence 007 in KITTI with parked cars

(d) Image from sequence 0011 in KITTI with poor light.

Figure 7: Examples of key-frames filtered by the simCLR-based method.

Finally, we observe that filtered frames common to both rule-based and simCLR
methods for JAAD are the ones of night-time frames when pedestrians first
appear ahead of the camera vehicle, from city scenes with shadows and poor
lighting on a snowed road. Also, we observe a very small fraction of frames
that are filtered by both methods. This occurs since the simCLR-based method
filters frames that are structurally dissimilar to the training frames, such
as snowed roads, rain, city to highway merging frames etc. On the other hand
the rule-based method filters frames based on the uncertainty in content with
respect to the previous and subsequent frames. Thus, based on the use-case and
outcome requirements for key-frame filtering speed (using simCLR-based method)
vs. generalizability across structural content (using the rule-based method),
using either method for isolating a minimal training data batch is effective
for scene understanding tasks.

To further assess the importance of filtered key-frames for transfer learning,
we further fine-tune the Faster-RCNN model using the filtered frames obtained
from the simCLR model. The object detection performances on the JAAD test
sequences (pedestrian only) are: mean average precision (mAP) of 70.91 and
mean intersection over union (mIOU) of 15.4. On the KITTI data set (vehicles
and pedestrians only), the fine-tuned object detector has mAP of 81.7 and mIOU
of 17.8, which is comparable to state-of-the-art models that use large volumes
of training images for object detection [7].

##  IV Conclusions and Discussion

Video data processing and tasks often require sifting through several hours
worth of video to create an efficient data pipeline that will then be used by
modeling and deployment pipelines in an operationalized machine learning
system. These video data pipelines rely on efficient tagging, storage and
retrieval of video frames to extract relevant data batches for the modeling
pipeline. For instance, fine-tuning an autonomous drive perception system for
highway perception or night-time object detection requires training on
relevant image samples that can be facilitated only if raw input frames were
tagged during storage for scene class, lighting conditions or foreground
content etc. Additionally, isolating relevant key-frames that improve scene
understanding algorithms and sending those images for annotation followed by
model fine-tuning in batches is key to continuous data-centric machine
learning deployment. In this work we present two methods to automate the
process of tagging each frame in video sequences and filtering key-frames
based on structural content to capture variability in image content from a
minimal training dataset.

The first structural content based method requires minimal parameterization
and relies on foreground objects and structural uncertainty information across
frames for scene classification with 64-72% accuracy across varying training
datasets. This method is most generalizable but suffers from a high processing
time of 0.32-0.5 seconds per image. The second method relies on high
dimensional encoding using a self-supervised deep learning model followed by
label spreading that achieves 66-93% accuracy of frame classification by
training on similar images only. This method is extremely fast with processing
times of 0.035 seconds per frame.

Future works will be directed towards combining the deep-learning encoding
with the sequential information from neighboring frames to further enhance the
automated frame tagging methods. Additionally, the proposed methods can be
extended for visual inspection and indoor camera systems for outlier and
threat detection tasks in future.

## References

  * [1] X. Yan, S. Z. Gilani, M. Feng, L. Zhang, H. Qin, and A. Mian, “Self-supervised learning to detect key frames in videos,” _Sensors_ , vol. 20, no. 23, p. 6941, 2020.
  * [2] D. Kang, P. Bailis, and M. Zaharia, “Challenges and opportunities in dnn-based video analytics: A demonstration of the blazeit video query engine.” in _CIDR_ , 2019.
  * [3] U. Sara, M. Akter, and M. S. Uddin, “Image quality assessment through fsim, ssim, mse and psnr—a comparative study,” _Journal of Computer and Communications_ , vol. 7, no. 3, pp. 8–18, 2019.
  * [4] W. Barhoumi and E. Zagrouba, “On-the-fly extraction of key frames for efficient video summarization,” _AASRI Procedia_ , vol. 4, pp. 78–84, 2013\.
  * [5] R. Savran Kızıltepe, J. Q. Gan, and J. J. Escobar, “A novel keyframe extraction method for video classification using deep neural networks,” _Neural Computing and Applications_ , pp. 1–12, 2021.
  * [6] A. Rasouli, I. Kotseruba, and J. K. Tsotsos, “Are they going to cross? a benchmark dataset and baseline for pedestrian crosswalk behavior,” in _Proceedings of the IEEE International Conference on Computer Vision Workshops_ , 2017, pp. 206–213.
  * [7] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2012.
  * [8] Y. Xu, X. Hu, and S. Peng, “Robust image deblurring using hyper laplacian model,” in _Asian Conference on Computer Vision_. Springer, 2012, pp. 49–60.
  * [9] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: towards real-time object detection with region proposal networks,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 39, no. 6, pp. 1137–1149, 2016\.
  * [10] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in _International conference on machine learning_. PMLR, 2020, pp. 1597–1607.
  * [11] D. Zhou, O. Bousquet, T. Lal, J. Weston, and B. Schölkopf, “Learning with local and global consistency,” _Advances in neural information processing systems_ , vol. 16, 2003.


# Adversarial Text Normalization

Joanna Bitton
Meta AI
<EMAIL_ADDRESS>
Maya Pavlova
University of Waterloo
<EMAIL_ADDRESS>

Ivan Evtimov
Meta AI
<EMAIL_ADDRESS>
###### Abstract

Text-based adversarial attacks are becoming more commonplace and accessible to
general internet users. As these attacks proliferate, the need to address the
gap in model robustness becomes imminent. While retraining on adversarial data
may increase performance, there remains an additional class of character-level
attacks on which these models falter. Additionally, the process to retrain a
model is time and resource intensive, creating a need for a lightweight,
reusable defense. In this work, we propose the Adversarial Text Normalizer, a
novel method that restores baseline performance on attacked content with low
computational overhead. We evaluate the efficacy of the normalizer on two
problem areas prone to adversarial attacks, i.e. Hate Speech and Natural
Language Inference. We find that text normalization provides a task-agnostic
defense against character-level attacks that can be implemented supplementary
to adversarial retraining solutions, which are more suited for semantic
alterations.

##  1 Introduction

Figure 1: This figure showcases that text normalization is able to restore
baseline scores on the Learning From The Worst (LFTW) test set for several
augmentation types. The model scores in this graph are the averaged scores
across all five LFTW models. To see the raw evaluation results, view Table 4
in the Appendix.

Natural language processing (NLP) models help preserve the integrity of
discourse in online social networks by detecting hate speech, misinformation,
and other content that violates community policies Halevy (2020). In these
application scenarios, classifiers operate under significantly more
adversarial conditions than in the standard paradigm of model development.
Users often post content that is heavily altered in order to induce worst-case
errors, dramatically reducing model performance relative to a standard test
set. Recent research in machine learning has made strides towards building
robust models to defend against sophisticated adversaries. Nevertheless, our
experience and experiments show that models remain vulnerable to many simple
and intuitive attacks.

One such class of highly-effective attacks are syntactic attacks, which
include adding punctuation and spacing, replacing fonts, and inserting zero-
width characters. Attackers commonly employ these methods with little or no
expert knowledge of machine learning algorithms, and constitute a large
proportion of practical threats to natural language processing models deployed
in industry. In our experiments on state-of-the-art robust NLP models, such
attacks can decrease a model’s performance by more than half. Thus, these
text-based attacks remain a serious and unaddressed problem for the at-scale
deployment of language models in integrity areas.

The literature has proposed methods to improve adversarial robustness by
retraining models on “hard” and adversarial examples Nie et al. (2020); Vidgen
et al. (2021) but we observe several open challenges with applying these
approaches. First, as our experiments demonstrate, even state-of-the-art
models trained on multiple iterations of adversarial data continue to lack
robustness to easily accessible syntactic attacks. For example, in Figure 2,
inserting zero-width characters reduces a hate speech classification model’s
accuracy to less than 15%. Second, these approaches are not amenable to quick
iteration. Whenever a new adversarial attack becomes more prevalent, a model
developer would need to collect new data or create synthetic examples of such
attacks, retrain the model, and redeploy it. This costs time, computational
resources and has an environmental impact (Wu et al., 2021).

To address these outstanding issues, we propose a novel lightweight and easily
extensible method for recovering model performance in the face of adversaries
who perform syntactic attacks. Our key insight is that many adversarial
modifications can be undone before they even reach the model with little
computational overhead. From a machine learning perspective, this can be
thought of as restoring the distribution of inference-time inputs to the
original distribution on which the model was trained. From a computer security
perspective, this method is analogous to sanitizing the inputs to programs so
as to make the input safe for further processing. In this work, we present the
design and implementation of a system called the Adversarial Text Normalizer
(ATN), that achieves this goal. This iterative approach produces a defense
mechanism that can be applied at scale in a lightweight fashion to ensure
robust model performance.

An important principle in computer security is that defense mechanisms should
be evaluated against adaptive adversaries (Petitcolas, 2011), i.e. those that
can adjust their techniques to actions by the defender. Therefore, we also
partner with red teaming experts skilled at creating novel adversarial inputs
to classical computer systems to conduct an adaptive evaluation of the ATN.
Our text normalizer can provide sufficient robustness gains even in the face
of such adaptive adversaries.

Our contributions are as follows:

  * •

We design and implement a system for undoing syntactic attacks on textual
models called the Adversarial Text Normalizer.

  * •

We conduct an adaptive attacker red teaming exercise to evaluate the ATN’s
performance against skilled human adversaries.

  * •

Through extensive experiments on three different benchmarks, we evaluate the
performance of the ATN and conclude that it successfully recovers the original
performance of a model when faced with syntactic attacks.

##  2 Related Work

Several papers have introduced benchmarks for adversarial attacks on NLP
systems. Attacks focused on preserving semantic content and grammar Jin et al.
(2020); Alzantot et al. (2018); Iyyer et al. (2018) have been shown to be
effective against state-of-the-art models at the cost of requiring a greater
understanding of the sentence structure and task context. In contrast, Eger
and Benz (2020) propose a benchmark of character level, orthographic
perturbations as more realistic attacks in general applications, attributing
the success of their high performance attacks to large out-of-vocabulary rates
and disruption to tokenization. Other work (Eger et al., 2019; Boucher et al.,
2021) investigates the replacement of characters with visually similar
embedding spaces and the insertion of zero-width characters, noting the
effectiveness of those methods against NLP models but marginal effect on human
legibility – especially when perturbing key offensive words. For such targeted
attacks, Rodriguez and Rojas-Galeano (2018) use a simple string matching
algorithm to filter obfuscated and negated key words (Rojas-Galeano, 2017),
focusing on a limited list of target vocables on each pass.

Our work focuses on the implementation of text normalization as a
computationally inexpensive and reusable solution to mitigate a range of
highly accessible but effective adversarial text attacks such as character
insertions, replacements, and censorship. Concurrent work in the NLI domain
has addressed the bias in model performance on classic test sets and
adversarial user attacks through iterative human-and-model-in-the-loop Nie et
al. (2020) data generation and model training. Similarly, Vidgen et al. (2021)
proposed a complementary approach with the amalgamation of targeted annotator
samples including challenging perturbations to generate adversarial data for
hate speech classification. Both works explored leveraging domain-experienced
annotator resources to progressively train more robust models with each
successive iteration. Other works on small text perturbations such as
adversarial typos, have proposed the use of robust token-level encodings Jones
et al. (2020) and preceding word recognition models Pruthi et al. (2019) as
reusable systems that are trained once and then reused across models and
tasks. In this work we explore a more lightweight, systematic correction layer
that does not require training to create model and task-agnostic defenses.

##  3 Methodology

###  3.1 Models and Datasets

We identify two natural language tasks with significant importance to
industrial applications and adversarial pressure. First, hate speech
classification is the problem of detecting statements that are likely to cause
harm and inject toxicity in online discourse. It has now become standard
practice for providers of services where people can post comments and discuss
content to employ hate speech classification models. These models are set up
as binary classification models that output a score for the “hatefulness” of a
given input statement. Second, Natural Language Inference (NLI) has been
adapted for the detection of misinformation Nie et al. (2020). In this
setting, NLI models aim to flag statements that do not receive support from
reputable sources or directly contradict information in them. Thus, a model is
given access to a set of support statements and a “hypothesis” and it outputs
a 3-way classification from among “supported,” “not supported,” and “not
enough information.” In the cases of “supported” or “not supported,” the model
also outputs the statement that supports or refutes the hypothesis.

Since both tasks are the subject of adversarial pressure, there have been
several proposed approaches to robustifying models trained on them. Most
notably, the Dynalab Vidgen et al. (2021) approach proactively samples “hard”
examples by asking human raters to conceive inputs that challenge the model.
Researchers then retrain the models and repeat the process for several rounds.
This paradigm helps achieve a large increase in robustness through the rounds,
so we evaluate our approaches on those models as the benchmark for state-of-
the-art robust performance.

For Hate Speech, we utilize the HateCheck (Röttger et al., 2021) dataset
(2,563 examples) and the Learning from the Worst (LFTW) (Vidgen et al., 2021)
test set (4,120 examples). The performance of state-of-the-art models trained
on adversarial data from Learning from the Worst (LFTW) were compared with and
without the addition of the text normalizer on both baseline and augmented
versions of the dataset. Additionally, we chose to evaluate NLI models on the
test sets from all three rounds of Adversarial NLI (Nie et al., 2020) (1,000,
1,000, and 1,200 examples respectively). We assessed the performance of these
baseline, augmented, and normalized datasets on five model architectures
trained on SNLI (Bowman et al., 2015), MNLI (Williams et al., 2018), FEVER
(Thorne et al., 2018) and all three rounds of Adversarial NLI. To see which
models we evaluate on, please review Table 1. We specifically choose models
already trained on adversarial datasets to assess opportunities for
improvement beyond retraining.

Problem Area | Model | # Parameters
---|---|---
Hate Speech | LFTW | 125,000,000
Adversarial NLI | DeBERTa | 140,000,000
Adversarial NLI | RoBERTa | 125,000,000
Adversarial NLI | T5 | 220,000,000
Adversarial NLI | BERT | 109,000,000
Adversarial NLI | ALBERT | 17,000,000
Table 1:  The models, associated problem areas, and number of parameters used
in our evaluations. All LFTW models, from rounds 1-4 and more, have an
identical number of parameters since they are all RoBERTa models.

###  3.2 Attacks

Augmentation | Output Text | Normalized Text
---|---|---
None | This is augmented text | This is augmented text
insert_punctuation_chars | Th.i.s ,is …a.ug;m!en’t?ed, ,te!x.t | This ,is …augmented, ,text
insert_whitespace_chars | T h i s is a u g m e n t e d text | This is augmented text
insert_zero_width_chars | This is augmented text | This is augmented text
merge_words | Thisis augmented text | Thisis augmented text
replace_fun_fonts |  | This is augmented text
replace_similar_chars | Th!s is @ugmented tex7 | Th!s is @ugmented tex7
replace_similar_unicode_chars |  | This is augmented text
simulate_typos | This is augmentde texht | This is augmentde texht
split_words | Th is is augment ed text | Th is is augment ed text
Table 2:  Examples of augmentations generated using the open-source library
AugLy (Papakipos and Bitton, 2022), leveraged in the analysis for adversarial
attacks on the text datasets, and their normalized counterparts. Note that
while the output of insert_zero_width_chars appears visually identical to the
original sentence, there are actually zero-width unicode characters embedded
throughout the entire string. We include augmentations that are not covered by
the normalizer (such as merge_words) in our evaluations to showcase that our
method does not further corrupt unknown attack types.

To attack the aforementioned datasets, we use the open-source augmentation
library AugLy (Papakipos and Bitton, 2022) to simulate various text-based
adversarial attacks commonly used on social media platforms. We focus on using
character-level attacks as opposed to attacks that can potentially add,
remove, or change full words in the text, to avoid perturbations in the
semantic meaning. The specific augmentations selected for the analysis are
listed in Table 2.

In addition to the synthetically generated attacks, we collected 98 samples of
hate speech text which were adversarially created and modified by individuals
in a cybersecurity Red Team. In the first half of the session, participants
were tasked with creating their own attacks based off of their prior knowledge
of text-based attacks seen online. In the second half of the session, they
were given direct access to the code implementation for the text normalizer
and were tasked to bypass it using targeted attacks.

Many of the attacks created were similar to the attacks of
insert_punctuation_chars, replace_fun_fonts, simulate_typos,
replace_similar_unicode_chars, and replace_similar_chars. The adversaries also
created letter repetition attacks, i.e. “hellllooooo", censored violating
text, and replaced words with emojis.

###  3.3 Adversarial Text Normalizer

The text normalizer is an isolated correction unit that can be placed in front
of models to target character-level attacks for various NLP tasks. The
normalizer is designed to be used as a preprocessing step prior to text
tokenization. For optimal computational efficiency, the operator is written in
torchscript, and can process approximately 77 examples per second on a server
with an Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz processor. This operator was
incorporated into the PyTorch model as a customizable data transform. The
algorithm relies on sophisticated string manipulation as a targeted defense
against known adversarial attacks, and is easily scaled up to support
additional attacks as the adversarial environment progresses.

Currently, the text normalizer supports removing three overarching categories
of text attacks:

  1. 1.

Character insertion: addition of characters such as punctuation marks,
whitespaces, Unicode characters, emojis, and more to separate characters in a
word with the intent to disrupt proper tokenization. This category includes
augmentation methods such as insert_punctuation_chars,
insert_whitespace_chars, and insert_zero_width_chars.

  2. 2.

Character replacement: substitution of standard Latin characters with visually
similar characters from other languages or Unicode characters with the intent
of obfuscation. This category includes augmentation methods such as
replace_fun_fonts, replace_similar_chars, and replace_similar_unicode_chars.

  3. 3.

Censorship of violating words: replacement of letters in violating words with
punctuation characters to avoid explicit content. For instance “kill" could be
censored as “k***", “k!ll", “k#*!" and more.

To undo the effects of a character insertion attack such as
insert_punctuation_chars, the text is first split by whitespaces to identify
‘words’. For each word, we then determine how many extraneous punctuation
characters have been inserted, ignoring punctuation marks at the beginning and
end, as such additions do not segment the word to disrupt tokenization. If the
amount of punctuation characters is below a set threshold or the word
resembles a URL, we do not modify the word and add it to our normalized
string. Otherwise, we replace the superfluous punctuation with spaces, strip
the string of excessive whitespace, and concatenate consecutive single
character entities together.

For character replacement attacks, we predefined multiple mappings between
Unicode characters and their keyboard character pairs, and performed a string
search method to reverse the replacement.

As for censorship attacks, a list of common user-censored toxic terms were
identified prior based on flagged user content. For each toxic term, a regex
for the censorship pattern was defined such that the first and last letters of
the word remain constant but any of the letters in between can be replaced by
punctuation characters - maintaining the same length as the original,
uncensored word. For every match found, we replace the censored string with
its uncensored pair accordingly.

##  4 Evaluation

In this section, we examine the performance of Hate Speech and Natural
Language Inference (NLI) models with respect to the original, the augmented,
and normalized datasets to evaluate the efficacy of the text normalizer. All
evaluations were conducted using Dynabench (Ma et al., 2021), in which AWS ECR
models are deployed as endpoints and Batch Transform jobs are run on AWS
Sagemaker to get dataset predictions. We roughly spent 38.36 CPU hours on
model inference (no GPUs were used).

###  4.1 Hate Speech

Figure 2: This figure showcases the results of evaluating Hate Speech models
against the baseline, augmented, and normalized HateCheck datasets. The model
scores in this graph are the averaged scores across all five LFTW models. To
see the raw evaluation results, view Table 5 in the Appendix.

To assess performance on adversarial hate speech data, we evaluated on five
models from the Learning from the Worst (LFTW) (Vidgen et al., 2021) paper
that were previously retrained on varying amounts of "rounds" of adversarial
hate speech data collection.

####  4.1.1 Learning from the Worst

We first evaluate on the test set from Learning from the Worst. Figure 1
showcases these results. Overall, the text normalizer maintains or improves
initial performance on all augmented datasets and the baseline. Across all
models, normalizing the insert_zero_width_chars,
replace_similar_unicode_chars, and replace_fun_fonts augmentations resulted in
the most significant performance gains, with a maximum of a 32.18% increase.
In between, normalizing insert_punctuation_chars and insert_whitespace_chars
had increases of at most 11.95%. For the LFTW R3 model, normalizing the
whitespace text resulted in a 1.27% loss in performance. This may be due to
the fact that not every whitespace character in the AugLy augmentation is
removed by the normalizer. As expected, other augmentations that aren’t
covered by the normalizer did not see any substantial gains or losses in
performance. To view the raw model scores, see Table 4 in the Appendix.

####  4.1.2 HateCheck

In addition to evaluating on the LFTW test set, we also evaluated on an out-
of-distribution dataset, HateCheck. Figure 2 displays these results. The
trends observed in the LFTW test set overall have agreement with the HateCheck
results. However, in this case, there were no losses in performance for
normalized augmentations covered by our method. The largest performance gain
was 48.89% by normalizing insert_zero_width_unicode_chars, and the smallest
performance gain was 6.1% by normalizing insert_punctuation_chars. To see the
raw evaluation results, please review Table 5 in the Appendix.

####  4.1.3 Red Team Attacks

Lastly, we evaluated the LFTW models on the Red Team dataset tasked to bypass
the text normalizer. The baseline scores across all models averaged at 39.52%
and the normalized scores averaged at 41.32%. In comparison to the
synthetically-augmented text, applying the normalizer resulted in a less
drastic increase in performance. This difference can be attributed to the fact
that a significant amount of the data was attacked similarly to
replace_similar_chars and simulate_typos, two augmentations not covered by our
defenses. To view the raw model scores, see Table 3 in the Appendix.

###  4.2 Natural Language Inference

To validate our results in another problem space, we evaluated on five Natural
Language Inference (NLI) models previously trained on a collection of
adversarial and benign NLI datasets. We evaluated the models on the baseline,
augmented, and normalized ANLI test sets from rounds 1-3.

Figure 3: This figure contains the results of evaluating ANLI models against
the baseline, augmented, and normalized rounds 1-3 test sets. The model scores
in this graph are the averaged scores across all five models. To see the raw
evaluation results, view Tables 6, 7, and 8 in the Appendix.

Overall, the results from the ANLI experiments align with the previous
insights discussed in the Hate Speech task. The replace_fun_fonts,
insert_zero_width_chars, and replace_similar_unicode_chars attacks were the
most performant, followed by insert_punctuation_chars,
insert_whitespace_chars, and finally the non-covered attack types. The largest
gain, 29.1%, was made by normalizing insert_zero_width_unicode_chars in the
ANLI R1 test set. To view the raw model scores, see Tables 6, 7, and 8 in the
Appendix.

##  5 Discussions

Retraining models on adversarial data has been proposed as a mitigation method
for adversarial attacks (Vidgen et al., 2021; Nie et al., 2020). Its benefits
are that it’s relatively simple to implement, shown to be effective
(Goodfellow et al., 2015), and doesn’t affect model throughput once deployed.
However, if a new adversarial attack were to be discovered on a system, the
turnaround time for deploying a robust model can be quite long. A developer
would need to (1) gather and annotate or systematically generate the attacked
data (2) retrain the model (3) redeploy the model. Steps (1) and (2) could be
highly nontrivial depending on the attack type and model size. In addition,
there can also be a significant monetary and environmental cost in retraining
a large model (Wu et al., 2021).

On the other hand, text normalization allows a developer to move fast. Instead
of collecting data, an engineer would simply need to write one additional
function to reverse said attack and test it. In addition, it’s lightweight and
there is no need to retrain, as this is a text preprocessing step. However,
text normalization cannot handle every attack type. Text normalization is best
suited to mitigate character-level attacks that do not change the semantic
meaning of the text, i.e.  syntactic attacks.

Another important consideration is that text normalization does affect real-
time performance. A balance must be found between intelligently mitigating
attacks and compute. In our use case, we limited the normalizer to
sophisticated string manipulation, as anything more would result in too much
compute. Hence, it becomes less feasible to build defenses that require
knowledge or understanding of words, etc.

Thus, we recommend the best way to mitigate adversarial attacks is to use a
combination of text normalization and retraining. Specifically, retraining
should be used for semantic adversarial attacks, and adversarial text
normalization for syntactic adversarial attacks. Despite all the models
evaluated on being retrained on adversarial data, they were still vulnerable
to character-level text attacks. However, together with the text normalizer,
we were able to increase performance and even return to baseline performance
at times. This study is meant to show that text normalization in general is a
viable approach to mitigating syntactic text-based attacks. This is certainly
not the final method, and we hope researchers extend this work to support
multilingual text.

##  6 Conclusion

We proposed a new method to mitigate text-based adversarial attacks, called
the Adversarial Text Normalizer. We evaluated the performance of models
retrained on adversarial data with and without the ATN. Our experiments show
that text normalization and retraining should be used together in order to
maintain baseline performance against a broad range of adversaries.

## Acknowledgements

The authors would like to thank Adina Williams, Tristan Thrush, and Kushal
Tirumala for their support in running experiments with the Dynabench platform.
The authors would also like to thank Luke Zettlemoyer, Caner Hazirbas, and
Stefan Hermanek for helpful discussions on the work in this paper. Finally,
the authors extend their gratitude to Aaron Grattafiori, Tom Ravenscroft, Adi
Ajit, Aimee Couglin, Athena Cheung, Ben Actis, Christopher Korban, Greg
Prosser, Hamza Kwisaba, James Burton, Jayson Grace, Martin Vigo, Nat Hirsch,
Ryan Hall, Sasha Hanganu, Tom Hebb, and Vlad Ionescu for participating in the
evaluation of the Adversarial Text Normalizer.

## Impact Statement

### Risks from the Work

Our work is meant to reduce risk to online discourse by enabling NLP models to
function robustly in an adversarial setting. We acknowledge that releasing
this work might enable stronger attacks against some systems, but we believe
it is important to discuss these defenses publicly for two reasons. First,
other practitioners and researchers can benefit from our findings in building
stronger defenses. Second, the history of security and cryptographic research
clearly demonstrates that robust systems are built only when they go through
ongoing attack/defend iteration cycles. To that end, we hope that our work
informs the next steps in building robust NLP systems. As with any NLP system
and computer technology, we acknowledge robustness may be at odds with safety
when the models defended with our mechanism are themselves used for nefarious
purposes.

### Use of Scientific Artifacts

In this work, we made heavy use of the Dynabench platform (Kiela et al., 2021;
Ma et al., 2021) and the models trained on data collected with it. We worked
closely with the creators of the platform and the models, and they were always
fully aware of our intentions. The Dynabench platform has been released under
the MIT license:
https://github.com/facebookresearch/dynabench/blob/main/LICENSE. We also used
the Adversarial NLI, HateCheck, and Learning from the Worst datasets. Each of
those do not contain identifying information and only associate a pseudonymous
annotator ID with each example. We verified this manually by looking at
samples from the datasets. The HateCheck and Learning from the Worst datasets
contain offensive language as part of the nature of the task they were set up
for. All datasets are in English and were created by English-speaking authors
and annotators in the United States.

### Humans Involved in the Research

We did not employ annotators or perform human subject experiments. We engaged
with a partner team in a collegial capacity to expand our insights into the
adversarial text normalizer and we discuss those findings here for the benefit
of the broader research community.

## References

  * Alzantot et al. (2018) Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018.  Generating natural language adversarial examples.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2890–2896, Brussels, Belgium. Association for Computational Linguistics.
  * Boucher et al. (2021) Nicholas Boucher, Ilia Shumailov, Ross Anderson, and Nicolas Papernot. 2021.  Bad characters: Imperceptible nlp attacks.  _arXiv preprint arXiv:2106.09898_.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015\.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642, Lisbon, Portugal. Association for Computational Linguistics.
  * Eger and Benz (2020) Steffen Eger and Yannik Benz. 2020.  From hero to zéroe: A benchmark of low-level adversarial attacks.  In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing_ , pages 786–803, Suzhou, China. Association for Computational Linguistics.
  * Eger et al. (2019) Steffen Eger, Gözde Gül Şahin, Andreas Rücklé, Ji-Ung Lee, Claudia Schulz, Mohsen Mesgar, Krishnkant Swarnkar, Edwin Simpson, and Iryna Gurevych. 2019.  Text processing like humans do: Visually attacking and shielding NLP systems.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1634–1647, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Goodfellow et al. (2015) Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015.  Explaining and harnessing adversarial examples.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Halevy (2020) Alon Y. Halevy. 2020.  Preserving integrity in online social media.  In _KDD ’20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020_ , page 3601\. ACM.
  * Iyyer et al. (2018) Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018.  Adversarial example generation with syntactically controlled paraphrase networks.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1875–1885, New Orleans, Louisiana. Association for Computational Linguistics.
  * Jin et al. (2020) Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020.  Is bert really robust? a strong baseline for natural language attack on text classification and entailment.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(05):8018–8025.
  * Jones et al. (2020) Erik Jones, Robin Jia, Aditi Raghunathan, and Percy Liang. 2020.  Robust encodings: A framework for combating adversarial typos.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2752–2765, Online. Association for Computational Linguistics.
  * Kiela et al. (2021) Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021.  Dynabench: Rethinking benchmarking in NLP.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4110–4124, Online. Association for Computational Linguistics.
  * Ma et al. (2021) Zhiyi Ma, Kawin Ethayarajh, Tristan Thrush, Somya Jain, Ledell Yu Wu, Robin Jia, Christopher Potts, Adina Williams, and Douwe Kiela. 2021.  Dynaboard: An evaluation-as-a-service platform for holistic next-generation benchmarking.  In _Advances in Neural Information Processing Systems_.
  * Nie et al. (2020) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020.  Adversarial NLI: A new benchmark for natural language understanding.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4885–4901, Online. Association for Computational Linguistics.
  * Papakipos and Bitton (2022) Zoe Papakipos and Joanna Bitton. 2022.  Augly: Data augmentations for robustness.
  * Petitcolas (2011) Fabien A. P. Petitcolas. 2011.  _Kerckhoffs’ Principle_, pages 675–675. Springer US, Boston, MA.
  * Pruthi et al. (2019) Danish Pruthi, Bhuwan Dhingra, and Zachary C. Lipton. 2019.  Combating adversarial misspellings with robust word recognition.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5582–5591, Florence, Italy. Association for Computational Linguistics.
  * Rodriguez and Rojas-Galeano (2018) Nestor Rodriguez and Sergio Rojas-Galeano. 2018.  Shielding google’s language toxicity model against adversarial attacks.
  * Rojas-Galeano (2017) Sergio Rojas-Galeano. 2017.  On obstructing obscenity obfuscation.  _ACM Trans. Web_ , 11(2).
  * Röttger et al. (2021) Paul Röttger, Bertie Vidgen, Dong Nguyen, Zeerak Waseem, Helen Margetts, and Janet Pierrehumbert. 2021.  HateCheck: Functional tests for hate speech detection models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 41–58, Online. Association for Computational Linguistics.
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018\.  FEVER: a large-scale dataset for fact extraction and VERification.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 809–819, New Orleans, Louisiana. Association for Computational Linguistics.
  * Vidgen et al. (2021) Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela. 2021.  Learning from the worst: Dynamically generated datasets to improve online hate detection.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1667–1682, Online. Association for Computational Linguistics.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
  * Wu et al. (2021) Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga Behram, James Huang, Charles Bai, Michael Gschwind, Anurag Gupta, Myle Ott, Anastasia Melnikov, Salvatore Candido, David Brooks, Geeta Chauhan, Benjamin Lee, Hsien-Hsin S. Lee, Bugra Akyildiz, Maximilian Balandat, Joe Spisak, Ravi Jain, Mike Rabbat, and Kim Hazelwood. 2021.  Sustainable AI: environmental implications, challenges and opportunities.  _CoRR_ , abs/2111.00364.

## Appendix

##  Appendix A AugLy Augmentation Generation

To augment the hate speech and natural language inference datasets, we chose
random parameters for each augmentation for every piece of text. The ranges we
used for every parameter are listed below. Since multiple AugLy augmentations
accept as input the same parameters, we do not break this down by augmentation
type (the same value was used across all augmentations). The following ranges
were inputted into random.uniform() and lists of options were inputted
random.choice():

  * •

aug_p: (0.3, 1.0)

  * •

aug_word_p: (0.3, 1.0)

  * •

aug_char_p: (0.1, 0.4)

  * •

granularity: ["char", "word", "all"]

  * •

vary_fonts: [True, False]

##  Appendix B Raw Data: Hate Speech and Natural Language Inference Analyses

Here, we provide the full evaluation results with exact numbers broken down by
attack type and model.

Model | Red Team | Normalized
---|---|---
LFTW R1 | 32.41 | 35.95
LFTW R2 | 40.61 | 41.32
LFTW R3 | 39.88 | 41.32
LFTW R4 | 42.01 | 43.68
All LFTW | 42.69 | 44.32
Table 3: Hate Speech Case Study Results by Model: Red Team Dataset

Augmentation LFTW R1 LFTW R2 LFTW R3 LFTW R4 All LFTW baseline 57.79 70.03
81.20 81.09 80.27 normalized 58.04 70.08 81.24 81.21 80.27
insert_punctuation_chars 43.34 50.48 48.53 49.23 50.74 normalized 50.83 58.48
60.48 63.40 56.43 insert_whitespace_chars 43.05 51.97 53.65 46.89 45.84
normalized 48.83 52.97 52.38 52.50 52.05 insert_zero_width_chars 35.04 50.73
51.04 48.96 46.87 normalized 58.37 70.23 81.29 81.14 80.38 merge_words 56.47
68.74 77.67 78.27 77.30 normalized 56.67 68.83 77.73 78.30 77.10
replace_fun_fonts 42.23 50.08 55.75 51.10 50.43 normalized 58.11 70.67 80.42
80.01 78.91 replace_similar_chars 52.43 56.21 52.39 52.36 56.17 normalized
52.48 56.76 53.00 52.76 56.19 replace_similar_unicode_chars 47.65 56.16 56.26
58.06 59.34 normalized 58.20 70.15 80.25 80.04 79.08 simulate_typos 53.95
60.48 61.61 59.93 61.81 normalized 53.99 60.55 61.34 60.31 61.74 split_words
52.45 55.74 59.27 58.36 60.22 normalized 52.38 56.38 59.32 58.77 60.82

Table 4: Hate Speech Case Study Results: LFTW

Augmentation LFTW R1 LFTW R2 LFTW R3 LFTW R4 All LFTW baseline 36.48 47.78
49.23 49.35 49.45 normalized 36.45 47.73 49.29 49.37 49.47
insert_punctuation_chars 13.18 35.05 32.52 30.09 18.09 normalized 28.19 41.15
40.78 44.67 38.95 insert_whitespace_chars 11.19 15.72 29.63 15.41 14.96
normalized 20.60 34.48 45.57 33.01 32.57 insert_zero_width_chars 0.00 21.28
30.50 8.37 0.58 normalized 36.45 47.73 49.29 49.37 49.47 merge_words 37.03
47.99 48.72 49.16 49.15 normalized 37.32 47.92 48.75 49.17 49.08
replace_fun_fonts 16.38 27.04 32.68 27.43 21.95 normalized 36.12 46.32 48.83
49.29 49.38 replace_similar_chars 33.86 47.63 47.57 47.77 47.09 normalized
34.10 47.47 47.46 47.80 47.05 replace_similar_unicode_chars 22.61 39.35 41.07
42.68 40.06 normalized 36.42 47.26 49.12 49.29 49.45 simulate_typos 32.46
44.16 42.84 46.49 45.88 normalized 32.57 43.81 42.79 46.44 45.75 split_words
33.76 47.19 44.11 47.21 46.49 normalized 33.20 46.44 43.58 47.00 46.04

Table 5: Hate Speech Case Study Results: HateCheck

Augmentation RoBERTa T5 BERT ALBERT DeBERTa baseline 62.10 58.90 53.80 63.00
65.00 normalized 61.80 58.90 54.00 62.90 65.00 insert_punctuation_chars 32.40
32.70 32.60 34.40 34.60 normalized 40.40 38.80 41.20 39.90 46.40
insert_whitespace_chars 36.40 38.50 40.20 33.60 33.20 normalized 38.00 39.80
39.50 34.60 35.10 insert_zero_width_chars 35.50 34.30 53.80 31.90 36.20
normalized 61.80 58.70 54.00 62.80 65.30 merge_words 60.10 57.20 48.00 58.30
59.90 normalized 60.00 57.30 48.10 58.50 59.70 replace_fun_fonts 40.00 50.10
36.40 62.30 39.10 normalized 60.50 58.50 51.80 63.10 64.00
replace_similar_chars 45.30 44.20 39.60 44.00 44.70 normalized 43.80 43.80
40.50 43.80 44.60 replace_similar_unicode_chars 44.30 42.10 38.20 53.40 44.90
normalized 61.00 58.00 52.40 62.00 63.60 simulate_typos 46.70 44.20 42.90
45.00 48.10 normalized 46.80 44.20 42.90 44.90 47.70 split_words 45.50 42.90
41.40 44.30 46.60 normalized 45.80 42.80 41.60 43.70 47.10

Table 6: Natural Language Inference Case Study Results: ANLI R1

Augmentation RoBERTa T5 BERT ALBERT DeBERTa baseline 46.50 46.80 44.80 46.50
44.50 normalized 46.20 46.80 44.90 46.60 44.40 insert_punctuation_chars 35.40
34.10 31.80 34.20 34.10 normalized 35.60 36.10 37.90 38.50 38.00
insert_whitespace_chars 35.30 36.70 35.50 33.90 34.40 normalized 33.60 36.50
35.80 34.80 35.30 insert_zero_width_chars 35.10 34.20 44.80 33.40 34.10
normalized 46.30 46.70 44.90 46.60 44.30 merge_words 45.60 46.60 46.70 44.30
42.80 normalized 45.50 46.60 46.60 44.50 42.50 replace_fun_fonts 36.70 42.20
35.90 46.40 36.80 normalized 44.80 45.90 44.80 46.30 44.00
replace_similar_chars 38.00 37.90 37.50 39.60 38.80 normalized 38.20 37.60
36.20 39.40 38.60 replace_similar_unicode_chars 39.40 38.30 36.90 42.00 37.60
normalized 46.00 46.20 44.30 46.20 44.40 simulate_typos 38.40 39.10 39.00
37.70 39.40 normalized 38.30 38.90 39.30 37.70 39.70 split_words 38.00 38.30
36.80 37.40 39.50 normalized 38.30 38.20 36.10 38.40 39.80

Table 7: Natural Language Inference Case Study Results: ANLI R2

Augmentation RoBERTa T5 BERT ALBERT DeBERTa baseline 45.58 44.83 44.00 44.17
45.83 normalized 45.08 44.83 43.67 43.58 45.33 insert_punctuation_chars 35.17
33.50 34.58 33.67 35.17 normalized 35.75 36.58 33.92 33.92 36.00
insert_whitespace_chars 33.67 35.17 36.33 34.33 34.92 normalized 34.75 36.25
37.33 35.00 35.00 insert_zero_width_chars 34.42 32.25 44.00 32.08 35.00
normalized 45.08 44.83 43.83 43.75 45.33 merge_words 43.42 44.58 41.42 40.33
44.92 normalized 42.83 44.58 41.42 40.00 44.42 replace_fun_fonts 38.08 40.92
37.08 44.25 37.50 normalized 45.33 44.92 43.58 43.58 44.83
replace_similar_chars 40.00 37.08 36.92 36.83 37.00 normalized 38.83 37.42
36.42 36.75 37.17 replace_similar_unicode_chars 38.67 38.00 36.17 41.17 37.67
normalized 44.58 45.00 43.08 43.58 44.67 simulate_typos 35.92 36.83 36.83
37.75 38.00 normalized 35.50 36.67 36.00 37.00 37.50 split_words 38.50 38.50
37.08 37.75 35.67 normalized 37.50 38.17 36.42 37.33 35.75

Table 8: Natural Language Inference Case Study Results: ANLI R3