# Conversational Group Detection With Deep Convolutional Networks

Mason Swofford
Stanford University
Stanford, CA
<EMAIL_ADDRESS>These authors contributed equally. John
Peruzzi11footnotemark: 1
Stanford University
Stanford, CA
<EMAIL_ADDRESS>Marynel Vázquez
Stanford University
Stanford, CA
<EMAIL_ADDRESS>
###### Abstract

Detection of interacting and conversational groups from images has
applications in video surveillance and social robotics. In this paper we build
on prior attempts to find conversational groups by detection of social
gathering spaces called o-spaces used to assign people to groups. As our
contributions to the task, we are the first paper to incorporate features
extracted from the room layout image, and the first to incorporate a deep
network to generate an image representation of the proposed o-spaces.
Specifically, this novel network builds on the PointNet architecture which
allows unordered inputs of variable sizes. We present accuracies which
demonstrate the ability to rival and sometimes outperform the best models, but
due to a data imbalance issue we do not yet outperform existing models in our
test results.

##  1 Introduction

###  1.1 Motivation

Conversational group detection from images has applications in video
surveillance and social robotics. Effective detection of groups allows a
mobile robot to navigate around people without interfering inappropriately
with social interactions. Sociologists define a facing formation, or
f-formation, as a socio-spatial formation in which people maintain a convex
space termed an o-space [6]. Detection of o-spaces and assigning people to
these o-spaces is a standard and sociologically consistent method for
detection of conversational groups. Robots which operate in public spaces,
particularly those which interact with and serve humans, can perform their
functions more naturally and seamlessly if able to recognize these social
formations and spaces.

###  1.2 Literature Review

There is ample literature on group detection which sets standards for error
analysis, group definition, and a baseline for performance. The best
performance to date has come not from a learning algorithm but rather a graph-
cuts clustering algorithm, which attained precisions from approximately .84 to
.65 on the Cocktail Party dataset depending on the threshold for accuracy used
[6] (see Evaluation Metrics for explanation of error analysis). We will
attempt to improve upon these results. Other attempts have included edge-
weighted graph algorithms, for instance one in which each node is a person and
the edge measures affinity between pairs [2]. There is a similar method with
weightings based on an attention metric, differing from the previous method by
exploiting social cues to determine edge weightings [7]. There is also a game
theoretic framework with probabilistic overlapping regions determined by an
attention metric which uses temporal data [8]. Finally, another successful
algorithm has been a voting-based algorithm where each individual gets a vote
for the o-space given by a Gaussian function for that individual [5]. All
attempts have performed worse or similar to the graph-cuts algorithm, so we
will focus on improving upon those results. Additionally, it is promising that
we are the first paper to apply deep learning to this task and the first paper
to use features from a layout image of the room.

###  1.3 Methodology Summary

Figure 1: Example O-Space [6]

Our data comes from standard datasets for f-formation detection, which are
videos of social interactions in confined spaces. These are annotated by
sociologists with position, orientation, and group assignment for every person
in various frames taken throughout the video. We use 2D coordinates and yaw
orientations of the people as features. Additionally, we manually transform
the camera images into a 2D map of the room, and extract features from this
map through deep convolutions.

Using the annotated features and image map, we run a deep network to output a
2D image of the room, with each pixel of the output image representing the
likelihood of that location containing an o-space. We then perform non-maximal
suppression and thresholding to arrive at our final predictions for the
o-space locations and greedily assign people to the nearest o-space to
determine our conversational groups.

##  2 Data Acquisition and Manipulation

###  2.1 Datasets

There exist multiple standard annotated datasets for the task of f-formation
detection. We elected to use the Cocktail Party dataset [5] due to its clarity
and known accurate annotations. There are four camera angles which record a 30
minute video of a 30 sq m room, in which 6 people wander around and
occasionally converse. One frame every three seconds is annotated with $x,y$
position and yaw orientation of each person, as well as which, if any,
conversational group they belong to. There are a total of 320 annotated frames
which we have divided with an 80%, 10%, 10% split into train, validation, and
testing sets, respectively. This division was not done randomly as is common
practice in most machine learning applications, but instead it was done
sequentially to prevent test groups and positions from resembling training
groups and positions, with the validation set acting as an additional buffer
between them.

Figure 2: Example Frame from Camera [5]

Our data therefore has both images and a file of annotated features. For
future work, we plan to incorporate other standard datasets such as the
Coffee-Break [6] or SALSA [1] datasets to acquire more training data.

###  2.2 Feature Extraction

Position in $x,y$ coordinates of all people is left as annotated from the
dataset. For head orientation, which was previously given in radians, we
bucketed into 16 possible discrete orientations of size 22.5$\degree$. This is
done in order to allow the network to learn that degrees wrap around. For
example, a value of 359$\degree$ would be more similar to 0$\degree$ than it
would to 300 $\degree$, and using continuous numerical values would not easily
allow the network to learn this trait. As well, modern networks to detect head
orientation algorithms often output yaw in discrete buckets rather than
continuous values for similar reasons [3].

We manually annotate the room image to create a 2D map of important features
of the room, with the hope that knowledge of locations of items such as walls
or tables will allow our algorithm to better identify social spaces.
Transformation of pixels from frames taken by cameras to overhead 2D
coordinates involves using a 3D to 2D calibration algorithm provided by the
dataset to convert an $x,y,z$ point in the room to a pixel from the camera
image. We set $z=0$ as we only desire $x,y$ coordinates and then visualize
$x,y,0$ points on the camera image to locate the room items in the $x,y$
plane. Modern 3D segmentation and classification algorithms will hopefully
automate this process in the future.

Figure 3: Overhead Room Image

###  2.3 Ground Truth from Labeled Data

As the number and size of groups is variable and therefore difficult to output
or even represent in a vectorized form, we instead have our algorithm output a
10x12 2D representation of the room, with each pixel of our 2D image
corresponding to a 0.5x0.5 meter space on the floor and containing a value
representing its likelihood of containing an o-space. To generate the true
image for our labeled data, we find the true o-spaces using a known algorithm
which minimizes distance of the o-spaces from every person in a group defined
by an o-space [9]. We label the pixels by treating the true o-spaces as the
means of multi-dimensional Gaussians. To output actual groups, we assign
people greedily to the o-spaces determined by our algorithm.

Figure 4: Ground Truth Image of 2 O-Spaces

###  2.4 Data Augmentation

Our model is very complex and the dataset consisted of only 320 annotated
frames. We thus augmented the data by flipping the room across the vertical
axis, horizontal axis, and both. To do this, we flipped the location and angle
of each person accordingly, leaving group annotations the same. We also
flipped the overhead room image before extracting its features. Through this,
we increased the size of our dataset to 1280 frames. Before this augmentation,
our model had difficulty learning, but afterward our accuracies in many cases
doubled or more.

##  3 Learning Methods

Our model can be broken down into three connected components: room feature
extraction, people feature extraction, and a fully connected network.

###  3.1 Extracting Room Features

For each room image, we wish to extract relevant features for our model. In
order to do so, we feed the images into MobileNet and extract the output from
block 9, approximately 3/4 of the way through the network. We theorize that
this layer is deep enough to contain useful features, but not to deep as to be
overly specialized for the image classification task it was trained on. We
chose MobileNet because it has the ability to reduce the input size as the
image progresses through the network. To do this, we chose an alpha=.25 which
reduces the image by 1/4 as it passes through the network. However, the output
image is still too high dimensional for our purposes, given our limited data.
Therefore, we chose to perform PCA on the outputs to reduce the image to 1024
dimensions. While this process is part of our deep network, we freeze all the
weights in MobileNet, as we do not have enough data to retrain it.

###  3.2 Extracting People Features

Each person is represented as an 18-dimensional vector consisting of the x
coordinate, y coordinate, and 16-dimensional one hot yaw, where the x and y
coordinates are normalized across the training set. However, since we do not
want the order in which we feed the people into our network to matter, and we
want to allow different numbers of people to be input into the network, since
different datasets have different numbers of people, we use an innovative
approach first used in PointNet [4].

We first create a 3D tensor of dimension 1xMx18 where M is the maximum number
of people possible (M=25 in our case) and fill the first 1xPx18 values with
our people vectors, where P is the number of people in a particular example
(P=6 for the Cocktail Party dataset), and leave the remaining 1x(M-P)x18
values zero. Then, we apply a series of 1x1 convolutions to the tensor to
slowly grow it to a 1xMxD tensor using a deep network, where D is our output
depth, which was determined by cross validation to be 1024. Note that in this
process the ordering of the person vectors is irrelevant because each vector
is growing without interacting with the other person vectors. Finally, a
symmetric function, in our case the max function, is applied across the final
dimension to obtain a D-dimensional vector which encodes information about all
of the input people and is independent of input order. Note that unlike the
room image feature extraction component, the weights in this layer are trained
in response to the output loss.

###  3.3 Generation of Groups

The final component of our neural network is a series of fully connected
layers that takes in the flattened MobileNet room features and PointNet people
features and outputs a 120-dimensional vector which corresponds to a flattened
version of our ground truth o-space map. To train our network, we evaluate our
output against the ground truth using the mean squared error loss. To get our
predicted groups, we convert our 120-dimensional vector to a 10x12 image and
perform non-maximal suppression and thresholding to obtain potential o-spaces.
Then, each person proposes an o-space given his position, orientation, and a
learned stride-length. We assign each person to the potential o-spaces
proposed by the non-maximal suppression neared to that person’s proposed
o-space. People assigned to the same o-space are then in the same group.

Figure 5: Maximal non-suppression to generate potential o-space centers Figure
6: Correctly assigning People to properly predicted o-spaces

##  4 Experiments

###  4.1 Hyper-Parameter Tuning

We experimented with various hyper-parameters. Our largest hyper-parameter was
our model architecture. First, we had to decide on our PointNet sub-
architecture. For this part, the output had to have enough dimensions to
adequately capture information on all the input people 18-dimensional vectors.
We experimented with output sizes of 512, 1024, and 2048 and settled on 1024
through cross validation. For our fully-connected architecture which inputs
the MobileNet features and PointNet features, we experimented with various
architectures and quickly found that only shallow networks worked. Our final
model had two layers with the first layer having 1024 neurons. When we used
deeper networks, the network was too complex and we had not enough data for
our network to learn.

We also experimented with various standard deviations for our Gaussian
representation of ground-truth o-spaces. Our initial Gaussian used 1 m
standard deviation but we found that when two groups were close together, the
deviation made it hard to distinguish the groups and our model almost
invariably predicted them as one large group. We then used .1m standard
deviation and this produced outputs which resembled one hot encodings.
However, this caused issues because given the high dimensional output, if our
network got the prediction wrong, it had trouble learning anything. This
caused our network to produce very poor outputs which were always either near
1 or 0. To combat both of these issues, we settled on .5m as our standard
deviation, because it allowed enough separation between groups that we could
distinguish them in our prediction, but our network could still learn.

We also tuned three hyper-parameters from group assigning: maximum distance
between groups, threshold for non-maximal suppression, and maximum distance to
assign to a group. Maximum distance between groups determines how close we
allow two potential group proposals to be. Threshold for non-maximal
suppression determines how high our probability of a local maximum in our
output image being a group needs to be for it to be a valid o-space proposal.
Maximum distance for assigning determines how far a person’s proposed o-space
can be from a potential group o-space before we do not allow that person to be
assigned to that group. We ran an iterative search over these three hyper-
parameters in our validation set to determine the ideal combination.

###  4.2 Data Imbalance

We found that our network was doing very well on the train and validation set
and outperforming existing models, so we decided to check our performance on
the test set. However, upon doing so, our f1 score dropped by $30\%$. We
decided to investigate why this occurred. We saw that over $75\%$ of our
training and validation data contained only one group, while only $44\%$ of
our test data contained one group. Since we assigned train, validation, and
test data sequentially to avoid training on images identical to test images,
we hypothesize that by the end of the cocktail party participants splintered
into smaller groups. We also hypothesized that we had over-fit to our training
and validation sets by only predicting large groups, and thus we decided to
retrain our model by more heavily weighting examples with multiple groups. We
present the results of both experiments.

##  5 Results

###  5.1 Evaluation Metrics

Standard accuracy evaluation metrics define a group as correctly estimated if
$\left\lceil T*|G|\right\rceil$ of their members are correctly estimated and
if no more than $1-\lceil T*|G|\rceil$ false subjects are identified, where
$|G|$ is the cardinality of the labeled group $G$ and $T$ is a defined
tolerance threshold. Standard values of $T$ to investigate are $2/3$ and $1$
[6]. It is standard to define $TP$ (true positive) to be a correctly detected
group, $FN$ (false negative) to be a non-detected group, and $FP$ (false
positive) to be a group that was detected but did not exist. We then measure
our accuracy with three metrics: precision, recall, and $F_{1}$ score.
Precision is $\frac{TP}{TP+FP}$, recall is $\frac{TP}{TP+FN}$ and $F_{1}$
score is $2*\frac{precision*recall}{precision+recall}$.

###  5.2 Results

As mentioned before, we present results from two experiments: one where all
examples are weighted equally and one where we assign a proportionally higher
weight to examples with multiple groups, in order to combat the data
imbalance. Results for both experiments can be seen in the tables below, as
well as those of the best performing model to date. Since all other prior
models were purely algorithmic and not learning methods, other literature only
has one accuracy to report instead of train/val/test.

Figure 7: Precision, Recall, F1 scores for unweighted model. Bold results
outperform the best published results Figure 8: Precision, Recall, F1 scores
for weighted model. Figure 9: Best Performing Model To Date [6]

###  5.3 Discussion

In our initial experiment with no weighting, our training and validation
accuracies often outperform the best models in existence. Our $T=1$ validation
$f_{1}$ score of $.69$ outperforms the previous best score of $.64$ which
itself far outperforms all models in existence. Our $T=2/3$ validation $f_{1}$
of $.82$ is close to the best score of $.85$ and similar to all other scores.
Our train accuracies outperform all models in existence. In fact, our train
and validation accuracies nearly match accuracies when we simply assign people
to the ground truth o-spaces. However, as discussed before, we experience a
huge decline in performance in the test set. This is due in part to over
fitting to validation and the high percentage validation examples with only
one group, but also because it is simply much more difficult to predict
multiple groups. For this reason it is slightly unfair to compare our test
accuracy to other models which were evaluated on the entire dataset, the
beginning of which was much friendlier to all algorithms. We hypothesize that
if re-run on a more representable portion of our dataset, our test accuracies
would improve to more standard values.

When weighting the data to attempt to balance the number of examples with
multiple groups with the single-group examples, we notice a slight drop in our
training and validation accuracies and a slight increase in our test
accuracies. However, this increase was not enough to make our test accuracies
rival best known models. The lack of improvement on the test data supports our
theory that the test data is simply much harder to predict than the
train/validation data and that our errors were influenced by an unlucky
train/test split.

Figure 10: Difficult to predict two nearby groups. Figure 11: Our model
sometimes shows the ability to detect two nearby groups

##  6 Conclusion and Future Work

Our model showed immense promise in its ability to outperform the best models
in its training and validation accuracies. However, due to a combination of
over-fitting and the difficult nature of the test dataset, we failed to
achieve a significant increased in performance. However, given that our model
was only trained on one dataset and the extent to which our results improved
when data was augmented, there is reason to believe that training on more data
will allow our model to generalize and replicate its impressive training and
validation performances. Thus, future development of our deep model which
incorporates more examples and is not unlucky to have its test data be the
most difficult subset of the data might significantly outperform existing
methods. Additionally, using more datasets in varied locations will allow the
extracted room features to be of more use, instead of simply indicating when
we rotated the room. One downside to a convolutional or deep model as opposed
to algorithmic methods proposed is that it becomes more difficult to work with
varied room sizes. For different room sizes, future work will need to add
padding to smaller room images so that spacial dimension are not affected. To
eventually switch to a practical deep model to be incorporated in a social
robot, many more datasets of different sizes must be used and the ability for
variable-sized rooms must be implemented.

## References

  * [1] X. Alameda-Pineda, J. Staiano, R. Subramanian, L. Batrinca, E. Ricci, B. Lepri, O. Lanz, and N. Sebe.  Salsa: A novel dataset for multimodal group behavior analysis.  IEEE transactions on pattern analysis and machine intelligence, 38(8):1707–1720, 2016.
  * [2] H. Hung and B. Kröse.  Detecting f-formations as dominant sets.  In Proceedings of the 13th international conference on multimodal interfaces, pages 231–238. ACM, 2011.
  * [3] S. Maji, L. Bourdev, and J. Malik.  Action recognition from a distributed representation of pose and appearance.  In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pages 3177–3184. IEEE, 2011.
  * [4] C. R. Qi, H. Su, K. Mo, and L. J. Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 1(2):4, 2017.
  * [5] F. Setti, O. Lanz, R. Ferrario, V. Murino, and M. Cristani.  Multi-scale f-formation discovery for group detection.  In Image Processing (ICIP), 2013 20th IEEE International Conference on, pages 3547–3551. IEEE, 2013.
  * [6] F. Setti, C. Russell, C. Bassetti, and M. Cristani.  F-formation detection: Individuating free-standing conversational groups in images.  PloS one, 10(5):e0123783, 2015.
  * [7] K. N. Tran, A. Bedagkar-Gala, I. A. Kakadiaris, and S. K. Shah.  Social cues in group formation and local interactions for collective activity analysis.  In VISAPP (1), pages 539–548, 2013.
  * [8] S. Vascon, E. Z. Mequanint, M. Cristani, H. Hung, M. Pelillo, and V. Murino.  A game-theoretic probabilistic approach for detecting conversational groups.  In Asian conference on computer vision, pages 658–675. Springer, 2014.
  * [9] M. Vázquez, A. Steinfeld, and S. E. Hudson.  Parallel detection of conversational groups of free-standing people and tracking of their lower-body orientation.  In Intelligent Robots and Systems (IROS), 2015 IEEE/RSJ International Conference on, pages 3010–3017. IEEE, 2015.


# SnAG: Scalable and Accurate Video Grounding

Fangzhou Mu*, Sicheng Mo*, Yin Li
University of Wisconsin-Madison

###### Abstract

Temporal grounding of text descriptions in videos is a central problem in
vision-language learning and video understanding. Existing methods often
prioritize accuracy over scalability — they have been optimized for grounding
only a few text queries within short videos, and fail to scale up to long
videos with hundreds of queries. In this paper, we study the effect of cross-
modal fusion on the scalability of video grounding models. Our analysis
establishes late fusion as a more cost-effective fusion scheme for long-form
videos with many text queries. Moreover, it leads us to a novel, video-centric
sampling scheme for efficient training. Based on these findings, we present
SnAG, a simple baseline for scalable and accurate video grounding. Without
bells and whistles, SnAG is 43% more accurate and 1.5$\times$ faster than
CONE, a state of the art for long-form video grounding on the challenging MAD
dataset, while achieving highly competitive results on short videos. Our code
is available at https://github.com/fmu2/snag_release.

00footnotetext: * indicates equal contribution

##  1 Introduction

Localizing moments within an untrimmed video based on text descriptions, also
known as temporal video grounding, involves joint reasoning about visual,
textual and temporal information, and thus remains a challenging problem in
video understanding. Video grounding is gaining traction amid the rapid
advances in vision-language learning, with a range of applications such as
intelligent personal assistants, human-robot interaction, and interactive
video editing.

Despite recent progress, one largely overlooked aspect of video grounding is
the _scalability_ of models with respect to video duration and the number of
text descriptions (_i.e_., queries). Previous methods (_e.g_., [70, 49]) are
primarily designed for grounding only a few queries within short video
snippets. However, the growing availability of long videos (_e.g_., on
streaming platforms) and demand to query their rich contents introduce a
paradigm shift, necessitating efficient grounding of large volumes of queries
in long videos. On latest long-form video grounding benchmarks [17, 48], these
methods struggle to scale up due to their high computational cost, and suffer
major performance degradation.

Figure 1: SnAG achieves the best accuracy and throughput simultaneously on the
MAD dataset [48] for long-form video grounding.

In this paper, we take a principled approach to studying the problem of
scalable video grounding, focusing on the design of cross-modal fusion—a key
model component that combines visual and textual information. Specifically, we
present an analysis that connects the computational cost of video grounding
models to their fusion schemes. Our analysis and empirical results challenge
the widely adopted early fusion, showing that late fusion scales better in
inference by amortizing the cost of long video processing among text queries.
Importantly, this insight further motivates the design of a novel, video-
centric sampling scheme for scalable training, without compromising grounding
accuracy.

Following these findings, we present SnAG, a scalable and accurate model for
long-form video grounding. SnAG features a minimalist, late-fusion design for
scalable inference, while supporting video-centric sampling for scalable
training. Notwithstanding its simplicity, SnAG achieves superior results
across all benchmarks designed for long-form video grounding. Without bells
and whistles, SnAG attains 13.75% R@5, tIoU=0.5 on MAD [48] with an inference
speed of 45.3 queries per second on a P100 GPU (Fig. 1), 43% more accurate and
1.5$\times$ faster than the concurrent method of CONE [18]. In the meantime,
SnAG is highly competitive on short-video benchmarks. It achieves 46.26% R@1,
tIoU=0.7 on Charades-STA [46], outperforming the previous state of the art
(SSRN [74]) by 3.61 absolute percentage points while being simpler in model
design.

Contributions. (1) We present one of the first studies to understand the
effect of fusion schemes on the computational cost of video grounding. (2) Our
analysis offers new insights and leads to a new training scheme for scalable
video grounding. (3) Based on our findings, we further deliver a simple model
that attains superior results across major benchmarks for long-form video
grounding while achieving significant efficiency gains in both training and
inference.

##  2 Related Work

Video grounding. Previous works on video grounding can be categorized based on
whether proposals are generated as moment candidates (two-stage _vs_. single-
stage), and how text queries inform action localization (cross-modal fusion).

Two-stage methods first generate temporal segments as proposals, then score
their “actionness” and optionally refine their boundaries. Early works [1, 14]
densely sample proposals using sliding windows and score them independently.
One line of work [59, 8, 58, 29, 57] subsequently conditions proposal
generation on sentence queries and/or video context to avoid dense sampling.
By contrast, another line of work [70, 53, 49, 15, 69] takes the opposite path
of enumerating all segments, yet organizes them in a 2D adjacency map to
reason about their relations.

Single-stage methods localize moments in a single shot without using
proposals, and thus are often more efficient than two-stage methods. Some
models decode moment boundaries from global features [62, 36, 25, 73, 24] or
learnable queries [22, 56, 26]. Others densely predict onset and offset
probabilities [16, 67, 44, 30, 71, 66, 37, 68], or classify and refine pre-set
anchors [6, 65, 61, 54, 31] at every moment. Most relevant to our work are
models that represent moment candidates as points [33, 63, 7, 28, 12].
Analogous to point-based object detectors [50], they classify every point in
time and regress moment boundaries relative to the points. Despite their
simplicity and efficiency, the performance of single-stage models lags behind
two-stage models on standard benchmarks made up of short videos.

Cross-modal fusion combines video and text information for video grounding.
Early models [1, 14] fuse video and text representations right before final
prediction. This simple, late-fusion approach is abandoned by subsequent
models, which turn to early fusion (_e.g_., [59, 65, 70, 63]) with
sophisticated model design (_e.g_., LSTM [6, 16, 59, 54, 30, 68], GCN [7, 49,
28], memory bank [8, 28]), feature hierarchies [53, 15, 36], losses [59, 37,
69, 24], and inference procedures [73, 71]. Early fusion is believed to
strengthen cross-modal reasoning, albeit at a price of model complexity.
Recent Transformer-based models [69, 22, 56, 3] inherit this early-fusion
design; video features and text embeddings are concatenated at input, and
attention-based feature fusion is carried out throughout the model.

Our method demonstrates a single-stage Transformer model with a simple late
fusion mechanism based on cross-attention. Late fusion allows us to amortize
the cost of video processing across many sentence queries, resulting in
scalable training and inference on long-form videos. Without bells and
whistles, our model outperforms all prior methods on long-video benchmarks,
and attains competitive performance on standard, short-video benchmarks.

Long-form video grounding. Recent efforts [17, 48] put forth new challenges
for sentence grounding in long videos. Previous methods tailored for short
videos [70, 67, 49] fall short on these benchmarks. They have no access to
long-range video context, require dense sliding window inference, and yield
unsatisfactory results. Concurrent works [18, 39] present multi-stage
approaches with sophisticated pre-processing, ranking and training procedures
for efficient long-form video grounding. However, their efficiency stems from
the restrictive assumption that moments are covered by short windows. By
contrast, SnAG offers a simple, single-stage solution with superior accuracy,
without making assumptions about moment duration.

Localizing moments in videos. Closely related to video grounding, temporal
action localization (_e.g_., [45, 34, 47]) aims to identify moments of actions
from within a closed set of action categories. Solutions to the two tasks
often evolve in tandem and share remarkably similar model design. SnAG is
inspired by our prior work [64], yet extends its design for the more
challenging task of video grounding, with the goal of scaling up video
grounding for long videos.

##  3 Scaling Up Video Grounding

Figure 2: (a) Cross-modal fusion is key to a video grounding model
$\mathcal{F}$. Models using _early fusion_ jointly encode video and sentence
query. SnAG revisits  _late fusion_ for scalable video grounding by decoupling
expensive video encoding and inexpensive query encoding. (b) Video-centric
model evaluation. With late fusion, the output of video encoder can be cached
and re-used by queries of the same video in both training and inference. (c)
Mini-batch sampling in training. Previous methods adopt _query-centric_
sampling (query → video) whereas SnAG resorts to _video-centric_ sampling
(video → many queries) for efficient training. (d) Model overview. SnAG is a
simple instantiation of late fusion and video-centric training for video
grounding. It separately encodes a video and its queries using Transformers,
applies simple cross-attention for cross-modal fusion, and decodes moments
represented as points using lightweight convolutional heads.

Our goal is to scale up video grounding for long-form videos with many
queries. Denote an input video $\mathbf{V}$ as a time-indexed sequence
$[\mathbf{v}_{t}]_{t=1}^{T}=[\mathbf{v}_{1},\mathbf{v}_{2},...,\mathbf{v}_{T}]$,
where $\mathbf{v}_{t}$ represents a short video clip centered at time
$t$.111Like prior works, we consider video features and text embeddings
extracted using pre-trained models (_e.g_., C3D [51] and BERT [11]). Yet the
formulation can be easily extended to raw video frames and text tokens.
$\mathbf{V}$ is paired with $N$ sentence queries
$\\{\mathrm{Q}_{j}\\}_{j=1}^{N}$, each describing a moment
$\mathbf{y}_{j}=(s_{j},e_{j})$ in the video.

Long-form video grounding is characterized by a great number of time steps $T$
(_e.g_., tens of thousands) and text queries $N$ (hundreds) for an input video
$\mathbf{V}$. With a large $T$, it is no longer feasible to feed the full
video $\mathbf{V}$ into a model, and thus more viable to consider a sliding
window setting. Specifically, a video $\mathbf{V}$ is further partitioned into
a set of $M$ snippets $\\{\mathbf{V}_{i=1}^{M}\\}$, each
$\mathbf{V}_{i}=[\mathbf{v}_{t}]_{t=t_{i}}^{t_{i}+T_{w}}$ taken from a local
time window of size $T_{w}$ that may overlap with others. We consider the
learning of a function $\mathcal{F}$, realized in a deep network, to predict
the start time $s_{j}$ and end time $e_{j}$ of the moment $\mathbf{y}_{j}$
given a snippet $\mathbf{V}_{i}$ and a query $\mathrm{Q}_{j}$.

###  3.1 Background: Cross-Modal Fusion

We review cross-modal fusion — a key design choice of $\mathcal{F}$, and
define the notation used for the rest of the paper.

Early fusion. A straightforward design is to fuse information from a video
snippet $\mathbf{V}_{i}$ and a text query $\mathrm{Q}_{j}$ at input (Fig.
2(a)). This early fusion strategy is widely adopted in prior works [65, 70,
63, 69, 22, 56], and can be written as

$\mathcal{F}_{E}(\mathbf{V}_{i},\mathrm{Q}_{j})\mapsto\mathbf{y}_{j},$ (1)

where $\mapsto$ defines any necessary post-processing of the model outputs
(_e.g_., re-scaling, non-maximum suppression) in order to decode the moment
$\mathbf{y}_{j}$.

Late fusion. Another possible design is to first embed the video snippet
$\mathbf{V}_{i}$ and text query $\mathrm{Q}_{j}$, then fuse their features for
grounding (Fig. 2(a)). This late fusion strategy follows

$\mathcal{F}_{L}(g(\mathbf{V}_{i}),h(\mathrm{Q}_{j}))\mapsto\mathbf{y}_{j},$
(2)

where $g(\cdot)$ is the video encoder and $h(\cdot)$ the text encoder. With
slight abuse of the symbols, $\mathcal{F}_{L}$ here denotes the rest part of
the grounding model, including feature fusion. The key is to keep
$\mathcal{F}_{L}$ lightweight, _i.e_., its cost should be a minor portion in
comparison to the rest of the model.

Design choice. The conventional wisdom favors the design of early fusion for
video grounding. Early fusion is believed to offer better interactions between
visual and textual features, resulting in strong empirical results. In what
follows, we analyze the computational cost of early and late fusion during
inference and training, and show that proper design of late fusion allows
significantly improved scalability in video duration ($M$) and number of text
queries ($N$).

###  3.2 Scalable Inference

We start by analyzing the computational cost of early _vs_. late fusion in
inference. Denote the computational cost of a function by
$\mathcal{C}(\cdot)$. For example, $\mathcal{C}(g)$ is the cost of the video
encoder $g(\cdot)$, and $\mathcal{C}(h)$ the cost of the text encoder
$h(\cdot)$.

Computational cost. We make two key observations regarding the computational
cost of a video grounding model: (a) a lightweight $\mathcal{F}_{L}$ is used
for late fusion, thus
$\mathcal{C}(\mathcal{F}_{L})\ll\mathcal{C}(g)+\mathcal{C}(h)$; and (b) the
video encoder $g(\cdot)$ has a much higher cost than the text encoder, thus
$\mathcal{C}(g)\gg\mathcal{C}(h)$.

The first observation stems from the design of late fusion. To justify the
second observation, it is important to notice that $\mathcal{C}(g)$ scales
w.r.t. the length of input video snippet (_i.e_., the local window size
$T_{w}$), whereas $\mathcal{C}(h)$ remains constant due to limited sequence
length of text queries. For example, a realization of $g(\cdot)$ using a
vanilla Transformer has a complexity of $\mathcal{O}(T_{w}^{2})$. Thus, using
a larger local window size naturally leads to
$\mathcal{C}(g)\gg\mathcal{C}(h)$.

Cost analysis: early _vs_. late fusion. Evaluating an early-fusion model on a
video with $M$ snippets and $N$ queries has a cost of $MN\
\mathcal{C}(\mathcal{F}_{E})$, whereas late fusion has a cost of $MN\
\mathcal{C}(\mathcal{F}_{L})+M\mathcal{C}(g)+N\mathcal{C}(h)$. This is because
an early-fusion model has to compare each snippet to every query, yet late
fusion allows re-using video features for all queries of the video (_i.e_.,
_video-centric_ evaluation, see Fig. 2(b)).

Let us assume that the cost of both models can be controlled by setting
$\mathcal{C}(\mathcal{F}_{E})=\mathcal{C}(\mathcal{F}_{L})+\mathcal{C}(g)+\mathcal{C}(h)$,
_i.e_., evaluating either model on a pair of video snippet and sentence query
has the same cost (post-processing excluded). With
$\mathcal{C}(\mathcal{F}_{L})=\alpha(\mathcal{C}(g)+\mathcal{C}(h))$ for some
small $\alpha$, the ratio between the cost of early and late fusion in
inference is

$\displaystyle\small\mathbf{R_{\text{inf}}}$
$\displaystyle=\frac{\mathcal{C}(\text{inference with early
fusion})}{\mathcal{C}(\text{inference with late fusion})}$ (3)
$\displaystyle=\frac{MN\ \mathcal{C}(\mathcal{F}_{L})+MN\ \mathcal{C}(g)+MN\
\mathcal{C}(h)}{MN\ \mathcal{C}(\mathcal{F}_{L})+M\ \mathcal{C}(g)+N\
\mathcal{C}(h)}$ (4)
$\displaystyle=\frac{MN(1+\alpha)}{MN\alpha+M\mathbf{R}_{g/(g+h)}+N(1-\mathbf{R}_{g/(g+h)})}$
(5)
$\displaystyle\approx\frac{1+\alpha}{\frac{1}{N}+\alpha}\to\begin{cases}N,&\alpha\to
0\\\ 1+\frac{1}{\alpha},&N\to\infty\end{cases},$ (6)

where $\mathbf{R}_{g/(g+h)}=\mathcal{C}(g)/(\mathcal{C}(g)+\mathcal{C}(h))$,
and the approximation in Eq. D is a result of $\mathbf{R}_{g/(g+h)}\approx 1$
with $\mathcal{C}(g)\gg\mathcal{C}(h)$. Eq. D reveals that early fusion is $N$
times the cost of late fusion as snippet length grows ($\alpha\to 0$), and is
particularly inefficient for videos with many queries ($N\to\infty$).

###  3.3 Scalable Training

We now show that late fusion further unlocks a novel training scheme that
scales well for long-form video grounding.

Training and mini-batch sampling. Given a set of paired videos, text queries,
and their moments $\\{\mathbf{V}^{(k)},\mathrm{Q}^{(k)},\mathbf{y}^{(k)}\\}$,
training a model $\mathcal{F}$ amounts to minimizing a loss function
$\mathcal{L}$ over mini-batches $B$ sampled from the dataset:

$\small\Sigma_{(k,i,j)\in B}\ \
\mathcal{L}\left(\mathcal{F}\left(\mathbf{V}^{(k)}_{i},\mathrm{Q}^{(k)}_{j}\right),\mathbf{y}^{(k)}_{j}\right).$
(7)

A key design largely overlooked in video grounding is the mini-batch sampling
scheme during training. With multiple queries associated with a video snippet,
a combination of late fusion and proper sampling scheme allows us to re-use
video representations in training as is done in inference.

Query-centric sampling. Virtually all prior works sample mini-batches
uniformly at random from all pairs of valid video snippets and text
queries.222A video snippet without any query is often discarded during
training. This can be interpreted as ancestral sampling: first draw a text
query (and its moment), then select a video snippet covering the moment:

$\small\begin{split}&\mathrm{Q}_{j^{\prime}}^{(k^{\prime})},\mathbf{y}_{j^{\prime}}^{(k^{\prime})}\sim
U\left(\\{(\mathrm{Q}_{j}^{(k)},\mathbf{y}_{j}^{(k)})\\}_{j\times
k}\right),\\\ &\mathbf{V}_{i^{\prime}}^{(k^{\prime})}\sim
U\left(\\{(\mathbf{V}_{i}^{(k^{\prime})},\mathrm{Q}_{j^{\prime}}^{(k^{\prime})},\mathbf{y}_{j^{\prime}}^{(k^{\prime})})_{+}\\}\right),\\\
\end{split}$ (8)

where $U(\cdot)$ denotes a discrete uniform distribution and
$\\{(\mathbf{V}_{i}^{(k^{\prime})},\mathrm{Q}_{j}^{(k^{\prime})},\mathbf{y}_{j^{\prime}}^{(k^{\prime})})_{+}\\}$
is the union of all valid triplets: a moment
$\mathbf{y}_{j^{\prime}}^{(k^{\prime})}$ of a text query
$\mathrm{Q}_{j^{\prime}}^{(k^{\prime})}$ in a video snippet
$\mathbf{V}^{(k^{\prime})}_{i}$.

This sampling process is repeated $|B|$ times to construct a mini-batch, with
$|B|$ the batch size. With a large dataset, the resulting mini-batch will
contain $|B|$ snippet-query pairs, each likely from a different video.

Video-centric sampling. To speed up the training of late fusion models, we
propose video-centric sampling that considers every video snippet
$\mathbf{V}_{i}^{(k)}$ as a sample. Specifically, we first draw a video based
on its importance score, then sample a snippet within the video, and finally
select _multiple_ text queries associated with the snippet. This is defined as
the following ancestral sampling:

$\small\begin{split}&k^{\prime}\sim
P(k)=\frac{M^{(k)}_{+}}{\Sigma_{k}M^{(k)}_{+}}\quad i^{\prime}\sim
P(i;k^{\prime})=\frac{N^{(k^{\prime})}_{i}}{\Sigma_{k}N^{(k^{\prime})}_{i}}\\\
&\\{\mathrm{Q}_{j^{\prime}}^{(k^{\prime})},\mathbf{y}_{j^{\prime}}^{(k^{\prime})}\\}^{B_{q}}\sim
U\left(\\{(\mathbf{V}_{i^{\prime}}^{(k^{\prime})},\mathrm{Q}_{j}^{(k^{\prime})},\mathbf{y}_{j}^{(k^{\prime})})_{+}\\}\right),\end{split}$
(9)

where $M^{(k)}_{+}$ denotes the number of valid snippets in a video
$\mathbf{V}^{(k)}$, and $N^{(k^{\prime})}_{i}$ the number of queries in a
snippet $\mathbf{V}^{(k^{\prime})}_{i}$. Importantly, a total of $B_{q}$
queries are drawn for the selected video snippet
$\mathbf{V}_{i^{\prime}}^{(k^{\prime})}$.

This sampling process is repeated $|B|/B_{q}$ times for a mini-batch, leading
to $|B|/B_{q}$ video snippets, each likely from a different video yet with
multiple ($B_{q}$) text queries. This mini-batch thus allows us to re-use the
video representations for the sampled snippets.

Cost analysis: query-centric _vs_. video-centric sampling. Fig. 2(c)
illustrates the key difference between query-centric and video-centric
sampling. Similar to our analysis in Section 3.2, it is easy to show that late
fusion combined with video-centric training enables substantial speedup for
long snippets ($\alpha\to 0$) and large $B_{q}$:

$\displaystyle\small\mathbf{R_{\text{train}}}$
$\displaystyle=\frac{\mathcal{C}(\text{late fusion + video-centric
training})}{\mathcal{C}(\text{late fusion + query-centric training})}$ (10)
$\displaystyle\approx\frac{1+\alpha}{\frac{1}{B_{q}}+\alpha}\to\begin{cases}B_{q},&\alpha\to
0\\\ 1+\frac{1}{\alpha},&B_{q}\to\infty\end{cases}.$ (11)

###  3.4 Scalable Video Grounding: An Instantiation

We now present an instantiation with late fusion and video-centric training
for scalable video grounding. Our instantiation, dubbed SnAG, is shown in Fig.
2(d). SnAG is a single-stage Transformer model where every time step
represents a moment candidate. It combines (a) a multi-scale Transformer-based
video encoder; (b) a Transformer-based text encoder; (c) cross-attentions for
late fusion; and (d) convolutional heads for moment decoding.

Video encoder $g$. We use the ActionFormer backbone [64] as our video encoder,
given its superior performance on the closely related task of temporal action
localization. It transforms input video features into a multi-scale
representation
$[\mathbf{Z}^{(l)}]_{l=1}^{L}=[\mathbf{Z}^{(1)},\mathbf{Z}^{(2)},...,\mathbf{Z}^{(L)}]$
using Transformer layers with local self-attentions [10]. We refer readers to
the original paper [64] for more details on model architecture.

Text encoder $h$. Our text encoder is a vanilla Transformer. It takes a query
$\mathrm{Q}$ with $K$ tokens $[\mathrm{q}_{1},...,\mathrm{q}_{K}]$ and outputs
their embeddings $\mathbf{E}=[\mathbf{e}_{1},...\mathbf{e}_{K}]$. This
computation is light compared to video encoding as $K\ll T_{w}$ for long
videos.

Cross-modal late fusion $\mathcal{F}_{L}$. Our model uses a simple variant of
cross-attention for fusion. Specifically, it modulates
$[\mathbf{Z}^{(l)}]_{l=1}^{L}$ with an affine transformation given by
$\mathbf{E}$:

$\displaystyle(\mathbf{W},\mathbf{B})^{(l)}$
$\displaystyle=\mathrm{MCA}(\mathrm{LN}(\mathbf{Z}^{(l)}),\mathrm{LN}(\mathbf{E})),$
(12) $\displaystyle\tilde{\mathbf{Z}}^{(l)}$
$\displaystyle=\mathbf{W}^{(l)}\odot\mathbf{Z}^{(l)}+\mathbf{B}^{(l)},$
$\displaystyle\mathbf{X}^{(l)}$
$\displaystyle=\tilde{\mathbf{c}}\odot\mathrm{MLP}(\mathrm{LN}(\tilde{\mathbf{Z}}^{(l)}))+\tilde{\mathbf{Z}}^{(l)},$

where $\mathrm{LN}$ is Layer Normalization [2], $\mathrm{MLP}$ is Multi-Layer
Perceptron, and $\mathrm{MCA}$ is Multi-head Cross-Attention [52], whose
output is split into affine weights $\mathbf{W}^{(l)}$ and biases
$\mathbf{B}^{(l)}$. $\tilde{\mathbf{c}}$ are learnable per-channel scales, and
$\mathbf{X}^{(l)}$ the modulated video representations prior to moment
decoding.

Moment decoding. After fusion, our model decodes each time step into a moment
candidate, similar to previous video grounding models [33, 63, 7, 28].
Briefly, a classification head predicts a score $p_{t}^{(l)}$ for all
$\mathbf{x}_{t}^{(l)}$ within $[\mathbf{X}^{(l)}]_{l=1}^{L}$, and a regression
head predicts the normalized distances from $t$ to the moment boundaries
$(d_{t}^{s},d_{t}^{e})$ if $\mathbf{x}_{t}^{(l)}$ is classified positive. Both
heads are lightweight, each comprising three 1D convolutions. The decoded
moment $\hat{\mathbf{y}}=(\hat{s},\hat{e})$ is given by

$\begin{gathered}(t,l)=\operatorname*{arg\,max}_{t,l}p_{t}^{(l)},\\\
\hat{s}=2^{l-1}\cdot(t-d_{t}^{s}),\quad\hat{e}=2^{l-1}\cdot(t+d_{t}^{e}).\end{gathered}$
(13)

At inference time, we further apply Soft-NMS [4] to merge overlapping moment
predictions.

Training and loss function. We use video-centric training as described in
Section 3.3. The loss function for a pair of video snippet and text query is
the sum of two terms: a Focal loss [27] $\mathcal{L}_{cls}$ for moment
classification and a Distance-IoU loss [72] $\mathcal{L}_{reg}$ defined on
positive time steps for distance regression. Formally,

$\small\mathcal{L}=\frac{1}{C}(\mathcal{L}_{cls}+\lambda_{reg}\mathds{1}_{t}^{(l)}\mathcal{L}_{reg}),$
(14)

where $\mathds{1}_{t}^{(l)}$ is an indicator function that evaluates to one on
positive time steps, $C=\sum_{t,l}\mathds{1}_{t}^{(l)}$ is the total number of
positive samples within the snippet, and $\lambda_{reg}$ balances the two loss
terms. Center sampling [64] is used for selecting positive time steps around
the center of a target moment.

##  4 Experiments

Figure 3: Visualization of dataset statistics. Circle radius is in proportion
to average number of queries per video. Long-video benchmarks (Ego4D-NLQ, MAD
and TACoS) consist of longer videos with more queries and exhibit low moment
coverage compared to short-video benchmarks (ANet-Captions and Charades).

| Model | R@1 | R@5
---|---|---|---
0.3 | 0.5 | Avg | 0.3 | 0.5 | Avg
SF + BERT | 2D-TAN [70] | 5.04 | 2.02 | 3.53 | 12.89 | 5.88 | 9.39
VSLNet [67] | 5.45 | 3.12 | 4.29 | 10.74 | 6.63 | 8.69
CONE [18] | 10.40 | 5.03 | 7.72 | 22.74 | 11.87 | 17.31
SOONet [39] | 8.00 | 3.76 | 5.88 | 22.40 | 11.09 | 16.75
SnAG (video-centric) | 9.83 | 6.83 | 8.33 | 27.93 | 19.27 | 23.60
SnAG (query-centric) | 10.43 | 7.15 | 8.79 | 27.59 | 19.33 | 23.46
EgoVLP | VSLNet [67] | 10.84 | 6.81 | 8.83 | 18.84 | 13.45 | 16.15
CONE [18] | 14.15 | 8.18 | 11.17 | 30.33 | 18.02 | 24.18
SnAG (video-centric) | 15.72 | 10.78 | 13.25 | 38.39 | 27.44 | 32.92
SnAG (query-centric) | 15.87 | 11.26 | 13.57 | 38.26 | 27.16 | 32.71

Table 1: Results on Ego4D-NLQ. SnAG outperforms all baselines. Best results
are in bold and second best underlined.

Our main experiments include extensive comparisons of SnAG to strong baselines
on five challenging benchmarks for video grounding. We further provide
ablations on cross-modal fusion and analyze the efficiency of SnAG in both
training and inference. More results are in the supplement.

Datasets. We categorize five benchmark datasets into two groups based on their
video and query statistics (Fig. 3).

Long videos, many queries. Ego4D-NLQ [17] 333All experiments are conducted
using the Ego4D _v1_ dataset. is a large collection of egocentric videos on
daily human activities. The videos are 3.5 to 20 minutes long with an average
of 11.6 queries. MAD [48] contains 1.2K hours of movies with 384K queries
transcribed from audio description. The videos are 47 to 202 minutes long.
TACoS [43] is a conventional benchmark with 10.1 hours of cooking videos, with
an average of 143.5 queries per video. These datasets expose the scalability
challenges that motivate our work and thus are the main focus of our
experiments.

Model | R@1 | R@5 | R@10 | R@50
---|---|---|---|---
0.1 | 0.3 | 0.5 | 0.1 | 0.3 | 0.5 | 0.1 | 0.3 | 0.5 | 0.1 | 0.3 | 0.5
CLIP [42] | 6.57 | 3.13 | 1.39 | 15.05 | 9.85 | 5.44 | 20.26 | 14.13 | 8.38 | 37.92 | 28.71 | 18.80
VLG-Net [49] | 3.64 | 2.76 | 1.65 | 11.66 | 9.31 | 5.99 | 17.39 | 14.56 | 9.77 | 39.78 | 34.27 | 24.93
Moment-DETR [22] | 0.31 | 0.24 | 0.16 | 1.52 | 1.14 | 0.28 | 2.79 | 2.06 | 1.20 | 11.08 | 7.97 | 4.71
CONE [18] | 8.90 | 6.87 | 4.10 | 20.51 | 16.11 | 9.59 | 27.20 | 21.53 | 12.82 | 43.36 | 34.73 | 20.56
SOONet [39] | 11.26 | 9.00 | 5.32 | 23.21 | 19.64 | 13.14 | 30.36 | 26.00 | 17.84 | 50.32 | 44.78 | 32.59
SnAG (video-centric) | 10.28 | 8.46 | 5.55 | 24.42 | 20.60 | 13.75 | 32.23 | 27.50 | 19.00 | 52.28 | 46.68 | 35.24
SnAG (query-centric) | 10.18 | 8.37 | 5.44 | 24.30 | 20.35 | 13.51 | 32.05 | 27.25 | 18.75 | 52.05 | 46.53 | 34.83

Table 2: Results on MAD. SnAG outperforms all baselines. Best results are in
bold and second best underlined. Figure 4: Visualization of moment
predictions. SnAG can (a) comprehend complex text queries with multiple
objects and actions; (b) reason about temporal ordering of events.

Short videos, few queries. Charades-STA [46] is an action recognition dataset
later annotated for video grounding [14]. The videos are $\sim$30 seconds long
with $\sim$2.4 queries per video. ActivityNet-Captions [21] was collected for
dense video captioning and later repurposed for video grounding. The videos
are two minutes long on average with a small set of $\sim$3.65 queries.
Several works [38, 60] advise against using these datasets as benchmarks for
the temporal biases in their moment annotations. Nevertheless, we report
results on these datasets for completeness.

Baselines. On TACoS, Charades and ANet-Captions, we compare to strong
baselines including two-stage [59, 58, 29] and single-stage [53, 73, 31]
models with various fusion operations (LSTM [54, 30], GCN [49, 28], co-
attention [25, 71], _etc_.). On Ego4D and MAD, we compare to their official
baselines (2D-TAN [70] and VSLNet [67] for Ego4D, CLIP [42] and VLG-Net [49]
for MAD) as well as the concurrent method of CONE [18] and SOONet [39] that
similarly address scalable video grounding. We discuss additional baselines in
the supplement.

Evaluation metric. We report Recall$@k$ at various temporal intersection-over-
union threshold $\theta$ (R@$k$, tIoU=$\theta$) for all datasets. It measures
the percentage of sentence queries with at least one of the top-$k$ moment
predictions whose temporal overlap with the ground-truth moment exceeds
$\theta$.

###  4.1 Long Videos, Many Queries

Experiment setup. For Ego4D, We consider (1) SlowFast [13] video features with
BERT [11] text features, and (2) EgoVLP [41] video and text features. We train
on the train split and report R@{1, 5}, tIoU={0.3, 0.5} on the val split. For
MAD, we use CLIP [42] features for both videos and text. We train on the train
split and report R@{1, 5, 10, 50}, tIoU={0.1, 0.3, 0.5} on the test split. For
TACoS, we use C3D video features [51] and GloVe [40] for sentence embeddings.
We train on the train split and report R@{1, 5}, tIoU={0.5, 0.7} on the test
split. We report results using both video-centric and query-centric training
on all datasets.

Results on Ego4D-NLQ (Table 1). SnAG sets the new state of the art, achieving
mean R@1 / R@5 scores of 8.79% / 23.60% with SlowFast/BERT features, and
13.57% / 32.92% with EgoVLP features, doubling or even tripling the results of
the official baselines [70, 67]. Compared to the strong baselines of CONE [18]
and SOONet [39], SnAG improves mean R@1 and R@5 respectively by more than 1.05
and 6.29 absolute percentage points without taking advantage of external
features.444CONE [18] leverages external CLIP [42] features in several steps.
Further, SnAG demonstrates a larger performance gain on the more stringent
tIoU threshold of 0.5, indicating more precise moment localization.

Results on MAD (Table 2). SnAG sets new records on the challenging MAD
dataset. It improves over CLIP [42] and VLG-Net [49] on mean R@1 / R@5 at
tIoU=0.5 by 3.90 / 7.76 absolute percentage points. In the meantime, it
outperforms CONE [18] and SOONet [39], with larger gain for more stringent
tIoU thresholds. Further, SnAG achieves stronger results as input snippet
length increases, reaching the best accuracy with $T_{w}=$16384 (Supplement
Fig. A). This scaling behavior highlights the strength of our scalable model
design for long-form video grounding.

Model | R@1 | R@5
---|---|---
0.3 | 0.5 | 0.3 | 0.5
(a) Fusion stage (early _vs_. late fusion)
No fusion | 10.85 | 7.85 | 46.46 | 31.19
SnAG-E | 53.34 | 44.01 | 79.56 | 69.41
SnAG-EL | 54.84 | 44.54 | 81.13 | 70.11
SnAG-M | 51.41 | 40.79 | 77.56 | 65.71
SnAG (ours) | 55.14 | 44.94 | 81.10 | 70.66
(b) Fusion operation (global _vs_. token-wise)
Cat | 49.39 | 39.24 | 77.31 | 65.36
Add | 51.91 | 41.81 | 77.98 | 66.51
Scale | 55.01 | 43.96 | 80.63 | 70.11
Add + Scale | 53.36 | 43.09 | 80.43 | 69.58
XAttn | 54.84 | 45.06 | 79.96 | 70.08
XAttn-affine (ours) | 55.14 | 44.94 | 81.10 | 70.66

Table 3: Ablations results on TACoS. (a) Late fusion yields comparable results
to early fusion (E, EL, M); (b) Fusion with token-wise sentence embeddings
(XAttn) produces better results than fusion with global sentence embeddings
(Cat, Add, Scale).

Results on TACoS (Table 4 (left)). SnAG achieves an R@1 / R@5 of 44.86% /
70.66% at tIoU=0.5, outperforming the strongest baseline of MATN [69] by a
significant margin (+7.5% R@1, +12.1% R@5). This confirms that our simple
single-stage design with late fusion is a strong alternative to multi-stage
models with sophisticated fusion mechanisms. Importantly, our scalable design
enables learning from full-resolution video features, whereas the baselines
often aggressively reduce feature resolution to contain their training and
inference cost on long-form videos. We hypothesize that full-resolution video
features capture detailed event dynamics and are key to the strong performance
of SnAG.

Result visualization. We visualize predictions of SnAG and the baselines on
TACoS and Ego4D-NLQ videos.555MAD videos are not publicly available due to
copyright issues. In Fig. 4(a), a long query describes the complex interplay
between two objects (pan and stove) and three actions (“take from”, “place
on”, and “turn on”). VLG-Net [49] recognized both objects yet none of the
actions, while 2D-TAN [70] identified one action (“take from”) but at the same
time missed one object (pan). By contrast, SnAG followed the query semantics
and produced accurate localization. The other query in Fig. 4(b) tests the
understanding of temporal context. SnAG leveraged the keyword “ _before_ ” to
identify the correct moment before the action (“pick up”), whereas VSLNet [67]
only attended to the object (can), thus mistakenly predicting a moment _after_
the action.

###  4.2 Short Videos, Few Queries

Method | _TACoS (C3D)_ | _Charades-STA (I3D)_ | _Charades-STA (C3D)_ | _ActivityNet-Captions (C3D)_
---|---|---|---|---
R@1 | R@5 | R@1 | R@5 | R@1 | R@5 | R@1 | R@5
0.3 | 0.5 | 0.3 | 0.5 | 0.5 | 0.7 | 0.5 | 0.7 | 0.5 | 0.7 | 0.5 | 0.7 | 0.5 | 0.7 | 0.5 | 0.7
CTRL ICCV 17 [14] | 18.32 | 13.30 | 36.69 | 25.42 | - | - | - | - | 23.63 | 8.89 | 58.92 | 29.52 | - | - | - | -
QSPN AAAI 19 [59] | - | - | - | - | - | - | - | - | 35.60 | 15.80 | 79.40 | 45.40 | 27.70 | 13.60 | 59.20 | 38.30
SCDM NeurIPS 19 [61] | 27.64 | 23.27 | 40.06 | 33.49 | 54.92 | 34.26 | 76.50 | 60.02 | - | - | - | - | 36.90 | 20.28 | 66.84 | 42.92
2D-TAN AAAI 20 [70] | 45.61 | 35.77 | 69.11 | 57.31 | 56.64 | 36.21 | 89.14 | 61.13 | 41.10 | 23.25 | 81.53 | 48.55 | 46.16 | 29.21 | 78.80 | 60.85
CBP AAAI 20 [54] | 27.31 | 24.79 | 43.64 | 37.40 | - | - | - | - | 36.80 | 18.87 | 70.94 | 50.19 | 35.76 | 17.80 | 65.89 | 46.20
DRN CVPR 20 [63] | - | 23.17 | - | 33.36 | 53.09 | 31.75 | 89.06 | 60.05 | 45.40 | 26.40 | 88.01 | 55.38 | 45.45 | 24.36 | 77.97 | 50.30
BPNet AAAI 21 [58] | 25.96 | 20.96 | - | - | 50.75 | 31.64 | - | - | 38.25 | 20.51 | - | - | 42.07 | 24.69 | - | -
CPNet AAAI 21 [25] | 42.61 | 28.29 | - | - | 60.27 | 38.74 | - | - | 40.32 | 22.47 | - | - | 40.56 | 21.63 | - | -
SMIN CVPR 21 [53] | 48.01 | 35.24 | 65.18 | 53.36 | 64.06 | 40.75 | 89.49 | 68.09 | - | - | - | - | 48.46 | 30.34 | 81.16 | 62.11
CBLN CVPR 21 [30] | 38.98 | 27.65 | 73.12 | 46.24 | 61.13 | 38.22 | 90.33 | 61.69 | 47.94 | 28.22 | 88.20 | 57.47 | 48.12 | 27.60 | 79.32 | 63.41
CPN CVPR 21 [71] | 47.69 | 36.33 | - | - | 51.07 | 31.54 | - | - | - | - | - | - | 45.10 | 28.10 | - | -
DeNet CVPR 21 [73] | - | - | - | - | 59.70 | 38.52 | 91.24 | 66.83 | - | - | - | - | 43.79 | - | 74.13 | -
MATN CVPR 21 [69] | 48.79 | 37.57 | 67.63 | 57.91 | - | - | - | - | - | - | - | - | 48.02 | 31.78 | 78.02 | 63.18
VLG-Net ICCVW 21 [49] | 45.46 | 34.19 | 70.38 | 56.56 | - | - | - | - | - | - | - | - | 46.32 | 29.82 | 77.15 | 63.33
APGN EMNLP 21 [29] | 40.47 | 27.86 | 59.98 | 47.12 | 62.58 | 38.86 | 91.24 | 62.11 | 48.20 | 29.37 | 89.05 | 58.49 | 48.92 | 28.64 | 78.87 | 63.19
IA-Net EMNLP 21 [31] | 37.91 | 26.27 | 57.62 | 46.39 | 61.29 | 37.91 | 89.78 | 62.04 | - | - | - | - | 48.57 | 27.95 | 78.99 | 63.12
RaNet EMNLP 21 [15] | 43.34 | 33.54 | 67.33 | 55.09 | 60.40 | 39.65 | 89.57 | 64.54 | - | - | - | - | 45.59 | 28.67 | 75.93 | 62.97
MGSL-Net AAAI 22 [31] | 42.54 | 32.27 | 63.39 | 50.13 | 63.98 | 41.03 | 93.21 | 63.85 | - | - | - | - | 51.87 | 31.42 | 82.60 | 66.71
MMN AAAI 22 [55] | 39.24 | 26.17 | 62.03 | 47.39 | - | - | - | - | - | - | - | - | 48.59 | 29.26 | 79.50 | 64.76
SSRN EMNLP 22 [74] | 45.10 | 34.33 | 65.26 | 51.85 | 65.59 | 42.65 | 94.76 | 65.48 | 50.39 | 31.42 | 90.68 | 59.94 | 54.49 | 33.15 | 84.72 | 68.48
G2L ICCV 23 [23] | 42.74 | 30.95 | 65.83 | 49.86 | - | - | - | - | - | - | - | - | 51.68 | 33.35 | 81.32 | 67.60
SnAG (video-centric) | 56.44 | 44.86 | 81.15 | 70.66 | 64.62 | 46.26 | 92.55 | 71.94 | 51.72 | 33.52 | 92.55 | 64.11 | 48.55 | 30.56 | 81.71 | 63.41
SnAG (query-centric) | 55.01 | 44.51 | 80.68 | 70.13 | 65.13 | 45.91 | 92.80 | 71.75 | 51.10 | 31.77 | 90.22 | 62.15 | 47.55 | 29.65 | 81.55 | 62.21

Table 4: Results on TACoS, Charades-STA and ActivityNet-Captions. SnAG
outperforms all baselines on TACoS and Charade-STA by a large margin, while
being highly competitive on ActivityNet-Captions. Best results are in bold and
second best underlined.

Experiment setup. We extract video features using both C3D [51] (pretrained on
Sports-1M [19]) and I3D [5] (pretrained on Kinetics [20]) for Charades-STA,
and use the official C3D features for ActivityNet-Captions. We use GloVe word
embeddings [40] to represent sentence queries for both datasets. We train on
their respective train splits, report R@{1, 5}, tIoU={0.5, 0.7} on the test
split of Charades-STA, and R@{1, 5}, tIoU={0.3, 0.5} on the val_2 split of
ActivityNet-Captions. Following standard practice, we resize all video
features to a uniform length of 256.

Results on Charades-STA (Table 4 (middle)). SnAG outperforms all previous
methods by a wide margin. It obtains 33.52% R@1 and 64.11% R@5 at tIoU=0.7
with C3D features, 2.10 and 4.52 absolute percentage points higher than the
latest SSRN [74] baseline. It further attains 46.26% R@1 and 71.94% R@5 at
tIoU=0.7 with I3D features, setting new state of the art on this competitive
benchmark.

Results on ActivityNet-Captions (Table 4 (right)). SnAG achieves competitive
results, reaching an R@1 / R@5 score of 30.56% / 63.41% at tIoU=0.7. It is
only outperformed by a few latest models [28, 74, 23] while being much
simpler.

###  4.3 Ablation Studies

We conduct extensive ablations on TACoS to study key design choices for cross-
modal fusion.

Early _vs_. late fusion. Does late fusion enable better scalability at the
expense of accuracy? We study three variants of SnAG with common early-fusion
designs: (1)  _SnAG-E_ with fusion before the pyramid; (2)  _SnAG-EL_ with
fusion before and after the pyramid; (3)  _SnAG-M_ with repeated fusion within
the pyramid. Table 3 confirms that early and late fusion yield comparable
results.

Figure 5: (a) Model capacity. T, E, M denotes TACoS, Ego4D and MAD,
respectively (also in (b)). Shaded bar: % of parameters, Colored bar: % of
MACs. SnAG places more parameters and compute in video encoder for less
expensive per-query evaluation (fusion + decoding). (b, c) Video-centric _vs_.
query-centric inference (b) and training (c). SnAG saves up to 50% training
time, 40% GPU memory and 80% test time, and delivers up to 40% faster
convergence relative to query-centric training / inference. (d) Effect of
$B_{q}$ on training efficiency and test accuracy on MAD. Training on MAD is
faster and takes less GPU memory as $B_{q}$ grows, while the test accuracy is
unaffected.

Choice of fusion operation. We study fusion operations used by strong
baselines, including point-wise addition, multiplication, and concatenation of
video representations with global sentence embeddings, as well as vanilla
cross-attention with token-level sentence embeddings. According to Table 3,
token-level embeddings produce better results, and our cross-attention variant
further boosts performance.

###  4.4 Efficiency Analysis

Finally, we conduct an efficiency analysis of our method.

Breakdown of model capacity. A key design choice of SnAG is to carefully
balance the model capacity among its video encoder, text encoder, and moment
decoder including the fusion module and the localization heads. For a detailed
breakdown of model capacity, we measure their respective parameter size and
MACs in percentage. Fig. 5(a) indicates that SnAG’s video encoder absorbs a
significant portion of parameters and compute, such that per-query evaluation
(_i.e_., text encoding, fusion, and decoding) occupies $\sim$50% of the total
computation. This design is naturally in favor of video-centric training and
inference, where the costly video encoding is shared among many queries.

Video-centric _vs_. query-centric training. Fig. 5(b) presents the training
time and GPU memory usage of video-centric training relative to conventional
query-centric training. The results show that the amount of saving correlates
with snippet length and number of queries. Video-centric training is thus
preferred for long videos with many queries.

$B_{q}$ in video-centric training. We further study how $B_{q}$, the number of
text query samples per video, trades off accuracy for efficiency in video-
centric training. On MAD, we fix the batch size $|B|=32$ for a mini-batch $B$,
vary $B_{q}$ between 1 and $|B|$, and report training time, GPU memory usage
and test accuracy relative to query-centric training (_i.e_., $B_{q}=1$). Fig.
5(d) reveals that test accuracy is unaffected even with a large $B_{q}=32$
(_i.e_., a single video per mini-batch), while training time and GPU memory
usage drop by a significant 55% and 42%, approaching the upper bound by our
analysis.

Video-centric _vs_. query-centric inference. We now ask to what extent the re-
use of video representations enabled by SnAG’s late fusion design accelerates
inference. Fig. 5(b) presents the percentage of time spent by our video-
centric inference scheme relative to query-centric inference, where the full
model is evaluated on each and every video-query pair. Consistent with earlier
findings on video-centric training, longer videos with more queries result in
more saving in inference time. On the challenging MAD dataset, this alone
brings a major reduction (65%) in test time.

Inference speed. With window filtering [18], inference of SnAG on the $val$
split of MAD takes 12 minutes on a single P100 GPU. This is 33% faster than
CONE [18] ($\sim$18 mins) and 63% faster than SOONet [39] ($\sim$32
mins).666SOONet reports an inference time of 8.2 mins on a single A100 GPU.

##  5 Conclusion

In this paper, we presented a simple and accurate model towards scaling up
video grounding. We provided extensive analysis on the complexity of video
grounding models from the perspective of cross-modal fusion, and argued that
late fusion is key to scalability in video duration and query size. Our model,
SnAG, is a simple instantiation of late fusion with minimalist design. It
achieves superior accuracy and efficiency on a spectrum of challenging
benchmarks and compares favorably to strong baselines including those
dedicated to efficient inference on long videos. We hope our analysis and
model design can shed light on video grounding and more broadly scalable video
understanding.

Acknowledgement: This work was partially supported by NIH grant R01OH012313,
by grants from McPherson Eye Research Institute and VCGRE at UW Madison, and
by the Army Research Lab under contract number W911NF-2020221.

## References

  * [1] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell.  Localizing moments in video with natural language.  In ICCV, 2017.
  * [2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  arXiv preprint arXiv:1607.06450, 2016.
  * [3] Wayner Barrios, Mattia Soldan, Alberto Mario Ceballos-Arroyo, Fabian Caba Heilbron, and Bernard Ghanem.  Localizing moments in long video via multimodal guidance.  In ICCV, 2023.
  * [4] Navaneeth Bodla, Bharat Singh, Rama Chellappa, and Larry S Davis.  Soft-NMS–improving object detection with one line of code.  In ICCV, 2017.
  * [5] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In CVPR, 2017.
  * [6] Jingyuan Chen, Xinpeng Chen, Lin Ma, Zequn Jie, and Tat-Seng Chua.  Temporally grounding natural sentence in video.  In EMNLP, 2018.
  * [7] Long Chen, Chujie Lu, Siliang Tang, Jun Xiao, Dong Zhang, Chilie Tan, and Xiaolin Li.  Rethinking the bottom-up framework for query-based video localization.  In AAAI, 2020.
  * [8] Shaoxiang Chen and Yu-Gang Jiang.  Semantic proposal for activity localization in videos via sentence query.  In AAAI, 2019.
  * [9] Feng Cheng, Mingze Xu, Yuanjun Xiong, Hao Chen, Xinyu Li, Wei Li, and Wei Xia.  Stochastic backpropagation: a memory efficient strategy for training video models.  In CVPR, 2022.
  * [10] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al.  Rethinking attention with performers.  In ICLR, 2021.
  * [11] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In NAACL, 2018.
  * [12] Xiang Fang, Daizong Liu, Pan Zhou, and Guoshun Nan.  You can ground earlier than see: An effective and efficient pipeline for temporal sentence grounding in compressed videos.  In CVPR, 2023.
  * [13] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  SlowFast networks for video recognition.  In ICCV, 2019.
  * [14] Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia.  TALL: Temporal activity localization via language query.  In ICCV, 2017.
  * [15] Jialin Gao, Xin Sun, Mengmeng Xu, Xi Zhou, and Bernard Ghanem.  Relation-aware video reading comprehension for temporal language grounding.  In EMNLP, 2021.
  * [16] Soham Ghosh, Anuva Agarwal, Zarana Parekh, and Alexander Hauptmann.  ExCL: Extractive clip localization using natural language descriptions.  In ACL, 2019.
  * [17] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al.  Ego4D: Around the world in 3,000 hours of egocentric video.  In CVPR, 2022.
  * [18] Zhijian Hou, Wanjun Zhong, Lei Ji, Difei Gao, Kun Yan, Wing-Kwong Chan, Chong-Wah Ngo, Zheng Shou, and Nan Duan.  CONE: An efficient coarse-to-fine alignment framework for long video temporal grounding.  In ACL, 2023.
  * [19] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei.  Large-scale video classification with convolutional neural networks.  In CVPR, 2014.
  * [20] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al.  The kinetics human action video dataset.  arXiv preprint arXiv:1705.06950, 2017.
  * [21] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles.  Dense-captioning events in videos.  In ICCV, 2017.
  * [22] Jie Lei, Tamara L Berg, and Mohit Bansal.  Detecting moments and highlights in videos via natural language queries.  NeurIPS, 2021.
  * [23] Hongxiang Li, Meng Cao, Xuxin Cheng, Yaowei Li, Zhihong Zhu, and Yuexian Zou.  G2L: Semantically aligned and uniform video grounding via geodesic and game theory.  In ICCV, 2023.
  * [24] Juncheng Li, Junlin Xie, Long Qian, Linchao Zhu, Siliang Tang, Fei Wu, Yi Yang, Yueting Zhuang, and Xin Eric Wang.  Compositional temporal grounding with structured variational cross-graph correspondence learning.  In CVPR, 2022.
  * [25] Kun Li, Dan Guo, and Meng Wang.  Proposal-free video grounding with contextual pyramid network.  In AAAI, 2021.
  * [26] Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shraman Pramanick, Difei Gao, Alex Jinpeng Wang, Rui Yan, and Mike Zheng Shou.  UniVTG: Towards unified video-language temporal grounding.  In ICCV, 2023.
  * [27] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In ICCV, 2017.
  * [28] Daizong Liu, Xiaoye Qu, Xing Di, Yu Cheng, Zichuan Xu, and Pan Zhou.  Memory-guided semantic learning network for temporal sentence grounding.  In AAAI, 2022.
  * [29] Daizong Liu, Xiaoye Qu, Jianfeng Dong, and Pan Zhou.  Adaptive proposal generation network for temporal sentence localization in videos.  In EMNLP, 2021.
  * [30] Daizong Liu, Xiaoye Qu, Jianfeng Dong, Pan Zhou, Yu Cheng, Wei Wei, Zichuan Xu, and Yulai Xie.  Context-aware biaffine localizing network for temporal sentence grounding.  In CVPR, 2021.
  * [31] Daizong Liu, Xiaoye Qu, and Pan Zhou.  Progressively guide to attend: An iterative alignment framework for temporal sentence grounding.  In EMNLP, 2021.
  * [32] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In International Conference on Learning Representations, 2019.
  * [33] Chujie Lu, Long Chen, Chilie Tan, Xiaolin Li, and Jun Xiao.  DEBUG: A dense bottom-up grounding approach for natural language video localization.  In EMNLP, 2019.
  * [34] Shugao Ma, Leonid Sigal, and Stan Sclaroff.  Learning activity progression in lstms for activity detection and early detection.  In CVPR, 2016.
  * [35] Niluthpol Chowdhury Mithun, Sujoy Paul, and Amit K Roy-Chowdhury.  Weakly supervised video moment retrieval from text queries.  In CVPR, 2019.
  * [36] Jonghwan Mun, Minsu Cho, and Bohyung Han.  Local-global video-text interactions for temporal grounding.  In CVPR, 2020.
  * [37] Guoshun Nan, Rui Qiao, Yao Xiao, Jun Liu, Sicong Leng, Hao Zhang, and Wei Lu.  Interventional video grounding with dual contrastive learning.  In CVPR, 2021.
  * [38] Mayu Otani, Yuta Nakashima, Esa Rahtu, and Janne Heikkilä.  Uncovering hidden challenges in query-based video moment retrieval.  In BMVC, 2020.
  * [39] Yulin Pan, Xiangteng He, Biao Gong, Yiliang Lv, Yujun Shen, Yuxin Peng, and Deli Zhao.  Scanning only once: An end-to-end framework for fast temporal grounding in long videos.  In ICCV, 2023.
  * [40] Jeffrey Pennington, Richard Socher, and Christopher D Manning.  GloVe: Global vectors for word representation.  In EMNLP, 2014.
  * [41] Kevin Qinghong Lin, Alex Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Zhongcong Xu, Difei Gao, Rongcheng Tu, Wenzhe Zhao, Weijie Kong, et al.  Egocentric video-language pretraining.  NeurIPS, 2022.
  * [42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In ICML, 2021.
  * [43] Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele, and Manfred Pinkal.  Grounding action descriptions in videos.  TACL, 2013.
  * [44] Cristian Rodriguez, Edison Marrese-Taylor, Fatemeh Sadat Saleh, Hongdong Li, and Stephen Gould.  Proposal-free temporal moment localization of a natural-language query in video using guided attention.  In WACV, 2020.
  * [45] Zheng Shou, Dongang Wang, and Shih-Fu Chang.  Temporal action localization in untrimmed videos via multi-stage cnns.  In CVPR, 2016.
  * [46] Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta.  Hollywood in homes: Crowdsourcing data collection for activity understanding.  In ECCV, 2016.
  * [47] Bharat Singh, Tim K Marks, Michael Jones, Oncel Tuzel, and Ming Shao.  A multi-stream bi-directional recurrent neural network for fine-grained action detection.  In CVPR, 2016.
  * [48] Mattia Soldan, Alejandro Pardo, Juan León Alcázar, Fabian Caba, Chen Zhao, Silvio Giancola, and Bernard Ghanem.  MAD: A scalable dataset for language grounding in videos from movie audio descriptions.  In CVPR, 2022.
  * [49] Mattia Soldan, Mengmeng Xu, Sisi Qu, Jesper Tegner, and Bernard Ghanem.  VLG-net: Video-language graph matching network for video grounding.  In ICCVW, 2021.
  * [50] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He.  FCOS: Fully convolutional one-stage object detection.  In ICCV, 2019.
  * [51] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri.  Learning spatiotemporal features with 3d convolutional networks.  In ICCV, 2015.
  * [52] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  NeurIPS, 2017.
  * [53] Hao Wang, Zheng-Jun Zha, Liang Li, Dong Liu, and Jiebo Luo.  Structured multi-level interaction network for video moment localization via language query.  In CVPR, 2021.
  * [54] Jingwen Wang, Lin Ma, and Wenhao Jiang.  Temporally grounding language queries in videos by contextual boundary-aware prediction.  In AAAI, 2020.
  * [55] Zhenzhi Wang, Limin Wang, Tao Wu, Tianhao Li, and Gangshan Wu.  Negative sample matters: A renaissance of metric learning for temporal grounding.  In AAAI, 2022.
  * [56] Sangmin Woo, Jinyoung Park, Inyong Koo, Sumin Lee, Minki Jeong, and Changick Kim.  Explore and match: End-to-end video grounding with transformer.  arXiv preprint arXiv:2201.10168, 2022.
  * [57] Shaoning Xiao, Long Chen, Jian Shao, Yueting Zhuang, and Jun Xiao.  Natural language video localization with learnable moment proposals.  In EMNLP, 2021.
  * [58] Shaoning Xiao, Long Chen, Songyang Zhang, Wei Ji, Jian Shao, Lu Ye, and Jun Xiao.  Boundary proposal network for two-stage natural language video localization.  In AAAI, 2021.
  * [59] Huijuan Xu, Kun He, Bryan A Plummer, Leonid Sigal, Stan Sclaroff, and Kate Saenko.  Multilevel language and vision integration for text-to-clip retrieval.  In AAAI, 2019.
  * [60] Yitian Yuan, Xiaohan Lan, Xin Wang, Long Chen, Zhi Wang, and Wenwu Zhu.  A closer look at temporal sentence grounding in videos: Dataset and metric.  In ACM MM-HUMA Workshop, 2021.
  * [61] Yitian Yuan, Lin Ma, Jingwen Wang, Wei Liu, and Wenwu Zhu.  Semantic conditioned dynamic modulation for temporal sentence grounding in videos.  NeurIPS, 2019.
  * [62] Yitian Yuan, Tao Mei, and Wenwu Zhu.  To find where you talk: Temporal sentence localization in video with attention based location regression.  In AAAI, 2019.
  * [63] Runhao Zeng, Haoming Xu, Wenbing Huang, Peihao Chen, Mingkui Tan, and Chuang Gan.  Dense regression network for video grounding.  In CVPR, 2020.
  * [64] Chen-Lin Zhang, Jianxin Wu, and Yin Li.  ActionFormer: Localizing moments of actions with transformers.  In ECCV, 2022.
  * [65] Da Zhang, Xiyang Dai, Xin Wang, Yuan-Fang Wang, and Larry S Davis.  MAN: Moment alignment network for natural language moment retrieval via iterative graph adjustment.  In CVPR, 2019.
  * [66] Hao Zhang, Aixin Sun, Wei Jing, Liangli Zhen, Joey Tianyi Zhou, and Siow Mong Rick Goh.  Parallel attention network with sequence matching for video grounding.  In Findings of ACL, 2021.
  * [67] Hao Zhang, Aixin Sun, Wei Jing, and Joey Tianyi Zhou.  Span-based localizing network for natural language video localization.  In ACL, 2020.
  * [68] Lingyu Zhang and Richard J Radke.  Natural language video moment localization through query-controlled temporal convolution.  In WACV, 2022.
  * [69] Mingxing Zhang, Yang Yang, Xinghan Chen, Yanli Ji, Xing Xu, Jingjing Li, and Heng Tao Shen.  Multi-stage aggregated transformer network for temporal language localization in videos.  In CVPR, 2021.
  * [70] Songyang Zhang, Houwen Peng, Jianlong Fu, and Jiebo Luo.  Learning 2d temporal adjacent networks for moment localization with natural language.  In AAAI, 2020.
  * [71] Yang Zhao, Zhou Zhao, Zhu Zhang, and Zhijie Lin.  Cascaded prediction network via segment tree for temporal video grounding.  In CVPR, 2021.
  * [72] Zhaohui Zheng, Ping Wang, Wei Liu, Jinze Li, Rongguang Ye, and Dongwei Ren.  Distance-IoU loss: Faster and better learning for bounding box regression.  In AAAI, 2020.
  * [73] Hao Zhou, Chongyang Zhang, Yan Luo, Yanjun Chen, and Chuanping Hu.  Embracing uncertainty: Decoupling and de-bias for robust temporal grounding.  In CVPR, 2021.
  * [74] Jiahao Zhu, Daizong Liu, Pan Zhou, Xing Di, Yu Cheng, Song Yang, Wenzheng Xu, Zichuan Xu, Yao Wan, Lichao Sun, et al.  Rethinking the video sampling and reasoning strategies for temporal sentence grounding.  In EMNLP Findings, 2022.

## Supplementary Material

In the supplementary material, we (1) elaborate on our cost analysis in the
main paper (Section A); (2) compare SnAG to latest methods with more advanced
learning paradigms (Section B); (3) analyze the effect of input snippet length
in training on model performance (Section C); (4) conduct error analysis of
our results in comparison to baseline results (Section D); (5) describe
additional implementation details (Section E); and (6) discuss the limitations
and future directions of our work (Section F). We hope that this document will
complement our main paper.

For sections, figures, tables, and equations, we use numbers (_e.g_., Sec. 1)
to refer to the main paper and capital letters (_e.g_., Sec. A) to refer to
this supplement.

##  A Cost Analysis

In Section 3.2 of the main paper, we have shown that late fusion, under the
assumption of $\mathcal{C}(\mathcal{F}_{L})\ll\mathcal{C}(g)+\mathcal{C}(h)$
and $\mathcal{C}(g)\ll\mathcal{C}(h)$, allows a maximum of $N$ times speedup
in theory at inference time. Here $N$ is the number of text queries per video,
$\mathcal{C}(g)$ and $\mathcal{C}(h)$ denotes the respective cost of video and
text encoding, and $\mathcal{C}(\mathcal{F}_{L})$ combines the cost of cross-
modal fusion and moment decoding.

In practice, for any realistic instantiation of $\mathcal{F}_{L}$, like our
SnAG, the cost of cross-modal fusion and moment decoding
$\mathcal{C}(\mathcal{F}_{L})$ can not be ignored. This partially breaks our
assumption of $\mathcal{C}(\mathcal{F}_{L})\ll\mathcal{C}(g)+\mathcal{C}(h)$.
To account for non-trivial cost $\mathcal{C}(\mathcal{F}_{L})$, we denote
$\mathcal{C}(\mathcal{F}_{L})=\alpha(\mathcal{C}(g)+\mathcal{C}(h))$, where
the positive coefficient $\alpha$ reflects the cost of $\mathcal{F}_{L}$
relative to the total cost of both encoders $g$ and $h$. For late fusion,
$\alpha$ is typically in the range of $(0,1)$. Following Section 3.2, we have

$\displaystyle\small\mathbf{R}_{\text{inf}}$
$\displaystyle=\frac{\mathcal{C}(\text{inference with early
fusion})}{\mathcal{C}(\text{inference with late fusion})}$ (A)
$\displaystyle=\frac{MN\ \mathcal{C}(\mathcal{F}_{L})+MN\ \mathcal{C}(g)+MN\
\mathcal{C}(h)}{MN\ \mathcal{C}(\mathcal{F}_{L})+M\ \mathcal{C}(g)+N\
\mathcal{C}(h)}$ (B)
$\displaystyle=\frac{MN(1+\alpha)}{MN\alpha+M\mathbf{R}_{g/(g+h)}+N(1-\mathbf{R}_{g/(g+h)})}$
(C)
$\displaystyle\approx\frac{1+\alpha}{\frac{1}{N}+\alpha}\to\begin{cases}N,&\alpha\to
0\\\ 1+\frac{1}{\alpha},&N\to\infty\end{cases},$ (D)

where the approximation in Equation D is a consequence of
$\mathbf{R}_{g/(g+h)}=\mathcal{C}(g)/(\mathcal{C}(g)+\mathcal{C}(h))\xrightarrow{\mathcal{C}(g)\gg\mathcal{C}(h)}1$.

Substituting $N$ with $B_{q}$ (the number of queries drawn for each training
snippet), we can derive similar efficiency result for our video-centric
training scheme (Section 3.3):

$\displaystyle\mathbf{R}_{\text{train}}\approx\frac{1+\alpha}{\frac{1}{B_{q}}+\alpha}\to\begin{cases}B_{q},&\alpha\to
0\\\ 1+\frac{1}{\alpha},&B_{q}\to\infty\end{cases}.$ (E)

For long-form video grounding (_i.e_., large $N$ and $B_{q}$), our extended
analysis indicates that late fusion remains highly efficient when $\alpha$ is
reasonably small. For example, a late-fusion model with $\alpha\to 1$ can in
theory bring a considerable 2$\times$ cost reduction in both training and
inference relative to early fusion. This is precisely the case with SnAG, as
Figure 5(a) of the main paper shows that SnAG’s encoders account for $\sim$50%
of the total computation.

Importantly, our empirical findings confirms this estimate. Figure 5(b, d) of
the main paper shows that video-centric training and inference of SnAG on MAD
is $\sim$50% more efficient in terms of runtime and GPU memory. On TACoS and
Ego4D, the saving in training is less prominent as we found that smaller
$B_{q}$ leads to slightly better results, yet the substantial reduction in
inference time echoes our cost analysis and demonstrates the efficiency of
SnAG.

Model | R@1 | R@5 | R@10 | R@50
---|---|---|---|---
0.1 | 0.3 | 0.5 | 0.1 | 0.3 | 0.5 | 0.1 | 0.3 | 0.5 | 0.1 | 0.3 | 0.5
$\dagger$ CLIP [42] | 9.30 | 4.65 | 2.16 | 18.96 | 13.06 | 7.40 | 24.30 | 17.73 | 11.09 | 39.79 | 32.23 | 23.21
$\dagger$ VLG-Net [49] | 5.60 | 4.28 | 2.48 | 16.07 | 13.14 | 8.78 | 23.64 | 19.86 | 13.72 | 45.35 | 39.77 | 30.22
$\dagger$ Moment-DETR [22] | 5.07 | 3.82 | 2.39 | 16.30 | 12.60 | 7.90 | 24.79 | 19.43 | 12.06 | 50.06 | 40.52 | 24.87
SnAG | 10.28 | 8.46 | 5.55 | 24.42 | 20.60 | 13.75 | 32.23 | 27.50 | 19.00 | 52.28 | 46.68 | 35.24
(a) Results on MAD [48].

Model | mean R@1 | mean R@5
---|---|---
$\dagger$ Moment-DETR [22] | 7.28 | 22.14
$\dagger$ VSL-Net [67] | 11.90 | 24.06
SnAG (EgoVLP) | 15.59 | 38.40
(b) Results on Ego4D-NLQ [17].

Table A: Comparison with Multimodal Guidance [12]. SnAG does not take audio as
input, yet outperforms existing models with Multimodal Guidance ($\dagger$)
using audio. Mean R@{1,5} are computed over tIoU={0.1,0.3,0.5}. For SnAG, we
report the better results between video-centric and query-centric training.

Model | R@1 | R@5
---|---|---
0.3 | 0.5 | 0.3 | 0.5
TCSF | 49.82 | 38.53 | 68.60 | 59.89
SnAG (C3D) | 56.44 | 44.86 | 81.15 | 70.66
(a) Results on TACoS [43].

Model | R@1 | R@5
---|---|---
0.5 | 0.7 | 0.5 | 0.7
TCSF | 53.85 | 37.20 | 90.86 | 58.95
SnAG (I3D) | 65.13 | 46.26 | 92.80 | 71.94
(b) Results on Charades-STA [46].

Model | R@1 | R@5
---|---|---
0.3 | 0.5 | 0.3 | 0.5
TCSF | 66.87 | 48.38 | 88.75 | 80.24
SnAG (C3D) | 63.58 | 48.55 | 89.55 | 81.71
(c) Results on ActivityNet-Captions [21].

Table B: Comparison with TCSF [12]. SnAG outperforms TCSF despite using frozen
video features. For SnAG, we report the better results between video-centric
and query-centric training.

Model | R@1
---|---
0.3 | 0.5 | avg
UniVTG | 7.28 | 3.95 | 5.62
UniVTG w/ PT | 11.74 | 7.54 | 9.64
SnAG (EgoVLP) | 15.87 | 11.26 | 13.57
(a) Results on Ego4D-NLQ [17].

Model | R@1
---|---
0.3 | 0.5 | avg
UniVTG | 51.44 | 34.97 | 43.21
UniVTG w/ PT | 56.11 | 43.44 | 49.78
SnAG (C3D) | 56.44 | 44.86 | 50.65
(b) Results on TACoS [43].

Model | R@1
---|---
0.5 | 0.7 | avg
UniVTG | 58.01 | 35.65 | 46.83
UniVTG w/ PT | 60.19 | 38.55 | 49.27
SnAG (I3D) | 65.13 | 46.26 | 55.70
(c) Results on Charades-STA [46].

Table C: Comparison with UniVTG [26]. SnAG consistently outperforms UniVTG
despite being simpler and learning from individual datasets without multi-task
pre-training. w/ PT means fine-tuning after pre-training. For SnAG, we report
the better results between video-centric and query-centric training.

##  B Additional Comparisons

We further compare SnAG to latest approaches that (1) consume additional input
modalities (_e.g_., audio), (2) are trained end to end on raw videos (as
opposed to using frozen video backbones); or (3) perform large-scale, multi-
task pre-training. We note that these are not fair comparisons and put SnAG at
a disadvantage. This comparative analysis, however, further highlights the
strength of SnAG.

Comparison with Barrios _et al_. [3]. Barrios _et al_. [3] has demonstrated a
method for long-form video grounding via Multimodal Guidance. The guidance
model takes audio in addition to video and text queries as input, and produces
a score to condition an existing grounding model.

Table A compares the performance of existing models with Multimodal Guidance
and SnAG. While the audio input provides additional cues to facilitate
accurate moment localization, it does not fully close the gap between the
existing models and SnAG. SnAG is 3.07 absolute percentage points higher in
R@1, tIoU=0.5 on MAD, and 1.67 points higher in mean R@1 on Ego4D. We
hypothesize that Multimodal Guidance will also boost the performance of SnAG.

Comparison with TCSF [12]. TCSF takes a compressed video and obtains features
from keyframes without explicitly decoding the video. Importantly, the feature
extractor is jointly trained with the grounding model. Joint training has
significant benefit as it allows video features to adapt to the grounding task
through end-to-end learning.

Table B compares the performance of TCSF and SnAG. Despite the extra capacity
to learn more powerful features, TCSF lags behind SnAG by a significant
margin. Consider the R@1, tIoU=0.5 scores — SnAG outperforms TCSF by 6.33
absolute percentage points on TACoS, and 11.28 absolute percentage points on
Charades. The two methods are on par on ActivityNet (48.38 _vs_. 48.55).

Comparison with UniVTG [26]. UniVTG pools large-scale datasets and annotations
from multiple video understanding tasks and learns a unified grounding model
through multi-task pre-training. The pre-trained model supports fine-tuning
and zero-shot inference on the training tasks, including temporal sentence
grounding.

Table C compares the performance of UniVTG and SnAG. While UniVTG is capable
of learning from diverse data and supervision signals, its performance lags
behind SnAG, despite using stronger SlowFast [13] and CLIP [42] features. We
again consider the R@1, tIoU=0.5 scores — SnAG achieves 11.26% on Ego4D,
44.85% on TACoS, and 64.62% on Charades, outperforming UniVTG by 3.72, 1.41
and 4.43 absolute percentage points, respectively.

Figure A: Ablations on snippet length in training. On both TACoS (a) and MAD
(b), SnAG delivers stronger test accuracy when trained on snippets of
increased length.

##  C Effect of Snippet Length

We conduct ablation studies on MAD and TACoS to understand the impact of
snippet length in training.

Experiment setting. On TACoS, we emulate feature sub-sampling in previous
methods [49, 70, 28] by resizing input video features to a uniform length of
128, 256 and 512, and compare to training using full-sized features (capped at
a length of 2304). On MAD, we are able to vary snippet length from 256 (1
minute) to 8192 (1 hour) thanks to SnAG’s scalable model design and training
scheme. We report R@1 and R@5 at tIoU=0.5 for both datasets.

Results. Figure A presents our results. Training on longer snippets constantly
leads to higher test accuracy for both datasets. On MAD, R@1, tIoU=0.5 grows
by a significant 73% (from 3.31% to 5.73%) even though all moments are short
(4.1 seconds on average) and can be fully covered by the shortest snippet
length of 256. We conjecture that negative moments that are distant in time
can provide strong learning signal to models like SnAG, which is capable of
processing long-form video inputs.

Figure B: Error analysis on TACoS. Our model consistently outperforms VLG-Net
[49] and 2D-TAN [70] on moments with low (XS) to high (XL) coverage and videos
with short (XS) to long (XL) duration. It is especially strong on low-coverage
moments (a) and long videos (b) thanks to its scalability with respect to
video duration.

##  D Error Analysis

To better understand our results, we present an error analysis and compare our
results with those from strong baselines of VLG-Net [49] and 2D-TAN [70]. This
analysis is conducted on the test split of TACoS.

Experiment setting. In our first experiment, we divide the ground-truth
moments into five bins according to their coverage (_i.e_., moment length
relative to full video length): Extra Small (XS: (0, 0.02]), Small (S: (0.02,
0.04]), Medium (M: (0.04, 0.08]), Large (L: (0.08, 0.16]) and Extra Large (XL:
(0.16, 1]). In our second experiment, we divide the videos into five bins
based on their duration in seconds: Extra Small (XS: (0, 125]), Small (S:
(125, 250]), Medium (M: (250, 500]), Large (L: (500, 750]), Extra Large (XL:
(750, $\infty$)). We report mean R@1 averaged over the tIoU thresholds of 0.3
and 0.5 for each bin.

Results. Figure B presents our results. While our model consistently
outperforms the baselines by a significant margin across all conditions, the
relative percentage of improvement is especially large on low-coverage moments
(XS and S) and long videos (L and XL). Note that both VLG-Net and 2D-TAN have
a complexity of $\mathcal{O}(T^{2})$ and thus need to sub-sample input video
features to a small length (256 for VLG-Net and 128 for 2D-TAN) to reduce
their computational cost. Intuitively, feature sub-sampling drops detailed
information about event dynamics and makes precise localization of short
moments more challenging. This becomes especially problematic for long videos
as the drop rate can be extremely large (_e.g_., 10$\times$ for XL videos).
Our results thus suggest that the ability to scale up to long videos is
critical for enhancing localization accuracy.

Dataset |  | Video
---
FPS
| Video
---
Feature
| Clip
---
Size
Stride |  | Snippet
---
Length
| Video
---
Dim
| Win
---
Size
| Video
---
Layers
| Text
---
Feature
| Text
---
Dim
| Text
---
Layers
| Batch
---
Size
$B_{q}$ | lr | Epochs
MAD | 5 | CLIP | 1 | 1 | 4096 | 512 | 17 | 9 | CLIP | 512 | 1 | 32 | 8 | 1e-4 | 8
Ego4D-NLQ | 30 | SlowFast | 32 | 16 | 2304 | 512 | 19 | 8 | BERT | 128 | 1 | 2 | 2 | 1e-4 | 10
EgoVLP | 384 | EgoVLP | 768 | 2 | 2 | 7
TACoS | 29.4 | C3D | 16 | 16 | 2304 | 128 | 19 | 8 | GloVe | 128 | 5 | 16 | 4 | 1e-3 | 15
Charades-STA | 24 | C3D | 16 | 4 | 256 | 256 | 5 | 7 | GloVe | 128 | 5 | 16 | 4 | 1e-3 | 10
I3D | 16 | 4
ANet-Captions | varying | C3D | 16 | 8 | 256 | 128 | 5 | 7 | GloVe | 128 | 5 | 16 | 2 | 1e-3 | 15

Table D: Implementation Details. We list key parameters for video encoding
(left), text encoding (middle), and model training (right).

##  E Implementation Details

We now present the implementation details of SnAG. Table D summarizes key
hyper-parameters we used in our experiments for feature extraction, training
and inference.

Video features. For TACoS [43], Charades-STA [46] and ActivityNet-Captions
[21], we extract clip-level video features using C3D network pre-trained on
Sports-1M [51], following prior approaches (_e.g_., [70, 49, 28]).

We decode TACoS videos at a frame rate of 29.4 frames per second (FPS) and
compute the 4096-dimensional, post-ReLU activations of _fc6_ layer as features
using non-overlapping clips of 16 frames. For Charades-STA, we use the RGB
frames provided on the official website. We use clips of 16 frames with a
stride of 4 frames (_i.e_., 75% overlap), and compute the 4096-dimensional,
post-ReLU activations of _fc7_ layer as features. We use official C3D features
for ActivityNet. For fair comparison with baselines, we additionally extract
two-stream I3D features for Charades-STA from I3D network pre-trained on
Kinetics [5] using clips of 16 frames with a stride of 4 frames. For Ego4D-NLQ
[17], we use official SlowFast features [13]as well as EgoVLP features [41],
both extracted from 30 FPS videos using clips of 32 frames with a stride of 16
frames (_i.e_., 50% overlap). For MAD [48], we use the official CLIP features
[42] extracted at 5 FPS.

Textual features. We use GloVe word embeddings [40] with 6B vocabulary for
TACoS, Charades-STA and ActivityNet-Captions. For Ego4D-NLQ, we use last-layer
token embeddings ($\mathrm{CLS}$ token excluded) from BERT [11] or EgoVLP text
encoder together with SlowFast or EgoVLP video features. For MAD, we use
official CLIP token embeddings ($\mathrm{SOT}$ and $\mathrm{EOT}$ tokens
excluded) from the last layer of text encoder.

Network architecture. For the video encoder, we apply a linear projection
followed by two 1D convolutional layers with kernel size 3 on the input
features prior to building the multi-scale video representation. The embedding
dimension, attention window size and number of scales in the multi-scale
Transformer network differ by dataset (see Table D for a summary). For the
text encoder, we apply a single Transformer layer for MAD and Ego4D-NLQ, and 5
layers for the other datasets. We use 4 attention heads for all Transformer
layers in the model. The classification and regression heads each contains two
1D convolutional layers with kernel size 3.

Training details. We randomly sample video snippets with a maximum sequence
length of 4096, 2304 and 2304 for MAD, Ego4D-NLQ and TACoS, respectively. We
resize all videos to a uniform length of 256 for Charades-STA and ActivityNet.
We use the AdamW optimizer [32] with the default beta values of (0.9, 0.999)
and a weight decay of 0.05. Mini-batch size (including $B_{q}$ for video-
centric training), learning rate and number of training epochs can be found in
Table D. We maintain an exponential moving average (EMA) of model parameters
over the entire course of training with a momentum of 0.999 and use the EMA
version of the trained model for inference.

Inference details. Following our video-centric inference protocol, we first
feed full videos into our model to compute the shared video representation,
and then input text queries and fuse them with the video representation.
Moments are then decoded from the fused representation and further merged
using SoftNMS [4].

##  F Discussion and Limitation

The design of SnAG is one of many possible instantiations of our analysis. As
long as late fusion and video-centric training is employed, alternative
designs can be considered, including two-stage proposal-based methods or DETR-
alike models. We anticipate that exploration of this design space may point to
an interesting future avenue.

Similar to prior methods, SnAG’s performance heavily depends on the quality of
pre-extracted video and sentence features. These features are from networks
pre-trained on a related yet different task (_e.g_., action recognition or
image-text matching) and may not be optimal for video grounding. Training of
SnAG further requires moment and sentence annotations that are expensive and
difficult to acquire. End-to-end training [9] of SnAG with less human
annotation [35] is thus a promising future direction.