[table]capposition=top capbtabboxtable[][]

# Glance and Focus: Memory Prompting for Multi-Event Video Question Answering

Ziyi Bai1,2, Ruiping Wang1,2, Xilin Chen1,2
1Key Laboratory of Intelligent Information Processing of Chinese Academy of
Sciences (CAS),
Institute of Computing Technology, CAS, Beijing, 100190, China
2University of Chinese Academy of Sciences, Beijing, 100049, China
<EMAIL_ADDRESS>{wangruiping<EMAIL_ADDRESS>
###### Abstract

Video Question Answering (VideoQA) has emerged as a vital tool to evaluate
agents’ ability to understand human daily behaviors. Despite the recent
success of large vision language models in many multi-modal tasks, complex
situation reasoning over videos involving multiple human-object interaction
events still remains challenging. In contrast, humans can easily tackle it by
using a series of episode memories as anchors to quickly locate question-
related key moments for reasoning. To mimic this effective reasoning strategy,
we propose the Glance-Focus model. One simple way is to apply an action
detection model to predict a set of actions as key memories. However, these
actions within a closed set vocabulary are hard to generalize to various video
domains. Instead of that, we train an Encoder-Decoder to generate a set of
dynamic event memories at the glancing stage. Apart from using supervised
bipartite matching to obtain the event memories, we further design an
unsupervised memory generation method to get rid of dependence on event
annotations. Next, at the focusing stage, these event memories act as a bridge
to establish the correlation between the questions with high-level event
concepts and low-level lengthy video content. Given the question, the model
first focuses on the generated key event memory, then focuses on the most
relevant moment for reasoning through our designed multi-level cross-attention
mechanism. We conduct extensive experiments on four Multi-Event VideoQA
benchmarks including STAR, EgoTaskQA, AGQA, and NExT-QA. Our proposed model
achieves state-of-the-art results, surpassing current large models in various
challenging reasoning tasks. The code and models are available at
https://github.com/ByZ0e/Glance-Focus.

##  1 Introduction

For intelligent agent that can assist humans in completing daily-life tasks,
complex event reasoning ability is particularly important. Multiple Event
Video Question Answering (Multi-Event VideoQA) task [14, 42, 16, 43] has
recently been proposed. These benchmarks require agents to reason about
complex videos depicting multiple human-object interactions (i.e., events or
situations). Furthermore, a wide scope of event-centric questions are
involved[14], ranging from event semantic descriptions to event sequencing and
dependencies to comprehensively evaluate an agent’s ability to understand
human behavior.

Figure 1: Comparison of different VideoQA solutions. The cross-modal common
feature space is visualized. (a) Cross-modal correspondence modeling. Weak
correspondence between the question with high-level event concepts and low-
level lengthy video leads to a wrong answer. (b) Human-like Glance-Focus
Strategy. Glancing stage: a set of episode memories are generated. Focusing
stage: memory prompts are used as anchors to quickly locate the keyframes for
reasoning.

Recently, the emerged large vision-language models [1, 3, 8, 25, 47, 50] have
demonstrated impressive generalization capabilities in VideoQA tasks. By
leveraging pre-training on large-scale cross-modal pairs, these models can
establish correlations between questions and videos to find answers
effectively. Such approaches proves effective for short videos containing
single or few events with one-step questions like “What is someone doing?”.
However, Multi-Event VideoQA poses distinct challenge. As shown in Figure
1(a), due to the longer video duration and more events involved, it is
challenging for the model to effectively find the visual clues from the
lengthy video content. Additionally, the event-centric questions often include
high-level event concepts while the semantics of video frames are relatively
low-level, making it even more difficult to directly locate key moments
related to the question.

This is very similar to the difficulties we humans encounter when addressing
reading comprehension tasks, where we must quickly extract question-related
context from long paragraphs. To ensure fast and effective reasoning, it is
common for humans to employ a strategy of first glancing through the story,
summarizing the storyline in our mind, and then using this episode memory to
quickly focus on the relevant context when answering questions.

We mimic such a reasoning strategy to tackle the challenging Multi-Event
VideoQA task with our proposed Glance-Focus model. It comprises two stages:
glancing and focusing. The purpose of the glancing stage is to generate a set
of episode memories that can efficiently summarize the multiple events
occurring in the video. A naive way is to directly train an action detection
model to obtain a set of actions. However, this would result in unnecessary
computation, and most importantly, it may not be practical for encapsulating
all possible actions in the real world using closed vocabularies.

Instead, we adopt a more flexible approach that automatically obtains a set of
dynamic event memories using a Transformer Encoder-Decoder architecture [39].
For datasets with closed event vocabularies, we employ bipartite matching
between the extracted memories and ground truth labels to explicitly obtain
event memories. Apart from supervised event matching approach, we additionally
design an unsupervised event generation way to get rid of the event-level
annotations of the video. We impose individual certainty and global diversity
constraints during event generation to ensure they can effectively summarize
the video content.

Finally, since these generated memories are still independent events, we
reorganize them into complete event memory prompts by arranging them in
sequence order with temporal information remaining. As shown in Figure 1(b),
since these memory prompts have high-level event semantics, they play an
anchor role in the multi-modal common feature space to help quickly locate the
related video content according to the question. Such as the example in Figure
1(b), we design a multi-level cross attention mechanism to first focus on the
event “putting down the blanket” before “taking the clothes” according to the
question, then focus on the related video content for answer prediction.

We conduct experiments on various recently released Multi-Event VideoQA
datasets, STAR [42], AGQA [14], EgoTaskQA[16] and NExT-QA[43]. The
experimental results demonstrate that our proposed Glance-Focus model achieves
state-of-the-art performance on all four datasets. We further verify the
effectiveness of our approach via model analysis, showing the outstanding
performance of key components in our method. Moreover, both quantitative and
qualitative results affirm that our proposed memory prompts play a significant
anchor role in multi-event reasoning by bridging the gap between questions and
videos.

##  2 Related Works

###  2.1 Video Question Answering

Early attempts[11, 49, 6, 51, 41] have made progress in improving complex
spatial-temporal contextual information modeling for VideoQA tasks. They
usually extract the appearance and motion features of video units (such as
clips or frames) separately and then let them interact with the question to
find an answer. To capture question-related spatial and temporal clues, co-
memory and co-attention mechanisms are proposed by MACN[11] and HME[6]
respectively. B2A[35] uses questions as a bridge to connect the appearance and
motion features to find the answers. These holistic models are effective at
aggregating spatial and temporal context, making them well-suited for general
one-step video reasoning tasks. However, they tend to encounter challenges
when confronted with more intricate compositional questions[14].

Therefore many studies[23, 15, 30, 5, 44, 45] have delved into the exploration
of hierarchical modeling strategies to address the challenges of multi-grained
video content representation. HCRN[23] proposes using reusable units stacked
hierarchically to gradually extract frame-level to video-level features
conditioned by motion and question. Graph-based methods such as L-GCN[15],
HAIR[30], and VGT[45] utilize extra object detection models, like Faster
R-CNN[37], to explicitly incorporate object-level features into the model.
They further organize the objects into a unified graph for fine-grained
reasoning. Although these hierarchical structures contain multi-grained clues,
few of them explicitly model event-level structure of videos. This limitation
poses a challenge in effectively handling high-level tasks, including event
prediction, explanation, and counterfactual reasoning.

###  2.2 Video-Language Pre-trained Models

Recently, Transformers-based large video-language models (VLMs) [52, 25, 2, 8,
47, 50, 12, 1, 3, 40] have accomplished comparable or superior results on
downstream tasks like VideoQA. These models harness the power of large-scale
image or video-text pairs collected from the Internet to pre-train cross-modal
correspondence models. They commonly employ proxy tasks like masked language
modeling and masked frame modeling [25, 50, 40] or contrastive learning [47,
2] during pre-training. Some models like ActBERT[52] and BridgeFormer[12]
focus on multi-grained semantics grounding by explicitly learning the
alignment of objects and actions across modalities. However, applying these
VLMs directly to Multi-Event VideoQA tasks is non-trivial. These models often
necessitate sparse sampling of lengthy videos, leading to a notable loss of
video semantics, and thus suffer from weak correspondence between questions
involving high-level event concepts and intricate video content[9, 16]. To
tackle this challenge, some retrieval-based cross-modal methods [48, 19, 36]
have been proposed recently to efficiently extract the keyframes of the video
for reasoning. Nonetheless, the use of fixed keyframes may pose difficulties
when dealing with multi-grained questions. Instead, we propose to employ the
dynamic and adaptable event memories for representing lengthy videos.

###  2.3 Multi-Event Modeling

Multi-event modeling is an essential ability for household robots to assist
humans in completing complex tasks. However, explicit multi-event modeling
still needs to be explored for VideoQA task. TSEA[9] proposes a heuristic
method to extract event-level features of videos. Several studies targeting at
multi-activity recognition and detection task[15, 34, 33] employ graph
structures to model complex relationships among multiple activities. P-GCN[15]
extracts activity proposals from videos and utilizes graph models to integrate
context. Ego-Topo[33] organizes egocentric videos into several zones and
learns the links between these zones for scene affordances and future action
prediction. Bridge-prompt[27] leverages the continuity of the natural language
to concatenate multiple discrete events by constructing a series of text
prompts. Similar to the multi-activity detection task, our goal is to obtain a
set of high-level key event from the video. However, directly training an
action detection model requires additional event-level annotations, which may
not always be available. Furthermore, it is impractical to use a closed set of
actions to encompass all possible events in the real-world. By contrast, our
proposed approach learns dynamic event memories from videos through either
unsupervised generation or supervised set-matching methods, depending on the
availability of event labels during the training process.

##  3 Approach: The Glance-Focus Model

We first give an overview of our model in Section 3.1. Next, we describe in
detail the two stages of our model, the Glancing stage and the Focusing stage
in Section 3.2 and Section 3.3 respectively. Finally, in Section 3.4 we
illustrate the overall loss functions.

Figure 2: The complex multi-event reasoning task is structured into two
stages: Glancing and Focusing. At the Glancing stage, a set of unordered event
memories are obtained by aggregating relevant input video frames. At the
Focusing stage, these event memories are reorganized according to their
sequential order. The resulting memory prompts act as bridges, allowing model
progressively focus on the related video frames for answer prediction using
multi-level cross-attention mechanism.

###  3.1 Overview

Given a video clip $V$ and a question $q$, a VideoQA model predicts the answer
$y$, formulated as follows:

$\hat{y}=\mathop{\arg\max}_{y\in\mathcal{A}}\mathcal{F_{\theta}}(y|q,V,\mathcal{A}),$
(1)

where $y$ is the predicted answer chosen from the candidate answers denoted as
$\mathcal{A}$, i.e., answer vocabulary for open-ended questions or choices for
multi-choice questions. $\theta$ is the trainable parameters of a VideoQA
model $\mathcal{F}$.

We propose a novel Glance-Focus VideoQA model with the Transformer Encoder-
Decoder framework[39]. As shown in Figure 2, the model answers the question
with two stages: 1) At the Glancing stage, a set of episode memories are
queried from the input video; 2) At the focusing stage, the model reorganizes
the memories into memory prompts and they will serve as anchors to localize
the question-related video content by multi-level focusing.

Input representations. For the input multi-event video of length $T$, we use a
pretrained visual backbone to extract its features
$\mathbf{X}\in\mathbb{R}^{T\times D}$ with dimension $D$. The video features
are fed into the model both in glancing and focusing stages. For questions,
following [31], we tokenize them using Byte-Pair Encoding (BPE) tokenization,
and embed them with pretrained RoBERTa language model[31] to get textual
tokens $\mathbf{w}=\\{w_{1},...,w_{L}\\}$ with question length $L$. Then they
are projected into the same dimension feature space as video features with a
projector $\mathcal{G}(.)$ to get question embeddings
$\mathbf{L}=\mathcal{G}(\mathbf{w})\in\mathbb{R}^{L\times D}$. The question
embeddings are fed into the model only at focusing stage.

###  3.2 Glancing Stage

It is challenging for the VideoQA model to directly find answers from lengthy
videos containing multiple complex events. To efficiently establish the
corresponding relationships between the high-level event concepts of the
questions and the low-level video unit features, we propose employing a quick
glance strategy that effectively extracts high-level episode memories from the
video to guide the subsequent reasoning.

To obtain the episode memory, a direct approach would be training an action
detection model to extract a set of actions from the video. However, this may
not be applicable in certain cases where video data lacks event annotations.
Moreover, it is impractical to encompass all possible events in the real world
with a close-set vocabulary. To address these issues, we propose a more
flexible and practical solution which can automatically learn a set of dynamic
memories $\mathbf{M=\\{m_{1},...,m_{N}\\}}\in\mathbb{R}^{N\times D}$ to
summarize the key $N$ events occurring in the videos. Given a group of
randomly initialized memory queries, we train a Transformer encoder-
decoder[39] to update them by gradually aggregating the related encoded video
features $\mathbf{X}$ through cross-attention interactions. Specifically, we
design both unsupervised generation and supervised matching approaches
depending on whether event-level annotations are available.

Unsupervised event memory generation. In this case, we employ the idea of
information maximization[21, 29] from the unsupervised learning to generate a
set of event memories. Our goal is to generate a collection of event memories.
To ensure that this set comprehensively captures all the crucial events within
the video, we impose two vital constraints. 1) Individual Certainty. Each
individual event is encouraged to be distinctive, thereby discouraging
individual frames from forming separate and independent events. 2) Global
Diversity. We aim to maintain diversity within the event set, preventing all
video frames from being clustered into only one single event.

Assuming that there are $C$ types of events that will occur in the video. For
individual certainty, we suppose higher classification confidence score.
Namely, the ideal category distribution is similar to one-hot encoding[21].
Specifically, a classifer header $f_{c}$ of a linear layer with softmax is
used to predict the event classification logits
$\hat{\mathbf{p}}_{m}=f_{c}(\mathbf{m})$. The individual certainty loss is
denoted as

$\mathcal{L}_{cert}(f_{c};\mathbf{M})=-\mathbb{E}_{\mathbf{m}\in\mathbf{M}}\sum_{i=1}^{C}\hat{\mathbf{p}}_{m}^{i}\log\hat{\mathbf{p}}_{m}^{i}.$
(2)

To ensure global diversity, we consider both the diversity of semantics and
temporal spans within the event set. To promote semantic diversity, we assign
category labels evenly across all memories. Namely, in contrast to the above
one-hot classification logits of each event, we aim for an even distribution.
The following $\mathcal{L}_{cls}$ is minimized to constitute the semantic
diversity loss

$\mathcal{L}_{cls}(f_{c};\mathbf{M})=\sum_{i=1}^{C}\hat{\mathbf{p}}^{i}\log\hat{\mathbf{p}}^{i},$
(3)

where
$\hat{\mathbf{p}}=\mathbb{E}_{\mathbf{m}\in\mathbf{M}}[f_{c}(\mathbf{m})]$ is
the mean output embedding of the all event memories of a video sample. Next,
to ensure temporal span diversity, we pursue a reduction in the temporal
overlap of all events. We use a temporal header $f_{t}$ of 3-layer FFN with
ReLU to predict the normalized temporal center coordinate $\hat{\delta}$ and
width $\hat{w}$ of events,
$\hat{\tau}=f_{t}(\mathbf{m})=[\hat{\delta},\hat{w}]\in[0,1]^{2}$. The
temporal IoU loss formulated as,

$\mathcal{L}_{iou}(f_{t};\mathbf{M})=\mathbb{E}_{\mathbf{m}\in\mathbf{M}}\sum_{i,j=1}^{N}IoU(\hat{\tau}_{i},\hat{\tau}_{j}),$
(4)

where IoU(.) is the normalized temporal IoU[24] given the paired temporal
prediction of events. The overall global diversity loss is the weighted sum of
$\mathcal{L}_{cls}$ and $\mathcal{L}_{iou}$

$\mathcal{L}_{div}(f_{c};f_{t};\mathbf{M})=\lambda_{cls}\mathcal{L}_{cls}+\lambda_{iou}\mathcal{L}_{iou}.$
(5)

Supervised event memory matching. For datasets with ground-truth event-level
labels, we similarly use the set prediction via bipartite matching [4, 24] to
extract event memories. Since the output of the Transformer is unordered, we
do not have a one-to-one correspondence between the predicted event memories
and the ground-truth events. Following [4, 24], we use the Hungarian algorithm
to find the optimal bipartite matching results. Given the padding ground-truth
event labels $E=\\{e_{i}\\}_{i=1}^{N}$,
$e_{i}=\\{c_{i},\tau_{i}=[\delta_{i},w_{i}]\\}$, where $c$ and $\tau$ are
class labels and the temporal labels respectively, the matching cost
$\mathcal{C}_{match}$ is defined as:

$\mathcal{C}_{match}=\sum_{i=1}^{N}[-\mathbf{1}_{c_{i}\neq\emptyset}\lambda_{cls}\hat{p}_{\pi(i)}^{c_{i}}+\mathbf{1}_{c_{i}\neq\emptyset}\lambda_{L1}||\tau_{i}-\hat{\tau}_{\pi(i)}||],$
(6)

where $\pi\in\Pi_{N}$ is the permutation of $N$ event memories. Then the goal
is to find the optimal $\pi$ to minimize the matching cost.

Memory prompting. At the glancing stage, the output is a set of unordered
event memories $\mathbf{M=\\{m_{1},...m_{N}\\}}$. We reorganize them into
$\widetilde{\mathbf{M}}$ in the sequential order according to their temporal
center coordinate $\\{\delta_{i}\\}$. The temporal positional embedding
$\textbf{P}_{t}\in\\{\Phi(i)|i\in[1,N]\\}$ is added to the corresponding
events to indicate their temporal positions in the video. And they will serve
as the memory prompts during multi-level attention at the following focusing
stage.

###  3.3 Focusing Stage

It is difficult to directly associate the high-level event semantics in the
questions with corresponding low-level videos. Thus the event memories
extracted through video glancing serve as useful anchors to prompt the model
quickly locate event segments related to the questions. Video features
$\mathbf{X}$, memory prompts $\widetilde{\mathbf{M}}+\mathbf{P}_{t}$, and
question embeddings $\mathbf{L}$ are concatenated as input. The Transformer
Encoder embeds these embeddings from different sources into a joint space
$[\mathbf{X}_{enc},\mathbf{M}_{enc},\mathbf{L}_{enc}]$.

Multi-level cross attentions. Given a set of answer queries
$\mathbf{a}\in\mathbb{R}^{N\times D}$, the Transformer Decoder exploits multi-
level cross attention to replace the original multi-head cross-attention to
guide the model gradually focus on the question-related visual clues in the
video.

a) Focus on memory. Given questions with high-level event semantics, we first
use question embeddings as queries to interact with memory prompts to guide
the model focus on the corresponding event memories $\mathbf{M}_{foc}$:

$\displaystyle\mathbf{Q}=\mathbf{L}_{enc},\mathbf{K}=\mathbf{V}=\mathbf{M}_{enc}$
(7) $\displaystyle\mathbf{M}_{foc}=Multihead(\mathbf{Q,K,V}).$ (8)

b) Focus on frame. Then the selected event memories prompt the model to
localize the corresponding video frames $\mathbf{X}_{foc}$ as visual clues for
specific reasoning:

$\displaystyle\mathbf{Q}=\mathbf{M}_{foc},\mathbf{K}=\mathbf{V}=\mathbf{X}_{enc}$
(9) $\displaystyle\mathbf{X}_{foc}=Multihead(\mathbf{Q,K,V}).$ (10)

c) Predict the answer. Finally, the answer queries find the answers
$\mathbf{\hat{a}}$ through reasoning across the focused video frames:

$\displaystyle\mathbf{Q}=\mathbf{a},\mathbf{K}=\mathbf{V}=\mathbf{X}_{foc}$
(11) $\displaystyle\mathbf{\hat{a}}=Multihead(\mathbf{Q,K,V}).$ (12)

Similarly to classifier header $f_{c}$, an answer classifier header $f_{a}$ of
a linear layer with softmax is used to predict the answer label from the last
token of answer queries $\hat{y}=f_{a}(\mathbf{\hat{a}})$.

###  3.4 Loss Functions

The first part of loss functions is the answer classification cross-entropy
loss $\mathcal{L}_{qa}$. The overall unsupervised loss is defined as a linear
combination:

$\mathcal{L}_{uns}(\mathcal{F}_{\theta};q;V;\mathcal{A};y)=\mathcal{L}_{qa}+\lambda_{cert}\mathcal{L}_{cert}+\mathcal{L}_{div}.$
(13)

Under the situation with ground-truth event-level labels, we turn to use them
for supervision by minimizing the event classification cross-entropy loss
$\mathcal{L}_{cls}$ and the temporal span $L1$ loss $\mathcal{L}_{L1}$ between
the matching memories with the ground-truth events:

$\mathcal{L}_{sup}(\mathcal{F}_{\theta};q;V;\mathcal{A};y;E)=\mathcal{L}_{qa}+\lambda_{cls}\mathcal{L}_{cls}+\lambda_{L1}\mathcal{L}_{L1}.$
(14)

##  4 Evaluation

###  4.1 Datasets

We evaluate our model on various recently proposed Multi-Event VideoQA
datasets. The comparison of them is shown in Table 1. Note that we use the v2
version of the AGQA dataset, which has more balanced distributions, as the
dataset creator recommended. We choose these benchmarks for a broad coverage
of: Video content. Including household and general activities, etc. Video
duration. The average video length ranges from 12 seconds to 44 seconds. Event
complexity. All these videos contain multiple events or actions, with the
average event number ranging from 2.7 to 8.8. Event complexity. The number of
event categories varies from 50 to 793.

Datasets | # Avg. | Video | #Event | #Videos | # QAs
---|---|---|---|---|---
| Events/Video | Length | Classes |  |
STAR[42] | 2.7 | 12s | 157 | 22K | 60K
AGQA v2[14] | 6.8 | 30s | 157 | 9.7K | 2.27M
EgoTaskQA[16] | 5 | 25s | 793 | 2K | 40K
NExT-QA[43] | 8.8 | 44s | 50 | 5.4K | 52K

Table 1: Comparison of the Multi-Event VideoQA benchmarks.

###  4.2 Implementation Details

For each benchmark, we follow standard protocols outlined by prior works for
data processing, metrics, and settings to ensure fair comparisons. For video
feature extraction, we use frozen S3D model [46] pre-trained on HowTo100M[32]
on STAR dataset and C3D[38] feature on EgoTaskQA dataset following[47, 16]. We
also try Faster-RCNN model[37] pretrained on the Visual Genome[22] and
CLIP(ViT-B/32) for more challenging AGQA and NExT-QA datasets. We employ a
standard 2-layer, 8-head Transformer Encoder-Decoder with hidden size $D$ of
512 as the backbone for our Glance-Focus model. According to the statistics of
common event categories, the query number $N$ is set to 10 since the maximum
average number of events per video in the current dataset is approximately
8.8. And the event class number $C$ is set to the ground truth overall class
number in the dataset or 100 if it is unknown by default. For training
details, we use dropout of 0.1, and initialize model weights using Xavier
init[13]. Adam optimizer[20] is used with a learning rate of 5e-6 to optimize
model parameters. All experiments are conducted on an NVIDIA GeForce RTX
3090Ti GPU. More details can be found in the supplementary material.

###  4.3 Comparison with State-of-the-arts

We conduct a comprehensive comparison with current state-of-the-arts (SOTA)
VideoQA methods, including traditional spatial-temporal models and large
vision language models. We compare our Glance-Focus model (shorted as GF model
in the following) with them on all four datasets. The results are shown in
Table 2, 3, 4 and 5 respectively. Overall, our proposed GF model achieves SOTA
results on all benchmarks, and obtains significant advantages in both the STAR
and the EgoTaskQA dataset. We will analyze different datasets one by one as
follows. Note that in Table 2-5 the red numbers highlight the significant
gains compared to prior SOTA results.

For STAR dataset, our proposed model obtains excellent reasoning performance
shown in Table 2. When facing all four different types of questions, GF
significantly surpasses the previous SOTAs, especially on the Sequence
category. Such results reflect that our model has strong temporal contextual
modeling ability. What’s more, note that our model trained without ground-
truth event labels, i.e., GF(uns), has achieved comparable results with the
one trained under supervision, i.e., GF(sup). It indicates that our proposed
unsupervised memory generation in the Glance stage can effectively extract key
events with strong discriminability.

Table 2: QA accuracies of SOTA methods on STAR test set (* indicates the val
set results).

Model | Interaction | Sequence | Prediction | Feasibility | Mean
---|---|---|---|---|---
ClipBERT[25] | 39.81 | 43.59 | 32.34 | 31.42 | 36.70
RESERVE-B[50] | 44.80 | 42.40 | 38.80 | 36.20 | 40.50
Flamingo-9B[1] |  - |  - |  - |  - | 43.40
AIO[40]* | 47.53 | 50.81 | 47.75 | 44.08 | 47.54
Temp[ATP][3]* | 50.63 | 52.87 | 49.36 | 40.61 | 48.37
MIST[10]* | 55.59 | 54.23 | 54.24 | 44.48 | 51.13
GF(uns) | 51.91 |  63.06(+8.83) | 54.89 | 45.57 | 53.86(+2.73)
GF(sup) | 56.10 | 61.27(+7.04) | 52.65 | 45.74 |  53.94(+2.81)

We also evaluate our model in the normal split of the EgoTaskQA dataset. From
Table 3, we can find that our model outperforms SOTA methods on almost all
question types. Our unsupervised GF model has strong performance especially on
the intent and explanatory question types. These questions evaluate the
understanding of the humans daily activities, which are all necessary for
building the household robots. And the model trained under supervision
achieves strong performance especially on object and predictive question
types, which indicates that by explicitly modeling the events happening in the
video, the model can obtain stronger event semantic understanding ability,
namely, to recognize the interacted objects and to predict the next action.

Table 3: QA accuracies of SOTA methods on EgoTaskQA normal split.

| Category | VisualBERT[26] | ClipBERT[25] | HCRN[23] | GF(uns) | GF(sup)
---|---|---|---|---|---|---
Scope | world | 39.73 | 42.15 | 44.27 | 44.65 | 46.53
intent | 44.51 | 40.94 | 49.77 |  52.07(+2.30) | 51.88
multi-agent | 26.29 | 27.63 | 31.36 | 33.17 | 34.31
Type | descriptive | 41.99 | 38.45 | 43.48 | 44.48 |  46.68(+3.20)
predictive | 30.37 | 31.50 | 36.56 | 39.00 |  40.36(+3.80)
counterfactual | 41.99 | 46.75 | 48.00 | 46.66 | 46.80
explanatory | 37.42 | 42.39 | 40.60 | 42.45 | 42.18
Semantic | action | 15.02 | 22.91 | 14.92 | 15.68 | 15.68
object | 23.26 | 21.80 | 45.31 | 45.97 |  50.98(+5.67)
state | 59.20 | 54.36 | 68.28 | 70.44 | 71.23
change | 68.27 | 66.58 | 67.38 | 68.69 | 69.71
Overall | open | 24.62 | 27.70 | 30.23 | 31.26 |  32.79(+2.56)
binary | 68.08 | 67.52 | 69.42 | 69.80 | 70.25
all | 37.93 | 39.87 | 42.20 | 43.06 |  44.27(+2.07)

As for AGQA dataset, from Table 4, our model also achieves the SOTA result
with object-level video features, i.e., GF(sup) with Faster R-CNN[37]
backbone. And the one trained with S3D features also gains comparable results.
We show several representative reasoning types of the dataset. Our model
obtains better results on some challenging question types, such as obj-rel
composition questions and temporal superlative questions like “the longest…”.
Overall, our model shows more clear advantage over SOTA on the challenging
Open questions, compared to the relatively simple Binary questions for which
the model could gain a 50% accuracy rate by blind guessing. The complete
breakdown results on all kinds of question types and a more thorough analysis
can be found in the Supplementary Material.

Table 4: QA accuracies of SOTA methods on AGQA v2 test set.

Method | HME[6] | HCRN[23] | AIO[40] | Temp[3] | MIST[10] | GF(uns) | GF(uns) | GF(sup)
---|---|---|---|---|---|---|---|---
Backbone | C3D | C3D | ViT | CLIP | CLIP | S3D | FasterRCNN | FasterRCNN
obj-rel | 37.42 | 40.33 | 48.34 | 50.15 | 51.68 | 52.93 | 54.31 |  54.96(+3.28)
superlative | 33.21 | 33.55 | 37.53 | 39.78 | 42.05 | 42.78 | 42.90 |  44.62(+2.57)
sequencing | 49.77 | 49.70 | 49.61 | 48.25 | 67.24 | 53.03 | 51.97 | 53.24
exist | 49.96 | 50.01 | 50.81 | 51.79 | 60.33 | 58.31 | 58.08 | 59.13
duration | 47.03 | 43.84 | 45.36 | 49.59 | 54.62 | 50.86 | 52.02 | 52.80
act. recog. | 5.43 | 5.52 | 18.97 | 18.96 | 19.69 |  22.08(+2.39) | 16.38 | 14.17
open | 31.01 | 36.34 | - | - | 50.56 | 53.06 | 55.66 |  56.07(+5.51)
binary | 48.91 | 47.97 | - | - | 58.28 | 53.61 | 53.52 | 54.17
all | 39.89 | 42.11 | 48.59 | 49.79 | 54.39 | 53.33 | 54.59 | 55.08

We also evaluate our model on the challenging NExT-QA dataset based on
different video backbones. Note that the NExT-QA dataset does not provide
event annotations. Therefore, we exploit the proposed unsupervised memory
generation in the Glance stage. As shown in Table 5, without event-level
supervision or applying any extra action detection model, the unsupervised GF
models with the same backbone as other methods still achieve SOTA
performances. Such results further justify the strong multi-event reasoning
ability of our method, considering that NExT-QA is currently the most complex
Multi-Event VideoQA benchmark with longer video duration (44s) and richer
events (8.8 actions per video).

What’s more, we also report the breakdown results by the different question
types. According to the results, GF has strong event reasoning ability
especially on Descriptive (Acc@D) questions, improving the accuracy by 3.61%,
and also on event relationship reasoning, achieving 2.31% accuracy gain on
Causal (Acc@C) questions. As recommended in ATP[3], we additionally report
results on its proposed ATP-hard subset of the NExT-QA validation set. This
subset filters out those “easy” questions that can be answered with a single
frame. The results in Table 5 show that our GF models significantly surpass
HGA[17] models on this subset by improving 5.22% accuracy overall. The above
results indicate that compared to HGA and other methods that directly model
the correspondence between question and long video through co-attention, our
method using event memory as an intermediate bridge to establish such
correlation for reasoning is more efficient.

Table 5: Accuracy (%) comparison on NExT-QA validation set and the ATP-hard
subset[3]. Acc@C, T, D, denote accuracy for Causal, Temporal, and Descriptive
questions respectively.

Methods | Backbone | val | val (ATP-hard subset)
---|---|---|---
Acc@C | Acc@T | Acc@D | Acc@All | Acc@C | Acc@T | Acc@All
ATP[3] | CLIP | 48.30 | 49.30 | 65.00 | 49.20 | 19.60 | 22.60 | 20.20
VQA-T[47] | S3D | 49.60 | 51.49 | 63.19 | 52.32 | - | - | -
IGV*[28] | C3D | 48.56 | 51.67 | 59.64 | 51.34 | - | - | -
Temp[ATP][3] | CLIP | 53.10 | 50.20 | 66.80 | 54.30 | 38.40 | 36.50 | 38.80
HGA[17] | C3D | 46.26 | 50.74 | 59.33 | 49.74 | 43.30 | 45.30 | 44.10
MIST[10] | CLIP | 54.62 | 56.64 | 66.92 | 57.18 | - | - | -
GF(uns) | S3D | 51.78 | 54.71 | 63.96 | 54.62(+2.30) | 48.28 | 49.95 | 48.96
GF(uns) | C3D | 53.59 | 55.71 | 66.80 | 56.33(+4.99) | 48.50 | 49.52 | 48.92
GF(uns) | CLIP |  56.93(+2.31) | 57.07 |  70.53(+3.61) | 58.83 |  48.65(+5.35) |  50.27(+4.97) |  49.32(+5.22)

###  4.4 Model Analysis

In this section, we evaluate the effectiveness of our proposed modules by
comparing different variants of the GF models and various baselines. We first
introduce the baselines and GF variants as follows. Baselines: 1) MDETR[18]
which utilizes same 2-layer Transformer Encoder-Decorder architecture. 2) An
Oracle model with ground-truth event labels concatenated to the questions as
textual input as an upper-bound model. GF Variants: 1) Glance-only. Model
which uses standard cross-attention to replace the multi-level attention
strategy. 2) Detect-Focus. Apply a frozen action detection model, such as
SlowFast[7], to replace the Glance stage of the model. We try to keep the
model parameters consistent.

Effect of memory prompting. Firstly, the performance of the Oracle model
significantly improves compared to the baseline model. It convinced us that
event memories do play an important prompting role in the Multi-Event VideoQA
task. Besides, it can be found that with event memory generation at the
glancing stage, the Glance-only model outperforms the MDETR baseline and the
model using a frozen action detection (Detect-Focus) with a large gap on both
datasets. Such results reflect the effectiveness of our proposed memory-
prompting strategy. Compared to additionally fine-tuning an action detection
model, our proposed Glance strategy can adaptively generate key event/action
features of the long videos without any annotations.

What’s more, we evaluate the effectiveness of different loss functions that
are designed during the event memory extraction in Table 7. We carefully
evaluate all pairs of losses on STAR, EgoTaskQA and AGQA datasets. Firstly,
the model with complete losses achieves the best results on both unsupervised
and supervised settings of all datasets. It indicates that each loss plays its
role in generating the event memories. Secondly, under the unsupervised
setting, the $\mathcal{L}_{iou}$ shows an important role in controlling the
temporal prediction of the events. For supervised settings, the
$\mathcal{L}_{cls}$ shows a more important role for the event semantic
supervision helps cross-modal grounding. Please see more discussions on the
loss functions in the Supplementary Material.

Model | STAR | EgoTaskQA
---|---|---
MDETR | 46.26 | 39.68
Detect-Focus | 49.85 | 42.62
Glance-only | 53.12 | 42.70
Glance-Focus | 53.94 | 44.27
Oracle(Upper) | 55.19 | 49.52

Table 6: Module ablations on STAR and EgoTaskQA datasets.

Effect of multi-level attention. From Table 6, compared to the Glance-only
model using standard cross-attention, the Glance-Focus model with our designed
multi-level attention shows stronger reasoning ability on both STAR and
EgoTaskQA datasets. Taking an example from STAR dataset, we further visualize
the attention weights in different levels in Figure 3, the darker the color,
the greater the weight. The attention map in the first row shows the
correspondence of the question embeddings with the generated memories. In this
level, the model focuses on the most related event “Throwing a towel”. The
attention map in the second row is the correspondence between the focused
event and the video embeddings. At this level, the model correctly attends to
the most related keyframes for answer prediction.

Table 7: Ablation study of the loss functions on all datasets.

Uns/Sup | $\mathcal{L}_{cls}^{uns}$ | $\mathcal{L}_{iou}$ | $\mathcal{L}_{cert}$ | $\mathcal{L}_{cls}^{sup}$ | $\mathcal{L}_{L1}$ | AGQA | STAR | EgoTaskQA
---|---|---|---|---|---|---|---|---
Uns | $\checkmark$ | $\checkmark$ |  |  |  | 54.12 | 53.62 | 42.69
Uns | $\checkmark$ |  | $\checkmark$ |  |  | 52.89 | 51.35 | 41.77
Uns |  | $\checkmark$ | $\checkmark$ |  |  | 54.27 | 53.72 | 42.85
Uns | $\checkmark$ | $\checkmark$ | $\checkmark$ |  |  | 54.59 | 53.86 | 43.06
Sup |  |  |  | $\checkmark$ |  | 54.26 | 53.69 | 43.87
Sup |  |  |  |  | $\checkmark$ | 53.19 | 53.03 | 42.64
Sup |  |  |  | $\checkmark$ | $\checkmark$ | 55.08 | 53.94 | 44.27

Figure 3: Visualization of the event memory generation and the multi-level
attention map.

##  5 Conclusion

Our proposed Glance-Focus model provides an innovative approach to the
challenging Multi-Event VideoQA task. With a set of event memories as prompts,
the model can efficiently establish the correspondence between high-level
questions and low-level video frames. Our experiments has shown that event
memories play a crucial role in locating question-related visual clues for
reasoning. The proposed Glance-Focus strategy can obtain a complete and
compact representation of videos without additional annotations, which can be
extended to more video comprehension tasks, such as video
captioning/summarization. We hope this study can inspire more future research
in the field of Multi-Event Video Understanding.

## Acknowledgements

This work is partially supported by National Key R&D Program of China No.
2021ZD0111901, and Natural Science Foundation of China under contracts Nos.
U21B2025, U19B2036.

## References

  * [1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  Advances in Neural Information Processing Systems, 2022.
  * [2] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman.  Frozen in time: A joint video and image encoder for end-to-end retrieval.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1728–1738, 2021.
  * [3] Shyamal Buch, Cristóbal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles.  Revisiting the “video” in video-language understanding.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2917–2927, 2022.
  * [4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In Proceedings of the European Conference on Computer Vision, pages 213–229, 2020.
  * [5] Long Hoang Dang, Thao Minh Le, Vuong Le, and Truyen Tran.  Hierarchical object-oriented spatio-temporal reasoning for video question answering.  International Joint Conference on Artificial Intelligence, 2021.
  * [6] Chenyou Fan, Xiaofan Zhang, Shu Zhang, Wensheng Wang, Chi Zhang, and Heng Huang.  Heterogeneous memory enhanced multimodal attention model for video question answering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1999–2007, 2019.
  * [7] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  Slowfast networks for video recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6202–6211, 2019.
  * [8] Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu.  Violet: End-to-end video-language transformers with masked visual-token modeling.  arXiv preprint arXiv:2111.12681, 2021.
  * [9] Difei Gao, Ruiping Wang, Ziyi Bai, and Xilin Chen.  Env-qa: a video question answering benchmark for comprehensive understanding of dynamic environments.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1675–1685, 2021.
  * [10] Difei Gao, Luowei Zhou, Lei Ji, Linchao Zhu, Yi Yang, and Mike Zheng Shou.  Mist: Multi-modal iterative spatial-temporal transformer for long-form video question answering.  arXiv preprint arXiv:2212.09522, 2022.
  * [11] Jiyang Gao, Runzhou Ge, Kan Chen, and Ram Nevatia.  Motion-appearance co-memory networks for video question answering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6576–6585, 2018.
  * [12] Yuying Ge, Yixiao Ge, Xihui Liu, Dian Li, Ying Shan, Xiaohu Qie, and Ping Luo.  Bridgeformer: Bridging video-text retrieval with multiple choice questions.  Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
  * [13] Xavier Glorot and Yoshua Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, pages 249–256. JMLR Workshop and Conference Proceedings, 2010.
  * [14] Madeleine Grunde-McLaughlin, Ranjay Krishna, and Maneesh Agrawala.  Agqa: A benchmark for compositional spatio-temporal reasoning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11287–11297, 2021.
  * [15] Deng Huang, Peihao Chen, Runhao Zeng, Qing Du, Mingkui Tan, and Chuang Gan.  Location-aware graph convolutional networks for video question answering.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 11021–11028, 2020.
  * [16] Baoxiong Jia, Ting Lei, Song-Chun Zhu, and Siyuan Huang.  Egotaskqa: Understanding human tasks in egocentric videos.  Advances in Neural Information Processing Systems, 2022.
  * [17] Pin Jiang and Yahong Han.  Reasoning with heterogeneous graph alignment for video question answering.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 11109–11116, 2020.
  * [18] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion.  Mdetr-modulated detection for end-to-end multi-modal understanding.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1780–1790, 2021.
  * [19] Sungdong Kim, Jin-Hwa Kim, Jiyoung Lee, and Minjoon Seo.  Semi-parametric video-grounded text generation.  arXiv preprint arXiv:2301.11507, 2023.
  * [20] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  International Conference on Learning Representations, 2015.
  * [21] Andreas Krause, Pietro Perona, and Ryan Gomes.  Discriminative clustering by regularized information maximization.  Advances in Neural Information Processing Systems, 23, 2010.
  * [22] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  International Journal of Computer Vision, 123:32–73, 2017.
  * [23] Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran.  Hierarchical conditional relation networks for video question answering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9972–9981, 2020.
  * [24] Jie Lei, Tamara L Berg, and Mohit Bansal.  Detecting moments and highlights in videos via natural language queries.  Advances in Neural Information Processing Systems, 34:11846–11858, 2021.
  * [25] Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu.  Less is more: Clipbert for video-and-language learning via sparse sampling.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7331–7341, 2021.
  * [26] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.  Visualbert: A simple and performant baseline for vision and language.  arXiv preprint arXiv:1908.03557, 2019.
  * [27] Muheng Li, Lei Chen, Yueqi Duan, Zhilan Hu, Jianjiang Feng, Jie Zhou, and Jiwen Lu.  Bridge-prompt: Towards ordinal action understanding in instructional videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19880–19889, 2022.
  * [28] Yicong Li, Xiang Wang, Junbin Xiao, Wei Ji, and Tat-Seng Chua.  Invariant grounding for video question answering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2928–2937, 2022.
  * [29] Jian Liang, Dapeng Hu, and Jiashi Feng.  Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation.  In International Conference on Machine Learning, pages 6028–6039, 2020.
  * [30] Fei Liu, Jing Liu, Weining Wang, and Hanqing Lu.  Hair: Hierarchical visual-semantic relational reasoning for video question answering.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1698–1707, 2021.
  * [31] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692, 2019.
  * [32] Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic.  Howto100m: Learning a text-video embedding by watching hundred million narrated video clips.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2630–2640, 2019.
  * [33] Tushar Nagarajan, Yanghao Li, Christoph Feichtenhofer, and Kristen Grauman.  Ego-topo: Environment affordances from egocentric video.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 163–172, 2020.
  * [34] Megha Nawhal and Greg Mori.  Activity graph transformer for temporal action localization.  arXiv preprint arXiv:2101.08540, 2021.
  * [35] Jungin Park, Jiyoung Lee, and Kwanghoon Sohn.  Bridge to answer: Structure-aware graph interaction network for video question answering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15526–15535, 2021.
  * [36] Tianwen Qian, Ran Cui, Jingjing Chen, Pai Peng, Xiaowei Guo, and Yu-Gang Jiang.  Locate before answering: Answer guided question localization for video question answering.  IEEE Transactions on Multimedia, 2023.
  * [37] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in Neural Information Processing Systems, pages 91–99, 2015.
  * [38] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri.  Learning spatiotemporal features with 3d convolutional networks.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4489–4497, 2015.
  * [39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in Neural Information Processing Systems, 30, 2017.
  * [40] Alex Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  All in one: Exploring unified video-language pre-training.  Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
  * [41] Jianyu Wang, Bing-Kun Bao, and Changsheng Xu.  Dualvgr: A dual-visual graph reasoning unit for video question answering.  IEEE Transactions on Multimedia, pages 3369–3380, 2021.
  * [42] Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan.  Star: A benchmark for situated reasoning in real-world videos.  In Advances in Neural Information Processing Systems, 2021.
  * [43] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua.  Next-qa: Next phase of question-answering to explaining temporal actions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9777–9786, 2021.
  * [44] Junbin Xiao, Angela Yao, Zhiyuan Liu, Yicong Li, Wei Ji, and Tat-Seng Chua.  Video as conditional graph hierarchy for multi-granular question answering.  Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
  * [45] Junbin Xiao, Pan Zhou, Tat-Seng Chua, and Shuicheng Yan.  Video graph transformer for video question answering.  In Proceedings of the European Conference on Computer Vision, pages 39–58, 2022.
  * [46] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy.  Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification.  In Proceedings of the European Conference on Computer Vision, pages 305–321, 2018.
  * [47] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid.  Just ask: Learning to answer questions from millions of narrated videos.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1686–1697, 2021.
  * [48] Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal.  Self-chained image-language model for video localization and question answering.  arXiv preprint arXiv:2305.06988, 2023.
  * [49] Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian.  Deep modular co-attention networks for visual question answering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6281–6290, 2019.
  * [50] Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi.  Merlot reserve: Multimodal neural script knowledge through vision and language and sound.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
  * [51] Zhou Zhao, Zhu Zhang, Shuwen Xiao, Zhenxin Xiao, Xiaohui Yan, Jun Yu, Deng Cai, and Fei Wu.  Long-form video question answering via dynamic hierarchical reinforced networks.  IEEE Transactions on Image Processing, pages 5939–5952, 2019.
  * [52] Linchao Zhu and Yi Yang.  Actbert: Learning global-local video-text representations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8746–8755, 2020.


# Social Data Analysis: A Study on Friend Rating Influence

Shubhanshu Gupta
Truebil
Mumbai, India
Email<EMAIL_ADDRESS>Vaibhavi Desai
Google India, Bangalore
Email<EMAIL_ADDRESS>Harsh Thakkar
Enterprise Information Systems Lab,
University of Bonn
Bonn, Germany
Email<EMAIL_ADDRESS>
###### Abstract

Social Networking accounts for a significant chunk of interest among various
online activities [11]. The proclivity of being social, online, has been
ingrained in us so much that we are actively producing content for the rest of
the world to see or take interest in our whereabouts, our meals, our opinions,
photographs etc. Yelp111Yelp (https://www.yelp.com/), seamlessly, integrates
this very aspect of people in its portal. It engages people to write reviews
about the businesses they have availed the services of, rate them, add
photographs, tags, follow other people and their activities, etc. In this
paper we examine and present the co-relation between a user’s rating and the
influence of the people, that the user follows, on the user for a particular
business. The group of people that the user follows is commonly referred as
friends of the user. We also analyze if a user can get influenced, if a
business has a certain number of reviews already present or if the reviews
have been written by elite reviewers (a reviewer who, according to Yelp, has
contributed exceptionally in engaging the community in the form of consistency
in writing reviews, as well as the quality of the reviews). Our analysis,
through correlation and regression techniques, is able to prove that the
user’s rating remains unaffected by the number of people a user was friends
with nor does the existing number of reviews and presence of elite reviewers
helps in influencing a user. What shapes a user’s rating is the overall
experience, that the user had at the restaurant.

###### Index Terms:

Yelp data analysis, social network analysis, regression, visualization,
correlation

##  I Introduction

The service based industry is growing rapidly in order to keep up with the
pace of our lifestyle changes and serve our data intensive needs
accordingly222source (https://www.plunkettresearch.com/industries/ecommerce-
internet-technology-market-research/). The recent galore of Internet based
service industry [1], has presented immense possibilities and control over
how, when and which service to choose from the plethora of existing ones [7,
8]. This trend has sparked the onset of various such Internet based solutions,
that enable the community to rate and/or review homogeneous genres of
businesses. It empowers anyone having a particular requirement to be able to
look anytime for the best possible service, which is available at a distance
convenient to them. Such community driven Online platforms also allow
businesses to serve their customers better [2, 10], enabling a unique and
transparent marketing channel for the businesses.

Yelp is one such very popular, community driven effort which provides a
platform to curate user reviews and ratings for local businesses, e.g.
restaurants, salons, auto services, healthcare services etc. The quality of a
business can be gauged by the average rating of the business, and the number
and the connotation of the reviews tend to support the rating [6]. Yelp allows
its users to formulate social circles within each other by adding other users
as friends. This enables the users to share with each other their
feedback/experience regarding satisfaction with respect to a particular
business. In such a social-driven rating scenario, it is often argued that the
user business service preference and reviews are prejudiced to a certain
degree derived by mutual friendship and trust [12, 7].

We observe, from the yelp data, that the businesses, Food and Restaurant,
contribute as the second largest category333https://www.yelp.com/factsheet of
all the business reviewed by Yelp users. We choose these two top categories of
businesses for our study. In this paper, we examine: (i) the impact of
friends’ experience on a user’s rating, if any. For instance, whether or not,
a user rating is indicative of the experience that his friends had; and (ii)
if there is any social influence of the number of reviewers or the presence of
elite reviewers, on a user while rating a restaurant. For our study we employ
the following approaches:

  * •

Firstly, we carry out Pearson Correlation test on user rating and user
friends’ average rating. Along with that, using P value test, we also conduct
a null hypothesis test. and,

  * •

Secondly, we perform Linear Regression over user rating and user friends’
average rating to gauge and visualize the relationship between the former and
later. Also, Linear Regression technique aids us with a few more statistical
measurements to verify our claims.

Both the approaches were also implemented for our second research objective,
.i.e, understanding relationship between: user rating and number of reviewers;
user rating and number of elite reviewers.

The remainder of this paper is organised as follows: In Section II we discuss
the related work with respect to our current work. Section III presents the
dataset and tools used in our work, Section  IV portrays the experimental
setup that we did in the process. In Section V we discuss the findings from
two main approaches employed and finally Section VI concludes the paper by
discussing the final outcome and the key statistics that helped us understand
the results.

##  II Related Work

Yelp provides a rich dataset which has engendered a lot of research in the
field of human behavior, social network analysis, recommendation techniques,
etc. Our study has an essence of each of the above mentioned fields. In this
section, we portray how our current work addresses the limitations of other
related research efforts. Moreover, we discuss the idea of the contribution of
various virtual or fictitious factors in influencing, primarily, the
engagement of the community on the Yelp platform, rather than the user rating.

One of our major arguments that a user rating is predominantly, a reflection
of his/her experience at the restaurant, is also proved with the help of Ridge
regression technique [9]. This study is, however, based on the sentiments of
the reviews written. The author zeroes down on a bag of 300 most popular and
frequently occurring words in the reviews, and their present the maximum
chances of influencing the ratings of a user. And these words describe the
customers’ experience, feel, facilities, ambiance, and quality of the
neighborhood of the restaurant.

In one of the other studies [4] which aligns to ours, in the sense, it looks
at the overall sensitivity of the fluctuations in a restaurant rating and
revenue as an attribute of 50 or more reviewers of that restaurant. It also
studies the same fluctuations as an attribute of elite status of the
reviewers. These are the people who have met certain criteria, decided by
Yelp, who are lauded with such status so that they could give some more
reliability or credibility to the reviews about the restaurant they rate. And
hence, the author implies that in both of the cases above, a restaurant’s
overall perception and rating improves when the numbers of reviews are more
than 50 and also that the rating is susceptible to people’s opinion based on
reviews by elite reviewers.

Whereas in our study, we touch a more humane aspect, in the sense that, how a
user while rating a restaurant would not necessarily change his/her opinion
altogether if he/she observes that, that restaurant had 50+ reviews or had
reviews written by elite reviewers. In fact, user’s opinions are formed as a
result of several other factors and the actual experiences. Although the
author’s observation that the number of friends a reviewer might have on the
website, may not necessarily impact his/her review about a restaurant, aligns
with our observations as well. Although there is one difference, in the sense,
that in this case, the author is weighting the overall rating by the number of
friends each reviewer has. Whereas we do a case-by-case study, by finding out
the pattern and behavior for every user rating for every restaurant versus the
average ratings given by the user’s friends to the same restaurant that the
user rated and then we expand and generalize this observation to the entire
dataset.

Quite some work has been done in the field of user star rating prediction. In
their study [5], the authors carry out user star rating prediction by
analyzing the user and business features using similarity measurement
algorithms. The study only takes into account, the user’s past ratings in
restaurants serving different categories of food. One of the other study,
predicts user start rating based solely on the text of user generated reviews
[3]. There also exists work, which discusses the contribution of social
network in predicting user rating [12]. This study makes use of latent-factor
model and then adds the user’s friends’ contribution to the business’s
ratings, in the model.

While our study does not discuss the textual analysis of user reviews, the
past ratings of the user or the latent-factor model in predicting user
reviews, but it certainly does a lot more analysis than the above three
studies. Our study discusses the various other aspects that determine how a
user would rate a restaurant. From analysing the impact of friends experience,
to the presence of elite reviewers and the number of overall reviewers, we
also discuss upon the extrensic and intrinsic features of a restaurant like
neighborhood quality, ambience and service quality among many others.

##  III Dataset and Tools

###  III-A Yelp Dataset

We use the data from the famous Yelp Dataset Challenge444Yelp dataset
(https://www.yelp.com/dataset_challenge). We have primarily used three subsets
of the Yelp dataset for our research, namely: business, reviews and user. All
these three subsets are in the form of individual JSON files. The dataset
statistics are as follows:

  * •

The business subset includes information of about 77K businesses and provides
the following information, about a particular business: type, unique id, name,
city, state, stars (which are rounded to half-stars), number of reviews, the
category that the business falls in.

  * •

The review subset includes information of about 2.2M reviews and tells us the
following attributes of every review:the type of review, to which business id
it belongs to, the user id who has written that review, number of stars given,
date, votes, and the review text.

  * •

The user subset on the other hand gives information about 552K users and each
user has: the unique id, the name, review counts, average stars of the user,
friend ids, elite status in years, date of joining (in the form of Yelping
since), compliments, fans, votes.

###  III-B User ratings

In all of the above cases, stars or ratings, range from 1 to 5. And usually,
the ratings are rounded off to half stars. So for example, a restaurant whose
average rating is of 3.45, would have a rounded-off rating of 3.5. Also, a
user is given an elite status for a particular year. He may or may not fall
under the elite category year-on-year. And one of the ways in which a user
usually makes a cut into the elite category, as per Yelp, is by contributing
really good quality content (reviews, photos, tips, check-ins etc) to the
platform.

###  III-C Tools

For our experiments, we used NumPy555NumPy library (http://www.numpy.org/),
Scikit-learn666Scikit learn (http://scikit-learn.org/stable/), Pandas777Pandas
data analysis library (http://pandas.pydata.org/), Matplotlib888Matplotlib
(http://matplotlib.org/) libraries and code base is on iPython notebook.
Moreover we specifically used the Pandas library for data analysis, Matplotlib
for visualizations, Scikit-learn for carrying out regression analysis and
NumPy for numerical calculations. We provide our complete source code publicly
for research purposes here999https://github.com/shubhanshu-gupta/yelp-data-
challenge.

##  IV Experimental setup

###  IV-A Trimming the data

As mentioned previously in Section I, we use only those businesses that are
mentioned in the ’Food’ and ’Restaurant’ category. Subsequently, the user
subset containing information of about 552K users is trimmed down to
approximately 250K by selecting only those users, who have atleast one (i.e.
$>=1$) friend in their social circle. We divide the reviews file, having
information of about 2.2M reviews, into multiple smaller segments for the ease
of data handling.

On performing a primitive data analysis on the first quarter of reviews
subset, we observe that the star category distribution is biased towards 4 and
5 stars, ref. Figure 1. These comprise of about 63% reviews against the all
the combined businesses having 1, 2, or 3 star ratings.

Figure 1: The star category distribution of the 1st quarter of reviews dataset

###  IV-B Joining relevant data subsets

Subsequently, the first quarter of reviews (about 556K) is inner-joined with
the key ids of users, so as to specifically get only those users who have
written a review. This gives us a dataset of about 394K records. Also, this
dataset is inner joined with the business subset on business ids (will be
addressed as ’business merged dataset’ in the rest of the paper), so as to
specifically get only those businesses that are rated or reviewed by our
select corpus of users. Having about 394K records, we select only those
businesses which fall in the ’Restaurant’ or ’Food’ categories, thereby
focusing only on 252K records.

###  IV-C Building data dictionaries

In order to handle this large amount of data (i.e. more than 252K rows)
efficiently, we build hash-maps or ordered dictionaries. The first dictionary
that we build is of user (key) and list of their friends (value). We traverse
two datasets: the user subset; the ’business merged dataset’. The former is
traversed in order to fetch users and their friends, thereby leading our way
to the first dictionary. And the latter, is grouped by business ids which
gives us the lists of user ids and user ratings, as values, for every business
id as the key - and this forms our second dictionary. This dictionary looks
something like ref. Figure 2

Figure 2: The Key (business id) - Value (list of dictionaries containing user
ids and user ratings) pairs of second dictionary

###  IV-D Deriving experiment-ready ’final dataset’

We traverse the business merged dataset along with using the ancillary
dictionaries to build a dataset of derived columns. So for every user and
business, the dataset finally comprises of derived columns including:

  * •

Number of friends who reviewed the same business as the user did

  * •

Average friends rating

  * •

Number of elite reviewers

  * •

Number of friends who were elite

  * •

Total number of reviewers etc.

This results in a dataset of about 90K records which contains the information
of about 4K unique restaurants. Hence, each record in our final dataset
consists of a business id, the user id of the reviewer, business average
rating, total friends this user has, average rating of these friends of user,
number of friends who rated this business id, total number of reviewers who
rated this business, & number of elite reviewers. We, from this point onwards,
address this dataset as ’final dataset’.

Figure 3: Star category distribution in the ’final dataset’

It can be visualized that the star rating distribution of the derived ’final
dataset’, has a little different pattern from its earlier counterpart ref.
Figure 3. Here, about 90K users give ratings which are more skewed towards 4.0
and 3.5 stars for about 4K unique businesses. They comprise of about 75% of
the total records whereas all the others combined, comprise of just 25% of the
total records.

##  V Social impact analysis on yelp ratings

In this section we analyze, whether the ratings given on Yelp have a social
edge or not. By social edge, we try to understand how a user rating for a
business could be influenced by his/her friends’ opinion for that same
business.

###  V-A Correlation analysis

Correlation basically measures the linear correlation between two variables,
user rating and friends’ average rating, in our case. The correlation value
varies between -1 to +1, wherein 0 indicates that there is no relationship
between the two variables. And the perfect correlation of -1 or +1 implies
that both the variables are exactly linearly related.

The correlation value for the friends’ average rating column and the user
rating column in our ’final dataset’ obtained is 0.3219. This, although, is on
a positive side, but is very weak to imply any strong linear relationship
between user rating and friends’ average rating. Our null hypothesis that,
there exists a linear relationship between user rating and user friends
average ratings is rejected by the meager p-value. The p-value is recorded
below a bare minimum threshold which made it indistinguishable and therefore,
came up to be 0.0. Hence, our alternative hypothesis, that there is an
invalidated relationship between user and friends average ratings, gets
proved.

Table I: The matrix of correlation between User and Friends’ Average Ratings |  | Friend’s average
---
rating
| User’s average
---
rating
| Friend’s average
---
rating
1.0000 | 0.3219
| User’s average
---
rating
0.3219 | 1.0000
Table II: Results of Pearson Correlation Test R value | P value | Stderr
---|---|---
0.3219 | 0.0 | 0.0040

###  V-B Regression analysis

In order to visualize and validate the above hypothesis, we further decided to
do a regression analysis on our ’final dataset’. We divided the ’final
dataset’ into 70-30 ratio of training and test data, respectively. The
dependent variable was friends’ average rating column and the independent
variable was user rating column. The coefficient value was 0.225, and an
intercept value was 2.984. And with the R square value of just 0.088, it means
that more than 90% of the variation is residing in the residuals. But, to be
sure about how truthful the value of R square was, in indicating how close our
predicted data is to the fitted regression line, we plot a scatterplot to get
a sense of how well the data seems to fit.

Figure 4: The combination of the scatter plot (black) between User and
Friends’ Average Ratings and the plot (blue) of the linear relationship
between the two.

The plot ref. Figure 4 is a visual evidence of the fact that as the user
rating increases or decreases, friends average rating does not follow the same
trend, and instead it proves that every person rates based on his/her own
experience with the business. The user rating and the predicted user friends’
average ratings are represented by the blue line, and the scatter plot of
friends’ average ratings and user ratings are represented by the black dots.
The complete vertical black dots indicate that for a user rating, for ex- 2,
the friends’ average could vary from 1 to 5, extensively. And same holds true
for higher user ratings of 4 and 5. This certainly affirms that the
correlation between Friends’ average ratings and user ratings is negligible,
which gives us the confidence to derive the conclusion that for a user who
rated highly (or low) for a business, the friends in his/her social group may
not have the same experience with the same business.

Furthermore, we also check for the presence of any correlation between the
user ratings, a business receives and the number of reviewers it has. We
normalize the dataset, comprising of number of reviewers column as the
dependent variable and user rating column as the independent variable.
Followed the same approach as of regression analysis and found that with a
meager R square value of 0.009, almost all the variations reside in residuals.
And to verify further, we also ran a Pearson test from the Scipy library and
obtain the R value to be 0.09, P value was in negative power of e, and std err
was 0.005. Adding up all of the above results, it’s pretty evident that a user
while rating a business does not get influenced by the number of reviews that
business has.

Finally, we also run similar experiments to understand whether there is any
influence of the presence of reviews by elite users on a general user rating.
And, since in all of the above experiments we mostly came up with a similar
conclusion that a user rates based on his/her experience solely, it came as a
no surprise that here also, a user does not get influenced while rating for a
business which has reviews by users who are of elite status.

###  V-C Results

Our study of the level of satisfaction or dissatisfaction for a user and
his/her experience at a restaurant, as a reflection of his/her social circle
on Yelp, number of reviews and the reputation of other reviewers is validated
by a number of statistical figures. We’ll go through them briskly to
understand how they shaped and affirmed our study.

Pearson correlation coefficient or the R value101010Pearson coefficient-
http://goo.gl/hSHI4I, measures how strongly two variables are related linearly
two each other. The coefficient which ranges between +1 to -1 (where both the
extremes depict best fit), indicate the level of association between those two
variables. A simple figure to understand the R value can be understood from
ref. Figure 5.

The second thing we elaborate is P value111111P value (http://goo.gl/xq4jmJ).
It helps in determining the significance of the results when a hypothesis is
tested. Our null hypothesis was that there existed a linear relationship
between user rating and friends’ average ratings. The value of P-value was so
meager which weighed our evidences (the data points in our final dataset) so
correctly that we were not only assured of our methodology by which we derived
our final dataset but also it helped us reject our null hypothesis. Usually
this happens when the P value is less than equal to 0.05.

Next, we elaborate our observations from linear regression analysis. It’s a
predictive modeling methodology which draws out a fine relationship (both, if
there is some or none at all) between a dependent variable and an independent
variable121212Regression Techniques (http://goo.gl/GqXEzh). Like, Pearson
correlation coefficient, it helps in indicating the strength of relationship
between the two variables and thereby, predicts the future patterns on the
similar lines. Hence, while performing the regression analysis (we performed
linear regression), we are served with multiple statistical figures like
coefficient, slope, intercept etc. The plot of our linear regression depicts
and the r square value measures the closeness of our data to the fitted (blue)
regression line. R square131313Interpreting R square (http://goo.gl/xj5ceu)
which can also be understood by the ratio of explained variation to the total
variation and it ranges between 0 and 1. In general, when the R square value
is high, the model represents a good fit our dataset. Since, our R square
value was so low, that also became a key indicator of the lack of variance
accounted by our regression model which made our data points spread all across
the fitted regression line. This was also supported by our residual plots
which, as a result housed more than 90% of our variations.

Figure 5: The figures seemingly portray the correlation between the data
points, with the appropriate R values.

##  VI Conclusions

In this paper, we examined and evaluated the presence or more aptly, the
absence of the contribution of factors that define how a user would rate
his/her experience with a business. We, in particular, identify whether a user
rating is more or less influenced, by modest extrinsic factors surrounding a
restaurant and several intrinsic factors that lie within a restaurant [4, 2].
For instance, the level of satisfaction or dissatisfaction reflected by a user
in their rating for a business could be driven by: (a) service (slow, good,
bad, etc.), (b) food quality, (c) quantity or ambiance with respect to the
price of the meal, (d) overall experience (and hygiene) at the locality and
also (e) the popularity of the neighborhood in which restaurant is situated
[10].

On the other hand, there is no direct relationship between how a group of
people, whom we simply follow on Yelp, might have rated a restaurant. In
reality we may not even know these people, and neither their tastes, or
preferences. And that’s why, a user’s social circle defined within Yelp’s
enclosure does not contribute much to a user’s rating - which has been
corroborated by our research findings. There is also no relationship between
how a user would rate a restaurant and how many reviews that restaurant
already holds or how many of the reviewers of the restaurant are/have been
elite as per Yelp. These factors are, probably, best to substantiate the
overall picture of a restaurant that Yelp portrays with the help of aggregated
reviews, ratings, photographs, etc. They do help to sort and filter out a
restaurant based on an individual’s or a group’s mood, taste and preference at
that point of time, but none of these factors determine how the individual or
a group would feel or experience, while they are actually at the restaurant,
having their meal.

## References

  * [1] J. Bughin, L. Corb, J. Manyika, O. Nottebohm, M. Chui, B. de Muller Barbat, and R. Said.  The impact of internet technologies: Search.  High Tech Practice, 2011.
  * [2] E. A. Dwyer.  Price, perceived value and customer satisfaction: A text-based econometric analysis of yelp! reviews.  2015\.
  * [3] M. Fan and M. Khademi.  Predicting a business star in yelp from its reviews text alone.  arXiv preprint arXiv:1401.0864, 2014.
  * [4] P. Hajas, L. Gutierrez, and M. S. Krishnamoorthy.  Analysis of yelp reviews.  arXiv preprint arXiv:1407.1443, 2014.
  * [5] J. Han, J. Kuang, and D. Lim.  Predicting yelp ratings from business and user characteristics.
  * [6] J. Huang, S. Rogers, and E. Joo.  Improving restaurants by extracting subtopics from yelp reviews.  iConference 2014 (Social Media Expo), 2014.
  * [7] Z. Iqbal, R. Verma, and R. Baran.  Understanding consumer choices and preferences in transaction-based e-services.  Journal of Service Research, 6(1):51–65, 2003.
  * [8] H. Li, C. Kuo, and M. G. Rusell.  The impact of perceived channel utilities, shopping orientations, and demographics on the consumer’s online buying behavior.  Journal of Computer-Mediated Communication, 5(2):0–0, 1999.
  * [9] M. Luca.  Reviews, reputation, and revenue: The case of yelp. com.  Com (September 16, 2011). Harvard Business School NOM Unit Working Paper, (12-016), 2011.
  * [10] D. Sharma.  Identifying influential factors for yelp business ratings.
  * [11] T. Smith.  The social media revolution.  International journal of market research, 51(4):559–561, 2009.
  * [12] W. Yang, Y. Yuan, and N. Zhang.  Predicting yelp ratings using user friendship network information.  2015\.