# From a Social Cognitive Perspective: Context-aware Visual Social
Relationship Recognition

Shiwei Wu, Chao Zhang, Joya Chen, Tong Xu, , Likang Wu, Yao Hu,
Enhong Chen  Shiwei Wu, Chao Zhang, Tong Xu, Likang Wu, and Enhong Chen are
with the Anhui Province Key Lab of Big Data Analysis and Application, School
of Data Science and School of Computer Science, University of Science and
Technology of China. E-mail: {dwustc<EMAIL_ADDRESS><EMAIL_ADDRESS>{tongxu<EMAIL_ADDRESS>Joya Chen is with the College
of Design and Engineering, National University of Singapore. E-mail:
<EMAIL_ADDRESS>Yao Hu is with the Xiaohongshu.Inc E-mail:
<EMAIL_ADDRESS>
###### Index Terms:

social relationship, multi-modal analysis, vision-linguistic contrasting.

People’s social relationships are often manifested through their surroundings,
with certain objects or interactions acting as symbols for specific
relationships, e.g., wedding rings, roses, hugs, or holding hands. This brings
unique challenges to recognizing social relationships, requiring understanding
and capturing the essence of these contexts from visual appearances. However,
current methods of social relationship understanding rely on the basic
classification paradigm of detected persons and objects, which fails to
understand the comprehensive context and often overlooks decisive social
factors, especially subtle visual cues. To highlight the social-aware context
and intricate details, we propose a novel approach that recognizes Contextual
Social Relationships (ConSoR) from a social cognitive perspective.
Specifically, to incorporate social-aware semantics, we build a lightweight
adapter upon the frozen CLIP to learn social concepts via our novel multi-
modal side adapter tuning mechanism. Further, we construct social-aware
descriptive language prompts (e.g., scene, activity, objects, emotions) with
social relationships for each image, and then compel ConSoR to concentrate
more intensively on the decisive visual social factors via visual-linguistic
contrasting. Impressively, ConSoR outperforms previous methods with a 12.2%
gain on the People-in-Social-Context (PISC) dataset and a 9.8% increase on the
People-in-Photo-Album (PIPA) benchmark. Furthermore, we observe that ConSoR
excels at finding critical visual evidence to reveal social relationships.

##  I Introduction

Figure 1: ConSoR excels at identifying decisive visual social cues. The
explicit clues (e.g., party scene in (a)) determine the probable intimate
relation, while ConSoR aids in uncovering implicit social cues (e.g., the
presence of the child and their cuddle interaction), further pinpointing the
accurate Couple relation. Additionally, the observed ‘transitivity’ property
in social relations also supports this conclusion. In (b), the visual-
linguistic contrasting helps to capture the undetected yet critical contextual
social cues (e.g., flowers and intimate stares), which have been overlooked by
previous methods.

> “No man is an island,
> entire of itself;
> every man is a piece of the continent,
> a part of the main.”
> — John Donne, No Man Is an Island

In today’s interconnected world, understanding social relationships has grown
increasingly vital across numerous domains. Gaining insights into social
relations contributes beneficially to human health behaviors, encompassing
chronic illness self-management and reducing suicidal tendencies [1, 2, 3];
facilitates the design of intelligent robots with emergent “social behavior”
[4, 5, 6, 7]; and also underpins semantic-related services, including
personalized content recommendations and targeted advertising [8, 9, 10].

Unfortunately, while we humans can easily comprehend social relations among
individuals through various comprehensive cues such as interactions, facial
expressions, clothing styles, common patterns, and rich contextual information
[11, 12], it remains an extremely challenging task for intelligent systems to
automatically capture social relations by comprehending these massive
potential cues, primarily due to the huge gap between visual evidence and
high-level semantics. This limitation severely restricts the application of
social relation factors in various domains. To address this issue, numerous
efforts have been made employing a wide range of technical approaches. Most
existing methods primarily focus on tackling this task through structural
modeling of people and detected objects based on interactions or co-
occurrences [13, 14, 15, 16], following a visual relation classification
paradigm from a purely visual perspective. However, these previous methods
cannot capture the critical yet implicit social clues effectively and often
miss the decisive factor since they are constrained within the limited
detected objects (e.g., within 80 categories in COCO-based [17] detectors),
and a lack of deep semantic knowledge about certain objects or interactions
(e.g., a man and a woman, both wearing wedding rings, are holding hands,
suggesting that they are a couple, while ‘rings’ are not even included in the
COCO object categories). To tackle this issue, we utilize a visual-linguistic
contrasting approach, encouraging models to concentrate on the social-decisive
visual factors, as well as incorporating social-aware semantics from pre-
trained CLIP [18] models, which further capture implicit social cues from the
comprehensive context.

Let’s examine Figure 1 as an example. While we can infer that the social
relation between the man and woman shown in Fig. 1a is likely intimate based
on explicit semantics(e.g., the party scene), it is still challenging to
distinguish the specific social relation among similar intimate relationships
(Couple, Family or Friend). However, by inducing the model to concentrate on
the social-aware contexts, ConSoR can capture the implicit yet critical
social-relevant contextual cues, such as the presence of a child in the scene
and their cuddling interaction. This helps to identify that the individuals
are parents with children, further confirming that the two adults are couple.
This scenario also exemplifies the ‘transitivity’ property in social
relations. Moreover, the visual-linguistic contrasting with the descriptive
social prompts approach helps ConSoR focus on the undetected yet decisive
social clues (flowers, intimate stares in Figure 1b), which have been ignored
by the previous approaches.

Motivated by these observations, in this paper, we propose ConSoR to
concentrate on the social-cognitive contexts through the visual-linguistic
contrasting approach, further recognizing social relations among individuals.
Specifically, we first introduce the novel Multi-modal Side Adapter Tuning
(MSAT), which transfers rich semantic knowledge from both CLIP visual and text
encoders into our lightweight multi-modal backbone through a hybrid feature
fusion strategy. Moreover, in the visual branch, we perform relation reasoning
from both context and interpersonal relation perspectives to examine the
visual factors that help determine social relations among individuals, without
the heavy object detector. Simultaneously, in the linguistic branch, we move
beyond simple one-hot vector classification, which overlooks the rich
information language offers. Instead, we utilize social-relevant contextual
corpora to generate descriptive prompts for each category label. Notably, the
candidate contextual social descriptive prompts for each image differ only in
the social relation category. Therefore, employing a visual-linguistic
contrasting approach allows the model to infer specific social relations more
accurately, concentrate on crucial social factors, and comprehend intrinsic
semantic connections among extensive social clues. This method also introduces
a degree of inherent explainability. Our technical contributions are
summarized as follows:

  * •

We propose ConSoR, which transfers rich semantics from CLIP via the novel
Multi-modal Side Adapter Tuning mechanism, and further employs a visual-
linguistic contrasting approach to steer models towards focusing on critical
social-decisive contexts.

  * •

Our innovative Contextual Interpersonal Reasoning module demonstrates a simple
yet effective approach for relation reasoning. This design facilitates not
only the modeling of the intrinsic social properties (e.g., transitivity,
symmetry, and reflexivity), but also the exploitation of potential contextual
cues without relying on a heavy object detector.

  * •

We incorporate linguistic information through descriptive social prompts
without extra annotations, fully utilizing the rich information language
offers, beyond the limitations of traditional one-hot vector classification.

  * •

Extensive validations on benchmark datasets demonstrate that ConSoR
significantly outperforms SOTA baselines.

##  II Related Work

In general, the related work could be roughly grouped into three parts, namely
Social Relation Theory, Social Relation Recognition in Computer Vision (CV)
area and Transfer CLIP Knowledge into Multi-modal Tasks.

###  II-A Social Relation Theory

The study of social relationships [12, 19, 20, 11, 4, 21, 22, 23, 24, 25] is
fundamental to social sciences, serving as the social cognitive sources across
numerous domains from social-affect health research [1, 2, 3] to socially
interactive robots [4, 5, 6, 7], as well as the visual social relation
recognition fields [26, 13, 15, 27, 28].

A widely endorsed unified theory of social relations [12] simplifies social
relations into four elementary prototypes, i.e., communal sharing (CS),
equality matching (EM), authority ranking (AR), and market pricing (MP). This
theory posits that the motivation, planning, production, comprehension,
coordination, and evaluation of human social life are largely based on
combinations of these four psychological models. Different taxonomies [13, 29,
27, 30] of social relations can also be viewed as combinations of these four
elementary forms of sociality. Each social relation exhibits unique properties
[4, 25, 22, 24, 21]; for example, CS and AR relations involve principles of
reflexivity and transitivity, while the CS relation is symmetric, AR is
antisymmetric [12]. To model these characteristics and adopt a holistic group
perspective, ConSoR introduces interpersonal modules, effectively representing
the structure and patterns of the interpersonal behavior cycle [11, 23, 22,
31, 32].

Further, social relations can be visually captured from a social-cognitive
perspective. For instance, individuals engaged in Communal Sharing (CS) often
seek similarity with others, a tendency that is particularly pronounced during
adolescence. In this stage, conforming to the norms of one’s reference group
or clique in aspects like dress, speech, and preferences becomes crucial [11,
12]. Additionally, specific objects serve as static markers representing
specific social relations. For example, European thrones, crowns, and scepters
may symbolize Authority Ranking (AR) relations, while relics, roses, and
wedding rings often represent CS relations. To model these nuances, we first
employ a multi-modal side adapter tuning mechanism, to incorporate social-
aware semantic knowledge from pre-trained CLIP models, enabling the model to
understand what specific objects and interactions represent. Subsequently, we
encourage the model to focus on these decisive social-visual context factors
and common social patterns through a visual-linguistic contrasting approach,
with social-aware descriptive language prompts.

###  II-B Social Relation Recognition in CV

Social relation recognition in computer vision (CV) [13, 14, 15, 28, 33, 26,
34, 35, 36, 37, 38, 39, 40, 41, 28, 42, 43] has attracted wide attention in
both academia and industry recently. This task involves predicting the social
relation between pairs of people in images, using both explicit and implicit
cues within each individual image. Along this line, two large-scale datasets,
namely The People in Photo Albums (PIPA) [44, 29] and the People in Social
Context (PISC) [13] are regarded as benchmarks for this task.

Most of the prior arts attempted to capture the potential information for
social relation inference via explicit modeling of detected human traits [29,
45, 44] and objects [13], or by incorporating knowledge graph [14]. For
instance, [45] utilized a deep neural network to capture social relation-
related attributes, such as gender, expression, and head pose, via detected
human face. [13] proposed a dual glance model where the first glance fixates
on modeling the individual pair of interests and the second glance deploys the
attention mechanism on context cues using detected objects. [14] exploited the
benefits by integrating structured knowledge graphs constructed by the
additional object detectors. Graph-based methods [37, 15, 28, 43] attempt to
model multiple relationships as graphs, exploring logical constraints among
various social relations. Though these models attempt to learn the specific
social patterns based on human interactions or co-occurrence, they may ignore
the critical yet subtle contextual social clues such as undetected objects due
to the constraints of pre-trained object detectors. Additionally, they can
hardly capture the intrinsic semantic connections among massive details due to
the relatively small dataset. To this end, we obtain the semantic contextual
information in a free-form way via the top-down attention mechanism, and
further explore the social-aware semantics via the visual-linguistic
contrasting approach.

###  II-C Transfer CLIP Knowledge into Multi-modal Tasks

Pre-trained on millions of text-image pairs using a contrastive approach, CLIP
[18] exhibits a remarkable ability to align visual and language modalities,
making it suitable for numerous multimodal downstream tasks [46, 47, 48, 49,
50, 51, 52]. For instance, ViLD [46] distills the knowledge from the teacher
CLIP model into a two-stage student detector, while SAN [47] attaches a side
network to reuse CLIP features and makes the predicted mask proposals
semantically aware in the context of the semantic segmentation task. PointCLIP
[53] designs an inter-view adapter to transfer 2D CLIP knowledge to the 3D
recognition task. Inspired by them, ConSoR first utilizes a multi-modal side
adapter network (MSAT) to transfer visual-linguistic semantics from CLIP,
which can also mitigate the severe overfitting issues in training. We also
find that the parameter-shared among both visual-linguistic adapters performs
better since it establishes a more semantically aligned cross-modal space.
Besides, there are several previous works that attempt to explore the
semantics of label text. Instead of mapping labels to numbers or one-hot
vectors, ActionCLIP [54] treats the action recognition task as a video-label
text matching problem. DetCLIP [55] and CapDet [56] further approach the
recognition task as a region-category matching task and utilize language to
unify different tasks. In contrast to merely using label text as language
input, ConSoR constructs social-aware descriptive language prompts for images,
and further encourages the models to focus on the social-decisive visual
contexts through the visual-linguistic contrasting.

##  III Methodology

Figure 2: This figure depicts the ConSoR framework, where the frozen visual
and linguistic CLIP encoders employ a shared multi-modal side adapter to learn
social-aware representations. Then, the contextual interpersonal reasoning
module extracts individual-pair features, followed by a CLIP-style visual-
linguistic contrasting head for social relationship classification.

To address the aforementioned problems, we propose the ConSoR framework as
shown in Figure 2, which recognizes social relationships visually from a
social cognitive perspective. Specifically, given an image $I$ and a social-
relation class set $R=\\{r_{1},r_{2},\dots,r_{C}\\}$ (where $C$ is the number
of social relations), we extend the side adapter tuning approach [57, 58, 47]
to the multi-modal field, to transfer rich social-aware semantics from CLIP
into our lightweight backbone. Moreover, we perform relation reasoning from
both contextual and interpersonal perspectives to examine the visual factors
and common social patterns that may suggest social relations. Then, we
construct social-aware descriptive language prompts for each image, and then
encourage the model to focus on social-decisive contexts via a visual-
linguistic contrasting approach. We now turn to illustrate the technical
details of the ConSoR framework in the following part.

###  III-A Multi-modal Side Adapter Tuning

Specific objects and interactions such as crowns, roses, wedding rings and
kiss often act as semantic markers of certain social relations. While these
meanings are readily understood by humans, models, particularly when trained
on relatively small datasets like those in the social relation recognition
field, struggle to grasp these social concepts. Thus, in this section, we
introduce the proposed Multimodal Side Adapter Tuning (MSAT) module, which can
efficiently transfer the social semantics from CLIP [18] models into our
lightweight multi-modal transformer backbone. By sharing parameters and latent
knowledge across the visual and linguistic branches, MSAT not only obtains
more semantically aligned representations across modalities, but also
mitigated the severe overfitting issues compared with finetuning CLIP
directly.

We denote the feature of the CLIP image and text encoder’s $i$-th layer as
$\mathcal{V}^{CLIP}_{i}\in\mathbb{R}^{L^{v}\times r^{v}}$ and
$\mathcal{T}^{CLIP}_{i}\in\mathbb{R}^{L^{t}\times r^{t}}$, where $L^{v}$ and
$L^{t}$ means the number of image patches and words, $r^{v}$ and $r^{t}$ is
the visual and text feature dimensions. For global image and text input
embedding in CLIP, we denote the [CLS] token in image patches and [EOT] token
in text sequences as $\mathcal{V}^{CLIP}\in\mathbb{R}^{r}$ and
$\mathcal{T}^{CLIP}\in\mathbb{R}^{r}$ (where $r$ is the default dimension of
CLIP). In addition, $\mathcal{V}^{SN}_{i}$ and $\mathcal{T}^{SN}_{i}$
represent the image and text output of our multi-modal side adapter network’s
$i$-th layer.

As shown in the side adapter network part of Figure 2, the input image is
split into $16\times 16$ patches and then embed into patch embeddings via a
linear embedding layer. Further, the position embeddings are added to form the
image representation $\mathcal{V}^{SN}_{0}$ for the side adapter network. For
text inputs, following [58], we use the downsampled linear projections of the
summation of word embeddings and position embeddings from the CLIP text
encoder $\mathcal{T}^{CLIP}_{0}$ as the text inputs $\mathcal{T}^{SN}_{0}$ of
the side adapter network, which significantly saves memory usage for a large
number of word embeddings.

To efficiently transfer the rich social-aware semantics from CLIP into our
lightweight backbone, we construct our lightweight side adapter network
comprising just 4 transformer layers and it is worth noting that we are the
first to explore the side adapter tuning in the multi-modal settings, finding
that the parameter-shared MSAT achieves the best performance when tuning both
modalities. For the visual branch, we incorporate the {0, 3, 6, 9, 12}-th
intermediate features of CLIP’s image encoder into the {0, 1, 2, 3, 4}-th
hidden states of the side adapter network, respectively. Similarly, for the
linguistic branch, the {3, 6, 9, 12}-th intermediate features of CLIP’s text
encoder are integrated into the {1, 2, 3, 4}-th hidden states of the side
adapter network. Specifically, we fuse the $i$-th intermediate CLIP visual
feature into the $j$-th layer of our side adapter network via the gated
mechanism as follows:

$\hat{\mathcal{V}}^{SN}_{j}=\mu_{j}^{v}\mathcal{V}^{SN}_{j}+(1-\mu_{j}^{v})\mathcal{V}^{CLIP}_{i},$
(1) $\mu_{j}^{v}=sigmoid(\frac{\alpha_{j}^{v}}{\tau}),$ (2)

where $\alpha_{j}^{v}$ is a learnable gate scalar for the visual branch and
$\tau$ is temperature. Similarly, we fuse the latent linguistic knowledge from
CLIP into our side adapter network.

$\hat{\mathcal{T}}^{SN}_{j}=\mu_{j}^{t}\mathcal{T}^{SN}_{j}+(1-\mu_{j}^{t})\mathcal{T}^{CLIP}_{i},$
(3) $\mu_{j}^{t}=sigmoid(\frac{\alpha_{j}^{t}}{\tau}),$ (4)

where $\alpha_{j}^{t}$ is a learnable gate scalar for the linguistic branch.

After fusing the multi-modal semantics from CLIP, we then utilize the side
adapter network, which consists of multi-head self-attention (MSA) layers
[59], to obtain the representation of the image
$\mathcal{V}^{SN}\in\mathbb{R}^{L^{v}\times r}$ and the feature of the
corresponding social prompt $\mathcal{T}^{SN}\in\mathbb{R}^{L^{t}\times r}$ as
follows,

$\mathcal{V}^{SN}_{j+1}=MSA(\hat{\mathcal{V}}^{SN}_{j})$ (5)
$\mathcal{T}^{SN}_{j+1}=MSA(\hat{\mathcal{T}}^{SN}_{j})$ (6)

By sharing parameters between the visual and linguistic spaces, the multi-
modal side adapter network learns representations that are more semantically
aligned across modalities, which significantly enhances the performance on
this task.

Figure 3: Illustration of our proposed interpersonal and contextual reasoning
modules. The interpersonal reasoning module can model the crucial properties
of social relations, such as transitivity and reflexivity. Furthermore, the
model attends to social-aware contexts, such as ‘holding hands’ in the image,
through the contextual reasoning module.

###  III-B Contextual Interpersonal Reasoning

After obtaining the social-aware semantic knowledge through the MSAT module,
we now turn our attention to examining which visual factors are beneficial for
social relation inference. Taking into account the impact from interpersonal
and contextual perspectives, we propose the Contextual Interpersonal Reasoning
Module (CIR), which consists of three parts, i.e., interpersonal reasoning,
contextual reasoning, and global context fusion. Some prior studies [13, 29]
treat each individual pair as mutually independent, neglecting the influence
of others. Although this approach can differentiate between individual
relationships, it often falls short in performance due to the oversight of
critical social relation properties such as transitivity, reflexivity, or
symmetry [12, 4, 25, 22, 24, 21]. Consider Figure 1 as an example: identifying
the parent-child relationship between the baby and the two adults can
reinforce our understanding of the couple relationship between the man and
woman, demonstrating the principle of transitivity.

Therefore, as shown in Figure 3, we utilize the interpersonal reasoning
submodule to model this crucial social pattern. Given $N$ person bounding
boxes $\\{b_{1},b_{2},\dots,b_{N}\\}$, we use ROIAlign [60] operation to
extract the features of the certain individual from the MSAT knowledge-
injected visual feature $\mathcal{V}^{SN}$ as follows:

$\mathbf{P}_{i}=ROIAlign(\mathcal{V}^{SN},b_{i}),$ (7)

where $\mathbf{P}_{i}\in\mathbb{R}^{r}$ is the person feature in the bounding
box $b_{i}$.

Then, we perform the interpersonal reasoning via the $N_{I}$ layers of multi-
head self-attention (MSA) mechanism,

$\hat{\mathbf{P}}=MSA(\mathbf{P})$ (8)

where $\mathbf{P}=\\{\mathbf{P}_{1},\dots,\mathbf{P}_{N}\\}$ represents the
concatenation of all $N$ individuals’ features, and
$\hat{\mathbf{P}}=\\{\hat{\mathbf{P}}_{1},\dots,\hat{\mathbf{P}}_{N}\\}$
denotes the features incorporating interpersonal influences.

Figure 4: The pipeline for constructing descriptive contextual social prompts.
With social prompts, ConSoR integrates rich linguistic information into visual
social relation tasks without requiring additional annotations. This
integration further enables the model to identify social-decisive contexts
through the following visual-linguistic contrasting classification.

Moreover, contextual information within an image plays a crucial role in
inferring underlying social relations. For example, in an office setting
characterized by laptops, monitors, and pens, the relationship between
individuals is likely to be that of colleagues. However, the focus on specific
contextual elements should vary among different individuals. Consider a scene
in a church crowded with people: when inferring the relationship between the
bride and groom on stage, the model should concentrate on items like roses,
the white dress, wedding rings, and their intimate interactions. In contrast,
when judging the relationships of others in the same scene, the model should
consider different contextual elements. Unfortunately, previous methods [45,
44, 13, 14, 27, 28] overlook this crucial social principle, they rely on
additional object detectors [61] to identify all the objects or pre-defined
traits in the image and then feed all the detected contexts with the target
individual-pair to infer the social relation. This kind of approach has three
primary drawbacks: first, it requires heavy object detectors and significantly
depends on their performance, which might miss subtle yet critical objects
such as wedding rings, as well as lack of capturing interaction details like
the holding hands in Figure 3; second, it applies the same contextual analysis
to all individuals in the image, potentially introducing noise in the
inference of specific individual-pair relations.

To this end, we propose the contextual reasoning submodule to discover the
social-decisive visual cues that may suggest social relationships among
individuals in a free-form manner, which can adaptively focus on specific
social-decisive contexts for different individual-pairs. Specifically, we
utilize a $N_{C}$ layers of Transformer-based Decoder to perform contextual
reasoning given the global feature map $\mathcal{V}^{SN}$,

$\hat{\mathbf{U}}=Concat(\hat{\mathbf{P}_{i}},\hat{\mathbf{P}_{j}})$ (9)

$\overline{\mathbf{U}}=Decoder(\hat{\mathbf{U}},\mathcal{V}^{SN})$ (10)

where $(\hat{\mathbf{P}_{i}},\hat{\mathbf{P}_{j}})$ are the target individual-
pair features. Through the cross-attention mechanism in Equation 10, the
potentially related objects could be discovered as shown in Figure 6.

Further, to incorporate the global context information $\mathcal{V}^{CLIP}$
from CLIP visual branch into the pair feature $\overline{\mathbf{U}}$, we
propose the global context fusion as follows,

$\mathbf{z}=sigmoid(\mathbf{W}_{g}^{u}\overline{\mathbf{U}}+\mathbf{W}_{g}^{clip}\mathcal{V}^{CLIP}+\mathbf{b}_{g}),$
(11)
$\mathbf{U}=\mathbf{z}\circ\overline{\mathbf{U}}+(1-\mathbf{z})\circ\mathcal{V}^{CLIP},$
(12)

where $\mathbf{W}_{g}^{u}$ and $\mathbf{W}_{g}^{clip}$ are gated matrices for
$\overline{\mathbf{U}}$ and $\mathcal{V}^{CLIP}$, $\circ$ means the element-
wise multiplication and $\mathbf{b}_{g}$ for the bias term. We finally obtain
the feature of the given individual-pair fusing with comprehensive semantic
contextual information $\mathbf{U}\in\mathbb{R}^{r}$ through the MSAT and CIR
modules.

###  III-C Descriptive Social Prompt Construction via Self-generated Visual-
vocabs

Beyond simple one-hot vector classification, previous approaches [54, 55, 56]
in CLIP downstream tasks have explored the use of semantic label text.
However, these methods fall short as they do not fully leverage the rich
contextual information available in the language, resulting in a lack of an
intermediate understanding of the rationale behind category selection. To
overcome this limitation, we introduce social-aware descriptive prompts
(termed ‘Social Prompts’) that articulate the contextual social states of
images. This approach not only encourages models to focus on social-decisive
factors through a visual-linguistic contrasting method, but also provides a
degree of inherent explainability. As illustrated in Figure 4, there are three
steps to construct the descriptive social prompt: i). visual-vocabs prompt
construction, ii). visual-vocabs zero-shot selection, and iii). social prompt
construction.

In visual-vocabs prompt construction, we use $4$ social-relevant corpora to
generate visual-vocabs based on the image, including scene categories
$\\{v_{1}^{SC},...,v_{|SC|}^{SC}\\}$ and scene attributes
$\\{v_{1}^{SA},...,v_{|SA|}^{SA}\\}$ from Place-365 dataset [62], objects
categories $\\{v_{1}^{OC},...,v_{|OC|}^{OC}\\}$ from ImageNet-1k dataset [63],
and emotion categories $\\{v_{1}^{E},...,v_{|E|}^{E}\\}$ from Plutchnik’s
Wheel of Emotions [64], where $|SC|$, $|SA|$, $|OC|$ and $|E|$ are the size of
respective corpora. Each corpus is a distinct set of social-relevant domain-
specific vocabulary and the statistic information and examples of corpora are
shown in Table LABEL:tab:corpora.

Utilizing CLIP’s robust zero-shot capabilities, we then select visual-vocabs
from each corpus in a zero-shot manner. We construct prompts with templates
depicted in Table LABEL:tab:template for each vocab in the corpus. Given an
image $I$ and all candidate visual-vocab prompts, we utilize CLIP dual encoder
to encode and further filter out the related visual-vocabs $Vocab_{I}$ based
on cosine similarity. We select top $5$ relevant visual-vocabs
$Vocab_{I}=\\{v_{t_{1}}^{SC},...,v_{t_{5}}^{SC},v_{t_{1}}^{SA},...,v_{t_{5}}^{SA},v_{t_{1}}^{OC},...,v_{t_{5}}^{OC},v_{t_{1}}^{E}\\}$
for each corpus (top $1$ for emotion corpus due to its relatively smaller
size).

Further, we construct social prompts with self-selected $Vocab_{I}$. For each
social relation $r_{i}\in R$, we prepend class name prompt ”In this photo, the
social relation of this person pair is [$r_{i}$].” with visual-vocab prompt
”The photo is taken in [$v_{t_{1}}^{SC}$],…,[$v_{t_{5}}^{SC}$]. This scene
attribute of the image are [$v_{t_{1}}^{SA}$],…,[$v_{t_{5}}^{SA}$]. There are
[$v_{t_{1}}^{OC}$],…,[$v_{t_{5}}^{OC}$] in the photo. This emotion in this
photo is [$v_{t_{1}}^{E}$].” to form the social prompt $P_{I}^{r_{i}}$. Then
we encode $P_{I}^{r_{i}}$ with our side adapter backbone and use the [EOT]
token from $\mathcal{T}^{SN}$ as the social prompt embedding, denoted as
$\mathbf{T}^{sp}_{r_{i}}$ for the candidate social relation $r_{i}\in R$.

TABLE I: Social-Relevant Corpora Corpora | #words | Examples
---|---|---
Scene Category [62] | $365$ | movie theater, classroom, office
Scene Attribute [62] | $94$ | sports, shopping, playing, camping
Object Category [63] | $1000$ | bow-tie, rubber, shopping basket
Emotion Category [64] | $24$ | joy, trust, interest, grief, surprise
TABLE II: Visual-vocabs prompt templates for each social-relevant contextual corpus. Corpora | Prompt Template
---|---
Scene Category [62] | ”The photo is taken in [$v_{i}^{SC}$].”
Scene Attribute [62] | ”The scene attribute of the image is [$v_{i}^{SA}$].”
Object Category [63] | ”There are [$v_{i}^{OC}$] in the photo.”
Emotion Category [64] | ”The emotion in this photo is [$v_{i}^{E}$].”
TABLE III: Experimental results on PISC and PIPA dataset. We outperform SOTA methods by achieving +12.2 mAP improvement on the PISC-Fine dataset, which is considered the primary metric for evaluating visual social relation recognition task. (Int.: Intimate, Non.: Non-Intimate, NoR.: No Relation, Fri.: Friend, Fam.: Family, Cou.: Couple, Pro.: Professional, Com.: Commercial.) The PIPA-Coarse/PIPA-Fine dataset is 5/16-class classification, respectively. Methods | PISC Coarse | PISC Fine | PIPA Coarse | PIPA Fine
---|---|---|---|---
Int. | Non. | NoR. | mAP | Fri. | Fam. | Cou. | Pro. | Com. | NoR. | mAP | Acc@1 | Acc@1
Pair-Box-CNN [13] | 70.3 | 80.5 | 38.8 | 65.1 | 30.2 | 59.1 | 69.4 | 57.5 | 41.9 | 34.2 | 48.2 | 65.9 | 58.0
Dual-Glance [13] | 73.1 | 84.2 | 59.6 | 79.7 | 35.4 | 68.1 | 76.3 | 70.3 | 57.6 | 60.9 | 63.2 | - | 59.6
GRM [14] | 81.7 | 73.4 | 65.5 | 82.8 | 59.6 | 64.4 | 58.6 | 76.6 | 39.5 | 67.7 | 68.7 | - | 62.3
MGR [34] | - | - | - | - | 64.6 | 67.8 | 60.5 | 76.8 | 34.7 | 70.4 | 70.0 | - | 64.4
SRG-GN [37] | - | - | - | - | 25.2 | 80.0 | 100.0 | 78.4 | 83.3 | 62.5 | 71.6 | - | 53.6
GR2N [15] | 81.6 | 74.3 | 70.8 | 83.1 | 60.8 | 65.9 | 84.8 | 73.0 | 51.7 | 70.4 | 72.7 | 72.3 | 64.3
HF-SRGR [36] | 89.1 | 87.0 | 75.5 | 84.6 | 82.2 | 39.4 | 33.2 | 60.0 | 47.7 | 71.8 | 73.3 | - | 65.9
GA-GCN [40] | 87.3 | 86.0 | 89.8 | 87.7 | 63.1 | 73.5 | 78.3 | 82.7 | 76.8 | 71.8 | 73.6 | - | 66.6
SRR-LGR [41] | 89.6 | 84.6 | 78.5 | 84.8 | 83.9 | 52.4 | 35.9 | 64.0 | 54.0 | 63.6 | 73.0 | - | 66.1
PRM [39] | - | - | - | - | 56.5 | 70.8 | 72.3 | 78.5 | 81.4 | 68.3 | 73.2 | 73.2 | 65.6
ConSoR (Ours) | 91.3 | 78.7 | 71.0 | 90.2 (+2.5) | 80.9 | 80.2 | 75.8 | 90.4 | 45.2 | 76.1 | 85.8 (+12.2) | 82.4 (+9.2) | 76.4 (+9.8)

###  III-D Visual-linguistic Contrasting Classification via Social Prompt

To further facilitate the model’s focus on social-decisive contexts, we employ
visual-linguistic contrasting between $\mathbf{U}$ from CIR module and the
candidate social prompt representations
$\\{\mathbf{T}^{sp}_{r_{1}},\dots,\mathbf{T}^{sp}_{r_{C}}\\}$. It is worth
noting that $\mathbf{T}^{sp}_{r_{i}}$ are slightly different, as the candidate
social prompts only vary in the prepended class name prompt. As a result,
social-relevant contextual clues can be further examined by applying
contrastive learning between $\mathbf{U}$ and
$\mathbf{T}^{sp}_{r_{i}}\in\\{\mathbf{T}^{sp}_{r_{1}},\dots,\mathbf{T}^{sp}_{r_{C}}\\}$.
Specifically, we compute the cosine similarity between the individual-pair
feature and all candidate social prompts features. Then, we apply softmax
activation to compute the cross-entropy loss as follows, The loss for ConSoR
can be written as:

$\mathbf{z}=[sim(\mathbf{T}^{sp}_{r_{1}},\mathbf{U}),\dots,sim(\mathbf{T}^{sp}_{r_{C}},\mathbf{U})]$
(13) $L_{ConSoR}=L_{CE}(softmax(\mathbf{z}),y_{u})$ (14)

where $sim(a,b)=a^{\top}b/(\|a\|\|b\|)$, $y_{u}$ denotes the social relation
label of the individual-pair, and the $L_{CE}$ is the cross entropy loss. By
employing a visual-linguistic contrasting classification paradigm, ConSoR is
able to adaptively and thoroughly investigate the crucial, social-decisive
contexts.

##  IV Experiments

In this section, we conduct detailed experiments to validate the effectiveness
of ConSoR.

###  IV-A Experimental Settings

Datasets. We utilize two benchmark datasets of social relation recognition for
effectiveness validation, i.e. PIPA dataset [29] and PISC dataset [13]. The
PIPA dataset divides social life into 5 broad domains, and 16 finer social
relations. Following [29, 15], we adopt the overall accuracy across all
categories for evaluation on PIPA dataset. The PISC dataset features a
hierarchical structure of 3 general relationship categories (intimate, non-
intimate, and no relation) and 6 detailed categories (friend, family, couple,
professional, commercial, and no relation). We follow the standard
train/validation/test split [13] for fair comparison. For the PISC dataset,
following [13], evaluation of methods is conducted using per-class recall and
mean Average Precision (mAP) metrics.

Implementation Details. We leverage AdamW [65] as the optimizer. We set the
learning rate of all trainable parameters to $10^{-4}$, weight decay to
$0.05$, training epoch to $6$, batch size to $32$111We utilize Distributed
Data Parallel training on $2$ GeForce RTX 3090 Ti GPUs and the batch size of
each gpu is $16$.. We utilize CosineAnnealing learning rate schedule to adjust
the learning rate. We adopt the pre-trained CLIP ViT-B/16 [18] as vision
encoder and the input images are resized to $224\times 224$. For MSAT, we set
the transformer to $4$ layers, the number of heads to $6$, the dimension of
transformer layers to $192$, the patch size to $16$, and temperature
$\tau=0.1$ in MSAT. We set the transformer layer number of interpersonal
reasoning and contextual reasoning to $1$ by default and the number of heads
to $8$.

###  IV-B Experimental Results

We compare our ConSoR with the following methods:

  * •

Pair-box-CNN [13]: Extracts features from two given individuals using ROI
pooling, then concatenates these features and applies a linear layer for
classification.

  * •

Dual-Glance [13]: Features are extracted by sending two cropped individual
patches and their union region to CNNs. Additionally, the introduced ‘second
glance’ mechanism utilizes surrounding proposals as contextual information to
refine the predictions.

  * •

GRM [14]: Constructs a knowledge graph comprising persons and objects within
an image, and employs a gated graph neural network to identify social
relations.

  * •

MGR [34]: The model utilizes both the person-object graph and the pose graph
of individuals to represent actions between people and objects, as well as
interactions among pairs of people. Subsequently, it infers social relations
using a graph convolutional network.

  * •

SRG-GN [37]: Employs a GRU [66] to process two types of information: single-
image data and relational attributes. It performs inference based on these
inputs and uses the final output of this inference process to classify social
relations.

  * •

GR2N [15]: Reasons out all relationships using multiple relation graphs, which
incorporate logical constraints among various social relations.

  * •

HF-SRGR [36]: Utilizes a social relation graph to extract and fuse hybrid
features. This approach incorporates an attention mechanism into Graph Neural
Networks (GNNs) [67] to generate features and interactions between pairs,
thereby enhancing the reasoning of interactions.

  * •

GA-GCN [40]: Introduces a gaze-aware graph convolutional network designed to
facilitate context-aware social relation inference. This is achieved through a
two-stream graph inference process that incorporates both gaze-aware attention
and distance-aware attention mechanisms.

  * •

SRR-LGR [41]: Leverages graph neural networks to deduce social relations by
analyzing interactions (local information) between social entities and the
global contextual information embedded in the constructed scene-relation
graph.

  * •

PRM [39]: Two ResNet [68] modules are employed: one to extract character
features, and the other to concentrate on the details of the entire graph,
thereby constructing a graph for each relationship.

The overall results are summarized in Table III. Through incorporating rich
semantics from CLIP models as well as encouraging models to focus on the
crucial social-decisive contexts via the visual-linguistic approach, ConSoR
significantly outperforms previous methods, achieving a +12.2 mAP increase on
the PISC-Fine dataset, which is considered the acknowledged primary setting in
the social relation recognition task. The improvement across most categories
suggests that ConSoR exhibits greater robustness compared to other methods.

TABLE IV: Comparison with ConSoR Variants. We performed ablations five times and conducted T-tests between each ablation and ConSoR. The results indicate that ConSoR is statistically significantly superior in most cases. Methods | PISC (mAP) | PIPA (Acc@1)
---|---|---
Coarse | Fine | Coarse | Fine
mean & std. | p-value | mean & std. | p-value | mean & std. | p-value | mean & std. | p-value
CLIP-ZeroShot | 36.20$\pm.00$ | $10^{-11}$ | 30.40$\pm.00$ | $10^{-9}$ | 34.80$\pm.00$ | $10^{-9}$ | 39.00$\pm.00$ | $10^{-7}$
CLIP-Frozen | 86.60$\pm.21$ | $10^{-9}$ | 77.72$\pm.61$ | $10^{-8}$ | 78.72$\pm.54$ | $10^{-5}$ | 73.36$\pm.43$ | 0.003
CLIP-Finetune | 88.22$\pm.26$ | $10^{-6}$ | 82.08$\pm.40$ | $10^{-6}$ | 77.82$\pm.67$ | $10^{-6}$ | 73.00$\pm.37$ | 0.002
ConSoR-Visual | 90.34$\pm.30$ | 0.109 | 84.62$\pm.45$ | 0.094 | 82.38$\pm.54$ | 0.877 | 75.84$\pm.36$ | 0.389
ConSoR-ObjDet | 89.94$\pm.25$ | 0.513 | 84.76$\pm.66$ | 0.274 | 81.28$\pm.57$ | 0.017 | 76.24$\pm 1.21$ | 0.860
ConSoR | 90.24$\pm.21$ | - | 85.80$\pm.51$ | - | 82.44$\pm.64$ | - | 76.38$\pm 1.22$ | -

It is worth noting that Dual-Glance [13] employs an additional object detector
[61] to include contextual object information. It then concatenates these
object features with the individual-pair feature for social relation
prediction. However, this approach can impede the model’s performance when the
contextual objects are non-informative or potentially cause interference.
Though GRM [14] and MGR [34] advance this approach by introducing a graph
attention mechanism, which adaptively reasons about the most relevant
contextual objects according to its proposed graph structure. Two significant
drawbacks remain: 1) The performance of these models heavily relies on the
accuracy of detection results. Crucial yet subtle social-decisive contexts,
such as wedding rings, are essential for social relation judgment but often
overlooked by these detector-based models. 2) The limited size of datasets in
the visual social relation recognition field hinders the model’s ability to
learn the specific meanings of certain objects. For instance, understanding
the relevance of thrones, crowns, and scepters for professional relations, or
roses and wedding gowns for couple relations, remains a challenge. However,
our detector-free ConSoR overcomes these limitations by first learning rich,
social-aware semantics from pre-trained CLIP models through the Multi-modal
Side Adapter Network module. It then attends to social-decisive contexts in a
free-form manner, enabled by the Contextual Interpersonal Reasoning module,
which allows for a more nuanced comprehension of visual social relationships
among individuals from a social cognitive perspective.

In addition, these previous models have overlooked the wealth of contextual
information present in language, treating this task merely as a classic
classification paradigm, i.e., mapping labels to one-hot vectors. In contrast,
our ConSoR innovatively models this task as a multi-modal problem, requiring
no additional annotations. It constructs social-aware descriptive language
prompts for each image and then encourages the model to focus on social-
decisive contexts through a visual-linguistic contrasting approach.

TABLE V: Efficiency analysis of ConSoR variants. All experiments were conducted using two Nvidia GeForce RTX 3090 Ti GPUs with 16 batch-size. ConSoR strikes a good balance between performance and complexity/efficiency Methods | Train.Params.(M) | TotalParams.(M) | GPUMem.(MB/GPU)
---|---|---|---
CLIP-ZeroShot | 0 | 149 | 1,628
CLIP-Frozen | 7.6 | 157 | 2,256
CLIP-Finetune | 157 | 157 | 5,706
ConSoR-Visual | 19.4 | 105 | 2,760
ConSoR-ObjDet | 20 | 214 | 4,535
ConSoR | 20 | 170 | 3,554

###  IV-C Ablation Study

To verify the contribution of each module in our ConSoR framework, we further
design several variants as well as ablations to conduct the ablation study on
our main dataset (i.e., PISC-Fine dataset).

####  IV-C1 Comparison with ConSoR variants.

We examine the benefits of each module in ConSoR by comparing the following
variants.

  * •

CLIP-ZeroShot: Employ only the raw image to perform zero-shot inference in
social relation recognition.

  * •

CLIP-Frozen: Employ the ROIAlign operation [60] to extract visual
representations of individuals from features encoded by the fixed CLIP image
encoder. Subsequently, uses a linear layer to classify the concatenated
features of individual pairs.

  * •

CLIP-Finetune: Similar to CLIP-Frozen, but finetune all CLIP parameters.

  * •

ConSoR-Visual: Use a side adapter network to fine-tune the CLIP image encoder
and utilizes the Contextual Interpersonal Reasoning (CIR) module for relation
reasoning. Subsequently, it predicts social relations using a linear layer.

  * •

ConSoR-ObjDet: Similar to ConSoR, we replace global feature $\mathcal{V}^{SN}$
in CIR decoder with detected-objects, following the same detector setting in
[14].

Figure 5: Ablation for ConSoR on PISC-Fine dataset. In Figure 5aa, ConR, IntR, GCF represents Context Reasoning, Interpersonal Reasoning and Global Context Fusion in CIR module. In Figure 5ab, SC, SA, OC, E means scene category, scene attribute, object category and emotion corpora, respectively. (a)  TABLE VI: Ablations on MSAT Modules. The incorporation of MSAT effectively mitigates severe overfitting issues and enhances efficiency. Methods | PISC (mAP) | PIPA (Acc@1) | Trainable Params. | GPU Memory
---|---|---|---|---
Coarse | Fine | Coarse | Fine | (Million) | (MB/GPU)
ConSoR-Visual w/o MSAT | 89.36$\pm.37$ | 83.58$\pm.42$ | 78.10$\pm.33$ | 72.98$\pm.70$ | 102 | 5,898
ConSoR-V | 90.34$\pm.30$ | 84.62$\pm.45$ | 82.38$\pm.54$ | 75.84$\pm.36$ | 19.4 | 2,760
ConSoR w/o MSAT | 66.94$\pm 2.72$ | 57.32$\pm 3.53$ | 66.46$\pm 1.66$ | 60.88$\pm 2.97$ | 166 | 9,596
ConSoR | 90.24$\pm.21$ | 85.80$\pm.51$ | 82.44$\pm.64$ | 76.38$\pm 1.22$ | 20 | 3,554

As illustrated in Table IV, firstly, it can be observed that CLIP-ZeroShot
notably underperforms compared to the other baselines and variants. This can
be attributed to the significant domain gaps that CLIP’s zero-shot capability
encounters in the social relation recognition domain, which renders its direct
application quite challenging. Subsequently, we employ the ROIAlign operation
to extract individual-pair features and utilize a single linear layer to
bridge the domain gap. The improved performance of CLIP-Frozen, which
surpasses previous methods [13, 14, 15], illustrates the great potential of
integrating rich semantic knowledge into the field of social relation
recognition. We further observe that the model encounters severe overfitting
issues with CLIP-Finetune, as the pre-trained model’s size is not compatible
with the relatively small dataset. Additionally, we find that the
incorporation of MSAT can effectively reduce the overfitting problem. Besides,
the integration of the CIR module allows the model to capture contextual
social clues, leading to a significantly improved performance in comparison to
other baselines. As demonstrated in Table V, our novel multi-modal side
adapter mechanism significantly reduces the model’s complexity, making it much
more lightweight (Trainable Parameters: $157M\rightarrow 19.4M$). By
incorporating language information from descriptive social prompts and further
directing the model’s focus towards social-decisive visual contexts through a
visual-linguistic contrasting approach, ConSoR achieves superior performance
compared to the ConSoR-Visual. It is worth noting that the performance of
ConSoR-Visual on the PISC-Coarse dataset is quite similar to ConSoR, as the
task involves a relatively simple 3-class classification and we observe that
the performance reaches a limit due to the presence of ambiguous relation
annotations within the dataset. Moreover, despite utilizing external heavy
detectors, ConSoR-ObjDet exhibits inferior performance compared to our
approach. This is because our simple yet effective CIR module is more adept at
capturing implicit, decisive social context cues. As illustrated in Table V,
ConSoR achieves an optimal balance between performance and computational
complexity/efficiency.

####  IV-C2 Impact of MSAT

In this part, we validate the effectiveness of MSAT in ConSoR. We propose four
variants of MSAT:

  * •

w/o SN: Utilizes a fixed CLIP encoder without a side adapter network.

  * •

TextSN/VisualSN: Employs a single side adapter network for fine-tuning the
CLIP text or visual encoders.

  * •

DualSN: This variant incorporates two side adapter networks, attaching them
separately to the CLIP visual encoder and the text encoder, with no shared
parameters between the two branches.

  * •

with vanilla ViT/BERT: Replaces the CLIP encoders with the standard vanilla
versions of ViT [69] 222https://huggingface.co/google/vit-base-patch16-384 and
BERT [70] 333https://huggingface.co/google/bert_uncased_L-12_H-512_A-8, each
trained separately on unimodal datasets. The only difference from ConSoR lies
in the encoder weights.

TABLE VII: Performance of different side adapter network methods on PISC-Fine dataset. Methods | mAP | Trainable Params. (M)
---|---|---
w/o SN | 83.4 | 16.5
TextSN | 84.8 | 18.9
VisualSN | 84.5 | 19.4
DualSN | 84.5 | 21.8
with vanilla ViT/BERT | 76.2 | 20.0
MSAT | 85.8 | 20.0
TABLE VIII: Ablation study of different fusion strategies for MSAT on PISC-Fine dataset. Each fusion layer contains the text and visual features specific to its corresponding layer. Strategies |  Fusion Layers |  mAP
---|---|---
w/o fusion |  - |  79.8
Single-layer Fusion |  0th layer |  80.2
3rd layer |  81.7
6th layer |  82.6
9th layer |  83.5
12th layer |  84.1
Multi-layers Fusion |  (9,12)-layers |  85.1
(6,9,12)-layers |  85.5
(3,6,9,12)-layers |  85.6
(0,3,6,9,12)-layers |  85.8
(b)  Figure 6: The visualization cases of ConSoR on PISC-Fine dataset. Texts
in green, blue, and red stand for ground truth, correct predictions, and wrong
predictions, respectively. SP. means social prompt. Best viewed in color.

As presented in Table VII, in the absence of a side adapter network, the model
struggles to adapt to the specific domain and exhibits poorer performance
compared to the model equipped with the side adapter network. Additionally, we
observe that multi-modal signals provide mutual benefits, as both TextSN and
VisualSN underperform MSAT despite having similar numbers of trainable
parameters. Moreover, though DualSN has more parameters than MSAT, by sharing
parameters between the visual and linguistic branches, MSAT establishes a more
semantically aligned cross-modal space, resulting in improved performance
compared to DualSN. Furthermore, compared to training on unimodal data, CLIP
acquires rich semantics through contrastive learning on a massive scale of 400
million image-text pairs, a process crucial for understanding social relations
in ConSoR. Consequently, using the standard vanilla versions of ViT [69] and
BERT [70], which are trained solely on unimodal data, proves to be suboptimal
($85.8mAP\rightarrow 76.2mAP$) as semantic knowledge sources in ConSoR.

As demonstrated in Table VI, the incorporation of MSAT not only enhances
performance by mitigating severe overfitting concerns associated with directly
applying the CLIP model as an encoder for relatively smaller datasets, but
also significantly improves the model’s efficiency.

To further examine the effectiveness of fusion strategies for MSAT, we conduct
an ablation study of various fusion strategies, as shown in Table VIII. We
find that incorporating CLIP features can significantly boost performance, and
integrating more layers leads to improved results. Additionally, the deeper
the layer of information integrated into MSAT, the better its performance, as
deeper features contain more high-level semantic information.

####  IV-C3 Influence of CIR.

As illustrated in Figure 5aa, we observe that incorporating interpersonal and
contextual reasoning significantly improves the model’s performance. Through
context reasoning, ConSoR more effectively identifies socially decisive
contexts, such as wedding rings, flowers, or scepters. Moreover, the ConSoR
decoder-based modeling approach not only achieves superior performance
compared to detector-based methods but is also more lightweight. Through
interpersonal reasoning, ConSoR effectively models crucial social relation
properties such as transitivity, reflexivity, and symmetry in a
straightforward yet efficient manner. Additionally, the inclusion of global
features significantly enhances performance, as the entire background
contributes to the understanding of social relations.

####  IV-C4 Influence of Descriptive Contextual Social Prompt.

We conducted an ablation study on the visual-vocab corpora to assess the
impact of descriptive contextual social prompts, as shown in Figure 5ab. The
findings reveal that a corpus enriched with social-relevant content enhances
performance. This improvement is attributed to the increased semantic
precision of social prompts describing the image, which more effectively
guides the model to identify social-decisive cues through subsequent visual-
linguistic contrasting.

####  IV-C5 Impact of CLIP Backbones.

The results presented in Figure 5ac show that ConSoR’s performance enhances as
the backbone expands. A larger backbone, containing more semantic knowledge,
underscores the excellent scalability of ConSoR.

###  IV-D Case Study

Finally, we turn to discuss some interesting rules based on several case
studies, which are summarized in Figure 6. Specifically, Figure 6a and 6b
demonstrate that by learning the social-aware semantic knowledge from CLIP,
ConSoR can identify specific elements and contexts within the image that are
essential for inferring social relations. For example, as shown in Figure 6a,
the model focuses on the microphone and camera, indicating that the situation
is more likely a professional interview context.

Moreover, we have noticed that the CIR module plays a crucial role in
identifying social relationships. As shown in Figure 6c, ConSoR demonstrates a
greater focus on contextual information, such as items on the shelf, which can
be attributed to the context reasoning submodule.

Furthermore, as shown in Figure 6d, ConSoR accurately infers the relationship
between two distant individuals by detecting the presence of an intermediate
girl. This demonstrates the ‘transitivity’ in social relations, highlighting
the significance of our interpersonal reasoning submodule.

Besides, Figure 6e and 6f illustrate the benefits of integrating descriptive
contextual social prompts into ConSoR, as the inclusion of construction site
and socializing prompts improve the model’s robustness on inferring social
relations.

Finally, as shown in Figure 6g and 6h, we observe that ConSoR cannot predict
“correctly” when faced with ambiguous labels, which is a challenging task even
for humans. It is interesting that ConSoR does not attend to the holding hands
aspect in Figure 6g, as it does not contribute to the social state.

##  V Conclusion

In this paper, we introduce ConSoR, to recognize visual social relationships
from a social cognitive perspective. By integrating social-aware semantics
derived from CLIP models, ConSoR achieves a deeper understanding of social-
aware contexts. This enhanced understanding enables it to excel at identifying
critical visual contextual evidence, revealing social relations through a
visual-linguistic contrasting approach. Extensive experiments on two benchmark
datasets demonstrate the superiority of ConSoR. By accurately interpreting
social relations from social cognitive perspectives, our work provides a solid
foundation for a better understanding of human behavior and society.

## References

  * [1] L. Tay, K. Tan, E. Diener, and E. Gonzalez, “Social relations, health behaviors, and health outcomes: A survey and synthesis,” Applied Psychology: Health and Well-Being, pp. 28–78, 2013.
  * [2] J. Holt-Lunstad, T. B. Smith, and J. B. Layton, “Social relationships and mortality risk: a meta-analytic review,” PLoS medicine, p. e1000316, 2010.
  * [3] D. Umberson and J. Karas Montez, “Social relationships and health: A flashpoint for health policy,” Journal of health and social behavior, pp. S54–S66, 2010.
  * [4] C. B. De Soto, “Learning a social structure.,” The Journal of Abnormal and Social Psychology, p. 417, 1960.
  * [5] R. Beckers, O. E. Holland, and J.-L. Deneubourg, “From local actions to global tasks: Stigmergy and collective robotics,” Prerational Intelligence: Adaptive Behavior and Intelligent Systems Without Symbols and Logic, pp. 1008–1022, 2000.
  * [6] J.-L. Deneubourg, S. Goss, N. Franks, A. Sendova-Franks, C. Detrain, and L. Chrétien, “The dynamics of collective sorting robot-like ants and ant-like robots,” in Proceedings of the International Conference on Simulation of Adaptive Behavior, pp. 356–365, 1991.
  * [7] T. Fong, I. Nourbakhsh, and K. Dautenhahn, “A survey of socially interactive robots,” Robotics and autonomous systems, pp. 143–166, 2003.
  * [8] J. Tang, S. Wang, X. Hu, D. Yin, Y. Bi, Y. Chang, and H. Liu, “Recommendation with social dimensions,” in AAAI, 2016.
  * [9] A. Khelloufi, H. Ning, S. Dhelim, T. Qiu, J. Ma, R. Huang, and L. Atzori, “A social-relationships-based service recommendation system for siot devices,” IEEE Internet of Things Journal, pp. 1859–1870, 2020.
  * [10] L. Guo, L. Sun, R. Jiang, Y. Luo, and X. Zheng, “Recommendation based on attributes and social relationships,” Expert Systems with Applications, p. 121027, 2023.
  * [11] F. Heider, The psychology of interpersonal relations.  Psychology Press, 2013.
  * [12] A. P. Fiske, “The four elementary forms of sociality: framework for a unified theory of social relations.,” Psychological review, p. 689, 1992.
  * [13] J. Li, Y. Wong, Q. Zhao, and M. S. Kankanhalli, “Dual-glance model for deciphering social relationships,” in ICCV, pp. 2650–2659, 2017.
  * [14] Z. Wang, T. Chen, J. Ren, W. Yu, H. Cheng, and L. Lin, “Deep Reasoning with Knowledge Graph for Social Relationship Understanding,” in IJCAI, pp. 1021–1028, 2018.
  * [15] W. Li, Y. Duan, J. Lu, J. Feng, and J. Zhou, “Graph-based social relation reasoning,” in ECCV, pp. 18–34, 2020.
  * [16] A. Kukleva, M. Tapaswi, and I. Laptev, “Learning interactions and relationships between movie characters,” in CVPR, pp. 9849–9858, 2020.
  * [17] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in ECCV, pp. 740–755, Springer, 2014.
  * [18] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., “Learning transferable visual models from natural language supervision,” in ICLR, pp. 8748–8763, 2021.
  * [19] P. Duncan-Jones, “The structure of social relationships: Analysis of a survey instrument: Part 1,” Social psychiatry, pp. 55–61, 1981.
  * [20] H. Tajfel, “Social psychology of intergroup relations,” Annual review of psychology, pp. 1–39, 1982.
  * [21] R. P. Abelson, E. E. Aronson, W. J. McGuire, T. M. Newcomb, M. J. Rosenberg, and P. H. Tannenbaum, “Theories of cognitive consistency: A sourcebook.,” 1968.
  * [22] C. B. De Soto and J. L. Kuethe, “Perception of mathematical properties of interpersonal relations,” Perceptual and Motor Skills, pp. 279–286, 1958.
  * [23] C. B. De Soto and J. L. Kuethe, “Subjective probabilities of interpersonal relationships.,” The Journal of Abnormal and Social Psychology, p. 290, 1959.
  * [24] C. B. De Soto, M. London, and S. Handel, “Social reasoning and spatial paralogic.,” Journal of Personality and Social Psychology, p. 513, 1965.
  * [25] C. B. De Soto and J. J. Bosley, “The cognitive structure of a social structure.,” The Journal of Abnormal and Social Psychology, p. 303, 1962.
  * [26] J. Li, Y. Wong, Q. Zhao, and M. S. Kankanhalli, “Visual social relationship recognition,” IJCV, pp. 1750–1764, 2020.
  * [27] X. Liu, W. Liu, M. Zhang, J. Chen, L. Gao, C. Yan, and T. Mei, “Social relation recognition from videos via multi-scale spatial-temporal reasoning,” in CVPR, pp. 3566–3574, 2019.
  * [28] S. Wu, J. Chen, T. Xu, L. Chen, L. Wu, Y. Hu, and E. Chen, “Linking the characters: Video-oriented social graph generation via hierarchical-cumulative gcn,” in ACM Multimedia, pp. 4716–4724, 2021.
  * [29] Q. Sun, B. Schiele, and M. Fritz, “A domain based approach to social relation recognition,” in CVPR, pp. 3481–3490, 2017.
  * [30] P. Vicol, M. Tapaswi, L. Castrejon, and S. Fidler, “Moviegraphs: Towards understanding human-centric situations from videos,” in CVPR, pp. 8581–8590, 2018.
  * [31] B. K. Tabbat, “The anatomy of dependence by takeo doi, md,” Bulletin of the Menninger Clinic, p. 660, 1976.
  * [32] F. N. Kerlinger, “Decision-making in japan,” Social Forces, pp. 36–41, 1951.
  * [33] T. Xu, P. Zhou, L. Hu, X. He, Y. Hu, and E. Chen, “Socializing the videos: A multimodal approach for social relation recognition,” TOMM, vol. 17, no. 1, pp. 1–23, 2021.
  * [34] M. Zhang, X. Liu, W. Liu, A. Zhou, H. Ma, and T. Mei, “Multi-granularity reasoning for social relation recognition from images,” in ICME, pp. 1618–1623, 2019.
  * [35] P. Dai, J. Lv, and B. Wu, “Two-stage model for social relationship understanding from videos,” in ICME, pp. 1132–1137, 2019.
  * [36] L. Li, L. Qing, Y. Wang, J. Su, Y. Cheng, and Y. Peng, “Hf-srgr: A new hybrid feature-driven social relation graph reasoning model,” The Visual Computer, pp. 1–14, 2021.
  * [37] A. Goel, K. T. Ma, and C. Tan, “An end-to-end network for generating social relationship graphs,” in CVPR, pp. 11186–11195, 2019.
  * [38] Z. Zhang, P. Luo, C.-C. Loy, and X. Tang, “Learning social relation traits from face images,” in ICCV, pp. 3631–3639, 2015.
  * [39] D. Lin, L. Wang, G. Shi, H. Xu, H. Li, B. Wu, and J. Ge, “Social relationship recognition based on relational self-attention mechanism,” in CSCWD, pp. 798–803, IEEE, 2022.
  * [40] X. Yang, F. Xu, K. Wu, Z. Xie, and Y. Sun, “Gaze-aware graph convolutional network for social relation recognition,” IEEE Access, pp. 99398–99408, 2021.
  * [41] L. Qing, L. Li, Y. Wang, Y. Cheng, and Y. Peng, “Srr-lgr: Local–global information-reasoned social relation recognition for human-oriented observation,” Remote Sensing, p. 2038, 2021.
  * [42] W. Peng, W. He, D. Xu, T. Xu, C. Zhu, and E. Chen, “Social context-aware GCN for video character search via scene-prior enhancement,” in ICME, IEEE, 2023.
  * [43] P. Qin, S. Wu, T. Xu, Y. Hao, F. Feng, C. Zhu, and E. Chen, “When i fall in love: Capturing video-oriented social relationship evolution via attentive gnn,” IEEE TCSVT, 2023.
  * [44] N. Zhang, M. Paluri, Y. Taigman, R. Fergus, and L. Bourdev, “Beyond frontal faces: Improving Person Recognition using multiple cues,” in CVPR, pp. 4804–4813, 2015.
  * [45] Z. Zhang, P. Luo, C.-C. Loy, and X. Tang, “Learning Social Relation Traits from Face Images,” in ICCV, pp. 3631–3639, 2015.
  * [46] X. Gu, T. Lin, W. Kuo, and Y. Cui, “Open-vocabulary object detection via vision and language knowledge distillation,” in ICLR, 2022.
  * [47] M. Xu, Z. Zhang, F. Wei, H. Hu, and X. Bai, “Side adapter network for open-vocabulary semantic segmentation,” CVPR, 2023.
  * [48] H. Fang, P. Xiong, L. Xu, and Y. Chen, “Clip2video: Mastering video-text retrieval via image clip,” arXiv preprint arXiv:2106.11097, 2021.
  * [49] J. A. Portillo-Quintero, J. C. Ortiz-Bayliss, and H. Terashima-Marín, “A straightforward framework for video retrieval using clip,” in Pattern Recognition, pp. 3–12, 2021.
  * [50] H. Luo, L. Ji, M. Zhong, Y. Chen, W. Lei, N. Duan, and T. Li, “Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning,” Neurocomputing, pp. 293–304, 2022.
  * [51] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical text-conditional image generation with clip latents,” arXiv preprint arXiv:2204.06125, 2022.
  * [52] H. Song, L. Dong, W. Zhang, T. Liu, and F. Wei, “Clip models are few-shot learners: Empirical studies on vqa and visual entailment,” in ACL, pp. 6088–6100, 2022.
  * [53] R. Zhang, Z. Guo, W. Zhang, K. Li, X. Miao, B. Cui, Y. Qiao, P. Gao, and H. Li, “Pointclip: Point cloud understanding by clip,” in CVPR, pp. 8552–8562, 2022.
  * [54] M. Wang, J. Xing, and Y. Liu, “Actionclip: A new paradigm for video action recognition,” arXiv preprint arXiv:2109.08472, 2021.
  * [55] L. Yao, J. Han, Y. Wen, X. Liang, D. Xu, W. Zhang, Z. Li, C. Xu, and H. Xu, “Detclip: Dictionary-enriched visual-concept paralleled pre-training for open-world detection,” NeurIPS, pp. 9125–9138, 2022.
  * [56] Y. Long, Y. Wen, J. Han, H. Xu, P. Ren, W. Zhang, S. Zhao, and X. Liang, “Capdet: Unifying dense captioning and open-world detection pretraining,” in CVPR, pp. 15233–15243, 2023.
  * [57] J. O. Zhang, A. Sax, A. Zamir, L. Guibas, and J. Malik, “Side-tuning: a baseline for network adaptation via additive side networks,” in ECCV, pp. 698–714, 2020.
  * [58] M. B. Yi-Lin Sung, Jaemin Cho, “Lst: Ladder side-tuning for parameter and memory efficient transfer learning,” in NeurIPS, 2022.
  * [59] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” NeurIPS, 2017.
  * [60] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in ICCV, pp. 2961–2969, 2017.
  * [61] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” NeurIPS, 2015.
  * [62] B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba, “Places: A 10 million image database for scene recognition,” TPAMI, 2017.
  * [63] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in CVPR, pp. 248–255, 2009.
  * [64] M. A. Mohsin and A. Beltiukov, “Summarizing emotions from text using plutchik’s wheel of emotions,” in ITIDS, pp. 291–294, 2019.
  * [65] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” ICLR, 2017.
  * [66] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, “Empirical evaluation of gated recurrent neural networks on sequence modeling,” arXiv preprint arXiv:1412.3555, 2014.
  * [67] J. Zhou, G. Cui, S. Hu, Z. Zhang, C. Yang, Z. Liu, L. Wang, C. Li, and M. Sun, “Graph neural networks: A review of methods and applications,” AI open, pp. 57–81, 2020.
  * [68] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in CVPR, pp. 770–778, 2016.
  * [69] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in ICLR, 2021.
  * [70] J. D. M.-W. C. Kenton and L. K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of naacL-HLT, p. 2, 2019.


# A Short Note on the Kinetics-700 Human Action Dataset

João Carreira
<EMAIL_ADDRESS>
Eric Noland
<EMAIL_ADDRESS>
Chloe Hillier
<EMAIL_ADDRESS>
Andrew Zisserman
<EMAIL_ADDRESS>

###### Abstract

We describe an extension of the DeepMind Kinetics human action dataset from
600 classes to 700 classes, where for each class there are at least 600 video
clips from different YouTube videos. This paper details the changes introduced
for this new release of the dataset, and includes a comprehensive set of
statistics as well as baseline results using the I3D neural network
architecture.

##  1 Introduction

The goal of the Kinetics project is to provide a large scale curated dataset
of video clips, covering a diverse range of human actions, that can be used
for training and exploring neural network architectures for modelling human
actions in video. This short paper describes the new version of the dataset,
called Kinetics-700.

The new dataset follows the same principles as Kinetics-400 [7] and
Kinetics-600 [2]: (i) The clips are from YouTube videos, last 10s, and have a
variable resolution and frame rate; (ii) for an action class, all clips are
from different YouTube videos. Kinetics-700 is almost a superset of
Kinetics-600: the number of classes is increased from 600 to 700, with all but
three of the Kinetics-600 classes retained. As in the case of Kinetics-600,
Kinetics-700 has 600 or more clips per human action class – this represents a
30% increase in the number of video clips, from around 500k to around 650k.
The statistics of the three Kinetics datasets are detailed in table 1.

In the new Kinetics-700 dataset there is a standard validation set, for which
labels have been publicly released, and also a held-out test set (where the
labels are not released). We encourage researchers to report results on the
standard validation set, unless they want to compare with participants of the
Activity-Net Kinetics challenge where the performance on the held-out test set
can be be measured only through the challenge evaluation
website111http://activity-net.org/challenges/2019/evaluation.html. The URLs of
Kinetics YouTube videos and temporal intervals can be obtained from
http://deepmind.com/kinetics.

Version | Train | Valid. | Test | Held-out Test | Total Train | Total | Classes
---|---|---|---|---|---|---|---
Kinetics-400 [7] | 250–1000 | 50 | 100 | 0 | 246,245 | 306,245 | 400
Kinetics-600 [2] | 450–1000 | 50 | 100 | around 50 | 392,622 | 495,547 | 600
Kinetics-700 | 450–1000 | 50 | 100 | 0 | 545,317 | 650,317 | 700
Table 1: Kinetics Dataset Statistics. The number of clips for each class in
the various splits (left), and the totals (right).

##  2 Data Collection Process

The data collection process evolved from Kinetics-400 to Kinetics-700,
although the overall pipeline is the same: 1) action class sourcing, 2)
candidate video matching, 3) candidate clip selection, 4) human verification,
5) quality analysis and filtering. In words, a list of class names is created,
then a list of candidate YouTube URLs is obtained for each class name, and
candidate 10s clips are sampled from the videos. These clips are sent to
humans who decide whether those clips contain the action class that they are
supposed to. Finally, there is an overall curation process including clip de-
duplication, and selecting the higher quality classes and clips. Full details
can be found in the original publication [7].

The main differences in the data collection process between Kinetics-400,
Kinetics-600 and 700 is in steps 1, 2 and 4: how action classes are sourced,
how candidate YouTube videos are matched with classes, and human verification.
In the following we detail these differences and the consequences of these
changes on the dataset. Note, as well as producing clips for entirely new
classes, it is necessary to ‘top up’ existing classes in Kinetics-600 since
YouTube videos are deleted or unlisted over time (about 3% per year).

It should be noted that the design of the collection process is not well
suited to finding action classes that progress over time. It is very well
suited to continual actions that exist over the length of the video (e.g.
‘juggling’, ‘drumming’), but not to those that have a progression from start
to middle to end (e.g. ‘dropping plates’, ‘getting out of car’).

###  2.1 Action class sourcing

The additional classes for Kinetics-700 over Kinetics-600 were partly sourced
from the lists of actions (or verbs) in recent human action datasets, such as
EPIC-Kitchens [4] and AVA [5]. Also, some existing classes in Kinetics-600
which were at quite a general level, e.g. ‘picking fruit’, were removed and
replaced by a number of fine-grained variations, for example: ‘picking
apples’, ‘picking blueberries’. We also introduced a number of more
imaginative classes, such as: ‘making slime’, ‘being in zero gravity’,
‘swimming with sharks’.

###  2.2 Candidate video matching

In Kinetics-700 we formally separated the ‘class name’ from the ‘query text’
used to search for that class. So, for example, to obtain the class ‘canoeing
or kayaking’, the query text could be canoeing and kayaking, and both would be
used. Another example is ‘abseiling’, which can be queried with both
‘abseiling’ or ‘rappelling’. Further more, the query text was translated into
three languages. In Kinetics-600 we had piloted this scheme by using both
English and Portuguese query texts, but in Kinetics-700 we extended it. We
describe next these multiple queries and how they are matched to the YouTube
corpus to obtain candidate videos.

Multiple queries. In order to get a better and larger pool of candidates for a
class, each query text was automatically translated from English into three
languages: French, Portuguese, and Spanish. These are three out of six
languages with the most native speakers in the world222According to
https://www.babbel.com/en/magazine/the-10-most-spoken-languages-in-the-world/,
and have large YouTube communities. We found that the machine translation had
adequate quality, though sometimes it introduced ambiguity. The query texts in
all four languages were used to obtain candidate videos.

Having multiple languages had the positive side effect of also promoting
slightly greater dataset diversity by incorporating a more well-rounded range
of cultures, ethnicities and geographies. In terms of continents, more than
50% of the clips are sourced from North America. However, the fraction of
clips from Latin America increased from 3% in Kinetics-400 to 8% in
Kinetics-700, thanks to adding Spanish and Portuguese language queries. Africa
is still the least represented continent, increasing from 0.8% in Kinetics-400
to 1% in Kinetics-700. These numbers are based on the 90% of videos that
contained location information.

Matching query text to YouTube videos. Rather than matching directly using
textual queries we found it beneficial to use weighted ngram representations
of the combination of the metadata of each video and the titles of related
ones. Importantly, these representations were compatible with multiple
languages. We combined this with standard title matching to get a robust
similarity score between a query and all YouTube videos. This meant that we
never ran out of candidates, although the human-verification yield of the
selected candidates became lower for smaller similarity values. This procedure
generates a far larger candidate pool than simply a binary match between the
query text and YouTube video title, say. Since the target length of a clip is
10s, videos longer than 5 minutes were discouraged.

###  2.3 Candidate clip selection and yield

Within a video, candidate clips are selected by using image classifiers. Image
classifiers are available for a large number of human actions. These
classifiers are obtained by tracking user actions on Google Image Search. For
example, for a search query “climbing tree”, user relevance feedback on images
is collected by aggregating across the multiple times that search query is
issued. This relevance feedback is used to select a high-confidence set of
images that can be used to train a “climbing tree” image classifier.
Classifiers corresponding to the class name are run at the frame level over
the selected videos for that class, and clips extracted around the top $k$
responses (where $k=2$). In cases where we could not find classifiers for the
class name, we used classifiers related to the query texts.

###  2.4 Human verification

The first and main annotation task in our pipeline asks human annotators if a
clip contains a particular action. This step was the same as in previous years
for Kinetics-700.

A difference to previous years was in the final human annotation stage, which
we previously did not crowdsource and instead did ourselves: we would go over
each individual class and look at all its animated-gif thumbnails while taking
into account potentially confusing classes (derived from classifier outputs).
Sometimes class names may allow for multiple types of videos – e.g. a class
named “jumping into pool” could have people diving or just jumping. If we had
a competing “diving” class then we would try to remove diving videos from
“jumping into pool”.

This was a painstaking manual effort, which we tried to crowdsource this year.
Since crowdsourcing requires limiting the size of individual tasks, we divided
class thumbnails into panels of 16 elements and had human workers clean up the
classes. Note that this provides them however with a tighter window into the
contents of each class.

Yield by class. It is interesting to see which classes gave the highest and
lowest yields in terms of the probability that a candidate clip was voted
positive for that class by three or more human annotators. The classes with
highest yield are given in table 2, and those with lowest yield are listed in
Appendix B.

Rank | Class | Yield
---|---|---
1 | busking | 0.9227
2 | spinning poi | 0.9227
3 | rope pushdown | 0.9091
4 | front raises | 0.8864
5 | zumba | 0.8864
6 | country line dancing | 0.8727
7 | ice skating | 0.8636
8 | shearing sheep | 0.8636
9 | arm wrestling | 0.8636
10 | bench pressing | 0.8545
11 | playing squash or racquetball | 0.8455
12 | playing accordion | 0.8318
Table 2: The classes that have the highest yield – measured as the proportion
of candidate clips that were judged positive for that class by three or more
annotators.

There are multiple factors involved here: whether the query is text that is
used to annotate videos; how general or specific the query text is for
obtaining relevant videos (for example, “acting in play” is already quite
ambiguous in English, and the current automatic translations are totally off,
e.g. for Portuguese it translates into what would translatee back as “acting
in game”); how well the clip is selected within a relevant video; and the
actual numbers of videos on YouTube for that action class. One notable common
element of the highest yields is that they are the type of actions where the
temporal position selected is not important – ‘playing guitar’ will be true at
almost any point over a long temporal period, and the video is easily
specified by the class name; in contrast ‘opening a letter’ only occurs over a
very specific and short time interval, and consequently could easily be missed
in a long video.

In general the high yield classes are successful in being included in the
Kinetics release, but conversely, only a small proportion of the low yield
classes survives.

##  3 From Kinetics-600 to Kinetics-700

As mentioned above, Kinetics-700 is an approximate superset of Kinetics-600 –
overall, 597 out of 600 classes are exactly the same in Kinetics-700 (although
some of the clips may have been replaced if the original videos have been
deleted). For the other classes, we renamed one (“passing american football
(not in game)” to “passing American football (not in game)”), and split
“chopping vegetables” and “picking fruit” into multiple subclasses.

In terms of the train/val/test split, there is a very small overlap between
the Kinetics-700 test set and Kinetics-600 train/val/test/hold out test (under
3%).

It is therefore largely safe to use models that have been trained on
Kinetics-600 to evaluate the Kinetics-700 test set (the activity-net
evaluation website explicitly ignores the predictions on those 3% clips when
evaluating on the test set). The full list of new classes in Kinetics-700 is
given in Appendix A.

##  4 Benchmark Performance

Acc. type | Valid | Test
---|---|---
Top-1 | 58.7 | 57.3
Top-5 | 81.7 | 79.9
$100.0-avg$(Top-1,Top-5) | 29.8 | 31.4
Table 3: Performance of an I3D model with RGB inputs on the Kinetics-700
dataset, without any test time augmentation (processing a center crop of each
video convolutionally in time). The first two rows show accuracy in
percentage, the last one shows the metric used at the Kinetics challenge
hosted by the ActivityNet workshop.

As a baseline model we used I3D [3], with standard RGB videos as input (no
optical flow). We trained the model from scratch on the Kinetics-700 training
set, picked hyper-parameters on validation, and report performance on
validation and test set. We used 32 P100 GPUs, batch size 5 videos, 64 frame
clips for training and 251 frames for testing. We trained using SGD with
momentum, starting with a learning rate of 0.1, decreasing it by a factor of
10 when the loss saturates. Results are shown in table 3. Hardest and easiest
classes are shown in fig. 1.

Figure 1: List of 20 easiest and 20 hardest Kinetics-700 classes sorted by
class accuracies obtained using the I3D-RGB model.

The top-1 accuracy on the validation set was 58.7 and on the test set was
57.3, which shows that both sets are similarly hard. On Kinetics-400 the
corresponding test set accuracy was 68.4 and on Kinetics-600 it was 71.7,
hence the task overall seems to have became considerably harder. This may
partially have to do with the way we have now crowdsourced the human
verification stage – it may be that workers did not strive as hard as we
previously did to make classes more unimodal. It is possible also that, since
we collected a full new test set, there is a little distribution shift between
train and test (but the validation performance is not very different from the
test performance).

Kinetics challenge. There was a first Kinetics challenge at the ActivityNet
workshop in CVPR 2017, using Kinetics-400. The second challenge occurred at
the ActivityNet workshop in CVPR 2018, this time using Kinetics-600. The
performance criterion used in the challenge is the average of Top-1 and Top-5
error. There was an improvement between the winning systems of the two
challenges, with error getting down from 12.4% (in 2017) to 11.0% (in 2018)
[1, 6]. The 2019 challenge featured the new Kinetics-700 dataset and had 15
participating teams. The top team was from JD AI Research and obtained 17.9%
error, considerably below our baseline – a single RGB I3D model – which
obtained 29.3% error.

##  5 Conclusion

We have described the new Kinetics-700 dataset, which in terms of clip counts
is 30% larger than Kinetics-600, and more than doubles the size of the
original Kinetics-400 dataset. It represents another step towards our original
goal of producing an action classification dataset with 1000 classes. Slides
illustrating some of what has been described in this paper can be found
online333https://drive.google.com/file/d/164kU_MFTKzmefbgOLntuiiTmADutl_x0/view.

### Acknowledgements:

The collection of this dataset was funded by DeepMind. The authors would like
to thank Jean-Baptiste Alyarac for checking and correcting the French machine
translated queries.

## References

  * [1] Y. Bian, C. Gan, X. Liu, F. Li, X. Long, Y. Li, H. Qi, J. Zhou, S. Wen, and Y. Lin.  Revisiting the effectiveness of off-the-shelf temporal modeling approaches for large-scale video classification.  arXiv preprint arXiv:1708.03805, 2017.
  * [2] J. Carreira, E. Noland, A. Banki-Horvath, C. Hillier, and A. Zisserman.  A short note about Kinetics-600.  arXiv preprint arXiv:1808.01340, 2018.
  * [3] J. Carreira and A. Zisserman.  Quo Vadis, Action Recognition? New Models and the Kinetics Dataset.  In IEEE International Conference on Computer Vision and Pattern Recognition CVPR, 2017.
  * [4] D. Damen, H. Doughty, G. Maria Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, and M. Wray.  Scaling Egocentric Vision: The EPIC-KITCHENS Dataset.  In Proc. ECCV, 2018.
  * [5] C. Gu, C. Sun, D. A. Ross, C. Vondrick, C. Pantofaru, Y. Li, S. Vijayanarasimhan, G. Toderici, S. Ricco, R. Sukthankar, et al.  AVA: A video dataset of spatio-temporally localized atomic visual actions.  CoRR, abs/1705.08421, 4, 2017.
  * [6] D. He, F. Li, Q. Zhao, X. Long, Y. Fu, and S. Wen.  Exploiting spatial-temporal modelling and multi-modal fusion for human action recognition.  arXiv preprint arXiv:1806.10319, 2018.
  * [7] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, M. Suleyman, and A. Zisserman.  The kinetics human action video dataset.  arXiv preprint arXiv:1705.06950, 2017.

##  Appendix A List of New Human Action Classes in Kinetics-700

This is the list of classes in Kinetics-700 that were not in Kinetics-600, or
that have been renamed.

  1. 1.

pouring wine

  2. 2.

walking on stilts

  3. 3.

listening with headphones

  4. 4.

dealing cards

  5. 5.

sanding wood

  6. 6.

splashing water

  7. 7.

digging

  8. 8.

chasing

  9. 9.

tossing salad

  10. 10.

playing cards

  11. 11.

moving baby

  12. 12.

bouncing ball (not juggling)

  13. 13.

helmet diving

  14. 14.

vacuuming car

  15. 15.

high fiving

  16. 16.

picking apples

  17. 17.

swimming with sharks

  18. 18.

cutting cake

  19. 19.

doing sudoku

  20. 20.

swimming with dolphins

  21. 21.

playing american football

  22. 22.

pouring milk

  23. 23.

entering church

  24. 24.

carrying weight

  25. 25.

taking photo

  26. 26.

saluting

  27. 27.

jumping sofa

  28. 28.

exercising arm

  29. 29.

playing oboe

  30. 30.

shooting off fireworks

  31. 31.

playing nose flute

  32. 32.

making latte art

  33. 33.

carving wood with a knife

  34. 34.

making slime

  35. 35.

looking in mirror

  36. 36.

shoot dance

  37. 37.

checking watch

  38. 38.

playing checkers

  39. 39.

seasoning food

  40. 40.

sieving

  41. 41.

gargling

  42. 42.

pulling espresso shot

  43. 43.

curling eyelashes

  44. 44.

shredding paper

  45. 45.

stacking dice

  46. 46.

surveying

  47. 47.

poaching eggs

  48. 48.

pulling rope (game)

  49. 49.

uncorking champagne

  50. 50.

eating nachos

  51. 51.

picking blueberries

  52. 52.

coughing

  53. 53.

filling cake

  54. 54.

shouting

  55. 55.

playing mahjong

  56. 56.

spinning plates

  57. 57.

spraying

  58. 58.

pretending to be a statue

  59. 59.

moving child

  60. 60.

steering car

  61. 61.

baby waking up

  62. 62.

treating wood

  63. 63.

playing piccolo

  64. 64.

letting go of balloon

  65. 65.

playing shuffleboard

  66. 66.

playing road hockey

  67. 67.

using megaphone

  68. 68.

squeezing orange

  69. 69.

being in zero gravity

  70. 70.

walking with crutches

  71. 71.

polishing furniture

  72. 72.

closing door

  73. 73.

grooming cat

  74. 74.

laying decking

  75. 75.

arresting

  76. 76.

rolling eyes

  77. 77.

ski ballet

  78. 78.

mixing colours

  79. 79.

metal detecting

  80. 80.

waxing armpits

  81. 81.

peeling banana

  82. 82.

cooking chicken

  83. 83.

carving marble

  84. 84.

filling eyebrows

  85. 85.

breaking glass

  86. 86.

playing rounders

  87. 87.

petting horse

  88. 88.

putting wallpaper on wall

  89. 89.

herding cattle

  90. 90.

playing billiards

  91. 91.

stacking cups

  92. 92.

blending fruit

  93. 93.

lighting candle

  94. 94.

decoupage

  95. 95.

crocheting

  96. 96.

playing slot machine

  97. 97.

silent disco

  98. 98.

being excited

  99. 99.

brushing floor

  100. 100.

opening coconuts

  101. 101.

milking goat

  102. 102.

slicing onion

  103. 103.

flipping bottle

##  Appendix B List of Low Yield Classes

This is the ranked list of classes that have lowest yield, where yield is the
probability that a candidate clip was voted positive for that class by three
or more human annotators. Bold indicates that the class was included in the
final dataset; most of the low yield classes were not included.

  1. 1.

opening letter 0.0019

  2. 2.

adding fish to aquarium 0.0033

  3. 3.

getting inside balloon 0.0034

  4. 4.

comforting 0.0036

  5. 5.

highlight text 0.0038

  6. 6.

riding giraffe 0.0047

  7. 7.

dropping plates 0.0057

  8. 8.

contemplating 0.0061

  9. 9.

whispering 0.0101

  10. 10.

grooming (person) 0.0106

  11. 11.

boarding train 0.0112

  12. 12.

buying fast food 0.0114

  13. 13.

Piling coins up 0.0114

  14. 14.

looking through telescope 0.0116

  15. 15.

breaking aquarium 0.0118

  16. 16.

using a crowbar 0.0127

  17. 17.

underlining 0.0128

  18. 18.

instant messaging 0.0133

  19. 19.

getting into a car 0.0134

  20. 20.

tossing coin 0.0146

  21. 21.

getting out of a car 0.0153

  22. 22.

checking mail 0.0157

  23. 23.

entering building 0.0177

  24. 24.

signing document 0.0179

  25. 25.

cutting in line 0.0179

  26. 26.

waiting at crossing 0.0179

  27. 27.

dunking biscuit 0.0185

  28. 28.

checking tickets 0.0188

  29. 29.

assembling bicycle 0.0196

  30. 30.

exiting building 0.0198

  31. 31.

unloading the trunk of a car 0.0198

  32. 32.

setting up fish tank 0.0198

  33. 33.

cutting squares 0.0201

  34. 34.

texting 0.0209

  35. 35.

playing underwater frisbee 0.0212

  36. 36.

riding zebra 0.0212