# Expanding Frozen Vision-Language Models without Retraining: Towards Improved
Robot Perception

Riley Tavassoli  Mani Amani  Reza Akhavian<EMAIL_ADDRESS>
###### Abstract

Vision-language models (VLMs) have shown powerful capabilities in visual
question answering and reasoning tasks by combining visual representations
with the abstract skill set large language models (LLMs) learn during
pretraining. Vision, while the most popular modality to augment LLMs with, is
only one representation of a scene. In human-robot interaction scenarios,
robot perception requires accurate scene understanding by the robot. In this
paper, we define and demonstrate a method of aligning the embedding spaces of
different modalities (in this case, inertial measurement unit (IMU) data) to
the vision embedding space through a combination of supervised and contrastive
training, enabling the VLM to understand and reason about these additional
modalities without retraining. We opt to give the model IMU embeddings
directly over using a separate human activity recognition model that feeds
directly into the prompt to allow for any nonlinear interactions between the
query, image, and IMU signal that would be lost by mapping the IMU data to a
discrete activity label. Further, we demonstrate our methodology’s efficacy
through experiments involving human activity recognition using IMU data and
visual inputs. Our results show that using multiple modalities as input
improves the VLM’s scene understanding and enhances its overall performance in
various tasks, thus paving the way for more versatile and capable language
models in multi-modal contexts.

###### keywords:

Multi-modal visual language models , Robot perception , Contrastive Learning

††journal: Information Fusion

[inst1]organization=San Diego State University,addressline=5500 Campanile Dr,
city=San Diego, postcode=92182, state=California, country=United States of
America

##  1 Introduction

Multi-modal research in vision, audio and language has gained traction in
recent years[1, 2], and now with current studies showing that Large language
models (LLMs) have the capabilities of complex question answering and
reasoning [3], there has been an influx of attention towards utilizing multi-
modal LLMs. Recent research on vision-language models has further shown these
reasoning capabilities can be extended to other modalities [4]. In this paper,
we propose a method that extends frozen, pretrained visual-language models to
understand inertial measurement unit (IMU) data while being extensible to any
other modality. This method of extending pretrained models without retraining
or finetuning reduces training costs dramatically in an era of deep learning
where it has become infeasible to train most models from scratch for the
majority of researchers and developers [5]. At these large sizes, models can
learn abstract, generalizable reasoning skills that are difficult to replicate
in smaller models [6]. Specifically, language models present a new paradigm of
foundation models that offer unlimited downstream use cases, with the
limitation of text being the singular modality. Vision-language models (VLMs)
have allowed for images to be interwoven with text, taking advantage of the
skills the base LLM learned while being trained on text. Flamingo [7] proposed
a novel VLM architecture where trainable layers were injected into the frozen
LLM. These new trainable layers require far less training than the base LLM
while allowing the raw image embeddings to be processed in a depth-wise manner
alongside the accompanying text. This results in the frozen LLM being capable
of understanding image embeddings that have been resampled to match the
distribution the LLM expects. This allows for the LLM’s in-context learning
capabilities to be used on images, making the model versatile and removing the
need for fine-tuning to a domain-specific dataset [8]. Instead of training new
layers or modules for every additional modality to be incorporated, any
modality can arbitrarily be aligned to share the embedding space of the vision
encoder through contrastive training. Consequently, the layers that translate
vision embeddings into representations the LLM understands also work on any
other modality that has been aligned with the vision embedding space.

Figure 1: Overview of the approach showing the concatenation of multiple modal
representations of a scene with a query yielding better, more semantic
responses

Most contrastive learning methods rely on large datasets, but with the methods
we propose in this paper, even modalities with relatively few examples can
sufficiently align their embedding space to the vision encoder. This idea also
addresses a growing demand for larger generalist models to use any modality to
enable users to take advantage of the abstract representations it has learned.
As such, our main contributions in this paper are as follows:

  1. 1.

A methodology that allows for the extension of frozen, pretrained vision-
language models to accommodate any number of modalities, broadening their
applicability and versatility.

  2. 2.

An understanding of how multi-modal inputs contribute to the development of
increasingly nuanced scene representations, adding depth and context to
machine understanding.

  3. 3.

A validated evidence that the integration of various modalities improves scene
understanding, a critical aspect of machine perception and cognition.

  4. 4.

A demonstration of how relatively small datasets can be used for contrastive
alignment.

###  1.1 Robot Perception and Human Activity Recognition (HAR)

The goal of this paper is to leverage VLMs for better scene understanding
toward improved robotics perception, especially in human-robot interaction
(HRI) scenarios. In this regard, human activity recognition (HAR) using
wearable devices can help robots better perceive their environment by gaining
an understanding of the type of activity in which the human counterpart is
engaged with. Because there already exist very competent HAR models, we choose
to supply the IMU embeddings directly in the prompt to assess model
performance on more granular aspects of scene understanding that are not
readily extractable with pre-existing models. For practical and automated HRI
applications, the HAR classification could also be retrieved from an auxiliary
model and appended to the query.

##  2 Related Work

HRI has garnered interest in recent years manufacturing due to the potential
production efficiency improvement. However, robots do not have the innate
contextual scene understanding capability humans latently possess [9]. To
remedy this issue, researchers have conducted extensive research into both
robot perception [10] and robot action [11]

For robotic action, RT-2 [12] is a visual-language-action (VLA) Model that
leverages visual-language models (VLM) such as PaLI-X [13] to generate action
commands for a robot based on a query. The model changes the backbone VLM’s
generated token to include robotic actions that are used as inputs to the low-
level robotic controller. Principally, the paper shows the potential of
adapting VLMs to VLAs by combining VLM pretraining with robotic data.

PaLM-E [14] is an embodied, multi-modal language model capable of executing
complex robotic tasks via planning and scene understanding. However, they use
a training paradigm different from the one presented in this paper whereby
modality encoders are trained end-to-end with a frozen LLM as the head of the
encoder, similar to[15]. Importantly, they highlight the ability of LLMs to
reason on multi-modal inputs in a way similar to language. There are several
other recent works, such as ImageBind [16], that integrate multiple modalities
into a unified embedding space, showing how a linear combination of embeddings
from multiple modalities yields a better representation. These developments
highlight the capabilities of multi-modal learning and underscore the
importance of exploring it further. Macaw-LLM [17] provides a new architecture
for multi-modal inputs, published as a vision-audio-text LLM with the
introduction of a new architecture containing an alignment module that aligns
the embeddings of multiple modalities to a common embedding space. The benefit
of what we design in this work is its ability to leverage pretrained models
without the need for a new architecture or retraining of the base model or an
alignment module. Works such as BlIP-2 [18] follow the same philosophy of
feeding different modalities to language models in a format that they can
understand and process through a specific pretraining regime. BLIP-2 combines
”off the shelf” frozen image encoders and frozen LLMs and proposes a
pretraining strategy that bridges the gap between modalities. [18] show a
novel architecture for a light-weight HAR model designed for processing videos
and trained contrastively on associated activity captions.VLMs have been
employed in the past for HAR. One such instance is VicTR, a model that
utilizes a joint video and text encoder to create video-conditioned text
tokens for improved HAR [19]. In another study, the authors developed a HAR
algorithm utilizing wide time-domain convolutional neural network and multi-
environment sensor data for daily behavior recognition while using
contribution significance analysis to assess the contribution of each sensor
to the detection of the activity [20].

PaLM-E’s approach to integrating sensor signals as inputs in a multi-modal
language model provided valuable insights into the potential capabilities of
LLMs to reason on multi-modal inputs in a similar manner to language. However,
they rely on a paradigm that requires the encoders to be trained end-to-end
with a frozen LLM, limiting the flexibility of the system. ImageBind [16]
integrates multiple modalities into a unified embedding space through
contrastive alignment, bypassing the high training cost of using the LLM
directly. Our work strives to develop a methodology that allows LLMs to
accommodate an arbitrary number of modalities without needing a new
architecture, an issue faced by works like Macaw-LLM [17].

##  3 Methodology

Figure 2 shows how raw inputs are processed through the VLM. An important
distinction is that we linearly combine the image and IMU embeddings for a
single example after having passed through the perceiver resampler but before
they pass through the gated cross attention layers. This linear combination of
encoded representations provides the VLM with a more holistic representation
of the scene. The training method that aligns the IMU encoder to the
pretrained visual encoder is outlined below.

Figure 2: The architecture of Flamingo VLMs extended to handle image-IMU pairs
of inputs

###  3.1 Dataset

We use the MMAct dataset [21] which consists of 35 different human actions
with varied durations of 2-10 seconds. Each sample is recorded on 4 different
cameras at different points in the room. The room is also set up in 4 distinct
ways with different obstacles or components. Example actions include talking
on the phone, running, and pulling a cart. We concatenate all IMU signals,
down sampling where necessary such that each signal is sampled at 50 Hz. Each
signal provides 3 channels, and with 4 signals (two accelerometers at
different parts of the body, a gyroscope, and a magnetometer), we attain a 12
channel IMU signal. We sample a window of length 256 from this data, padding
with zeros when the signal is not long enough. We randomly sample a frame from
the corresponding video. We use a batch size of 512 and train for a maximum of
100 epochs, early stopping once the validation loss is minimized. The total
train size is 6093 examples.

###  3.2 Modality Encoders

In this work, we extend visual language models to understand IMU data encoded
using a transformer-based encoder in combination with a 1-d convolution
without retraining the visual language model. To train this encoder, we
contrastively optimize temporally overlapping image-IMU pairs to have a large
cosine similarity using a frozen, pretrained ViT-L/14 [22] as the visual
encoder. An extension of CLIP for video, X-CLIP [23], has previously been
explored by the authors for HAR [24]. The presented work seeks to show the
capability of extending VLMs understanding to multiple modalities with no
retraining. Therefore, we are constrained to the frozen visual encoder the VLM
was trained with. This is because as we contrastively train our IMU encoder to
share the embedding space of the vision encoder, it is necessary that this
shared embedding space towards which the IMU encoder optimizes is the same as
the embedding space the VLM was trained to understand. Had we used a different
vision encoder to contrastively train the IMU encoder, the pretrained VLM
would not understand the IMU embeddings without retraining. Here, we are
inspired by the work presented in ImageBind [16] to train arbitrary modality
encoders to align their embeddings with a desired embedding space.

###  3.3 Contrastive Pretraining

Contrastive learning, a subfield of unsupervised learning, works by learning a
representation of its inputs such that similar inputs result in similar
vectors and dissimilar inputs yield dissimilar vectors [25]. It has been
successfully applied in a variety of machine learning tasks, ranging from
image and speech recognition to natural language understanding, largely due to
its effectiveness in learning rich, meaningful embeddings from unlabeled data.
Multi-modal contrastive learning is still an active area of research [26, 27]
where the loss functions with which we optimize over are just beginning to be
explored. When contrastively training an encoder for a modality with a
temporal dimension such as IMU data, the window size is directly correlated
with information content which makes it an important hyperparameter to tune
and optimize for good representation quality [28]. We utilize a symmetric
cross entropy loss objective, also known as the infoNCE loss [29, 30], in
order to train our IMU encoder model. The loss maximizes the dot product of
matching pairs in a batch and minimizes the dot product of negative pairs.
This was most recently popularized in a multi-modal context with CLIP [22].

$L_{\text{{infoNCE}}}=-\sum_{(i,j)\in
P}\log\left(\frac{e^{\text{{CoSim}}(z_{i},z_{j})/\tau}}{\sum_{k=1}^{N}e^{\text{{CoSim}}(z_{i},z_{k})/\tau}}\right)$
(1)

For every pair (i,j) in set P, which represents positive pairs of data
instances, we compute the cosine similarity CoSim between the respective
representations $z_{i}$ and $z_{j}$. This similarity score is scaled by a
temperature parameter $\tau$ to control the sharpness of the distribution. The
logarithm of this ratio is then computed, and the loss is the negative sum
over all positive pairs. This formulation encourages the network to maximize
similarity for positive pairs and minimize similarity for negative pairs,
where positive pairs are defined as images and overlapping IMU windows, and
negative pairs are images and non-overlapping IMU windows. We also add a
supervised loss term to the loss function, mapping the embedded IMU
representation to class logits with a linear head. This enforces a constraint
on the embedding space that keeps embedded actions linearly separable. With
the addition of this supervised loss term, we observed more specific, distinct
outputs from the VLM when given IMU embeddings.

Rather than computing the infoNCE and supervised losses on the outputs from
the encoders, we further process both encoded representations by passing them
through the frozen, pretrained perceiver resampler module. This outputs a
predefined set of latent vectors that are resampled representations of the
input. For our implementation, we map an encoded sequence length of 256 with
dimension 1024 to a length of 64 with the perceiver resampler. We then average
pool along the sequence dimension for both image and IMU embeddings to obtain
1-d vectors of size 1024 for each sample. It is with these representations we
compute the infoNCE and supervised loss terms. In our empirical tests, this
process of including the perceiver resampler module grounds the representation
the IMU encoder learns more rigidly. We observed this in testing different
iterations on an activity recognition sub-task where we prompt the VLM with
only IMU embeddings to identify the action being performed. IMU encoders
trained without the perceiver resampler exhibited far worse performance on
this task, such that when combining the IMU embeddings with vision embeddings,
worse performance could sometimes be observed. Our hypothesis for why we see
better performance with this architecture is that the inclusion of the
perceiver resampler strictly constrains the features learned by the IMU
encoder to have a similar distribution to the features of the image encoder.
When computing loss on the embeddings that are output from the encoders rather
than the perceiver resampler, the loss is far noisier whereas the perceiver
resampler processes embeddings of both modalities into a shared distribution.
This contrastive and supervised learning objective enables the IMU encoder to
learn a meaningful mapping from raw sensor data to a representative embedding
space that aligns with the image encoder. Most unsupervised methods,
contrastive learning included, require large amounts of data. This paper
explores how a relatively small training dataset of around 6,000 image-IMU
pairs can be leveraged to align an IMU encoder with a vision encoder.

###  3.4  Multi-Modal Large Language Model

We utilize VLMs as a high-level reasoning module to better understand a scene
given various modal representations. We use an implementation of the Otter VLM
[31], a continuation of Open Flamingo [32], the open-sourced version of the
original DeepMind paper, Flamingo [7]. Otter is further trained on an
instruction dataset to support multi-modal in-context instruction tuning which
involves conditioning the language model on the corresponding media, such as
an image, that corresponds to a caption or an instruction-response pair [31,
33]. This makes Otter deliver more guided outputs when we effectively design
prompts.

VLMs such as Otter are autoregressive decoder-only models, with image
embeddings represented in the tokenizer with a special token. The image
embeddings, or any other modality’s embeddings, are passed through a module
introduced in the original Flamingo paper called the Perceiver Resampler which
takes as input the sequence output of a transformer encoder and outputs a
fixed set of resampled tokens as based on a set of learnable latent queries.
This allows for variably sized inputs to be mapped to the same length of
tokens, and it allows the frozen, pretrained language model to resample the
static image embeddings throughout the layers of the LLM.

Because Otter was trained on an instruction-tuning dataset, the model learns
to structure its response to follow the preceding in-context examples which
allows us to query the model’s understanding of the scene. In this paper, we
show that the addition of the IMU embeddings in the prompt allows the VLM to
better reason about a scene and more wholly understand the activities of the
humans present in the visual input.

##  4 Experiments

Below, we show the capabilities of the pretrained Otter model on semantic
scene understanding tasks when provided vision data, IMU data, and a
combination of both. We take advantage of conditional generation by prepending
our query with two example question-response pairs to update the model’s
output distribution to be more in line with our expectations. Figure 3 shows
model responses given different combinations of input modalities.

Figure 3: In-context generation with only IMU, only images, and both
modalities

##  5 Results

We evaluated the effectiveness of our contrastive alignment process by mapping
the embeddings of the IMU and image data for each of the 35 classes via
t-distributed stochastic neighbor embedding (t-SNE), a technique used to
visualize high-dimensional data in a way that shows underlying data
distributions in a 2-dimensional representation that can easily be plotted
[34]. Figure 4 shows the result of this visualization where each class is
represented by a different color, and the clusters suggest distinctive
patterns in the data.

Figure 4: t-SNE visualization of video and IMU encoder embeddings across 35
classes

The video encoder embeddings display clear clusters, suggesting that the model
successfully extracted meaningful patterns from the data. However, these
clusters do not align with the specific activities performed as there are no
clear color groupings, an outcome we anticipated. This is because the image
encoder was not specifically fine-tuned for HAR. The IMU encoder embeddings
lack some of the structure present in the image embeddings, suggesting that
the contrastive alignment of encoders did not fully align the two models to
share the exact same embedding space, but the class distribution is far more
organized which allows the model to better understand a user’s actions as
evidenced the very clear color groups. Further, the two modalities are
fundamentally capturing different characteristics of the scene, which is by
design, but that does mean that the embedding space of the IMU encoder will
naturally have a different structure even after alignment. For example, the
IMU data more closely corresponds to what a person is doing, e.g. two
different people doing the same activity have more similar IMU data than
images of two people doing the same activity due to the potential for
different backgrounds, environments, or peripheral objects. Because the IMU
data contains less total information than the associated images, there will be
some breakdown in structure where the image embeddings more finely correspond
to a given input.

We also test how linearly combining the embeddings from both modalities
changes the shared embedding space when visualized with t-SNE. In our
experiments, we see that the weights used in linearly combining the two modal
embeddings interpolate between the structure of the video and IMU embedding
spaces. For Figure 4, we weight the vision embeddings 80% and the IMU
embeddings 20%. In practice, these values must be empirically tuned to
maintain the structure of the desired embedding space while gaining some
smaller amount of information from the new embedding space. ImageBind [16]
exploits the linear combination of vectors in multi-modal embeddings spaces
for the task of video retrieval, obtaining better results when using audio in
conjunction with vision data.

This emergent structure of grouped examples of the same activity that is
present in the IMU embedding space and not in the vision embedding space
indicates that the raw IMU distribution is more implicitly linked to the
activity label. We view this as a feature allowing the two modalities to
naturally complement one another, the IMU data encoding the kinematic,
activity data the vision encoder struggles to encode. This point can be seen
in Table 1. This table shows the linear probe performance of a supervised HAR
model trained by video only, IMU only, and combined video-IMU data. The IMU
embeddings naturally encode information about an individual’s action with far
less noise than is present in an image. When combining modalities, we
concatenate the output embeddings of each encoder, mapping the combined vector
to class logits. This shows that the contrastive alignment of modalities can
provide novel information to a pretrained model that otherwise would not be
present in the unimodal data. This hypothesis warrants further investigation
in future studies.

Modality |  Training Loss |  Test Loss |  Training Accuracy (%) |  Test Accuracy (%)
---|---|---|---|---
Video |  1.0428 |  1.4748 |  65.07 |  52.41
IMU |  0.4052 |  1.2468 |  91.83 |  64.47
Combined |  0.2138 |  0.8753 |  94.58 |  74.46
Table 1: Supervised activity recognition for different modalities. Despite
sharing the same embedding space, each modality still preserves unique
information, as reflected in the increased performance when combining
embeddings

##  6 Conclusion

In this paper we have proposed a methodology to extend pretrained vision
language models to any modality by only training the new encoder. We have
shown the ability of a contrastive and supervised objective to sufficiently
map an encoder to the pretrained vision encoder’s embedding space thereby
taking advantage of the pretrained VLM. Further, we have shown how multiple
modalities leads to a more robust and general scene representation and
highlighted its potential in collaborative robotics.

###  6.1 Future Work

Future work can explore the effects of larger VLMs or VLMs with different
architectures with multi-modal fine-tuning. The model size can prove as a
limitation to the quality of the models responses, however, larger models will
have longer inference times which could prove as an issue in different
implementations. We plan to implement multi-modal fine-tuning to models such
as MiniGPT-4 [35] which uses a base Vicuna model [36] as the backbone LLM to
assess and compare their capabilities with the Otter model. The MiniGPT-4
utilizes a Q-former with a frozen pretrained vision encoder compared to gated
cross-attention layers which the flamingo model uses.

Another area we plan to explore is the assessment of information quality of
each modality. We hypothesize that modalities can have varying levels of
generalizable information regarding the activity and how we identify and
account for these discrepancies. Previous work such as ImageBind uses multi-
modal embedding space arithmetic to linearly combine embeddings of different
modalities to yield a more information-dense embedding vector. [16]. The
outcome of the presented work will be ultimately used in the context of HRI
for better robot perception. The authors are currently exploring HRI scenarios
in the context of construction activities where visual data from robot cameras
and IMU data from wearable sensors by construction workers are used to enhance
robot perception as seen in Figure 5.

Figure 5: A researcher investigating multi-modal robot perception for human-
robot collaboration

Other avenues of future research can be implementing the methodology
introduced in RT-2 which consists of utilizing robotic data through the VLM
pretraining. We introduce the feasibility of extending VLMs to any number of
modalities and experiment with the viability of implementing modality
extensions on VLAs. The study of contrastive alignment of modality encoders to
a shared embedding space is an avenue we plan to explore with new training
objectives and data-dependent significance analysis across multi-modal
representations.

###  6.2 Limitations

The Otter model uses MPT-7b as its backbone, making it fast for inference, but
with technical limitations in its hallucinations. Further, because the dataset
of video frame-IMU pairs used, MMAct, is relatively small, we do not attain a
1:1 representation between video and IMU data. This is expected as IMU data
intrinsically has a data distribution distinct from any correlated video
frames. Another downfall of extending modalities without pretraining is that
the learnable model parameters have not been trained for multi-modal
processing, potentially causing an increase in hallucinations. Current
research indicates that poor training and low-quality training data has a
direct effect on the degrees of hallucination [37].

## Conflict of Interest

The authors declare that they do not identify any personal relationships or
financial ties that would affect the contents or the publishing of this paper.

## Data Availability

Source code and data will be made available upon request.

## CRediT authorship contribution statement

Riley Tavassoli Conceptualization, Methodology, Software, Investigation,
Validation, Writing - Original Draft, Writing - Review & Editing,
Visualization.
Mani Amani: Investigation, Validation, Visualization, Writing - Original
Draft, Writing - Review & Editing, Software.
Reza Akhavian: Project administration, Funding acquisition, Writing - Review &
Editing, Supervision.

## Declaration of Funding

The presented work has been supported by the U.S. National Science Foundation
(NSF) CAREER Award through the grant # CMMI 2047138. The authors gratefully
acknowledge the support from the NSF. Any opinions, findings, conclusions, and
recommendations expressed in this paper are those of the authors and do not
necessarily represent those of the NSF.

## References

  * [1] S. Uppal, S. Bhagat, D. Hazarika, N. Majumder, S. Poria, R. Zimmermann, A. Zadeh, Multimodal research in vision and language: A review of current and emerging trends, Information Fusion 77 (2022) 149–171.  doi:https://doi.org/10.1016/j.inffus.2021.07.009.
URL https://www.sciencedirect.com/science/article/pii/S1566253521001512

  * [2] N. H. Nguyen, D. T. Vo, K. Van Nguyen, N. L.-T. Nguyen, Openvivqa: Task, dataset, and multimodal fusion models for visual question answering in vietnamese, Information Fusion 100 (2023) 101868.  doi:https://doi.org/10.1016/j.inffus.2023.101868.
URL https://www.sciencedirect.com/science/article/pii/S1566253523001847

  * [3] J. Huang, K. C.-C. Chang, Towards reasoning in large language models: A survey (2023).  arXiv:2212.10403.
  * [4] S. Liu, L. Fan, E. Johns, Z. Yu, C. Xiao, A. Anandkumar, Prismer: A vision-language model with an ensemble of experts (2023).  arXiv:2303.02506.
  * [5] Z. Wang, J. Wohlwend, T. Lei, Structured pruning of large language models, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, 2020.  doi:10.18653/v1/2020.emnlp-main.496.
URL https://doi.org/10.18653%2Fv1%2F2020.emnlp-main.496

  * [6] S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, Y. Zhang, Sparks of artificial general intelligence: Early experiments with gpt-4 (2023).  arXiv:2303.12712.
  * [7] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Monteiro, J. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Binkowski, R. Barreira, O. Vinyals, A. Zisserman, K. Simonyan, Flamingo: a visual language model for few-shot learning (2022).  arXiv:2204.14198.
  * [8] N. D. Palo, A. Byravan, L. Hasenclever, M. Wulfmeier, N. Heess, M. Riedmiller, Towards a unified agent with foundation models (2023).  arXiv:2307.09668.
  * [9] J. Fan, P. Zheng, S. Li, Vision-based holistic scene understanding towards proactive human–robot collaboration, Robotics and Computer-Integrated Manufacturing 75 (2022) 102304.  doi:https://doi.org/10.1016/j.rcim.2021.102304.
URL https://www.sciencedirect.com/science/article/pii/S0736584521001848

  * [10] A. Amorim, D. Guimares, T. Mendona, P. Neto, P. Costa, A. P. Moreira, Robust human position estimation in cooperative robotic cells, Robotics and Computer-Integrated Manufacturing 67 (2021) 102035.  doi:https://doi.org/10.1016/j.rcim.2020.102035.
URL https://www.sciencedirect.com/science/article/pii/S0736584520302465

  * [11] G. Evangelou, N. Dimitropoulos, G. Michalos, S. Makris, An approach for task and action planning in human–robot collaborative cells using ai, Procedia CIRP 97 (2021) 476–481, 8th CIRP Conference of Assembly Technology and Systems.  doi:https://doi.org/10.1016/j.procir.2020.08.006.
URL https://www.sciencedirect.com/science/article/pii/S2212827120314931

  * [12] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, P. Florence, C. Fu, M. G. Arenas, K. Gopalakrishnan, K. Han, K. Hausman, A. Herzog, J. Hsu, B. Ichter, A. Irpan, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, L. Lee, T.-W. E. Lee, S. Levine, Y. Lu, H. Michalewski, I. Mordatch, K. Pertsch, K. Rao, K. Reymann, M. Ryoo, G. Salazar, P. Sanketi, P. Sermanet, J. Singh, A. Singh, R. Soricut, H. Tran, V. Vanhoucke, Q. Vuong, A. Wahid, S. Welker, P. Wohlhart, J. Wu, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, B. Zitkovich, Rt-2: Vision-language-action models transfer web knowledge to robotic control, in: TODO, 2023.
  * [13] X. Chen, J. Djolonga, P. Padlewski, B. Mustafa, S. Changpinyo, J. Wu, C. R. Ruiz, S. Goodman, X. Wang, Y. Tay, S. Shakeri, M. Dehghani, D. Salz, M. Lucic, M. Tschannen, A. Nagrani, H. Hu, M. Joshi, B. Pang, C. Montgomery, P. Pietrzyk, M. Ritter, A. Piergiovanni, M. Minderer, F. Pavetic, A. Waters, G. Li, I. Alabdulmohsin, L. Beyer, J. Amelot, K. Lee, A. P. Steiner, Y. Li, D. Keysers, A. Arnab, Y. Xu, K. Rong, A. Kolesnikov, M. Seyedhosseini, A. Angelova, X. Zhai, N. Houlsby, R. Soricut, Pali-x: On scaling up a multilingual vision and language model (2023).  arXiv:2305.18565.
  * [14] D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. H. Vuong, T. Yu, W. Huang, Y. Chebotar, P. Sermanet, D. Duckworth, S. Levine, V. Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, I. Mordatch, P. R. Florence, Palm-e: An embodied multimodal language model, ArXiv abs/2303.03378 (2023).
  * [15] M. Tsimpoukelli, J. L. Menick, S. Cabi, S. Eslami, O. Vinyals, F. Hill, Multimodal few-shot learning with frozen language models, Advances in Neural Information Processing Systems 34 (2021) 200–212.
  * [16] R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, I. Misra, Imagebind: One embedding space to bind them all (2023).  arXiv:2305.05665.
  * [17] C. Lyu, M. Wu, L. Wang, X. Huang, B. Liu, Z. Du, S. Shi, Z. Tu, Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration (2023).  arXiv:2306.09093.
  * [18] J. Li, D. Li, S. Savarese, S. Hoi, Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models (2023).  arXiv:2301.12597.
  * [19] K. Kahatapitiya, A. Arnab, A. Nagrani, M. S. Ryoo, Victr: Video-conditioned text representations for activity recognition (2023).  arXiv:2304.02560.
  * [20] Y. Li, G. Yang, Z. Su, S. Li, Y. Wang, Human activity recognition based on multienvironment sensor data, Information Fusion 91 (2023) 47–63.  doi:https://doi.org/10.1016/j.inffus.2022.10.015.
URL https://www.sciencedirect.com/science/article/pii/S1566253522001841

  * [21] Q. Kong, Z. Wu, Z. Deng, M. Klinkigt, B. Tong, T. Murakami, Mmact: A large-scale dataset for cross modal human action understanding, in: The IEEE International Conference on Computer Vision (ICCV), 2019.
  * [22] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, I. Sutskever, Learning transferable visual models from natural language supervision, in: ICML, 2021.
  * [23] B. Ni, H. Peng, M. Chen, S. Zhang, G. Meng, J. Fu, S. Xiang, H. Ling, Expanding language-image pretrained models for general video recognition (2022).  arXiv:2208.02816.
  * [24] F. Shahavaz, R. Tavassoli, R. Akhavian, Robust activityrecognition for adaptive workerrobot interaction using transfer learning, in: The 2023 ASCE International Conference on Computing in Civil Engineering, CORVALLIS, OR, 2023\.
  * [25] P. H. Le-Khac, G. Healy, A. F. Smeaton, Contrastive representation learning: A framework and review, IEEE Access 8 (2020) 193907–193934.  doi:10.1109/ACCESS.2020.3031549.
  * [26] S. Mai, Y. Zeng, H. Hu, Learning from the global view: Supervised contrastive learning of multimodal representation, Information Fusion 100 (2023) 101920.  doi:https://doi.org/10.1016/j.inffus.2023.101920.
URL https://www.sciencedirect.com/science/article/pii/S1566253523002361

  * [27] R. Nakada, H. I. Gulluk, Z. Deng, W. Ji, J. Zou, L. Zhang, Understanding multimodal contrastive learning and incorporating unpaired data, in: International Conference on Artificial Intelligence and Statistics, PMLR, 2023, pp. 4348–4380.
  * [28] Y. Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, P. Isola, What makes for good views for contrastive learning?, in: H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, H. Lin (Eds.), Advances in Neural Information Processing Systems, Vol. 33, Curran Associates, Inc., 2020, pp. 6827–6839.
URL
https://proceedings.neurips.cc/paper_files/paper/2020/file/4c2e5eaae9152079b9e95845750bb9ab-
Paper.pdf

  * [29] H. O. Song, Y. Xiang, S. Jegelka, S. Savarese, Deep metric learning via lifted structured feature embedding (2015).  arXiv:1511.06452.
  * [30] A. van den Oord, Y. Li, O. Vinyals, Representation learning with contrastive predictive coding (2019).  arXiv:1807.03748.
  * [31] B. Li, Y. Zhang, L. Chen, J. Wang, Y. Jingkang, L. Ziwei, Otter: A multi-modal model with in-context instruction tuning, arXiv preprint arXiv:2305.03726 (2023).
  * [32] A. Awadalla, I. Gao, J. Gardner, J. Hessel, Y. Hanafy, W. Zhu, K. Marathe, Y. Bitton, S. Gadre, J. Jitsev, S. Kornblith, P. W. Koh, G. Ilharco, M. Wortsman, L. Schmidt, Openflamingo (Mar. 2023).  doi:10.5281/zenodo.7733589.
URL https://doi.org/10.5281/zenodo.7733589

  * [33] B. Li, Y. Zhang, L. Chen, J. Wang, F. Pu, J. Yang, C. Li, Z. Liu, Mimic-it: Multi-modal in-context instruction tuning (2023).  arXiv:2306.05425.
  * [34] G. E. Hinton, S. Roweis, Stochastic neighbor embedding, in: S. Becker, S. Thrun, K. Obermayer (Eds.), Advances in Neural Information Processing Systems, Vol. 15, MIT Press, 2002.
URL
https://proceedings.neurips.cc/paper_files/paper/2002/file/6150ccc6069bea6b5716254057a194ef-
Paper.pdf

  * [35] D. Zhu, J. Chen, X. Shen, X. Li, M. Elhoseiny, Minigpt-4: Enhancing vision-language understanding with advanced large language models, arXiv preprint arXiv:2304.10592 (2023).
  * [36] W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, E. P. Xing, Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality (March 2023).
URL https://lmsys.org/blog/2023-03-30-vicuna/

  * [37] P. K. Choubey, A. R. Fabbri, J. Vig, C.-S. Wu, W. Liu, N. F. Rajani, Cape: Contrastive parameter ensembling for reducing hallucination in abstractive summarization (2022).  arXiv:2110.07166.


# PCIE_LAM Solution for Ego4D Looking At Me Challenge

Kanokphan Lertniphonphan
Lenovo Research
<EMAIL_ADDRESS>Jun Xie
Lenovo Research
<EMAIL_ADDRESS>Yaqing Meng
Chinese Academy of Sciences
<EMAIL_ADDRESS>Shijing Wang
Beijing Jiaotong University
<EMAIL_ADDRESS>Feng Chen
Lenovo Research
<EMAIL_ADDRESS>Zhepeng Wang
Lenovo Research
<EMAIL_ADDRESS>
###### Abstract

This report presents our team’s ’PCIE_LAM’ solution for the Ego4D Looking At
Me Challenge at CVPR2024. The main goal of the challenge is to accurately
determine if a person in the scene is looking at the camera wearer, based on a
video where the faces of social partners have been localized. Our proposed
solution, InternLSTM, consists of an InternVL image encoder and a Bi-LSTM
network. The InternVL extracts spatial features, while the Bi-LSTM extracts
temporal features. However, this task is highly challenging due to the
distance between the person in the scene and the camera movement, which
results in significant blurring in the face image. To address the complexity
of the task, we implemented a Gaze Smoothing filter to eliminate noise or
spikes from the output. Our approach achieved the 1st position in the looking
at me challenge with 0.81 mAP and 0.93 accuracy rate. Code is available at
https://github.com/KanokphanL/Ego4D_LAM_InternLSTM

##  1 Introduction

Recently, an egocentric video captured using a wearable camera has gained
significant importance in the fields of computer vision and robotics. In the
Ego4D social benchmark, the interaction between the camera wearer and the
participant is highlighted in the Looking At Me (LAM) challenge [2].

Various approaches [2, 4] have been proposed to accurately analyze the
interaction of the participant with the camera wearer in image sequences. In
[4], landmarks and head pose estimation techniques were employed to improve
spatial feature refinement. Subsequently, the spatial feature refinement and
temporal dynamic refinement were applied on [2].

In this work, we introduce InternLSTM, a method for extracting spatial and
temporal features from sequences of images to enhance performance.
Furthermore, we apply a smoothing filter in the post-processing stage to
eliminate spikes caused by motion blur and low quality face image.

##  2 Method

###  2.1 InternLSTM

Using the strong visual representation capabilities of large language models,
we introduce InternLSTM based on InternVL[1]. It consists of the InternVL
image encoder, Bi-LSTM, and a classification head as shown in figure 1. The
InternVL image encoder was freezed during training, while all other modules
undergo training.

Initially, the spatial features of the image encoder are down-sampled. Then,
the Bi-LSTM network extracts temporal features, capturing intermediary frame
features. A classification head with 4 fully connected layers is utilized
along with the cross entropy loss function.

Figure 1:  The Training and Inference Framework for InternLSTM.  | Validation | Test
---|---|---
Method | mAP ($\uparrow$) | Acc ($\uparrow$) | mAP ($\uparrow$) | Acc ($\uparrow$)
InternLSTM | 84.23% | 94.79% | 76.57% | 90.00%
\+ Horizontal flip when training | 83.66% | 91.93% | 76.94% | 80.04%
\+ TTA | - | - | 77.18% | 80.30%
Table 1: InternLSTM performance on different training and testing setting. | Validation | Test
---|---|---
Method | mAP ($\uparrow$) | Acc ($\uparrow$) | mAP ($\uparrow$) | Acc ($\uparrow$)
GazeLSTM [2] | 79.40% | 92.78% | 74.54% | 87.04%
GazePose [4] | 81.89% | 91.84% | 76.95% | 85.08%
GazePose with Gaze Smoothing | 82.69% | 92.44% | 78.50% | 85.92%
Table 2: Baseline and GazePose performance

Meanwhile, since reversing images should not impact the prediction output, we
augment the original training dataset $S^{train}$ by horizontal flip to create
$S^{train}_{flip}$. Both $S^{train}$ and $S^{train}_{flip}$ are used in the
training. When testing on the test dataset, we also flip the original test
dataset. Applying InternLSTM to the original test dataset $S^{test}$ and the
flipped test dataset $S^{test}_{flip}$, we obtain $R$ and $R_{flip}$,
respectively. Subsequently, softmax is used to obtain the final score.

###  2.2 Test-Time Augmentation (TTA) and Ensemble

During the validation and testing, we implemented TTA to enhance performance.
Specifically, only the horizontal flip technique was utilized in TTA for this
LAM dataset.

In addition to InternLSTM, we reproduced the baseline results [2] and GazePose
[4]. To combine these results, ensemble methods [3] were utilized. The overall
ensemble score was computed as the mean of the combined prediction scores.

###  2.3 Gaze Smoothing

In this dataset, images from the wearable camera contain significant blurring
in the face image. The image with motion blur reduces the performance of gaze
direction or head pose estimation from the face image. Moreover, transitions
in behavior take some time compared to sudden changes induced by motion noise.
To address this, a median smoothing filter was applied as a post-processing to
mitigate spikes in prediction scores. This filter calculates the median value
within a specified time window and replaces the central score with the median
value.

##  3 Experiments

###  3.1 Dataset and Evaluation metric

We conducted experiments using the Ego4D social benchmarks [2]. The dataset
comprises 572 clips, each lasting 5 minutes. 389 and 50 clips are allocated
for training and validation, respectively, with the remainder set aside for
testing. Ground truth includes a binary label for each face in every frame
within the time segment showing a person looking at the camera wearer. Mean
Average Precision (mAP) and Top-1 accuracy are used for evaluation.

###  3.2 Experimental Setting

First, we employed InternVL on 4 A100 (80GB) GPUs to extract features from all
images, including both original and flipped version. Then, InternLSTM was
trained with a batch size of 128 using the Adam optimizer and a learning rate
of 1e-4.

Figure 2:  The InternLSTM performance varies with different image qualities.
The model incorrectly predicted labels for blurry images as shown in the right
image.

###  3.3 Results

The results from InternLSTM are presented in table 1. We experimented
InternLSTM using original training images and flipped augmentation. Our
InternLSTM model was initially trained without any augmentation and achieved
84.23% mAP and 94.79% accurancy on the validation set. Then we train the model
with horizontal flip and achieved 83.66% mAP and 91.93% accuracy. Finally, TTA
(horizontal flip only) was applied during inference and achieve the highest
score in test set.

InternLSTM performs well in predicting results when images are clear. However,
its predictions deviate from the ground truth when the image quality is low,
as illustrated in figure 2.

We also reproduced the baseline [2] and GazePose [4] approaches as shown in
table 2. We added Gaze smoothing to GazePose results and improved the
performance in both validation and testing.

From the InternLSTM results, data augmentation and TTA improved the testing
results, but the validation results did not show the same improvement for our
model. Comparing the mAP in table 1 and 2, a higher mAP in validation may not
always result in better testing performance. This issue also occurred with
other approaches we attempted, making model selection challenging.

Our final results combined outputs from several approaches including
InternLSTM, GazeLSTM, and GazePose with various parameter configurations. The
results after applying gaze smoothing are presented in table 3. Our approach
achieved the 1st position in the Ego4D Looking At Me leaderboard with an mAP
of 0.81 and an accuracy of 0.93.

Team | mAP ($\uparrow$) | Acc ($\uparrow$)
---|---|---
PCIE_LAM (Ours) | 0.81 | 0.93
listen | 0.80 | 0.87
PKU-WICT-MIPL | 0.79 | 0.92
ydejie | 0.79 | 0.92
Dejie (gb6) | 0.79 | 0.92
Host_24191_Team | 0.66 | 0.74
Table 3: Ego4D Looking at me Challenge Leaderboard

##  4 Conclusion

We introduce InternLSTM consists of an InternVL image encoder and a Bi-LSTM
for spatial and temporal feature extraction from sequence images. The data
augmentation and TTA helped improve model performance. We also applied
ensemble and gaze smoothing to increase accuracy and reduce noise from the
outputs. Our next step is to enhance robustness of the model over the blur
input images.

## References

  * Chen et al. [2023] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai.  Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.  _arXiv preprint arXiv:2312.14238_ , 2023.
  * et al. [2021] Kristen Grauman et al.  Ego4d: Around the world in 3,000 hours of egocentric video.  _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 18973–18990, 2021.
  * Solovyev et al. [2021] Roman Solovyev, Weimin Wang, and Tatiana Gabruseva.  Weighted boxes fusion: Ensembling boxes from different object detection models.  _Image and Vision Computing_ , 107:104117, 2021.
  * [4] Xiyu Wei, Dejie Yang, Yuxin Peng, and Yang Liu.  Team pku-wict-mipl ego4d look at me challenge 2023 technical report.