# GRASP: A novel benchmark for evaluating language GRounding And Situated
Physics understanding in multimodal language models

Serwan Jassim1, Mario Holubar2, Annika Richter1,
Cornelius Wolff1, Xenia Ohmer1111Shared senior authorship, Elia
Bruni1111Shared senior authorship
1Osnabrück University, 2University of Amsterdam
{serwan.jassim, annika.richter, xenia.ohmer<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

This paper presents GRASP, a novel benchmark to evaluate the language
grounding and physical understanding capabilities of video-based multimodal
large language models (LLMs). This evaluation is accomplished via a two-tier
approach leveraging Unity simulations. The initial level tests for language
grounding by assessing a model’s ability to relate simple textual descriptions
with visual information. The second level evaluates the model’s understanding
of ‘Intuitive Physics’ principles, such as object permanence and continuity.
In addition to releasing the benchmark, we use it to evaluate several state-
of-the-art multimodal LLMs. Our evaluation reveals significant shortcomings in
current models’ language grounding and intuitive physics. These identified
limitations underline the importance of benchmarks like GRASP to monitor the
progress of future models in developing these competencies.

GRASP: A novel benchmark for evaluating language GRounding And Situated
Physics understanding in multimodal language models

Serwan Jassim1, Mario Holubar2, Annika Richter1, Cornelius Wolff1, Xenia
Ohmer1111Shared senior authorship, Elia Bruni1111Shared senior authorship
1Osnabrück University, 2University of Amsterdam {serwan.jassim,
annika.richter, xenia.ohmer<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

The remarkable progress of large language models (LLMs) has sparked intense
debate about their potential for achieving genuine machine cognition and
human-level comprehension. A very important question is whether these models
possess an understanding of the physical world, even though they learn
primarily through next-word prediction, so in a fundamentally ungrounded
manner. Recently, multimodal LLMs have started to make their mark, learning
not just from vast text databases but also from extensive visual inputs.
GPT-4, for example, can process inputs consisting of arbitrarily interlaced
text and images (OpenAI, 2023). However, it remains unclear whether
multimodality enhances language grounding and physical understanding. With
GRASP, we propose a novel evaluation benchmark to answer these questions for
video-based LLMs.

Most research on physical understanding in LLMs has been centered around tasks
that assess a model’s ability to correctly link a textual question to its
corresponding textual answer (e.g., Gordon et al., 2012; Hendrycks et al.,
2021). For instance, consider a datapoint in the “Choice of plausible
alternatives” (COPA) dataset (Gordon et al., 2012): Given the text “I poured
water into the glass.”, the model has to choose one of the two effects “The
water quenched my thirst.” or “The glass became full.” Under a conservative
interpretation, this test evaluates whether the model has learned to map the
input text pattern onto the correct output text pattern from examples of
similar sentence pairs in its vast training dataset. It certainly does not
evaluate whether the model knows what water looks and behaves like and whether
the model genuinely understands that pouring liquid into a glass will fill up
that glass.

GRASP advances the evaluation of language grounding and physical grounding by
leveraging multimodality: Questions about objects and their behaviors are
connected to an external environment. Specifically, we use the
Unity111https://unity.com/ game engine to simulate various scenes (in the form
of videos) as a proxy for the real world. The model has to answer textual
questions about these scenes, which are designed to assess its ability to
relate text to the physical world (_grounding_) as well as its ability to
understand the basic principles of physics that govern how objects behave in
the world (_Intuitive Physics_).

Based on these two aspects, the GRASP dataset is comprised of two levels.
Level 1 tests for grounding: The model has to detect simple objects and
recognize object features, relative positions, and direction of motion. This
level tests whether a model can map between simple textual descriptions and
visual inputs. Passing Level 1 is necessary, if the model is to generate
statements about the physical properties and behaviors of objects in a visual
scene. It is therefore a requirement for passing Level 2 which tests for
Intuitive Physics: The model has to judge the physical plausibility of video
sequences, where simple objects behave according to, or violate, Intuitive
Physics concepts such as object permanence or gravity. The knowledge and
abilities of LLMs are tied to their accessibility through language. Thus, the
two levels evaluate a model’s ability to “perceive” the environment and to
“reason” about the physical events therein within the constraints of its
language interface.

This language interface differentiates GRASP from other image- or video-based
datasets for Intuitive Physics, which were largely developed to train and
evaluate dedicated Intuitive Physics models (lacking language) (e.g. Battaglia
et al., 2016; Watters et al., 2017; Piloto et al., 2022). Most relevant to our
approach is the recently developed Physical
Concepts222https://github.com/deepmind/physical_concepts dataset (Piloto et
al., 2022), which, similar to our Level 2, consists of simulated videos of
physically plausible and implausible events. Instead of targeting dedicated
Intuitive Physics models, GRASP is designed to evaluate whether language
grounding and Intuitive Physics are a subset of the abilities that naturally
emerge in multimodal LLMs.

To this end, GRASP extends and improves existing datasets in several ways.
First, it presents novel stimuli to test for grounding (Level 1). While
grounding is naturally absent in vision-only models it is a key prerequisite
for a (question-based) evaluation of Intuitive Physics in multimodal LLMs.
Second, it significantly broadens the range of Intuitive Physics concepts that
can be tested (Level 2). While Physical Concepts comprises five concepts, our
benchmark comprises seven and provides multiple experiments for some concepts
as well as combinations of concepts. Lastly, by disseminating the Unity source
code along with the benchmark data, we enable the community to customize and
expand the benchmark as multimodal LLMs become more sophisticated.

Accompanying the release of GRASP, we provide scores for three state-of-the-
art multimodal LLMs. In addition, we run a qualitative analysis, relying
solely on descriptions of the benchmark scenes. This approach allows us to
assess the models’ textual understanding of intuitive physics decoupled from
their visual component. Our findings reveal that, despite their impressive
capabilities, current multimodal LLMs are still lacking in both language
grounding and intuitive physics understanding. These shortcomings emphasize
the necessity for using benchmarks like GRASP to monitor the progress of
future models in terms of these capabilities.

##  2 Related Work

GRASP takes inspiration from psychology research on Intuitive Physics in early
development. It is related to other datasets that have been created to develop
neural network models of Intuitive Physics but targets language models. We
study grounding and physics understanding in LLMs but instead of looking at
text-based models, we focus on recently developed video-based multimodal LLMs.

##### Multimodal LLMs.

In recent years, LLMs have demonstrated remarkable capabilities in language
understanding and reasoning tasks. Following the success of early foundation
models like GPT-2 Radford et al. (2019) and BERT Devlin et al. (2019),
development has moved towards a steady increase in model size and training
data. Models with hundreds of billions of parameters, such as GPT-3 Brown et
al. (2020), PaLM Chowdhery et al. (2022), Gopher Rae et al. (2022) and BLOOM
Scao et al. (2023) have exhibited exceptional generalization and in-context
learning performance Brown et al. (2020); Wei et al. (2022).

With the recent widespread success of LLMs, researchers have also explored
their use for processing multi-modal inputs. A key idea utilized by seminal
works such as Flamingo Alayrac et al. (2022) and BLIP-2 Li et al. (2023a) is
the alignment of a pretrained visual model with the textual embedding space of
the LLM. LLaVA Liu et al. (2023) and MiniGPT-4 Zhu et al. (2023) combine this
technique with instruction tuning to deliver an end-to-end chatbot with visual
reasoning abilities.

Recent work has extended this approach to video data. VideoChat Li et al.
(2023b) was among the first to do so, followed by Video-ChatGPT Maaz et al.
(2023). Video-LLaMA Zhang et al. (2023) adds an additional audio processing
branch to the pipeline, and PandaGPT Su et al. (2023) takes the idea further
with support for six different modalities including thermal images and IMU
sensor data. Although these models differ in implementation and dataset
generation, they are all based on the premise of aligning the embeddings of
pretrained foundation models through a learnable interface.

##### Intuitive Physics in early development.

To evaluate and train intuitive physics in LLMs, we will rely on extensive
research on the subject from developmental psychology. In particular, we will
focus on fundamental concepts of intuitive physics that have been identified
in this research, such as object permanence Baillargeon et al. (1985);
Baillargeon (1987); Spelke et al. (1992), occlusion Hespos and Baillargeon
(2001), gravity (Kim and Spelke, 1992, 1999; Spelke et al., 1992), and inertia
(Kim and Spelke, 1999; Spelke et al., 1992, 1994). Studies in this field
typically employ simple experimental setups with geometric shapes that are
part of physically possible or physically impossible events and assess
understanding with the so-called violation-of-expection (VoE) paradigm. This
paradigm is based on the idea that infants will show surprise—measured through
their behavioral or physiological response—when an event violates their
expectations. We draw inspiration from known experimental setups to develop
the videos for GRASP. Since LLMs possess language, we can inquire about the
plausibility or implausibility of the scenes directly, mapping the VoE
paradigm onto a binary classification problem.

##### Neural network models of Intuitive Physics.

Advances in deep learning and AI have paved the way for the development of
dedicated models capable of learning Intuitive Physics from data (for an
overview, see Duan et al., 2022). One of the earliest examples is the so-
called Interaction Network (Battaglia et al., 2016). It received explicit
information about objects and their relationships for a given scene and was
trained on next-state prediction. Scenes included n-body systems, bouncing
balls, and springs colliding with rigid bodies. Since then, the research focus
has shifted to learning Intuitive Physics from raw visual inputs. For example,
Watters et al. (2017) extended the Interaction Network with a convolutional
neural network to process image inputs. Other prominent examples include
neural networks predicting the behavior of block towers (Lerer et al., 2016)
or the dynamics of robot-object interactions (Agrawal et al., 2016). As a part
of model development and evaluation, several Intuitive Physics benchmarks have
been developed. Among others, the Physical Concepts (Piloto et al., 2022),
IntPhys (Riochet et al., 2022), InfLevel (Weihs et al., 2022), and AVoE
(Dasgupta et al., 2021a, b) benchmarks utilize simulated videos to assess
physical concepts in computer vision models. Like GRASP, they focus on
established concepts and VoE experiments from the developmental psychology
literature. GRASP can be considered an expansion of these benchmarks: It adds
an entirely new level to test for grounding, as well as more concepts, more
scenes per concept, and combinations of concepts. Besides, unlike all the
approaches above, it is designed to evaluate LLMs and therefore uses a
language interface, allowing for analyses of the models’ physics understanding
within and across the modalities of language and vision.

##### Grounding and physics understanding in LLMs.

Before the advent of multimodal LLMs, it has often been argued that LLMs—being
trained on text—fail to relate language to the physical world (e.g. Bisk et
al., 2020). Progress on neural networks with grounded language abilities
largely happened in research fields stemming from image captioning (Mitchell
et al., 2012), such as visual question answering (VQA) Antol et al. (2015),
instruction following (e.g. Anderson et al., 2018; Das et al., 2018), and
visual commonsense reasoning Zellers et al. (2019). Among others, the VQA
datasets CLEVR Johnson et al. (2017) and CLEVRER Yi et al. (2020) have been
developed in the context of this research. The former tests for compositional
language and elementary visual reasoning using images of 3D shapes. The latter
tests for various visual reasoning capabilities based on videos of colliding
objects (again 3D shapes). At the time CLEVRER was released, SOTA models
performed well on descriptive tasks but poorly on causal ones requiring
explanation, prediction or counterfactual reasoning. Now, multimodal LLMs
promise to provide extensive language capabilities while being able to relate
between text and images or videos. For example, GPT-4 outperforms SOTA models
on various VQA benchmarks (https://openai.com/research/gpt-4). GRASP provides
a novel, and extensive benchmark to test fundamental grounding and Intuitive
Physics skills in a question-answering setting.

##  3 Benchmark Design

GRASP is a two-level benchmark, with each level containing multiple visual
tests. These tests were modeled in the Unity simulator and compiled into a
dataset in the form of videos.333Although most Level 1 tests do not entail
dynamics and could therefore be represented as images, we decided to capture
them as videos to ensure consistency across all Level 1 and 2 tests. All
videos are ten seconds long and were generated at 50 frames per second.

###  3.1 Level 1 (Grounding)

In the initial stage of our GRASP benchmark, we evaluate the elementary visual
understanding capabilities of LLMs. This stage comprises tests that assess
basic visual comprehension and lay the groundwork for higher-order reasoning
required in the subsequent level. Our premise is that models struggling at
this foundational phase will likely encounter difficulties in the next stage,
where they must discern and reason about more complex physical interactions.
This approach ensures a sequential build-up in the complexity of tasks,
aligning with the natural progression of cognitive development.

We categorize the tests that make up this part of GRASP as

  * •

Shape & Color: A cube or sphere of random color is spawned at a random
location on a table.

  * •

Directionality: A ball rolls either forward, backward, right, or left.

  * •

Object Ordering: A random sequence of balls or cubes of random color are
spawned on a table. Three versions of this test exist with two, three, and
four objects.

  * •

Relational Position: A ball is spawned either to the left or right of a
barrier.

Visual examples of these tests are displayed in Figure 1. Importantly, the
elements used in this phase—such as objects in various shapes, colors, and
positional relationships—are consistently present in the next stage.

Figure 1: Tests in Level 1 of GRASP: (a) Shape & Color, (b) Directionality
(ball is rolling from right to left as indicated by red arrow), (c) Object
Ordering, (d) Relational Position

####  3.1.1 Prompts

We construct this level of the benchmark as a binary classification
problem444In the GRASP benchmark, both levels are structured as binary
classification tasks. Despite Level 1 tests initially suggesting a multi-class
format, we chose binary classification for uniformity and simplicity. This
approach facilitates easier parsing of model outputs and ensures compatibility
across levels, allowing models to respond with just two distinct tokens. by
generating positive and negative samples from videos as follows: After the
video is served to the models, we use a prompt that proposes an observation
and asks whether it is true. The proposed observation is either an accurate
description of the video (pos. sample) or a false one (neg. sample). In the
case of the example in Figure 1c, the model is prompted with “From left to
right, the following objects are on the table: white ball, white cube, white
ball, red ball. Is this true?” (pos. sample) and “From left to right, the
following objects are on the table: red ball, white ball, white ball, white
cube. Is this true?” (neg. sample). A complete list of the prompts is given in
Appendix A.1.

For the Object Ordering test, we exclusively create negative samples by
permuting the existing objects, specifically to assess ordering accuracy. In
contrast, for all other tests, we sample entirely different observations.

In order to evaluate the sensitivity of the models with respect to the
prompts, we propose a set of alternative prompts that allow for answers other
than yes and no, e.g. “Which direction is the ball rolling?” for the
Directionality test. These prompts were introduced for all, except for the
Object Ordering test, and can be found in Appendix A.2.

###  3.2 Level 2 (Intuitive Physics)

Following the concept of VoE experiments in psychological research, this level
contains videos that are either plausible or implausible with respect to
physical concepts and events. The concepts/events for which we test in our
benchmark are

  * •

Continuity: Objects cannot teleport in space and time, they can only move
along continuous paths (Spelke et al., 1992).

  * •

Solidity: Objects cannot overlap in space and time, they can only move along
clear paths (Spelke et al., 1992).

  * •

Unchangeableness: Objects cannot spontaneously change their size, shape, and
color (Baillargeon et al., 2012).

  * •

Gravity: Objects move downward without existing support (Spelke et al., 1992).

  * •

Collision: Objects get displaced when hit by moving objects (Baillargeon,
1995).

  * •

Object Permanence: Objects continue to exist when they are occluded
(Baillargeon, 1995).

  * •

Inertia: Objects do not spontaneously alter their motion (Spelke et al.,
1992).

Figure 2: Examples from the Level 2 benchmark. The first and second rows
display plausible and implausible versions respectively for each experiment.
These examples test for the understanding of the solidity, continuity (a), and
inertia principle (b).

We adopted 14 tests from the psychological research literature and simulated
them in Unity. Illustrations of these tests can be found in Figure 2 and
Appendix F and their distribution across the physical concepts and events is
shown in Table 1. Since it is not always possible to strictly disentangle all
the concepts in the tests (e.g. in the implausible scenario displayed in
Figure 2a, the ball could have either teleported behind (continuity violation)
or rolled through the barrier (solidity violation)), there is a discrepancy
between the sum of the distribution in Table 1 and the number of experiments.

From the psychological literature, we adopt the use of occluders to hide
physical manipulations in experiments. This ensures that models need to infer
plausibility using the entire history of a video instead of individual frames.

Table 1: Test distribution over physical principles/events that are evaluated in Level 2 of GRASP (OP = Object Permanence). Continuity | Solidity | Inertia | Gravity
---|---|---|---
4 | 2 | 3 | 4
Collision | OP | Unchangeableness
1 | 3 | 2

####  3.2.1 Visual Randomization

In GRASP, models are validated against a set of videos per test to ensure that
results are representative. A rich variation in visuals was achieved across
videos by randomly sampling object colors/textures, background colors, camera
angles, movement speeds, and start delays across all tests. Furthermore, in
asymmetric tests we randomly mirrored the experimental setup, and where
applicable, we also ran randomized test-specific settings (see Appendix C).

####  3.2.2 Prompts

The task associated with this level involves evaluating the physical
plausibility of generated videos, constituting a binary classification task.
In contrast to the previous level, negative and positive samples of a test
share the same prompt and are distinguished by the video contents. All prompts
share the same template:

$\begin{gathered}\text{The video you're seeing was generated by a}\\\
\text{simulator. Given how objects behave on earth, is}\\\ \text{$\langle$
{observation} $\rangle$ plausible? Your answer should}\\\ \text{be based on
the events in the video and ignore}\\\ \text{the quality of the simulation.
Answer only with}\\\ \text{yes or no.}\end{gathered}$

In this case $\langle$observation$\rangle$ denotes a phrase that hints at what
the model should give attention to when infering about the physical
plausibility, e.g. “the trajectory of the ball“ or “the final position of the
ball“. The prompts include an instruction to ignore details of the simulator
in an attempt to prevent models from using visual imperfections in their
reasoning. For the complete list of prompts, see Appendix B.

Additionally, we introduce prompts containing scene descriptions for each
plausible and implausible experiment. These prompts are served to the models
without videos in order to assess their textual understanding of intuitive
physics. For the list of scene descriptions, see Appendix D

##  4 Experiments

###  4.1 Models

We consider several multimodal large language models that can perform video-
based question answering.

Video-ChatGPT Maaz et al. (2023) leverages LLaVA Liu et al. (2023) as a
vision-language model and adapts it to video data, fine-tuning the model on a
dataset of 100,000 video-instruction pairs to enable understanding of temporal
dynamics. The version of LLaVA used is LLaVA-Lightning-7B-v1-1, which itself
is comprised of CLIP ViT-L/14 Radford et al. (2021) as a visual encoder and
Vicuna-7B-1.1 Chiang et al. (2023) as a language decoder.

Video-LLaMA Zhang et al. (2023) enables simultaneous visual and auditory
understanding using a multi-branch cross-modal pre-training framework. The
vision-language branch uses CLIP ViT-G/14 Radford et al. (2021) and BLIP-2
Q-Former Li et al. (2023a) and is trained using video-, as well as image-text
data.

PandaGPT Su et al. (2023) uses the joint embeddings of ImageBind Girdhar et
al. (2023) to enable a Vicuna model to reason about image, video, depth,
thermal, and IMU data.

###  4.2 Experimental Setup

Experiments are conducted on a NVIDIA 3090 GPU. For the models that have
different sizes available, we use the 7B parameter variant. Parameters used
for prompting the models are outlined in Appendix E.

Each video-prompt pair is evaluated three times with a different seed per
model. For quantitative evaluation, responses are only counted as valid if
they begin with the word “yes” or “no”; the rest of the response is considered
irrelevant. Responses that do not adhere to this are counted as incorrect.

Since PandaGPT and Video-ChatGPT do not allow for text-only inputs, we only
compare Video-LLaMA to GPT-4 in the test with scene descriptions.

##  5 Results

Table 2: Accuracy for all models on part of GRASP’s Level 1 using binary question prompts. Results are displayed for positive and negative samples as well as an average accuracy. Correct answer are “yes“ for pos. samples and “no“ for neg. samples. Test | Video-LLaMA | PandaGPT | Video-ChatGPT
---|---|---|---
pos. | neg. | avg. | pos. | neg. | avg. | pos. | neg. | avg.
Shape & Color | 0.98 | 0.10 | 0.54 | 1.00 | 0.00 | 0.50 | 0.30 | 0.73 | 0.51
Directionality | 0.57 | 0.19 | 0.38 | 0.98 | 0.03 | 0.50 | 0.61 | 0.38 | 0.49
Relational Position | 0.98 | 0.01 | 0.50 | 1.00 | 0.00 | 0.50 | 0.69 | 0.28 | 0.49
Ordering (2 obj.) | 0.94 | 0.04 | 0.49 | 1.00 | 0.00 | 0.50 | 0.05 | 0.93 | 0.49
Ordering (3 obj.) | 0.92 | 0.05 | 0.48 | 1.00 | 0.00 | 0.50 | 0.06 | 0.92 | 0.49
Ordering (4 obj.) | 0.91 | 0.05 | 0.48 | 1.00 | 0.00 | 0.50 | 0.01 | 0.98 | 0.50
Table 3: Accuracy for all models on GRASP’s Level 1 using open-ended question prompts. Test | Video-LLaMA | PandaGPT | Video-ChatGPT
---|---|---|---
Shape & Color | 0.84 | 0.49 | 0.55
Directionality | 0.17 | 0.19 | 0.24
Relational Position | 0.39 | 0.50 | 0.50
Table 4: Accuracy for all models on GRASP’s Level 2. Results are displayed for implausible and plausible scenes as well as an average accuracy. Correct answer are “yes“ for plausible scenes and “no“ for implausible scenes. OP denotes object permanence. The tests are numbered following Appendix F. Test | Video-LLaMA | PandaGPT | Video-ChatGPT
---|---|---|---
# | principle/event | plaus. | impl. | avg. | plaus. | impl. | avg. | plaus. | impl. | avg.
1 | unchangeableness | 0.48 | 0.00 | 0.24 | 0.00 | 1.00 | 0.50 | 0.94 | 0.06 | 0.50
2 | unchangeableness | 0.58 | 0.02 | 0.30 | 0.09 | 0.87 | 0.48 | 0.85 | 0.20 | 0.52
4 | solidity & continuity | 0.58 | 0.12 | 0.35 | 0.00 | 1.00 | 0.50 | 0.22 | 0.78 | 0.50
5 | solidity & continuity | 0.50 | 0.02 | 0.26 | 0.00 | 1.00 | 0.50 | 0.45 | 0.47 | 0.46
6 | OP | 0.62 | 0.09 | 0.35 | 0.41 | 0.56 | 0.48 | 0.11 | 0.85 | 0.48
7 | OP | 0.38 | 0.01 | 0.20 | 0.00 | 0.99 | 0.49 | 0.51 | 0.39 | 0.45
8 | OP | 0.51 | 0.13 | 0.32 | 0.00 | 1.00 | 0.50 | 0.07 | 0.93 | 0.50
9 | inertia | 0.54 | 0.01 | 0.27 | 0.00 | 1.00 | 0.50 | 0.61 | 0.46 | 0.54
11 | gravity | 0.61 | 0.02 | 0.32 | 0.13 | 0.86 | 0.50 | 0.68 | 0.31 | 0.49
12 | gravity & inertia | 0.42 | 0.01 | 0.21 | 0.00 | 1.00 | 0.50 | 0.40 | 0.67 | 0.53
13 | gravity & inertia | 0.56 | 0.00 | 0.28 | 0.00 | 0.95 | 0.48 | 0.70 | 0.27 | 0.48
14 | continuity | 0.43 | 0.02 | 0.23 | 0.00 | 1.00 | 0.50 | 0.28 | 0.74 | 0.51
15 | continuity & gravity | 0.50 | 0.04 | 0.27 | 0.00 | 1.00 | 0.50 | 0.45 | 0.58 | 0.51
16 | collision | 0.74 | 0.07 | 0.40 | 0.00 | 1.00 | 0.50 | 0.56 | 0.36 | 0.46

Table 2 presents accuracy scores for the multimodal LLMs tested on Level 1
(grounding) in the binary classification task. The average scores across all
models and tests indicate a performance close to chance (0.50). Video-LLaMA
even displays below chance performance on the Directionality test because it
generated ambiguous answers that could not be parsed.

Examining scores on positive and negative samples highlights distinctive
behaviors among the models. Video-LLaMA exhibits a consistent bias toward
responding “yes” to prompts, while Video-ChatGPT displays a more dynamic bias,
shifting between “yes” and “no” responses across different tests. PandaGPT, in
turn, consistently responds with “yes” regardless of the prompt, with some
exceptions in the Directionality test.

Measured accuracies for Level 1 with open-ended question prompts are presented
in Table 3. For the Directionality and Relational Position tasks the results
coincide with the binary prompts for all models (in this case, accuracies of
0.25 and 0.50 equal chance performance for Directionality and Relational
Position respectively). The Shape & Color task with open-ended questions
constitutes an exception to the poor performances on Level 1: All models
perform quite well here, especially considering that there are no restrictions
to the solution space with respect to the color of the object.

Results for Level 2 (Intuitive Physics) are listed in Table 4. The models
generally display chance performance across concepts and scenes. As for Level
1, Video-LLaMA performs below chance because it generates answers that cannot
be parsed. The models’ poor performance on Level 2 is not surprising given
that they already failed on Level 1, which assesses basic grounding abilities
necessary to answer questions about the videos at Level 2.

Evaluation with Video-LLaMA using only scene descriptions yields an average
accuracy of 0.44. In comparison, GPT-4 achieves an accuracy of 0.77 on this
task.

##  6 Discussion

Our results show that the tested multimodal LLMs lack a basic perceptual
understanding of simulated scenes. Besides some understanding of simple shapes
and colors, the models fail to answer basic questions about (relative)
positions of static objects and movement of objects (Level 1). Because of
that, it is not surprising that they also fail to judge the physical
plausibility of simple object behaviors (Level 2).

One potential reason for the poor performance of the models could be the
discrepancy between the simulated videos in our benchmarks and the real-world
training data of the models. In other words, our benchmark data is out-of-
distribution (OOD) for the model. Generating a controlled dataset of real-
world videos that test fundamental aspects of grounding and Intuitive Physics
is challenging. Still, it would be interesting to conduct a comparison with
selected examples of such stimuli in future work. Either way, GRASP is useful
as a challenging benchmark. Due to its OOD nature it tests for “true”
understanding that generalizes to novel, and in our case abstracted,
scenarios.

As demonstrated in the Level 1 comparison, employing an alternative prompt can
result in a substantial performance boost. This suggests that the observed
performance can also be partly attributed to the nature of the prompts.
Consequently, investigating the significant impact of prompts on models poses
an important future research question. Within the context of GRASP, we view
this observation as an additional challenge to be addressed by future models.

Future work will encompass the analysis of additional multimodal LLMs.
Considering the recency of video-based multimodal LLMs (all the evaluated
models were released this year), their capabilities may improve significantly
in the near future. Compared to text-based LLMs, which at this point contain
hundreds of billions of parameters (Naveed et al., 2023), video-based
multimodal LLMs are at least one order of magnitude smaller (Zhang et al.,
2023; Su et al., 2023; Maaz et al., 2023). Supported by the obervation that
GPT-4 significantly ouperforms Video-LLaMA in the scene descriptions test, an
increase in model size alone might lead to the emergence of relevant language
grounding and physical reasoning capabilities. Image-based multimodal LLMs,
such as GPT-4, have proven remarkably adept at answering complex and detailed
questions about images. Therefore, we aim to create an image-based version of
GRASP and to compare image- and video-based models. Throughout future
developments in multimodal LLM capabilities, GRASP will prove instrumental in
tracking the progression of these models, testing their grounding and physical
comprehension capabilities against demanding data sets.

The limitations observed in current models’ performance on GRASP’s Level 1
stress the need for additional basic perceptual tests to allow for a more
detailed analysis. Furthermore, a future expansion to the benchmark may
involve introducing a subsequent level that requires models to address
challenges within an interactive simulated environment. Lastly, to disentangle
failure attribution between the language and vision components of multimodal
LLMs, we plan the inclusion of a comprehensive textual benchmark for Intuitive
Physics.

##  7 Conclusion

GRASP introduces a robust grounding and Intuitive Physics benchmark tailored
for multimodal LLMs. By using simulated videos to model basic perceptual tasks
and faithfully reproducing experiments from the developmental psychology
research within simulation, GRASP serves as a comprehensive evaluation
platform. Results across both benchmark tiers demonstrate the challenging
nature of GRASP. Notably, the results showcase a lacking perceptual
understanding of simulated scenes by existing models, stressing the need for
further development in this domain. Our future plans involve expanding the
benchmark to facilitate the research within the intersection of language and
perception.

## References

  * Agrawal et al. (2016) Pulkit Agrawal, Ashvin V. Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. 2016.  Learning to poke by poking: Experiential learning of intuitive physics.  In _Proceedings of the 30th International Conference on Neural Information Processing Systems (NeurIPS)_ , volume 29. Curran Associates, Inc.
  * Alayrac et al. (2022) Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. 2022.  Flamingo: a visual language model for few-shot learning.  _ArXiv Preprint_ , arXiv:2204.14198.
  * Anderson et al. (2018) Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. 2018.  Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Antol et al. (2015) Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015.  VQA: Visual question answering.  In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_.
  * Baillargeon (1995) Renee Baillargeon. 1995.  Physical reasoning in infancy.  In Michael S. Gazzaniga, editor, _The Cognitive Neurosciences_ , pages 181–204. MIT Press.
  * Baillargeon (1987) Renée Baillargeon. 1987.  Object permanence in 3½- and 4½-month-old infants.  _Developmental Psychology_ , 23(5):655–664.
  * Baillargeon et al. (1985) Renée Baillargeon, Elizabeth S. Spelke, and Stanley Wasserman. 1985.  Object permanence in five-month-old infants.  _Cognition_ , 20:191–208.
  * Baillargeon et al. (2012) Renée Baillargeon, Maayan Stavans, Di Wu, Yael Gertner, Peipei Setoh, Audrey K. Kittredge, and Amélie Bernard. 2012.  Object individuation and physical reasoning in infancy: An integrative account.  _Language learning and development: The official journal of the Society for Language Development_ , 8(1):4–46.
  * Battaglia et al. (2016) Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, and Koray kavukcuoglu. 2016.  Interaction networks for learning about objects, relations and physics.  In _Proceedings of the 30th International Conference on Neural Information Processing Systems (NeurIPS)_ , NeurIPS’16, pages 4509–4517, Red Hook, NY, USA. Curran Associates Inc.
  * Bisk et al. (2020) Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian. 2020.  Experience grounds language.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8718–8735, Online. Association for Computational Linguistics.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  _ArXiv Preprint_ , arXiv:2005.14165.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.  Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality.  _LMSYS Org Blog_.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, …, and Noah Fiedel. 2022.  Palm: Scaling language modeling with pathways.  _ArXiv Preprint_ , arXiv:2204.02311.
  * Das et al. (2018) Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. 2018.  Embodied question answering.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Dasgupta et al. (2021a) Arijit Dasgupta, Jiafei Duan, Marcelo H Ang Jr, Yi Lin, Su-hua Wang, Renée Baillargeon, and Cheston Tan. 2021a.  A benchmark for modeling violation-of-expectation in physical reasoning across event categories.  _ArXiv Preprint_ , arXiv:2111.0882.
  * Dasgupta et al. (2021b) Arijit Dasgupta, Jiafei Duan, Marcelo H Ang Jr, and Cheston Tan. 2021b.  Avoe: A synthetic 3d dataset on understanding violation of expectation for artificial cognition.  _ArXiv Preprint_ , arXiv:2110.05836.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _ArXiv Preprint_ , arXiv:1810.04805.
  * Duan et al. (2022) Jiafei Duan, Arijit Dasgupta, Jason Fischer, and Cheston Tan. 2022.  A survey on machine learning approaches for modelling intuitive physics.  In _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22_ , pages 5444–5452. International Joint Conferences on Artificial Intelligence Organization.  Survey Track.
  * Girdhar et al. (2023) Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. 2023.  Imagebind: One embedding space to bind them all.  _ArXiv Preprint_ , arXiv:2305.05665.
  * Gordon et al. (2012) Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. 2012.  SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning.  In _*SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)_ , pages 394–398, Montréal, Canada. Association for Computational Linguistics.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021.  Measuring massive multitask language understanding.  In _International Conference on Learning Representations (ICLR)_.
  * Hespos and Baillargeon (2001) Susan J. Hespos and Renée Baillargeon. 2001.  Infants’ knowledge about occlusion and containment events: a surprising discrepancy.  _Psychological Science_ , 12(2):141–147.
  * Johnson et al. (2017) Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. 2017.  CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Kim and Spelke (1992) In K. Kim and Elizabeth S. Spelke. 1992.  Infants’ sensitivity to effects of gravity on visible object motion.  _Journal of Experimental Psychology: Human Perception and Performance_ , 18(2):385–393.
  * Kim and Spelke (1999) In K. Kim and Elizabeth S. Spelke. 1999.  Perception and understanding of effects of gravity and inertia on object motion.  _Developmental Science_ , 2(3):339–362.
  * Lerer et al. (2016) Adam Lerer, Sam Gross, and Rob Fergus. 2016.  Learning physical intuition of block towers by example.  In _Proceedings of The 33rd International Conference on Machine Learning (ICML)_ , volume 48 of _Proceedings of Machine Learning Research_ , pages 430–438. PMLR.
  * Li et al. (2023a) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023a.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _ArXiv Preprint_ , arXiv:2301.12597.
  * Li et al. (2023b) KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2023b.  Videochat: Chat-centric video understanding.  _ArXiv Preprint_ , arXiv:2305.06355.
  * Liu et al. (2023) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023.  Visual instruction tuning.  _ArXiv Preprint_ , arXiv:2304.08485.
  * Maaz et al. (2023) Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. 2023.  Video-ChatGPT: Towards detailed video understanding via large vision and language models.  _ArXiv Preprint_ , arXiv:2306.05424.
  * Mitchell et al. (2012) Margaret Mitchell, Jesse Dodge, Amit Goyal, Kota Yamaguchi, Karl Stratos, Xufeng Han, Alyssa Mensch, Alex Berg, Tamara Berg, and Hal Daumé III. 2012.  Midge: Generating image descriptions from computer vision detections.  In _Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 747–756, Avignon, France. Association for Computational Linguistics.
  * Naveed et al. (2023) Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. 2023.  A comprehensive overview of large language models.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 technical report.  _ArXiv Preprint_ , arXiv:2303.08774.
  * Piloto et al. (2022) Luis S. Piloto, Ari Weinstein, Peter Battaglia, and Matthew Botvinick. 2022.  Intuitive physics learning in a deep-learning model inspired by developmental psychology.  _Nature Human Behaviour_ , 6(9):1257–1267.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning transferable visual models from natural language supervision.  _ArXiv Preprint_ , arXiv:2103.00020.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Rae et al. (2022) Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, …, and Geoffrey Irving. 2022.  Scaling language models: Methods, analysis & insights from training gopher.  _ArXiv Preprint_ , arXiv:2112.11446.
  * Riochet et al. (2022) Ronan Riochet, Mario Ynocente Castro, Mathieu Bernard, Adam Lerer, Rob Fergus, Véronique Izard, and Emmanuel Dupoux. 2022.  Intphys 2019: A benchmark for visual intuitive physics understanding.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 44(9):5016–5025.
  * Scao et al. (2023) BigScience Workshop: Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, …, and Thomas Wolf. 2023.  BLOOM: A 176B-parameter open-access multilingual language model.  _ArXiv Preprint_ , arXiv:2211.05100.
  * Spelke et al. (1992) Elizabeth S. Spelke, Karen Breinlinger, Janet Macomber, and Kristen Jacobson. 1992.  Origins of knowledge.  _Psychological review_ , 99(4):605–632.
  * Spelke et al. (1994) Elizabeth S. Spelke, Gary Katz, Susan E. Purcell, Sheryl M. Ehrlich, and Karen Breinlinger. 1994.  Early knowledge of object motion: continuity and inertia.  _Cognition_ , 51(1):131–176.
  * Su et al. (2023) Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. 2023.  PandaGPT: One model to instruction-follow them all.  _ArXiv Preprint_ , arXiv:2305.16355.
  * Watters et al. (2017) Nicholas Watters, Andrea Tacchetti, Théophane Weber, Razvan Pascanu, Peter Battaglia, and Daniel Zoran. 2017.  Visual interaction networks: Learning a physics simulator from video.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems (NeurIPS)_ , NeurIPS’17, pages 4542–4550, Red Hook, NY, USA. Curran Associates Inc.
  * Wei et al. (2022) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022.  Emergent abilities of large language models.  _ArXiv Preprint_ , arXiv:2206.07682.
  * Weihs et al. (2022) Luca Weihs, Amanda Yuile, Renée Baillargeon, Cynthia Fisher, Gary Marcus, Roozbeh Mottaghi, and Aniruddha Kembhavi. 2022.  Benchmarking progress to infant-level physical reasoning in AI.  _Transactions on Machine Learning Research_.
  * Yi et al. (2020) Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. 2020.  CLEVRER: Collision events for video representation and reasoning.  In _International Conference on Learning Representations (ICLR)_.
  * Zellers et al. (2019) Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019.  From recognition to cognition: Visual commonsense reasoning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Zhang et al. (2023) Hang Zhang, Xin Li, and Lidong Bing. 2023.  Video-llama: An instruction-tuned audio-visual language model for video understanding.  _ArXiv Preprint_ , arXiv:2306.02858.
  * Zhu et al. (2023) Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023.  MiniGPT-4: Enhancing vision-language understanding with advanced large language models.  _ArXiv Preprint_ , arXiv:2304.10592.

##  Appendix A Level 1 Prompts

###  A.1 Binary Questions

##### Shape & Color

Is it true that a …is on the table?

##### Directionality

Is it true that the ball is rolling …in the video?

##### Object Ordering

From left to right, the following objects are on the table: ob1, obj2, …,
objN. Is this true?

##### Relational Position

The ball is on the …side of the black barrier. Is this true?

###  A.2 Open Questions

##### Shape & Color

What color is the object on the table? / Is the object on the table a cube or
a ball?

##### Directionality

Which direction is the ball rolling?

##### Relational Position

Is the ball on the left or right side of the black barrier?

##  Appendix B Level 2 Prompts

The tests are numbered following Appendix F.

##### Test 1

The video you’re seeing was generated by a simulator. Given how objects behave
on earth, is the outcome of the experiment plausible? Your answer should be
based on the events in the video and ignore the quality of the simulation.
Answer only with yes or no.

##### Test 2

The video you’re seeing was generated by a simulator. Given how objects behave
on earth, is the outcome of the experiment plausible? Your answer should be
based on the events in the video and ignore the quality of the simulation.
Answer only with yes or no.

##### Test 3

The video you’re seeing was generated by a simulator. Given how objects behave
on earth, is the final position of the ball plausible? Your answer should be
based on the events in the video and ignore the quality of the simulation.
Answer only with yes or no.

##### Test 4

The video you’re seeing was generated by a simulator. Given how objects behave
on earth, is the final position of the ball plausible? Your answer should be
based on the events in the video and ignore the quality of the simulation.
Answer only with yes or no.

##### Test 5

The video you’re seeing was generated by a simulator. Given how objects behave
on earth, is the location of the cube plausible? Your answer should be based
on the events in the video and ignore the quality of the simulation. Answer
only with yes or no.

##### Test 6

The video you’re seeing was generated by a simulator. Given how objects behave
on earth, is the trajectory of the ball plausible? Your answer should be based
on the events in the video and ignore the quality of the simulation. Answer
only with yes or no.

##### Test 7

The video you’re seeing was generated by a simulator. Given how objects behave
on earth, is the trajectory of the rotating plank plausible? Your answer
should be based on the events in the video and ignore the quality of the
simulation. Answer only with yes or no.

##### Test 8

The video you’re seeing was generated by a simulator. Given how objects behave
on earth, is the trajectory of the ball plausible? Your answer should be based
on the events in the video and ignore the quality of the simulation. Answer
only with yes or no.

##### Test 9

The video you’re seeing was generated by a simulator. Given how objects behave
on earth, is the trajectory of the ball plausible? Your answer should be based
on the events in the video and ignore the quality of the simulation. Answer
only with yes or no.

##### Test 10

The video you’re seeing was generated by a simulator. Given how objects behave
on earth, is the final position of the ball plausible? Your answer should be
based on the events in the video and ignore the quality of the simulation.
Answer only with yes or no.

##### Test 11

The video you’re seeing was generated by a simulator. Given how objects behave
on earth, is the trajectory of the ball plausible? Your answer should be based
on the events in the video and ignore the quality of the simulation. Answer
only with yes or no.

##### Test 12

The video you’re seeing was generated by a simulator. Given how objects behave
on earth, is the final position of the ball plausible? Your answer should be
based on the events in the video and ignore the quality of the simulation.
Answer only with yes or no.

##### Test 13

The video you’re seeing was generated by a simulator. Given how objects behave
on earth, is the final position of the ball plausible? Your answer should be
based on the events in the video and ignore the quality of the simulation.
Answer only with yes or no.

##### Test 14

The video you’re seeing was generated by a simulator. Given how objects behave
on earth, is the interaction between the balls plausible, assuming they are of
the same mass? Your answer should be based on the events in the video and
ignore the quality of the simulation. Answer only with yes or no.

##  Appendix C Test-Specific Randomizations

The tests are numbered following Appendix F.

##### Test 1

  * •

Position of objects

  * •

Number of objects

##### Test 3

  * •

Position of both barriers

  * •

Orientation of the experiment (left-to-right/right-to-left)

##### Test 4

  * •

Random gap size in platform

##### Test 9

  * •

Angle of the plank

  * •

Orientation of the experiment (left-to-right/right-to-left)

##### Test 13

  * •

Orientation of the experiment (left-to-right/right-to-left)

##### Test 14

  * •

Angle of the plank

  * •

Orientation of the experiment (left-to-right/right-to-left)

##  Appendix D Scene Descriptions

The tests are numbered following Appendix F.

##### Test 1

  * •

Plausible: Several objects sit on a flat surface at different positions. The
objects have different shapes and colors. A mechanism moves an occluder such
that the objects cannot be seen for a short while and then the occluder is
removed again. All objects are still in the same position as before and have
the same shape and color.

  * •

Implausible: Several objects sit on a flat surface at different positions. The
objects have different shapes and colors. A mechanism moves an occluder such
that the objects cannot be seen for a short while and then the occluder is
removed again. All objects are still in the same position as before but one
has changed their shape or color.

##### Test 2

  * •

Plausible: A sphere is rolling into the field of view. It rolls from one side
to another, slowing down on the way. There is an occluding object, larger than
the sphere, in the foreground. The sphere rolls past the occluding object.
When the sphere rolls behind that object it disappears from the field of view
and then reappears on the other side of the object. When the sphere reappears
it looks exactly the same as before. The sphere continues to roll until it
reaches a wall where it comes to rest.

  * •

Implausible: A sphere is rolling into the field of view. It rolls from one
side to another, slowing down on the way. There is an occluding object, larger
than the sphere, in the foreground. The sphere rolls past the occluding
object. When the sphere rolls behind that object it disappears from the field
of view and then reappears on the other side of the object. When the sphere
reappears it has a different color. The sphere continues to roll until it
reaches a wall where it comes to rest.

##### Test 3

  * •

Plausible: A sphere is rolling on a flat, solid plank. At the back of the
plank is a wall. The sphere comes into the field of view on one side of the
plank and rolls towards the other side. The occluder is standing at the
frontal edge of the plank, parallel to the wall at the back. It is higher than
the sphere but does not extend all the way to the ceiling. Two walls are
visible above the occluder. The walls have a certain lateral distance from
each other and extend all the way from the front to the back of the plank. The
sphere rolls along the plank until it disappears behind the occluder. After a
short while, the occluder is removed by a mechanism. The removal reveals the
bottom parts of the two walls, which reach all the way to the plank at the
bottom. The sphere has come to rest at the first wall along its trajectory.

  * •

Implausible: A sphere is rolling on a flat, solid plank. At the back of the
plank is a wall. The sphere comes into the field of view on one side of the
plank and rolls towards the other side. The occluder is standing at the
frontal edge of the plank, parallel to the wall at the back. It is higher than
the sphere but does not extend all the way to the ceiling. Two walls are
visible above the occluder. The walls have a certain lateral distance from
each other and extend all the way from the front to the back of the plank. The
sphere rolls along the plank until it disappears behind the occluder. After a
short while, the occluder is removed by a mechanism. The removal reveals the
bottom parts of the two walls, which reach all the way to the plank at the
bottom. The sphere has come to rest at the second wall along its trajectory.

##### Test 4

  * •

Plausible: There is a room enclosed by walls at the sides and at the back. The
room has a lower level and an upper level, defined by a lower floor at the
very bottom and an upper floor that is parallel to the lower floor but a
certain height above it. Both floors are solid and extend all the way to the
wall at the back and the walls at the side. An occluder blocks the field of
view at the front of the room. It starts at the lower floor and extends over
the upper floor. A sphere falls into the room from above and disappears behind
the occluder. The part of the occluder that reaches over the upper floor is
higher and wider than the diameter of the sphere. The occluder is removed by a
mechanism. Both floors are completely visible. The sphere lies on the upper
floor.

  * •

Implausible: There is a room enclosed by walls at the sides and at the back.
The room has a lower level and an upper level, defined by a lower floor at the
very bottom and an upper floor that is parallel to the lower floor but a
certain height above it. Both floors are solid and extend all the way to the
wall at the back and the walls at the side. An occluder blocks the field of
view at the front of the room. It starts at the lower floor and extends over
the upper floor. A sphere falls into the room from above and disappears behind
the occluder. The part of the occluder that reaches over the upper floor is
higher and wider than the diameter of the sphere. The occluder is removed by a
mechanism. Both floors are completely visible. The sphere lies on the lower
floor.

##### Test 5

  * •

Plausible: There is a mechanism moving two occluders in parallel. There is a
little cube on the left side of the two occluders. The cube is smaller than
the occluders. The occluders are moved sideways from right to left until the
left occluder covers the cube. Another mechanism is used to move a pin. The
pin is moved horizontally from left to right behind the occluders. It pushes
out the cube from behind the left occluder to a position between the
occluders. After pushing out the cube, the pin is retracted.

  * •

Implausible: There is a mechanism moving two occluders in parallel. There is a
little cube between the two occluders. The cube is smaller than the occluders.
The occluders are moved sideways from right to left until the right occluder
reaches and then covers the cube. Another mechanism is used to move a pin. The
pin is moved horizontally from left to right behind the occluders. It pushes
out the cube from behind the left occluder to a position between the two
occluders. After pushing out the cube, the pin is retracted.

##### Test 6

  * •

Plausible: A sphere is rolling from one side to another, passing behind an
occluder on the way. In the middle of the surface is an occluder that is
higher on the left and the right side and lower in the middle. At the low part
in the middle, the height of the occluder exceeds the diameter of the sphere.
The sphere disappears behind the occluder on one side and reappears on the
other side. While it is behind the occluder, it remains entirely out of sight.
The scene is viewed right from the front, with a line of sight exactly at the
top of the lower part of the occluder.

  * •

Implausible: A sphere is rolling from one side to another, passing behind an
occluder on the way. In the middle of the surface is an occluder that is
higher on the left and the right side and lower in the middle. At the low part
in the middle, the height of the occluder is less than the diameter of the
sphere. The sphere disappears behind the occluder on one side and reappears on
the other side. While it is behind the occluder, it remains entirely out of
sight. The scene is viewed right from the front, with a line of sight exactly
at the top of the lower part of the occluder.

##### Test 7

  * •

Plausible: A solid object is standing on a solid and flat surface. A certain
mechanism is used to move a plank in a rotating motion. The plank starts out
flat on the surface in front of the object. It is rotated upwards until it
stands vertically on the surface and occludes the object, and rotated further
until it is above the object. Its movement stops and then it is rotated back
until it lies flat in its initial position again. While the plank is rotating
back the object is revealed in its initial position.

  * •

Implausible: A solid object is standing on a solid and flat surface. A certain
mechanism is used to move a plank in a rotating motion. The plank starts out
flat on the surface in front of the object. It is rotated upwards until it
stands vertically on the surface, and rotated further until it lies flat on
the surface again on the other side, where the object was standing before. Its
movement stops and then it is rotated back until it lies flat in its initial
position again. While the plank is rotating back the object is revealed in its
initial position.

##### Test 8

  * •

Plausible: A sphere is rolling onto a flat plane that is enclosed by a
straight and rigid wall on the other side. There is a white line that runs
through the middle of the plane. The sphere rolls onto the plane with a
lateral offset from the line. It rolls diagonally towards the wall and reaches
the wall exactly at the white line in the middle. When it reaches the wall it
bounces off and rolls back on the other side of the white line. The angle
between the sphere’s trajectory and the white line is the same as on the way
towards the wall. When the sphere disappears from view at the back of the
plane, it is now on the opposite side of the white line compared to when it
entered the plane, while the lateral offset is the same.

  * •

Implausible: A sphere is rolling onto a flat plane that is enclosed by a
straight and rigid wall on the other side. There is a white line that runs
through the middle of the plane. The sphere rolls onto the plane with a
lateral offset from the line. It rolls diagonally towards the wall and reaches
the wall exactly at the white line in the middle. When it reaches the wall it
bounces off and rolls back on the same side of the white line. The angle
between the sphere’s trajectory and the white line is the same as on the way
towards the wall. When the sphere disappears from view at the back of the
plane, it is still on the same side of the white line compared to when it
entered the plane and the lateral offset is the same.

##### Test 9

  * •

Plausible: There is a diagonally lying plank. A sphere rolls down the plank
from one side to the other without slowing down.

  * •

Implausible: There is a diagonally lying plank. A sphere rolls up the plank
from one side to the other without slowing down.

##### Test 10

  * •

Plausible: A sphere is falling down into the field of view. The sphere is
falling towards a solid, flat surface. It falls until it disappears behind an
occluder. When the occluder is removed, the sphere is lying behind it on the
surface.

  * •

Implausible: A sphere is falling down into the field of view. The sphere is
falling towards a solid, flat surface. It falls until it disappears behind an
occluder. When the occluder is removed, the sphere is hovering above the
surface behind it.

##### Test 11

  * •

Plausible: A sphere is rolling sideways on a flat solid surface until it
reaches the edge of that surface. It drops down continuing to move in the same
sideways direction while it is falling. When it hits the ground it continues
rolling in that direction.

  * •

Implausible: A sphere is rolling sideways on a flat solid surface until it
reaches the edge of that surface. It drops down exactly vertically along the
edge and stops moving upon reaching the ground.

##### Test 12

  * •

Plausible: A sphere is rolling onto a flat plane from one side. The plane is
enclosed by rigid walls to the left, right, and front of the sphere (in a
U-shape). The end of the plane, where the sphere is rolling towards, is
covered. The sphere rolls towards one of the corners, slowing down, until it
disappears under the cover. The cover is removed automatically by an inbuilt
mechanism. The removal of the cover reveals the frontal wall and the two
corners. The sphere has come to rest in the corner it was rolling towards. The
other corner is enclosed by an additional plank that runs diagonally from the
wall at the front to the wall on the side, such that the walls and the plank
form a triangle. There is no object inside the triangle.

  * •

Implausible: A sphere is rolling onto a flat plane from one side. The plane is
enclosed by rigid walls to the left, right, and front of the sphere (in a
U-shape). The end of the plane, where the sphere is rolling towards, is
covered. The sphere rolls towards one of the corners, slowing down, until it
disappears under the cover. The cover is removed automatically by an inbuilt
mechanism. The removal of the cover reveals the frontal wall and the two
corners. The corner on the other side of the sphere is empty. The corner that
the sphere was rolling towards is enclosed by an additional plank that runs
diagonally from the wall at the front to the wall on the side, such that the
walls and the plank form a triangle. The sphere has come to rest inside that
triangle.

##### Test 13

  * •

Plausible: A sphere is rolling slowly on a flat, solid plank. Below that plank
is another plank that runs in parallel. The sphere comes into the field of
view from one side and rolls towards the other side until it disappears behind
an occluder. A mechanism removes the occluder. The removal reveals a gap in
the upper plank on which the sphere was rolling. The gap is wider than the
diameter of the sphere. The sphere is now on the lower plank and is at rest.

  * •

Implausible: A sphere is rolling slowly on a flat, solid plank. Below that
surface is another plank that runs in parallel. The sphere comes into the
field of view from one side and rolls towards the other side until it
disappears behind an occluder. A mechanism removes the occluder. The removal
reveals a gap in the upper plank on which the sphere was rolling. The gap is
wider than the diameter of the sphere. The sphere is now on the other side of
the gap, still on the upper plank.

##### Test 14

  * •

Plausible: A sphere is rolling down an incline. At the bottom of the incline,
directly in the trajectory of the rolling sphere is another sphere of the same
size and weight as the moving sphere. The rolling sphere is relatively fast.
When it collides with the sphere at the bottom, it pushes that sphere away in
the direction of movement. Both spheres continue to roll in that direction
until they reach a wall. The first sphere (that pushed the second sphere)
comes to rest right behind the sphere that it pushed. The sphere that was
pushed comes to rest at the wall.

  * •

Implausible: A sphere is rolling down an incline. At the bottom of the
incline, directly in the trajectory of the rolling sphere is another sphere of
the same size and weight as the moving sphere. The rolling sphere is
relatively fast. When it collides with the sphere at the bottom, the sphere at
the bottom remains at its exact original location. The sphere that was rolling
comes to rest next to the sphere at the bottom.

##  Appendix E Model Parameters

Generation parameters for each model are left at their default values.
Responses are generated with num_beams = 1 and temperature = 1.0.

Where available, system prompts were modified from their defaults so as not to
clash with the instruction to give one-word replies:

##### Video-ChatGPT

"You are Video-ChatGPT, a large vision-language assistant. You are able to
understand the video content that the user provides, and answer their
questions correctly. Follow the instructions carefully and give short answers
based on the provided video."

##### Video-LLaMA

"You will be able to see a video once I provide it to you. Please answer my
questions."

##  Appendix F Intuitive Physics Tests

Figure 3: Level 2 (Intuitive Physics) tests: The first and second rows display
plausible and implausible versions respectively for each experiment. The tests
assess the understanding of (1) unchangeableness, (2) unchangeableness, (3)
solidity & continuity, (4) solidity & continuity. Figure 4: Level 2 (Intuitive
Physics) tests: The first and second rows display plausible and implausible
versions respectively for each experiment. The tests assess the understanding
of (5) object permanence, (6) object permanence, (7) object permanence, (8)
inertia. Figure 5: Level 2 (Intuitive Physics) tests: The first and second
rows display plausible and implausible versions respectively for each
experiment. The tests assess the understanding of (9) gravity, (10) gravity &
inertia, (11) gravity & inertia, (12) continuity. Figure 6: Level 2 (Intuitive
Physics) tests: The first and second rows display plausible and implausible
versions respectively for each experiment. The tests assess the understanding
of (9) gravity & continuity, (10) collisions.