†† * Authors contributed equally, 1 University of California, Berkeley 2
Deepmind, Corresponding authors<EMAIL_ADDRESS><EMAIL_ADDRESS>
# Towards Understanding How Machines Can Learn Causal Overhypotheses

Eliza Kosoy*,1, David M. Chan*,1, Adrian Liu1, Jasmine Collins1, Bryanna
Kaufmann1, &Sandy Han Huang2, Jessica B. Hamrick2, John Canny1, Nan Rosemary
Ke2, Alison Gopnik1

###### Abstract

Recent work in machine learning and cognitive science has suggested that
understanding causal information is essential to the development of
intelligence. The extensive literature in cognitive science using the “blicket
detector” environment shows that children are adept at many kinds of causal
inference and learning. We propose to adapt that environment for machine
learning agents. One of the key challenges for current machine learning
algorithms is modeling and understanding causal overhypotheses: transferable
abstract hypotheses about sets of causal relationships. In contrast, even
young children spontaneously learn and use causal overhypotheses. In this
work, we present a new benchmark—a flexible environment which allows for the
evaluation of existing techniques under variable causal overhypotheses—and
demonstrate that many existing state-of-the-art methods have trouble
generalizing in this environment. The code and resources for this benchmark
are available at https://github.com/CannyLab/casual_overhypotheses.

##  1 Introduction

Over the past few years, deep learning has made impressive progress in many
areas, including reinforcement learning, natural language processing, and
computer vision. However, most state-of-the-art algorithms still pale in
comparison to humans, even children, for out-of-distribution generalization
and fast adaptation to new tasks. In contrast, causal modeling has long
concerned itself not merely with accurate modeling of in-distribution data,
but also the accurate recovery of underlying causal mechanisms (and their true
graphical relations) capable of explaining out-of-distribution data. Hence
causal modeling holds promise for achieving systematic generalization (Bengio
et al., 2019; Schölkopf et al., 2021; Ke et al., 2021).

Causal overhypotheses are a key component of causal learning—they enable
humans to learn causal models from a sparse amount of data (Griffiths and
Tenenbaum, 2009), because they reduce the set of possible causal relationships
to consider. Despite the recent surge of machine learning datasets and
environment for causal inference and learning (Ahmed et al., 2020; McDuff et
al., 2021; Wang et al., 2021a; Ke et al., 2021), the causal overhypotheses for
these environments and datasets are unclear. In this work, we evaluate
existing machine learning algorithms for causal overhypothesis learning.
Instead of proposing a new environment for learning causal overhypotheses, we
draw inspiration from recent cognitive science development, and use a virtual
blicket detector environment, which has been used for understanding causal
overhypotheses learning in children (Kosoy et al., 2022).

A “blicket detector” is a machine that lights up and plays music when some
combinations of objects but not others are placed on it (Gopnik and Sobel,
2000; Lucas et al., 2014). The central question is whether an agent can learn
that a particular set of causal events will lead to the lighting-up effect,
and use that knowledge to design novel interventions on the machine. The
causal relationship is entirely determined by the pattern of conditional
dependencies and interventions, rather than requiring intuitive physics
knowledge or visual understanding. Although these tasks may seem simple, and
are easily mastered by children, they are challenging for current learning
algorithms, as we will see later.

Several features of this environment and the tasks it allows make it
particularly useful as a benchmark for machine learning algorithms. First,
causal representations are more powerful and structured than mere statistical
generalizations, though both can be systematically inferred from statistical
information. Many researchers have argued that such causal representations are
crucial for both human and general AI intelligence. The goals of causal
inference, namely learning which actions will alter the environment in
particular ways, are similar to those of standard reinforcement learning, but
adding causal representations and inferences makes such learning far more
effective.

Second, and unlike some existing causal environments (Ke et al., 2021; Wang et
al., 2021a), the blicket environment enables the inference of both specific
causal structure and more general features of causal structure, such as
whether causal systems are conjunctive or disjunctive, stochastic or
deterministic. Learning these “overhypotheses” about causal structure
(Griffiths and Tenenbaum, 2009) is especially important because such
inferences can constrain the search for causal structure in the future, a
search that can rapidly become unwieldy. Third, this environment allows for
active learning and exploration in a way that is both sufficiently constrained
to be practical and that also allows for informative interventions.

Most significantly, and again unlike existing environments, research has
already shown that even preschool children can easily manipulate and explore
this environment, generate appropriate data, and rapidly learn both particular
causal structure and overhypotheses about causal structure (Kosoy et al.,
2022). We can then directly compare both the overall performance and the
behavior of various agents in these tasks to the performance and behavior of
children. Young children are a particularly informative baseline group. They
do not have the extensive education and experience of typical adults, which
might make comparisons to artificial agents challenging, but they are
nevertheless effective causal learners and able to make broad yet accurate
generalizations from small sample sizes, in contrast to many current machine
learning systems (Gopnik, 2012; Gopnik et al., 2017).

In what follows, we look at a spectrum of algorithms ranging from
reinforcement learning, to imitation learning, to the use of pre-trained
language models, to better understand how these different methods perform on
the proposed environment and tasks. We find that such algorithms, in contrast
to children, only converge on a solution after an extensive number of trials
or if they are given all the possible settings and outcomes as training data.
This suggests that these tasks are an interesting challenge for machine
learning algorithms. In order for machines to perform as well as children do,
algorithms must reason about the sequence of observations seen, extract causal
overhypotheses from those observations and use them for exploration—which
current methods fall short of doing.

##  2 Related Work

Our work relates to prior work on exploration in reinforcement learning (RL),
multi-task RL, causal learning in RL, and large language models.

#### Exploration in Reinforcement Learning

Recent exploration algorithms for deep reinforcement learning typically add an
exploration bonus to the task reward; please refer to Amin et al. (2021) for a
comprehensive survey. This bonus could be based on novelty (Bellemare et al.,
2016; Ostrovski et al., 2017; Martin et al., 2017; Tang et al., 2017; Machado
et al., 2018a), dynamics prediction error (Schmidhuber, 1991; Pathak et al.,
2017), uncertainty (Osband et al., 2016; Burda et al., 2018), or disagreement
(Pathak et al., 2019). The vast majority of existing exploration methods do
not involve causal reasoning, whereas our proposed environment requires
algorithms to learn and use causal overhypotheses in order to explore
effectively, to solve the task.

#### Multi-task and Causal RL Benchmarks

There exist multi-task RL benchmarks featuring robotics (Yu et al., 2019;
James et al., 2020), physical reasoning (Bakhtin et al., 2019; Allen et al.,
2020), and video games (Cobbe et al., 2018; Machado et al., 2018b; Nichol et
al., 2018; Chevalier-Boisvert et al., 2018). Unfortunately, since it is not
clear what the relevant causal overhypotheses for these environments are, it
is difficult to evaluate how causal information influences agents’
exploration.

RL benchmarks for causal discovery include Causal World (Ahmed et al., 2020),
Causal City (McDuff et al., 2021), Alchemy (Wang et al., 2021a), ACRE (Zhang
et al., 2021), and the work of Ke et al. (2021). However, many of these
environments either lack clear causal hypotheses or do not allow for
controlling overhypotheses. In addition, these environments primarily focus on
causal induction or generalization, rather than exploration (though see
Sontakke et al., 2021). In contrast, the blicket environment in this work is
designed to measure agents’ ability to explore using causal overhypotheses.
Moreover, children have not been tested on any of these existing environments,
whereas in the blicket environment, prior work has shown that children as
young as age four are able to learn causal overhypotheses and use these to
explore effectively (Kosoy et al., 2022). It can be informative to compare the
exploration and performance of RL approaches to that of children.

#### Language models for reasoning tasks

Large language models such as GPT (Radford et al., 2019; Brown et al., 2020)
and PALM (Chowdhery et al., 2022) are trained on massive amounts of data, and
they have been shown to be able to express uncertainty and perform common
sense reasoning up to an extent (Lin et al., 2022). In this work, we probe the
the causal reasoning capabilities of GPT-3 and PALM using textual descriptions
of the virtual blicket environment.

##  3 The “Blicket Detector” Environment

Figure 1: A screenshot of the embodied version of the blicket detector.

The blicket detector is commonly used in cognitive science to study causal
inference and reasoning in children (Gopnik and Sobel, 2000; Lucas et al.,
2014). Typically these studies are conducted in person. Recently Kosoy et al.
(2022) proposed an open-sourced virtual blicket detector environment (Figure
1), focused on studying how children learn and use causal overhypotheses in
exploration. This virtual environment has several advantages: it enables not
only tracking interactions with children, but also an exact comparison between
children and machine learning agents via the OpenAI Gym interface (Brockman et
al., 2016a). Please refer to Appendix A for examples of the virtual blicket
detector in various states.

#### Causal Overhypotheses

The causal structure of this environment is hierarchical: the higher level
specifies the causal relationship, and the lower level specifies which objects
are blickets. Following Kosoy et al. (2022), we consider two causal
relationships—disjunctive and conjunctive—which require that at least one or
at least two blickets are placed on the detector in order for it to light up,
respectively. More concretely, suppose there are three objects, $A$, $B$, and
$C$, where $A$ and $B$ are blickets. In the conjunctive case, the detector
would only light up if $AB$ or $ABC$ were placed. In the disjunctive case, any
combination of objects except for just $C$ would cause the detector to light
up.

Figure 2: A simplified representation of the examples that children see in the
_given hypothesis_ training phase.

#### Prior Experiments with Children

Since we would like to compare the performance of machine learning agents to
that of children in this environment, we aim to make our evaluation of machine
learning algorithms comparable to how children are evaluated in Kosoy et al.
(2022). Their evaluation used a $2\times 2$ experimental design, with a _given
hypotheses_ or _not given hypotheses_ training phase, combined with a
_disjunctive_ or _conjunctive_ causal relationship in the test phase. In the
training phase, they showed the child two different blicket detectors, with
three unique objects each, and for each detector demonstrated three examples
of placing objects on the detector and checking whether it lights up. In the
_given hypotheses_ case, one could infer from the examples that one detector
was conjunctive and the other was disjunctive, as shown in Figure 2. Whereas
in the _not given hypotheses_ case, one could not infer from the examples
whether the detectors were conjuctive or disjunctive. In the test phase, the
children were given the opportunity to freely interact with a third blicket
detector with three new objects. When they were done exploring, they were
asked to respond yes or no to which objects were blickets. In their
experiments with children from ages four to six, Kosoy et al. (2022) found
that children exhibit a diverse range of exploration strategies, which
suggests that they are exploring based on a rich set of causal overhypotheses,
formed from their prior knowledge of how objects and mechanisms behave.

#### Our Experiments with Models

To adapt the blicket environment for agent learning in this work, we made the
following design choices regarding the observations, actions, reward, and
termination conditions.

  * •

Observations: We could allow the algorithms to observe the same environment as
the children, but this places RL algorithms at a significant disadvantage,
since they would need to not only understand causal structures, but also learn
visual inputs and object detection. Thus, we choose to evaluate the algorithms
in a purely symbolic environment where the objects are represented by one-hot
indices.

  * •

Actions: In the experiments with children, actions consist of placing a
blicket onto the detector, removing a blicket from the detector, and pressing
the “check” button to evaluate the detector’s state. For RL algorithms, we
simplify this process by allowing the agent to place multiple objects
simultaneously, automatically “checking” the state of the detector with every
action, and automatically resetting the detector after each check. This means
that the agent gets feedback with every action, which significantly improved
training stability.

  * •

Reward: The reward function should capture whether the algorithm has learned
the causal overhypothesis of an environment. To do this, we evaluate the
models using a quiz-based framework. Models are allowed to make as many
exploration steps as needed, and then trigger an action which switches to the
evaluation mode. In the evaluation mode, models receive as input a blicket and
must produce an action indicating if the object is a blicket or not. They
receive a reward of $1$ for identifying a correct blicket, and a reward of
$-1$ for incorrectly labeling an object (i.e., for both false positives and
false negatives). We also explored several reward modifications. In one
modification, models were asked to disambiguate between Disjunctive and
Conjunctive environments, with +1 reward for identifying the correct modality.

The environment is implemented using the standard OpenAI Gym (Brockman et al.,
2016b) interface, allowing it to be used across many different pre-existing
machine learning architectures and algorithms. We hope that using this
standard interface allows for expanded research in this field of causal
overhypothesis learning.

##  4 Benchmark Causal Learning in the Blicket Environment

Results from Kosoy et al. (2020) suggest that children can explore
efficiently, especially given the causal overhypotheses. In this work, we
evaluate how a spectrum of different machine learning models perform on the
blicket detector tasks. Solving these tasks requires reasoning about the
sequence of observations seen, extracting causal overhypotheses from those
observations, and using these extracted overhypotheses for exploration.

We first evaluate several popular reinforcement learning algorithms—A2C (Mnih
et al., 2016), PPO2 (Schulman et al., 2017), and Q-learning (Watkins and
Dayan, 1992)—on this task. We also evaluate imitation learning algorithms,
including behaviour cloning with decision transformers (Chen et al., 2021).
Finally, we apply pre-trained language models (Brown et al., 2020; Chowdhery
et al., 2022) to this task, since they have been shown to be capable of
performing common-sense reasoning and expressing uncertainty to an extent (Lin
et al., 2022).

###  4.1 Deep Reinforcement Learning Algorithms

We evaluate the performance of two pupular deep reinforcement learning
algorithms, Advantage Actor Criric (A2C) (Mnih et al., 2016) and Proximal
Policy Optimization (PPO2) (Schulman et al., 2017), on the blicket
environment. For each algorithm, we use several policy variants: a standard
MLP policy (no memory), an LSTM-based policy, and an LSTM-based policy with
layer normalization with two hidden dimension variants of 256 and 512. For all
of these policies, we found that a network with a hidden size of 512 obtained
the optimal performance See Appendix B for learning hyperparameters. We train
all of these algorithms in the given hypothesis scenario, where the agent is
exposed during training to all possible overhypotheses, and asked to perform
well given these scenarios.

#### Experimental Design

We terminate training either after 3 million environment steps or when the
agent obtains maximum reward for 500 consecutive episodes, whichever comes
first. Each episode has 25 timesteps, so each agent is exposed to at most
120,000 episodes. Experiments are run using a single Nvidia Titan-X Maxwell
GPU, and take less than three hours to complete. To evaluate whether the agent
can generalize to additional causal situations, we also train agents on five
held-out scenarios: holding out all of the conjunctive overhypotheses, holding
out all of the disjunctive overhypotheses, and holding out either one
conjunctive overhypothesis or one disjunctive overhypothesis.

#### Results

3(a) shows the performance of A2C and PPO2 when none of the hypotheses are
held out. PPO2 outperforms A2C in almost all scenarios, achieving higher
rewards faster. Further, the LSTM models clearly outperform the non memory-
based models; this is expected, since causal learning requires memory.
Unfortunately, none of the algorithms perform well on held-out causal
examples, as shown in 3(c). This suggests that they are primarily learning to
memorize the causal patterns, and thus are incapable of generalization. This
conclusion comes with a caveat: because there are only a handful of possible
hypotheses, it may be possible that we do not have enough data to perform well
on held-out samples. Agents are, however, able to easily distinguish between
conjunctive and disjunctive environments—3(d) shows that after very few steps,
in the held-out situation (with both overhypotheses), agents can distinguish
conjunctive from disjunctive environments, even though they are unable to
determine which objects exactly are blickets.

(a) Performance of PPO2 and A2C algorithms with MLP, LSTM, and Layer-Norm LSTM
policies on the blicket environment.

(b) Performance when models are forced to explore for K steps before entering
the quiz phase.

(c) Performance on several over-hypothesis variants. Conjunctive Only: Train
on only conjunctive hypotheses, test on both conjunctive and disjunctive.
Disjunctive Only: Train on only disjunctive hypotheses, test on both
conjunctive and disjunctive. Leave one out modifier: Train on 2/3 (or 4/6 in
the case of both) of the hypothesis within the overhypothesis, and test on the
remaining hypotheses.

(d) Accuracy of models over time when determining if the environment is
conjunctive or disjunctive.

Figure 3: Experiments with standard deep reinforcement learning algorithms

###  4.2 Q-Learning

We also train tabular Q-Learning on the symbolic version of the blicket
environment. We append the full history of previous observations to the
current observation in order to give the agent memory. The Q-values are
initialized to zero. We used $\epsilon$-greedy exploration with an exploration
probability of $0.1$, and we found the best learning rate was $0.95$.
Q-learning is able to learn the task very quickly due to a small search
space—it took an average of $70$ episodes and $292$ steps for the agent to
converge to maximum reward. However, tabular Q-learning is incapable of
generalizing to new scenarios, so we do not test Q-Learning agents in the
held-out scenarios.

###  4.3 Behavior Cloning

Recently Chen et al. (2021) introduced the Decision Transformer, a simple
transformer-based approach to imitation learning, shown to outperform most
existing behavior cloning methods. The decision transformer works by applying
a causally-masked transformer to predict the reward-to-go of a flattened
sequence of (state, action, reward, next state) tuples, with an
$\ell^{2}$-norm loss. The model then chooses actions during test time that
maximize the predicted reward-to-go.

#### Experimental Design

In this work, we adapt the model to predict discrete actions in our space by
adding a sigmoid activation to the action predictions and altering the action
prediction loss accordingly. We evaluate the model using a target reward of 3,
which corresponds to identifying all of the blickets correctly. The decision
transformer was trained using a batch size of 128, a K-value of 30, embedding
dimension of 128, 3 layers, one head, and dropout of 0.1. The weights are
optimized using ADAM, with a learning rate of $1e^{-4}$ and weight decay of
$1e^{-4}$. We also compare against standard behavior cloning, using an MLP
with a hidden-dimension of 128, and trained using the same optimizer as the
decision transformer. Both models are trained for 10 epochs using one Nvidia
Titan-X Maxwell GPU, which takes less than one hour, and the best validation
set checkpoint is used for each during test time. Models are evaluated on 100
rollouts in the environment, and the mean reward is reported.

When training the decision transformer model, we found that in some cases,
random (and even the PPO2/A2C trained) models were unable to explore
efficiently, as they entered the quiz environment too soon. Note that random
exploration will enter the quiz environment after $t$ steps with with
probability $p=1-\frac{1}{2^{t}}$. Thus, we found it helpful to force the
policy to explore for several steps before allowing it to enter the quiz
environment. While this process helps the decision transformer, it negatively
affects the performance of the standard RL models, as shown in 3(b).

#### Results

Table 1 shows the performance of the decision transformer and behavior cloning
models when applied to datasets generated by several policies. As we can see,
while expert policies allow for higher rewards using the decision transformer,
allowing the model additional forced exploration time is the most important
factor. This suggests that on their own, the A2C and PPO2 trained policies do
not lead to sufficient exploration for learning a strong model of the reward,
whereas forcing additional exploration (even if it is random) is much more
useful. Notably, standard behavior cloning performs very poorly, as copying
actions with the same local observations under different overhypotheses will
likely lead to incorrect or uninformative actions.

Model |  Pre-Training Dataset | Reward | FCA
---|---|---|---
Decision Transformer |  Random | $0.16\pm 1.747$ | $0.67$
Decision Transformer |  Random (5-step forced exploration) | $2.06\pm 0.998$ | $0.71$
Decision Transformer |  Random (10-step forced exploration) | $2.16\pm 0.872$ | $0.73$
Decision Transformer |  Random (15-step forced exploration) | $2.37\pm 0.633$ | $0.77$
Decision Transformer |  A2C-LSTM | $0.14\pm 1.847$ | $0.63$
Decision Transformer |  PPO2-LSTM | $0.26\pm 0.990$ | $0.66$
Decision Transformer |  A2C-LayerNorm-LSTM | $0.18\pm 0.983$ | $0.62$
Decision Transformer |  PPO2-LayerNorm-LSTM | $0.74\pm 1.460$ | $0.64$
Behavior Cloning |  Random | $0.22\pm 0.975$ | $0.61$
Behavior Cloning |  Random (5-step forced exploration) | $-0.16\pm 0.987$ | $0.58$
Behavior Cloning |  Random (10-step forced exploration) | $0.14\pm 0.990$ | $0.60$
Behavior Cloning |  Random (15-step forced exploration) | $0.04\pm 0.999$ | $0.61$
Behavior Cloning |  A2C-LSTM | $0.10\pm 0.995$ | $0.55$
Behavior Cloning |  PPO2-LSTM | $0.06\pm 0.998$ | $0.63$
Behavior Cloning |  A2C-LayerNorm-LSTM | $0.04\pm 2.087$ | $0.59$
Behavior Cloning |  PPO2-LayerNorm-LSTM | $0.12\pm 0.995$ | $0.68$
Table 1: Performance of the imitation learning models with different pre-
training datasets. Reward is on the blicket-quiz task: +1 for correctly
identifying a blicket, and -1 for incorrectly identifying a blicket. FCA
refers to Forced Choice Accuracy, the accuracy of the model in determining if
the environment is conjunctive or disjunctive.

###  4.4 Large Language Models

Recently large language models have shown promising performance on a wide
variety of tasks, including logical inference and common-sense reasoning. Such
language models include GPT-3 (Brown et al., 2020), PaLM (Chowdhery et al.,
2022), MT-NLG (Smith et al., 2022), and Chinchilla (Hoffmann et al., 2022),
which all use transformer-based architectures and are trained autoregressively
on a large corpus of text, to predict the next token when given a sequence of
tokens. These language models have also shown promising performance on tasks
that involve causal reasoning, such as identifying causal effects from
observational data (Veitch et al., 2020) and inferring causality for when
drugs lead to adverse effects (Wang et al., 2021b) However, these models have
not yet been applied to a task that involves reasoning across a set of
overhypotheses, as is required in the blicket environment.

Motivated by this, here we investigate the performance of two such language
models, GPT-3 and PaLM, on the blicket environment. One can apply a language
model to a target task via purely text-based interaction with the model, by
providing a prompt to the model and evaluating its output. This prompt could
be _freeform_ , in which it has no set structure, or it could be _few-shot_ ,
where it contains a small number of examples of correct prompt-output pairs,
followed by the test prompt. Instead of text-based interaction, another option
is to fine-tune the model on the target task. This involves training the model
on data from the target task, and can require a sizeable amount of data. To
make the evaluation more similar to how children are introduced to the task,
we focus on only text-based interaction with the model, including both
freeform and few-shot prompts.

As in the experiments with children, there are four conditions: _given_ versus
_not given hypotheses_ , and _disjunctive_ versus _conjunctive_. In the
_disjunctive_ case, one of the three objects is a blicket. In the
_conjunctive_ case, two of the three objects are blickets. We evaluate how
well the models can: 1) identify the blickets and 2) identify whether the
causal structure is disjunctive or conjuctive. For GPT-3, we use the OpenAI
API111Available at https://beta.openai.com/overview, and for PaLM we use an
internally available interface.

#### Freeform Prompt

In the freeform prompt, we provide the model with text that is as similar as
possible to what children receive in the blicket experiment. The only
difference is that we replace the visual components with text descriptions. We
include the exact prompts in Appendix C. The prompt starts with a description
of interacting with a striped blicket machine and then a dotted blicket
machine, each of which has three unique objects. There are three interaction
examples given per machine, for example “If we put the blue pyramid on the
machine, then it does not light up”. In the _given hypotheses_ condition, the
striped blicket machine is conjunctive and the dotted blicket machine is
disjunctive. In the _not given hypotheses_ condition, there is not enough
information to determine whether the striped and dotted blicket machines are
disjunctive or conjuctive. Thus the _given hypotheses_ condition defines the
space of overhypotheses, whereas the _not given hypotheses_ condition does
not. The prompt then introduces a new blicket machine, along with examples of
interactions with the machine and whether it lights up or not. Finally we ask
the model which objects are blickets and whether the new machine is more
similar to the striped or dotted machine.

#### Few-shot Prompt

In the few-shot prompt, we structure the input into two prompt-output
demonstrations, one per machine, containing the same information as is
provided in the freeform prompt. Unlike the information given to children, in
the outputs we explicitly state which objects are blickets. In addition, for
the _given hypotheses_ condition, in the preamble we define the disjunctive
and conjunctive hypotheses: “A striped machine needs two blickets to make it
light up, and a dotted machine needs one blicket to make it light up”, and in
the outputs we state whether this is more similar to a striped or dotted
machine. We need to structure the output like this because the output of the
model will have the same structure.

The results are reported in Table 2.222For GPT-3, we use the text-davinci-002
model, with temperature 0.7, maximum length 256, and frequency and presence
penalties 0. For PaLM, we use a temperature of 0 to obtain the greedy, or one-
best decoding. When given the freeform input, the PaLM model output continues
indefinitely, so we simply took the first two sentences of the output and
ignored the rest. We find that when given hypotheses, GPT-3 and PaLM are
almost always able to select the correct causal structure, but they are not
always able to select the correct blickets. In particular, GPT-3 frequently
names too many objects as blickets, including those associated with other
machines. In contrast, PaLM never identified objects as blickets that were not
one of the three objects in the test task.

In the two-shot setting, PaLM performs best when the space of overhypotheses
is covered perfectly by the two examples given, as one would expect. However,
when this space is not covered, i.e. in the _not given hypothesis_ setting,
PaLM struggles in the two-shot setting because the test example’s causal
structure does not match either of the two given examples. We tried adding
chain-of-thought reasoning in the prompt for PaLM, but were not able to
improve results for either the freeform or two-shot setting.

Condition |  Model & Input | Blickets Chosen | Causal Structure
---|---|---|---
Given hypotheses, disjunctive |  GPT-3, freeform | 1/1 correct, 6 wrong | correct
|  PaLM, freeform | 1/1 correct, 1 wrong | wrong
|  PaLM, two-shot | 1/1 correct | correct
Given hypotheses, conjunctive |  GPT-3, freeform | 2/2 correct, 1 wrong | correct
|  PaLM, freeform | 2/2 correct | correct
|  PaLM, two-shot | 2/2 correct | correct
Not given hypotheses, disjunctive |  GPT-3, freeform | 1/1 correct, 7 wrong | —
|  PaLM, freeform | 0/1 correct | —
|  PaLM, two-shot | 1/2 correct, 1 wrong | —
Not given hypotheses, conjunctive |  GPT-3, freeform | 2/2 correct, 7 wrong | —
|  PaLM, freeform | 2/2 correct | —
|  PaLM, two-shot | 1/2 correct, 1 wrong | —
Table 2: Results for GPT-3 and PaLM for the four conditions. Bold font
indicates a fully correct answer. For the _not given hypotheses_ setting,
there is not a correct answer possible for causal structure, because there is
not enough information to determine whether the blicket detectors in the
training phase are disjunctive or conjunctive.

##  5 Discussion & Conclusion

In this work, we looked into evaluating and understanding how machine learning
models learn causal overhypotheses, by evaluating these models in the blicket
environment. In contrast to existing benchmark tasks, in which there is a
fixed causal structure, this environment focuses on the need for _causal
overhypotheses_ in order to explore effectively to determine the underlying
causal structure. We focused on three categories of state-of-the-art
methods—deep RL, behavior cloning, and large language models—for tasks in this
environment. In blicket detector experiments, children are able to learn
causal overhypotheses from only a handful of observations and can apply these
overhypotheses to explore effectively for a new situation (Kosoy et al.,
2022). In contrast, our experiments indicate that state-of-the-art machine
learning algorithms have difficulty learning and using causal overhypotheses
for exploration and inference—we saw this in the weak performance of deep RL
algorithms on held-out environments and in the tendency of decision
transformer models to under-explore. In our experiments with language models,
we provide the same observations that the children were given in Kosoy et al.
(2022), along with a full set of examples for the new situation (thus removing
the need for exploration). Despite this, language models struggle when the
hypotheses are not given, and are not able to express uncertainty about the
causal structure in that case.

Given that understanding and leveraging causal structure is essential to
developing general intelligence, this work highlights an opportunity for
improvement in this direction, and provides a set of concrete benchmark tasks
to measure improvement. One direction of future work is to build machine
learning models that can better learning causal overhypotheses. Modular
architectures have shown to be helpful in understanding causal hypotheses of
the environment (Goyal et al., 2019, 2021; Ke et al., 2021); it would be
promising to explore such models for causal overhypotheses understanding.
Another direction of future work is to improve on exploration in RL agents by
explicitly learning and incorporating causal overhypotheses, in order to
narrow down the search over possibilities. Furthermore, an interesting
direction is to train machine learning models on children’s exploration
trajectories from Kosoy et al. (2020). This may lead to new insights for
improving causal overhypotheses learning for machine learning models. A main
limitation of this work is that we evaluated only two types of causal
overhypotheses–conjunctive and disjunctive. These are particular types of
overhypotheses on the causal graph structure. In the real world, there exist
many other types of overhpotheses, as well as conditional probability
distributions. It would be interesting to see if one can extend either the
blicket or other RL environments (such as Ahmed et al. (2020); Ke et al.
(2021)) to include other types of causal overpotheses.

##  6 Acknowledgements

The authors would like to acknowledge the support of the following agencies
for research funding and computing support: DeepMind, The DARPA Machine Common
Sense grant (MESS: Model-Building, Exploratory, Social System
#N66001-19-2-4034) and The Templeton foundation (Grant: The Development of
Curiosity in Childhood #61475). We would like to thank the following
children’s museums for providing us with a space in which to run the
experiments: Bay Area Discovery Museum, Children’s Creativity Museum and The
Lawrence Hall of Science. We would also like to thank the following research
assistants for conducing the research with children: Zane Levin, Elijah
Phipps, Janie Dent, Jenna Levin, Nikita Kumar and Athena Leong.

## References

## References

  * Ahmed et al. [2020] O. Ahmed, F. Träuble, A. Goyal, A. Neitz, Y. Bengio, B. Schölkopf, M. Wüthrich, and S. Bauer.  Causalworld: A robotic manipulation benchmark for causal structure and transfer learning.  _arXiv preprint arXiv:2010.04296_ , 2020.
  * Allen et al. [2020] K. R. Allen, K. A. Smith, and J. B. Tenenbaum.  Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoning.  _Proceedings of the National Academy of Sciences_ , 117(47):29302–29310, 2020.  ISSN 0027-8424.  doi: 10.1073/pnas.1912341117.  URL https://www.pnas.org/content/117/47/29302.
  * Amin et al. [2021] S. Amin, M. Gomrokchi, H. Satija, H. van Hoof, and D. Precup.  A survey of exploration methods in reinforcement learning.  _arXiv preprint arXiv:2109.00157_ , 2021.
  * Bakhtin et al. [2019] A. Bakhtin, L. van der Maaten, J. Johnson, L. Gustafson, and R. Girshick.  Phyre: A new benchmark for physical reasoning.  In _Advances in Neural Information Processing Systems_ , pages 5082–5093, 2019.
  * Bellemare et al. [2016] M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos.  Unifying count-based exploration and intrinsic motivation.  In _Advances in neural information processing systems_ , pages 1471–1479, 2016.
  * Bengio et al. [2019] Y. Bengio, T. Deleu, N. Rahaman, R. Ke, S. Lachapelle, O. Bilaniuk, A. Goyal, and C. Pal.  A meta-transfer objective for learning to disentangle causal mechanisms.  _arXiv preprint arXiv:1901.10912_ , 2019.
  * Brockman et al. [2016a] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba.  Openai gym, 2016a.
  * Brockman et al. [2016b] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba.  Openai gym, 2016b.
  * Brown et al. [2020] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Burda et al. [2018] Y. Burda, H. Edwards, A. Storkey, and O. Klimov.  Exploration by random network distillation.  _arXiv preprint arXiv:1810.12894_ , 2018.
  * Chen et al. [2021] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch.  Decision transformer: Reinforcement learning via sequence modeling.  _arXiv preprint arXiv:2106.01345_ , 2021.
  * Chevalier-Boisvert et al. [2018] M. Chevalier-Boisvert, D. Bahdanau, S. Lahlou, L. Willems, C. Saharia, T. H. Nguyen, and Y. Bengio.  Babyai: A platform to study the sample efficiency of grounded language learning.  In _International Conference on Learning Representations_ , 2018.
  * Chowdhery et al. [2022] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ , 2022.
  * Cobbe et al. [2018] K. Cobbe, O. Klimov, C. Hesse, T. Kim, and J. Schulman.  Quantifying generalization in reinforcement learning.  _arXiv preprint arXiv:1812.02341_ , 2018.
  * Gopnik [2012] A. Gopnik.  Scientific thinking in young children: Theoretical advances, empirical research, and policy implications.  _Science_ , 337(6102):1623–1627, 2012.
  * Gopnik and Sobel [2000] A. Gopnik and D. M. Sobel.  Detecting blickets: how young children use information about novel causal powers in categorization and induction.  _Child Development_ , 71(5):1205–1222, 2000.
  * Gopnik et al. [2017] A. Gopnik, S. O’Grady, C. G. Lucas, T. L. Griffiths, A. Wente, S. Bridgers, R. Aboody, H. Fung, and R. E. Dahl.  Changes in cognitive flexibility and hypothesis search across human life history from childhood to adolescence to adulthood.  _Proceedings of the National Academy of Sciences_ , 114(30):7892–7899, 2017.
  * Goyal et al. [2019] A. Goyal, A. Lamb, J. Hoffmann, S. Sodhani, S. Levine, Y. Bengio, and B. Schölkopf.  Recurrent independent mechanisms.  _arXiv preprint arXiv:1909.10893_ , 2019.
  * Goyal et al. [2021] A. Goyal, A. Didolkar, N. R. Ke, C. Blundell, P. Beaudoin, N. Heess, M. C. Mozer, and Y. Bengio.  Neural production systems.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Griffiths and Tenenbaum [2009] T. L. Griffiths and J. B. Tenenbaum.  Theory-based causal induction.  _Psychological review_ , 116(4):661, 2009.
  * Hill et al. [2018] A. Hill, A. Raffin, M. Ernestus, A. Gleave, A. Kanervisto, R. Traore, P. Dhariwal, C. Hesse, O. Klimov, A. Nichol, M. Plappert, A. Radford, J. Schulman, S. Sidor, and Y. Wu.  Stable baselines.  https://github.com/hill-a/stable-baselines, 2018.
  * Hoffmann et al. [2022] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre.  Training compute-optimal large language models.  _arXiv preprint arXiv:2203.15556_ , 2022.
  * James et al. [2020] S. James, Z. Ma, D. R. Arrojo, and A. J. Davison.  Rlbench: The robot learning benchmark & learning environment.  _IEEE Robotics and Automation Letters_ , 5(2):3019–3026, 2020.
  * Ke et al. [2021] N. R. Ke, A. R. Didolkar, S. Mittal, A. Goyal, G. Lajoie, S. Bauer, D. J. Rezende, M. C. Mozer, Y. Bengio, and C. Pal.  Systematic evaluation of causal discovery in visual model based reinforcement learning.  _arXiv preprint arXiv:2107.00848_ , 2021.
  * Kosoy et al. [2020] E. Kosoy, J. Collins, D. M. Chan, S. Huang, D. Pathak, P. Agrawal, J. Canny, A. Gopnik, and J. B. Hamrick.  Exploring exploration: Comparing children with rl agents in unified environments.  _arXiv preprint arXiv:2005.02880_ , 2020.
  * Kosoy et al. [2022] E. Kosoy, A. Liu, J. Collins, D. M. Chan, J. B. Hamrick, N. R. Ke, S. H. Huang, B. Kaufmann, J. Canny, and A. Gopnik.  Learning causal overhypotheses through exploration in children and computational models.  _arXiv preprint arXiv:2202.10430_ , 2022.
  * Lin et al. [2022] S. Lin, J. Hilton, and O. Evans.  Teaching models to express their uncertainty in words.  _arXiv preprint arXiv:2205.14334_ , 2022.
  * Lucas et al. [2014] C. G. Lucas, S. Bridgers, T. L. Griffiths, and A. Gopnik.  When children are better (or at least more open-minded) learners than adults: Developmental differences in learning the forms of causal relationships.  _Cognition_ , 131(2):284–299, 2014.
  * Machado et al. [2018a] M. C. Machado, M. G. Bellemare, and M. Bowling.  Count-based exploration with the successor representation.  _arXiv preprint arXiv:1807.11622_ , 2018a.
  * Machado et al. [2018b] M. C. Machado, M. G. Bellemare, E. Talvitie, J. Veness, M. Hausknecht, and M. Bowling.  Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents.  _Journal of Artificial Intelligence Research_ , 61:523–562, 2018b.
  * Martin et al. [2017] J. Martin, S. N. Sasikumar, T. Everitt, and M. Hutter.  Count-based exploration in feature space for reinforcement learning.  _arXiv preprint arXiv:1706.08090_ , 2017.
  * McDuff et al. [2021] D. McDuff, Y. Song, J. Lee, V. Vineet, S. Vemprala, N. Gyde, H. Salman, S. Ma, K. Sohn, and A. Kapoor.  Causalcity: Complex simulations with agency for causal discovery and reasoning.  _arXiv preprint arXiv:2106.13364_ , 2021.
  * Mnih et al. [2016] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu.  Asynchronous methods for deep reinforcement learning.  In _International conference on machine learning_ , pages 1928–1937. PMLR, 2016.
  * Nichol et al. [2018] A. Nichol, V. Pfau, C. Hesse, O. Klimov, and J. Schulman.  Gotta learn fast: A new benchmark for generalization in rl.  _arXiv preprint arXiv:1804.03720_ , 2018.
  * Osband et al. [2016] I. Osband, C. Blundell, A. Pritzel, and B. Van Roy.  Deep exploration via bootstrapped dqn.  _Advances in neural information processing systems_ , 29:4026–4034, 2016.
  * Ostrovski et al. [2017] G. Ostrovski, M. G. Bellemare, A. van den Oord, and R. Munos.  Count-based exploration with neural density models.  In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_ , pages 2721–2730. JMLR. org, 2017.
  * Pathak et al. [2017] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell.  Curiosity-driven exploration by self-supervised prediction.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , pages 16–17, 2017.
  * Pathak et al. [2019] D. Pathak, D. Gandhi, and A. Gupta.  Self-supervised exploration via disagreement.  _arXiv preprint arXiv:1906.04161_ , 2019.
  * Radford et al. [2019] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Schmidhuber [1991] J. Schmidhuber.  Curious model-building control systems.  In _Proc. international joint conference on neural networks_ , pages 1458–1463, 1991.
  * Schölkopf et al. [2021] B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio.  Towards causal representation learning.  _arXiv preprint arXiv:2102.11107_ , 2021.
  * Schulman et al. [2017] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Smith et al. [2022] S. Smith, M. Patwary, B. Norick, P. LeGresley, S. Rajbhandari, J. Casper, Z. Liu, S. Prabhumoye, G. Zerveas, V. Korthikanti, E. Zhang, R. Child, R. Y. Aminabadi, J. Bernauer, X. Song, M. Shoeybi, Y. He, M. Houston, S. Tiwary, and B. Catanzaro.  Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model.  _arXiv preprint arXiv:2201.11990_ , 2022.
  * Sontakke et al. [2021] S. A. Sontakke, A. Mehrjou, L. Itti, and B. Schölkopf.  Causal curiosity: Rl agents discovering self-supervised experiments for causal representation learning.  In _International Conference on Machine Learning_ , pages 9848–9858. PMLR, 2021.
  * Tang et al. [2017] H. Tang, R. Houthooft, D. Foote, A. Stooke, O. X. Chen, Y. Duan, J. Schulman, F. DeTurck, and P. Abbeel.  # exploration: A study of count-based exploration for deep reinforcement learning.  In _Advances in neural information processing systems_ , pages 2753–2762, 2017.
  * Veitch et al. [2020] V. Veitch, D. Sridhar, and D. Blei.  Adapting text embeddings for causal inference.  In J. Peters and D. Sontag, editors, _Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI)_ , volume 124 of _Proceedings of Machine Learning Research_ , pages 919–928, August 2020.
  * Wang et al. [2021a] J. X. Wang, M. King, N. Porcel, Z. Kurth-Nelson, T. Zhu, C. Deck, P. Choy, M. Cassin, M. Reynolds, F. Song, G. Buttimore, D. P. Reichert, N. Rabinowitz, L. Matthey, D. Hassabis, A. Lerchner, and M. Botvinick.  Alchemy: A structured task distribution for meta-reinforcement learning.  _arXiv preprint arXiv:2102.02926_ , 2021a.
  * Wang et al. [2021b] X. Wang, X. Xu, W. Tong, R. Roberts, and Z. Liu.  Inferbert: A transformer-based causal inference framework for enhancing pharmacovigilance.  _Frontiers in Artificial Intelligence_ , 4, 2021b.
  * Watkins and Dayan [1992] C. J. Watkins and P. Dayan.  Q-learning.  _Machine learning_ , 8(3):279–292, 1992.
  * Yu et al. [2019] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine.  Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning.  _arXiv preprint arXiv:1910.10897_ , 2019.
  * Zhang et al. [2021] C. Zhang, B. Jia, M. Edmonds, S.-C. Zhu, and Y. Zhu.  Acre: Abstract causal reasoning beyond covariation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10643–10653, 2021.

## Appendix

##  Appendix A Additional Environment Details

(a) No objects on the detector

(b) One object on the detector, but detector does not light up.

(c) Two objects, triggering the detector

Figure A.1: A visualization of some of the possible states of the blicket
environment when rendered in Unity.

##  Appendix B A2C and PPO2 Hyperparameters

Table B.1 gives the set of hyper-parameters for the PPO2 and A2C algorithms
used in the paper. We use the reference implementations from Stable Baselines
(Hill et al., 2018) for our standard RL models. For model size variants, we
use the standard policy networks available in Hill et al. (2018) with no
additional modification (asides from the hidden dimension, as specified in the
main paper).

| A2C | PPO2
---|---|---
Discount Factor | 0.99 | 0.99
Steps/Update | 5 | 128
Value Function Coefficient | 0.25 | 0.5
Entropy Coefficient | 0.01 | 0.01
Learning Rate | 0.0007 | 0.00025
LR Schedule | Constant | Constant
Gradient Clipping (Max Norm) | 0.5 | 0.5
GAE Bias/Variance Lambda | - | 0.95
Table B.1: Hyperparameters for A2C and PPO2 RL algorithms.

##  Appendix C Language Model Prompts

The prompts we used for GPT-3 and PaLM are based on the instructions and
examples that children are given in Kosoy et al. (2022).

###  C.1 Freeform Prompts

We modelled the freeform prompt as closely as possible to the experiment done
with children in Kosoy et al. (2022). The main difference is that we replace
the visual components with textual descriptions. The freeform prompt first
explains that blicket machines turn on when objects called blickets are placed
on them. It explains that some objects are blickets and some are not, and
machines need either one, two, or three blickets placed on it in order to turn
on. Next, the prompt introduces a striped blicket machine and three objects,
and gives three examples of whether the machine lights up or not when certain
objects are placed on it—for example, “If we put the blue pyramid on the
machine, then it does not light up”. Then the prompt does the same for the
dotted blicket machine and three different objects. Finally, the prompt does
the same for a new blicket machine with three different objects, and asks
which of these objects are blickets, and whether this new machine works like
the striped machine or like the dotted machine.

We tested the language models in four conditions, each with its own prompt.
The conditions consist of either hypothesis given or not given, combined with
either disjunctive or conjunctive causal structure. The new machine either has
a conjunctive or disjunctive causal structure, depending on what the condition
is. In the conditions where the hypothesis is given, it is clear from the
examples that the striped machine has a conjunctive causal structure and the
dotted machine has a disjunctive structure. In the conditions where the
hypothesis is not given, it is _not_ clear from the examples whether the
striped and dotted machines have conjunctive or disjunctive structure. Thus in
the not-given hypothesis conditions, when the model is asked whether the new
machine works like the striped or dotted machine, the correct response is to
be unsure.

Below are the exact freeform prompts for all four conditions; the same prompts
are given to both GPT-3 and PaLM.

  * •

All conditions: A blicket detector is a special kind of machine, objects that
are different colors and shapes either make the machine turn on or not. If the
object is a blicket and placed on the machine then the machine will turn on.
Sometimes 1, 2 or 3 blickets make the machine turn on. Our goal is to make the
machine turn on and figure out which shapes make it do so.
_< insert given/not-given hypothesis portion>_
_< insert disjunctive/conjunctive portion>_
Can you tell me which objects are blickets? Does this checkerboard pattern
blicket detector behave like the striped pattern blicket detector or like the
dotted pattern blicket detector?

  * •

Given hypotheses: First I have a striped pattern blicket detector, it behaves
in the following way: I have 3 objects, one blue pyramid, one green cube and
one orange sphere. First I put the blue pyramid on the striped pattern blicket
machine and it does not light up. Then I put the orange sphere on the striped
pattern blicket machine and it does not light up. Then I put the blue pyramid
and the orange sphere on the striped pattern blicket machine and it did light
up!
Then I have a dotted pattern blicket detector. I have 3 different objects now,
a yellow cylinder, a purple cone, and a red dome. First I put the purple cone
on the dotted pattern blicket detector and it did light up! Then I put the
yellow cylinder on the dotted pattern blicket detector and it does not light
up. Then I put the yellow cylinder and the purple cone on the dotted pattern
blicket detector and it did light up!

  * •

Not-given hypotheses: First I have a striped pattern blicket detector, it
behaves in the following way: I have 3 objects, one blue pyramid, one green
cube and one orange sphere. First I put the blue pyramid on the striped
pattern blicket detector and it does not light up. Then I put the green cube
on the striped pattern blicket detector and it does not light up. Then I put
the blue pyramid and the orange sphere on the striped pattern blicket detector
and it did light up!
Then I have a dotted pattern blicket detector. I have 3 different objects now,
a yellow cylinder, a purple cone, and a red dome. First I put the purple cone
on the dotted pattern blicket detector and it does not light up. Then I put
the yellow cylinder on the dotted pattern blicket detector and it does not
light up. Then I put the red half dome and the purple cone on the dotted
pattern blicket detector and it did light up!

  * •

Disjunctive: Then I have a checkerboard pattern blicket detector. I have 3 new
objects, a teal prism, a pink frustum and a brown torus. This machine could
work like the dotted patterned blicket detector or it could work like the
striped pattern blicket detector. First I put the brown torus on the
checkerboard pattern blicket detector and it does light up! Then I put the
pink frustum on the checkerboard pattern blicket detector and it does not
light up. Then I put the teal prism on the checkerboard pattern blicket
detector and it does not light up. Then I put the brown torus and the pink
frustum on the checkerboard pattern blicket detector and it did light up! Then
I put the teal prism and the brown torus on the checkerboard pattern blicket
detector and it did light up! Then I put the teal prism and the pink frustum
and the brown torus on the checkerboard pattern blicket detector and it did
light up! Then I put the pink frustum and the teal prism on the checkerboard
pattern blicket detector and it does not light up.

  * •

Conjunctive: Then I have a checkerboard pattern blicket detector. I have 3 new
objects, a teal prism, a pink frustum and a brown torus. This machine could
work like the dotted patterned blicket detector or it could work like the
striped pattern blicket detector. First I put the brown torus on the
checkerboard pattern blicket detector and it does not light up. Then I put the
pink frustum and the brown torus on the checkerboard pattern blicket detector
and it does not light up. Then I put the teal prism on the checkerboard
pattern blicket detector and it does not light up. Then I put the teal prism
and the pink frustum on the checkerboard pattern blicket detector and it does
not light up. Then I put the pink frustum on the checkerboard pattern blicket
detector and it does not light up. Then I put the teal prism and the brown
torus on the checkerboard pattern blicket detector and it does light up! Then
I put the teal prism and the pink frustum and the brown torus on the
checkerboard pattern blicket detector and it did light up!

Below are the outputs that GPT-3 produces for each of the conditions:

  * •

Given hypothesis, disjunctive: The objects that are blickets are the blue
pyramid, the orange sphere, the purple cone, the yellow cylinder, the brown
torus, the pink frustum and the teal prism. This checkerboard pattern blicket
detector behaves like the dotted pattern blicket detector.

  * •

Given hypothesis, conjunctive: The objects that are blickets are the teal
prism, the pink frustum and the brown torus. This checkerboard pattern blicket
detector behaves like the striped pattern blicket detector.

  * •

Not-given hypothesis, disjunctive: The objects that are blickets are the blue
pyramid, the orange sphere, the yellow cylinder, the purple cone, the red
dome, the teal prism, the pink frustum, and the brown torus. This checkerboard
pattern blicket detector behaves like the striped pattern blicket detector.

  * •

Not-given hypothesis, conjunctive: The objects that are blickets are the blue
pyramid, the green cube, the orange sphere, the yellow cylinder, the purple
cone, the red dome, the teal prism, the pink frustum, and the brown torus.
This checkerboard pattern blicket detector behaves like the dotted pattern
blicket detector.

Below are the outputs that PaLM produces for each of the conditions. We only
take into account the first two sentences of the output when evaluating the
model’s performance.

  * •

Given hypothesis, disjunctive: The answer is that the checkerboard pattern
blicket detector behaves like the striped pattern blicket detector. The
blickets are the brown torus and the pink frustum. The reason why the
checkerboard pattern blicket detector behaves like the striped pattern blicket
detector is because the machine only turns on when the brown torus and the
pink frustum are on the machine. The machine does not turn on when the brown
torus and the teal prism are on the machine. The machine does not turn on when
the pink frustum and the teal prism are on the machine. The machine does not
turn on…

  * •

Given hypothesis, conjunctive: The checkerboard pattern blicket detector
behaves like the striped pattern blicket detector. The blickets are the teal
prism and the brown torus. The checkerboard pattern blicket detector behaves
like the dotted pattern blicket detector. The blickets are the teal prism and
the brown torus. The checkerboard pattern blicket detector behaves like the
striped pattern blicket detector. The blickets are the teal prism and the pink
frustum. The checkerboard pattern blicket detector behaves like the dotted
pattern blicket detector. The blickets are the…

  * •

Not-given hypothesis, disjunctive: This is a very difficult problem. It is not
possible to figure out which objects are blickets and which are not. It is not
possible to figure out how the checkerboard pattern blicket detector works.
This is a very difficult problem. It is not possible to figure out which
objects are blickets and which are not. It is not possible to figure out how
the checkerboard pattern blicket detector works. This is a very difficult
problem. It is not possible to figure out which objects are blickets and which
are not. It is not possible to figure out how the checkerboard…

  * •

Not-given hypothesis, conjunctive: The checkerboard pattern blicket detector
behaves like the striped pattern blicket detector. The blickets are the teal
prism and the brown torus. The checkerboard pattern blicket detector behaves
like the dotted pattern blicket detector. The blickets are the teal prism and
the brown torus. The checkerboard pattern blicket detector behaves like the
striped pattern blicket detector. The blickets are the teal prism and the pink
frustum. The checkerboard pattern blicket detector behaves like the dotted
pattern blicket detector. The blickets are the…

###  C.2 Few-Shot Prompts

The few-shot prompts consist of a preface, two input-output examples, and a
new input (that the model is must provide the corresponding output for). The
input describes the three objects and three examples of whether the machine
lights up or not when certain combinations of objects are placed on it. The
output specifies which of the three objects are blickets, and whether the
machine is like the striped or dotted machine.

Below are the exact texts of the few-shot prompts that we used as inputs to
the PaLM model.

  * •

All conditions: If we put objects on the machine, then it will either light up
or not. Some objects are blickets, and others are not. A striped machine needs
two blickets to make it light up, and a dotted machine needs one blicket to
make it light up. Please identify which objects are blickets and whether the
machine behaves like the striped machine or the dotted machine.
_< insert given/not-given hypothesis portion>_
_< insert disjunctive/conjunctive portion>_

  * •

Given hypotheses:
Input: There is a blue pyramid, a green cube, and an orange sphere. If we put
the blue pyramid on the machine, then it does not light up. If we put the
orange sphere on the machine, then it does not light up. If we put the blue
pyramid and the orange sphere on the machine, then it does light up.
Output: The blue pyramid and orange sphere are blickets. This machine behaves
like the striped machine.
Input: There is a yellow cylinder, a purple cone, and a red dome. If we put
the purple cone on the machine, then it does light up. If we put the yellow
cylinder on the machine, then it does not light up. If we put the yellow
cylinder and the purple cone on the machine, then it does light up.
Output: The purple cone is a blicket. This machine behaves like the dotted
machine.

  * •

Not-given hypotheses:
Input: There is a blue pyramid, a green cube, and an orange sphere. If we put
the blue pyramid on the machine, then it does not light up. If we put the
green cube on the machine, then it does not light up. If we put the blue
pyramid and the orange sphere on the machine, then it does light up.
Output: The orange sphere is a blicket, and the blue pyramid is maybe a
blicket.
Input: There is a yellow cylinder, a purple cone, and a red dome. If we put
the purple cone on the machine, then it does light up. If we put the yellow
cylinder on the machine, then it does not light up. If we put the red dome and
the purple cone on the machine, then it does light up.
Output: The red dome is a blicket and the purple cone is maybe a blicket.

  * •

Disjunctive:
There is a teal prism, a pink frustum, and a brown torus. If we put the brown
torus on the machine, then it does light up. If we put the pink frustum and
the brown torus on the machine, then it does light up. If we put the teal
prism on the machine, then it does not light up. If we put the teal prism and
the pink frustum on the machine, then it does not light up. If we put the pink
frustum on the machine, then it does not light up. If we put the teal prism
and the brown torus on the machine, then it does light up. If we put the teal
prism and the pink frustum and the brown torus on the machine, then it does
light up.
Output:

  * •

Conjunctive:
Input: There is a teal prism, a pink frustum, and a brown torus. If we put the
brown torus on the machine, then it does not light up. If we put the pink
frustum and the brown torus on the machine, then it does not light up. If we
put the teal prism on the machine, then it does not light up. If we put the
teal prism and the pink frustum on the machine, then it does not light up. If
we put the pink frustum on the machine, then it does not light up. If we put
the teal prism and the brown torus on the machine, then it does light up. If
we put the teal prism and the pink frustum and the brown torus on the machine,
then it does light up.
Output:

Below are the outputs that PaLM produces for each of the conditions:

  * •

Given hypothesis, disjunctive: The brown torus is a blicket. This machine
behaves like the dotted machine.

  * •

Given hypothesis, conjunctive: The teal prism and brown torus are blickets.
This machine behaves like the striped machine.

  * •

Not-given hypothesis, disjunctive: The brown torus is a blicket, and the pink
frustum is maybe a blicket.

  * •

Not-given hypothesis, conjunctive: The teal prism is a blicket, and the pink
frustum is maybe a blicket.