# Human irrationality: both bad
and good for reward inference

Lawrence Chan  University of California, Berkeley
<EMAIL_ADDRESS>Andrew Critch  University of California, Berkeley
<EMAIL_ADDRESS>Anca Dragan  University of California, Berkeley
<EMAIL_ADDRESS>
###### Abstract

Assuming humans are (approximately) rational enables robots to infer reward
functions by observing human behavior. But people exhibit a wide array of
irrationalities, and our goal with this work is to better understand the
effect they can have on reward inference. The challenge with studying this
effect is that there are many types of irrationality, with varying degrees of
mathematical formalization. We thus operationalize irrationality in the
language of MDPs, by altering the Bellman optimality equation, and use this
framework to study how these alterations would affect inference.

We find that wrongly modeling a systematically _irrational_ human as noisy-
_rational_ performs a lot worse than correctly capturing these biases – so
much so that it can be better to skip inference altogether and stick to the
prior! More importantly, we show that an _irrational_ human, when _correctly_
modelled, can communicate _more_ information about the reward than a
_perfectly rational_ human can. That is, if a robot has the correct model of a
human’s irrationality, it can make an even _stronger_ inference than it ever
could if the human were rational. Irrationality fundamentally helps rather
than hinder reward inference, but it needs to be correctly accounted for. We
explain this theoretically by showing that irrationality can increase the
mutual information between the policy and the reward. We also replicate the
result in an autonomous driving domain with myopic/greedy demonstrations.

##  I Introduction

Motivated by difficulty in reward specification [21], inverse reinforcement
learning (IRL) methods estimate a reward function from human demonstrations
[24, 1, 20, 17, 23]. The central assumption behind these methods is that human
behavior is rational, i.e., optimal with respect to their reward (cumulative,
in expectation). Unfortunately, decades of research in behavioral economics
and cognitive science [6] have unearthed a deluge of _irrationalities_ , i.e.,
of ways in which people deviate from optimal decision making: hyperbolic
discounting, scope insensitivity, illusion of control, decision noise, loss
aversion, to name a few. As a community, we are starting to account for some
of these irrationalities in different ways [35, 36, 32, 26, 12, 30]. In this
work, we seek to understand how the _"ground truth"_ human irrationality (type
and amount) pairs with the robot’s _model_ of the human to affect the
_performance_ of reward inference.

On the human _model_ side, prior work has shown that a more accurate model of
the ground-truth human behavior will lead to better reward learning
performance. However, what we still do not understand is: how big is the
difference between reward learning performance for an irrational human when
the human is correctly modelled (as irrational), as opposed to incorrectly
modelled (as rational), for different types of irrationality? A very large
difference would mean we should keep pushing towards better human models.

On the _ground truth_ human side, we may intuit that a perfectly rational
human would be the ideal human for reward inference and that a robot using a
rational model will perform the best reward inference, i.e. that the (rational
human, rational model) pair is ideal. Even if this proves true, it still
raises questions about how other human-model pairs fare. For example, just how
hard does human irrationality make reward learning, assuming the robot
accurately models the human? While this might seem hypothetical because we
cannot dictate what kind of human interacts with the robot, the answer is
nonetheless important in order to understand the fundamental limits of reward
learning. Namely, what are the effects of irrationality on the upper bounds on
performance, if the robot correctly models the human? If some irrationality
types lead to inherently ambiguous behavior, such that no robot could perform
well, this is something the reward learning community needs to be aware of.

Figure 1: On the left, the rational human (dark gray) behaves in the same way
regardless of their speed preference. On the right, the myopic (short horizon)
human (cyan) behaves _differently_ depending on their speed preference:
merging and staying in lane vs. myopically starting to pass on the right to
make more progress. Myopia induces this diversity of behavior as a function of
the human’s reward, increasing the amount of information in the behavior. This
makes it fundamentally easier for a robot to differentiate what the reward is.

One challenge with conducting such an analysis is that there are many
irrationalities in the psychology and behavioral economics literature, with
varying degrees of mathematical formalization versus empirical description. To
structure the space for our analysis, we operationalize irrationalities in the
language of MDPs by systematically enumerating possible deviations from the
Bellman equation – imperfect maximization, deviations from the true transition
function, etc. This gives us a formal framework in which we can simulate
irrational behavior, run reward inference, and study its performance. Armed
with this formalism, we then explore the various impacts of irrationality on
reward learning in three families of environments: small random MDPs, a more
legible gridworld MDP, and an autonomous driving domain drawn from the
literature [28].

###  I-A Irrationality can make human behavior _more_ informative.

Our most surprising finding comes from analyzing the question of how "ground
truth" human irrationality affects reward inference, when the correct model is
known to the robot. We expected that perfectly rational behavior is more
informative than suboptimal, biased behavior. Instead, we found that certain
irrationalities actually _improve_ the quality of reward inference. When the
human exhibits certain biases, correctly modeling these biases can help make
stronger inferences – not only compared to incorrect modeling, but even
compared to the best-case performance when learning from a perfectly rational
human! That is, in all three environments: an irrational human correctly
modelled as irrational by the robot leads to better reward inference than a
rational human correctly modelled as rational, for some types and amounts of
irrationality.

We explain this theoretically from the perspective of the _mutual information_
between the human behavior and the reward parameters, proving that some
irrationalities are arbitrarily more informative than rational behavior. An
intuitive example is in Fig. 1, where myopic behavior varies with the reward,
but rational behavior does not – this makes it fundamentally possible for the
robot to figure out the reward from the myopic behavior, but not from the
rational one. We prove that there are cases where the rational behavior is not
informative at all, whereas some irrational behavior achieves the information-
theoretic upper bound; we also prove that the popular Boltzmann-rationality
model is arbitrarily more informative than perfect rationality. While knowing
the human irrationality model is hard, this result is overall very
encouraging: it is probably easier to learn irrationality models of people,
than to make people act rationally – so in the end, it is a good thing that
irrational behavior is the more informative one.

Figure 2: In Sec. II-C, we modify the components of the Bellman update to
cover different types of irrationalities: changing the max into a softmax to
capture noise (Boltzmann), changing the transition function to capture
optimism/pessimism or the illusion of control, changing the reward values to
capture the nonlinear perception of gains and losses (Prospect), changing the
average reward over time into a maximum (Extremal), and changing the
discounting to capture more myopic decision-making.

###  I-B Unmodelled irrationality leads to remarkably poor reward inference.

We continue with an analysis of what the _model_ that the robot uses should
be, if the human is in fact irrational. It might seem that we can’t
immediately benefit from the knowledge that irrationalities help inference
unless we have a comprehensive understanding of human decision-making, and so
we should just stick to the status quo of modeling people as rational.
However, prior work [30, 32] has already shown benefits of algorithms that
approximately capture irrationality. What we contribute is the finding that
modeling irrational humans as (noisily-)rational (instead of accounting for
their biases) can lead to worse outcomes than not performing inference at all
and just using the prior (Sec. V). Encouragingly, we also find evidence that
even just modeling the human’s irrationality only _approximately_ , i.e.
getting the type of irrationality correct but not the exact parameters, allows
a robot to outperform modeling the human as noisily-rational. This reinforces
prior works that studied specific algorithms for capturing specific biases
with a more systematic analysis (Sec. V-B).

###  I-C Contributions

Overall, we contribute 1) a theoretical and empirical analysis of the effects
of different irrationalities on reward inference, 2) a way to systematically
formalize and cover the space of irrationalities in order to conduct such an
analysis, and 3) evidence and explanation for the importance and benefit of
accounting for irrationality during inference. The implications are that we
should probably stop assuming rationality (because it can lead to worse
inference than sticking to the prior), and further prior work on learning
models of irrationality (because there is a lot of information to be exploited
from irrational behavior). While basic intuition says that it is good to model
irrationality because people are irrational, what we argue in this paper is
that there is an additional reason for modeling irrationality: the way people
are irrational communicates additional information about the reward, and
correct modeling will enable robots to tap into that.

##  II Framework: Biases as deviations from the Bellman update

###  II-A Exploring biases through simulation

While ideally we would recruit human subjects with different irrationalities
and measure how well we can learn rewards, this is prohibitive because we do
not get to dictate someone’s irrationality type: people exhibit a mix of them,
some yet to be discovered. Further, measuring the accuracy of inference from
observing real humans is complicated by the fact that we lack ground truth
access to the human’s reward function. For instance, suppose we asked subjects
to produce a set of (behavior, reward function) pairs. We could then try to
predict the reward functions from the behaviors. But how did we, the
experimenters, infer the reward functions from the people? If we are wrong in
our assumptions about which irrationalities are affecting their behavior
and/or explicit reports of rewards, we would remain deluded about the
subjects’ true preferences.

To address these issues, we _simulate_ human behavior subject to different
irrationalities, run reward inference, and measure the performance against the
ground truth, i.e., the accuracy of a Bayesian posterior on the reward
parameter given the (simulated) human’s inputs.

###  II-B Background and formalism

Consider an Uncertain-Reward MDP (URMDP) [4, 27, 8]
$\mathcal{M}=(S,A,\\{P_{{s},{a}}\\},\gamma,\Theta,p,r)$, consisting of finite
state and action sets $S$ and $A$, distributions over states
$\\{P_{{s},{a}}\\}$ representing the result of taking action $a$ in state $s$,
discount rate $\gamma\in[0,1)$, a (finite) set of reward parameters $\Theta$,
a prior distribution $p\in\Delta(S\times\Theta)$ over starting states and
reward parameters, and a parameterized state-action reward function
$r:\Theta\times S\times A\times S\rightarrow\mathbb{R}$, where
$r_{\theta}({s},{a},{s^{\prime}})$ represents the reward received.

We assume that the human’s policy $\pi$ satisfies $\pi=d(\theta)$, where $d$
is an (environment-specific) planner $d:\Theta\rightarrow\Pi$ that returns a
(possibly stochastic) policy given a particular reward parameter $\theta$. The
rational human uses a planner $d_{\textrm{Rational}}$ that, given a reward
parameter $\theta$, returns a policy that maximizes its expected value. On the
other hand, we say that an human is irrational if its planner returns policies
with lower expected value than the optimal policy, for at least one
$\theta\in\Theta$.

###  II-C Types and degrees of irrationality

There are many possible irrationalities that people exhibit [6], far more than
what we could study in one paper. To provide good coverage of this space, we
start from the Bellman update, and systematically manipulate its terms and
operators to produce a variety of different irrationalities that deviate from
the optimal MDP policy in complementary ways. For instance, operating on the
discount factor can model myopic behavior, while operating on the transition
function can model optimism or the illusion of control. We parametrize each
irrationality so that we can manipulate its “intensity" or deviation from
rationality. Figure 2 summarizes our approach, which we detail below.

####  II-C1 Rational human

In our setup, the rational human does value iteration using the Bellman update
from Fig. 2 with ground truth models and high $\gamma$. Our models change this
update to produce different types of non-rational behavior.

####  II-C2 Modifying the max operator: Boltzmann

Boltzmann-rationality modifies the maximum over actions $\max_{a}$ with a
Boltzmann operator with a parameter $\beta$:

$V_{i+1}(s)=\textrm{Boltz}^{\beta}_{a}\sum_{s^{\prime}\in
S}P_{{s},{a}}{(s^{\prime})}\left(r_{\theta}(s,a,s^{\prime})+\gamma
V_{i}(s^{\prime})\right),$

where $\textrm{Boltz}^{\beta}(\mathbf{x})={\sum_{i}x_{i}e^{\beta
x_{i}}}/{\sum_{i}e^{\beta x_{i}}}$ [36, 3]. This is the most popular
stochastic model used in reward inference[36, 3, 13]. After computing the
value function, the Boltzmann-rational planner $d_{\textrm{Boltz}}$ returns a
policy where the probability of an action is proportional to the exponential
of the $Q$-value of the action:

$\pi(a|s)\propto e^{\beta Q_{\theta}(s,a)}.$

The constant $\beta$ is called the _rationality constant_ , because as
$\beta\to\infty$, the human choices approach perfect rationality (optimality),
whereas $\beta=0$ produces uniformly random choices.

####  II-C3 Modifying the transition function

Our next set of irrationalities manipulate the transition function away from
reality.

##### Illusion of Control.

People often overestimate their ability to control random events [34]. To
model this, we consider humans that use the Bellman update:

$V_{i+1}(s)=\max_{a}\sum_{s^{\prime}\in
S}P_{{s},{a}}^{n}{(s^{\prime})}\left(r_{\theta}(s,a,s^{\prime})+\gamma
V_{i}(s^{\prime})\right)$

where
$P_{{s},{a}}^{n}{(s^{\prime})}=\left(P_{{s},{a}}{(s^{\prime})}\right)^{n}/\sum_{s^{\prime\prime}\in
S}\left(P_{{s},{a}}{(s^{\prime\prime})}\right)^{n}$. As $n\rightarrow\infty$,
the human acts as if it exists in a deterministic environment. As
$n\rightarrow 0$, the human acts as if it had an equal chance of transitioning
to every possible successor state.

##### Optimism/Pessimism

Many people systematically overestimate or underestimate their chance
experiencing of positive over negative events [31]. We model this using humans
that modify the probability they get outcomes based on the value of those
outcomes:

$V_{i+1}(s)=\max_{a}\sum_{s^{\prime}\in
S}P_{{s},{a}}^{\omega}{(s^{\prime})}\left(r_{\theta}(s,a,s^{\prime})+\gamma
V_{i}(s^{\prime})\right)$

where $P_{{s},{a}}^{\omega}{(s^{\prime})}\propto
P_{{s},{a}}{(s^{\prime})}e^{\omega\left(r_{\theta}(s,a,s^{\prime})+\gamma
V_{i}(s^{\prime})\right)}$. $\omega$ controls how pessimistic or optimistic
the human is. As $\omega\rightarrow+\infty$ (respectively,
$\omega\rightarrow-\infty$), the human becomes increasingly certain that good
(bad) transitions will happen. As $\omega\rightarrow 0$, the human approaches
the rational human.

####  II-C4 Modifying the reward

Next, we consider humans that use the modified Bellman update:

$V_{i+1}(s)=\max_{a}\sum_{s^{\prime}\in
S}P_{{s},{a}}{(s^{\prime})}\left(f(r_{\theta}(s,a,s^{\prime}))+\gamma
V_{i}(s^{\prime})\right)$

where $f:\mathbb{R}\rightarrow\mathbb{R}$ is some scalar function. This is
equivalent to solving the MDP with reward $f\circ r_{\theta}$, and allows us
to model human behavior such as loss aversion and scope insensitivity.

##### Prospect Bias

Kahneman and Tversky [19] inspires us to consider a particular family of $f$s:

$f_{c}(r)=\begin{cases}\log(1+|r|)&r>0\\\ 0&r=0\\\
-c\log(1+|r|)&r<0\end{cases}$

$c$ controls how loss averse the human is. As $c\rightarrow\infty$, the human
primarily focuses on avoiding negative rewards. As $c\rightarrow 0$, the human
focuses on maximizing positive rewards.

####  II-C5 Modifying the sum between reward and future value:

##### Extremal

People seem to exhibit duration neglect, sometimes only caring about the
maximum intensity of an experience [10]. We model this using Bellman update:

$\displaystyle V_{i+1}(s)=\max_{a}\sum_{s^{\prime}\in
S}P_{{s},{a}}{(s^{\prime})}\max\begin{cases}r_{\theta}(s,a,s^{\prime})\\\
(1-\alpha)r_{\theta}(s,a,s^{\prime})+\alpha V_{i}(s^{\prime})\end{cases}$

These humans maximize the expected maximum reward along a trajectory, instead
of the expected sum of rewards. As $\alpha\rightarrow 1$, the human maximizes
the expected maximum reward received along the full trajectory. As
$\alpha\rightarrow 0$, the human becomes greedy and maximizes the immediate
reward.

####  II-C6 Modifying the discounting

##### Myopic Discount

In practice, people are sometimes myopic, only considering near-term rewards.
One way to model this is to decrease gamma in the Bellman update. At
$\gamma=\gamma^{*}$, the discount rate specified by the environment, this is
the rational human. As $\gamma\rightarrow 0$, the human becomes greedy and
only acts to maximize immediate reward.

##### Myopic Value Iteration

As another way to model human myopia, we consider a human that performs only
$h$ steps of Bellman updates. That is, this human cares equally about rewards
for a horizon $h$, and discount to 0 reward after that. As
$h\rightarrow\infty$, this human becomes rational. If $h=1$, this human only
cares about the immediate reward.

##### Hyperbolic Discounting

Human also exhibit hyperbolic discounting, with a high discount rate for the
immediate future and a low discount rate for the far future [15]. Alexander
and Brown [2] formulate this as the following Bellman update:

$V_{i+1}(s)=\max_{a}\sum_{s^{\prime}\in
S}P_{{s},{a}}{(s^{\prime})}\frac{r_{\theta}(s,a,s^{\prime})+V_{i}(s^{\prime})}{1+kV_{i}(s^{\prime})}$

$k$ modulates how much the human prefers rewards now versus the future. As
$k\rightarrow 0$, this human becomes a rational human without discounting.

##  III Exploring the effects of known biases on reward inference

Armed with our framework for characterizing irrationality, we test its
implications for reward inference. We start by investigating the effects in
random MDPs.

###  III-A Experimental Design: Exact Inference in Random MDPs.

##### Independent Variables.

We manipulate the type of the planner, and vary the degree parameters for
each. We use different environments, sample different ground truth reward
parameters, and test different trajectory lengths for the demonstrated
behavior ($T=3$, $15$, and $30$ state-actions).

##### Dependent Measures.

To separate the inference difficulty caused by suboptimal inference from the
difficulty caused by human irrationality, we perform the exact Bayesian update
on the trajectory $\xi$ [25], which gives us the posterior on $\theta$ given
$\xi$,
$P(\theta|\xi)=\frac{P(\xi|\theta)P(\theta)}{\int_{\theta^{\prime}}P(\xi|\theta^{\prime})P(\theta^{\prime})}$.
Our primary metric is the expected log loss of this posterior:

$\displaystyle\textrm{Log Loss}(\theta|\xi)=E_{{\theta^{*}},\xi\sim
d(\theta^{*})}\left[-\log P(\theta^{*}|\xi)\right].$

A low log loss implies that we are assigning a high likelihood to the true
$\theta^{*}$. Note that in this case, the log loss is equal to the entropy of
the posterior $H(\theta|\xi)=H(\theta)-\mathbf{I}(\theta;\xi)$.

For each environment and irrationality type, we calculate the performance of
reward inference on trajectories of a fixed length $T$. To sample a trajectory
of length $T$ from a human, we fix $\theta^{*}$ and start state $s$. Then, we
generate rollouts starting from state $s$ until $T$ state, action pairs have
been sampled from $\pi=d(\theta^{*})$. We repeat this procedure 10 times for
each start state.

Figure 3: The log loss (lower = better) of the posterior as a function of the
parameter we vary for each irrationality type, on the random MDP environments.
For the irrationalities that interpolate to the rational planner, we denote
the value that is closest to rational using a dashed vertical line. Every
irrationality except Prospect Bias all have parameter settings that outperform
the rational planner. The error bars show the standard error of the mean,
calculated by 1000 bootstraps across environments.

##### Simulation Environment.

We used MDPs with 10 states and 2 actions, where each $(s,a)$-pair has 2
random, nonzero transition-probabilities. We used $\gamma=0.99$ and start
trajectories in every state without reward. In these $\theta$ is a vector of
length 3, where $\theta_{i}$ is the reward received from transitions out of
state $i$ (and all other rewards are 0). We discretized each $\theta_{i}$ with
4 values, leading to $|\Theta|=64$. We generated 20 such random MDPs.

Figure 4: The analog of Fig. 3 for the gridworld. Error bars are the standard
error of the mean. The findings are surprisingly similar as with the random
MDPs. Note the more limited x-axis ranges we used in this experiment.

###  III-B Results

Fig. 3 plots the log loss for each irrationality for random MDPs. The degree
affects reward inference, with many settings naturally resulting in worse
inference, especially at the extremes. However, what is surprising is that
_every_ type except Prospect Bias has at least one degree (parameter) setting
that results in _better_ inference with enough data: we see that most
irrationality types can be more informative than rational behavior. The more
data we have, the more drastic the difference (T=30 results in both better
inference and larger difference relative to rational).

We put this to the test with a repeated-measures ANOVA with planner type as
the independent variable, using the data from the best parameter setting from
each type and T=30, and environment as a random effect. We find a significant
main effect for planner type ($F(8,806372)=6102.93,\ p<0.001$). A post-hoc
analysis with Tukey HSD corrections confirmed that every irrationality except
Prospect improved inference compared to the fully rational planner ($p<.001$).
For $T=30$, Optimism with $\omega=3.16$ performed best, followed by Illusion
of Control with $n=0.1$ and Boltzmann with $\beta=10$.

Figure 5: Myopic value iteration ($h=5$), here presented in blue) produces different policies for $\theta^{*}=(4,1)$ vs. $\theta^{*}=(4,0)$: while the rational expert (whose policy is represented in black) always detours around the hole and attempts to reach the larger reward, myopia causes the myopic expert to go for the smaller source of reward when it is non-zero. TABLE I: Autonomous Driving Results: Merging | Bayesian IRL | CIOC
---|---|---
Horizon | Log Loss | Information Gain | Cosine Similarity | Normalized Return
3 | 0.690 | 0.696 | 0.999 | 0.939
5 | 0.824 | 0.562 | 0.940 | 0.981
7 | 1.383 | 0.004 | 0.350 | 0.856

##  IV Does this effect generalize?

In the domain of random MDPs, we found that not only does irrationality affect
reward inference, certain irrationalities can actually result in better
inference. In this section, we probe the generality of this finding
empirically, and explain it theoretically.

###  IV-A Gridworld

Random MDPs lack structure, so we first test a toy environment that adds
natural structure based on OpenAI Gym’s ‘Frozen-Lake-v0’ [5] (Fig. 5). Fig. 4
shows the results in the gridworld: they are surprisingly (and reassuringly)
similar to the random MDPs. This suggests that irrationalities can indeed help
inference across differently structured MDPs. By inspecting the policies, we
see that the irrational humans were able to outperform the rational human by
disambiguating between $\theta$s that the rational human could not. To
visualize this, we give an example of how the policy of several irrational
humans differ on the gridworld when the rational human’s policies are
identical in Fig. 5. Additional examples are given in Fig. 9 and Fig. 10 in
the appendix.

###  IV-B Autonomous Driving

Even if known irrationality helps reward inference both empirically and
theoretically in small MDPs, the question still remains whether this result
will matter in practice. Next, we investigate the effect of human
irrationality on reward inference in a continuous state and action self-
driving car domain [28]. Switching to this real-world domain means we can no
longer plan exactly, so we use model predictive control. It also means we can
no longer run exact inference, so we approximate Bayesian IRL through samples,
and also test continuous IOC (CIOC, Levine and Koltun [22]) which recovers an
approximate MLE. We measure the cosine similarity between the estimate and
$\theta^{*}$, as well as the normalized reward of the trajectory when
optimizing the recovered estimate. We use the merging task from Fig. 1 (more
details in the appendix).

We report our results in Table I. As the columns relating to Bayesian IRL
show, decreasing the MPC planning horizon significantly decreases the log loss
of the Bayesian IRL posterior. As with the gridworld results in Sec. IV-A, the
reason for this is that the shorter planning horizons exhibit more diverse
behavior (as a function of the reward). Of the 4 reward settings we used, MPC
with horizon 3 and 5 produced two different qualitative behaviors, whereas all
rewards led to qualitatively similar behavior with horizon 7 (depicted in Fig.
1). When the weight on target speed was large enough, MPC with horizon 3 and 5
would produce a trajectory that overtakes the front car by going off the road.
In all other cases, the human would merge between the two other cars.

(a)

(b)

Figure 6: (a) A comparison of reward inference using a correct model of the
irrationality type, versus always using a Boltzmann-rational model
($\beta=10$), on the random MDPs (left) and the car environment (right). The
impairment due to model misspecification greatly outweighs the variation in
inference performance caused by various irrationalities. The error bars show
the standard error of the mean, calculated by the bootstrap across
environments. (b) An example of why assuming Boltzmann is bad when the ground
truth human is Myopic in the gridworld environment - the Boltzmann rational
agent would take the trajectory depicted only if the reward at the bottom was
not much less than the reward at the top. A myopic human with $n\leq 4$,
however, only "sees" the reward at the bottom. Consequently, inferring the
preferences of the myopic agent as if it were Boltzmann leads to poor
performance in this case.

We found similar results for CIOC: decreasing the MPC planning horizon
increases the cosine similarity between the true and CIOC-recovered rewards.
The return of optimizing the CIOC-recovered rewards is also higher when the
human’s planning horizon is 3 or 5 than 7.

Overall, yet again we find that irrationality (in this case specifically
myopia) improves reward inference by producing more diverse behavior as a
function of the reward.

###  IV-C Theoretical Analysis

We now investigate this phenomenon theoretically. We show that not only can
rational behavior be arbitrarily less informative than irrational behavior,
but also that this applies to Boltzmann-rationality.

####  IV-C1 Informativeness as mutual information.

The mutual information
$\mathbf{I}(\theta;d(\theta))=H(\theta)-H(\theta|d(\theta))$ between the
policy and the reward parameters allows us to quantify the informativeness of
a planner. As the conditional entropy $H(\theta|d(\theta))=E\left[-\log
p(\theta|d(\theta)\right]$ is equal to the log loss of the posterior under the
true model, the mutual information upper bounds how much better (in terms of
log loss) _any_ inference procedure can do, relative to the prior. A known
insight is that planners that optimize for maximizing the discounted sum of
rewards _are not the same_ as those that optimize for being informative [11,
16]:
$\arg\max_{d}E_{\theta}[V^{d(\theta)}_{\theta}]\not=\arg\max_{d}\mathbf{I}(\theta;d(\theta))$.
While this insight has been tested empirically, we begin with a theoretical
lens for understanding it.

####  IV-C2 Irrationalities exist that are arbitrarily better for inference
than rationality.

We first consider deterministic planners: planners that return deterministic
policies. We show that there are cases where the rational behavior is not
informative at all, whereas some (irrational) deterministic planner achieves
the theoretical upper bound on informativeness.

######  Proposition 1.

There exists a family of URMDPs with state spaces of any size, such there
exists a deterministic planner $d_{\textrm{Irrational}}$ satisfying
$\mathbf{I}(\theta,d_{\textrm{Irrational}}(\theta))=\log|\Theta|$ and
$\mathbf{I}(\theta,d_{\textrm{Rational}}(\theta))=0$.

####  IV-C3 Boltzmann-rationality is (arbitrarily) more informative than full
rationality.

Of course, the upper bound is attained by _some_ irrational planner. It
demonstrates that an human can perform better than rational when specifically
optimizing for informativeness. But this is an artificial, contrived kind of
irrationality. In fact, prior work that maximized informativeness did so by
solving a _more difficult_ problem than rationality [11, 16]. Here, we provide
evidence that Boltzmann-rationality, a standard model of stochastic choice,
outperforms full rationality for reward inference. (Proof in appendix.)

######  Proposition 2.

There exists a family of one-state two-action MDPs, with arbitrarily large
$|\Theta|$ such that
$\mathbf{I}(\theta,d_{\textrm{Boltz}}(\theta))=\log|\Theta|$ and
$\mathbf{I}(\theta,d_{\textrm{Rational}}(\theta))=0$.

##  V Effects of misspecification on reward inference

We see that irrationalities sometimes hinder, but sometimes help reward
inference. So far, the robot had access to the type (and degree of
irrationality) during inference. Next, we ask how important it is to know
this. Can we not bother with irrationality, make the default assumption (of
Boltzmann-rationality), and run inference?

###  V-A Assuming noisy rationality can lead to very poor inference

Fig. 6(a) suggests that the answer is no. We start by comparing inference with
the true model on random MDPs versus with assuming the standard Boltzmann
model as a default. The results are quite striking: not knowing the correct
irrationality harms inference tremendously.111Shah et al. [30] proposed a way
to model irrationality and analyzed its benefit over assuming Boltzmann; the
benefit was very limited, which they attributed due to their deep learning
model’s brittleness compared to exact planning. Here, we compare to perfect
modeling to analyze the headroom that modeling has. We then confirm that
misspecification greatly harms inference in the autonomous driving
environment. In Fig. 6(b), we provide an example of why assuming Boltzmann-
rationality when the human is Myopic leads to bad inference. This emphasizes
the importance of understanding irrationality when doing reward inference
going forward.

###  V-B Approximate models of irrationality might be enough

This finding is rather daunting, as perfect models of irrationality are very
challenging to develop. But do they need to be perfect? Our final analysis
suggests that the answer is no as well. In Fig. 7, we report the log loss of
inference with the correct type, but under misspecification of the parameter.
Encouragingly, we find that in many cases, merely getting the type of the
human irrationality correct is actually sufficient to lead to much better
inference than assuming Boltzmann rationality. Further, we also find evidence
that the robot does not need to get the type exactly right either: as shown in
Fig. 8, if the robot accounts for the human’s myopia, but gets the _type_ of
the myopia wrong, this still leads to significantly better inference than
assuming Boltzmann rationality.

Figure 7: The log loss (lower = better) of various models under parameter
misspecification. Each x-axis shows the parameter that the robot assumes. The
orange line represents the performance when the robot makes the faulty
assumption that the human is Boltzmann-rational. In many cases, the robot
perform better than by assuming Boltzmann-rational just by getting the type of
the planner correct, even if they don’t get the exact parameter correct. The
error bars show the standard error of the mean, calculated by the bootstrap
across environments. Figure 8:  The log loss (lower = better) of two myopic
humans under type misspecification. On the left, the human performs myopic
value iteration (Myopic $h$), but the robot assumes the human has a myopic
discount rate $\gamma$ (Myopic $\gamma$). On the right, the human has a myopic
discount rate $\gamma$ but the robot assumes myopic value iteration. However,
in both cases, this leads to better inference than assuming Boltzmann-
rationality.

##  VI Discussion

####  VI-1 Summary

In this work, we analyzed how the _"ground truth"_ human pairs with the
robot’s _model_ of the human to affect the performance of reward inference. To
investigate this, we defined irrationalities as deviations from the Bellman
update (Sec. II-C).

On the _model_ side, we found that if the human is irrational, incorrectly
modeling them via the standard noisy-rationality assumption can lead to so
poor performance of reward inference, that it could be better to stick with
the prior and skip inference. While it is natural for incorrect models to lead
to worse performance, this gap is particularly daunting. Fortunately, even
approximate models of irrationalities may be enough - merely getting the
_type_ of the human irrationality correct led to significantly better
inference than assuming noisy-rationality.

On the _ground truth_ human side, we asked what the best-case performance of
learning is for different human irrationalities – that is, if the robot models
the human correctly, how well can it hope to infer the reward? We expected
that noisy and irrational humans would be less informative about what they
want than perfectly rational humans, and therefore make the reward inference
problem inherently more difficult. However, we found that irrationality can
actually make human behavior _more_ informative, by increasing the mutual
information between reward and policy. In three domains, we found that
irrational behavior varies more as a function of the reward, and ultimately
results in better learning performance. This is not only a theoretically
interesting observation about the upper bound of the problem, but also
encourages further research on modeling human irrationality, so that robots
can benefit from this additional information in human behavior.

####  VI-2 Limitations and Future Work

Much of our results are in the relatively random MDP and gridworld
environments. Though we confirmed our key results in a higher-dimensional,
continuous autonomous driving task, future work is needed to replicate these
results in more realistic domains. Future work is also needed to develop
reward learning algorithms that can account for a variety human biases in
larger domains.

Further, our paper presents an analysis, not a practical algorithm for
improving reward learning. Our most interesting finding, and the one we
dedicate most of the paper to, is the fact that irrationalities can improve
reward inference. Though we showed that even approximate models of
irrationality suffice to outperform the standard noisily-rational assumption,
actually benefiting from irrationality still requires precise models of the
human’s irrationality. Nonetheless, we think the result is still important,
because it encourages treating biases more seriously. In the end, our paper
makes a perhaps fortunate observation – that the irrational human, when
correctly modelled, is better than the rational human correctly: since real
humans are likely not perfectly rational, this might be good news in the end.
In fact, is is probably easier to learn irrationality models of people, than
to make people act rationally – so in the end, it is a good thing that
irrational behavior is the more informative one.

What is more, our results also point to an untapped opportunity: training
people to act irrationality (e.g. myopically) when demonstrating behavior in
order to be more informative to our robots. User studies with real people will
be important to understand how to make use of this opportunity.

## References

  * Abbeel and Ng [2004] Pieter Abbeel and Andrew Y Ng.  Apprenticeship learning via inverse reinforcement learning.  In _Proceedings of the twenty-first international conference on Machine learning_ , page 1. ACM, 2004.  URL: http://people.eecs.berkeley.edu/~russell/classes/cs294/s11/readings/Abbeel+Ng:2004.pdf.
  * Alexander and Brown [2010] William H Alexander and Joshua W Brown.  Hyperbolically discounted temporal difference learning.  _Neural computation_ , 22(6):1511–1527, 2010\.
  * Asadi and Littman [2017] Kavosh Asadi and Michael L Littman.  An alternative softmax operator for reinforcement learning.  In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_ , pages 243–252. JMLR. org, 2017.
  * Bagnell et al. [2001] J Andrew Bagnell, Andrew Y Ng, and Jeff G Schneider.  Solving uncertain markov decision processes.  2001\.
  * Brockman et al. [2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba.  Openai gym.(2016).  _arXiv preprint arXiv:1606.01540_ , 2016.
  * Chipman [2014] Susan E. F. Chipman.  _The Oxford Handbook of Cognitive Science_.  Oxford University Press, 11 2014.  ISBN 9780199842193.  URL https://www.oxfordhandbooks.com/view/10.1093/oxfordhb/9780199842193.001.0001/oxfordhb-9780199842193.
  * Cully et al. [2015] Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean-Baptiste Mouret.  Robots that can adapt like animals.  _Nature_ , 521(7553):503–507, 2015.
  * Desai [2017] Nishant Desai.  Uncertain reward-transition mdps for negotiable reinforcement learning.  Technical report, UC Berkeley, Berkeley, California, USA, 2017.
  * Devin et al. [2017] Coline Devin, Abhishek Gupta, Trevor Darrell, Pieter Abbeel, and Sergey Levine.  Learning modular neural network policies for multi-task and multi-robot transfer.  In _2017 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 2169–2176. IEEE, 2017.
  * Do et al. [2008] Amy M Do, Alexander V Rupert, and George Wolford.  Evaluations of pleasurable experiences: The peak-end rule.  _Psychonomic Bulletin & Review_, 15(1):96–98, 2008.
  * Dragan et al. [2013] Anca D Dragan, Kenton CT Lee, and Siddhartha S Srinivasa.  Legibility and predictability of robot motion.  In _Proceedings of the 8th ACM/IEEE international conference on Human-robot interaction_ , pages 301–308. IEEE Press, 2013.
  * Evans et al. [2016] Owain Evans, Andreas Stuhlmüller, and Noah Goodman.  Learning the preferences of ignorant, inconsistent agents.  In _Thirtieth AAAI Conference on Artificial Intelligence_ , 2016.  URL: https://pdfs.semanticscholar.org/31bf/e42e77a572bd83c0529e0f03bc3dc8af52c2.pdf.
  * Fisac et al. [2017] Jaime F Fisac, Monica A Gates, Jessica B Hamrick, Chang Liu, Dylan Hadfield-Menell, Malayandi Palaniappan, Dhruv Malik, S Shankar Sastry, Thomas L Griffiths, and Anca D Dragan.  Pragmatic-pedagogic value alignment.  _arXiv preprint arXiv:1707.06354_ , 2017.  URL: https://arxiv.org/pdf/1707.06354.pdf.
  * Fu et al. [2018] Justin Fu, Katie Luo, and Sergey Levine.  Learning robust rewards with adverserial inverse reinforcement learning.  In _International Conference on Learning Representations_ , 2018.  URL https://openreview.net/forum?id=rkHywl-A-.
  * Grüne-Yanoff [2015] Till Grüne-Yanoff.  Models of temporal discounting 1937–2000: An interdisciplinary exchange between economics and psychology.  _Science in context_ , 28(4):675–713, 2015.
  * Hadfield-Menell et al. [2016] Dylan Hadfield-Menell, Stuart J Russell, Pieter Abbeel, and Anca Dragan.  Cooperative inverse reinforcement learning.  In _Advances in neural information processing systems_ , pages 3909–3917, 2016.
  * Jameson and Kreindler [1973] Antony Jameson and Eliezer Kreindler.  Inverse problem of linear optimal control.  _SIAM Journal on Control_ , 11(1):1–19, 1973\.  URL: https://epubs.siam.org/doi/pdf/10.1137/0311001.
  * Jing et al. [2019] Mingxuan Jing, Xiaojian Ma, Wenbing Huang, Fuchun Sun, and Huaping Liu.  Task transfer by preference-based cost learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 2471–2478, 2019.
  * Kahneman and Tversky [2013] Daniel Kahneman and Amos Tversky.  Prospect theory: An analysis of decision under risk.  In _Handbook of the fundamentals of financial decision making: Part I_ , pages 99–127. World Scientific, 2013.
  * Kalman [1964] Rudolf Emil Kalman.  When is a linear control system optimal?  _Journal of Basic Engineering_ , 86(1):51–60, 1964.  URL: https://asmedigitalcollection.asme.org/fluidsengineering/article-abstract/86/1/51/392203.
  * Lehman et al. [2018] Joel Lehman, Jeff Clune, Dusan Misevic, Christoph Adami, Lee Altenberg, Julie Beaulieu, Peter J Bentley, Samuel Bernard, Guillaume Beslon, David M Bryson, et al.  The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary computation and artificial life research communities.  _arXiv preprint arXiv:1803.03453_ , 2018.
  * Levine and Koltun [2012] Sergey Levine and Vladlen Koltun.  Continuous inverse optimal control with locally optimal examples.  _arXiv preprint arXiv:1206.4617_ , 2012.
  * Mombaur et al. [2010] Katja Mombaur, Anh Truong, and Jean-Paul Laumond.  From human to humanoid locomotion—an inverse optimal control approach.  _Autonomous robots_ , 28(3):369–383, 2010.  URL: https://link.springer.com/content/pdf/10.1007/s10514-009-9170-7.pdf.
  * Ng et al. [2000] Andrew Y Ng, Stuart J Russell, et al.  Algorithms for inverse reinforcement learning.  In _Icml_ , volume 1, page 2, 2000.  URL: http://ai.stanford.edu/~ang/papers/icml00-irl.pdf.
  * Ramachandran and Amir [2007] Deepak Ramachandran and Eyal Amir.  Bayesian inverse reinforcement learning.  In _IJCAI_ , volume 7, pages 2586–2591, 2007.
  * Reddy et al. [2018] Sid Reddy, Anca Dragan, and Sergey Levine.  Where do you think you’re going?: Inferring beliefs about dynamics from behavior.  In _Advances in Neural Information Processing Systems_ , pages 1454–1465, 2018.
  * Regan and Boutilier [2011] Kevin Regan and Craig Boutilier.  Robust online optimization of reward-uncertain mdps.  In _Twenty-Second International Joint Conference on Artificial Intelligence_ , 2011.
  * Sadigh et al. [2016] Dorsa Sadigh, Shankar Sastry, Sanjit A Seshia, and Anca D Dragan.  Planning for autonomous cars that leverage effects on human actions.  In _Robotics: Science and Systems_ , volume 2. Ann Arbor, MI, USA, 2016.
  * Sadigh et al. [2017] Dorsa Sadigh, Anca D. Dragan, Shankar Sastry, and Sanjit A Seshia.  Active preference-based learning of reward functions.  In _Robotics: Science and Systems (RSS)_ , 2017.
  * Shah et al. [2019] Rohin Shah, Noah Gundotra, Pieter Abbeel, and Anca D Dragan.  On the feasibility of learning, rather than assuming, human biases for reward inference.  _arXiv preprint arXiv:1906.09624_ , 2019.  URL: https://arxiv.org/pdf/1906.09624.pdf.
  * Sharot et al. [2007] Tali Sharot, Alison M Riccardi, Candace M Raio, and Elizabeth A Phelps.  Neural mechanisms mediating optimism bias.  _Nature_ , 450(7166):102–105, 2007.
  * Singh et al. [2017] Sumeet Singh, Jonathan Lacotte, Anirudha Majumdar, and Marco Pavone.  Risk-sensitive inverse reinforcement learning via semi-and non-parametric methods.  _arXiv preprint arXiv:1711.10055_ , 2017.
  * Taylor and Stone [2009] Matthew E Taylor and Peter Stone.  Transfer learning for reinforcement learning domains: A survey.  _Journal of Machine Learning Research_ , 10(Jul):1633–1685, 2009.
  * Thompson [1999] Suzanne C Thompson.  Illusions of control: How we overestimate our personal influence.  _Current Directions in Psychological Science_ , 8(6):187–190, 1999.
  * Ziebart et al. [2008] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey.  Maximum entropy inverse reinforcement learning.  In _Aaai_ , volume 8, pages 1433–1438. Chicago, IL, USA, 2008.
  * Ziebart et al. [2010] Brian D Ziebart, J Andrew Bagnell, and Anind K Dey.  Modeling interaction via the principle of maximum causal entropy.  2010\.

##  VII Additional Details for Random MDPs

To calculate the policies corresponding to particular planners with a
particular $\theta$, we perform value iteration using the Bellman equation
until convergence (defined as $||V_{i+1}-V_{i}||_{\infty}<0.001$).

##  VIII Details for Gridworld

In this section, we investigate the effects of irrationality on inference in
an MDP based on OpenAI Gym’s ‘Frozen-Lake-v0’ [5]. This a small 5x5 gridworld
(Fig. 5), consisting of three types of cells: ice, holes, and rewards. The
human can start in any ice cell. At each ice cell, the human can move in one
of the four cardinal directions. With probability $0.8$, they will go in that
direction. With probability $0.2$, they will instead go in one of the two
adjacent directions. Holes and rewards are terminal states, and return the
human back to their start state. They receive a penalty of $-10$ for falling
into a hole and $\theta_{i}\in[0,4]$ ($|\Theta|=25$) for entering into the
$i$th reward cell. As with the random MDPs, to calculate the policies
corresponding to particular planners, we perform value iteration using the
Bellman equation until convergence (defined as
$||V_{t+1}-V_{t}||_{\infty}<0.001$)

To visualize why inference quality is improved, we show examples of how the
policy of several irrational humans differ on the gridworld when the rational
human’s policies are identical in Fig. 9, Fig. 5, and Fig. 10.

Finally, Fig. 6(b) example of why using the wrong model for reward inference
leads to bad inference. In it, the reward inference algorithm assumes that the
human in Boltzmann when it is actually Myopic. The Boltzmann rational agent
would take this trajectory only if the reward at the bottom was not much less
than the reward at the top. The myopic agent with $n\leq 4$, however, only
"sees" the reward at the bottom. Consequently, inferring the preferences of
the myopic agent as if it were Boltzmann leads to poor performance in this
case.

Figure 9: Optimism bias ($\omega=3.16$) produces different actions for
$\theta^{*}=(4,1)$ vs. $\theta^{*}=(1,4)$ in the states shown: the rational
policy is to go away from the hole regardless of $\theta$, but an optimistic
human takes the chance and goes for the larger reward – up in the first case,
down in the second. Figure 10: Boltzmann ($\beta=100$) produces different
policies for $\theta^{*}=(1,1)$ vs. $\theta^{*}=(4,4)$: when $||\theta||$ is
larger, the policy becomes closer to that of the rational human, as the
differences in $Q$-values becomes larger.

##  IX Details for Autonomous Driving Environment

Even if known irrationality helps reward inference both empirically and
theoretically in small MDPs, the question still remains whether this result
will matter in practice. As a result, we investigate the effect of human
irrationality on reward inference in the self-driving car domain [28].

Simulation environment. As in previous work in the car domain [28, 29], we
model the dynamics of cars using a point-mass model. The state of each car is
a 4-dimensional vector ${s}=[x~{}y~{}h~{}v]$, where $x,y$ are the coordinates
of the car, $h$ is the heading, and $v$ is the speed. The control input for
the car is a two dimensional vector $a=[u_{1}~{}u_{2}]$, where $u_{1}$ is the
steering input and $u_{2}$ is the acceleration. We also include a friction
coefficient $\alpha$. The dynamics model of the vehicle is:

$[\dot{x}~{}\dot{y}~{}\dot{h}~{}\dot{v}]=[v\cdot\cos{h}~{}~{}~{}~{}v\cdot\sin{h}~{}~{}~{}~{}v\cdot
u_{1}~{}~{}~{}~{}u_{2}-\alpha\cdot v].$

For ease of simulation, we discretized the simulation along the time dimension
using the following dynamics:

$\displaystyle\Delta{{s}}_{t}$
$\displaystyle=[\Delta{x}_{t}~{}\Delta{y}_{t}~{}\Delta{h}_{t}~{}\Delta{v}_{t}]$
$\displaystyle=[\bar{v}_{t}\cos{h_{t}}dt~{}~{}\bar{v}_{t}\sin{h_{t}}dt~{}~{}\bar{v}_{t}u_{1}dt~{}~{}(u_{2}-\alpha
v_{t})dt],$

where $\Delta{{s}}_{t}={s}_{t+1}-{s}_{t}$ and $\bar{v}_{t}=v_{t}+0.5\cdot
u_{2}\cdot dt$. In our experiments, we used $dt=0.1s$.

Reward functions in this environment are assumed to be a linear combination of
features:

$r_{\theta}(s,a)=\theta^{\top}f(s,a),~{}~{}f:S\times
A\rightarrow\mathbb{R}^{n}$

Due to environment complexity, we can’t solve for optimal trajectories in our
environment directly. Instead, we suppose that the planner of is performing
Model Predictive Control (MPC) at every iteration – that is, it will plan a
finite horizon sequence of actions to maximize its reward, execute the first
action in the sequence, then replan. As a basic model of irrationality, we
consider shorter planning horizons. We assume that the reward inference
procedure knows the planninng horizon exactly.

##### Bayesian IRL in the car environment

First, we consider the analogue of the results in Sec. III-B in the car
environment: what is the log-loss of the posterior on $\theta$, given
different planning horizons? Since the demonstrations used in this work are
generated via trajectory optimization (and not an inherently stochastic
process), using Bayesian IRL requires us to specify a “fake" observation model
$P(\xi|\theta)$.

We use the following distribution, normal in feature space, for
$P(\xi|\theta)$:

$P(\xi|\theta)\propto
e^{\left(E[f(\xi)]-E[f(\xi^{*}_{\theta}]\right)^{T}\Sigma_{\xi_{\theta}^{*}}^{-1}\left(E[f(\xi)]-E[f(\xi^{*}_{\theta}]\right)},$

where $E[f(\xi^{*}_{\theta}]$ is the expectation of the features of the
optimal trajectory $\xi^{*}_{\theta}$ for $r_{\theta}$.

##### Maximum likelihood IRL in cars

In practice, full Bayesian inference is completely intractable for complicated
domains such as cars. Instead, the state of the art for reward inference are
approximate, maximum-likelihood estimate (MLE) based methods.To study the
effects of irrationality on MLE-based inference methods, we also perform
Levine and Koltun’s Continuous IOC with Locally Optimal Examples (CIOC) [22],
which uses a Boltzmann model of the human with a second order Laplace
approximation of the normalizing constant. Since our humans are actually
locally optimal, we rectify the bias induced by the Boltzmann model by using a
large $\beta=10^{4}$. To rectify the issue of local optimization in CIOC, we
initialize the optimization procedure with the true reward weights.

As CIOC returns an (approximate) MLE estimate for the reward parameter
$\theta$, we cannot use the log-loss metric. Instead, we evaluate the reward
functions by the cosine similarity of $\theta^{*}$ and $\hat{\theta}$, as well
as the normalized return of the trajectory when optimizing the recovered
$\hat{\theta}$. (We normalize the returns so that the optimal trajectory has
return $1$ and the trajectory that goes forward at constant speed $0$.)

Driving Scenario: Merging Our experiments were performed in a simple merging
environment (Fig. 1). In it, the human wants to merge into the right lane
while trying to maintain its $1.2$ forward speed. In addition to the human
car, the right lane contains two constant velocity cars, traveling at $0.8$
speed. The features of this environment are composed of a squared penalty for
deviating from $1.2$ forward speed, features for the squared distances to the
medians of each of the lanes, a feature for the minimum squared distance to
any of the medians of the lanes, and a smooth collision feature.

For the Bayesian IRL scenario, we considered four different reward functions
on this domain, consisting of varying the weight on the penalty for deviating
from the target speed. All the other weights are unchanged. In particular, we
considered $\theta_{\textrm{speed}}\in\\{0.5,1,2,4\\}$. In the CIOC scenario,
we used $\theta_{\textrm{speed}}=1$. We also consider 3 different planning
horizons: $h=3$, $h=5$, and $h=7$. This means we had 12 conditions for
Bayesian IRL and 3 for CIOC.

##  X Proofs for Sec. IV-C

First, note that for deterministic planners, there that there are $|A|^{|S|}$
such policies, and thus we have $\mathbf{I}(\theta;d(\theta))\leq
H(d(\theta))\leq|S|\log|A|$ (as the entropy of a discrete random variable $X$
is bounded above by the logarithm of the size of its support $|X|$).
Similarly, we also have that $\mathbf{I}(\theta;d(\theta))\leq
H(\theta)\leq\log|\Theta|$. For deterministic planners, we thus have

$\displaystyle\mathbf{I}(\theta;d(\theta))\leq\min(\log|\Theta|,|S|\log|A|),$

We will now prove a stronger version of proposition 3.

######  Proposition 3.

There exists a family of URMDPs with state and action spaces of any size, such
that the rational planner provides no information, and there exists a
deterministic planner that provides $min(\log|\Theta|,|S|\log|A|)$ bits worth
of information.

###### Proof.

Consider a set of environments where $\Theta=\\{1,2,...,|A|^{|S|}\\}$, the
prior $p$ is uniform over $\Theta$ (and thus $H(\theta)=\log|\Theta|$), and
where

$\displaystyle r_{\theta}(s,a,s^{\prime})$
$\displaystyle=\begin{cases}\theta&a=a^{*}\\\ 0&a\not=a^{*}\end{cases}$

for some $a^{*}$. Then the unique optimal policy has $\pi(s)=a^{*}$ for every
$\theta$ (as $\theta>0$). This implies that

$\mathbf{I}(\theta,d_{\textrm{Rational}}(\theta))\leq
H(d_{\textrm{Rational}}(\theta))=0.$

However, the planner $d^{\prime}:\theta\mapsto\pi^{(\theta)}$, with some fixed
some ordering $\\{\pi^{(i)}\\}_{i\in\\{1,2,...,|A|^{|S|}\\}}$ of the possible
policies, satisfies $H(\theta|d^{\prime}(\theta))=0$ and thus
$\mathbf{I}(\theta;d^{\prime}(\theta))=H(\theta)=\log|\Theta|=|S|\log|A|$. ∎

By choosing $A$ such that $|A|\geq|\theta|^{1/|S|}$, we get the Proposition 3
in the text.

######  Proposition 4.

There exists a family of one-state two-action MDPs, with arbitrarily large
$|\Theta|$ such that
$\mathbf{I}(\theta,d_{\textrm{Boltz}}(\theta))=\log|\Theta|$ and
$\mathbf{I}(\theta,d_{\textrm{Rational}}(\theta))=0$.

###### Proof.

Suppose first action $a_{1}$ has reward $\theta$, where
$\theta\in\\{1,2,...,-|\Theta|\\}$, and the second action $a_{2}$ has reward
$0$. Then the rational planner $d_{\textrm{Rational}}$ will always return a
policy that always takes the first action, while $d_{\textrm{Boltz}}(\theta)$
is the policy

$\displaystyle\pi_{\theta}(a_{1}|s_{1})$
$\displaystyle=\frac{e^{\beta\left(\theta+V(s_{1})\right)}}{e^{\beta\left(\theta+V(s_{1})\right)}+e^{\beta
V(s_{1})}}=\frac{1}{1+e^{-\beta\theta}}$
$\displaystyle\pi_{\theta}(a_{2}|s_{1})$
$\displaystyle=\frac{1}{1+e^{\beta\theta}}$

This mapping is injective, and so $H(\theta|d_{\textrm{Boltz}}(\theta))=0$ and
thus $\mathbf{I}(\theta;d_{\textrm{Boltz}}(\theta))=H(\theta)=\log|\Theta|$. ∎

##  XI More theory

##### Why recover the reward parameter?

Even if irrationalities can help with inference, a natural question why we
wish to infer the reward. If the actual goal is acting optimally, a rational
human would be great because we can just imitate them. In addition to the fact
that real people are not perfectly rational, the reason we focus on inference
is that imitation is not enough in some cases, e.g. if we need to behave in
new environments [33, 9, 18], or transfer the policy to a robot with different
dynamics [7, 14, 26].

In fact, there are cases where being robust to changes in dynamics requires
identifying the reward parameters. Indeed, there exist such cases where it is
impossible to identify the parameters from rational humans, but possible from
Boltzmann humans:

######  Proposition 5.

There exists an URMDP $M$ and a set of new transition probabilities
$\\{\\{P_{{s},{a}}^{(i)}\\}\\}_{i\in\\{1,2,...,|\Theta|\\}}$ such that
$\log|\Theta|$ bits of information are needed to compute the optimal policy
under all the transition probabilities; and where
$\mathbf{I}(\theta,d_{\textrm{Rational}}(\theta))=0$, whereas
$\mathbf{I}(\theta,d_{\textrm{Boltz}}(\theta))=\log|\Theta|$.

###### Proof.

We construct a 2 state, 2 action URMDP. Label the two states ${s}_{1}$ and
${s}_{2}$. Let $\Theta=\\{1,2,...,|\Theta|\\}$, the prior be uniform over
$\Theta$, and let $r(s,a,s_{1})=\theta$ for all $s,a$ and
$r(s,a_{1},s_{2})=0$, $r(s,a_{2},s_{2})=-1$ for all $s$.
Under the original transition probabilities $\\{P_{{s},{a}}\\}$, $a_{1}$ leads
deterministically to $s_{1}$ while $a_{2}$ deterministically leads to $s_{2}$.
Then, the unique optimal policy is $\pi(s)=a^{*}$ for every $\theta$ (as
$\theta>0$). This implies that
$\mathbf{I}(\theta,d_{\textrm{Rational}}(\theta))=0$. On the other hand,
$d_{\textrm{Boltz}}$ is an injective map (as in Proposition 2), and thus
$\mathbf{I}(\theta;d_{\textrm{Boltz}}(\theta))=\log|\Theta|$.
Now, we construct $\\{P_{{s},{a}}^{(i)}\\}$ such that $a_{1}$ is optimal for
$\theta\leq i-1$, and $a_{2}$ is optimal for $\theta\geq i$. First, let
$a_{1}$ deterministically lead to $a_{2}$, thus, leading to reward 0. Next,
let $a_{2}$ lead to $s_{1}$ with probability $\frac{2}{2i+1}$. Therefore:

$\displaystyle E_{s^{\prime}\sim
P_{s,a_{2}}^{(i)}}[r_{\theta}(s,a_{2},s^{\prime})]$
$\displaystyle=\frac{2}{2i+1}\theta-\frac{2i-1}{2i+1}$

Then for $\theta\geq i$, we have $E_{s^{\prime}\sim
P_{s,a_{2}}^{(i)}}[r_{\theta}(s,a_{2},s^{\prime})]\geq\frac{1}{2i+1}>0$ and
for $\theta\leq i-1$, we have $E_{s^{\prime}\sim
P_{s,a_{2}}^{(i)}}[r_{\theta}(s,a_{2},s^{\prime})]\leq-\frac{1}{2i+1}<0$.
By construction, in order to compute the optimal policy for every new
transition probabilities $\\{P_{{s},{a}}^{(i)}\\}$, we must know the value of
$\theta$, and thus need $\log|\Theta|$ bits of information. ∎