ifaamas [AAMAS ’24]Proc. of the 23rd International Conference on Autonomous
Agents and Multiagent Systems (AAMAS 2024)May 6 – 10, 2024 Auckland, New
ZealandN. Alechina, V. Dignum, M. Dastani, J.S. Sichman (eds.) 2024 2024 926
Harvard University Cambridge USA Harvard University Cambridge USA Harvard
University Cambridge USA Harvard University Cambridge USA Harvard University
Cambridge USA

# Reinforcement Learning Interventions on Boundedly Rational Human Agents in
Frictionful Tasks

Eura Nofshin<EMAIL_ADDRESS>,  Siddharth Swaroop  ,  Weiwei Pan  ,
Susan Murphy  and  Finale Doshi-Velez

###### Abstract.

Many important behavior changes are _frictionful_ ; they require individuals
to expend effort over a long period with little immediate gratification. Here,
an artificial intelligence (AI) agent can provide personalized interventions
to help individuals stick to their goals. In these settings, the AI agent must
personalize _rapidly_ (before the individual disengages) and _interpretably_ ,
to help us understand the behavioral interventions. In this paper, we
introduce Behavior Model Reinforcement Learning (BMRL), a framework in which
an AI agent intervenes on the parameters of a Markov Decision Process (MDP)
belonging to a _boundedly rational human agent_. Our formulation of the human
decision-maker as a planning agent allows us to attribute undesirable human
policies (ones that do not lead to the goal) to their maladapted MDP
parameters, such as an extremely low discount factor. Furthermore, we propose
a class of tractable human models that captures fundamental behaviors in
frictionful tasks. Introducing a notion of _MDP equivalence_ specific to BMRL,
we theoretically and empirically show that AI planning with our human models
can lead to helpful policies on a wide range of more complex, ground-truth
humans.

###### Key words and phrases:

Reinforcement learning; Personalization; Agent-based modeling of humans;
Bounded rationality

##  1\. Introduction

In many AI+human applications of behavior change, AI agents assist the human
in performing _frictionful_ tasks, where making progress toward the human’s
goal requires sustained effort over time with little immediate gratification.
Examples include physical therapy (PT) programs, adherance to scheduled
medication, or passing an online course. Two key challenges for AI agents in
these settings are rapid personalization (Wang et al., 2021; Park and Lee,
2023; Tabatabaei et al., 2018) and learning interpretable policies for
intervention (Trella et al., 2022; Xu, 2022). In frictionful tasks, since
effort exerted by the human does not reap immediate benefits, the AI agent
must learn a personalized intervention policy for each human in a small number
of interactions, or risk disengagement. These policies must also be
interpretable to experts in behavioral science so that they can discover which
interventions work for which individuals, and investigate why.

Grounded in behavioral literature that treats humans as sequential decision-
makers (e.g. (Taylor et al., 2021a, b; Niv, 2009; Shteingart and Loewenstein,
2014; Zhou et al., 2018)), we model the human as an agent planning under a
“maladapted” Markov Decision Process (MDP). In maladapted human MDPs, the
optimal policy does not reach the human’s stated goal; for example, in
physical therapy (PT), the goal may be a rehabilitated shoulder and the
maladapted MDP parameter may be an extremely low discount rate, $\gamma$. This
results in myopic decision-making, wherein an individual forgoes the long-term
goal (rehabilitated shoulder) to avoid experiencing friction in the short-term
(unpleasantness of PT). The AI agent helps the individual achieve their long-
term goal by changing the maladapted human MDP (and thereby the optimal
policy).

While there is existing reinforcement learning (RL) literature for optimizing
interventions on human utility functions (i.e. reward) in maladapted MDPs (Yu
and Ho, 2022; Zhou et al., 2018; Mintz et al., 2023), interventions on
$\gamma$ have not been optimized from an RL perspective. On the other hand, in
behavioral science, humans have been observed to use a problematically low
$\gamma$ (Story et al., 2014) and scientists have developed interventions to
change a human’s $\gamma$ (e.g. (Magen et al., 2008)). However, no work
optimizes for _when_ and with what mechanisms to intervene on the parameters
of the human’s maladapted MDP.

In this paper, we introduce a _flexible_ and _behaviorally interpretable_
framework called “Behavior-Model RL” (BMRL). In BMRL, the human is modeled as
an RL agent, whose actions are _behaviors_ , such as performing or skipping
PT; the AI agent provides personalized assistance by delivering
_interventions_ on the human’s maladapted MDP parameters. By linking the
behaviors of our human agents to their MDP parameters, BMRL allows us to
_interpret_ the mechanism behind the human’s maladapted decision-making. Our
framework is also more _flexible_ than existing ones, since we allow the AI
agent’s actions to include operations on any part of the human MDP (such as
$\gamma$). By solving for the AI agent’s optimal policy, we learn the best set
of interventions to change the human agent’s behavior and to help the human
reach their goal.

Unfortunately, current RL approaches have two major drawbacks when used to
solve for the optimal AI agent policy in BMRL. First, most planning methods
are too data-intensive for our setting, in which personalization occurs
online. For example, online algorithms in robotics require thousands of
interactions to learn reasonable policies (e.g. in (Yang et al., 2020; Thabet
et al., 2019; Tebbe et al., 2021)), but in frictionful tasks, we are limited
to _tens to hundreds_ of interactions (Trella et al., 2022). Second, existing
planning methods model the human as a black-box transition or value function.
Unfortunately, in learning black-box representations of the human agent, we
lose the ability to interpretably attribute human behavior to their MDP
parameters.

In this paper, we propose a tractable planning method for the AI agent in our
BMRL framework. Our method provides the AI agent with a useful inductive bias,
in the form of a human model that captures important behavioral patterns in
frictionful tasks. Specifically, we identify a small, behaviorally-grounded
model of the human that the AI agent can leverage to rapidly personalize
interventions, including previously under-explored interventions on $\gamma$.
Then, we introduce the concept of “AI equivalence” to identify a class of more
complex human models for which AI policies learned in our simple human model
can be lifted with no loss of performance. In our empirical analysis, we test
whether AI planning with our small model is robust to complex human models
that are not covered by our equivalence result. Throughout all of this, our
small model preserves scientific interpretability– in fact, it has an
analytical solution for the human behavior policy– which allows experts to
inspect and learn from the AI policies.

##  2\. Related Works

_Computational modeling of human behaviors._ Behavioral scientists have
developed and verified several computational models of _dynamic_ human
decision-making. Unlike _static_ models, such as Social Cognitive Theory
(Bandura, 1999), _dynamic_ models of decision-making apply to interactive
human-AI settings, since they capture person-level variation and changes over
time, as in Zhang et al. (2022). Scientists developed these models to explain
_offline data_ from frictionful settings such as health (e.g. (Martin et al.,
2018; Zhang et al., 2021; Wang et al., 2021)), energy (Mogles et al., 2018),
and experience sampling (Khanshan et al., 2023) or to capture broader
behaviors such as risk (Liu et al., 2019) and adherence (Pirolli, 2016).
However, these models involve too many latent variables– corresponding to
internal human processes– to facilitate rapid AI learning from _online data_.
In contrast, we propose a minimal, behaviorally-grounded model, one whose set
of latent parameters is small and structured enough that our AI can learn.

_Computational modeling of human agent deficiencies._ RL is frequently used to
model the complex mechanisms underlying human behavior, from the firing of
dopaminergic neurons in the brain (e.g. in (Niv, 2009; Shteingart and
Loewenstein, 2014)) to frictionful tasks such as mindful eating (Taylor et
al., 2021a), weight loss (Aswani et al., 2019), and smoking cessation (Taylor
et al., 2021b). Although these works use RL to model humans, the models
themselves are not used to enrich planning for an AI agent. One exception is
inverse reinforcement learning, in which the AI agent infers the human agent’s
rewards (e.g. (Zhi-Xuan et al., 2020; Brown et al., 2019)), transitions (Reddy
et al., 2018), discount factor (Giwa and Lee, 2021), or entire MDP (Shah et
al., 2019; Evans et al., 2016; Jarrett et al., 2021), but does not _intervene_
on the parts of the human MDP that are maladapted. When the AI agent does
intervene, the changes are limited to the human’s reward (Yu and Ho, 2022;
Tabrez et al., 2019; Zhou et al., 2018; Mintz et al., 2023) or states (Chen et
al., 2022; Reddy et al., 2021). Our BMRL framework is flexible enough to
incorporate AI interventions on multiple parts of the human MDP, including the
discount factor or transitions.

_Equivalence of (human) MDP models._ In RL, there are notions of equivalence
that can reduce larger human MDPs to smaller, more manageable ones.
Equivalence, as defined in bisimulations (Givan et al., 2003), homomorphism
(Ravindran and Barto, 2002), and approximate homomorphisms (e.g. (Ravindran
and Barto, 2004; van der Pol et al., 2020)), requires that one human MDP
strictly preserves the transition and reward functions of another, given a
mapping between the state and action spaces. State abstraction methods, which
equate the optimal value function between two human-level MDPs, are less
strict (Li et al., 2006). However, these equivalences are still _stricter than
necessary_ in our setting, where we only care that the human MDPs are similar
enough that the AI agent policy will not differ. Furthermore, the simpler MDPs
recovered by these methods are not guaranteed to be behaviorally valid or
interpretable. In our approach, we define two human MDPs as equivalent if they
lead to the same _AI optimal policies_ , and we use this definition to build
up to more complex human MDPs from a behaviorally interpretable one.

##  3\. The Behavior Model RL (BMRL) Framework for AI Interventions

Figure 1. Overview of BMRL. The human agent interacts with the environment as
in standard RL. The AI agent’s actions affect the human agent. The human agent
+ environment form the AI environment.

We define a formal framework, called BMRL, in which an AI agent learns to
intervene on a human agent’s maladapted MDP parameters (overview in fig. 1).

###  3.1. Assumptions on human agent

In BMRL, human agents perform optimal planning on (subconscious) knowledge of
their MDP,

$\mathcal{M}_{h}=\langle\mathcal{S}_{h},\mathcal{A}_{h},T_{h},R_{h},\gamma_{h},s_{g},s_{d}\rangle,$
(1)

where $s_{g},s_{d}\in\mathcal{S}_{h}$ are absorbing goal (e.g. a rehabilitated
shoulder) and disengagement states (e.g. quitting PT).

Though in general, it is possible for the human’s perception of the states
$\mathcal{S}_{h}$, actions $\mathcal{A}_{h}$ and transitions $T_{h}$ to be
maladapted, in this paper we assume that the human’s perception matches the
true environment. On the other hand, we allow the human’s rewards $R_{h}$ and
discount $\gamma_{h}$ to vary by perception. For example, one may skip PT
because of a tendency to ignore long-term rewards (low $\gamma_{h}$) while
another may skip PT because they find the workout to be extremely unpleasant
(bad $R_{h}$).

We assume that at any point the human subconsciously “knows” their own MDP,
solves for the optimal policy, and uses it to select actions. In future work,
BMRL can extend to sub-optimal human planning. Despite being optimal, our
human agents are still boundedly rational because their MDP is maladapted.
That is, under certain values of $\gamma_{h},R_{h}$, even an optimal human
policy will _never_ lead to the goal state (e.g. if the path to the goal
reward is laced with extremely negative rewards). The existence of maladapted
MDPs in humans is shown in behavior science, where myopic discounting has been
linked to excessive alcohol intake (Story et al., 2014) or miscalibrated
rewards have been linked to unhealthy eating (Taylor et al., 2021a). Despite
subconscious knowledge of their own MDP, our human agents are still boundedly
rational because (1) they may not be _conscious_ of their deficiencies and
unable to target them; (2) even if aware, they may still struggle to change
their deficiencies. In both cases, behavioral interventions (delivered by the
AI agent) can help.

###  3.2. AI agent

Our AI agent encourages the human agent toward the goal by intervening on the
human’s decision-making parameters, such as $\gamma_{h}$. To do so, the AI
agent plans according to an MDP,

$\mathcal{M}_{AI}=\langle\mathcal{S}_{AI},\mathcal{A}_{AI},T_{AI},R_{AI},\gamma_{AI}\rangle,$
(2)

with known rewards $R_{AI}$ and unknown transitions $T_{AI}$.

Upon observing state $s_{AI}=[s_{h},a_{h}]$, which consists of the human’s
current state and _previous_ action, the AI agent must decide whether to
intervene on the human’s discount ($a_{AI}=a_{\gamma}$), reward
($a_{AI}=a_{R}$), or to do nothing ($a_{AI}=0$). In practice, a discounting
intervention $a_{\gamma}$ could be “episodic future thinking,” where
individuals imagine future events as if they are presently occurring (Brown
and Stein, 2022); this could executed as a guided activity in-app. A common
intervention on reward $a_{R}$ is to offer extrinsic rewards, such as badges
(Fanfarelli et al., 2015). Domain experts would determine how the
interventions are executed, e.g. if the burden intervention should be a badge,
motivational message, or cash.

To encourage policies that quickly lead to the goal state, the AI agent
receives a positive reward when the human reaches the goal state, a negative
reward when the human disengages, and a negative reward for the “cost” of
intervening. The AI’s transitions factorize into two probability
distributions,
$T_{AI}(s_{AI},a_{AI},s_{AI}^{\prime})=P(s_{h}^{\prime}|s_{h},a_{h})P(a_{h}^{\prime}|s_{h},a_{AI})=T_{h}(s_{h},a_{h},s_{h}^{\prime})\pi_{h}(a_{h}^{\prime}|s_{h},a_{AI})$.
The first distribution refers to the human-level transitions $T_{h}$. The
second distribution is over human actions; it is the human policy that results
from the AI’s intervention on the human’s MDP. Importantly, we assume that the
effect of AI actions on the human MDP is _temporary_. For example, if the AI
agent increases the human’s discount factor $\gamma_{h}$ to
$\gamma_{h}^{\prime}$ in the current time step, the human’s discounting will
have reverted to $\gamma_{h}$ at the next time step.

In table 1, we provide a comparison on what the AI and human agents separately
know and observe. Note that all of the AI agent’s unknown parameters pertain
to the human MDP $\mathcal{M}_{h}$ and are contained in the AI’s transitions
$T_{AI}$. Instead of explicitly learning $\mathcal{M}_{h}$ to form $T_{AI}$,
we could directly estimate $T_{AI}$ or $Q_{AI}^{*}$ using standard model-based
or model-free techniques. However, by learning $\mathcal{M}_{h}$, we take
advantage of the known structure of the problem; the better the AI’s model of
$\mathcal{M}_{h}$, the better the inductive bias for forming $T_{AI}$ (and
therefore $\pi_{AI}^{*}$).

| Human agent | AI agent
---|---|---
Knows… | $\mathcal{S}_{h},\mathcal{A}_{h},T_{h},R_{h},\gamma_{h}$ | $\mathcal{S}_{AI},\mathcal{A}_{AI},R_{AI}$
Does not know… | — |  $T_{AI}$ (includes $T_{h},R_{h},\gamma_{h}$)
Observes… | $\mathcal{S}_{h},\mathcal{A}_{h},\mathcal{A}_{AI}$ | $\mathcal{S}_{h},\mathcal{A}_{h},\mathcal{A}_{AI}$
Table 1. Overview of what is known, unknown, and observable to the human and
AI agent. the AI agent does not know (and must infer) the human agent’s MDP
($R_{h},\gamma_{h}$) and the true environmental transitions ($T_{h}$).

##  4\. Rapid personalization in BMRL via a simple human model

###  4.1. Chainworlds: a simple human model that captures progress-based
decision-making

In this section we define _chainworlds_ , a class of simple human MDPs that
the AI agent will use as a stand-in model for the _true_ human decision-making
process. Chainworlds are based on the observation that many frictionful tasks
contain a notion of human progress toward a goal; for example, in PT, the
progress toward a rehabilitated shoulder may be summarized by the current
strength of the joint. We summarize these “progress-based” settings with a
“progress-only” class of human MDPs, shown in fig. 2, which we call
_chainworlds_ and denote $\mathscr{M}_{\text{chain}}$.

(a) $a_{h}=0$

(b) $a_{h}=1$

Figure 2. Graphical representation of the chainworld.

Each element of $\mathscr{M}_{\text{chain}}$ is as follows:

  * •

_States $s_{h}\in\\{s_{0},s_{1},\ldots,s_{N}=s_{g},s_{d}\\}$._ The $N$ states
are 1-D, discrete, and represent progress toward the goal. The goal state at
the end of the chain, $s_{N}=s_{g}$ means that the human has rehabilitated
their shoulder. The disengagement state $s_{d}$ means that the human has
disengaged from PT.

  * •

_Actions $a_{h}\in\\{0,1\\}$._ The human decides to perform ($a_{h}=1$) or not
perform ($a_{h}=0$) the goal-directed behavior. In the future, this could be
extended to categorical actions. That said, many important applications have
binary actions, such as ”exercise or not” in PT, ”smoke or not” in smoking
cessation, and ”adhered or not” in medication adherence.

  * •

_Rewards._ The human’s utility function is the reward,

$R_{h}(s,a,s^{\prime})=\begin{cases}r_{b},&a=1\\\ r_{\ell},&s^{\prime}<s\\\
r_{g},&s=s_{g}\\\ r_{d},&s=s_{d}.\end{cases}$ (3)

Goal behaviors, such as doing PT, incur a cost representing burden $r_{b}<0$.
Similarly, losing progress incurs $r_{\ell}<0$. The goal and disengagement
states have positive utility, $r_{g}>0$ and $r_{d}>0$.

  * •

_Transitions._ The human knows that there is $p_{g}$ probability that they
will move toward the goal as a result of the behavior, $p_{\ell}$ probability
that they will lose progress from abstaining, and $p_{d}$ probability that
they will disengage from abstaining. These probabilities are fixed across
states, except for the first state $s_{0}$, which has a separate probability
of disengagement $p_{d0}\geq p_{d}$.

  * •

_Discount._ The human exponentially discounts future rewards via
$\gamma_{h}\in[0,1)$. We leave other behaviorally relevant forms of
discounting, such as hyperbolic discounting (Fedus et al., 2019), as future
work.

  * •

_Effect of AI interventions._ When $a_{AI}=a_{\gamma}$ the human’s discount
$\gamma_{h}$ _increases_ by $\Delta_{\gamma}>0$, and when $a_{AI}=a_{b}$ the
human’s burden $r_{b}<0$ _decreases_ by $\Delta_{b}$. We clip
$\gamma_{h}+\Delta_{\gamma}$ to be between $0$ and $1$.

Each individual is an instance of the chainworld,
$\mathcal{M}_{\theta}\in\mathscr{M}_{\text{chain}},$ with parameters $\theta$
= $\\{r_{b}$, $r_{\ell}$, $r_{g}$, $r_{d}$, $p_{g}$, $p_{\ell}$, $p_{d}$,
$p_{d0}$, $\gamma_{h}$, $\Delta_{\gamma}$, $\Delta_{b}\\}$. For example, some
people tend to prioritize short-term rewards (with a low $\gamma_{h}$) while
others prioritize long-term rewards (with a high $\gamma_{h}$). The parameters
$\theta$ must be inferred by the AI.

_Closed-form Solutions for Human Policies in Chainworlds._ Chainworlds are
inspectable to behavioral experts because there is an analytical solution for
the optimal value function (all derivations in appendix A). For a chainworld
MDP $\mathcal{M}_{\theta}\in\mathscr{M}_{\text{chain}}$, the optimal value
function maximizes between the value of a policy that always pursues the goal,
$\pi_{g}(s_{n})=1$, and the value of a policy that always chooses to
disengage, $\pi_{d}(s_{n})=0$, where $s_{n}$ for $n\in{0,\ldots,N}$ refers to
the $n$-th state on the chain. The value of goal pursuit is,

$\displaystyle V^{\pi_{g}}_{\theta}(s_{n})=r_{g}\left(\frac{\gamma
p_{g}}{z}\right)^{N-n}+r_{b}\left(\frac{1-(\gamma
p_{g}/z)^{N-n}}{1-\gamma}\right),$ (4)

where $z=1-\gamma(1-p_{g})$. The value of goal pursuit,
$V^{\pi_{g}}_{\theta}(s_{n})$, trades off between the long-term utility of the
goal (the $r_{g}$ term) and the burden one accumulates to get there (the
$r_{b}$ term). The value of disengagement is,

$\displaystyle\begin{split}V^{\pi_{d}}_{\theta}(s_{n})=r_{d}\left(\frac{\gamma\
p_{d0}}{v}\right)\left(\frac{p_{\ell}\ \gamma}{u}\right)^{n}+(\gamma\ p_{d}\
r_{d}+p_{\ell}\ r_{\ell})\left(\frac{1-(\gamma
p_{\ell}/u)^{n}}{1-\gamma(1-p_{d})}\right),\end{split}$ (5)

where $v=1-\gamma(1-p_{d0})$ and $u=1-\gamma(1-p_{d}-p_{\ell})$. The first
term in the equation (with $r_{d}$), represents the value of disengagement
from state $0$, after having lost all prior progress. The second term
represents the value of disengagement after state $0$, which factors in the
cost of disengagement $r_{d}$ and of losing progress $r_{\ell}$.

These equations allow us to hypothesize about the diverse space of AI actions
that will encourage the human towards the goal, such as actions to increase
the human’s level of motivation (increasing $r_{g}$) or that highlight the
consequences of quitting (decreasing $r_{d}$).

###  4.2. Different humans yield different AI policies

At this point, we have fully specified an AI MDP as defined in section 3.2, in
which the human MDP is a chainworld
$\mathcal{M}_{\theta}\in\mathscr{M}_{\text{chain}}$. Solving this AI MDP will
yield an optimal AI policy, which is the best intervention plan for a given
human with parameters $\theta$. Importantly fig. 3 demonstrates that
personalization is necessary because humans with different $\theta$ require
different optimal AI policies.

(a) Highly myopic human ($\boldsymbol{\gamma=0.1}$) with high burden
($\boldsymbol{r_{b}=-2}$).

(b) Highly myopic human ($\boldsymbol{\gamma=0.1}$) with low burden
($\boldsymbol{r_{b}=-0.3}$).

Figure 3. Example of different optimal AI policies for two humans with
different chainworld parameters. Each square is a chainworld state. An $a_{b}$
means AI should select action to reduce $r_{b}$, while $a_{\gamma}$ means AI
should select action to increase $\gamma$. Red solid and blue dotted lines
show start and end of intervention window.

##  5\. Theoretical analysis: When is chainworld good enough?

In this section, we define an _equivalence class_ of more complex human MDPs
for which an AI agent that plans with the chainworld can still learn the
optimal policy.

######  Definition  (AI equivalence of human MDPs).

We consider two human MDPs $\mathcal{M}_{h}\equiv\widehat{\mathcal{M}}_{h}$
under state mapping $f:\mathcal{S}_{h}\rightarrow\widehat{\mathcal{S}}_{h}$
and action mapping $g_{s}:\mathcal{A}_{h}\rightarrow\widehat{\mathcal{A}}_{h}$
if the corresponding optimal AI policies are equal, so that
$\pi_{AI}^{*}\left(\ [s_{h},a_{h}]\ \right)=\widehat{\pi}_{AI}^{*}\left(\
[f(s_{h}),g_{s_{h}}(a_{h})]\ \right)$ for all
$[s_{h},a_{h}]\in\mathcal{S}_{AI}$.

The state mapping $f$ and (state-specific) action mapping $g_{s}$ let us map
from the state and action space of the one MDP to the other. In terms of the
chainworld, our definition states that if the optimal AI action in the
chainworld MDP is the same as the optimal AI action in the true MDP for all
states (after applying the mappings), then the two are equivalent.

Our equivalence in section 5 is not as strict as the homomorphisms
equivalence. Unlike homomorphisms, we _do not_ seek human MDPs that have the
same rewards and transitions as chainworld. In fact, we do not even seek MDPs
that result in the same optimal human policy as chainworld. Instead, we only
care that the two human MDPs are similar enough to result in the same _optimal
AI policy._ As a result, we get the largest set of human MDPs that admits
simple planning of optimal interventions by the AI agent.

###  5.1. Optimal AI policies for chainworld MDPs

Under section 5, the class of MDPs that is equivalent to chainworlds is
determined by the space of AI policies that chainworlds can express. In this
section, we show that all chainworld MDPs
$\mathcal{M}_{\theta}\in\mathscr{M}_{\text{chain}}$ result in AI optimal
policies that follow a “three-window format,” which we refer to as
$\bar{\Pi}$. Throughout this section, we describe the AI policy in terms of
the chainworld states, $s_{n}$, where $n$ refers to $n$-th state on the chain;
even though the _previous_ human actions are technically part of the AI state,
they do not affect the _best current_ action in the AI’s optimal policy.

A “three-window” AI policy consists of: window 1 (no intervention is effective
enough to make human perform the behavior), window 2 (intervention window),
and window 3 (human performs behavior without intervention). Two examples are
in fig. 3. The size of these windows varies and may even be $0$. For example,
if the interventions have no effect ($\Delta_{\gamma}=0,\Delta_{b}=0$) then
the intervention window will be size $0$. The three windows are a consequence
of how the AI’s action affects the human’s optimal policy; when the AI agent
intervenes on the human, it changes the human’s MDP parameters, which in turn,
might change the human’s optimal policy.

To succinctly describe the human’s optimal policy, we introduce “human
thresholds” $t$ in section 5.1; when the human is in a state past the
threshold, their optimal policy is to pursue the goal. A human with a smaller
threshold $t$ will pursue the goal from farther away. An effective AI action
is one that moves the threshold $t$ to a state _preceding_ the human’s current
state, so that the human chooses to move.

######  Definition  (Human threshold).

For a chainworld $\mathcal{M}_{\theta}\in\mathscr{M}_{\text{chain}}$, define
$t\in\\{0,\ldots,N-1\\}$ as the threshold where $\pi_{\theta}^{*}(s_{n})=0$
for $n\leq t$ and $\pi_{\theta}^{*}(s_{n})=1$ for $n>t$.

Even if the AI agent _can_ intervene to prompt the human toward the goal,
whether or not the optimal AI _does_ intervene depends on the configuration of
the AI rewards. If intervening has negligible cost, then the AI agent will
intervene as soon as it is able. On the other hand, if there is a high cost,
then the AI agent will wait until the human is closer to the goal, to minimize
the total number of interventions needed. We define AI threshold $t_{AI}$
below, as the point at which the reward of reaching the goal outweighs the
cost of interventions required to reach it:

######  Definition  (AI threshold).

For a human chainworld $\mathcal{M}_{\theta}\in\mathscr{M}_{\text{chain}}$ and
AI MDP $\mathcal{M}_{AI}$, define AI threshold $t_{AI}\in\\{0,\ldots,N-1\\}$
as the chainworld state in which the value of the goal is greater than the
value of disengagement. For states $s_{n}$ where $n>t_{AI}$, the AI values are
$V_{AI}^{\pi_{g}}(s_{n})>V_{AI}^{\pi_{d}}(s_{n})$, and for states where $n\leq
t_{AI}$, the AI values are $V_{AI}^{\pi_{g}}(s_{n})\leq V_{AI}^{\pi_{d}}$.

The human and AI thresholds define the intervention windows for the AI policy
in section 5.1.

[Chainworld AI policies] Suppose we are given:

  * •

An AI MDP
$\mathcal{M}_{AI}=\langle\mathcal{S}_{AI},\mathcal{A}_{AI},T_{AI},R_{AI},\gamma_{AI}\rangle$,
where the actions are to do nothing ($a_{AI}=0$), intervene on the discount
($a_{AI}=a_{\gamma}$), or to intervene on burden ($a_{AI}=a_{b}$)

  * •

A human MDP $\mathcal{M}_{\theta}\in\mathscr{M}_{\text{chain}}$, which results
in human thresholds $t_{h}^{0}$, $t_{h}^{\gamma}$, and $t_{h}^{b}$ under AI
actions $0$, $a_{\gamma}$, and $a_{b}$, respectively

Let
$t_{h}^{\text{min}}=\min\left\\{t_{h}^{0},t_{h}^{\gamma},t_{h}^{b}\right\\}$
denote the earliest human threshold as a result of _any AI action_. Let
$t_{AI}$ denote the AI intervention threshold, as in section 5.1. Then, the
optimal AI policy is,

$\pi_{AI}^{*}(s_{n})=\begin{cases}0,&n\leq t_{h}^{\text{min}}\\\
0,&t_{h}^{\text{min}}<n\leq t_{AI}\\\
a_{\gamma},&\max\\{t_{AI},t_{h}^{\gamma}\\}<n\leq t_{h}^{0}\\\
a_{b},&\max\\{t_{AI},t_{h}^{b}\\}<n\leq t_{h}^{0}\\\
0,&n>t_{h}^{0},\end{cases}$ (6)

and $\pi_{AI}^{*}$ belongs to the three-window policy class, $\bar{\Pi}$. The
proof is in section B.1. Note that if both $a_{b}$ and $a_{\gamma}$ are valid
options in the intervention window (when $t_{AI}<n\leq t_{h}^{0}$), then the
AI agent will prefer the less expensive intervention. Theorem 5.1 shows that
every chainworld results in an optimal AI policy belonging to $\bar{\Pi}$.
Theorem 5.1 shows the reverse; for any human MDP whose corresponding AI policy
is $\pi_{AI}\in\bar{\Pi}$, there exists a chainworld MDP whose AI policy is
also $\pi_{AI}$.

[Chainworld equivalence class] If human MDP $\mathcal{M}_{h}$ has
corresponding AI policy $\pi_{AI}\in\bar{\Pi}$, then $\exists\theta$ for
$\mathcal{M}_{\theta}\in\mathscr{M}_{\text{chain}}$ such that
$\mathcal{M}_{\theta}\equiv\mathcal{M}_{h}$.

Proof in section B.2. Theorem 5.1 means that _any human MDP_ that results in a
three-window AI policy—that is, consists of three regions: impossible to help,
can be helped by the AI, and does not need help— belongs to the chainworld
equivalence class. In section 6, we will show that the AI agent can plan
interventions using chainworld as a substitute for another human MDP in the
same class, without any loss in performance.

###  5.2. Realistic human models that are equivalent to chainworld

Ultimately, we care that the chainworld equivalence class contains _realistic_
models of humans that align with the behavioral literature. In this section,
we provide examples of human MDPs that capture a meaningful behavior not
covered by chainworlds, yet whose optimal AI policy is still in the
equivalence class $\bar{\Pi}$.

_Monotonic chainworlds._ In monotonic chainworlds, the closer one gets to the
goal, the higher the relative value of pursuing it.

######  Definition  (Monotonic chainworlds).

For a monotonic chainworld $\mathcal{M}$, the value of goal-pursuit increases
closer to the goal: $V^{\pi_{g}}(s_{n})-V^{\pi_{d}}(s_{n})$ $\leq
V^{\pi_{g}}(s_{n+1})-V^{\pi_{d}}(s_{n+1})$ for all states $n=1,\ldots,N-1$.

For example, consider chainworlds in which the probability of disengagement
$p_{d}$ decreases the closer the agent is to the goal (the human is less
likely the quit the closer they are to recovery). Monotonic chainworlds relate
to the goal-gradient hypothesis, which states that motivation to reach a goal
increases with proximity (Mutter and Kundisch, 2014). In section C.1, we prove
that all monotonic chainworlds are AI equivalent to our chainworld.

_Progress worlds._ Progress worlds, while potentially multi-dimensional, have
a one-dimensional notion of progress.

######  Definition  (Progress worlds).

Suppose $\mathcal{M}$ is a $D$ dimensional, path-connected graph with an
absorbing goal state $s_{g}$, an absorbing disengagement state $s_{d}$, and
actions that allow movement between states on the graph. Let $d(s,s^{\prime})$
denote the shortest graph distance from $s$ to $s^{\prime}$. $\mathcal{M}$ is
a progress world if $d(s,s_{d})=d(s^{\prime},s_{d})$ and
$d(s,s_{g})=d(s^{\prime},s_{g})$ for all pairs of
$s,s^{\prime}\in\mathcal{S}$.

In our PT example, “progress” may depend on a combination of metrics such as
joint strength, the ability to perform daily tasks, and so on. We show in
section C.4 that worlds in which states can be mapped to a one-dimensional
distance are equivalent to our chainworlds. This type of equivalence is simple
yet useful, as it lets us reduce high-dimensional worlds to a single dimension
of interest. Definition 5.2 restricts us to graphs in which all shortest paths
between the disengagement and goal state are of the same length. Intuitively,
this means that a single chainworld can represent all paths (and therefore,
the entire world). Though not all graphs are progress worlds, in our empirical
experiments, we test the chainworld AI’s robustness to graphs that break this
definition.

_Multi-chain worlds._ In multi-chain worlds, there is a principle dimension
that corresponds to progress toward the goal (as in our simple chainworld) but
there may be several additional dimensions associated with different ways of
dropping out.

######  Definition  (Multi-chain worlds).

A multi-chain world $\mathcal{M}$ consists of $C$ chains, each of length
$N_{c}$. The first chain, $c=0$, is the _goal chain_ ; when the human reaches
the end of this chain, they have reached the goal. The remaining chains,
$s_{1},\ldots,s_{C-1}$, are disengagement chains; when the human reaches the
end of _any_ of these chains, they disengage. When $a=1$, the human moves
along the goal chain with probability $p_{0}$ while staying still in the
disengagement chains. When $a=0$, the human stays still in the goal chain and
(independently) moves along each of the $c$ disengagement chains with
probability $p_{c}$.

In our PT example, the principle chain might still correspond to the overall
strength of the joint as a measure of progress toward recovery. Additional
chains, corresponding to the level of motivation, level of pain, etc., may all
represent mechanisms that cause disengagement. This form of multi-chain
reflects how disengagement is described in the behavioral literature (e.g.
(Moshe et al., 2022; Moroshko et al., 2011)). In section C.5.1 we show
equivalence to multi-chain worlds whose disengagement chains are of length
$2$, which corresponds to real-world situations in which one of many factors
can abruptly trigger disengagement at any point (e.g. the PT patient is re-
injured).

_Negative effect worlds._ These are chainworlds in which the AI intervention
has the opposite intended effect on the human.

######  Definition  (Negative effect worlds).

A negative effect world $\mathcal{M}$ is defined exactly as the chainworld,
except that $\Delta_{\gamma}<0$ (AI intervention on discount $\gamma_{h}$
_decreases_ it) or $\Delta_{b}>0$ (AI intervention on burden $r_{b}$
_increases_ it).

The efficacy of a behavioral intervention is known to vary by individual (e.g.
(Bryan et al., 2021)). In section C.2, we prove that negative effect worlds
result in AI policies that correspond to chainworlds where the intervention
has _no effect_ (i.e. $\Delta_{\gamma}=0$ and $\Delta_{b}=0$).

##  6\. Empirical Analysis: Testing Robustness of Chainworld

We test how AI planning using chainworld benefits performance, especially as
we remove our assumptions and make the _true_ human model dissimilar to
chainworld.

###  6.1. Setup

All experiments are over $200$ trials of $15$ episodes each, and each trial
corresponds to a human whose MDP parameters $\theta$ are sampled. Not all
settings of $\theta$ correspond to individuals that can reach their goal—for
example, consider a human whose burden is so high that no AI intervention can
make them act. Here, we report results for the subset of sampled humans that
can reach the goal under the _oracle AI policy_. Doing so preserves the
relative ordering of method performances and reduces noise; in fig. 11 we give
an example of results that include individuals who never reach the goal.

_Baselines._ Our baselines are ways to learn the AI policy online. Using data
$\mathcal{D}_{AI}=\\{(s_{AI},a_{AI},s^{\prime}_{AI},r_{AI})\\}$, the _model-
free_ approach directly estimates $Q_{AI}^{*}$ via Q-learning. The _model-
based_ method estimates $T_{AI}$ using the observed transitions and then
solves for $\pi_{AI}^{*}$ with certainty equivalence. Both approaches bypass
the need for explicitly solving for a human policy. The _always
$\boldsymbol{\gamma}$_ and _always $\boldsymbol{B}$_ are “no personalization,”
in which the AI policy is to always intervene on $\gamma$ and $B$,
respectively. Our method, _chainworld_ , estimates the parameters $\theta$
from $\mathcal{D}_{AI}$.

###  6.2. Results under _no model misspecification_

In perfect conditions, the AI agent can use chainworld to reach oracle-level
performance in the fewest episodes. When the true human matches our inductive
bias, i.e. both are chainworlds, we achieve the fastest personalization in
fig. 4. In contrast, model-free requires hundreds of episodes before it learns
policies that are better than random (which we demonstrate in fig. 10 of the
appendix).

Figure 4. When the true human model is a chainworld, our method rapidly
personalizes. Plot is AI rewards (y-axis) over multiple episodes (x-axis).
Lines in upper-left personalize quicker.

(a)

(b) 18 total states

(c) 40 total states

(d) 70 total states

Figure 5. Chainworld scales to large gridworlds. Example gridworld on left. Going right, the grid’s width (X) and height (Y) increases. Assumption | Equiv? | Low misspecification | High misspecification
---|---|---|---
|  | Chainworld (ours) | Top baseline | Chainworld (ours) | Top baseline
Noise in burden $r_{b}$ | No | $\boldsymbol{-14.47\pm 3.63}$ | $\boldsymbol{-14.43\pm 3.63}$ | $\boldsymbol{-35.96\pm 3.36}$ | $\boldsymbol{-33.43\pm 3.4}$
Noise in utility of goal $r_{g}$ | No | $\boldsymbol{-5.53\pm 1.71}$ | $-14.76\pm 3.38$ | $\boldsymbol{-6.9\pm 2.22}$ | $-14.66\pm 3.34$
Noise in utility of progress loss $r_{\ell}$ | No | $\boldsymbol{-5.97\pm 1.94}$ | $-14.78\pm 3.39$ | $\boldsymbol{-11.01\pm 3.29}$ | $\boldsymbol{-15.43\pm 3.54}$
Noise in utility of disen. $r_{d}$ | No | $\boldsymbol{-8.08\pm 2.58}$ | $-15.18\pm 3.44$ | $\boldsymbol{-13.38\pm 3.54}$ | $\boldsymbol{-14.63\pm 3.41}$
Noise in prob. of disen. $p_{d}$ | No | $\boldsymbol{-5.03\pm 1.46}$ | $-14.78\pm 3.39$ | $\boldsymbol{-6.41\pm 2.45}$ | $\boldsymbol{-12.13\pm 4.05}$
Noise in prob. of disen. at state 0, $p_{d0}$ | No | $\boldsymbol{-5.8\pm 1.86}$ | $-14.81\pm 3.4$ | $\boldsymbol{-5.83\pm 1.86}$ | $-14.36\pm 3.3$
Noise in prob. of losing progress $p_{\ell}$ | No | $\boldsymbol{-5.05\pm 1.51}$ | $-14.78\pm 3.39$ | $\boldsymbol{-5.19\pm 1.81}$ | $-13.38\pm 4.13$
Noise in prob. of making progress $p_{g}$ | No | $\boldsymbol{-5.82\pm 1.77}$ | $-15.24\pm 3.49$ | $\boldsymbol{-19.38\pm 4.34}$ | $\boldsymbol{-17.85\pm 3.72}$
Noise in discount $\gamma_{h}$ | No | $\boldsymbol{-7.75\pm 2.42}$ | $-15.83\pm 3.56$ | $\boldsymbol{-20.7\pm 4.03}$ | $\boldsymbol{-21.19\pm 3.93}$
Params. fixed across states | Yes | — | — |  |
Mapping many dimensions to chainworld | Yes | — | — |  |
Wrong distance to goal in mapping | No | $\boldsymbol{-21.18\pm 3.84}$ | $\boldsymbol{-15.62\pm 3.15}$ | $-35.8\pm 3.8$ | $\boldsymbol{-24.52\pm 3.3}$
Wrong distance to disengagement in mapping | No | $\boldsymbol{-10.11\pm 2.44}$ | $\boldsymbol{-15.62\pm 2.44}$ | $\boldsymbol{-27.27\pm 3.86}$ | $\boldsymbol{-24.52\pm 3.3}$
Diseng. from multiple factors | Yes | — | — |  |
Human selects actions non-optimally | No | $\boldsymbol{-7.23\pm 2.27}$ | $-16.01\pm 4.01$ | $\boldsymbol{-24.27\pm 3.85}$ | $\boldsymbol{-23.39\pm 3.68}$
AI intervention has negative effect | Yes |  |  |  |
Table 2.  Reward earned by the AI in episode six. Each row is an assumption
violated by the environment. Chainworld is better than or within 95%
confidence interval of the top-performing baseline (out of five total
baselines) in all but one setting. Conditions marked with “yes” in the
“Equivalence?” column were shown in section 5.2 to preserve theoretical
equivalence under misspecification.

Our method’s performance scales to high-dimensional human models equivalent to
the chainworld. In the prior theoretical section, we provided examples of
human MDPs that reduce to the chainworld. The gridworld in fig. 5(a) is one
such world since it is a type of distance world. In fig. 5, our method still
personalizes the fastest in increasingly large state spaces, because the
number of chainworld parameters is invariant to the size of the gridworld. On
the other hand, model-based degrades; it is _worse_ than the personalization-
free baselines and the same as random baselines, even after $15$ episodes.
This is because the transition matrix that model-based must estimate scales
with the size of the gridworld. Model-free approaches are even more
inefficient in the 2-D setting than in the 1-D chainworld.

###  6.3. Robustness results under _model misspecification_

In true frictionful settings, the AI agent will encounter humans that are more
sophisticated than the chainworld. Our remaining experiments in table 2 test
if AI performance is robust to misspecification when we remove our assumptions
about humans. In section 5.2, we theoretically showed that a subset of these
assumptions can be removed without affecting the AI. The remaining assumptions
we test empirically, and we show our method is more robust to increasing
levels of misspecification than baselines. The definition of “low” vs. “high”
misspecification is specific to the experiment.

(a) Chainworld robust to low and high mis.

(b) Chainworld is robust to low mis.

(c) Environment challenges all methods.

Figure 6. Examples of robustness experiments. Chainworld is robust to all
levels of misspecification fig. 6(a), robust to low levels of misspecification
with maintainence at high levels fig. 6(b), and all methods, including oracle,
struggle to perform well in fig. 6(c). Details and plots for all environments
in section D.1 and section E.3, respectively.

_Experiment on noise in chainworld parameters._ In this experiment, we test AI
performance when the true human model is a chainworld whose parameters vary
each timestep due to noise. This mimics situations in which unobservable
factors, such as mood, affect parameters, such as burden $r_{b}$. We vary each
parameter in isolation. Our comparison must account for the domains of
different parameters, since $\gamma_{h}\in(0,1)$ while rewards such as
$r_{b}\in\mathbb{R}$. At each timestep, the parameter of interest $x$ is
sampled uniformly from $x\sim\text{Uniform}(\bar{x}-\epsilon
c,\bar{x}+\epsilon c)$, where $\bar{x}$ is the mean parameter value for that
individual and the noise level is determined by the parameter range $c$ and
the error level $\epsilon\in[0,1]$. We set parameter range $c=5$ for reward
parameters and to $c=1$ for transition parameters and $\gamma_{h}$. We define
low misspecification as is $\epsilon=0.1$ and high misspecification as
$\epsilon=0.5$.

Experiment on action selection. Instead of selecting actions via the optimal
policy, humans in this experiment select actions according to softmax policy,
$\pi_{h}(a|s)\propto\exp\\{Q_{h}(s,a)/\epsilon\\}$, where $\epsilon$ is the
level of noise. We define low misspecification as $\epsilon=0.05$ and high
misspecification as $\epsilon=0.2$.

Experiment on misspecified mapping. This experiment tests robustness to
differences in _model structure_. The true human is no longer a chainworld,
but a gridworld as in fig. 5(a). However, the gridworld in this experiment is
no longer equivalent to our chainworld because the goal state $s_{g}$ is not
in the lower-right corner at $[X,0]$. In fact, the equivalence degrades as
$\epsilon$ increases for $[X,\epsilon]$. We set the grid dimensions as
$X=8,Y=5$ and define low misspecification as $\epsilon=1$ and high
misspecification as $\epsilon=4$.

We are robust to low levels of misspecification. In table 2, our method
outperforms baselines in _9 out of 12 robustness experiments_ under low levels
of misspecification. With high misspecification, when our method is not the
best, it falls within two standard errors of the next-best method in all but
one condition.

Some humans are difficult to intervene on overall, even for the oracle. All
methods, including the oracle, earn fewer rewards when the burden parameter
$r_{b}$ is noisy (see fig. 6(c)). This indicates that it is particularly
important to model $r_{b}$ well in frictionful tasks. For example, we may
ensure that features predictive of burden, such as mood, are part of the AI’s
state space, so that we can estimate $r_{b}$.

To reduce (non-equivalent) human models to the chainworld, it is important
that we capture distance to goal well. Since chainworld is one-dimensional, it
can only represent worlds whose multi-dimensional states can be mapped to one
dimension. When such a mapping is not possible, we must choose between
capturing progress toward goal (e.g. how far does the patient feel from
shoulder recovery?) or distance from disengagement (e.g. how close to giving
up does the patient feel?). Under the “wrong distance to goal / disengagement
mapping” condition in table 2, we show that capturing progress toward goal
matters more. This implies that chainworlds can still be applied to settings
where we cannot model all factors that lead to disengagement, so long as we
have an accurate way of measuring the human’s progress to the goal.

##  7\. Conclusion and Future Work

In this paper, we introduced Behavior Model Reinforcement Learning (BMRL), a
framework for AI agents to intervene on human agents performing frictionful
tasks. We proposed a simple model of the human agent– the chainworld– that the
AI agent can use to rapidly personalize. Using a novel definition of
equivalence between human models in BMRL, we defined a theoretical class of
human MDPs that chainworld can generalize to and showed that this class
contains behaviorally meaningful models of humans.

Our chainworlds are not psychologically verified human models; in future work,
we will formally test the modeling assumptions with user studies. To apply
BMRL in the real world, we must also consider the ethics of AI intervention.
Mainly, we must ensure the AI does not manipulate the human. BMRL should only
be used for people who already have a long-term goal, and the AI must not
change that goal. Subgroup fairness should also be considered during learning
and personalization.

Although we aimed to be comprehensive in testing chainworld’s robustness,
there were limitations to our approach. First, we did not evaluate how
multiple misspecifications may compound to affect AI performance. Second, our
analyses assumed that the mapping from the true MDP to the chainworld is
given. In some applications this is reasonable; in PT, a domain expert is
likely to know which factors contribute to a patient’s perception of
“progress” (the mapping from a distance world to a chainworld). In other
cases, one will need to learn this mapping in conjunction with the chainworld
parameters.

We made several simplifying assumptions on the human + AI interactions. We
avoided a POMDP formulation by assuming that there are no delayed effects of
the AI’s actions on the human MDP. However, habituation (reduced effectiveness
of repeated interventions) is a well-studied phenomenon in digital
interventions (e.g. (Gotzian, 2023)). Furthermore, we avoided multi-agent RL
by assuming that the human is _not learning_ , and instead, is solving an
(implicitly) known MDP at each time step. We did not consider suboptimality of
the human agent’s planning, such as (small) fixed-horizon planning. Finally
(and excitingly), BMRL is adaptable to more diverse AI interventions. Our
paper focused exclusively on interventions to the human’s discount and reward.
In many applications, the human’s perception of state, actions, and
transitions may also be impaired. Similarly, behavioral interventions on
perceptions of state, actions, and transitions exist and could be incorporated
into our framework.

##  8\. Acknowledgements

This material is based upon work supported by the National Science Foundation
under Grant No. IIS-2107391 and the National Institute of Biomedical Imaging
and Bioengineering of the National Institutes of Health under OD P41EB028242.
Any opinions, findings, and conclusions or recommendations expressed in this
material are those of the author(s) and do not necessarily reflect the views
of the National Science Foundation. ES’s work was supported by a gift fund
from Benshi.ai and the National Science Foundation Graduate Research
Fellowship Program under Grant No. DGE2140743.

## References

  * (1)
  * Aswani et al. (2019) Anil Aswani, Philip Kaminsky, Yonatan Mintz, Elena Flowers, and Yoshimi Fukuoka. 2019.  Behavioral modeling in weight loss interventions.  _European journal of operational research_ 272, 3 (2019), 1058–1072.
  * Bandura (1999) Albert Bandura. 1999\.  Social cognitive theory: An agentic perspective.  _Asian journal of social psychology_ 2, 1 (1999), 21–41.
  * Brown et al. (2019) Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. 2019.  Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In _International conference on machine learning_. PMLR, California USA, 783–792.
  * Brown and Stein (2022) Jeremiah Michael Brown and Jeffrey Scott Stein. 2022.  Putting prospection into practice: Methodological considerations in the use of episodic future thinking to reduce delay discounting and maladaptive health behaviors.  _Frontiers in Public Health_ 10 (2022), 1020171.
  * Bryan et al. (2021) Christopher J Bryan, Elizabeth Tipton, and David S Yeager. 2021\.  Behavioural science is unlikely to change the world without a heterogeneity revolution.  _Nature human behaviour_ 5, 8 (2021), 980–989.
  * Chen et al. (2022) Kaiqi Chen, Jeffrey Fong, and Harold Soh. 2022.  Mirror: Differentiable deep social projection for assistive human-robot communication. In _Robotics: Science and Systems_. Robotics: Science and Systems, New York USA.
  * Evans et al. (2016) Owain Evans, Andreas Stuhlmüller, and Noah Goodman. 2016\.  Learning the preferences of ignorant, inconsistent agents. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 30. AAAI, Arizona USA.
  * Fanfarelli et al. (2015) Joseph Fanfarelli, Stephanie Vie, and Rudy McDaniel. 2015\.  Understanding digital badges through feedback, reward, and narrative: a multidisciplinary approach to building better badges in social environments.  _Communication Design Quarterly Review_ 3, 3 (2015), 56–60.
  * Fedus et al. (2019) William Fedus, Carles Gelada, Yoshua Bengio, Marc G. Bellemare, and Hugo Larochelle. 2019\.  Hyperbolic Discounting and Learning over Multiple Horizons.  arXiv:1902.06865 [stat.ML]
  * Givan et al. (2003) Robert Givan, Thomas Dean, and Matthew Greig. 2003\.  Equivalence notions and model minimization in Markov decision processes.  _Artificial Intelligence_ 147, 1-2 (2003), 163–223.
  * Giwa and Lee (2021) Babatunde H Giwa and Chi-Guhn Lee. 2021.  Estimation of Discount Factor in a Model-Based Inverse Reinforcement Learning Framework.  https://hdl.handle.net/1807/125220
  * Gotzian (2023) Lisa Gotzian. 2023\.  Modeling the decreasing intervention effect in digital health: a computational model to predict the response for a walking intervention.  https://doi.org/10.31219/osf.io/6v7d5
  * Jarrett et al. (2021) Daniel Jarrett, Alihan Hüyük, and Mihaela Van Der Schaar. 2021\.  Inverse decision modeling: Learning interpretable representations of behavior. In _International Conference on Machine Learning_. PMLR, PMLR, Virtual, 4755–4771.
  * Khanshan et al. (2023) Alireza Khanshan, Pieter Van Gorp, and Panos Markopoulos. 2023\.  Simulating Participant Behavior in Experience Sampling Method Research. In _Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems_ (¡conf-loc¿, ¡city¿Hamburg¡/city¿, ¡country¿Germany¡/country¿, ¡/conf-loc¿) _(CHI EA ’23)_. Association for Computing Machinery, New York, NY, USA, Article 250, 7 pages.  https://doi.org/10.1145/3544549.3585586
  * Li et al. (2006) Lihong Li, Thomas J Walsh, and Michael L Littman. 2006\.  Towards a unified theory of state abstraction for MDPs.
  * Liu et al. (2019) Quanying Liu, Haiyan Wu, and Anqi Liu. 2019.  Modeling and Interpreting Real-world Human Risk Decision Making with Inverse Reinforcement Learning.  arXiv:1906.05803 [cs.LG]
  * Magen et al. (2008) Eran Magen, Carol S Dweck, and James J Gross. 2008\.  The hidden-zero effect: Representing a single choice as an extended sequence reduces impulsive choice.  _Psychological Science_ 19, 7 (2008), 648–649.
  * Martin et al. (2018) Cesar A Martin, Daniel E Rivera, Eric B Hekler, William T Riley, Matthew P Buman, Marc A Adams, and Alicia B Magann. 2018.  Development of a control-oriented model of social cognitive theory for optimized mHealth behavioral interventions.  _IEEE Transactions on Control Systems Technology_ 28, 2 (2018), 331–346.
  * Mintz et al. (2023) Yonatan Mintz, Anil Aswani, Philip Kaminsky, Elena Flowers, and Yoshimi Fukuoka. 2023.  Behavioral analytics for myopic agents.  _European Journal of Operational Research_ 310, 2 (2023), 793–811.
  * Mogles et al. (2018) Nataliya Mogles, Julian Padget, Elizabeth Gabe-Thomas, Ian Walker, and JeeHang Lee. 2018\.  A computational model for designing energy behaviour change interventions.  _User Modeling and User-Adapted Interaction_ 28 (2018), 1–34.
  * Moroshko et al. (2011) Irena Moroshko, Leah Brennan, and Paul O’Brien. 2011\.  Predictors of dropout in weight loss interventions: a systematic review of the literature.  _Obesity reviews_ 12, 11 (2011), 912–934.
  * Moshe et al. (2022) Isaac Moshe, Yannik Terhorst, Sarah Paganini, Sandra Schlicker, Laura Pulkki-Råback, Harald Baumeister, Lasse B Sander, and David Daniel Ebert. 2022.  Predictors of dropout in a digital intervention for the prevention and treatment of depression in patients with chronic back pain: secondary analysis of two randomized controlled trials.  _Journal of Medical Internet Research_ 24, 8 (2022), e38261.
  * Mutter and Kundisch (2014) Tobias Mutter and Dennis Kundisch. 2014.  Behavioral mechanisms prompted by badges: The goal-gradient hypothesis. In _ICIS 2014 Proceedings_ , Vol. 12. ICIS, New Zealand.
  * Niv (2009) Yael Niv. 2009.  Reinforcement learning in the brain.  _Journal of Mathematical Psychology_ 53, 3 (2009), 139–154.
  * Park and Lee (2023) Joonyoung Park and Uichin Lee. 2023.  Understanding Disengagement in Just-in-Time Mobile Health Interventions.  _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ 7, 2 (2023), 1–27.
  * Pirolli (2016) Peter Pirolli. 2016\.  A computational cognitive model of self-efficacy and daily adherence in mHealth.  _Translational behavioral medicine_ 6, 4 (2016), 496–508.
  * Ravindran and Barto (2002) Balaraman Ravindran and Andrew G Barto. 2002.  Model minimization in hierarchical reinforcement learning. In _Abstraction, Reformulation, and Approximation: 5th International Symposium, SARA_ , Vol. 2371. Springer, Springer, Berlin, Heidelberg, Canada, 196–211.
  * Ravindran and Barto (2004) Balaraman Ravindran and Andrew G Barto. 2004.  Approximate homomorphisms: A framework for non-exact minimization in Markov decision processes.
  * Reddy et al. (2018) Siddharth Reddy, Anca D. Dragan, and Sergey Levine. 2018\.  Where do you think you’re going? inferring beliefs about dynamics from behavior. In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ (Montréal, Canada) _(NIPS’18)_. Curran Associates Inc., Red Hook, NY, USA, 1461–1472.
  * Reddy et al. (2021) Siddharth Reddy, Sergey Levine, and Anca Dragan. 2021\.  Assisted perception: optimizing observations to communicate state. In _Conference on Robot Learning_. PMLR, PMLR, London UK, 748–764.
  * Shah et al. (2019) Rohin Shah, Noah Gundotra, Pieter Abbeel, and Anca Dragan. 2019.  On the feasibility of learning, rather than assuming, human biases for reward inference. In _International Conference on Machine Learning_. PMLR, PMLR, California, USA, 5670–5679.
  * Shteingart and Loewenstein (2014) Hanan Shteingart and Yonatan Loewenstein. 2014.  Reinforcement learning and human behavior.  _Current opinion in neurobiology_ 25 (2014), 93–98.
  * Story et al. (2014) Giles W Story, Ivo Vlaev, Ben Seymour, Ara Darzi, and Raymond J Dolan. 2014.  Does temporal discounting explain unhealthy behavior? A systematic review and reinforcement learning perspective.  _Frontiers in behavioral neuroscience_ 8 (2014), 76.
  * Tabatabaei et al. (2018) Seyed Amin Tabatabaei, Mark Hoogendoorn, and Aart van Halteren. 2018.  Narrowing reinforcement learning: Overcoming the cold start problem for personalized health interventions. In _PRIMA 2018: Principles and Practice of Multi-Agent Systems: 21st International Conference_. Springer, Springer, Tokyo Japan, 312–327.
  * Tabrez et al. (2019) Aaquib Tabrez, Shivendra Agrawal, and Bradley Hayes. 2019\.  Explanation-Based Reward Coaching to Improve Human Performance via Reinforcement Learning. In _ACM/IEEE International Conference on Human-Robot Interaction (HRI)_. IEEE, Korea, 249–257.  https://doi.org/10.1109/HRI.2019.8673104
  * Taylor et al. (2021a) Véronique A Taylor, Isabelle Moseley, Shufang Sun, Ryan Smith, Alexandra Roy, Vera U Ludwig, and Judson A Brewer. 2021a.  Awareness drives changes in reward value which predict eating behavior change: Probing reinforcement learning using experience sampling from mobile mindfulness training for maladaptive eating.  _Journal of behavioral addictions_ 10, 3 (2021), 482–497.
  * Taylor et al. (2021b) Véronique A Taylor, Isabelle Moseley, Shufang Sun, Ryan Smith, Alexandra Roy, Vera U Ludwig, and Judson A Brewer. 2021b.  Awareness drives changes in reward value which predict eating behavior change: Probing reinforcement learning using experience sampling from mobile mindfulness training for maladaptive eating.  _Journal of behavioral addictions_ 10, 3 (2021), 482–497.
  * Tebbe et al. (2021) Jonas Tebbe, Lukas Krauch, Yapeng Gao, and Andreas Zell. 2021\.  Sample-efficient reinforcement learning in robotic table tennis. In _2021 IEEE international conference on robotics and automation (ICRA)_. IEEE, IEEE, China, 4171–4178.
  * Thabet et al. (2019) Mohammad Thabet, Massimiliano Patacchiola, and Angelo Cangelosi. 2019.  Sample-efficient deep reinforcement learning with imaginary rollouts for human-robot interaction. In _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, IEEE, Macau, 5079–5085.
  * Trella et al. (2022) Anna L Trella, Kelly W Zhang, Inbal Nahum-Shani, Vivek Shetty, Finale Doshi-Velez, and Susan A Murphy. 2022.  Designing reinforcement learning algorithms for digital interventions: pre-implementation guidelines.  _Algorithms_ 15, 8 (2022), 255.
  * van der Pol et al. (2020) Elise van der Pol, Thomas Kipf, Frans A. Oliehoek, and Max Welling. 2020.  Plannable Approximations to MDP Homomorphisms: Equivariance under Actions.  arXiv:2002.11963 [cs.LG]
  * Wang et al. (2021) Shihan Wang, Chao Zhang, Ben Kröse, and Herke van Hoof. 2021\.  Optimizing adaptive notifications in mobile health interventions systems: reinforcement learning from a data-driven behavioral simulator.  _Journal of medical systems_ 45 (2021), 1–8.
  * Xu (2022) Xuhai Xu. 2022.  Towards Future Health and Well-being: Bridging Behavior Modeling and Intervention. In _Adjunct Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology_. Association for Computing Machinery, New York, USA, 1–5.
  * Yang et al. (2020) Yuxiang Yang, Ken Caluwaerts, Atil Iscen, Tingnan Zhang, Jie Tan, and Vikas Sindhwani. 2020\.  Data efficient reinforcement learning for legged robots. In _Conference on Robot Learning_. PMLR, PMLR, Virtual, 1–10.
  * Yu and Ho (2022) Guanghui Yu and Chien-Ju Ho. 2022.  Environment Design for Biased Decision Makers. In _Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI)_. International Joint Conferences on Artificial Intelligence Organization, Austria, 592–598.
  * Zhang et al. (2022) Chao Zhang, Joaquin Vanschoren, Arlette van Wissen, Daniël Lakens, Boris de Ruyter, and Wijnand A IJsselsteijn. 2022.  Theory-based habit modeling for enhancing behavior prediction in behavior change support systems.  _User Modeling and User-Adapted Interaction_ 32, 3 (2022), 389–415.
  * Zhang et al. (2021) Chao Zhang, Shihan Wang, Henk Aarts, and Mehdi Dastani. 2021\.  Using Cognitive Models to Train Warm Start Reinforcement Learning Agents for Human-Computer Interactions.  arXiv:2103.06160 [cs.AI]
  * Zhi-Xuan et al. (2020) Tan Zhi-Xuan, Jordyn Mann, Tom Silver, Josh Tenenbaum, and Vikash Mansinghka. 2020.  Online bayesian goal inference for boundedly rational planning agents.  _Advances in neural information processing systems_ 33 (2020), 19238–19250.
  * Zhou et al. (2018) Mo Zhou, Yonatan Mintz, Yoshimi Fukuoka, Ken Goldberg, Elena Flowers, Philip Kaminsky, Alejandro Castillejo, and Anil Aswani. 2018\.  Personalizing mobile fitness apps using reinforcement learning. In _CEUR workshop proceedings_ , Vol. 2068. NIH Public Access, CEUR workshop proceedings, Japan.

##  Appendix A Optimal value functions in chainworlds

In this section, we solve for the analytical solution of the optimal value
function (and therefore the optimal policy) for chainworlds
$\mathcal{M}_{\theta}\in\mathscr{M}_{\text{chain}}$.

In our setting, once the optimal action is to go right in a given state, the
best strategy is to continue going right in subsequent states that are closer
to the goal. That is, if $\pi^{*}(w)=1$, then $\pi^{*}(w+1)=1$. The opposite
is also true; if the optimal action is to stay in place in a given state, then
the best strategy in a state that is farther away from the goal is also to
stay in place — if $\pi^{*}(w)=0$, then $\pi^{*}(w-1)=0$.

In other words, the optimal value function maximizes between a policy that
goes to the goal state $\pi_{g}$ and a policy that goes to disengagement
$\pi_{d}$. Specifically, for MDP
$\mathcal{M}_{\theta}\in\mathscr{M}_{\text{chain}}$, the corresponding optimal
value function $V^{*}_{\theta}$ is
$V^{*}_{\theta}(s)=\max\\{V^{\pi_{d}}(s),V^{\pi_{g}}(s)\\},$ and the optimal
policy $\pi_{\theta}^{*}$ is
$\pi^{*}_{\theta}(s)=\mathbb{I}\\{V^{\pi_{g}}(s)>V^{\pi_{d}}(s)\\},$ for all
$s\in\mathcal{S}$.

###  A.1. Derivation of $V^{\pi_{g}}$

We will start by deriving $V^{\pi_{g}}$ for states close to the goal state
$s_{g}$, and generalize these findings. First, note that
$V^{\pi_{g}}(s_{N})=V^{\pi_{g}}(s_{g})=r_{g}$ because $s_{g}$ is absorbing.

Next, we will derive the value of a state which is right before the goal
state, $s_{N-1}$. Recall that when $a=1$ the human moves right with
probability $p_{g}$ and stays in place with probability $1-p_{g}$. The human
always receives a reward of $r_{b}$ for choosing $a=1$. Using Bellman
recursion for the value function results in,

$\displaystyle\begin{split}&V^{\pi_{g}}(s_{N-1})\\\
&=r(s_{N-1},a=1)+\gamma\sum\limits_{s^{\prime}}P(s^{\prime}|s=s_{N-1},a=1)V^{\pi_{g}}(s^{\prime})\\\
&=r_{b}+\gamma\left[p_{g}V^{\pi_{g}}(s_{N})+(1-p_{g})V^{\pi_{g}}(s_{N-1})\right]\\\
&=r_{b}+\gamma\left[p_{g}r_{g}+(1-p_{g})V^{\pi_{g}}(s_{N-1})\right]\\\
&=r_{b}+\gamma p_{g}r_{g}+\gamma(1-p_{g})V^{\pi_{g}}(N-1)\\\ &=r_{b}+\gamma
p_{g}r_{g}+\gamma(1-p_{g})[r_{b}+\gamma
p_{g}r_{g}+\gamma(1-p_{g})V^{\pi_{g}}(s_{N-1})]\\\ &=\underbrace{r_{b}+\gamma
p_{g}r_{g}}_{\text{when }s=s_{N-1}\text{ at
time}0}+\underbrace{\gamma(1-p_{g})[r_{b}+\gamma p_{g}r_{g}]}_{\text{when
}s=s_{N-1}\text{ at time
}1}+\underbrace{\gamma^{2}(1-p_{g})^{2}V^{\pi_{g}}(s_{N-1})}_{\text{when
}s=s_{N-1}\text{ at time}2}+\ldots\\\
&=\sum\limits_{t=0}^{\infty}\gamma^{t}(1-p_{r})^{t}[r_{b}+\gamma
p_{g}r_{g}]\\\ &=\frac{p_{g}\gamma r_{g}+r_{b}}{1-\gamma(1-p_{g})}\\\
&=\frac{p_{g}\gamma r_{g}+r_{b}}{z},\end{split}$ (7)

for $z=1-\gamma(1-p_{g})$. Next, using a similar strategy, we derive the value
of a state which is _two spaces_ away from the goal state, $s_{N-2}$:

$\displaystyle\begin{split}&V^{\pi_{g}}(s_{s_{N-2}})\\\
&=r(s=s_{N-2},a=1)+\gamma\sum\limits_{s^{\prime}}P(s^{\prime}|s=s_{N-2},a=1)V^{\pi_{g}}(s^{\prime})\\\
&=r_{b}+\gamma\left[p_{g}V^{\pi_{g}}(s_{N-1})+(1-p_{g})V^{\pi_{g}}(s_{N-2})\right]\\\
&=r_{b}+\gamma
p_{g}V^{\pi_{g}}(s_{N-1})+\gamma(1-p_{g})V^{\pi_{g}}(s_{N-2})\\\
&=\underbrace{r_{b}+\gamma p_{g}V^{\pi_{g}}(s_{N-1})}_{\text{ when
}s=s_{N-2}\text{ at time 0}}+\underbrace{\gamma(1-p_{g})[r_{b}+\gamma
p_{g}V^{\pi_{g}}(s_{N-1})]}_{\text{ when }s=s_{N-2}\text{ at time
1}}+\underbrace{\gamma^{2}(1-p_{g})^{2}[r_{b}+\gamma
p_{g}V^{\pi_{g}}(s_{N-1})]}_{\text{ when }s=s_{N-2}\text{ at time
2}}+\ldots\\\ &=\sum\limits_{t=0}^{\infty}\gamma^{t}(1-p_{g})^{t}[r_{b}+\gamma
p_{g}V^{\pi_{g}}(s_{N-1})]\\\ &=\frac{r_{b}+\gamma
p_{g}V^{\pi_{g}}(s_{N-1})}{1-\gamma(1-p_{g})}\\\ &=\frac{r_{b}+\gamma
p_{g}\frac{\gamma p_{g}r_{g}+r_{b}}{z}}{z}\\\
&=\frac{\gamma^{2}p_{g}^{2}r_{g}}{z^{2}}+\frac{\gamma
p_{g}r_{b}}{z^{2}}+\frac{r_{b}}{z}.\\\ \end{split}$ (8)

In general, we can apply the Bellman equation to “recursively” expand the form
of the value function, so that the value at a given state $s_{n}$ can be
written as an infinite geometric series:

$\displaystyle
V^{\pi_{g}}(s_{n})=\sum\limits_{t=0}^{\infty}\gamma^{t}(1-p)^{t}\left[r_{b}+\gamma
p_{g}V^{\pi_{g}}(s_{n+1})\right]$ (9)

We will apply eq. 9 to our final derivation of $V^{\pi_{g}}(s_{N-3})$:

$\displaystyle\begin{split}&V^{\pi_{g}}(s_{N-3})\\\
&=\sum\limits_{t=0}^{\infty}\gamma^{t}(1-p_{g})^{t}[r_{b}+\gamma
p_{g}V^{\pi_{g}}(s_{N-2})]\\\ &=\frac{r_{b}+\gamma
p_{g}V^{\pi_{g}}(s_{N-2})}{1-\gamma(1-p_{g})}\\\ &=\frac{r_{b}+\gamma
p_{g}\left(\frac{\gamma^{2}p^{2}r_{g}}{z^{2}}+\frac{\gamma
p_{g}r_{b}}{z^{2}}+\frac{r_{b}}{z}\right)}{z}\\\
&=\frac{\gamma^{3}p_{g}^{3}}{z^{3}}r_{g}+\frac{\gamma^{2}p_{g}^{2}}{z^{3}}r_{b}+\frac{\gamma
p_{g}}{z^{2}}r_{b}+\frac{1}{z}r_{b}.\\\ \end{split}$ (10)

In general, for any state $s_{N-\delta}$, the value function is:

$\displaystyle\begin{split}&V^{\pi_{g}}(s_{N-\delta})\\\
&=r_{g}\left(\frac{\gamma\
p_{g}}{z}\right)^{\delta}+\frac{r_{b}}{z}\sum\limits_{n=0}^{\delta-1}\left(\frac{\gamma\
p_{g}}{z}\right)^{n}\\\ &=r_{g}\left(\frac{\gamma\
p_{g}}{z}\right)^{\delta}+r_{b}\left(\frac{1-(\gamma\
p_{g}/z)^{\delta}}{1-\gamma}\right),\end{split}$ (11)

where $z=1-\gamma(1-p_{g})$.

###  A.2. Derivation of $V^{\pi_{d}}$

This derivation is similar in nature to the one on $V^{\pi_{g}}$. Note that
$V^{\pi_{d}}(s_{0})=r_{d}\left(\frac{\gamma
p_{d0}}{1-\gamma(1-p_{d0})}\right)$. We will begin by solving for
$V^{\pi_{d}}(s_{1})$:

$\displaystyle\begin{split}&V^{\pi_{d}}(s_{1})=\sum\limits_{s^{\prime}}P(s^{\prime}|s=s_{1},a=0)[\
r(s=s_{1},a=0,s^{\prime})+\gamma V^{\pi_{d}}(s^{\prime})\ ]\\\
&=\underbrace{p_{d}[0+\gamma
V^{\pi_{d}}(s_{d})]}_{s^{\prime}=s_{d}}+\underbrace{p_{\ell}[r_{\ell}+\gamma
V^{\pi_{d}}(s_{0})]}_{s^{\prime}=s_{0}}+\underbrace{(1-p_{\ell}-p_{d})[0+\gamma
V^{\pi_{d}}(s_{1})]}_{s^{\prime}=s_{1}}\\\ &=\gamma
p_{d}r_{d}+p_{\ell}r_{\ell}+p_{\ell}\gamma
V^{\pi_{d}}(s_{0})+(1-p_{\ell}-p_{d})\gamma V^{\pi_{d}}(s_{1})\\\
&=\sum\limits_{t=0}^{\infty}\gamma^{t}(1-p_{\ell}-p_{d})^{t}[\gamma
p_{d}r_{d}+p_{\ell}r_{\ell}+p_{\ell}\gamma V^{\pi_{d}}(s_{0})]\\\
&=\frac{\gamma p_{d}r_{d}+p_{\ell}r_{\ell}+p_{\ell}\gamma
V^{\pi_{d}}(s_{0})}{1-\gamma(1-p_{\ell}-p_{d})}\\\ &=r_{d}\left(\frac{\gamma
p_{d}}{u}\right)+r_{\ell}\left(\frac{p_{\ell}}{u}\right)+r_{d}\left(\frac{\gamma^{2}p_{\ell}p_{d0}}{uz}\right),\end{split}$
(12)

where $u=1-\gamma(1-p_{\ell}-p_{d})$ and $z=1-\gamma(1-p_{d0})$. In the same
way, $V^{\pi_{d}}(s_{2})$ is:

$\displaystyle\begin{split}&V^{\pi_{d}}(s_{2})\\\
&=\sum\limits_{t=0}^{\infty}\gamma^{t}(1-p_{\ell}-p_{d})^{t}[\gamma
p_{d}r_{d}+p_{\ell}r_{\ell}+p_{\ell}\gamma V^{\pi_{d}}(s_{1})]\\\
&=\frac{\gamma p_{d}r_{d}+p_{\ell}r_{\ell}+p_{\ell}\gamma
V^{\pi_{d}}(s_{1})}{u}\\\ &=r_{d}\left(\frac{\gamma
p_{d}}{u}\right)+r_{\ell}\left(\frac{p_{\ell}}{u}\right)+\frac{\gamma
p_{\ell}}{u}\left(r_{d}\left(\frac{\gamma
p_{d}}{u}\right)+r_{\ell}\left(\frac{p_{\ell}}{u}\right)+r_{d}\left(\frac{\gamma^{2}p_{\ell}p_{d0}}{uz}\right)\right)\\\
&=r_{d}\left(\frac{\gamma
p_{d}}{u}\right)+r_{\ell}\left(\frac{p_{\ell}}{u}\right)+r_{d}\left(\frac{\gamma^{2}p_{\ell}p_{d}}{u^{2}}\right)+r_{\ell}\left(\frac{\gamma
p_{\ell}^{2}}{u^{2}}\right)+r_{d}\left(\frac{\gamma^{3}p_{\ell}^{2}p_{d0}}{u^{2}z}\right)\end{split}$
(13)

This yields to a general form:

$V^{\pi_{d}}(s_{n})=r_{d}\left(\frac{\gamma
p_{d0}}{z}\right)\left(\frac{\gamma p_{\ell}}{u}\right)^{n}+(\gamma
p_{d}r_{d}+p_{\ell}r_{\ell})\sum\limits_{t=0}^{n}\left(\frac{\gamma
p_{\ell}}{u}\right)^{n}=r_{d}\left(\frac{\gamma
p_{d0}}{z}\right)\left(\frac{\gamma p_{\ell}}{u}\right)^{n}+(\gamma
p_{d}r_{d}+p_{\ell}r_{\ell})\left(\frac{1-(\gamma
p_{\ell}/u)^{n}}{1-\gamma(1-p_{d})}\right)$ (14)

##  Appendix B AI policies for chainworld humans

(a) Human’s optimal policy when $a_{AI}=0$. The human is planning in an MDP
with default values for $\gamma,r_{b}$.

(b) Human’s optimal policy when $a_{AI}=a_{\gamma}$. The human is planning in
an MDP with default $r_{b}$ and increased
$\gamma^{\prime}=\gamma+\Delta_{\gamma}$.

(c) Human’s optimal policy when $a_{AI}=a_{b}$. The human is planning in an
MDP with default $\gamma$ and decreased $r_{b}-\Delta_{b}$.

(d) Example of AI’s optimal policy, given the example thresholds for
$t_{h}^{0},t_{h}^{\gamma},t_{h}^{b}$ from fig. 7(a), fig. 7(b), fig. 7(c)

Figure 7. Example of how the human’s optimal policies for chainworld following
AI actions $a_{AI}=0$ (fig. 7(a)), $a_{AI}=a_{\gamma}$ (fig. 7(b)), and
$a_{AI}=a_{b}$ (fig. 7(c)) result in the AI optimal policy in fig. 7(d).

###  B.1. Proof that chainworld AI has a 3-window policy

The following is the proof for section 5.1.

###### Proof.

We will prove this on a case-by-case basis.

The optimal AI policy is takes action $\boldsymbol{0}$ for states
$\boldsymbol{s_{n}}$ where $\boldsymbol{n\leq t_{h}^{\text{min}}}$. We will
prove by negation.

Let $\pi_{1}$ be defined as in eq. 6. Suppose $\pi_{1}$ is not optimal. This
implies that there must exist some optimal policy, $\pi_{2}$, whose actions
are non-zero for a subset of $\\{s_{m}\\}$ states, where
$m<t_{h}^{\text{min}}$.

Note that,

$V^{\pi_{1}}(s_{m})=r_{AI}^{d}\left(\frac{\gamma
p_{d}}{1-\gamma(1-p_{d})}\right),$

because the human will not take action in states before the threshold
$t_{h}^{\text{min}}$, so disengagement is inevitable. Similarly,

$V^{\pi_{2}}(s_{m})=r_{AI}^{i}+r_{AI}^{d}\left(\frac{\gamma
p_{d}}{1-\gamma(1-p_{d})}\right),$

where the human outcome remains the same, but the AI agent receives additional
penalty $r_{AI}^{i}$ for sending an intervention.

Since $\pi_{2}$ is an optimal policy, $V^{\pi_{2}}(s_{m})\geq
V^{\pi_{1}}(s_{m})\implies r_{AI}^{i}+r_{AI}^{d}\left(\frac{\gamma
p_{d}}{1-\gamma(1-p_{d})}\right)>r_{AI}^{d}\left(\frac{\gamma
p_{d}}{1-\gamma(1-p_{d})}\right)\implies r_{AI}^{i}>0$. This cannot be true,
since $r_{AI}^{i}<0$ by construction.

The optimal AI policy takes action $\boldsymbol{0}$ for states
$\boldsymbol{s_{n}}$ where $\boldsymbol{n>t_{h}^{0}}$. We will prove by
negation.

Let $\pi_{1}$ be defined as in eq. 6. Suppose $\pi_{1}$ is not optimal. This
implies that there must exist some optimal policy, $\pi_{2}$, whose actions
are non-zero for a subset of $\\{s_{m}\\}$ states, where $m>t_{h}^{0}$.

Note that,

$V^{\pi_{1}}(s_{m})=r_{AI}^{g}\left(\frac{\gamma_{AI}\
p_{g}}{1-\gamma_{AI}(1-p_{g})}\right)^{N-m},$

because the human will always take action in states after the threshold
$t_{h}^{0}$, so they will always reach the goal state. Similarly,

$V^{\pi_{2}}(s_{m})\leq r_{AI}^{i}+r_{AI}^{g}\left(\frac{\gamma_{AI}\
p_{g}}{1-\gamma_{AI}(1-p_{g})}\right)^{N-m},$

where the human outcome remains the same, but the AI agent receives additional
penalty $r_{AI}^{i}$ for sending an intervention in this state, and (possibly)
subsequent states.

Since $\pi_{2}$ is an optimal policy, $V^{\pi_{2}}(s_{m})\geq
V^{\pi_{1}}(s_{m})\implies r_{AI}^{i}+r_{AI}^{g}\left(\frac{\gamma_{AI}\
p_{g}}{1-\gamma_{AI}(1-p_{g})}\right)^{N-m}>r_{AI}^{g}\left(\frac{\gamma_{AI}\
p_{g}}{1-\gamma_{AI}(1-p_{g})}\right)^{N-m}$. This cannot be true, since
$r_{AI}^{i}<0$ by construction.

The optimal AI policy takes action $\boldsymbol{0}$ when
$\boldsymbol{t_{h}^{\text{min}}<n\leq t_{AI}}$ and takes action
$\boldsymbol{a_{\gamma}}$ when $\boldsymbol{t_{AI}<n\leq t_{h}^{0}}$.

Without loss of generality, assume $t_{h}^{\gamma}<t_{h}^{b}$.

By construction, an episode in the chainworld is finite with two absorbing
states, so the optimal AI policy must choose weather to influence the human
toward the goal or disengagement state.

Let $\pi_{AI}^{g}$ denote the _highest value_ policy to the goal state. Such a
policy has two behaviors. First, the policy will always take
$\pi_{AI}^{g}(s_{n})=a_{\gamma}$ in states where the AI agent can move the
threshold before the current state; this corresponds to states $s_{n}$ where
$t_{h}^{\text{min}}\leq<n<t_{h}^{0}$. This is because _witholding
intervention_ on some states by taking action $\pi_{AI}^{g}(s_{n})=0$ means
that the human will not pursue the goal, since $n<t_{h}^{0}$, and the AI agent
will receive a disengagement penalty. Second, the goal policy will always take
action $0$ in states $s_{n}$ where $n\geq t_{h}^{0}$, as we showed earlier in
the proof. The value of $\pi_{AI}^{g}$ is,

$V^{\pi_{AI}^{g}}(s_{n})=r_{AI}^{g}\left(\frac{\gamma_{AI}\
p_{g}}{1-\gamma_{AI}(1-p_{g})}\right)^{N-n}+r_{AI}^{i}\left(\frac{1-(\gamma_{AI}\
p_{g}/(1-\gamma_{AI}(1-p_{g})))^{N-n}}{1-(\gamma_{AI}\
p_{g}/(1-\gamma_{AI}(1-p_{g})))}\right),$

which is the discounted reward of the goal reduced by the cost of the
interventions to reach the goal.

Let $\pi_{AI}^{d}$ denote the _highest value_ policy to disengagement, which
takes action $\pi_{AI}^{d}(s_{n})=0$ in states $s_{n}$ where
$t_{h}^{\text{min}}<n\leq t_{h}^{0}$. This is for the same reason as we showed
earlier in the proof; when disengagement is inevitable, it is better to
withold intervention to avoid the additional cost. The value of $\pi_{AI}^{d}$
is,

$V^{\pi_{AI}^{d}}(s_{n})=r_{AI}^{d}\left(\frac{\gamma
p_{d}}{1-\gamma(1-p_{d})}\right).$

When $V^{\pi_{AI}^{d}}(s_{n})\geq V^{\pi_{AI}^{g}}(s_{n})$, the value of
disengagement outweighs the cost of reaching the goal, and the optimal AI
policy will take action $0$. By definition, $V^{\pi_{AI}^{d}}(s_{n})\geq
V^{\pi_{AI}^{g}}(s_{n})$ when $n\leq t_{AI}$. So, the optimal AI policy will
take action $0$ when $n\leq t_{AI}$.

Similarly, when $V^{\pi_{AI}^{d}}(s_{n})<V^{\pi_{AI}^{g}}(s_{n})$, the optimal
AI policy will take action $a_{\gamma}$. By definition,
$V^{\pi_{AI}^{d}}(s_{n})<V^{\pi_{AI}^{g}}(s_{n})$ when $n>t_{AI}$. So, the
optimal AI policy will take action $a_{\gamma}$ when $n>t_{AI}$. ∎

###  B.2. Proof that chainworlds can cover the entire space of 3-window
policies

The following is the proof for section 5.1.

###### Proof.

Note that the optimal chainworld AI policy in eq. 6 depends on four
quantities: the human thresholds $t_{h}^{0},t_{h}^{\gamma},t_{h}^{b}$ and the
AI policy threshold $t_{AI}$. Since the AI policy threshold is a direct result
of the human thresholds and the AI MDP, it suffices to show in this proof that
there exists a $\theta$ that can produce _all possible values of_
$t_{h}^{0},t_{h}^{\gamma},t_{h}^{b}$.

We will prove this in three parts, and each part will be similar in structure:

  1. (1)

First, we will show that there exist chainworld parameters without considering
AI intervention effects ($\theta\setminus\\{\Delta_{\gamma},\Delta_{b}\\}$)
that can define any $t_{h}^{0}$.

  2. (2)

Then, we will show that there exists a $\Delta_{b}$ that can define any
$t_{h}^{b}$, given the chainworld parameters from step (1).

  3. (3)

Finally, we will show that there exists a $\Delta_{\gamma}$ that can define
any $t_{h}^{\gamma}$ given the chainworld parameters from step (1).

There exists $\boldsymbol{\theta\setminus\\{\Delta_{\gamma},\Delta_{b}\\}}$
that can define any $t_{h}^{0}$. We will refer to $t_{h}^{0}$ as $t_{0}$ for
brevity. By section 5.1, any $t_{0}\in\\{0,\ldots,N-1\\}$ must satisfy two
constraints:

$V^{\pi_{g}}(s_{t_{0}})<V^{\pi_{d}}(s_{t_{0}})\And
V^{\pi_{g}}(s_{t_{0}+1})>V^{\pi_{d}}(s_{t_{0}+1}).$

We will prove that there exists
$\theta\setminus\\{\Delta_{\gamma},\Delta_{b}\\}$ so that these constraints
are always satisfied.

Let $r_{d}=0,r_{\ell}=0,p_{g}=1,p_{d}=0,p_{d0}=0,p_{e}ll=0$. The constraints
become:

$\displaystyle\begin{split}&V^{\pi_{g}}(s_{t_{0}})<V^{\pi_{d}}(s_{t_{0}})\And
V^{\pi_{g}}(s_{t_{0}+1})>V^{\pi_{d}}(s_{t_{0}+1})\\\ &\implies
r_{g}\gamma^{N-t_{0}}-r_{b}\left(\frac{1-\gamma^{N-t_{0}}}{1-\gamma}\right)>0\And
r_{g}\gamma^{N-t_{0}-1}-r_{b}\left(\frac{1-\gamma^{N-t_{0}-1}}{1-\gamma}\right)<0\\\
&\implies
r_{g}>r_{b}\left(\frac{1-\gamma^{N-t_{0}}}{\gamma^{N-t_{0}}(1-\gamma)}\right)\And
r_{g}<r_{b}\left(\frac{1-\gamma^{N-t_{0}-1}}{\gamma^{N-t_{0}-1}(1-\gamma)}\right)\\\
&\implies
r_{b}\left(\frac{1-\gamma^{N-t_{0}}}{\gamma^{N-t_{0}}(1-\gamma)}\right)<r_{g}<r_{b}\left(\frac{1-\gamma^{N-t_{0}-1}}{\gamma^{N-t_{0}-1}(1-\gamma)}\right).\end{split}$
(15)

These constraints are satisfied so long as there exists valid chainworld
parameters so that the following inequality holds:

$\displaystyle\begin{split}&\implies
r_{b}\left(\frac{1-\gamma^{N-t_{0}}}{\gamma^{N-t_{0}}(1-\gamma)}\right)<r_{b}\left(\frac{1-\gamma^{N-t_{0}-1}}{\gamma^{N-t_{0}-1}(1-\gamma)}\right)\\\
&\implies\left(\frac{1-\gamma^{N-t_{0}}}{\gamma^{N-t_{0}}}\right)<\left(\frac{1-\gamma^{N-t_{0}-1}}{\gamma^{N-t_{0}-1}}\right)\\\
&\implies\gamma(1-\gamma^{N-t_{0}-1})<1-\gamma^{N-t_{0}}\\\
&\implies\gamma<1.\end{split}$ (16)

So, there exists

$\displaystyle\begin{split}&\theta\setminus\\{\Delta_{\gamma},\Delta_{b}\\}=\\{p_{g}=1,p_{d}=0,p_{d0}=0,p_{\ell}=0,r_{d}=0,r_{\ell}=0,\\\
&\quad\quad\quad\quad\quad\ \ \quad r_{g}\text{ (that satisfies condition in
\lx@cref{creftype~refnum}{eq: t-0-conditions})},r_{b}\text{ (that satisfies
condition in \lx@cref{creftype~refnum}{eq:
t-0-conditions})},\gamma<1\\},\end{split}$ (17)

which defines any $t_{0}\in\\{0,\ldots,N-1\\}$.

There exists an AI effect on burden $\boldsymbol{\Delta_{b}}$ that can define
any human threshold $\boldsymbol{t_{h}^{b}}$. We will refer to the human
threshold following a burden intervention $t_{h}^{b}$ as $t_{b}$ for brevity.
By section 5.1, the threshold $t_{b}$ must satisfy the following constraints:

$V^{\pi_{g}}(s_{t_{b}};r_{b}+\Delta_{b})<V^{\pi_{d}}(s_{t_{b}})\And
V^{\pi_{g}}(s_{t_{b}+1};r_{b}+\Delta_{b})<V^{\pi_{d}}(s_{t_{b}}),$

where $V^{\pi_{g}}(\cdot;r_{b}+\Delta_{b})$ represents the human’s value of
goal pursuit under the burden $r_{b}+\Delta_{b}$.

Suppose $\theta\setminus\\{\Delta_{\gamma},\Delta_{b}\\}$ is defined as in eq.
17. Then,

$\displaystyle\begin{split}&V^{\pi_{g}}(s_{t_{b}};r_{b}+\Delta_{b})<V^{\pi_{d}}(s_{t_{b}})\And
V^{\pi_{g}}(s_{t_{b}+1};r_{b}+\Delta_{b})>V^{\pi_{d}}(s_{t_{b}+1})\\\
&\implies
r_{g}\gamma^{N-t_{b}}-(r_{b}+\Delta_{b})\left(\frac{1-\gamma^{N-t_{b}}}{1-\gamma}\right)>0\And
r_{g}\gamma^{N-t_{b}-1}-(r_{b}+\Delta_{b})\left(\frac{1-\gamma^{N-t_{b}-1}}{1-\gamma}\right)<0\\\
&\frac{r_{g}(1-\gamma)\gamma^{N-t_{b}}}{1-\gamma^{N-t_{b}}}<\Delta<\frac{r_{g}(1-\gamma)\gamma^{N-t_{b}-1}}{1-\gamma^{N-t_{b}-1}}.\end{split}$
(18)

These constraints are satisfied so long as there exists parameters so that the
following inequality holds:

$\displaystyle\begin{split}&\frac{r_{g}(1-\gamma)\gamma^{N-t_{b}}}{1-\gamma^{N-t_{b}}}<\frac{r_{g}(1-\gamma)\gamma^{N-t_{b}-1}}{1-\gamma^{N-t_{b}-1}}\\\
&\implies\frac{\gamma^{N-t_{b}}}{1-\gamma^{N-t_{b}}}<\frac{\gamma^{N-t_{b}-1}}{1-\gamma^{N-t_{b}-1}}\\\
&\implies\gamma^{N-t_{b}}<\gamma^{N-t_{b}-1}\\\ &\implies\gamma<1,\end{split}$
(19)

which is true by definition. So, there exists $\Delta_{b}$ that defines any
$t_{b}\in{0,\ldots,N-1}$.

There exists an AI effect on discounting $\boldsymbol{\Delta_{\gamma}}$ that
can define any human threshold $\boldsymbol{t_{h}^{\gamma}}$. We will refer to
the human threshold following a discount intervention $t_{h}^{\gamma}$ as
$t_{\gamma}$ for brevity. By definition section 5.1, the threshold
$t_{\gamma}$ must satisfy the constraints:

$V^{\pi_{g}}(s_{t_{\gamma}};\gamma+\Delta_{\gamma})<V^{\pi_{d}}(s_{t_{\gamma}};\gamma+\Delta_{\gamma})\And
V^{\pi_{g}}(s_{t_{\gamma}+1};\gamma+\Delta_{\gamma})<V^{\pi_{d}}(s_{t_{\gamma}+1};\gamma+\Delta_{\gamma}),$

where $V^{\pi_{g}}(\cdot;\gamma+\Delta_{\gamma})$ represents the human’s value
of goal under the discount rate $\gamma+\Delta_{\gamma}$. The same applies to
$V^{\pi_{d}}(\cdot;\gamma+\Delta+\gamma)$.

Suppose $\theta\setminus\\{\Delta_{\gamma},\Delta_{b}\\}$ is defined as in eq.
17. Then,

$\displaystyle\begin{split}&V^{\pi_{g}}(s_{t_{\gamma}};\gamma+\Delta_{\gamma})<V^{\pi_{d}}(s_{t_{\gamma}};\gamma+\Delta_{\gamma})\And
V^{\pi_{g}}(s_{t_{\gamma}+1};\gamma+\Delta_{\gamma})<V^{\pi_{d}}(s_{t_{\gamma}+1};\gamma+\Delta_{\gamma})\\\
&\implies
r_{g}(\gamma+\Delta_{\gamma})^{N-t_{\gamma}}-r_{b}\left(\frac{1-(\gamma+\Delta_{\gamma})^{N-t_{\gamma}}}{1-(\gamma+\Delta_{\gamma})}\right)>0\And
r_{g}(\gamma+\Delta_{\gamma})^{N-t_{\gamma}-1}-r_{b}\left(\frac{1-(\gamma+\Delta_{\gamma})^{N-t_{\gamma}-1}}{1-(\gamma+\Delta_{\gamma})}\right)<0\\\
&\implies
r_{g}\left(\frac{(\gamma+\Delta_{\gamma})^{N-t_{\gamma}}(1-\gamma-\Delta_{\gamma})}{1-(\gamma+\Delta)^{N-t_{\gamma}}}\right)<r_{b}<r_{g}\left(\frac{(\gamma+\Delta_{\gamma})^{N-t_{\gamma}-1}(1-\gamma-\Delta_{\gamma})}{1-(\gamma+\Delta)^{N-t_{\gamma}-1}}\right)\end{split}$
(20)

These constraints are satisfied so long as there exists parameters so that the
following inequality holds:

$\displaystyle\begin{split}&r_{g}\left(\frac{(\gamma+\Delta_{\gamma})^{N-t_{\gamma}}(1-\gamma-\Delta_{\gamma})}{1-(\gamma+\Delta)^{N-t_{\gamma}}}\right)<r_{g}\left(\frac{(\gamma+\Delta_{\gamma})^{N-t_{\gamma}-1}(1-\gamma-\Delta_{\gamma})}{1-(\gamma+\Delta)^{N-t_{\gamma}-1}}\right)\\\
&\implies\frac{(\gamma+\Delta_{\gamma})^{N-t_{\gamma}}}{1-(\gamma+\Delta)^{N-t_{\gamma}}}<\frac{(\gamma+\Delta_{\gamma})^{N-t_{\gamma}-1}}{1-(\gamma+\Delta)^{N-t_{\gamma}-1}}\\\
&\implies\frac{\gamma+\Delta_{\gamma}}{1-(\gamma+\Delta_{\gamma})^{N-t_{\gamma}}}<\frac{1}{1-(\gamma+\Delta_{\gamma})^{N-t_{\gamma}-1}}\\\
&\implies\gamma+\Delta_{\gamma}<1,\end{split}$ (21)

which is true by definition. So, there exists $\Delta_{\gamma}$ that defines
any $t_{b}\in{0,\ldots,N-1}$.

Furthermore, there exists chainworld parameters $\theta$ that define any
$t_{h}^{0},t_{h}^{\gamma},t_{h}^{b}$, and therefore, any AI policy
$\pi_{AI}\in\bar{\Pi}$. ∎

##  Appendix C Equivalence Proofs

Throughout this section, we will distinguish chainworld parameters from
parameters in other worlds with a $\widehat{\cdot}$. For example, we will
refer to the human’s goal utility $r_{g}$ with $\widehat{r}_{g}$.

###  C.1. Proof of equivalence with _monotonic chainworlds_

[Chainworld and monotonic chainworld equivalence] If
$\mathcal{M}\in\mathscr{M}_{\text{mono}}$, then there exists
$\widehat{\mathcal{M}}\in\mathscr{M}_{\text{chain}}$ such that
$\mathcal{M}\equiv\widehat{\mathcal{M}}$ with identity mapping $f(s)=s$ and
$g_{s}(a)=a$.

###### Proof.

Like our chainworlds, optimal human policies in monotonic chainworlds are
defined by a threshold. Specifically, under each AI intervention, monotonic
chainworlds result in the thresholds $t_{h}^{0},t_{h}^{\gamma},t_{h}^{b}$
under AI actions $0,a_{\gamma},a_{b}$, respectively. As a result, the proof
from section 5.1 holds exactly for monotonically increasing chainworlds. ∎

###  C.2. Proof of equivalence with _negative effect of AI intervention_

[Chainworld equivalence under negative effect of AI intervention] If
$\mathcal{M}$ has the same states, actions, rewards, transitions, and discount
as a chainworld except that $\Delta_{b}>0$ or $\Delta_{\gamma}<0$, then there
exists a chainworld MDP $\mathcal{M}_{\theta}\in\mathscr{M}_{\text{chain}}$
such that $\mathcal{M}_{\theta}\equiv\mathcal{M}$.

###### Proof.

  * •

As it is defined in eq. 6, the AI’s optimal policy _only depends on the
minimum human threshold_ ,
$t_{h}^{\text{min}}=\min\left\\{t_{h}^{0},t_{h}^{0},t_{h}^{0}\right\\}$

  * •

If the AI agent’s intervention on $\gamma$ has the negative intended effect,
then $t_{h}^{\gamma}>t_{h}^{0}$.

  * •

Similarly, if the AI intervention on $r_{b}$ has the negative intended effect,
then $t_{h}^{b}>t_{h}^{0}$.

  * •

As a result, $t_{h}^{\text{min}}=t_{h}^{0}$.

  * •

As shown in section 5.1, if $t_{h}^{\text{min}}=t_{h}^{0}$, then this results
in an optimal AI policy where $\pi_{AI}^{*}(s)=0$ for all states
$s\in\mathcal{S}_{AI}$.

  * •

$\pi_{AI}^{*}\in\bar{\Pi}$, because this is a “three-window” AI policy where
the intervention window is size $0$.

  * •

Since $\pi_{AI}^{*}\in\bar{\Pi}$, it is in the equivalence class of
chainworlds.

∎

###  C.3. Proof that AI equivalence is achieved if human MDPs are equivalent

In the remaining proofs, we will first equate the rewards and transitions of
two human MDPs and show that this carries into AI equivalence. In this
section, we prove that two human-level MDPs with the same rewards and
transitions result in two AI MDPs with the same optimal policy.

Suppose we are given two different human MDPs with matching discount factors,
$\mathcal{M}_{h}=\langle\mathcal{S}_{h},\mathcal{A}_{h},T_{h},R_{h},\gamma_{h}\rangle\text{
and
}\widehat{\mathcal{M}}_{h}=\langle\widehat{\mathcal{S}}_{h},\widehat{\mathcal{A}}_{h},\widehat{T}_{h},\widehat{R}_{h},\gamma_{h}\rangle,$
whose rewards and transitions are equivalent under some mapping between the
state and action spaces. Specifically, under a state mapping
$f:\mathcal{S}_{h}\rightarrow\widehat{\mathcal{S}}_{h}$ and (state-specific)
action mapping
$g_{s_{h}}:\mathcal{A}_{h}\rightarrow\widehat{\mathcal{A}}_{h}$,

$T_{h}(s_{h},a_{h},s^{\prime}_{h})=\widehat{T}_{h}(\
f(s_{h}),g_{s_{h}}(a_{h}),f(s^{\prime}_{h})\ )\text{ and
}R_{h}(s_{h},a_{h})=\widehat{R}_{h}(\ f(s_{h}),g_{s_{h}}(a_{h})\
),\quad\quad\forall s_{h}\in\mathcal{S}_{h},A_{h}\in\mathcal{A}_{h}.$

Assume that both human agents follow the same action selection strategy. For
example, both agents select actions according to the _optimal policy_ for
$\mathcal{M}$ and $\widehat{\mathcal{M}}$.

Suppose we are also given two AI MDPs that correspond to each of the
respective human MDPs,

$\mathcal{M}_{AI}=\langle\mathcal{S}_{AI},\mathcal{A}_{AI},T_{AI},R_{AI},\gamma_{AI}\rangle\text{
and
}\widehat{\mathcal{M}}_{AI}=\langle\widehat{\mathcal{S}}_{AI},{\mathcal{A}}_{AI},\widehat{T}_{AI},\widehat{R}_{AI},\gamma_{AI}\rangle,$

with matching discount functions and action spaces. The AI rewards are also
mapping under the mappings so that

$R_{AI}([s_{h},a_{h}],a_{AI})=\widehat{R}_{AI}(\
[f(s_{h}),g_{s_{h}}(a_{h})],a_{AI}\ ),\quad\quad\forall
s_{h}\in\mathcal{S}_{h},A_{h}\in\mathcal{A}_{h}.$

Then, the optimal AI policies are equal, where

$\pi_{AI}^{*}(\ [s_{h},a_{h}]\ )=\widehat{\pi}_{AI}^{*}(\ [\ f(s_{h})\ ,\
g_{s_{h}}(a_{h})\ ]\ ),\quad\quad\forall
s_{h}\in\mathcal{S}_{h},A_{h}\in\mathcal{A}_{h}.$

###### Proof.

$\displaystyle\begin{split}&\pi^{*}_{AI}(\ [s_{h},a_{h}]\ )\\\
&=\operatorname*{arg\,max}\limits_{a_{AI}}Q_{AI}^{*}(\ [s_{h},a_{h}]\
,a_{AI})\\\
&=\operatorname*{arg\,max}\limits_{a_{AI}}R_{AI}([s_{h},a_{h}],a_{AI})+\gamma_{AI}\sum\limits_{s_{h}^{\prime},a_{h}^{\prime}}T_{AI}([s_{h},a_{h}],a_{AI},[s_{h}^{\prime},a_{h}^{\prime}])\operatorname*{arg\,max}\limits_{a_{AI}^{\prime}}Q_{AI}^{*}(\
[s_{h}^{\prime},a_{h}^{\prime}]\ ,a_{AI})\\\
&=\operatorname*{arg\,max}\limits_{a_{AI}}R_{AI}([s_{h},a_{h}],a_{AI})+\gamma_{AI}\sum\limits_{s_{h}^{\prime},a_{h}^{\prime}}T_{h}(s_{h},a^{\prime}_{h},s_{h}^{\prime})\pi_{h}(a_{h}\
|\ s_{h},a_{AI})\operatorname*{arg\,max}\limits_{a_{AI}^{\prime}}Q_{AI}^{*}(\
[s_{h}^{\prime},a_{h}^{\prime}]\ ,a_{AI}).\end{split}$ (22)

Since we are given $R_{AI}\equiv\widehat{R}_{AI},T_{h}\equiv\widehat{T}_{h}$
under mappings $f,g$,

$\displaystyle\begin{split}&=\operatorname*{arg\,max}\limits_{a_{AI}}\widehat{R}_{AI}(\
[f(s_{h}),g_{s_{h}}(a_{h})],a_{AI}\
)+\gamma_{AI}\sum\limits_{s_{h}^{\prime},a_{h}^{\prime}}\widehat{T}_{h}(\
f(s_{h}),g_{s_{h}}(a_{h}),f(s^{\prime}_{h})\ )\pi_{h}(a_{h}\ |\
s_{h},a_{AI})\operatorname*{arg\,max}\limits_{a_{AI}^{\prime}}Q_{AI}^{*}(\
[s_{h}^{\prime},a_{h}^{\prime}]\ ,a_{AI}).\end{split}$ (23)

Note that $T_{h}\equiv\widehat{T}_{h}\text{ and
}R_{h}\equiv\widehat{R}_{h}\text{ under }f,g\implies
Q_{h}^{*}(s_{h},a_{h})=\widehat{Q}_{h}^{*}(\ f(s_{h}),g_{s_{h}}(a_{h})\
)\forall s_{h},a_{h}\implies\pi_{h}(s_{h},a_{h})=\widehat{\pi}_{h}(\
f(s_{h}),g_{s_{h}}(a_{h})\ )\forall s_{h},a_{h}$. So,

$\displaystyle\begin{split}&=\operatorname*{arg\,max}\limits_{a_{AI}}\widehat{R}_{AI}(\
[f(s_{h}),g_{s_{h}}(a_{h})],a_{AI}\
)+\gamma_{AI}\sum\limits_{s_{h}^{\prime},a_{h}^{\prime}}\widehat{T}_{h}(\
f(s_{h}),g_{s_{h}}(a_{h}),f(s^{\prime}_{h})\ )\widehat{\pi}_{h}(\
g_{s_{h}}(a_{h})\ |\ f(s_{h}),a_{AI}\
)\operatorname*{arg\,max}\limits_{a_{AI}^{\prime}}Q_{AI}^{*}(\
[s_{h}^{\prime},a_{h}^{\prime}]\ ,a_{AI})\\\
&=\operatorname*{arg\,max}\limits_{a_{AI}}\widehat{Q}_{AI}^{*}(\
[f(s_{h}),g_{s_{h}}(s_{h})],g_{s_{h}}\ )\\\ &=\widehat{\pi}_{AI}^{*}(\
[f(s_{h}),g_{s_{h}}(s_{h})]\ ).\end{split}$ (24)

∎

###  C.4. Proof of equivalence for _progress worlds_

(a) Example of a progress world, because $s$ and $s^{\prime}$ are the same
distance from $s_{g}$ and are the same distance from $s_{d}$.

(b) Example of a graph that is _not_ a progress world, because $s$ and
$s^{\prime}$ are the same distance from $s_{g}$ but different distances from
$s_{d}$.

######  Definition  (Progress worlds).

Let
$\mathcal{M}=\langle\mathcal{S},\mathcal{A},T,R,\gamma,\Delta_{\gamma},\Delta_{b}\rangle$
denote a graph MDP, defined as follows:

  * •

$\mathcal{S}=\mathcal{S}_{1}\times\mathcal{S}_{2}\times\ldots\times\mathcal{S}_{D}$.
The states are $D$ dimensional and discrete, where $\mathcal{S}_{d}$ refers to
the set of discrete states in the $d$-th dimension. There is an absorbing goal
state $s_{g}\in\mathcal{S}$, and an absorbing disengagement state
$s_{d}\in\mathcal{S}$.

  * •

$\mathcal{A}=\mathcal{A}_{g}\cup\mathcal{A}_{d}$. Actions allow movement
bewteen states on the graph. The set of action $\mathcal{A}_{g}$ are actions
that lead closer to the goal state. The set of actions $\mathcal{A}_{d}$ are
actions that lead closer to the disengagement state.

  * •

The graph must be path-connected. The transitions are parametrized by $p$; the
agent moves in the intended direction with probability $p$ and stays in place
with probability $1-p$. The states $s_{d}$ and $s_{g}$ are absorbing.

  * •

${R}_{\text{prog}}(s,a)=\begin{cases}r_{d}>0,&s=s_{d}\\\ r_{g}>0,&s=s_{g}\\\
r_{b}<0,&a\in\mathcal{A}_{g}\end{cases}$.

  * •

$\gamma\in(0,1)$

  * •

$\Delta_{\gamma},\Delta_{b}$. AI action $a=a_{\gamma}$ increases $\gamma$ by
$\Delta_{\gamma}$ and AI action $a=a_{b}$ reduces $r_{b}$ by $\Delta_{b}$.

In summary, a progress world
$\mathcal{M}_{\theta}\in{\mathscr{M}}_{\text{prog}}$ is parameterized by
$\theta=\\{\underbrace{p}_{\text{transitions}},\underbrace{r_{g},r_{d},r_{b}}_{\text{rewards}},\underbrace{\gamma}_{\text{discount}},\underbrace{\Delta_{\gamma},\Delta_{b}}_{\text{app
effect}}\\}$.

AI equivalence with chainworld holds for a subset of progress worlds. This is
the subset of worlds for which no two states on the graph are the same
distance from the goal state $s_{g}$, but different distances from the
disengagement state $s_{d}$ (see example in fig. 8(b)). At a high level, this
means that all shortest paths between the goal and disengagement state are the
same length, which means that a single chainworld can represent all paths (and
therefore, the entire world). We prove this in section C.4.

[Chainworld and progress world equivalence] Suppose
$\mathcal{M}_{\theta}\in{\mathscr{M}}_{\text{prog}}$. Let $d(s,s^{\prime})$
denote the shortest graph distance from $s$ to $s^{\prime}$. Then, there
exists $\widehat{\mathcal{M}}_{\widehat{\theta}}\in\mathscr{M}_{\text{chain}}$
such that $\mathcal{M}_{\theta}\equiv\widehat{\mathcal{M}}_{\widehat{\theta}}$
under state mapping,

$f(s)=\begin{cases}\widehat{s}_{d(s,s_{d})-1},&d(s,s_{d})>0\\\
\widehat{s}_{d},&d(s,s_{d})=0,\end{cases}$

and action mapping $g_{s}(a)=\mathbb{I}\left\\{a\in\mathcal{A}_{g}\right\\}$,
where actions in the progress-world that move the human toward the goal
correspond to chainworld actions $\widehat{a}=1$.

###### Proof.

Consider the following chainworld parameters $\widehat{\theta}$:

  * •

Length of chain $\widehat{N}=d(s_{g},s_{d})$

  * •

Goal reward $\widehat{r}_{g}=r_{g}$

  * •

Disengagement reward $\widehat{r}_{d}=r_{d}$

  * •

Progress loss reward $\widehat{r}_{\ell}=0$

  * •

Burden reward $\widehat{r}_{b}=r_{b}$

  * •

Probability of moving toward goal $\widehat{p}_{g}=p$

  * •

Probability of losing progress $\widehat{p}_{\ell}=p$

  * •

Probability of disengagement $\widehat{p}_{d}=0$

  * •

Probability of disengagement at state $0$ $\widehat{p}_{d0}=p$

  * •

Discount factor $\widehat{\gamma}=\gamma$

  * •

Effect of AI intervention on discount
$\widehat{\Delta}_{\gamma}=\Delta_{\gamma}$

  * •

Effect of AI intervention on burden $\widehat{\Delta}_{b}=\Delta_{b}$

Notes | $d(s,s_{d})$ | $a$ | $d(s^{\prime},s_{d})$ | $f(s)$ | $g_{s}(a)$ | $f(s^{\prime})$ | $T(s,a,s^{\prime})$ | $\widehat{T}(\ f(s),g_{s}(a),f(s^{\prime})\ )$
---|---|---|---|---|---|---|---|---
$d(s,s_{d})>0$ | $d(s,s_{d})$ | $a\in\mathcal{A}_{g}$ | $d(s,s_{d})+1$ | $\widehat{s}_{d(s,s_{d})-1}$ | $1$ | $\widehat{s}_{d(s,s_{d})}$ | $p$ | $\widehat{p}_{g}$
$d(s,s_{d})>0$ | $d(s,s_{d})$ | $a\in\mathcal{A}_{g}$ | $d(s,s_{d})$ | $\widehat{s}_{d(s,s_{d})-1}$ | $1$ | $\widehat{s}_{d(s,s_{d})}$ | $1-p$ | $1-\widehat{p}_{g}$
$d(s,s_{d})>1$ | $d(s,s_{d})$ | $a\in\mathcal{A}_{d}$ | $d(s,s_{d})-1$ | $\widehat{s}_{d(s,s_{d})-1}$ | $0$ | $\widehat{s}_{d(s,s_{d})}$ | $p$ | $\widehat{p}_{\ell}$
$d(s,s_{d})>1$ | $d(s,s_{d})$ | $a\in\mathcal{A}_{d}$ | $d(s,s_{d})$ | $\widehat{s}_{d(s,s_{d})-1}$ | $0$ | $\widehat{s}_{d(s,s_{d})-1}$ | $1-p$ | $1-\widehat{p}_{\ell}$
| $d(s,s_{d})=1$ | $a\in\mathcal{A}_{d}$ | $d(s,s_{d})=0$ | $\widehat{s}_{0}$ | $0$ | $\widehat{s}_{d}$ | $p$ | $\widehat{p}_{d0}$
$d(s,s_{d})=1$ | $1$ | $a\in\mathcal{A}_{d}$ | $0$ | $\widehat{s}_{0}$ | $0$ | $\widehat{s}_{d}$ | $1-p$ | $1-\widehat{p}_{d0}$
Table 3. Equivalence of progress world transitions. All possible _progress world_ transitions in $T$ are equivalent to the _chainworld_ transitions in $\widehat{T}$ under mappings $f$ and $g$. Transitions $T(s,a,s^{\prime})$ with probability $0$ are not shown; these are clearly still $0$ probability under $\widehat{T}(\ f(s),g_{s}(a),f(s^{\prime})\ )$, since the grouped rows sum to $1$ for both $T$ and $\widehat{T}$. Notes | $s$ | $a$ | $s^{\prime}$ | $f(s)$ | $g_{s}(a)$ | $f(s^{\prime})$ | $R(s,a)$ | $\widehat{R}(\ f(s),g_{s}(a)\ )$
---|---|---|---|---|---|---|---|---
| $s_{d}$ | — | — | $\widehat{s}_{d}$ | — | — | $r_{d}$ | $\widehat{r}_{d}$
| $s_{g}$ | — | — | $\widehat{s}_{g}$ | — | — | $r_{g}$ | $\widehat{r}_{g}$
$d(s^{\prime},s_{d})<d(s,s_{d})$ where $s^{\prime}$ results from action $a$ | — | $a$ | — | — | 1 | — | $r_{b}$ | $\widehat{r}_{b}$
Table 4. Equivalence of progress world rewards. All possible _progress world_
rewards in $R$ are equivalent to the _chainworld_ rewards in $\widehat{R}$
under mappings $f$ and $g$. We use “—” to represent any action or state. For
all other $s,a,s^{\prime}$ combinations not shown,
$R(s,a,s^{\prime})=\widehat{R}(\ f(s),g_{s}(a),f(s^{\prime})\ )=0$.

In table 3, we show that $T(s,a,s^{\prime})=\widehat{T}_{\widehat{\theta}}(\
f(s),g_{s}(a),f(s)\ )$ for all
$s\in\mathcal{S},a\in\mathcal{A},s^{\prime}\in\mathcal{S}$. In table 4, we
show that $R(s,a,s^{\prime})=\widehat{R}_{\widehat{\theta}}(\
f(s),g_{s}(a),f(s)\ )$ for all
$s\in\mathcal{S},a\in\mathcal{A},s^{\prime}\in\mathcal{S}$. As a result, we
can invoke section C.3.

∎

###  C.5. Proof of equivalence with _multi-chain disengagement worlds_

######  Definition  (Multi-chain disengagement worlds).

Let ${\mathscr{M}}_{\text{multi}}$ denote the class of multi-chain MDPs, so
that an MDP
$\mathcal{M}\in{\mathscr{M}}_{\text{multi}}=\langle\mathcal{S},\mathcal{A},T,R,\gamma,\Delta_{\gamma},\Delta_{b}\rangle$
is defined as follows:

  * •

$s=[s_{0},s_{2},\ldots,s_{C}]$, where $s_{c}\in\\{0,\ldots,N_{c}\\}$ denotes
the current placement along chain $c$ of length $N_{c}$.

    * –

The first chain, $c=0$, represents the _goal chain_ ; when the human reaches
the end of this chain, they have reached the goal. The set of goal states is
$\mathcal{S}_{g}=\\{s\ |\ s_{0}=N_{0}\\}$.

    * –

The remaining chains, $s_{1},\ldots,s_{C}$, represent _disengagement chains_.
When the human reaches the end of any of these chains, they disengage. The set
of disengagement states is $\mathcal{S}_{d}=\\{s\ |\ \exists
s_{c}=N_{c}\forall c\in\\{1,\ldots,C\\}\\}$

  * •

$\mathcal{A}=\\{0,1,2\\}$. The action $a=1$ allows the human to move along
goal chain $c=0$. The action $a=0$ allows the human to move along the
disengagement chain $c>0$. The action $a=2$ allows the human recover, by
moving backwards on all the disengagement chains.

  * •

$T$. Each chain $c$ is associated with a probability of movement, $p_{c}$,
conditioned on actions as described below:

    * –

When $a=0$, the human loses progress in the goal chain with probability
$p_{\ell}$ and independently moves along each disengagement chain $c>0$ with
probability $p_{c}^{0}$.

    * –

When $a=1$, the human moves along each chain with probability $p_{c}^{1}$.

    * –

When $a=2$, the human moves backwards on each disengagement chain $c>0$ with
probability $1$. The human stays still in the goal chain.

  * •

$R(s,a,s^{\prime})=\begin{cases}r_{d}&s_{c}=N_{c}\text{ for }0<1\leq C\text{
(reached end of a disengagement chain)}\\\ r_{g}&s_{0}=N_{0}\text{ (reached
end of goal chain)}\\\ r_{\ell}&s_{0}^{\prime}<s_{0}\\\ r_{b},&a=1\text{ or
}a=2\end{cases}$,

where $r_{d}>0$, $r_{g}>0$, $r_{b}<0$, $r_{l}<0$.

  * •

$\gamma>0$

  * •

$\Delta=[\Delta_{\gamma},\Delta_{b}]$. AI action $a_{AI}=a_{\gamma}$ increases
$\gamma$ by $\Delta_{\gamma}$ and AI action $a_{AI}=a_{b}$ reduces $r_{b}$ by
$\Delta_{b}$.

In summary, a multi-chain world
$\mathcal{M}_{\theta}\in{\mathscr{M}}_{\text{multi}}$ is parameterized by
$\theta=\\{\underbrace{p_{\ell},p_{1}^{0},p_{2}^{0},\ldots,p_{C}^{0},p_{0}^{1},p_{2}^{1},\ldots,p_{C}^{1}}_{\text{transitions}},\underbrace{r_{g},r_{d},r_{\ell},r_{b}}_{\text{rewards}},\underbrace{\gamma}_{\text{discount}},\underbrace{\Delta_{\gamma},\Delta_{b}}_{\text{app
effect}}\\}$. We prove equivalence for two subsets of multi-chain
disengagement worlds, shown in fig. 9.

(a) Case A

(b) Case B

Figure 9. Two types of multi-chain disengagement worlds

####  C.5.1. Multi-chain Disengagement Case A

[Chainworld equivalence to case A] Suppose
$\mathcal{M}_{\theta}\in{\mathscr{M}}_{\text{multi}}$ is a multi-chain MDP
like the one shown in fig. 9(a), where:

  * •

There are $C$ chains and the disengagement chains are all of length $2$
($N_{c}=2$ for $c=1,\ldots,C-1$). Note that the recovery action $a=2$ is no
longer available in this setting, because the disengagement state is absorbing
and the disengagement chains are of length $2$, so that once the human moves
along any disengagement chain, they have disengaged and cannnot recover.

  * •

When the agent takes action to move along the goal chain, it stays still in
all other disengagement chains. Specifically, movement along all disengagement
chains is impossible when $a=1$ so that $p_{c}^{1}=0$ for all
$c=1,\ldots,N_{c}$. Movement along the goal chain is still possible,
$p_{0}^{1}>0$.

  * •

When $a=0$, the agent either loses progress or disengages, but not both.

Then, there exists a chainworld
$\widehat{\mathcal{M}}_{\widehat{\theta}}\in\mathscr{M}_{\text{chain}}$ such
that $\mathcal{M}_{\theta}\equiv\widehat{\mathcal{M}}_{\widehat{\theta}}$
under state mapping,

$f(s=[s_{0},s_{1},\ldots,s_{C-1}])=\begin{cases}\widehat{s}_{s_{0}},&s_{0}<N_{0}\text{
and }s_{c}=0\text{ for all }c=1,\ldots,C-1\\\ \widehat{s}_{d},&s_{c}=1\text{
for any }c\geq 1\\\ \widehat{s}_{g}&s_{0}=N_{0}\end{cases}$

and action mapping $g_{s}(a)=\mathbb{I}\left\\{a>0\right\\}.$

###### Proof.

Consider the following chainworld parameters $\widehat{\theta}$:

  * •

Length of chain $\widehat{N}=N_{0}$

  * •

Goal reward $\widehat{r}_{g}=r_{g}$

  * •

Disengagement reward $\widehat{r}_{d}=r_{d}$

  * •

Progress loss reward $\widehat{r}_{\ell}=r_{\ell}$

  * •

Burden reward $\widehat{r}_{b}=r_{b}$

  * •

Probability of moving toward goal $\widehat{p}_{g}=p_{0}^{1}$

  * •

Probability of losing progress $\widehat{p}_{\ell}=p_{\ell}$

  * •

Probability of disengagement
$\widehat{p}_{d}=1-\prod\limits_{c=1}^{C}(1-p_{c}^{0})$

  * •

Probability of disengagement at state $0$
$\widehat{p}_{d0}=1-\prod\limits_{c=1}^{C}(1-p_{c}^{0})$

  * •

Discount factor $\widehat{\gamma}=\gamma$

  * •

Effect of AI intervention on discount
$\widehat{\Delta}_{\gamma}=\Delta_{\gamma}$

  * •

Effect of AI intervention on burden $\widehat{\Delta}_{b}=\Delta_{b}$

Notes | $s$ | $a$ | $s^{\prime}$ | $f(s)$ | $g_{s}(a)$ | $f(s^{\prime})$ | $T(s,a,s^{\prime})$ | $\widehat{T}(\ f(s),g_{s}(a),f(s^{\prime})\ )$
---|---|---|---|---|---|---|---|---
$n=\\{0,\ldots,N_{0}-1\\}$ | $[n,0,0,\ldots,0]$ | $1$ | $[n,0,0,\ldots,0]$ | $\widehat{s}_{n}$ | $1$ | $\widehat{s}_{n}$ | $1-p_{0}^{1}$ | $1-\widehat{p}_{g}$
$n=\\{0,\ldots,N_{0}-1\\}$ | $[n,0,0,\ldots,0]$ | $1$ | $[n+1,0,0,\ldots,0]$ | $\widehat{s}_{n}$ | $1$ | $\widehat{s}_{n+1}$ | $p_{0}^{1}$ | $\widehat{p}_{g}$
$n=\\{1,\ldots,N_{0}-1\\}$ | $[n,0,0,\ldots,0]$ | $0$ | $[n,0,0,\ldots,0]$ | $\widehat{s}_{n}$ | $0$ | $\widehat{s}_{n}$ | $\prod\limits_{c=1}^{C}(1-p_{c}^{0})-p_{\ell}$ | $1-\widehat{p}_{\ell}-\widehat{p}_{d}$
$n=\\{1,\ldots,N_{0}-1\\}$ | $[n,0,0,\ldots,0]$ | $0$ | $[n,\ldots,s_{c},\ldots],\exists s_{c}=1$ | $\widehat{s}_{n}$ | $0$ | $\widehat{s}_{d}$ | $1-\prod\limits_{c=1}^{C}(1-p_{c}^{0})$ | $\widehat{p}_{d}$
$n=\\{1,\ldots,N_{0}-1\\}$ | $[n,0,0,\ldots,0]$ | $0$ | $[n-1,0,0,\ldots,0]$ | $\widehat{s}_{n}$ | $0$ | $\widehat{s}_{n-1}$ | $p_{\ell}$ | $\widehat{p}_{\ell}$
| $[0,0,\ldots,0]$ | $0$ | $[0,0,\ldots,0]$ | $\widehat{s}_{0}$ | $0$ | $\widehat{s}_{0}$ | $\prod\limits_{c=1}^{C}(1-p_{c}^{0})$ | $1-\widehat{p}_{d}$
| $[0,0,\ldots,0]$ | $0$ | $[n,\ldots,s_{c},\ldots],\exists s_{c}=1$ | $\widehat{s}_{0}$ | $0$ | $\widehat{s}_{d}$ | $1-\prod\limits_{c=1}^{C}(1-p_{c}^{0})$ | $\widehat{p}_{d}$
Table 5. Equivalence of multi-chain case A transitions. All possible _multi-chain world_ transitions in $T$ are equivalent to the _chainworld_ transitions in $\widehat{T}$ under mappings $f$ and $g$. Transitions $T(s,a,s^{\prime})$ with probability $0$ are not shown; these are clearly still $0$ probability under $\widehat{T}(\ f(s),g_{s}(a),f(s^{\prime})\ )$, since the grouped rows sum to $1$ for both $T$ and $\widehat{T}$. Notes | $s$ | $a$ | $s^{\prime}$ | $f(s)$ | $g_{s}(a)$ | $f(s^{\prime})$ | $R(s,a)$ | $\widehat{R}(\ f(s),g_{s}(a)\ )$
---|---|---|---|---|---|---|---|---
| $[N_{0},\ldots]$ | — | — | $\widehat{s}_{g}$ | — | — | $r_{g}$ | $\widehat{r}_{g}$
| $[s_{0},\ldots,s_{c},\ldots],\exists s_{c}=1$ | — | — | $\widehat{s}_{d}$ | — | — | $r_{d}$ | $\widehat{r}_{d}$
| $[n,0,0,\ldots,0]$ | — | $[n-1,0,0,\ldots,0]$ | $\widehat{s}_{n}$ | 0 | $\widehat{s}_{n-1}$ | $r_{\ell}$ | $\widehat{r}_{\ell}$
| — | $1$ | — | — | $1$ | — | $r_{b}$ | $\widehat{r}_{b}$
Table 6. Equivalence of multi-chain case A rewards. All possible _multi-chain
world_ rewards in $R$ are equivalent to the _chainworld_ rewards in
$\widehat{R}$ under mappings $f$ and $g$. We use “—” to represent any action
or state. For all other $s,a,s^{\prime}$ combinations not shown,
$R(s,a,s^{\prime})=\widehat{R}(\ f(s),g_{s}(a),f(s^{\prime})\ )=0$.

In table 5, we show that $T(s,a,s^{\prime})=\widehat{T}_{\widehat{\theta}}(\
f(s),g_{s}(a),f(s)\ )$ for all
$s\in\mathcal{S},a\in\mathcal{A},s^{\prime}\in\mathcal{S}$. In table 6, we
show that $R(s,a,s^{\prime})=\widehat{R}_{\widehat{\theta}}(\
f(s),g_{s}(a),f(s)\ )$ for all
$s\in\mathcal{S},a\in\mathcal{A},s^{\prime}\in\mathcal{S}$. As a result, we
can invoke section C.3. ∎

####  C.5.2. Multi-chain Disengagement Case B

This world represents situations in which the human makes progress toward a
goal (e.g. a rehabilitated shoulder), but must also manage an additional
factor that may cause them to disengage (e.g. exercise fatigue). To manage
this, the human has an additional action, $a=2$, which allows them to recover
from fatigue in exchange for not making progress toward the goal (e.g. taking
a rest day).

[Chainworld equivalence to case B] If
$\mathcal{M}_{\theta}\in{\mathscr{M}}_{\text{multi}}$ is a multi-chain MDP
with $C=2$, a goal chain ($c=0$) and a disengagement chain ($c=1$), rewards
$r_{d}=0,r_{\ell}=0$, and transitions:

  * •

When $a=0$, the agent stays still in the goal chain ($p_{\ell}=0$) and always
moves along the disengagement chain ($p_{1}^{0}=1$)

  * •

When $a=1$, the agent deterministically makes progress along both chains,
$p_{0}^{1}=p_{1}^{1}=1$.

  * •

When $a=2$, the agent deterministically moves backward on the disengagement
chain.

then there exists
$\widehat{\mathcal{M}}_{\widehat{\theta}}\in\mathscr{M}_{\text{chain}}$ such
that $\mathcal{M}_{\theta}\equiv\widehat{\mathcal{M}}_{\widehat{\theta}}$
under state mapping,

$f(s=[s_{0},s_{1}])=2N_{0}-N_{0}+s_{0}-\mathbb{I}\left\\{N_{0}-s_{0}>N_{1}-s_{1}\right\\}(N_{0}-s_{0}-N_{1}+s_{1})$

,

and action mapping $g_{s}(a)=\mathbb{I}\left\\{a>0\right\\}.$

###### Proof.

Consider the following chainworld parameters $\widehat{\theta}$:

  * •

Length of chain $\widehat{N}=2N_{0}$

  * •

Goal reward $\widehat{r}_{g}=r_{g}$

  * •

Disengagement reward $\widehat{r}_{d}=r_{d}$

  * •

Progress loss reward $\widehat{r}_{\ell}=0$

  * •

Burden reward $\widehat{r}_{b}=r_{b}$

  * •

Probability of moving toward goal $\widehat{p}_{g}=1$

  * •

Probability of losing progress $\widehat{p}_{\ell}=1$

  * •

Probability of disengagement $\widehat{p}_{d}=0$

  * •

Probability of disengagement at state $0$, $\widehat{p}_{d0}=1$

  * •

Discount factor $\widehat{\gamma}=\gamma$

  * •

Effect of AI intervention on discount
$\widehat{\Delta}_{\gamma}=\Delta_{\gamma}$

  * •

Effect of AI intervention on burden $\widehat{\Delta}_{b}=\Delta_{b}$

Notes | $s$ | $a$ | $s^{\prime}$ | $f(s)$ | $g_{s}(a)$ | $f(s^{\prime})$ | $T(s,a,s^{\prime})$ | $\widehat{T}(\ f(s),g_{s}(a),f(s^{\prime})\ )$
---|---|---|---|---|---|---|---|---
Movement along chain $0$ and chain $1$ | $[s_{0},s_{1}]$ | $1$ | $[s_{0}+1,s_{1}+1]$ | $\widehat{s}_{f(s)}$ | $1$ | $\widehat{s}_{f(s)+1}$ | $1$ | $\widehat{p}_{g}=1$
Movement along only chain $1$ | $[s_{0},s_{1}]$ | $0$ | $[s_{0},s_{1}+1]$ | $\widehat{s}_{f(s)}$ | $0$ | $\widehat{s}_{f(s)-1}$ | $1$ | $\widehat{p}_{\ell}=1$
Reaching end of chain $1$ | $[s_{0},N_{1}-1]$ | $0$ | $[s_{0},N_{1}]$ | $\widehat{s}_{f(s)}$ | $0$ | $\widehat{s}_{f(s)-1}$ | $1$ | $\widehat{p}_{d0}=1$
Backwards along chain $1$, for $s_{1}>0$ | $[s_{0},s_{1}]$ | $2$ | $[s_{0},s_{1}-1]$ | $\widehat{s}_{f(s)}$ | $1$ | $\widehat{s}_{f(s)+1}$ | $1$ | $\widehat{p}_{g}=1$
Table 7. Equivalence of multi-chain case B transitions. All possible _multi-chain_ transitions in $T$ are equivalent to the _chainworld_ transitions in $\widehat{T}$ under mappings $f$ and $g$. Transitions $T(s,a,s^{\prime})$ with probability $0$ are not shown; these are clearly still $0$ probability under $\widehat{T}(\ f(s),g_{s}(a),f(s^{\prime})\ )$, since all rows are $1$ for both $T$ and $\widehat{T}$. Notes | $s$ | $a$ | $s^{\prime}$ | $f(s)$ | $g_{s}(a)$ | $f(s^{\prime})$ | $R(s,a)$ | $\widehat{R}(\ f(s),g_{s}(a)\ )$
---|---|---|---|---|---|---|---|---
| — | — | $[N_{0},s_{1}]$ | $\widehat{s}_{g}$ | — | — | $r_{g}$ | $\widehat{r}_{g}$
| — | — | $[s_{0},N_{1}]$ | $\widehat{s}_{d}$ | — | — | $r_{d}$ | $\widehat{r}_{d}$
| — | $2$ | — | — | $1$ | — | $r_{b}$ | $\widehat{r}_{b}$
| — | $1$ | — | — | $1$ | — | $r_{b}$ | $\widehat{r}_{b}$
Table 8. Equivalence of multi-chain case B rewards. All possible _multi-chain_
rewards in $R$ are equivalent to the _chainworld_ rewards in $\widehat{R}$
under mappings $f$ and $g$. We use “—” to represent any action or state. For
all other $s,a,s^{\prime}$ combinations not shown,
$R(s,a,s^{\prime})=\widehat{R}(\ f(s),g_{s}(a),f(s^{\prime})\ )=0$.

In table 7, we show that $T(s,a,s^{\prime})=\widehat{T}_{\widehat{\theta}}(\
f(s),g_{s}(a),f(s)\ )$ for all
$s\in\mathcal{S},a\in\mathcal{A},s^{\prime}\in\mathcal{S}$. In table 8, we
show that $R(s,a,s^{\prime})=R_{\widehat{\theta}}(\ f(s),g_{s}(a),f(s)\ )$ for
all $s\in\mathcal{S},a\in\mathcal{A},s^{\prime}\in\mathcal{S}$. As a result,
we can invoke section C.3.

∎

##  Appendix D Experimental Details

###  D.1. Environment descriptions

Throughout the experiments, we fix (do not sample) the following parameters
per individual, to make the methods easier to compare:
$p_{g}=1,\Delta_{\gamma}=0.3,\Delta_{b}=-0.4$. All methods (excluding the
oracle) do not have access to any of these parameters and must infer them.

_Chainworld environment_ The chainworld environment is described in the main
body of the text. Every individual’s parameters are sampled uniformly from the
following ranges:

  * •

$r_{b}:[-1,-0.2]$

  * •

$r_{d}:[0,1]$

  * •

$r_{\ell}:[-1,0]$

  * •

$r_{g}:[5,15]$

  * •

$\gamma_{h}:[0.01,0.99]$

  * •

$p_{d}:[0.1,0.5]$

  * •

$p_{d0}:[p_{d},0.5]$

  * •

$p_{\ell}:[0,0.4]$

_Noisy parameters experiments._ The _mean_ parameter value for each chainworld
human is sampled as in the standard chainworld environment above. Then, every
timestep, the parameter of interest is sampled uniformly from a range
surrounding this mean, as described in the main body of the text. For example,
if the environment is testing sensitivity to noise in burden $r_{b}$, then
every timestep, $r_{b}\sim\text{Uniform}(\bar{r}_{b}-c,\bar{r}_{b}+c)$, where
$\bar{r}_{b}$ is the mean burden and $c=\epsilon 5$ is the range such that
$\epsilon\in[0,1]$ is the error level and $5$ is the range assigned to the
reward parameters.

_Distance mapping (gridworld) experiments._ The gridworld environment is
described in the main body of the text. The gridworld has width $X$ and height
$Y$. Every individual’s parameters are sampled uniformly from the following
ranges:

  * •

$r_{b}:[-1,-0.2]$

  * •

$\gamma_{h}:[0.01,0.99]$

  * •

$p:[0.5,1.0]$

  * •

$r_{g}:[5\frac{X}{8},10\frac{X}{8}]$

  * •

$r_{d}:[0,\frac{Y}{5}]$

We scale the values of the rewards to the size of the gridworld.

###  D.2. Definition of AI agent

The AI actions are always $\\{0,a_{\gamma},a_{b}\\}$ for all experiments and
the transition is computed directly off of human agent transitions.

_AI states._ The AI states $s_{AI}=[s_{h},a_{h}]$ are composed of the human’s
current state and previous reward. An AI that uses a chainworld has state
space of size $N\times 2$. An AI that plans directly in the gridworld has
state space of size $X\times Y\times 4$.

_AI rewards._ In all experiments, the rewards are as follows:

$R_{AI}(a_{AI},a_{AI})=\begin{cases}1,&s_{h}=s_{g}\\\ -50,&s_{h}=s_{d}\\\
-1,&a_{AI}\neq 0\\\ -0.5,&\text{otherwise}\end{cases}$ (25)

_AI discount_ We use an AI agent discount of $\gamma_{AI}=0.99$.

###  D.3. Optimizing chainworld parameters

The AI agent must infer the chainworld parameters $\theta$ from the data
$\mathcal{D}_{AI}=\\{(s_{AI},a_{AI},r_{AI},s^{\prime}_{AI})\\}$. Maximizing
the likelihood of the data corresponds to maximizing the likelihood of the
observed transitions, since the chainworld parameters are all contained within
the AI’s transition function:

$P(\mathcal{D}_{AI}\ |\
\theta)=T_{AI}^{\theta}(s_{AI},a_{AI},s^{\prime}_{AI}).$

We follow a simple maximization scheme in which we randomly sample possible
values of $\theta$ and select the one with the highest likelihood. The
candidate $\theta$’s are sampled uniformly from the following ranges:

  * •

$r_{b}:[-1,0]$

  * •

$r_{d}:[0,5]$

  * •

$r_{\ell}:[-5,0]$

  * •

$r_{g}:[5,50]$

  * •

$\gamma_{h}:[0.01,0.99]$

  * •

$p_{g}:[0,1]$

  * •

$p_{\ell}$: see below description

  * •

$p_{d}$: see below description

  * •

$p_{d0}:[0,1]$

  * •

$\tau:[0.01,0.3]$

  * •

$\Delta_{\gamma}:[0,1]$

  * •

$\Delta_{b}:[-1,0]$

The parameters $p_{d}$ and $p_{\ell}$ are constrainted so that
$p_{d}+p_{\ell}\leq 1$. To sample them so that they respect this constraint,
we sample them uniformly from a triangle whose vertices are at
$[0,0],[0,1],[1,0]$. The parameter $\tau$ refers to the noise level in the
softmax action selection policy.

##  Appendix E Additional experiments / results

###  E.1. Effect of learning rate for model-free baseline

In our experiments, the model-free baseline is given a learning rate of $0.9$.
Throughout the results, the model-free baseline performs poorly– equal to
using a random AI policy. This is because it requires _much more data_ to
estimate the optimal value function $Q_{AI}^{*}$ well. As we show in fig. 10,
the model-free baseline requires at least $100$ episodes to learn a policy
that outperforms random.

Figure 10. Model-free baseline performance under different learning rates (LR)
in the chainworld environment.

###  E.2. Results with and without filtering for individuals that cannot reach
goal

(a) Included humans that cannot reach goal state

(b) Filtered out humans that cannot reach goal state

Figure 11. Results with (left) and without (right) individuals that will not
reach the goal state under the oracle AI policy.

###  E.3. Full plots from robustness experiments

Figure 12. Full results for environments where parameter varies every
timestep.

Figure 13. Full results for gridworld environments in which goal state $s_{g}$
moves (and therefore gridworld is not equivalent to chainworld). Chainworld is
correctly modeling distance to goal in top row and distance to disengagement
in bottom row.

Figure 14. Full results for environments in which human follows softmax action
selection.