$\liminf_{T\to\infty}a_{T}\inf_{\widehat{f}_{T}}\sup_{m\in\mathcal{M}}R(\widehat{f}_{T},f_{0})\geq\liminf_{T\to\infty}a_{T}\inf_{\widehat{f}_{T}}\sup_{m\in\mathcal{M}}\mathrm{E}\left[\int_{\mathbb{R}^{d}}|\widehat{f}_{T}(y)-f_{0}(y)|^{2}\Pi_{m}(dy)\right].$
(B.10)

Now, using the definition of $\Pi_{m}$, we can easily check that $\Pi_{m}$ has
the density given by

$\pi_{m}(y)=\int_{\mathbb{R}^{d}}p_{m}(x,y)\Pi_{m}(dx),\qquad
y\in\mathbb{R}^{d}.$

We have by (B.4)

$\inf_{y\in[0,1]^{d}}\pi_{m}(y)\geq\alpha\inf_{y\in[0,1]^{d}}\varphi(y)\Pi_{m}(\mathcal{C}).$

By Markov’s inequality and (B.9), we obtain

$\displaystyle
1-\Pi_{m}(\mathcal{C})=\Pi_{m}(V>R)\leq\frac{1}{R}\int_{\mathbb{R}^{d}}V(x)\Pi_{m}(dx)\leq\frac{1}{3}.$

Hence we conclude

$\inf_{y\in[0,1]^{d}}\pi_{m}(y)\geq\frac{\alpha}{3}\inf_{y\in[0,1]^{d}}\varphi(y).$

Consequently, there is a constant $\rho>0$ depending only on $\mathbf{c},d$
and $\nu$ such that

$\mathrm{E}\left[\int_{[0,1]^{d}}|\widehat{f}_{T}(y)-f_{0}(y)|^{2}\Pi_{m}(dy)\right]\geq\rho\mathrm{E}\left[\int_{[0,1]^{d}}|\widehat{f}_{T}(y)-f_{0}(y)|^{2}dy\right]$
(B.11)

for any estimator $\widehat{f}_{T}$. Combining (B.10) with (B.11) gives the
desired result. ∎

###### Proof of Theorem 4.1.

We write
$\mathcal{M}_{A}=\mathcal{M}\big{(}\mathbf{c},q,\mathbf{d},\mathbf{t},\bm{\beta},A\big{)}$
for short. For each $m\in\mathcal{M}_{A}$ and $T\in\mathbb{N}$, we denote by
$P_{m,T}$ the law of the random vector
$\mathbf{Y}_{T}:=(Y_{-d+1},\dots,Y_{T})^{\prime}$ when $Y_{t}$ are defined by
(4.1). Moreover, we denote by $\mathrm{E}_{m,T}[\cdot]$ the expectation under
$P_{m,T}$.

Now, note that any estimator based on the observation $\\{Y_{t}\\}_{t=1}^{T}$
is also an estimator based on $\mathbf{Y}_{T}$. Therefore, according to
Theorem 2.7 in Tsybakov (2009) and Lemma B.1, it suffices to show that there
is a constant $A>0$ having the following property: For sufficiently large
$T\in\mathbb{N}$, there are an integer $M\geq 1$ and functions
$m_{0},m_{1},\dots,m_{M}\in\mathcal{M}_{A}$ such that

$\|m_{j}-m_{k}\|_{2}^{2}\geq\kappa\phi_{T}\quad\text{for all }0\leq j<k\leq M$
(B.12)

and

$P_{j}\ll P_{0}\quad\text{for all }j=1,\dots,M$ (B.13)

and

$\frac{1}{M}\sum_{j=1}^{M}\mathrm{E}_{m_{j},T}\left[\log\frac{dP_{j}}{dP_{0}}\right]\leq\frac{1}{9}\log
M,$ (B.14)

where $\kappa>0$ is a constant independent of $T$ and $P_{j}:=P_{m_{j},T}$ for
$j=0,1,\dots,M$.

By the proof of (Schmidt-Hieber, 2020, Theorem 3), there is a constant $A>0$
having the following property: For any $T\in\mathbb{N}$, there are an integer
$M\geq 1$ and functions
$f_{(0)},\dots,f_{(M)}\in\mathcal{G}\big{(}q,\mathbf{d},\mathbf{t},\bm{\beta},A\big{)}$
satisfying the following condition:

  1. ($\star$)

For all $0\leq j<k\leq M$,

$T\|f_{(j)}-f_{(k)}\|_{2}^{2}\leq\frac{\log M}{9}$ (B.15)

and

$\|f_{(j)}-f_{(k)}\|_{2}^{2}\geq\kappa\phi_{T},$ (B.16)

where $\kappa>0$ is a constant depending only on $\mathbf{t}$ and
$\bm{\beta}$.

For each $j=1,\dots,M$, we define the function
$m_{j}:\mathbb{R}^{d}\to\mathbb{R}$ as

$m_{j}(x)=\begin{cases}f_{(j)}(x)&\text{if }x\in[0,1]^{d},\\\
0&\text{otherwise}.\end{cases}$

It is evident that $m_{0},m_{1},\dots,m_{M}\in\mathcal{M}_{A}$ when $c_{0}\geq
A$. In the following we show that these $m_{j}$ satisfy (B.12)–(B.14).

First, (B.12) immediately follows from (B.16). Next, it is straightforward to
check (B.13) and

$\displaystyle\frac{dP_{j}}{dP_{0}}(\mathbf{Y}_{T})=\prod_{t=1}^{T}\frac{g(Y_{t}-m_{j}(Y_{t-1},\dots,Y_{t-d}))}{g(Y_{t}-m_{0}(Y_{t-1},\dots,Y_{t-d}))}$

for every $j=1,\dots,d$, where $g$ is the standard normal density. Hence, with
$X_{t}=(Y_{t-1},\dots,Y_{t-d})^{\prime}$,

$\displaystyle\mathrm{E}_{m_{j},T}\left[\log\frac{dP_{j}}{dP_{0}}(\mathbf{Y}_{T})\right]$
$\displaystyle=\frac{1}{2}\sum_{t=1}^{T}\mathrm{E}_{m_{j},T}\left[m_{0}(X_{t})^{2}-m_{j}(X_{t})^{2}+2Y_{t}(m_{j}(X_{t})-m_{0}(X_{t}))\right]$
$\displaystyle=\frac{1}{2}\sum_{t=1}^{T}\mathrm{E}_{m_{j},T}\left[(m_{j}(X_{t})-m_{0}(X_{t}))^{2}\right].$

When $t>d$, conditional on $X_{t-d}$, $X_{t}$ has the density given by

$\prod_{i=1}^{d}g(y_{i}-m(y_{i+1},\dots,y_{d},Y_{t-d-1},\dots,Y_{t-d-i})),\qquad
y\in\mathbb{R}^{d},$

which is bounded by 1. Thus

$\displaystyle\mathrm{E}_{m_{j},T}\left[\log\frac{dP_{j}}{dP_{0}}(\mathbf{Y}_{T})\right]\leq
2A^{2}d+\frac{T}{2}\int_{\mathbb{R}^{d}}(m_{j}(x)-m_{0}(x))^{2}dx\leq
2A^{2}d+\frac{\log M}{18},$

where the last inequality follows from (B.15). Also, by (B.15) and (B.16),
$\kappa T\phi_{T}\leq(\log M)/9$. Since $T\phi_{T}\to\infty$ as $T\to\infty$,
we have $2A^{2}d\leq(\log M)/18$ for sufficiently large $T$. For such $T$, we
have (B.14). This completes the proof. ∎

###  B.4. Proof of Theorem 4.2

Let $\kappa=\max_{i=0,\dots,q}t_{i}/(2\beta_{i}^{*})$. By the proof of Theorem
1 in Schmidt-Hieber (2020), there exist constants $C_{0},C_{S}>0$ such that

$\sup_{m\in\mathcal{M}(\mathbf{c},q,\mathbf{d},\mathbf{t},\bm{\beta},A)}\inf_{f\in\mathcal{F}_{\sigma}(L_{T},N_{T},B_{T},F,S_{T})}\|f-m\|_{\infty}^{2}\leq
C_{0}\phi_{T}=C_{0}T^{-1/(\kappa+1)}$

with $S_{T}:=C_{S}T^{\kappa/(\kappa+1)}\log T$. So the desired result follows
by Proposition 4.1.∎

###  B.5. Proof of Theorem 4.3

For each $m\in\mathcal{M}_{0}(\mathbf{c})$ and $T\in\mathbb{N}$, we denote by
$P_{m,T}$ the law of the random vector
$\mathbf{Y}_{T}:=(Y_{-d+1},\dots,Y_{T})^{\prime}$ when $Y_{t}$ are defined by
(4.1). Moreover, we denote by $\mathrm{E}_{m,T}[\cdot]$ the expectation under
$P_{m,T}$.

Now, note that any estimator based on the observation $\\{Y_{t}\\}_{t=1}^{T}$
is also an estimator based on $\mathbf{Y}_{T}$. Therefore, by Lemma B.1, it
suffices to prove

$\liminf_{T\to\infty}T\inf_{\widehat{f}_{T}}\sup_{m\in\mathcal{M}_{\Phi}^{0}(\mathbf{c},n_{s},C)}\operatorname{E}[\|\widehat{f}_{T}-f_{0}\|_{2}^{2}]>0.$

For each $T=1,2,\dots,$ define $m^{\pm}_{T}:=\pm\frac{1}{2\sqrt{T}}\varphi$
and write $P_{\pm}=P_{m^{\pm}_{T},T}$. Note that
$m^{\pm}_{T}\in\mathcal{M}_{\Phi}^{0}(\mathbf{c},n_{s},C)$ by assumption.
Also, by construction,

$\|m^{+}_{T}-m^{-}_{T}\|_{2}=\frac{1}{\sqrt{T}}\|\varphi\|_{2}=\frac{1}{\sqrt{T}}.$

Moreover, as in the proof of Theorem 4.1, we can show that $P_{+}\ll P_{-}$
and

$\displaystyle\mathrm{E}_{m^{+}_{T},T}\left[\log\frac{dP_{+}}{dP_{-}}(\mathbf{Y}_{T})\right]$
$\displaystyle\leq
2c_{0}^{2}d+\frac{T}{2}\int_{\mathbb{R}^{d}}(m^{+}_{T}(x)-m^{-}_{T}(x))^{2}dx=2c_{0}^{2}d+\frac{1}{2},$

where we used the assumptions $\operatorname{supp}(\varphi)\subset[0,1]^{d}$
and $\|\varphi\|_{\infty}\leq c_{0}$. Consequently, by Eq.(2.9) and Theorem
2.2 in Tsybakov (2009),

$\liminf_{T\to\infty}\inf_{\widehat{f}_{T}}\sup_{m\in\mathcal{M}_{\Phi}^{0}(\mathbf{c},n_{s},C)}P_{m,T}\left(\|\widehat{f}_{T}-f_{0}\|_{2}\geq{1\over
2\sqrt{T}}\right)>0.$

Since

$\displaystyle\liminf_{T\to\infty}T\inf_{\widehat{f}_{T}}\sup_{m\in\mathcal{M}_{\Phi}^{0}(\mathbf{c},n_{s},C)}\operatorname{E}[\|\widehat{f}_{T}-f_{0}\|_{2}^{2}]$
$\displaystyle\geq\frac{1}{4}\liminf_{T\to\infty}\inf_{\widehat{f}_{T}}\sup_{m\in\mathcal{M}_{\Phi}^{0}(\mathbf{c},n_{s},C)}P_{m,T}\left(\|\widehat{f}_{T}-f_{0}\|_{2}\geq{1\over
2\sqrt{T}}\right),$

we complete the proof.∎

###  B.6. Proof of Theorem 4.4

We are going to apply Proposition 4.1. Fix
$m\in\mathcal{M}_{\Phi}^{0}(\mathbf{c},n_{s},C)$ arbitrarily. By definition,
$m$ is of the form

$m(x)=\sum_{i=1}^{n_{s}}\theta_{i}\varphi_{i}(A_{i}x-b_{i}),$

where $A_{i}\in\mathbb{R}^{d\times
d},b_{i}\in\mathbb{R}^{d},\theta_{i}\in\mathbb{R}$ and $\varphi_{i}\in\Phi$
with $|\det
A_{i}|^{-1}\vee|A_{i}|_{\infty}\vee|b_{i}|_{\infty}\vee|\theta_{i}|\leq C$ for
$i=1,\dots,n_{s}$. Since
$\Phi\subset\mathrm{AP}_{\operatorname{ReLU},d}(C_{1},C_{2},D,r)$ by
assumption, for every $i$, there exist parameters $L_{i},N_{i},B_{i},S_{i}>0$
such that $L_{i}\vee N_{i}\vee S_{i}\leq C_{1}(\log_{2}T)^{r}$ and $B_{i}\leq
C_{2}T$ hold and there exists an
$f_{i}\in\mathcal{F}_{\operatorname{ReLU}}(L_{i},N_{i},B_{i})$ such that
$|\theta(f_{i})|_{0}\leq S_{i}$ and
$\|f_{i}-\varphi_{i}\|_{L^{2}([-D,D]^{d})}^{2}\leq 1/T.$ Define

$f(x)=\sum_{i=1}^{n_{s}}\theta_{i}f_{i}(A_{i}x-b_{i}),\qquad
x\in\mathbb{R}^{d}.$

Then

$\displaystyle\|f-m\|_{L^{2}([0,1]^{d})}$ $\displaystyle\leq
C\sum_{i=1}^{n_{s}}\sqrt{\int_{[0,1]^{d}}|f_{i}(A_{i}x-b_{i})-\varphi_{i}(A_{i}x-b_{i})|^{2}dx}$
$\displaystyle\leq
C\sum_{i=1}^{n_{s}}\sqrt{\int_{[-(d+1)C,(d+1)C]^{d}}|f_{i}(y)-\varphi_{i}(y)|^{2}|\det
A_{i}|^{-1}dx}$ $\displaystyle\leq C^{3/2}n_{s}T^{-1/2},$

where we used the assumption $D\geq(d+1)C$ for the last inequality. Also, note
that $\|m\|_{\infty}\leq\sum_{i=0}^{d}c_{i}\leq F$ because
$m\in\mathcal{M}_{0}(\mathbf{c})$ and $\sum_{i=1}^{d}c_{i}<1$. Hence, with
$\widetilde{f}=(-F)\vee(f\wedge F)$, we have $|\widetilde{f}-m|\leq|f-m|$.
Thus $\|\widetilde{f}-m\|_{L^{2}([0,1]^{d})}^{2}\leq C^{3}n_{s}^{2}/T$.
Therefore, the proof is completed once we show that there exists a constant
$C_{S}>0$ such that
$\widetilde{f}\in\mathcal{F}_{\operatorname{ReLU}}(L_{T},N_{T},B_{T},F,C_{S}\log^{r}T)$
for sufficiently large $T$.

By Lemmas II.3–II.4 and A.8 in Elbrächter et al. (2021), there exists a
constant $C^{\prime}_{S}>0$ such that
$f\in\mathcal{F}_{\operatorname{ReLU}}(L_{T},N_{T},B_{T})$ and
$|\theta(f)|_{0}\leq C^{\prime}_{S}\log^{r}T$ for sufficiently large $T$.
Also, note that $x\wedge F=-\operatorname{ReLU}(F-x)+F$ and
$x\vee(-F)=\operatorname{ReLU}(x+F)-F$ for all $x\in\mathbb{R}$. Thus we have
$\widetilde{f}\in\mathcal{F}_{\operatorname{ReLU}}(L_{T}+4,N_{T},B_{T},F,4C^{\prime}_{S}\log^{r}T+20)$
for sufficiently large $T$ by Lemma II.3 in Elbrächter et al. (2021). ∎

##  Appendix C Technical tools

Here we collect technical tools we used in the proofs. Let $\mathcal{A}$ and
$\mathcal{B}$ be two $\sigma$-fields of a probability space
$(\Omega,\mathcal{T},\operatorname{P})$. The $\beta$-mixing coefficient
between $\mathcal{A}$ and $\mathcal{B}$ is defined by

$\beta(\mathcal{A},\mathcal{B})={1\over 2}\sup\left\\{\sum_{i\in I}\sum_{j\in
J}\left|\operatorname{P}(A_{i}\cap
B_{j})-\operatorname{P}(A_{i})\operatorname{P}(B_{j})\right|\right\\},$

where the maximum is taken over all finite partitions $\\{A_{i}\\}_{i\in
I}\subset\mathcal{A}$ and $\\{B_{j}\\}_{j\in J}\subset\mathcal{B}$ of
$\Omega$.

######  Lemma C.1 (Lemma 5.1 in Rio (2013)).

Let $\mathcal{A}$ be a $\sigma$-field in a probability space
$(\Omega,\mathcal{T},\operatorname{P})$ and $X$ be a random variable with
values in some Poilsh space. Let $U$ be a random variable with uniform
distribution over $[0,1]$, independent of the $\sigma$-field generated by $X$
and $\mathcal{A}$. Then there exists a random variable $X^{\ast}$, with the
same law as $X$, independent of $X$, such that $\operatorname{P}(X\neq
X^{\ast})=\beta(\mathcal{A},\sigma(X))$ where $\sigma(X)$ denote the
$\sigma$-field generated by $X$. Furthermore $X^{\ast}$ is measurable with
respect to the $\sigma$-field generated by $\mathcal{A}$ and $(X,U)$.

######  Lemma C.2 (Lemma 1.4 in de la Peña et al. (2004)).

Let $\\{d_{i}\\}$ be a sequence of variables adapted to an increasing sequence
of $\sigma$-fields $\\{\mathcal{F}_{i}\\}$. Assume that the $d_{i}$’s are
conditionally symmetric (i.e.
$\mathcal{L}(d_{i}|\mathcal{F}_{i-1})=\mathcal{L}(-d_{i}|\mathcal{F}_{i-1})$,
where $\mathcal{L}(d_{i}|\mathcal{F}_{i-1})$ is the conditional law of $d_{i}$
given $\mathcal{F}_{i-1}$). Then
$\exp\left(\lambda\sum_{i=1}^{n}d_{i}-\lambda^{2}\sum_{i=1}^{n}d_{i}^{2}/2\right)$,
$n\geq 1$, is a supermartingale with mean $\leq 1$, for all
$\lambda\in\mathbb{R}$.

######  Lemma C.3.

Let $\\{d_{i}\\}$ be a martingale difference sequence with respect to a
filtration $\\{\mathcal{F}_{i}\\}$. Assume $\mathrm{E}[d_{i}^{2}]<\infty$ for
all $i$. Then

$\operatorname{E}\left[\exp\left(\lambda\sum_{i=1}^{n}d_{i}-\frac{\lambda^{2}}{2}\left(\sum_{i=1}^{n}d_{i}^{2}+\sum_{i=1}^{n}\operatorname{E}[d_{i}^{2}\mid\mathcal{F}_{i-1}]\right)\right)\right]\leq
1$

for all $n\geq 1$ and $\lambda\in\mathbb{R}$.

###### Proof.

Define a process $M=\\{M_{t}\\}_{t\in[0,\infty)}$ as
$M_{t}=\sum_{i=1}^{\lfloor t\rfloor}\lambda d_{i}$ for $t\geq 0$. It is
straightforward to check that $M$ is an $\\{\mathcal{F}_{\lfloor
t\rfloor}\\}$-martingale and its continuous martingale part is identically
equal to 0. Moreover, the compensator of the process $\\{\sum_{i=1}^{\lfloor
t\rfloor}\\{(-\lambda d_{i})\vee 0\\}^{2}\\}_{t\geq 0}$ is
$\\{\sum_{i=1}^{\lfloor t\rfloor}\mathrm{E}[\\{(-\lambda d_{i})\vee
0\\}^{2}\mid\mathcal{F}_{i-1}]\\}_{t\geq 0}$ by Eq.(3.40) of (Jacod and
Shiryaev, 2003, Ch. I). Therefore, by Proposition 4.2.1 in Barlow et al.
(1986), the process

$\left\\{\exp\left(M_{t}-\frac{1}{2}\left(\sum_{i=1}^{\lfloor
t\rfloor}\\{(\lambda d_{i})\vee 0\\}^{2}+\sum_{i=1}^{\lfloor
t\rfloor}\operatorname{E}[\\{(-\lambda d_{i})\vee
0\\}^{2}\mid\mathcal{F}_{i-1}]\right)\right)\right\\}_{t\in[0,\infty)}$

is an $\\{\mathcal{F}_{\lfloor t\rfloor}\\}$-supermartingale. Hence

$\displaystyle
1\geq\operatorname{E}\left[\exp\left(\lambda\sum_{i=1}^{n}d_{i}-\frac{1}{2}\left(\sum_{i=1}^{n}\\{(\lambda
d_{i})\vee 0\\}^{2}+\sum_{i=1}^{n}\operatorname{E}[\\{(-\lambda d_{i})\vee
0\\}^{2}\mid\mathcal{F}_{i-1}]\right)\right)\right].$

Since $\\{(\lambda d_{i})\vee 0\\}^{2}\leq\lambda^{2}d_{i}^{2}$ and
$\\{(-\lambda d_{i})\vee 0\\}^{2}\leq\lambda^{2}d_{i}^{2}$, the desired result
follows from the monotonicity of the exponential function. ∎

######  Lemma C.4 (Theorem 1.2 in de la Peña et al. (2004)).

Let $B\geq 0$ and $A$ be two random variables satisfying
$\mathrm{E}\left[\exp\left(\lambda A-{\lambda^{2}\over
2}B^{2}\right)\right]\leq 1$ for all $\lambda\in\mathbb{R}$. Then for all
$y>0$,

$\displaystyle\mathrm{E}\left[{y\over\sqrt{B^{2}+y^{2}}}\exp\left({A^{2}\over
2(B^{2}+y^{2})}\right)\right]$ $\displaystyle\leq 1.$

######  Lemma C.5 (Proposition 8 in Ohn and Kim (2022)).

Let $L\in\mathbb{N}$, $N\in\mathbb{N}$, $B\geq 1$, $F>0$, and $S>0$. Then for
any $\delta\in(0,1)$,

$\displaystyle\log
N\left(\delta,\mathcal{F}_{\sigma}(L,N,B,F,S),\|\cdot\|_{\infty}\right)$
$\displaystyle\leq 2S(L+1)\log\left((L+1)(N+1)B\delta^{-1}\right).$

######  Lemma C.6 (Proposition 10 in Ohn and Kim (2022)).

Let $L\in\mathbb{N}$, $N\in\mathbb{N}$, $B\geq 1$, $F>0$, and $\tau>0$. Let

$\displaystyle\check{\mathcal{F}}_{\sigma,\tau}(L,N,B,F,S)$
$\displaystyle:=\left\\{f\in\mathcal{F}_{\sigma}(L,N,B,F):\|\theta(f)\|_{\text{clip},\tau}\leq
S\right\\}.$

Then for any $\delta\in(\tau(L+1)((N+1)B)^{L+1},1)$,

$\displaystyle\log
N\left(\delta,\check{\mathcal{F}}_{\sigma}(L,N,B,F,S),\|\cdot\|_{\infty}\right)$
$\displaystyle\leq
2S(L+1)\log\left({(L+1)(N+1)B\over\delta-\tau(L+1)((N+1)B)^{L+1}}\right).$

## References

  * Barlow et al. (1986) Barlow, M. T., S. D. Jacka, and M. Yor (1986).  Inequalities for a pair of processes stopped at a random time.  Proceedings of the London Mathematical Society 52(3), 142–172.
  * Bauer and Kohler (2019) Bauer, B. and M. Kohler (2019).  On deep learning as a remedy for the curse of dimensionality in nonparametric regression.  Annals of Statistics 47(4), 2261–2285.
  * Bhattacharya and Lee (1995) Bhattacharya, R. and C. Lee (1995).  On geometric ergodicity of nonlinear autoregressive models.  Statistics & Probability Letters 22(4), 311–315.
  * Chen and Chen (2000) Chen, M. and G. Chen (2000).  Geometric ergodicity of nonlinear autoregressive models with changing conditional variances.  Canadian Journal of Statistics 28(3), 605–614.
  * Chen and Tsay (1993) Chen, R. and R. S. Tsay (1993).  Functional-coefficient autoregressive models.  Journal of the American Statistical Association 88(421), 298–308.
  * Cline and Pu (2004) Cline, D. B. and H.-M. H. Pu (2004).  Stability and the Lyapounov exponent of threshold AR-ARCH models.  Annals of Applied Probability 14(4), 1920–1949.
  * Davydov (1973) Davydov, Y. A. (1973).  Mixing conditions for Markov chains.  Theory of Probability and Its Applications 18(2), 321–338.
  * de la Peña et al. (2004) de la Peña, V. H., M. J. Klass, and T. L. Lai (2004).  Self-normalized processes: exponential inequalities, moment bounds and iterated logarithm laws.  Annals of Probability, 1902–1933.
  * Elbrächter et al. (2021) Elbrächter, D., D. Perekrestenko, P. Grohs, and H. Bölcskei (2021).  Deep neural network approximation theory.  IEEE Transactions on Information Theory 67(5), 2581–2623.
  * Exterkate (2013) Exterkate, P. (2013).  Model selection in kernel ridge regression.  Computational Statistics and Data Analysis 68, 1–16.
  * Fan and Yao (2008) Fan, J. and Q. Yao (2008).  Nonlinear Time Series: Nonparametric and Parametric Methods. Springer.
  * Granger and Teräsvirta (1993) Granger, C. W. and T. Teräsvirta (1993).  Modelling Nonlinear Economic Relationships. Oxford University Press.
  * Györfi et al. (2002) Györfi, L., M. Kohler, A. Krzyzak, and H. Walk (2002).  A Distribution-Free Theory of Nonparametric Regression.  Springer.
  * Haggan and Ozaki (1981) Haggan, V. and T. Ozaki (1981).  Modelling nonlinear random vibrations using an amplitude-dependent autoregressive time series model.  Biometrika 68(1), 189–196.
  * Hairer (2006) Hairer, M. (2006).  Ergodic properties of Markov processes.  Lecture notes.  http://www.hairer.org/notes/Markov.pdf.
  * Hairer and Mattingly (2011) Hairer, M. and J. C. Mattingly (2011).  Yet another look at Harris’ ergodic theorem for Markov chains.  In Seminar on Stochastic Analysis, Random Fields and Applications VI, pp. 109–117. Springer.
  * Hansen (2008) Hansen, B. E. (2008).  Uniform convergence rates for kernel estimation with dependent data.  Econometric Theory 24(3), 726–748.
  * Hastie et al. (2009) Hastie, T., R. Tibshirani, and J. Friedman (2009).  The Elements of Statistical Learning (second ed.).  Springer.
  * Hayakawa and Suzuki (2020) Hayakawa, S. and T. Suzuki (2020).  On the minimax optimality and superiority of deep neural network learning over sparse parameter spaces.  Neural Networks 123, 343–361.
  * Hoffmann (1999) Hoffmann, M. (1999).  On nonparametric estimation in nonlinear AR(1)-models.  Statistics & Probability Letters 44(1), 29–45.
  * Imaizumi and Fukumizu (2019) Imaizumi, M. and K. Fukumizu (2019).  Deep neural networks learn non-smooth functions effectively.  In Proceedings of Machine Learning Research, Volume 89, pp. 869–878. PMLR.
  * Jacod and Shiryaev (2003) Jacod, J. and A. Shiryaev (2003).  Limit Theorems for Stochastic Processes (second ed.).  Springer.
  * Kingma and Ba (2015) Kingma, D. P. and J. L. Ba (2015).  Adam: A method for stochastic optimization.  In Y. Bengio and Y. LeCun (Eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
  * Kohler and Krzyzak (2020) Kohler, M. and A. Krzyzak (2020).  On the rate of convergence of a deep recurrent neural network estimate in a regression problem with dependent data.  arXiv:2011.00328.
  * Kohler and Langer (2021) Kohler, M. and S. Langer (2021).  On the rate of convergence of fully connected deep neural network regression estimates.  Annals of Statistics 49(4), 2231–2249.
  * Liu and Wu (2010) Liu, W. and W. B. Wu (2010).  Simultaneous nonparametric inference of time series.  Annals of Statistics 38(4), 2388–2421.
  * Lu and Jiang (2001) Lu, Z. and Z. Jiang (2001).  $L_{1}$ geometric ergodicity of a multivariate nonlinear AR model with an ARCH term.  Statistics & Probability Letters 51(2), 121–130.
  * Masry (1996a) Masry, E. (1996a).  Multivariate local polynomial regression for time series: uniform strong consistency and rates.  Journal of Time Series Analysis 17(6), 571–599.
  * Masry (1996b) Masry, E. (1996b).  Multivariate regression estimation local polynomial fitting for time series.  Stochastic Processes and their Applications 65(1), 81–101.
  * Nakada and Imaizumi (2020) Nakada, R. and M. Imaizumi (2020).  Adaptive approximation and generalization of deep neural network with intrinsic dimensionality.  Journal of Machine Learning Research 21, 174–1.
  * Oga and Koike (2021) Oga, A. and Y. Koike (2021).  Drift estimation for a multi-dimensional diffusion process using deep neural networks.  arXiv:2112.13332.
  * Ogihara (2021) Ogihara, T. (2021).  Misspecified diffusion models with high-frequency observations and an application to neural networks.  Stochastic Processes and their Applications 142, 245–292.
  * Ohn and Kim (2022) Ohn, I. and Y. Kim (2022).  Nonconvex sparse regularization for deep neural networks and its optimality.  Neural Computation 34(2), 476–517.
  * Probst and Boulesteix (2018) Probst, P. and A.-L. Boulesteix (2018).  To tune or not to tune the number of trees in random forest.  Journal of Machine Learning Research 18, 1–18.
  * Rio (2013) Rio, E. (2013).  Inequalities and limit theorems for weakly dependent sequences.  https://cel.archives-ouvertes.fr/cel-00867106v2.
  * Robinson (1983) Robinson, P. M. (1983).  Nonparametric estimators for time series.  Journal of Time Series Analysis 4(3), 185–207.
  * Rudin (1987) Rudin, W. (1987).  Real and Complex Analysis (third ed.).  McGraw-Hill.
  * Schmidt-Hieber (2019) Schmidt-Hieber, J. (2019).  Deep ReLU network approximation of functions on a manifold.  arXiv:1908.00695.
  * Schmidt-Hieber (2020) Schmidt-Hieber, J. (2020).  Nonparametric regression using deep neural networks with ReLU activation function.  Annals of Statistics 48(4), 1875–1897.
  * Suzuki (2019) Suzuki, T. (2019).  Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality.  International Conference on Learning Representations.  https://openreview.net/forum?id=H1ebTsActm.
  * Suzuki and Nitanda (2021) Suzuki, T. and A. Nitanda (2021).  Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic Besov space.  Advances in Neural Information Processing Systems 34.
  * Teräsvirta (1994) Teräsvirta, T. (1994).  Specification, estimation, and evaluation of smooth transition autoregressive models.  Journal of the American Statistical Association 89(425), 208–218.
  * Tjøstheim (1990) Tjøstheim, D. (1990).  Non-linear time series and Markov chains.  Advances in Applied Probability 22(3), 587–611.
  * Tran (1993) Tran, L. T. (1993).  Nonparametric function estimation for time series by local average estimators.  Annals of Statistics 21(2), 1040–1057.
  * Truong (1994) Truong, Y. K. (1994).  Nonparametric time series regression.  Annals of the Institute of Statistical Mathematics 46(2), 279–293.
  * Tsuji and Suzuki (2021) Tsuji, K. and T. Suzuki (2021).  Estimation error analysis of deep learning on the regression problem on the variable exponent Besov space.  Electronic Journal of Statistics 15(1), 1869–1908.
  * Tsybakov (2009) Tsybakov, A. B. (2009).  Introduction to Nonparametric Estimation. Springer.
  * van der Vaart and Wellner (1996) van der Vaart, A. W. and J. A. Wellner (1996).  Weak Convergence and Empirical Processes with Applications to Statistics. Springer.
  * Vogt (2012) Vogt, M. (2012).  Nonparametric regression for locally stationary time series.  The Annals of Statistics 40(5), 2601–2633.
  * Xia and Li (1999) Xia, Y. and W. K. Li (1999).  On single-index coefficient regression models.  Journal of the American Statistical Association 94(448), 1275–1285.
  * Xia et al. (2007) Xia, Y., W.-K. Li, and H. Tong (2007).  Threshold variable selection using nonparametric methods.  Statistica Sinica 17, 265–287.
  * Zhang (2010) Zhang, T. (2010).  Analysis of multi-stage convex relaxation for sparse regularization.  Journal of Machine Learning Research 11(3), 1081–1107.
  * Zhao and Wu (2008) Zhao, Z. and W. B. Wu (2008).  Confidence bands in nonparametric time series regression.  Annals of Statistics 36(4), 1854–1878.


# A Quantitative Analysis of Activities of Daily Living: Insights into
Improving Functional Independence with Assistive Robotics

Laura Petrich1, Jun Jin1, Masood Dehghan1 and Martin Jagersand1 1L. Petrich,
J. Jin, M. Dehghan and M. Jagersand are with Department of Computing Science,
University of Alberta, Canada<EMAIL_ADDRESS>
###### Abstract

Human assistive robotics have the potential to help the elderly and
individuals living with disabilities with their Activities of Daily Living
(ADL). Robotics researchers focus on assistive tasks from the perspective of
various control schemes and motion types. Health research on the other hand
focuses on clinical assessment and rehabilitation, arguably leaving important
differences between the two domains. In particular, little is known
quantitatively on which ADLs are typically carried out in a persons everyday
environment - at home, work, etc. Understanding what activities are frequently
carried out during the day can help guide the development and prioritization
of robotic technology for in-home assistive robotic deployment. This study
targets several lifelogging databases, where we compute (i) ADL task frequency
from long-term low sampling frequency video and Internet of Things (IoT)
sensor data, and (ii) short term arm and hand movement data from 30 fps video
data of domestic tasks. Robotics and health care communities have differing
terms and taxonomies for representing tasks and motions. In this work, we
derive and discuss a robotics-relevant taxonomy from quantitative ADL task and
motion data in attempt to ameliorate taxonomic differences between the two
communities. Our quantitative results provide direction for the development of
better assistive robots to support the true demands of the healthcare
community.

##  I Introduction

Activities of Daily Living (ADL) can be a challenge for individuals living
with upper-body disabilities and assistive robotic arms have the potential to
help increase functional independence [1]. Assistive robot arms, such as the
wheelchair-mountable Kinova Jaco [2] (Fig. 1) and Manus/iArm [3], have been
commercially available for over a decade. Such devices can increase
independence, decrease the caregiver load, and reduce healthcare costs [4].
Robot arms have the potential to be as important to individuals living with
upper body disabilities as power wheelchairs have become to those with lower
body disabilities. However, outside of research purposes, only a few hundred
assistive arms, primarily in Europe and North America, are practically
deployed and in use. The gap between assistive robotic research and healthcare
needs impedes the wide adoption of assistive robot products. Healthcare
professionals, assistive technology users, and researchers have differing
biases towards what tasks are of high priority to focus efforts on. For
assistive robotics research, knowing which ADLs are most important to support,
as well as the necessary performance parameters for these tasks will be
crucial to increase usability and deployment. In order to build an assistive
robotic task taxonomy that focuses on functional independence, it is
imperative to understand how the health care community defines independence;
to this end we briefly review the World Health Organization Disability
Assessment Schedule (WHODAS2.0) [5]. [6]. This classification was primarily
developed to determine an individuals level of disability and design an
appropriate rehabilitation plan, not to guide assistive robotics research. To
the best of the authors knowledge, in healthcare literature there does not
appear to be quantitative studies or statistics breaking down individual ADL
tasks and motions by able-bodied or individuals living with disabilities.
Health care and robotic domains use different taxonomies to classify and
everyday activity tasks and motions [7, 8, 9, 10, 11, 12]. By merging these
taxonomies and quantifying health care needs with robotic capabilities we seek
to bridge the two, often separate, communities. This would provide the
robotics community with guidance as to which tasks could make a large impact
to patient populations if implemented on assistive robotic systems.

Figure 1: Jaco assistive arm [2]. Wheelchair-mounted assistive robot arms can
help those living with disabilities carry out their Activities of Daily Living
(ADLs), such as picking up objects, eating, drinking, opening doors, operating
appliances, etc. Figure 2: Measures of Functional Ability that are defined as
essential in healthcare communities: Activities of Daily Living (ADLs) [7] and
Instrumental Activities of Daily Living (IADLs) [8]. ADLs are basic self-care
tasks, whereas IADLs are more complex skills. Together they represent what a
person needs to be able to manage on their own in order to live independently.

In the field of Computer Science, recent interest in video object and activity
recognition [13, 14] along with life-logging capture has resulted in numerous
public data-sets [15]. In this work we analyzed over 30 such data-sets in
order to extract everyday tasks of high importance and relevant motion data
[16].

This paper aims to mitigate the gap dividing the health care and robotics
communities. Contributions include:

  1. 1.

We build a task taxonomy consolidating the taxonomic differences between the
robotics and healthcare communities for the purpose of further analyzing ADL
tasks and the motions they are composed of.

  2. 2.

We analyze long term video-recordings from publicly available life-logging
data. From the video data we extract ADL task frequencies, that quantify how
often a human performs particular ADLs.

  3. 3.

From higher frame-rate video recordings of human kitchen activities, we
analyze human arm and hand motion data to quantify the speed and variability
of human movement.

  4. 4.

We discuss how the task frequency and human motion characterization can
prioritize what robotics techniques will have high impact in assistance
robotics for elderly and disabled.

##  II Societal and Economic Impacts

The use of robotics to help increase functional independence in individuals
living with upper limb disabilities has been studied since the 1960’s. We
distinguish here between a physically independent robot arm, typically mounted
on the users wheelchair, and a smart prosthesis, attached to an amputated
limb, with the former being our group of interest. The United States Veterans
Affairs estimate that approximately 150,000 Americans could benefit from
currently commercially available wheelchair-mounted robot arms [17]. With
improved functionality, reliability, and ease of use deployment to larger
populations could be possible.

What is the magnitude of need and potential for robotic assistance in the
world? Many countries in the west and Asia have an aging populations and
disabilities also affect younger populations, e.g. from accidents, disease, or
inheritance. Definitions and quality of statistics on disability differs
across nations and are difficult to integrate globally. Canada has a multi-
ethnic population and characteristics similar to other industrialized nations.
The proportion of seniors (age 65+) in the population is steadily increasing,
with seniors comprising a projected 23.1% of the population by 2031 [18]. In
2014, seniors constituted only 14% of the population, but consumed 46% of
provincial public health care dollars [19]. A growing number of elderly and
disabled, supported by a dwindling young population is putting pressure both
on government budgets and available health care personnel. Today, individuals
with lower-body impairments and the elderly are able to independently move
around using power wheelchairs. In the near future wheelchair-mounted robot
arms could help increase independence and reduce care needs for those living
with reduced upper-limb function.

Statistics Canada found that from 2001 - 2006 there was a 20.5% increase in
those identifying as having a disability, corresponding to over 2.4 million
people in Canada [20]. One in twenty Canadians living with disabilities
regularly receive assistance with at least one ADL on a daily basis, although
not all of which will require the use of wheelchair-mounted arms. This
suggests that there is a significant need and potential market for robotic
solutions in Canada and similar countries across the world. Some individuals
may prefer automation integration with their smart homes, and some may require
both cognitive and physical assistance. While artificial intelligence might
provide some basic cognitive support, such as planning of the days tasks and
reminders, it cannot eliminate the need for human contact and support.
However, robotic assistance can free up humans from mundane chores, allowing
more time for caregivers to focus on high quality help and personal
interaction. A four year study of assistive arm users in care homes found that
a robot reduced the nursing assistance need by 40% from 3.7h/day to 2.8h/day
[21]. While cost savings from reduced nursing are already significant (about
$20,000/year in an industrialised economy), further savings and increased
independence came from half of the robot users being able to move out of
assistive living with one quarter obtaining jobs.

An advantage of wheelchair-mounted arms is that they are with the person at
all times. Nursing care is typically only for morning and evening routines for
those who live independently. Imagine yourself dropping something important
and having to wait all day before someone is able to help you retrieve it.

##  III Activities of Daily Living, Self-Care, and Functional Independence

Figure 3: The major life domains of functioning and disability as set out in
the World Health Organization Disability Schedule 2.0 (WHODAS2.0); a
standardized cross-cultural measurement of health status based on the
International Classification of Functioning, Disability and Health. WHODAS2.0
can be used to measure the impact of health conditions, monitor intervention
effectiveness and estimate the burden of physical and mental disorders across
all major life domains. Physical motion activities relevant to this paper are
highlighted in bold

The International Classification of Functioning, Disability and Health (ICF)
provides a framework for determining the overall health of individuals and
populations [6]. Disability information is an important indicator of a
population’s health status, as it shows the impact that functional limitations
have on independence. This concept is known as functional disability, or the
limitations one may experience in performing independent living tasks [22]. A
quantification of functional disability includes measures of both Activities
of Daily Living (ADLs) [7] and Instrumental Activities of Daily Living (IADLs)
[8] (Fig. 2); in this work we will refer to these collectively as ADLs. The
World Health Organization further developed the World Health Organization
Disability Assessment Schedule (WHODAS2.0) from the ICF as a standardized,
cross-cultural measure of functioning and disability across all life domains
[23]. Figure 3 highlights these major life domains with associated tasks; the
tasks most relevant to robotics research are emphasized in italics.

A common approach that drives research is to ask patients and caregivers for
their preferences when it comes to robotic assistance [24, 25]. Notably,
preferences vary and user opinions shift over time. In particular, a survey of
67 users surveyed both before and after they received and used an assistive
robotic arm found that caregivers tend to favor essential tasks, such as
taking medication. Pre-automation patients favor picking up dropped objects
and leisure-related tasks, with a shift more towards work-related tasks post-
automation [17]. Combining user preferences with quantitative ADL data will be
important for robotics researchers to consider when deciding what tasks should
be focused on.

##  IV A Task Taxonomy for Arm Manipulation

Robotic capabilities can be built bottom-up by designing control methods for
individual motions (i.e. motor primitives) which can then be combined to solve
specific tasks [26]. The same motions can potentially be used to solve
different ADLs that fall within healthcare taxonomies. Dexterous in-hand
manipulation requires different contact configurations and manipulation
taxonomies have been developed to compensate for these various configurations
[27]. Robot arm manipulation is generally thought of as a 6-DOF Euclidean
(end-effector) transform, thus requiring no taxonomy. Contrarily, ADL tasks
naturally contain a variety of movements with different DOFs, as well as
contact and precision motions. This suggests that an ADL-based taxonomy could
help guide the development of control subroutines tailored to those specific
requirements and that the composition of such subroutines will be capable of
solving a broad variety of tasks.

Figure 4: High-Level Taxonomy of Assistive Robot Tasks and Motions, and how
they intersect with example WHODAS tasks from Figure 3.

Figure 4 introduces a high-level taxonomy of robotic tasks. There are three
general categories relevant to assistive robotic applications: non-physical
cognitive, locomotion-based mobility, and arm-hand manipulation tasks. In this
work we will focus on arm and hand manipulations. In applied robotics, the
robot gripper is typically used for grasping while the arm is responsible for
gross pose alignment and contact point decisions. There is much work to be
done before robotic systems will be able to utilize fine dexterous finger
manipulation motions humans use for ADLs. Commonly the robot gripper just
grasps and the robot arm has to perform both coarse and fine manipulations
[12]. Coarse reaching motions are mostly a 3-DOF translation and requires
moderate accuracy. Fine motions can be further subdivided into contact and
non-contact motions depending on the desired outcome. Non-contact 6-DOF fine
motions can be used to bring an object into alignment with the target location
before putting the object down or inserting it. Although most applied robotics
is performed using position-based control, some studies take contact forces
into account, either through impedance control or sensing and modeling of the
surface for hybrid position-force control [28]. Surface contact data allows
for human-like control strategies to overcome sensing and actuation
inaccuracies by utilizing practices such as feeling a table and sliding across
it before picking up a small object.

##  V Wheelchair-mounted assistive arms

A lightweight robotic arm can be attached to a wheelchair to assist with ADLs
[17]. With such a device, users with limited upper limb functionality are able
to independently carry out a larger subset of their daily tasks. While there
are about 2 million robot arms deployed in industry, only two assistive robot
manufacturers have over 100 assistive arms deployed with disabled humans,
namely, Exact Dynamics (Manus and iARM) [3] and Kinova (JACO and MICO) [29].
These arms are lightweight with integrated controllers and cost around USD
20,000-35,000 with a gripper. For example, the Kinova JACO robotic arm weighs
5.7kg (12.5lbs) and comes with a wheelchair attachment kit. It is capable of
grasping and moving objects up to 1.5kg, Fig. 1. The Manus/iARM has similar
specifications.

In published assistive robotics research a variety of commercial robot arms
are used and several new prototype arms have been designed, however neither
new robots nor new methods for motion control or Human Robot Interaction (HRI)
have reached noticeable deployment [30]. The few hundred deployed JACO and
Manus arms still use basic joystick position-based teleoperation, where a 2 or
3 Degree of Freedom (DOF) joystick is mapped to a subset of the Cartesian arm
translation and wrist rotation controls [2, 31]. To complete 6-DOF tasks the
user needs to switch between various Cartesian planes, known as mode
switching, which can be tedious and cognitively taxing.

Novel user interfaces have been implemented in research settings and rely on a
variety of input signals for shared autonomy, such as gestures, eye gaze,
electromyography (EMG), electroencephalography (EEG), and
electrocorticographic (ECoG). Gesture-based systems allow the user to specify
an object to manipulate by either pointing [32] or clicking on it through a
touch screen interface [33, 34] and then the robotic arm would autonomously
move towards the target object [35]. Eye gaze can be used in place of gestures
to capture an individuals intended actions and drive robot control [36].
Neural interface systems (i.e. ECoG and EEG) work by mapping neuronal activity
to multidimensional control signals that are used to drive robot arm movements
[37]. Hochberg et al. highlight the potential of ECoG-based control methods,
although it requires an invasive surgical procedure in order to implant the
microelectrode array. EEG- and EMG-based methods provide an intuitive, non-
invasive alternative for closed-loop robot control using brain and muscle
activity [38, 39]. Recently, Struijk et al. developed a wireless intraoral
tongue interface device that enables individuals with tetraplegia to use their
tongue to control a 7-DOF robotic arm and 3 finger gripper [40].

##  VI ADL Evaluation from Lifelogging Data

Lifelogging data is a valuable source of quantitative ADL and human motion
information. Lifelogging involves long-term recording of all activities
performed by an individual throughout the course of the day, usually through a
video camera, and occasionally using other types of sensors [15]. While
lifelogging research has been published for over two decades [41], hardware
and method innovation has made the field grow greatly within the past few
years [16]. Small, wearable cameras, such as the Microsoft Lifecam [42], with
a longer recording duration has made it more practical compared to the analog
video cameras and recorders used in initial research. New methods for
recognizing objects and actions has driven Computer Vision (CV) research
interests to explore lifelogging data, which has been found to be a source of
more realistic “in-the-wild” data than typical CV benchmarks [43, 44].

In this work we evaluated over 30 lifelogging datasets111For a detailed table
of specific datasets investigate please visit http://webdocs.cs.ualberta.ca/
vis/ADL/, most of which targeted the performance of a particular algorithm
(e.g. video object recognition in home environments) and therefore did not
encompass the full day. These datasets typically did not have a statistically
sound sampling over all objects and tasks in order to meet our analysis
inclusion criteria for this work. We found that long term video recordings of
several days or more were done at 1-2 frames per minute (fpm), making these
data useful to analyze gross ADL task frequency and duration, but not suitable
for studying detailed timing of individual arm and hand motions. An additional
downfall of the low fpm video datasets is that they fail to capture daily
tasks which are repeated with high frequency but are performed quickly, such
as opening doors or turning on lights. Another category of datasets had
regular video rate recordings of specific tasks, at 30 frames per second
(fps), making the detailed timings of individual arm and hand motions
possible. We were able to choose three sources of data for analysis: two from
long duration recordings in order to extract ADL task frequency and duration
[15, 45], and one from short-term recordings of individual tasks [46].

Figure 5: In the NCTIR Lifelog Dataset [47] 3 people wore lifelogging cameras
for a total of 79 days, collectively. These provide images of the individuals
arms and egocentric environment at a rate of 2 fpm. Due to the low frame rate,
fine arm and hand motions are not available, but actions are instead inferred
from context using visual concepts automatically computed from the images.

###  VI-A ADL Task Frequency Analysis

To compute quantitative data on ADL task frequency and duration we analyzed
both egocentric lifelogging videos (referred to as ‘NTCIR’ [15, 47]), and
exocentric data from Internet-of-Things type sensing built into home objects
(referred to as ‘MIT’) [45]. Example lifelogging images from the NTCIR dataset
are shown in Fig. 5. The use of complementary sensing turned out to be
important for capturing a broader set of tasks. Similar to other CV research,
we were able to infer actions from automatically computed visual concepts
[46]. Our supplementary web page (footnote 1) contains the visual context to
actions inference bindings, so readers can replicate results or add other
rules and actions to classify. We hand-labeled a small portion of the data to
verify the accuracy of the automatic computations. This enabled us to label
in-home data sequences spanning multiple days according to what ADLs were
carried out at particular times and compute their statistics. Figure 6
illustrates the frequency of the most common ADL tasks found in these
datasets.

Figure 6: Human ADL task frequencies from MIT IoT sensors (yellow bars), and
NTCIR lifelogging video (blue bars). The largest bar measure is the most
accurate, as explained in the text. The 2 frame/minute video analysed from
NCTIR misses fast cabinet door and drawer openings to retrieve objects, so
under counts doors. MIT under counts electronics, as mobile devices were not
sensored. Door and drawer opening and robot feeding are high priority tasks
robot researchers already publish on, while hand washing is high priority task
where robot assistance has not been studied.

We have grouped tasks together that correspond with robot skills rather than
specific healthcare ADL/ICF codes. Some events are detected more reliably by
the embedded sensors used in MIT, others only in the lifelogging videos. For
examples sensors detect quick events more reliably that the lifelogging video
data misses. In contrast, outdoor activities are only captured in the video
data. By combining results from both datasets, we were able to obtain a better
quantitative measure of task significance.

Opening and closing doors is the most frequent task at 94 times per day; this
category includes doors between rooms, cabinet doors and drawers. Our
rationale for including cabinet doors and drawers is that the robot would
approach each situation in the same fashion as a standard door. We believe the
MIT data was more accurate since the ‘door opening’ data was obtained from
built in door sensors; the low video frequency (2 fpm) of the NTCIR data
presented low accuracy with the automatic visual concepts extraction by
missing quick openings, particularly of cabinet doors and drawers to retrieve
objects. Following door opening, electronics is the second most frequent task
performed during the day; the electronics category refers to the use of
electronic handheld devices and was dominated by smart phone use. These
devices were mostly not covered by the MIT sensors, but were detected in the
NTCIR video data. Drinking and eating were essential tasks in both studies,
with a frequency of 8.8/day from NTCIR and 4.4/day from MIT. MIT-data captured
hand washing by the number of faucet openings/closing (ie. turning the sink on
and off resulted in two tasks), which overestimated hand washing frequency. We
removed this outlier and relied on the NTCIR results of 4.7/day. These results
capture the actions of able adults, and hence can guide robotics researchers
both what to implement, and how - a task that is frequent and executed quickly
by a human such as door openings need to be easy and fast for a disabled to do
with their robot. This depends on the physical velocity of the robot, as well
as the time and cognitive load it takes the user to handle the human-robot
interface. Door openings are covered in the literature e.g. , and robot
feeding has been studied for over 30 years, with some prominent recent results
[24, 48]. By contrast, hand washing would also be high-priority. Hand washing
has been studied in assistive Computer Vision [49], to prompt Alzheimer
patients though the steps, but we know of no robotics researchers to have
attempted this highly important ADL. Yet, we know anecdotally that disabled
users of robot arms use the robot to support their own arm (please see the
accompanying video). Hence, it should be possible to study motion programming
where a robot arm brings the human arm and hand under a water tap (the water
tap can be automatically activated as is already common).

###  VI-B Arm and Hand Motion Analysis.

From high frame-rate video datasets we were able to extract the number and
timings of individual arm and hand motions required to perform a particular
ADL and, for a few tasks, similar timings for robot execution. The Georgia
Tech Egocentric Activity Datasets (GTEA Gaze+)
222http://www.cbi.gatech.edu/fpv/ contain full frame rate (30 fps) video
recordings of humans performing domestic tasks [46]. We analyzed the annotated
GTEA Gaze+ dataset, which contained approximately 25GB of annotated kitchen
activity videos to extract individual human motion timings performed during
these tasks (Fig. 7).

Figure 8 illustrates four common motions out of the 33 captured in the GTEA
Gaze+ dataset. Notably, human motions were far faster than typical assistive
robot motions. For example, as seen in the histogram, reach motions that take
just a second for a human, can take anywhere from ten seconds to several
minutes in published HRI solutions [50]. This has implications for how many
tasks a robot system can practically substitute in a day without taking up an
excessive amount of time. In other motions, such as pouring liquids, the task
itself constrains the human to proceed rather slowly. The door task covers
both lightweight cabinet doors and drawers, along with heavier doors (e.g.
refrigerator); with lighter doors, the human times approached that of an
unconstrained reach, despite the more challenging physical constraint of
hinged or sliding motion, while heavier doors represent the long tail of the
time distribution. Unlike NTCIR, GTEA Gaze+ is not a representative sampling
of all human activities. It is still notable that the number of reaches is
three times the number of door openings (1800 reaches versus 600 door and
drawer openings over 11 hours of video).

In the following table the frequency (occurrences per hour) and mean human
task execution time are presented. The tasks involve kitchen activities - food
preparation, but movement times are likely typical of other human activities.
It is notable how quickly human moves and how many movements we make.
Replicating human motion speed and agility is a gold standard to benchmark
robots against.

Task | freq | time
---|---|---
Reach and pick item | 88 | 1.5s
Reach and place item | 84 | 1.2s
Turn switch on or off | 10 | 2.1s
Wash hands or items | 3 | 6.7s
Flip food in pan | 2 | 4.9s
Transfer food (e.g. to plate) | 6 | 8.6s

Figure 7: The GTEA Gaze+ dataset contains 7 kitchen activities performed by 6
persons. We analyzed the frequency and mean human execution time of each human
activity. Top 4 frequent activities are as shown above.

##  VII Discussion

Door opening/closing, drinking/eating, hand washing and toileting would
arguably be the most essential to support for assistive robot arm and hand
systems, out of all the ADL tasks analyzed in this work. The first three are
relatively feasible to accomplish given the payload capacity of current
robotic arms.

Activities such as using electronics (primarily smartphones), socializing, and
reading could be physically aided by robot arms, but since these activities
are not inherently physical, alternative solutions are possible and can be a
simpler and more reliable solution (e.g. hands-free phone use and other
computational automation).

Toileting is a high priority task that involves transferring from a wheelchair
to the toilet. Assistive arms do not support this, but there are specialized
transfer devices - also useful for transfer from beds - that are generally
used in health care, and be employed in peoples homes.

Overall, there is great potential for supporting ADLs for those living with
disabilities as well as the elderly. Over the past few decades there has been
an increasing demand for health care services due to the rising elderly and
disability populations [51]. Assistive robots can help bridge this gap by
alleviating the labour burden for health care specialists and caregivers.
Furthermore, an assistive robot could help one perform ADL they are otherwise
incapable of managing on their own, thus increasing functional independence.

Figure 8: Timing histograms for four common human motions. Human arm and hand
motions are very quick and accurate, just seconds long. By contrast current
robots are slow.

However, challenges remain before these robots will reach mainstream adoption,
including but not limited to: system costs, task completion times, and ease of
use via user interfaces. Currently costing around USD 30,000, an arm is a
significant expense for an individual, who may already have a limited income.
While western health insurance often covers expensive prostheses for amputees,
only in the Netherlands does insurance cover a wheelchair mounted arm.

Speed of robot motion, which affects task completion time, is another
challenge. While a human reach takes just 1-2s (Fig. 8), published assistive
robots take 40-90s, resulting in robot solutions that are magnitudes slower
[52, 32, 50]. In the GTEA Gaze+ kitchen tasks, humans performed 160 reaches
per hour. Substituting robot reaches would turn a moderate 30 minute meal
preparation and eating time into a 2 hour ordeal. Anecdotal comments from
users of assistive robot arms are that everyday morning kitchen and bathroom
activities, which an able person easily performs in less than an hour, takes
them several hours.

Robots may solve tasks differently than humans as robots are often limited to
grasping one item at a time, while humans can handle many. When setting a
table we will for instance pick several utensils at a time from a drawer. In
restaurants, waiters can clear a table for four, and handle all the plates,
utensils, glasses, etc. in their hands and arms. Analysing the publicly
available TUM Kitchen Data Set of activity sequences recorded in a kitchen
environment [53], we found that the robot strategy on average required 1.6
times more movements than a human. Users of assistive robots adopt compromises
to deal with the speed and accuracy of robots. For example, foods and drinks
that can be consumed while held statically in front of the humans face by the
robot, e.g. eating a snack bar, or drinking with a straw, are far quicker to
consume than those requiring numerous robot reach motions, such as eating a
bowl of cereal.

User interfaces need improvements. Currently deployed arms are, as mentioned
before, joystick operated, while most research is on autonomous movement, e.g.
autonomously delivering a piece of food once the system has detected an open
mouth [54, 48]. Sheridan’s conventional scale from tele-operation to autonomy
[55], has been redefined by Goodrich to have seamless human-robot
collaboration as the goal rather than robot autonomy [31].

We, and others, have found that users generally prefer to have continuous in-
the-loop control [32, 56]. Someone may change their mind midway through
autonomous food delivery, and may instead open their mouth to say something -
only to get their mouth stuffed with food. In very recent work a low
dimensional control space is learned from demonstrations. This allows a human
user to have direct control over a 6DOF motion using a low DOF HRI, such as a
joystick [57, 58]. Getting the balance right between human interaction and
semi-autonomous assistive systems will be challenging. Currently, most
research is evaluated with a few participants trying it for about an hour each
in a research lab setting. We expect that new HRI solutions will need to be
deployed longer term in real users homes in order to properly evaluate
usability.

##  VIII Conclusion

In this paper we presented assistive robotics for Activities of Daily Living -
ADL from both from a health care perspective and robotics perspective. We
analyzed human ADL task frequency from public life-logging datasets and
computed motion timings from public Computer Vision data. Overall, reach
motions (to grasp objects) and door openings (including cabinets and drawers)
were the most frequent motions. Drinking, eating and hand washing are other
high priority tasks that can be addressed by current assistive robot arms.
Toileting and dressing, while ranking just below, are generally thought to be
more challenging for robotics, since they require the transfer of body weight.
Detailed data on frequency and duration information for all analyzed tasks and
motions, as well as the analysis methods are available on the companion
website http://webdocs.cs.ualberta.ca/~vis/ADL

## References

  * [1] C. Smarr, C. B. Fausset, and W. A. Rogers, “Understanding the Potential for Robot Assistance for Older Adults in the Home Environment,” Atlanta, GA: Georgia Tech, School of Psychology, Tech. Rep., 2010.
  * [2] P. Archambault and P. S. Archambault, “Evaluation of the JACO robotic arm: clinico- economic study for powered wheelchair users with upper-extremity disabilities Evaluation of the JACO robotic arm,” _Proceedings of the 2011 IEEE Int. Conf. on Rehabilitation Robotics (ICORR)_ , 2011.
  * [3] B. J. Driessen, H. Evers, and J. Woerden, “MANUS — a wheelchair-mounted rehabilitation robot,” _Proc Inst Mech Eng [H]_ , vol. 215, pp. 285–290.
  * [4] G. Romer, H. Stuyt, and A. Peters, “Cost-savings and economic benefits due to the assistive robotic manipulator (arm),” _Proceedings of the 2005 IEEE 9th International Conference on Rehabilitation Robotics_ , pp. 201 – 204, 01 2005\.
  * [5] T. B. Üstün, S. Chatterji, N. Kostanjsek, J. Rehm, C. Kennedy, J. Epping-Jordan, S. Saxena, M. von Korff, and C. Pull, “Developing the world health organization disability assessment schedule 2.0,” _Bulletin of the World Health Organization_ , vol. 88, no. 11, pp. 815–823, 2010.
  * [6] World Health Organization (WHO), “International Classification of Funcitoning, Disability and Health,” _World Health Organization_ , no. Version 2.1a, Clinician Form, pp. 1–15, 2003. [Online]. Available: http://www.who.int/classifications/icf/icfchecklist.pdf?ua=1
  * [7] S. Katz, “Assessing Self-maintenance: Activities of Daily Living , Mobility , and Instrumental Activities,” _Journal of the American Geriatrics Society_ , vol. 31, pp. 721–727, 1979.
  * [8] M. P. Lawton and E. M. Brody, “Assessment of Older People: Self-Maintaining and Instrumental Activities of Daily Living,” _The Gerontologist_ , vol. 9, no. 3, pp. 179–186, 1969. [Online]. Available: http://gerontologist.oxfordjournals.org/cgi/doi/10.1093/geront/9.3˙Part˙1.179
  * [9] B. Langdon, H. Wang, and D. Ding, “Analysis of Assistive Robotic Manipulator (ARM) Performance Based on a Task Taxonomy,” _RESNA Annual Conference_ , 2016. [Online]. Available: https://www.resna.org/sites/default/files/conference/2016/emerging˙technology/langdon.html
  * [10] A. Bloomfield, Y. Deng, J. Wampler, P. Rondot, D. Harth, M. McManus, and N. Badler, “A taxonomy and comparison of haptic actions for disassembly tasks,” _Proceedings - IEEE Virtual Reality_ , vol. 2003-Janua, no. March, pp. 225–231, 2003.
  * [11] A. Dollar, “Classifying Human Hand Use and the Activities of Daily Living,” _Springer Tracts in Advanced Robotics_ , vol. 95, 2014.
  * [12] I. M. Bullock and A. M. Dollar, “Classifying human manipulation behavior,” _IEEE International Conference on Rehabilitation Robotics_ , 2011.
  * [13] P. Wang and A. F. Smeaton, “Using visual lifelogs to automatically characterize everyday activities,” _Information Sciences_ , vol. 230, pp. 147–161, 2013. [Online]. Available: http://dx.doi.org/10.1016/j.ins.2012.12.028
  * [14] P. Wang, L. Sun, S. Yang, A. Smeaton, and C. Gurrin, “Characterizing everyday activities from visual lifelogs based on enhancing concept representation,” _Comput. Vis. Image Underst._ , vol. 148, pp. 181–192, 2016.
  * [15] C. Gurrin, H. Joho, F. Hopfgartner, L. Zhou, and R. Albatal, “NTCIR Lifelog: The First Test Collection for Lifelog Research,” _39th International ACM SIGIR conference_ , pp. 705–708, 2016.
  * [16] M. Bolanos, M. Dimiccoli, and P. Radeva, “Toward Storytelling from Visual Lifelogging: An Overview,” _IEEE Transactions on Human-Machine Systems_ , vol. 47, no. 1, pp. 77–90, 2017.
  * [17] C.-S. Chung, H. Wang, and R. A. Cooper, “Functional assessment and performance evaluation for assistive robotic manipulators: Literature review,” _The Journal of Spinal Cord Medicine_ , vol. 36, no. 4, pp. 273–289, 2013. [Online]. Available: http://www.tandfonline.com/doi/full/10.1179/2045772313Y.0000000132
  * [18] “The sustainability of health care spending in canada 2017,” https://www.fraserinstitute.org/sites/default/files/sustainability-of-health-care-spending-in-canada-2017.pdf, accessed: 2018-07-30.
  * [19] “Health care in canada, 2011. a focus on seniors and aging,” https://secure.cihi.ca/, accessed: 2018-07-30.
  * [20] “Help with activities of daily living for people with a disability,” https://www150.statcan.gc.ca/n1/pub/11-008-x/2010002/article/11345-eng.htm#n1, accessed: 2018-07-30.
  * [21] G. Romer, H. Stuyt, and A. Peters, “Cost-savings and economic benefits due to the assistive robotic manipulator (arm),” vol. 2005, 01 2005, pp. 201 – 204\.
  * [22] W. Spector and J. Fleishman, “Combining activities of daily living with instrumental activities of daily living to measure functional disability,” _Journal of Gerontology_ , vol. 53, no. I, pp. 46–56, 1998. [Online]. Available: http://psychsocgerontology.oxfordjournals.org/content/53B/1/S46.short
  * [23] S. Federici, M. Bracalenti, F. Meloni, and J. V. Luciano, “World Health Organization disability assessment schedule 2.0: An international systematic review,” _Disability and Rehabilitation_ , vol. 39, no. 23, pp. 2347–2380, 2017.
  * [24] J. M. Beer, C.-A. Smarr, T. L. Chen, A. Prakash, T. L. Mitzner, C. C. Kemp, and W. A. Rogers, “The domesticated robot: design guidelines for assisting older adults to age in place,” _Proceedings of the seventh annual ACM/IEEE international conference on Human-Robot Interaction - HRI ’12_ , pp. 335–342, 2012. [Online]. Available: http://ieeexplore.ieee.org/xpls/abs˙all.jsp?arnumber=6249576
  * [25] C. A. Stanger, C. Anglin, W. S. Harwin, and D. P. Romilly, “Devices for Assisting Manipulation: A Summary of User Task Priorities,” _IEEE Transactions on Rehabilitation Engineering_ , vol. 2, no. 4, pp. 256–265, 1994\.
  * [26] J. Kober and J. Peters, “Learning motor primitives for robotics,” _2009 IEEE International Conference on Robotics and Automation_ , pp. 2112–2118, 2009.
  * [27] I. M. Bullock, R. R. Ma, and A. M. Dollar, “A Hand-Centric Classification of Human and Robot Dexterous Manipulation,” _IEEE Transactions on Haptics_ , vol. 6, no. 2, pp. 129–144, 2013. [Online]. Available: http://ieeexplore.ieee.org/xpl/articleDetails.jsp
  * [28] A. Leite, F. Lizarralde, and L. Hsu, “Hybrid vision-force robot control for tasks on unknown smooth surfaces,” in _Robotics and Automation, 2006. ICRA 2006. Proceedings 2006 IEEE International Conference on_ , May 2006, pp. 2244–2249.
  * [29] A. Campeau-lecours, H. Lamontagne, and S. Latour, “Kinova Modular Robot Arms for Service Robotics Applications,” _International Journal of Robotics Applications and Technologies_ , vol. 5, pp. 49–71, 2017.
  * [30] S. S. Groothuis, S. Stramigioli, and R. Carloni, “Lending a helping hand: Toward novel assistive robotic arms,” _IEEE Robotics and Automation Magazine_ , vol. 20, no. 1, pp. 20–29, 2013.
  * [31] M. A. Goodrich and A. C. Schultz, “Human-Robot Interaction: A Survey,” _Foundations and Trends® in Human-Computer Interaction_ , vol. 1, no. 3, pp. 203–275, 2007. [Online]. Available: http://www.nowpublishers.com/article/Details/HCI-005
  * [32] C. Quintero, R. Tatsambon, M. Gridseth, and M. Jagersand, “Visual pointing gestures for bi-directional human robot interaction in a pick-and-place task,” _Proceedings - IEEE International Workshop on Robot and Human Interactive Communication_ , pp. 349–354, 2015.
  * [33] M. Jagersand and R. Nelson, “Visual space task specification, planning and control,” _Proceedings of International Symposium on Computer Vision - ISCV_ , pp. 521–526, 1995.
  * [34] M. Gridseth, O. Ramirez, C. Quintero, and M. Jagersand, “ViTa: Visual task specification interface for manipulation with uncalibrated visual servoing,” _Proceedings - IEEE International Conference on Robotics and Automation_ , pp. 3434–3440, 2016.
  * [35] K. M. Tsui, A. Behal, D. Kontak, and H. A. Yanco, “I want that: Human-in-the-loop control of a wheelchair-mounted robotic arm,” _Applied Bionics and Biomechanics_ , vol. 8, no. 1, pp. 127–147, 2011.
  * [36] H. Admoni and S. Srinivasa, “Predicting user intent through eye gaze for shared autonomy,” _AAAI Fall Symposium - Technical Report_ , pp. 298–303, 2016.
  * [37] L. R. Hochberg, D. Bacher, B. Jarosiewicz, N. Y. Masse, J. D. Simeral, J. Vogel, S. Haddadin, J. Liu, S. S. Cash, P. Van Der Smagt, and J. P. Donoghue, “Reach and grasp by people with tetraplegia using a neurally controlled robotic arm,” _Nature_ , pp. 372–375, 2013.
  * [38] A. F. Salazar-Gomez, J. Delpreto, S. Gil, F. H. Guenther, and D. Rus, “Correcting robot mistakes in real time using EEG signals,” _Proceedings - IEEE International Conference on Robotics and Automation_ , pp. 6570–6577, 2017.
  * [39] M. V. Liarokapis, P. K. Artemiadis, P. T. Katsiaris, K. J. Kyriakopoulos, and E. S. Manolakos, “Learning human reach-to-grasp strategies: Towards EMG-based control of robotic arm-hand systems,” _Proceedings - IEEE International Conference on Robotics and Automation_ , pp. 2287–2292, 2012.
  * [40] L. N. A. Struijk, L. L. Egsgaard, R. Lontis, M. Gaihede, and B. Bentsen, “Wireless intraoral tongue control of an assistive robotic arm for individuals with tetraplegia,” _Journal of NeuroEngineering and Rehabilitation_ , vol. 14, pp. 1–8, 2017.
  * [41] S. Mann, “Wearable computing: a first step toward personal imaging,” _Computer_ , vol. 30, no. 2, pp. 25–32, 1997.
  * [42] G. Wilson, D. Jones, P. Schofield, and D. Martin, “Experiences of using a wearable camera to record activity, participation and health-related behaviours: Qualitative reflections of using the sensecam,” _Digital Health_ , vol. 2, 11 2016.
  * [43] H. Pirsiavash and D. Ramanan, “Detecting activities of daily living in first-person camera views,” _Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , pp. 2847–2854, 2012.
  * [44] A. Fathi, X. Ren, and J. M. Rehg, “Learning to recognize objects in egocentric activities,” _Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , pp. 3281–3288, 2011.
  * [45] E. M. Tapia, S. S. Intille, and K. Larson, “Activity recognisation in Home Using Simple state changing sensors,” _Pervasive Computing_ , vol. 3001, pp. 158–175, 2004. [Online]. Available: http://link.springer.com/10.1007/b96922
  * [46] A. Fathi, Y. Li, and J. M. Rehg, “Learning to recognize daily actions using gaze,” _Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)_ , vol. 7572 LNCS, no. PART 1, pp. 314–327, 2012.
  * [47] C. Gurrin, H. Joho, F. Hopfgartner, L. Zhou, R. Gupta, R. Albatal, and D.-T. Dang-Nguyen, “Overview of NTCIR-13 Lifelog-2 Task,” _Proceedings of the 13th NTCIR Conference on Evaluation of Information Access Technologies_ , pp. 6–11, 2016.
  * [48] E. Gordon, X. Meng, M. Barnes, T. Bhattacharjee, and S. Srinivasa, “Adaptive robot-assisted feeding: An online learning framework for acquiring previously-unseen food items,” _arXiv: Robotics_ , 2019.
  * [49] J. Hoey, P. Poupart, A. v. Bertoldi, T. Craig, C. Boutilier, and A. Mihailidis, “Automated handwashing assistance for persons with dementia using video and a partially observable markov decision process,” _Comput. Vis. Image Underst._ , vol. 114, no. 5, p. 503–519, May 2010. [Online]. Available: https://doi.org/10.1016/j.cviu.2009.06.008
  * [50] K. Muelling, A. Venkatraman, J. Valois, J. Downey, J. Weiss, S. Javdani, M. Hebert, A. Schwartz, J. Collinger, and J. Bagnell, “Autonomy infused teleoperation with application to brain computer interface controlled manipulation,” _Autonomous Robots_ , vol. 41, pp. 1401–1422, 2017.
  * [51] Department of Economic and Social Affairs: Population Division, _World Population Ageing 2017 - Highlights_ , 2017. [Online]. Available: http://www.un.org/en/development/desa/population/publications
  * [52] D.-j. Kim, R. Hazlett-knudsen, H. Culver-godfrey, G. Rucks, T. Cunningham, D. Portée, J. Bricout, Z. Wang, and A. Behal, “How Autonomy Impacts Performance and Satisfaction: Results From a Study With Spinal Cord Injured Subjects Using an Assistive Robot,” _IEEE Transactions on Systems, Man, and Cybernetics, Part A: Systems and Humans_ , vol. 42, no. 1, pp. 2–14, 2012\.
  * [53] M. Tenorth, J. Bandouch, and M. Beetz, “The TUM kitchen data set of everyday manipulation activities for motion tracking and action recognition,” _Journal of NeuroEngineering and Rehabilitation_ , pp. 1089–1096, 2009.
  * [54] D. Park, Y. Hoshi, H. Mahajan, H. Kim, Z. Erickson, W. Rogers, and C. Kemp, “Active robot-assisted feeding with a general-purpose mobile manipulator: Design, evaluation, and lessons learned,” _Robotics and Autonomous Systems_ , vol. 124, 10 2019.
  * [55] T. B. Sheridan, _Humans and Automation: System Design and Research Issues_ , 1st ed. New York, NY: Wiley-Interscience, 2002.
  * [56] D. Kim, R. Hazlett-Knudsen, H. Culver-Godfrey, G. Rucks, T. Cunningham, D. Portee, J. Bricout, Z. Wang, and A. Behal, “How autonomy impacts performance and satisfaction: Results from a study with spinal cord injured subjects using an assistive robot,” _Systems, Man and Cybernetics, Part A: Systems and Humans, IEEE Transactions on_ , vol. 42, pp. 2 – 14, 02 2012.
  * [57] H. J. Jeon, D. P. Losey, and D. Sadigh, “Shared autonomy with learned latent actions,” _ArXiv_ , vol. abs/2005.03210, 2020.
  * [58] C. Quintero, M. Dehghan, O. Ramirez, M. Ang, and M. Jägersand, “Flexible virtual fixture interface for path specification in tele-manipulation,” _2017 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 5363–5368, 2017.