$\displaystyle\sqrt{\int_{\mathcal{H}}\left(f^{\star}(x)-\hat{f}_{n}(x)\right)^{2}\differential\mathcal{L}(x)}$
$\displaystyle+\sqrt{\int_{\mathcal{H}}\left(\hat{f}_{n}(x)-\hat{f}_{n,N}(x)\right)^{2}\differential\mathcal{L}(x)}.$

Caponnetto and De Vito (2007, Thm. 1) shows that

$\sqrt{\int_{\mathcal{H}}\left(f^{\star}(x)-\hat{f}_{n}(x)\right)^{2}\differential\mathcal{L}(x)}=\mathcal{O}_{\mathbb{P}}\left(n^{-\frac{bc}{2(bc+1)}}\right).$

Note that with the setting of Theorem 3.11, it is straigthforward to check
Hypotheses 1 and 2 in Caponnetto and De Vito (2007). Hence, it just remains to
show that

$E_{n,N}=\sqrt{\int_{\mathcal{H}}\left(\hat{f}_{n}(x)-\hat{f}_{n,N}(x)\right)^{2}\differential\mathcal{L}(x)}=\mathcal{O}_{\mathbb{P}}\left(n^{-\frac{bc}{2(bc+1)}}\right).$

Now, before applying Theorem 3.4, we show that
$c_{n}=\|\hat{f}_{n}\|_{\mathcal{H}_{K}}$ there is bounded in probability. The
developments in Szabó et al. (2016, Sect. 9.1) (using Caponnetto and De Vito
(2007)), although written for a specific Hilbert space $\mathcal{H}$ based on
mean embeddings of distributions, can actually be seen to hold for a general
$\mathcal{H}$ as in the context of Theorem 3.11. These developments yield

$\displaystyle
c_{n}^{2}=\mathcal{O}_{\mathbb{P}}\left(\frac{1}{\lambda^{2}n^{2}}+\frac{\mathcal{N}(\lambda)}{n\lambda}+\frac{\mathcal{B}(\lambda)}{\lambda^{2}n^{2}}+\frac{\mathcal{A}(\lambda)}{\lambda^{2}n}+\mathcal{B}(\lambda)+1\right).$

Above, the quantities $\mathcal{N}(\lambda)$, $\mathcal{B}(\lambda)$ and
$\mathcal{A}(\lambda)$ are defined in Caponnetto and De Vito (2007, Sect. 5.2)
and it is shown in Proposition 3 there that
$\mathcal{N}(\lambda)=\mathcal{O}(\lambda^{-1/b})$,
$\mathcal{B}(\lambda)=\mathcal{O}(\lambda^{c-1})$ and
$\mathcal{A}(\lambda)=\mathcal{O}(\lambda^{c})$. Hence, it is simple to check
that $c_{n}=\mathcal{O}_{\mathbb{P}}(1)$.

Then, Theorem 3.4 yields, with $\mathcal{B}_{n}$ the $\sigma$-algebra
generated by $(\mu_{i},Y_{i})_{i=1}^{n}$, and with a constant $c_{1}$,

$\displaystyle\mathbb{E}[E_{n,N}|\mathcal{B}_{n}]$
$\displaystyle\leq\frac{c_{1}(Y_{\max}+c_{n})}{\lambda
N}+\frac{c_{1}(Y_{\max}+c_{n})}{\lambda\sqrt{n}\sqrt{N}}$
$\displaystyle+\left(1+\frac{\sqrt{N}}{\sqrt{n}}\right)^{-1}\left(\frac{c_{1}(Y_{\max}+c_{n})}{\lambda
n}+\frac{c_{1}(Y_{\max}+c_{n})}{\lambda^{2}n\sqrt{N}}\right).$

With Markov inequality, applied conditionally to $\mathcal{B}_{n}$ this
implies that

$\displaystyle E_{n,N}$
$\displaystyle=\mathcal{O}_{\mathbb{P}}\left(\frac{Y_{\max}+c_{n}}{\lambda
N}+\frac{Y_{\max}+c_{n}}{\lambda\sqrt{n}\sqrt{N}}\right)$
$\displaystyle+\left(1+\frac{\sqrt{N}}{\sqrt{n}}\right)^{-1}\mathcal{O}_{\mathbb{P}}\left(\frac{Y_{\max}+c_{n}}{\lambda
n}+\frac{Y_{\max}+c_{n}}{\lambda^{2}n\sqrt{N}}\right).$

Let us now use that $c_{n}=\mathcal{O}_{\mathbb{P}}(1)$ and that $\lambda
n^{\frac{b}{bc+1}}$ is lower and upper bounded. Furthermore, let $a$ be as in
the theorem statement, so that $N/n^{a}$ is lower-bounded. This yields

$\displaystyle E_{n,N}$
$\displaystyle=\mathcal{O}_{\mathbb{P}}\left(n^{\frac{b}{bc+1}-a}+n^{\frac{b}{bc+1}-\frac{1}{2}-\frac{a}{2}}\right)$
(C.1)
$\displaystyle+\min(1,n^{\frac{1}{2}-\frac{a}{2}})\mathcal{O}_{\mathbb{P}}\left(n^{\frac{b}{bc+1}-1}+n^{\frac{2b}{bc+1}-1-\frac{a}{2}}\right).$

We now study whether the bound in (C.1) can be of order
$\mathcal{O}_{\mathbb{P}}\left(n^{-\frac{bc}{2(bc+1)}}\right)$ with $a\leq 1$.
Necessary and sufficient conditions for this are

$\displaystyle\frac{b}{bc+1}-a\leq-\frac{bc}{2(bc+1)},\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\
\frac{b}{bc+1}-\frac{1}{2}-\frac{a}{2}\leq-\frac{bc}{2(bc+1)},$
$\displaystyle\frac{b}{bc+1}-1\leq-\frac{bc}{2(bc+1)},\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\
\frac{2b}{bc+1}-1-\frac{a}{2}\leq-\frac{bc}{2(bc+1)}.$

Using that we aim for $a\leq 1$, so that the third condition is implied by the
first one, the conditions are equivalent to

$\displaystyle a\geq\frac{b+\frac{bc}{2}}{bc+1},\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\
a\geq\frac{2b-1}{bc+1},\leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ a\geq\frac{4b-bc-2}{bc+1}.$

The three lower bounds on $a$ above are smaller or equal to $1$ if and only if
$b(1-\frac{c}{2})\leq\frac{3}{4}$. Hence, in this case
$a=\max(\frac{b+\frac{bc}{2}}{bc+1},\frac{2b-1}{bc+1},\frac{4b-bc-2}{bc+1})\leq
1$ indeed yields
$E_{n,N}=\mathcal{O}_{\mathbb{P}}\left(n^{-\frac{bc}{2(bc+1)}}\right)$ with
$N/n^{a}$ lower-bounded.

We now consider the case $b(1-\frac{c}{2})>\frac{3}{4}$, and we now study
whether the bound in (C.1) can be of order
$\mathcal{O}_{\mathbb{P}}\left(n^{-\frac{bc}{2(bc+1)}}\right)$ with $a>1$.
Necessary and sufficient conditions for this are

$\displaystyle\frac{b}{bc+1}-a\leq-\frac{bc}{2(bc+1)},\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\
\frac{b}{bc+1}-\frac{1}{2}-\frac{a}{2}\leq-\frac{bc}{2(bc+1)},$
$\displaystyle\frac{1}{2}-\frac{a}{2}+\frac{b}{bc+1}-1\leq-\frac{bc}{2(bc+1)},\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\
\frac{1}{2}-\frac{a}{2}+\frac{2b}{bc+1}-1-\frac{a}{2}\leq-\frac{bc}{2(bc+1)}.$

These conditions are equivalent to

$\displaystyle a\geq\frac{b+\frac{bc}{2}}{bc+1},\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\
a\geq\frac{2b-\frac{1}{2}}{bc+1}.\leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ $

Hence, when $b(1-\frac{c}{2})>\frac{3}{4}$, taking
$a=\max(\frac{b+\frac{bc}{2}}{bc+1},\frac{2b-\frac{1}{2}}{bc+1})$, we indeed
have $E_{n,N}=\mathcal{O}_{\mathbb{P}}\left(n^{-\frac{bc}{2(bc+1)}}\right)$
with $N/n^{a}$ lower-bounded. This concludes the proof.

##  Appendix D Proofs for Section 4.1

###  D.1 Proof of Lemma 4.1

For $p>0$, we let $\mathcal{C}^{p}(\Omega)$ be the space of functions
$f:\Omega\to\mathbb{R}$ that are $\lfloor p\rfloor$ times differentiable, with
$\lfloor.\rfloor$ the integer part and with
$\|f\|_{\mathcal{C}^{p}(\Omega)}<\infty$, where

$\|f\|_{\mathcal{C}^{p}(\Omega)}=\sum_{\beta=0}^{\lfloor
p\rfloor}\sum_{|\alpha|=\beta}\|D^{\alpha}f\|_{\infty}.$

Above $\alpha=(\alpha_{1},\ldots,\alpha_{d})\in\mathbb{N}^{d}$ with
$\sum_{\ell=1}^{d}\alpha_{\ell}=\beta$ and
$D^{\alpha}=\partial^{\beta}/\partial_{x_{1}}^{\alpha_{1}}\cdots\partial_{x_{d}}^{\alpha_{d}}$.
The space $\mathcal{C}^{p}(\Omega)$ is endowed with the norm
$\|\cdot\|_{\mathcal{C}^{p}(\Omega)}$.

A distance between two measures $\mu,\nu\in\mathcal{P}(\Omega)$ can thus be
defined as

$\|\mu-\nu\|_{p}=\sup_{f\in\mathcal{C}^{p}(\Omega),\
\|f\|_{\mathcal{C}^{p}(\Omega)}\leq 1}\int
f(x)(\differential\mu(x)-\differential\nu(x)).$

Fix $p>d$. It can be seen from the proof of González-Sanz, Loubes and Niles-
Weed (2022, Thm. 2.1) and from Carlier and Laborde (2020, Prop. 3.1) that we
have, with a constant $C_{\Omega}$,

$||g^{\mu^{N}}-g^{\mu}-\mathcal{B}\mathcal{A}(\mu^{N}-\mu)||_{\mathcal{L}^{2}(\mathcal{U})}\leq
C_{\Omega}\|\mu^{N}-\mu\|_{p}^{2},$

with $\mathcal{B}:\mathcal{L}^{2}(\mathcal{U})\to\mathcal{L}^{2}(\mathcal{U})$
being a bounded linear operator—with unimportant shape for us—and
$\mathcal{A}$ defined by, for $\eta_{1},\eta_{2}\in\mathcal{P}(\Omega)$,
$u\in\Omega$,

$(\mathcal{A}(\eta_{1}-\eta_{2}))(u)=\int_{\Omega}e^{\|u-v\|^{2}}\differential(\eta_{1}-\eta_{2})(v).$

We call $a_{N}=\mathcal{B}\mathcal{A}(\mu^{N}-\mu)$ and

$b_{N}=g^{\mu^{N}}-g^{\mu}-\mathcal{B}\mathcal{A}(\mu^{N}-\mu).$

As a consequence, $g^{\mu^{N}}-g^{\mu}=a_{N}+b_{N}$.

First, the proof of del Barrio et al. (2023, Eq. 4.13) yields
$\mathbb{E}[\|\mu^{N}-\mu\|_{p}^{2s}]\leq\frac{C_{\Omega}}{N^{s}}$ (up to
increasing the value of $C_{\Omega}$). From this, we obtain
$\mathbb{E}[\|b_{N}\|_{\mathcal{L}^{2}(\mathcal{U})}^{s}]\leq
C_{\Omega}^{s}\mathbb{E}[\|\mu^{N}-\mu\|_{p}^{2s}]\leq\frac{C_{\Omega}^{1+s}}{N^{s}}$.
Then, with a constant $c_{1}$, and letting
$\|\mathcal{B}\|=\sup_{\|h\|_{\mathcal{L}^{2}(\mathcal{U})}\leq
1}\|\mathcal{B}h\|_{\mathcal{L}^{2}(\mathcal{U})}$ (the operator norm), we
have
$\mathbb{E}[\|a_{N}\|_{\mathcal{L}^{2}(\mathcal{U})}^{s}]\leq\|\mathcal{B}\|^{s}\mathbb{E}[\|\mathcal{A}(\mu^{N}-\mu)\|_{\mathcal{L}^{2}(\mathcal{U})}^{s}]\leq\|\mathcal{B}\|^{s}c_{1}\mathbb{E}[\|\mu^{N}-\mu\|_{p}^{s}]\leq\frac{\|\mathcal{B}\|^{s}c_{1}\sqrt{C_{\Omega}}}{N^{s/2}}.$

Finally, the adjoint operator
$\mathcal{B}^{\star}:\mathcal{L}^{2}(\mathcal{U})\to\mathcal{L}^{2}(\mathcal{U})$
satisfies, for any $h\in\mathcal{L}^{2}(\mathcal{U})$,

$\langle
h,\mathcal{B}\mathcal{A}(\mu^{N}-\mu)\rangle_{\mathcal{L}^{2}(\mathcal{U})}=\langle\mathcal{B}^{\star}(h),\mathcal{A}(\mu^{N}-\mu)\rangle_{\mathcal{L}^{2}(\mathcal{U})}.$

Taking expectation first and then applying Fubini’s theorem, we obtain

$\mathbb{E}\left[\langle
h,\mathcal{B}\mathcal{A}(\mu^{N}-\mu)\rangle_{\mathcal{L}^{2}(\mathcal{U})}\right]=\mathbb{E}\left[\langle\mathcal{B}^{\star}(h),\mathcal{A}(\mu^{N}-\mu)\rangle_{\mathcal{L}^{2}(\mathcal{U})}\right]\\\
=\int(\mathcal{B}^{\star}(h))(u)\mathbb{E}\left[\int
e^{\|u-y\|^{2}}(\differential\mu^{N}-\differential\mu)(y)\right]\differential\mathcal{U}(u)=0.$

The proof is concluded.

##  Appendix E Proofs for Section 4.2

###  E.1 Proof of Lemma 4.3

As in the statement of Condition 3.2, the analysis here is conducted
conditionally to $(\mu_{i},Y_{i})_{i=1}^{n}$ and we use the notation
$\mathbb{E}_{n}$ and $\mathbb{P}_{n}$ to denote the conditional expectation
and probability given $(\mu_{i},Y_{i})_{i=1}^{n}$. The independence property
of Condition 3.2 clearly holds, together with the property on
$(b_{N,i})_{i=1}^{n}$. Let $x\in\mathcal{H}_{k}$ be fixed. We have

$\displaystyle\mathbb{E}_{n}\left[\left\langle
x,x_{N,i}\right\rangle_{\mathcal{H}_{k}}-\left\langle
x,x_{i}\right\rangle_{\mathcal{H}_{k}}\right]=$
$\displaystyle\mathbb{E}_{n}\left[\left\langle
x,\frac{1}{N}\sum_{j=1}^{N}k(X_{i,j},\cdot)\right\rangle_{\mathcal{H}_{k}}-\left\langle
x,\int_{\Omega}k(t,\cdot)\differential\mu_{i}(t)\right\rangle_{\mathcal{H}_{k}}\right]$
$\displaystyle=$
$\displaystyle\mathbb{E}_{n}\left[\frac{1}{N}\sum_{j=1}^{N}x(X_{i,j})-\int_{\Omega}x(t)\differential\mu_{i}(t)\right]$
$\displaystyle=$ $\displaystyle 0.$

It remains to show the moment bound on $a_{N,i}$. For $t\in\Omega$, we let
$k_{t}=k(t,\cdot)\in\mathcal{H}_{k}$. Note that

$a_{N,i}=\frac{1}{N}\sum_{j=1}^{N}\left(k_{X_{i,j}}-\int_{\Omega}k_{t}\differential\mu_{i}(t)\right)$

is an average of i.i.d. random centered elements of $\mathcal{H}_{k}$. These
random elements have norm bounded by $B=2\sup_{u\in\Omega}\sqrt{k(u,u)}$.
Hence, using Caponnetto and De Vito (2007, Prop. 2 p. 345), as in Szabó et al.
(2016, Sect. 7.3.1), we obtain, for $\eta>0$,

$\mathbb{P}_{n}\left(\|a_{N,i}\|_{\mathcal{H}_{k}}\geq
2\left(\frac{2B}{n}+\frac{B}{\sqrt{n}}\right)\log(2/\eta)\right)\leq\eta.$

From there, it is simple to show that for any $s>0$, there is a constant
$c_{s}$ such that $\mathbb{E}_{n}[\|a_{N,i}\|_{\mathcal{H}_{k}}^{s}]\leq
c_{s}N^{-s/2}$. This completes the proof.

##  Appendix F Proofs for Section 4.3

###  F.1  Proof of Lemma 4.6

We write the proof only for the case $d\geq 2$, since the proof for $d=1$ uses
similar arguments and is simpler. Fix $\delta$ with $0<\delta<\epsilon$. Let
us fix $i$ and a realization of $\mu_{i}$ for which the almost sure statements
in the lemma hold, and let us work conditionally to this realization of
$\mu_{i}$. Let us fix $\theta\in\mathcal{S}^{d-1}$. We let $\mathcal{K}$ be
the support of $\mu_{i}$. We let $\theta,v_{2},\ldots,v_{d}$ be an orthonormal
basis of $\mathbb{R}^{d}$. We let $g:\mathbb{R}^{d}\to\mathbb{R}$ be the
density of $\mu_{i}$ (which is zero outside of $\mathcal{K}\subseteq\Omega$)
and we let $h:\mathbb{R}^{d}\to\mathbb{R}$ be the function defined by, for
$x\in\mathbb{R}^{d}$,

$g(x)=h(x^{\top}\theta,x^{\top}v_{2},\ldots,x^{\top}v_{d}).$

We let similarly $I_{\mathcal{K}}:\mathbb{R}^{d}\to\mathbb{R}$ be the function
defined by, for $x\in\mathbb{R}^{d}$,

$\mathrm{1}_{x\in\mathcal{K}}=I_{\mathcal{K}}(x^{\top}\theta,x^{\top}v_{2},\ldots,x^{\top}v_{d}).$

Note that
$\inf\\{h(x_{1},\ldots,x_{d});I_{\mathcal{K}}(x_{1},\ldots,x_{d})=1\\}=\inf\\{g(x);x\in\mathcal{K}\\}$.
Similarly

$\sup\\{h(x_{1},\ldots,x_{d});I_{\mathcal{K}}(x_{1},\ldots,x_{d})=1\\}=\sup\\{g(x);x\in\mathcal{K}\\}.$

Note also that, by assumption, $I_{\mathcal{K}}$ is zero outside of
$[-L,L]^{d}$, where $L=\kappa\sqrt{d}$. Consider the set

$\displaystyle S_{1}=$ $\displaystyle\Big{\\{}\
x_{1}\in\mathbb{R};\leavevmode\nobreak\ \text{the set}\leavevmode\nobreak\
\\{x_{2},\ldots,x_{d}\in\mathbb{R}\leavevmode\nobreak\
\text{s.t.}\leavevmode\nobreak\ h(x_{1},\ldots,x_{d})>0\\}$
$\displaystyle\leavevmode\nobreak\ \text{has non-zero Lebesgue masure in
$\mathbb{R}^{d-1}$}\Big{\\}}.$

Exploiting the convexity of the support of $\mu_{i}$, this set $S_{1}$ is a
segment of the form $[x_{\inf},x_{\sup}]$, $(x_{\inf},x_{\sup}]$,
$[x_{\inf},x_{\sup})$ or $(x_{\inf},x_{\sup})$, with
$[-\tau,\tau]\subseteq(x_{\inf},x_{\sup})$. Then for
$x_{1}\in(x_{\inf},x_{\sup})$, the density of $\theta^{\top}X_{i,1}$ at
$x_{1}$, written $g_{1}(x_{1})$, is

$\int_{[-L,L]^{d-1}}I_{\mathcal{K}}(x_{1},x_{2},\ldots,x_{d})h(x_{1},x_{2},\ldots,x_{d})\differential
x_{2}\cdots\differential x_{d}.$ (F.1)

By the definition of the set $S_{1}$, we see that the density $g_{1}$ is
strictly positive on $(x_{\inf},x_{\sup})$, which shows that
$F_{\mu_{i,\theta}}$ is bijective from $(x_{\inf},x_{\sup})$ to $(0,1)$. We
write $F_{i,\theta}^{-1}$ for its inverse function.

If $F_{i,\theta}^{-1}(\delta)\leq 0$, consider the following. Since the
function $hI_{\mathcal{K}}$ is bounded by $T$, there is a deterministic
constant $c_{1}>0$ such that there are $d-1$ segments
$[r_{2},s_{2}],\ldots,[r_{d},s_{d}]\subset\mathbb{R}$ of length at least
$c_{1}$ such that for some
$\bar{x}_{1}\in[x_{\inf},F_{i,\theta}^{-1}(\delta/2)]$,
$\bar{x}_{1}\theta+[r_{2},s_{2}]v_{2}+\cdots+[r_{d},s_{d}]v_{d}\subseteq\mathcal{K}$.
Considering the convex hull of the union of
$\bar{x}_{1}\theta+[r_{2},s_{2}]v_{2}+\cdots+[r_{d},s_{d}]v_{d}$ and
$B(0,\tau)$, that belongs to $\mathcal{K}$ which is convex, since $h$ is
lower-bounded when $I_{\mathcal{K}}$ is non-zero in (F.1), the density $g_{1}$
is lower-bounded on $[F_{i,\theta}^{-1}(\delta),0]$, by a deterministic
constant.

If $F_{i,\theta}^{-1}(1-\delta)\geq 0$, by the same reasoning, this density is
also lower-bounded on $[0,F_{i,\theta}^{-1}(1-\delta)]$, by a deterministic
constant. In the end, in all cases, the density $g_{1}$ is lower-bounded on
$[F_{i,\theta}^{-1}(\delta),F_{i,\theta}^{-1}(1-\delta)]$, by a deterministic
constant.

From (F.1), and since $h$ is upper bounded by $T$, then also $g_{1}$ is upper
bounded by $c_{2}$, with a deterministic constant $c_{2}$.

Let us now show that $g_{1}$ is differentiable on $(x_{\inf},x_{\sup})$, with
derivative bounded by a deterministic constant $c_{3}$. For this, we will use
that $h$ is differentiable on $\\{x;I_{\mathcal{K}}(x)=1\\}$, with gradient
bounded in Euclidean norm by $T$. Consider the set $S_{x_{1}}$ of the
$x_{2},\ldots,x_{d}$ such that the value of $x^{\prime}_{1}\mapsto
I_{\mathcal{K}}(x^{\prime}_{1},x_{2},\ldots,x_{d})$ is not locally constant at
$x_{1}$. It is sufficient so show that for $x_{1}\in(x_{\inf},x_{\sup})$,
$S_{x_{1}}$ has zero Lebesgue measure. This enables to conclude with the
dominated convergence theorem applied to (F.1).

Since $x_{1}\in(x_{\inf},x_{\sup})$, by convexity, there exists
$\tilde{x}_{2},\ldots,\tilde{x}_{d}$ such that
$(x_{1},\tilde{x}_{2},\ldots,\tilde{x}_{d})$ is in the interior of
$\\{x;I_{\mathcal{K}}(x)=1\\}$. Consider also
$(\bar{x}_{2},\ldots,\bar{x}_{d})$ in the interior of the convex set
$C_{x_{1}}$ defined by
$C_{x_{1}}=\\{x_{2},\ldots,x_{d};I_{\mathcal{K}}(x_{1},\ldots,x_{d})=1\\}$.
Then by convexity, $(\bar{x}_{2},\ldots,\bar{x}_{d})\not\in S_{x_{1}}$.
Consider then $(\bar{x}_{2},\ldots,\bar{x}_{d})\not\in C_{x_{1}}$. Since
$\\{x;I_{\mathcal{K}}(x)=1\\}$ is closed, as $\mathcal{K}$ is a probabilistic
support, then $(\bar{x}_{2},\ldots,\bar{x}_{d})\not\in S_{x_{1}}$.

Hence, we have shown that $S_{x_{1}}$ is included in the boundary of the
closed convex set $C_{x_{1}}$. This set has non-zero $d-1$-Lebesgue measure
because $x_{1}\in(x_{\inf},x_{\sup})$. Hence, it is well-known that this
boundary of $C_{x_{1}}$ has zero $d-1$-Lebesgue measure. Hence, indeed $g_{1}$
is differentiable on $(x_{\inf},x_{\sup})$, with derivative bounded by a
deterministic constant $c_{3}$.

Finally, we can exploit that the derivative of $F_{\mu_{i,\theta}}$ is $g_{1}$
to conclude that $F_{i,\theta}^{-1}$ is twice differentiable on
$(\delta,1-\delta)$ with first and second derivatives bounded in absolute
value, by deterministic constants.

###  F.2 Proof of Lemma 4.7

Let us fix $i$ and a realization of $\mu_{i}$ for which the almost sure
statements in Condition 4.5 hold, and let us work conditionally to this
realization of $\mu_{i}$. In particular, in the proof, the notations
$\mathbb{P}$ and $\mathbb{E}$ denote the conditional probability and
expectation given $\mu_{i}$. Fix $\theta\in\mathcal{S}^{d-1}$ and
$t\in(\epsilon,1-\epsilon)$. For $j=1,\ldots,N$, we let
$V_{j}=\theta^{\top}X_{i,j}$. We let $G$ and $G^{(N)}$ be the c.d.f. and
empirical c.d.f. of $V_{1},\ldots,V_{N}$. We also let $U_{j}=G(V_{j})$,
$j=1,\ldots,N$ and we remark that $U_{1},\ldots,U_{N}$ are i.i.d random
variables uniformly distributed on $[0,1]$, since $G$ is assumed to be
bijective from $(a_{i}(\theta),b_{i}(\theta))$ to $(0,1)$.

We let $V_{(1)}\leq\cdots\leq V_{(N)}$ be the order statistics of
$V_{1},\ldots,V_{N}$ and we define $U_{1},\cdots,U_{N}$ with $U_{j}=G(V_{j})$.
We let $U_{(1)}\leq\cdots\leq U_{(N)}$, breaking ties in the same way as for
$V_{(1)}\leq\cdots\leq V_{(N)}$. We let $\ell_{N}(t)\in\\{1,\ldots,N\\}$ be
the smallest integer larger or equal to $Nt$. For convenience, we may write
$\ell=\ell_{N}(t)$.

If $\epsilon>0$, we write $A_{N}$ for the event

$A_{N}=\\{V_{(\ell)}\in[G^{-1}(\delta),G^{-1}(1-\delta)]\\}$

and we let $\mathbf{1}_{A_{N}}$ be its indicator function and $A_{N}^{c}$ be
its complement event. If $\epsilon=0$, we let $\mathbf{1}_{A_{N}}=1$ by
convention. If $\epsilon>0$, note that $V_{(\ell)}\leq G^{-1}(\delta)$ implies
that $G^{(N)}(G^{-1}(\delta))\geq\ell/n$ and thus $G^{(N)}(G^{-1}(\delta))\geq
t\geq\epsilon$. Hence, using Hoeffding inequality, one can see that there are
deterministic constants $c_{1}$ and $c_{2}$ (not depending on $N,t,\theta$)
with $c_{2}>0$ such that $\mathbb{P}(V_{(\ell)}\leq G^{-1}(\delta))\leq
c_{1}e^{-c_{2}N}$. Reasoning similarly on the event $V_{(\ell)}\geq
G^{-1}(1-\delta)$, up to changing $c_{1}$ and $c_{2}$, we have

$\mathbb{P}\left(A_{N}^{c}\right)\leq c_{1}e^{-c_{2}N}.$ (F.2)

Of course, (F.2) also holds for $\epsilon=0$. Then, writing
$G^{(N)-1}=(G^{(N)})^{-1}$,

$\displaystyle\mathbb{E}\left[G^{(N)-1}(t)-G^{-1}(t)\right]=$
$\displaystyle\underbrace{\mathbb{E}\left[\mathbf{1}_{A_{N}^{c}}G^{(N)-1}(t)-\mathbf{1}_{A_{N}^{c}}G^{-1}(t)\right]}_{=B_{1}}$
$\displaystyle+\underbrace{\mathbb{E}\left[\mathbf{1}_{A_{N}}G^{(N)-1}(t)-\mathbf{1}_{A_{N}}G^{-1}(t)\right]}_{=B_{2}}.$

From (F.2), and because the values of $G^{(N)-1}(t)$ and $G^{-1}(t)$ are
bounded by $\max\\{||x||;x\in\Omega\\}$, we obtain

$|B_{1}|\leq\frac{c_{3}}{N},$ (F.3)

for a constant $c_{3}$. Next,

$B_{2}=\mathbb{E}\left[\mathbf{1}_{A_{N}}V_{(\ell)}-\mathbf{1}_{A_{N}}G^{-1}(t)\right]=\mathbb{E}\left[\mathbf{1}_{A_{N}}G^{-1}(U_{(\ell)})-\mathbf{1}_{A_{N}}G^{-1}(t)\right].$

By a Taylor expansion, exploiting the fact that the event $A_{N}$ holds in the
right-most expectation above, we obtain, with a random
$\xi\in(\delta,1-\delta)$, writing $G^{-1^{\prime}}$ and
$G^{-1^{\prime\prime}}$ for the first and second derivatives of $G^{-1}$ on
$(\delta,1-\delta)$,

$\displaystyle B_{2}=$
$\displaystyle\mathbb{E}\left[\mathbf{1}_{A_{N}}G^{-1^{\prime}}(t)\left(U_{(\ell)}-t\right)+\frac{\mathbf{1}_{A_{N}}}{2}G^{-1^{\prime\prime}}(\xi)\left(U_{(\ell)}-t\right)^{2}\right]$
$\displaystyle=$ $\displaystyle
G^{-1^{\prime}}(t)\mathbb{E}\left[U_{(\ell)}-t\right]-G^{-1^{\prime}}(t)\mathbb{E}\left[\mathbf{1}_{A_{N}^{c}}\left(U_{(\ell)}-t\right)\right]+\mathbb{E}\left[\frac{\mathbf{1}_{A_{N}}}{2}G^{-1^{\prime\prime}}(\xi)\left(U_{(\ell)}-t\right)^{2}\right].$

Above, $U_{(\ell)}$ follows the $\mathcal{B}(\ell,N+1-\ell)$ distribution,
where $\mathcal{B}$ stands for the Beta distribution. Hence, in the above
display, the first expectation is of order $1/N$ since $|Nt-\ell|\leq 1$. The
second expectation is of order $1/N$ from the same arguments as before (F.3).
The quantities $G^{-1^{\prime}}(t)$ and $G^{-1^{\prime\prime}}(\xi)$ are
bounded by $c^{(2)}$ from Condition 4.5 because $t$ and $\xi$ are in
$(\delta,1-\delta)$. Hence, the third expectation above is of order $1/N$ also
because $U_{(\ell)}$ follows the $\mathcal{B}(\ell,N+1-\ell)$ distribution.
Thus, using (F.3), we have

$\left|\mathbb{E}\left[G^{(N)-1}(t)-G^{-1}(t)\right]\right|\leq\frac{c_{4}}{N},$
(F.4)

for a constant $c_{4}$.

Then for $\theta\in\mathcal{S}^{d-1}$ and $t\in(\epsilon,1-\epsilon)$, we let
$F^{(N)-1}_{\mu_{i,\theta}}=(F^{(N)}_{\mu_{i,\theta}})^{-1}$, where
$F^{(N)}_{\mu_{i,\theta}}$ is the empirical c.d.f. of
$\theta^{\top}X_{i,1},\ldots,\theta^{\top}X_{i,N}$, and we define

$a_{N,i}(\theta,t)=F^{(N)-1}_{\mu_{i,\theta}}(t)-\mathbb{E}\left[F^{(N)-1}_{\mu_{i,\theta}}(t)\right]$

and

$b_{N,i}(\theta,t)=\mathbb{E}\left[F^{(N)-1}_{\mu_{i,\theta}}(t)\right]-F_{\mu_{i,\theta}}^{-1}(t).$

Hence
$a_{N,i}(\theta,t)+b_{N,i}(\theta,t)=F^{(N)-1}_{\mu_{i,\theta}}(t)-F_{\mu_{i,\theta}}^{-1}(t)=x_{N,i}(\theta,t)-x_{i}(\theta,t)$.
Also, (3.3) is simple to show. Since $b_{N,i}$ is deterministic, (F.4) implies
(3.2). It thus remains to prove (3.1).

Let us fix $s\geq 1$. We have, using Jensen’s inequality twice,

$\displaystyle\mathbb{E}[\|a_{N,i}\|_{\mathcal{H}}^{s}]=$
$\displaystyle\mathbb{E}\left[\left(\frac{1}{1-2\epsilon}\int_{\mathcal{S}^{d-1}}\int_{\epsilon}^{1-\epsilon}\left(F^{(N)-1}_{\mu_{i,\theta}}(t)-\mathbb{E}\left[F^{(N)-1}_{\mu_{i,\theta}}(t)\right]\right)^{2}\differential\Lambda(\theta)\differential
t\right)^{s/2}\right]$
$\displaystyle\leq\sqrt{\mathbb{E}\left[\left(\frac{1}{1-2\epsilon}\int_{\mathcal{S}^{d-1}}\int_{\epsilon}^{1-\epsilon}\left(F^{(N)-1}_{\mu_{i,\theta}}(t)-\mathbb{E}\left[F^{(N)-1}_{\mu_{i,\theta}}(t)\right]\right)^{2}\differential\Lambda(\theta)\differential
t\right)^{s}\right]}$
$\displaystyle\leq\sqrt{\mathbb{E}\left[\frac{1}{1-2\epsilon}\int_{\mathcal{S}^{d-1}}\int_{\epsilon}^{1-\epsilon}\left(F^{(N)-1}_{\mu_{i,\theta}}(t)-\mathbb{E}\left[F^{(N)-1}_{\mu_{i,\theta}}(t)\right]\right)^{2s}\differential\Lambda(\theta)\differential
t\right]}.$ (F.5)

We now fix $t\in(\epsilon,1-\epsilon)$ and $\theta\in\mathcal{S}^{d-1}$. We
use the same notation as above: $G$, $V_{1},\ldots,V_{N}$,
$U_{1},\ldots,U_{N}$, $A_{N}$. We then study the above integrand

$\displaystyle
F^{(N)-1}_{\mu_{i,\theta}}(t)-\mathbb{E}\left[F^{(N)-1}_{\mu_{i,\theta}}(t)\right]$
$\displaystyle=$ $\displaystyle
G^{(N)-1}(t)-\mathbb{E}\left[G^{(N)-1}(t)\right]$ $\displaystyle=$
$\displaystyle V_{(\ell)}-\mathbb{E}V_{(\ell)}$ $\displaystyle=$
$\displaystyle G^{-1}(U_{(\ell)})-\mathbb{E}G^{-1}(U_{(\ell)})$
$\displaystyle=$
$\displaystyle\underbrace{\mathrm{1}_{A_{N}}G^{-1}(U_{(\ell)})-\mathbb{E}[\mathrm{1}_{A_{N}}G^{-1}(U_{(\ell)})]}_{C_{2}}+\underbrace{\mathrm{1}_{A_{N}^{c}}G^{-1}(U_{(\ell)})-\mathbb{E}[\mathrm{1}_{A_{N}^{c}}G^{-1}(U_{(\ell)})]}_{C_{1}}.$

From the same arguments as before (F.3), we have

$\mathbb{E}[|C_{1}|^{2s}]\leq\frac{c_{s,5}}{N^{s}},$ (F.6)

for a constant $c_{s,5}$ (not depending on $N$, $\theta$, $t$).

To study $C_{2}$, we use similarly as before that under the event $A_{N}$
there is a random $\xi\in(\delta,1-\delta)$ such that

$\displaystyle C_{2}=$
$\displaystyle\mathrm{1}_{A_{N}}G^{-1}\left(\frac{\ell}{N+1}\right)+\mathrm{1}_{A_{N}}G^{-1^{\prime}}\left(\xi\right)\left(U_{(\ell)}-\frac{\ell}{N+1}\right)$
$\displaystyle-\mathbb{E}\left[\mathrm{1}_{A_{N}}G^{-1}\left(\frac{\ell}{N+1}\right)\right]-\mathbb{E}\left[\mathrm{1}_{A_{N}}G^{-1^{\prime}}\left(\xi\right)\left(U_{(\ell)}-\frac{\ell}{N+1}\right)\right].$

Above, $|G^{-1^{\prime}}\left(\xi\right)|$ is bounded by $c^{(2)}$ from
Condition 4.5 since $A_{N}$ holds. Using (F.2) as above, we can show, for a
constant $c_{s,6}$,

$\mathbb{E}[|C_{2}|^{2s}]\leq
c_{s,6}\mathbb{E}\left[\left|U_{(\ell)}-\frac{\ell}{N+1}\right|^{2s}\right]+\frac{c_{s,6}}{N^{s}}.$

We can finally use Skorski (2023, Thm. 3), together with a simple induction,
to obtain

$\mathbb{E}[|C_{2}|^{2s}]\leq\frac{c_{s,7}}{N^{s}}$

for a constant $c_{s,7}$. Combined with (F.2) and (F.6), we thus obtain that
(3.1) holds for $s\geq 1$. From Jensen’s inequality, (3.1) also holds for
$s\leq 1$ and the proof is concluded.

## References

  * Arjovsky, Chintala and Bottou (2017) [author] Arjovsky, MartinM., Chintala, SoumithS. and Bottou, LéonL. (2017). Wasserstein Generative Adversarial Networks. In International Conference on Machine Learning.
  * Bachoc et al. (2017) [author] Bachoc, FrançoisF., Gamboa, FabriceF., Loubes, Jean-MichelJ.-M. and Venet, NilN. (2017). A Gaussian process regression model for distribution inputs. IEEE Transactions on Information Theory 64 6620–6637.
  * Bachoc et al. (2020) [author] Bachoc, FrançoisF., Suvorikova, AlexandraA., Ginsbourger, DavidD., Loubes, Jean-MichelJ.-M. and Spokoiny, VladimirV. (2020). Gaussian processes with multidimensional distribution inputs via optimal transport and Hilbertian embedding. Electronic Journal of Statistics 14 2742–2772.
  * Bachoc et al. (2023) [author] Bachoc, FrançoisF., Béthune, LouisL., González-Sanz, AlbertoA. and Loubes, Jean-MichelJ.-M. (2023). Gaussian processes on distributions based on regularized optimal transport. In International Conference on Artificial Intelligence and Statistics.
  * Berlinet and Thomas-Agnan (2011) [author] Berlinet, AlainA. and Thomas-Agnan, ChristineC. (2011). Reproducing kernel Hilbert spaces in probability and statistics. Springer Science & Business Media.
  * Bernton et al. (2019) [author] Bernton, EspenE., Jacob, Pierre EP. E., Gerber, MathieuM. and Robert, Christian PC. P. (2019). On parameter estimation with the Wasserstein distance. Information and Inference: A Journal of the IMA 8 657–676.
  * Caponnetto and De Vito (2007) [author] Caponnetto, AndreaA. and De Vito, ErnestoE. (2007). Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics 7 331–368.
  * Carlier and Laborde (2020) [author] Carlier, GuillaumeG. and Laborde, MaximeM. (2020). A differential approach to the multi-marginal schrödinger system. SIAM Journal on Mathematical Analysis 52 709-717.
  * Catalano, Lijoi and Prünster (2021) [author] Catalano, MartaM., Lijoi, AntonioA. and Prünster, IgorI. (2021). Measuring dependence in the Wasserstein distance for Bayesian nonparametric models. The Annals of Statistics 49 2916–2947.
  * Christmann and Steinwart (2010) [author] Christmann, AndreasA. and Steinwart, IngoI. (2010). Universal kernels on non-standard input spaces. In Advances in Neural Information Processing Systems.
  * Cuturi (2013) [author] Cuturi, MarcoM. (2013). Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems.
  * Cuturi et al. (2022) [author] Cuturi, MarcoM., Meng-Papaxanthos, LaetitiaL., Tian, YingtaoY., Bunne, CharlotteC., Davis, GeoffG. and Teboul, OlivierO. (2022). Optimal Transport Tools (OTT): A JAX Toolbox for all things Wasserstein. arXiv preprint arXiv:2201.12324.
  * del Barrio et al. (2023) [author] del Barrio, EustasioE., Sanz, Alberto GonzálezA. G., Loubes, Jean-MichelJ.-M. and Niles-Weed, JonathanJ. (2023). An improved central limit theorem and fast convergence rates for entropic transportation costs. SIAM Journal on Mathematics of Data Science 5 639-669.
  * Flamary and Courty (2017) [author] Flamary, RémiR. and Courty, NicolasN. (2017). POT Python Optimal Transport library.
  * Flaxman, Wang and Smola (2015) [author] Flaxman, Seth RS. R., Wang, Yu-XiangY.-X. and Smola, Alexander JA. J. (2015). Who supported Obama in 2012? Ecological inference through distribution regression. In International Conference on Knowledge Discovery and Data Mining.
  * Flaxman et al. (2016) [author] Flaxman, SethS., Sutherland, Danica JD. J., Wang, Yu-XiangY.-X. and Teh, Yee WhyeY. W. (2016). Understanding the 2016 US Presidential Election using ecological inference and distribution regression with census microdata. arXiv preprint arXiv:1611.03787.
  * Genevay (2019) [author] Genevay, AudeA. (2019). Entropy-regularized optimal transport for machine learning, PhD thesis, Paris Sciences et Lettres (ComUE).
  * González-Sanz and Hundrieser (2023) [author] González-Sanz, AlbertoA. and Hundrieser, ShayanS. (2023). Weak limits for empirical entropic optimal transport: beyond smooth costs. arXiv preprint arXiv:2305.09745.
  * González-Sanz, Loubes and Niles-Weed (2022) [author] González-Sanz, AlbertoA., Loubes, Jean-MichelJ.-M. and Niles-Weed, JonathanJ. (2022). Weak limits of entropy regularized Optimal Transport; potentials, plans and divergences. arXiv preprint arXiv:2207.07427.
  * Hastie et al. (2009) [author] Hastie, TrevorT., Tibshirani, RobertR., Friedman, Jerome HJ. H. and Friedman, Jerome HJ. H. (2009). The elements of statistical learning: data mining, inference, and prediction 2\. Springer.
  * Kimeldorf and Wahba (1971) [author] Kimeldorf, GeorgeG. and Wahba, GraceG. (1971). Some results on Tchebycheffian spline functions. Journal of Mathematical Analysis and Applications 33 82–95.
  * Klatt (2017) [author] Klatt, MarcelM. (2017). Package ‘Barycenter’. R package.
  * Kolouri, Rohde and Hoffmann (2018) [author] Kolouri, SoheilS., Rohde, Gustavo KG. K. and Hoffmann, HeikoH. (2018). Sliced Wasserstein distance for learning Gaussian mixture models. In IEEE Conference on Computer Vision and Pattern Recognition.
  * Kolouri, Zou and Rohde (2016) [author] Kolouri, SoheilS., Zou, YangY. and Rohde, Gustavo KG. K. (2016). Sliced Wasserstein kernels for probability distributions. In IEEE Conference on Computer Vision and Pattern Recognition.
  * Ledoux and Talagrand (1991) [author] Ledoux, MichelM. and Talagrand, MichelM. (1991). Probability in Banach Spaces: isoperimetry and processes 23\. Springer Science & Business Media.
  * Manole, Balakrishnan and Wasserman (2022) [author] Manole, TudorT., Balakrishnan, SivaramanS. and Wasserman, LarryL. (2022). Minimax confidence intervals for the sliced Wasserstein distance. Electronic Journal of Statistics 16 2252–2345.
  * Meunier, Pontil and Ciliberto (2022) [author] Meunier, DimitriD., Pontil, MassimilianoM. and Ciliberto, CarloC. (2022). Distribution regression with sliced Wasserstein kernels. In International Conference on Machine Learning.
  * Muandet et al. (2012) [author] Muandet, KrikamolK., Fukumizu, KenjiK., Dinuzzo, FrancescoF. and Schölkopf, BernhardB. (2012). Learning from distributions via support measure machines. In Advances in Neural Information Processing Systems.
  * Muandet et al. (2017) [author] Muandet, KrikamolK., Fukumizu, KenjiK., Sriperumbudur, BharathB. and Schölkopf, BernhardB. (2017). Kernel mean embedding of distributions: A review and beyond. Foundations and Trends® in Machine Learning 10 1–141.
  * Murphy (2014) [author] Murphy, Gerald JG. J. (2014). C*-algebras and operator theory. Academic press.
  * Niles-Weed and Berthet (2022) [author] Niles-Weed, JonathanJ. and Berthet, QuentinQ. (2022). Minimax estimation of smooth densities in Wasserstein distance. The Annals of Statistics 50 1519–1540.
  * Oliva et al. (2014) [author] Oliva, JunierJ., Neiswanger, WillieW., Póczos, BarnabásB., Schneider, JeffJ. and Xing, EricE. (2014). Fast distribution to real regression. In International Conference on Artificial Intelligence and Statistics.
  * Pedersen (2012) [author] Pedersen, Gert KG. K. (2012). Analysis now 118\. Springer Science & Business Media.
  * Peyré, Cuturi and Solomon (2016) [author] Peyré, GabrielG., Cuturi, MarcoM. and Solomon, JustinJ. (2016). Gromov-Wasserstein averaging of kernel and distance matrices. In International Conference on Machine Learning.
  * Peyré and Cuturi (2019) [author] Peyré, GabrielG. and Cuturi, MarcoM. (2019). Computational optimal transport. Foundations and Trends® in Machine Learning 11 355–607.
  * Póczos et al. (2013) [author] Póczos, BarnabásB., Singh, AartiA., Rinaldo, AlessandroA. and Wasserman, LarryL. (2013). Distribution-free distribution regression. In International Conference on Artificial Intelligence and Statistics.
  * Portnoy (2012) [author] Portnoy, StephenS. (2012). Nearly root-n approximation for regression quantile processes. The Annals of Statistics 40 1714–1736.
  * Rahimi and Recht (2007) [author] Rahimi, AliA. and Recht, BenjaminB. (2007). Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems.
  * Ramsay and Silverman (2007) [author] Ramsay, James OJ. O. and Silverman, Bernard WB. W. (2007). Applied functional data analysis: methods and case studies. Springer.
  * Schölkopf and Smola (2002) [author] Schölkopf, BernhardB. and Smola, Alexander JA. J. (2002). Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT press.
  * Skorski (2023) [author] Skorski, MaciejM. (2023). Bernstein-type bounds for beta distribution. Modern Stochastics: Theory and Applications 10 211–228.
  * Smola et al. (2007) [author] Smola, AlexA., Gretton, ArthurA., Song, LeL. and Schölkopf, BernhardB. (2007). A Hilbert space embedding for distributions. In International Conference on Algorithmic Learning Theory.
  * Srivastava, Li and Dunson (2018) [author] Srivastava, SanveshS., Li, ChengC. and Dunson, David BD. B. (2018). Scalable Bayes via barycenter in Wasserstein space. The Journal of Machine Learning Research 19 312–346.
  * Szabó et al. (2015) [author] Szabó, ZoltánZ., Gretton, ArthurA., Póczos, BarnabásB. and Sriperumbudur, BharathB. (2015). Two-stage sampled learning theory on distributions. In International Conference on Artificial Intelligence and Statistics.
  * Szabó et al. (2016) [author] Szabó, ZoltánZ., Sriperumbudur, BharathB., Póczos, BarnabásB. and Gretton, ArthurA. (2016). Learning theory for distribution regression. The Journal of Machine Learning Research 17 5272–5311.
  * Thi Thien Trang et al. (2021) [author] Thi Thien Trang, BuiB., Loubes, Jean-MichelJ.-M., Risser, LaurentL. and Balaresque, PatriciaP. (2021). Distribution regression model with a Reproducing Kernel Hilbert Space approach. Communications in Statistics-Theory and Methods 50 1955–1977.
  * van der Vaart and van Zanten (2009) [author] van der Vaart, Aad WA. W. and van Zanten, J HarryJ. H. (2009). Adaptive Bayesian estimation using a Gaussian random field with inverse gamma bandwidth. The Annals of Statistics 37 2655–2675.
  * van der Vaart and Wellner (2013) [author] van der Vaart, A.A. and Wellner, J.J. (2013). Weak convergence and empirical processes: with applications to statistics. Springer Science & Business Media.
  * Villani (2003) [author] Villani, CedricC. (2003). Topics in Optimal Transportation. American mathematical society, Providence, Rhode Island.


# Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for
Autonomous Real-World Reinforcement Learning

Jingyun Yang∗, Max Sobol Mark∗, Brandon Vu, Archit Sharma, Jeannette Bohg,
Chelsea Finn ∗Equal contribution. All authors are affiliated with Department
of Computer Science, Stanford University. Toyota Research Institute provided
funds to support this work. Contact<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The pre-train and fine-tune paradigm in machine learning has had dramatic
success in a wide range of domains because the use of existing data or pre-
trained models on the internet enables quick and easy learning of new tasks.
We aim to enable this paradigm in robotic reinforcement learning, allowing a
robot to learn a new task with little human effort by leveraging data and
models from the Internet. However, reinforcement learning often requires
significant human effort in the form of manual reward specification or
environment resets, even if the policy is pre-trained. We introduce RoboFuME,
a reset-free fine-tuning system that pre-trains a multi-task manipulation
policy from diverse datasets of prior experiences and self-improves online to
learn a target task with minimal human intervention. Our insights are to
utilize calibrated offline reinforcement learning techniques to ensure
efficient online fine-tuning of a pre-trained policy in the presence of
distribution shifts and leverage pre-trained vision language models (VLMs) to
build a robust reward classifier for autonomously providing reward signals
during the online fine-tuning process. In a diverse set of five real robot
manipulation tasks, we show that our method can incorporate data from an
existing robot dataset collected at a different institution and improve on a
target task within as little as 3 hours of autonomous real-world experience.
We also demonstrate in simulation experiments that our method outperforms
prior works that use different RL algorithms or different approaches for
predicting rewards. Project website: https://robofume.github.io

##  I Introduction

In many domains that involve machine learning, a widely successful paradigm
for learning task-specific models is to first pre-train a general-purpose
model from an existing diverse prior dataset, and then adapt the model with a
small addition of task-specific data [1, 2, 3, 4, 5]. This paradigm is
attractive to real-world robot learning, since collecting data on a robot is
expensive, and fine-tuning an existing model on a small task-specific dataset
could substantially improve the data efficiency for learning a new task. Pre-
training a policy with offline reinforcement learning and then fine-tuning it
with online reinforcement learning is a natural way to implement this paradigm
in robotics. However, numerous challenges arise when using this recipe in
practice. First, off-the-shelf robot datasets often use different objects,
fixture placements, camera viewpoints, and lighting conditions compared to the
local robot platform. This causes non-trivial distribution shifts between pre-
training and online fine-tuning data, which makes effectively fine-tuning a
robot policy difficult. Indeed, most existing works only show the benefit of
the pre-train and fine-tune paradigm where the robot uses the same hardware
instance in both pre-training and fine-tuning phases [6, 7]. Second, training
or fine-tuning a policy in the real world often requires extensive human
supervision, which includes manually resetting the environment between trials
[8, 9, 10] and engineering reward functions [11, 12, 7]. In this work, our
goal is to address these two challenges and develop a practical framework that
enables robot fine-tuning with minimal time and human effort.

Over the past few years, there has been a lot of progress in designing
efficient and autonomous reinforcement learning algorithms. However, no
existing system could both utilize diverse demonstration datasets and learn
with minimal human supervision, without the need for human-engineered reward
functions and manual environment resets. Some works propose to reduce the need
for manual environment resets using reset-free RL [11, 7, 13], where an agent
alternates between running a task policy and a reset policy during training
while updating both with online experience. However, these works do not
leverage diverse off-the-shelf robot datasets. Recent advances in offline RL
algorithms have enabled policies to leverage diverse offline data and improve
further via online fine-tuning [14, 15], but these new methods have not been
integrated into a system that aims at minimizing human supervision in the
fine-tuning phase. There are also works that propose to eliminate the need for
human-specified reward functions by learning reward prediction models [16, 17,
18, 13], but we found that many of these proposed models can be brittle when
deployed in a real-world RL fine-tuning setup. In summary, although prior
works have presented individual components that are vital to building a
working system for efficient and human-free robot learning, it is not clear
which components one should use to put together such a system and how.

We design RoboFuME, a system that enables autonomous and efficient real-world
robot learning by leveraging diverse offline datasets and online fine-tuning.
Our system operates in two phases. In the pre-training phase, we assume access
to a diverse prior dataset, a few task demonstrations and reset demonstrations
of the target task, and a small collection of sample failure observations in
the target task. From this data, we learn a language-conditioned, multi-task
policy with offline RL. To cope with the distribution shift between the
offline dataset and online interactions, we need an algorithm that could
effectively digest diverse offline data, and display robust fine-tuning
performance when placed into an environment different from those seen in the
offline dataset. We find that calibrated offline RL techniques [15], by
underestimating predicted values of the learned policy from offline data and
correcting the scale of the learned Q-values, make sure that the pre-trained
policy can effectively digest diverse offline data and continuously improve
during online adaptation. To ensure the online fine-tuning phase requires
minimal human feedback, we need to remove the need for reward engineering by
learning a reward predictor. Our insight is to take a large vision-language
model (VLM) to provide a robust pre-trained representation and fine-tune it
with a small amount of in-domain data so that it is tailored for the reward
classification setup. Pre-trained VLMs have already been trained on internet-
scale visual and language data. This makes the model more robust to lighting
and camera positioning variations than the models used in prior works. In the
fine-tuning phase, a robot adapts the policy in the real world autonomously by
alternating between attempting the task and attempting to reset the
environment to the initial state distribution of the task. Meanwhile, the
agent uses the pre-trained VLM model as a surrogate reward for updating the
policy.

We evaluate our framework by pre-training it on the Bridge dataset [19] and
testing it on a diverse set of real-world downstream tasks: cloth folding,
cloth covering, sponge pick-and-place, placing lid on a pot, and putting a pot
in a sink. We find that our system provides substantial improvements over
offline-only methods with as little as 3 hours of real-world training. We
perform more quantitative experiments in a simulation setup, where we
illustrate that our method outperforms imitation learning and offline RL
methods that either do not perform fine-tuning online or do not incorporate
diverse prior data.

Our main contributions include (1) a fully autonomous system for pre-training
from a prior robot dataset and fine-tuning on an unseen downstream task with a
minimal number of resets and learned reward labels; (2) a method for fine-
tuning pre-trained vision-language models and using them to construct a
surrogate reward for downstream RL training.

##  II Related Work

Offline RL. Offline RL algorithms [20, 21, 22, 23, 24, 25] provide a framework
for initializing robot manipulation policies from offline demonstrations or
interaction datasets. Such algorithms can also be extended to include an
online fine-tuning phase after training a policy offline [26, 27, 28, 29, 30,
31, 32, 15]. Our work utilizes a recent offline RL algorithm, calibrated
Q-learning (CalQL) [15], a state-of-the-art method that effectively learns
from offline data and continuously improves the policy’s performance online by
explicitly correcting the scale of the learned Q-values. We show that
integrating CalQL helps our framework effectively utilize diverse prior
datasets that have large distribution shifts from real-world online
interactions.

Reset-free RL. Training an RL policy on a real robot typically requires manual
environment resets. To eliminate such need to manually reset environments,
prior works have studied approaches to learn robot policies in a ‘reset-free’
setup. Some work [33, 34, 11, 35, 36] cast the ‘reset-free’ learning problem
as a multi-task learning problem, observing that by learning a set of tasks
where some of the tasks could reset others, an agent could then be trained to
perform all of those tasks without needing manual resets. Other works [37, 38,
39, 12, 17, 13, 7] learn both a task policy and a reset policy for performing
the task and resetting to the initial state distribution. Our work takes an
approach between the two classes of approaches, learning a language-
conditioned multi-task policy that can perform both the target task and the
reset for the target task. Most of these prior works learn from scratch rather
than incorporating prior data and assume that a reward function is available.
ARIEL [7] combines incorporating prior data with reset-free learning but
assumes a hand-crafted reward function for each environment. They also collect
their own prior dataset on the same robot hardware set-up as their target
task. MEDAL++ [13] learns a reward classifier with demonstration and online
interaction data via adversarial training, but does not consider incorporating
diverse prior data. Leveraging diverse, off-the-shelf prior demonstration
datasets is desirable since these datasets are readily available to use and
can help a system obtain a policy initialization for efficient fine-tuning on
a target task. Our system offers an approach to both incorporate diverse prior
data and improve the autonomy of the fine-tuning phase by learning a model for
predicting rewards. In particular, we found out that by leveraging diverse
demonstration data, our system requires only about 3 hours of training in the
real world compared to 10-30 hours in MEDAL++.

Reward learning. Early works have studied the problem of learning a reward or
cost function in imitation learning. These works leverage inverse optimal
control (IOC) or inverse reinforcement learning (IRL) to extract a reward
function directly from expert demonstrations [40, 41, 42]. With the advent of
deep neural networks, more recent works have explored learning a reward model
for an imitation learning or RL policy [43, 44, 45, 46, 47, 17, 13]. When
using classifier-based reward models in reinforcement learning, RL agents can
exploit the learned model by exploring states unseen during classifier
training, tricking the model to output incorrect rewards. To solve such an
exploitation issue, many works that learn reward models leverage adversarial
learning, where a system learns a discriminator that identifies states similar
to those in demonstrations as positives and those visited by the policy as
negatives [44, 47, 17, 13]. However, prior work has found this training
objective to be sensitive to distribution shifts between offline and online
setups, such as lighting and camera view changes [48]. In this work, we fine-
tune vision language models (VLM), pre-trained on internet-scale data, to
construct a reward model. Large scale pre-training can learn representations
that are robust to natural variations such as lighting, camera shifts and
distractors [49, 16].

Leveraging pre-trained representations as reward predictors. Several recent
works have shown positive results in utilizing pre-trained vision models [50,
16], large language models (LLMs) [51] or vision language models (VLMs) [52]
as reward predictors. We tried VIP [16], a method that pre-trains a visual
representation for generating dense reward functions for novel robotic tasks,
and found it insufficient for the real-world robot fine-tuning setup. In this
work, we fine-tune a pre-trained VLM [53] and find that it performs most
effectively as a reward model. Our proposed system is flexible and can easily
be adapted to use other pre-trained visual representations and VLMs.

##  III Preliminaries

The goal of our method is to leverage diverse prior demonstration datasets and
learn a novel target task autonomously in a robot hardware instance that is
distinct from the one used to collect the datasets. Our method assumes access
to a prior dataset
$\mathcal{D}_{\text{prior}}=\cup_{j=1}^{N}\mathcal{D}_{j}=\cup_{j=1}^{N}\\{(s_{i}^{j},a_{i}^{j},s^{\prime
j}_{i})\\}_{i=1}^{K}$, which consists of demonstrations of $N$ different tasks
$\tau_{1},\ldots\tau_{k}$. We assume that all demonstration data uses image
observations. The method will be tested on a downstream task $\tau_{f}$, which
is different from any of the prior tasks.

To facilitate learning on the downstream task, we also assume the availability
of a small set of target task demos $\mathcal{D}_{f}$, target task reset demos
$\mathcal{D}_{b}$, and target task failure states $\mathcal{D}_{\frownie{}}$.
The reset demos $\mathcal{D}_{b}$ come from the reset task $\tau_{b}$ which
resets the environment from an end state of $\tau_{f}$ to the initial state
distribution of $\tau_{f}$. The failure states $\mathcal{D}_{\frownie{}}$
consist entirely of image observations that correspond to unsuccessful states
and are collected to aid with the VLM reward learning. In addition to all the
given data ($\mathcal{D}_{\text{prior}}$, $\mathcal{D}_{f}$,
$\mathcal{D}_{b}$, $\mathcal{D}_{\frownie{}}$), each task $\tau$ is also
accompanied with a language description $l$.

##  IV RoboFuME

Our work focuses on designing an efficient and scalable framework for pre-
training on a diverse set of prior demonstrations and autonomously fine-tuning
on target tasks. Our system consists of an offline pre-training phase and an
online fine-tuning phase. In Section IV-A, we discuss how we pre-train a
language-conditioned multi-task policy on diverse data that can be fine-tuned
online efficiently. Online fine-tuning requires a reward function to label
successes and failures. In Section IV-B, we introduce a VLM-based classifier
for providing a reward signal to the policy in the fine-tuning phase. Finally,
in Section IV-C, we describe how to autonomously adapt the pre-trained policy
in the fine-tuning phase by utilizing the VLM-based reward classifier as a
reward signal and chaining forward and backward behaviors to practice the task
with minimal human interventions.

###  IV-A Pre-Training a Multi-Task Policy on Diverse Prior Data

Prior work has shown that training a policy using a conservative Q-value
function is an effective way to obtain a good policy from an offline dataset
[22, 24]. However, fine-tuning can be critical to learn competent policies as
prior data may not provide sufficient coverage, especially for new tasks or
scenes. We leverage CalQL [15] which modifies the conservative Q-learning
algorithm CQL such that it enables efficient online fine-tuning by enforcing
calibration on the Q-function (i.e. making the Q-value of the learned policy
no lower than the Monte-Carlo returns in the prior dataset). CalQL allows us
to improve the pre-trained policy efficiently with respect to online
interactions.

CalQL requires the training of an actor and a critic. Since we use image
observations, we additionally train an encoder $\phi(s_{\text{img}})$ that
projects the images into a lower-dimensional space before giving them as
inputs to the actor and critic. The encoder $\phi$ is a 4-layer CNN, and is
optimized exclusively against the critic loss. To best utilize the multi-task
data, we encode task descriptions $l$ using pre-trained CLIP embeddings,
resulting in an embedding ${z=\text{CLIP}(l)}$ which is used as the task
representation. The policy then takes as inputs a concatenation of the encoded
image observation $\phi(s_{\text{img}})$, task representation $z$, and
proprioceptive information $s_{p}$, processes the concatenated vector through
an MLP, and produces the output action $a$.

In addition to updating the policy using CalQL, we regularize policy learning
with a behavior cloning (BC) loss, which encourages the behaviors to stay
close to the seen demonstrations. Not only does this regularization improve
performance of the offline pre-training, but we find that it also makes it
less likely for the autonomous fine-tuning procedure to exploit false positive
rewards from the VLM reward model. The weight of the BC regularization term is
chosen such that the scales of the RL loss and the BC loss are similar
throughout the pre-training phase. We train the policy $\pi$ and the critic
$Q$ with datasets
$\mathcal{D}_{\text{prior}},\mathcal{D}_{f},\mathcal{D}_{b}$. After the
offline learning phase, the policy and critic contain knowledge of all tasks
in the prior data and the target task.

###  IV-B Fine-Tuning A Vision-Language Model for Rewards

To improve the autonomy of the policy fine-tuning phase, our agent needs to
perform online fine-tuning without manually labeled or engineered reward
functions. To achieve this, we propose to fine-tune off-the-shelf vision-
language models as reward predictors. Leveraging existing vision-language
models offers a number of benefits compared to utilizing a pre-trained visual
representation or training a reward model from scratch using in-domain data:
First, VLMs are trained on an Internet-scale dataset that contains diverse
image and language contents. Such models possess better inductive biases and
thus, can be more robust to natural shifts, such as perturbations to lighting
conditions, or distractor objects that might be seen at test time. Second,
since VLMs can take both visual and language information as input, they
provide a natural interface for communicating the current observation and
current task to the model when requesting a reward label.

We design a VLM-based reward model that takes the current observation and the
task name as input and outputs a binary label of whether the current
observation corresponds to a successful state or an unsuccessful state with
respect to the task. Given a task name (eg. ‘put green cabbage into sink’), we
first use GPT4 to convert the name to a short question that could serve as a
prompt to know if the task has been completed or not (eg. ‘is green cabbage
placed in the sink?’). Then, we pass the converted prompt to a VLM together
with the current image of the environment. The VLM outputs a sparse binary
reward, returning success if the ‘yes’ token has a higher probability than
‘no’ token.

We use MiniGPT4 [53] as the VLM for receiving (image, task prompt) pairs and
answering whether the task has been successfully completed. We find the zero-
shot performance of the pre-trained VLM to be unsatisfactory. To improve the
VLM’s performance for reward modeling, we fine-tune it using the prior and
target task data. In particular, for every demonstration, the last 3 states
are used as success states and the ground truth answer is labeled as ‘Yes’;
for all other states, we label the ground-truth answer as ‘No’. To provide the
model with more information about failed states, we collect a small dataset
$\mathcal{D}_{\frownie{}}$ of images that correspond to unsuccessful states
for the forward and backward target tasks. We find in our experiments that
fine-tuning leads to a more accurate reward model.

###  IV-C Autonomous Online Fine-tuning

The offline pre-training phase produces a single language-conditioned policy
$\pi(\cdot|s,l)$ that can perform the target and reset tasks when provided
their respective language instructions $l_{f}$ and $l_{b}$. The policy is then
deployed in a hardware setup for further online fine-tuning. The outline of
our pre-training and fine-tuning pipeline is presented in Algorithm 1.

Since we aim for a fully autonomous setup, we roll out the policy in a reset-
free manner, alternating between attempting the target task $\tau_{f}$ with
$\pi(\cdot|s,l_{f})$ and the reset task $\tau_{b}$ with $\pi(\cdot|s,l_{b})$.
We use the fine-tuned VLM from the previous subsection as the sparse reward
function for the RL algorithm. When the VLM predicts the task has been
completed successfully, we terminate the episode and switch the language
instruction for the policy to complete the other task. In addition to
switching tasks upon completion as predicted by the VLM, we switch after a
fixed number of timesteps (150) to ensure the robot does not become stuck in
bad states. As mentioned in Section IV-A, we fine-tune the policy using CalQL
with an additional BC regularization term on the critic. We find that without
a BC regularization term, behaviors degrade over the course of training. By
constraining the policy to stay close to the expert demonstrations from the
target and reset tasks, the agent becomes less likely to exploit false-
positives from the VLM reward model. We use the same fixed BC regularization
weight throughout fine-tuning as we did on the offline pre-training phase. Our
fine-tuning pipeline is implemented on top of the implementation of MEDAL++
[13]. Please refer to this work for more details on our training procedure.

Initialize agent $\mathcal{A}=\\{\phi,\pi,Q\\}$ and pre-trained VLM $\hat{r}$.

Initialize forward and backward tasks $\tau_{f},\tau_{b}$.

// Prepare data and train VLM reward classifier.

$\mathcal{D_{\text{prior}}},\mathcal{D}_{f},\mathcal{D}_{b},\mathcal{D}_{\frownie{}}\leftarrow$
load_data().

$\hat{r}\leftarrow$
finetune_vlm($\hat{r},\\{\mathcal{D_{\text{prior}}},\mathcal{D}_{f},\mathcal{D}_{b},\mathcal{D}_{\frownie{}}\\}$).

// Offline pre-training phase.

$\mathcal{A}$.update_buffer($\mathcal{D}_{\text{prior}},\mathcal{D}_{f},\mathcal{D}_{b}$).

for  _$t=1$ to $T_{\text{offline}}$_ do

$\mathcal{A}$.update_params_with_calql().

// Online fine-tuning phase.

$s\leftarrow$ env.reset(); $l\leftarrow\tau_{f}.$get_task_lang().

for  _$t=1$ to $T_{\text{online}}$_ do

$a\leftarrow\mathcal{A}$.act($s,l$); $s^{\prime}\leftarrow$ env.step($a$).

$\mathcal{A}$.update_buffer({$s,a,s^{\prime},\hat{r}(s)$}).

for  _$i=1$ to $N_{\text{utd\\_ratio}}$_ do

$\mathcal{A}$.update_params_with_calql().

if  _switch_ then

// Switch task after a fixed interval.

$l\leftarrow$ env.switch($\tau_{f},\tau_{b}$).get_task_lang().

if  _interrupt_ then

// Allow occasional human intervention.

$s\leftarrow$ env.reset(); $l\leftarrow\tau_{f}.$get_task_lang().

else

$s\leftarrow s^{\prime}$.

Algorithm 1 RoboFuME

##  V Experiments

We design our experiments to answer the following questions: Is our method
able to improve its performance through near autonomous online interactions?
How does our proposed VLM reward function mechanism compare to existing
alternatives? And, how does each component of RoboFuME or data affect the
performance of our method?

###  V-A Real Robot Experiments

Figure 2: Illustrations of the five real-world evaluation tasks. (a) Sweep
candies to the top of the tray. (b) fold the yellow cloth. (c) cover a red
wooden cube using the cloth. (d) place the lid on top of the metallic pot. (e)
move the orange pot from the sink to the drying rack.

Setup. We evaluate RoboFuME on five different real-robot manipulation tasks.
We use a WidowX 250 robotic arm with a single third-person camera (Logitech
C920, resizing images to 100x100 pixels). Figure 2 shows the five tasks we
fine-tune and evaluate on. Our method runs autonomously executing back and
forth the target task and the reset target task for a fixed number of steps or
until the VLM predicts success. For tasks involving deformable objects (the
two cloth tasks) we manually reset the object to the initial forward pose
every 15-25 episodes, and for the rest of the tasks we reset every 30-35
episodes. Tasks that use the kitchen-sink environment (pot lid and pot pnp)
frequently experience episode interruptions when the robot arm applies more
than the maximum allowed torque, for example, when close to the sink borders.
All tasks use 50 forward and 50 backward demos for the target task, and fewer
than 20 combined trajectories of failures. We use demos from the BridgeDataV2
[19, 54] for pre-training our language-conditioned policy, selecting
approximately 1,000 trajectories with relevant behaviors per task.

Results. Table I shows the results of our method after pretraining (labeled
offline) and after autonomous fine-tuning (labeled FT 30k steps), comparing
with a behavior cloning (BC) baseline. BC trains a language-conditioned policy
on all the prior and target data. After 30k steps of autonomous online
interaction, our method shows relative improvement of 51% upon the pre-trained
performance, and outperforms BC by 58% on an average. For pick and place tasks
(pot lid and pot pnp), the fine-tuned policy was more likely to retry the
action if it initially failed to grasp the object. For candy sweeping, BC and
the pre-trained policy were prone to overshooting and pushing on the border of
the tray after the first sweep, whereas fine-tuning the policy enabled the
policy to chain multiple sweeping attempts for higher success. Additionally,
we find that policies learned by RoboFuME (both offline and after fine-tuning)
to be more robust to scene distractors on the candy sweeping task, as reported
in Table III. The policies were trained without any distractors, but multiple
objects not seen during training were placed in the background during
evaluation. RoboFuME policies retained 68% of its original performance,
compared to BC which retained only 10% of its original performance. We
hypothesize that BC might be more sensitive to spurious features, whereas
RoboFuME learns from more predictive features, leading to more robust
policies.

Figure 3: Performance of our method on three simulated environments. We report the success rate over the course of training, averaged over three seeds. Our method RoboFuME outperforms BC, ARIEL+VLM [7], and MEDAL++ [13] consistently on all three domains. Task | BC |  | RoboFuME
---
(offline)
| RoboFuME
---
(FT 30k steps)
Cloth Covering | 45% | 60% | 80%
Cloth Folding | 60% | 70% | 85%
Candy Sweeping | 31% | 47% | 66%
Pot Lid | 60% | 40% | 95%
Pot PNP | 45% | 35% | 55%
TABLE I: Real-robot results on 5 manipulation tasks. Our method significantly
improves over both offline-only and BC performance after 30k steps of online
interaction (2-4 hours). For the Candy sweeping we report the average
percentage of candies out of a total of 7 that are placed in the top third of
the tray by the end of the evaluation. For all other tasks, we report success
rate over 20 trials.

###  V-B Simulation Experiments and Ablations

Figure 4: Performance of our method on the Vase simulated task with different
actor-critic update objectives. Fine-tuning with CalQL is critical to obtain
stable improvements on this task, as training with CQL, AWAC, or SAC yields
poor performance. We also find that language conditioned policies perform
slightly better than one-hot task IDs in simulation.

We use a suite of simulated robotic manipulation environments to ablate
contributions of different components of our algorithm. We test on three
simulated environments used in [6]. We consider three bin-sorting tasks in
which different objects (a vase, a tiny bench, and a dumbbell weight) have to
be placed on the correct bin based on the language instruction, given only a
sparse binary reward. We provide 10 forward and reset demonstrations for each
task, 30 failure demos, and 10 demos each for 20 prior tasks that show picking
and placing diverse objects on the same environment. For all methods that
require online experience, we reset the environment every 1,000 environment
steps, i.e. every 25 episodes of interactions. We compare our method against
the following baselines: (1) BC behavior clones on all prior and target data;
(2) MEDAL++ learns separate forward and backward policies from target forward
and backward task demonstrations and performs reset-free learning using an
adversarially trained classifier as a reward signal; (3) MEDAL++ with prior
data modifies MEDAL++ to a single language-conditioned multi-task policy and
adds all prior demonstration data into the replay buffer; (4) ARIEL+VLM
modifies ARIEL [7] to use our VLM reward models as reward signal, instead of a
handcrafted ground-truth reward. The results of our simulation experiments are
presented in Figure 3. In all simulation tasks, our method RoboFuME
consistently outperforms prior methods, achieving success rates at least 20%
higher than all baselines within 200k steps of online fine-tuning.

Ablations on RL Algorithm Design Choices. We evaluate our method trained with
different critic and actor optimization procedures on the Vase simulated task,
shown in Figure 4. Training with CalQL was the only method that yielded strong
improvements in this task, with the other methods either failing completely or
obtaining very poor performance. We find that training without the CalQL
stabilizes training, while the losses for other methods would explode given
the limited data.

Figure 5: Performance of our method on the simulated Vase task using different
reward functions. Our method uses a fine-tuned VLM reward function and
outperforms VICE rewards, whereas CNN and VIP rewards fail to improve online.

Ablations on Reward Models. We compare our VLM reward function against other
choices of automatic reward functions on the Vase simulated task in Figure 5.
VICE [47] adversarially trains a binary classifier using positive samples from
successful demonstrations, and labeling online experience as negative. We find
that offline pre-training sufficiently limits the exploitation of the frozen
VLM reward, outperforming VICE and thus, bypassing the need for adversarially
trained reward functions. Such adversarial training can often learn to
discriminate based on spurious shifts in the real world, such as lighting or
scene changes, leading to instability in training outside simulation. VIP [16]
trains a representation function such that the distance in representation
space between the current observation and a goal image can be used to
construct a dense reward function. We find that in the Vase simulated task,
VIP fails to obtain good behaviors. Qualitatively, we observe VIP to be prone
to false positives, which are exploited by the RL algorithm. To test the
importance of the VLM large-scale pre-training compared to our fine-tuning
procedure, we train a CNN classifier from scratch using the same data as we
used to fine-tune the VLM, leading to unsatisfactory performance compared to
fine-tuning a VLM.

How Accurate is the VLM Reward? We analyze the performance of the VLM reward
over the course of fine-tuning for real-robot experiments. In Table II, we
report the false positive rate, false negative rate, accuracy, and precision
metrics for the VLM reward. The metrics are computed on the data collected
during fine-tuning against a hand-engineered ground truth reward. We observe
that while false negative rates are high, false positive rates are low across
all tasks. This asymmetry is crucial for successful RL fine-tuning, as RL
policies can learn poor behaviors by exploiting false positives, but labeling
some successful rollouts as negatives does not necessarily impede learning.

How Important is Diverse Prior Data and Language Conditioning? We ablate the
contribution of diverse prior data and language-conditioned policies to
RoboFuME by evaluating the offline performance on the candy sweeping task,
reported in Table IV. When pre-training without using prior data, that is,
exclusively using target data, our method is able to sweep less than half the
amount of candies on average. Similarly, we find that one-hot task encodings
perform substantially worse than language-conditioned policies, as the prior
dataset used in real-robot training is larger and more diverse compared to the
simulation experiments.

Task | FP | FN | Accuracy | Precision
---|---|---|---|---
Cloth Covering | 6.3% | 80.9% | 89.4% | 15.3%
Cloth Folding | 1.2% | 59.8% | 84.1% | 92.0%
Pot PNP | 6.1% | 81.3% | 86.9% | 24.3%
TABLE II: VLM reward model accuracy during real robot fine-tuning. The low false positive (FP) rate indicates that online training has minimal reward exploitation. Task | BC |  | RoboFuME
---
(offline)
| RoboFuME
---
(fine-tuned @30k)
Candy Sweeping | 31% $\to$ 3% | 47% $\to$ 31% | 66% $\to$ 45%
TABLE III: Robustness of learned policy to distractors. Entries in this table
show the performance of the learned policy “before” $\rightarrow$ “after”
adding distractors to the scene in the candy-sweeping task. Our system learns
a policy that is much more robust to the distractors.

Task RoboFuME (offline) RoboFuME w/o Prior Data RoboFuME w/o Language Cond.
Candy Sweeping 47% 23% 13%

TABLE IV: Evaluating effectiveness of prior data and language conditioned
policies. Results show that using prior data and using language conditioning
positively affected the offline performance of our system.

##  VI Conclusion and Future Work

We introduced an autonomous framework that leverages existing diverse prior
robot demonstration datasets and improves performance in a new robot
manipulation skill by finetuning online. By combining state-of-the-art
offline-to-online RL algorithms, reset-free RL, and VLM-based reward models,
our framework can fine-tune efficiently and nearly autonomously. Integrating
this work with new VLM models that can exhibit robust zero-shot performance on
unseen manipulation tasks and improving the reset efficiency of this framework
are promising directions for future research.

## References

  * [1] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, _et al._ , “Improving language understanding by generative pre-training,” 2018.
  * [2] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [3] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, _et al._ , “Language models are unsupervised multitask learners,” _OpenAI blog_ , vol. 1, no. 8, p. 9, 2019.
  * [4] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, _et al._ , “Language models are few-shot learners,” _Advances in neural information processing systems_ , vol. 33, pp. 1877–1901, 2020.
  * [5] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 16 000–16 009.
  * [6] A. Kumar, A. Singh, F. Ebert, Y. Yang, C. Finn, and S. Levine, “Pre-training for robots: Offline rl enables learning new tasks from a handful of trials,” _arXiv preprint arXiv:2210.05178_ , 2022.
  * [7] H. R. Walke, J. H. Yang, A. Yu, A. Kumar, J. Orbik, A. Singh, and S. Levine, “Don’t start from scratch: Leveraging prior data to automate robotic reinforcement learning,” in _Conference on Robot Learning_. PMLR, 2023, pp. 1652–1662.
  * [8] V. Kumar, E. Todorov, and S. Levine, “Optimal control with learned local models: Application to dexterous manipulation,” in _2016 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2016, pp. 378–383.
  * [9] A. Ghadirzadeh, A. Maki, D. Kragic, and M. Björkman, “Deep predictive policy training using reinforcement learning,” in _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2017, pp. 2351–2358.
  * [10] K. Ploeger, M. Lutter, and J. Peters, “High acceleration reinforcement learning for real-world juggling with binary rewards,” in _Conference on Robot Learning_. PMLR, 2021, pp. 642–653.
  * [11] A. Gupta, J. Yu, T. Z. Zhao, V. Kumar, A. Rovinsky, K. Xu, T. Devlin, and S. Levine, “Reset-free reinforcement learning via multi-task learning: Learning dexterous manipulation behaviors without human intervention,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021, pp. 6664–6671.
  * [12] C. Sun, J. Orbik, C. M. Devin, B. H. Yang, A. Gupta, G. Berseth, and S. Levine, “Fully autonomous real-world reinforcement learning with applications to mobile manipulation,” in _Conference on Robot Learning_. PMLR, 2022, pp. 308–319.
  * [13] A. Sharma, A. M. Ahmed, R. Ahmad, and C. Finn, “Self-improving robots: End-to-end autonomous visuomotor reinforcement learning,” _arXiv preprint arXiv:2303.01488_ , 2023.
  * [14] I. Kostrikov, A. Nair, and S. Levine, “Offline reinforcement learning with implicit q-learning,” _arXiv preprint arXiv:2110.06169_ , 2021.
  * [15] M. Nakamoto, Y. Zhai, A. Singh, M. S. Mark, Y. Ma, C. Finn, A. Kumar, and S. Levine, “Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning,” _arXiv preprint arXiv:2303.05479_ , 2023.
  * [16] Y. J. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V. Kumar, and A. Zhang, “Vip: Towards universal visual reward and representation via value-implicit pre-training,” _arXiv preprint arXiv:2210.00030_ , 2022.
  * [17] A. Sharma, R. Ahmad, and C. Finn, “A state-distribution matching approach to non-episodic reinforcement learning,” _arXiv preprint arXiv:2205.05212_ , 2022.
  * [18] Y. J. Ma, W. Liang, V. Som, V. Kumar, A. Zhang, O. Bastani, and D. Jayaraman, “Liv: Language-image representations and rewards for robotic control,” _arXiv preprint arXiv:2306.00958_ , 2023.
  * [19] F. Ebert, Y. Yang, K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and S. Levine, “Bridge data: Boosting generalization of robotic skills with cross-domain datasets,” _arXiv preprint arXiv:2109.13396_ , 2021\.
  * [20] Y. Wu, G. Tucker, and O. Nachum, “Behavior regularized offline reinforcement learning,” _arXiv preprint arXiv:1911.11361_ , 2019.
  * [21] X. B. Peng, A. Kumar, G. Zhang, and S. Levine, “Advantage-weighted regression: Simple and scalable off-policy reinforcement learning,” _arXiv preprint arXiv:1910.00177_ , 2019.
  * [22] S. Levine, A. Kumar, G. Tucker, and J. Fu, “Offline reinforcement learning: Tutorial, review, and perspectives on open problems,” _arXiv preprint arXiv:2005.01643_ , 2020.
  * [23] T. Yu, G. Thomas, L. Yu, S. Ermon, J. Y. Zou, S. Levine, C. Finn, and T. Ma, “Mopo: Model-based offline policy optimization,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 14 129–14 142, 2020.
  * [24] A. Kumar, A. Zhou, G. Tucker, and S. Levine, “Conservative q-learning for offline reinforcement learning,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 1179–1191, 2020.
  * [25] W. Zhou, S. Bajracharya, and D. Held, “Plas: Latent action space for offline reinforcement learning,” in _Conference on Robot Learning_. PMLR, 2021, pp. 1719–1735.
  * [26] N. Ashvin, D. Murtaza, G. Abhishek, and L. Sergey, “Accelerating online reinforcement learning with offline datasets,” _CoRR, vol. abs/2006.09359_ , 2020.
  * [27] R. Rafailov, T. Yu, A. Rajeswaran, and C. Finn, “Offline reinforcement learning from images with latent space models,” in _Learning for Dynamics and Control_. PMLR, 2021, pp. 1154–1168.
  * [28] J. Lyu, X. Ma, X. Li, and Z. Lu, “Mildly conservative q-learning for offline reinforcement learning,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 1711–1724, 2022.
  * [29] A. Beeson and G. Montana, “Improving td3-bc: Relaxed policy constraint for offline learning and stable online fine-tuning,” _arXiv preprint arXiv:2211.11802_ , 2022.
  * [30] J. Wu, H. Wu, Z. Qiu, J. Wang, and M. Long, “Supported policy optimization for offline reinforcement learning,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 31 278–31 291, 2022.
  * [31] S. Lee, Y. Seo, K. Lee, P. Abbeel, and J. Shin, “Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble,” in _Conference on Robot Learning_. PMLR, 2022, pp. 1702–1712.
  * [32] M. S. Mark, A. Ghadirzadeh, X. Chen, and C. Finn, “Fine-tuning offline policies with optimistic action selection,” in _Deep Reinforcement Learning Workshop NeurIPS 2022_ , 2022.
  * [33] K. Lu, A. Grover, P. Abbeel, and I. Mordatch, “Reset-free lifelong learning with skill-space planning,” _arXiv preprint arXiv:2012.03548_ , 2020.
  * [34] S. Ha, P. Xu, Z. Tan, S. Levine, and J. Tan, “Learning to walk in the real world with minimal human effort,” _arXiv preprint arXiv:2002.08550_ , 2020\.
  * [35] A. Gupta, C. Lynch, B. Kinman, G. Peake, S. Levine, and K. Hausman, “Demonstration-bootstrapped autonomous practicing via multi-task reinforcement learning,” _arXiv preprint arXiv:2203.15755_ , vol. 1, 2022\.
  * [36] K. Xu, Z. Hu, R. Doshi, A. Rovinsky, V. Kumar, A. Gupta, and S. Levine, “Dexterous manipulation from images: Autonomous real-world rl via substep guidance,” in _2023 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2023, pp. 5938–5945.
  * [37] B. Eysenbach, S. Gu, J. Ibarz, and S. Levine, “Leave no trace: Learning to reset for safe and autonomous reinforcement learning,” _arXiv preprint arXiv:1711.06782_ , 2017.
  * [38] H. Zhu, J. Yu, A. Gupta, D. Shah, K. Hartikainen, A. Singh, V. Kumar, and S. Levine, “The ingredients of real-world robotic reinforcement learning,” _arXiv preprint arXiv:2004.12570_ , 2020.
  * [39] A. Sharma, A. Gupta, S. Levine, K. Hausman, and C. Finn, “Autonomous reinforcement learning via subgoal curricula,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 18 474–18 486, 2021.
  * [40] A. Y. Ng, S. Russell, _et al._ , “Algorithms for inverse reinforcement learning.” in _Icml_ , vol. 1, 2000, p. 2.
  * [41] P. Abbeel and A. Y. Ng, “Apprenticeship learning via inverse reinforcement learning,” in _Proceedings of the twenty-first international conference on Machine learning_ , 2004, p. 1.
  * [42] B. D. Ziebart, A. L. Maas, J. A. Bagnell, A. K. Dey, _et al._ , “Maximum entropy inverse reinforcement learning.” in _Aaai_ , vol. 8. Chicago, IL, USA, 2008, pp. 1433–1438.
  * [43] J. Ho, J. Gupta, and S. Ermon, “Model-free imitation learning with policy optimization,” in _International conference on machine learning_. PMLR, 2016, pp. 2760–2769.
  * [44] J. Ho and S. Ermon, “Generative adversarial imitation learning,” _Advances in neural information processing systems_ , vol. 29, 2016.
  * [45] C. Finn, S. Levine, and P. Abbeel, “Guided cost learning: Deep inverse optimal control via policy optimization,” in _International conference on machine learning_. PMLR, 2016, pp. 49–58.
  * [46] J. Fu, K. Luo, and S. Levine, “Learning robust rewards with adversarial inverse reinforcement learning,” _arXiv preprint arXiv:1710.11248_ , 2017\.
  * [47] J. Fu, A. Singh, D. Ghosh, L. Yang, and S. Levine, “Variational inverse control with events: A general framework for data-driven reward definition,” in _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , ser. NIPS’18. Red Hook, NY, USA: Curran Associates Inc., 2018, p. 8547–8556.
  * [48] A. Singh, L. Yang, K. Hartikainen, C. Finn, and S. Levine, “End-to-end robotic reinforcement learning without reward engineering,” _arXiv preprint arXiv:1904.07854_ , 2019.
  * [49] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta, “R3m: A universal visual representation for robot manipulation,” _arXiv preprint arXiv:2203.12601_ , 2022.
  * [50] P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, S. Levine, and G. Brain, “Time-contrastive networks: Self-supervised learning from video,” in _2018 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2018, pp. 1134–1141.
  * [51] W. Yu, N. Gileadi, C. Fu, S. Kirmani, K.-H. Lee, M. G. Arenas, H.-T. L. Chiang, T. Erez, L. Hasenclever, J. Humplik, _et al._ , “Language to rewards for robotic skill synthesis,” _arXiv preprint arXiv:2306.08647_ , 2023.
  * [52] Y. Du, K. Konyushkova, M. Denil, A. Raju, J. Landon, F. Hill, N. de Freitas, and S. Cabi, “Vision-language models as success detectors,” _arXiv preprint arXiv:2303.07280_ , 2023.
  * [53] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “Minigpt-4: Enhancing vision-language understanding with advanced large language models,” _arXiv preprint arXiv:2304.10592_ , 2023.
  * [54] H. Walke, K. Black, A. Lee, M. J. Kim, M. Du, C. Zheng, T. Zhao, P. Hansen-Estruch, Q. Vuong, A. He, _et al._ , “Bridgedata v2: A dataset for robot learning at scale,” _arXiv preprint arXiv:2308.12952_ , 2023.