# Examples of cyclic polynomially bounded operators that are not similar to
contractions, II

Maria F. Gamal  St. Petersburg Branch
V. A. Steklov Institute of Mathematics
Russian Academy of Sciences
Fontanka 27, St. Petersburg
191023, Russia<EMAIL_ADDRESS>
###### Abstract.

The question if polynomially bounded operator is similar to a contraction was
posed by Halmos and was answered in the negative by Pisier. His counterexample
is an operator of infinite multiplicity, while all its restrictions on
invariant subspaces of finite multiplicity are similar to contractions. In
[G16], cyclic polynomially bounded operators which are not similar to
contractions was constructed. The construction was based on a perturbation of
the sequence of finite dimensional operators which is uniformly polynomially
bounded, but is not uniformly completely polynomially bounded, constructed by
Pisier. In this paper, a cyclic polynomially bounded operator $T_{0}$ such
that $T_{0}$ is not similar to a contraction and
$\omega_{a}(T_{0})=\mathbb{O}$, is constructed. Here
$\omega_{a}(z)=\exp(a\frac{z+1}{z-1})$, $z\in\mathbb{D}$, $a>0$, and
$\mathbb{D}$ is the open unit disk. To obtain such $T_{0}$, a slight
modification of the construction from [G16] is needed.

###### Key words and phrases:

Polynomially bounded operator, similarity, contraction, unilateral shift,
isometry, $C_{0}$-contraction, $C_{0}$-operator

###### 2010 Mathematics Subject Classification:

Primary 47A60; Secondary 47A65, 47A16, 47A20.

##  1\. Introduction

Let $T$ be an operator on a Hilbert space. $T$ is called polynomially bounded,
if there exists a constant $C$ such that

$\|p(T)\|\leq C\|p\|_{\infty}\text{ \ for every polynomial }p,$

where $\|\cdot\|_{\infty}$ denotes the uniform norm on the open unit disk
$\mathbb{D}$.

An operator $T$ is called a contraction if $\|T\|\leq 1$. Every contraction is
polynomially bounded (von Neumann inequality, see, for example, [SFBK,
Proposition I.8.3]).

The question if a polynomially bounded operator is similar to a contraction
was posed by Halmos [H] and was answered in the negative by Pisier [Pi]. In
particular, the sequence of finite dimensional operators which is uniformly
polynomially bounded, but is not uniformly completely polynomially bounded was
constructed in [Pi] (see also [DP]; see [Pa], [Pi], or [DP] for the definition
of completely polynomially boundedness).

In [G16, Theorem 6.3], a perturbation of the sequence described above was
constructed. The obtained perturbed sequence was used to construct examples of
cyclic polynomially bounded operators which are not similar to contractions
([G16, Theorem 7.1 and Sec. 2]). In this paper, a slight modification of [G16,
Theorem 7.1] is given which allows to construct a cyclic absolutely continuous
polynomially bounded operator $T_{0}$ such that $T_{0}$ is not similar to a
contraction and $\omega_{a}(T_{0})=\mathbb{O}$, where $a>0$ and
$\omega_{a}(z)=\exp(a\frac{z+1}{z-1})$ ($z\in\mathbb{D}$) (Theorem 3.7).

We need the following notations and definitions.

Let $\mathcal{H}$ be a (complex, separable) Hilbert space, and let
$\mathcal{M}$ be its (linear, closed) subspace. By $I_{\mathcal{H}}$ and
$P_{\mathcal{M}}$ the identical operator on $\mathcal{H}$ and the orthogonal
projection from $\mathcal{H}$ onto $\mathcal{M}$ are denoted, respectively.

For an operator $T\colon\mathcal{H}\to\mathcal{H}$, a subspace $\mathcal{M}$
of $\mathcal{H}$ is called _invariant subspace_ of $T$, if
$T\mathcal{M}\subset\mathcal{M}$. The complete lattice of all invariant
subspaces of $T$ is denoted by $\operatorname{Lat}T$.

The multiplicity $\mu_{T}$ of an operator $T\colon\mathcal{H}\to\mathcal{H}$
is the minimum dimension of its reproducing subspaces:

$\mu_{T}=\min\\{\dim E:E\subset\mathcal{H},\ \
\bigvee_{n=0}^{\infty}T^{n}E=\mathcal{H}\\}.$

An operator $T$ is called cyclic, if $\mu_{T}=1$.

It is well known and easy to see that if $\mathcal{M}\in\operatorname{Lat}T$,
then

(1.1)
$\mu_{P_{\mathcal{M}^{\perp}}T|_{\mathcal{M}^{\perp}}}\leq\mu_{T}\leq\mu_{T|_{\mathcal{M}}}+\mu_{P_{\mathcal{M}^{\perp}}T|_{\mathcal{M}^{\perp}}}$

(see, for example, [N, II.D.2.3.1]).

Let $T$ and $R$ be operators on Hilbert spaces $\mathcal{H}$ and
$\mathcal{K}$, respectively, and let $X:\mathcal{H}\to\mathcal{K}$ be a
(linear, bounded) operator such that $X$ intertwines $T$ and $R$, that is,
$XT=RX$. If $X$ is unitary, then $T$ and $R$ are called unitarily equivalent,
in notation: $T\cong R$. If $X$ is invertible, that is, $X$ has the bounded
inverse $X^{-1}$, then $T$ and $R$ are called similar, in notation: $T\approx
R$. If $X$ is a quasiaffinity, that is, $\ker X=\\{0\\}$ and
$\operatorname{clos}X\mathcal{H}=\mathcal{K}$, then $T$ is called a
quasiaffine transform of $R$, in notation: $T\prec R$. If $T\prec R$ and
$R\prec T$, then $T$ and $R$ are called quasisimilar, in notation: $T\sim R$.
If $\operatorname{clos}X\mathcal{H}=\mathcal{K}$, we write $T\buildrel
d\over{\prec}R$. It follows immediately from the definition of the relation
$\buildrel d\over{\prec}$ that if $T\buildrel d\over{\prec}R$, then
$\mu_{R}\leq\mu_{T}$.

If $T$, $R$, and $X$ are operators such that $XT=RX$, and
$\mathcal{M}\in\operatorname{Lat}T$, then
$\operatorname{clos}X\mathcal{M}\in\operatorname{Lat}R$. The mapping

(1.2) $\mathcal{J}_{X}\colon\operatorname{Lat}T\to\operatorname{Lat}R,\ \
\mathcal{J}_{X}\mathcal{M}=\operatorname{clos}X\mathcal{M},\ \ \
\mathcal{M}\in\operatorname{Lat}T,$

is a lattice isomorphism if and only if it is a bijection, see [B, VII.1.19].

Symbols $\mathbb{D}$ and $\mathbb{T}$ denote the open unit disc and the unit
circle, respectively. The normalized Lebesgue measure on $\mathbb{T}$ is
denoted by $m$. The symbol $H^{\infty}$ denotes the Banach algebra of all
bounded analytic functions in $\mathbb{D}$. The uniform norm on $\mathbb{D}$
is denoted by the symbol $\|\cdot\|_{\infty}$. The symbol $H^{2}$ denotes the
Hardy space on $\mathbb{D}$, $L^{2}=L^{2}(\mathbb{T},m)$,
$H^{2}_{-}=L^{2}\ominus H^{2}$. The symbol $P_{-}$ denotes the orthogonal
projection from $L^{2}$ onto $H^{2}_{-}$. The simple unilateral and bilateral
shifts $S$ and $U_{\mathbb{T}}$ are the operators of multiplication by the
independent variable on $H^{2}$ and on $L^{2}$, respectively. It is well known
that $\mu_{S}=1$ and $\mu_{U_{\mathbb{T}}}=1$. For an inner function
$\theta\in H^{\infty}$ the subspace $\theta H^{2}$ is invariant for $S$, put

(1.3) $\mathcal{K}_{\theta}=H^{2}\ominus\theta H^{2}\ \ \text{ and }\ \
T_{\theta}=P_{\mathcal{K}_{\theta}}S|_{\mathcal{K}_{\theta}}.$

By (1.1), $\mu_{T_{\theta}}=1$.

If $T$ is a polynomially bounded operator, then $T=T_{a}\dotplus T_{s}$, where
$T_{a}$ is an _absolutely continuous (a.c.)_ polynomially bounded operator,
that is, $H^{\infty}$-functional calculus is well defined for $T_{a}$, and
$T_{s}$ is similar to a singular unitary operator, see [M] or [K16]. (Although
many results on polynomially bounded operators that will be used in the
present paper were originally proved by Mlak, we will refer to [K16] for the
convenience of references.) In the paper, absolutely continuous polynomially
bounded operators are considered. Clearly, $S$ and $U_{\mathbb{T}}$ are
contraction, and, consequently, polynomially bounded. It is well known that
$S$ and $U_{\mathbb{T}}$ are a.c.. An a.c. polynomially bounded operator $T$
is called a $C_{0}$-operator, if there exists $\varphi\in H^{\infty}$ such
that $\varphi(T)=\mathbb{O}$, see [BP]; if a $C_{0}$-operator is a
contraction, it is called a $C_{0}$-contraction, see [SFBK] and [B]. For an
inner function $\theta$, $T_{\theta}$ is a $C_{0}$-contraction, because
$\theta(T_{\theta})=\mathbb{O}$.

The paper is organized as follows. In Sec. 2 polynomially bounded operators
that are quasiaffine transforms of the unilateral shift $S$ are considered. In
many results is assumed that the operator $T$ under consideration has an
invariant subspace $\mathcal{M}$ such that $T|_{\mathcal{M}}\approx S$. If
$T\prec S$ and $T$ is a contraction, then $T$ always has such invariant
subspace (a particular case of [K07], see also [SFBK, Theorem IX.3.5]). The
author does not know if this result can be generalized on polynomially bounded
operators. In Sec. 3 Theorem 2.4 is applied to construct the polynomially
bounded operator $T_{0}$ such that $T_{0}$ is not similar to a contraction and
$\omega_{a}(T_{0})=\mathbb{O}$, where $a>0$ and
$\omega_{a}(z)=\exp(a\frac{z+1}{z-1})$ ($z\in\mathbb{D}$). (All subsequent
results from Sec. 2 are not used in Sec. 3.)

##  2\. On operators that are quasiaffine transforms of a unilateral shift

An operator $T$ is called _power bounded_ , if $\sup_{n\geq
0}\|T^{n}\|<\infty$. For a power bounded operator $T$ the _isometric
asymptote_ $(Y,V)$ is defined in [K89]. Here $V$ is an isometry, and $Y$ is an
intertwining mapping: $YT=VY$. When it will be convenient, we will called by
the isometric asymptote of $T$ the isometry $V$ itself, while $Y$ will be
called the _canonical intertwining mapping_ (sf. [K89] and [K16]).

Let $\mathcal{M}\in\operatorname{Lat}T$. It follows immediately from the
construction of the isometric asymptote and [K89, Theorem 1] that
$T|_{\mathcal{M}}$ is similar to an isometry if and only if $Y|_{\mathcal{M}}$
is left invertible, i.e., there exists $c>0$ such that $\|Yx\|\geq c\|x\|$ for
every $x\in\mathcal{M}$. The _unitary asymptote_ for a power bounded operator
$T$ is the minimal unitary extension of its isometric asymptote, see [K89].

Clearly, a polynomially bounded operator is power bounded. The unitary
asymptotes for polynomially bounded operators are considered in [K16].

In the following proposition, the idea from [F] is used. Since this
proposition can be easily checked directly, its proof is omitted.

######  Proposition 2.1.

Suppose that $T\colon\mathcal{H}\to\mathcal{H}$,
$V\colon\mathcal{K}\to\mathcal{K}$, and $Y\colon\mathcal{H}\to\mathcal{K}$ are
operators, and $YT=VY$. Furthermore, suppose that
$\mathcal{M}\in\operatorname{Lat}T$ is such that $Y|_{\mathcal{M}}$ is left
invertible. Put $\mathcal{H}_{0}=\mathcal{H}\ominus\mathcal{M}$,
$T_{0}=P_{\mathcal{H}_{0}}T|_{\mathcal{H}_{0}}$,
$\mathcal{K}_{1}=Y\mathcal{M}$,
$\mathcal{K}_{0}=\mathcal{K}\ominus\mathcal{K}_{1}$,
$Y_{2}=P_{\mathcal{K}_{1}}Y|_{\mathcal{H}_{0}}$,
$Y_{0}=P_{\mathcal{K}_{0}}Y|_{\mathcal{H}_{0}}$, $V_{1}=V|_{\mathcal{K}_{1}}$,
$V_{2}=P_{\mathcal{K}_{1}}V|_{\mathcal{K}_{0}}$,
$V_{0}=P_{\mathcal{K}_{0}}V|_{\mathcal{K}_{0}}$,

$Z\colon\mathcal{M}\oplus\mathcal{H}_{0}\to\mathcal{K}_{1}\oplus\mathcal{H}_{0},\
\ \ \
Z=\begin{pmatrix}Y|_{\mathcal{M}}&Y_{2}\cr\mathbb{O}&I_{\mathcal{H}_{0}}\end{pmatrix},$

and

$R\colon\mathcal{K}_{1}\oplus\mathcal{H}_{0}\to\mathcal{K}_{1}\oplus\mathcal{H}_{0},\
\ \ \ R=\begin{pmatrix}V_{1}&V_{2}Y_{0}\cr\mathbb{O}&T_{0}\end{pmatrix}.$

Then $Y_{0}T_{0}=V_{0}Y_{0}$, $Z$ is invertible, $R=ZTZ^{-1}$, and

$(I_{\mathcal{K}_{1}}\oplus Y_{0})R=V(I_{\mathcal{K}_{1}}\oplus Y_{0}).$

Moreover, if $\operatorname{clos}Y\mathcal{H}=\mathcal{K}$, then
$\operatorname{clos}Y_{0}\mathcal{H}_{0}=\mathcal{K}_{0}$, and if $\ker
Y=\\{0\\}$, then $\ker Y_{0}=\\{0\\}$.

######  Lemma 2.2.

Suppose that $C>0$, $g$, $\theta$, $\omega\in H^{\infty}$, $\theta$ is inner,
and

(2.1) $\|P_{-}\overline{\theta}\omega u\|\leq C\|gu\|\ \ \text{ for every }\
u\in H^{2}.$

Furthermore, suppose that $\mathcal{H}_{0}$ is a Hilbert space, and
$X_{0}\colon\mathcal{K}_{\theta}\to\mathcal{H}_{0}$ is an operator. Put

(2.2) $X\colon H^{2}\to\theta H^{2}\oplus\mathcal{H}_{0},\ \ \ \
X=\begin{pmatrix}g(S)|_{\theta H^{2}}&P_{\theta
H^{2}}g(S)|_{\mathcal{K}_{\theta}}\cr\mathbb{O}&X_{0}\end{pmatrix}.$

Then there exists $C_{1}>0$ (which depends on $C$, $\|g\|_{\infty}$,
$\|\omega\|_{\infty}$, and $\|X_{0}\|$) such that

$\|X\omega u\|\leq C_{1}\|gu\|\ \ \ \text{ for every }u\in H^{2}.$

###### Proof.

Let $u\in H^{2}$. We have

$X\omega u=(gP_{\theta H^{2}}\omega u+P_{\theta
H^{2}}gP_{\mathcal{K}_{\theta}}\omega u)\oplus
X_{0}P_{\mathcal{K}_{\theta}}\omega u.$

Therefore,

$\|X\omega u\|^{2}=\|gP_{\theta H^{2}}\omega u+P_{\theta
H^{2}}gP_{\mathcal{K}_{\theta}}\omega
u\|^{2}+\|X_{0}P_{\mathcal{K}_{\theta}}\omega u\|^{2}.$

We have $gP_{\theta H^{2}}\omega u+P_{\theta
H^{2}}gP_{\mathcal{K}_{\theta}}\omega u=g\omega
u-P_{\mathcal{K}_{\theta}}gP_{\mathcal{K}_{\theta}}\omega u$,

$\|P_{\mathcal{K}_{\theta}}gP_{\mathcal{K}_{\theta}}\omega
u\|\leq\|g\|_{\infty}\|P_{\mathcal{K}_{\theta}}\omega u\|,\ \ \text{ and }\ \
\|P_{\mathcal{K}_{\theta}}\omega u\|=\|\theta P_{-}\overline{\theta}\omega
u\|=\|P_{-}\overline{\theta}\omega u\|.$

Therefore,

$\displaystyle\|gP_{\theta H^{2}}\omega u$ $\displaystyle+P_{\theta
H^{2}}gP_{\mathcal{K}_{\theta}}\omega u\|^{2}\leq(\|g\omega
u\|+\|P_{\mathcal{K}_{\theta}}gP_{\mathcal{K}_{\theta}}\omega u\|)^{2}$
$\displaystyle\leq 2(\|g\omega
u\|^{2}+\|P_{\mathcal{K}_{\theta}}gP_{\mathcal{K}_{\theta}}\omega u\|^{2})\leq
2(\|\omega\|_{\infty}^{2}\|gu\|^{2}+\|g\|_{\infty}^{2}\|P_{-}\overline{\theta}\omega
u\|^{2}).$

Thus,

$\displaystyle\|X\omega u\|^{2}$ $\displaystyle\leq
2(\|\omega\|_{\infty}^{2}\|gu\|^{2}+\|g\|_{\infty}^{2}\|P_{-}\overline{\theta}\omega
u\|^{2})+\|X_{0}\|^{2}\|P_{-}\overline{\theta}\omega u\|^{2}$
$\displaystyle=2\|\omega\|_{\infty}^{2}\|gu\|^{2}+(2\|g\|_{\infty}^{2}+\|X_{0}\|^{2})\|P_{-}\overline{\theta}\omega
u\|^{2}.$

It remains to apply (2.1). ∎

In the following lemma, only part (i) is used in the sequel.

######  Lemma 2.3.

Suppose that $g$, $\theta$, $\omega\in H^{\infty}$, and $\theta$ is inner.

  1. (i)

If there exist $\eta$, $h\in H^{\infty}$ such that

(2.3) $\omega=g\eta+\theta h,$

then there exists $C>0$ such that (2.1) is fulfilled.

  2. (ii)

If $g$ is outer and (2.1) is fulfilled with some $C>0$, then there exist
$\eta$, $h\in H^{\infty}$ such that (2.3) is fulfilled.

###### Proof.

(i) Let $u\in H^{2}$. We have $P_{-}\overline{\theta}\omega
u=P_{-}\overline{\theta}(g\eta+\theta h)u=P_{-}\overline{\theta}g\eta u$.
Therefore, $\|P_{-}\overline{\theta}\omega u\|\leq\|g\eta
u\|\leq\|\eta\|_{\infty}\|gu\|$.

(ii) For $\delta>0$ define an outer function $\psi_{\delta}$ by the formula

$|\psi_{\delta}(\zeta)|=\begin{cases}1/|g(\zeta)|,&\text{ if
}|g(\zeta)|\geq\delta,\\\ 1,&\text{ if }|g(\zeta)|<\delta.\end{cases}$

Since $g$ is outer,

(2.4) $(g\psi_{\delta})(z)\to 1\ \ \text{ for every }z\in\mathbb{D},\ \ \text{
when }\delta\to 0.$

Clearly, $\|g\psi_{\delta}\|_{\infty}\leq\max(\|g\|_{\infty},1)$. Set
$C_{1}=\max(\|g\|_{\infty},1)$. Applying (2.1) with $\psi_{\delta}u$ instead
of $u$, we obtain that

$\sup_{u\in H^{2},\|u\|\leq
1}\|P_{-}\overline{\theta}\omega\psi_{\delta}u\|\leq CC_{1}\ \ \text{ for
every }\delta>0.$

By the Nehari theorem (see, for example, [N, Theorem I.B.1.3.2]), there exist
$\varphi_{\delta}\in H^{\infty}$ such that

$\|\overline{\theta}\omega\psi_{\delta}-\varphi_{\delta}\|_{\infty}\leq
CC_{1}\ \ \text{ for every }\delta>0.$

We have

$\displaystyle\|g\varphi_{\delta}\|_{\infty}=\|\theta
g\varphi_{\delta}\|_{\infty}\leq\|\theta g\varphi_{\delta}-\omega
g\psi_{\delta}\|_{\infty}+\|\omega g\psi_{\delta}\|_{\infty}$
$\displaystyle\leq\|g\|_{\infty}CC_{1}+\|\omega\|_{\infty}C_{1}$
$\displaystyle\text{ for every }\delta>0.$

By the Vitali–Montel theorem, there exist a sequence $\delta_{j}\to_{j}0$ and
$h$, $\eta\in H^{\infty}$ such that

(2.5) $(g\varphi_{\delta_{j}})(z)\to_{j}h(z)\ \text{ and }\
(\omega\psi_{\delta_{j}}-\theta\varphi_{\delta_{j}})(z)\to_{j}\eta(z)\ \
\text{ for every }z\in\mathbb{D}.$

It follows from (2.4) and (2.5) that $\omega-\theta h=g\eta$. ∎

######  Theorem 2.4.

Suppose that $g$, $\eta$, $h$, $\theta$, $\omega\in H^{\infty}$, $\theta$ and
$\omega$ are inner, and $\omega=g\eta+\theta h$. Furthermore, suppose that
$\mathcal{H}_{0}$ is a Hilbert space,
$T_{0}\colon\mathcal{H}_{0}\to\mathcal{H}_{0}$,
$X_{0}\colon\mathcal{K}_{\theta}\to\mathcal{H}_{0}$,
$Y_{0}\colon\mathcal{H}_{0}\to\mathcal{K}_{\theta}$ are operators,
$Y_{0}X_{0}=g(T_{\theta})$, $X_{0}T_{\theta}=T_{0}X_{0}$,
$Y_{0}T_{0}=T_{\theta}Y_{0}$. Define $X$ by (2.2), put $Y=I_{\theta
H^{2}}\oplus Y_{0}$,

(2.6) $T\colon\theta H^{2}\oplus\mathcal{H}_{0}\to\theta
H^{2}\oplus\mathcal{H}_{0},\ \ \ \ T=\begin{pmatrix}S|_{\theta
H^{2}}&P_{\theta
H^{2}}S|_{\mathcal{K}_{\theta}}Y_{0}\cr\mathbb{O}&T_{0}\end{pmatrix},$

and $\mathcal{M}=\operatorname{clos}X\omega H^{2}$. Then
$\mathcal{M}\in\operatorname{Lat}T$, $T|_{\mathcal{M}}\approx S$, and
$Y\mathcal{M}=\vartheta\omega H^{2}$, where $\vartheta$ is the inner factor of
$g$. Moreover,

(2.7) $XS=TX,\ \ \ YT=SY,\ \text{ and }\ \ YX=g(S).$

###### Proof.

The equalities (2.7) follow immediately from the definitions of $T$, $X$, and
$Y$. The inclusion $\mathcal{M}\in\operatorname{Lat}T$ follows from the first
equality in (2.7). By Lemma 2.3, (2.1) is fulfilled. We will show that there
exists $c>0$ such that

(2.8) $\|Yy\|\geq c\|y\|\ \ \text{ for every }y\in\mathcal{M}.$

It is sufficient to prove (2.8) for $y=X\omega u$, where $u\in H^{2}$, due to
the definition of $\mathcal{M}$ and the continuity of $Y$. For such $y$,
taking into account the third equality in (2.7) and the assumption that
$\omega$ is inner, the condition (2.8) can be rewritten as follows:

$\|gu\|\geq c\|X\omega u\|\ \ \text{ for every }u\in H^{2}.$

This estimate is fulfilled by Lemma 2.2. Thus, (2.8) is proved.

By (2.8), the operator $Y|_{\mathcal{M}}\colon\mathcal{M}\to Y\mathcal{M}$ is
invertible. By the second equality in (2.7), $Y|_{\mathcal{M}}$ realizes the
relation $T|_{\mathcal{M}}\approx S|_{Y\mathcal{M}}$. We have

$Y\mathcal{M}=Y\operatorname{clos}X\omega H^{2}=\operatorname{clos}YX\omega
H^{2}=\operatorname{clos}g(S)\omega H^{2}=\vartheta\omega H^{2},$

where $\vartheta$ is the inner factor of $g$. Clearly, $S|_{\vartheta\omega
H^{2}}\cong S$. ∎

The following simple lemma can be considered as a partial inverse to Theorem
2.4.

######  Lemma 2.5.

Suppose $g$, $\eta$, $h$, $\omega\in H^{\infty}$, $\omega$ is inner, and
$1=g\eta+\omega h$. Furthermore, suppose $T\colon\mathcal{H}\to\mathcal{H}$ is
a polynomially bounded operator, $Y\colon\mathcal{H}\to H^{2}$ is a
quasiaffinity, $(Y,S)$ is the isometric asymptote of $T$, $X\colon
H^{2}\to\mathcal{H}$ is an operator such that $YX=g(S)$. Set
$\mathcal{M}=\operatorname{clos}\omega(T)\mathcal{H}$. If
$T|_{\mathcal{M}}\approx S$, then $T\approx S$.

###### Proof.

By assumption, $Y$ realizes the relation $T\prec S$. By [M] or [K16,
Proposition 16], $T$ is a.c.. Thus, the operator $\omega(T)$ is well defined.
Since $Y\omega(T)=\omega(S)Y$ and $\operatorname{clos}Y\mathcal{H}=H^{2}$, we
conclude that $\operatorname{clos}Y\mathcal{M}=\omega H^{2}$. Suppose that
$T|_{\mathcal{M}}\approx S$. By [K89, Theorem 1], $Y|_{\mathcal{M}}$ is left
invertible, in particular,
$Y\mathcal{M}=\operatorname{clos}Y\mathcal{M}=\omega H^{2}$. Let $u\in H^{2}$.
There exists $x\in\mathcal{M}$ such that $\omega u=Yx$. We have

$\displaystyle u$ $\displaystyle=(g\eta+\omega h)u=g(S)(\eta u)+h(S)(\omega
u)=(YX)(\eta u)+h(S)Yx$ $\displaystyle=Y(X(\eta u)+h(T)x).$

We conclude that $Y\mathcal{H}=H^{2}$. Since $\ker Y=\\{0\\}$, $Y$ is
invertible by the Closed Graph Theorem. ∎

The following lemma is Nevanlinna’s theorem, see [N, Corollary I.B.3.3.2].

######  Lemma 2.6 (Nevanlinna’s theorem).

Suppose that $\theta$, $g$, $\eta\in H^{\infty}$, $\|g\eta\|_{\infty}<1$, and
$\theta$ is an inner function. Then there exists an inner function $\omega\in
H^{\infty}$ such that $\omega-g\eta\in\theta H^{\infty}$.

The following lemma is well known, see, for example, [S, Lemma 2] or [SFBK,
Proposition X.1.1].

######  Lemma 2.7.

Suppose that $\gamma$, $\theta$, $f\in H^{\infty}$, $\gamma$ and $\theta$ are
inner, $\theta$ and the inner factor of $f$ are relatively prime. Then there
exists $t\in\mathbb{C}$ such that $\gamma$ and the inner factor of $f+t\theta$
are relatively prime.

######  Lemma 2.8.

Suppose that $\gamma$, $\theta\in H^{\infty}$ are inner functions, $T_{0}$ is
a polynomially bounded operator, $Y_{0}$ is a quasiaffinity, and
$Y_{0}T_{0}=T_{\theta}Y_{0}$. Then there exist a quasiaffinity $X_{0}$ and
$g\in H^{\infty}$ such that $X_{0}T_{\theta}=T_{0}X_{0}$,
$Y_{0}X_{0}=g(T_{\theta})$, and $\gamma$ and the inner factor of $g$ are
relatively prime.

###### Proof.

By [BP], there exists a contraction $R_{0}$ such that $R_{0}\prec T_{0}$. From
the relations $R_{0}\prec T_{0}\prec T_{\theta}$ and the fact that
$T_{\theta}$ is a $C_{0}$-contraction we conclude that $T_{\theta}\prec R_{0}$
by [SFBK, Theorem X.5.7] or [B, Proposition III.5.32]. We obtain that
$T_{0}\sim T_{\theta}$. Denote by $X_{0}$ the quasiaffinity which realizes the
relation $T_{\theta}\prec T_{0}$. We have
$Y_{0}X_{0}\in\\{T_{\theta}\\}^{\prime}$. By [SFBK, Theorem X.2.10] or [B,
Proposition III.1.21], there exists $f\in H^{\infty}$ such that
$Y_{0}X_{0}=f(T_{\theta})$. Since $f(T_{\theta})$ is a quasiaffinity, $\theta$
and the inner factor of $f$ are relatively prime. By Lemma 2.7, there exists
$t\in\mathbb{C}$ such that $\gamma$ and the inner factor of $f+t\theta$ are
relatively prime. Put $g=f+t\theta$. Then $Y_{0}X_{0}=g(T_{\theta})$. ∎

######  Lemma 2.9.

Suppose that $T$ is a polynomially bounded operator and $T\prec S$. Then $S$
is the isometric asymptote of $T$.

###### Proof.

Since $T$ is polynomially bounded, by [BP], there exists a contraction $R$
such that $R\prec T$. Clearly, $R\prec S$. By [KS, Proposition 9] or [G12,
Lemma 2.1], the isometric asymptote of $R$ is $S$. Denote by $Y_{R}$ and
$Y_{T}$ the canonical intertwining mappings for $R$ and $T$, and by $X$ and
$Y$ the quasiaffinities which realize the relations $R\prec T$ and $T\prec S$,
respectively. Let $V_{T}$ be the isometric asymptote of $T$. By [K89, Theorem
1(a)], there exist operators $Z_{1}$ and $Z_{2}$ such that
$Z_{1}S=V_{T}Z_{1}$, $Y_{T}X=Z_{1}Y_{R}$, $Z_{2}V_{T}=SZ_{2}$, and
$Y=Z_{2}Y_{T}$. Since the ranges of $Y_{T}$ and $Y$ are dense, the ranges of
$Z_{1}$ and $Z_{2}$ are also dense. We obtain $S\buildrel d\over{\prec}V_{T}$
and $V_{T}\buildrel d\over{\prec}S$. By [K16, Proposition 16], $T$ is a.c..
Therefore, $V_{T}$ is a.c.. Thus, $V_{T}\cong S$. ∎

######  Theorem 2.10.

Suppose that $T\colon\mathcal{H}\to\mathcal{H}$ is a polynomially bounded
operator, $T\prec S$, $\mathcal{M}\in\operatorname{Lat}T$, and
$T|_{\mathcal{M}}\approx S$. Then there exist the operators
$X_{1}$,$X_{2}\colon H^{2}\to\mathcal{H}$, $Y\colon\mathcal{H}\to H^{2}$, the
functions $g_{1}$, $g_{2}\in H^{\infty}$ and
$\mathcal{N}\in\operatorname{Lat}T$ such that

(2.9) $YT=SY,\ \ X_{k}S=TX_{k},\ \ YX_{k}=g_{k}(S)\ \text{ and }\ \
X_{k}Y=g_{k}(T),\ \ \ k=1,2,$

the inner factors of $g_{1}$ and $g_{2}$ are relatively prime,
$T|_{\mathcal{N}}\approx S$, and $\mathcal{M}\vee\mathcal{N}=\mathcal{H}$.

###### Proof.

By Lemma 2.9, the isometric asymptote of $T$ is $S$. Denote by $Y$ the
canonical intertwining mapping. Then $Y|_{\mathcal{M}}$ is left invertible.
Let $\theta\in H^{\infty}$ be an inner function such that $Y\mathcal{M}=\theta
H^{2}$. Let $T_{0}$ and $Y_{0}$ be defined as in Proposition 2.1. Without loss
of generality, we can suppose that $T$ has the form (2.6) and $Y=I_{\theta
H^{2}}\oplus Y_{0}$. Then $Y_{0}$ is a quasiaffinity which realizes the
relation $T_{0}\prec T_{\theta}$.

Denote by $X_{10}$ and $g_{1}$ the quasiaffinity and the function from Lemma
2.8 applied to $\gamma=\theta$, $\theta$, $T_{0}$, and $Y_{0}$. Denote by
$\vartheta_{1}$ the inner factor of $g_{1}$. By Lemma 2.8, $\theta$ and
$\vartheta_{1}$ are relatively prime. Denote by $X_{1}$ the operator defined
by (2.2) applied to $g_{1}$ and $X_{10}$.

Take $\eta\in H^{\infty}$ such that $\|g_{1}\eta\|_{\infty}<1$, and $\theta$
and the inner factor of $\eta$ are relatively prime. By Lemma 2.6, there exist
$\omega$, $h\in H^{\infty}$ such that $\omega$ is inner and
$\omega=g_{1}\eta+\theta h$. Put $\mathcal{N}=\operatorname{clos}X_{1}\omega
H^{2}$. By Theorem 2.4, $\mathcal{N}\in\operatorname{Lat}T$,
$T|_{\mathcal{N}}\approx S$, and $Y\mathcal{N}=\vartheta_{1}\omega H^{2}$.
Furthermore, (2.7) is fulfilled for $T$, $X_{1}$, $Y$, and $g_{1}$. Note that
$\theta$ and $\vartheta_{1}\omega$ are relatively prime.

Denote by $X_{20}$ and $g_{2}$ the quasiaffinity and the function from Lemma
2.8 applied to $\gamma=\vartheta_{1}$, $\theta$, $T_{0}$, and $Y_{0}$. Denote
by $\vartheta_{2}$ the inner factor of $g_{2}$. By Lemma 2.8, $\vartheta_{1}$
and $\vartheta_{2}$ are relatively prime. Denote by $X_{2}$ the operator
defined by (2.2) applied to $g_{2}$ and $X_{20}$. Then (2.7) is fulfilled for
$T$, $X_{2}$, $Y$, and $g_{2}$.

It is easy to see from (2.6) and (2.7) that

$g(T)=\begin{pmatrix}g(S)|_{\theta H^{2}}&P_{\theta
H^{2}}g(S)|_{\mathcal{K}_{\theta}}Y_{0}\\\ \mathbb{O}&g(T_{0})\end{pmatrix}$

for every $g\in H^{\infty}$. The equations (2.9) follow from this equation and
(2.7).

It is well known that if (2.9) are fulfilled, and the inner factors of $g_{1}$
and $g_{2}$ are relatively prime, then the mapping $\mathcal{J}_{Y}$ defined
by (1.2) is a lattice isomorphism. We have

$\mathcal{J}_{Y}(\mathcal{M}\vee\mathcal{N})=\mathcal{J}_{Y}\mathcal{M}\vee\mathcal{J}_{Y}\mathcal{N}=\theta
H^{2}\vee\vartheta_{1}\omega H^{2}=H^{2}$

(because $\theta$ and $\vartheta_{1}\omega$ are relatively prime). Therefore,
$\mathcal{M}\vee\mathcal{N}=\mathcal{H}$. ∎

######  Remark 2.11.

If $T$ is a contraction such that $T\prec S$, then there exists
$\mathcal{M}\in\operatorname{Lat}T$ such that $T|_{\mathcal{M}}\approx S$ by
[K07] or [SFBK, Theorem IX.3.5]. Thus, $T$ satisfies to the conditions of
Theorem 2.10.

######  Lemma 2.12.

Suppose that $T$ is a polynomially bounded operator, $Y$ is a quasiaffinity,
and $YT=SY$. Then the mapping $\mathcal{J}_{Y}$ defined by (1.2) is a lattice
isomorphism.

###### Proof.

By [BP], there exists a contraction $R$ such that $R\prec T$. Clearly, $R\prec
S$. By Remark 2.11, the conclusion of Theorem 2.10 is fulfilled for $R$. Using
the relation $R\prec T$, it is easy to see that there exist operators
$X_{1}$,$X_{2}\colon H^{2}\to\mathcal{H}$ and the functions $g_{1}$, $g_{2}\in
H^{\infty}$ such that the inner factors of $g_{1}$ and $g_{2}$ are relatively
prime and (2.9) is fulfilled for $T$ and $Y$. Since the inner factors of
functions $g_{1}$ and $g_{2}$ are relatively prime, $\mathcal{J}_{Y}$ is a
lattice-isomorphism.∎

######  Remark 2.13.

The relations (2.9) are proved for contractions $T$ such that $T\prec S$ in
[G03, Corollary 2.5] in other way.

The following lemma follows from the definition of a.c. polynomially bounded
operators (see [G17, Lemma 2.2] for the proof).

######  Lemma 2.14.

Suppose that $T$ is a polynomially bounded operator, and
$\mathcal{M}\in\operatorname{Lat}T$. Then $T$ is a.c. if and only if
$T|_{\mathcal{M}}$ and $P_{\mathcal{M}^{\perp}}T|_{\mathcal{M}^{\perp}}$ are
a.c..

The following lemma can be easily checked directly, therefore, its proof is
omitted.

######  Lemma 2.15.

Suppose that $T\colon\mathcal{H}\to\mathcal{H}$ is an operator, $\mathcal{M}$,
$\mathcal{N}\in\operatorname{Lat}T$, and
$\mathcal{H}=\mathcal{M}\vee\mathcal{N}$. Put
$\mathcal{E}=\mathcal{M}\cap\mathcal{N}$. Then the quasiaffinity
$P_{\mathcal{M}^{\perp}}|_{\mathcal{N}\ominus\mathcal{E}}$ realizes the
relation

$P_{\mathcal{N}\ominus\mathcal{E}}T|_{\mathcal{N}\ominus\mathcal{E}}\prec
P_{\mathcal{M}^{\perp}}T|_{\mathcal{M}^{\perp}}.$

The following proposition is closed to [T] and [KS, Proposition 20].

######  Proposition 2.16.

Suppose that $T\colon\mathcal{H}\to\mathcal{H}$ is a polynomially bounded
operator, $\mathcal{M}_{1}$, $\mathcal{M}_{2}\in\operatorname{Lat}T$,
$T|_{\mathcal{M}_{k}}\prec S$, $k=1,2$,
$\mathcal{M}_{1}\vee\mathcal{M}_{2}=\mathcal{H}$ and
$\mathcal{M}_{1}\cap\mathcal{M}_{2}\neq\\{0\\}$. Then $T$ is a.c., $T\buildrel
d\over{\prec}S$, and $S$ is the isometric asymptote of $T$. Moreover, if
$T\not\prec S$, then $T|_{\ker Y}$ is a $C_{0}$-operator, where $Y$ is a
canonical intertwining mapping.

###### Proof.

Put $\mathcal{E}=\mathcal{M}_{1}\cap\mathcal{M}_{2}$. We can suppose that
$\mathcal{E}\neq\mathcal{M}_{2}$. Denote by $X$ a quasiaffinity which realizes
the relation $T|_{\mathcal{M}_{2}}\prec S$. There exists an inner function
$\theta\in H^{\infty}$ such that $\mathcal{J}_{X}\mathcal{E}=\theta H^{2}$,
where $\mathcal{J}_{X}$ is defined by (1.2). Put

$T_{0}=P_{\mathcal{M}_{2}\ominus\mathcal{E}}T|_{\mathcal{M}_{2}\ominus\mathcal{E}},\
\ \ T_{1}=P_{\mathcal{M}_{1}^{\perp}}T|_{\mathcal{M}_{1}^{\perp}},\ \ \text{
and }\ \
X_{0}=P_{\mathcal{K}_{\theta}}X_{0}|_{\mathcal{M}_{2}\ominus\mathcal{E}}.$

Clearly, $X_{0}T_{0}=T_{\theta}X_{0}$. Taking into account that
$\mathcal{J}_{X}$ is a lattice isomorphism (by Lemma 2.12), it is easy to see
that $X_{0}$ is a quasiaffinity. Therefore, $T_{0}$ is a.c. ([K16, Proposition
16]) and $\theta(T_{0})=\mathbb{O}$.

By Lemma 2.15, $T_{0}\prec T_{1}$. By [M] or [K16, Proposition 16] and Lemma
2.14, $T$ is a.c.. Furthermore,

(2.10) $\theta(T_{1})=\mathbb{O}.$

Let $(Y,U)$ be the unitary asymptote of $T$. By [K89, Theorem 3] and Lemma
2.9, $U\cong U_{\mathbb{T}}$, and $Y|_{\mathcal{M}_{1}}$ realizes the relation
$T|_{\mathcal{M}_{1}}\prec U_{\mathbb{T}}|_{\xi H^{2}}$ for some $\xi\in
L^{\infty}$ such that $|\xi|=1$ a.e. with respect to $m$.

If $\operatorname{clos}Y\mathcal{H}=L^{2}$, then $P_{\xi
H^{2}_{-}}Y|_{\mathcal{M}_{1}^{\perp}}$ realizes the relation $T_{1}\buildrel
d\over{\prec}P_{\xi H^{2}_{-}}U_{\mathbb{T}}|_{\xi H^{2}_{-}}$, which
contradicts to (2.10). Thus, $\xi
H^{2}\subset\operatorname{clos}Y\mathcal{H}\neq L^{2}$. Since
$\operatorname{clos}Y\mathcal{H}\in\operatorname{Lat}U_{\mathbb{T}}$, we
conclude that $\operatorname{clos}Y\mathcal{H}=\xi_{1}H^{2}$ for some
$\xi_{1}\in L^{\infty}$ such that $|\xi_{1}|=1$ a.e. with respect to $m$.
Since $U_{\mathbb{T}}|_{\xi_{1}H^{2}}\cong S$, we conclude that $S$ is the
isometric asymptote of $T$.

Put $\mathcal{E}_{0}=\ker Y$ and
$\mathcal{E}_{1}=\operatorname{clos}P_{\mathcal{M}_{1}^{\perp}}\mathcal{E}_{0}$.
Then $\mathcal{E}_{0}\in\operatorname{Lat}T$ and
$\mathcal{E}_{0}\cap\mathcal{M}_{1}=\\{0\\}$. Therefore,
$\mathcal{E}_{1}\in\operatorname{Lat}T_{1}$ and
$P_{\mathcal{E}_{1}}|_{\mathcal{E}_{0}}$ realizes the relation
$T|_{\mathcal{E}_{0}}\prec T_{1}|_{\mathcal{E}_{1}}$. By (2.10),
$\theta(T|_{\mathcal{E}_{0}})=\mathbb{O}$. Thus, $T|_{\mathcal{E}_{0}}$ is a
$C_{0}$-operator. ∎

##  3\. Existence of a polynomially bounded operator with the minimal function
$\exp(a\frac{z+1}{z-1})$ which is not similar to a contraction

For $\lambda\in\mathbb{D}$ denote by $b_{\lambda}$ a Blaschke factor:
$b_{\lambda}(z)=\frac{|\lambda|}{\lambda}\frac{\lambda-z}{1-\overline{\lambda}z}$,
$z\in\mathbb{D}$. For $w\in\mathbb{D}$ put

(3.1) $\beta_{w}(z)=\frac{w-z}{1-\overline{w}z},\ \ \ z\in\mathbb{D}.$

Clearly, $\beta_{w}\circ\beta_{w}(z)=z$ for every $z\in\mathbb{D}$. For every
$\varphi\in H^{\infty}$ we have $\varphi\circ\beta_{w}\in H^{\infty}$,
$\|\varphi\circ\beta_{w}\|_{\infty}=\|\varphi\|_{\infty}$, and

$b_{\lambda}\circ\beta_{w}=\zeta_{w,\lambda}b_{\beta_{w}(\lambda)},\ \ \text{
where }\zeta_{w,\lambda}\in\mathbb{T}.$

For $a>0$ and $0<\alpha<1$ put

(3.2) $\omega_{a}(z)=\exp\Bigl{(}a\frac{z+1}{z-1}\Bigr{)}\ \text{ and }\ \
g_{\alpha}(z)=\exp\Bigl{(}-\Bigl{(}\frac{1+z}{1-z}\Bigr{)}^{\alpha}\Bigr{)},\
\ z\in\mathbb{D}.$

We have $\omega_{a}$, $g_{\alpha}\in H^{\infty}$, $\omega_{a}$ is inner, and
$g_{\alpha}$ is outer by [N, Example I.A.4.3.7, p. 71].

The following lemma can be easily proved by induction. Therefore, its proof is
omitted.

######  Lemma 3.1.

Let $a>0$, and let $0<\alpha<1$. Let $\omega_{a}$ and $g_{\alpha}$ be defined
in (3.2). Let $n\in\mathbb{N}$. Then there exist functions $\psi_{\omega,n}$,
$\psi_{g,n}$ analytic in $\mathbb{D}$ such that

$\omega_{a}^{(n)}=\omega_{a}\psi_{\omega,n}\ \ \text{ and }\ \
g_{\alpha}^{(n)}=g_{\alpha}\psi_{g,n}.$

Furthermore,

$\psi_{\omega,n}(z)\\!=\\!\sum_{l=1}^{K_{\omega,n}}\frac{c_{\omega,n,l}}{(1-z)^{k_{\omega,n,l}}}\text{
and
}\psi_{g,n}(z)\\!=\\!\sum_{l=1}^{K_{g,n}}c_{g,n,l}\Bigl{(}\frac{1+z}{1-z}\Bigr{)}^{d_{1,l,n}}\\!(1-z)^{d_{2,n,l}},$

where $K_{\omega,n}$, $K_{g,n}$, $k_{\omega,n,l}\in\mathbb{N}$,
$c_{\omega,n,l}$, $c_{g,n,l}$, $d_{1,l,n}$, $d_{2,n,l}\in\mathbb{R}$ .

######  Lemma 3.2.

Let $a>0$, and let $0<\alpha<1$. Let $\omega_{a}$ and $g_{\alpha}$ be defined
in (3.2). Put

$\displaystyle\psi_{g,n,k}$ $\displaystyle=\frac{n!}{k!(n-k)!}\psi_{g,n-k},\ \
\ 1\leq k\leq n,$ $\displaystyle\psi_{g,k,k}$ $\displaystyle=1,\
\psi_{g,n,k}=0,\ \ \ n\leq k-1,\ \ \psi_{\omega,0}=1.$

For $M\in\mathbb{N}$ set $\Psi_{M}=[\psi_{g,n,k}]_{n,k=0}^{M-1}$ and define
the functions $\kappa_{M,n}$, $0\leq n\leq M-1$, by the formula

$[\kappa_{M,n}]_{n=0}^{M-1}=\frac{\omega_{a}}{g_{\alpha}}\cdot\Psi_{M}^{-1}\cdot[\psi_{\omega,n}]_{n=0}^{M-1}.$

Then $\kappa_{M,n}$ are functions analytic in $\mathbb{D}$, and
$\kappa_{M,n}(z)\to 0$ when $z\in(0,1)$, $z\to 1$ for every $0\leq n\leq M-1$.

###### Proof.

We have $\Psi_{M}$ is a lower triangular matrix-function, and the elements of
the main diagonal of $\Psi_{M}$ are equal to $1$. Therefore, $\Psi_{M}$ is
invertible. The elements of the matrix-function
$\Psi_{M}^{-1}\cdot[\psi_{\omega,n}]_{n=0}^{M-1}$ have the form

$\sum_{l=1}^{K}c_{l}\Bigl{(}\frac{1+z}{1-z}\Bigr{)}^{d_{1,l}}(1-z)^{d_{2,l}}$

for some $K\in\mathbb{N}$ and $c_{l}$, $d_{1,l}$, $d_{2,l}\in\mathbb{R}$
(depended on $\alpha$, $a$, $M$, and the indices of the considered element of
the matrix).

Set $t=\frac{1+z}{1-z}$. Then $1-z=\frac{2}{1+t}$, and
$\frac{\omega_{a}(z)}{g_{\alpha}(z)}=\exp(-at+t^{\alpha})$. Furthermore,
$t\to+\infty$ when $z\in(0,1)$, $z\to 1$. We have
$t^{d_{1}}(1+t)^{d_{2}}\exp(-at+t^{\alpha})\to 0$ when $t\to+\infty$ for every
$d_{1}$, $d_{2}\in\mathbb{R}$. The conclusion of the lemma follows from this
relation. ∎

The following lemma is an emphasizing of [G16, Lemma 3.5].

######  Lemma 3.3.

Suppose that $\Lambda\subset\mathbb{D}$ is finite, $1\leq k_{\lambda}<\infty$
for every $\lambda\in\Lambda$, and
$B=\prod_{\lambda\in\Lambda}b_{\lambda}^{k_{\lambda}}$. Then there exists
$C>0$ which depends on $B$ such that

$\operatorname{dist}(\varphi,(B\circ\beta_{w})H^{\infty})\leq
C\max_{\lambda\in\Lambda,0\leq k\leq
k_{\lambda}-1}|\varphi^{(k)}(\beta_{w}(\lambda))|$

for every $\varphi\in H^{\infty}$ and every $w\in\mathbb{D}$.

###### Proof.

We have
$\operatorname{dist}(\varphi,(B\circ\beta_{w})H^{\infty})=\operatorname{dist}(\varphi\circ\beta_{w},BH^{\infty})$
for every $\varphi\in H^{\infty}$ and every $w\in\mathbb{D}$. By [G16, Lemma
3.5], there exists $C>0$ (which depends on $B$) such that

$\operatorname{dist}(\psi,BH^{\infty})\leq C\max_{\lambda\in\Lambda,0\leq
k\leq k_{\lambda}-1}|\psi^{(k)}(\lambda)|\ \ \text{ for every }\psi\in
H^{\infty}.$

We need to estimate $|(\varphi\circ\beta_{w})^{(k)}(\lambda)|$ for $1\leq
k\leq k_{\lambda}-1$. By [G16, Lemma 3.3], there exist functions
$c_{kl}\colon\mathbb{D}\to\mathbb{C}$ such that
$\sup_{\mathbb{D}}|c_{kl}|<\infty$ for every $k\geq 1$, $0\leq l\leq k-1$, and
for every analytic function $\varphi\colon\mathbb{D}\to\mathbb{C}$ and every
$w\in\mathbb{D}$

$(\varphi\circ\beta_{w})^{(k)}(z)=\sum_{l=1}^{k}\varphi^{(l)}(\beta_{w}(z))\frac{c_{k,k-l}(w)}{(1-\overline{w}z)^{k+l}}.$

Therefore,

$\displaystyle|(\varphi\circ\beta_{w})^{(k)}(\lambda)|\leq\sum_{l=1}^{k}|\varphi^{(l)}(\beta_{w}(\lambda))|\frac{|c_{k,k-l}(w)|}{|1-\overline{w}\lambda|^{k+l}}$
$\displaystyle\leq\\!\max_{\lambda\in\Lambda,0\leq k\leq
k_{\lambda}-1}\\!|\varphi^{(k)}(\beta_{w}(\lambda))|\cdot\max_{\lambda\in\Lambda}(k_{\lambda}-1)\cdot\\!\sup_{{1\leq
l\leq k,1\leq k\leq
k_{\lambda}-1,}\atop{w\in\mathbb{D}}}\\!\frac{|c_{k,k-l}(w)|}{(1-\max_{\lambda\in\Lambda}|\lambda|)^{k+l}}$

for $1\leq k\leq k_{\lambda}-1$. ∎

######  Theorem 3.4.

Suppose that $C>0$, $a>0$, $0<\alpha<1$, and $\\{B_{N}\\}_{N}$ is a sequence
of finite Blaschke products with zeros from $(0,1)$. Then there exists a
sequence $\\{\rho_{N}\\}_{N}\subset(0,1)$ such that for any $w\in[\rho_{N},1)$
there exists $\eta_{N,w}\in H^{\infty}$ such that

(3.3) $\operatorname{dist}(\eta_{N,w},(B_{N}\circ\beta_{w})H^{\infty})\leq C\
\ \text{ and }\ \
\omega_{a}-g_{\alpha}\eta_{N,w}\in(B_{N}\circ\beta_{w})H^{\infty},$

where $\beta_{w}$ is defined in (3.1), and $\omega_{a}$ and $g_{\alpha}$ are
defined in (3.2).

###### Proof.

Let $N$ be fixed. Denote by $C_{N}$ a constant from Lemma 3.3 applied to
$B_{N}=\prod_{\lambda\in\Lambda}b_{\lambda}^{k_{\lambda}}$. By Lemma 3.2,
there exists $\rho_{1N}\in(0,1)$ such that $|\kappa_{k_{\lambda},k}(z)|\leq
C/C_{N}$ for every $z\in[\rho_{1N},1)$, $0\leq k\leq k_{\lambda}-1$,
$\lambda\in\Lambda$.

Since $\beta_{w}(\lambda)\to 1$ when $w\to 1$ for every
$\lambda\in\mathbb{D}$, and $\beta_{w}(\lambda)\in(0,1)$ when $0<\lambda<w<1$,
there exists $\rho_{N}\in(0,1)$ such that $\beta_{w}(\lambda)\in[\rho_{1N},1)$
when $w\in[\rho_{N},1)$ for every $\lambda\in\Lambda$.

Since $\Lambda$ is a finite set, and $k_{\lambda}<\infty$ for every
$\lambda\in\Lambda$, we have that for every $w\in\mathbb{D}$ there exists
$\eta_{N,w}\in H^{\infty}$ such that

(3.4)
$\eta_{N,w}^{(k)}(\beta_{w}(\lambda))=\kappa_{k_{\lambda},k}(\beta_{w}(\lambda))\
\text{ for every }\ \lambda\in\Lambda\ \text{ and }\ 0\leq k\leq
k_{\lambda}-1.$

By Lemma 3.3, $\operatorname{dist}(\eta_{N,w},(B\circ\beta_{w})H^{\infty})\leq
C$ for $w\in[\rho_{N},1)$.

The inclusion in (3.3) is fulfilled if and only if

(3.5)
$\omega_{a}^{(k)}(\beta_{w}(\lambda))-(g_{\alpha}\eta_{N,w})^{(k)}(\beta_{w}(\lambda))=0\
\ \text{ for all }\lambda\in\Lambda,\ 0\leq k\leq k_{\lambda}-1.$

Clearly,

$(g_{\alpha}\eta_{N,w})^{(n)}(z)=\sum_{k=0}^{n}\frac{n!}{k!(n-k)!}g_{\alpha}^{(n-k)}(z)\eta_{N,w}^{(k)}(z)\
\text{ for }\ n\in\mathbb{N}\ \text{ and }z\in\mathbb{D}.$

Therefore,

$[(g_{\alpha}\eta_{N,w})^{(k)}(z)]_{k=0}^{k_{\lambda}-1}=g_{\alpha}(z)\cdot\Psi_{k_{\lambda}}(z)\cdot[\eta_{N,w}^{(k)}(z)]_{k=0}^{k_{\lambda}-1}$

for every $z\in\mathbb{D}$, where $\Psi_{k_{\lambda}}$ is defined in Lemma
3.2. By (3.4) and Lemmas 3.1 and 3.2,

$\displaystyle[(g_{\alpha}\eta_{N,w})^{(k)}(z)]_{k=0}^{k_{\lambda}-1}$
$\displaystyle=g_{\alpha}(z)\cdot\Psi_{k_{\lambda}}(z)\cdot[\kappa_{k_{\lambda},k}(z)]_{k=0}^{k_{\lambda}-1}=\omega_{a}(z)\cdot[\psi_{\omega,k}(z)]_{k=0}^{k_{\lambda}-1}$
$\displaystyle=[\omega_{a}^{(k)}(z)]_{k=0}^{k_{\lambda}-1}\ \ \text{ when }\
z=\beta_{w}(\lambda),\ \lambda\in\Lambda.$

Thus, the equalities (3.5) are fulfilled, and the inclusion in (3.3) is
proved. ∎

The following theorem is a version of [G16, Theorem 3.8]. The condition (3.6)
from the theorem is the generalized Carleson condition, see [N, Theorem
II.C.3.2.14, p. 164].

######  Theorem 3.5.

Suppose that $C>0$, $\\{\rho_{N}\\}_{N}\subset(0,1)$, $g\in H^{\infty}$, and
$g^{(k)}(r)\to 0$ when $r\in(0,1)$, $r\to 1$, for every $k\geq 0$.
Furthermore, suppose that $B_{N}$ are finite Blaschke products with zeros from
$(0,1)$, $\varphi_{N}\in H^{\infty}$, $\varphi_{N}(\lambda)\neq 0$ for every
$\lambda\in\mathbb{D}$ such that $B_{N}(\lambda)=0$, for every index $N$. Then
there exist $\delta>0$ and sequences of $w_{N}\in[\rho_{N},1)$, of
$\zeta_{N}\in\mathbb{T}$, and of $\psi_{N}\in H^{\infty}$ such that the
product $\prod_{N}\zeta_{N}B_{N}\circ\beta_{w_{N}}$ converges,

(3.6)
$\bigl{|}\prod_{N}\zeta_{N}(B_{N}\circ\beta_{w_{N}})(z)\bigr{|}\geq\delta\inf_{N}|(B_{N}\circ\beta_{w_{N}})(z)|\
\ \text{ for every }\ z\in\mathbb{D},$

$g-\psi_{N}\cdot\varphi_{N}\circ\beta_{w_{N}}\in(B_{N}\circ\beta_{w_{N}})H^{\infty}$,
and $\operatorname{dist}(\psi_{N},(B_{N}\circ\beta_{w_{N}})H^{\infty})\leq C$.

###### Proof.

The proof of [G16, Theorem 3.8] is beginned from the definition of the
sequence $\\{r_{N}\\}_{N}\subset(0,1)$. To prove the present theorem, it needs
to replace $r_{N}$ by $\max(r_{N},\rho_{N})$ and to repeat the proof of [G16,
Theorem 3.8]. ∎

The following theorem is a version of [G16, Theorem 7.1].

######  Theorem 3.6.

Let $a>0$, and let $0<\alpha<1$. Let $\omega_{a}$ and $g_{\alpha}$ be defined
in (3.2). Then there exist a function $\eta\in H^{\infty}$, an operator
$T\colon\mathcal{H}\to\mathcal{H}$, a Blaschke product $B$ with zeros from
$(0,1)$, and quasiaffinities $X\colon\mathcal{H}\to\mathcal{K}_{B}$,
$Y\colon\mathcal{K}_{B}\to\mathcal{H}$ such that

(3.7) $\omega_{a}-g_{\alpha}\eta\in BH^{\infty},$

$T$ is polynomially bounded, $T$ is not similar to a contraction,

(3.8) $XT=T_{B}X,\ \ \ YT_{B}=TY,\ \ \text{ and }\ \ XY=g_{\alpha}(T_{B}),$

where $T_{B}$ is defined in (1.3).

###### Proof.

Let $\\{T_{N}\\}_{N}$ and $\\{B_{N}\\}_{N}$ be the sequences of operators and
of finite Blaschke products with zeros from $(0,1)$ from [G16, Theorem 6.3],
respectively. Let $C>0$ be fixed. Denote by $\mathcal{H}_{N}$ the finite
dimensional spaces on which $T_{N}$ acts. There exist invertible operators
$X_{N}\colon\mathcal{H}_{N}\to\mathcal{K}_{B_{N}}$,
$Y_{N}\colon\mathcal{K}_{B_{N}}\to\mathcal{H}_{N}$ such that
$X_{N}T_{N}=T_{B_{N}}X_{N}$, $Y_{N}T_{B_{N}}=T_{N}Y_{N}$, $\|X_{N}\|\leq C$,
$\|Y_{N}\|\leq C$. By [SFBK, Theorem X.2.10], there exist functions
$\varphi_{N}\in H^{\infty}$ such that $X_{N}Y_{N}=\varphi_{N}(T_{B_{N}})$.
Note that $\varphi_{N}(\lambda)\neq 0$ for every $\lambda\in\mathbb{D}$ such
that $B_{N}(\lambda)=0$, for every index $N$.

Let $\\{\rho_{N}\\}_{N}$ be the sequence from Theorem 3.4 applied to $C$, $a$,
$\alpha$, and $\\{B_{N}\\}_{N}$. Applying Theorem 3.5 to $C$,
$\\{\rho_{N}\\}_{N}$, $g_{\alpha}$, sequences of $B_{N}$ and of $\varphi_{N}$
we obtain $\delta>0$ and sequences of $w_{N}\in[\rho_{N},1)$, of
$\zeta_{N}\in\mathbb{T}$ and of $\psi_{N}\in H^{\infty}$ which satisfy the
conclusion of Theorem 3.5. Put

$B=\prod_{N}\zeta_{N}B_{N}\circ\beta_{w_{N}}\ \ \text{ and }\ \
T=\bigoplus_{N}\beta_{w_{N}}(T_{N}).$

By [G16, Theorem 6.3] and [G16, Corollary 1.2], $T$ is polynomially bounded,
and $T$ is not similar to a contraction.

Let $\eta_{N,w_{N}}$ be from Theorem 3.4. Then (3.3) is fulfilled for every
$N$ with $w=w_{N}$. By (3.6), (3.3), and [N, Theorem II.C.3.2.14, p. 164],
there exists a function $\eta\in H^{\infty}$ such that
$\eta-\eta_{N,w_{N}}\in(B_{N}\circ\beta_{w_{N}})H^{\infty}$ for every $N$. We
have

$\omega_{a}-g_{\alpha}\eta=\omega_{a}-g_{\alpha}\eta_{N,w_{N}}+g_{\alpha}(\eta_{N,w_{N}}-\eta)\in(B_{N}\circ\beta_{w_{N}})H^{\infty}$

for every $N$ due to (3.3). Thus, (3.7) is proved.

Equalities (3.8) are proved exactly as in the proof of [G16, Theorem 7.1]. ∎

######  Theorem 3.7.

Let $a>0$, and let $\omega_{a}$ be defined in (3.2). Then there exists a
cyclic a.c. polynomially bounded operator $T_{0}$ such that $T_{0}$ is not
similar to a contraction and $\omega_{a}(T_{0})=\mathbb{O}$.

###### Proof.

Take $0<\alpha<1$, and apply Theorem 3.6 to $a$ and $\alpha$. Denote the
polynomially bounded operator and quasiaffinities constructed in Theorem 3.6
by $T_{10}$, $X_{10}$, and $Y_{10}$, respectively. Define $X$ and $T$ by (2.2)
and (2.6), respectively, with $\theta=B$, where $B$ is the Blaschke product
obtained in Theorem 3.6. By [G16, Corollary 2.3], $T$ is polynomially bounded,
and $T$ is not similar to a contraction. Set $Y=I_{BH^{2}}\oplus Y_{10}$.
Clearly, $Y$ is a quasiaffinity. Since $g_{\alpha}$ is outer, $X$ is a
quasiaffinity (sf. [G16, Lemma 2.1 and Proposition 2.4]). Taking into account
(2.7), we obtain that $T\sim S$. Therefore, $T$ is cyclic, and, by [M] or
[K16, Proposition 16], $T$ is a.c..

Put

$\mathcal{M}=\operatorname{clos}X\omega_{a}H^{2}.$

By Theorem 2.4, $T|_{\mathcal{M}}\approx S$, and
$Y\mathcal{M}=\omega_{a}H^{2}$ (because $g_{\alpha}$ is outer). Put

$T_{0}=P_{\mathcal{M}^{\perp}}T|_{\mathcal{M}^{\perp}}.$

Then $T_{0}$ is cyclic, because $T$ is cyclic (see (1.1)), and $T_{0}$ is
a.c., because $T$ is a.c. ([M] or [K16, Propositions 14 and 35]). It is easy
to see that $\omega_{a}(T_{0})=\mathbb{O}$. Indeed, denote by $\mathcal{H}$
the space in which $T$ acts. We have

$\mathcal{M}=\operatorname{clos}X\omega_{a}H^{2}=\operatorname{clos}X\omega_{a}(S)H^{2}=\operatorname{clos}\omega_{a}(T)XH^{2}=\operatorname{clos}\omega_{a}(T)\mathcal{H}.$

Therefore,
$\omega_{a}(T_{0})=P_{\mathcal{M}^{\perp}}\omega_{a}(T)|_{\mathcal{M}^{\perp}}=\mathbb{O}$.

Finally, $T_{0}$ is not similar to a contraction by [C, Corollary 4.2].
Indeed, if $T_{0}$ is similar to a contraction, we can apply [C, Corollary
4.2] to $T$, because $T|_{\mathcal{M}}$ is similar to an isometry, and $T$ is
power bounded. Then $T$ must be similar to a contraction, a contradiction.∎

.

## References

  * [1]
  * [B] H. Bercovici,_Operator theory and arithmetic in $H^{\infty}$_, Math. Surveys Monogr., Amer. Math. Soc., v. 26, 1988.
  * [BP] H. Bercovici and B. Prunaru, Quasiaffine transforms of polynomially bounded operators, _Arch. Math. (Basel)_ , 71 (1998), 384–387.
  * [C] G. Cassier, Generalized Toeplitz operators, restriction to invariant subspaces and similarity problems, _J. Oper. Theory_ , 53 (2005), no. 1, 49–89.
  * [DP] K. R. Davidson and V. I. Paulsen, Polynomially bounded operators, _J. reine angew. Math._ , 487 (1997), 153–170.
  * [F] S. H. Ferguson, Polynomially bounded operators and Ext groups, _Proc. Amer. Math. Soc._ , 124 (1996), 2779–2785.
  * [G03] M. F. Gamal’, $C_{\cdot 0}$-contractions: a Jordan model and lattices of invariant subspaces, _Algebra i Analiz_ , 15 (2003), no. 5, 198–227 (in Russian); English transl.: _St. Petersburg Math. J._ , 15 (2003), no. 5, 773–793.
  * [G12] M. F. Gamal’, On the shift index of contractions, _Acta Sci. Math. (Szeged)_ , 78 (2012), 279–290.
  * [G16] M. F. Gamal’, Examples of cyclic polynomially bounded operators that are not similar to contractions, _Acta Sci. Math. (Szeged)_ , 82 (2016), 597–628.
  * [G17] M. F. Gamal’, A sufficient condition for the similarity of a polynomially bounded operator to a contraction, _Zap. Nauchn. Sem. S.-Peterburg. Otdel. Mat. Inst. Steklov. (POMI)_ , 456 (2017), 77–95 (in Russian); English transl.: arxiv:1803.10174
  * [H] P. R. Halmos, Ten problems in Hilbert space, _Bull. Am. Math. Soc._ , 76 (1970), 887–933.
  * [K89] L. Kérchy, Isometric asymptotes of power bounded operators, _Indiana Univ. Math. J._ , 38 (1989), 173–188.
  * [K07] L. Kérchy, Shift-type invariant subspaces of contractions, _J. Funct. Anal._ , 246 (2007), 281–301.
  * [K16] L. Kérchy, Quasianalytic polynomially bounded operators, in: _Operator Theory: the State of the Art_ , Theta, Bucharest, 2016, 75–101.
  * [KS] L. Kérchy and A. Szalai, Asymptotically cyclic quasianalytic contractions, _Studia Math._ , 223 (2014), 53–75.
  * [M] W. Mlak, Algebraic polynomially bounded operators, _Ann. Polon. Math._ , 29 (1974), 133–139.
  * [N] N. K. Nikolski, _Operators, functions, and systems: an easy reading. Volume I: Hardy, Hankel, and Toeplitz, Volume II: Model operators and systems_ , Math. Surveys and Monographs 92, AMS, 2002.
  * [Pa] V. I. Paulsen, Every completely polynomially bounded operator is similar to a contraction, _J. Funct. Anal._ , 55 (1984), 1–17.
  * [Pi] G. Pisier, A polynomially bounded operator on Hilbert space which is not similar to a contraction, _J. Amer. Math. Soc._ , 10 (1997), no. 2, 351–369.
  * [S] B. Sz.-Nagy, Diagonalization of matrices over $H^{\infty}$, _Acta Sci. Math. (Szeged)_ , 38 (1976), 223–238.
  * [SFBK] B. Sz.-Nagy, C. Foias, H. Bercovici and L. Kérchy, _Harmonic analysis of operators on Hilbert space_ , Springer, New York, 2010.
  * [T] K. Takahashi, On contractions without disjoint invariant subspaces, _Proc. Amer. Math. Soc._ , 110 (1990), 935–937.


# SARC: Soft Actor Retrospective Critic

Sukriti Verma
Carnegie Mellon University
<EMAIL_ADDRESS>
&Ayush Chopra
MIT Media Lab
<EMAIL_ADDRESS>
Jayakumar Subramanian
Adobe
<EMAIL_ADDRESS>
&Mausoom Sarkar
Adobe
<EMAIL_ADDRESS>
&Nikaash Puri
Adobe
<EMAIL_ADDRESS>
&Piyush Gupta
Adobe
<EMAIL_ADDRESS>
&Balaji Krishnamurthy
Adobe
<EMAIL_ADDRESS>
Work done while at Adobe

###### Abstract

The two-time scale nature of SAC, which is an actor-critic algorithm, is
characterised by the fact that the critic estimate has not converged for the
actor at any given time, but since the critic learns faster than the actor, it
ensures eventual consistency between the two. Various strategies have been
introduced in literature to learn better gradient estimates to help achieve
better convergence. Since gradient estimates depend upon the critic, we posit
that improving the critic can provide a better gradient estimate for the actor
at each time. Utilizing this, we propose Soft Actor Retrospective Critic
(SARC), where we augment the SAC critic loss with another loss term -
retrospective loss - leading to faster critic convergence and consequently,
better policy gradient estimates for the actor. An existing implementation of
SAC can be easily adapted to SARC with minimal modifications. Through
extensive experimentation and analysis, we show that SARC provides consistent
improvement over SAC on benchmark environments. We plan to open-source the
code and all experiment data at https://github.com/sukritiverma1996/SARC.

##  1 Introduction

The idea of applying and extending successful supervised learning techniques
from the field of deep learning to the field of reinforcement learning has
long existed and helped achieve remarkable success in deep RL over this
decade. One of the early breakthroughs, Deep Q-Networks (DQN) [12, 13],
leveraged advances from the field of deep learning to train an end-to-end deep
reinforcement learning agent, which achieved better than human-level
performance in several challenging Atari games. Before DQNs, reinforcement
learning could only be applied to tasks having low-dimensional state spaces or
tasks where a small number of features could be extracted. With DQNs, it
became possible to learn policies directly from high-dimensional input. This
subsequently gave rise to a plethora of deep RL algorithms that combined deep
learning with reinforcement learning. Broadly, these algorithms fall into
three classes - critic-only (like DQN, DDQN [23] etc.), actor-only (REINFORCE
[26]) and actor-critic (SAC [6], PPO [16] etc.) algorithms, In this work, we
seek to improve the Soft Actor-Critic Algorithm (SAC) [6], which is one of the
current state-of-the-art actor-critic algorithms in Deep RL, by taking
inspiration from a recent advance in the field of supervised deep learning -
Retrospective Loss.

Actor-critic methods such as PPO, A3C [14], DPG [17], TD3 [5] and SAC have
yielded great success in recent times [24, 2]. With actor-critic algorithms,
the central idea is to split the learning into two key modules: i) Critic,
which learns the policy-value function given the actor’s policy. ii) Actor,
which uses the critic’s policy-value function to estimate a policy gradient
and improve its policy.

One iteration of policy improvement in actor-critic methods [10] involves two
interwoven steps: one step of critic learning followed by one step of actor
learning. These steps are interwoven for the following reason. Given a fixed
critic, policy improvement and actor learning is determined by policy gradient
estimates computed using the policy-value function provided by the critic.
Given a fixed actor, the policy remains unchanged for the critic and hence,
there exists a unique value function for the critic to learn. This allows the
application of policy evaluation algorithms to learn this value function. A
good value function, in turn enables accurate policy gradient estimates for
the actor [18, 27].

Several advances in actor-critic methods such as TRPO [15], PPO, A3C and DDPG
[11] have come from improving actor learning and discovering more stable ways
of updating the policy. A few actor-critic algorithms, such as TD3 and SAC,
have advanced state-of-the-art by improving critic learning. Both of these
algorithms learn two value functions rather than one to counter the
overestimation bias [21] in value learning. TD3 updates the critic more
frequently than the actor to minimize the error in the value function [5]. SAC
uses the entropy of the policy as a regulariser [6]. In this work, we propose
to further improve critic learning in SAC by applying a regulariser that
accelerates critic convergence.

This regulariser is inspired from a recent technique that improves performance
in the supervised learning setting called retrospective loss [8]. When
retrospective loss is minimized along with the task-specific loss, the
parameters at the current training step are guided towards the optimal
parameters while being pulled away from the parameters at a previous training
step. In the supervised setting, retrospective loss results in better test
accuracy.

Using this loss as a retrospective regulariser applied to the critic,
accelerates the convergence of the critic. Due to the two time-scale nature of
actor-critic algorithms [3, 10], causing the critic to learn faster leads to
better policy gradient estimates for the actor, as gradient estimates depend
on the value function learned by the critic. Better policy gradient estimates
for the actor in turn lead to better future samples.

Bringing it all together, in this work, we propose a novel actor-critic
method, Soft Actor Retrospective Critic (SARC), which improves the existing
Soft Actor-Critic (SAC) method by applying the aforementioned retrospective
regulariser on the critic. We perform extensive empirical demonstration and
analysis to show that SARC leads to better actors in terms of sample
complexity as well as final return in several standard benchmark RL tasks
provided in the DeepMind Control Suite and PyBullet Environments. We also show
how SARC does not degrade performance for tasks where SAC already achieves
optimal performance. To help reproducibility and to help convey that it takes
minimal editing and essentially no added computation cost to convert an
existing implementation of SAC to SARC, we open-source the code and all
experiment data at https://github.com/sukritiverma1996/SARC.

##  2 Related Work

Out of the three broad classes of RL algorithms, critic-only methods can
efficiently learn an optimal policy [9] (implicitly derived from the optimal
action value function) under tabular setting. But under continuous or
stochastic settings, function approximation can cause critic-only methods to
diverge [25]. Actor-only methods can be applied to these settings but they
suffer from high variance when computing the policy gradient estimates. Actor-
critic algorithms combine the advantages of critic-only methods and actor-only
methods. The key difference between actor-critic and actor-only algorithms is
that actor-critic algorithms estimate an explicit critic or policy-value
function instead of the Monte Carlo return estimated by the actor-only
algorithms. In the sequel, we restrict attention to actor-critic algorithms. A
more detailed comparison can be found in Sutton and Barto [18].

Methods that model an explicit policy, such as actor-only and actor-critic
algorithms, consist of two steps. i) Policy evaluation: Calculating the
policy-value function for the current policy, and ii) Policy improvement:
Using the value function to learn a better policy.

It is customary in RL to run these steps simultaneously utilizing the two
time-scale stochastic approximation algorithm [3, 19, 10] rather than running
these steps to convergence individually. This is the approach used in actor-
critic algorithms, where the critic learns the policy-value function given the
actor’s policy and the actor improves it’s policy guided by policy gradient
estimates computed using the policy-value function provided by the critic.

RL algorithms can be on-policy or off-policy. Off-policy actor-critic
algorithms like Deep Deterministic Policy Gradient (DDPG) [11] and Twin
Delayed DDPG (TD3) [5] often have a better sample complexity than on-policy
algorithms. DDPG is a deep variant of the deterministic policy gradient
algorithm [17]. It combines the actor-critic approach with insights from DQN.
However, DDPG suffers from an overestimation bias [21] in learning the
Q-values. This hinders policy improvement. TD3 addresses this by learning two
Q-functions and using the smaller of the two to compute the policy gradient.
This favors underestimations. TD3 also proposes delaying policy updates until
the value estimates have converged.

On-policy algorithms provide better stability but are not as sample efficient
and often employ replay buffers to improve sample efficiency. Trust Region
Policy Optimization (TRPO) [15] proposes updating the policy by optimizing a
surrogate objective, constrained on the KL-divergence between the previous
policy and the updated policy. Actor Critic using Kronecker-Factored Trust
Region (ACKTR) [28] uses kronecker-factored approximated curvature to perform
a similar trust region update. Proximal Policy Optimization (PPO) [16]
proposes to match the performance of TRPO with only first-order optimization.

Soft Actor-Critic (SAC) [6] is an off-policy algorithm that has both the
qualities of sample efficiency and stability. SAC optimizes a stochastic
policy with entropy regularization. The policy is trained to maximize a trade-
off between expected return and entropy. Increasing entropy results in more
exploration, which can help accelerate learning. It can also prevent the
policy from converging to a local optimum.

#### Our contribution

In this work, we propose Soft Actor Restrospective Critic (SARC), an algorithm
that improves SAC by applying a retrospective regulariser on the critic
inspired from retrospective loss [8]. We empirically validate this claim by
comparing and analysing SARC with SAC, TD3 and DDPG on multiple continuous
control environments.

##  3 Soft Actor Retrospective Critic

###  3.1 Premise

In actor-critic methods, we note that the actor learning algorithm
incrementally improves its policy, using the critic’s performance estimate,
while the best performance is unknown. However, in contrast to this, the
critic learning algorithm is a supervised learning one albeit with a moving
target. This is the key observation that enables us to import retrospective
loss from supervised learning to actor-critic methods.

#### Retrospective loss and critic learning

We begin by describing our notation. Let
$(x^{i},y^{i})_{i=1}^{n_{d}}\in\mathcal{D}$ denote an input and output (ground
truth) training data point, with $n_{d}=|\mathcal{D}|$ denoting the size of
the training data and $x^{i},y^{i}$ belonging to the spaces
$\mathcal{X}\subset\mathbb{R}^{n_{x}}$ and
$\mathcal{Y}\subset\mathbb{R}^{n_{y}}$ respectively, where
$n_{x},n_{y}\in\mathbb{N}$ are the respective dimensions of the input and
output. Furthermore, let $G_{\theta}:\mathcal{X}\to\mathcal{Y}$ denote a
mapping from the input space to the output space, parameterized by
$\theta\in\mathbb{R}^{n_{\theta}}$, where $n_{\theta}\in\mathbb{N}$ denotes
the number of parameters in this mapping. Let
$\hat{y}^{i}_{\theta}=G_{\theta}(x^{i})$ denote the predicted output for the
data point $x^{i}$ using parameter $\theta$. The supervised learning objective
is to learn a parameter $\theta^{*}$, which minimizes the error between the
ground truth output $y^{i}$ and the predicted output
$\hat{y}^{i}_{\theta^{*}}$ for all $i\in\\{1,\dots,n_{d}\\}$. Various metrics
can be used to represent this loss, but for our illustration, we choose a loss
based on the $L^{2}$ metric. We call this loss as the original loss,
$\mathcal{L}_{\texttt{ori}}(\theta)$, which is defined as:

$\mathcal{L}_{\texttt{ori}}(\theta)=\frac{1}{n_{d}}\sum_{i=1}^{n_{d}}{(y^{i}-\hat{y}^{i}_{\theta})}^{2}$
(1)

We paraphrase the definition of retrospective loss from [8], and modify it
slightly as follows:

######  Definition 1.

[adapted from [8]] Retrospective loss is defined in the context of an
iterative parameter estimation scheme and is a function of the parameter
values at two different iterations (times)—the current iteration and a
previous iteration. Let $t,p$ denote the current and previous times
respectively, and we so have $p<t$. Then, the retrospective loss at time $t$,
is comprised of two components - the original loss and the retrospective
regularizer as given below:

$\displaystyle\mathcal{L}_{\texttt{tot}}(\theta_{t},\theta_{p},(x^{i},y^{i})_{i=1}^{n_{d}})$
$\displaystyle=\mathcal{L}_{\texttt{ori}}(\theta_{t},(x^{i},y^{i})_{i=1}^{n_{d}})$
$\displaystyle+\mathcal{L}_{\texttt{ret}}(\theta_{t},\theta_{p},(x^{i},y^{i})_{i=1}^{n_{d}}),$
(2)

where the retrospective regularizer $\mathcal{L}_{\texttt{ret}}$ is defined
as:

$\displaystyle\mathcal{L}_{\texttt{ret}}(\theta_{t},\theta_{p},(x^{i},y^{i})_{i=1}^{n_{d}})=$
$\displaystyle\frac{1}{n_{d}}\sum_{i=1}^{n_{d}}\Bigl{(}(\kappa+1)\textup{d}_{\texttt{ret}}(y^{i},\hat{y}^{i}_{\theta_{t}})-\kappa\textup{d}_{\texttt{ret}}(\hat{y}^{i}_{\theta_{t}},\hat{y}^{i}_{\theta_{p}})\Bigr{)},$
(3)

where $\kappa>0$ and $\textup{d}_{\texttt{ret}}(\cdot,\cdot)$ is any metric on
vector spaces, such as $L^{1}$. We term the family of functions as given in
(1), defined by parameters $\kappa$ and the metric
$\textup{d}_{\texttt{ret}}(\cdot,\cdot)$ as retrospective loss functions.

When retrospective loss is minimized along with the task-specific loss, the
parameters at the current training step are guided towards the optimal
parameters while being pulled away from the parameters at a previous training
step. In the supervised setting, retrospective loss results in better test
accuracy.

We posit that accelerating critic learning can help actor realise better
gradient estimates which results in improved policy learning. In lieu with
this goal, we propose that using the retrospective loss as a regularizer in
the critic objective will accelerate critic learning. In Section 5, we present
experiments that demonstrate improved policy learning for the actor achieved
with the use of retrospective loss as a regularizer in the critic objective.
Furthermore, in Section 6 we observe that even for slowly moving targets
(ground truths), as is the case for the critic, retrospective loss yields
convergence acceleration.

###  3.2 Formalising SARC

As the popular Soft Actor-Critic (SAC) algorithm offers both the qualities of
sample efficient learning as well as stability [6] (Section 2), we therefore,
specifically extend SAC in this work. In principle, retrospective
regularization can be incorporated with other actor critic methods as well and
would be an interesting direction for future exploration. Our algorithm is
summarized in Algorithm 1, where the basic SAC code is reproduced from
SpinningUp [1] and the additions are highlighted in blue. From an algorithmic
perspective, SARC is a minimal modification (3 lines) over SAC.

1: Input: initial policy parameters $\theta$, Q-function parameters
$\phi_{1}$, $\phi_{2}$, empty replay buffer $\mathcal{D}$

2: Set target parameters equal to main parameters
$\phi_{\text{targ},1}\leftarrow\phi_{1}$,
$\phi_{\text{targ},2}\leftarrow\phi_{2}$

3: Previous Q-function parameters set to initial
$\phi_{prev,1}\leftarrow\phi_{1},\phi_{prev,2}\leftarrow\phi_{2}$

4: repeat

5: Observe state $s$, sample $a\sim\pi_{\theta}(.|s)$

6: Execute $a$ in environment

7: Observe next state $s^{\prime}$, reward $r$, and done signal $d$ to
indicate whether $s^{\prime}$ is terminal

8: Store $(s,a,s^{\prime},r,d)$ in replay buffer $D$

9: If $d$ = 1 then $s^{\prime}$ is terminal, reset environment

10: if it’s time to update then

11: for $j$ in range numUpdates do

12: Sample batch, $B$ = ${(s,a,s^{\prime},r,d)}$ from $D$

13: Compute targets for Q-functions
$\displaystyle\tilde{a}^{\prime}\sim\pi_{\theta}(\cdot|s^{\prime})$
$\displaystyle
y(r,s^{\prime},d)=r+\gamma(1-d)\left(\min_{i=1,2}Q_{\phi_{\text{targ},i}}(s^{\prime},\tilde{a}^{\prime})-\alpha\log\pi_{\theta}(\tilde{a}^{\prime}|s^{\prime})\right)$
$\displaystyle\text{MSE-
Loss}(s,a,s^{\prime},r,d)=\left(Q_{\phi_{i}}(s,a)-y(r,s^{\prime},d)\right)^{2}\text{,
for }i=1,2$

14: Update Q-functions
$\displaystyle\nabla_{\phi_{i}}\frac{1}{|B|}\Bigl{(}\sum_{(s,a,r,s^{\prime},d)\in
B}\text{MSE-Loss}(s,a,r,s^{\prime},d)$
$\displaystyle+\mathcal{L}_{\texttt{ret}}(\phi_{i},\phi_{prev,i},y(r,s^{\prime},d))\Bigr{)}\text{,
for }i=1,2$

15: Update policy  $\displaystyle\nabla_{\theta}\frac{1}{|B|}\sum_{s\in B}$
$\displaystyle\Big{(}\min_{i=1,2}Q_{\phi_{i}}(s,\tilde{a}_{\theta}(s))-$
$\displaystyle\alpha\log\pi_{\theta}\left(\left.\tilde{a}_{\theta}(s)\right|s\right)\Big{)},$
where $\tilde{a}_{\theta}(s)$ is a sample from $\pi_{\theta}(\cdot|s)$ which
is differentiable wrt $\theta$ via the reparametrization trick.

16: Update target Q-functions
$\displaystyle\phi_{targ,i}\leftarrow\rho\phi_{targ,i}+(1-\rho)\phi_{i}$
$\displaystyle\text{for }i=1,2$

17: Update previous Q-functions $\displaystyle\phi_{prev,i}\leftarrow\phi_{i}$
$\displaystyle\text{for }i=1,2$

18: end for

19: end if

20: until convergence

Algorithm 1 Soft Actor Retrospective Critic

#### Learning in SARC

Here we reproduce the basic SAC actor and critic learning algorithms [6, 1]
with the retrospective regularizer added to the critic loss, which yields the
SARC algorithm. The update equations are as given below. The actor is
represented by a function $\pi$, parametrized with $\theta$. Given any state
$s^{\prime}$, the action $\tilde{a}^{\prime}$ is sampled as:

$\tilde{a}^{\prime}\sim\pi_{\theta}(\cdot|s^{\prime}).$ (4)

The targets for the two critics in SARC are then computed as in SAC:

$\displaystyle y(r,s^{\prime},d)=r+\gamma(1-d)\Bigl{(}\min_{i=1,2}$
$\displaystyle
Q_{\phi_{\text{targ},i}}(s^{\prime},\tilde{a}^{\prime})-\alpha\log\pi_{\theta}(\tilde{a}^{\prime}|s^{\prime})\Bigr{)}$
(5)

where $y$ is the target, $r$ is the per-step reward function, $d$ is the done
signal, $\phi_{i}$ are the critic parameters, and $\phi_{\text{targ, i}}$ are
the corresponding critic parameters for the target. The loss for the critic is
then estimated as:

$\displaystyle\mathcal{L}$
$\displaystyle=\bigl{(}Q_{\phi_{i}}(s,a)-y(r,s^{\prime},d)\bigr{)}^{2}$
$\displaystyle+\mathcal{L}_{\texttt{ret}}(\phi_{i},\phi_{prev,i},y(r,s^{\prime},d))\text{,
for }i=1,2$ (6)

where $\mathcal{L}_{\texttt{ret}}$ is given by (1). And finally, the actor is
updated as in the original SAC algorithm.

##  4 Experimental Setup

To demonstrate that SARC outperforms SAC and is competitive with existing
state-of-the-art actor-critic methods such as TD3 and DDPG, we conduct an
exhaustive set of experiments. For all of our experiments, we use the
implementations provided in SpinningUp [1]. SpinningUp is an open-source deep
RL library.

We modify the original implementation of SAC, provided in SpinningUp, to SARC.
Modifying an existing SAC implementation to SARC is minimal and
straightforward, requiring addition of a few lines and essentially no compute
as presented in Algorithm 1. Within SARC critic loss, for the retrospective
regularizer, we use $\kappa=2$ across all tasks, environments and analyses.

We conduct our evaluations and analysis across various continuous control
tasks. Concretely, we use 6 tasks provided in the DeepMind Control Suite [20]
and 5 tasks provided in PyBullet Environments [4]. For all experiments, we
retain all of the default hyperparameters specified originally in the
SpinningUp library and let each agent train for 2 million timesteps in the
given environment. In our experiments, the same hyperparameters worked for all
environments and tasks. The actor and critic network size used in all
experiments is [256, 256], unless specified otherwise. This is the default
value for the network size specified originally in the SpinningUp library. All
models were trained using CPUs on a Linux machine running Ubuntu 16.04 with
2nd Generation Intel Xeon Gold Processor.

In Section 5, we compare our proposed algorithm SARC to SAC, TD3 and DDPG
across 6 tasks provided in the DeepMind Control Suite. In Section 6, we
compare retrospective loss with an alternate baseline technique for critic
acceleration and analyse our hypothesis of faster critic convergence. We also
compare SARC with SAC under various modified experimental settings to further
establish our claim. We make all of our code available at:
https://github.com/sukritiverma1996/SARC. We also make all of our data used to
construct the graphs and report the figures available at:
https://github.com/sukritiverma1996/SARC.

##  5 Results

(a) Walker-Stand

(b) Walker-Walk

(c) Finger-Spin

(d) Cheetah-Run

(e) Reacher-Easy

(f) Reacher-Hard

Figure 1: Results for SARC (red), SAC (green), TD3 (purple) and DDPG (blue)
across 6 DeepMind Control Suite Tasks. The x-axis shows the timesteps in each
environment. The y-axis shows the average value and standard deviation band of
return across 5 seeds (10 test episode returns per seed). It can be observed
that SARC improves SAC in all environments. SARC also outperforms TD3 and DDPG
in most environments.

In this section, we present comparisons between SARC, SAC, TD3 and DDPG. We
perform these evaluations on 6 DeepMind Control Suite tasks: Cheetah-Run,
Finger-Spin, Walker-Walk, Walker-Stand, Reacher-Easy, Reacher-Hard. To ensure
fairness, consistency and, to rule out outlier behavior, we run five trials
for each experiment presented in this paper, each with a different preset
random seed. Each experiment uses the same set of 5 seeds.

Table 1: Test episode returns of SARC, SAC, TD3 and DDPG on 6 DeepMind Control Suite tasks after training for 2M steps. Environment | Average Test Episode Return
---|---
TD3 | DDPG | SAC | SARC
Walker-Stand | 959.48 | 716.99 | 697.33 | 970.8
Walker-Walk | 797.87 | 770.08 | 886.28 | 961.67
Finger-Spin | 827.02 | 179.3 | 315.28 | 970.94
Cheetah-Run | 789.31 | 500.52 | 8.22 | 382.69
Reacher-Easy | 799.62 | 913.92 | 978.52 | 978.32
Reacher-Hard | 771.06 | 497.4 | 973.08 | 970.02

Table 1 shows the average value across 5 seeds (10 test episode returns per
seed) obtained for each of the 6 tasks after training for 2 million timesteps.
It can be observed that, at the end of training, SARC outperforms or is at par
with existing approaches on 5 out of 6 environments presented in bold.

In figure 1, we present the mean Monte Carlo returns over 10 test episodes at
various steps during training. The x-axis shows the timesteps in each
environment and the y-axis shows the mean and standard deviation band of above
specified returns across 5 seeds.

It can be observed that, on Finger-Spin, Walker-Stand and Walker-Walk, SARC
achieves a higher return value faster than any other algorithm. On Reacher-
Easy and Reacher-Hard, SARC overlaps with SAC and continues to provide
improvement over TD3 and DDPG. SARC substantially improves SAC, particularly
on Walker-Stand, Walker-Walk, Cheetah-Run, and Finger-Spin. To ensure and
demonstrate that SARC provides a consistent improvement over SAC even under
modified experimental settings, we restrict further analyses to these 4
environments. The experiments as described in this section have also been
performed on 5 tasks provided in PyBullet Environments [4]. These can be found
in Appendix A.

##  6 Discussion

First, we explore the effect of alternate baseline strategy to accelerate
critic convergence in SAC and compare it with SARC. Next, we present evidence
of accelerated critic convergence in SARC. Finally, we demonstrate that SARC
significantly outperforms SAC for increased network sizes. We note that, for
ease of exposition, we restrict the comparative scope of this analysis to SAC,
which is the most similar baseline and also learns a soft critic.

###  6.1 Increasing critic update frequency for SAC

(a) Finger-Spin

(b) Cheetah-Run

(c) Walker-Stand

Figure 2: Comparing SARC and delayed SAC. The x-axis shows the timesteps in
each environment. The y-axis shows the average value and standard deviation
band of return across 5 seeds (10 test episode returns per seed). Compared to
delayed SAC (yellow), SARC (red) provides a more reliable and potent
improvement over SAC.

An alternate baseline strategy to accelerate convergence of the critic
relative to the actor can be to update critic more frequently. A similar idea
has been proposed in TD3 [5], where the critic updates more frequently than
the actor.

In this section, we compare SARC against a version of SAC where the critic
updates twice as frequently as the actor. We term this as "delayed SAC". We do
this for three environments: Finger-Spin, Cheetah-Run and Walker-Stand. Figure
2 has the results of this investigation. The x-axis shows the timesteps in
each environment and the y-axis shows the mean and standard deviation band of
Mean Monte Carlo returns over 10 test episodes across 5 seeds. It can be
observed that while delayed SAC provides some improvement over original SAC,
it is not as reliable and not as potent as the improvement provided by SARC.

###  6.2 Modifying entropy regularization coefficient

In this section, we explore the effect of modifying the entropy regularization
coefficient, $\alpha$ in SAC and SARC. It may be the case that the significant
performance gain that SARC is offering over SAC may disappear if the entropy
regularizer, $\alpha$, is tuned.

All experiments presented in Section 5 had value of $\alpha=0.2$. In this
section, we look at two more values of $\alpha$: 0.1 and 0.4. All other
hyperparameters are held at fixed values as described in Section 4. We do this
for three environments: Finger-Spin, Walker-Stand and Walker-Walk. Figure 3
has the results of this investigation. Mean Monte Carlo returns over 10 test
episodes at various steps during training have been plotted. The x-axis shows
the timesteps in each environment and the y-axis shows the mean and standard
deviation band of above specified returns across 5 seeds. It can be observed
that SARC continues to provide consistent improvement over SAC across values
of $\alpha$.

(a) Finger-Spin, $\alpha=0.1$

(b) Walker-Stand, $\alpha=0.1$

(c) Walker-Walk, $\alpha=0.1$

(d) Finger-Spin, $\alpha=0.4$

(e) Walker-Stand, $\alpha=0.4$

(f) Walker-Walk, $\alpha=0.4$

Figure 3: Results for SARC (red) and SAC on tuning the entropy regularization
coefficient, $\alpha$. The x-axis shows the timesteps in each environment. The
y-axis shows the average value and standard deviation band of return across 5
seeds (10 test episode returns per seed). It can be seen that SARC outperforms
SAC across different values of $\alpha$.

###  6.3 Tracking SARC critic convergence with respect to Monte Carlo policy
evaluation

We train one SAC agent and one SARC agent for 2 million timesteps. During
training, we save the intermediate actor and critic at every 10,000 timesteps.
Given an intermediate actor-critic pair, we compute the mean squared
difference between Monte Carlo estimate of the return of the current policy
and Q($s_{0}$,$a_{0}$) learned by the critic for that policy. This difference
is termed as “Q-Error”. If the critic value function has converged to the
correct value for a given policy, then this difference should be zero. The
idea is to track and compare how fast the critic converges.

We do this for three environments: Finger-Spin, Cheetah-Run, Walker-Stand.
Figure 4 has the results of this investigation. The average “Q-Error” over 5
seeds has been plotted. The graphs also show the standard deviation band
across the 5 random seeds. It can be seen that SARC critic converges faster
than the SAC critic on Finger-Spin and Walker-Stand.

(a) Finger-Spin

(b) Cheetah-Run

(c) Walker-Stand

Figure 4: Tracking critic convergence with respect to Monte Carlo policy
evaluation across training. The x-axis shows the timesteps in each
environment. The y-axis shows the mean and standard deviation band of
“Q-Error” across 5 seeds. It can be seen that SARC (red) critic converges
faster.

###  6.4 Increasing actor and critic network complexity

It may be possible that SAC can perform as well as SARC with a bigger network.
In this section, we compare SAC and SARC on tasks from DeepMind Control Suite
after increasing the network sizes for actor and critic to [400, 300]. This is
another commonly found architecture in literature relating to continuous
control [7]. In our previous experiments, we were using network size [256,
256].

In figure 5, we present the mean Monte Carlo returns over 10 test episodes at
various steps during training. The x-axis shows the timesteps in each
environment and the y-axis shows the mean and standard deviation band of above
specified returns across 5 seeds. It can be observed that even on increasing
the network sizes, SARC continues to provide consistent improvement in episode
return value over SAC. The experiments as described in this section have also
been performed on 5 tasks provided in PyBullet Environments [4]. These can be
found in Appendix B.

(a) Walker-Stand

(b) Walker-Walk

(c) Finger-Spin

(d) Cheetah-Run

(e) Reacher Easy

(f) Reacher Hard

Figure 5: Even after increasing network size to [400,300], SARC (red)
continues to provide consistent gains over SAC. The x-axis shows the timesteps
in each environment. The y-axis shows the average value and standard deviation
band of return across 5 seeds (10 test episode returns per seed).

##  7 Conclusion and Future Work

In this work, we presented Soft Actor Retrospective Critic (SARC). SARC builds
on and improves SAC by applying a retrospective regulariser on the critic.
This retrospective regulariser is inspired from retrospective loss, a recent
technique that improves accuracy in the supervised setting. We proposed that
using the retrospective loss as a regularizer in the critic objective
accelerates critic learning. Due to the two-time scale nature of actor-critic
algorithms, accelerating critic convergence can provide a better gradient
estimate for the actor throughout training and improve final performance. We
validated our claims of improvement in final performance through extensive
experimental evaluation and analysis on multiple continuous control
environments. We also compared retrospective loss with an alternate baseline
technique for critic acceleration. We presented evidence of critic
acceleration in SARC. We also tuned the SAC entropy regularizer, $\alpha$ to
demonstrate that SARC continues providing performance gain across different
values of $\alpha$. As a future direction, we aim to extend retrospective
regularization to other actor-critic algorithms.

## References

  * Achiam [2018] Joshua Achiam.  Spinning Up in Deep Reinforcement Learning.  2018\.
  * Bahdanau et al. [2016] Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio.  An actor-critic algorithm for sequence prediction.  _arXiv preprint arXiv:1607.07086_ , 2016.
  * Borkar and Konda [1997] Vivek S Borkar and Vijaymohan R Konda.  The actor-critic algorithm as multi-time-scale stochastic approximation.  _Sadhana_ , 22(4):525–543, 1997.
  * Coumans and Bai [2016] Erwin Coumans and Yunfei Bai.  Pybullet, a python module for physics simulation for games, robotics and machine learning.  2016\.
  * Fujimoto et al. [2018] Scott Fujimoto, Herke Hoof, and David Meger.  Addressing function approximation error in actor-critic methods.  In _International Conference on Machine Learning_ , pages 1587–1596. PMLR, 2018.
  * Haarnoja et al. [2018] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In _International Conference on Machine Learning_ , pages 1861–1870. PMLR, 2018.
  * Henderson et al. [2018] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger.  Deep reinforcement learning that matters.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32, 2018.
  * Jandial et al. [2020] Surgan Jandial, Ayush Chopra, Mausoom Sarkar, Piyush Gupta, Balaji Krishnamurthy, and Vineeth Balasubramanian.  Retrospective loss: Looking back to improve training of deep neural networks.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 1123–1131, 2020.
  * Jin et al. [2018] Chi Jin, Zeyuan Allen-Zhu, Sebastien Bubeck, and Michael I Jordan.  Is q-learning provably efficient?  _arXiv preprint arXiv:1807.03765_ , 2018.
  * Konda and Tsitsiklis [2000] Vijay R Konda and John N Tsitsiklis.  Actor-critic algorithms.  In _Advances in neural information processing systems_ , pages 1008–1014. Citeseer, 2000.
  * Lillicrap et al. [2015] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra.  Continuous control with deep reinforcement learning.  _arXiv preprint arXiv:1509.02971_ , 2015.
  * Mnih et al. [2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller.  Playing atari with deep reinforcement learning.  _arXiv preprint arXiv:1312.5602_ , 2013.
  * Mnih et al. [2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.  Human-level control through deep reinforcement learning.  _nature_ , 518(7540):529–533, 2015.
  * Mnih et al. [2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu.  Asynchronous methods for deep reinforcement learning.  In _International conference on machine learning_ , pages 1928–1937. PMLR, 2016.
  * Schulman et al. [2015] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz.  Trust region policy optimization.  In _International conference on machine learning_ , pages 1889–1897. PMLR, 2015.
  * Schulman et al. [2017] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Silver et al. [2014] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller.  Deterministic policy gradient algorithms.  In _International conference on machine learning_ , pages 387–395. PMLR, 2014.
  * Sutton and Barto [2018] Richard S Sutton and Andrew G Barto.  _Reinforcement learning: An introduction_.  MIT press, 2018.
  * Sutton et al. [1999] Richard S Sutton, David A McAllester, Satinder P Singh, Yishay Mansour, et al.  Policy gradient methods for reinforcement learning with function approximation.  In _NIPs_ , volume 99, pages 1057–1063. Citeseer, 1999.
  * Tassa et al. [2018] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al.  Deepmind control suite.  _arXiv preprint arXiv:1801.00690_ , 2018.
  * Thrun and Schwartz [1993] Sebastian Thrun and Anton Schwartz.  Issues in using function approximation for reinforcement learning.  In _Proceedings of the Fourth Connectionist Models Summer School_ , pages 255–263. Hillsdale, NJ, 1993.
  * Todorov et al. [2012] E. Todorov, T. Erez, and Y. Tassa.  Mujoco: A physics engine for model-based control.  In _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , pages 5026–5033, 2012.  doi: 10.1109/IROS.2012.6386109.
  * Van Hasselt et al. [2016] Hado Van Hasselt, Arthur Guez, and David Silver.  Deep reinforcement learning with double q-learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 30, 2016.
  * Wang et al. [2016] Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas.  Sample efficient actor-critic with experience replay.  _arXiv preprint arXiv:1611.01224_ , 2016.
  * Wiering [2004] Marco A Wiering.  Convergence and divergence in standard and averaging reinforcement learning.  In _European Conference on Machine Learning_ , pages 477–488. Springer, 2004.
  * Williams [1992] Ronald J Williams.  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  _Machine learning_ , 8(3-4):229–256, 1992.
  * Wu et al. [2020] Yue Wu, Weitong Zhang, Pan Xu, and Quanquan Gu.  A finite time analysis of two time-scale actor critic methods.  _arXiv preprint arXiv:2005.01350_ , 2020.
  * Wu et al. [2017] Yuhuai Wu, Elman Mansimov, Shun Liao, Roger Grosse, and Jimmy Ba.  Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation.  _arXiv preprint arXiv:1708.05144_ , 2017.

##  Appendix A PyBullet Results for Section 5: Results

Table 2: Test episode returns of SARC and SAC on 5 PyBullet Environments after
training for 2M steps.

Environment | Avg Test Ep Return
---|---
SAC | SARC
Ant | 1581.59 | 2333.26
Half Cheetah | 1701.19 | 2734.0
Hopper | 2497.83 | 1975.58
Reacher | 20.55 | 20.05
Walked-2D | 1093.64 | 1761.26

(a) Hopper

(b) Reacher

(c) Ant

(d) Half Cheetah

(e) Walker-2D

Figure 6: Results for SARC (red) and SAC on 5 PyBullet Environments. The
x-axis shows the timesteps in each environment. The y-axis shows the average
value and standard deviation band of return across 5 seeds (10 test episode
returns per seed). It can be observed that SARC improves SAC in most
environments even after modifying the underlying physics engine.

We have demonstrated SARC using tasks from the DeepMind Control Suite which
uses the MuJoCo Physics Engine [22]. In this section, we present comparisons
on a set of environments that use the Bullet Physics Engine [4], an open-
source physics simulator. We perform these evaluations on 5 PyBullet [4]
environments: Ant, HalfCheetah, Hopper, Reacher, Walker-2D.

Table 2 shows the average value across 5 seeds (10 test episode returns per
seed) obtained for each of the 5 tasks after training for 2 million timesteps.
It can be observed that, at the end of training, SARC outperforms or is at par
with existing approaches on 4 out of 5 environments presented in bold.

In figure 6, we present the mean Monte Carlo returns over 10 test episodes at
various steps during training. The x-axis shows the timesteps in each
environment and the y-axis shows the mean and standard deviation band of above
specified returns across 5 seeds.

It can be observed that, on Ant, Half-Cheetah and Walker-2D, SARC achieves a
higher return value faster than SAC. On Hopper and Reacher, SARC overlaps with
and does not degrade the performance of SAC. Even on changing the underlying
physics simulator, SARC continues providing consistent improvement over SAC.

We also hope that using an open-source physics simulator will help with
reproducibility and reach of our work.

(a) AntBulletEnv-v0

(b) HalfCheetahBulletEnv-v0

(c) HopperBulletEnv-v0

(d) ReacherBulletEnv-v0

(e) Walker2DBulletEnv-v0

Figure 7: Results for SARC (red) and SAC on PyBullet Envrironments after
increasing Actor and Critic network sizes to [400,300]. Even after increasing
network size to [400,300], SARC (red) continues to provide consistent gains
over SAC. The x-axis shows the timesteps in each environment. The y-axis shows
the average value and standard deviation band of return across 5 seeds (10
test episode returns per seed)

##  Appendix B PyBullet Results for Section 6.4: Increasing actor and critic
network complexity

Figure 7 shows results for SARC and SAC on PyBullet Envrironments after
increasing Actor and Critic network sizes to [400,300]. Even after increasing
network size to [400,300], SARC continues to provide consistent gains over
SAC.