$\cdot\mid(\textnormal{{latent}}\\!:A);(\textnormal{{obs}}\\!:B)\vdash_{\Sigma}m_{\mathsf{m}}\mathbin{\textnormal{\ooalign{\vphantom{$\div$}\raisebox{4.09024pt}{\scalebox{1.2}{.}}\cr$\sim$\cr\vphantom{$\div$}\raisebox{-0.21529pt}{\scalebox{1.2}{.}}\cr}}}\tau_{\mathsf{m}}\mid(\textnormal{{latent}}\\!:\boldsymbol{1});(\textnormal{{obs}}\\!:\boldsymbol{1}).$

Usually, the program $m_{\mathsf{m}}$ does _not_ receive any branch
selections, i.e., $A$ is $\mathbin{\varoplus}$-free and $B$ is
$\mathbin{\binampersand}$-free. Given a concrete observation $\sigma_{o}:B$,
Bayesian inference is the problem of approximating the _posterior_ $\llbracket
m_{\mathsf{m}}\rrbracket_{\sigma_{o}}$, a measure _conditioned_ with respect
to $\sigma_{o}$, defined by

$\llbracket
m_{\mathsf{m}}\rrbracket_{\sigma_{o}}(S_{\ell})\mathrel{\overset{\underset{\textnormal{def}}{}}{=}}\frac{\int_{S_{\ell}}\mathbf{P}_{m_{\mathsf{m}}}(\sigma_{\ell},\sigma_{o})\lambda_{\llbracket
A\rrbracket}(d\sigma_{\ell})}{\int\mathbf{P}_{m_{\mathsf{m}}}(\sigma_{\ell},\sigma_{o})\lambda_{\llbracket
A\rrbracket}(d\sigma_{\ell})},$

where $S_{\ell}$ is a measurable set in $\llbracket A\rrbracket$, i.e., a set
of guidance traces of type $A$.

#### Guide programs

In our system, we implement a guide program $m_{\mathsf{g}}$ as a coroutine
that works with the model program $m_{\mathsf{m}}$ and provides the latent
channel with guide type $A$ that $m_{\mathsf{m}}$ consumes, i.e.,

$\displaystyle\cdot\mid\varnothing;(\textnormal{{latent}}\\!:A)\vdash_{\Sigma}m_{\mathsf{g}}\mathbin{\textnormal{\ooalign{\vphantom{$\div$}\raisebox{4.09024pt}{\scalebox{1.2}{.}}\cr$\sim$\cr\vphantom{$\div$}\raisebox{-0.21529pt}{\scalebox{1.2}{.}}\cr}}}\tau_{\mathsf{g}}\mid\varnothing;(\textnormal{{latent}}\\!:\boldsymbol{1}),$
$\displaystyle\cdot\mid(\textnormal{{latent}}\\!:A);(\textnormal{{obs}}\\!:B)\vdash_{\Sigma}m_{\mathsf{m}}\mathbin{\textnormal{\ooalign{\vphantom{$\div$}\raisebox{4.09024pt}{\scalebox{1.2}{.}}\cr$\sim$\cr\vphantom{$\div$}\raisebox{-0.21529pt}{\scalebox{1.2}{.}}\cr}}}\tau_{\mathsf{m}}\mid(\textnormal{{latent}}\\!:\boldsymbol{1});(\textnormal{{obs}}\\!:\boldsymbol{1}).$

The coroutine-based paradigm folds the model and guide programs into a single
entity; thus, during the inference, both the model and guide coroutines
execute. However, to distinguish the two measures defined by the model and
guide, respectively, we define a denotation for the guide $m_{\mathsf{g}}$,
accompanied by the model $m_{\mathsf{m}}$ and conditioned on a concrete
observation $\sigma_{o}:B$, as a measure defined by

$\llbracket
m_{\mathsf{g}}\rrbracket^{m_{\mathsf{m}}}_{\sigma_{o}}(S_{\ell})\mathrel{\overset{\underset{\textnormal{def}}{}}{=}}\int_{S_{\ell}}[\mathbf{P}_{m_{\mathsf{m}}}(\sigma_{\ell},\sigma_{o})\neq
0]\cdot\mathbf{P}_{m_{\mathsf{g}}}(\sigma_{\ell})\lambda_{\llbracket
A\rrbracket}(d\sigma_{\ell}),$

where $S_{\ell}$ is a measurable set in $\llbracket A\rrbracket$, i.e., a set
of guidance traces of type $A$.

To justify the inclusion of
$[\mathbf{P}_{m_{\mathsf{m}}}(\sigma_{\ell},\sigma_{o})\neq 0]$ in the
denotation of the guide program, we consider _possible_ traces for a model-
guide system. A combination of traces $(\sigma_{\ell},\sigma_{o})$ is said to
be _possible_ for the model program $m_{\mathsf{m}}$ and the guide program
$m_{\mathsf{g}}$, if
$\emptyset\mid(\textnormal{{latent}}\\!:\sigma_{\ell});(\textnormal{{obs}}\\!:\sigma_{o})\mathrel{\vdash_{\mathsf{red}}}m_{\mathsf{m}}\mathrel{\Downarrow}v_{\mathsf{m}}$
and
$\emptyset\mid\varnothing;(\textnormal{{latent}}\\!:\sigma_{\ell})\mathrel{\vdash_{\mathsf{red}}}m_{\mathsf{g}}\mathrel{\Downarrow}v_{\mathsf{g}}$
for some values $v_{\mathsf{m}}$ and $v_{\mathsf{g}}$.

###### Lemma 0 (Lem. 5.1).

Suppose that $A$ is $\mathbin{\varoplus}$-free, $B$ is
$\mathbin{\binampersand}$-free, and

$\displaystyle\cdot\mid\varnothing;(\textnormal{{latent}}\\!:A)$
$\displaystyle\vdash_{\Sigma}m_{\mathsf{g}}\mathbin{\textnormal{\ooalign{\vphantom{$\div$}\raisebox{4.09024pt}{\scalebox{1.2}{.}}\cr$\sim$\cr\vphantom{$\div$}\raisebox{-0.21529pt}{\scalebox{1.2}{.}}\cr}}}\tau_{\mathsf{g}}\mid\varnothing;(\textnormal{{latent}}\\!:\boldsymbol{1}),$
$\displaystyle\cdot\mid(\textnormal{{latent}}\\!:A);(\textnormal{{obs}}\\!:B)$
$\displaystyle\vdash_{\Sigma}m_{\mathsf{m}}\mathbin{\textnormal{\ooalign{\vphantom{$\div$}\raisebox{4.09024pt}{\scalebox{1.2}{.}}\cr$\sim$\cr\vphantom{$\div$}\raisebox{-0.21529pt}{\scalebox{1.2}{.}}\cr}}}\tau_{\mathsf{m}}\mid(\textnormal{{latent}}\\!:\boldsymbol{1});(\textnormal{{obs}}\\!:\boldsymbol{1}).$

Then a combination of traces $(\sigma_{\ell},\sigma_{o})$ is possible for the
model $m_{\mathsf{m}}$ and the guide $m_{\mathsf{g}}$ if and only if
$\mathbf{P}_{m_{\mathsf{m}}}(\sigma_{\ell},\sigma_{o})\neq 0$.

###### Proof.

  * •

The “if” direction:

$\mathbf{P}_{m_{\mathsf{m}}}(\sigma_{\ell},\sigma_{o})\neq 0$ (assumption)

$\emptyset\mid(\textnormal{{latent}}\\!:\sigma_{\ell});(\textnormal{{obs}}\\!:\sigma_{o})\vdash
m_{\mathsf{m}}\mathrel{\Downarrow^{w}}v$ for some $v,w$ such that $w>0$
(definition)

$\emptyset\mid(\textnormal{{latent}}\\!:\sigma_{\ell});(\textnormal{{obs}}\\!:\sigma_{o})\mathrel{\vdash_{\mathsf{red}}}m_{\mathsf{m}}\mathrel{\Downarrow}v$
(Cor. B.9)

$A$ is $\mathbin{\varoplus}$-free (assumption)

$\emptyset\mid\varnothing;(\textnormal{{latent}}\\!:\sigma_{\ell})\vdash
m_{\mathsf{g}}\mathrel{\Downarrow^{w^{\prime}}}v^{\prime}$ for some
$v^{\prime},w^{\prime}$ such that $w^{\prime}>0$ (Thm. 4.6)

$\emptyset\mid\varnothing;(\textnormal{{latent}}\\!:\sigma_{\ell})\mathrel{\vdash_{\mathsf{red}}}m_{\mathsf{g}}\mathrel{\Downarrow}v^{\prime}$
(Cor. B.9)

  * •

The “only if” direction:

$\emptyset\mid(\textnormal{{latent}}\\!:\sigma_{\ell});(\textnormal{{obs}}\\!:\sigma_{o})\mathrel{\vdash_{\mathsf{red}}}m_{\mathsf{m}}\mathrel{\Downarrow}v$
for some $v$ (assumption)

$\emptyset\mid(\textnormal{{latent}}\\!:\sigma_{\ell});(\textnormal{{obs}}\\!:\sigma_{o})\vdash
m_{\mathsf{m}}\mathrel{\Downarrow^{w}}v$ for some $w>0$ (Cor. B.9)

$\sigma_{\ell}:A$, $\sigma_{o}:B$ (Thm. 4.4)

$\mathbf{P}_{m_{\mathsf{m}}}(\sigma_{\ell},\sigma_{o})=w>0$ (definition)

∎

#### Absolute continuity

Recall that a measure $\mu$ is said to be _absolutely continuous_ with respect
to a measure $\nu$, if $\mu$ and $\nu$ are defined on the same measurable
space, and $\nu(S)\neq 0$ for every measurable set $S$ for which $\mu(S)\neq
0$.

We prove that for a model-guide pair, guide types serve as certificates for
absolute continuity.

###### Theorem 0 (Thm. 5.2).

Suppose that

$\displaystyle\cdot\mid\varnothing;(\textnormal{{latent}}\\!:A)$
$\displaystyle\vdash_{\Sigma}m_{\mathsf{g}}\mathbin{\textnormal{\ooalign{\vphantom{$\div$}\raisebox{4.09024pt}{\scalebox{1.2}{.}}\cr$\sim$\cr\vphantom{$\div$}\raisebox{-0.21529pt}{\scalebox{1.2}{.}}\cr}}}\tau_{\mathsf{g}}\mid\varnothing;(\textnormal{{latent}}\\!:\boldsymbol{1}),$
$\displaystyle\cdot\mid(\textnormal{{latent}}\\!:A);(\textnormal{{obs}}\\!:B)$
$\displaystyle\vdash_{\Sigma}m_{\mathsf{m}}\mathbin{\textnormal{\ooalign{\vphantom{$\div$}\raisebox{4.09024pt}{\scalebox{1.2}{.}}\cr$\sim$\cr\vphantom{$\div$}\raisebox{-0.21529pt}{\scalebox{1.2}{.}}\cr}}}\tau_{\mathsf{m}}\mid(\textnormal{{latent}}\\!:\boldsymbol{1});(\textnormal{{obs}}\\!:\boldsymbol{1}),$

$A$ is $\mathbin{\varoplus}$-free, $B$ is $\mathbin{\binampersand}$-free, and
$\sigma_{o}:B$ such that
$\int\mathbf{P}_{m_{\mathsf{m}}}(\sigma_{\ell},\sigma_{o})\lambda_{\llbracket
A\rrbracket}(d\sigma_{\ell})>0$. Then the measure $\llbracket
m_{\mathsf{m}}\rrbracket_{\sigma_{o}}$ is absolutely continuous with respect
to the measure $\llbracket
m_{\mathsf{g}}\rrbracket^{m_{\mathsf{m}}}_{\sigma_{o}}$, and vice versa.

###### Proof.

We first claim that for any $\sigma_{\ell}\in\llbracket A\rrbracket$, it holds
that $\mathbf{P}_{m_{\mathsf{m}}}(\sigma_{\ell},\sigma_{o})\neq 0$ if and only
if $[\mathbf{P}_{m_{\mathsf{m}}}(\sigma_{\ell},\sigma_{o})\neq
0]\cdot\mathbf{P}_{m_{\mathsf{g}}}(\sigma_{\ell})\neq 0$.

($\implies$):

Assume $\mathbf{P}_{m_{\mathsf{m}}}(\sigma_{\ell},\sigma_{o})\neq 0$. Thus,
there exists $v_{1}$ such that
$\emptyset\mid(\textnormal{{latent}}\\!:\sigma_{\ell});(\textnormal{{obs}}\\!:\sigma_{o})\vdash
m_{\mathsf{m}}\mathrel{\Downarrow^{w_{1}}}v_{1}$, where
$w_{1}\mathrel{\overset{\underset{\textnormal{def}}{}}{=}}\mathbf{P}_{m_{\mathsf{m}}}(\sigma_{\ell},\sigma_{o})$.
Because $A$ is $\mathbin{\varoplus}$-free, we can apply Thm. 4.6 to program
$m_{\mathsf{g}}$. Thus, there exist $w_{2},v_{2}$ such that
$\emptyset\mid\varnothing;(\textnormal{{latent}}\\!:\sigma_{\ell})\vdash
m_{\mathsf{g}}\mathrel{\Downarrow^{w_{2}}}v_{2}$ and $w_{2}>0$. In other
words, we have $\mathbf{P}_{m_{\mathsf{g}}}(\sigma_{\ell})>0$. Then, we
conclude that $[\mathbf{P}_{m_{\mathsf{m}}}(\sigma_{\ell},\sigma_{o})\neq
0]\cdot\mathbf{P}_{m_{\mathsf{g}}}(\sigma_{\ell})\neq 0$.

($\impliedby$):

Assume $[\mathbf{P}_{m_{\mathsf{m}}}(\sigma_{\ell},\sigma_{o})\neq
0]\cdot\mathbf{P}_{m_{\mathsf{g}}}(\sigma_{\ell})\neq 0$. Thus, we have both
$\mathbf{P}_{m_{\mathsf{m}}}(\sigma_{\ell},\sigma_{o})\neq 0$ and
$\mathbf{P}_{m_{\mathsf{g}}}(\sigma_{\ell})\neq 0$. Then, we conclude that
$\mathbf{P}_{m_{\mathsf{m}}}(\sigma_{\ell},\sigma_{o})\neq 0$ directly.

Fix a measurable set $S_{\ell}$ in $\llbracket A\rrbracket$. It suffices to
show that $\llbracket m_{\mathsf{m}}\rrbracket_{\sigma_{o}}(S_{\ell})=0$ if
and only if $\llbracket
m_{\mathsf{g}}\rrbracket^{m_{\mathsf{m}}}_{\sigma_{o}}(S_{\ell})=0$. We
conclude by the following reasoning:

$\displaystyle\llbracket m_{\mathsf{m}}\rrbracket_{\sigma_{o}}(S_{\ell})=0$
$\displaystyle\iff\int_{S_{\ell}}\mathbf{P}_{m_{\mathsf{m}}}(\sigma_{\ell},\sigma_{o})\lambda_{\llbracket
A\rrbracket}(d\sigma_{\ell})=0$ $\displaystyle\iff\lambda_{\llbracket
A\rrbracket}(\mathbf{P}_{m_{\mathsf{m}}}({\cdot},\sigma_{o})\cdot\mathrm{I}_{S_{\ell}})=0$
$\displaystyle\iff\lambda_{\llbracket
A\rrbracket}(\\{\mathbf{P}_{m_{\mathsf{m}}}({\cdot},\sigma_{o})\cdot\mathrm{I}_{S_{\ell}}>0\\})=0$
$\displaystyle\iff\lambda_{\llbracket
A\rrbracket}(\\{\mathbf{P}_{m_{\mathsf{m}}}({\cdot},\sigma_{o})>0\wedge\mathrm{I}_{S_{\ell}}>0\\})=0$
$\displaystyle\iff\lambda_{\llbracket
A\rrbracket}(\\{([\mathbf{P}_{m_{\mathsf{m}}}({\cdot},\sigma_{o})\neq
0]\cdot\mathbf{P}_{m_{\mathsf{g}}}({\cdot}))>0\wedge\mathrm{I}_{S_{\ell}}>0\\})=0$
$\displaystyle\iff\lambda_{\llbracket
A\rrbracket}(\\{[\mathbf{P}_{m_{\mathsf{m}}}({\cdot},\sigma_{o})\neq
0]\cdot\mathbf{P}_{m_{\mathsf{g}}}({\cdot})\cdot\mathrm{I}_{S_{\ell}}>0\\})=0$
$\displaystyle\iff\lambda_{\llbracket
A\rrbracket}([\mathbf{P}_{m_{\mathsf{m}}}({\cdot},\sigma_{o})\neq
0]\cdot\mathbf{P}_{m_{\mathsf{g}}}({\cdot})\cdot\mathrm{I}_{S_{\ell}})=0$
$\displaystyle\iff\int_{S_{\ell}}[\mathbf{P}_{m_{\mathsf{m}}}(\sigma_{\ell},\sigma_{o})\neq
0]\cdot\mathbf{P}_{m_{\mathsf{g}}}(\sigma_{\ell})\lambda_{\llbracket
A\rrbracket}(d\sigma_{\ell})=0$ $\displaystyle\iff\llbracket
m_{\mathsf{g}}\rrbracket^{m_{\mathsf{m}}}_{\sigma_{o}}(S_{\ell})=0.$

∎

#### Importance sampling (IS)

Recall the operational rule below for a single step in the IS algorithm: given
a model program $m_{\mathsf{m}}$, a guide program $m_{\mathsf{g}}$, and a
concrete observation $\sigma_{o}$, IS performs joint execution of the two
programs to draw a sample $\sigma_{\ell}$ with density $w_{\mathsf{g}}$ and
compute $\frac{w_{\mathsf{m}}}{w_{\mathsf{g}}}$ as the importance of
$\sigma_{\ell}$. ∅∣∅; $(\textnormal{{latent}}\\!:\sigma_{\ell})$ ⊢m_g
$\mathrel{\Downarrow^{w_{\mathsf{g}}}}$ _
∅∣$(\textnormal{{latent}}\\!:\sigma_{\ell})$ ;
$(\textnormal{{obs}}\\!:\sigma_{o})$ ⊢m_m
$\mathrel{\Downarrow^{w_{\mathsf{m}}}}$ _ m_g; m_m; σ_o ⊢_is^w_g
$\langle\sigma_{\ell},\nicefrac{{w_{\mathsf{m}}}}{{w_{\mathsf{g}}}}\rangle$
Define a density function
$\textsc{is}(\sigma_{\ell})\mathrel{\overset{\underset{\textnormal{def}}{}}{=}}w_{\mathsf{g}}\cdot\frac{w_{\mathsf{m}}}{w_{\mathsf{g}}}=w_{\mathsf{m}}$
on the space $\llbracket A\rrbracket$ of guidance traces for the computation
of IS. Then, the measure defined by IS can be defined as
$\mu_{\textsc{is}}(S_{\ell})\mathrel{\overset{\underset{\textnormal{def}}{}}{=}}\int_{S_{\ell}}\textsc{is}(\sigma_{\ell})\lambda_{\llbracket
A\rrbracket}(d\sigma_{\ell})$.

###### Lemma C.2.

Suppose that

$\displaystyle\cdot\mid\varnothing;(\textnormal{{latent}}\\!:A)$
$\displaystyle\vdash_{\Sigma}m_{\mathsf{g}}\mathbin{\textnormal{\ooalign{\vphantom{$\div$}\raisebox{4.09024pt}{\scalebox{1.2}{.}}\cr$\sim$\cr\vphantom{$\div$}\raisebox{-0.21529pt}{\scalebox{1.2}{.}}\cr}}}\tau_{\mathsf{g}}\mid\varnothing;(\textnormal{{latent}}\\!:\boldsymbol{1}),$
$\displaystyle\cdot\mid(\textnormal{{latent}}\\!:A);(\textnormal{{obs}}\\!:B)$
$\displaystyle\vdash_{\Sigma}m_{\mathsf{m}}\mathbin{\textnormal{\ooalign{\vphantom{$\div$}\raisebox{4.09024pt}{\scalebox{1.2}{.}}\cr$\sim$\cr\vphantom{$\div$}\raisebox{-0.21529pt}{\scalebox{1.2}{.}}\cr}}}\tau_{\mathsf{m}}\mid(\textnormal{{latent}}\\!:\boldsymbol{1});(\textnormal{{obs}}\\!:\boldsymbol{1}),$

$A$ is $\mathbin{\varoplus}$-free, $B$ is $\mathbin{\binampersand}$-free, and
$\sigma_{o}:B$ such that
$\int\mathbf{P}_{m_{\mathsf{m}}}(\sigma_{\ell},\sigma_{o})\lambda_{\llbracket
A\rrbracket}(d\sigma_{\ell})>0$. Then
$\mu_{\textnormal{{is}}}\propto\llbracket
m_{\mathsf{m}}\rrbracket_{\sigma_{o}}$.

###### Proof.

By Thm. 5.2, we know that the posterior $\llbracket
m_{\mathsf{m}}\rrbracket_{\sigma_{o}}$ is absolutely continuous with respect
to $\llbracket m_{\mathsf{g}}\rrbracket^{m_{\mathsf{m}}}_{\sigma_{o}}$. Thus,
$\llbracket m_{\mathsf{m}}\rrbracket_{\sigma_{o}}$ is also absolutely
continuous with respect to $\mu_{\textsc{is}}$. In other words, the density
function $\textsc{is}({\cdot})$ is positive at all possible latent variables
$\sigma_{\ell}$ in the posterior. Then, for any $S_{\ell}$ such that
$\llbracket m_{\mathsf{m}}\rrbracket_{\sigma_{o}}(S_{\ell})\neq 0$, we have

$\llbracket
m_{\mathsf{m}}\rrbracket_{\sigma_{o}}(S_{\ell})=\frac{\int_{S_{\ell}}\mathbf{P}_{m_{\mathsf{m}}}(\sigma_{\ell},\sigma_{o})\lambda_{\llbracket
A\rrbracket}(d\sigma_{\ell})}{\int\mathbf{P}_{m_{\mathsf{m}}}(\sigma_{\ell},\sigma_{o})\lambda_{\llbracket
A\rrbracket}(d\sigma_{\ell})}=\frac{\int_{S_{\ell}}\textsc{is}(\sigma_{\ell})\lambda_{\llbracket
A\rrbracket}(d\sigma_{\ell})}{\int\mathbf{P}_{m_{\mathsf{m}}}(\sigma_{\ell},\sigma_{o})\lambda_{\llbracket
A\rrbracket}(d\sigma_{\ell})}=\frac{\mu_{\textsc{is}}(S_{\ell})}{\int\mathbf{P}_{m_{\mathsf{m}}}(\sigma_{\ell},\sigma_{o})\lambda_{\llbracket
A\rrbracket}(d\sigma_{\ell})},$

and we conclude by the fact the the denominator is a constant. ∎

#### Variational inference (VI)

Recall that we parameterize the guide program $m_{\mathsf{g},\theta}$ by a
vector $\theta\in\Theta$ of parameters, and use KL divergence as the distance
metric, which is defined by

$\mathrm{KL}(\mu\parallel\nu)\mathrel{\overset{\underset{\textnormal{def}}{}}{=}}\int
p_{\mu}(\sigma_{\ell})(\log p_{\mu}(\sigma_{\ell})-\log
p_{\nu}(\sigma_{\ell}))\lambda_{\llbracket A\rrbracket}(d\sigma_{\ell}),$

where $\mu$ and $\nu$ are measures on a space $\llbracket A\rrbracket$ of
guidance traces of type $A$ with densities $p_{\mu}$ and $p_{\nu}$,
respectively, and $\mu$ is absolutely continuous with respect to $\nu$. The
rule below formulates the computation of KL divergence for a specific
$\theta$, via joint execution of the two programs. ∅∣∅;
$(\textnormal{{latent}}\\!:\sigma_{\ell})$ ⊢m_g,θ
$\mathrel{\Downarrow^{w_{\mathsf{g}}}}$ _
∅∣$(\textnormal{{latent}}\\!:\sigma_{\ell})$ ;
$(\textnormal{{obs}}\\!:\sigma_{o})$ ⊢m_m
$\mathrel{\Downarrow^{w_{\mathsf{m}}}}$ _ m_g,θ; m_m; σ_o ⊢_vi^w_g
$\langle\sigma_{\ell},\log w_{\mathsf{m}}-\log w_{\mathsf{g}}\rangle$ The rule
can be seen as defining a map $\sigma_{\ell}\mapsto w_{\mathsf{g}}\cdot(\log
w_{\mathsf{m}}-\log w_{\mathsf{g}})$, which is the integrand of the divergence
$\mathrm{KL}(\llbracket
m_{\mathsf{g},\theta}\rrbracket^{m_{\mathsf{m}}}_{\sigma_{o}}\parallel\llbracket
m_{\mathsf{m}}\rrbracket_{\sigma_{o}})$.

###### Lemma C.3.

Suppose that

$\displaystyle\cdot\mid\varnothing;(\textnormal{{latent}}\\!:A)$
$\displaystyle\vdash_{\Sigma}m_{\mathsf{g},\theta}\mathbin{\textnormal{\ooalign{\vphantom{$\div$}\raisebox{4.09024pt}{\scalebox{1.2}{.}}\cr$\sim$\cr\vphantom{$\div$}\raisebox{-0.21529pt}{\scalebox{1.2}{.}}\cr}}}\tau_{\mathsf{g}}\mid\varnothing;(\textnormal{{latent}}\\!:\boldsymbol{1}),$
$\displaystyle\cdot\mid(\textnormal{{latent}}\\!:A);(\textnormal{{obs}}\\!:B)$
$\displaystyle\vdash_{\Sigma}m_{\mathsf{m}}\mathbin{\textnormal{\ooalign{\vphantom{$\div$}\raisebox{4.09024pt}{\scalebox{1.2}{.}}\cr$\sim$\cr\vphantom{$\div$}\raisebox{-0.21529pt}{\scalebox{1.2}{.}}\cr}}}\tau_{\mathsf{m}}\mid(\textnormal{{latent}}\\!:\boldsymbol{1});(\textnormal{{obs}}\\!:\boldsymbol{1}),$

$A$ is $\mathbin{\varoplus}$-free, $B$ is $\mathbin{\binampersand}$-free, and
$\sigma_{o}:B$ such that
$\int\mathbf{P}_{m_{\mathsf{m}}}(\sigma_{\ell},\sigma_{o})\lambda_{\llbracket
A\rrbracket}(d\sigma_{\ell})>0$. Then, $\mathrm{KL}(\llbracket
m_{\mathsf{g},\theta}\rrbracket^{m_{\mathsf{m}}}_{\sigma_{o}}\parallel\llbracket
m_{\mathsf{m}}\rrbracket_{\sigma_{o}})$ is well-defined.

###### Proof.

By Thm. 5.2, we know that $\llbracket
m_{\mathsf{g},\theta}\rrbracket^{m_{\mathsf{m}}}_{\sigma_{o}}$ is absolutely
continuous with respect to $\llbracket m_{\mathsf{m}}\rrbracket_{\sigma_{o}}$.
Thus, the KL divergence used in VI is well-defined. ∎

#### Markov-Chain Monte Carlo (MCMC)

We focus on _Metropolis-Hastings_ (MH), which constructs the transition kernel
from a _proposal_ subroutine, and a probabilistic decision subroutine that
either accepts the proposed random sample, or rejects it and keeps the old
one. To implement proposal subroutines in our system, we extend the core
calculus such that guidance traces can be used as first-class data. Then we
implement the proposal subroutine as a procedure $g$ whose argument is a
guidance trace on the channel for latent random variables. The operational
rule below formulates a single step in the MH algorithm; given a proposal
procedure $g$, a model program $m_{\mathsf{m}}$, a concrete observation
$\sigma_{o}$, and the current latent trace $\sigma_{\ell}$, MH first performs
joint execution of $\mathsf{call}(g;\sigma_{\ell})$ and $m_{\mathsf{m}}$ to
generate a new latent trace $\sigma_{\ell}^{\prime}$ with density
$w_{\mathsf{fwd}}$, and then uses the new $\sigma_{\ell}^{\prime}$ and the old
$\sigma_{\ell}$ to calculate a _backward_ density $w_{\mathsf{bwd}}$. MH then
uses these densities to compute an _acceptance ratio_ $\alpha$, and then
accepts the new sample $\sigma_{\ell}^{\prime}$ with probability $\alpha$.
∅∣∅; $(\textnormal{{latent}}\\!:\sigma_{\ell}^{\prime})$
⊢$\mathsf{call}(g;\sigma_{\ell})$ $\mathrel{\Downarrow^{w_{\mathsf{fwd}}}}$ _
∅∣$(\textnormal{{latent}}\\!:\sigma_{\ell}^{\prime})$ ;
$(\textnormal{{obs}}\\!:\sigma_{o})$ ⊢m_m
$\mathrel{\Downarrow^{w^{\prime}_{\mathsf{m}}}}$ _
∅∣∅; $(\textnormal{{latent}}\\!:\sigma_{\ell})$
⊢$\mathsf{call}(g;\sigma_{\ell}^{\prime})$
$\mathrel{\Downarrow^{w_{\mathsf{bwd}}}}$ _
∅∣$(\textnormal{{latent}}\\!:\sigma_{\ell})$ ;
$(\textnormal{{obs}}\\!:\sigma_{o})$ ⊢m_m
$\mathrel{\Downarrow^{w_{\mathsf{m}}}}$ _
α= min(1, w’m⋅wbwdwm⋅wfwd) g ;m_m; σ_o ⊢_mh σ_ℓw_fwd ⋅α⇒ σ_ℓ’ The MH algorithm
specifies a transition density
$f_{1}(\sigma_{\ell},\sigma_{\ell}^{\prime})\mathrel{\overset{\underset{\textnormal{def}}{}}{=}}w_{\mathsf{fwd}}\cdot\alpha$
and an unchanged density
$f_{2}(\sigma_{\ell},\sigma_{\ell}^{\prime})\mathrel{\overset{\underset{\textnormal{def}}{}}{=}}w_{\mathsf{fwd}}\cdot(1-\alpha)$.
Then we can use the two density functions to construct the MH kernel
$\kappa_{\textsc{mh}}(\sigma,S)\mathrel{\overset{\underset{\textnormal{def}}{}}{=}}\int_{S}(f_{1}(\sigma,\sigma^{\prime})+f_{2}(\sigma,\sigma^{\prime})\cdot[\sigma\in
S])\lambda_{\llbracket A\rrbracket}(d\sigma^{\prime})$.

###### Lemma C.4.

Suppose that

$\displaystyle\textnormal{{old}}:|A|\mid\varnothing;(\textnormal{{latent}}\\!:A)$
$\displaystyle\vdash_{\Sigma}m_{\mathsf{g}}\mathbin{\textnormal{\ooalign{\vphantom{$\div$}\raisebox{4.09024pt}{\scalebox{1.2}{.}}\cr$\sim$\cr\vphantom{$\div$}\raisebox{-0.21529pt}{\scalebox{1.2}{.}}\cr}}}\tau_{\mathsf{g}}\mid\varnothing;(\textnormal{{latent}}\\!:\boldsymbol{1}),$
$\displaystyle\cdot\mid(\textnormal{{latent}}\\!:A);(\textnormal{{obs}}\\!:B)$
$\displaystyle\vdash_{\Sigma}m_{\mathsf{m}}\mathbin{\textnormal{\ooalign{\vphantom{$\div$}\raisebox{4.09024pt}{\scalebox{1.2}{.}}\cr$\sim$\cr\vphantom{$\div$}\raisebox{-0.21529pt}{\scalebox{1.2}{.}}\cr}}}\tau_{\mathsf{m}}\mid(\textnormal{{latent}}\\!:\boldsymbol{1});(\textnormal{{obs}}\\!:\boldsymbol{1}),$

$m_{\mathsf{g}}$ is the procedure body of the proposal program $g$, which has
a single parameter old with type $|A|$ that describes first-class guidance
traces, $A$ is $\mathbin{\varoplus}$-free, $B$ is
$\mathbin{\binampersand}$-free, and $\sigma_{o}:B$ such that
$\int\mathbf{P}_{m_{\mathsf{m}}}(\sigma_{\ell},\sigma_{o})\lambda_{\llbracket
A\rrbracket}(d\sigma_{\ell})>0$. Then the posterior $\llbracket
m_{\mathsf{m}}\rrbracket_{\sigma_{o}}$ is stationary for the kernel
$\kappa_{\textnormal{{mh}}}$.

###### Proof.

We can extend Thm. 5.2 by allowing the environments to contain values and then
using Thm. B.7 instead of Thm. 4.6 in the proof. Then, we derive that for any
latent variables $\sigma$, the posterior $\llbracket
m_{\mathsf{m}}\rrbracket_{\sigma_{o}}$ is absolutely continuous with respect
to the measure $\kappa_{\textsc{mh}}(\sigma,{\cdot})$; that is, the guide
program is able to sample any latent variables in the posterior, no matter
what the current sample is. We then conclude by applying the Metropolis-
Hastings-Green theorem (Green, 1995). ∎


11institutetext: Chen Qin 22institutetext: Department of Electrical and
Electronic Engineering, Imperial College London, London, UK
Institute for Digital Communications, School of Engineering, University of
Edinburgh, UK
22email<EMAIL_ADDRESS>33institutetext: Daniel Rueckert
44institutetext: Department of Computing, Imperial College London, London, UK
Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
44email<EMAIL_ADDRESS>
# Artificial Intelligence-Based Image Reconstruction in Cardiac Magnetic
Resonance

Chen Qin and Daniel Rueckert

###### Abstract

Artificial intelligence (AI) and Machine Learning (ML) have shown great
potential in improving the medical imaging workflow, from image acquisition
and reconstruction to disease diagnosis and treatment. Particularly, in recent
years, there has been a significant growth in the use of AI and ML algorithms,
especially Deep Learning (DL) based methods, for medical image reconstruction.
DL techniques have shown to be competitive and often superior over
conventional reconstruction methods in terms of both reconstruction quality
and computational efficiency. The use of DL-based image reconstruction also
provides promising opportunities to transform the way cardiac images are
acquired and reconstructed. In this chapter, we will review recent advances in
DL-based reconstruction techniques for cardiac imaging, with emphasis on
cardiac magnetic resonance (CMR) image reconstruction. We mainly focus on
supervised DL methods for the application, including image post-processing
techniques, model-driven approaches and $k$-space based methods. Current
limitations, challenges and future opportunities of DL for cardiac image
reconstruction are also discussed.

###### Keywords:

Artificial Intelligence, Deep Learning, Cardiac Imaging, Cardiac Magnetic
Resonance (CMR), Dynamic Image Reconstruction

##  1 Introduction

Artificial intelligence (AI) and Machine Learning (ML), and in particular Deep
Learning (DL), have been active and fast-moving areas of research in the field
of medical imaging. These techniques have great potential for transforming
clinical practice in the coming years pesapane2018artificial . DL-based
methods have also led to significant improvement across various applications
in cardiac imaging, such as cardiac image segmentation chen2020deep ;
bai2018automated , myocardial motion tracking duchateau2020machine ;
qin2018joint ; qin2020biomechanics  and image enhancement
oktay2017anatomically ; oktay2016multi . Cardiac image reconstruction, as one
of the key step in cardiac imaging workflow, has also witnessed the important
growth of the use of DL algorithms, though the development of which is
relatively new in comparison to other cardiac imaging applications.

In cardiac imaging, cardiac magnetic resonance (CMR) imaging is one of the
most useful and important imaging modalities for the non-invasive assessment
of morphology and function of the heart. It provides images with excellent
soft tissue contrast as well as high spatial and temporal resolution and is
therefore widely used for the assessment of cardiovascular diseases. However,
the acquisition speed of CMR is limited, which is on account of the need of
obtaining images of high temporal and spatial resolution, as well as the
cardiac and respiratory-induced motion that needs to be taken into
consideration during the acquisition process bustin2020compressed .

To improve and accelerate the CMR acquisition process, several major advances
have been made over the last few decades. Parallel imaging (PI)
pruessmann1999sense ; griswold2002generalized ; sodickson1997simultaneous  is
one of the techniques frequently used to accelerate the MR acquisition. In PI,
the data is acquired using an array of multiple independent receiver coils,
allowing for a reduction in the number of phase-encoding steps and enabling
the combination of undersampled data from each receiver coil given the
additional coil information. Particularly, for dynamic CMR imaging where
substantial correlations in $k$-space and time exist, most PI-based imaging
strategies have been designed to acquire part of the desired $k$-$t$
measurements and then reconstruct images by exploiting spatio-temporal
redundancies in combination with coil sensitivity information
kellman2001adaptive ; breuer2005dynamic ; huang2005k . Undersampled image
reconstruction approaches such as compressed sensing (CS) have also been
proposed for CMR image reconstruction. CS-based methods lustig2007sparse ;
ye2019compressed  exploit signal sparsity with some sparse transforms such as
finite differences or wavelets operators, and recover original image from
undersampled $k$-space data using nonlinear reconstructions. For dynamic CMR
image reconstruction, a common practice for CS-based approaches is effectively
to exploit the spatio-temporal redundancies via enforcing the signal sparsity
in spatio-temporal or its transform domain against the consistency of the
undersampled acquired data, such as some state-of-the-art $k$-$t$ methods
lustig2006kt ; jung2007improved ; otazo2010combination . Combination of CS
with low-rank schemes lingala2011accelerated ; otazo2015low , spatio-temporal
partial separability zhao2012image  and patch-based regularisation framework
yoon2014motion ; mohsin2017accelerated  have also been proposed to exploit the
spatio-temporal correlations or geometric similarities for the dynamic CMR
image reconstruction.

Though CS-based methods have shown great potential for accelerated CMR
imaging, they often impose strong assumptions on the data and require
nontrivial manual adjustments of hyperparameters depending on the application.
In addition, reconstruction speed of these methods is often slow due to the
iterative optimisation used, and in the context of dynamic CMR imaging, the
additional time domain further increases the computational demand. To enable
progress beyond the limitations of conventional CS-based reconstruction
methods, DL approaches have been proposed for CMR image reconstruction. These
DL solutions can be roughly categorised as image post-processing techniques,
model-driven image reconstruction methods or $k$-space interpolation
approaches. Most of these approaches typically rely on large databases of MR
scans as training data to exploit the prior knowledge. DL approaches have
shown great performance in achieving high image reconstruction quality with
extremely fast reconstruction speed, which is promising to further accelerate
CMR acquisition as well as to improve the reconstruction.

In this chapter, we will first briefly introduce the problem formulation of
CMR image reconstruction as an inverse problem, and then we will discuss the
recent advances in DL for CMR image reconstruction with a major focus on
supervised learning. Finally, current limitations, challenges and
opportunities of DL for CMR reconstruction will be discussed.

##  2 Problem Formulation for Image Reconstruction

A general forward model for the MR imaging process can be formulated as

$\mathbf{y}=\mathbf{Ex}+\mathbf{n},$ (1)

where $\mathbf{x}\in\mathbb{C}^{N_{x}}$ denotes the complex-valued MR images
to be recovered, $\mathbf{y}\in\mathbb{C}^{N_{y}}$ represents the undersampled
$k$-space measurements corrupted by additive Gaussian noise
$\mathbf{n}\in\mathbb{C}^{N_{y}}$, and
$\mathbf{E}:\mathbb{C}^{N_{x}}\to\mathbb{C}^{N_{y}}$ is a linear forward
operator which describes the encoding process modelling the MR physics. Here
$N_{x}$ and $N_{y}$ define the dimensions of the reconstruction $\mathbf{x}$
and the $k$-space data $\mathbf{y}$ ($N_{y}\ll N_{x}$ in the undersampling
scenario) and $\mathbf{E}$ includes the coil sensitivity maps (if available),
Fourier transform as well as the undersampling mask according to the
underlying multi-coil or single-coil setting. In dynamic CMR imaging, the
measurements $\mathbf{y}$ are acquired in $k$-$t$ space with an additional
temporal dimension $N_{\textsc{T}}$, and $\mathbf{x}$ is the reconstructed
image sequence.

Reconstructing image $\mathbf{x}$ from measurements $\mathbf{y}$ is an ill-
posed inverse problem. To ensure that there exists a solution that is unique
and plausible, the reconstruction problem is commonly formulated as a
regularised optimisation problem of the form:

$\hat{\mathbf{x}}={\text{argmin}}_{\mathbf{x}}\
\mathcal{R}(\bf{x})+\lambda\|\bf{y}-\mathbf{E}\mathbf{x}\|^{2}_{2}.$ (2)

Here $\mathcal{R}$ expresses the regularisation terms that act on $\mathbf{x}$
and $\hat{\mathbf{x}}$ is recovered via enforcing the regularisation against
consistency with the acquired undersampled $k$-space data. For CS-based
approaches, the regularisation terms $\mathcal{R}$ are often employed in some
specific transform domain with sparse representations: Common choices are
wavelet transforms lustig2007sparse  or total variation block2007undersampled
. For low-rank based methods, global or local low-rankness of the dynamic
images is exploited via regularising the rank or nuclear norm of $\mathbf{x}$
to estimate the missing samples. Here, the model weight $\lambda$ allows the
adjustment of data fidelity based on the noise level of the acquired
measurements $\mathbf{y}$.

To solve the inverse problem formulated as Eq. 2, various optimisation schemes
can be employed ye2019compressed , such as alternating directional method of
multiplier (ADMM) wang2008new , split Bregman iteration goldstein2009split
and primal-dual algorithm chambolle2011first . A common practice to optimise
Eq. 2 is to use variable splitting techniques ramani2010parallel , where an
auxiliary variable $\mathbf{z}$ with the constraint $\mathbf{z}=\mathbf{x}$ is
introduced to decouple the regularisation term from the data fidelity term.
This reformulates Eq. 2 to minimize the following cost function via the
penalty method:

${\text{argmin}}_{\mathbf{x},\mathbf{z}}\
\mathcal{R}(\mathbf{z})+\lambda\|\mathbf{y}-\mathbf{E}\mathbf{x}\|_{2}^{2}+\mu\|\mathbf{x}-\mathbf{z}\|^{2}_{2},$
(3)

Here $\mu$ is a penalty parameter. By applying alternating minimisation over
$\mathbf{x}$ and $\mathbf{z}$, Eq. 3 can be solved via the following iterative
procedures:

$\displaystyle\mathbf{z}^{(i+1)}$ $\displaystyle={\text{argmin}}_{\mathbf{z}}\
\mathcal{R}(\mathbf{z})+\mu\|\mathbf{x}^{(i)}-\mathbf{z}\|^{2}_{2}$ (4a)
$\displaystyle\mathbf{x}^{(i+1)}$ $\displaystyle={\text{argmin}}_{\mathbf{x}}\
\lambda\|\mathbf{y}-\mathbf{E}\mathbf{x}\|^{2}_{2}+\mu\|\mathbf{x}-\mathbf{z}^{(i+1)}\|^{2}_{2},$
(4b)

where $\mathbf{x}^{(0)}$ is commonly a zero-filled reconstruction used as an
initialisation and $\mathbf{z}$ can be seen as an intermediate state of the
optimisation process. For MRI reconstruction, Eq. 4b is often regarded as a
data consistency (DC) step.

##  3 Deep learning for CMR Image Reconstruction

Deep learning approaches have gained in popularity for MRI reconstruction in
recent years, due to their excellent capabilities in reconstructing high-
quality images at extremely fast reconstruction speeds. For CMR image
reconstruction, most of the DL methods can be generally categorised into three
main groups: (1) image reconstruction via DL-based image post-processing; (2)
model-driven unrolled DL for iterative reconstruction; and (3) $k$-space based
interpolation. Schematic illustrations of these DL techniques for CMR image
reconstruction are shown in Fig. 1.

Figure 1: Illustration of different types of DL-based approaches for CMR image
reconstruction. (a) Learning image post-processing for CMR image
reconstruction. (b) Learning unrolled optimisation for model-driven iterative
reconstruction. (c) DL for k-space based CMR reconstruction. DNN: deep neural
networks, DC: data consistency.

###  3.1 CMR Reconstruction via DL-based Image Post-processing

The first group of methods for CMR reconstruction relies on image post-
processing techniques for enhancement in the image domain. These approaches
learn an end-to-end mapping from the zero-filled reconstructions to the fully-
sampled reference image without any requirement for prior knowledge about the
image acquisition process. For instance, U-net architectures are commonly used
for the CMR image post-processing to reduce noise-like image artefacts
hauptmann2019real ; kofler2019spatio . They receive inputs in form of
reconstructed, undersampled CMR sequences and learn to de-alias the images via
multi-scale downsampling and synthesis paths with skip connections. To enforce
temporal consistency between frames of cardiac cine sequences, the U-net can
be equipped with either 3D convolutions to perform on the entire image
sequences ($x$-$y$-$t$) hauptmann2019real  or 2D convolutions on the 2D
spatio-temporal domain ($x$-$t$) for more efficient training kofler2019spatio
. These models hauptmann2019real ; kofler2019spatio  showed better image
quality and faster reconstruction speed recovered from undersampled 2D golden-
angle radial cine CMR when compared to CS image reconstruction techniques.

Since these methods solely work on information from the initial zero-filling
image data and information extracted from training samples, they can also be
classified as purely data-driven approaches. Though these methods are simple
and efficient in reconstructing images, one significant drawback is that they
cannot guarantee the consistency of the reconstructed image with respect to
the measured $k$-space data. This limits their performance in comparison to
the model-driven approaches which will be discussed in the following.

###  3.2 Model-driven DL for CMR Reconstruction

Another type of approaches for CMR image reconstruction are model-driven DL
techniques based on unrolled schemes inspired by conventional iterative CS
reconstruction, which also represents the majority of the work in this field.
These methods utilise the physical domain knowledge with the incorporation of
raw $k$-space data, combining the power of data-driven learning with the
physics-derived model-based framework (Eq. 2). In contrast to CS-based methods
where regularisation terms need to be specified manually beforehand, DL-based
reconstruction approaches implicitly learn such prior information and
regularisation from the routinely performed MR data.

In detail, model-driven DL-based solutions schlemper2017deep ;
schlemper2018deep ; qin2019convolutional ; aggarwal2018modl ;
hammernik2018learning ; duan2019vs  propose to learn the unrolled optimisation
and embed the iterative reconstruction process in a learning setting based on
Eq. 4. Specifically, Eq. 4a is commonly modelled via leveraging deep neural
networks (DNNs) to learn the prior knowledge from the training data itself
schlemper2018deep ; qin2019convolutional ; aggarwal2018modl . This allows the
model to fully exploit the information contained in the population data for
the regularisation and alleviates the need to explicitly pre-define them based
on certain assumptions. Accordingly, Eq. 4b can be derived as close-form
solutions, which then can be embedded as DC layers in DNNs via exact update
for DC schlemper2018deep ; qin2019convolutional ; duan2019vs  or through
numerical optimisation such as conjugate gradient updates aggarwal2018modl ;
biswas2019dynamic ; mardani2018deep , depending on the way to solve Eq. 4b.
Therefore, model-driven approaches based on Eq. 4 can be generally represented
as the compact form:

$\displaystyle{\mathbf{z}}^{(i+1)}$
$\displaystyle=\textnormal{DNN}({\bf{x}}^{(i)}),$ (5a)
$\displaystyle{\bf{x}}^{(i+1)}$
$\displaystyle=\textnormal{DC}({\bf{z}}^{(i+1)};\mathbf{y},\mathbf{E},\lambda,\mu),$
(5b)

which alternates the reconstruction process between image de-aliasing and DC
steps for a pre-defined number of iterations. This represents the most
commonly used scheme for DL-based CMR reconstruction. For other types of
model-driven approaches (including applications not limited to CMR), please
refer to hammernik2020machine  for details.

Based on types of deep-learned regularisation priors, we generally group these
unrolled DL techniques into two classes: image-domain-regularisation and
complementary-regularisation approaches. More specific network architectures
and applied scenarios in cardiac imaging are introduced hereafter.

#### Learning Unrolled Optimisation with Image Domain Regularisation

##### Unrolled Convolutional Neural Networks

Typical approaches for constructing Eq. 5a are to employ convolutional neural
networks (CNNs) as regularisation operators in image domain. State-of-the-art
methods such as DC-CNN schlemper2017deep ; schlemper2018deep  and MoDL
aggarwal2018modl ; biswas2019dynamic  used 5 layers of CNN with a residual
connection for 10 iterations. To exploit spatio-temporal redundancies,
Schlemper et al. schlemper2018deep  proposed to incorporate a data sharing
(DS) layer in a 3D ($x$-$y$-$t$) CNN cascade network, which allows to utilise
similar information contained in neighboring $k$-space samples. The method was
performed on retrospectively undersampled single-coil cardiac cine MR images,
where the single-coil setting enables solution to Eq. 4b to be analytically
computed as an exact update for DC step. For multi-coil acquisition,
variational network (VN) hammernik2018learning  proposed to embed a combined
PI and CS reconstruction in a DL-based unrolled gradient descent scheme. An
extension of it has also been applied for dynamic cine CMR via learning the
complex spatio-temporal convolutions corresponding to spatio-temporal total
variation regularisation hammernik2019dynamic . Beyond applications on 2D
dynamic cine CMR, Fuin et al. fuin2019variational  has extended VN to a multi-
scale VN architecture for fast reconstruction of undersampled motion-
compensated free-breathing whole-heart 3D ($x$-$y$-$z$) CMRA, and CINENet
kustner2020cinenet  was introduced for highly prospectively undersampled 3D
Cartesian cardiac cine data via exploiting spatio-temporal redundancies by
cascaded and complex-valued (3+1)D spatial and temporal convolutions.

##### Unrolled Convolutional Recurrent Neural Networks

Another type of representative approach for learning the deep priors in Eq. 5a
is via recurrent architectures. Recurrent neural networks (RNNs) are known to
be effective in processing sequential information with an internal state
acting as “memory” and thus have been widely employed for modelling the
iterative reconstruction steps qin2019convolutional ; zhou2020dudornet ;
huang2020dynamic ; wang2019pyramid . As one of the pioneering work in this
direction, CRNN-MRI qin2019convolutional  proposed to embed the iterative
optimisation scheme (Eq. 4) into a recurrent network structure with hidden-to-
hidden connections between each iteration step, which allows effective
information propagation along iterations and avoids redundant computations.
Furthermore, to fully exploit spatio-temporal redundancies, bidirectional RNNs
were also designed to model the sequential dependencies along temporal axis
together with recurrence over iteration dimension. An illustrative diagram of
the model is shown in Fig. 2, where unfolded structures of each layer are
given with more details. The method was validated on retrospectively
undersampled single-coil cardiac cine MRI, which achieved better and more
efficient reconstructions compared to CNN architectures with fewer number of
network parameters.

Figure 2: (a) Architecture of CRNN-MRI network qin2019convolutional  for CMR
image reconstruction. (b) The structure of CRNN-MRI when unfolded over
iterative reconstruction steps. (c) The structure of bidirectional RNN layer
(BCRNN-t-i) when unfolded over the time sequence. The method is able to
jointly exploit spatio-temporal redundancies and iteration dependencies in a
recurrent fashion with a single framework.

##### Motion Compensated Dynamic CMR Reconstruction

Due to the presence of motion patterns in dynamic CMR acquisition, combining
motion estimation and motion compensation with model-driven DL-based
reconstruction methods has also been explored. End-to-end motion compensated
reconstruction framework huang2020dynamic ; huang2019dynamic ;
seegoolam2019exploiting  commonly consists of DL-based motion estimation
components qin2018joint  estimating motion information between frames within
undersampled MRI sequences seegoolam2019exploiting  or with respect to fully
sampled reference frames huang2020dynamic . The estimated motion then can be
exploited to generate motion compensated images to be infused into
reconstruction process huang2019dynamic ; huang2020dynamic  or to propagate
$k$-space data along the full temporal dynamics via data consistency
seegoolam2019exploiting  for better reconstruction quality. These approaches
have been validated on single-coil 2D cardiac cine MRI data with retrospective
undersampling, and incorporating motion information into reconstruction
process has shown to be effective in reconstructing aggressively accelerated
data seegoolam2019exploiting .

#### Learning Unrolled Optimisation with Complementary Regularisation

Similar to conventional CS-based approaches, model-driven DL for CMR
reconstruction can also be learnt with complementary regularisation. Typical
$k$-$t$ methods lustig2006kt ; jung2007improved ; otazo2010combination
exploit signal sparsity and regularise the data in the combined spatial and
temporal Fourier ($x$-$f$) domain for dynamic CMR reconstruction. This can
also be similarly leveraged with DL by learning deep priors in complementary
image and $x$-$f$ spaces. Qin et al. qin2019k  proposed the $k$-$t$ NEXT
approach along this direction, which alternates the learned DL-based
regularisation between both image and $x$-$f$ domains in an iterative fashion
for single-coil dynamic CMR reconstruction. Comparison results of $k$-$t$ NEXT
against other model-driven DL approaches and CS-based method are shown in Fig.
3, where it can be seen that exploiting complementary domain knowledge can
achieve better performance than single image-domain reconstructions, with
better preservation of fine details and sharp structures. An extension of it
has also been investigated for multi-coil dynamic CMR data with a
complementary time-frequency domain network based on variable splitting
technique in PI setting qin2020complementary . In addition to the $x$-$f$
regularisation, MoDL-SToRM biswas2019dynamic  combines MoDL reconstruction
aggarwal2018modl  along with a complementary image regularisation penalty,
i.e., SmooThness regularisation on manifolds (SToRM) prior poddar2015dynamic ,
to recover free-breathing cardiac cine MR images. Similarly, this framework
alternates between a learned regularisation of the image using CNN, an
analytically defined SToRM prior, and a conjugate gradient DC step for multi-
channel measurements. Besides, the complementary information can also be
exploited in both $k$-space and image space, and this has been shown in
wang2019dimension  where the frequency domain network and image domain network
were combined and trained with a multi‐supervised technique for the single-
coil 2D cardiac cine MR image reconstruction. All these above methods have
demonstrated the benefits of combining complementary regularisation, achieving
better performance than single image-domain approaches.

Figure 3: CMR image reconstruction for 9-fold accelerated single-coil data
along with their error maps. (a) Fully sampled reference (b) $9\times$
undersampled image (c, d) $k$-$t$ FOCUSS jung2007improved  (e, f) CRNN-MRI
qin2019convolutional  (g, h) DS+3DCNN schlemper2018deep  (i, j) $k$-$t$ NEXT
qin2019k . Model-driven DL approaches outperform CS-based $k$-$t$ FOCUSS.
Learning with complementary regularisation ($k$-$t$ NEXT) achieves better
reconstruction than methods with single domain regularisation.

###  3.3 DL for K-space based CMR Reconstruction

An alternative class of techniques for MRI reconstruction is based on
$k$-space interpolation. This is represented by methods such as GRAPPA
(generalized autocalibrating partially parallel acquisitions)
griswold2002generalized  which recovers the missing data directly in $k$-space
by employing autocalibrating signal (ACS) estimated for each subject. Similar
to this, a CNN-based approach has also been proposed to improve the $k$-space
interpolation for PI-based MRI reconstruction. The RAKI (robust artificial-
neural-networks for $k$-space interpolation) reconstruction approach
akccakaya2019scan  enables non-linear estimation of missing $k$-space lines
from acquired multi-coil data via using CNNs trained on ACS data for $k$-space
interpolation. This has been applied for reconstruction of retrospectively
undersampled myocardial 2D T1 mapping data with improved noise resilience.
Note, that RAKI is a scan-specific and database-free DL approach, which does
not require training datasets but relies on ACS data and separate CNN training
for each scan.

###  3.4 Network Learning

In general, most of the aforementioned supervised DL methods rely on large
databases of training data to learn the population-based prior information and
the mapping between pairs of undersampled data and reference MRI scans. Due to
the limited availability of the acquired images, most approaches consider
retrospectively simulating the undersampled data from fully-sampled raw
$k$-space data, which are then used as input and target for DL models during
the training process. To train these supervised models, metrics that measure
the similarity between the reconstructed image $\mathbf{x}$ and the reference
image $\mathbf{x}_{\text{ref}}$ are used as loss functions. One commonly used
loss function is the mean-squared-error (MSE) which measures the L2 distance
between $\mathbf{x}$ and $\mathbf{x}_{\text{ref}}$ in the training set
$\Omega$:

$\mathcal{L}_{\text{mse}}(\theta)=\frac{1}{n_{\Omega}}\sum\limits_{(\mathbf{x},\mathbf{x}_{\text{ref}})\in\Omega}{\left\|{{{\mathbf{x}}_{\text{ref}}}-{{\mathbf{x}}}}\right\|_{2}^{2}},$
(6)

Here $\theta$ is the network parameters and ${n_{\Omega}}$ is the number of
training samples. Though the MSE metric is easy to optimise, it often tends to
generate blurry reconstructions due to the averaging effect. To remedy this,
the L1 distance or structural similarity index measure (SSIM) are also often
employed as the loss function to improve upon the sharpness of structures,
such that:

$\mathcal{L}_{\text{ssim}}(\theta)=\frac{1}{n_{\Omega}}\sum\limits_{(\mathbf{x},\mathbf{x}_{\text{ref}})\in\Omega}\left(1-\text{SSIM}(\mathbf{x},\mathbf{x}_{\text{ref}})\right).$
(7)

Additionally, adversarial loss functions goodfellow2014generative  have also
gained in popularity for reconstructing images with improved perceptual
quality. Such a loss function has also been used for MRI reconstruction
yang2017dagan ; seitzer2018adversarial ; quan2018compressed . The adversarial
approach works by learning a discriminator to discriminate whether a given
sample is from a reconstruction network (generator) or from the reference
distribution. The adversarial loss has shown to be effective in generating
images with sharper textures and finer details compared to MSE. However, it is
known to be unstable in training and is also prone to hallucinate unrealistic
artefacts. This is problematic for MRI reconstruction in preserving the data
fidelity and anatomical structures. Therefore, the adversarial loss is often
combined with the content losses such as MSE and L1 norm to inform the
reconstruction with image contents. An alternative way to balance the data
fidelity and perceptual quality uses deep network interpolation qin2020deep ,
where these two effects can be trade off with a interpolation coefficient in
parameter space. Finally, perceptual loss terms ledig2017photo  have also been
introduced to improve the reconstruction details based on feature
representations of the VGG network simonyan2014very , and this has also been
applied for CMR reconstruction seitzer2018adversarial .

##  4 Discussion

Most of the current research shows exciting and promising developments of AI
algorithms in image reconstruction for cardiac imaging. These are mostly
attributed to their strong ability in recovering high quality images with fast
reconstruction speed, which therefore can further increase the acceleration
capability. Current DL-based approaches, especially models learning an
unrolled optimisation schlemper2018deep ; qin2019convolutional ;
wang2019dimension ; biswas2019dynamic ; fuin2019variational ;
kustner2020cinenet , have shown to outperform conventional CS-based methods
for CMR reconstruction, and their application in PI also indicates their
potentiality in facilitating fast single-breath-hold 2D cardiac cine imaging.
Though promising results have been achieved in recent developments, there are
still some challenges yet to be solved for their use in clinical applications.
One of the limitations of current studies for CMR image reconstruction is
their limited validation in clinical practice. Most of the existing DL-based
works focus on reconstructing images with retrospectively undersampling based
on models trained on simulated undersampled data. Though such retrospective
study can provide useful insights and initial validation of employed methods,
it still remains unclear how well these methods can perform on prospectively
undersampled data, which is a more clinically practical scenario. In addition,
majority of current DL research for dynamic CMR image reconstruction works on
simulated single-coil acquisition setting, whereas only a handful of works
propose to address the more practical and commonly used multi-coil
acquisitions. This is mainly due to the increased computational and model
complexities introduced by the multi-coil data as well as the extra temporal
dimension. Thus, efficient DL models for PI-based dynamic CMR image
reconstruction are also highly desirable. Besides, as most work investigates
reconstruction with Cartesian undersampling patterns, efforts towards non-
uniform undersampling strategies such as radial sampling and golden angle
sampling are also necessary, which represent the commonly used sampling
strategies in acceleration of 2D cardiac MR imaging in practice.

Another issue with DL-based CMR image reconstruction methods is their
generalisation ability when presented with data that looks different to the
data used in training. It is not yet systematically evaluated how well DL
models trained on MR scans with specific scanners, pulse sequences, imaging
resolutions or acceleration factors can generalise to other MR scans with
different acquisition parameters. Furthermore, the generalisation ability of
these models on unseen pathologies still needs to be explored in detail. For
instance, it will be interesting and useful to see if models trained on
healthy cohorts can faithfully recover the pathological patterns of patients
with cardiovascular diseases. Though a recent study qin2020complementary  has
made efforts towards this direction and shows promising generalisation
potential of the DL-based approach, more thorough and systematic evaluations
are still needed for a wider validation on this aspect including radiologists’
evaluation to improve their translation into clinical practice.

Additionally, in comparison to conventional optimisation-based CS methods,
training DL models normally requires large databases of training data.
However, such large training database is not available for most of the CMR
image reconstruction studies, and currently most existing works rely on a
small set of training samples with a great deal of data augmentations to
remedy this. Though such strategy has shown to be effective, it could still
limit the performance potential of such DL-models as well as their validations
due to the limited amount of available data. The FastMRI datasets
knoll2020fastmri  provide large databases of knee ($\sim$1000) and brain
($\sim$7000) MR images together with the corresponding k-space data, which
brings a great opportunity for accelerating MR imaging with AI. However, such
a large archive of images and their corresponding raw k-space data is still
lacking in cardiac imaging, and efforts towards building a large CMR database
will be helpful to further push the developments of the use of AI in this
field. Besides, methods beyond supervised learning such as self-supervised
learning and unsupervised learning will also be interesting to be explored to
reduce the expensive need of fully sampled reference data for training, and
models in transfer learning and domain adaptation can also be investigated to
effectively exploit the large amount of available data in other domains for
CMR reconstruction.

##  5 Conclusion

Based on recent advancements in AI technology, the field of cardiac image
reconstruction also sees its exciting developments with the use of AI
algorithms in recent years. This chapter provides an overview of these
developments with a major focus on introducing the existing popular DL-based
approaches for CMR image reconstruction as well as the current challenges.
Though DL approaches for CMR image reconstruction are not yet established for
clinical applicability and will require more thorough evaluation, they have
already shown a huge potential for accelerating and improving the CMR image
acquisition and reconstruction workflow.

## References

  * (1) Aggarwal, H.K., Mani, M.P., Jacob, M.: MoDL: Model-based deep learning architecture for inverse problems.  IEEE Trans. Med. Imag. 38(2), 394–405 (2018)
  * (2) Akçakaya, M., Moeller, S., Weingärtner, S., Uğurbil, K.: Scan-specific robust artificial-neural-networks for k-space interpolation (RAKI) reconstruction: Database-free deep learning for fast imaging.  Magn. Reson. Med. 81(1), 439–453 (2019)
  * (3) Bai, W., Sinclair, M., Tarroni, G., Oktay, O., Rajchl, M., Vaillant, G., Lee, A.M., Aung, N., Lukaschuk, E., Sanghvi, M.M., et al.: Automated cardiovascular magnetic resonance image analysis with fully convolutional networks.  Journal of Cardiovascular Magnetic Resonance 20(1), 65 (2018)
  * (4) Biswas, S., Aggarwal, H.K., Jacob, M.: Dynamic MRI using model-based deep learning and SToRM priors: MoDL-SToRM.  Magn. Reson. Med. 82(1), 485–494 (2019)
  * (5) Block, K.T., Uecker, M., Frahm, J.: Undersampled radial MRI with multiple coils. iterative image reconstruction using a total variation constraint.  Magn. Reson. Med. 57(6), 1086–1098 (2007)
  * (6) Breuer, F.A., Kellman, P., Griswold, M.A., Jakob, P.M.: Dynamic autocalibrated parallel imaging using temporal grappa (tgrappa).  Magn. Reson. Med. 53(4), 981–985 (2005)
  * (7) Bustin, A., Fuin, N., Botnar, R.M., Prieto, C.: From compressed-sensing to artificial intelligence-based cardiac MRI reconstruction.  Frontiers in Cardiovascular Medicine 7, 17 (2020)
  * (8) Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging.  Journal of mathematical imaging and vision 40(1), 120–145 (2011)
  * (9) Chen, C., Qin, C., Qiu, H., Tarroni, G., Duan, J., Bai, W., Rueckert, D.: Deep learning for cardiac image segmentation: A review.  Frontiers in Cardiovascular Medicine 7, 25 (2020)
  * (10) Duan, J., Schlemper, J., Qin, C., Ouyang, C., Bai, W., Biffi, C., Bello, G., Statton, B., O’Regan, D.P., Rueckert, D.: VS-Net: Variable splitting network for accelerated parallel MRI reconstruction.  In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 713–722. Springer (2019)
  * (11) Duchateau, N., King, A.P., De Craene, M.: Machine learning approaches for myocardial motion and deformation analysis.  Frontiers in Cardiovascular Medicine 6, 190 (2020)
  * (12) Fuin, N., Bustin, A., Kuestner, T., Botnar, R., Prieto, C.: A variational neural network for accelerating free-breathing whole-heart coronary MR angiography.  In: Proc. ISMRM 27th Annu. Meeting Exhibit, pp. 11–16 (2019)
  * (13) Goldstein, T., Osher, S.: The split Bregman method for L1-regularized problems.  SIAM journal on imaging sciences 2(2), 323–343 (2009)
  * (14) Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets.  In: Advances in neural information processing systems, pp. 2672–2680 (2014)
  * (15) Griswold, M.A., Jakob, P.M., Heidemann, R.M., Nittka, M., Jellus, V., Wang, J., Kiefer, B., Haase, A.: Generalized autocalibrating partially parallel acquisitions (GRAPPA).  Magn. Reson. Med. 47(6), 1202–1210 (2002)
  * (16) Hammernik, K., Klatzer, T., Kobler, E., Recht, M.P., Sodickson, D.K., Pock, T., Knoll, F.: Learning a variational network for reconstruction of accelerated MRI data.  Magn. Reson. Med. 79(6), 3055–3071 (2018)
  * (17) Hammernik, K., Knoll, F.: Machine learning for image reconstruction.  In: Handbook of Medical Image Computing and Computer Assisted Intervention, pp. 25–64. Elsevier (2020)
  * (18) Hammernik, K., Schloegl, M., Kobler, E., Stollberger, R., Pock, T.: Dynamic multicoil reconstruction using variational networks.  In: Proc. ISMRM 27th Annu. Meeting Exhibit, p. 4656 (2019)
  * (19) Hauptmann, A., Arridge, S., Lucka, F., Muthurangu, V., Steeden, J.A.: Real-time cardiovascular MR with spatio-temporal artifact suppression using deep learning–proof of concept in congenital heart disease.  Magn. Reson. Med. 81(2), 1143–1156 (2019)
  * (20) Huang, F., Akao, J., Vijayakumar, S., Duensing, G.R., Limkeman, M.: k-t GRAPPA: A k-space implementation for dynamic MRI with high reduction factor.  Magn. Reson. Med. 54(5), 1172–1184 (2005)
  * (21) Huang, Q., Xian, Y., Yang, D., Qu, H., Yi, J., Wu, P., Metaxas, D.N.: Dynamic MRI reconstruction with end-to-end motion-guided network.  Medical Image Analysis p. 101901 (2020)
  * (22) Huang, Q., Yang, D., Qu, H., Yi, J., Wu, P., Metaxas, D.: Dynamic MRI reconstruction with motion-guided network.  In: International Conference on Medical Imaging with Deep Learning, pp. 275–284. PMLR (2019)
  * (23) Jung, H., Ye, J.C., Kim, E.Y.: Improved k–t BLAST and k–t SENSE using FOCUSS.  Physics in Medicine & Biology 52(11), 3201 (2007)
  * (24) Kellman, P., Epstein, F.H., McVeigh, E.R.: Adaptive sensitivity encoding incorporating temporal filtering (TSENSE).  Magn. Reson. Med. 45(5), 846–852 (2001)
  * (25) Knoll, F., Zbontar, J., Sriram, A., Muckley, M.J., Bruno, M., Defazio, A., Parente, M., Geras, K.J., Katsnelson, J., Chandarana, H., et al.: fastmri: A publicly available raw k-space and dicom dataset of knee images for accelerated mr image reconstruction using machine learning.  Radiology: Artificial Intelligence 2(1), e190007 (2020)
  * (26) Kofler, A., Dewey, M., Schaeffter, T., Wald, C., Kolbitsch, C.: Spatio-temporal deep learning-based undersampling artefact reduction for 2D radial cine MRI with limited training data.  IEEE Trans Med Imag 39(3), 703–717 (2019)
  * (27) Küstner, T., Fuin, N., Hammernik, K., Bustin, A., Qi, H., Hajhosseiny, R., Masci, P.G., Neji, R., Rueckert, D., Botnar, R.M., et al.: CINENet: deep learning-based 3D cardiac CINE MRI reconstruction with multi-coil complex-valued 4D spatio-temporal convolutions.  Scientific reports 10(1), 1–13 (2020)
  * (28) Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al.: Photo-realistic single image super-resolution using a generative adversarial network.  In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4681–4690 (2017)
  * (29) Lingala, S.G., Hu, Y., DiBella, E., Jacob, M.: Accelerated dynamic MRI exploiting sparsity and low-rank structure: k-t SLR.  IEEE Trans. Med. Imag. 30(5), 1042–1054 (2011)
  * (30) Lustig, M., Donoho, D., Pauly, J.M.: Sparse MRI: The application of compressed sensing for rapid MR imaging.  Magn. Reson. Med. 58(6), 1182–1195 (2007)
  * (31) Lustig, M., Santos, J.M., Donoho, D.L., Pauly, J.M.: kt SPARSE: High frame rate dynamic MRI exploiting spatio-temporal sparsity.  In: Proc. ISMRM 13th Annu. Meeting Exhibit, p. 2420 (2006)
  * (32) Mardani, M., Gong, E., Cheng, J.Y., Vasanawala, S.S., Zaharchuk, G., Xing, L., Pauly, J.M.: Deep generative adversarial neural networks for compressive sensing MRI.  IEEE Trans Med Imag 38(1), 167–179 (2018)
  * (33) Mohsin, Y.Q., Lingala, S.G., DiBella, E., Jacob, M.: Accelerated dynamic MRI using patch regularization for implicit motion compensation.  Magn. Reson. Med. 77(3), 1238–1248 (2017)
  * (34) Oktay, O., Bai, W., Lee, M., Guerrero, R., Kamnitsas, K., Caballero, J., de Marvao, A., Cook, S., O’Regan, D., Rueckert, D.: Multi-input cardiac image super-resolution using convolutional neural networks.  In: International conference on medical image computing and computer-assisted intervention, pp. 246–254. Springer (2016)
  * (35) Oktay, O., Ferrante, E., Kamnitsas, K., Heinrich, M., Bai, W., Caballero, J., Cook, S.A., De Marvao, A., Dawes, T., O‘Regan, D.P., et al.: Anatomically constrained neural networks (ACNNs): application to cardiac image enhancement and segmentation.  IEEE Trans. Med. Imag. 37(2), 384–395 (2017)
  * (36) Otazo, R., Candès, E., Sodickson, D.K.: Low-rank plus sparse matrix decomposition for accelerated dynamic MRI with separation of background and dynamic components.  Magn. Reson. Med. 73(3), 1125–1136 (2015)
  * (37) Otazo, R., Kim, D., Axel, L., Sodickson, D.K.: Combination of compressed sensing and parallel imaging for highly accelerated first-pass cardiac perfusion MRI.  Magn. Reson. Med. 64(3), 767–776 (2010)
  * (38) Pesapane, F., Codari, M., Sardanelli, F.: Artificial intelligence in medical imaging: threat or opportunity? Radiologists again at the forefront of innovation in medicine.  European radiology experimental 2(1), 35 (2018)
  * (39) Poddar, S., Jacob, M.: Dynamic MRI using smoothness regularization on manifolds (SToRM).  IEEE Trans. Med. Imag. 35(4), 1106–1115 (2015)
  * (40) Pruessmann, K.P., Weiger, M., Scheidegger, M.B., Boesiger, P.: SENSE: sensitivity encoding for fast MRI.  Magn. Reson. Med. 42(5), 952–962 (1999)
  * (41) Qin, C., Bai, W., Schlemper, J., Petersen, S.E., Piechnik, S.K., Neubauer, S., Rueckert, D.: Joint learning of motion estimation and segmentation for cardiac MR image sequences.  In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 472–480. Springer (2018)
  * (42) Qin, C., Duan, J., Hammernik, K., Schlemper, J., Küstner, T., Botnar, R., Prieto, C., Price, A.N., Hajnal, J.V., Rueckert, D.: Complementary time-frequency domain networks for dynamic parallel mr image reconstruction.  Magnetic Resonance in Medicine 86(6), 3274–3291 (2021)
  * (43) Qin, C., Schlemper, J., Caballero, J., Price, A.N., Hajnal, J.V., Rueckert, D.: Convolutional recurrent neural networks for dynamic MR image reconstruction.  IEEE Trans. Med. Imag. 38(1), 280–290 (2019)
  * (44) Qin, C., Schlemper, J., Duan, J., Seegoolam, G., Price, A., Hajnal, J., Rueckert, D.: k-t NEXT: Dynamic MR image reconstruction exploiting spatio-temporal correlations.  In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 505–513. Springer (2019)
  * (45) Qin, C., Schlemper, J., Hammernik, K., Duan, J., Summers, R.M., Rueckert, D.: Deep network interpolation for accelerated parallel MR image reconstruction.  Proc. ISMRM 28th Annu. Meeting Exhibit (2020)
  * (46) Qin, C., Wang, S., Chen, C., Qiu, H., Bai, W., Rueckert, D.: Biomechanics-informed neural networks for myocardial motion tracking in MRI.  In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 296–306 (2020)
  * (47) Quan, T.M., Nguyen-Duc, T., Jeong, W.K.: Compressed sensing MRI reconstruction using a generative adversarial network with a cyclic loss.  IEEE Trans. Med. Imag. 37(6), 1488–1497 (2018)
  * (48) Ramani, S., Fessler, J.A.: Parallel MR image reconstruction using augmented Lagrangian methods.  IEEE Trans Med Imag 30(3), 694–706 (2010)
  * (49) Schlemper, J., Caballero, J., Hajnal, J.V., Price, A.N., Rueckert, D.: A deep cascade of convolutional neural networks for MR image reconstruction.  In: IPMI, pp. 647–658. Springer (2017)
  * (50) Schlemper, J., Caballero, J., Hajnal, J.V., Price, A.N., Rueckert, D.: A deep cascade of convolutional neural networks for dynamic MR image reconstruction.  IEEE Trans. Med. Imag. 37(2), 491–503 (2018)
  * (51) Seegoolam, G., Schlemper, J., Qin, C., Price, A., Hajnal, J., Rueckert, D.: Exploiting motion for deep learning reconstruction of extremely-undersampled dynamic MRI.  In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 704–712. Springer (2019)
  * (52) Seitzer, M., Yang, G., Schlemper, J., Oktay, O., Würfl, T., Christlein, V., Wong, T., Mohiaddin, R., Firmin, D., Keegan, J., et al.: Adversarial and perceptual refinement for compressed sensing MRI reconstruction.  In: International conference on medical image computing and computer-assisted intervention, pp. 232–240. Springer (2018)
  * (53) Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition.  In: International Conference on Learning Representations (2015)
  * (54) Sodickson, D.K., Manning, W.J.: Simultaneous acquisition of spatial harmonics (SMASH): fast imaging with radiofrequency coil arrays.  Magn. Reson. Med. 38(4), 591–603 (1997)
  * (55) Wang, P., Chen, E.Z., Chen, T., Patel, V.M., Sun, S.: Pyramid convolutional RNN for MRI reconstruction.  arXiv preprint arXiv:1912.00543 (2019)
  * (56) Wang, S., Ke, Z., Cheng, H., Jia, S., Ying, L., Zheng, H., Liang, D.: DIMENSION: Dynamic MR imaging with both k-space and spatial prior knowledge obtained via multi-supervised network training.  NMR in Biomedicine p. e4131 (2019)
  * (57) Wang, Y., Yang, J., Yin, W., Zhang, Y.: A new alternating minimization algorithm for total variation image reconstruction.  SIAM Journal on Imaging Sciences 1(3), 248–272 (2008)
  * (58) Yang, G., Yu, S., Dong, H., Slabaugh, G., Dragotti, P.L., Ye, X., Liu, F., Arridge, S., Keegan, J., Guo, Y., et al.: DAGAN: Deep de-aliasing generative adversarial networks for fast compressed sensing MRI reconstruction.  IEEE Trans. Med. Imag. 37(6), 1310–1321 (2017)
  * (59) Ye, J.C.: Compressed sensing MRI: a review from signal processing perspective.  BMC Biomedical Engineering 1(1), 1–17 (2019)
  * (60) Yoon, H., Kim, K.S., Kim, D., Bresler, Y., Ye, J.C.: Motion adaptive patch-based low-rank approach for compressed sensing cardiac cine MRI.  IEEE Trans Med Imag 33(11), 2069–2085 (2014)
  * (61) Zhao, B., Haldar, J.P., Christodoulou, A.G., Liang, Z.P.: Image reconstruction from highly undersampled (k, t)-space data with joint partial separability and sparsity constraints.  IEEE Trans Med Imag 31(9), 1809–1820 (2012)
  * (62) Zhou, B., Zhou, S.K.: DuDoRNet: Learning a dual-domain recurrent network for fast MRI reconstruction with deep T1 prior.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4273–4282 (2020)