#  Axion Quality Problem and Non-Minimal Gravitational Coupling
in the Palatini Formulation

Dhong Yeon Cheong<EMAIL_ADDRESS>Department of Physics and IPAP,
Yonsei University, Seoul 03722, Republic of Korea  Koichi Hamaguchi  hama@hep-
th.phys.s.u-tokyo.ac.jp Department of Physics, University of Tokyo, Tokyo
113–0033, Japan  Kavli Institute for the Physics and Mathematics of the
Universe (Kavli IPMU), University of Tokyo, Kashiwa 277–8583, Japan  Yoshiki
Kanazawa<EMAIL_ADDRESS>Department of Physics,
University of Tokyo, Tokyo 113–0033, Japan  Sung Mook Lee
<EMAIL_ADDRESS>Department of Physics and IPAP, Yonsei University,
Seoul 03722, Republic of Korea  Theoretical Physics Department, CERN, 1211
Geneva 23, Switzerland  Natsumi Nagata<EMAIL_ADDRESS>Department of Physics, University of Tokyo, Tokyo 113–0033, Japan  Seong Chan
Park<EMAIL_ADDRESS>Department of Physics and IPAP, Yonsei University,
Seoul 03722, Republic of Korea  Korea Institute for Advanced Study, Seoul
02455, Republic of Korea

###### Abstract

In axion models, the global U(1) Peccei-Quinn (PQ) symmetry is explicitly
broken by non-perturbative effects of gravity, such as axionic wormholes. The
gravitational violation of the PQ symmetry due to wormholes is large enough to
invalidate the PQ mechanism, which is entitled as the axion quality problem.
Recently, a novel solution to this quality problem was suggested, where the
non-minimal coupling of the axion field to gravity $\xi$ is introduced to
suppress the wormhole contribution. In this work, we revisit the problem in a
different but equally valid formulation of gravity, namely the Palatini
formulation, where the Ricci scalar is solely determined by connection. We
first find the axionic wormhole solution in the Palatini formulation, taking
the full dynamical radial mode as well as the axial mode, then show that the
quality problem is still resolved with the non-minimal coupling $\xi$. The
requested lower bound of $\xi$ in the Palatini formulation turns out to be
slightly higher than that in the metric formulation.

††preprint: CERN-TH-2022-167

##  I Introduction

The axion Weinberg (1978); Wilczek (1978),  which is a pseudo-Nambu-Goldstone
boson associated with the spontaneous breaking of the global U(1) Peccei-Quinn
(PQ) symmetry Peccei and Quinn (1977a, b), is introduced as a solution to the
strong CP problem. Below the QCD scale, the axion field obtains a periodic
potential induced by QCD instantons to settle down to an exactly CP-conserving
vacuum. However, even a tiny violation of this symmetry can  jeopardize the PQ
solution to the strong CP problem,  leading to the mechanism being extremely
sensitive to the quality of the PQ symmetry Dine (2022).

From an effective field theory perspective, we expect that there should be
higher dimensional operators of the form $\Delta V\sim
c\,\Phi^{n}/M_{P}^{n-4}\sim\frac{cf_{a}^{n}}{M_{P}^{n-4}}\cos\left(\frac{n\phi}{f_{a}}+\delta\right)$
where $\Phi$ is a complex PQ scalar and $\phi$ is the associated axion field
with a decay constant $f_{a}$ Hook (2019). Indeed, it is expected that gravity
may spoil any global symmetry  at the least Kallosh  _et al._ (1995); Banks
and Seiberg (2011); Witten (2018); Harlow and Ooguri (2019, 2021) and the PQ
symmetry may not be an exception. This violation of the U(1) symmetry
generally displaces the axion field from the CP-conserving minimum of the
axion potential. If the deviation is too large, the induced value of the
neutron electric dipole moment becomes incompatible with the experimental
upper bound Abel  _et al._ (2020), and hence the PQ mechanism does not work as
a viable solution to the strong CP problem. This is called the ‘axion quality
problem’ Kamionkowski and March-Russell (1992); Holman  _et al._ (1992); Barr
and Seckel (1992) and there have been many possible solutions suggested in the
literature Witten (1984); Randall (1992); Cheng and Kaplan (2001); Izawa  _et
al._ (2002); Choi (2004); Lillard and Tait (2018).

Meanwhile, an axionic wormhole Giddings and Strominger (1988a); Lee (1988);
Kallosh  _et al._ (1995), which is a gravitational instanton in Euclidean
spacetime, is a well-known  case that explicitly shows the gravitational
violation of the U(1) symmetry. It has a non-trivial topology characterized by
a global U(1) charge and induces effective local operators that violate the
U(1) symmetry  through computing quantum transition amplitudes in a semi-
classical approximation Coleman (1988); Giddings and Strominger (1988b); Rey
(1989); Abbott and Wise (1989). (See also Refs. Alonso and Urbano (2019);
Hebecker  _et al._ (2018).)  Assuming these non-perturbative effects as the
source of symmetry breaking, the coefficients of these operators have a
characteristic exponentially suppressed term consisting of a coefficient of
order $c\sim e^{-S}$ with $S$ corresponding to the wormhole action Abbott and
Wise (1989); Coleman and Lee (1990).111 See, for example, Ref. Svrcek and
Witten (2006) for other sources of explicit PQ-violation such as stringy
instantons. In order to  issue the quality problem, the wormhole action should
be large, $S\gtrsim 190$, for its effect to be suppressed enough Kallosh  _et
al._ (1995); Alvey and Escudero (2021).

Whether these wormholes induce the quality problem depends sensitively on the
UV model that realizes the axion at low energies Alvey and Escudero (2021). In
a simple model with a periodic scalar field Giddings and Strominger (1988a);
Lee (1988), the wormhole action is $S\sim{\color[rgb]{0,0,0}n}M_{P}/f_{a}$;
thus the value becomes large enough when the decay constant is small as
$f_{a}\lesssim 10^{16}~{}\rm{GeV}$. However, the result drastically changes if
the axion corresponds to a phase component of a complex scalar field together
with a dynamical radial component, $\Phi=fe^{i\theta}/\sqrt{2}$. In this case,
the field value of $f$ stays near the Planck scale at the throat of the
wormhole, and the size of the wormhole also becomes close to the Planck
length. The wormhole action now scales logarithmically as
$S\sim{\color[rgb]{0,0,0}n}\log(M_{P}/f_{a})$ and it cannot grow sufficiently.
Therefore, we still suffer from the quality problem Kallosh  _et al._ (1995);
Alonso and Urbano (2019); Alvey and Escudero (2021).

A recent work Hamaguchi  _et al._ (2022) suggested a  novel solution to the
quality problem even in the presence of the dynamical radial field, by
introducing a non-minimal gravitational coupling of a complex scalar field,
$\xi$. It is found that the quality problem is avoided for $\xi\gtrsim 2\times
10^{3}$ having a sufficiently large wormhole action. We take this solution
seriously, as the non-minimal coupling $\xi$ is allowed in any effective
theory as long as the term is consistent with the symmetries of the theory.
Indeed, the implications of the term have been widely explored in inflationary
cosmology Futamase and Maeda (1989); Bezrukov and Shaposhnikov (2008); Park
and Yamaguchi (2008); Bauer and Demir (2008); Tamanini and Contaldi (2011);
Hertzberg (2010); Hamada  _et al._ (2014, 2015); Jinno  _et al._ (2019, 2020);
Cheong  _et al._ (2022a, b).

One potential loophole of the suggested solution in Ref. Hamaguchi  _et al._
(2022) is noted: gravity can be equally valid when formulated in different
ways. Indeed, distinctively from the conventional metric formulation taken in
Ref. Hamaguchi  _et al._ (2022), we can alternatively choose the Palatini
formulation of gravity Einstein (1925); Ferraris  _et al._ (1982) that takes
the connection as an independent degree of freedom apart from the metric.
While both the metric and Palatini formulations are equivalent within the so-
called minimal gravity model of the Einstein-Hilbert action, the equivalence
is generally broken in non-minimal models Ferraris  _et al._ (1994); Magnano
and Sokolowski (1994). Since we are not able to  distinctively rule out either
formulation at the current stage of our experimental knowledge, in this paper,
we pursue to reconsider the Palatini formulation and examine if the noble
solution to the quality problem remains valid. 222The intrinsic difficulty of
the experimental probes becomes more transparent when one considers the
Einstein frame. In this frame, the gravity sector becomes canonical in both
formulations of gravity and all effects are recast to the scalar potential
deformation, which non-minimally couples to the Ricci scalar in the Jordan
frame. As long as we are not able to probe scalar field values  large enough
for the deformation of the potential  to be significant, $f\gtrsim M_{P}/\xi$,
the differences in observables become negligible. On the other hand, for
$f(R)$ gravity, there could be constraints for Palatini formulation coming
from the fact that it violates the equivalence principle Olmo (2007); Wu  _et
al._ (2018).

##  II Model

The action for a complex scalar field $\Phi=\frac{f}{\sqrt{2}}e^{i\theta}$
with non-minimal coupling with gravity in the Palatini formulation is given as

$\displaystyle S$ $\displaystyle=\int
d^{4}x\sqrt{|g|}\left[-\frac{M^{2}+2\xi|\Phi|^{2}}{2}R(\Gamma)+|\partial_{\mu}\Phi|^{2}+V(|\Phi|)\right],$
(1)

where $g=\det g_{\mu\nu}$, the mass parameter $M$ is defined as
$M^{2}=M_{P}^{2}-\xi f_{a}^{2}$ with the Planck mass $M_{P}=1/\sqrt{8\pi
G}\approx 2.4\times 10^{18}~{}{\rm GeV}$ and
$V(|\Phi|)=\lambda\left(|\Phi|^{2}-f_{a}^{2}/2\right)^{2}$. The vacuum
expectation value of $\Phi$ is
$\sqrt{\langle\Phi^{\dagger}\Phi\rangle}=f_{a}/\sqrt{2}$ such that the
gravitational coupling becomes canonical at the vacuum. In order to keep the
correct sign of the kinetic term of the graviton, we request $M^{2}\geq 0$, or
equivalently $\xi\leq M_{P}^{2}/f_{a}^{2}$. The Ricci scalar $R(\Gamma)$ is
obtained from the Ricci tensor, which is explicitly given as

$\displaystyle
R_{\mu\nu}(\Gamma)=\partial_{\mu}\Gamma^{\lambda}_{\lambda\nu}-\partial_{\lambda}\Gamma^{\lambda}_{\mu\nu}+\Gamma^{\lambda}_{\mu\sigma}\Gamma^{\sigma}_{\lambda\nu}-\Gamma^{\sigma}_{\mu\nu}\Gamma^{\lambda}_{\lambda\sigma}.$
(2)

Due to the absence of second-order derivatives, unlike the metric formulation,
the Gibbons-Hawking-York boundary term York (1972); Gibbons and Hawking (1977)
is not necessary for the Palatini formulation. We take the Euclidean geometry
with spherical symmetry, $ds^{2}=dr^{2}+a(r)^{2}d^{2}\Omega_{3}$, where $r$ is
the Euclidean time, $d^{2}\Omega_{3}$ is the line element on the three-
dimensional unit sphere and $a$ is the radius of the sphere. We assume that
$f$ and $\theta$ depend only on $r$, taking the spherical symmetry into
account.

##  III Analysis in Palatini formulation

###  III.1 Wormhole Solutions

In a semi-classical approximation, a wormhole is a saddle-point solution in
the Euclidean path integral, with a boundary condition on the canonical
momentum of the axion field  Kallosh  _et al._ (1995); Alonso and Urbano
(2019); Hebecker  _et al._ (2018). Expanding Eq. (1), we have

$\displaystyle S=\int d^{4}x\sqrt{|g|}$
$\displaystyle\left[-\frac{M_{P}^{2}}{2}\Omega^{2}(f)R+\frac{1}{2}(\partial_{\mu}f)^{2}\right.$
$\displaystyle\left.\quad\quad\quad\quad+\frac{1}{2}f^{2}(\partial_{\mu}\theta)^{2}+V(f)\right]~{},$
(3)

where

$\displaystyle\Omega^{2}(f)\equiv 1+\frac{\xi(f^{2}-f_{a}^{2})}{M_{P}^{2}}.$
(4)

The variation with respect to $\theta$ gives

$\displaystyle\partial_{\mu}\left(\sqrt{g}f^{2}\partial^{\mu}\theta\right)=0,$
(5)

motivating us to define a conserved current,

$\displaystyle J^{\mu}=\sqrt{g}f^{2}\partial^{\mu}\theta(r),$ (6)

which is associated with shift symmetry. The conserved charge is quantized as

$\displaystyle 2\pi^{2}a^{3}f^{2}\theta^{\prime}(r)=n\in\mathbb{Z},$ (7)

due to the $2\pi$-periodicity of the axion and the spherical symmetry of the
metric. The prime($\prime$) denotes the derivative with respect to $r$. The
integer $n\neq 0$ corresponds to the charge going through the wormhole and
characterizes a wormhole solution.

To take this constraint into account properly, one can either impose the
condition Eq. (7) from the beginning and plug this into the action in Eq. (3),
or introduce the constraint as a Lagrange multiplier Lee (1988); Alvey and
Escudero (2021).333For rigorous treatments on this point, see Ref. Coleman and
Lee (1990). For sure, both ways give identical results. Note that in the
former way, the kinetic term of $\theta$ gives an additional effective
potential that corresponds to a ‘centrifugal force’. As we will see, this is
the origin of the large field value at the wormhole throat once the radial
field $f$ is taken to be dynamical.

From the variational principle with respect to $\Gamma^{\lambda}_{\mu\nu}$, we
find

$\displaystyle\nabla_{\lambda}[M_{P}^{2}\Omega^{2}(f)\sqrt{g}g^{\mu\nu}]=0,$
(8)

or

$\displaystyle\Gamma^{\lambda}_{\mu\nu}=\bar{\Gamma}^{\lambda}_{\mu\nu}+\delta^{\lambda}_{\mu}\partial_{\nu}\omega+\delta^{\lambda}_{\nu}\partial_{\mu}\omega-
g_{\mu\nu}\partial^{\lambda}\omega,$ (9)

where

$\displaystyle\omega(f)\equiv\log|\Omega(f)|.$ (10)

The first term in Eq. (9) is the Levi-Civita connection,

$\displaystyle\bar{\Gamma}^{\lambda}_{\mu\nu}=\frac{1}{2}g^{\lambda\alpha}(g_{\mu\alpha,\nu}+g_{\alpha\nu,\mu}-g_{\mu\nu,\alpha})~{},$
(11)

while the last three terms are additional terms that are absent in the metric
formulation and depend on the non-minimal coupling. We also obtain

$\displaystyle R$ $\displaystyle=g^{\mu\nu}R_{\mu\nu}(\Gamma)$
$\displaystyle=-6\left(\frac{a^{\prime\prime}}{a}+\frac{a^{\prime
2}}{a^{2}}-\frac{1}{a^{2}}\right)-6\left[{\omega}^{\prime
2}+{\omega}^{\prime\prime}+3\frac{a^{\prime}}{a}{\omega}^{\prime}\right].$
(12)

From the variation with respect to $g_{\mu\nu}$, we find

$\displaystyle\Omega^{2}\left[a^{\prime
2}-1+2aa^{\prime}{\omega}^{\prime}+a^{2}{\omega}^{\prime 2}\right]$
$\displaystyle\quad=-\frac{a^{2}}{3M_{P}^{2}}\left[-\frac{1}{2}f^{\prime
2}+V(f)+\frac{n^{2}}{8\pi^{4}f^{2}a^{6}}\right],$ (13)

and

$\displaystyle\Omega^{2}\left[2aa^{\prime\prime}+a^{\prime
2}-1+4aa^{\prime}{\omega}^{\prime}+a^{2}{\omega}^{\prime
2}+2a^{2}{\omega}^{\prime\prime}\right]$
$\displaystyle\quad=-\frac{a^{2}}{M_{P}^{2}}\left[\frac{1}{2}f^{\prime
2}+V(f)-\frac{n^{2}}{8\pi^{4}f^{2}a^{6}}\right],$ (14)

while from the variation with respect to $f$, we obtain

$\displaystyle
f^{\prime\prime}+3\frac{a^{\prime}}{a}f^{\prime}-\frac{dV}{df}+\frac{n^{2}}{4\pi^{4}f^{3}a^{6}}$
$\displaystyle\quad=6\xi f\left[\frac{a^{\prime\prime}}{a}+\frac{a^{\prime
2}}{a^{2}}-\frac{1}{a^{2}}+{\omega}^{\prime
2}+{\omega}^{\prime\prime}+3\frac{a^{\prime}}{a}{\omega}^{\prime}\right].$
(15)

We solve Eqs. (13), (14) and (15) numerically with the following boundary
conditions,

$\displaystyle a^{\prime}(0)=0,$ $\displaystyle f^{\prime}(0)=0,$
$\displaystyle f(\infty)=f_{a}.$ (16)

For convenience, we introduce dimensionless parameters,

$\displaystyle\rho\equiv\sqrt{3\lambda}M_{P}r,$ $\displaystyle
A\equiv\sqrt{3\lambda}M_{P}a,$ $\displaystyle F\equiv\frac{f}{\sqrt{3}M_{P}}.$
(17)

We show the graphs of $F(\rho)$ and $A(\rho)$ in Fig. 1 and Fig. 2,
respectively. First, for $\xi=0$ and $\xi=M_{P}^{2}/f_{a}^{2}$, both the
metric (dashed) and Palatini (solid) formulations give  identical results. In
particular, as found in Ref. Hamaguchi  _et al._ (2022), the solution for
$\xi=M_{P}^{2}/f_{a}^{2}$ (the induced gravity model) is identical to the
Giddings-Strominger (GS) wormhole corresponding to $f(r)=f_{a}$ Giddings and
Strominger (1988a).

Figure 1: $F(\rho)$ for several values of $\xi$. The solid and dashed lines
are for the Palatini and metric formulations, respectively.  Figure 2:
$A(\rho)$ for several values of $\xi$. The solid and dashed lines are for the
Palatini and metric formulations, respectively.

To understand this trend, it is illuminating to consider the action Eq. (3) in
the Einstein frame, with a metric redefinition
$g_{\mu\nu}\rightarrow\Omega^{-2}g_{\mu\nu}$ Accetta  _et al._ (1990). In this
frame,

$\displaystyle S_{E}=\int d^{4}x\sqrt{g}$
$\displaystyle\left[-\frac{M_{P}^{2}}{2}\left(R-\frac{3\zeta(\partial_{\mu}\Omega^{2})^{2}}{2\Omega^{4}}\right)\right.$
$\displaystyle~{}~{}\left.+\frac{1}{2\Omega^{2}}(\partial_{\mu}f)^{2}+\frac{1}{2\Omega^{2}}f^{2}(\partial_{\mu}\theta)^{2}+\frac{V}{\Omega^{4}}\right]~{},$
(18)

where $\zeta=0,1$ in the Palatini and metric formalisms, respectively, and we
have not canonicalized the kinetic term of $f$ as it is irrelevant for the
discussion. Note that, in the induced-gravity limit with
$\Omega^{2}=f^{2}/f_{a}^{2}$, the coefficient of $(\partial_{\mu}\theta)^{2}$
becomes constant. This shows the decoupling between $f$ and $\theta$ in this
limit, hence the reason for staying in the GS solution with the absence of the
additional force more explicitly.

For intermediate values of $\xi$, the behavior of the solutions differs
between the metric and Palatini formulations. The difference is especially
noticeable at the wormhole throat, as shown in Fig. 3. This figure shows that
in the Palatini case, the radial field $f$ stays near the Planck scale
(corresponding to $F(0)\sim 1$)  except for extremely close values to
$\xi=M_{P}^{2}/f_{a}^{2}$ and quickly transits to the GS wormhole near the
induced-gravity limit $\xi=M_{P}^{2}/f_{a}^{2}$.  As $\xi$ increases from 0,
the radius of the wormhole throat $a(0)$ initially decreases and then
increases drastically near $\xi=M_{P}^{2}/f_{a}^{2}$ in the Palatini case,
while it monotonically increases in the metric case. We have also confirmed
that the $\lambda$-dependence of the wormhole solutions for small $\lambda$s
is negligible, resulting in identical wormhole geometries for both the metric
and Palatini cases, leading to robust predictions.

One may wonder if there is  no problem with the size of the wormhole throat
being comparable to or even smaller than the Planck length. In fact, such a
large non-minimal coupling also introduces a perturbative unitarity cutoff
$\Lambda_{J}$ to the theory, where $J$ denotes the Jordan frame Burgess  _et
al._ (2009); Barbon and Espinosa (2009); Burgess  _et al._ (2010); Hertzberg
(2010); Bezrukov  _et al._ (2011); Bauer and Demir (2011); Shaposhnikov  _et
al._ (2020); Antoniadis  _et al._ (2021).444The cutoff scales in the Einstein
frame, $\Lambda_{E}$, and in the Jordan frame, $\Lambda_{J}$, are related
through a conformal factor $\Omega$, as $\Lambda_{J}=\Omega\Lambda_{E}$. It is
known that,555We note that the perturbative unitarity cutoff of the complex
U(1) scalar is higher than the SU(2) scalar doublet as the SM Higgs. This
difference is nicely discussed and summarized in the addendum of Ref.
Antoniadis  _et al._ (2021). for small field values

$\displaystyle\Lambda_{J}\left(f\ll\frac{M_{P}}{\sqrt{\xi}}\right)\simeq\begin{dcases}M_{P}/\xi&\text{(metric)}\\\
M_{P}/\sqrt{\xi}&\text{(Palatini)}\end{dcases}$ (19)

while for large field values

$\displaystyle\Lambda_{J}\left(f\gg\frac{M_{P}}{\sqrt{\xi}}\right)\simeq\begin{dcases}\xi
f^{2}/M_{P}&\text{(metric)}\\\
\sqrt{\xi}f^{2}/M_{P}&\text{(Palatini)}.\end{dcases}$ (20)

Near the throat, the field value of $f$ can be as large as the Planck scale,
but the cutoff scale also increases as $\Lambda_{J}^{(M)}\sim\xi M_{P}$ and
$\Lambda_{J}^{(P)}\sim\sqrt{\xi}M_{P}$ for the metric and Palatini cases,
respectively, giving a cutoff much larger than the Planck scale. This partly
justifies the self-consistency of our calculation666We, however, note in
passing that although our calculation for wormholes is self-consistent, if we
consider $f$ as an inflaton, a large value of $\xi$ as in Eq. (22) may still
give rise to a unitarity problem at the stage of preheating for the metric
formulation Ema  _et al._ (2017); Sfakianakis and van de Vis (2019); Ema  _et
al._ (2021). On the other hand, if the quartic coupling $\lambda$ is less than
$\mathcal{O}(1)$, the unitarity problem may not occur for the Palatini
formulation, for any value of $\xi$ Ema  _et al._ (2021). This difference
between the two formulations is highly related to the dependence of the cutoff
scale on $\xi$ as given in Eq. (19).  in the semi-classical regime while the
wormhole throat becomes smaller than the Planck length for large $\xi$ in the
Palatini case, as depicted in Fig. 2.

Figure 3: $F(0)$ and $A(0)$ as functions of $\xi$. The solid and dashed lines
are for the Palatini and metric formulations, respectively.

###  III.2 Quality Problem

Figure 4:  Values of wormhole action as a function of $\xi$. The blue-solid
and red-dashed lines are for the Palatini and metric formulations,
respectively. The horizontal gray line shows a lower bound to solve the
quality problem, $S=190$.

The PQ-violating operators are exponentially suppressed by the wormhole
action. The wormhole action is almost proportional to the PQ charge $n$ for a
given $\xi$. Thus, we focus on the contributions of the wormholes with $n=1$
to the quality problem. We now compute the wormhole action,

$\displaystyle S=2\pi^{2}\int_{0}^{\infty}dr~{}a^{3}\left[f^{\prime
2}+3M_{P}^{2}\Omega^{2}\left\\{\frac{a^{\prime\prime}}{a}+\frac{a^{\prime}}{a}{\omega}^{\prime}+{\omega}^{\prime\prime}\right\\}\right].$
(21)

In Fig. 4, we show the wormhole action as a function of $\xi$ in both Palatini
and metric formulations. For $0<\xi<M_{P}^{2}/f_{a}^{2}$, the wormhole action
in the Palatini formulation is found to be smaller than that in the metric
formulation. For $\xi=0$ and $\xi=M_{P}^{2}/f_{a}^{2}$, both formulations give
the same result, as expected. Note that the quality problem can be solved for
$S\gtrsim 190$ Kallosh  _et al._ (1995); Alvey and Escudero (2021). This
corresponds to $\xi\gtrsim 1\times 10^{4}$ in the Palatini case, which is
roughly an order of magnitude more stringent than that in the metric case,
$\xi\gtrsim 2\times 10^{3}$ Hamaguchi  _et al._ (2022). It is also worth
noting that, for the Palatini case, the wormhole action increases even when
the radius of the throat of the wormhole decreases.

Figure 5:  $f_{a}$-$\xi$ boundaries that marginally solve the quality problem
in the Palatini (blue-solid) and metric (red-dashed) formulations. Regions
above each line correspond to values that are safe from wormhole solutions
spoiling the axion quality. The right gray region corresponds to values
$\xi>M_{P}^{2}/f_{a}^{2}$.

In Fig. 5, we also depict the $f_{a}$-$\xi$ plane which shows the parameter
space that solves the axion quality problem with axionic wormholes, for both
Palatini (blue-solid) and metric (red-dashed) formulations. Note that the
required values of $\xi$ not to have the quality problem hardly depend on the
PQ symmetry breaking scale $f_{a}$. This $f_{a}$ independence may be
understood as follows. As shown in Fig. 6, for a fixed value of $\xi$, the
solutions of $F(\rho)$ and $A(\rho)$ are almost independent of $f_{a}$ except
for a large value of $\rho$. On the other hand, the contribution to the action
$S$ is dominated by the region of small $\rho$ near the wormhole throat. Thus,
the region of large $\rho$ (which is sensitive to $f_{a}$) hardly contributes
to the $S$.

Figure 6: $F(\rho)$ for several values of $f_{a}$. The solid and dashed lines
are for the Palatini and metric formulations, respectively.

##  IV Conclusion and discussions

In this work, we have discussed the effect of the non-minimal coupling to
gravity $\xi$ on the axion quality problem in the Palatini formulation. In
this formulation, the affine connection and the metric are independent a
priori. Hence, for non-zero $\xi$, the affine connection is different from the
Levi-Civita connection of metric formulation, and so are the physical
consequences compared to the usual metric formulation.

We have found that the presence of additional terms in the affine connection
does affect the wormhole solution, resulting in a smaller wormhole action
compared to that in the metric formulation. As a result, a larger value of the
non-minimal coupling, $\xi\gtrsim 1\times 10^{4}$, is required in order to
avoid the quality problem. We have also checked  that our calculation does not
violate the perturbative unitarity of the theory with large non-minimal
coupling thanks to large radial field values at the wormhole throat.

More specifically, as $\xi$ increases, the wormhole throat decreases at first
while the wormhole action value increases. For larger $\xi$ near the induced
gravity limit, there is a rapid convergence to the Gidding-Stronminger
solution. This coincidence happens in both the metric and Palatini
formulations, and could be understood as a decoupling between the axion
$\theta$ and the radial mode $f$, as explicitly shown in the Einstein frame.

Lastly, as mentioned in the introduction, a scalar field with a large non-
minimal coupling is sometimes considered  in the context of inflation which
gives a consistent fit to CMB observations. In our case, the radial field $f$
can play the role of the inflaton. To be consistent with the measurement of
the scalar amplitude of the power spectrum $A_{s}\simeq 2.1\times 10^{-9}$
Akrami  _et al._ (2020) with about  60 e-folds, we request a  substantial
value of $\xi$ for successful inflation Bezrukov and Shaposhnikov (2008);
Bauer and Demir (2008),

$\displaystyle\xi\simeq\begin{dcases}4.9\times
10^{4}\sqrt{\lambda}&(\text{metric})\\\ 1.4\times
10^{10}\lambda&(\text{Palatini}).\end{dcases}$ (22)

Note that a larger value of $\xi$ is needed for the Palatini case and it can
easily satisfy the condition to solve the axion quality problem when
$\lambda>10^{-6}$.

## Acknowledgments

NN would like to thank KIAS for inviting him to an online seminar, where this
work was initiated. We thank Miguel Escudero, Fabrizio Rompineve, and Ryusuke
Jinno for helpful discussions. This work is supported in part by the Grant-in-
Aid for Innovative Areas (No.19H05810 [KH], No.19H05802 [KH], No.18H05542
[NN]), Scientific Research B (No.20H01897 [KH and NN]), Young Scientists
(No.21K13916 [NN]), and JSPS Fellows (No.21J20445 [YK]). This work is
supported in part by National Research Foundation grants funded by the Korean
government (MSIT) (NRF-2019R1A2C1089334), (NRF-2021R1A4A2001897) [SCP] and
Korea-CERN Theoretical Physics Collaboration and Developing Young High-Energy
Theorists fellowship program (NRF-2012K1A3A2A0105178151) [SML]. SML is
supported by the Hyundai Motor Chung Mong-Koo Foundation Scholarship.

## References

  * Weinberg (1978) S. Weinberg, Phys. Rev. Lett. 40, 223 (1978).
  * Wilczek (1978) F. Wilczek, Phys. Rev. Lett. 40, 279 (1978).
  * Peccei and Quinn (1977a) R. D. Peccei and H. R. Quinn, Phys. Rev. Lett. 38, 1440 (1977a).
  * Peccei and Quinn (1977b) R. D. Peccei and H. R. Quinn, Phys. Rev. D 16, 1791 (1977b).
  * Dine (2022) M. Dine, (2022), arXiv:2207.01068 [hep-ph] .
  * Hook (2019) A. Hook, PoS TASI2018, 004 (2019), arXiv:1812.02669 [hep-ph] .
  * Kallosh  _et al._ (1995) R. Kallosh, A. D. Linde, D. A. Linde, and L. Susskind, Phys. Rev. D 52, 912 (1995), arXiv:hep-th/9502069 .
  * Banks and Seiberg (2011) T. Banks and N. Seiberg, Phys. Rev. D 83, 084019 (2011), arXiv:1011.5120 [hep-th] .
  * Witten (2018) E. Witten, Nature Phys. 14, 116 (2018), arXiv:1710.01791 [hep-th] .
  * Harlow and Ooguri (2019) D. Harlow and H. Ooguri, Phys. Rev. Lett. 122, 191601 (2019), arXiv:1810.05337 [hep-th] .
  * Harlow and Ooguri (2021) D. Harlow and H. Ooguri, Commun. Math. Phys. 383, 1669 (2021), arXiv:1810.05338 [hep-th] .
  * Abel  _et al._ (2020) C. Abel _et al._ , Phys. Rev. Lett. 124, 081803 (2020), arXiv:2001.11966 [hep-ex] .
  * Kamionkowski and March-Russell (1992) M. Kamionkowski and J. March-Russell, Phys. Lett. B 282, 137 (1992), arXiv:hep-th/9202003 .
  * Holman  _et al._ (1992) R. Holman, S. D. H. Hsu, T. W. Kephart, E. W. Kolb, R. Watkins, and L. M. Widrow, Phys. Lett. B 282, 132 (1992), arXiv:hep-ph/9203206 .
  * Barr and Seckel (1992) S. M. Barr and D. Seckel, Phys. Rev. D 46, 539 (1992).
  * Witten (1984) E. Witten, Phys. Lett. B 149, 351 (1984).
  * Randall (1992) L. Randall, Phys. Lett. B 284, 77 (1992).
  * Cheng and Kaplan (2001) H.-C. Cheng and D. E. Kaplan, (2001), arXiv:hep-ph/0103346 .
  * Izawa  _et al._ (2002) K. I. Izawa, T. Watari, and T. Yanagida, Phys. Lett. B 534, 93 (2002), arXiv:hep-ph/0202171 .
  * Choi (2004) K.-w. Choi, Phys. Rev. Lett. 92, 101602 (2004), arXiv:hep-ph/0308024 .
  * Lillard and Tait (2018) B. Lillard and T. M. P. Tait, JHEP 11, 199 (2018), arXiv:1811.03089 [hep-ph] .
  * Giddings and Strominger (1988a) S. B. Giddings and A. Strominger, Nucl. Phys. B 306, 890 (1988a).
  * Lee (1988) K.-M. Lee, Phys. Rev. Lett. 61, 263 (1988).
  * Coleman (1988) S. R. Coleman, Nucl. Phys. B 307, 867 (1988).
  * Giddings and Strominger (1988b) S. B. Giddings and A. Strominger, Nucl. Phys. B 307, 854 (1988b).
  * Rey (1989) S.-J. Rey, Phys. Rev. D 39, 3185 (1989).
  * Abbott and Wise (1989) L. F. Abbott and M. B. Wise, Nucl. Phys. B 325, 687 (1989).
  * Alonso and Urbano (2019) R. Alonso and A. Urbano, JHEP 02, 136 (2019), arXiv:1706.07415 [hep-ph] .
  * Hebecker  _et al._ (2018) A. Hebecker, T. Mikhail, and P. Soler, Front. Astron. Space Sci. 5, 35 (2018), arXiv:1807.00824 [hep-th] .
  * Coleman and Lee (1990) S. R. Coleman and K.-M. Lee, Nucl. Phys. B 329, 387 (1990).
  * Svrcek and Witten (2006) P. Svrcek and E. Witten, JHEP 06, 051 (2006), arXiv:hep-th/0605206 .
  * Alvey and Escudero (2021) J. Alvey and M. Escudero, JHEP 01, 032 (2021), arXiv:2009.03917 [hep-ph] .
  * Hamaguchi  _et al._ (2022) K. Hamaguchi, Y. Kanazawa, and N. Nagata, Phys. Rev. D 105, 076008 (2022), arXiv:2108.13245 [hep-th] .
  * Futamase and Maeda (1989) T. Futamase and K.-i. Maeda, Phys. Rev. D 39, 399 (1989).
  * Bezrukov and Shaposhnikov (2008) F. L. Bezrukov and M. Shaposhnikov, Phys. Lett. B 659, 703 (2008), arXiv:0710.3755 [hep-th] .
  * Park and Yamaguchi (2008) S. C. Park and S. Yamaguchi, JCAP 08, 009 (2008), arXiv:0801.1722 [hep-ph] .
  * Bauer and Demir (2008) F. Bauer and D. A. Demir, Phys. Lett. B 665, 222 (2008), arXiv:0803.2664 [hep-ph] .
  * Tamanini and Contaldi (2011) N. Tamanini and C. R. Contaldi, Phys. Rev. D 83, 044018 (2011), arXiv:1010.0689 [gr-qc] .
  * Hertzberg (2010) M. P. Hertzberg, JHEP 11, 023 (2010), arXiv:1002.2995 [hep-ph] .
  * Hamada  _et al._ (2014) Y. Hamada, H. Kawai, K.-y. Oda, and S. C. Park, Phys. Rev. Lett. 112, 241301 (2014), arXiv:1403.5043 [hep-ph] .
  * Hamada  _et al._ (2015) Y. Hamada, H. Kawai, K.-y. Oda, and S. C. Park, Phys. Rev. D 91, 053008 (2015), arXiv:1408.4864 [hep-ph] .
  * Jinno  _et al._ (2019) R. Jinno, K. Kaneta, K.-y. Oda, and S. C. Park, Phys. Lett. B 791, 396 (2019), arXiv:1812.11077 [gr-qc] .
  * Jinno  _et al._ (2020) R. Jinno, M. Kubota, K.-y. Oda, and S. C. Park, JCAP 03, 063 (2020), arXiv:1904.05699 [hep-ph] .
  * Cheong  _et al._ (2022a) D. Y. Cheong, S. M. Lee, and S. C. Park, JCAP 02, 029 (2022a), arXiv:2111.00825 [hep-ph] .
  * Cheong  _et al._ (2022b) D. Y. Cheong, K. Kohri, and S. C. Park, JCAP 10, 015 (2022b), arXiv:2205.14813 [hep-ph] .
  * Einstein (1925) A. Einstein, Verlag der Koeniglich-Preussichen Akademie der Wissenschaften 22, 414 (July, 1925).
  * Ferraris  _et al._ (1982) M. Ferraris, M. Francaviglia, and C. Reina, General Relativity and Gravitation 14 no. 3, 243 (Mar, 1982).
  * Ferraris  _et al._ (1994) M. Ferraris, M. Francaviglia, and I. Volovich, Class. Quant. Grav. 11, 1505 (1994), arXiv:gr-qc/9303007 .
  * Magnano and Sokolowski (1994) G. Magnano and L. M. Sokolowski, Phys. Rev. D 50, 5039 (1994), arXiv:gr-qc/9312008 .
  * Olmo (2007) G. J. Olmo, Phys. Rev. Lett. 98, 061101 (2007), arXiv:gr-qc/0612002 .
  * Wu  _et al._ (2018) J. Wu, G. Li, T. Harko, and S.-D. Liang, Eur. Phys. J. C 78, 430 (2018), arXiv:1805.07419 [gr-qc] .
  * York (1972) J. W. York, Jr., Phys. Rev. Lett. 28, 1082 (1972).
  * Gibbons and Hawking (1977) G. W. Gibbons and S. W. Hawking, Phys. Rev. D 15, 2752 (1977).
  * Accetta  _et al._ (1990) F. S. Accetta, A. Chodos, and B. Shao, Nucl. Phys. B 333, 221 (1990).
  * Burgess  _et al._ (2009) C. P. Burgess, H. M. Lee, and M. Trott, JHEP 09, 103 (2009), arXiv:0902.4465 [hep-ph] .
  * Barbon and Espinosa (2009) J. L. F. Barbon and J. R. Espinosa, Phys. Rev. D 79, 081302 (2009), arXiv:0903.0355 [hep-ph] .
  * Burgess  _et al._ (2010) C. P. Burgess, H. M. Lee, and M. Trott, JHEP 07, 007 (2010), arXiv:1002.2730 [hep-ph] .
  * Bezrukov  _et al._ (2011) F. Bezrukov, A. Magnin, M. Shaposhnikov, and S. Sibiryakov, JHEP 01, 016 (2011), arXiv:1008.5157 [hep-ph] .
  * Bauer and Demir (2011) F. Bauer and D. A. Demir, Phys. Lett. B 698, 425 (2011), arXiv:1012.2900 [hep-ph] .
  * Shaposhnikov  _et al._ (2020) M. Shaposhnikov, A. Shkerin, and S. Zell, JCAP 07, 064 (2020), arXiv:2002.07105 [hep-ph] .
  * Antoniadis  _et al._ (2021) I. Antoniadis, A. Guillen, and K. Tamvakis, JHEP 08, 018 (2021), [Addendum: JHEP 05, 074 (2022)], arXiv:2106.09390 [hep-th] .
  * Ema  _et al._ (2017) Y. Ema, R. Jinno, K. Mukaida, and K. Nakayama, JCAP 02, 045 (2017), arXiv:1609.05209 [hep-ph] .
  * Sfakianakis and van de Vis (2019) E. I. Sfakianakis and J. van de Vis, Phys. Rev. D 99, 083519 (2019), arXiv:1810.01304 [hep-ph] .
  * Ema  _et al._ (2021) Y. Ema, R. Jinno, K. Nakayama, and J. van de Vis, Phys. Rev. D 103, 103536 (2021), arXiv:2102.12501 [hep-ph] .
  * Akrami  _et al._ (2020) Y. Akrami _et al._ (Planck), Astron. Astrophys. 641, A10 (2020), arXiv:1807.06211 [astro-ph.CO] .


11institutetext: Japan Advanced Institute of Science and Technology, Ishikawa
9231292, JAPAN
11email<EMAIL_ADDRESS>
# SketchMeHow: Interactive Projection Guided Task Instruction with User
Sketches

Haoran Xie  Yichen Peng  Hange Wang  Kazunori Miyata

###### Abstract

In this work, we propose an interactive general instruction framework
SketchMeHow to guidance the common users to complete the daily tasks in real-
time. In contrast to the conventional augmented reality-based instruction
systems, the proposed framework utilizes the user sketches as system inputs to
acquire the users’ production intentions from the drawing interfaces. Given
the user sketches, the designated task instruction can be analyzed based on
the subtask division and spatial localization for each task. The projector-
camera system is adopted in the projection guidance to the end-users with the
spatial augmented reality technology. To verify the proposed framework, we
conducted two case studies of domino arrangement and bento production. From
our user studies, the proposed systems can help novice users complete the
tasks efficiently with user satisfaction. We believe the proposed SketchMeHow
can broaden the research topics in sketch-based real-world applications in
human-computer interaction111This is a pre-peer review, pre-print version of
this article. The final authenticated version is available online at
proceedings of HCII 2021(International Conference on Human-Computer
Interaction)..

###### Keywords:

User sketch Spatial augmented reality Projector camera system User guidance
Task instruction.

##  1 Introduction

In our daily lives, numerous tasks required special skills or adequate
experience, which usually frustrate the common and amateur users to make any
attempt. In addition, the daily tasks may be tedious and complex procedures,
and it is difficult and challenging to conduct creative activities. In the
research fields of creativity support in human-computer interaction, the
previous works mainly focused on the graphical user interface in the computer
environment. In this work, we aim to help the common user to complete the
designated tasks under the computed guidance in real-world applications.

To solve this issue, the common approaches are spatial computing techniques of
instruction systems in a workspace, such as augmented reality approaches with
head-mounted display and projection, see-through displays such as tablet [2,
12]. Considering the system stability and cognitive loads of the instruction
systems, we adopt the spatial augmented reality technology with projector-
camera system [5]. Especially, the depth camera was used for sensing the depth
information for instruction generation rather than the standard web cameras.
In the previous instruction works [5, 23], the users are asked to complete the
designated task with the given target where the requirement of satisfying the
users’ design intention is absent. In this work, we combine the creativity
support and instruction systems to help users to conduct tasks with their own
designs. To achieve this research purpose, we utilized the freehand user
sketches as system input which are difficult to be handled due to the
ambiguous and abstract visual representations.

Therefore, we propose an interactive instruction system, SketchMeHow that can
provide the users with projection-based guidance in real-time with spatial
augmented reality technique. As shown in Figure 1, the system input of
SketchMeHow may include freehand line drawing or color illustration. With the
help of user guidance in projector-camera systems, the completed tasks can be
achieved at low time costs and high success rates.

Figure 1: The proposed interaction framework of SketchMeHow.

To verify the proposed SketchMeHow framework, we conducted two usage
applications for task instruction from user sketches: domino block arrangement
and bento production. In the block arrangement system, the stroke input is
analyzed with the individual block, and the gradient colors are projected on
the blocks for guiding the correct positions. Finally, the domino blocks can
be collapsed successfully. In the bento production system, we adopt both
stroke and color of user sketches as input. The food ingredients are
recommended based on user input and arranged with the projected color
guidance. The accomplished bento is in compliance with the user sketches well.
From our user study and subjective questionnaires, we verified that the
proposed framework of SketchMeHow can provide the interactive projection
guidance to help non-skilled users to complete theirs desired target in
specific tasks. The proposed systems are effective and useful compared with
the conventional approaches without guidance. We believe SketchMeHow can
facilitate sketch-based systems for real-world applications.

##  2 Related Work

###  2.1 Sketch-based Interface

Freehand sketches can present the users’ design intention and have been
adopted intensively in two-dimensional content design. Teddy proposed the
sketch-based interface to construct 3D polygonal models from freehand drawn
sketches [11]. A sketch-based interface was usually adopted in the image and
shape retrieval [3] and drawing guidance [26]. Recently, the deep learning
algorithms were widely developed in the sketch user interface, such as facial
image editing [19] and drawing [10], anime image editing [15], image
translation [6] and normal map estimation [7]. Besides static applications of
image editing, the physical dynamics were controlled with user sketches such
as hair [27] and fluid [9]. In this work, we especially focus on the sketch
interfaces for task instructions with augmented reality approaches.

###  2.2 Sketch-based Guidance

It is challenging and useful to adopt the sketch-based design for real-world
applications with interactive user guidance. Sketch and run used the sketch-
based interface for robot control based on a top-down camera view [21]. Lighty
proposed the guidance system of room lights by sketching a target illumination
area on tablet devices [16]. However, these proposed systems were designed for
automatic machine control where the user guidance was absent such task
instruction. To solve this issue, Sketch2Domino proposed a sketch-based
guidance system for domino toppling production [17]. Sketch2Bento proposed the
guide system for ingredient arrangement of designing lunch boxes [24]. In this
work, we aim to propose a general instruction framework for sketch-based user
guidance in real-world applications.

###  2.3 Projection-based Guidance

The spatial augmented reality adopts interactive projection for various real-
world applications. Illuminating clay was proposed as a tangible interface for
landscape design [18], Flagg et al. worked on the implementation to guide oil
painting using the projection techniques [4]. There are several works to adopt
projection-based guidance in fabrication purposes, such as sculpting [20]. For
large-scale fabrication, BalloonFab guided the user to create the large-scale
balloon art [25]. A similar approach was used in newspaper sculpture [14]. For
daily activities, the projection guidance can also be helpful in solving the
puzzle of Rubik’s cubes [1], calligraphy practice [8], and sports training
[22]. For room-scale projection, Roomalive was proposed for multiple
projector-camera systems [13]. In contrast to these projection-based guidance
approaches, we first adapt user sketches as system control to meet user design
intentions.

##  3 Framework Overview

Figure 2: The proposed interaction framework of SketchMeHow.

Figure 2 shows the proposed framework of SketchMeHow which is composed of
three modules on the separated platforms: system input, task instruction
analysis, and projection-based guidance. For the input module, a drawing user
interface is developed with the stoke and coloring function with pen-based
interaction on a tablet or real workspace. For the task instruction analysis
module, the input hand-drawn sketches are cleaned with stroke smoothing to
exclude the infeasible instructions. According to the requirements of specific
works, we divide the whole sketch instruction into separated sub-tasks with
the elemental objects, such as individual blocks in element construction.
Finally, the locations of each task object are calculated based on the
subtasks. For the projection-based guidance module, we adopt the projector-
camera system with the depth camera. The system calibration is conducted based
on multi-pin correction to adjust the projector and camera coordinates. Then,
the current workspace is detected with depth differences from the captured
depth maps. With the calculated task instruction, the guidance information is
projected onto the workspace overlapped with the calculated operations. After
the iterative processes of subtasks, the desired productions can be achieved
with guidance.

###  3.1 Drawing Interface

The sketching interface can be developed based on the graphical user interface
or tangible physical interface. The examples of user interfaces are shown in
Figure 3. For the graphical drawing interface, the canvas is used for stroke-
based interaction as Figure 3(a). Besides the canvas, the user can select
tools from the toolbars includes pencil, eraser, line, and different shapes.
In addition, the user can select the stroke sharpness and stroke color. For
the physical drawing interface, it is convenient for the user to draw with a
pen-type input device. A simple solution is to use the retro-reflective marker
on the pen tip in a depth camera-projector system as Figure Figure 3(b). In
our case studies, we implemented both graphical and physical drawing
interfaces in the SketchMeHow framework.

Figure 3: Drawing interface can be implemented as graphical user interface on
computer (a) or the 3D spatial interface on real desk (b).

###  3.2 Spatial Augmented Reality

Due to the absence of head-mounted devices, it is more convenient in
instruction guidance systems using interactive projection [5]. As shown in
Figure 4(a), a projector-Camera System is usually adopted in the setup of
spatial augmented reality system. The analyzed results from the drawing
interface are projected on the workspace. In the prototype systems in our case
study, we used a depth camera to capture the depth map or image recognition in
infrared frames. An example setup of the proposed system is shown in 4(b). We
used the pin-points tool to ensure the appropriate calibration of projector-
camera system.

Figure 4: Spatial augmented reality system (a); an example setup for bento
design (b). Figure 5: Projection guidance for task instruction (a) and the
system usage of the proposed SketchMeHow framework with bento production (b).

###  3.3 User Guidance

In our prototype systems, the depth data of captured depth map is used to
guide the arrangement of instruction elements. For the projector calibration,
the guidance image projected onto the workspace is distorted to match the
coordinates of the depth view. In addition, we stored the pixel positions for
each color in the scaled guidance image to provide guidance for each sub-
tasks. The guidance image is projected onto the workspace with different
colors to help complete tasks as shown in Figure 5(a). We map the stored color
list to the depth data to obtain the areas, and the user can place the objects
under the corresponding tasks. For example, a black color denotes the area
where the object should not be placed. In addition, the green color refers to
the correct placement, and the red color for incorrect placement. An example
of the usage of the proposed instruction system is shown in Figure 5(b).

##  4 Case Study

To verify the proposed SketchMeHow framework, we conducted two case studies of
domino arrangement and bento production.

###  4.1 Domino Arrangement

To support designing a complex chain reaction such as domino blocks, we
implemented a sketch-based instruction system for the domino placement
according to the user customized sketch in this case study [17]. The prototype
system applied interactive projection-mapping technology to guide users to
places the domino blocks. Users were allowed to design the blueprint of the
arrangement by the freehand sketches without considering the physical
limitations of domino placement following the rules of chain reaction.

####  4.1.1 System Overview

Figure 6: The prototype system of the proposed domino arrangement system.

As shown in Figure 6, users can draw the sketch for the intention design of
the domino arrangement. The proposed system analyzed the stroke data of input
sketches to make the domino blocks collapse successfully in a valid
arrangement where the vertices are excessively dense or the angles are quite
acute. After that, the users can locate the domino blocks following the
projection guidance to confirm the target positions.

####  4.1.2 System Environment

Similar to the example setup in Figure 4, we set the working space (70 cm-high
desk) with a laser projector (LG HF80JG, 2000 lumen), and a depth camera
(Microsoft Kinect V2, the resolution of depth image is 512 x 424). The
projector and camera were set vertically 85.1 cm and 77.8 cm above the desk.
The size of the projection area was 57.2 cm x 32.1 cm. In this study, standard
domino blocks (23 mm wide, 46 mm high and 8 mm thick) were used which was
certified by the Japan Domino Association.

####  4.1.3 Projection Guidance

The users can use a pen stick with the marker to draw a sketch directly on the
projection area. The arrangement proposal according to the drawn target is
projected as visual guidance of the valid domino’s positions, so that the user
can arrange the domino blocks in the order following the design. For the
correction of the valid domino’s position, a gradient-colored circle is
projected at the center of the placed domino blocks for the user to confirm
whether they have been placed correctly. The colors of the circles are set in
gradient colors in terms of the distance from the target position. The circle
on the top of the block is in red color if the distance is larger than 2.0 cm
from the target position. The circle turns yellow about 1 cm away. The circle
turns green when is correctly placed. Note that the target positions of domino
are shown by white rectangles from the beginning.

####  4.1.4 User Study

We conducted a comparison study to verify the effectiveness of the proposed
system. we compared the domino arrangement with and without the proposed
system. In conventional approach without the systems, the participants were
asked to place the domino blocks following the hand-drawn sketches on a
printed paper without explicit guidance on the intervals between the domino
blocks. The criterion for a successful design was that all the domino blocks
could collapsed with one simple push. 12 graduate student participants (11
males, 1 female) joined the comparison study.

For our prototype system, the participants were asked to arrange the domino
blocks with the help of the visual guidance. The user can start the
arrangement from any location, because the arrangement order is non-trivial.
Note that we adopted the same blueprint design used in the conventional
approach. Both operational time costs and the results were recorded. In
addition, we conducted a questionnaire for the subjective evaluations of the
prototype system. The questionnaire had four questions: a) Which approach is
easier for arranging domino blocks? b) Which perfection do you have confidence
in? c) Which result gave you a sense of accomplishment? d) Which approach was
more interesting?

Figure 7: The prototype system of the proposed bento production system.

###  4.2 Bento Production

In this case study, we implemented an ingredient placement support system for
bento (lunch box) production based on the user hand-drawn sketches. The
prototype system allows users to make the bento following creative intentions
without relying on existing recipes.

####  4.2.1 System Overview

As illustrated in Figure 7, the user first sketches the desired arrangement
image with the drawing interface. Then, the system provides the projection
guidance to lunch box from the sketch input. Under the help of the production
guidance, the user can complete the ingredient arrangement for desired bento.
Finally, the real-time guidance is achieved based on the current working
progress.

####  4.2.2 System Environment

As shown in Figure 4, we set the similar setup for domino arrangement where
the desk height is 77 cm, a projector (EPSON dreamio EF-100, 2000 lumen), a
Kinect V2 depth camera, and a pen tablet (Wacom Intous CTL-6100WL). The
projector was placed 89 cm above the desk, while the height of camera was
80cm. The size of the lunch box was 20 cm $\times$ 13.5 cm $\times$ 4 cm.
Because the light reflection may influence the captured infrared image, we
placed a kitchen paper inside the box. We adjusted the canvas size of drawing
interface to match the size ratio of the lunch box to be used. We set the
canvas size to be 600px $\times$ 405px as shown in Figure 3.

####  4.2.3 Projection Guidance

The color selection is decided according to the colors of the prepared food
ingredients. We investigated the common food ingredients which are often used
in character bento production and determined the color of the ingredients
correlated to the sketch. The ingredients used in the prototype system were
broccoli, fried egg, crab stick, and sausage. Therefore, the drawing interface
allows the use of green, yellow, orange, and pink colors according to the
above ingredients.

We use the depth sensor to detect the positions of food ingredients placed by
the user. The depth map of the workspace without any ingredients is used as
the environment map. In the comparison with the depth map with food
ingredients, the environment map is excluded for noisy data. In our
implementation, we decided the depth difference of 8.0 mm as the threshold
value to detect the placement of food ingredients. If 70% of the projected
color area has been filled with food ingredients, the arrangement is
considered to be correct with the projected green color. On the contrary, the
red color will be projected when the ingredients are placed outside of the
target area. The arrangement is considered to be completed if the area of red
color is less than 20% of the non-target area. The arrangement area then
switches to the next step with a different color. If the subtasks of all
colors are completed, a white color is projected.

####  4.2.4 User Study

We conducted a comparison study to verify the effectiveness of the arrangement
support with the prototype system. 12 participants were recruited in two
groups: Group A with the proposed system; Group B without the proposed system.
All the participants were asked to arrange a lunch box. We evaluated the time
costs and the levels of completion with a questionnaire in five-point scale
(“did you complete your lunch box the way you wanted?”). For Group B, the
participants created bento by drawing the blueprint on paper as a reference.
For Group A, the participants used the guidance of the proposed system. All
the participants were asked to fill a questionnaire about the sketching
interface and the arrangement support system after the task. In our
questionnaire survey, we designed three questions items with 5-Point Likert
scale (1 for strongly disagree, 5 for strongly agree). The questions include:
Q1: Did you follow the guidance?; Q2: Easy to observe the guidance; Q3: Easy
to understand the guidance.

##  5 Results

In this section, the task instruction and evaluation results of two case
studies developed in SketchMeHow framework are discussed, and we explore the
further potential application with the proposed framework.

###  5.1 Domino Arrangement

Figure 8: The collapsed results (c) of domino arrangement (b) under user
sketches (a).

Figure 8 shows the final arrangements of the domino matched the participants’
sketches well. The operational time cost of with/without the domino
arrangement system is shown in Figure 9(a) . The average time cost using our
system was 140 seconds (min 109 s, max 185 s). The average time cost without
the system was 155 seconds (min 88 s, max 231 s). The variation and average of
the processing time costs were reduced using the proposed system. We counted
the number of participants who accidentally knocked down the domino blocks
during arrangement which had 4 using the proposed system, and 8 without the
system. Therefore, it is verified that SketchMeHow can not only shorten the
arrangement time cost but also provide stability.

Figure 9: Arrangement time cost (a) and the results of questionnaire (b).

Figure 9(b) shows the results of the questionnaire in 5-Point Likert scale
with the prototype system. It is verified that the proposed system has higher
evaluation than without the system except for question c). In the case of the
sense of accomplishment, the visual guidance that asked the participants to
follow the projected positions may make participants feel boring. This issue
can be improved by providing users more freedom of the modification in stroke
drawing and domino blocks arrangement.

###  5.2 Bento Production

Figure 10: Users sketches and the arranged bento box using the proposed
system.

Figure 10 shows examples of bento produced in the proposed SketchMeHow
framework. Figure 11 shows the comparison results of production time cost of
with/without the proposed system and level of completion (5 for the highest
degree of completion). The average production time cost of Group B is 280.2
seconds, while that of Group A with the proposed system was 250.6 seconds. We
found that the production time cost decreased with the proposed system. For
the level of completion, the average score for Group B was 2.4, while Group A
was 3.3. Group B achieved lower scores because the participants had difficulty
to arrange ingredients without any guidance. The participant spent more time
in production because the participants may hesitate to adjust the positions of
ingredients to match the sketch.

Figure 11: From left to right: the production time costs (a), level of
completion (b), and the questionnaire results (c).

Figure 10(c) shows The results of questionnaire that the scores of Q2 (4.1)
and Q3 (4.3) were relatively higher than Q1 (3.6), which confirmed most of the
participants were satisfied with the system guidance and the prototype system.
Scores for Q1 was above 3.0, so the participants can follow the guidance well.
Meanwhile, we found that the size of the lunch box is small and lack of
ingredient choice in the study. The user may fail to fit the ingredients to
the prompted position due to the inaccuracy between the food size and the
prompted area.

###  5.3 Discussion

####  5.3.1 Cooperative Instruction System

In contrast to the instruction systems with head-mounted displays and tablets
[5, 12], SketchMeHow can support not only individual use and also cooperative
working styles. It is possible for multiple users to collaborate in sketch
drawing and task completion.

####  5.3.2 Real World Applications

Not limited to the prototype systems developed in this work, we believe
various real world applications may benefit from the proposed SketchMeHow
framework, such as assembly tasks of hand craft and furniture construction,
and fabrication work of hand-made objects and arts.

####  5.3.3 Large Scale System

In our case study, the desk-based tasks were adopted for framework
verification. However, it is also feasible to support large scale instruction
system, such as architecture-scale fabrication. For large-scale systems, the
volumetric projection calibration may be required for accurate visual
guidance, such as layered projection mapping technique [25].

##  6 Conclusion

In this work, we proposed a general sketch-based task instruction framework,
SketchMeHow with the users’ hand-drawn sketches as system inputs. The
framework consists of the drawing interfaces in graphical and physical styles,
the computation platform for task analysis, and the projection guidance with
projector-camera systems. To verify the proposed SketchMeHow framework, we
conducted two case studies of domino arrangement and bento production. In
terms of the user studies in two prototype systems, it is verified that
SketchMeHow framework can help novice users in task instruction in low
computation cost and high completion levels. The participants were satisfied
with the usage and visual guidance provided by the prototype systems.

Based on the free comments in our case studies, the participants reported that
the accuracy of recognition should be improved. In future, we plan to adopt
depth sensor with higher resolution of depth images than the Kinect, such as
the RealSense D455 depth camera. We believe that a higher resolution depth
sensor can also reduce the noise data in prototype implementation. Under the
SketchMeHow framework, the instruction system with user sketches can improve
both the work efficiency and user satisfaction of our daily activities.

## Acknowledgement

The authors thank the anonymous reviewers for their valuable comments. We
greatly thank Yuki Mishima, Yamato Igarashi, Ryoma Miyauchi, Masahiro Okawa,
Haruka Kanayama, Shogo Yoshida, Shogo Okada for idea discussion and experiment
design. We would thank Ze Wang for sharing the codes of drawer interface. This
project has been partially funded by JAIST Research Grant, Hayao Nakayama
Foundation for Science & Technology and Culture Grant A and JSPS KAKENHI grant
JP20K19845, Japan.

## References

  * [1] Ajisaka, S., Hara, S., Matsuchi, M., Luo, S., Yoshida, S., Xie, H., Miyata, K.: Learning rubik’s cube through user operation history. In: 2020 Nicograph International (NicoInt). pp. 43–46 (2020). https://doi.org/10.1109/NicoInt50878.2020.00015, https://doi.org/10.1109/NicoInt50878.2020.00015
  * [2] Blattgerste, J., Renner, P., Strenge, B., Pfeiffer, T.: In-situ instructions exceed side-by-side instructions in augmented reality assisted assembly. In: Proceedings of the 11th PErvasive Technologies Related to Assistive Environments Conference. p. 133–140. PETRA ’18, Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3197768.3197778, https://doi.org/10.1145/3197768.3197778
  * [3] Eitz, M., Richter, R., Boubekeur, T., Hildebrand, K., Alexa, M.: Sketch-based shape retrieval. ACM Transactions on Graphics (TOG) 31(4), 31:1–31:10 (2012). https://doi.org/10.1145/2185520.2185527, https://doi.org/10.1145/2185520.2185527
  * [4] Flagg, M., Rehg, J.M.: Projector-guided painting. In: Proceedings of the 19th annual ACM symposium on User interface software and technology. pp. 235–244 (2006)
  * [5] Funk, M., Kosch, T., Schmidt, A.: Interactive worker assistance: Comparing the effects of in-situ projection, head-mounted displays, tablet, and paper instructions. p. 934–939. UbiComp ’16, Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2971648.2971706, https://doi.org/10.1145/2971648.2971706
  * [6] Ghosh, A., Zhang, R., Dokania, P.K., Wang, O., Efros, A.A., Torr, P.H., Shechtman, E.: Interactive sketch & fill: Multiclass sketch-to-image translation. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV). pp. 1171–1180. IEEE, Long Beach, CA, USA (2019). https://doi.org/10.1109/ICCV.2019.00126
  * [7] He, Y., Xie, H., Zhang, C., Yang, X., Miyata, K.: Sketch-based normal map generation with geometric sampling. In: Nakajima, M., Kim, J.G., Lie, W.N., Kemao, Q. (eds.) International Workshop on Advanced Imaging Technology (IWAIT) 2021. vol. 11766, pp. 261 – 266. International Society for Optics and Photonics, SPIE (2021). https://doi.org/10.1117/12.2590760, https://doi.org/10.1117/12.2590760
  * [8] He, Z., Xie, H., Miyata, K.: Interactive projection system for calligraphy practice. In: 2020 Nicograph International (NicoInt). pp. 55–61 (2020). https://doi.org/10.1109/NicoInt50878.2020.00018, https://doi.org/10.1109/NicoInt50878.2020.00018
  * [9] Hu, Z., Xie, H., Fukusato, T., Sato, T., Igarashi, T.: Sketch2vf: Sketch-based flow design with conditional generative adversarial network. Computer Animation and Virtual Worlds 30(3-4), e1889 (2019). https://doi.org/https://doi.org/10.1002/cav.1889, https://onlinelibrary.wiley.com/doi/abs/10.1002/cav.1889, e1889 cav.1889
  * [10] Huang, Z., Peng, Y., Hibino, T., Zhao, C., Xie, H., Fukusato, T., Miyata, K.: dualface: Two-stage drawing guidance for freehand portrait sketching. CoRR abs/2104.12297 (2021), https://arxiv.org/abs/2104.12297
  * [11] Igarashi, T., Matsuoka, S., Tanaka, H.: Teddy: A sketching interface for 3d freeform design. In: Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH). pp. 409–416. ACM, New York, USA (1999)
  * [12] Jasche, F., Hoffmann, S., Ludwig, T., Wulf, V.: Comparison of different types of augmented reality visualizations for instructions. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. CHI ’21, Association for Computing Machinery, New York, NY, USA (2021). https://doi.org/10.1145/3411764.3445724, https://doi.org/10.1145/3411764.3445724
  * [13] Jones, B., Sodhi, R., Murdock, M., Mehra, R., Benko, H., Wilson, A., Ofek, E., MacIntyre, B., Raghuvanshi, N., Shapira, L.: Roomalive: Magical experiences enabled by scalable, adaptive projector-camera units. In: Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology. p. 637–644. UIST ’14, Association for Computing Machinery, New York, NY, USA (2014). https://doi.org/10.1145/2642918.2647383, https://doi.org/10.1145/2642918.2647383
  * [14] Li, S., Yoshida, S., Arihara, K., Nakashima, K., Peng, Y., Xie, H., Sato, T., Miyata, K.: Skeleton-based interactive fabrication for large-scale newspaper sculpture. In: 2021 Nicograph International (NicoInt). pp. 74–81 (2021). https://doi.org/10.1109/NICOINT52941.2021.00021, https://doi.org/10.1109/NICOINT52941.2021.00021
  * [15] Luo, S., Xie, H., Miyata, K.: Sketch-based anime hairstyle editing with generative inpainting. In: 2021 Nicograph International (NicoInt). pp. 7–14 (2021). https://doi.org/10.1109/NICOINT52941.2021.00009, https://doi.org/10.1109/NICOINT52941.2021.00009
  * [16] Noh, S.t., Hashimoto, S., Yamanaka, D., Kamiyama, Y., Inami, M., Igarashi, T.: Design and enhancement of painting interface for room lights. The Visual Computer 30, 467–478 (05 2013). https://doi.org/10.1007/s00371-013-0872-7
  * [17] Peng, Y., Mishima, Y., Igarashi, Y., Miyauchi, R., Okawa, M., Xie, H., Miyata, K.: Sketch2domino: Interactive chain reaction design and guidance. In: 2020 Nicograph International (NicoInt). pp. 32–38 (2020). https://doi.org/10.1109/NicoInt50878.2020.00013, https://doi.org/10.1109/NicoInt50878.2020.00013
  * [18] Piper, B., Ratti, C., Ishii, H.: Illuminating clay: A 3-d tangible interface for landscape analysis. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. p. 355–362. CHI ’02, Association for Computing Machinery, New York, NY, USA (2002). https://doi.org/10.1145/503376.503439, https://doi.org/10.1145/503376.503439
  * [19] Portenier, T., Hu, Q., Szabó, A., Bigdeli, S.A., Favaro, P., Zwicker, M.: Faceshop: Deep sketch-based face image editing. ACM Transactions on Graphics (TOG) 37(4) (2018). https://doi.org/10.1145/3197517.3201393
  * [20] Rivers, A., Adams, A., Durand, F.: Sculpting by numbers. ACM Trans. Graph. 31(6) (Nov 2012). https://doi.org/10.1145/2366145.2366176, https://doi.org/10.1145/2366145.2366176
  * [21] Sakamoto, D., Honda, K., Inami, M., Igarashi, T.: Sketch and run: A stroke-based interface for home robots. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. p. 197–200. CHI ’09, Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1518701.1518733, https://doi.org/10.1145/1518701.1518733
  * [22] Sano, Y., Sato, K., Shiraishi, R., Otsuki, M.: Sports support system: Augmented ball game for filling gap between player skill levels. In: Proceedings of the 2016 ACM International Conference on Interactive Surfaces and Spaces. pp. 361–366 (2016)
  * [23] Smith, E., Semple, G., Evans, D., McRae, K., Blackwell, P.: Augmented instructions: Analysis of performance and efficiency of assembly tasks. In: Chen, J.Y.C., Fragomeni, G. (eds.) Virtual, Augmented and Mixed Reality. Industrial and Everyday Life Applications. pp. 166–177. Springer International Publishing, Cham (2020)
  * [24] Wang, H., Kanayama, H., Peng, Y., Yoshida, S., Xie, H., Okada, S., Miyata, K.: Sketch2bento: Sketch-based arrangement guidance for lunch boxes. In: 2021 Nicograph International (NicoInt). pp. 114–114 (2021). https://doi.org/10.1109/NICOINT52941.2021.00032, https://doi.org/10.1109/NICOINT52941.2021.00032
  * [25] Xie, H., Peng, Y., Chen, N., Xie, D., Chang, C.M., Miyata, K.: Balloonfab: Digital fabrication of large-scale balloon art. In: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. p. 1–6. CHI EA ’19, Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3290607.3312947, https://doi.org/10.1145/3290607.3312947
  * [26] Xie, J., Hertzmann, A., Li, W., Winnemöller, H.: Portraitsketch: face sketching assistance for novices. In: Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST). pp. 407–417. ACM, New York, USA (2014). https://doi.org/10.1145/2642918.2647399
  * [27] Xing, J., Kazi, R.H., Grossman, T., Wei, L.Y., Stam, J., Fitzmaurice, G.: Energy-brushes: Interactive tools for illustrating stylized elemental dynamics. In: Proceedings of the 29th Annual Symposium on User Interface Software and Technology. p. 755–766. UIST ’16, Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2984511.2984585, https://doi.org/10.1145/2984511.2984585