# Elliptic inequalities with nonlinear convolution and Hardy terms in cone-
like domains

Marius Ghergu111School of Mathematics and Statistics, University College
Dublin, Belfield, Dublin 4, Ireland<EMAIL_ADDRESS>ORCID:
0000-0001-9104-5295 222Institute of Mathematics Simion Stoilow of the Romanian
Academy, 21 Calea Grivitei St., 010702 Bucharest, Romania and Zhe Yu333School
of Mathematics and Statistics, University College Dublin, Belfield, Dublin 4,
Ireland<EMAIL_ADDRESS>; ORCID: 0000-0002-2261-4981

###### Abstract

We study the inequality $\displaystyle-\Delta u-\frac{\mu}{\lvert
x\rvert^{2}}u\geq(|x|^{-\alpha}*u^{p})u^{q}$ in an unbounded cone
$\mathcal{C}_{\Omega}^{\rho}\subset\mathbb{R}^{N}$ ($N\geq 2$) generated by a
subdomain $\Omega$ of the unit sphere $S^{N-1}\subset\mathbb{R}^{N},$
$p,q,\rho>0$, $\mu\in\mathbb{R}$ and $0\leq\alpha<N$. In the above,
$|x|^{-\alpha}*u^{p}$ denotes the standard convolution operator in the cone
$\mathcal{C}_{\Omega}^{\rho}$. We discuss the existence and nonexistence of
positive solutions in terms of $N,p,q,\alpha,\mu$ and $\Omega$. Extensions to
systems of inequalities are also investigated.

Keywords: Semilinear elliptic inequalities; Hardy terms; nonlinear convolution
term; cone-like domains; a priori estimates

2020 AMS MSC: 35J15; 35J47; 35A23; 35A01

##  1 Introduction

Let $\Omega$ be a subdomain of the unit sphere $\SS\subset\mathbb{R}^{N}(N\geq
2)$ such that $\Omega$ is connected and relatively open in $\SS$. Set

$\mathcal{C}_{\Omega}=\\{(r,\omega)\in\mathbb{R}^{N}:r>0\text{ and
}\omega\in\Omega\\}.$

For any $\rho>0$ and $0\leq\rho_{1}<\rho_{2}<\infty$, we define

$\begin{split}\mathcal{C}_{\Omega}^{\rho}&=\\{(r,\omega)\in\mathbb{R}^{N}:r>\rho\text{
and }\omega\in\Omega\\},\\\
\mathcal{C}_{\Omega}^{\rho_{1},\rho_{2}}&=\\{(r,\omega)\in\mathbb{R}^{N}:\rho_{1}<r<\rho_{2}\text{
and }\omega\in\Omega\\}.\end{split}$

This paper is concerned with the study of positive solutions to the following
semi-linear elliptic inequality with convolution and Hardy terms:

${\mathscr{L}}_{H}u:=-\Delta u-\frac{\mu}{\lvert
x\rvert^{2}}u\geq(|x|^{-\alpha}*u^{p})u^{q}\;\;\;\text{ in
}\;\;\mathcal{C}_{\Omega}^{\rho}.$ (1.1)

In the above inequality we assume $\mu\in\mathbb{R}$, $p,q>0$ and
$0\leq\alpha<N$. The quantity $|x|^{-\alpha}\ast u^{p}$ represents the
convolution operation in $\mathcal{C}_{\Omega}^{\rho}$ given by

$|x|^{-\alpha}\ast
u^{p}=\int_{\mathcal{C}_{\Omega}^{\rho}}\frac{u^{p}(y)}{|x-y|^{\alpha}}dy\,,\quad
x\in\mathcal{C}_{\Omega}^{\rho}.$

In this paper we shall be interested in classical positive solutions of (1.1),
that is, functions $u\in C^{2}(\mathcal{C}_{\Omega}^{\rho})\cap
C(\overline{\mathcal{C}_{\Omega}^{\rho}})$ such that

$u(x)>0\;,\;\;|x|^{-\alpha}*u^{p}<\infty\quad\mbox{ for all
}\;x\in{\mathcal{C}}_{\Omega}^{\rho},$ (1.2)

and $u$ satisfies (1.1) pointwise in $\mathcal{C}_{\Omega}^{\rho}$. We do not
prescribe any condition on $u$ on the boundary
$\partial\mathcal{C}_{\Omega}^{\rho}$ of the cone.

The prototype model

$-\Delta u=\big{(}|x|^{-1}\ast u^{2}\big{)}u\quad\mbox{ in }\;\mathbb{R}^{3},$
(1.3)

has been around for nearly a century, being introduced in 1928 by D.R. Hartree
[13, 14, 15] in the form of time-dependent equations given by

$i\psi_{t}+\Delta\psi+(|x|^{-1}\ast|\psi|^{2})\psi=\psi\quad\mbox{in
}{\mathbb{R}}^{3}\times(0,\infty).$ (1.4)

We may regard (1.4) in connection with the Schrödinger-Newton (or Schrödinger-
Poisson) equation, namely

$i\psi_{t}+\Delta\psi+\phi\psi=\psi\quad\mbox{in
}{\mathbb{R}}^{3}\times(0,\infty),$ (1.5)

where $\phi$ is a gravitational potential representing the interaction of the
particle with its own gravitational field and satisfies the Poisson equation

$\Delta\phi=4\pi G|\psi|^{2}\quad\mbox{ in }\mathbb{R}^{3}.$

Thus, one may write the gravitational potential as a convolution, namely

$\phi(x)=4\pi G\big{(}|x|^{-1}\ast|\psi|^{2}\big{)},$

which plugged into (1.5) leads us (modulo a scaling of coefficients) to (1.4).
Looking for a solution $\psi$ of (1.4) in the form $\psi(x,t)=e^{-it}u(x)$ we
see that $u$ satisfies (1.3).

More recently, the equation (1.3) is encountered in the literature under the
name of Choquard or Choquard-Pekar equation. Precisely, (1.3) was introduced
in 1954 by S.I. Pekar [26] as a model in quantum theory of a Polaron at rest
(see also [7]) and in 1976, P. Choquard used (1.3) in a certain approximation
to Hartree-Fock theory of one component plasma (see [20]). The first
mathematical study of (1.3) appeared in the mid 1970s and is due to E.H. Lieb
[20], followed by P.-L. Lions’ works [21, 22].

The inequality

$-\Delta u\geq(|x|^{-\alpha}\ast u^{p})u^{q}\quad\mbox{ in
}\mathbb{R}^{N}\setminus\overline{B}_{1},$ (1.6)

was first discussed in [25]. Let us note that (1.6) corresponds to (1.1) in
the cone $\mathcal{C}^{1}_{\Omega}$ where $\Omega=S^{N-1}$. The approach in
[25] relies on the so-called Agmon-Allegretto-Piepenbrink positivity
principle. Quasilinear versions of (1.6) (again in exterior domains) are
studied in [9] while the polyharmonic case is discussed in [10]( see also [4,
5, 6, 8, 11, 12] for more contexts in which (1.6) occurs).

To the best of our knowledge, this is the first work dealing with semilinear
elliptic inequalities in cone-like domains featuring convolution terms. The
local cases

$-\Delta u=u^{p}\quad\mbox{ and }\quad-\Delta u\geq u^{p}$

in a cone-like domain have been studied in [1, 2, 16, 17, 23]. In the current
work we derive an a priori estimate result which we provide in Proposition 3.1
below. Our method is inspired by [3] (see also [9]) where integral estimates
are obtained to study local inequalities of type
$-\Delta_{p}u\geq|x|^{\sigma}u^{q}$ in $\mathbb{R}^{N}\setminus B_{1}$. Due to
the geometric features of our cone domain $\mathcal{C}_{\Omega}^{\rho}$, we
cannot handle $W^{1,1}_{loc}$-solutions as in [3] and [9], and thus have to
restrict ourselves to the class $C^{2}(\mathcal{C}_{\Omega}^{\rho})\cap
C(\overline{\mathcal{C}_{\Omega}^{\rho}})$. The test function we construct
(see (3.2) below) is not a proper
$C^{\infty}_{c}(\mathcal{C}_{\Omega}^{\rho})$ function, rather a mere element
in the space $C^{2}(\overline{\mathcal{C}_{\Omega^{\prime}}^{R,4R}})$, where
$\Omega^{\prime}\subset\subset\Omega$ is a subdomain and $R>\rho$ is large.
Nonetheless, Proposition 3.1 below provides us with the right tool to derive
the nonexistence of a positive solution to (1.1) under some conditions on the
exponents and parameters.

Apart from the a priori estimates described above, another crucial tool in our
approach is the Hardy inequality on cone-like domains established in [23]:

$\int_{\mathcal{C}_{\Omega}^{\rho}}|\nabla\varphi|^{2}\geq
C_{H,\Omega}\int_{\mathcal{C}_{\Omega}^{\rho}}\frac{\varphi^{2}}{|x|^{2}}dx\quad\mbox{
for all }\varphi\in C^{\infty}_{c}(\mathcal{C}_{\Omega}^{\rho}),$ (1.7)

where $C_{H,\Omega}=\lambda_{1}+(\frac{N-2}{2})^{2}$ and $\lambda_{1}$ denotes
the first eigenvalue of the Laplace-Beltrami operator on $\Omega$ subject to
Dirichlet boundary condition. As obtained in [23] the constant $C_{H,\Omega}$
in (1.7) is optimal. Throughout this paper, we denote by $\phi$ the
corresponding eigenfunction which we normalize as $\phi>0$ in $\Omega$ and
$\max_{\overline{\Omega}}\phi=1$.

For $\mu\leq C_{H,\Omega}$ the quadratic equation

$\gamma(\gamma+N-2)=\lambda_{1}-\mu$ (1.8)

has two real roots which satisfy $\gamma_{*}<0\leq\gamma^{*}$.

Our first result on (1.1) establishes conditions under which no positive
solutions exist.

######  Theorem 1.1.

(Nonexistence)

Assume $0\leq\alpha<N$, $p,q>0$ and $\mu\in\mathbb{R}$.

  1. (a)

If $\mu>C_{H,\Omega}$ then (1.1) has no positive solutions.

  2. (b)

If $\mu\leq C_{H,\Omega}$ and one of the following conditions hold, then (1.1)
has no positive solutions.

    1. (i)

$\displaystyle p\leq\frac{N-\alpha}{|\gamma_{*}|}$.

    2. (ii)

$\displaystyle p+q<1+\frac{N-\alpha+2}{\lvert\gamma_{*}\rvert}$.

    3. (iii)

$\displaystyle p+q=1+\frac{N-\alpha+2}{|\gamma_{*}|}$ and $q>1$.

We next turn to the existence of positive solutions to (1.1). Under the
assumption

$q>1+\max\left\\{0,\frac{2-\alpha}{|\gamma_{*}|}\right\\},$ (1.9)

we provide optimal conditions in terms of $p,q,\alpha$ and $\mu$ for the
existence of a positive solution. Precisely, we have:

######  Theorem 1.2.

(Existence)

Assume $\mu\leq C_{H,\Omega}$, $0\leq\alpha<N$ and (1.9) holds. Then,
inequality (1.1) has a positive solution in $\mathcal{C}_{\Omega}^{\rho}$ if
and only if

$p>\frac{N-\alpha}{|\gamma_{*}|}\quad\mbox{ and }\;\quad
p+q>1+\frac{N-\alpha+2}{|\gamma_{*}|}.$ (1.10)

The existence and nonexistence regions for (1.1) in the positive quadrant of
the $pq$-plane are depicted in Figure 1 below in the case $0\leq\alpha<N$.

Figure 1: The existence region (dark shaded) and nonexistence region (light
shaded) for positive solutions of (1.1) in the case $0\leq\alpha<N$.

For $\alpha=0$ the inequality (1.1) reads

$\mathscr{L}_{H}u\geq\Big{(}\int_{\mathcal{C}_{\Omega}^{\rho}}u^{p}(y)dy\Big{)}u^{q}\quad\mbox{
in }\;\mathcal{C}_{\Omega}^{\rho}.$ (1.11)

As a consequence of Theorem 1.1 and Theorem 1.2 we deduce the following
result.

######  Corollary 1.3.

  1. (i)

If $\mu>C_{H,\Omega}$ then inequality (1.11) has no positive solutions.

  2. (ii)

If $\mu\leq C_{H,\Omega}$ and $q>1+\frac{2}{|\gamma_{*}|}$ then (1.11) has a
positive solution if and only if $p>\frac{N}{|\gamma_{*}|}$.

Our approach can be extended to system of inequalities of type

$\begin{cases}\displaystyle{\mathscr{L}}_{H}u:=-\Delta u-\frac{\mu}{\lvert
x\rvert^{2}}u\geq\big{(}|x|^{-\alpha}\ast v^{p}\big{)}v^{q}\\\\[7.22743pt]
\displaystyle{\mathscr{L}}_{H}v:=-\Delta v-\frac{\mu}{\lvert x\rvert^{2}}v\geq
u^{s}\end{cases}\quad\mbox{ in }\;\mathcal{C}_{\Omega}^{\rho},$ (1.12)

where $p,q>0$ and $s>1$. The system (1.12) contains a combination of the local
nonlinearity $u^{s}$ and nonlocal nonlinearities given by the convolution
operator. As in the case of a single inequality, we say that $(u,v)$ is a
positive solution of (1.12) if:

  * •

$u,v\in C^{2}(\mathcal{C}_{\Omega}^{\rho})\cap
C(\overline{\mathcal{C}_{\Omega}^{\rho}})$ and $\big{(}|x|^{-\alpha}\ast
v^{p}\big{)}v^{q}<\infty$ for all $x\in\mathcal{C}_{\Omega}^{\rho}$.

  * •

$u,v>0$ in $\mathcal{C}_{\Omega}^{\rho}$ satisfy (1.12) pointwise.

Semilinear elliptic systems of inequalities in cone-like domains were
discussed in [18, 24] (see also [19] for time dependent inequalities). In the
case of (1.12) we are able to identify a critical curve that separates the
existence and nonexistence regions of positive solutions to (1.12).

The main result on the system (1.12) is stated below.

######  Theorem 1.4.

Assume $\mu\in\mathbb{R}$, $p,q>0$ and $0\leq\alpha<N$.

  1. (a)

If $\mu>C_{H,\Omega}$ then (1.12) has no positive solutions.

  2. (b)

Assume $\mu\leq C_{H,\Omega}$.

    1. (i)

If $s>1$, $p+q\geq 2$ and

$p\leq\frac{N-\alpha}{|\gamma_{*}|}\quad\mbox{or}\quad
p+q<\frac{1}{s}\Big{(}1+\frac{2}{|\gamma_{*}|}\Big{)}+\frac{N-\alpha+2}{|\gamma_{*}|},$

then (1.12) has no positive solutions.

    2. (ii)

If $\displaystyle s>1+\frac{2}{|\gamma_{*}|}$, $\displaystyle
q>1+\max\Big{\\{}0,\frac{2-\alpha}{|\gamma_{*}|}\Big{\\}}$ and

$p>\frac{N-\alpha}{|\gamma_{*}|}\quad\mbox{and}\quad
p+q>\frac{1}{s}\Big{(}1+\frac{2}{|\gamma_{*}|}\Big{)}+\frac{N-\alpha+2}{|\gamma_{*}|},$

then (1.12) has positive solutions.

Theorem 1.4 yields a new critical curve given by

$p+q=\frac{1}{s}\Big{(}1+\frac{2}{|\gamma_{*}|}\Big{)}+\frac{N-\alpha+2}{|\gamma_{*}|},$

which defines the regions of existence and nonexistence of a positive solution
to (1.12). We may regard the quantity $p+q$ as the global exponent of $v$ in
the first inequality of (1.12).

The remaining of this work is organised as follows. In Section 2 we collect
some preliminary results consisting of various integral estimates. In Section
3 we derive an a priori estimate for positive solutions of (1.1). This is one
of the main tools in the proof of Theorem 1.1 which is presented in Section 4.
Section 5 is devoted to the proof of Theorem 1.2 while the proof of Theorem
1.4 is given in Section 6. Throughout this article by $c_{0}$, $c$, $C$,
$C_{1}$, $C_{2}$, $\ldots$ we denote positive constants whose value may change
on every occurrence.

##  2 Preliminary Results

In this section we derive some preliminary results for our approach.

######  Lemma 2.1.

If $u\in C^{2}(\mathcal{C}_{\Omega}^{\rho})$ is positive and satisfies
$\mathscr{L}_{H}u\geq 0$ in $\mathcal{C}_{\Omega}^{\rho}$, then $\mu\leq
C_{H,\Omega}$.

###### Proof.

Let $\varphi\in C_{c}^{\infty}(\mathcal{C}_{\Omega}^{\rho})$. Multiply the
inequality $\mathscr{L}_{H}u\geq 0$ by $\frac{\varphi^{2}}{u}$ and integrate
over $\mathcal{C}_{\Omega}^{\rho}$. We deduce

$-\int_{\mathcal{C}_{\Omega}^{\rho}}\frac{\varphi^{2}}{u}\Delta
udx\geq\mu\int_{\mathcal{C}_{\Omega}^{\rho}}\frac{\varphi^{2}}{\lvert
x\rvert^{2}}dx.$

By Green’s formula, one has

$\begin{split}-\int_{\mathcal{C}_{\Omega}^{\rho}}\frac{\varphi^{2}}{u}\Delta
udx&=\int_{\mathcal{C}_{\Omega}^{\rho}}\nabla
u\nabla\Big{(}\frac{\varphi^{2}}{u}\Big{)}dx\\\
&=\int_{\mathcal{C}_{\Omega}^{\rho}}\left(\frac{2\varphi}{u}\nabla\varphi\cdot\nabla
u-\frac{\varphi^{2}}{u^{2}}|\nabla u|^{2}\right)dx\\\
&\leq\int_{\mathcal{C}_{\Omega}^{\rho}}\lvert\nabla\varphi\rvert^{2}dx.\end{split}$

It follows that

$\int_{\mathcal{C}_{\Omega}^{\rho}}\lvert\nabla\varphi\rvert^{2}dx\geq-\int_{\mathcal{C}_{\Omega}^{\rho}}\frac{\varphi^{2}}{u}\Delta
udx\geq\mu\int_{\mathcal{C}_{\Omega}^{\rho}}\frac{\varphi^{2}}{\lvert
x\rvert^{2}}dx.$

Furthermore, by Hardy’s inequality (1.7) and the optimality of the constant
$C_{H,\Omega}$ we have $\mu\leq C_{H,\Omega}$. ∎

In the result below we derive various integral estimates on cone domains
involving the kernel $|x|^{-\alpha}$.

######  Lemma 2.2.

Let $\Omega\subset\SS$ be a subdomain, $\rho>0$ and $0\leq\alpha<N$.

  1. (i)

If $f\in L^{1}_{loc}(\mathcal{C}_{\Omega}^{\rho})$ is positive, then there
exists $C>0$ such that

$\int_{\mathcal{C}_{\Omega}^{\rho}}\frac{f(y)}{|x-y|^{\alpha}}dy\geq
C|x|^{-\alpha}\quad\mbox{ for all }x\in\mathcal{C}_{\Omega}^{2\rho}.$

  2. (ii)

If $f:\mathcal{C}_{\Omega}^{\rho}\to[0,\infty)$ is measurable and there exists
a subdomain $\Omega_{0}\subset\subset\Omega$ and $c,\beta>0$ such that
$f(x)\geq c|x|^{-\beta}$ in $\mathcal{C}_{\Omega_{0}}^{\rho}$, then

$\left\\{\begin{aligned}
&\int_{\mathcal{C}_{\Omega}^{\rho}}\frac{f(y)}{|x-y|^{\alpha}}dy=+\infty&&\quad\mbox{
if }\;N\geq\alpha+\beta\\\\[3.61371pt]
&\int_{\mathcal{C}_{\Omega}^{\rho}}\frac{f(y)}{|x-y|^{\alpha}}dy\geq
C|x|^{N-\alpha-\beta}&&\quad\mbox{ if
}\;N<\alpha+\beta\end{aligned}\right.\quad\mbox{ in
}\;\mathcal{C}_{\Omega}^{2\rho},$

where $C>0$ is a constant.

  3. (iii)

If $f:\mathcal{C}_{\Omega}^{\rho}\to[0,\infty)$ is measurable and satisfies

$\displaystyle f(x)\leq c|x|^{-\beta}\log^{\tau}(1+|x|)\quad\mbox{ in
}\;\mathcal{C}_{\Omega}^{\rho},$

for some $c>0$, $\tau\geq 0$ and $\beta>N-\alpha>0$, then there exists a
positive constant $C>0$ such that

$\int_{\mathcal{C}_{\Omega}}\frac{f(y)}{|x-y|^{\alpha}}dy\leq\left\\{\begin{aligned}
&C|x|^{N-\alpha-\beta}\log^{\tau}(1+|x|)&&\quad\mbox{ if }\;\beta<N\\\
&C|x|^{-\alpha}\log^{1+\tau}(1+|x|)&&\quad\mbox{ if }\;\beta=N\\\
&C|x|^{-\alpha}\log^{\tau}(1+|x|)&&\quad\mbox{ if
}\;\beta>N\end{aligned}\right.\quad\mbox{ in }\;\mathcal{C}_{\Omega}^{2\rho}.$

###### Proof.

(i) For any $x\in\mathcal{C}_{\Omega}^{2\rho}$ and
$y\in\mathcal{C}_{\Omega}^{3\rho/2,2\rho}$ we have $|x-y|\leq|x|+|y|\leq
2|x|$. Thus,

$|x|^{-\alpha}\ast
f\geq\int_{\mathcal{C}_{\Omega}^{3\rho/2,2\rho}}\frac{f(y)}{|x-y|^{\alpha}}dy\geq
C\int_{\mathcal{C}_{\Omega}^{3\rho/2,2\rho}}\frac{f(y)}{(2|x|)^{\alpha}}dy=C|x|^{-\alpha}.$

(ii) We use the spherical coordinates: each $x\in\mathcal{C}_{\Omega_{0}}$ can
be written as

$\begin{cases}x_{1}=r\sin\theta_{1}\sin\theta_{2}\cdots\sin\theta_{N-2}\sin\theta_{N-1},\\\\[7.22743pt]
\displaystyle x_{k}=r\cos\theta_{k-1}\prod_{j=k}^{N-1}\sin\theta_{j},\;2\leq
k\leq N-2,\\\\[7.22743pt] x_{N-1}=r\cos\theta_{N-1},\end{cases}$ (2.1)

where $r>0$. Because $\Omega_{0}$ is a subdomain of $\SS$, each $\theta_{k}$
($1\leq k\leq N-1$) belongs to a nondegenerate interval
$I_{k}\subset\mathbb{R}$.

Let $|y|\geq 2|x|\geq 2\rho$. Then $|x-y|\leq|x|+|y|\leq 3|y|/2$ so that

$\displaystyle|x|^{-\alpha}\ast f$ $\displaystyle\geq
c\int_{\mathcal{C}_{\Omega_{0}}^{2|x|}}\frac{|y|^{-\beta}}{|x-y|^{\alpha}}dy\geq
C\int_{\mathcal{C}_{\Omega_{0}}^{2|x|}}|y|^{-\alpha-\beta}dy$
$\displaystyle=C\int_{2|x|}^{\infty}r^{N-1-\alpha-\beta}\,dr=\begin{cases}+\infty&\mbox{
if }N\geq\alpha+\beta,\\\\[3.61371pt] C|x|^{N-\alpha-\beta}&\mbox{ if
}N<\alpha+\beta,\end{cases}$

and the conclusion follows.

(iii) Let $x\in\mathcal{C}_{\Omega}^{2\rho}$. We split the convolution
integral as follow:

$\int_{\mathcal{C}_{\Omega}^{\rho}}\frac{f(y)}{|x-y|^{\alpha}}dy=\left\\{\int_{\mathcal{C}_{\Omega}^{2\lvert
x\rvert}}+\int_{\mathcal{C}_{\Omega}^{\lvert x\rvert/2,2\lvert
x\rvert}}+\int_{\mathcal{C}_{\Omega}^{\rho,\lvert
x\rvert/2}}\right\\}\frac{f(y)}{|x-y|^{\alpha}}dy.$ (2.2)

To estimate the first integral in the right-hand side of (2.2) let
$y\in\mathcal{C}_{\Omega}^{2|x|}$. Then $\lvert
x-y\rvert\geq|y|-|x|\geq\frac{\lvert y\rvert}{2}$ so that

$\displaystyle\int_{\mathcal{C}_{\Omega}^{2|x|}}\frac{f(y)}{|x-y|^{\alpha}}dy$
$\displaystyle\leq
C\int_{\mathcal{C}_{\Omega}^{\rho}}\frac{|y|^{-\beta}\log^{\tau}(1+|y|)}{|x-y|^{\alpha}}dy$
(2.3) $\displaystyle\leq
C\int_{\mathcal{C}_{\Omega}^{\rho}}\frac{|y|^{-\beta}\log^{\tau}(1+|y|)}{(|y|/2)^{\alpha}}dy$
$\displaystyle\leq C\int_{2\lvert
x\rvert}^{\infty}t^{N-\alpha-\beta-1}\log^{\tau}(1+t)dt$ $\displaystyle\leq
C\lvert x\rvert^{N-\alpha-\beta}\log^{\tau}(1+\lvert x\rvert).$

For $y\in\mathcal{C}_{\Omega}^{\lvert x\rvert/2,2\lvert x\rvert}$ we have
$\lvert x-y\rvert\leq|x|+|y|\leq 3\lvert x\rvert$. Then

$\displaystyle\int_{\mathcal{C}_{\Omega}^{\lvert x\rvert/2,2\lvert
x\rvert}}\frac{f(y)}{|x-y|^{\alpha}}dy$ $\displaystyle\leq C\lvert
x\rvert^{-\beta}\log^{\tau}(1+\lvert x\rvert)\int_{\lvert x-y\rvert\leq
3\lvert x\rvert}\frac{dy}{\lvert x-y\rvert^{\alpha}}$ (2.4) $\displaystyle\leq
C\lvert x\rvert^{N-\alpha-\beta}\log^{\tau}(1+\lvert x\rvert).$

Finally, for $y\in\mathcal{C}_{\Omega}^{\rho,\lvert x\rvert/2}$ we have
$\frac{\lvert x\rvert}{2}\leq\lvert x-y\rvert\leq\frac{3\lvert x\rvert}{2}$
and

$\begin{split}\int_{\mathcal{C}_{\Omega}^{\rho,\lvert
x\rvert/2}}\frac{f(y)}{|x-y|^{\alpha}}dy&\leq C\lvert
x\rvert^{-\alpha}\int_{\mathcal{C}_{\Omega}^{\rho,\lvert x\rvert/2}}\lvert
y\rvert^{-\beta}\log^{\tau}(1+\lvert y\rvert)dy\\\\[3.61371pt] &=C\lvert
x\rvert^{-\alpha}\int_{\rho}^{\lvert
x\rvert/2}t^{N-\beta-1}\log^{\tau}(1+t)dt.\end{split}$

It follows that

$\int_{\mathcal{C}_{\Omega}^{\rho,\lvert
x\rvert/2}}\frac{f(y)}{|x-y|^{\alpha}}dy\leq C\left\\{\begin{aligned} &\lvert
x\rvert^{N-\alpha-\beta}\log^{\tau}(1+\lvert x\rvert)&&\text{ if }\beta<N,\\\
&\lvert x\rvert^{-\alpha}\log^{1+\tau}(1+\lvert x\rvert)&&\text{ if
}\beta=N,\\\ &\lvert x\rvert^{-\alpha}\log^{\tau}(1+\lvert x\rvert)&&\text{ if
}\beta>N.\end{aligned}\right.$ (2.5)

Now, we combine (2.2)-(2.5) to achieve the conclusion. ∎

######  Lemma 2.3.

(see [23, Theorem 4.2]) Suppose $\mu\leq C_{H,\Omega}$ and $u\in
C^{2}(\mathcal{C}^{\rho}_{\Omega})$ is such that

$\mathscr{L}_{H}u\geq 0\quad\mbox{ and }\quad u>0\quad\mbox{ in
}\;\mathcal{C}^{\rho}_{\Omega}.$

Then, for any subdomain $\Omega_{0}\subset\subset\Omega$ there exists
$c_{0}>0$ such that

$u(x)\geq c_{0}|x|^{\gamma_{*}}\qquad\mbox{ in
}\;\mathcal{C}^{2\rho}_{\Omega_{0}}.$ (2.6)

######  Lemma 2.4.

(see [23, Theorem 1.2])  Suppose $\mu\leq C_{H,\Omega}$, $q>1$ and
$\beta\in\mathbb{R}$. Then, the inequality

$\mathscr{L}_{H}u\geq|x|^{\beta}u^{q}\quad\mbox{ in
}\;\mathcal{C}_{\Omega}^{\rho},$

has positive solutions if and only if $\displaystyle
q>1+\frac{\beta+2}{|\gamma_{*}|}$.

##  3 An a priori estimate

In this section we establish the following a priori estimate for positive
solutions of (1.1) in $\mathcal{C}^{\rho}_{\Omega}$.

######  Proposition 3.1.

Let $\rho>0$ and $u\in C^{2}(\mathcal{C}_{\Omega}^{\rho})\cap
C(\overline{\mathcal{C}_{\Omega}^{\rho}})$ be a positive solution of (1.1) in
$\mathcal{C}_{\Omega}^{\rho}$. For $R>\max\\{\rho,e\\}$ denote
$\eta_{R}(x)=\eta(\,\lvert x\rvert/R)$ where $\eta\in C_{c}^{1}(\mathbb{R})$
is such that

$0\leq\eta\leq 1,\;\;\;\text{supp }\eta\subset[1,4]\;\;\;\text{ and
}\eta\equiv 1\text{ on }[2,3].$

Then, for any $\lambda>4$ and $0\leq m<1$, there exists
$C=C(N,m,\mu,\alpha)>0$ such that

$\left(\int_{C_{\Omega}^{\rho}}u^{(p+q-m)/2}\phi(\omega)^{1/2}\eta_{R}^{\lambda/2}dx\right)^{2}\leq
CR^{\alpha-2}\int_{C_{\Omega}^{\rho}}u^{1-m}\phi(\omega)\eta_{R}^{\lambda-2}dx,$
(3.1)

where $\phi>0$ is the first eigenfunction of $-\Delta_{S^{N-1}}$ in $\Omega$
such that $\max_{\overline{\Omega}}\phi=1$.

###### Proof.

Let $\\{\Omega_{\varepsilon}\\}_{0<\varepsilon<1}$ be a sequence of smooth
subdomains in $\Omega$ such that

$\Omega_{\varepsilon}\subset\overline{\Omega}_{\varepsilon^{\prime}}\quad\mbox{
for all }\;0<\varepsilon<\varepsilon^{\prime}<1$

and

$\bigcup_{0<\varepsilon<1}\Omega_{\varepsilon}=\Omega.$

Let $\phi_{\varepsilon}$ denote the first eigenfunction of $-\Delta_{S^{N-1}}$
in $\Omega_{\varepsilon}$ which satisfies
$\max_{\overline{\Omega}_{\varepsilon}}\phi_{\varepsilon}=1$. Notice that for
all $\omega\in\Omega$ one has $\phi_{\varepsilon}(\omega)\to\phi(\omega)$ as
$\varepsilon\to 0$. Define the test function $\varphi$ as

$\varphi=u^{-m}\phi_{\varepsilon}\eta_{R}^{\lambda}.$ (3.2)

Then, supp$\,\varphi=\overline{\mathcal{C}_{{\Omega}_{\varepsilon}}^{R,4R}}$
and

$\nabla\varphi=-mu^{-m-1}\phi_{\varepsilon}\eta_{R}^{\lambda}\nabla
u+u^{-m}\nabla(\phi_{\varepsilon}\eta_{R}^{\lambda}).$ (3.3)

Multiplying (1.1) by $\varphi$ and integrating over
$\mathcal{C}_{\Omega}^{\rho}$ we find

$\int_{\mathcal{C}_{{\Omega}_{\varepsilon}}^{\rho}}(|x|^{-\alpha}*u^{p})u^{q}\varphi\leq-\int_{\mathcal{C}_{{\Omega}_{\varepsilon}}^{\rho}}\varphi\Delta
u-\mu\int_{\mathcal{C}_{{\Omega}_{\varepsilon}}^{\rho}}\frac{u}{\lvert
x\rvert^{2}}\varphi.$

By Green’s formula and (3.3) it follows that

$\displaystyle\int_{\mathcal{C}_{\Omega_{\varepsilon}}^{\rho}}(|x|^{-\alpha}*u^{p})u^{q-m}\phi_{\varepsilon}\eta_{R}^{\lambda}$
$\displaystyle+m\int_{\mathcal{C}_{{\Omega}_{\varepsilon}}^{R,4R}}u^{-m-1}\lvert\nabla
u\rvert^{2}\phi_{\varepsilon}\eta_{R}^{\lambda}$ (3.4)
$\displaystyle\leq\int_{\mathcal{C}_{{\Omega}_{\varepsilon}}^{R,4R}}u^{-m}\nabla
u\nabla(\phi_{\varepsilon}\eta_{R}^{\lambda})-\mu\int_{\mathcal{C}_{{\Omega}_{\varepsilon}}^{R,4R}}\frac{u}{\lvert
x\rvert^{2}}\varphi$
$\displaystyle=\frac{1}{1-m}\int_{\mathcal{C}_{{\Omega}_{\varepsilon}}^{R,4R}}\nabla
u^{1-m}\nabla(\phi_{\varepsilon}\eta_{R}^{\lambda})-\mu\int_{\mathcal{C}_{{\Omega}_{\varepsilon}}^{R,4R}}\frac{u}{\lvert
x\rvert^{2}}\varphi.$

By the divergence formula, we have

$\int_{\mathcal{C}_{{\Omega}_{\varepsilon}}^{R,4R}}\nabla
u^{1-m}\nabla(\phi_{\varepsilon}\eta_{R}^{\lambda})=\int_{\partial\mathcal{C}_{{\Omega}_{\varepsilon}}^{R,4R}}u^{1-m}\nabla(\phi_{\varepsilon}\eta_{R}^{\lambda})\cdot\nu
d\sigma-\int_{\mathcal{C}_{{\Omega}_{\varepsilon}}^{R,4R}}u^{1-m}\Delta(\phi_{\varepsilon}\eta_{R}^{\lambda})dx,$
(3.5)

where $\nu$ is the outer unit normal vector at
$\partial\mathcal{C}_{{\Omega}_{\varepsilon}}^{R,4R}$. We next claim that

$\nabla(\phi_{\varepsilon}\eta_{R}^{\lambda})\cdot\nu\leq 0\;\;\;\text{ on
}\;\partial\mathcal{C}_{{\Omega}_{\varepsilon}}^{R,4R}.$ (3.6)

For $x\in\partial\mathcal{C}_{{\Omega}_{\varepsilon}}^{R,4R}$ we write
$x=(r,\omega)$ where $r=\lvert x\rvert$ and $\omega=\frac{x}{\lvert x\rvert}$.
Then, for any
$x=(r,\omega)\in\partial\mathcal{C}_{{\Omega}_{\varepsilon}}^{R,4R}$ we have

$(r,\omega)\in(\\{R\\}\times\overline{\Omega}_{\varepsilon})\cup(\\{4R\\}\times\overline{\Omega}_{\varepsilon})\cup([R,4R],\partial\Omega_{\varepsilon}).$

By the properties of $\eta_{R}$, we have

$\eta_{R}(x)=|\nabla\eta_{R}(x)|=0\quad\mbox{ whenever
}\;x=(r,\omega)\in(\\{R\\}\times\overline{\Omega}_{\varepsilon})\cup(\\{4R\\}\times\overline{\Omega}_{\varepsilon}).$

Thus,

$\nabla(\phi_{\varepsilon}\eta_{R}^{\lambda})\cdot\nu=0\;\;\;\text{ on
}\;(\\{R\\}\times\overline{\Omega}_{\varepsilon})\cup(\\{4R\\}\times\overline{\Omega}_{\varepsilon}).$

Now, let $x=(r,\omega)\in([R,4R],\partial\Omega_{\varepsilon})$ so that
$\phi_{\varepsilon}(\omega)=0$ and then

$\nabla(\phi_{\varepsilon}\eta_{R}^{\lambda})=\phi_{\varepsilon}\nabla\eta_{R}^{\lambda}+\eta_{R}^{\lambda}\nabla\phi_{\varepsilon}=\frac{1}{\lvert
x\rvert}\eta_{R}^{\lambda}\nabla_{S^{N-1}}\phi_{\varepsilon}.$ (3.7)

Recall that $\nabla_{\SS}\phi_{\varepsilon}(\omega)$ lies in the tangent plane
to $\SS$ at $\omega=\frac{x}{|x|}\in\partial\Omega_{\varepsilon}$. Further,
$\nu(x)$ and $\nabla_{\SS}\phi_{\varepsilon}(\omega)$ are both located in the
tangent plane to $S^{N-1}$ at
$\omega=\frac{x}{|x|}\in\partial\Omega_{\varepsilon}$ and have opposite
direction. This comes from the fact that
$\partial\Omega_{\varepsilon}=\\{\phi_{\varepsilon}=0\\}$ is a level set of
$\phi_{\varepsilon}$ and
$\nabla\phi_{\varepsilon}(x)=\frac{1}{|x|}\nabla_{\SS}\phi_{\varepsilon}(\omega)$
points towards inside of $\Omega_{\varepsilon}$. Thus,

$\nu(x)=-\frac{\nabla_{\SS}\phi_{\varepsilon}(\omega)}{|\nabla_{\SS}\phi_{\varepsilon}(\omega)|}.$

Hence, by (3.7) we find

$\nabla\big{(}\phi_{\varepsilon}\eta^{\lambda}_{R}\big{)}(x)\cdot\nu(x)=-\frac{\eta_{R}^{\lambda}(x)}{|x|}|\nabla_{\SS}\phi_{\varepsilon}(\omega)|\leq
0.$

This proves our claim (3.6). Combining (3.4), (3.5) and (3.6) we deduce

$\int_{\mathcal{C}_{{\Omega}_{\varepsilon}}^{\rho}}(|x|^{-\alpha}*u^{p})u^{q-m}\phi_{\varepsilon}\eta_{R}^{\lambda}\leq\int_{\mathcal{C}_{{\Omega}_{\varepsilon}}^{R,4R}}u^{1-m}\left\\{\frac{|\Delta(\phi_{\varepsilon}\eta_{R}^{\lambda})|}{1-m}+|\mu|\frac{\phi_{\varepsilon}\eta_{R}^{\lambda}}{|x|^{2}}\right\\}.$
(3.8)

One has

$\Delta(\phi_{\varepsilon}\eta_{R}^{\lambda})=\eta_{R}^{\lambda}\Delta\phi_{\varepsilon}+2\nabla\phi_{\varepsilon}\nabla\eta_{R}^{\lambda}+\phi_{\varepsilon}\Delta\eta_{R}^{\lambda}.$
(3.9)

By direct computation, we find

$\begin{split}\lvert\eta_{R}^{\lambda}\Delta\phi_{\varepsilon}\rvert&=\frac{\lambda_{1}}{\lvert
x\rvert^{2}}\eta_{R}^{\lambda}\phi_{\varepsilon}\leq
CR^{-2}\eta_{R}^{\lambda}\phi_{\varepsilon},\\\\[3.61371pt]
\lvert\phi_{\varepsilon}\Delta\eta_{R}^{\lambda}\rvert&=\lambda\phi_{\varepsilon}\eta_{R}^{\lambda-2}\Big{|}(\lambda-1)\lvert\nabla\eta_{R}\rvert^{2}+\eta_{R}\Delta\eta_{R}\Big{|}\leq
CR^{-2}\eta_{R}^{\lambda-2}\phi_{\varepsilon},\\\\[3.61371pt]
\nabla\phi_{\varepsilon}\nabla\eta_{R}^{\lambda}&=\frac{\lambda}{R}\eta_{R}^{{}^{\prime}}\eta_{R}^{\lambda-1}\frac{1}{\lvert
x\rvert}\nabla_{S^{N-1}}\phi_{\varepsilon}\cdot\frac{x}{\lvert
x\rvert}.\end{split}$

Recall that $\nabla_{S^{N-1}}\phi_{\varepsilon}$ lies in the tangent plane to
the unit sphere $S^{N-1}$ at $\frac{x}{\lvert x\rvert}\in\Omega_{\varepsilon}$
and is also orthogonal to $x$. Thus,
$\nabla\phi_{\varepsilon}\nabla\eta_{R}^{\lambda}=0$ and (3.9) yields

$|\Delta(\phi_{\varepsilon}\eta_{R}^{\lambda})|\leq
CR^{-2}\eta_{R}^{\lambda-2}\phi_{\varepsilon}(\omega)\;\;\text{ in
}\;\mathcal{C}_{\Omega_{\varepsilon}}^{R,4R}.$ (3.10)

Now, from (3.8) and (3.10) we find

$\int_{\mathcal{C}_{\Omega_{\varepsilon}}^{\rho}}(|x|^{-\alpha}*u^{p})u^{q-m}\phi_{\varepsilon}\eta_{R}^{\lambda}\leq
CR^{-2}\int_{\mathcal{C}_{\Omega_{\varepsilon}}^{R,4R}}u^{1-m}\phi_{\varepsilon}\eta_{R}^{\lambda-2}.$
(3.11)

Furthermore, for $x,y\in\mathcal{C}_{\Omega_{\varepsilon}}^{\rho,4R}$, we have
$\lvert x-y\rvert\leq\lvert x\rvert+\lvert y\rvert\leq 8R$. This yields

$|x|^{-\alpha}\ast
u^{p}\geq\int_{\mathcal{C}_{\Omega_{\varepsilon}}^{\rho,4R}}\frac{u^{p}(y)}{|x-y|^{\alpha}}dy\geq
CR^{-\alpha}\int_{\mathcal{C}_{\Omega_{\varepsilon}}^{\rho,4R}}u^{p}(y)dy.$

Thus, (3.11) implies

$\left(\int_{\mathcal{C}_{\Omega_{\varepsilon}}^{\rho,4R}}u^{p}\right)\left(\int_{\mathcal{C}_{\Omega_{\varepsilon}}^{R,4R}}u^{q-m}\phi_{\varepsilon}\eta_{R}^{\lambda}\right)\leq
CR^{\alpha-2}\int_{\mathcal{C}_{\Omega_{\varepsilon}}^{R,4R}}u^{1-m}\phi_{\varepsilon}\eta_{R}^{\lambda-2}.$
(3.12)

Finally, by Holder’s inequality we conclude that

$\left(\int_{\mathcal{C}_{\Omega_{\varepsilon}}^{R,4R}}u^{(p+q-m)/2}\phi_{\varepsilon}^{1/2}(\omega)\eta_{R}^{\lambda/2}dx\right)^{2}\leq
CR^{\alpha-2}\int_{\mathcal{C}_{\Omega_{\varepsilon}}^{R,4R}}u^{1-m}\phi_{\varepsilon}(\omega)\eta_{R}^{\lambda-2}dx.$

Letting $\varepsilon\to 0$ in the above estimate we deduce (3.1). ∎

##  4 Proof of Theorem 1.1

Part (a) follows directly from Lemma 2.1. We next assume $\mu\leq
C_{H,\Omega}$ and prove part (b) in Theorem 1.1.

(i) Assume that (1.1) has a positive solution $u$ in
$\mathcal{C}_{\Omega}^{\rho}$ and let $\Omega_{0}\subset\subset\Omega$ be a
proper subdomain of $\Omega$. From Lemma 2.3 we derive $u\geq
c_{0}|x|^{\gamma_{*}}$ in $\mathcal{C}_{\Omega_{0}}^{2\rho}$ where $c_{0}>0$
is a constant. If $p\leq\frac{N-\alpha}{|\gamma_{*}|}$, then, by Lemma 2.2(ii)
it follows that $|x|^{-\alpha}\ast u^{p}=\infty$ in
$\mathcal{C}_{\Omega}^{2\rho}$ which contradicts condition (1.2).

(ii) We divide our argument into two steps.

Step 1:  If $p+q\leq 1$ then (1.1) has no positive solutions.

Assume by contradiction that (1.1) has a positive solution $u$.

If $p+q=1$, then, by taking $m=q\in(0,1)$ in (3.12) we find

$\displaystyle\int_{\mathcal{C}^{R,4R}_{\Omega}}u^{p}$
$\displaystyle\geq\int_{\mathcal{C}^{R,4R}_{\Omega}}u^{p}\phi(\omega)\eta^{\lambda-2}_{R}$
$\displaystyle\geq
CR^{2-\alpha}\Big{(}\int_{\mathcal{C}^{\rho,4R}_{\Omega}}u^{p}\Big{)}\Big{(}\int_{\mathcal{C}^{\rho,4R}_{\Omega}}\phi(\omega)\eta_{R}^{\lambda}\Big{)},$

where $\eta_{R}$ is the test function defined in the statement of Proposition
3.1. The above estimate implies

$C\geq
R^{2-\alpha}\int_{\mathcal{C}^{\rho,4R}_{\Omega}}\phi(\omega)\eta_{R}^{\lambda}\geq
CR^{2-\alpha+N}\,,$

which yields a contradiction as $R\to\infty$.

Assume now $p+q<1$. By letting $m=p+q\in(0,1)$ in (3.1) we find

$\displaystyle\int_{\mathcal{C}^{R,4R}_{\Omega}}u^{1-m}$
$\displaystyle\geq\int_{\mathcal{C}^{R,4R}_{\Omega}}u^{1-m}\phi(\omega)\eta_{R}^{\lambda-2}$
(4.1) $\displaystyle\geq
CR^{2-\alpha}\left(\int_{\mathcal{C}^{R,4R}_{\Omega}}u^{(p+q-m)/2}\phi(\omega)^{1/2}\eta^{\lambda/2}_{R}\right)^{2}$
$\displaystyle=CR^{2-\alpha}\left(\int_{\mathcal{C}^{R,4R}_{\Omega}}\phi(\omega)^{1/2}\eta^{\lambda/2}_{R}\right)^{2}$
$\displaystyle\geq CR^{2N-\alpha+2}.$

Let $\Omega_{0}\subset\subset\Omega$ be a subdomain and
$\rho_{0}>\max\\{1,\rho\\}$. For $x\in\mathcal{C}_{\Omega_{0}}$,
$|x|=R>\rho_{0}$, we can find a real number $\delta\in(0,1)$ depending only on
$\Omega$ and $\Omega_{0}$ such that
$B_{R\delta}(x)\subset\mathcal{C}_{\Omega}$. Replacing eventually $\Omega_{0}$
with a smaller subdomain, we may assume that diam$(\Omega_{0})<\delta/6$.
Thus, for any $x\in\mathcal{C}_{\Omega_{0}}$, $|x|=R$ one has (see Figure 2
below):

$D:=\mathcal{C}_{\Omega_{0}}^{R-R\delta/6,R+R\delta/6}\subset
B_{R\delta/3}(x)\subset\mathcal{C}_{\Omega}^{\rho}.$ (4.2)

Figure 2: The domain $D$ and the ball $B_{R\delta/3}(x)$ in (4.2) on which one
applies the Harnack inequality.

With a similar argument as in the Proposition 3.1 in which we replace the pair
$(R,4R)$ by $(R-R\delta/6,R+R\delta/6)$ and $\Omega$ by $\Omega_{0}$, we
derive as in (4.1) that

$\int_{D}u^{1-m}\geq CR^{2N-\alpha+2}.$ (4.3)

By the weak Harnack inequality (see [27, Theorem 1.2]) and (4.3) we find

$\displaystyle u(x)$ $\displaystyle\geq\inf_{B_{R\delta/3}(x)}u$
$\displaystyle\geq
C\left(\frac{1}{(R\delta/3)^{N}}\int_{B_{R\delta/3}(x)}u^{1-m}\right)^{1/(1-m)}$
$\displaystyle\geq
C\left(\frac{1}{(R\delta/3)^{N}}\int_{D}u^{1-m}\right)^{1/(1-m)}$
$\displaystyle\geq CR^{(N-\alpha+2)/(1-m)}.$

Hence,

$u(x)\geq C|x|^{\zeta}\quad\mbox{ in }\;\mathcal{C}_{\Omega_{0}}^{\rho_{0}},$

where $\zeta=\frac{N-\alpha+2}{1-m}>0$. This contradicts condition (1.2) since
for $x\in\mathcal{C}_{\Omega}^{\rho_{0}}$ fixed we have

$\displaystyle|x|^{-\alpha}\ast u^{p}$
$\displaystyle\geq\int_{y\in\mathcal{C}_{\Omega}^{\rho},\,|y|>|x|}\frac{u^{p}(y)}{|x-y|^{\alpha}}dy\geq
C\int_{\mathcal{C}_{\Omega}^{\rho},\,|y|>|x|}\frac{|y|^{p\zeta}}{(2|y|)^{\alpha}}dy=\infty,$

by using the fact that $\zeta>0$ and $0\leq\alpha<N$.

Step 2:  If $p+q>1$ and (1.1) has a positive solution, then $p+q\geq
1+\frac{N-\alpha+2}{|\gamma_{*}|}$.

Let $m\in(0,1)$ be close to 1 such that $p+q>2-m$. Let

$\kappa=\frac{p+q-m}{2-2m}>1\quad\mbox{ and
}\quad\kappa^{\prime}=\frac{\kappa}{\kappa-1}>1.$ (4.4)

With the same notations as in Proposition 3.1 and using Hölder’s inequality we
find

$\int_{\mathcal{C}^{\rho}_{\Omega}}u^{1-m}\phi(\omega)\eta_{R}^{\lambda-2}\leq\left(\int_{\mathcal{C}^{\rho}_{\Omega}}u^{(p+q-m)/2}\phi(\omega)^{1/2}\eta_{R}^{\lambda/2}\right)^{1/\kappa}\left(\int_{\mathcal{C}^{\rho}_{\Omega}}\phi(\omega)^{\gamma}\eta_{R}^{\sigma}\right)^{1/\kappa^{\prime}},$
(4.5)

where

$\gamma=\frac{2\kappa-1}{2(\kappa-1)}>1\quad\mbox{ and
}\quad\sigma=\frac{2(\lambda-2)\kappa-\lambda}{2(\kappa-1)}>0.$

Combining (4.5) and (3.1) we deduce

$\displaystyle\left(\int_{\mathcal{C}^{\rho}_{\Omega}}u^{(p+q-m)/2}\phi(\omega)^{1/2}\eta_{R}^{\lambda/2}\right)^{2-1/\kappa}$
$\displaystyle\leq
CR^{\alpha-2}\left(\int_{\mathcal{C}^{\rho}_{\Omega}}\phi(\omega)^{\gamma}\eta_{R}^{\sigma}\right)^{1/\kappa^{\prime}}$
(4.6) $\displaystyle\leq CR^{\alpha-2+N/\kappa^{\prime}}.$

Take a subdomain $\Omega_{0}\subset\subset\Omega$. Then, for some constant
$c>0$ we have

$\phi>c>0\quad\mbox{ in }\;\Omega_{0},$ (4.7)

and the estimate (4.6) yields

$\left(\int_{\mathcal{C}^{\rho}_{\Omega_{0}}}u^{(p+q-m)/2}\eta_{R}^{\lambda/2}\right)^{2-1/\kappa}\leq
CR^{\alpha-2+N/\kappa^{\prime}}\quad\mbox{ for }R>2\rho\mbox{ large}.$ (4.8)

We next use the estimate (2.6) in Lemma 2.3 together with (4.8) to derive

$\displaystyle CR^{\alpha-2+N/\kappa^{\prime}}$
$\displaystyle\geq\left(\int_{\mathcal{C}^{\rho}_{\Omega_{0}}}u^{(p+q-m)/2}\eta_{R}^{\lambda/2}\right)^{2-1/\kappa}$
(4.9)
$\displaystyle\geq\left(c\int_{\mathcal{C}^{\rho}_{\Omega_{0}}}|x|^{(p+q-m)\gamma_{*}/2}\eta_{R}^{\lambda/2}\right)^{2-1/\kappa}$
$\displaystyle\geq
c\left(R^{(p+q-m)\gamma_{*}/2}\int_{\mathcal{C}^{R,4R}_{\Omega_{0}}}\eta_{R}^{\lambda/2}\right)^{2-1/\kappa}$
$\displaystyle\geq
cR^{\big{(}N+\frac{p+q-m}{2}\gamma_{*}\big{)}(2-1/\kappa)}\quad\mbox{ for
}R>2\rho\mbox{ large}.$

This implies

$\Big{(}N+\frac{p+q-m}{2}\gamma_{*}\Big{)}\Big{(}2-\frac{1}{\kappa}\Big{)}\leq\alpha-2+\frac{N}{\kappa^{\prime}},$

where $\kappa$, $\kappa^{\prime}$ are given in (4.4). This last estimates
finally gives

$p+q\geq 1+\frac{N-\alpha+2}{|\gamma_{*}|}.$

(iii) Assume $p+q=1+\frac{N-\alpha+2}{|\gamma_{*}|}$ and $q>1$. From the
estimate (2.6) of Lemma 2.3 and Lemma 2.2(ii) we deduce

$|x|^{-\alpha}\ast u^{p}\geq C|x|^{N-\alpha+p\gamma_{*}}\quad\mbox{ in
}\;\mathcal{C}_{\Omega}^{2\rho}.$

Thus, $u$ satisfies

$\mathscr{L}_{H}u\geq|x|^{N-\alpha+p\gamma_{*}}u^{q}\quad\mbox{ in
}\;\mathcal{C}_{\Omega}^{2\rho}.$

By Lemma 2.4 with $\beta=N-\alpha+p\gamma_{*}$ the above inequality has no
positive solutions if $p+q=1+\frac{N-\alpha+2}{|\gamma_{*}|}$.

∎

##  5 Proof of Theorem 1.2

We shall discuss the construction of a positive solution to (1.1) separately
for the cases $\mu<C_{H,\Omega}$ and $\mu=C_{H,\Omega}$.

Case 1: $\mu<C_{H,\Omega}$. Then $\gamma_{*}<-\frac{N-2}{2}<0$ and we can find
$\gamma_{*}<\gamma<-\frac{N-2}{2}$ close to $\gamma_{*}$ such that
$p\lvert\gamma\rvert\neq N$ and

$p>\frac{N-\alpha}{\lvert\gamma\rvert}\,,\;\;\;q>1+\frac{2-\alpha}{\lvert\gamma\rvert}\;\;\;\text{
and }\;\;\;p+q>1+\frac{N-\alpha+2}{\lvert\gamma\rvert}.$ (5.1)

Since $\mu<C_{H,\Omega}$ and $\gamma_{*}<\gamma<-\frac{N-2}{2}$, we have

$\lambda_{1}-\mu>\gamma(\gamma+N-2).$ (5.2)

We construct a positive solution $U$ in the form $U=Cu$, where $C>0$ is a
constant and

$u(x)=\phi(\omega)\lvert x\rvert^{\gamma}.$ (5.3)

We first check that $u$ given by (5.3) satisfies (1.2). Indeed, since
$p>\frac{N-\alpha}{\lvert\gamma\rvert}$ and $0\leq\alpha<N$, we have

$\displaystyle|x|^{-\alpha}*u^{p}$
$\displaystyle=\int_{\mathcal{C}_{\Omega}^{\rho,2|x|}}\frac{u^{p}(y)}{|x-y|^{\alpha}}dy+\int_{\mathcal{C}_{\Omega}^{2|x|}}\frac{u^{p}(y)}{|x-y|^{\alpha}}dy$
(5.4)
$\displaystyle\leq\rho^{p\gamma}\\!\int_{|x-y|<3|x|}\frac{dy}{|x-y|^{\alpha}}+2^{\alpha}\int_{|y|>2|x|}|y|^{-\alpha+p\gamma}dy$
$\displaystyle\leq
C\int_{0}^{3|x|}\\!\\!t^{N-\alpha-1}dt+C\int_{2|x|}^{\infty}t^{N-\alpha+p\gamma-1}dt$
$\displaystyle<\infty.$

Thus, $u$ satisfies the second condition of (1.2). By (5.2) we find

$\mathscr{L}_{H}u=\phi(\omega)\lvert
x\rvert^{\gamma-2}\left(\lambda_{1}-\mu-\gamma(\gamma+N-2)\right)\geq
c\phi(\omega)\lvert x\rvert^{\gamma-2}\quad\text{ in
}\mathcal{C}_{\Omega}^{\rho},$ (5.5)

for some $c>0$. Applying Lemma 2.2(iii) for $f(x)=u^{p}(x)\leq\lvert
x\rvert^{p\gamma}$ in $\mathcal{C}_{\Omega}^{\rho/2}$, we deduce

$\begin{aligned} (K_{\alpha,\beta}*u^{p})u^{q}\leq
C\phi(\omega)^{q}\begin{cases}\lvert x\rvert^{N-\alpha+(p+q)\gamma}&\text{if
}N-\alpha<p\lvert\gamma\rvert<N\\\\[3.61371pt] \lvert
x\rvert^{N-\alpha+q\gamma}&\text{if
}p\lvert\gamma\rvert>N\end{cases}\end{aligned}\quad\mbox{in
}\;\mathcal{C}_{\Omega}^{\rho}.$ (5.6)

Note that by (5.1) we have

$\gamma-2>N-\alpha+(p+q)\gamma\quad\mbox{and}\quad\gamma-2>-\alpha+q\gamma.$

Using the above estimates and $q>1$, from (5.5)-(5.6) we deduce

${\mathscr{L}}_{H}u\geq C_{1}(|x|^{-\alpha}*u^{p})u^{q}\quad\mbox{in
}\;\;\mathcal{C}_{\Omega}^{\rho},$ (5.7)

where $C_{1}>0$ is a constant. Let now $C=C_{1}^{1/(p+q-1)}$. Then, from (5.7)
we deduce that $U=Cu$ is a positive solution of (1.1).

Case 2: $\mu=C_{H,\Omega}$. Then, the quadratic equation (1.8) has two equal
roots given by $\gamma_{*}=\gamma^{*}=\frac{2-N}{2}<0$. We construct a
solution $u$ in the form

$u(x)=\phi(\omega)\lvert x\rvert^{\gamma_{*}}\log^{\tau}(\sigma\lvert
x\rvert)\quad\mbox{ where }\sigma>\frac{4}{\rho}\quad\mbox{ and
}\;\;0<\tau<1.$ (5.8)

By direct computation we have

$\displaystyle\mathscr{L}_{H}u$ $\displaystyle=\tau(1-\tau)\phi(\omega)\lvert
x\rvert^{\gamma_{*}-2}\log^{\tau-2}(\sigma\lvert x\rvert)\;\;\;\text{ in
}\mathcal{C}_{\Omega}^{\rho}.$ (5.9)

Also, by Lemma 2.2(iii) for

$f=u^{p}\leq C|x|^{p\gamma_{*}}\log^{p\tau}(1+\lvert x\rvert)\quad\mbox{in
}\;\mathcal{C}_{\Omega}^{\rho/2},$

we find the existence of a constant $C>0$ such that for all
$x\in\mathcal{C}_{\Omega}^{\rho}$ one has

$(|x|^{-\alpha}*u^{p})u^{q}\leq C\phi^{q}(\omega)\begin{cases}\lvert
x\rvert^{N-\alpha+(p+q)\gamma_{*}}\log^{(p+q)\tau}(1+\lvert x\rvert)&\text{if
}\;N-\alpha<p|\gamma_{*}|<N,\\\ \lvert
x\rvert^{-\alpha+q\gamma_{*}}\log^{1+(p+q)\tau}(1+\lvert x\rvert)&\text{if
}\;p|\gamma_{*}|=N,\\\ \lvert
x\rvert^{-\alpha+q\gamma_{*}}\log^{q\tau}(1+\lvert x\rvert)&\text{if
}\;p|\gamma_{*}|>N.\end{cases}$ (5.10)

Notice that

$\gamma_{*}-2>\max\\{-\alpha+q\gamma_{*},N-\alpha+(p+q)\gamma_{*}\\}.$

From (5.9), (5.10) and the above inequality we deduce that $u$ satisfies
(5.7). Similar to the Case 1 above, $U=Cu$ is a solution of (1.1) for a
specific constant $C>0$. ∎

##  6 Proof of Theorem 1.4

Part (a) follows directly from Lemma 2.1.

(b)(i) Assume (1.12) has a positive solution $(u,v)$. Take a proper subdomain
$\Omega_{0}\subset\subset\Omega$. From Lemma 2.3 we derive $v\geq
c_{0}|x|^{\gamma_{*}}$ in $\mathcal{C}_{\Omega_{0}}^{2\rho}$ where $c_{0}>0$
is a constant. If $p\leq\frac{N-\alpha}{|\gamma_{*}|}$, then, by Lemma 2.2(ii)
it follows that $|x|^{-\alpha}\ast v^{p}=\infty$ in
$\mathcal{C}_{\Omega}^{2\rho}$ which contradicts the definition of the
solution to (1.12).

It remains to raise a contradiction if $p,q>0$ satisfy $p+q\geq 2$ and

$p+q<\frac{1}{s}\Big{(}1+\frac{2}{|\gamma_{*}|}\Big{)}+\frac{N-\alpha+2}{|\gamma_{*}|}.$
(6.1)

With the same notations as in Proposition 3.1 let us take
$\varphi=\phi_{\varepsilon}\eta_{R}^{\lambda}$ as a test function in (1.12);
this corresponds to $m=0$ in (3.1). We use the same approach as in Proposition
3.1 to find

$\left(\int_{\mathcal{C}^{\rho}_{\Omega}}v^{(p+q)/2}\phi(\omega)^{1/2}\eta^{\lambda/2}_{R}\right)^{2}\leq
CR^{\alpha-2}\int_{\mathcal{C}^{\rho}_{\Omega}}u\phi(\omega)\eta_{R}^{\lambda-2}$
(6.2)

and

$\int_{\mathcal{C}^{\rho}_{\Omega}}u^{s}\phi(\omega)\eta^{\lambda}_{R}\leq
CR^{-2}\int_{\mathcal{C}^{\rho}_{\Omega}}v\phi(\omega)\eta_{R}^{\lambda-2}.$
(6.3)

We next claim that

$\int_{\mathcal{C}^{\rho}_{\Omega}}v\phi(\omega)\eta_{R}^{\lambda-2}\leq
CR^{N\big{(}1-2/(p+q)\big{)}+(\alpha-2)/(p+q)}\left(\int_{\mathcal{C}^{\rho}_{\Omega}}u\phi(\omega)\eta_{R}^{\lambda-2}\right)^{1/(p+q)}.$
(6.4)

If $p+q=2$, the above estimate follows directly from (6.2). Indeed, using the
fact that $0<\phi,\eta_{R}\leq 1$ in $\mathcal{C}_{\Omega}^{\rho}$ and
$\lambda-2>\lambda/2$, from (6.2) we have

$\int_{\mathcal{C}^{\rho}_{\Omega}}v\phi(\omega)\eta_{R}^{\lambda-2}\leq\int_{\mathcal{C}^{\rho}_{\Omega}}v^{(p+q)/2}\phi(\omega)^{1/2}\eta^{\lambda/2}_{R}\leq
CR^{(\alpha-2)/2}\left(\int_{\mathcal{C}^{\rho}_{\Omega}}u\phi(\omega)\eta_{R}^{\lambda-2}\right)^{1/2},$

which yields (6.4) in the case $p+q=2$.

If $p+q>2$, then by Hölder’s inequality and (6.2) we have

$\displaystyle\int_{\mathcal{C}^{\rho}_{\Omega}}v\phi(\omega)\eta_{R}^{\lambda-2}$
$\displaystyle\leq\left(\int_{\mathcal{C}^{\rho}_{\Omega}}v^{(p+q)/2}\phi(\omega)^{1/2}\eta^{\lambda/2}_{R}\right)^{2/(p+q)}\left(\int_{\mathcal{C}^{\rho}_{\Omega}}\phi(\omega)^{\tau_{1}}\eta^{\tau_{2}}_{R}\right)^{1-2/(p+q)}$
$\displaystyle\leq
CR^{N\big{(}1-2/(p+q)\big{)}}\left(\int_{\mathcal{C}^{\rho}_{\Omega}}v^{(p+q)/2}\phi(\omega)^{1/2}\eta^{\lambda/2}_{R}\right)^{2/(p+q)}$
$\displaystyle\leq
CR^{N\big{(}1-2/(p+q)\big{)}}\left(R^{\alpha-2}\int_{\mathcal{C}^{\rho}_{\Omega}}u\phi(\omega)\eta_{R}^{\lambda-2}\right)^{1/(p+q)}$
$\displaystyle=CR^{N\big{(}1-2/(p+q)\big{)}+(\alpha-2)/(p+q)}\left(\int_{\mathcal{C}^{\rho}_{\Omega}}u\phi(\omega)\eta_{R}^{\lambda-2}\right)^{1/(p+q)},$

where

$\tau_{1}=\frac{p+q-1}{p+q-2}>1\quad\mbox{ and
}\quad\tau_{2}=\frac{(p+q)(\lambda-2)-\lambda}{p+q-2}>1.$

This concludes the proof of our claim (6.4). By (6.3) and Hölder’s inequality
we have

$\displaystyle\int_{\mathcal{C}^{\rho}_{\Omega}}u\phi(\omega)\eta_{R}^{\lambda-2}$
$\displaystyle\leq\left(\int_{\mathcal{C}^{\rho}_{\Omega}}u^{s}\phi(\omega)\eta^{\lambda}_{R}\right)^{1/s}\left(\int_{\mathcal{C}^{\rho}_{\Omega}}\phi(\omega)\eta^{\lambda-2s/(s-1)}_{R}\right)^{1-1/s}$
(6.5) $\displaystyle\leq
CR^{N(1-1/s)}\left(\int_{\mathcal{C}^{\rho}_{\Omega}}u^{s}\phi(\omega)\eta^{\lambda}_{R}\right)^{1/s}$
$\displaystyle\leq
CR^{N(1-1/s)-2/s}\left(\int_{\mathcal{C}^{\rho}_{\Omega}}v\phi(\omega)\eta^{\lambda-2}_{R}\right)^{1/s}.$

Using (6.4) in (6.5) we find

$\left(\int_{\mathcal{C}^{R,4R}_{\Omega}}v\phi(\omega)\eta^{\lambda-2}_{R}\right)^{1-1/(s(p+q))}\leq
CR^{\tau}\quad\mbox{ for $R>\rho$ large,}$ (6.6)

where

$\tau=\frac{1}{p+q}\Big{(}N\big{(}1-\frac{1}{s}\big{)}-\frac{2}{s}+N(p+q-2)+\alpha-2\Big{)}.$

Let $\Omega_{0}\subset\subset\Omega$ be a smooth subdomain. Then, by Lemma
2.3, there exists $c_{0}>0$ such that

$v(x)\geq c_{0}|x|^{\gamma_{*}}\quad\mbox{ in
}\;\mathcal{C}_{\Omega_{0}}^{2\rho}.$ (6.7)

Using (6.6) and (6.7) we find

$CR^{\tau}\geq\left(c\int_{\mathcal{C}^{2R,3R}_{\Omega_{0}}}|x|^{\gamma_{*}}\right)^{1-1/(s(p+q))}\geq
CR^{(N+\gamma_{*})\big{(}1-1/(s(p+q))\big{)}}\quad\mbox{ for $R>2\rho$
large.}$

This yields

$\frac{N+\gamma_{*}}{s}\Big{(}s(p+q)-1\Big{)}\leq
N\Big{(}1-\frac{1}{s}\Big{)}-\frac{2}{s}+N(p+q-2)+\alpha-2,$

which implies

$p+q\geq\frac{1}{s}\Big{(}1+\frac{2}{|\gamma_{*}|}\Big{)}+\frac{N-\alpha+2}{|\gamma_{*}|}.$

and this contradicts condition (6.1).

(ii) Assume $s>1+2/|\gamma_{*}|$, $q>1+(2-\alpha)/|\gamma_{*}|$ and that

$p|\gamma_{*}|>N-\alpha\quad\mbox{ and }\quad
p+q>\frac{1}{s}\Big{(}1+\frac{2}{|\gamma_{*}|}\Big{)}+\frac{N-\alpha+2}{|\gamma_{*}|}.$
(6.8)

Case 1: $\mu<C_{H,\Omega}$. Note that the second inequality of (6.8) is
equivalent to

$\Big{(}p+q-\frac{1}{s}\Big{)}|\gamma_{*}|>N-\alpha+2+\frac{2}{s}.$

Thus, we may find $\gamma_{*}<b<-\frac{N-2}{2}$ such that

$s>1+\frac{2}{|b|}\,,\qquad q>1+\frac{2-\alpha}{|b|}\,,\qquad p|b|>N-\alpha$
(6.9)

and

$\Big{(}p+q-\frac{1}{s}\Big{)}|b|>N-\alpha+2+\frac{2}{s}.$ (6.10)

By moving $b$ closer to $\gamma_{*}$, we may also assume $p|b|\neq N$. As in
the proof of Theorem 1.2, this helps us to avoid the logarithmic terms in
Lemma 2.2(iii). From (6.10) we deduce

$\frac{b-2}{s}>N-\alpha+2+(p+q)b.$ (6.11)

On the other hand, from (6.9)1 we have $s>1+2/|b|$ so

$\frac{b-2}{s}>b.$ (6.12)

Thus, from (6.11) and (6.12) we deduce

$\frac{b-2}{s}>\max\Big{\\{}b,N-\alpha+2+(p+q)b\Big{\\}}.$

We thus may find $a\in(\gamma_{*},b)$ such that

$0>\frac{b-2}{s}>a>\max\Big{\\{}b,N-\alpha+2+(p+q)b\Big{\\}}.$ (6.13)

Since $\mu<C_{H,\Omega}$ and $\gamma_{*}<a<b<-\frac{N-2}{2}$ we have

$\lambda_{1}-\mu>\max\Big{\\{}a(a+N-2),b(b+N-2)\Big{\\}}.$ (6.14)

Let now $u(x)=\phi(\omega)|x|^{a}$ and $v(x)=\phi(\omega)|x|^{b}$. Using
(6.9)3 we have $p|b|>N-\alpha$ and then with the same estimates as in (5.4) we
deduce $|x|^{-\alpha}\ast v^{p}<\infty$ for all
$x\in\mathcal{C}_{\Omega}^{\rho}$. Also, since by (6.13) one has $(b-2)/s>a$,
we obtain

$\displaystyle\mathscr{L}_{H}v$
$\displaystyle=\Big{(}\lambda_{1}-\mu-b(b+N-2)\Big{)}\phi(\omega)|x|^{b-2}$
(6.15) $\displaystyle\geq C\phi(\omega)^{s}|x|^{as}$
$\displaystyle=Cu^{s}\quad\mbox{ in }\;\mathcal{C}_{\Omega}^{\rho}.$

It remains to check that $u$ satisfies the first inequality of the system
(1.12). To this aim, let us first estimate the nonlocal term
$(|x|^{-\alpha}\ast v^{p}\big{)}v^{q}$ in $\mathcal{C}_{\Omega}^{\rho}$. We
apply Lemma 2.2(iii) for $f=v^{p}\leq|x|^{pb}$ in
$\mathcal{C}^{\rho/2}_{\Omega}$ and $\beta=p|b|$. We obtain

$|x|^{-\alpha}\ast v^{p}\leq C\begin{cases}|x|^{N-\alpha+pb}&\quad\mbox{ if
}p|b|<N\\\ |x|^{-\alpha}&\quad\mbox{ if }p|b|>N\end{cases}\quad\mbox{ in
}\;\mathcal{C}_{\Omega}^{\rho},$

and then

$\big{(}|x|^{-\alpha}\ast v^{p}\big{)}v^{q}\leq
C\begin{cases}\phi(\omega)^{q}|x|^{N-\alpha+(p+q)b}&\quad\mbox{ if }p|b|<N\\\
\phi(\omega)^{q}|x|^{-\alpha+qb}&\quad\mbox{ if }p|b|>N\end{cases}\quad\mbox{
in }\;\mathcal{C}_{\Omega}^{\rho}.$ (6.16)

Case 1a: $p|b|<N$. Let us observe that by (6.13) we have
$a-2>N-\alpha+(p+q)b$. Using this fact together with (6.14) and (6.16) we find

$\displaystyle\mathscr{L}_{H}u$
$\displaystyle=\Big{(}\lambda_{1}-\mu-a(a+N-2)\Big{)}\phi(\omega)|x|^{a-2}$
(6.17) $\displaystyle\geq C\phi(\omega)^{q}|x|^{N-\alpha+(p+q)b}$
$\displaystyle\geq C\big{(}|x|^{-\alpha}\ast v^{p}\big{)}v^{q}\quad\mbox{ in
}\;\mathcal{C}_{\Omega}^{\rho}.$

Case 1b: $p|b|>N$. From (6.13) we have $0>a>b$ and from (6.9)2 we know that
$q>1+(2-\alpha)/|b|$. Thus, $a-2>b-2>-\alpha+qb$. As in Case 1a above, by
(6.14) and (6.16) we deduce

$\mathscr{L}_{H}u\geq C\phi(\omega)|x|^{a-2}\geq
C\phi(\omega)^{q}|x|^{-\alpha+qb}\geq C\big{(}|x|^{-\alpha}\ast
v^{p}\big{)}v^{q}\quad\mbox{ in }\;\mathcal{C}_{\Omega}^{\rho}.$ (6.18)

From (6.15) and (6.17)-(6.18) there exists a constant $C>0$ such that

$\begin{cases}\displaystyle\mathscr{L}_{H}u\geq C\big{(}|x|^{-\alpha}\ast
v^{p}\big{)}v^{q}\\\\[3.61371pt] \displaystyle\mathscr{L}_{H}v\geq
Cu^{s}\end{cases}\quad\mbox{ in }\;\mathcal{C}_{\Omega}^{\rho}.$ (6.19)

Let now $C_{1},C_{2}>0$ be given by

$C_{1}^{s-1/(p+q)}=C^{1+1/(p+q)}\quad\mbox{ and }\quad
C_{2}^{p+q-1/s}=C^{1+1/s}.$

Then $(U,V)=(C_{1}u,C_{2}v)$ is a positive solution of the system (1.12).

Case 2: $\mu=C_{H,\Omega}$. Let $u=v$ be given by (5.8). We have seen in the
proof of Theorem 1.2 that $(u,v)$ satisfies $\mathscr{L}_{H}u\geq
C\big{(}|x|^{-\alpha}\ast v^{p}\big{)}v^{q}$ in $\mathcal{C}_{\Omega}^{\rho}$.

Using $s>1+\frac{2}{|\gamma_{*}|}$ one has $\gamma_{*}-2>s\gamma_{*}$ and a
direct calculation yields

$\mathscr{L}_{H}v\geq C\phi(\omega)\lvert
x\rvert^{\gamma_{*}-2}\log^{\tau-2}(\sigma\lvert x\rvert)\geq
C\phi(\omega)^{s}\lvert x\rvert^{s\gamma_{*}}\log^{s\tau}(\sigma\lvert
x\rvert)=Cu^{s}\;\;\;\text{ in }\mathcal{C}_{\Omega}^{\rho}.$

Thus, $(u,v)$ satisfies (6.19) and then as in Case 1 above we deduce that
$(U,V)=(C_{1}u,C_{2}v)$ is a positive solution of (1.12). This concludes our
proof. ∎

## References

  * [1] C. Bandle, Positive solutions of Emden equations in cone-like domains, Progress in Nonlinear Differential Equations Appl., 7 (1992), 71–75.
  * [2] C. Bandle and M. Essen, On positive solutions of Emden equations in cone-like domains, Arch. Rational Mech. Anal., 112 (1990), 319–338.
  * [3] M.F. Bidaut-Véron and S. Pohozaev, Nonexistence results and estimates for some nonlinear elliptic problems, J. Analyse Math. 84 (2001), 1–49.
  * [4] H. Chen and F. Zou, Classification of isolated singularities of positive solutions for Choquard equations, J. Differential Equations 261 (2016), 6668–6698.
  * [5] H. Chen and F. Zou, Isolated singularities of positive solutions for Choquard equations in sublinear case, Commun. Contemp. Math. 20 (2018), Article No. 1750040.
  * [6] H. Chen and F. Zou, Qualitative properties of singular solutions of Choquard equations, Adv. Differential Equations 28 (2023), 35–72.
  * [7] J. T. Devreese and A. S. Alexandrov, Advances in Polaron Physics,  Springer Series in Solid-State Sciences, Springer, 159 (2010).
  * [8] M. Ghergu, Partial Differential Inequalities with Nonlinear Convolution Terms, SpringerBriefs in Mathematics, 2022.
  * [9] M. Ghergu, P. Karageorgis and G. Singh, Positive solutions for quasilinear elliptic inequalities and systems with nonlocal terms, J. Differential Equations 268 (2020), 6033–6066.
  * [10] M. Ghergu, Y. Miyamoto and V. Moroz, Polyharmonic inequalities with nonlocal terms. J. Differential Equations 296 (2021), 799–821.
  * [11] M. Ghergu and S.D. Taliaferro, Pointwise bounds and blow-up for Choquard-Pekar inequalities at an isolated singularity, J. Differential Equations 261 (2016), 189–217.
  * [12] M. Ghergu and S.D. Taliaferro, Isolated Singularities in Partial Differential Inequalities. Encyclopedia of Mathematics and its Applications, 161. Cambridge University Press, Cambridge, 2016.
  * [13] D.R. Hartree, The wave mechanics of an atom with a non-Coulomb central field, Part I. Theory and Methods, Math. Proc. Cambridge Philosophical Soc., 24 (1928), 89–110.
  * [14] D.R. Hartree, The wave mechanics of an atom with a non-Coulomb central field, Part II. Some Results and Discussion, Math. Proc. Cambridge Philosophical Soc., 24 (1928), 111–132.
  * [15] D.R. Hartree, The wave mechanics of an atom with a non-Coulomb central field, Part III. Term Values and Intensities in Series in Optical Spectra, Math. Proc. Cambridge Philosophical Soc., 24 (1928), 426–437.
  * [16] V. Kondratiev, V. Liskevich and V. Moroz, Positive solutions to superlinear second-order divergence type elliptic equations in cone-like domains, Ann. Inst. H. Poincaré C Anal. Non Linéaire 22 (2005), 25–43.
  * [17] V. Kondratiev, V. Liskevich, V. Moroz and Z. Sobol, A critical phenomenon for sublinear elliptic equations in cone-like domains, Bull. London Math. Soc. 37 (2005), 585–591.
  * [18] G.G. Laptev, On the nonexistence of solutions of elliptic differential inequalities in conic domains, (in Russian) Mat. Zametki 71 (2002), 855–866; English translation in Math. Notes 71 (2002), 782–793.
  * [19] G.G. Laptev, Nonexistence of solutions for parabolic inequalities in unbounded cone-like domains via the test function method, J. Evol. Equations 2 (2002), 459–470.
  * [20] E. H. Lieb, Existence and uniqueness of the minimizing solution of Choquard’s nonlinear equation, Studies Appl. Math. 57 (1976/77), 93–105.
  * [21] P.-L. Lions, The Choquard equation and related questions, Nonlinear Anal. (1980), 1063–1072.
  * [22] P.-L. Lions, The concentration-compactness principle in the calculus of variations. The locally compact case. II, Ann. Inst. H. Poincaré Anal. Non Linéaire (1984), 223–283.
  * [23] V. Liskevich, S. Lyakhova and V. Moroz, Positive solutions to singular semilinear elliptic equations with critical potential on cone-like domains, Adv. Differential Equations 4 (2006), 361–398.
  * [24] E. Mitidieri and S.I. Pohozaev, A priori estimates and blow up of solutions to nonlinear partial differential equations, Proc. Steklov Inst. Math. 234 (2001), 1-367.
  * [25] V. Moroz and J. Van Schaftingen, Nonexistence and optimal decay of supersolutions to Choquard equations in exterior domains, J. Differential Equations, 254 (2013), 3089–3145.
  * [26] S. Pekar, Untersuchung über die Elektronentheorie der Kristalle, Akademie Verlag, Berlin, 1954.
  * [27] N.S. Trudinger, On Harnack type inequalities and their application to quasilinear equations, Comm. Pure Appl. Math. 20 (1967), 721–747.


# Simple Local Attentions Remain Competitive for Long-Context Tasks

Wenhan Xiong, Barlas Oğuz, Anchit Gupta, Xilun Chen, Diana Liskovich,
Omer Levy, Wen-tau Yih, Yashar Mehdad

Meta AI
<EMAIL_ADDRESS>
###### Abstract

Many NLP tasks require processing long contexts beyond the length limit of
pretrained models. In order to scale these models to longer text sequences,
many efficient long-range attention variants have been proposed. Despite the
abundance of research along this direction, it is still difficult to gauge the
relative effectiveness of these models in practical use cases, _e.g._ , if we
apply these models following the pretrain-and-finetune paradigm. In this work,
we aim to conduct a thorough analysis of these emerging models with large-
scale and controlled experiments. For each attention variant, we pretrain
large-size models using the same long-doc corpus and then finetune these
models for real-world long-context tasks. Our findings reveal pitfalls of an
existing widely-used long-range benchmark and show none of the tested
efficient attentions can beat a simple local window attention under standard
pretraining paradigms. Further analysis on local attention variants suggests
that even the commonly used attention-window overlap is not necessary to
achieve good downstream results — using disjoint local attentions, we are able
to build a simpler and more efficient long-doc QA model that matches the
performance of Longformer (Beltagy et al., 2020) with half of its pretraining
compute.

##  1 Introduction

The quadratic complexity of Transformer architectures make it prohibitive to
apply large state-of-the-art pretrained models to full-length documents. To
efficiently handle longer text while still maintaining the capacity of
attention-based models, a long list of low-complexity attention variants have
been proposed and many claim to effectively capture long-range dependencies.
Typical paradigms of these architecture innovations involve learnable sparse
attention patterns (Kitaev et al., 2020; Tay et al., 2020a; Roy et al., 2021),
fixed local patterns Beltagy et al. (2020); Ainslie et al. (2020); Zaheer et
al. (2020) and attention matrix approximation methods Wang et al. (2020);
Choromanski et al. (2021); Xiong et al. (2021). While most of these studies
have reported numbers on long sequence inputs, they tend to adopt quite
different benchmarks. For instance, Reformer (Kitaev et al., 2020) is tested
on the 64k-chunk enwik8 dataset for unidirectional language modeling;
Performer (Choromanski et al., 2021) reports masked language modeling (MLM)
perplexity on the PG-19 book corpus and protein sequences; Linformer (Wang et
al., 2020) shows MLP perplexity with various input length, while most of
documents in their pretrain corpus are short documents.111Multiple short
documents are concatenated to form long sequences. The divergence of
evaluation protocols make it hard to compare the relative performance of each
attention variant and it is also unknown how they perform in more practical
use cases, which typically involve large-scale pretraining and downstream
finetuning.

Another line of work such as Longformer (Beltagy et al., 2020) and ETC
(Ainslie et al., 2020) conduct experiments on real-world long-context tasks
such as long document QA and summarization. These methods only test simple
local attention patterns, _i.e._ , each token can only attend a small set of
nearby tokens. To reduce the pretraining cost, these models are all
initialized from the RoBERTa (Liu et al., 2019) checkpoint222By extending and
copying the position embeddings and reusing all other parameters before
further long-doc pretraining. While this paradigm is useful to achieve strong
downstream performance, it is not ideal for a fair comparison of all available
attention mechanisms since some of them use different parametrization that is
incompatible with the vanilla transformer attention.

A recently proposed benchmark (Tay et al., 2021), named long-range arena
(LRA), aims to address the devoid of unified evaluation with a bundle of long-
sequence tasks along with controlled hyperparameter settings. However, the
text-related tasks in this benchmark are either automatically generated or
artificially lengthened by enforcing byte-level inputs, making them rather
synthetic. With a fixed byte-level vocabulary and pre-specified model size,
all models are trained from scratch with the same epoch limit on each dataset.
While the evaluation protocol is consistent across architectures, this setup
still deviates from the common paradigm of applying Transformer models, _i.e._
, standard tokenization like BPE or wordpiece, large-scale pretraining
followed and task-specific finetuning (Devlin et al., 2019). Thus, an
important question yet to be addressed is whether whether the performance on
these artificially datasets are indicative for real-world long-context tasks.

In this work, our goal is to better understand the effectiveness of various
attention mechanisms through a systematic study on practical long-context
tasks. Instead of only relying on language modeling or synthetic tasks, we
test each model under the standard pretraining-and-finetuning paradigm. For a
fair comparison, we implement these attentions under a unified framework and
test them using the same Transformer architecture333We only modify the
attention calculation within the multi-head attention blocks used by RoBERTa-
large. We pretrain each model using a large corpus that contains mostly long
documents and then finetune them on tasks like long-document question
answering, full document retrieval and classification. Our experiments show
the discrepancies between the commonly used LRA benchmark and downstream
results (after pretraining). Additionally, our analysis on the best local
attention models allows us to further simplify these models and results in a
more efficient long-context encoder. More specifically, the key findings of
this paper include:

  * •

With proper tuning, we find that all the tested models can achieve similar
level of performance on the LRA benchmark while their performance diverges
significantly on large-scale pretraining and downstream tasks;

  * •

In our experiments, the other attention paradigms barely outperform the class
of simple local attentions on downstream long-context tasks when using similar
pretraining compute;

  * •

As a result of our further analysis on the best performing attention variants,
we are able to build a long-doc QA model that is on-par with Longformer while
being 2x more efficient.

##  2 Preliminaries of Tested Attention Variants

We study three representative classes of efficient attentions:

#### Fixed local patterns.

These methods restrict each token to only attend a local window of tokens. The
long-range interactions are achieved by the depth of the model. We consider
two variants of these models, the token-wise local window attention (Local
Window) proposed in Beltagy et al. (2020) where each token attends to the same
number of tokens on each side, and a simplified and easy-to-implement
blockwise version (Blockwise LW) (Zaheer et al., 2020) where each token
attends to tokens in the same block and half of the tokens in the left/right
blocks. A visualization comparing these two models is shown in Figure 1.

Figure 1: Attention pattern visualization of two types of local attentions:
Left: Local window attention as in Longformer, with window size 2; Right:
Blockwise local window attention with block size 2. The rows represent the
tokens in the sequence and the columns represent the tokens being attended to.

#### Learnable sparse attention patterns.

Instead of relying on the inductive bias of locality, methods like Reformer
(Kitaev et al., 2020) and Sinkhorn Attention Tay et al. (2020a) allows the
model to adaptively select tokens to attend to. Briefly, Reformer uses a
learnable hashing function to bucket the sequence and each token only attends
to tokens in the same bucket; Sinkhorn uses a learnable sorting function to
learn a permutation of the input segments and each token will attend to tokens
in its own segment and the corresponding segment after permutation.

#### Kernel-based/Low-rank methods.

This class of methods use matrix approximation methods to approximate the full
attention function. For sequence length $L$ and the hidden dimension $d$,
Linformer (Wang et al., 2020) simply uses a projection matrix ($L\times k$) to
reduce the length of key and value feature matrix, i.e., from $L\times d$ to
$k\times d$ ($k\ll L$). Nyström (Xiong et al., 2021) attention adopts a
classic matrix approximation method which reconstructs the full attention
matrix using a sampled sub-matrix. Performer (Choromanski et al., 2021)
eliminates the need of explicitly calculating the $L\times L$ attention matrix
by using a random feature method that can approximate the softmax kernel with
only dot-product operations.

#### Hybrid attention.

In additional to these representative methods in each class, our study also
includes the more recent Long-Short attention (Zhu et al., 2021) which has a
similar compression component as in Linformer and combines it with local
attentions. Unlike Linformer’s compression component that is simply
implemented as a standalone projection matrix, Long-Short proposes an input-
dependent compression layer, which can adaptively reduce the sequence length.

Due to space limit, we refer the readers to the corresponding papers and a
recent survey (Tay et al., 2020b) for more thorough model descriptions.

#### A note on global tokens.

For many practical NLP tasks, _e.g._ , classification or entailment, the final
layer of the models usually requires a single sequence-level representation as
input. For local attention models, it is common practice (Beltagy et al.,
2020; Zaheer et al., 2020) to mark a single or a small number of tokens as
global tokens and allow these tokens to attend to and be attended by all other
tokens. Without incurring much computational cost, these global tokens are
important to get better sequence representations and achieve good downstream
results. While the mechanism of global tokens has not been used in models with
learnable attention patterns, it is straightforward to augment Reformer and
Sinkhorn with global tokens using gather operations in standard neural network
packages, as their attention scores are still calculated by dot product and
softmax operations. Thus, in our experiments, except for the kernel-based/low-
rank methods, we augment all other models with global tokens to offset the
potential performance gap resulting from this trick.

##  3 Experiment Setup

We restrict our studies to encoder-only models and leave the analysis of
generative models to future work. We begin by implementing a collection of
efficient attentions with a unified framework (Lefaudeux et al., 2021), with
which we can consistently plug these models in our pretraining-and-finetuning
pipeline.

###  3.1 LRA Experiments

Following recent work on efficient long-range attentions, we take the LRA
benchmark as our first set of experiments. As our focus here is on NLP tasks,
we consider a subset of LRA tasks with text inputs, _i.e._ , the ListOps, IMDB
sentiment analysis and text matching tasks. All tasks are formulated as
classification problems: ListOps requires the model to predict the correct
output of an expression (10-way classification), sentiment analysis is to
predict the positive/negative labels of IMDB reviews and text matching aims to
predict citation link between papers. We follow the hyperparameter settings of
recent work (Xiong et al., 2021; Zhu et al., 2021). Two-layer Transformer
encoders are used across all tasks and enough training updates is allowed to
ensure convergence444The limit of training updates is arbitrarily set in LRA
and various work have reported hugely improved results on the text matching
task, simply by running more training steps.. Note that this is different from
the setup proposed in the original LRA benchmark, where different tasks adopt
different model sizes. It is observed from recent work that two-layer models
with smaller dimensions are sufficient to achieve similar or better results
than previously reported results. The final classification layer is added on
top of the representations of [CLS] tokens which are prepended to each
sequence.

###  3.2 Pretraining and Downstream Tasks

For practical NLP application, large-scale pretraining has become an
indispensable ingredient to fully unlock the power of Transformer models.
Thus, we conjecture that the results on the LRA benchmark, where each model is
trained from scratch using small datasets, may not be a good indicator for
practical long-context scenarios. Our second set of experiments aims to test
all these attention variants at scale.

#### Pretraining Resource.

Following Beltagy et al. (2020), we compile a corpus that contains mostly long
documents, including Stories Trinh and Le (2018), RealNews Zellers et al.
(2019), Books corpus Zhu et al. (2015) and English Wikipedia. To make the
experiments manageable and relevant for standard GPU hardware, we restrict
each model’s memory usage close to the 16GB threshold when taking 4,096 tokens
in each training batch. We control the batch size and training update across
all models: we use a batch size of 256 sequences ($2^{20}$ tokens) and
pretrain each model using the standard masked language modeling objective for
100k updates. We find that all models’ training curves almost stabilize after
this amount of training steps. We use 32 A100 GPUs for pretraining and all
model runs are finished within around 2 days.

#### Pretraining Architecture

In contrast to Longformer (Beltagy et al., 2020) and Bigbird (Zaheer et al.,
2020) where the models are initialized from RoBERTa before pretraining on long
documents, we pretrain these models from scratch, as our goal here is to
ensure fair comparison and not all architectures can reuse weights from a
standard transformer model. In particular, Nyström and Performer do not use
the standard dot-product and softmax to compute attention probabilities,
making their parameters not compatible with common models like RoBERTa or
BERT. Furthermore, other models like Linformer or LongShort introduce
additional parameters inside the attention module. In our initial experiments,
we observe initializing from the RoBERTa put these models at significant
disadvantage compared to other models (e.g., local window attention) that are
more compatible with vanilla transformers. Apart from the expanded position
embedding matrix and the attention blocks, the architecture hyperparameters
are consistent with RoBERTa-large. For both LRA and the large-scale
experiments, we adopt the pre layer-normalization trick (Xiong et al., 2020)
for feedforward and attention blocks. This usually results in better
performance in LRA and turns out to be essential for several models in the
pretraining experiments.555Linformer and Performer cannot reach reasonable
perplexity without pre-layer normalization. Additional model-specifc
architecture settings and models’ average memory usage can be found in the
Appendix.

#### Downstream Datasets and Metrics.

We consider practical tasks that naturally involve long documents. We test on
extractive QA over long documents, long document classification and document
retrieval. For the first two tasks, we use TriviaQA and Hyperpartisan
classification respectively, both of which have been used in existing long
Transformer work Beltagy et al. (2020). For full document retrieval, we
construct the dataset based on recent open-domain QA work (Lee et al., 2019)
that uses passage-level retrievers. We take an existing passage corpus from
Karpukhin et al. (2020) and reconstruct the document-level corpus. We consider
a document to be positive if it includes the answer passage. We reported
token-level answer exact match and F1 for extractive QA and the classification
accuracy for Hyperpartisan. For the retrieval task, for the ease of
experiments, we reported the mean reciprocal rank on the dev set666For each
question, the ground-truth document will be ranked with all documents (both
positive and negative) corresponding to the dev-set questions., which has been
shown to correlate well with final retrieval metric like answer recall (Oguz
et al., 2021). We conduct grid search for all tasks and report the best dev
results. Given the small size of the Hyperpartisan dataset, we reported
averaged results from 4 random seeds.

#### Task-specific Architectures for Finetuning.

We use standard architectures for the finetuning tasks: for extractive QA, a
single-layer MLP span predictor is added on top the output token
representations; the classification task uses a binary MLP classifier that
takes the [CLS] vector as input. For retrieval, we share the query and
document encoder using our pretrained models and use dot product of the [CLS]
vectors as the similarity score. For models that are compatible with global
tokens, we use all the question tokens as global tokens in the QA task and use
a single global token at the start of the sequences for both classification
and retrieval. Except for the Hyperpartisan dataset, the document lengths of
the other two datasets usually exceed 4,096 tokens after tokenization. In
these cases, we drop the tokens outside the models’ position range. We put
further implementation details and each task’s length statistics in the
Appendix.

##  4 Results and Analysis

Model | ListOps | Text | Matching | Avg Acc | GFlops
---|---|---|---|---|---
_Learnable attention pattern_ |
Sinkhorn | 37.6 | 63.8 | 80.4 | 60.6 | 0.289
LSH | 37.9 | 62.5 | 80.5 | 60.3 | 0.273
_Low-rank/kernel-based approximation_ |
Linformer | 37.7 | 61.9 | 78.4 | 59.3 | 0.271
Nystrom | 37.9 | 66.1 | 81.0 | 61.7 | 0.256
Performer | 37.1 | 66.1 | 79.8 | 61.0 | 0.205
_Hybrid attention_ |
Long-Short | 37.7 | 65.7 | 81.6 | 61.7 | 0.199
_Fixed attention pattern_ |
Local Window | 37.4 | 65.7 | 81.6 | 61.6 | 0.153
Blockwise LW | 37.4 | 65.6 | 81.3 | 61.4 | 0.146
Table 1: LRA results with our reimplementation. We use similar hyperparameter
settings (compute/memory complexity) as recent work, i.e., Nyström (Xiong et
al., 2021) and LongShort (Zhu et al., 2021)

###  4.1 Models Perform Similarly in LRA

We report our reimplemented LRA results in Table 1. While previous work (Tay
et al., 2021) have shown clear performance gap between different models, we
find that with proper tuning, the results of several models could be
significantly improved, (_e.g._ , Sinkhorn, Linformer, Reformer, Performer)
and there is no significant performance gap between any of the models when
using similar level of compute (measure by FLOPS). It is worth noting that
these improved results are not obtained by increasing the complexity of models
(_e.g._ , by using larger bucket size in Sinkhorn), as our implementation
either use similar or smaller size models compared to existing work. Also note
that while the single global token we added to Sinkhorn and LSH might be
essential for some performance gains, it only brings trivial computation
overhead.

###  4.2 Pretraining and Downstream Tasks

Model | MLM Train Perplexity
---|---
Linformer | 4.31
Performer | 6.36
Blockwise LW | 4.04
Table 2: Training perplexity of our best fixed local attention and other
faster attention variants. Each model uses similar GPU memory and training
time.

We now evaluate these models on practical benchmarks that involve real-world
long documents. As shown in Table 3, after we scale up the experiments and
control the memory consumption of each models, we see more clear differences
between these models than what we observe in LRA. Clearly, fixed local
attentions remain to be strong baselines. However, in contrast to LRA, we
observe local attentions are significantly better than the other attention
variants, for both pretraining perplexity and downstream task results. The
only exception in terms of the pretraining perplexity is the hybrid Long-Short
attention, which already integrates a local attention component: it achieves
better perplexity than fixed local attentions, but the downstream results are
at most on par with much simpler models like Blockwise LW. It is worth noting
that while we only control the training updates and memory usage in Table 3,
the conclusion still holds if we control the training time of each model: We
compare the training perplexity of Blockwise LW attention and other faster
models with fixed training time in Table 2.

Even though our LRA experiments also study tasks with text inputs, we see
clear discrepancies between the two sets of experiments. Apart from models
with fixed local attention patterns, improvements on these text LRA tasks
often do not transfer to the standard scaled pretraining-finetuning
experiments. For instance, while Performer can outperform most of the non-
local attention methods on LRA, it performs poorly on both large-scale MLM and
downstream long-context tasks. Similarly, while Nyström is significant better
than LSH in LRA on average, we observe the opposite trend in Table 3. Among
the three tasks, only ListOps is loosely aligned with the MLM perplexity.
However, the gaps between each models on this task are still too narrow to be
indicative.

Given that large-scale pretraining has become the gold-standard paradigm to
build state-of-the-art NLP models. Our findings here call for more careful and
reliable evaluation of lots of existing and emerging long-range attentions. On
the other hand, our results also reveal that the local context might still be
highly essential even in long context tasks. In the following section, we
conduct further analysis on local attention models and attempt to identify the
key ingredients of building strong NLP models for downstream long-context
tasks.

Models | MLM Pretraining | Downstream Tasks
---|---|---
PPL $\downarrow$ | k word/sec $\uparrow$ | TriviaQA | Doc Retrieval | Hyperpartisan
_Learnable attention pattern_
Sinkhorn | 4.03 | 11.8 | 63.3/68.5 | 80.9 | 95.0
LSH | 3.63 | 10.0 | 62.9/67.5 | 83.6 | 92.2
_Low-rank/kernel-based approximation_
Linformer | 4.14 | 24.6 | 59.8/65.2 | 80.3 | 88.7
Nystrom | 3.79 | 9.5 | 51.5/57.3 | 83.1 | 89.5
Performer | 5.58 | 17.2 | 24.5/31.9 | 66.8 | 94.9
_Hybrid attention_
Long-Short | 3.36 | 8.4 | 66.5/71.4 | 84.5 | 91.5
_Fixed local attention pattern_
Sliding Window | 3.47 | 9.2 | 65.6/70.7 | 83.2 | 95.3
Blockwise LW | 3.39 | 13.5 | 68.1/72.9 | 85.0 | 95.0
Table 3: MLM pretraining and downstream task results.

###  4.3 Analysis on Local Attentions

As we have seen in section 4.2, models that compute exact attention for local
contexts around each token achieve better results. Moreover, the Blockwise LW
variant performs the best even it does not guarantee a balanced left and right
context window for each token. Given these intriguing findings, we aim to
investigate the follow questions: How effective are the long-range mechanism
in local attention models? and Whether the studied long-context tasks still
mostly rely on locality bias?

Model | MLM PPL | TriviaQA | NQ Doc Retrieval | Hyperpartisan
---|---|---|---|---
Blockwise LW | 3.39 | 68.1/72.9 | 85.0 | 95.0
\- w/o overlap | 3.52 | 68.4/73.2 | 86.3 | 96.5
\- w/o overlap & global tokens | 3.54 | 56.5/61.0 | 85.4 | 94.6
Table 4: Ablation of the Blockwise LW Model.

#### Ablation Study.

In the Blockwise LW model, there are two mechanisms that enable long-range
connections: the global tokens and the attention window overlap, _i.e._ , each
token will additionally attend to half the tokens in the neighboring blocks
and the receptive field increases with model depth. While both are adopted as
common practice in existing work (Zaheer et al., 2020; Beltagy et al., 2020),
we study the isolated effect of each component in both pretraining and
finetuning experiments. For the non-overlap variant, we increase the block
size by a factor of 2 such that the amount of tokens each token attends to
remains the same. We show the results in Table 4. Surprisingly, we see
different stories in terms of MLM pretraining and downstream tasks. While both
mechanisms are useful for achieving lower MLM perplexity, only the global-
token mechanism seems important for downstream tasks. Note that in the
document retrieval tasks, removing both mechanisms results in slightly better
performance. Now the model is only able to use the first block of whole
document for retrieval. While this seems to suggest that this task is highly
local and involves strong positional bias777Many answer context appears in the
beginning of the Wikipedia page., the gap might be too trivial to be
conclusive. Additionally, we only use a single global token for this task, it
is likely that assigning more global tokens, _e.g._ , at passage boundaries,
could bring additional improvements. Investigating the particular task further
is beyond scope of this work. In terms of the effect of attention-window
overlap, it is expected that this scheme is crucial for lower perplexity: it
not only enables more distant dependencies but also reduces the number of
"boundary tokens" which can only attend to one side of the context. However,
it is counter-intuitive that the overlapping attention links between
neighboring blocks, which adds more long-range information, results in worse
downstream performance. Also note that this observation is consistent for all
the task we studied. There are two possible implications of this finding: 1)
the tested tasks still highly depend on locality bias, _i.e._ , most of
important information can be captured solely from the local bias or 2) the
overlapping scheme is not effective at capturing the long-range dependency in
downstream tasks. To confirm either hypothesis, we conduct another set of
experiments with models that have access to different sizes of context.

#### On Locality Bias.

We take the non-overlapping variant and experiment with various block sizes to
see whether longer context is actually important to studied tasks. We show the
results in Table 5 and the pretraining curves in Figure 2. While the long-
range connections brought by the attention overlaps is not helpful for
downstream results, we see that increasing the local block sizes does
consistently improve both pretraining and downstream performance although the
improvement becomes modest beyond block size 256. It is also interesting that
the models with smaller blocksizes converge faster at the early stage of
pretraining. This suggests a staged pretraining process might be more
efficient than directly training from long sequences, which aligns with Press
et al. (2021)’s finding on unidirectional LMs. Overall, this set of
experiments suggest that increasing model’s capabilities to capture longer
context is generally helpful for both pretraining and downstream tasks.
However, using overlapping attention windows is not an effective way to make
use of more context. Thus, we hypothesize the MLM perplexity improvements of
overlapping local attentions might mainly come from the reduction of the
“boundary" tokens instead of the abilities to capture long-range dependencies.
For downstream tasks, the issue of “boundary" tokens is not that essential and
the introduction of the overlapping attention windows might disrupt the
effective modeling of local context, as the attention module needs to extract
both local and distant information from the same set of tokens.888As the depth
of the model increase, the tokens’ representation will be added information of
more distant tokens.

Blocksize | Val PPL | TriviaQA Ans F1
---|---|---
64 | 4.16 | 68.9
128 | 3.74 | 70.7
256 | 3.52 | 73.2
512 | 3.39 | 73.5
Table 5: Pretraining and long-doc QA results of the non-overlapping blockwise attention with various block sizes. Figure 2: Pretraining curves of the non-overlapping block attentions with various context windows. Blocksize | Speed $\uparrow$ | Ans EM/F1
---|---|---
Longformer (64k) | 6.6k | 73.1/77.8
Blockwise LW w/o overlap (64k) | 14.8k | 73.2/77.9
Table 6: Comparing with Longformer with TriviaQA when initializing the models
from RoBERTa. Speed is measure by thousand word per second during pretraining.

#### Initializing from Existing Short Models.

While we train all models from scratch for the sake of fair comparison,
existing state-of-the-art long context models like Longformer (Beltagy et al.,
2020) or BigBird (Zaheer et al., 2020) usually initialize their longer models
from an extensively pretrained short model like RoBERTa (Liu et al., 2019).
With simple techniques like positional embedding copying, a strong long-
context encoder can be initialized without the need of pretraining from
scratch. To test our findings from the above analysis in this setting, we
follow the same scheme but use the non-overlapping block attention as
discussed in section 4.3. We compare this model with Longformer (based on
Sliding Window attention) as it uses the same long-doc corpus and pretrain-
and-finetune pipeline (_e.g.,_ packages and downstream data processing) as our
experiments.999Note that while BigBird has a similar overlapping local
attention and outperforms Longformer, it uses a larger corpus, much more
pretraining compute and different finetune pipelines, making a direct
comparison difficult.  Same as our setting in section 4.2, here we control the
batch size and number of training updates: we use a batch size of 64 and train
the model for 64k steps. Note that as we drop the attention window overlaps,
the model is 2x more efficient than Longformer: Given the same window/block
size $B$ and sequence length $L$, the complexity of the non-overlapping block
attention is $L\times B$ compared to Longformer’s $2L\times B$. We show the
TriviaQA results in Table 6, where the speed is measured by words per second
during pretraining. With only half of the pretraining compute, our model with
disjoint attention blocks achieves slightly better performance than
Longformer. This confirms that our findings about the attention overlap from
above section is still valid when the models are not trained from scratch.

##  5 Related Work

#### Long-Range Context in Language Models.

Various studies have investigated the effective usage of distant context in
unidirectional language models. Khandelwal et al. (2018) look into the context
usage of LSTM LMs and find that these models are only capable to make full use
of the nearby 50 tokens and the longer range context is only roughly captured,
_i.e._ , excluding detailed information such as word orders. Similarly,
O’Connor and Andreas (2021) study the mid- and long-range context usage in
transformer LMs, by manipulating the ordering and lexical information in the
text. Their experiments show that while long-range context is usually helpful,
most of the usable information is carried by local ordering statistics and
non-function words instead of detailed content like sentence orders. These
observations provide a possible explanation of our ablation experiments in
section 4.3 that adding overlaps to attention windows does not yield better
downstream results, despite allowing the capture of more long-range
interaction. Press et al. (2021) observe diminishing returns as they increase
the context length when using sliding windows at inference time. They propose
a staged training paradigm which train LMs from smaller context to longer
ones. This paradigm can more efficiently use the training compute and achieves
lower perplexity compared to directly training with long sequences. Given that
models with smaller attention windows converge faster at early training steps
(Figure 2), the staged training might also benefit MLM pretraining but further
investigation is required to validate whether it can also bring downstream
improvements.

#### Other Long-Range Architectures.

Instead of modifying the attention calculation, other work propose to augment
transformers with parametric long-term memories. Transformer-XL (Dai et al.,
2019) maintains frozen activations of previous tokens in memory and uses them
as additional inputs. To handle the shift of positional information of these
activations, it also requires a relative position encoding mechanism which
brings additional computation cost. The Compressive Transformer (Rae et al.,
2020) takes a similar scheme but propose to use compression modules to account
for even further memories. Both methods cannot be directly applied to long-
context understanding tasks. Under the scheme of kernel-based methods,
Katharopoulos et al. (2020); Peng et al. (2021); Schlag et al. (2021) also
attempt to linearize the softmax with kernel methods. The core ideas of these
methods are similar to Performer and they only differ in the choice of kernel
functions. Outside of the transformer families, a recent work (Lei, 2021)
proposes to augment recurrent LMs with minimal attention blocks. It is much
more efficient while achieving stronger LM perplexity compared to Transformer-
XL. However, it is still unknown whether this model has the scaling ability as
transformer architectures.

##  6 Conclusion

We present a systematic and practical study of recent proposed efficient
attention variants on real-world long-context NLP tasks. In contrast to
existing work, we are the first to test these models with a set of unified and
large-scale experiments. Our results reveal the gap between a widely used
benchmark and practical downstream tasks after conducting large-scale
pretraining. Among all the studied attention methods, we find that the
simplest local attentions outperform other complex attention paradigms on
downstream tasks. We also show that existing local attention models can be
further simplified by removing the attention-window overlap, resulting in
faster model that achieves similar or better results. Importantly, our work
calls for more careful and practical evaluation protocols while developing
long-context NLP models.

## References

  * Ainslie et al. (2020) Joshua Ainslie, Santiago Ontañón, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. 2020.  ETC: encoding long and structured inputs in transformers.  In _EMNLP (1)_ , pages 268–284. Association for Computational Linguistics.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  abs/2004.05150.
  * Choromanski et al. (2021) Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sarlós, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J. Colwell, and Adrian Weller. 2021.  Rethinking attention with performers.  In _ICLR_. OpenReview.net.
  * Dai et al. (2019) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019.  Transformer-XL: Attentive language models beyond a fixed-length context.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2978–2988, Florence, Italy. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6769–6781, Online. Association for Computational Linguistics.
  * Katharopoulos et al. (2020) Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. 2020.  Transformers are rnns: Fast autoregressive transformers with linear attention.  In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 5156–5165. PMLR.
  * Khandelwal et al. (2018) Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. 2018.  Sharp nearby, fuzzy far away: How neural language models use context.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 284–294, Melbourne, Australia. Association for Computational Linguistics.
  * Kitaev et al. (2020) Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020.  Reformer: The efficient transformer.  In _ICLR_. OpenReview.net.
  * Lee et al. (2019) Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019.  Latent retrieval for weakly supervised open domain question answering.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6086–6096, Florence, Italy. Association for Computational Linguistics.
  * Lefaudeux et al. (2021) Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, and Susan Zhang. 2021.  xformers: A modular and hackable transformer modelling library.  https://github.com/facebookresearch/xformers.
  * Lei (2021) Tao Lei. 2021.  When attention meets fast recurrence: Training language models with reduced compute.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7633–7648, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * O’Connor and Andreas (2021) Joe O’Connor and Jacob Andreas. 2021.  What context features can transformer language models use? In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 851–864, Online. Association for Computational Linguistics.
  * Oguz et al. (2021) Barlas Oguz, Kushal Lakhotia, Anchit Gupta, Patrick S. H. Lewis, Vladimir Karpukhin, Aleksandra Piktus, Xilun Chen, Sebastian Riedel, Wen-tau Yih, Sonal Gupta, and Yashar Mehdad. 2021.  Domain-matched pre-training tasks for dense retrieval.  _arXiv_ , abs/2107.13602.
  * Peng et al. (2021) Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A. Smith, and Lingpeng Kong. 2021.  Random feature attention.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
  * Press et al. (2021) Ofir Press, Noah A. Smith, and Mike Lewis. 2021.  Shortformer: Better language modeling using shorter inputs.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5493–5505, Online. Association for Computational Linguistics.
  * Rae et al. (2020) Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. 2020.  Compressive transformers for long-range sequence modelling.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Roy et al. (2021) Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. 2021.  Efficient content-based sparse attention with routing transformers.  _Transactions of the Association for Computational Linguistics_ , 9:53–68.
  * Schlag et al. (2021) Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. 2021.  Linear transformers are secretly fast weight programmers.  In _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 9355–9366. PMLR.
  * Tay et al. (2020a) Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. 2020a.  Sparse sinkhorn attention.  In _ICML_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 9438–9447. PMLR.
  * Tay et al. (2021) Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. 2021.  Long range arena : A benchmark for efficient transformers.  In _ICLR_. OpenReview.net.
  * Tay et al. (2020b) Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2020b.  Efficient transformers: A survey.  _arXiv_ , abs/2009.06732.
  * Trinh and Le (2018) Trieu H. Trinh and Quoc V. Le. 2018.  A simple method for commonsense reasoning.  abs/1806.02847.
  * Wang et al. (2020) Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. 2020.  Linformer: Self-attention with linear complexity.  _arXiv_ , abs/2006.04768.
  * Xiong et al. (2020) Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. 2020.  On layer normalization in the transformer architecture.  In _International Conference on Machine Learning_ , pages 10524–10533. PMLR.
  * Xiong et al. (2021) Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. 2021.  Nyströmformer: A nyström-based algorithm for approximating self-attention.  In _AAAI_ , pages 14138–14148. AAAI Press.
  * Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020.  Big bird: Transformers for longer sequences.  In _NeurIPS_.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019.  Defending against neural fake news.  In _NeurIPS_ , pages 9051–9062.
  * Zhu et al. (2021) Chen Zhu, Wei Ping, Chaowei Xiao, Mohammad Shoeybi, Tom Goldstein, Anima Anandkumar, and Bryan Catanzaro. 2021.  Long-short transformer: Efficient transformers for language and vision.  _NeurIPS_ , abs/2107.02192.
  * Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015.  Aligning books and movies: Towards story-like visual explanations by watching movies and reading books.  In _ICCV_ , pages 19–27. IEEE Computer Society.

##  Appendix A Appendix

Downstream Task | Hyperparameter Grid
---|---
TriviaQA | learning rate: 1e-5, 3e-5, 5e-6;
| warmup ratio: 0%, 10% of total steps;
| random seed: 42, 3, 4321;
| batch size: 32;
| max epochs: 10
NQ Doc Retrieval | learning rate: 1e-5, 5e-6, 3e-5;
| random seed: 42, 3;
| batch size: 8;
| max epochs: 10
Hyperpartisan | learning rate: 1e-5, 3e-5;
| random seed: 42, 3, 5, 1992;
| batch size: 8;
| max epochs: 40
Table 7: Hyperparamters of downstream finetuning. TriviaQA | Hyperpartisan | NQ doc Retrieval
---|---|---
Average|P95% | Average|P95% | Average|P95%
769.8|2,067.0 |  3,333.9|11,444.3 |  6,732.9|17,493.4
Table 8: Document length statistics in the tested downstream datasets.

#### Downstream Task Details.

On TriviaQA, there are usually multiple matched spans in the document, we
train the model to maximize the marginalized probability of all matched spans.
The prediction head in the classification task is defined as a 2-layer MLP
with tanh activations. For the retrieval task, we follow existing passage
retrieval methods and use in-batch documents as negative retrieval targets.
The loss is simply a cross-entropy loss defined over the scores of all
documents in the batch. All the models are finetuned using the Adam optimizer
with linear decays. We conduct grid search for all the tested models. The
hyperparameters for all the three tasks are shown in Table 7. In Table 8, we
show the average and the 95% percentile of the document lengths in each
dataset. As mentioned in the main text, we drop the tokens exceeding 4,096
tokens.

#### Pretraining Details.

Our pretraining pipeline is implemented with
fairseq101010https://fairseq.readthedocs.io/en/latest/. We control the memory
usage of each model by adjusting model-specifc hyperparameters. The details in
shown in Table 9. Due to different model designs, we are not able to exactly
control the memory consumption. However, the tested local attentions typical
requires less GPU memory than all the other models.

Model | Avg Memory Usage (GB) | Architecture Setting
---|---|---
Sinkhorn | 14.2 | block size: $128$
LSH | 18.2 | num of hash functions: $4$; chunk size: $16$
Linformer | 17.2 | compression ratio: $8$
Nystrom | 16.3 | num of landmarks: $256$; convolution kernel size: $35$;
Performer | 14.2 | random feature dimension: $256$; kernel function: $relu$
Long-Short | 16.3 | block size: $128$; num of landmarks: $32$
Sliding Window | 15.3 | attention window size: $256$
Blockwise LW | 15.1 | block size: $128$; overlap: $64$
Blockwise LW w/o global toks | 14.7 | block size: $128$
Blockwise LW w/o overlap | 13.4 | block size: $256$
Blockwise LW w/o overlap & global | 13.2 | block size: $256$
Table 9: Model-specific architecture settings and each model’s GPU memory
usage when feeding in a single sequence of 4,096 tokens.