# Neural homogenization and the physics-informed neural network for the
multiscale problems

Wing Tat Leung111Department of Mathematics, University of California, Irvine,
USA, Guang Lin222Department of Mathematics and Mechanical Engineering, Purdue
University, Zecheng Zhang 333Department of Mathematics, Purdue University

###### Abstract

Physics-informed neural network (PINN) is a data-driven approach to solve
equations. It is successful in many applications; however, the accuracy of the
PINN is not satisfactory when it is used to solve multiscale equations.
Homogenization is a way of approximating a multiscale equation by a
homogenized equation without multiscale property; it includes solving cell
problems and the homogenized equation. The cell problems are periodic; and we
propose an oversampling strategy which greatly improves the PINN accuracy on
periodic problems. The homogenized equation has constant or slow dependency
coefficient and can also be solved by PINN accurately. We hence proposed a
3-step method, neural homogenization based PINN (NH-PINN), to improve the PINN
accuracy for solving multiscale problems with the help of the homogenization.
We apply our method to solve three equations which represent three different
homogenization. The results show that the proposed method greatly improves the
PINN accuracy in particular when the scaling is small. Besides, we also find
that the PINN aided homogenization may achieve better accuracy than the
numerical methods driven homogenization; PINN hence is a potential alternative
to implementing the homogenization.

##  1 Introduction

In the recent decade, the problems having features at multiple scales are of
rising importance. For example, in biomedical applications, to obtain a more
accurate model, people need to incorporate the cell-level information to the
tissue-level model. In material application, people are interested in the
macroscopic properties of some complex composite materials which are
composited in a microscopic scale. In petroleum application [20, 18, 9, 14],
the fluid flow in the reservoir is heavily depending on the heterogeneous of
the porous media.  It is not easy to solve the equations by the classical
methods. To capture the multiscale properties, one needs to use a fine mesh
[3, 7, 12, 5, 2]; or one can use the multiscale finite element methods which
can save the computation cost but it is still not easy for some problems.
Hence researchers also studied the data driven approaches to solve the
multiscale problems.

Physics-informed neural network (PINN) is a neural network approach of solving
the partial differential equations (PDE) [22, 17, 16]. The idea of the PINN is
to approximate the solution of the PDE by a network. The PINN has been widely
used in solving both forward and inverse problems [25, 16]. Compared to the
classical numerical methods, PINN is a mesh free method and hence can
interpret the solution without a mesh. Moreover, PINN has no CFL constrain and
is easy to be used to solve time-dependent problems. For the equation with a
convection term, PINN is also powerful and one does not need to consider the
directions of the flow, which reduce the difficulty. Besides, PINN can be used
to solve the equations with the mixed direction derivatives such as paraxial
approximation [5]. The traditional numerical methods are usually hard to
implement and involve intense computation. Compared to other neural network
methods of solving the PDE [26, 6], PINN is a sample free method which does
not rely on label, we hence can use PINN on solving problems not limited to
UQ.

Although researchers have applied PINN to solve many mathematical problems,
there are only a few works about multiscale problems. In [24], the authors
point out that the classical PINN formulation is unable to capture the
multiscale property of the solution; they later proposed a method which
applies the Fourier transformation to prepossess the input to solve the
problem. Their interest in the multiscale is in the solution; this is
different from many existing real life multiscale problems we have discussed
before [15, 10, 26, 8, 4]. In this work, we mainly focus on the multiscale
problems whose multiscale property comes from the equation. One typical
example is the 2D elliptic equation which models the porous media flow. The
permeability of the equation has a multiscale nature and brings multiscale to
the solution; for example, the permeability $\kappa(x)=\sin(x)+\sin(10x)$.

We hence seek help from the PINN method; however, we find that the classical
PINN cannot give us an accurate prediction. The relative error is very large
and the training is not robust to the hyper-parameters and randomness. More
precisely, the relative error varies a lot for different runs even if we keep
all hyper-parameters the same. To alleviate the problems of the classical PINN
and take the advantages of the PINN, we propose a homogenization based
approach.

Homogenization is one of the classic ways for solving an equation with highly
heterogeneous medium [1, 13, 19, 23]. Basically, throughout the homogenization
process, we will obtain a homogenized equation of the original equation such
that the solution of this homogenized equation can capture the macroscopic
behavior of the solution of the original equation. The parameters of the
homogenized equation are normally homogeneous or smooth such that the equation
can be solved numerical in a coarse mesh. For example, we consider a diffusion
equation with a periodic media $\kappa$:

$-\nabla\cdot(\kappa(\cfrac{x}{\epsilon})\nabla u_{\epsilon})=f.$ (1)

By homogenization theory, we can obtain a homogenized equation:

$-\nabla\cdot(\kappa^{*}\nabla u_{0})=f.$ (2)

where $\kappa^{*}$ is the homogenized parameter which is a constant tensor,
and we have $u_{0}$ is converging to $u_{\epsilon}$ in $L_{2}$-norm as
$\epsilon\rightarrow 0$. Thus, one can solve the homogenized equation in a
coarse grid and use the homogenized solution $u_{0}$ to approximate the
solution of the original equation $u_{\epsilon}$.

We now propose to solve the multiscale problems by PINN with the help of
homogenization [11]. We find that the homogenization can be implemented by
PINN very naturally. The classical homogenization consists of three steps. The
first step is to solve the cell problems on a unit cube. The cell problems are
equipped with a periodic boundary condition and the permeability has no fast
dependency. It should be noted that we find the periodic PDE is not easily
solved by PINN when the dimension is high; however, we propose an oversampling
trick which greatly improves the performance of solving the periodic problems.
In all, the cell problems can be solved by PINN method accurately and
efficiently.

The second step is to evaluate the homogenized permeability which has no
multiscale property and then obtain the homogenized equations. This step
usually involves finding the derivatives of the cell problem solutions. If the
solutions are approximated by a network, the derivatives can be easily derived
by the auto differentiation of the software. Once the cell problems are solved
by the network accurately, we can anticipate getting an accurate homogenized
equation.

Lastly, we can solve the homogenized equation. The permeability associated
with this equation is usually constant or depends on the slow variable only,
the PINN has shown its power in solving such kind of equations. We conclude
that the PINN can be used to implement the entire homogenization process; one
is able to solve the multiscale PDEs by PINN with the help of the
homogenization.

We summarize the contributions of this work as follows:

  1. 1.

We observe that the accuracy of the PINN degenerates when solving the
multiscale PDE; we also explain the reason of the failure. Please note that in
this work, we focus on the PDE whose multiscale property originates from the
equation coefficients, in particular, we are interested in the multiscale
permeability with multiple frequencies.

  2. 2.

We propose a 3-step approach, neural homogenization based PINN (NH-PINN).
Instead of solving the multiscale equation by PINN directly, we first apply
PINN to solve the cell problems which are used to derive the homogenized
equation; the homogenized equation can then be easily solved by PINN.

  3. 3.

We propose an oversampling strategy to solve the periodic PDE by PINN; this
method greatly improves the accuracy of the PINN when solving the high
dimensional periodic problems.

  4. 4.

We conduct four numerical experiments which represent three different types of
homogenization. The predictions are very accurate and are much better than the
classical PINN in particular when the scaling is small.

  5. 5.

In addition to the improved accuracy of the PINN, we also observe that NH-PINN
can improve the homogenization accuracy. That is, if we apply PINN to
implement homogenization, the solution may be more accurate than the
traditional numerical methods (eg, finite element methods) driven
homogenization. We hence suggest that PINN may be a potential alternative of
implementing the homogenization.

The rests of the paper are organized as follow. In Section 2, we review the
basics of the PINN; the performance of applying classical PINN to solve
multiscale problems is also presented in this Section. Next, we will review
the homogenization method in Section 3. We will give the details of one
homogenization of the elliptic equations. Our numerical examples are not
limited to the elliptic operator, we hence also present the homogenization of
the reaction diffusion equation in the Appendix (A). We conduct four
experiments and they are shown in the Section 5.

##  2 Physics-informed neural network (PINN)

In this section, we briefly review the physics-informed neural network (PINN)
for solving the PDE; for the inversion problems, please refer to [25, 16] for
details. PINN is a data driven method to solve the PDE. The solution of the
PDE is approximated by the network and the target is to minimize the error of
this approximation. The PINN has been applied to solve various of problems;
however, it is rarely used in solving multiscale problems. Suppose we are
solving the following system in the domain $\Omega$,

$\displaystyle\mathcal{L}(u)$ $\displaystyle=f\text{ in }\Omega$
$\displaystyle\mathcal{B}(u)$ $\displaystyle=b\text{ on }\partial\Omega$

where $\mathcal{L}$ is a differential operator and $\mathcal{B}$ is the
boundary condition operator. $f$ is the given source term, $b$ is the given
boundary condition. The idea of the PINN is to build a map from $\Omega$ to
the solution by a network $\mathcal{F}_{\beta}(\cdot)$, where $\beta$ is the
parameters associated with the network. More precisely, we will solve the
following minimization problem:

$\min_{\beta}\frac{w_{1}}{N_{f}}\sum_{i=1}^{N_{f}}|\mathcal{L}(\mathcal{F}_{\beta}(p_{i}))-f(p_{i})|^{2}+\frac{w_{2}}{N_{b}}\sum_{i=1}^{N_{b}}|\mathcal{B}(\mathcal{F}_{\beta}(q_{i}))-b(q_{i})|^{2},$
(3)

where $w_{1}+w_{2}=1$ are the positive weights; $\\{p_{i}\\}\subset\Omega$,
$\\{q_{i}\\}\subset\partial\Omega$ and $N_{f},N_{b}$ are the number of points
used in discretizing the domain and boundary respectively. It should be noted
that if the PDE has other constrains such as the initial condition
$\mathcal{I}$, we just need to sample some points $z_{i}$, substitute the
points in the network $\mathcal{F}_{\beta}(\cdot)$, then approximate the
constrains by $\mathcal{I}(\mathcal{F}_{\beta}(\cdot))$ and include this loss
$\mathcal{I}(\mathcal{F}_{\beta}(\cdot))-I(\cdot)$ in Equation (3), where
$I(\cdot)$ is the given innitial conditions. PINN is easy to be implemented.

Since $\mathcal{F}_{\beta}$ approximate the solution and is smooth (neural
network), the differential operator $\mathcal{L}(\mathcal{F}_{\beta}(\cdot))$
can be evaluated with the auto differentiation package of the modern deep
learning software easily. The minimization problem can then be solved by the
variants of the gradient descent algorithm.

###  2.1 Failure of the PINN

PINN has been used to solve various problems, however the PINN performance of
solving multiscale problems has been compromised. For example, let us consider
the following elliptic equation:

$\displaystyle-\nabla\cdot(\kappa\nabla u(x))=f,x\in\Omega,$ $\displaystyle
u=0,x\in\partial\Omega,$

where $\Omega=[0,\pi]$ and $f=\sin(x)$. This is a standard elliptic problem
with a homogeneous Dirichlet boundary condition. If $\kappa(x)$ is a constant
or has no multiscale property, the PINN performs very well; however if the
permeability $\kappa(x)$ presents some multiscale property, for example,
$\kappa(x)=0.5\sin(2\pi x/\epsilon)+2$, where $\epsilon=\frac{1}{8}$, we
cannot obtain a satisfied solution (with a low relative error) using the
classical PINN. We solve the above problem and demonstrate the results in
Figure (1).

Figure 1: 1D elliptic problem without slow dependency (classical PINN). Left:
demonstration of the permeability $\kappa(x)=0.5\sin(2\pi x/\epsilon)+2$,
where $\epsilon=1/8$. Middle: learnt solution by the classical PINN vs the
reference solution. Right: relative error as a function of the training
epochs. The average error of the last 500 epochs is $0.987471$. In this
example, learning rate is 0.0001 and $w_{1}=w_{2}=0.5$ and we train the
network with Adam gradient descent for 2000 epochs. We test with different
sets of the hyper-parameters such as the learning rate and the weights of the
loss, however, we cannot obtain a satisfied result.

We will give more examples about the direct application of PINN on solving
multiscale problems later in Section (5) and provide our homogenization
strategy; that is, we propose to solve the multiscale problems in three steps
with the help of the homogenization. Our numerical experiments show that the
accuracy of the PINN is greatly improved.

###  2.2 Intuitive explanation of the failure

Now the question is why PINN fails in solving the multiscale equation. We give
an explanation motivated by the finite element methods (FEM). For the standard
FEM, it is well known that a coarse mesh FEM solution cannot resolve the
multiscale solution. For example, in 1-d case, the multiscale solution
$u_{\epsilon}$ satisfying

$-\partial_{x}\Big{(}a(\frac{x}{\epsilon})\partial_{x}u_{\epsilon}\Big{)}=f,$

converging to the homogeneous solution $u_{0}$ satisfying

$-\partial_{x}\Big{(}a^{*}(x)\partial_{x}u_{0}\Big{)}=f,$

in $L_{2}$-norm when $\epsilon$ goes to $0$ where $a^{*}$ is the harmonic
average of $a$ [11]. However, if we use linear finite element method in a
coarse grid, we have $\partial_{x}\phi_{i}$ is a piecewise constant for any
linear basis function $\phi_{i}$. Thus, we have the finite element solution
$u_{FEM}$ satisfying

$\int_{\Omega}a(\frac{x}{\epsilon})\partial_{x}u_{FEM}\partial_{x}\phi_{i}\approx\int_{\Omega}\overline{a}(x)\partial_{x}u_{FEM}\partial_{x}\phi_{i},\forall\phi_{i}$

where $\overline{a}$ is the average of $a$. We can check that the harmonic
average and normal average can have a large difference in some cases. For
example, if $a=15\sin^{2}(x)+1$, we have $a^{*}=4$ but $\overline{a}=8.5$.
Thus, we have $u_{FEM}\approx\frac{8}{17}u_{0}\approx u_{\epsilon}$ and the
relative error is
$\cfrac{\|u_{FEM}-u_{\epsilon}\|_{L_{2}}}{\|u_{\epsilon}\|_{L_{2}}}\approx\cfrac{9}{17}>50\%$.
We remark that using higher order coarse mesh finite element method in a
higher dimensional case will have a similar behavior.

It is shown in paper [21] that the network tends to learn the low frequency
solution (globally varies without local fluctuation) first. The low frequency
solution is similar to the solution in a coarse grid. If we use the multiscale
loss function to train the solution, it will tend to obtain a FEM solution-
like solution since when the solution is smooth, the multiscale residual is
similar to the the residual with average parameters. PINN hence fails in
solving multiscale problems.

One way to verify our hypothesis is: we can initialize the PINN network such
that it can capture the low frequency component of the solution; if the
network cannot capture the low frequency as the training goes, this implies
that the multiscale operator misleads the network and then the training fails.
Our proposed method (NH-PINN) can train a network which learns the coarse
scale solution by the homogenization; we hence can use NH-PINN network as the
initialization to the classical PINN network; and then train the network
further with the classical PINN loss and settings. Our experiments show that
the coarse scale solution initialization cannot improve the classical PINN.
This indicates that the set of parameters which results in an approximation of
the coarse scale solution may not be an optimum of the PINN loss. The
optimization algorithm performs well; however the loss function itself cannot
resolve the multiscale features. This verifies our hypothesis of the PINN
failure. The details of the experiments can be found in Section 5.

##  3 Homogenization

As we mentioned in the introduction, to obtain the homogenizated parameter, we
need to use the solution of the cell problems. For the completeness of the
paper, in this section, we briefly revisit the derivation of the homogenizated
equation of a diffusion problem and introduce the cell problem in this case.
We also included a discussion about the homogenization of a diffusion-reaction
equation in the appendix.

We here give an example of the homogenization of multi-dimension elliptic
problems. Consider we are solving:

$\displaystyle-\frac{\partial}{\partial
x_{i}}\bigg{(}a_{ij}(x/\epsilon)\frac{\partial}{\partial
x_{j}}u_{\epsilon}(x)\bigg{)}=f(x),x\in\Omega$ (4)

with $u_{\epsilon}(x)=0$ on $\partial\Omega$; here we use the Einstein
notation. We seek for $u_{\epsilon}(x)$ in the asymptotic expansion:

$\displaystyle u_{\epsilon}(x)=u_{0}(x,x/\epsilon)+\epsilon
u_{1}(x,x/\epsilon)+\epsilon^{2}u_{2}(x,x/\epsilon)...$ (5)

where $u_{j}(x,y)$ are periodic in $y=x/\epsilon$. Denote

$\displaystyle A^{\epsilon}=-\frac{\partial}{\partial
x_{i}}\bigg{(}a_{ij}(x/\epsilon)\frac{\partial}{\partial x_{j}}\bigg{)}$

It is not hard to check that:

$\displaystyle
A^{\epsilon}=\epsilon^{-2}A_{1}+\epsilon^{-1}A_{2}+\epsilon^{0}A_{3},$

where ,

$\displaystyle A_{1}=-\frac{\partial}{\partial
y_{i}}\bigg{(}a_{ij}(y)\frac{\partial}{\partial y_{j}}\bigg{)},$
$\displaystyle A_{2}=-\frac{\partial}{\partial
x_{i}}\bigg{(}a_{ij}(y)\frac{\partial}{\partial
y_{j}}\bigg{)}-\frac{\partial}{\partial
y_{i}}\bigg{(}a_{ij}(y)\frac{\partial}{\partial x_{j}}\bigg{)}$ $\displaystyle
A_{3}=-\frac{\partial}{\partial
x_{i}}\bigg{(}a_{ij}(y)\frac{\partial}{\partial x_{j}}\bigg{)}.$

We hence have $A_{1}u_{\epsilon}+A_{2}u_{\epsilon}+A_{3}u_{\epsilon}=f$.
Equating the terms with the same power, it follows that,

$\displaystyle A_{1}u_{0}=0,$ (6) $\displaystyle A_{1}u_{1}+A_{2}u_{0}=0,$ (7)
$\displaystyle A_{1}u_{2}+A_{2}u_{1}+A_{3}u_{0}=f.$ (8)

Substitute the $A_{1}$ and the equation (6) becomes:

$\displaystyle-\frac{\partial}{\partial
y_{i}}\bigg{(}a_{ij}(y)\frac{\partial}{\partial y_{j}}\bigg{)}u_{0}=0$

The theory of the second-order ODE implies that $u_{0}$ is independent of $y$
and this will further simplify the equation (7); it follows that,

$\displaystyle-\frac{\partial}{\partial
y_{i}}\bigg{(}a_{ij}(y)\frac{\partial}{\partial
y_{j}}\bigg{)}u_{1}=\bigg{(}\frac{\partial}{\partial
y_{i}}a_{ij}(y)\bigg{)}\frac{\partial}{\partial x_{j}}u_{0}.$

$u_{1}(x,y)$ can be solved by introducing $\chi_{j}(y)$ which is the solution
of the problem:

$\displaystyle-\frac{\partial}{\partial
y_{i}}\bigg{(}a_{ij}(y)\frac{\partial}{\partial
y_{j}}\bigg{)}\chi_{j}=\frac{\partial}{\partial y_{i}}a_{ij}(y),$ (9)
$\displaystyle\chi_{j}\text{ is periodic in $y$ with mean $0$}.$ (10)

The above problems (10) are called cell problems and they are to be solved in
one period of $y$, or, in the unit cell $Y=[0,1]^{d}$ where $d$ is the
dimension of the problem. $u_{1}$ can be then written as:

$\displaystyle u_{1}(x,y)=\chi_{j}\frac{\partial u_{0}}{\partial x_{j}}(x).$

Finally, we have,

$\displaystyle-\frac{\partial}{\partial
y_{i}}\bigg{(}a_{ij}(y)\frac{\partial}{\partial
y_{j}}\bigg{)}u_{2}=A_{2}u_{1}+A_{3}u_{0}-f.$

The ODE has a solution only if the right hand side has zero mean in one period
of $y$, i.e.,

$\displaystyle\int_{Y}(A_{2}u_{1}+A_{3}u_{0}-f)dy=0.$

This solvability condition then gives,

$\displaystyle-\frac{\partial}{\partial
x_{i}}\bigg{(}a^{*}_{ij}\frac{\partial}{\partial x_{j}}\bigg{)}u_{0}=f,$ (11)

where $a^{*}_{ij}=\int_{Y}(a_{ij}+a_{ik}\frac{\chi_{j}}{\partial y_{k}})dy$ is
called the homogenized coefficient and $u_{0}$ is the homogenized solution.

######  Remark.

Note that $a_{ij}(x/\epsilon)$ has no slow dependency $x$. If $a_{ij}(\cdot)$
also depends on slow variable $x$, the cell problems solutions also have the
$x$ dependency, i.e.,

$\displaystyle-\frac{\partial}{\partial
y_{i}}\bigg{(}a_{ij}(x,y)\frac{\partial}{\partial
y_{j}}\bigg{)}\chi_{j}(x,y)=\frac{\partial}{\partial y_{i}}a_{ij}(x,y),$

$u_{1}(x,y)$ then becomes:

$\displaystyle u_{1}(x,y)=\chi_{j}(x,y)\frac{\partial u_{0}}{\partial
x_{j}}(x)$

The homogenized coefficient $a^{*}(x)$ then has slow dependency since we only
integrate in $y$ of $\chi_{j}(x,y)$. This case will be illustrated by in
Section (5.3).

We now briefly discuss the convergence property of the homogenization. The
goal is to estimate the remainder $R$ which is defined as:

$\displaystyle u_{\epsilon}=u_{0}+\epsilon u_{1}+R.$

$R$ then satisfies that $|R|<C\epsilon$ where $C$ is a constant. We also have
the energy estimate $\int a|\frac{d}{dx}R|^{2}\leq C\epsilon$.

##  4 Neural homogenization based PINN (NH-PINN)

In this section, we introduce our proposed method: neural homogenization based
PINN (NH-PINN). We have seen in Section 2 (and will see more examples in
Section 5) that PINN is unable to solve the multiscale problems with a
satisfied accuracy; however from the Section 3, the homogenized equation loses
its multiscale property since the homogenized coefficient is either constant
or depends only on the slow variable; besides, the previous studies of the
PINN have shown that the PINN can deal with equations with constant
coefficients; this motivates us to use homogenization.

The idea is to decompose the original hard problem into easier and solvable
problems. Asymptotic expansion homogenization consists of three steps and we
have shown that the homogenized solution converges to the real solution in
Section 3; if each of the steps can be solved by PINN easily, we believe that
the accuracy of the final data driven (by PINN) solution can approximate the
real solution closely. Our method consists of three steps:

  1. 1.

Solve the cell problems using PINN.

  2. 2.

Evaluate the homogenized coefficients.

  3. 3.

Solve the homogenized equation using PINN.

The first step is to solve the cell problems. Since there is no fast
dependency, the cell problems can be solved easily by PINN. It should be noted
that the cell problems are equipped with the periodic boundary condition; we
find that the performance is satisfactory when solving the 1D periodic
problem; however, for the high dimensional cases, the PINN cannot give a good
result. To deal with this issue, we propose an oversampling trick which
facilities the PINN with a few more sampling points. Our numerical experiments
show that this trick greatly improves the PINN performance for handling $2D$
periodic problem. We introduce this trick in detail in Section (4.1).

The second step is to evaluate the coefficients. This step usually involves
calculating the derivatives of the cell problem solutions. The cell problems
are solved by PINN, i.e., the solution is approximated by a network which is
smooth, hence with the help of the auto differentiation, these derivatives can
be calculated very easily. This is one of the benefits of our method.

The last step is to solve the homogenized equation with PINN. Since PINN has
shown its power in solving the equations with constant or slow varying
coefficients, we do not expect any difficulty in this step.

There are three error sources in our problem. Firstly, the error comes from
the homogenization, however, this error is small, in particular when
$\epsilon$ is small and the scales are very different. The other two errors
come from the PINN solver accuracy when solving the cell problems and the
homogenized equation; we will show in the numerical section that all three
errors can be controlled but the error of solving the homogenized equation
dominates the total error of our proposed method.

###  4.1 Oversampling

As we have discussed before, each cell problem is equipped with a periodic
boundary condition. We find that the generalization error is large when we
solve the $2D$ equation with a periodic boundary condition. Hence, we are
going to introduce an oversampling trick which is used to aid PINN to solve an
equation with periodic boundary conditions. Our experiments show that this
trick greatly improves the training for the $2D$ problems. For simplicity,
will illustrate this idea in $1D$ case and the extension to the high
dimensional cases are similar.

Suppose $\Omega=[x_{0},x_{t}]$ and recall (3), we then have $x_{0}=q_{1}$ and
$x_{t}=q_{2}$. Since we are solving the periodic problem, the minimization
problem then becomes:

$\displaystyle\min_{\beta}\frac{w_{1}}{N_{f}}\sum_{i=1}^{N_{f}}|\mathcal{L}(\mathcal{F}_{\beta}(p_{i}))-f(p_{i})|^{2}+\frac{w_{2}}{2}\sum_{i=1}^{2}|(\mathcal{F}_{\beta}(q_{1})-\mathcal{F}_{\beta}(q_{2})|^{2},$
(12)

where the second term above indicates the periodic boundary condition. To
enforce the boundary learning, we oversample some points on both sides of the
domain. Denote $\Delta x=1/(N_{f}-1)$, we then include additional 2 sets of
points $\\{q_{i}^{left}\\}_{i=1}^{N_{o}}$ and
$\\{q_{i}^{right}\\}_{i=1}^{N_{o}}$ , where $N_{o}$ is the number of
oversampling layers and $N_{o}<N_{f}-2$. In this work, we consider the uniform
mesh, we hence have, $q_{i+1}^{left}-q_{i}^{left}=\Delta x$ and
$q_{i+1}^{right}-q_{i}^{right}=\Delta x$. We also require:

$\displaystyle q^{left}_{N_{o}}+x_{t}-x_{0}=x_{t}-\Delta x,$ $\displaystyle
q^{right}_{1}-(x_{t}-x_{0})=x_{0}+\Delta x.$

Due to the above assumptions, we have $q_{i}^{left}+x_{t}-x_{0}\in\\{p_{j}\\}$
and $q_{i}^{right}-x_{t}+x_{0}\in\\{p_{j}\\}$ for all $i=1,...,N_{o}$. If we
denote $q_{i}^{left}+x_{t}-x_{0}:=p^{left}_{i}$ and
$q_{i}^{right}-x_{t}+x_{0}=p^{right}_{i}$, the minimization becomes:

$\displaystyle\min_{\beta}\bigg{\\{}\frac{w_{1}}{N_{f}}\sum_{i=1}^{N_{f}}|\mathcal{L}(\mathcal{F}_{\beta}(p_{i}))-f(p_{i})|^{2}+\frac{w_{2}}{2}\sum_{i=1}^{2}|(\mathcal{F}_{\beta}(q_{1})-\mathcal{F}_{\beta}(q_{2})|^{2}$
$\displaystyle+\frac{w_{3}}{N_{o}}\sum_{i=1}^{N_{0}}\bigg{(}|(\mathcal{F}_{\beta}(q_{i}^{left})-\mathcal{F}_{\beta}(p_{i}^{left})|^{2}+|(\mathcal{F}_{\beta}(q_{i}^{right})-\mathcal{F}_{\beta}(p_{i}^{right})|^{2}\bigg{)}\bigg{\\}},$

where $w_{1}+w_{2}+w_{3}=1$ are positive constants.

##  5 Numerical examples

In this section, we perform numerical experiments to demonstrate our proposed
method (neural homogenization based PINN: NH-PINN). Three equations will be
considered; they have the different cell problems and the corresponding
homogenized equations are also different. We will first define all necessary
notations in Section (5.1). The remaining sections are the experiment results.

It should be noted that our reference solutions are obtained by the finite
element method. This includes solving the cell problems and the homogenized
equations. To evaluate the homogenized coefficients, one usually needs to
calculate the derivatives of the cell problems and do the integration. For the
reference solution, we use the finite difference scheme to evaluate the
derivatives; and we use the quadrature rule to compute the integration.

###  5.1 Notations

We first introduce the notations and define the relative errors. Suppose we
are solving the following equation with a proper boundary condition:

$\displaystyle L(a(x,x/\epsilon),u_{\epsilon})=f,x\in\Omega$ (13)

where $L$ is a well defined operator; $a(x,x/\epsilon)$ is the coefficient
which contributes the multiscale property to the system; and $f$ is the source
with a proper regularity. $u_{\epsilon}$ is the reference solution which is
obtained by the finite element methods (FEM) with the fine mesh. To get the
homogenized equation, we first compute the cell problems; the solutions of the
cell problems are the key component in determining the homogenized equation.
The cell problems can be solved by PINN and the traditional numerical methods.
We denote the the homogenized equation as

$\displaystyle L^{*}(b^{*}(x),v(x))=f,$ (14)

if $b^{*}(x)$ (cell problems) is computed by the FEM; here $v(x)$ is the FEM
solution to the system. We denote the homogenized equation as

$\displaystyle L^{*}(a^{*}(x),\cdot)=f,$ (15)

if $a^{*}(x)$ (cell problems) is computed by PINN. If the system is solved by
the PINN, the solution will be denoted as $p(x)$; and if this equation is
solved by the FEM, the solution will be denoted as $w(x)$. This solution
$w(x)$ can quantify the errors of the PINN method in solving the homogenized
equation (15). The notations of the different solutions are summarized in
Table (1).

Solution notation | Cell problems solver | Homogenized equation solver
---|---|---
$p(x)$ | PINN | PINN
$v(x)$ | FEM | FEM
$w(x)$ | PINN | FEM
Table 1: Notations of the solutions. $u_{\epsilon}$ which is not listed here
is the reference solution which solves the multiscale PDE directly by fine
scale finite element methods.

We compute the relative errors to quantify the accuracy of the proposed method
(NH-PINN) and the definitions are given as follow,

$\displaystyle
e_{1}=\frac{\|p(x)-u_{\epsilon}(x)\|}{\|u_{\epsilon}\|},e_{2}=\frac{\|w(x)-u_{\epsilon}(x)\|}{\|u_{\epsilon}(x)\|},e_{3}=\frac{\|p(x)-w(x)\|}{\|w(x)\|},e_{4}=\frac{\|v(x)-u_{\epsilon}(x)\|}{\|u_{\epsilon}\|}$

where $\|.\|$ is the $L_{2}$ norm.  $e_{1}$ is the error of the proposed
method (NH-PINN); this error is the ultimately measure of NH-PINN and the
number will be compared with the relative error of the classical PINN. $e_{2}$
is the error coming from the cell problems since we fix the homogenized
equation solvers; if this error is big, this means the cell problems are not
well solved by the PINN. $e_{3}$ measures the PINN solver accuracy of solving
the homogenized equation; if this error is small, it can then show that the
homogenized equation is easy to be solved by PINN. $e_{4}$ is the error of the
homogenization implemented by the classical numerical methods, this is
theoretical lower bound, NH-PINN cannot be better than this.

###  5.2 2D elliptic equation

The first example is a $2D$ elliptic problem; the detailed homogenization is
presented in Section (3). We consider the following $2D$ elliptic equation:

$\displaystyle-\frac{\partial}{\partial
x_{i}}\big{(}a(\frac{x}{\epsilon})\frac{\partial}{\partial
x_{i}}u_{\epsilon}(x)\big{)}=f(x),x\in\Omega,$ (16) $\displaystyle
u_{\epsilon}(x)=0,x\in\partial\Omega.$ (17)

In our examples, $\Omega=[0,1]^{2}$ and the permeability
$a(x/\epsilon)=2+\sin(2\pi x_{1}/\epsilon)\cos(2\pi x_{2}/\epsilon)$ and
$\epsilon=\frac{1}{8}$ (shown in Figure (2) ). The source is
$f(x)=\sin(x_{1})+\cos(x_{2})$.

Figure 2: Left: permeability of the 2D elliptic problem. Note that
$\epsilon=1/8$. Right: the reference solution.

We first present the results of solving the Equation (17) using the classical
PINN; we use the same network structure and the result of the prediction is
shown in Figure (3).

Figure 3: $2D$ elliptic problem by the classical PINN. Relative error as a
function of the training epochs, the entire history (left), the last $1500$
epochs (right). We train the network for $4000$ epochs with Adam gradient
descent (learning rate 0.0001). The average relative error of the last 500
epochs is $0.900758$;

For NH-PINN, to solve the cell problem, we use a 4-layer network ($2\times
64\rightarrow 64\times 64\rightarrow 64\times 64\rightarrow 64\times 1$)
activated by Tanh. The domain is discretized with a $101\times 101$ mesh; we
use these points and additional 2-layer oversampling in each side of the
domain in the training. We find that the relative errors for the oversampling
methods are very robust to the learning rate, the weights of the losses and
other hyper-parameters; that is, relative error drops to the similar value
even if we change some of the hyper-parameters. We do not observe the similar
results for the standard method without oversampling and hence do a series
experiments and then choose the best results. The results for $\chi_{1}$ and
$\chi_{2}$ are shown in Figure (4) and Figure (5) respectively.

Figure 4: $2D$ elliptic cell problem $\chi_{1}$. Relative error as a function
of the training epochs for $\chi_{1}$. Left: all training epochs; right: the
last 4000 epochs. The average relative errors of the last 300 epochs for the
oversampling and without oversampling are $0.072034$ and $0.326963$
respectively.

Figure 5: $2D$ elliptic cell problem $\chi_{2}$. Relative error as a function
of the training epochs for $\chi_{2}$. Left: all training epochs; right: the
last 4000 epochs. The average relative errors of the last 300 epochs for the
oversampling and without oversampling are $0.071396$ and $0.312961$
respectively.

For solving the learnt homogenized equation, we use a network with the same
structure as the cell problem. The domain is discretized with a uniform mesh
of the size $21\times 21$; and these grid points are then used in the
training. To test the network performance, we use a $101\times 101$ mesh. The
relative errors are plotted in Figure (6) and we are interested in the
relative errors after the training is stabilized, i.e., we compute the average
errors of the last $500$ epochs. The numbers are shown in the Table (2).

Figure 6: $2D$ elliptic cell problem $e_{1}$ and $e_{3}$ relative errors with respective to the training epochs. Left: history of all training epochs; right: history of the last 1200 epochs. The average relative errors of the last 500 epochs are: $e_{1}=0.082994$ and $e_{3}=0.076604$. $e_{1}$ | $e_{2}$ | $e_{3}$ | $e_{4}$
---|---|---|---
$0.082994$ | $0.0212534$ | $0.076604$ | $0.021316$
Table 2: Relative errors for the $2D$ elliptic problem.

Since $2D$ solution is not easy to observe, we also calculate the error
$u_{reference}-u_{NH-PINN}$ and plot the $2D$ error distribution; please check
Figure (7) for the illustration.

Figure 7: $2D$ elliptic error distribution. The error is calculated as the
difference between the reference solution and the NH-PINN solution.

####  5.2.1 Interpretation of the results

We can see from the Table (2) and Figure (10), the $e_{1}$ drops to around
$8\%$; this is a great improvement when compared to the classical method
(relative error $0.900758$) which we apply the PINN directly to Equation (13).
$e_{2}$ is close to $e_{4}$ which is the theoretical optimal relative error,
this means that the cell problems can be solved by PINN very accurately;
besides, $e_{1}$ is slightly bigger than $e_{3}$, together these two
comparisons imply that most of the errors come from solving the homogenized
equation. $e_{3}$ is around $0.076604$, this is the error of solving the
homogenized equation using PINN; however, athe homogenized equation is much
easier to be solved by PINN when compared to the multiscale PDE.

One interesting fact is that $e_{2}<e_{4}$. We observe the similar phenomenon
for some other examples. We have an intuitive explanation for this phenomenon.
The problem may originate from the evaluation of the
$\partial\chi_{i}/\partial y$. For the traditional method, the easiest way to
calculate the derivative is the finite difference method; computationally,
this approximation contributes an error and as a result, $a^{*}$ may not be
evaluated exactly. On the other hand, for NH-PINN method, we approximate
$\chi_{i}$ by the network. The derivatives can be derived by the auto
differentiation. If $\chi_{i}$ are approximated accurately, we may expect a
small error in the derivatives when compared to the exact derivatives.
Consequently, we obtain a better $a^{*}$ and this results in a smaller error.
The derivatives can be approximated by some high-order methods; however, the
computational costs may be higher than the neural network method. We observe
the similar situation in some other examples, we hence conclude that our
proposed NH-PINN method of solving the cell problems potentially can give an
accurate approximation of the homogenized coefficients in a relative easier
way; as a result, the final solution of the resulting homogenized equation may
also be improved.

####  5.2.2 Transfer learning

In this section, we present a transfer learning result which can explain the
reason of the PINN failure. As we have discussed in Section (2.2), we can
initialize the training of the classical PINN with a set of parameters which
can capture the low frequency component of the solution. Since we solve the
homogenized equation with the network as the classical PINN and the
homogenized solution is the coarse scale component, the trained network of NH-
PINN method can be used the initialization of the classical PINN directly. All
settings including the weights and learning rate are kept the same; but we get
an inaccurate result. The relative error blows up very quickly and the
stabilised solution has a relative error ($0.909675$, please check Figure (8))
which is similar as applying PINN directly to the multiscale PDE. This implies
that the set of parameters which gives an approximation to the coarse scale
solution is not optimal (local and global) of the multiscale loss of the
classical PINN; the multiscale loss prevents the network from learning the
coarse scale solution.

Figure 8: 2D elliptic transfer learning of classical PINN.

###  5.3 1D slowly varying periodic coefficients

In this section, we consider again an elliptic equation. The difference is
that the coefficient $a(x,x/\epsilon)$ has a slow dependency $x$; as a result,
the homogenization process is different: we need to solve a cell problem for
each $x$ in the domain; besides, the $a^{*}$ in the homogenized equation also
depends on $x$. We consider the following elliptic equation:

$\displaystyle-\frac{d}{dx}\big{(}a(x,\frac{x}{\epsilon})\frac{d}{dx}u_{\epsilon}\big{)}=f,x\in[0,\pi],$
(18) $\displaystyle u_{\epsilon}(0)=u_{\epsilon}(\pi)=0.$ (19)

In this example, $a(x,x/\epsilon)=0.5\sin(2\pi x/\epsilon)+\sin(x)+2$ where
$\epsilon=\frac{1}{8}$, it is demonstrated in Figure (9) Left, we can observe
the high frequency oscillations. The source $f(x)=\sin(x)$.

Figure 9: 1D slowly varying elliptic problem. Left: $a(x,x/\epsilon)$, where
$\epsilon=1/8$. Right: NH-PINN solution vs the reference solution. The
relative eror $e_{1}=0.005819$.

We first present the results of solving Equation (19) using the classical
PINN. We use the same network structure as NH-PINN detailed later and the
result of the prediction is shown in Figure (10).

Figure 10: $1D$ slowly varying elliptic problem by the classical PINN.
Relative error as a function of the training epochs, the entire history(left),
the last $500$ epochs (middle) and the solution (right). We train the network
for $1000$ epochs with Adam gradient descent (learning rate 0.0001).
$\omega_{1}=1/11$ and $\omega_{2}=10/11$. The average relative error of the
last 100 epochs is $0.980968$; however we observe a climb of the error (check
the middle image). We vary the learning rate and the loss weights, however,
for all the combinations, we fail to get a satisfactory result.

For NH-PINN, to solve each cell problem, we use a network of structure
$1\times 64\rightarrow 64\times 64\rightarrow 64\times 64\rightarrow 64\times
1$; the network is activated with Tanh as usual. To train the network, we use
$101$ uniform spaced points. We solve the cell problems by PINN and denote the
resulting homogenized coefficient as $a^{*}_{p}(x)$; the relative error is
then calculated as $e_{a}=\|a^{*}(x)-a^{*}_{p}(x)\|/\|a^{*}(x)\|=0.010304$. We
present $a^{*}(x)$ and $da^{*}(x)/dx$ in Figure (11).

Figure 11: $a^{*}(x)$ and $da^{*}(x)/dx$ of the slowly varying problem.

Lastly, we need to solve the learnt homogenized equation. We use a network
with the same structure as before and $101$ points for training. The model is
tested with $401$ points. The history of the relative errors $e_{1}$ and
$e_{3}$ are shown in Figure (12). We also compute the average relative errors
(when the training is stable) in Table (3); more precisely, $e_{1}$ and
$e_{3}$ are calculated as the average errors of the last $500$ epochs of the
training. The solution is demonstrated in Figure (9) Right.

Figure 12: 1D slowly varying problem $e_{1}$ and $e_{3}$ relative errors with respective to the training epochs. The average relative errors of the last 500 epochs are: $e_{1}=0.005819$ and $e_{3}=0.003665$. $e_{1}$ | $e_{2}$ | $e_{3}$ | $e_{4}$
---|---|---|---
$0.005819$ | $0.007668$ | $0.003665$ | $0.006201$
Table 3: Relative errors for the $1D$ slowly varying elliptic problem.

####  5.3.1 Interpretation of the results

We can see from the Table (3) and the Figure (10) that, the relative error
$e_{1}$ of NH-PINN is much better than the one of the classical PINN
($0.980968$). This again shows that our method improves the classical PINN
performance.  There is one comment about this example. Different from the
other two examples, the homogenized coefficients $a^{*}(x)$ can be evaluated
exactly. We hence do not observe that $e_{2}<e_{4}$ as the other two examples;
however, we can see from the Table (3) that $e_{1}<e_{4}$. This is not
observed in the other two examples. This is probably due to the FEM solver
accuracy is lower than the PINN. Since $e_{3}$ is very small, this indicates
that the PINN solver is very accurate. It follows that the total error (error
in the homogenized coefficients and error in solving the homogenized equation)
of PINN is even smaller than the theoretical upper bound.

####  5.3.2 Transfer learning

Similar as the first example, we use the trained network of NH-PINN as the
initialization of the classical PINN directly. All settings including the
weights and learning rate are kept the same; but we get an inaccurate result
with a relative error $1.005152$. The relative error increases and stabilizes
at the similar value as before (please check Figure (13)).

Figure 13: 1d slow varying elliptic transfer learning of classical PINN.

###  5.4 Diffusion and reaction equation

In this section, we consider another example whose cell problems and the
homogenized equation are different with before; The detailed homogenization
process can be seen in the Appendix (A). We are going to test the NH-PINN with
two different scales. The purpose of the experiments is to show that NH-PINN
is able to deal with different scales; however, the classical PINN fails for
both scales and performs even worse when $\epsilon$ is small. We consider the
following example:

$\displaystyle\frac{\partial u_{\epsilon}}{\partial t}-D\nabla\cdot\nabla
u_{\epsilon}+\frac{1}{\epsilon}r(\frac{x}{\epsilon})u_{\epsilon}=f,x\in\Omega,$
(20) $\displaystyle u_{\epsilon}(x)=0,x\in\partial\Omega.$ (21)

In our example, $\Omega=[-\pi,\pi]$. We set $r(x/\epsilon)=\cos(x/\epsilon)$.
We use different $\epsilon$ and present the results in the following two
sections.

####  5.4.1 Scale $\epsilon=1/10$

In this section, we set $\epsilon=\frac{1}{10}$ and $D=2$, the $r(x/\epsilon)$
is demonstrated in Figure (14) Right. The source is $f(x)=\sin(2\pi x)$.

Figure 14: Diffusion and reaction problem. Left: $r(x)=cos(x/\epsilon)$, where
$\epsilon=1/10$. Right: NH-PINN solution vs the reference solution. The
relative error $e_{1}=0.012388$.

As we have discussed before, the classical PINN cannot give us an accurate
prediction; the results of applying PINN directly are shown in Figure (15).

Figure 15: Diffusion and reaction problem with $\epsilon=1/10$ solved by the
classical PINN. Relative error of the last $7000$ epochs (left), the last
$1000$ epochs (middle) and the solution (right) of the classical PINN. We
train the network for $10000$ epochs with Adam gradient descent (learning rate
0.015). $\omega_{1}=1/7$, $\omega_{2}=5/7$ and $\omega_{3}=1/7$, where
$\omega_{3}$ is the weight of the initial condition. The average relative
error of the last 100 epochs is $0.774558$; however we observe a climb of the
error (check the middle image). We vary the learning rate and the loss
weights, however, for all the combinations, we fail to get a satisfactory
result.

For NH-PINN, the cell problem is solved with a 4-layer network of the
structure $1\times 64\rightarrow 64\times 64\rightarrow 64\times 64\rightarrow
64\times 1$; the network is activated by the Tanh function. For the network
training, we use $101$ points uniformed placed in the domain and additional
2-layer oversampling on each side of the domain. The relative error in $N$
drops to $0.053309$ when the training is stable (we take the average of the
last 500 epochs). The solution and training history of the cell problem are
shown in Figure (16).

Figure 16: Diffusion and reaction equation cell problem (23) with
$\epsilon=1/10$. Left: relative error with respected to the training epochs;
note: only last 7000 epochs are shown. The average relative error in $N(y)$ of
the last $500$ epochs is $0.053309$. Right: N(y) reference vs learnt.

To solve the homogenized equation, we use another 4-layer network activated by
Tanh; the network structure is the same to the one used in the cell problems
except that the first layer’s dimension is $2\times 64$. The spatial domain
and the temporal domain are both discretized with $100$ uniformed placed mesh
points; the grid points are then used in the training. We will test the the
solution at the terminal time and uniformly place $201$ points in space. The
training epoch is set to be $10,000$ and we use the standard Adam gradient
descent algorithm. The history of the relative errors are shown in Figure
(17); and the average relative errors when the training is stable are
presented in Table (4). We also solve Equation (21) using the classical PINN;
we use the same network structure and the result of the prediction is shown in
Figure (15).

Figure 17: Diffusion and reaction problem with $\epsilon=1/10$, $e_{1}$ and $e_{3}$ relative error as a function of the training epochs. Left: history of all training epochs; right: history of the last 2000 epochs. The average relative errors of the last 500 epochs are: $e_{1}=0.012388$ and $e_{3}=0.012448$. $e_{1}$ | $e_{2}$ | $e_{3}$ | $e_{4}$
---|---|---|---
$0.012388$ | $0.0012916$ | $0.012448$ | $0.0012917$
Table 4: Relative errors for the diffusion reaction problem.

From the Table (4) and the Figure (15), firstly we can observe that the
relative error $e_{1}$ of NH-PINN is much better than the classical PINN. If
we look at the Figure (17), the relative errors of NH-PINN are still decaying.
Since $e_{2}$ and $e_{4}$ are very closed to each other ($e_{2}<e_{4}$), we
conclude that the PINN aided homogenization is a potential alternative to the
traditional numerical driven homogenization. Finally, because $e_{2}$ is
small, this implies that most errors of the method still come from solving the
homogenized equation with PINN; however, this has been much improved when
compared to applying PINN on the multiscale PDE. The transfer learning is the
same as before, the stabilized relative error is $0.265721$ and cannot give us
a better result (please check Figure (18)).

Figure 18: Diffusion reaction equation with $\epsilon=1/10$ transfer learning
of classical PINN.

####  5.4.2 Scale $\epsilon=1/50$

In this section, we set $\epsilon=\frac{1}{50}$ and $D=2$, the $r(x/\epsilon)$
is demonstrated in Figure (19) Right. The source is $f(x)=\sin(2\pi x)$.

Figure 19: Diffusion and reaction problem. Left: $r(x)=cos(x/\epsilon)$, where
$\epsilon=1/50$. Right: NH-PINN solution vs the reference solution. The
relative error $e_{1}=0.018582$.

As we have discussed before, the classical PINN cannot give us an accurate
prediction; the results of applying PINN directly are shown in Figure (20).

Figure 20: Diffusion and reaction problem with $\epsilon=1/50$ solved by the
classical PINN. Relative error of the last $10,000$ epochs (left), the last
$7,000$ epochs (middle) and the solution (right) of the classical PINN. We
train the network for $10000$ epochs with Adam gradient descent (learning rate
0.05). $\omega_{1}=1/7$, $\omega_{2}=5/7$ and $\omega_{3}=1/7$, where
$\omega_{3}$ is the weight of the initial condition. The average relative
error of the last 100 epochs is $1.000237$; however we observe a climb of the
error (check the middle image). We vary the learning rate and the loss
weights, however, for all the combinations, we fail to get a satisfactory
result.

For $\epsilon=1/50$, we use the same network setting as before. The relative
error of $N$ drops to $0.057602$ when the training is stable (we take the
average of the last 500 epochs). The solution and training history of the cell
problem are shown in Figure (21).

Figure 21: Diffusion and reaction equation cell problem (23) with
$\epsilon=1/50$. Left: relative error with respected to the training epochs;
note: only last 7000 epochs are shown. The average relative error in $N(y)$ of
the last $500$ epochs is $0.057602$. Right: N(y) reference vs learnt.

For the homogenized equation with $\epsilon=1/50$, we use the same setting as
the $\epsilon=1/10$ case. The history of the relative errors are shown in
Figure (22); and the average relative errors when the training is stable are
presented in Table (5). We also solve Equation (21) using the classical PINN;
we use the same network structure and the result of the prediction is shown in
Figure (20).

Figure 22: Diffusion and reaction problem with $\epsilon=1/50$, $e_{1}$ and $e_{3}$ relative error as a function of the training epochs. Left: history of all training epochs; right: history of the last 2000 epochs. The average relative errors of the last 500 epochs are: $e_{1}=0.012388$ and $e_{3}=0.022184$. $e_{1}$ | $e_{2}$ | $e_{3}$ | $e_{4}$
---|---|---|---
$0.018582$ | $0.011637$ | $0.022184$ | $0.0116435$
Table 5: Relative errors for the diffusion reaction problem, $\epsilon=1/50$.

Similar as the $\epsilon=1/10$ case, firstly we can observe from the Table (5)
and the Figure (20) that the relative error $e_{1}$ of NH-PINN is much better
than the classical PINN. Since $e_{2}<e_{4}$ are closed to each other, we
conclude that the PINN aided homogenization is a potential alternative to the
traditional numerical driven homogenization. The transfer learning (check
Figure (23) ) is the same as before ( with a relative error $0.506884$) and
cannot give us a better result.

Figure 23: Diffusion reaction equation with $\epsilon=1/50$ transfer learning
of classical PINN.

##  6 Conclusion

Multiscale problems widely exist in the real life; to solve multiscale
problems, fine scale solvers are required, but the computational cost is very
high. Researchers consider solving the multiscale problem with the data-driven
approaches. In this work, we propose to solve the multiscale problems by the
physics-informed neural network (PINN) with the help of the homogenization. We
first find that the classical PINN is unable to solve the multiscale problems.
Homogenization is a 3-step PDE technique which is used to approximate the
multiscale equation by a homogenized equation. We find that all steps of the
homogenization can be implemented by the PINN accurately. In particular, we
propose an oversampling strategy which greatly improves PINN accuracy for
solving the periodic problems; this technique is used in the first step of the
homogenization. We conduct several experiments and find that the accuracy of
PINN is greatly improved by our method. We also observe that PINN assisted
homogenization is also an accurate approach of implementing homogenization; we
hence conclude that the PINN is a potential alternative to the numerical
driven homogenization.

##  Appendix A Homogenization of the diffusion reaction equation

In this section, we provide the details of the homogenizing of a diffusion
reaction equation. The problem is defined as:

$\displaystyle\frac{\partial u_{\epsilon}}{\partial t}-D\nabla\cdot\nabla
u_{\epsilon}+\frac{1}{\epsilon}r(x/\epsilon)u_{\epsilon}=f,$ (22)

where $\int_{Y}r(y)dy=0$ for the solvability; $Y=[0,1]^{d}$ is the unit cube.
We seek the same asymptotic expansion as (5) and by equating the power of
$\epsilon$, we have:

$\displaystyle-D\nabla\cdot\nabla u_{0}=0$

We have $u_{0}$ is independent of $y$ and we can then further simplifies the
$\epsilon^{-1}$ term,

$\displaystyle-D\nabla\cdot\nabla u_{1}=r_{0}u_{0},$

where $u_{1}(x,y)$ is double periodic in $y$ with period $Y$. The cell problem
can then be defined as:

$\displaystyle-D\nabla\cdot\nabla N(y)=-r(y),y\in Y.$ (23)

The problem has the double periodic boundary condition and we assume $N(y)$
has zero average in $Y$; the $\int_{Y}r(y)dy=0$ guarantees the equation is
solvable. $u_{1}$ can be then expressed as:

$\displaystyle u_{1}(x,y)=N(y)u_{0}(x).$

Finally for $\epsilon^{0}$ term,

$\displaystyle\frac{\partial u_{0}}{\partial t}-D\nabla\cdot\nabla
u_{0}-D\Delta_{xy}u_{1}-D\nabla\cdot\nabla u_{2}+r(t)u_{1}=f(x).$

Integrate the above equation over $y$ in one period $Y$, we finally have,

$\displaystyle\frac{\partial u_{0}}{\partial t}-D\nabla\cdot\nabla
u_{0}+r^{*}u_{0}=f,$ (24)

where the homogenized coefficient $r^{*}$ is:

$\displaystyle\int_{Y}r(y)N(y)dy.$

## References

  * [1] A. Bensoussan, J.-L. Lions, and G. Papanicolaou.  Asymptotic analysis for periodic structures, volume 374.  American Mathematical Soc., 2011.
  * [2] B. Chetverushkin, E. Chung, Y. Efendiev, S.-M. Pun, and Z. Zhang.  Computational multiscale methods for quasi-gas dynamic equations.  Journal of Computational Physics, 440:110352, 2021.
  * [3] E. Chung, Y. Efendiev, and T. Y. Hou.  Adaptive multiscale model reduction with generalized multiscale finite element methods.  Journal of Computational Physics, 320:69–95, 2016.
  * [4] E. Chung, Y. Efendiev, W. T. Leung, S.-M. Pun, and Z. Zhang.  Multi-agent reinforcement learning accelerated mcmc on multiscale inversion problem.  arXiv preprint arXiv:2011.08954, 2020.
  * [5] E. Chung, Y. Efendiev, S.-M. Pun, and Z. Zhang.  Computational multiscale methods for parabolic wave approximations in heterogeneous media.  arXiv preprint arXiv:2104.02283, 2021.
  * [6] E. Chung, W. T. Leung, S.-M. Pun, and Z. Zhang.  A multi-stage deep learning based algorithm for multiscale model reduction.  Journal of Computational and Applied Mathematics, 394:113506, 2021\.
  * [7] E. T. Chung, Y. Efendiev, and W. T. Leung.  Constraint energy minimizing generalized multiscale finite element method.  Computer Methods in Applied Mechanics and Engineering, 339:298–319, 2018.
  * [8] E. T. Chung, Y. Efendiev, W. T. Leung, and Z. Zhang.  Cluster-based generalized multiscale finite element method for elliptic pdes with random coefficients.  Journal of Computational Physics, 371:606–617, 2018.
  * [9] Y. Efendiev, L. Durlofsky, and S. Lee.  Modeling of subgrid effects in coarse-scale simulations of transport in heterogeneous porous media.  Water Resources Research, 36(8):2031–2041, 2000.
  * [10] Y. Efendiev, J. Galvis, and T. Y. Hou.  Generalized multiscale finite element methods (gmsfem).  Journal of computational physics, 251:116–135, 2013.
  * [11] Y. Efendiev and T. Y. Hou.  Multiscale finite element methods: theory and applications, volume 4.  Springer Science & Business Media, 2009.
  * [12] Y. R. Efendiev, T. Y. Hou, and X.-H. Wu.  Convergence of a nonconforming multiscale finite element method.  SIAM Journal on Numerical Analysis, 37(3):888–910, 2000.
  * [13] B. Engquist and P. E. Souganidis.  Asymptotic and numerical homogenization.  Acta Numerica, 17:147–190, 2008.
  * [14] E. Gildin, M. Ghasemi, A. Romanovskay, and Y. Efendiev.  Nonlinear complexity reduction for fast simulation of flow in heterogeneous porous media.  In SPE Reservoir Simulation Symposium. OnePetro, 2013.
  * [15] T. Y. Hou and X.-H. Wu.  A multiscale finite element method for elliptic problems in composite materials and porous media.  Journal of computational physics, 134(1):169–189, 1997.
  * [16] G. Lin, Y. Wang, and Z. Zhang.  Multi-variance replica exchange stochastic gradient mcmc for inverse and forward bayesian physics-informed neural network.  arXiv preprint arXiv:2107.06330, 2021.
  * [17] L. Liu, T. Zeng, and Z. Zhang.  A deep neural network approach on solving the linear transport model under diffusive scaling.  arXiv preprint arXiv:2102.12408, 2021.
  * [18] X. Ma, M. Al-Harbi, A. Datta-Gupta, and Y. Efendiev.  An efficient two-stage sampling method for uncertainty quantification in history matching geological models.  SPE Journal, 13(01):77–87, 2008.
  * [19] A. A. Pankov.  G-convergence and homogenization of nonlinear partial differential operators, volume 422.  Springer Science & Business Media, 2013.
  * [20] P. Popov, G. Qin, L. Bi, Y. Efendiev, R. E. Ewing, and J. Li.  Multiphysics and multiscale methods for modeling fluid flow through naturally fractured carbonate karst reservoirs.  SPE Reservoir Evaluation & Engineering, 12(02):218–231, 2009.
  * [21] N. Rahaman, A. Baratin, D. Arpit, F. Draxler, M. Lin, F. Hamprecht, Y. Bengio, and A. Courville.  On the spectral bias of neural networks.  In International Conference on Machine Learning, pages 5301–5310. PMLR, 2019.
  * [22] M. Raissi, P. Perdikaris, and G. E. Karniadakis.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations.  Journal of Computational Physics, 378:686–707, 2019.
  * [23] L. Tartar.  Compensated compactness and applications to partial differential equations.  In Nonlinear analysis and mechanics: Heriot-Watt symposium, volume 4, pages 136–212, 1979.
  * [24] S. Wang, H. Wang, and P. Perdikaris.  On the eigenvector bias of fourier feature networks: From regression to solving multi-scale pdes with physics-informed neural networks.  Computer Methods in Applied Mechanics and Engineering, 384:113938, 2021.
  * [25] L. Yang, X. Meng, and G. E. Karniadakis.  B-pinns: Bayesian physics-informed neural networks for forward and inverse pde problems with noisy data.  Journal of Computational Physics, 425:109913, 2021.
  * [26] Z. Zhang, E. T. Chung, Y. Efendiev, and W. T. Leung.  Learning algorithms for coarsening uncertainty space and applications to multiscale simulations.  Mathematics, 8(5):720, 2020.