# LSA-PINN: Linear Boundary Connectivity Loss for Solving PDEs on Complex
Geometry

Jian Cheng Wong∗
*: equal contribution Institute of High Performance Computing
Agency for Science, Technology and Research
Singapore
<EMAIL_ADDRESS>Pao-Hsiung Chiu∗
Institute of High Performance Computing
Agency for Science, Technology and Research
Singapore
<EMAIL_ADDRESS>Chinchun Ooi
My Ha Dao
Institute of High Performance Computing
Agency for Science, Technology and Research
Singapore
<EMAIL_ADDRESS>Institute of High Performance Computing
Agency for Science, Technology and Research
Singapore
<EMAIL_ADDRESS>Yew-Soon Ong
School of Computer Science and Engineering
Nanyang Technological University
Singapore
<EMAIL_ADDRESS>
###### Abstract

We present a novel loss formulation for efficient learning of complex dynamics
from governing physics, typically described by partial differential equations
(PDEs), using physics-informed neural networks (PINNs). In our experiments,
existing versions of PINNs are seen to learn poorly in many problems,
especially for complex geometries, as it becomes increasingly difficult to
establish appropriate sampling strategy at the near boundary region. Overly
dense sampling can adversely impede training convergence if the local gradient
behaviors are too complex to be adequately modelled by PINNs. On the other
hand, if the samples are too sparse, existing PINNs tend to overfit the near
boundary region, leading to incorrect solution. To prevent such issues, we
propose a new Boundary Connectivity (BCXN) loss function which provides linear
local structure approximation (LSA) to the gradient behaviors at the boundary
for PINN. Our BCXN-loss implicitly imposes local structure during training,
thus facilitating fast physics-informed learning across entire problem domains
with order of magnitude sparser training samples. This LSA-PINN method shows a
few orders of magnitude smaller errors than existing methods in terms of the
standard L2-norm metric, while using dramatically fewer training samples and
iterations. Our proposed LSA-PINN does not pose any requirement on the
differentiable property of the networks, and we demonstrate its benefits and
ease of implementation on both multi-layer perceptron and convolutional neural
network versions as commonly used in current PINN literature.

###### Index Terms:

physics-informed neural networks, local structure approximation, boundary
connectivity, partial differential equations

##  I Introduction

Figure 1: PINNs learning the solution of 2D N-S equations in a complex
geometry (wavy channel flow problem, $Re=100$). Our LSA-PINN method can learn
accurate solution with fewer (50% less) training samples and faster speed (50%
less training iterations).

Physics-informed neural networks (PINNs) have emerged as a promising method
for learning the solution of dynamical system from the governing physics [1].
PINNs have recently been studied for a wide range of physical phenomena and
applications across science and engineering domains — electromagnetic, fluid
dynamics, heat transfer, etc [2, 3]. The distinctive feature of PINNs is the
use of governing physics law, typically in the form of partial differential
equations (PDEs), as the learning objective. This physics-informed learning
constrains the PINN from violating the underlying physics at all training
points sampled from the problem domain.

Existing PINNs evaluate the PDE constraints in their training loss by either
automatic differentiation (AD) or numerical differentiation (ND)-type method.
AD-loss has become a popular choice because it can exactly compute the
derivative terms at any sample location directly from the input and the
network weights. While both methods have their pros and cons, ND-loss can be
flexibly implemented across many different neural network (NN) architectures,
including both multi-layer perceptrons (MLPs) and convolutional neural
networks (CNNs), because they do not require the NN to retain
differentiability, unlike AD. Recent studies [4, 5, 6] have suggested that ND-
type methods and especially coupled-automatic-numerical differentiation
(CAN)-loss [6] can more robustly and efficiently produce accurate solutions
with fewer training samples, whereas conventional AD-loss is prone to failure
during training. This is because ND-type methods approximate high order
derivatives using _PINN output from neighbouring samples_ , hence, they can
effectively connect sparse samples into piecewise regions via these local
approximations, thereby facilitating fast physics-informed learning across the
entire domain with sparser samples.

When dealing with irregular geometries, it becomes difficult for existing
PINNs to perfectly connect training samples in the domain’s interior to the
boundary. Failing to do so can cause undesirable training failure as the PINN
starts to overfit at the near boundary region. Since many PDEs of practical
interest are boundary-value problems, it is desirable to have the PINNs model
the correct boundary behaviors. While adding dense samples to better refine
the piecewise local regions near the boundary may improve accuracy, the extent
to which sampling needs to be increased is empirical, and a denser sampling
strategy may adversely impede training convergence. Moreover, it may not be
feasible to locally refine the near boundary sampling density for PINNs with
CNN-architecture.

Hence, we propose a loss function formulation in this work that helps provide
linear approximation to the local gradient behaviour at the boundary, thereby
restoring connectivity between domain boundary and near boundary interior
samples. Such local structure approximation also provides certain structural
bias during PINN training to prevent inaccurate representation near the
boundaries. This new boundary connectivity (BCXN)-loss function is key to a
novel class of LSA-PINN method which can more efficiently learn the solution
to PDEs with sparser training samples, regardless of domain geometry; see
example in Fig. 1. In addition, this method can be jointly implemented with
other PINN advances in optimization such as loss balancing, domain
decomposition and adaptive sampling.

In the rest of this work, we present comprehensive experiments to demonstrate
i) the flexible implementation of LSA-PINN method for both MLP and CNN
architectures; and ii) the effectiveness of LSA-PINN for learning multiple
complex fluid dynamical systems, including for meta-modelling. Compared to
conventional PINNs with the AD- and ND-loss, our LSA-PINNs with linear BCXN-
loss are shown to be capable of tackling challenging PDE problems while using
sparser training samples, hence expanding the exciting potential of PINNs for
learning complex, real-world dynamics.

##  II Related Work

Efficient sampling in PINNs. The theoretical limit of physics-informed loss
learning in relation to training samples has been provided by prior studies
[7, 8]. To improve PINN training speed for practical applications [9], several
studies have focused on efficient sampling strategies such as importance
sampling, adaptive sampling, and sequential sampling to reduce the amount of
training samples required during PINN training [10, 11, 12, 13, 14, 15, 16,
17, 18]. Domain decomposition and parallelization strategies have also been
explored to speed up training [19, 20, 21, 22, 23]. Our LSA-PINN differs from
these works in that we make physics-informed learning more robust in the
sparse sample regime via a newly-proposed BCXN-loss.

CNN architecture and numerical differentiation (ND)-type loss for PINNs. CNN-
based formulation allows us to design and train larger, more powerful
networks, hence it has potential to be scalable for more complex, large-scale
PDE problem [24, 4, 25, 26, 27, 28]. However, cumbersome coordinate
transformations may need to be performed to better handle irregular domains.
CNN-architecture PINNs usually utilize ND methods for computing the PDE loss
on their input grid, which is much cheaper to compute as compared to AD-based
loss [4]. Hybrid frameworks that couple both AD- and ND-type loss have also
been proposed for MLP architecture PINNs to unify the advantages of both
methods [19, 29, 5, 6]. Our method further augments the efficiency and
applicability of ND-type loss on irregular geometries.

Enforcing BC constraint in PINN loss. It is important for the PINNs to
prioritize learning the correct boundary behaviors [30, 31]. The BCs are
usually enforced into the PINN loss as a soft constraint via penalty method.
Various strategies have been explored in this context to dynamically calibrate
the relative important between PDE and BC constraints during PINN training
[32, 33, 34, 35, 36, 37, 38, 39, 40, 41]. There are also other approaches to
bypass the loss balancing issue. For example, one can devise an ansatz
function such that the BCs are exactly satisfied by construction or implicitly
formulate the BC constraints into the PDE loss [42, 43, 44, 45, 46, 47, 48],
leaving only single loss term from PDE residuals to be optimized. Another
example is the use of augmented Lagrangian method to impose the BC constraints
into PINN loss [49]. Our method implicitly incorporates BCs into the PDE loss
via a different strategy.

##  III Preliminary

###  III-A Governing physics - incompressible Navier-Stokes (N-S) equations

Fundamentally, PINNs have been shown to be applicable to many different
physical systems. The present study focuses on learning fluid dynamics with
PINNs. We consider fluid problems where the governing physics are the steady-
state, incompressible N-S equations derived from the conservation of mass and
momentum:

$\displaystyle\nabla\cdot\vec{u}=0$ (1a)
$\displaystyle\left(\vec{u}\cdot\nabla\right)\vec{u}=Re^{-1}\Delta\vec{u}-\nabla
p$ (1b)

In the above PDEs, the primitive variables $\vec{u}$ and $p$ are velocity
vector and pressure while Reynolds number ($Re$) represents the ratio of
inertial to viscous forces. $\vec{u}$ consists of 2 components
$\left(u,v\right)$ for a 2D case. While this work uses Cartesian coordinates,
this formulation is extendable to other coordinate systems.

###  III-B MLP- and CNN-architecture PINNs

We use a fully connected DNN architecture (e.g., MLP, CNN) to represent the
solution of the dynamical process U $=[\vec{u},p]^{T}$. For MLPs, the input
$\vec{x}$ is a point coordinate in spatial domain. For CNNs, the input X is a
tensor with a fixed shape, comprising the entire (discretized) spatial domain.
The accuracy of the PINN outputs U($\vec{x}$;w) given input $\vec{x}$ is
determined by the network parameters w, which are optimized w.r.t. the PINN
loss function during training. The PINN loss function is defined as the
composition of a PDE loss component $\left(L_{PDE}\right)$ and a BC loss
component $\left(L_{BC}\right)$:

$\displaystyle L_{PINN}$
$\displaystyle=\lambda_{PDE}L_{PDE}+\lambda_{BC}L_{BC}$ (2a)
$\displaystyle\begin{split}L_{PDE}&=\big{\|}\nabla\cdot\vec{u}(\vec{x};\textbf{{w}})\big{\|}_{\Omega}^{2}+\big{\|}\left(\vec{u}(\vec{x};\textbf{{w}})\cdot\nabla\right)\vec{u}(\vec{x};\textbf{{w}})\\\
&-Re^{-1}\Delta\vec{u}(\vec{x};\textbf{{w}})+\nabla
p(\vec{x};\textbf{{w}})\big{\|}_{\Omega}^{2}\end{split}$ (2b) $\displaystyle
L_{BC}$
$\displaystyle=\big{\|}B[u(\vec{x};\textbf{{w}})]-\vec{u}(\vec{x})\big{\|}_{\partial\Omega}^{2}$
(2c)

The PDE loss penalizes deviation from governing N-S equations for the PINN
output U($\vec{x}$;w) over the fluid domain $\vec{x}\in\Omega$, whereas the BC
loss penalizes deviation from the desired Dirichlet boundary condition
$\vec{u}_{BC}(\vec{x})$ which constrains the velocity field at the domain
boundary $\vec{x}\in\partial\Omega$. The relative weights $\lambda_{PDE}$ and
$\lambda_{BC}$ in Eq.(2)a control the trade-off between different loss
components.

The PDE and BC loss components are defined over a continuous domain, but for
practical reasons, we evaluate the discretized PINN loss from a finite set of
$n$ samples $D=\\{(\vec{x}_{i})\\}_{i=1}^{n}$ during training. The CNN-
architecture PINNs naturally acquire an input with equidistantly spaced grid
where all the samples are coming from. For the MLP-architecture PINNs, it is
also convenient to design training samples based on equidistantly spaced grid,
then $m(<n)$ samples can be randomly drawn to compute the PINN loss during
each stochastic gradient descent (SGD) mini-batch training iteration.

Although it is ideal for a PINN to satisfy the PDE constraint on a very dense
set of samples, i.e., $\Delta\vec{x}\rightarrow 0$ where $\Delta\vec{x}$ is
the distance between two adjacent sample points, training such PINN may not be
computationally feasible even with state-of-the-art SGD variants. For a more
complex physics problem, training on dense samples is correspondingly more
computationally expensive, and is also more likely to become unsuccessful due
to adverse impact by some local region with very complex gradient behaviours,
e.g., abnormally high gradient [50, 51, 52, 53, 54, 40]. Given the same
problem, training PINNs on sparser samples means fewer restrictions (the PDE
constraint that needs to be satisfied) and an increased likelihood of avoiding
problematic gradient points, thereby accelerating training. The _relaxation of
PDE constraint via reducing the sample density makes learning physically
meaningful solution feasible_ , although it may introduce some approximation
error to the solution. Note that there is still an open question as to how
best balance sample density and solution accuracy. Methods that can
consistently produce more accurate solutions with sparse training samples like
LSA-PINN are hugely advantageous for learning complex, real-world dynamics.

###  III-C PINN learning challenge on sparse samples

When samples are sparse relative to the local complexity, a highly flexible,
over-parameterized PINN can become “overfit”, leading to incorrect solution
even as the PDE constraint is fulfilled at all sample points (i.e., the
convergence is good). For AD-loss PINN, this phenomenon can occur at any
location within the computation domain. On the contrary, ND-loss are more
robust to sample density, because ND-type methods approximate the derivative
terms by using _output from neighbouring samples_ , e.g., a finite difference-
type stencil, which connects sparse samples into piecewise continuous regions
to facilitate fast training across the entire problem domain. The use of
lower-order local structure approximation for the derivative terms also
provides certain structural bias for PINNs to reduce overfitting. For example,
abnormally high gradients are regularized so that it is easier for PINNs to
learn the solution, with the accuracy being dependent on the approximation
employed.

However, ND-loss can still exhibit training failure if the near-boundary
training samples do not perfectly connect to the domain boundary through their
stencils (as illustrated in Fig. 2). Their outputs essentially act as free
parameters during optimization for satisfying the PDE constraint at near-
boundary samples, and the PINN training tends to “overfit” the near-boundary
region to more easily jointly satisfy the PDE and BC constraints, leading to
incorrect solution. _This becomes a major issue for ND-loss when dealing with
irregular geometries_ , because it is almost impossible to design sample
locations such that their stencils perfectly connect the domain boundary to
other training samples.

One potential solution is to increase the sampling density near the boundary,
i.e., $\Delta\vec{x}\rightarrow 0$, to avoid such over-fitting. This however
increases the complexity of the optimization problem and can cause adverse
effects on training. It is also very challenging to locally refine the
sampling density for PINNs with CNN-architecture. To resolve this issue
without increasing the sample density, we propose BCXN-loss to impose local
structure approximation to govern the near boundary gradient behaviour and
restore connectivity between domain boundary and near-boundary samples.

(a) Stencil points connect to boundary

(b) Stencil points fall outside domain

(c) Extrapolation along normal direction

Figure 2: (a)-(b) Schematic of a near-boundary training sample (red circle)
and its stencil points (square) typically used for evaluating PDE constraint
in ND-loss. BC information is passed to stencils in fluid domain (blue square)
(a) successfully in regular domain; and (b) unsuccessfully in an irregular
domain due to under-defined stencils (unfilled square) outside the domain. (c)
Schematic of related definitions under the present LSA-PINN framework. BCXN-
loss enforces a constraint across 3 points: external stencil point A, boundary
point P, and mirror point Q inside fluid domain.

##  IV Method

###  IV-A Enforcing linear constraint at near-boundary samples

This section outlines the proposed LSA-PINN method. Here we denote the
external (out-of-domain) stencil of a near-boundary sample as ES point(s). As
per Fig. 2c, the boundary condition $\vec{u}_{BC}$ and field value
$\vec{u}_{MI}$ are defined at the boundary point P and a chosen mirror point Q
inside the domain, while the field value $\vec{u}_{ES}$ at ES point A is the
value to be determined.

By employing Taylor series expansion on points A and Q relative to point P
along $\overline{AQ}$ with respect to local coordinate $n$ (normal direction
of surface), the following can be derived:

$\displaystyle\vec{u}_{ES}=\vec{u}_{BC}+\overline{AP}\frac{\partial\vec{u}_{BC}}{\partial
n}+\frac{\overline{AP}^{2}}{2}\frac{\partial^{2}\vec{u}_{BC}}{\partial
n^{2}}+O(\overline{AP}^{3})$ (3a)
$\displaystyle\vec{u}_{MI}=\vec{u}_{BC}-\overline{PQ}\frac{\partial\vec{u}_{BC}}{\partial
n}+\frac{\overline{PQ}^{2}}{2}\frac{\partial^{2}\vec{u}_{BC}}{\partial
n^{2}}+O(\overline{PQ}^{3})$ (3b)

where $\overline{AP}$ and $\overline{PQ}$ are the distances between A and P
and between P and Q, and $\overline{AQ}=\overline{AP}+\overline{PQ}$. The
field value $\vec{u}_{ES}$ at point A can then be derived as:

$\vec{u}_{ES}=\vec{u}_{MI}+(\vec{u}_{BC}-\vec{u}_{MI})\times\frac{\overline{AQ}}{\overline{PQ}}+O(\overline{AQ}^{2})$
(4)

From the above equation, it can be seen that $\vec{u}_{ES}$ is linearly
constrained by $\vec{u}_{BC}$ and $\vec{u}_{MI}$. While the mirror point Q can
be conveniently chosen at the center of stencil, i.e., the near-boundary
sample location, we choose mirror point Q along the normal direction of
boundary P (ref. to Fig. 2c) as it is more general and performs better in our
experiments. Hence, $\overline{AP}=\overline{PQ}$, and Eq. (4) becomes:

$\vec{u}_{ES}=2\vec{u}_{BC}-\vec{u}_{MI}$ (5)

A similar process can be applied for the derivation of corresponding linear
constraints for Neumann- and Robin-type boundary conditions.

###  IV-B Boundary connectivity (BCXN)-loss with direct forcing

We can _directly apply Eq. ( 5) to compute the field value $\vec{u}_{ES}$ for
any external stencil point_ during the evaluation of PDE constraint on near-
boundary samples with ND-type schemes. This direct forcing approach has an
association to the direct forcing immersed boundary methods in numerical
computing. We can then use the following BCXN-loss:

$L_{LSA-PINN}=L_{PDE(BCXN)}$ (6)

as our LSA-PINN’s PDE loss term whereby the PDE constraints on near-boundary
samples are modulated by the local linear constraint. In such implementation,
the BCs are implicitly infused into the training loss, obviating the BC loss
term.

###  IV-C Procedure to evaluate the field value $\vec{u}_{MI}$ at mirror point

Note that PINN methods in general require a routine to sample from inside the
geometry of interest as well as the boundary. As compared to conventional
PINNs, LSA-PINN involves additional steps. We first describe the steps to
compute the mirror point location (along normal direction):

  * •

Determine whether a stencil is external. In this study, we first construct a
level set function $\phi$ of the geometry of interest [55], where the shortest
distance $\overline{AP}$ between the stencil point A and boundary point P can
be found. From the sign of level set function, we can determine whether a
stencil is external.

  * •

Compute mirror point location for external stencil. Once $\overline{AP}$ is
determined, we compute the location of mirror point Q inside the fluid domain,
with the distance $\overline{AP}=\overline{PQ}$.

In practice, given a fixed set of training samples, all the external stencil
points and their mirror locations can be pre-computed. On the other hand, the
field value $\vec{u}_{MI}$ at mirror points depends on the LSA-PINN’s output
and are evaluated inside a training iteration:

  * •

MLP-architecture. The $\vec{u}_{MI}$ value can be directly obtained by
evaluating $\vec{u}_{MI}(\vec{x};\textbf{{w}})$ at the mirror point.

  * •

CNN-architecture. When the location of mirror point does not coincide with the
CNN grid, the following inverse-distance-weighted interpolation function can
be utilized to obtain the $\vec{u}_{MI}$ value at mirror point [56].

$\displaystyle u(\tilde{x},\tilde{y})=\Sigma
u(x,y)\phi^{\prime}\Bigl{(}\frac{x-\tilde{x}}{h}\Bigr{)}\phi^{\prime}\Bigl{(}\frac{y-\tilde{y}}{h}\Bigr{)}$
(7a)
$\displaystyle\phi^{\prime}\Bigl{(}\frac{x-\tilde{x}}{h}\Bigr{)}=\phi\Bigl{(}\frac{x-\tilde{x}}{h}\Bigr{)}/\Sigma\phi\Bigl{(}\frac{x-\tilde{x}}{h}\Bigr{)}$
(7b)
$\displaystyle\phi^{\prime}\Bigl{(}\frac{y-\tilde{y}}{h}\Bigr{)}=\phi\Bigl{(}\frac{y-\tilde{y}}{h}\Bigr{)}/\Sigma\phi\Bigl{(}\frac{y-\tilde{y}}{h}\Bigr{)}$
(7c)
$\displaystyle\phi(r)=\begin{cases}\frac{1}{8}\Bigl{(}3-2|r|+\sqrt{1+4|r|+4r^{2}}\Bigr{)},&|r|\leq
1\\\ \frac{1}{8}\Bigl{(}5-2|r|-\sqrt{-7+12|r|-4r^{2}}\Bigr{)},&1<|r|\leq 2\\\
0,&|r|>2\end{cases}$ (7d)

$(\tilde{x},\tilde{y})$ is the location of mirror point to be interpolated,
while $(x,y)$ is the location of training samples inside the fluid domain. In
this study, $h$ is chosen as $\Delta x$, which leads to a $2\Delta x$
interpolation radius. Once $\vec{u}_{MI}$ and $\vec{u}_{BC}$ are evaluated, we
can compute $\vec{u}_{ES}$ based on Eq. (5) during training.

##  V Results

Figure 3: Comparison of $u$-, $v$-velocity and pressure $p$ contour between
the ground truth and PINN solutions (median MSE from 10 runs), for the wavy
channel flow problem, $Re=100$.

In this work, we study the performance of the proposed LSA-PINN for learning
multiple complex fluid dynamical systems, using both MLP and CNN
architectures. We further demonstrate that LSA-PINN is broadly applicable to
different ND-type training losses, including ND-type (BCXN-ND) and CAN-type
(BCXN-CAN) schemes.

The LSA-PINN (BCXN-ND-, BCXN-CAN-loss) is compared with baseline PINN (ND-,
CAN-, AD-loss) models, based on mean squared error (MSE) of the respective
PINN solutions’ velocity vector $\vec{u}$ relative to ground truth. To
facilitate comparison, identical network architectures, initialization
distribution, and training settings are employed. Relevant settings are
summarized in Table VI and Table VII in Appendix A-A. For each experiment, we
perform and present results based on 10 independent runs with different
initialization. The ground truth solutions are obtained by an in-house
numerical solver based on the improved divergence-free condition compensated
(IDFC) method [57]. The 3 fluid dynamics test cases are:

  * •

2D semi-circle lid-driven cavity flow, $Re=1000$. The fluid flow inside the
semi-circle cavity is driven by a lid velocity $u_{lid}=1,v_{lid}=0$ at the
top wall. No-slip condition $(u=v=0)$ is applied to the other wall. There is a
primary eddy near the cavity’s center.

  * •

2D lid-driven cavity flow, $Re=1000$. The fluid flow inside a $1\times 1$ unit
square cavity is driven by a lid velocity $u_{lid}=1,v_{lid}=0$ at the top
wall. No-slip condition is applied to the other walls. The lid-driven cavity
flow has been widely chosen as a benchmark case for many numerical methods due
to the complex physics encapsulated within. At $Re=1000$, there is a primary
eddy near the center of the cavity, with secondary and tertiary eddies at the
bottom-right and bottom-left regions.

  * •

2D wavy channel flow, $Re=100$. The fluid flow passing through a long wavy
channel is studied. The inlet profile at left boundary is defined as
$u(0,y)=-\frac{3}{2}y^{2}+\frac{3}{2},v=0$. A non-slip condition is applied to
the top and bottom walls, while outlet boundary conditions $(\frac{\partial
u}{\partial x}=\frac{\partial v}{\partial x}=p=0)$ are applied to the right
boundary.

###  V-A MLP-architecture LSA-PINN

In the MLP-architecture experiment, we evaluate the LSA-PINN’s performance and
compare them to other AD-, CAN-, and ND-loss baseline methods previously
published for the MLP architecture. All models were trained on a relatively
sparse set of training sample (4,230, 10,400, and 15,200 points for semi-
circle lid-driven cavity, lid-driven cavity, and wavy channel flow problems,
respectively) for 300,000 iterations using Adam algorithm. The average MSE and
relative L2 error across multiple independent runs are summarized in Table I.

TABLE I: Model Performance Comparisons for MLP-architecture PINNs

Problem Model MSE Rel. Error Semi-circle lid-driven cavity flow, $Re=1000$
BCXN-CAN $6.21\times 10^{-5}$ $3.55\times 10^{-2}$ BCXN-ND $1.61\times
10^{-4}$ $5.95\times 10^{-2}$ CAN $2.82\times 10^{-2}$ $7.62\times 10^{-1}$ ND
$3.91\times 10^{-2}$ $8.56\times 10^{-1}$ AD (SIREN) $1.74\times 10^{-2}$
$5.95\times 10^{-1}$ AD (Vanilla) $5.46\times 10^{-2}$ $9.83\times 10^{-1}$
Lid-driven cavity flow, $Re=1000$ BCXN-CAN $5.10\times 10^{-5}$ $3.38\times
10^{-2}$ CAN $4.23\times 10^{-2}$ $9.77\times 10^{-1}$ AD (SIREN) $3.47\times
10^{-2}$ $8.87\times 10^{-1}$ Wavy channel flow, $Re=100$ BCXN-CAN $1.34\times
10^{-4}$ $7.28\times 10^{-2}$ CAN $5.96\times 10^{-1}$ $2.55\times 10^{0}$ AD
(SIREN) $3.52\times 10^{-2}$ $7.19\times 10^{-1}$

  1. (I)

Implementation for CAN-PINN models based on previous work [6].

  2. (II)

Implementation for SIREN-PINN models based on previous work [58, 59, 60].

  3. (III)

Implementation for vanilla PINN models based on previous work [1].

The results indicate that our present LSA-PINN method can successfully learn a
good solution with low MSE across a diverse set of problems with irregular
domains and complex physics. All 3 test cases show noticeable improvement in
solution accuracy ($>$2 orders of magnitude lower $\vec{u}$ MSE) with the LSA-
PINN method. The baseline PINNs fail to produce a reasonable solution, i.e.,
they cannot properly learn the correct flow with current sampling density and
training iteration, see example in Fig. 3.

Case study of information propagation. Information propagation from the
boundary to the inner domain has been suggested to be critical for successful
PINN modelling [31, 16]. Hence, we investigate the evolution of the PDE
residual across multiple training runs for the semi-circle lid-driven cavity
with and without BCXN-loss to better understand the impact of enhanced
connectivity and local structure approximation imposed by the BCXN-loss.
Illustrative contour plots of the PDE residuals and velocity at various
training iterations are provided in Fig. 4.

Figure 4: Contour plots of PDE error residual and velocity for (a) LSA-PINN
(BCXN-ND) and (b) PINN (ND), for the semi-circle lid-driven cavity flow
problem, $Re=1000$.

As described in the previous sections, the ND methods typically fail to
converge to a good model even after 300,000 training iterations but this can
be improved via the addition of BCXN-loss during training. The velocity
contours clearly illustrate the improved convergence. In addition, there is a
striking difference in the distribution of PDE residuals across the domain
during the early stages of training. In a typical ND-PINN, the PDE residuals
rapidly reduce to zero, especially in the parts of the domain far away from
the boundary condition. This also corresponds to the ND-loss PINN velocity
profile converging to a wrong solution, and is indicative of the information
from the boundary condition being lost within the domain. On the other hand,
the PDE residuals within the domain persist for a much longer period when the
BCXN-loss is included, which is indicative of information from the top
boundary being persistently transmitted to the inner domain until the LSA-PINN
model converges. This is also consistent with the physical dynamics of the
system, as we note that the lid-driven cavity flow, as the name suggests, is
heavily dependent on the top boundary’s motion to drive the dynamical flow
within the entire domain.

Case study of samples with different boundary connectivity. The lid-driven
cavity test case has a regular domain, but we chose Cell-centered (whereby
near-boundary points need support points outside the domain) sample locations
to show that the proposed method can be useful for this kind of scenarios. To
further illustrate the impact of having near-boundary stencil points outside
of the domain, we also investigate the impact of Cell-centered sampling points
against Node-centered (whereby near-boundary points use support points that
are coincident with the domain boundary). These two sets of sampling points
are tested on a lid-driven cavity flow at $Re=1000$ with different batch sizes
and point sampling density for the BCXN-CAN, CAN and AD models, and are
trained for $300,000$ iterations each. The MSE and L2 relative errors are
collated across multiple independent runs and tabulated in Table II. A
qualitatively good solution for this problem corresponds to an MSE of $1\times
10^{-3}$ and below.

TABLE II: Model Performance for Different Sampling Densities. [C] and [N]
refer to cell- and node-centered sample arrangements.

Batch Size Points Model $50\times 50$ $100\times 100$ MSE Rel. Error MSE Rel.
Error $500$ [C] BCXN-CAN $2.63\times 10^{-4}$ $7.75\times 10^{-2}$ $3.06\times
10^{-3}$ $2.63\times 10^{-1}$ CAN & AD $>1\times 10^{-2}$ $>5\times 10^{-1}$
$>1\times 10^{-2}$ $>5\times 10^{-1}$ [N] CAN $>1\times 10^{-2}$ $>5\times
10^{-1}$ $>1\times 10^{-2}$ $>5\times 10^{-1}$ AD $>1\times 10^{-2}$ $>5\times
10^{-1}$ $>1\times 10^{-2}$ $>5\times 10^{-1}$ $1000$ [C] BCXN-CAN $2.23\times
10^{-4}$ $7.13\times 10^{-2}$ $3.97\times 10^{-4}$ $9.47\times 10^{-2}$ CAN &
AD $>1\times 10^{-2}$ $>5\times 10^{-1}$ $>1\times 10^{-2}$ $>5\times 10^{-1}$
[N] CAN $>1\times 10^{-2}$ $>5\times 10^{-1}$ $>1\times 10^{-2}$ $>5\times
10^{-1}$ AD $>1\times 10^{-2}$ $>5\times 10^{-1}$ $>1\times 10^{-2}$ $>5\times
10^{-1}$ $2000$ [C] BCXN-CAN $2.36\times 10^{-4}$ $7.34\times 10^{-2}$
$7.14\times 10^{-5}$ $4.02\times 10^{-2}$ CAN & AD $>1\times 10^{-2}$
$>5\times 10^{-1}$ $>1\times 10^{-2}$ $>5\times 10^{-1}$ [N] CAN $7.41\times
10^{-4}$ $1.32\times 10^{-1}$ $>1\times 10^{-2}$ $>5\times 10^{-1}$ AD
$>1\times 10^{-2}$ $>5\times 10^{-1}$ $>1\times 10^{-2}$ $>5\times 10^{-1}$
$4000$ [C] BCXN-CAN $2.38\times 10^{-5}$ $2.31\times 10^{-2}$ CAN & AD
$>1\times 10^{-2}$ $>5\times 10^{-1}$ [N] CAN $2.94\times 10^{-4}$ $8.24\times
10^{-2}$ AD $>1\times 10^{-2}$ $>5\times 10^{-1}$

From this experiment, we notice that the Node-centered CAN-loss PINN model has
improved performance relative to the Cell-centered version, for a larger batch
size and training samples. This further emphasizes our hypothesis that
accurate modelling of the boundary conditions is an important factor in
obtaining good performance. As this is a square domain, the use of a sampling
point distribution that has perfect connectivity with the boundary of the
domain naturally leads to improved performance.

In addition, the inclusion of our proposed BCXN-loss on a Cell-centered LSA-
PINN can improve model performance to achieve a lower error than the properly
aligned baseline PINN models, especially for sparse training samples setting.
This suggests that the use of LSA-PINN can afford flexibility in the choice of
sampling strategies, whereas conventional sampling strategies may require
sampling that maintains connectivity with the boundaries.

Case study with different sampling density. From the above experiment, we
notice that a 4x increase in sampling density (from $50\times 50$ to
$100\times 100$) can improve model performance. Hence, we further study the
lid-driven cavity flow problem to understand the trade-off between convergence
in accuracy and sampling density. We train on a denser (i.e., $200\times 200$)
set of samples until the accuracy of the CAN and AD models start to improve.
Note that denser sampling requires larger training iteration and batch size,
and the tuning of these training hyper-parameters can be very time consuming.
Although all PINN models can eventually achieve improved accuracy with denser
sampling, Fig. 5 indicates that it is much harder and slower to learn from a
dense set of samples, thereby demonstrating the benefits of LSA-PINN.
Encouragingly, LSA-PINN learns more accurate solutions ($>$1 order of
magnitude lower $\vec{u}$ MSE) than the baseline CAN- and AD-loss PINNs with
less ($<$50%) training iterations, while using fewer ($<$50%) training samples
for the lid-driven cavity flow problem.

Figure 5: PINNs learning the solution of lid-driven cavity flow under (a)
sparse and (b) dense training scenarios respectively. The convergence trends
of $\vec{u}$ MSE are plotted. Bold lines indicate the median convergence path,
and the shaded areas indicate the inter-percentile range from
$10^{th}-90^{th}$ percentiles.

###  V-B CNN-architecture LSA-PINNs

The BCXN-loss method does not pose any requirement on the differentiable
property of the networks, hence it can be freely implemented in any neural
network architecture, including CNN. Hence, we compare the LSA-PINN (BCXN-ND)
to baseline PINN (ND) method previously published for the CNN architecture and
summarize their average MSE and relative L2 error across multiple independent
runs in Table III. The U-Net architecture and training settings used are
summarized in Table VII. All PINN models for this section are trained on a
single discretized 2D input which consists of 3 channels, namely the x-,
y-coordinate, and an indicator to differentiate fluid and non-fluid domain.
They were trained for 200,000 iterations using Adam algorithm on a relatively
sparse set of training samples, i.e., the semi-circle and square lid-driven
cavity flow problems use 3,930 and 10,000 interior points which belong to the
fluid domain respectively.

TABLE III: Model Performance Comparisons for CNN-architecture PINNs

Problem Model MSE Rel. Error Semi-circle lid-driven cavity flow, $Re=1000$
BCXN-ND $1.14\times 10^{-4}$ $4.92\times 10^{-2}$ ND $3.96\times 10^{-2}$
$8.55\times 10^{-1}$ Lid-driven cavity flow, $Re=1000$ BCXN-ND $2.52\times
10^{-5}$ $4.46\times 10^{-2}$ ND $2.38\times 10^{-2}$ $1.00\times 10^{0}$
Transient semi-circle lid-driven cavity flow, $Re=500$ BCXN-ND $3.95\times
10^{-4}$ $1.29\times 10^{-1}$ ND $9.66\times 10^{-3}$ $6.81\times 10^{-1}$

Like the MLP experiment, LSA-PINN for CNNs show a noticeable improvement in
the solution accuracy (2-3 orders of magnitude lower $\vec{u}$ MSE) over the
baseline ND-loss PINN. The results indicate that LSA-PINN can effectively
learn the solution of fluid dynamic problems with U-Net architecture and that
the BCXN-loss can benefit CNN-architecture PINNs by remedying training issues
stemming from a discretized input being unable to exactly resolve complex
geometries.

Transient 2D semi-circle lid-driven cavity flow, $Re=500$. In addition to the
steady-state semi-circle lid-driven cavity flow, we further evaluate the
improvement obtained from application of CNN-architecture LSA-PINN to the
modelling of a transient version of this problem. In this instance, the
problem’s initial condition corresponds to that of a fluid domain at rest (at
t = 0). The PINN model is then used to model the evolution of the flow within
a semi-circle lid-driven cavity from $t=0$ to $t=2$, $\Delta{t}=0.1$. Model
settings are provided in Table VII. The transient PINN models have additional
$t$ channel in their input.

The performance of LSA-PINN (BCXN-ND) and baseline PINN (ND) are compared in
Table III. The average errors for all time points between $t=0$ and $t=2.0$
yield MSEs of 3.95e-4 and 9.66e-3 with and without the inclusion of BCXN-loss
respectively. Hence, the inclusion of BCXN-loss in this transient problem
improves MSE by $>1$ order of magnitude. Velocity contour plots of the BCXN-ND
and ND models are provided in Fig. 6 to illustrate the differences in
convergence.

Figure 6: Velocity contours of the transient simulation at $t=0.5$, $t=1.0$
and $t=2.0$ for the ground truth and PINN solutions.

Performance with increasing problem complexity. To better understand the
improvement in accuracy obtainable by the BCXN-loss, we also investigate the
utility of LSA-PINN for modelling more complex problems more efficiently.
Hence, we train both LSA-PINN and baseline PINN for the semi-circle lid-driven
cavity flow across a range of $Re$. As $Re$ increases, the complexity of the
developed flow typically requires more sophisticated algorithms and training
samples for better capture of the non-linear effects.

The average MSE and relative L2 error obtained across multiple runs for
various $Re$ are presented in Table IV. We clearly see that the performance
difference is increasingly pronounced for more complicated (higher $Re$)
problems, thus demonstrating that strategies like LSA-PINN methods are
increasingly important for more complex problems.

TABLE IV: Model Performance for the 2D Semi-circular Lid-Driven Cavity Problem

Metric Model Reynolds Number (Re) $50$ $100$ $200$ $400$ $1000$ MSE BCXN-ND
$2.51\times 10^{-5}$ $2.76\times 10^{-5}$ $3.56\times 10^{-5}$ $4.58\times
10^{-5}$ $1.14\times 10^{-4}$ ND $2.98\times 10^{-4}$ $2.79\times 10^{-4}$
$3.74\times 10^{-4}$ $2.33\times 10^{-3}$ $3.96\times 10^{-2}$ Rel. Error
BCXN-ND $0.0258$ $0.0269$ $0.0305$ $0.0335$ $0.0492$ ND $0.0823$ $0.0732$
$0.0780$ $0.211$ $0.855$

Meta-PINN model. The CNN-architecture PINN models represent the entire spatial
domain as a single input. We can train the model on multiple samples, each
representing a different scenario such as changes in geometry or physical
behaviour. Such a meta-PINN model can be used to predict new scenarios. Hence,
we further study the ability of LSA-PINN to learn the solutions to multiple
flow scenarios concurrently (spanning $Re=50-1000$) on semi-circle lid-driven
cavity geometry. The model architecture and training setting is given in
Appendix A-A Table VII. The LSA-PINN (BCXN-ND) model is used to predict
$Re=300,500,600,700,900$ scenarios which are not used during training. The
results are summarized in Fig. 7. The present LSA-PINN method successfully
learns a good solution with low $\vec{u}$ MSE (i.e., $<$1e-4) for most of the
$Re$ scenarios, and achieves good prediction accuracy.

Figure 7: LSA-PINN learning the solutions of multiple 2D semi-circle lid-
driven cavity flow scenarios concurrently, using CNN architecture. The
$\vec{u}$ MSE between ground truth and LSA-PINN solution for train and test
$Re$ are plotted. Inset are $u$\- and $v$-velocity contours of the ground
truth and LSA-PINN solutions for $Re=600$ and $Re=900$.

###  V-C Impact of BCXN-loss on training time

We further compare the time taken to train a model with and without the BCXN-
loss on the MLP- and CNN-type models as applied to the semi-circle lid-driven
cavity problem at $Re=1000$. The average MSE and relative L2 error obtained
are averaged across 10 independent runs, and the minimum and maximum wall-
clock time are recorded. All experiments are run with standard Tensorflow and
Keras implementation on a 20-core workstation with an Intel Xeon Gold 6248
processor and an Nvidia RTX 2080 Ti GPU.

Quantitative comparisons for the MLP-architecture and CNN-architecture models
are presented in Table V. For fair comparison within architecture types, all
MLP models were run for an identical number of training iterations (300,000)
with the same batch size (1,000) and model size while all CNN models were
similarly run for an identical number of training iterations (200,000) and
with the same model size.

TABLE V: Model Performance for different PINN architectures Model | Method | MSE | Rel. Error | Time (min)
---|---|---|---|---
MLP | BCXN-CAN | $6.21\times 10^{-5}$ | $3.55\times 10^{-2}$ | $55-57$
MLP | BCXN-ND | $1.61\times 10^{-4}$ | $5.95\times 10^{-2}$ | $23-24$
MLP | CAN | $2.82\times 10^{-2}$ | $7.62\times 10^{-1}$ | $40-46$
MLP | ND | $3.91\times 10^{-2}$ | $8.56\times 10^{-1}$ | $20-22$
MLP | AD (SIREN) | $1.74\times 10^{-2}$ | $5.95\times 10^{-1}$ | $58-61$
MLP |  AD (Vanilla) | $5.46\times 10^{-2}$ | $9.83\times 10^{-1}$ | $50-55$
CNN | BCXN | $1.14\times 10^{-4}$ | $4.92\times 10^{-2}$ | $126-136$
CNN | ND | $3.96\times 10^{-2}$ | $8.55\times 10^{-1}$ | $121-137$

From this study, we note that the inclusion of BCXN-loss across all the cases
only slightly increases the amount of time required for model training. This
increase was on the order of less than 5 % for the CNN-type architecture and
the ND-MLP-type model. Critically, while the time taken may have increased
slightly, the MSE typically decreases more than 1 order of magnitude. In many
instances, this trade-off between model training time and performance with the
inclusion of a BCXN-loss is probably preferable.

##  VI Conclusion

We present a LSA-PINN method which can more efficiently learn the solution to
PDEs for complex geometry with sparse training samples. This is accomplished
by enforcing a linear constraint during the training implicitly. While linear
constraints are applied in this work due to truncation of the Taylor series
expansion in the derivation, it is worth studying if different, higher order
approximations can achieve better performance in future work. Overall, our
comprehensive experimental studies demonstrate practical advantages of LSA-
PINN on diverse test problems in fluid dynamics, improving the accuracy of
solutions by orders of magnitude even while requiring much less training
iterations and samples.

## References

  * [1] M. Raissi, P. Perdikaris, and G. E. Karniadakis, “Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations,” _Journal of Computational physics_ , vol. 378, pp. 686–707, 2019.
  * [2] G. E. Karniadakis, I. G. Kevrekidis, L. Lu, P. Perdikaris, S. Wang, and L. Yang, “Physics-informed machine learning,” _Nature Reviews Physics_ , vol. 3, no. 6, pp. 422–440, 2021.
  * [3] S. Cuomo, V. S. Di Cola, F. Giampaolo, G. Rozza, M. Raissi, and F. Piccialli, “Scientific machine learning through physics-informed neural networks: Where we are and what’s next,” _arXiv preprint arXiv:2201.05624_ , 2022.
  * [4] H. Gao, L. Sun, and J.-X. Wang, “Phygeonet: Physics-informed geometry-adaptive convolutional neural networks for solving parameterized steady-state pdes on irregular domain,” _Journal of Computational Physics_ , vol. 428, p. 110079, 2021.
  * [5] Z. Fang, “A high-efficient hybrid physics-informed neural networks based on convolutional neural network,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2021.
  * [6] P.-H. Chiu, J. C. Wong, C. Ooi, M. H. Dao, and Y.-S. Ong, “Can-pinn: A fast physics-informed neural network based on coupled-automatic–numerical differentiation method,” _Computer Methods in Applied Mechanics and Engineering_ , vol. 395, p. 114909, 2022.
  * [7] Y. Lu, H. Chen, J. Lu, L. Ying, and J. Blanchet, “Machine learning for elliptic pdes: fast rate generalization bound, neural scaling law and minimax optimality,” _arXiv preprint arXiv:2110.06897_ , 2021.
  * [8] S. Mishra and R. Molinaro, “Estimates on the generalization error of physics-informed neural networks for approximating a class of inverse problems for pdes,” _IMA Journal of Numerical Analysis_ , vol. 42, no. 2, pp. 981–1022, 2022.
  * [9] S. Markidis, “The old and the new: Can physics-informed deep-learning replace traditional linear solvers?” _Frontiers in big Data_ , p. 92, 2021.
  * [10] C. Anitescu, E. Atroshchenko, N. Alajlan, and T. Rabczuk, “Artificial neural network methods for the solution of second order boundary value problems,” _Computers, Materials and Continua_ , vol. 59, no. 1, pp. 345–359, 2019.
  * [11] L. McClenny and U. Braga-Neto, “Self-adaptive physics-informed neural networks using a soft attention mechanism,” _arXiv preprint arXiv:2009.04544_ , 2020\.
  * [12] C. L. Wight and J. Zhao, “Solving allen-cahn and cahn-hilliard equations using the adaptive physics informed neural networks,” _arXiv preprint arXiv:2007.04542_ , 2020.
  * [13] M. A. Nabian, R. J. Gladstone, and H. Meidani, “Efficient training of physics-informed neural networks via importance sampling,” _Computer-Aided Civil and Infrastructure Engineering_ , vol. 36, no. 8, pp. 962–977, 2021.
  * [14] L. Lu, X. Meng, Z. Mao, and G. E. Karniadakis, “Deepxde: A deep learning library for solving differential equations,” _SIAM Review_ , vol. 63, no. 1, pp. 208–228, 2021.
  * [15] K. O. Lye, S. Mishra, D. Ray, and P. Chandrashekar, “Iterative surrogate model optimization (ismo): an active learning algorithm for pde constrained optimization with deep neural networks,” _Computer Methods in Applied Mechanics and Engineering_ , vol. 374, p. 113575, 2021.
  * [16] A. Daw, J. Bu, S. Wang, P. Perdikaris, and A. Karpatne, “Rethinking the importance of sampling in physics-informed neural networks,” _arXiv preprint arXiv:2207.02338_ , 2022.
  * [17] R. Mattey and S. Ghosh, “A novel sequential method to train physics informed neural networks for allen cahn and cahn hilliard equations,” _Computer Methods in Applied Mechanics and Engineering_ , vol. 390, p. 114474, 2022.
  * [18] C. Wu, M. Zhu, Q. Tan, Y. Kartha, and L. Lu, “A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks,” _Computer Methods in Applied Mechanics and Engineering_ , vol. 403, p. 115671, 2023.
  * [19] A. D. Jagtap, E. Kharazmi, and G. E. Karniadakis, “Conservative physics-informed neural networks on discrete domains for conservation laws: Applications to forward and inverse problems,” _Computer Methods in Applied Mechanics and Engineering_ , vol. 365, p. 113028, 2020.
  * [20] A. D. Jagtap and G. E. Karniadakis, “Extended physics-informed neural networks (xpinns): A generalized space-time domain decomposition based deep learning framework for nonlinear partial differential equations.” in _AAAI Spring Symposium: MLPS_ , 2021.
  * [21] K. Shukla, A. D. Jagtap, and G. E. Karniadakis, “Parallel physics-informed neural networks via domain decomposition,” _Journal of Computational Physics_ , vol. 447, p. 110683, 2021.
  * [22] S. Dong and Z. Li, “Local extreme learning machines and domain decomposition for solving linear and nonlinear partial differential equations,” _Computer Methods in Applied Mechanics and Engineering_ , vol. 387, p. 114129, 2021.
  * [23] K. Li, K. Tang, T. Wu, and Q. Liao, “D3m: A deep domain decomposition method for partial differential equations,” _IEEE Access_ , vol. 8, pp. 5283–5294, 2019.
  * [24] N. Wandel, M. Weinmann, and R. Klein, “Learning incompressible fluid dynamics from scratch–towards fast, differentiable fluid models that generalize,” _arXiv preprint arXiv:2006.08762_ , 2020.
  * [25] ——, “Teaching the incompressible navier–stokes equations to fast neural surrogate models in three dimensions,” _Physics of Fluids_ , vol. 33, no. 4, p. 047117, 2021.
  * [26] R. Ranade, C. Hill, and J. Pathak, “Discretizationnet: A machine-learning based solver for navier–stokes equations using finite volume discretization,” _Computer Methods in Applied Mechanics and Engineering_ , vol. 378, p. 113722, 2021.
  * [27] N. Wandel, M. Weinmann, M. Neidlin, and R. Klein, “Spline-pinn: Approaching pdes without data using fast, physics-informed hermite-spline cnns,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 36, no. 8, 2022, pp. 8529–8538.
  * [28] P. Ren, C. Rao, Y. Liu, J.-X. Wang, and H. Sun, “Phycrnet: Physics-informed convolutional-recurrent network for solving spatiotemporal pdes,” _Computer Methods in Applied Mechanics and Engineering_ , vol. 389, p. 114399, 2022.
  * [29] S. K. Mitusch, S. W. Funke, and M. Kuchta, “Hybrid fem-nn models: Combining artificial neural networks with the finite element method,” _Journal of Computational Physics_ , vol. 446, p. 110651, 2021.
  * [30] Y. Shin, J. Darbon, and G. E. Karniadakis, “On the convergence of physics informed neural networks for linear second-order elliptic and parabolic type pdes,” _arXiv preprint arXiv:2004.01806_ , 2020.
  * [31] S. Wang, S. Sankaran, and P. Perdikaris, “Respecting causality is all you need for training physics-informed neural networks,” _arXiv preprint arXiv:2203.07404_ , 2022.
  * [32] M. Elhamod, J. Bu, C. Singh, M. Redell, A. Ghosh, V. Podolskiy, W.-C. Lee, and A. Karpatne, “Cophy-pgnn: Learning physics-guided neural networks with competing loss functions for solving eigenvalue problems,” _ACM Transactions on Intelligent Systems and Technology (TIST)_ , 2020.
  * [33] R. Bischof and M. Kraus, “Multi-objective loss balancing for physics-informed deep learning,” _arXiv preprint arXiv:2110.09813_ , 2021.
  * [34] X. Jin, S. Cai, H. Li, and G. E. Karniadakis, “Nsfnets (navier-stokes flow nets): Physics-informed neural networks for the incompressible navier-stokes equations,” _Journal of Computational Physics_ , vol. 426, p. 109951, 2021\.
  * [35] P. Thanasutives, M. Numao, and K.-i. Fukui, “Adversarial multi-task learning enhanced physics-informed neural networks for solving partial differential equations,” in _2021 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2021, pp. 1–9.
  * [36] S. Maddu, D. Sturm, C. L. Müller, and I. F. Sbalzarini, “Inverse dirichlet weighting enables reliable training of physics informed neural networks,” _Machine Learning: Science and Technology_ , vol. 3, no. 1, p. 015026, 2022\.
  * [37] T. de Wolff, H. C. Lincopi, L. Martí, and N. Sanchez-Pi, “Mopinns: an evolutionary multi-objective approach to physics-informed neural networks,” in _Proceedings of the Genetic and Evolutionary Computation Conference Companion_ , 2022, pp. 228–231.
  * [38] R. van der Meer, C. W. Oosterlee, and A. Borovykh, “Optimally weighted loss functions for solving pdes with neural networks,” _Journal of Computational and Applied Mathematics_ , vol. 405, p. 113887, 2022.
  * [39] Z. Xiang, W. Peng, X. Liu, and W. Yao, “Self-adaptive loss balanced physics-informed neural networks,” _Neurocomputing_ , vol. 496, pp. 11–34, 2022.
  * [40] X. Huang, H. Liu, B. Shi, Z. Wang, K. Yang, Y. Li, M. Wang, H. Chu, J. Zhou, F. Yu _et al._ , “A universal pinns method for solving partial differential equations with a point source.”
  * [41] S. Wang, Y. Teng, and P. Perdikaris, “Understanding and mitigating gradient flow pathologies in physics-informed neural networks,” _SIAM Journal on Scientific Computing_ , vol. 43, no. 5, pp. A3055–A3081, 2021.
  * [42] I. E. Lagaris, A. Likas, and D. I. Fotiadis, “Artificial neural networks for solving ordinary and partial differential equations,” _IEEE transactions on neural networks_ , vol. 9, no. 5, pp. 987–1000, 1998.
  * [43] I. E. Lagaris, A. C. Likas, and D. G. Papageorgiou, “Neural-network methods for boundary value problems with irregular boundaries,” _IEEE Transactions on Neural Networks_ , vol. 11, no. 5, pp. 1041–1049, 2000.
  * [44] K. S. McFall and J. R. Mahan, “Artificial neural network method for solution of boundary value problems with exact satisfaction of arbitrary boundary conditions,” _IEEE Transactions on Neural Networks_ , vol. 20, no. 8, pp. 1221–1233, 2009.
  * [45] J. Berg and K. Nyström, “A unified deep artificial neural network approach to partial differential equations in complex geometries,” _Neurocomputing_ , vol. 317, pp. 28–41, 2018.
  * [46] M. A. Nabian and H. Meidani, “A deep learning solution approach for high-dimensional random differential equations,” _Probabilistic Engineering Mechanics_ , vol. 57, pp. 14–25, 2019.
  * [47] S. Karumuri, R. Tripathy, I. Bilionis, and J. Panchal, “Simulator-free solution of high-dimensional stochastic elliptic partial differential equations using deep neural networks,” _Journal of Computational Physics_ , vol. 404, p. 109120, 2020.
  * [48] Z. Wang and Z. Zhang, “A mesh-free method for interface problems using the deep learning approach,” _Journal of Computational Physics_ , vol. 400, p. 108963, 2020.
  * [49] L. Lu, R. Pestourie, W. Yao, Z. Wang, F. Verdugo, and S. G. Johnson, “Physics-informed neural networks with hard constraints for inverse design,” _SIAM Journal on Scientific Computing_ , vol. 43, no. 6, pp. B1105–B1132, 2021.
  * [50] O. Fuks and H. A. Tchelepi, “Limitations of physics informed machine learning for nonlinear two-phase transport in porous media,” _Journal of Machine Learning for Modeling and Computing_ , vol. 1, no. 1, 2020.
  * [51] C. Michoski, M. Milosavljević, T. Oliver, and D. R. Hatch, “Solving differential equations using deep neural networks,” _Neurocomputing_ , vol. 399, pp. 193–212, 2020.
  * [52] A. A. Ramabathiran and P. Ramachandran, “Spinn: Sparse, physics-based, and partially interpretable neural networks for pdes,” _Journal of Computational Physics_ , vol. 445, p. 110600, 2021.
  * [53] W. Ji, W. Qiu, Z. Shi, S. Pan, and S. Deng, “Stiff-pinn: Physics-informed neural network for stiff chemical kinetics,” _The Journal of Physical Chemistry A_ , vol. 125, no. 36, pp. 8098–8106, 2021.
  * [54] D. Lucor, A. Agrawal, and A. Sergent, “Physics-aware deep neural networks for surrogate modeling of turbulent natural convection,” _arXiv preprint arXiv:2103.03565_ , 2021.
  * [55] S. Osher and N. Paragios, _Geometric level set methods in imaging, vision, and graphics_. Springer Science & Business Media, 2003.
  * [56] P.-H. Chiu and H. J. Poh, “Development of an improved divergence-free-condition compensated coupled framework to solve flow problems with time-varying geometries,” _International Journal for Numerical Methods in Fluids_ , vol. 93, no. 1, pp. 44–70, 2021.
  * [57] P.-H. Chiu, “An improved divergence-free-condition compensated method for solving incompressible flows on collocated grids,” _Computers & Fluids_, vol. 162, pp. 39–54, 2018.
  * [58] V. Sitzmann, J. Martel, A. Bergman, D. Lindell, and G. Wetzstein, “Implicit neural representations with periodic activation functions,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 7462–7473, 2020.
  * [59] S. Wang, H. Wang, and P. Perdikaris, “On the eigenvector bias of fourier feature networks: From regression to solving multi-scale pdes with physics-informed neural networks,” _Computer Methods in Applied Mechanics and Engineering_ , vol. 384, p. 113938, 2021.
  * [60] J. C. Wong, C. Ooi, A. Gupta, and Y.-S. Ong, “Learning in sinusoidal spaces with physics-informed neural networks,” _IEEE Transactions on Artificial Intelligence_ , 2022.

##  Appendix A Appendix

###  A-A MLP and CNN experimental settings

TABLE VI: PINN (MLP) architecture and training settings used for experiments
in Section V-A

Problem type Forward Forward Forward Test case 2D semi-circle lid-driven
cavity flow, $Re=1000$ 2D lid-driven cavity flow, $Re=1000$ 2D wavy channel
flow, $Re=100$ Governing physics eqns. 2D incompressible NS eqns. 2D
incompressible NS eqns. 2D incompressible NS eqns. MLP architecture for LSA-
PINN, ND-PINN $\&$ AD-PINN $(x,y)-64–30–30–30–$ $[30–30–30–(\hat{u})$,
$30–30–30–(\hat{v})$, $30–30–30–(\hat{p})]$ $(x,y)-128–30–30–30–$
$[30–30–30–(\hat{u})$, $30–30–30–(\hat{v})$, $30–30–30–(\hat{p})]$
$(x,y)-128–40–40–40–$ $[40–40–40–(\hat{u})$, $40–40–40–(\hat{v})$,
$40–40–40–(\hat{p})]$ ND scheme for LSA-PINN $\&$ ND-PINN CAN & finite
difference CAN CAN Training sample (equidistantly spaced samples @ fluid
domain $\&$ boundary) $3,930+300$ $10,000+400$ $14,400+800$ Batch size $1,000$
$2,000$ $1,000$ Training iterations $300,000$ $300,000$ $300,000$

  1. (I)

For the MLP architecture, the numbers in between input and output represent
the number of nodes in each hidden layer. For example,
$(x)–64–20–20–20–(\hat{u})$ indicates a single input $x$, followed by 4 hidden
layers with 64, 20, 20 and 20 nodes in each layer, and a single output
$\hat{u}$.

  2. (II)

We incorporate the sinusoidal mapping [60] into the first hidden layer of PINN
and initialize its weights by sampling from a normal distribution
$N(0,\sigma^{2}),\sigma=1$. The subsequent hidden layers use “sine”
activation, except a “linear” activation function is used in the final
(output) layer, and their weights are initialized by He uniform distribution.

  3. (III)

Batch size: number of random sample used for 1 evaluation of
$L_{PINN}(=\lambda_{PDE}~{}L_{PDE}+\lambda_{BC}~{}L_{BC})$ and $L_{PINN(df-
BCXN)}(=\lambda_{PDE}~{}L_{PDE(df-BCXN)})$. We used a default
$\lambda_{PDE}=1$ and $\lambda_{BC}=1$ unless otherwise mentioned.

  4. (IV)

A training iteration: 1 evaluation of $L_{PINN}$ or $L_{PINN(df-BCXN)}$ for
backpropagating the weight gradients. We use an initial learning of 1e-3 and
reduce it on plateauing, until a min. learning rate of 5e-6 is reached.

TABLE VII: PINN (CNN) architecture and training settings used in the
experiments in Section V-B

Problem type Forward Forward Forward Meta-modelling Test case 2D semi-circle
lid-driven cavity flow, $Re=1000$ 2D lid-driven cavity flow, $Re=1000$ 2D
transient semi-circular lid-driven cavity flow, $Re=500$ 2D semi-circle lid-
driven cavity flow,$Re=1000$ Governing physics eqns. 2D incompressible NS
eqns. 2D incompressible NS eqns. 2D transient incompressible NS eqns. 2D
incompressible NS eqns. CNN (U-Net) architecture for LSA-PINN $\&$ ND-PINN
$(\textit{{X}}_{(56\times 112\times 3)})–$,
$8–8–\downarrow–16–16–\downarrow–$, $32–32–\uparrow–64–64–\uparrow–$,
$32–32–\uparrow–16–16–\uparrow–[$, $8–8–8–(\hat{\textit{{U}}}_{u,56\times
112})$, $8–8–8–(\hat{\textit{{U}}}_{v,56\times 112})$,
$8–8–8–(\hat{\textit{{U}}}_{p,56\times 112})$ $(\textit{{X}}_{(104\times
104\times 3)})–$, $8–8–\downarrow–16–16–\downarrow–$,
$32–32–\uparrow–64–64–\uparrow–$, $32–32–\uparrow–16–16–\uparrow–[$,
$8–8–8–(\hat{\textit{{U}}}_{u,104\times 104})$,
$8–8–8–(\hat{\textit{{U}}}_{v,104\times 104})$,
$8–8–8–(\hat{\textit{{U}}}_{p,104\times 104})$ $(\textit{{X}}_{(56\times
112\times 4)})–$, $8–8–\downarrow–16–16–\downarrow–$,
$32–32–\uparrow–64–64–\uparrow–$, $32–32–\uparrow–16–16–\uparrow–[$,
$8–8–8–(\hat{\textit{{U}}}_{u,56\times 112})$,
$8–8–8–(\hat{\textit{{U}}}_{v,56\times 112})$,
$8–8–8–(\hat{\textit{{U}}}_{p,56\times 112})$ $(\textit{{X}}_{(56\times
112\times 4)})–$, $8–8–\downarrow–16–16–\downarrow–$,
$32–32–\uparrow–64–64–\uparrow–$, $32–32–\uparrow–16–16–\uparrow–[$,
$8–8–8–(\hat{\textit{{U}}}_{u,56\times 112})$,
$8–8–8–(\hat{\textit{{U}}}_{v,56\times 112})$,
$8–8–8–(\hat{\textit{{U}}}_{p,56\times 112})$ Kernel size $4\times 4$ $6\times
6$ $4\times 4$ $6\times 6$ ND scheme for LSA-PINN $\&$ ND-PINN Finite
difference Finite difference Finite difference Finite difference Training
sample  / batch size 1 / 1 1 / 1 21 / 2 6 / 1 Training iterations $200,000$
$200,000$ $100,000$ $500,000$

  1. (I)

For the U-net architecture, the numbers in between input and output represent
the number of filters in each hidden layer. Each hidden layer consists of
convolution operation, batch normalization, and nonlinear “sine” activation.
Also, $\downarrow$ represents a $2\times 2$ max-pooling operation, and
$\uparrow$ represents a $2\times 2$ up-sampling operation. The network weights
are initialized by He uniform distribution.

  2. (II)

Batch size: number of sample used for 1 evaluation of
$L_{PINN}(=\lambda_{PDE}~{}L_{PDE}+\lambda_{BC}~{}L_{BC})$ and $L_{PINN(df-
BCXN)}(=\lambda_{PDE}~{}L_{PDE(df-BCXN)})$. We used a default
$\lambda_{PDE}=1$ and $\lambda_{BC}=1$, unless otherwise mentioned.

  3. (III)

A training iteration: 1 evaluation of $L_{PINN}$ or $L_{PINN(df-BCXN)}$ for
backpropagating the weight gradients. We use an initial learning of 1e-3 and
reduce it on plateauing, until a min. learning rate of 5e-6 is reached.

  4. (IV)

The semi-circle and regular lid-driven cavity geometries consist of 3,930 and
10,000 interior points which belong to the fluid domain respectively.