# INTERLAYER SYNCHRONIZATION IN TIME-VARYING KURAMOTO-SAKAGUCHI NETWORKS

Muhittin Cenk Eser  Department of Physics, Eastern Mediterranean University,
99628 Famagusta, North Cyprus,via Mersin 10, Turkey  Mustafa Riza  Department
of Physics, Eastern Mediterranean University, 99628 Famagusta, North
Cyprus,via Mersin 10, Turkey

###### Abstract

We investigate the abrupt transition from low interlayer synchrony to high
interlayer synchrony in a system of two identical layers of non-locally
coupled Kuramoto-Sakaguchi oscillators using time-switching of the interlayer
topology, while keeping the intralayer topology constant. As a measure of
interlayer synchronization we introduce the interlayer order parameter, and
determine the interlayer order parameter first in the static case as a
function of the number of interlayer links observing a monotonically smooth
transition to the fully synchronized state in the case that all nodes in layer
1 are connected to their corresponding nodes in layer 2. In the second step,
we analyze the synchronization behavior of the multiplex network for a time-
varying interlayer topology, observing abrupt transitions from the state of
low synchrony to the state of high synchrony for a significantly lower number
of interlayer connections. These abrupt transitions, even if not to full
synchrony, appear only because of the periodic change of the interlayer
topology, fostering the transition to occur for a significantly lower number
of interlayer connections. We interpret this phenomenon as the shrinking, and
ultimately, the nearly disappearance of the basin of attraction of a
desynchronized network state. A comprehensive statistical analysis of this
phenomenon is needed to interpret the results properly.

††preprint: AIP/123-QED

One of the first encouragements for analyzing networks composed of Kuramoto-
Sakaguchi oscillators originates from neuroscience. The synaptic coupling can
vary depending on the action on the neurons changing the coupling between the
neurons Gerstner et al. (1996); Clopath et al. (2010). Furthermore, the
analysis of the impact of varying coupling ranges on the spiking behavior and
its timings in neural networks was carried out in Markram et al. (1997);
Abbott and Nelson (2000); Caporale and Dan (2008); Popovych et al. (2013);
Lücken et al. (2016). Apart from neuroscience, many models of robust networks
are studied in chemical Jain and Krishna (2001), biological, and social
networks Gross and Blasius (2008), and many different systems. This study
analyzes the effect of a time-varying interlayer coupling on the
synchronization of a two-layer network composed of two weakly coupled rings of
identical Kuramoto-SakaguchiSakaguchi and Kuramoto (1986); Pikovsky and
Rosenblum (2015) phase oscillators. We limit our investigation to one
particular coupling composition, a non-locally coupled ring of Kuramoto-
Sakaguchi oscillators resulting the two layers to be in the chimera state.
Then we couple the two layers one-to-one weakly in order not to disturb the
intralayer dynamics, statically as well as time-varyingly.

Synchronization is an extraordinary characteristic of the dynamics of the
networks of connected oscillators Pikovsky et al. (2001); Strogatz (2001);
Albert and Barabási (2002); Boccaletti et al. (2018). Different
synchronization regions are known, like a group of synchronization where the
network is divided into two clusters of synchronous components Dahms et al.
(2012), or partial synchronization dynamics, which is chimera patterns where
all networks are divided into coexisting regions of synchronized and
desynchronized patterns Kuramoto and Battogtokh (2002); Abrams and Strogatz
(2004); Panaggio and Abrams (2015). Chimera patterns were also investigated in
various topological networks Aoki and Aoyagi (2009); Kasatkin and Nekorkin
(2016); Berner et al. (2019a).

Those properties are studied in a variety of networks experimentally as well
as theoretically Pasemann (1995); Bressloff et al. (1997); Yanchuk and Wolfrum
(2008); Bonnin (2009); Omelchenko et al. (2013). Especially, the important
properties of a network of neurons have been investigated in Compte et al.
(2003); Sporns (2013); Popovych et al. (2011). Lately, related models have
been considered for an all-to-all coupling topology, called the global
coupling topology, Ren and Zhao (2007); Aoki and Aoyagi (2009); Avalos-Gaytán
et al. (2018); Kasatkin et al. (2017); Berner et al. (2019a, b). Adaptive
Kuramoto-Sakaguchi type networks show distinct complex topological responses
on globally coupled networks. Critical, stable multi-frequency groups appear
when the oscillators are divided into strongly coupled oscillators with a
similar average frequency. This characteristics does not appear in the
classical Kuramoto or Kuramoto-Sakaguchi phase oscillator dynamics.
Furthermore, the groups demonstrate how to control a hierarchical composition;
for instance, their dimensions are substantially different Kasatkin et al.
(2017); Berner et al. (2019b). This development is also described for adaptive
neural networks of Morris-Lecar, Hindmarsh-Rose, and Hodgkin-Huxley models
with either spike-timing-dependent plasticity rule Popovych et al. (2015);
Chakravartula et al. (2017); Röhr et al. (2019). In networks modeling the
brain, hierarchy and modularity have recently been introduced Bassett et al.
(2008); Meunier et al. (2010); Bassett et al. (2010).

Commonly, the connecting structure of real-world systems is time-dependent,
e.g., the brain neuronal plasticity, network of scientific collaborations, or
mutation processes in evolutionary biology. To incorporate this feature into a
description, many theoretical studies on synchronization have considered the
effects of networks with time-varying topologies. Studies show that such
settings have massive impact on the transition to synchronization Gómez-
Gardenes et al. (2011); Leyva et al. (2012); Zhang et al. (2015); Boccaletti
et al. (2016). In fact, the synchronizability of networks is enhanced for
different implementations of the topological time dependencies, i.e., the
transitions to synchronization generally occur for lower values of the
coupling parameters. The example are provided by, for instance, evolution
along commutative graphs Boccaletti et al. (2006), adaptation of weights Zhou
and Kurths (2006), on-off coupling Chen et al. (2009), different rewiring
frequencies Kohar et al. (2014), and random connections Zhou et al. (2019).
More recently, and of particular interest for this study, the enhancement of
synchronizability due to time-varying topology has also been shown in networks
in which the interacting structure is subdivided into different layers, i.e.,
multilayer networks. For instance, in a network of Hindmarsh-Rose neurons in
which the coupling structure of both electrical and chemical synapses is time-
varying, it has been found that the coupling strength required for the
achievement of complete synchrony is significantly lower for increasing the
rewiring frequency Rakshit et al. (2018). Moreover, the enhancement effect has
been also observed in a network with multiple layers in which only the
structure of specific layers is time-varying Rakshit et al. (2019).
Furthermore, it has been shown that diverse partial synchronization patterns
can be generated by multiplexing of adaptive networks Berner et al. (2020).
Recently, it has been found that noisy modulation of inter-layer coupling
strength, called multiplexing noise, induces the inter-layer synchronization
of spatio–temporal patterns Vadivasova et al. (2020). Despite such a
significant understanding of the role played by time-dependent topologies in
facilitating synchronization, some important issues are still await
unresolved. In particular, the influence of the topological time-dependencies
in the characteristics of the transitions to synchronization is mostly
unknown.

In section I we review shortly the Kuramoto-Sakaguchi Network model, outlining
the differences between the original Kuramoto model and the extension given by
Kuramoto and Sakaguchi. In the following section II we define the network of
two layers composed of identical Kuramoto-Sakaguchi oscillators in the chimera
regime. Section III reviews first the global and local order parameters
introduced by Kuramoto as a measure to find the regions of coherence and
decoherence in a single layer network. If the system reaches full synchrony,
the local order parameters for both layers coincide perfectly. As a second
measure of interlayer synchronization, we introduced the interlayer order
parameter indicating numerically the level synchrony between the two layers.
In section LABEL:sec:results we analyze and discuss the behavior of the
Kuramoto-Sakaguchi network, first in absence of interlayer links, showing that
both layers are in the chimera state, then by increasing the number of
interlayer links reaching full synchrony only in the case when all nodes in
layer 1 are connected to their corresponding nodes in layer 2. Finally, we
employ the idea of a time-varying interlayer topology, where the interlayer
links are removed after a certain time period and the same number of
interlayer links is instantaneously randomly connected again. The analysis of
the interlayer order parameter as a function of the number of interlayer links
shows sharp transitions from low synchrony to high synchrony. The high
synchrony case, i.e. not having reached full synchrony, shows that the regions
of coherence and incoherence coincide, but while the local order parameters
for the coherent regions are identical equal to $1$, the local order
parameters for the incoherent regions do not match perfectly. We finally close
with the conclusion of the results.

##  I The Kuramoto-Sakaguchi Network model

To get a deeper understanding on the interlayer synchronization phenomenon, we
decided to extend the explosive synchronization model to a network consisting
of two weakly coupled, identical Kuramoto Sakaguchi layers. Again, we employ
non-local coupling to ensure the establishment of chimera states. In contrast
to the FHN oscillator, non-local coupling is not sufficient to create Chimera
states. Therefore, the so-called Kuramoto-Sakaguchi oscillator Sakaguchi and
Kuramoto (1986) is used. Kuramoto discussed network of phase oscillators in
its simplest form in Kuramoto (1984). For a single layer, the Kuramoto system
can be written as

$\dot{\theta}_{i}(t)=\omega_{i}+\frac{K}{N}\sum_{j}\sin\left(\theta_{i}-\theta_{j}\right).$
(1)

For this model, it is not possible to generate chimera states with just
employing non-local coupling. The extension of the Kuramoto model by the so-
called phase-lag parameter $\alpha$ opened the door to observe chimera states
even for phase oscillators like the Kuramoto oscillator.

$\dot{\theta}_{i}(t)=\omega_{i}+\frac{K}{N}\sum_{j}\sin\left(\theta_{i}-\theta_{j}+\alpha\right).$
(2)

Clearly, the Kuramoto model (1) is the special case of the Kuramoto-Sakaguchi
model (2) for $\alpha=0$. Kuramoto and Battogtokh showed the existence of
coherent and incoherent domains for the Kuramoto-Sakaguchi model in Kuramoto
and Battogtokh (2002), without naming it Chimera state. The terminology
’Chimera state’ was created by Abrams and Strogatz in Abrams and Strogatz
(2004). Later, Panaggio and Abrams gave a summary of the phenomenon of Chimera
states in Panaggio and Abrams (2015).

##  II The Kuramoto-Sakaguchi Model in double layer multiplex networks

We consider a network constructed of two identical layers of non-locally
coupled Kuramoto-Sakaguchi oscillators. The interlayer coupling is weak and
one-to-one, i.e. the node $i$ in layer 1 is, if connected, only connected to
its corresponding node $i$ in layer 2. These kind of networks as
aforementioned are called multiplex networks. The interlayer coupling is
selected as weak in order not to influence the intralayer dynamics of the the
individual layers.

The consider the following system.

$\displaystyle\dot{\theta}_{i}^{(1)}(t)$ $\displaystyle=\omega_{i}^{(1)}+$
$\displaystyle\sigma_{1}\sum_{j=i-R_{1}}^{i+R_{1}}\sin\left(\theta_{i}^{(1)}-\theta_{j}^{(1)}+\alpha^{(1)}\right)+$
(3)
$\displaystyle+\sigma_{12}\sin\left(\theta_{i}^{(1)}-\theta_{i}^{(2)}\right)$
$\displaystyle\dot{\theta}_{i}^{(2)}(t)$ $\displaystyle=\omega_{i}^{(2)}+$
$\displaystyle\sigma_{2}\sum_{j=i-R_{2}}^{i+R_{2}}\sin\left(\theta_{i}^{(2)}-\theta_{j}^{(2)}+\alpha^{(2)}\right)+$
$\displaystyle+\sigma_{21}\sin\left(\theta_{i}^{(2)}-\theta_{i}^{(1)}\right),$

where $\omega_{i}^{(j)}$ is called the natural frequency of the node $i$ in
layer $j$.

##  III Order parameter for phase oscillators

###  III.1 Global and Local Order parameter

As a measure of the simultaneous coexistence of coherence and incoherence in a
network, namely the Chimera states, Kuramoto and Battogtokh introduce global
order parameter $Z(t)$ in Kuramoto and Battogtokh (2002).

$Z(t)=\frac{1}{N}\left|\sum_{k=1}^{N}e^{i\theta_{k}(t)}\right|,$ (4)

where $\theta_{k}(t)$ is the phase of the $k^{th}$-oscillator at the instant
$t$. There are two limiting cases for the global order parameter $Z(t)$. If
$Z(t)=1$ the network is coherent, whereas if $Z(t)=0$, the network is
incoherent. So, if $0<Z(t)<1$ the network has domains of coherence and
decoherence, consequently the global order parameter can be used to identify
the existence of Chimera states.

Alongside with the global order parameter, the local order parameter
$Z_{k}(t)$ can be used as a measure to determine the regions of coherence and
decoherence in the network.

$Z_{k}(t)=\frac{1}{2R+1}\left|\sum_{i=k-R}^{k+R}e^{i\theta_{k}(t)}\right|,$
(5)

where $R\leq N/2$.

###  III.2 Interlayer synchronization order parameter

Synchronization in multilayer networks has only been studied based on the
intralayer local order parameter $Z_{k}(t)$ as defined in (5). This intralayer
order parameter helps us to identified the regions of coherence and
decoherence. On the other hand,a measure of interlayer synchronization has not
been introduced yet in the literature. Therefore, to fill this gap we
introduce the time-averaged interlayer order parameter as following

$Z=\frac{1}{N}\left|\left\langle\sum_{i=1}^{N}e^{i\left|\theta_{i}^{(1)}(t)-\theta_{i}^{(2)}(t)\right|}\right\rangle_{t}\right|=\\\
=\frac{1}{NT}\left|\int_{0}^{T}\sum_{i=1}^{N}e^{i\left|\theta_{i}^{(1)}(t)-\theta_{i}^{(2)}(t)\right|}dt\right|.$
(6)

Using this definition we will determine the interlayer synchronization
behavior of the two layer multiplex system of identical Kuramoto-Sakaguchi
oscillators as given in equation (3). If the phase difference between node $i$
in layer 1 and the corresponding node $i$ in layer 2 vanishes,
$e^{i\left|\theta_{i}^{(1)}-\theta_{i}^{(2)}\right|}=1$, so if all nodes are
synchronized, i.e. their phase difference of the corresponding nodes vanishes,
the time averaged interlayer order parameter as defined in equation (6)
becomes $Z=1$.

##  IV Results and Discussion

As aforementioned, the two layers are set to be identical. Therefore, the
intralayer coupling strengths $\sigma_{1}$ and $\sigma_{2}$ of the links in
layer 1 and layer 2, respectively, have to be equal, i.e.
$\sigma_{1}=\sigma_{2}$. Moreover, the coupling ranges $r_{1}=R_{1}/N$ and
$r_{2}=R_{2}/N$ together with the coupling radii $R_{1}$ and $R_{2}$ have to
be equal to ensure the the two layers to be identical. As the mean frequency
is conserved in the Kuramoto model, and has no effect other than a constant
phase shift, for simplicity we will select them uniformly as
$\omega_{i}^{(j)}=0$. Additionally, the phase-lag parameter $\alpha^{(1)}$ in
layer 1 and layer 2 $\alpha^{(2)}$, i.e.
$\alpha^{(1)}=\alpha^{(2)}=\pi/2-0.1=1.47$. Finally, layer 1 and layer 2 will
be coupled symmetrically and weakly to each other, i.e.
$\sigma_{12}=\sigma_{21}\ll\sigma_{1}=\sigma_{2}$.

Let us first summarize all parameters used in the following table:

Parameter | Description of the Parameter
---|---
$N=300$ | Number of nodes per layer
$\omega_{i}^{(1)}=\omega_{i}^{(2)}=0$ | natural frequency
$\sigma_{1}=\sigma_{2}=0.1$ | intralayer coupling strength
$r_{1}=r_{2}=0.35$ | intralayer coupling range
$R_{1}=r_{1}\cdot N=105$ | intralayer coupling radius for layer 1
$R_{2}=r_{2}\cdot N=105$ | intralayer coupling radius for layer 2
$\sigma_{12}=\sigma_{21}=0.01$ | interlayer coupling strength
$\alpha^{(1)}=\alpha^{(2)}=1.47$ | phase-lag parameter
$N_{12}$ | Number of interlayer links
Table 1: List of all parameters used.

Proceeding to the analysis of the time-varying interlayer topology of the
double layer multiplex Kuramoto-Sakaguchi network, we will use the parameters
stated in table 1, following the procedure outlined in Cenk Eser et al. (2021)
for the FHN oscillator. First, we show using the spacetime plots and the local
order parameters that both layers are in the chimera state for random initial
conditions in the case that there is no interlayer connection.

Figure 1: For the case of no interlayer connection, i.e. $N_{12}=0$, the
spacetime plots for layer 1 (a) and layer 2 (b), as well as the local order
parameters for layer 1 (c) and layer 2 (d) have been depicted.

For random initial conditions and no interlayer link, figure 1 shows that for
the selected parameters stated in table 1 layer 1 and layer 2 are in the
Chimera state. The spacetime plots in figure 1 (a) for layer 1 and figure 1
(b) for layer 2 together with the local order parameter at $t=3000$ in figure
1 (c) and (d) show that both layers are in the the chimera regime and that the
size of the domains of coherence and decoherence are the same. So, we can
state once again, that the size of the domains of coherence and decoherence
are independent of the initial conditions.

In order to check if the proposed network (3) reaches full synchrony, we
connect all nodes in layer 1 to their corresponding nodes in layer 2, i.e. the
number of interlayer links $N_{12}=N-300$.

Figure 2: $N_{12}=300$

Figure 2 shows the spacetime plots for (a) layer 1 and (b) layer 2, as well as
the local order parameter for (c) layer 1 and (d) layer 2 in the case
$N_{12}=N=300$. Evidently, the regions of coherence and decoherence in both
layers now overlap if all nodes in layer 1 are connected to its corresponding
node in layer 2. The reason for the selection of a weak coupling strengths was
to ensure that the interlayer coupling does not impact the intralayer dynamics
of the individual layers notably. As one can see clearly in figure 2 the
chimera patterns have not been influenced, i.e. they still appear, and the
size of the domains of coherence and decoherence has not changed, or if only
insignificantly. It is clear that the domains of coherence and decoherence
appear now at the same nodes in the corresponding layers. In figure 3 (b) we
show for the case of full synchrony, the combination of figure 2 (c) and (d)
clearly indicating the overlap of the domains in both layers. In the next
step, we increase gradually the number of interlayer links and determine the
interlayer order parameters as a function of the number of interlayer links
$N_{12}$.

Figure 3: (a) Interlayer synchronization order parameter as a function of the
number of interlayer links $N_{12}$ with not switching. (b) local order
parameter for the case $N_{12}=N=300$

In figure 3 (a) we can see that the interlayer order parameter increases
monotonically with increasing number of interlayer links $N_{12}$, reaching
full synchrony, i.e. Z=1, only for $N_{12}=N=300$. So, we can conclude, full
synchronization can only be reached if all nodes in layer 1 are connected to
the corresponding node in layer 2. In this case, as aforementioned the regions
of coherence and decoherence coincide illustrated using the local order
parameter $Z_{k}(t)$ exemplarily for $t=3000$ as illustrated in figure 3 (b).

Finally, following the process we already used in Cenk Eser et al. (2021), we
want to investigate if it possible to reach synchrony for a significantly
fewer number of interlayer links than $N_{12}=N=300$. Analogously to the
analysis of the network of the FHN oscillators, for a given number of
interlayer links $N_{12}$ we start now removing the randomly distributed links
after a given time, called the switching time $T_{S}$, and immediately
connecting the same number of interlayer links randomly again. Statistically,
the distribution of the connections after each time interval $T_{S}$ is most
likely different to the distribution of connections before. So, we determine
the interlayer order parameter for different switching times as a function of
the number of interlayer links $N_{12}$.

Figure 4: Interlayer synchronization order parameter as a function of the
number of interlayer links $N_{12}$ for switching times (a) $T_{S}=5$,(b)
$T_{S}=35$,(c) $T_{S}=45$,(d) $T_{S}=55$,(e) $T_{S}=65$,and (f) $T_{S}=95$.

Figure 4 depicts the interlayer order parameter as a function of the number of
interlayer links $N_{12}$ for $T_{S}=5$ in (a), $T_{S}=35$ in (b), $T_{S}=45$
in (c), $T_{S}=55$ in (d),$T_{S}=65$ in (e), and $T_{S}=95$ in (f). We can
observe, that for every switching time we can observe abrupt changes in the
interlayer order parameter. Unfortunately, we could not find a combination of
switching time $T_{S}$ and number of interlayer links where the system reaches
full synchrony between the two layers. Interestingly, although we could not
pin down the situation where the system makes a transition from the
desynchronized to the fully synchronized state yet, the system shows for the
every switching time an abrupt change in the interlayer oder parameter,
leading to partial synchronization of the coherent regions of the two layers.
The most significant transition between low interlayer order parameter
($Z\approx 0.3$) to high interlayer order parameter ($Z\approx 0.7$) could be
observed for $T_{S}=5$, indicating that the two layers are widely
synchronized. Let us have a closer look at the case of switching time
$T_{S}=5$ and $N_{12}=70$, where we could observe the biggest jump in the
interlayer order parameter. Therefore, we will first show the spacetime plots
with their corresponding local order parameters. In order to show, the effect
we will combine the interlayer order parameters for both layers in one graph.

Figure 5: For switching time $T_{S}=5$ and $N_{12}=70$ space time plot for
layer 1 (a) and layer 2 (b) and local order parameter for layer 1 (c) and
layer 2 (d) Figure 6: For switching time $T_{S}=5$ and $N_{12}=70$ local order
parameter for layer 1 and layer 2 2

In figure 5 we can observe, that the regions of coherence and decoherence in
layer 1 and layer 2 are at the same positions, indicating that the weak
interlayer coupling made the regions of coherence and decoherence in layer 1
and layer 2 coincide. A closer look at the local order parameter, by plotting
the local order parameters for layer 1 and layer 2 in figure 6 shows, that the
local order parameters for the coherent domains coincide, but the local order
parameters for the incoherent domains do not overlap as perfect. But,
nevertheless we can interpret this as a transition to a interlayer synchrony.
Furthermore, it seems that the combination of switching and the change of the
number of interlayer links impacts the domains of decoherence. A more careful
statistical analysis has to be carried out to show how switching impacts the
intralayer behavior of the individual layers.

We see again that the basin of attraction of the desynchronized state shrinks,
but we could not yet find the condition where the basin of attraction of the
desynchronized state vanishes. This is just an intermediate result of this
phenomenon. As we have clear indicators that full synchrony exists for the
case $N_{12}=N$ switched or not switched, and we can observe certain abrupt
transitions, further research will reveal the results for an abrupt transition
from the desynchronized state to the fully synchronized state, where the basin
of attraction of the desynchronized state vanishes, has to be carried out.

As already discussed above, although we could not yet observe the transition
from a desynchronized state to full synchrony, nevertheless we can observe the
intriguing phenomenon of abrupt transitions from states of lower synchrony to
higher synchrony with the implementation of switching. We also illustrated for
the case of the biggest jump in the interlayer order parameter, that the
regions of coherence and decoherence overlap, while the incoherence seems to
behave different in the different layers. We also can state, that the
difference in the order parameter at the transitions seems to be depending on
the switching time. Also two abrupt changes of the order parameter could be
observed. With shorter switching times, the average time one node is connected
to its corresponding node is higher. A comprehensive statistical analysis of
this phenomenon is needed to interpret the results properly.

##  V Conclusions

In analogy to the study of the FHN oscillator we extended our study to the
Kuramoto-Sakaguchi oscillator to investigate if such a collective transition
also appears for phase oscillators. As a measure of synchronization we
introduced the interlayer order parameter as counter part to the
synchronization error in the FHN case. The parameters for the Kuramoto-
Sakaguchi network were selected in such a way that the individual layers show
Chimera patterns under non-local intralayer coupling. Non-local intralayer
coupling is not sufficient to impose Chimera behavior, therefore the phase-lag
parameter had to be selected analogously to the coupling angle in the cross-
coupling matrix of the FHN system as $\alpha=\pi/2-0.1$. With weak interlayer
coupling strength, we studied the interlayer synchronization behavior of the
Kuramoto-Sakaguchi first for static interlayer topology with varying number of
interlayer links, observing a smooth transition from the desynchronized state
to the fully synchronized state with increasing number of interlayer links.
Full synchrony can only be reached in case of all nodes in layer 1 being
connected to the corresponding node in layer 2. Then, the interlayer
connection topology is changed for a fixed number of interlayer links at given
time-intervals, namely after each time-interval $T_{S}$ all $N_{12}$ links are
removed and connected randomly at different positions. Reconnection for
individual links is not prohibited in two succeeding time-intervals. We
observe for different switching times $T_{S}$ abrupt changes in the interlayer
order parameter for varying number of interlayer links $N_{12}$. These sharp
changes in the interlayer order parameter as a function of the number of
interlayer links indicates, that under certain conditions the domains of
coherence and incoherence overlap for both networks. The local order
parameters in this case show that the coherent parts of the layers coincide,
whereas the incoherent parts do not match perfectly resulting in a interlayer
order parameter $Z<1$. This phenomenon has been shown explicitly for the case
$T_{S}=5$ and $N=70$. The state of full synchrony, could not be established
yet for a number of interlayer conditions $N_{12}<N$ using time-varying
interlayer connection switching. Further statistical analyses have to be
conducted to finalize this work. The partial synchronization of the interlayer
networks for the coherent domains indicates a collective behavior of the nodes
in the coherent domain, whereas the incoherent domains seem not to synchronize
perfectly. Further studies are needed to explain this behavior. Of course,
also in this case we can observe that switching leads to a shrinking basin of
attraction of the desynchronized states without vanishing up to our current
knowledge.

###### Acknowledgements.

## Data Availability

The data that support the findings of this study are available from the
authors upon reasonable request.

## References

  * Gerstner et al. (1996) W. Gerstner, R. Kempter, J. L. Van Hemmen, and H. Wagner, “A neuronal learning rule for sub-millisecond temporal coding,” _Nature_ , vol. 383, no. 6595, pp. 76–78, 1996.
  * Clopath et al. (2010) C. Clopath, L. Büsing, E. Vasilaki, and W. Gerstner, “Connectivity reflects coding: a model of voltage-based stdp with homeostasis,” _Nature neuroscience_ , vol. 13, no. 3, pp. 344–352, 2010.
  * Markram et al. (1997) H. Markram, J. Lübke, M. Frotscher, and B. Sakmann, “Regulation of synaptic efficacy by coincidence of postsynaptic aps and epsps,” _Science_ , vol. 275, no. 5297, pp. 213–215, 1997.
  * Abbott and Nelson (2000) L. F. Abbott and S. B. Nelson, “Synaptic plasticity: taming the beast,” _Nature neuroscience_ , vol. 3, no. 11, pp. 1178–1183, 2000.
  * Caporale and Dan (2008) N. Caporale and Y. Dan, “Spike timing–dependent plasticity: a hebbian learning rule,” _Annu. Rev. Neurosci._ , vol. 31, pp. 25–46, 2008.
  * Popovych et al. (2013) O. V. Popovych, S. Yanchuk, and P. A. Tass, “Self-organized noise resistance of oscillatory neural networks with spike timing-dependent plasticity,” _Scientific reports_ , vol. 3, no. 1, pp. 1–6, 2013.
  * Lücken et al. (2016) L. Lücken, O. V. Popovych, P. A. Tass, and S. Yanchuk, “Noise-enhanced coupling between two oscillators with long-term plasticity,” _Physical Review E_ , vol. 93, no. 3, p. 032210, 2016.
  * Jain and Krishna (2001) S. Jain and S. Krishna, “A model for the emergence of cooperation, interdependence, and structure in evolving networks,” _Proceedings of the National Academy of Sciences_ , vol. 98, no. 2, pp. 543–547, 2001.
  * Gross and Blasius (2008) T. Gross and B. Blasius, “Adaptive coevolutionary networks: a review,” _Journal of the Royal Society Interface_ , vol. 5, no. 20, pp. 259–271, 2008\.
  * Sakaguchi and Kuramoto (1986) H. Sakaguchi and Y. Kuramoto, “A soluble active rotater model showing phase transitions via mutual entertainment,” _Progress of Theoretical Physics_ , vol. 76, no. 3, pp. 576–581, 1986.
  * Pikovsky and Rosenblum (2015) A. Pikovsky and M. Rosenblum, “Dynamics of globally coupled oscillators: Progress and perspectives,” _Chaos: An Interdisciplinary Journal of Nonlinear Science_ , vol. 25, no. 9, p. 097616, 2015.
  * Pikovsky et al. (2001) A. Pikovsky, M. Rosenblum, and J. Kurths, _Synchronization: A Universal Concept in Nonlinear Sciences_ , ser. Cambridge Nonlinear Science Series. Cambridge University Press, 2001\.
  * Strogatz (2001) S. H. Strogatz, “Exploring complex networks,” _nature_ , vol. 410, no. 6825, pp. 268–276, 2001.
  * Albert and Barabási (2002) R. Albert and A.-L. Barabási, “Statistical mechanics of complex networks,” _Reviews of modern physics_ , vol. 74, no. 1, p. 47, 2002.
  * Boccaletti et al. (2018) S. Boccaletti, A. N. Pisarchik, C. I. Del Genio, and A. Amann, _Synchronization: from coupled systems to complex networks_. Cambridge University Press, 2018.
  * Dahms et al. (2012) T. Dahms, J. Lehnert, and E. Schöll, “Cluster and group synchronization in delay-coupled networks,” _Physical Review E_ , vol. 86, no. 1, p. 016202, 2012.
  * Kuramoto and Battogtokh (2002) Y. Kuramoto and D. Battogtokh, “Coexistence of coherence and incoherence in nonlocally coupled phase oscillators,” _Nonlinear Phenomena in Complex Systems_ , vol. 5, pp. 380–385, 2002.
  * Abrams and Strogatz (2004) D. M. Abrams and S. H. Strogatz, “Chimera states for coupled oscillators,” _Physical review letters_ , vol. 93, no. 17, p. 174102, 2004.
  * Panaggio and Abrams (2015) M. J. Panaggio and D. M. Abrams, “Chimera states: coexistence of coherence and incoherence in networks of coupled oscillators,” _Nonlinearity_ , vol. 28, no. 3, p. R67, 2015.
  * Aoki and Aoyagi (2009) T. Aoki and T. Aoyagi, “Co-evolution of phases and connection strengths in a network of phase oscillators,” _Physical review letters_ , vol. 102, no. 3, p. 034101, 2009.
  * Kasatkin and Nekorkin (2016) D. Kasatkin and V. Nekorkin, “Dynamics of the phase oscillators with plastic couplings,” _Radiophysics and Quantum Electronics_ , vol. 58, no. 11, pp. 877–891, 2016.
  * Berner et al. (2019a) R. Berner, J. Fialkowski, D. Kasatkin, V. Nekorkin, S. Yanchuk, and E. Schöll, “Hierarchical frequency clusters in adaptive networks of phase oscillators,” _Chaos: An Interdisciplinary Journal of Nonlinear Science_ , vol. 29, no. 10, p. 103134, 2019.
  * Pasemann (1995) F. Pasemann, “Characterization of periodic attractors in neural ring networks,” _Neural Networks_ , vol. 8, no. 3, pp. 421–429, 1995.
  * Bressloff et al. (1997) P. Bressloff, S. Coombes, and B. De Souza, “Dynamics of a ring of pulse-coupled oscillators: Group-theoretic approach,” _Physical review letters_ , vol. 79, no. 15, p. 2791, 1997.
  * Yanchuk and Wolfrum (2008) S. Yanchuk and M. Wolfrum, “Destabilization patterns in chains of coupled oscillators,” _Physical Review E_ , vol. 77, no. 2, p. 026212, 2008.
  * Bonnin (2009) M. Bonnin, “Waves and patterns in ring lattices with delays,” _Physica D: Nonlinear Phenomena_ , vol. 238, no. 1, pp. 77–87, 2009.
  * Omelchenko et al. (2013) I. Omelchenko, E. Omel’chenko, P. Hövel, and E. Schöll, “When nonlocal coupling between oscillators becomes stronger: patched synchrony or multichimera states,” _Physical review letters_ , vol. 110, no. 22, p. 224101, 2013.
  * Compte et al. (2003) A. Compte, M. V. Sanchez-Vives, D. A. McCormick, and X.-J. Wang, “Cellular and network mechanisms of slow oscillatory activity (< 1 hz) and wave propagations in a cortical network model,” _Journal of neurophysiology_ , vol. 89, no. 5, pp. 2707–2725, 2003.
  * Sporns (2013) O. Sporns, “Network attributes for segregation and integration in the human brain,” _Current opinion in neurobiology_ , vol. 23, no. 2, pp. 162–171, 2013.
  * Popovych et al. (2011) O. V. Popovych, S. Yanchuk, and P. A. Tass, “Delay-and coupling-induced firing patterns in oscillatory neural loops,” _Physical review letters_ , vol. 107, no. 22, p. 228102, 2011.
  * Ren and Zhao (2007) Q. Ren and J. Zhao, “Adaptive coupling and enhanced synchronization in coupled phase oscillators,” _Physical Review E_ , vol. 76, no. 1, p. 016207, 2007\.
  * Avalos-Gaytán et al. (2018) V. Avalos-Gaytán, J. A. Almendral, I. Leyva, F. Battiston, V. Nicosia, V. Latora, and S. Boccaletti, “Emergent explosive synchronization in adaptive complex networks,” _Physical Review E_ , vol. 97, no. 4, p. 042301, 2018.
  * Kasatkin et al. (2017) D. Kasatkin, S. Yanchuk, E. Schöll, and V. Nekorkin, “Self-organized emergence of multilayer structure and chimera states in dynamical networks with adaptive couplings,” _Physical Review E_ , vol. 96, no. 6, p. 062211, 2017.
  * Berner et al. (2019b) R. Berner, E. Scholl, and S. Yanchuk, “Multiclusters in networks of adaptively coupled phase oscillators,” _SIAM Journal on Applied Dynamical Systems_ , vol. 18, no. 4, pp. 2227–2266, 2019.
  * Popovych et al. (2015) O. V. Popovych, M. N. Xenakis, and P. A. Tass, “The spacing principle for unlearning abnormal neuronal synchrony,” _PloS one_ , vol. 10, no. 2, p. e0117205, 2015.
  * Chakravartula et al. (2017) S. Chakravartula, P. Indic, B. Sundaram, and T. Killingback, “Emergence of local synchronization in neuronal networks with adaptive couplings,” _PloS one_ , vol. 12, no. 6, p. e0178975, 2017.
  * Röhr et al. (2019) V. Röhr, R. Berner, E. L. Lameu, O. V. Popovych, and S. Yanchuk, “Hierarchical clusters in neuronal populations with plasticity,” _arXiv preprint arXiv:1908.04103_ , 2019.
  * Bassett et al. (2008) D. S. Bassett, E. Bullmore, B. A. Verchinski, V. S. Mattay, D. R. Weinberger, and A. Meyer-Lindenberg, “Hierarchical organization of human cortical networks in health and schizophrenia,” _Journal of Neuroscience_ , vol. 28, no. 37, pp. 9239–9248, 2008.
  * Meunier et al. (2010) D. Meunier, R. Lambiotte, and E. T. Bullmore, “Modular and hierarchically modular organization of brain networks,” _Frontiers in neuroscience_ , vol. 4, p. 200, 2010.
  * Bassett et al. (2010) D. S. Bassett, D. L. Greenfield, A. Meyer-Lindenberg, D. R. Weinberger, S. W. Moore, and E. T. Bullmore, “Efficient physical embedding of topologically complex information processing networks in brains and computer circuits,” _PLoS computational biology_ , vol. 6, no. 4, p. e1000748, 2010.
  * Gómez-Gardenes et al. (2011) J. Gómez-Gardenes, S. Gómez, A. Arenas, and Y. Moreno, “Explosive synchronization transitions in scale-free networks,” _Phys. Rev. Lett._ , vol. 106, no. 12, p. 128701, 2011.
  * Leyva et al. (2012) I. Leyva, R. Sevilla-Escoboza, J. Buldú, I. Sendina-Nadal, J. Gómez-Gardeñes, A. Arenas, Y. Moreno, S. Gómez, R. Jaimes-Reátegui, and S. Boccaletti, “Explosive first-order transition to synchrony in networked chaotic oscillators,” _Phys. Rev. Lett._ , vol. 108, no. 16, p. 168702, 2012.
  * Zhang et al. (2015) X. Zhang, S. Boccaletti, S. Guan, and Z. Liu, “Explosive synchronization in adaptive and multilayer networks,” _Phys. Rev. Lett._ , vol. 114, no. 3, p. 038701, 2015.
  * Boccaletti et al. (2016) S. Boccaletti, J. Almendral, S. Guan, I. Leyva, Z. Liu, I. Sendiña-Nadal, Z. Wang, and Y. Zou, “Explosive transitions in complex networks’ structure and dynamics: Percolation and synchronization,” _Physics Reports_ , vol. 660, pp. 1–94, 2016.
  * Boccaletti et al. (2006) S. Boccaletti, D.-U. Hwang, M. Chavez, A. Amann, J. Kurths, and L. M. Pecora, “Synchronization in dynamical networks: Evolution along commutative graphs,” _Phys. Rev. E_ , vol. 74, no. 1, p. 016102, 2006.
  * Zhou and Kurths (2006) C. Zhou and J. Kurths, “Dynamical weights and enhanced synchronization in adaptive complex networks,” _Phys. Rev. Lett._ , vol. 96, no. 16, p. 164102, 2006.
  * Chen et al. (2009) L. Chen, C. Qiu, and H. Huang, “Synchronization with on-off coupling: Role of time scales in network dynamics,” _Phys. Rev. E_ , vol. 79, no. 4, p. 045101, 2009.
  * Kohar et al. (2014) V. Kohar, P. Ji, A. Choudhary, S. Sinha, and J. Kurths, “Synchronization in time-varying networks,” _Phys. Rev. E_ , vol. 90, no. 2, p. 022812, 2014\.
  * Zhou et al. (2019) S. Zhou, Y. Guo, M. Liu, Y.-C. Lai, and W. Lin, “Random temporal connections promote network synchronization,” _Phys. Rev. E_ , vol. 100, no. 3, p. 032302, 2019.
  * Rakshit et al. (2018) S. Rakshit, B. K. Bera, D. Ghosh, and S. Sinha, “Emergence of synchronization and regularity in firing patterns in time-varying neural hypernetworks,” _Phys. Rev. E_ , vol. 97, no. 5, p. 052304, 2018.
  * Rakshit et al. (2019) S. Rakshit, B. K. Bera, J. Kurths, and D. Ghosh, “Enhancing synchrony in multiplex network due to rewiring frequency,” _Proceedings of the Royal Society A_ , vol. 475, no. 2230, p. 20190460, 2019.
  * Berner et al. (2020) R. Berner, J. Sawicki, and E. Schöll, “Birth and stabilization of phase clusters by multiplexing of adaptive networks,” _Physical review letters_ , vol. 124, no. 8, p. 088301, 2020.
  * Vadivasova et al. (2020) T. Vadivasova, A. Slepnev, and A. Zakharova, “Control of inter-layer synchronization by multiplexing noise,” _Chaos: An Interdisciplinary Journal of Nonlinear Science_ , vol. 30, no. 9, p. 091101, 2020.
  * Kuramoto (1984) Y. Kuramoto, “Cooperative dynamics of oscillator communitya study based on lattice of rings,” _Progress of Theoretical Physics Supplement_ , vol. 79, pp. 223–240, 1984.
  * Cenk Eser et al. (2021) M. Cenk Eser, E. S. Medeiros, M. Riza, and A. Zakharova, “Edges of inter-layer synchronization in multilayer networks with time-switching links,” _Chaos: An Interdisciplinary Journal of Nonlinear Science_ , vol. 31, no. 10, p. 103119, 2021. [Online]. Available: https://doi.org/10.1063/5.0065310


* [109] N. Reimers and I. Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  arXiv preprint arXiv:1908.10084, 2019.
  * [110] L. F. Ribeiro, P. H. Saverese, and D. R. Figueiredo.  struc2vec: Learning node representations from structural identity.  In SIGKDD, pages 385–394, 2017.
  * [111] M. T. Ribeiro, S. Singh, and C. Guestrin.  ” why should i trust you?” explaining the predictions of any classifier.  In SIGKDD, pages 1135–1144, 2016.
  * [112] N. Sarafianos, X. Xu, and I. A. Kakadiaris.  Adversarial representation learning for text-to-image matching.  In Proceedings of the IEEE ICCV, pages 5814–5824, 2019.
  * [113] J. Shen, Y. Qu, W. Zhang, and Y. Yu.  Adversarial representation learning for domain adaptation.  stat, 1050:5, 2017.
  * [114] T. Shen, T. Lei, R. Barzilay, and T. Jaakkola.  Style transfer from non-parallel text by cross-alignment.  In NIPS, pages 6830–6841, 2017.
  * [115] C. Shi, M. Xu, Z. Zhu, W. Zhang, M. Zhang, and J. Tang.  Graphaf: a flow-based autoregressive model for molecular graph generation.  arXiv preprint arXiv:2001.09382, 2020.
  * [116] A. Sinha, Z. Shen, Y. Song, H. Ma, D. Eide, B.-j. P. Hsu, and K. Wang.  An overview of microsoft academic service (mas) and applications.  In WWW’15, pages 243–246, 2015.
  * [117] P. Smolensky.  Information processing in dynamical systems: Foundations of harmony theory.  Technical report, Colorado Univ at Boulder Dept of Computer Science, 1986\.
  * [118] F.-Y. Sun, J. Hoffmann, and J. Tang.  Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization.  arXiv preprint arXiv:1908.01000, 2019.
  * [119] F.-Y. Sun, M. Qu, J. Hoffmann, C.-W. Huang, and J. Tang.  vgraph: A generative model for joint community detection and node representation learning.  In NIPS, pages 512–522, 2019.
  * [120] K. Sun, Z. Lin, and Z. Zhu.  Multi-stage self-supervised learning for graph convolutional networks on graphs with few labeled nodes.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5892–5899, 2020.
  * [121] K. Sun, Z. Zhu, and Z. Lin.  Multi-stage self-supervised learning for graph convolutional networks.  arXiv preprint arXiv:1902.11038, 2019.
  * [122] Y. Sun, S. Wang, Y. Li, S. Feng, X. Chen, H. Zhang, X. Tian, D. Zhu, H. Tian, and H. Wu.  Ernie: Enhanced representation through knowledge integration.  arXiv preprint arXiv:1904.09223, 2019.
  * [123] J. Tang, M. Qu, M. Wang, M. Zhang, J. Yan, and Q. Mei.  Line: Large-scale information network embedding.  In WWW’15, pages 1067–1077, 2015.
  * [124] J. Tang, J. Zhang, L. Yao, J. Li, L. Zhang, and Z. Su.  Arnetminer: extraction and mining of academic social networks.  In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 990–998, 2008.
  * [125] W. L. Taylor.  “cloze procedure”: A new tool for measuring readability.  Journalism quarterly, 30(4):415–433, 1953.
  * [126] Y. Tian, D. Krishnan, and P. Isola.  Contrastive multiview coding.  arXiv preprint arXiv:1906.05849, 2019.
  * [127] Y. Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, and P. Isola.  What makes for good views for contrastive learning.  arXiv preprint arXiv:2005.10243, 2020.
  * [128] M. Tschannen, O. Bachem, and M. Lucic.  Recent advances in autoencoder-based representation learning.  arXiv preprint arXiv:1812.05069, 2018.
  * [129] M. Tschannen, J. Djolonga, P. K. Rubenstein, S. Gelly, and M. Lucic.  On mutual information maximization for representation learning.  arXiv preprint arXiv:1907.13625, 2019.
  * [130] A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu.  Wavenet: A generative model for raw audio.  In 9th ISCA Speech Synthesis Workshop, pages 125–125.
  * [131] A. Van den Oord, N. Kalchbrenner, L. Espeholt, O. Vinyals, A. Graves, et al.  Conditional image generation with pixelcnn decoders.  In NIPS, pages 4790–4798, 2016.
  * [132] A. van den Oord, O. Vinyals, et al.  Neural discrete representation learning.  In NIPS, pages 6306–6315, 2017.
  * [133] A. Van Oord, N. Kalchbrenner, and K. Kavukcuoglu.  Pixel recurrent neural networks.  In ICML, pages 1747–1756, 2016.
  * [134] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.  Attention is all you need.  In NIPS, pages 5998–6008, 2017.
  * [135] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio.  Graph attention networks.  arXiv preprint arXiv:1710.10903, 2017.
  * [136] P. Veličković, W. Fedus, W. L. Hamilton, P. Liò, Y. Bengio, and R. D. Hjelm.  Deep graph infomax.  arXiv preprint arXiv:1809.10341, 2018.
  * [137] H. Wang, J. Wang, J. Wang, M. Zhao, W. Zhang, F. Zhang, X. Xie, and M. Guo.  Graphgan: Graph representation learning with generative adversarial nets.  In AAAI, 2018.
  * [138] P. Wang, S. Li, and R. Pan.  Incorporating gan for negative sampling in knowledge representation learning.  In AAAI, 2018.
  * [139] T. Wang and P. Isola.  Understanding contrastive representation learning through alignment and uniformity on the hypersphere.  arXiv preprint arXiv:2005.10242, 2020.
  * [140] Z. Wang, Q. She, and T. E. Ward.  Generative adversarial networks: A survey and taxonomy.  arXiv preprint arXiv:1906.01529, 2019.
  * [141] C. Wei, L. Xie, X. Ren, Y. Xia, C. Su, J. Liu, Q. Tian, and A. L. Yuille.  Iterative reorganization with weak spatial constraints: Solving arbitrary jigsaw puzzles for unsupervised representation learning.  In CVPR, pages 1910–1919, 2019.
  * [142] Z. Wu, Y. Xiong, S. X. Yu, and D. Lin.  Unsupervised feature learning via non-parametric instance discrimination.  In CVPR, pages 3733–3742, 2018.
  * [143] Q. Xie, M.-T. Luong, E. Hovy, and Q. V. Le.  Self-training with noisy student improves imagenet classification.  In CVPR, pages 10687–10698, 2020.
  * [144] W. Xiong, J. Du, W. Y. Wang, and V. Stoyanov.  Pretrained encyclopedia: Weakly supervised knowledge-pretrained language model.  arXiv preprint arXiv:1912.09637, 2019.
  * [145] K. Xu, J. Li, M. Zhang, S. S. Du, K.-i. Kawarabayashi, and S. Jegelka.  How neural networks extrapolate: From feedforward to graph neural networks.  arXiv preprint arXiv:2009.11848, 2020.
  * [146] X. Yan, I. Misra, A. Gupta, D. Ghadiyaram, and D. Mahajan.  Clusterfit: Improving generalization of visual representations.  arXiv preprint arXiv:1912.03330, 2019.
  * [147] J. Yang, D. Parikh, and D. Batra.  Joint unsupervised learning of deep representations and image clusters.  In CVPR, pages 5147–5156, 2016.
  * [148] Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le.  Xlnet: Generalized autoregressive pretraining for language understanding.  In NIPS, pages 5754–5764, 2019.
  * [149] Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning.  Hotpotqa: A dataset for diverse, explainable multi-hop question answering.  arXiv preprint arXiv:1809.09600, 2018.
  * [150] J. You, B. Liu, Z. Ying, V. Pande, and J. Leskovec.  Graph convolutional policy network for goal-directed molecular graph generation.  In NIPS, pages 6410–6421, 2018.
  * [151] J. You, R. Ying, X. Ren, W. Hamilton, and J. Leskovec.  Graphrnn: Generating realistic graphs with deep auto-regressive models.  In ICML, pages 5708–5717, 2018.
  * [152] Y. You, T. Chen, Y. Sui, T. Chen, Z. Wang, and Y. Shen.  Graph contrastive learning with augmentations.  arXiv preprint arXiv:2010.13902, 2020.
  * [153] Y. You, T. Chen, Z. Wang, and Y. Shen.  When does self-supervision help graph convolutional networks?  arXiv preprint arXiv:2006.09136, 2020.
  * [154] F. Zhang, X. Liu, J. Tang, Y. Dong, P. Yao, J. Zhang, X. Gu, Y. Wang, B. Shao, R. Li, and K. Wang.  Oag: Toward linking large-scale heterogeneous entity graphs.  In KDD’19, pages 2585–2595, 2019.
  * [155] J. Zhang, Y. Dong, Y. Wang, J. Tang, and M. Ding.  Prone: fast and scalable network representation learning.  In IJCAI, pages 4278–4284, 2019.
  * [156] M. Zhang, Z. Cui, M. Neumann, and Y. Chen.  An end-to-end deep learning architecture for graph classification.  In AAAI, 2018.
  * [157] R. Zhang, P. Isola, and A. A. Efros.  Colorful image colorization.  In ECCV, pages 649–666. Springer, 2016.
  * [158] R. Zhang, P. Isola, and A. A. Efros.  Split-brain autoencoders: Unsupervised learning by cross-channel prediction.  In CVPR, pages 1058–1067, 2017.
  * [159] Z. Zhang, X. Han, Z. Liu, X. Jiang, M. Sun, and Q. Liu.  Ernie: Enhanced language representation with informative entities.  arXiv preprint arXiv:1905.07129, 2019.
  * [160] D. Zhu, P. Cui, D. Wang, and W. Zhu.  Deep variational network embedding in wasserstein space.  In SIGKDD, pages 2827–2836, 2018.
  * [161] J.-Y. Zhu, R. Zhang, D. Pathak, T. Darrell, A. A. Efros, O. Wang, and E. Shechtman.  Toward multimodal image-to-image translation.  In NIPS, pages 465–476, 2017.
  * [162] C. Zhuang, A. L. Zhai, and D. Yamins.  Local aggregation for unsupervised learning of visual embeddings.  In Proceedings of the IEEE ICCV, pages 6002–6012, 2019.
  * [163] B. Zoph, G. Ghiasi, T.-Y. Lin, Y. Cui, H. Liu, E. D. Cubuk, and Q. V. Le.  Rethinking pre-training and self-training.  arXiv preprint arXiv:2006.06882, 2020.
  * [164] B. Zoph and Q. V. Le.  Neural architecture search with reinforcement learning.  arXiv preprint arXiv:1611.01578, 2016.

## REVISION HISTORY

  * •

v2/v3 (June 2020): correct several typos and mistakes.

  * •

v4 (July 2020): add papers newly published; add a new theoretical analysis
part for contrastive objective; add semi-supervised self-training’s connection
with self-supervised contrastive learning.

  * •

v5 (March 2021): add papers newly published; update statistics in Fig. 2 and
Fig. 7; add a motivation section to better introduce the reason of using SSL;
remove the preliminary section.

|  Xiao Liu is a senior undergraduate student with the Department of Computer
Science and Technology, Tsinghua University. His main research interests
include data mining, machine learning and knowledge graph. He has published a
paper on KDD.
---|---
|  Fanjin Zhang is a PhD candidate in the Department of Computer Science and
Technology, Tsinghua University. She got her bachelor degree from the
Department of Computer Science and Technology, Nanjing Unviersity. Her
research interests include data mining and social network.
---|---
|  Zhenyu Hou is an undergraduate with the department of Computer Science and
Technology, Tsinghua University. His main research interests include graph
representation learning and reasoning.
---|---
|  Li Mian received bachelor degree(2020) from Department of Computer Science,
Beijing Institute of Technology. She is now admitted into a graduate program
in Georgia Institute of Technology. Her research interests focus on data
mining, natural language processing and machine learning.
---|---
|  Zhaoyu Wang is a graduate student with the Department of Computer Science
and Technology of Anhui University. His research interests include data
mining, natural language processing and their applications in recommender
systems.
---|---
|  Jing Zhang received the master and PhD degree from the Department of
Computer Science and Technology, Tsinghua University. She is an assistant
professor in Information School, Renmin University of China. Her research
interests include social network mining and deep learning.
---|---
|  Jie Tang received the PhD degree from Tsinghua University. He is full
professor in the Department of Computer Science and Technology, Tsinghua
University. His main research interests include data mining, social network,
and machine learning. He has published over 200 research papers in top
international journals and conferences.
---|---


approximate solution of the PDEs, as well as the question that is often asked,
namely, does a smaller training error imply more accurate predictions?

(v) Many multiphysics and multiscale complex phenomena occur in systems with
complicated geometry that must be incorporated into the algorithms, which is
not an easy task due to the required computation time. Although some efforts
have been made to address such questions [185-188], much remains to be
explored.

(vi) Even when it is clear one needs a synthesis of two or more approaches,
say a combination of a machine learning algorithm and intensive numerical
simulation, one needs to be equipped with, for example, optimization theory
and theory of PDEs. In addition, there is always tremendous need for yet
faster numerical simulation and analysis. The combination of such branches of
science is openning up new research venues.

In addition, every new approach or algorithm requires benchmarks for checking
its accuracy and efficiency. When dealing with huge amounts of data for
complex phenomena and systems, such benchmarks must provide a meaningful
evaluation of the algorithms. Selecting such benchmarks is also not an easy
task and requires careful considerations, as does the task of selecting the
way by which such data should be made publicly available, a way that is
accessible to a larger number of potential users.

In terms of moving in the direction of much wider use of such algorithms, we
recall that one reason that platforms for conventional computations, such as
OpenFOAM [189] for simulation of fluid flow and transport processes, and the
FEniCS [190] that solves differential equations by finite-element method, are
popular is that they are user-friendly. Thus, for example, in the area of
applications of the physics- and date-informed algorithms or the symbolic
regression methods, in order to make such approaches “everyday tools” of
research and development, they must also be user-friendly, and provide tools
of visualization and tracking the variables as they evolve in space and time.

ACKNOWLEDGMENTS

Over the past several years, I have benefitted greatly from stimulating
discussions and fruitful collaboration with many colleague who worked with me
on some of the problems described in this Perspective. I am particularly
grateful to Felipe de Barros, Jinwoo Im, Serveh Kamrava, Joachim Peinke, Reza
Rahimi Tabar, Pejman Tahmasebi, and Sami Masri.

  * [1]

J.H. Seinfeld and S.N. Pandis, Atmospheric Chemistry and Physics (Wiley, New
York, 1998).

  * [2]

D. Simpson, Long-period modelling of photochemical oxidants in Europe. Model
calculations for July 1985, Atmos. Environ. 26, 1609 (1992).

  * [3]

A. Heidarinasab, B. Dabir, and M. Sahimi, Multiresolution wavelet-based
simulation of transport and photochemical reactions in the atmosphere, Atmos.
Environ. 38, 6381 (2004).

  * [4]

P. Tahmasebi, S. Kamrava, T. Bai, and M. Sahimi, Machine learning in geo- and
environmental sciences: From small to large scale, Adv. Water Resour. 142,
103619 (2020).

  * [5]

G.E. Karniadakis, I.G. Kevrekidis, L. Lu, P. Perdikaris, S. Wang, and L. Yang,
Physics-informed machine learning, Nature Rev. Phys. 3, 422 (2021).

  * [6]

M. Reichstein, G. Camps-Valls, B. Stevens, M. Jung, J. Denzler, N. Carvalhais,
and Mr Prabhat, Deep learning and process understanding for data-driven earth
system science, Nature 566, 195 (2019).

  * [7]

S. Kamrava, P. Tahmasebi, and M. Sahimi, Simulating fluid Flow in complex
porous materials: Integrating the governing equations with deep-layered
machines, NPJ Comput. Mater. 7, 127 (2021).

  * [8]

M. Alber, A.B. Tepole, W.R. Cannon, S. De, S. Dura-Bernal, K. Garikipati, G.
Karniadakis, W.W. Lytton, P. Perdikaris, L. Petzold, and E. Kuhl, Integrating
machine learning and multiscale modeling — perspectives, challenges, and
opportunities in the biological, biomedical, and behavioral sciences, NPJ
Digit. Med. 2, 1 (2019).

  * [9]

M. Sahimi, Flow and Transport in Porous Media and Fractured Rock, 2nd. ed.
(Wiley-VCH, Weinheim, 2011).

  * [10]

M. Sahimi and S.E. Tajer, Self-affine distributions of the bulk density,
elastic moduli, and seismic wave velocities of rock, Phys. Rev. E 71, 046301
(2005).

  * [11]

Z. Zhang and J.C. Moore, Mathematical and Physical Fundamentals of Climate
Change (Elsevier, Amsterdam, 2015), Chapter 9.

  * [12]

G. Cressman, An operational objective analysis system, Mon. Wea. Rev. 87, 367
(1959).

  * [13]

R.E. Kalman, A new approach to linear filtering and prediction problems,
Trans. ASME, J. Basic Engineeering 87, 35 (1960).

  * [14]

G. Evensen, Using the extended Kalman filter with a multilayer quasi-
geostrophic ocean model, J. Geophys. Res. 97, 17905 (1992).

  * [15]

G. Evensen, Sequential data assimilation with a nonlinear quasi-geostrophic
model using monte carlo methods to forecast error statistics, J. Geophys. Res.
99, 10143 (1994).

  * [16]

P.L. Houtekamer and H.L. Mitchell, Data assimilation using an ensemble Kalman
filter technique, Mont. Wea. Rev. 126, 796 (1998).

  * [17]

P.L. Houtekamer and H.L. Mitchell, Ensemble Kalman filtering, Quarterly J.
Roy. Meteorol. Soc. 131, 3269 (2005).

  * [18]

H. Li, S.J. Qin, T.T. Tsotsis, and M. Sahimi, Computer simulation of gas
generation and transport in landfills. VI. Dynamic updating of the model using
the ensemble Kalman filter, Chem. Eng. Sci. 74, 69 (2012).

  * [19]

H. Li, T.T. Tsotsis, M. Sahimi, and S.J. Qin, Ensembles-based and GA-based
optimization for landfill gas production, AIChE J. 60, 2063 (2014).

  * [20]

See, for example, C.M. Bishop, Neural networks and their applications, Rev.
Sci. Instrum. 65, 1803 (1994)

  * [21]

S. Torquato, Random Heterogeneous Materials (Springer, New York, 2002).

  * [22]

M. Sahimi, Heterogeneous Materials, Volumes I and II (Springer, New York,
2003).

  * [23]

S. Kamrava, P. Tahmasebi, and M. Sahimi, Linking morphology of porous media to
their macroscopic permeability by deep learning, Transp. Porous Media 131, 427
(2020).

  * [24]

S. Kamrava, J. Im, F.P.J. de Barros, and M. Sahimi, Estimating dispersion
coefficient in flow through heterogeneous porous media by a deep convolutional
neural network, Geophys. Res. Lett. 48, e2021GL094443 (2021).

  * [25]

H. Wu, W.Z. Fang, Q. Kang, W.Q. Tao, and R. Qiao, Predicting effective
diffusivity of porous media from images by deep learning, Sci. Rep. 9, 20387
(2019).

  * [26]

N. Alqahtani, F. Alzubaidi, R.T. Armstrong, P. Swietojanski, and P.
Mostaghimi, Machine learning for predicting properties of porous media from 2d
X-ray images, J. Pet. Sci. Eng. 184, 106514 (2020).

  * [27]

K.M. Graczyk and M. Matyka, Predicting porosity, permeability, and tortuosity
of porous media from images by deep learning, Sci. Rep. 10, 21488 (2020).

  * [28]

L. Zhou, L. Shi, and Y. Zha, Seeing macro-dispersivity from hydraulic
conductivity field with convolutional neural network, Adv. Water Resour. 138,
103545 (2020).

  * [29]

S. Kamrava, P. Tahmasebi, and M. Sahimi, Enhancing images of shale formations
by a hybrid stochastic and deep learning algorithm, Neural Networks 118, 310
(2019).

  * [30]

S. Kamrava, P. Tahmasebi, and M. Sahimi, Physics- and image-based prediction
of fluid flow and transport in complex porous membranes and materials by deep
learning, J. Membr. Sci. 622, 119050 (2021).

  * [31]

S. Ioffe and C. Szegedy, Batch normalization: Accelerating deep network
training by reducing internal covariate shift, arXiv:1502.03167V3 (2015).

  * [32]

S. Kullback and R.A. Leibler, On information and sufficiency, Annal. Math.
Statist. 22, 79 (1951).

  * [33]

H. Andreä, N. Combaret, J. Dvorkin, E. Glatt, J. Han, M. Kabel, Y. Keehm, F.
Krzikalla, M. Lee, C. Madonna, M. Marsh, T. Mukerji, E.H. Saenger, R. Sain, N.
Saxena, S. Ricker, A. Wiegmann, and X. Zhan, Digital rock physics
benchmarks—Part I: imaging and segmentation, Comput. Geosci. 50, 25 (2013).

  * [34]

G. Kissas, Y. Yang, E. Hwuang, W.R. Witschey, J.A. Detre, and P. Perdikaris,
Machine learning in cardiovascular flows modeling: predicting arterial blood
pressure from non-invasive 4D flow MRI data using physicsinformed neural
networks, Comput. Methods Appl. Mech. Eng. 358, 112623 (2020)

  * [35]

X. Glorot and Y. Bengio, Understanding the difficulty of training deep
feedforward neural networks, in, Proceedings of the thirteenth international
conference on artificial intelligence and statistics, AISTATS 2010, Chia
Laguna Resort, Sardinia, Italy (May 2010), 249-256.

  * [36]

Y. Zhu, N. Zabaras, P.S. Koutsourelakis, and P. Perdikaris, Physics-
constrained deep learning for high-dimensional surrogate modeling and
uncertainty quantification without labeled data, J. Comput. Phys. 394, 56
(2019).

  * [37]

N. Geneva and N. Zabaras, Modeling the dynamics of PDE systems with physics-
constrained deep auto-regressive networks, J. Comput. Phys. 403, 109056
(2020).

  * [38]

J.L. Wu, K. Kashinath, A. Albert, D. Chirila, Prabhat, and H. Xiao, Enforcing
statistical constraints in generative adversarial networks for modeling
chaotic dynamical systems, J. Comput. Phys. 406, 109209 (2020).

  * [39]

A. Kashefi, D. Rempe, and L.J. Guibas, A point-cloud deep learning framework
for prediction of fluid flow fields on irregular geometries, Phys. Fluids 33,
027104 (2021).

  * [40]

Y. LeCun and Y. Bengio, Convolutional networks for images, speech, and time
series, in The Handbook of Brain Theory and Neural Networks, edited by M.A.
Arbib (MIT Press, Cambridge, (1995).

  * [41]

J. Winkens, J. Linmans, B.S. Veeling, T.S. Cohen, and M. Welling, Improved
semantic segmentation for histopathology using rotation equivariant
convolutional networks, in Proceedings of the 1st Conference on Medical
Imaging with Deep Learning (MIDL 2018), Amsterdam, The Netherlands (2018).

  * [42]

T. Cohen, M. Weiler, B. Kicanaoglu, and M. Welling, Gauge equivariant
convolutional networks and the icosahedral CNN, in Proceedings of the 36th
International Conference on Machine Learning, Long Beach, California, 97, 1321
(2019).

  * [43]

H. Owhadi and G.R. Yoo, Kernel flows: from learning kernels from data into the
abyss, J. Comput. Phys. 389, 22 (2019).

  * [44]

M. Raissi, P. Perdikaris, and G.E. Karniadakis, Numerical Gaussian processes
for time-dependent and nonlinear partial differential equations, SIAM J. Sci.
Comput. 40, A172 (2018).

  * [45]

B. Hamzi and H. Owhadi, Learning dynamical systems from data: a simple cross-
validation perspective, part I: parametric kernel flows, Physica D 421, 132817
(2021).

  * [46]

S. Wang, X. Yu, and P. Perdikaris, When and why PINNs fail to train: a neural
tangent kernel perspective, J. Comput. Phys. 449, 110768 (2022).

  * [47]

S. Wang, H. Wang, and P. Perdikaris, On the eigenvector bias of Fourier
feature networks: from regression to solving multi-scale PDEs with physics-
informed neural networks, Comput. Methods Appl. Mech. Eng. 384, 113938 (2021).

  * [48]

H. Owhadi, Do ideas have shape? Plato’s theory of forms as the continuous
limit of artificial neural networks, https://arxiv.org/abs/2008.03920 (2020).

  * [49]

J. Zhou, G. Cui, S. Hu, Z. Zhang, C. Yang, Z. Liu, L. Wang, C. Li, and M. Sun,
Graph neural networks: A review of methods and applications, AI Open 1, 57
(2020).

  * [50]

A. Mathews, M. Francisquez, J. Hughes, and D. Hatch, Uncovering edge plasma
dynamics via deep learning from partial observations, Phys. Rev. E 104, 025205
(2021).

  * [51]

D. Pfau, J.S. Spencer, A.G. Matthews, and W.M.C. Foulkes, Ab initio solution
of the many-electron Schrödinger equation with deep neural networks, Phys.
Rev. Res. 2, 033429 (2020).

  * [52]

G.P. Pun, R. Batra, R. Ramprasad, and Y. Mishin, Physically informed
artificial neural networks for atomistic modeling of materials, Nat. Commun.
10, 2339 (2019).

  * [53]

D. Li, K. Xu, J.M. Harris, and E. Darve, Coupled time-lapse full-waveform
inversion for subsurface flow problems using intrusive automatic
differentiation, Water Resour. Res. 56, e2019WR027032 (2020).

  * [54]

W. Zhu, K. Xu, E. Darve, and G.C. Beroza, A general approach to seismic
inversion with automatic differentiation, Computer Geosci. 151, 104751 (2021).

  * [55]

J. Behler and M. Parrinello, Generalized neural-network representation of
high-dimensional potential-energy surfaces, Phys. Rev. Lett. 98, 146401
(2007).

  * [56]

L. Zhang, J. Han, H. Wang, R. Car, and E. Weinan, Deep potential molecular
dynamics: a scalable model with the accuracy of quantum mechanics, Phys. Rev.
Lett. 120, 143001 (2018).

  * [57]

W. Jia, H. Wang, M. Chen, D. Lu, L. Lin, R. Car, E. Weinan, and L. Zhang,
Pushing the limit of molecular dynamics with ab initio accuracy to 100 million
atoms with machine learning, arXiv:2005.00223 (2020).

  * [58]

A. Nakata, J.S. Baker, S.Y. Mujahed, J.T.L. Poulton, S. Arapan, J. Lin, Z.
Raza, S. Yadav, L. Truflandier, T. Miyazaki, and D.R. Bowler, Large scale and
linear scaling DFT with the CONQUEST code, J. Chem. Phys. 152, 164112 (2020).

  * [59]

R. Mantegna and H. E. Stanley, An Introduction to Econophysics: Correlations
and Complexities in Finance (Cambridge University Press, New York, 2000).

  * [60]

P. Manshoor, S. Saberi, M. Sahimi, J. Peinke, A.F. Pacheco, and M.R. Rahimi
Tabar, Turbulencelike behavior of seismic time series, Phys. Rev. Lett. 102,
014101 (2009).

  * [61]

P.Ch. Ivanov, L.A. Amaral, A.L. Goldberger, S. Havlin, M.G. Rosenblum, Z.R.
Struzik, and H.E. Stanley, Multifractality in human heartbeat dynamics, Nature
(London) 399, 461 (1999).

  * [62]

Y. Ashkenazy, P.Ch. Ivanov, S. Havlin, C.-K. Peng, A.L. Goldberger, and H.E.
Stanley, magnitude and sign correlations in heartbeat fluctuations, Phys. Rev.
Lett. 86, 1900 (2001).

  * [63]

J.D. Hamilton, Time Series Analysis  (Princeton University Press, Princeton,
1994).

  * [64]

H. Kantz and T. Schreiber, Nonlinear Time Series Analysis (Cambridge
University Press, London, 2003).

  * [65]

R.H. Stoffer and S. David Time Series Analysis and Its Applications (Springer-
Verlag, Berlin, 2006).

  * [66]

R. Friedrich and J. Peinke, Description of a turbulent cascade by a Fokker-
Planck equation, Phys. Rev. Lett. 78, 863 (1997).

  * [67]

J. Davoudi and M.R. Rahimi Tabar, Theoretical model for the Kramers-Moyal
description of turbulence cascades, Phys. Rev. Lett. 82, 1680 (1999).

  * [68]

R. Friedlich, J. Peinke, M. Sahimi, and M.R. Rahimi Tabar, Approaching
complexity by stochastic methods: From biological systems to turbulence, Phys.
Rep. 506, 87 (2011).

  * [69]

B.B. Mandelbrot and J.W. van Ness, Fractional Brownian motions, fractional
noises, and applications, SIAM Rev. 10, 422 (1968).

  * [70]

G.R. Jafari, S.M. Fazeli, F. Ghasemi, S.M. Vaez Allaei, M.R. Rahimi Tabar, A.
Iraji zad, and G. Kavei, stochastic analysis and regeneration of rough
surfaces, Phys. Rev. Lett. 91, 226101 (2003).

  * [71]

F. Ghasemi, M. Sahimi, J. Peinke, and M.R. Rahimi Tabar, Analysis of non-
stationary data for heart-rate fluctuations in terms of drift and diffusion
coefficients, J. Biol. Phys. 32, 117 (2006).

  * [72]

H. Risken, The Fokker-Planck Equation, 2nd ed. (Springer, Berlin, 1996).

  * [73]

F. Ghasemi, M. Sahimi, J. Peinke, R. Friedrich, G.R. Jafari, and M.R. Rahimi
Tabar, Markov analysis and Kramers-Moyal expansion of nonstationary stochastic
processes with an application to the fluctuations in the oil price, Phys. Rev.
E 75, 060102(R) (2007).

  * [74]

J.P. Bouchaud and R. Cont, A Langevin approach to stock market fluctuations
and crashes, Eur. Phys. J. B 6, 543 (1998).

  * [75]

F. Ghasemi, J. Peinke, M. Sahimi, and M.R. Rahimi Tabar, Regeneration of
stochastic processes: An inverse method, Euro. Phys. J. B 47, 411 (2005).

  * [76]

R.F. Pawula, Approximation of the linear Boltzmann equation by the Fokker-
Planck equation, Phys. Rev. 162, 186 (1967).

  * [77]

M. Anvari, K. Lehnertz, M.R. Rahimi Tabar, and J. Peinke, Disentangling the
stochastic behavior of complex time series, Sci. Rep. 6, 35435 (2016).

  * [78]

M.R. Rahimi Tabar, Analysis and Data-Based Reconstruction of Complex Nonlinear
Dynamical Systems: Using the Methods of Stochastic Processes (Springer, Bern,
2019).

  * [79]

M.T. Giraudo and L. Sacerdote, Jump-diffusion processes as models for neuronal
activity, Biosystems 40, 75 (1997).

  * [80]

R. Sirovich, L. Sacerdote, and A.E.P. Villa, Cooperative behavior in a jump
diffusion model for a simple network of spiking neurons, Math. Biosci. Eng.
11, 385 (2014).

  * [81]

E. Daly and A. Porporato, Probabilistic dynamics of some jump-diffusion
systems, Phys. Rev. E 73, 026108 (2006).

  * [82]

R. Cont and P. Tankov, Financial Modelling with Jump Processes (Chapman &
Hall, Boca Raton, 2004).

  * [83]

J. Prusseit and K. Lehnertz, Stochastic qualifiers of epileptic brain
dynamics, Phys. Rev. Lett. 98, 138103 (2007).

  * [84]

K. Lehnertz, Epilepsy and nonlinear dynamics, J. Biol. Phys. 34, 253 (2008).

  * [85]

L.R. Gorjão, J. Heysel, K. Lehnertz, and M.R. Rahimi Tabar, Analysis and data-
driven reconstruction of bivariate jump-diffusion processes, Phys. Rev. E 100,
062127 (2019)

  * [86]

F. Nikakhtar, L. Parkavosi, M.R. Rahimi Tabar, M. Sahimi, K. Lehnertz, and U.
Feudel, Data-driven reconstruction of stochastic dynamical equations based on
statistical moments, Phys. Rev. Lett. (to be published).

  * [87]

M.R. Rahimi Tabar, F. Nikakhtar, L. Parkavosi, A. Akhshi, K. Lehnertz, and U.
Feudel, Revealing higher-order interactions in high-dimensional complex
systems: A data-driven approach, to be published.

  * [88]

F. Hourdin, T. Mauritsen, A. Gettelman, J.-C. Golaz, V. Balaji, Q. Duan, D.
Folini, D. Ji, D. Klocke, Y. Qian, F. Rauser, C. Rio, L. Tomassini, M.
Watanabe, and D. Williamson, The art and science of climate model tuning,
Bull. Am. Meteorol. Soc. 98, 589 (2017)

  * [89]

S. Scher, Toward data-driven weather and climate forecasting: Approximating a
simple general circulation model with deep learning, Geophys. Res. Lett. 45,
12 (2018).

  * [90]

P.D. Dueben and P. Bauer, Challenges and design choices for global weather and
climate models based on machine learning, Geosci. Model Development 11, 3999
(2018).

  * [91]

D.C. Park, A time series data prediction scheme using bilinear recurrent
neural network, in Proceedings of the 2010 International Conference on
Information Science and Applications, Seoul, South Korea (2010), p. 1, doi:
10.1109/ICISA.2010.5480383.

  * [92]

R. Fablet, S. Ouala, and C. Herzet, Bilinear residual neural network for the
identification and forecasting of geophysical dynamics, in Proceedings of 26th
European Signal Processing Conference (EUSIPCO), Rome, Italy (2018), p. 1477,
doi: 10.23919/EUSIPCO.2018.8553492.

  * [93]

K. Pathak, B. Hunt, M. Girvan, Z. Lu, and E. Ott, Model-free prediction of
large spatiotemporally chaotic systems from data: A reservoir computing
approach, Phys. Rev. Lett. 120, 024102 (2018).

  * [94]

P. Laloyaux, M. Bonavita, M. Dahoui, J. Farnan, S. Healy, E. Hólm, and S.
Lang, Towards an unbiased stratospheric analysis, Q. J. Roy. Meteorol. Soc.
146, 2392 (2020).

  * [95]

S. Rasp, M.S. Pritchard, and P. Gentine, Deep learning to represent subgrid
processes in climate models, Proc. Natl. Acad. Sci. USA 115, 9684 (2018).

  * [96]

W.D. Collins et al., The formulation and atmospheric simulation of the
community atmosphere model version 3 (CAM3), J. Clim. 19, 2144 (2006).

  * [97]

J. Brajard, A. Carrassi, M. Bocquet, and L. Bertino, Combining data
assimilation and machine learning to infer unresolved scale parametrisation,
Phil. Trans. Roy. Soc. London 379, 20200086 (2021).

  * [98]

L. De Cruz, J. Demaeyer, and S. Vannitsem, The modular arbitrary-order ocean-
atmosphere model: MAOOAM v1.0, Geosci. Model Development 9, 2793 (2016).

  * [99]

J. Bongard and H. Lipson, Automated reverse engineering of nonlinear dynamical
systems, Proc. Natl. Acad. Sci. U.S.A. 104, 9943 (2007).

  * [100]

M. Schmidt and H. Lipson, Distilling free-form natural laws from experimental
data, Science 324, 81 (2009).

  * [101]

J. Bongard, V. Zykov, and H. Lipson, Resilient machines through continuous
self-modeling, Science 314, 1118 (2066).

  * [102]

M. Sahimi and P. Tahmasebi, Reconstruction, optimization, and design of
heterogeneous materials and media: Basic principles, computational algorithms,
and applications, Phys. Rep. 939, 1 (2021).

  * [103]

S. Kirkpatrick, C.D. Gelatt, Jr., and M.P. Vecchi, Optimization by simulated
annealing, Science 220, 671 (1983).

  * [104]

S. Katoch, S. Singh Chauhan, and V. Kumar, A review on genetic algorithm:
past, present, and future, Multimed. Tools. Appl. 80, 8091 (2021).

  * [105]

S.J. Russell and P. Norvig, Artificial Intelligence: A Modern Approach, 2nd
ed. (Prentice Hall, Upper Saddle River, 2003) pp. 111–114.

  * [106]

J. Im, F.P.J. de Barros, S. Masri, M. Sahimi, and R.M. Ziff, Data-driven
discovery of the governing equations for transport in heterogeneous media by
stochastic optimization, Phys. Rev. E, to be published

  * [107]

Y. Gefen, A. Aharony, and S. Alexander, Anomalous diffusion on percolation
clusters, Phys. Rev. Lett. 50, 77 (1983).

  * [108]

D. Stauffer and A. Aharony, Introduction to Perclation Theory, 2nd. ed.
(Taylor and Francis, London, 1994).

  * [109]

M. Sahimi, Applications of Percolation Theory, 2nd. ed. (Springer, New York,
2023).

  * [110]

B. O’Shaughnessy and I. Procaccia, Analytical solutions for diffusion on
fractal objects, Phys. Rev. Lett. 54, 455 (1985).

  * [111]

M. Giona and H.E. Roman, Fractional diffusion equation for transport phenomena
in random media, Physica A 185, 87 (1992).

  * [112]

R. Metzler, W.G. Glöckle, and T.F. Nonnenmacher, Fractional model equation for
anomalous diffusion, Physica A 211, 13 (1994).

  * [113]

Y. He, S. Burov, R. Metzler, and E. Barkai, Random time-scale invariant
diffusion and transport coefficients, Phys. Rev. Lett. 101, 058101 (2008).

  * [114]

A. Bunde and J. Dräger, Localization in disordered structures: Breakdown of
the self-averaging hypothesis, Phys. Rev. E 52, 53 (1995).

  * [115]

A. Pacheco-Pozo and I.M. Sokolov, Universal fluctuations and ergodicity of
generalized diffusivity on critical percolation clusters, J. Phys. A 55,
345001 (2022).

  * [116]

K. Garbrecht, M. Aguilo, A. Sanderson, A. Rollett, R.M. Kirby, and J.
Hochhalter, Interpretable machine learning for texture-dependent constitutive
models with automatic code generation for topological optimization, Integr.
Mater. Manuf. Innov. 10, 373 (2021)

  * [117]

P.J. Schmid, Dynamic mode decomposition of numerical and experimental data, J.
Fluid Mech. 656, 5 (2010).

  * [118]

B.O. Koopman, Hamiltonian systems and transformation in Hilbert space, Proc.
Natl. Acad. Sci. USA 17, 315 (1931).

  * [119]

M.R. Jovanović, P.J. Schmid, and J.W. Nichols, Sparsity-promoting dynamic mode
decomposition, Phys. Fluids 26, 024103 (2014).

  * [120]

S.L. Brunton, J.L. Proctor, and J.N. Kutz, Discovering governing equations
from data by sparse identification of nonlinear dynamical systems, Proc. Natl.
Acad. Sci. U.S.A. 113, 3932 (2016).

  * [121]

F. Takens, Detecting strange attractors in turbulence, Lect. Notes. Math. 898,
366 (1981).

  * [122]

H. Ye, R.J. Beamish, S.M. Glaser, S.C. H. Grant, C.-H. Hsieh, L.J. Richards,
J.T. Schnute, and G. Sugihara, Equation-free mechanistic ecosystem forecasting
using empirical dynamic modeling, Proc. Natl. Acad. Sci. U.S.A. 112, E1569
(2015).

  * [123]

G. Berkooz, P. Holmes, and J.L. Lumley, The proper orthogonal decomposition in
the analysis of turbulent flows, Annu. Rev. Fluid Mech 25 , 539 (1993).

  * [124]

D. Ruelle and F. Takens, On the nature of turbulence, Commun. Math. Phys. 20,
167 (1971).

  * [125]

C.P. Jackson, A finite-element study of the onset of vortex shedding in flow
past variously shaped bodies, J. Fluid Mech. 182, 23 (1987).

  * [126]

Z. Zebib, Stability of viscous flow past a circular cylinder, J. Eng. Math.
21, 155 (1987).

  * [127]

B.R. Noack, K. Afanasiev, M. Morzynsk, D. Tadmor, and F. Thiele, A hierarchy
of low-dimensional models for the transient and post-transient cylinder wake,
J. Fluid Mech. 497, 335 (2003).

  * [128]

T. Colonius and K. Taira, A fast immersed boundary method using a nullspace
approach and multi-domain far-field boundary conditions, Comput. Methods Appl.
Mech. Eng. 197, 2131 (2008).

  * [129]

See, https://github.com/dynamicslab/pysindy.

  * [130]

B.M. de Silva, K. Champion, M. Quade, J.-C. Loiseau, J.N. Kutz, and S.L.
Brunton, PySINDy: a Python package for the sparse identification of nonlinear
dynamics from data, J. Open Source Software 5, 2104 (2020).

  * [131]

K. Champion, B. Lusch, J.N. Kutz, and S.L. Brunton, Data-driven discovery of
coordinates and governing equations, Proc. Natl. Acad. Sci. U.S.A. 116, 22445
(2019).

  * [132]

C. Rackauckas, Y. Ma, J. Martensen, C. Warner, K. Zubov, R. Supekar, D.
Skinner, A. Ramadhan, and A. Edelman, Universal differential equations for
scientific machine learning, arXiv:2001.04385 (2020).

  * [133]

M. Gelbrecht, N. Boers, and J. Kurths, Neural partial differential equations
for chaotic systems, New J. Phys. 23, 043005 (2021).

  * [134]

M. Kalia, S.L. Brunton, H.G.E. Meijer, C. Brune, and J.N. Kutz, Learning
normal form autoencoders for data-driven discovery of universal, parameter-
dependent governing equations, arXiv:2106.05102 (2021).

  * [135]

M. Sorokina, S. Sygletos, and S. Turitsyn, Sparse identification for nonlinear
optical communication systems: SINO method, Optics Express 24, 30433 (2016).

  * [136]

M. Dam, M. Brøns, J.J. Rasmussen, V. Naulin, and J.S. Hesthaven, Sparse
identification of a predator-prey system from simulation data of a convection
model, Phys. Plasmas 24, 022310 (2017).

  * [137]

J.-C. Loiseau and S.L. Brunton, Constrained sparse Galerkin regression, J.
Fluid Mech. 838, 42 (2018).

  * [138]

J.-C. Loiseau, B.R. Noack, and S.L. Brunton. Sparse reduced-order modeling:
sensor-based dynamics to full-state estimation, J. Fluid Mech. 844, 459
(2018).

  * [139]

L. Boninsegna, F. Nüske, and C. Clementi, Sparse learning of stochastic
dynamical equations, J. Chem. Phys. 148, 241723 (2018).

  * [140]

P. Gel$\beta$, S. Klus, J. Eisert, and C. Schütte, Multidimensional
approximation of nonlinear dynamical systems, J. Comput. Nonlinear Dyn. 14,
061006 (2019).

  * [141]

S. Thaler, L. Paehler, and N.A. Adams. Sparse identification of truncation
errors, J. Comput. Phys. 397, 108851 (2019).

  * [142]

K. Kaheman, E. Kaiser, B. Strom, J.N. Kutz, and S.L. Brunton, Learning
discrepancy models from experimental data, arXiv:1909.08574v1 (2019).

  * [143]

J.-C. Loiseau, Data-driven modeling of the chaotic thermal convection in an
annular thermosyphon, Theor. Comput. Fluid Dyn. 34, 339 (2020).

  * [144]

S. Beetham and J. Capecelatro, Formulating turbulence closures using sparse
regression with embedded form invariance, Phys. Rev. Fluid. 5, 084611 (2020).

  * [145]

M. Schmelzer, R.P. Dwight, and P. Cinnella, Discovery of algebraic Reynolds-
stress models using sparse symbolic regression, Flow, Turbulence and
Combustion 104, 579 (2020).

  * [146]

B.M. de Silva, D.M. Higdon, S.L. Brunton, and J.N. Kutz, Discovery of physics
from data: universal laws and discrepancies, Front. Artif. Intell. 3, 25
(2020).

  * [147]

J.J. Bramburger and J.N. Kutz, Poincaré maps for multiscale physics discovery
and nonlinear Floquet theory, Physica D 408, 132479 (2020).

  * [148]

J.L. Callaham, J.-C. Loiseau, G. Rigas and S.L. Brunton, Nonlinear stochastic
modelling with Langevin regression, Proc. Roy. Soc. Lond. A 477, 20210092
(2021).

  * [149]

J.J. Bramburger, J.N. Kutz, and S.L. Brunton, Data-driven stabilization of
periodic orbits, IEEE Access 9, 43504 (2021).

  * [150]

D.E. Shea, S.L. Brunton, and J.N. Kutz, SINDy-BVP: Sparse identification of
nonlinear dynamics for boundary value problems, Phys. Rev. Res. 3, 023255
(2021).

  * [151]

S. Beetham, R.O. Fox, and J. Capecelatro, Sparse identification of multiphase
turbulence closures for coupled fluid–particle flows, J. Fluid Mech. 914, A11
(2021).

  * [152]

A.A. Kaptanoglu, K.D. Morgan, C.J. Hansen, and S.L. Brunton, Physics-
constrained, low-dimensional models for MHD: First-principles and data-driven
approaches, Phys. Rev. E 104, 015206 (2021).

  * [153]

N. Deng, B.R. Noack, M. Morzyński, and L.R. Pastur, Galerkin force model for
transient and post-transient dynamics of the fluidic pinball, J. Fluid Mech.
918, A4 (2021).

  * [154]

Y. Guan, S.L. Brunton, and I. Novosselov, Sparse nonlinear models of chaotic
electroconvection, Roy. Soc. Open Sci. 8, 202367 (2021).

  * [155]

J.L. Callaham, S.L. Brunton, and J.-C. Loiseau, On the role of nonlinear
correlations in reduced-order modeling, J. Fluid Mech. 938, A1 (2022).

  * [156]

J. L. Callaham, G. Rigas, J.-C. Loiseau, and S.L. Brunton, An empirical mean-
field model of symmetry-breaking in a turbulent wake, Sci. Adv. 8, eabm4786
(2022).

  * [157]

H. Schaeffer and S.G. McCalla, Sparse model selection via integral terms,
Phys. Rev. E 96, 023302 (2017).

  * [158]

D.A. Messenger and D.M. Bortz, Weak SINDy for partial differential equations,
J. Comput. Phys. 443, 110525 (2021).

  * [159]

D.R. Gurevich, P.A. Reinbold, and R.O. Grigoriev, Robust and optimal sparse
regression for nonlinear PDE models, Chaos 29, 103113 (2019).

  * [160]

P.A. Reinbold, D.R. Gurevich, and R.O. Grigoriev, Using noisy or incomplete
data to discover models of spatiotemporal dynamics, Phys. Rev. E 101, 010202
(2020).

  * [161]

P.K.A. Reinbold, L.M. Kageorge, M.F. Schatz, and R.O. Grigoriev, Robust
learning from noisy, incomplete, high-dimensional experimental data via
physically constrained symbolic regression, Nat. Commun. 12, 3219 (2021).

  * [162]

E.P. Alves and F. Fiuza, Data-driven discovery of reduced plasma physics
models from fully-kinetic simulations, Phys. Rev. Res. 4, 033192 (2022).

  * [163]

H. Mori, Transport, collective motion, and Brownian motion, Prog. Theor. Phys.
33, 423 (1965).

  * [164]

R. Zwanzig, Nonlinear generalized Langevin equations, J. Stat. Phys. 9, 215
(1973).

  * [165]

G.F. Mazenko, Nonequilibrium Statistical Mechanics (Wiley-VCH, Weinheim,
2006).

  * [166]

D.J. Evans and G. Morriss, Statistical Mechanics of Nonequilibrium Liquids
(Cambridge University Press, Cambridge, 2008).

  * [167]

C. Hij́on, P. Espãnol, E. Vanden-Eijnden, and R. Delgado-Buscalioni, Mori-
Zwanzig formalism as a practical computational tool, Faraday Discuss. 144, 301
(2010).

  * [168]

A. J. Chorin, O. H. Hald, and R. Kupferman, Optimal prediction and the Mori-
Zwanzig representation of irreversible processes, Proc. Natl. Acad. Sci. USA
97, 2968 (2000).

  * [169]

S.K.J. Falkena, C. Quinn, J. Sieber, J. Frank, and H.A. Dijkstra, Derivation
of delay equation climate models using the Mori-Zwanzig formalism, Proc. R.
Soc. A 475, 20190075 (2019).

  * [170]

A. Gouasmi, E.J. Parish, and K. Duraisamy, A priori estimation of memory
effects in reduced-order models of nonlinear systems using the Mori-Zwanzig
formalism, Proc. R. Soc. LOnd. A 473, 20170385 (2017).

  * [171]

Y. Tian, Y.T. Lin, M. Anghel, and D. Livescu, Data-driven learning of Mori-
Zwanzig operators for isotropic turbulence, Phys. Fluids 33, 125118 (2021).

  * [172]

E.J. Parish and K. Duraisamy, Non-Markovian closure models for large eddy
simulations using the Mori-Zwanzig formalism, Phys. Rev. Fluids 2, 014604
(2017).

  * [173]

W. Chu and X. Li, The Mori-Zwanzig formalism for the derivation of a
fluctuating heat conduction model from molecular dynamics, Commun. Math. Sci.
17, 539 (2019).

  * [174]

Y.T. Lin, Y. Tian, D. Livescu, and M. Anghel, Data-driven learning for the
Mori-Zwanzig formalism: A generalization of the Koopman learning framework,
SIAM J. Appl. Dyn. Syst. 20, 101137/21M1401759 (2021).

  * [175]

E.J. Parish and K. Duraisamy, A dynamic subgrid scale model for large eddy
simulations based on the Mori-Zwanzig formalism, J. Comput. Phys. 349, 154
(2017).

  * [176]

S. Maeyama and T.-H. Watanabe, Extracting and modeling the effects of small-
scale fluctuations on large-scale fluctuations by Mori-Zwanzig projection
operator method, J. Phys. Soc. Jpn. 89, 024401 (2020).

  * [177]

J. Li and P. Stinis, Mori-Zwanzig reduced models for uncertainty
quantification, J. Comput. Dyn. 6, 39 (2019).

  * [178]

P. Stinis, Higher order Mori-Zwanzig models for the Euler equations,
Multiscale Modeling Simul. 6, 10.1137/06066504X (2007).

  * [179]

T. Qin, K. Wu, and D. Xiu, Data driven governing equations approximation using
deep neural networks, J. Comput. Phys. 395, 620 (2019).

  * [180]

D.M. DiPietro, S. Xiong, and B. Zhu, Sparse symplectically integrated neural
networks, in Advances in Neural Information Processing Systems 33 (NeurIPS
2020), edited by H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H.
Lin, 34th Conference on Neural Information Processing Systems, Vancouver,
Canada (2020).

  * [181]

L. Lu, P. Jin, and G.E. Karniadakis, Learning nonlinear operators via DeepONet
based on the universal approximation theorem of operators, Nat. Machine Intel.
3, 218 (2021).

  * [182]

B. Reyes, A.A. Howard, P. Perdikaris, and A.M. Tartakovsky, Learning unknown
physics of non-Newtonian fluids, Phys. Rev. Fluids 6, 073301 (2021).

  * [183]

S. Cai, Z. Wang, L. Lu, T.A. Zaki, and G.E. Karniadakis, DeepM&Mnet: inferring
the electroconvection multiphysics fields based on operator approximation by
neural networks, J. Comput. Phys. 436, 110296 (2020).

  * [184]

Z. Mao, L. Lu, O. Marxen, T.A. Zaki, and G.E. Karniadakis, DeepM&Mnet for
hypersonics: Predicting the coupled flow and finite-rate chemistry behind a
normal shock using neural-network approximation of operators, J. Comput. Phys.
447, 110698 (2021)

  * [185]

Y. Shin, J. Darbon, and G.E. Karniadakis, On the convergence of physics
informed neural networks for linear second-order elliptic and parabolic type
PDEs, Commun. Comput. Phys. 28, 2042 (2020).

  * [186]

S. Mishra and R. Molinaro, Estimates on the generalization error of physics
informed neural networks (PINNs) for approximating PDEs, IMA J. Numer.
Analysis 42, 981 (2022).

  * [187]

S. Mishra and R. Molinaro, Estimates on the generalization error of physics
informed neural networks (PINNs) for approximating PDEs II: a class of inverse
problems, arXiv:2007.01138v2 (2021)

  * [188]

Y. Shin, Z. Zhang, and G.E. Karniadakis, Error estimates of residual
minimization using neural networks for linear PDEs, arXiv:2010.08019 (2020).

  * [189]

H. Jasak, A. Jemcov, and Z. Tuković, OpenFOAM: A C++ library for complex
physics simulations, International Workshop on Coupled Methods in Numerical
Dynamics IUC, Dubrovnik, Croatia (September 2007).

  * [190]

M. Alnæs, J. Blechta, J. Hake, A. Johansson, B. Kehlet, A. Logg, C.
Richardson, J. Ring, M.E. Rohnes, and G.N. Wells, The FEniCS project version
1.5, Arch. Numer. Softw. 3, 9 (2015).

Figure 1: The architecture of the PIRED network, with $E_{i}$ and $D_{i}$
indicating the encoder and decoder blocks; $\sigma^{2}$ being the cost
function, $x^{i}$ the input, and the pressure $P^{j}$ and fluid velocity
$|v|^{j}$ are the output [7].

Figure 2: Computational efficiency and accuracy of the PIRED. Comparison of
the cost $\sigma^{2}$ for (a) training and (b) test data of pressure $p$ and
fluid velocity $v$.

Figure 3: Comparison of the actual permeabilities $K$ and the predictions by
the PIRED network for (a) 300 2D images of the membrane, and (b) for 100
images of the Fontainebleau sandstone. $K$ is normalized according to
$(K-K_{\rm min})/(K_{\rm max}-K_{\rm min})$ [7].

Figure 4: Schematic representation of a $Y-$shaped bifurcating arterial system
and its 1D center-lines used in the reduced model [25].

Figure 5: Flow through a $Y-$shaped bifurcation. (a) Comparison of the
computed velocity, $u(t)=v_{x}(t)$ wave, obtained by the discontinuous
Galerkin (DG, red) method with those predicted by the PINN with (blue) and
without non-dimensionalization (black) at the middle point of channel 1 [the
left channel in Fig. 4]. (b) Same as in (a), but for the pressure wave [25].

Figure 6: The drift coefficient $D^{(1)}(x)$ for two classes of patients, the
healthy ones, and those with congestive heart failure (CHF).

Figure 7: The diffusion coefficient $D^{(2)}(x)$ for two classes of patients,
the healthy ones, and those with congestive heart failure (CHF).

Figure 8: (a) Implantation scheme of intracranial electrodes in a patient with
seizures originating in the left mesial temporal lobe: temporal-lateral grid
electrode ($8$ contacts, denoted by GL), two temporal-basal strip electrodes
(4 contacts each, denoted by TB), and a hippocampal depth electrode (10
contacts, denoted by TL). The most anterior contact (TL1) is located ventral
to the amygdala and the most posterior contact (TL10) is located within the
hippocampus. The latter electrode samples the epileptic focus. (b) Segments of
the iEEG time series recorded during the seizure-free interval from within the
epileptic focus (red; contact TL4) and from a distant brain region (black;
contact GLC6) [77].

Figure 9: Sample results computed based on the data for an epilepsy patient
with an epileptic focus in the left mesial temporal lobe, showing, (a) the
diffusion coefficients; (b) jump amplitudes, and (c) jump rates, together with
the respective probability distribution functions estimated from normalized
iEEG time series, recorded during the seizure-free interval from within the
epileptic focus (red, contact TL4 in Fig. 8) and the data from a distant brain
region (black, contact GLC6, as shown in Fig. 8) [77]. Figure 10: Longitudinal
and five-year temporal averages. (A) Mean convective and radiative subgrid
heating rates $\Delta T_{\rm phy}$. (B) Mean temperature $T$ of SPCAM and
biases of NNCAM and CTRLCAM relative to SPCAM. The dashed black line denotes
the approximate position of the tropopause. (C) Mean short- (solar) and
longwave (thermal) net fluxes at the top of the atmosphere and precipitation.
The latitude axis is area weighted [95].

Figure 11: Precipitation histogram of time-step (30 min) accumulation. The bin
width is 3.9 mm$\cdot$d-1. Solid lines show the simulations for reference sea
surface temperature (SST). Dashed lines denote simulations for warming up by
+4-K (see the original Rreference [95]). The neural network in the +4-K case
is NNCAM-ref + 4 K [95].

Figure 12: The vortex shedding past a cylinder is the result of a Hopf
bifurcation. Because the Navier-Stokes equations have quadratic nonlinearity,
one must use a mean-field model with a separation of timescales, where a fast
mean-field deformation is slave to the slow vortex shedding dynamics. The
parabolic slow manifold is shown (left), with the unstable fixed point (C),
mean flow (B), and vortex shedding (A). A proper orthogonal decomposition
(POD) basis and shift mode were used to reduce the dimension of the problem
(middle right). The identified dynamics closely match the true trajectory in
POD coordinates, and capture the quadratic nonlinearity and timescales
associated with the mean-field model [120].

Figure 13: Normalized Frobenius norm of the learned memory kernel for
observable set as a function of normalized time delay. Two types of spatial
filters, Gaussian and box filters, with various filtering length scales, were
applied to the physical-space variables [171].