# DMInet: an accurate and highly flexible deep learning framework for drug
membrane interaction with membrane selectivity

Guang Chen<EMAIL_ADDRESS>[

###### Abstract

Drug membrane interaction is a very significant bioprocess to consider in drug
discovery. Here, we propose a novel deep learning framework coined DMInet to
study drug-membrane interactions that leverages large-scale Martini coarse-
grained molecular simulations of permeation of drug-like molecules across six
different lipid membranes. The network of DMInet receives three inputs, viz.,
the drug-like molecule, membrane type and spatial distance across membrane
thickness, and predicts the potential of mean force with structural resolution
across the lipid membrane and membrane selectivity. Inheriting from coarse-
grained Martini representation of organic molecules and combined with deep
learning, DMInet has the potential for more accelerated high throughput
screening in drug discovery across a much larger chemical space than that can
be explored by physics-based simulations alone. Moreover, DMInet is highly
flexible in its nature and holds the possibilities for other properties
prediction without significant change of the architecture. Last but not least,
the architecture of DMInet is general and can be applied to other membrane
problems involving permeation and selection.

###### keywords:

Deep Learning, DMInet, Drug Membrane Interaction, Martini Coarse-Graining,
Membrane Selectivity

IMS] Institute of Materials Science, University of Connecticut, Storrs CT,
06269

##  1 Introduction

Drug membrane interactions (DMI), a transmembrane activity happening in a soft
interface between blood and mammalian membrane, is a necessary process to
consider for treating diseased cells in drug delivery and many other
therapeutic approaches 1, 2. As such, study of passive permeation of drug-like
molecules across phospholipid membranes has not only fundamental significance
in physical chemistry or biochemistry, but also very practical meanings in
pharmaceutical engineering.

DMI renders an important bioprocess to consider in drug discovery, for
example, in the design of effective antimicrobials for bacterial persisters
which cause persistent infections 3, 4. It has been reported that these
pathogens result in around 2.8 million infections and even 35,000 death in the
United States each year 5. Different from the majority of bacteria which show
significant biosynthetic process, bacterial persisters are dormant, making the
common antibiotics targeting biosynthetic processes intractable 6. The
membrane-targeting agents are promising yet underexploited for treating
persisters, which can disrupt the integrity and the functionality of bacterial
membranes, and ultimately kill the persisters 6, 7. A crucial need associated
with antibiotic design is to fight against multidrug resistance evolved in
pathogens during bacteria-antibiotics competition. Naturally occurring
antimicrobial peptides (AMPs) is considered as a promising technique for this
treatment, which has been widely studied 8, 9, 10. The idea is also to disrupt
the membrane functionality of microbial pathogens. Therefore, an important
property for these potential agents is the membrane selectivity. That is, it
should target bacterial membrane bilayer only, while selectively avoiding
mammalian membranes. As a result, a systematic study of potential drugs across
a big chemical space interacting with multi-membrane is of great values, which
can bring immense information and insights to DMI, and more importantly, boost
discovery of potent drugs.

However, a detailed and thorough understanding of DMI is still lacking 11.
Scientific challenges associated with this problem come from the intrinsic
features of drugs/membranes and the existing study methods. For example, how
the concentration of the drug, and membrane types and compositions affect the
interactions between them and further affect the membrane selectivity. For
potential drugs, the complexity of the possible compositions results in a huge
yet discrete chemical space, which makes existing study methods, either
experiments or computations, intractable as most optimization methods are
gradient-based. Experiments guided by theory and computational simulations
have been the mainstreams in drug discovery, though both methods have
unavoidable limitations, such as time- and cost-consuming.

Figure 1: Schematic of physical modeling of drug-membrane interaction. a: CGMD
simulation of the free energy profile of a sample molecule using umbrella
sampling (the PMF profile is drawn after data from Kim et al. 12); b: the
lipid molecules and corresponding Martini coarse-grained representation
(adapted from Ref. 13).

Despite this limitation, high throughput computation (HTC) for screening has
been deemed as the most applicable and successful approach for novel material
design 14, 15 and drug discovery 7, 12. The development of physics-based
molecular dynamics (MD) simulations have enabled enormous progresses in DMI
and provided nanoscale structural resolution on the insertion of drugs into
lipid bilayers 10. In MD simulations, the free energy profile or potential of
mean force (PMF) along the membrane thickness can be extracted, which grands
rich information of DMI and membrane selectivity, as shown in Figure 1. To
illustrate, Kim et al. 12 applied high throughput screening for antibiotic
discovery and bithionol was discovered as a candidate. All-atom MD (AAMD)
simulations were then employed to showcase the difference of the PMF profiles
of bithionol across bacterial (mimicked by membranes without cholesterol) and
mammalian membranes (mimicked by membranes with cholesterol). As seen from the
PMF profile in Figure 1a, this drug can selectively permeate the bacterial
membrane whereas avoid mammalian membrane. This HTC-screened antibiotic was
validated very well by further experiments for the membrane selectivity. Their
work demonstrated the capability of physics-based modeling in study of DMI
problems. Though AAMD simulation is indeed a great tool to study DMI, the
shortcomings associated with it are also very significant. For example, the
computation time of a compound across a lipid membrane using AAMD is
approximated to be $10^{5}$ CPU hours 16. The atomic level resolution in turn
severely restricts the system size that can be modeled, which thus limits the
number of compounds per study only up to 10 compounds 11.

To address this issue, coarse graining (CG) of all-atom system for computation
speedup is widely adopted 17, 18. By lumping several atoms into a so-called
superatom or CG bead, CG mapping can considerably reduce the degree of freedom
of a biological system, and thus allows for simulation of larger systems and
longer-lasting bioprocess. One noteworthy example is the Martini CG approach,
which has proved very successful in modeling of biological systems 19, 20.
Notably, it has been showed that using Martini CG techniques, the speedup is
about $10^{6}$ compared to AAMD simulations, which leads to an exploration of
about 500,000 molecules in study of drug-membrane permeability 11.

However, the exhaustive simulations using CGMD alone are still daunting since
the combination is exponentially increased when longer beads are considered.
Nonetheless, CGMD simulations have fostered big data for DMI problems in
recent years 21, which makes it very suitable for data-driven studies using
advanced machine learning (ML) or deep learning (DL) algorithms. The usage of
DL in drug-membrane interactions in turn enables the exploration in a much
larger chemical space. Yet, an accurate and effective deep learning framework
to take advantage of Martini CGMD simulations of DMI is still limited. The
difficulty comes from effective feature representation of Martini drugs and
membranes, as feature representation severely affects DL model performance 22.
Using SMILES string 23 for organic molecules from CGMD simulation losses the
one-to-one mapping characteristic. It is because in coarse graining many
different organic molecules can map to the same Martini CG molecules, and
thereby sharing the same PMF profiles. This physically sensible rule results
in poor performances of ML/DL models using SMILES as the molecular feature.
For example, in a study 24 using DNN model and SMILES as features for
prediction of free energy barrier ($\Delta G$), the reported coefficient of
determination or $R^{2}$ score was limited to 0.69.

To address this difficulty, we develop an accurate and highly flexible deep
learning framework, i.e., DMInet for DMI study. It can uniquely encode Martini
CG drugs and multi-membranes, and thus is potential for high throughput
screening across a vast chemical space. In addition, since it considers multi-
membranes, insights are endowed by this model on membrane selectivity in
design of potent drugs. Last but not least, it is highly flexible in its
architecture. New CG molecules, membrane types, and/or other inputs can be
easily feed into the network.

Figure 2: Schematic of the DMInet architecture, which receives three inputs:
the Martini CG molecule, the membrane and the spatial distance along the axis
of the membrane thickness.

##  2 Results

###  2.1 DMInet Architecture

As illustrated in Figure 2, the proposed DMInet model receives three distinct
inputs: the image-based Martini CG molecules, the integer-labeled membrane
types, and the numerical spatial distance along the membrane thickness. In
general, various operations can be applied on each of them before the
concatenation of the temporal outputs into a complete deep neural network
(Figure 2). That is, convolutional neural network (CNN) is employed for the
image-based CG molecules, while deep neural network (DNN) is applied for the
membrane type and the spatial distance.

Subsequent to these neural networks, the temporary outputs are then flattened
and concatenated into a dense layer which is followed by hidden layers and an
ultimate output layer denoting the PMF values. In this way, the PMF profile
with structural resolutions can be predicted for drug-membrane interactions
and membrane selectivity.

The architecture of DMInet has high flexibility in that the layers of the CNN
and DNN models in three components can be tuned for sake of accuracy as per
the quantity and quality of data at hand. Namely, the number of convolutional
layers in the CNN model and the number of dense layers in the two DNN models
can be set accordingly. Furthermore, it can be simplified to be a combination
of a single CNN model and a DNN model depending on the number of inputs of the
problem being studied. The flexibility of this DL framework allows for not
only drug-membrane interactions, but also wider applications to other problems
involving multi-component interactions. Here, for the sake of accuracy and
robustness of the model given the amount of data points in the database, only
CNN-related operations are carried out.

In building the CNN part of the DMInet using the Martini CG molecules, the
convolutional operation followed by max pooling and batch normalization were
applied. The output of CNN-model were flattened and concatenated with the
other two inputs. For detailed convolutional operations and the final model
architecture, readers are referred to the Supporting Information.

Figure 3: Comparison of DMInet predictions and true PMF profiles of six unseen
drug-membrane pairs.

###  2.2 Predictions of the PMF of Unseen Drug-like Molecules Across Membranes

DMInet is trained on a training dataset with $80\%$ of the total dataset,
whereas the remaining $20\%$ data are not seen by DMInet. Its prediction
ability is tested on this unseen data, which contains 126 drug-membrane data
points. The training hyperparameter settings and model training information
can be found in the Material and Model Section and the Supporting Information.

Figure 3 gives the comparison of the PMF profile from DMInet prediction and
the ground truth of six randomly selected samples. The complete comparison of
all other test samples can be found in the Supporting Information. The
distribution of the $R^{2}$ correlation score on the prediction is given in
Figure 4a. One can see that DMInet can predict the whole PMF profile with
reasonable accuracy. With this predicted PMF profile, insights are offered on
whether a certain drug is energetically favorable or unfavorable to permeate
across a lipid bilayer. Additionally, for a given drug-like molecule, the PMF
profile of permeating across different membranes can grant information of
membrane selectivity, which is of great practical significance in drug
discovery.

Using the predicted PMF profile, the free energy barriers can be extracted of
drug-like molecules across lipid membranes, which is plotted in Figure 4b. A
$R^{2}$ score of 0.98 is obtained using DMInet, which significantly
outperforms the DNN model using the SMILES string as feature representation
24. It demonstrates the advantages of using Martini CG as molecular
representation than the SMILES strings of molecules.

Figure 4: Evaluations of DMInet on the unseen test dataset. a: the $R^{2}$
distribution of the correlations between DMInet predictions and ground truth
on the 126 unseen data; b: the comparison of DMInet prediction and ground
truth on the free energy barrier for the total dimers.

###  2.3 Predictions of the PMF of An External Trimers Data

In addition to the interpolation test of DMInet as showed in the previous
example, the extrapolation test using the trimers permeating across the DOPC
membrane 25 is conducted. The objective of this test is twofold. First, it is
meant to show that DMInet can accommodate more complex Martini CG molecules
than dimers. Second, it is a way to test the generalization ability of DMInet
for extrapolation.

Note that in this external data 25, only the free energy barrier values of 694
linear trimers are available rather than the whole PMF profile. DMInet can
predict the whole PMF profile, which is then processed to extract the free
energy barrier values. The $R^{2}$ correlation by DMInet predictions of this
test is given in Figure 5a. One can see that the performance of this
extrapolation test is worse than that of the previous example. However, the
degradation of the $R^{2}$ and MAE in prediction is not unexpected. From the
perspective of machine learning, the extrapolation ability of general ML
models are not guaranteed since the extrapolated data may not fall into the
same range as the training data (dimers only). This is also expected from the
physical chemistry of the Martini CG molecules. For molecules with longer
beads, the range of $\Delta G$ is larger than that of shorter beads 22.

To showcase the ability of DMInet on membrane selectivity, a randomly selected
Martini CG molecule (N0-C1-C4) is adopted for PMF profile prediction across
six membranes. The predicted PMF profile is given in Figure 5b. As can be seen
from this result, DMInet can differentiate across various membrane types. It’s
shown that DMInet can be a powerful tool for drug discovery with membrane
selectivity.

This test shows that DMInet can allow for predictions of complex Martini CG
molecules, which holds the potential for exploration in a more diversified
chemical space. However, for sake of accuracy, the extrapolation test should
be carefully done and interpreted. One reasonable approach, on the other hand,
is to sample uniform and representative data of CG molecules, and then apply
DMInet for reasonable interpolations. As a result, the computational cost can
be reduced.

Figure 5: Model prediction on a trimer dataset. a: the prediction of the free
energy barrier for the trimers; b: PMF profile of Martini CG molecule
(N0-C1-C4) across six membranes.

##  3 Discussion

In this work, a deep-learning framework, namely DMInet has been proposed for
drug-membrane interactions with the ability to differentiate distinct
membranes. DMInet uses the Martini CG beads as the molecular representation of
drugs, and so its reliability and accuracy depend on the Martini CG procedure
and the transferrability of the CG approach. Since the Martini CG forcefield
adopts a top-down approach to reproduce the partitioning free energies 19, it
has very good transferrability and has been widely employed for CG modeling of
biomolecules such as lipids and proteins 20, 26. The efficiency of DMInet is
mainly dependent on the efforts for CG mapping which can be easy or difficult
as per the complexity of the molecules.

While DMInet has demonstrated great potential in study of drug-membrane
interactions in accelerated exploration of drug candidates among a larger
chemical space for drug discovery, one may note there are several limitations
in this work. For example, the molecules in the database is limited to Martini
CG dimers and membranes are confined to only six types. Additionally, only the
linear Martini CG molecules are considered while the nonlinear ones, e.g. ring
molecules, are not studied. However, one should note that all these
limitations are inherited either from the database (only linear Martini dimers
are contained) or from the Automartini package 27 because of the difficulty in
converting complex molecules into corresponding Martini CG molecules, rather
than associated with the DL framework of DMInet. In fact, DMInet is very
general and has no difficulties to accommodate for complex molecules. The
process would be trivial as all key points are introduced.

As CGMD simulations have been widely adopted for high throughput screening in
drug discovery, more and more data will be available. Therefore, it is very
beneficial to apply DL based methods to study these problems. A DL method can
in turn reduces the chemical space for sampling using physic-based simulations
as DL models can have exceptional interpolation capability. However, a vital
point in applying DL methods to drug related problems is the molecular
representation of organic molecules which heavily impacts on the performances
of ML/DL models. For example, in a study 22 using DNN for $\Delta G$
prediction, the reported $R^{2}$ was limited to 0.69 while in this work the
reported $R^{2}$ is as high as $0.98$. The deficiency comes from the molecular
representation as using SMILES string notation for organic molecules from CGMD
simulation losses the one-to-one mapping characteristic. Conversely, DMInet
naturally retains the one-to-one mapping advantages by using the Martini CG
molecules as features. As a result, it is superior of DMInet to make more
accurate predictions.

In addition, DMInet is very flexible in that the number of inputs and ML
models used for processing of each input can be varied. For example, the
spatial distance (the 3rd input) can be replaced with other quantities, e.g.,
drug concentration or membrane thickness, to build more complex models with
more diversified inputs. This flexibility is one of the advantages associated
with DMInet. Moreover, it has the potential to improve itself once more
computational data is available. One of the future directions the authors are
focusing on is to collect computational data from cholesterol-rich membranes
and normal membranes to empower DMInet for membrane selectivity problems in
design of antibacterial drug.

##  4 Conclusions

In summary, we presented a novel deep learning framework (DMInet) which has
been showed effective to study drug-membrane interactions and membrane
selectivity. DMInet is designed as an effective framework to take advantage of
Martini CGMD simulations for accurate PMF predictions of drug-like molecules
across membranes. DMInet holds the promise for accelerated drug discovery as
more and more data are becoming available. We hope this framework can be
helpful in drug design and serve as a basis for development of more powerful
deep learning platforms to tackle more complex biochemical problems.

##  5 Material and Model

###  5.1 Database and Feature representation

The dataset for ML model development in this work is from large scale CGMD
simulations using Martini force field 21. This dataset contains all possible
dimers (CG representation of organic molecules by 2 CG beads) of polar, apolar
and nonpolar types. The enumeration of these 14 unique single beads gives rise
to 105 CG dimers. In addition, the dataset also includes 6 different lipid
membranes, i.e., DAPC (1,2-diarachidonoyl-sn-glycero-3-phosphocholine), DIPC
(1,2-dilinoleoyl-sn-glycero-3-phosphocholine), DLPC (1,2-dilauroyl-sn-
glycero-3-phosphocholine), DOPC (1,2-dioleoyl-sn-glycero-3-phosphocholine),
DPPC (1,2-dipalmitoyl-sn-glycero-3-phosphocholine), and POPC
(1-palmitoyl-2-oleoyl-sn-glycero-3-phosphocholine) membranes. The combination
of 105 CG molecules and 6 membranes results in 630 unique data points of drug-
membrane pairs. In this dataset, the PMF profile was obtained by umbrella
sampling technique 28 along the direction of membrane thickness (4.1 nm) with
a step size of 0.1 nm. Therefore, the database containing CG molecules,
different membranes and PMF profiles is very suitable for the purpose of this
study on membrane permeability and membrane selectivity in drug-membrane
interactions.

Figure 6: Feature representation of CG beads. a: the adjacency matrix for the
C1-C5 bead; b: the degree matrix for the C1-C5 bead.

To develop a ML model, the features (CG molecules and membranes in this work)
should be numerically encoded so that they can be feasibly applied to ML
models as inputs. For the representation of CG molecules or dimers, the idea
of graph representation was employed. A square matrix of 14 in width by 14 in
height is applied for Martini CG molecules, which is the summation of an
adjacency matrix and a degree matrix 29 (Figure 6) so that each CG molecule
can be uniquely represented. For more detailed information, readers are
referred to the Supporting Information. Note that this representation can
capture the symmetry of the linear Martini molecules. In addition, this
representation of the CG molecule is general and can be applied for longer
Martini molecules and even ring molecules.

The membrane type can be easily encoded using one-hot encoding by a vector
with a length of 6. However, considering the limit of the data and ML model
performance, in this work they are treated as categorical variables. Namely,
an integer is assigned to each of them as DAPC=0, DIPC=1, DLPC=2, DOPC=3,
DPPC=4, and POPC=5. The spatial distances are numerical numbers and can be
represented by a vector.

###  5.2 ML model development

In developing the DMInet, a train/test split ratio of 0.8/0.2 was applied so
that the training dataset can be used for model development and the unseen
test dataset can be saved for model evaluation. In training the model using
the training dataset, the checkpoints and early stopping methods were applied
so that the training process can be automatically ceased when the best
possible model is obtained. As a result, the overfitting possibility can be
largely reduced. The model was developed under the Tensorflow platform 30 in
which all aforementioned modules and techniques can be feasibly implemented.
For more detailed information of the hyperparameters used and learning curves,
readers are referred to the Supporting Information.

Mathematically, the DMInet seeks to find an approximation function
$f:(\mathbb{R}^{d_{1}\times
d_{1}},\mathbb{R}^{d_{2}},\mathbb{R}^{d_{3}})\mapsto\mathbb{R}^{d}$ where
$d_{1}$, $d_{2}$, and $d_{3}$ are respectively the dimension of the image,
membrane and spatial distance vector, and $d$ is the dimension of the PMF
profile. The training objective is to find the weights and biases of DMInet by
minimizing the loss or cost function, which is defined as the mean squared
error (MSE) of the predicted and true PMF values:

$\mathcal{L}(w,b)=\frac{1}{m}\sum^{m}_{i=1}(y_{i}-\hat{y}_{i})^{2}$ (1)

where $y_{i}$ and $\hat{y}_{i}$ are the true label and predicted values. $m$
is the number of training samples.

The evaluation metric for model performance is the $R^{2}$ score:

$R^{2}=1-\frac{\sum^{n}_{i=1}(y_{i}-\hat{y}_{i})^{2}}{\sum^{n}_{i=1}(y_{i}-\frac{1}{n}\sum^{n}_{j=1}y_{j})^{2}}$
(2)

and mean absolute error (MAE) of $\Delta G$:

$\text{MAE}=\frac{1}{n}\sum^{n}_{i=1}|\Delta G_{i}-\Delta\hat{G}_{i}|$ (3)

where $n$ is the number of test samples.

## References

  * Seydel and Wiese 2009 Seydel, J. K.; Wiese, M. _Drug-membrane interactions: analysis, drug distribution, modeling_ ; John Wiley & Sons, 2009; Vol. 15
  * Lucio et al. 2010 Lucio, M.; Lima, J.; Reis, S. Drug-membrane interactions: significance for medicinal chemistry. _Curr. Med. Chem._ 2010, _17_ , 1795–1809
  * Fisher et al. 2017 Fisher, R. A.; Gollan, B.; Helaine, S. Persistent bacterial infections and persister cells. _Nat. Rev. Microbiol._ 2017, _15_ , 453
  * Hart et al. 2019 Hart, E. M.; Mitchell, A. M.; Konovalova, A.; Grabowicz, M.; Sheng, J.; Han, X.; Rodriguez-Rivera, F. P.; Schwaid, A. G.; Malinverni, J. C.; Balibar, C. J., et al. A small-molecule inhibitor of BamA impervious to efflux and the outer membrane permeability barrier. _Proc. Natl. Acad. Sci. U.S.A._ 2019, _116_ , 21748–21757
  * Avesar et al. 2017 Avesar, J.; Rosenfeld, D.; Truman-Rosentsvit, M.; Ben-Arye, T.; Geffen, Y.; Bercovici, M.; Levenberg, S. Rapid phenotypic antimicrobial susceptibility testing using nanoliter arrays. _Proc. Natl. Acad. Sci. U.S.A._ 2017, _114_ , E5787–E5795
  * Hurdle et al. 2011 Hurdle, J. G.; O’neill, A. J.; Chopra, I.; Lee, R. E. Targeting bacterial membrane function: an underexploited mechanism for treating persistent infections. _Nat. Rev. Microbiol._ 2011, _9_ , 62–75
  * Kim et al. 2018 Kim, W.; Zhu, W.; Hendricks, G. L.; Van Tyne, D.; Steele, A. D.; Keohane, C. E.; Fricke, N.; Conery, A. L.; Shen, S.; Pan, W., et al. A new class of synthetic retinoid antibiotics effective against bacterial persisters. _Nature_ 2018, _556_ , 103–107
  * Epand and Vogel 1999 Epand, R. M.; Vogel, H. J. Diversity of antimicrobial peptides and their mechanisms of action. _Biochim. Biophys. Acta Biomembr._ 1999, _1462_ , 11–28
  * Tossi et al. 2000 Tossi, A.; Sandri, L.; Giangaspero, A. Amphipathic, $\alpha$-helical antimicrobial peptides. _Pept. Sci._ 2000, _55_ , 4–30
  * Zhao et al. 2013 Zhao, J.; Zhao, C.; Liang, G.; Zhang, M.; Zheng, J. Engineering antimicrobial peptides with improved antimicrobial and hemolytic activities. _J. Chem. Inf. Model._ 2013, _53_ , 3280–3296
  * Menichetti et al. 2019 Menichetti, R.; Kanekal, K. H.; Bereau, T. Drug–membrane permeability across chemical space. _ACS Cent. Sci._ 2019, _5_ , 290–298
  * Kim et al. 2019 Kim, W.; Zou, G.; Hari, T. P.; Wilt, I. K.; Zhu, W.; Galle, N.; Faizi, H. A.; Hendricks, G. L.; Tori, K.; Pan, W., et al. A selective membrane-targeting repurposed antibiotic with activity against persistent methicillin-resistant Staphylococcus aureus. _Proc. Natl. Acad. Sci. U.S.A._ 2019, _116_ , 16529–16534
  * 13 Martini: coarse grain forcefield for biomolecules. http://cgmartini.nl/index.php, Accessed: 09-20-2020
  * Curtarolo et al. 2013 Curtarolo, S.; Hart, G. L.; Nardelli, M. B.; Mingo, N.; Sanvito, S.; Levy, O. The high-throughput highway to computational materials design. _Nat. Mater._ 2013, _12_ , 191–201
  * Mounet et al. 2018 Mounet, N.; Gibertini, M.; Schwaller, P.; Campi, D.; Merkys, A.; Marrazzo, A.; Sohier, T.; Castelli, I. E.; Cepellotti, A.; Pizzi, G., et al. Two-dimensional materials from high-throughput computational exfoliation of experimentally known compounds. _Nat. Nanotechnol._ 2018, _13_ , 246–252
  * Menichetti et al. 2017 Menichetti, R.; Kanekal, K. H.; Kremer, K.; Bereau, T. In silico screening of drug-membrane thermodynamics reveals linear relations between bulk partitioning and the potential of mean force. _J. Chem. Phys._ 2017, _147_ , 125101
  * Klein and Shinoda 2008 Klein, M. L.; Shinoda, W. Large-scale molecular dynamics simulations of self-assembling systems. _Science_ 2008, _321_ , 798–800
  * Flores et al. 2012 Flores, S. C.; Bernauer, J.; Shin, S.; Zhou, R.; Huang, X. Multiscale modeling of macromolecular biosystems. _Briefings Bioinf._ 2012, _13_ , 395–405
  * Marrink et al. 2007 Marrink, S. J.; Risselada, H. J.; Yefimov, S.; Tieleman, D. P.; De Vries, A. H. The MARTINI force field: coarse grained model for biomolecular simulations. _J. Phys. Chem. B_ 2007, _111_ , 7812–7824
  * Monticelli et al. 2008 Monticelli, L.; Kandasamy, S. K.; Periole, X.; Larson, R. G.; Tieleman, D. P.; Marrink, S.-J. The MARTINI coarse-grained force field: extension to proteins. _J. Chem. Theory Comput._ 2008, _4_ , 819–834
  * Hoffmann et al. 2020 Hoffmann, C.; Centi, A.; Menichetti, R.; Bereau, T. Molecular dynamics trajectories for 630 coarse-grained drug-membrane permeations. _Sci. Data_ 2020, _7_ , 1–7
  * Chen et al. 2020 Chen, G.; Shen, Z.; Iyer, A.; Ghumman, U. F.; Tang, S.; Bi, J.; Chen, W.; Li, Y. Machine-Learning-Assisted De Novo Design of Organic Molecules and Polymers: Opportunities and Challenges. _Polymers_ 2020, _12_ , 163
  * Weininger 1988 Weininger, D. SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules. _J. Chem. Inform. Comput. Sci._ 1988, _28_ , 31–36
  * Chen et al. 2020 Chen, G.; Shen, Z.; Li, Y. A machine-learning-assisted study of the permeability of small drug-like molecules across lipid membranes. _Phys. Chem. Chem. Phys._ 2020, _22_ , 19687–19696
  * Hoffmann et al. 2019 Hoffmann, C.; Menichetti, R.; Kanekal, K. H.; Bereau, T. Controlled exploration of chemical space by machine learning of coarse-grained representations. _Phys. Rev. E_ 2019, _100_ , 033302
  * López et al. 2013 López, C. A.; Sovova, Z.; van Eerden, F. J.; de Vries, A. H.; Marrink, S. J. Martini force field parameters for glycolipids. _J. Chem. Theory Comput._ 2013, _9_ , 1694–1708
  * Bereau and Kremer 2015 Bereau, T.; Kremer, K. Automated parametrization of the coarse-grained Martini force field for small organic molecules. _J. Chem. Theory Comput._ 2015, _11_ , 2783–2791
  * Kästner 2011 Kästner, J. Umbrella sampling. _Wiley Interdiscip. Rev.: Comput. Mol. Sci._ 2011, _1_ , 932–942
  * Shmilovich et al. 2020 Shmilovich, K.; Mansbach, R. A.; Sidky, H.; Dunne, O. E.; Panda, S. S.; Tovar, J. D.; Ferguson, A. L. Discovery of Self-Assembling $\pi$-Conjugated Peptides by Active Learning-Directed Coarse-Grained Molecular Simulation. _J. Phys. Chem. B_ 2020, _124_ , 3873–3891
  * Abadi et al. 2016 Abadi, M.; Barham, P.; Chen, J.; Chen, Z.; Davis, A.; Dean, J.; Devin, M.; Ghemawat, S.; Irving, G.; Isard, M., et al. Tensorflow: A system for large-scale machine learning. 12th $\\{$USENIX$\\}$ symposium on operating systems design and implementation ($\\{$OSDI$\\}$ 16). 2016; pp 265–283


# Machine-learning-assisted construction of appropriate rotating frame

Yoshihiro Michishita<EMAIL_ADDRESS>RIKEN Center for Emergent
Matter Science (CEMS), Wako, Saitama 351-0198, Japan

###### Abstract

Machine learning with neural networks is now becoming a more and more powerful
tool for various tasks, such as natural language processing, image
recognition, winning the game, and even for the issues of physics. Although
there are many studies on the application of machine learning to numerical
calculation and the assistance of experimental detection, the methods of
applying machine learning to find the analytical method are poorly studied. In
this letter, we propose methods to use machine learning to find the analytical
methods. We demonstrate that the recurrent neural networks can “derive” the
Floquet-Magnus expansion just by inputting the time-periodic Hamiltonian to
the neural networks, and derive the appropriate rotating frame in the
periodically-driven system. We also argue that this method is also applicable
to finding other theoretical frameworks in other systems.

††preprint: APS

Introduction. – Machine learning with neural networks (NN) is now becoming a
powerful tool for various tasks, such as natural language processing[1], image
recognition[2, 3, 4], and winning the game[5]. As for the issues of physics,
machine learning can be used for phase detection[6, 7, 8, 9, 10, 11], solving
the equilibrium state[12, 13, 14] or steady state[15], materials
informatics[16, 17, 18, 19] and noise reduction of the experimental
measurement results[20]. While experimental detection and numerical
calculation with the aid of machine learning are now making great progress,
one might have a natural question:

Can we develop theoretical analysis methods with the aid of machine learning?

Before tackling this question, let us consider how we ourselves have developed
the theoretical analysis methods so far. One of the central techniques is
scale separation, which leads to the reduction and the perturbation theory. A
typical example is a derivation of the Langevin equation, in which, by
utilizing the time-scale separation between the microparticles’ motion and the
Brownian particles’ motion, we perform the reduction of the degrees of freedom
of the microparticles and get the stochastic equation of motion.[21] Other
examples are the dimensional reduction of nonlinear dynamical systems[22], the
derivation of the Heisenberg model from the Hubbard model in half-filling and
large interaction limit[23], the renormalization group methods[24], and so on.
However, in general, it is a non-trivial problem to find and separate the fast
process and the slow process of the system, and we have to perform an
appropriate unitary transformation or projection and get the frame in which
the scale separation is apparent. [25, 26, 27, 28, 29, 30, 31, 32]

In periodically-driven systems, we introduce an appropriate rotating frame
(RF) associated with the time-periodic unitary transformation and separate
fast and slow modes. It is known that, under high-frequency driving where the
frequency is larger enough than the energy scale of the system, the system
stays the Floquet prethermalized before going to the infinite temperature
state[33, 34, 35], and we can engineer the desired state in the Floquet
prethermalized state.[36, 37, 38, 39, 40, 41, 42, 43] In the high-frequency
regime, we can construct an appropriate RF with high-frequency expansion.[44]
In such an appropriate RF, the effective static Hamiltonian describes the
Floquet prethermalized state, and the dressed driving term describes the
heating rate.[45, 44] Therefore, finding an appropriate unitary transformation
or projection and a scale separation is highly beneficial, while it is usually
difficult.

In this letter, we propose a method to search for the appropriate frame with
desirable properties using machine learning techniques. Our method has the
advantage that it is enough to set the desirable properties as a loss function
and, thus, should be versatile. Moreover, our method has a possibility of
finding “implicit” scale separation, which is difficult for the human to find,
and the related appropriate frame. As a first step, in this letter, we
demonstrate that machine learning can derive the appropriate rotating frame
when there is an apparent scale separation. As concrete systems, we focus on a
periodically-driven system under high-frequency driving and its appropriate
rotating frame.

We first show the concrete procedure to construct the RF in periodically-
driven systems with the aid of machine learning and demonstrate it for a
simple model. Then, we argue how we can generalize the above procedure to
derive an appropriate unitary transformation under which we can easily solve
the issues of physics. Our results show that machine learning is a powerful
tool also in exploring theoretical analysis methods.

Concrete procedure to construct the rotating frame with the aid of machine
learning. – The procedures to derive the RF are summarized in Figure. 1, and
it can be divided into two steps. Below, we first review the Floquet theory
and the RF shortly, and then explain each step.

Figure 1: Schematic picture of machine-learning-assisted construction of
appropriate rotating frame.

We consider the time-dependent Hamiltonian as

$\displaystyle\hat{H}(t)=\hat{H}_{0}+\hat{V}(t),$ (1)

where $\hat{H}_{0}$ is the system Hamiltonian without the driving and
$\hat{V}(t)$ is the driving term which depends on time. Under the time-
dependent unitary transformation, the Schrödinger equation
$i\frac{d}{dt}\ket{\psi(t)}=\hat{H}(t)\ket{\psi(t)}$ can be written as

$\displaystyle i\frac{d}{dt}\ket{\tilde{\psi}(t)}$ $\displaystyle\equiv$
$\displaystyle
i\frac{d}{dt}\hat{U}(t)\ket{\psi(t)}=\hat{H}_{r}(t)\ket{\tilde{\psi}(t)},$ (2)
$\displaystyle\hat{U}(t)$ $\displaystyle=$
$\displaystyle\exp\Bigl{[}i\hat{K}(t)\Bigr{]},$ (3)
$\displaystyle\hat{H}_{r}(t)$ $\displaystyle\equiv$
$\displaystyle\hat{U}(t)\hat{H}(t)\hat{U}^{\dagger}(t)+i\bigl{(}\frac{d}{dt}\hat{U}(t)\bigr{)}\hat{U}^{\dagger}(t)$
(4) $\displaystyle=$
$\displaystyle\hat{U}(t)\Bigl{(}\hat{H}(t)-i\frac{d}{dt}\Bigr{)}\hat{U}^{\dagger}(t),$
(5)

where we call $\hat{H}_{r}(t)$ as the dressed Hamiltonian, which describes the
dynamics of $\ket{\tilde{\psi}(t)}$ from the reference frame of $\hat{U}(t)$.

When we focus on the periodically-driven systems, where
$\hat{H}(t)=\hat{H}(t+T)$ and $\hat{V}(t)=\hat{V}(t+T)$ are satisfied, it is
known that there is the periodic time-dependent unitary transformation
$\hat{U}_{F}(t)=\hat{U}_{F}(t+T)$, which make the dressed Hamiltonian time-
independent, that is, $\hat{H}_{r}(t)=\hat{H}_{F}$. We call $\hat{H}_{F}$
Floquet Hamiltonian, $\ket{\tilde{\psi}(t)}=\hat{U}_{F}(t)\ket{\psi(t)}$,
whose dynamics is described by the Floquet Hamiltonian, the exact rotating
frame, and $\hat{K}(t)$ the micromotion operator or the kick operator. In the
high-frequency regime, we can construct this exact rotating frame
perturbatively with high-frequency expansion such as the Floquet-Magnus
expansion[33, 34] and van-Vleck expansion[36, 46]. If we calculate up to the
finite order $\mathcal{O}(1/\Omega^{n})$, the dressed Hamiltonian slightly
depends on time by $\mathcal{O}(1/\Omega^{n+1})$. Fortunately, when we
calculate the heating rate of the system under the high-frequent driving, it
is more beneficial to calculate the high-frequency expansion up to the second
or third order [44] and we call this benefitial RF an appropriate RF.

In the following, supposing that the physicists in the world do not know the
high-frequency expansion methods and the appropriate RF but want to derive the
appropriate time-periodic unitary transformation in which the dressed
Hamiltonian slightly depends on time, we “derive” the high-frequency expansion
and the appropriate RF with the aid of machine learning.

In the first step (step 1), we numerically derive the RF with the recurrent
neural networks (RNN) [47, 48, 49]. We input the model Hamiltonian $H(t)$ and
the kick operator $K(t)$, which is calculated by the NN, and get the output as
the time derivative of the kick operator $K^{\prime}(t)$. Then, we can
calculate the kick operator $K(t+\delta t)=K(t)+K^{\prime}(t)\delta t$ and the
dressed Hamiltonian $H_{r}(t)=U(t)H(t)U^{\dagger}(t)-iU(t)\\{U(t+\delta
t)-U(t)\\}/\delta t$. We set the boundary condition $K(t=0)=0$; therefore,
$K(t)$ is given at $t=0$, and we can input it into the NN. We note that, in
this letter, we use $\hat{O}$ when it describes the operator form and $O$ when
it describes the matrix form numerically. Because we want to get the RF in
which the time-dependence of the dressed Hamiltonian is small, we set the loss
function at the time step $t$ of learning as,

$\displaystyle l(t)=\sum_{n}\gamma^{n}||H_{r}(t)-H_{r}(t-n\delta t)||,$ (6)

where $||M||=\mathrm{Tr}\Bigl{[}M^{2}\Bigr{]}$ and $\gamma$ is the discount
rate, which is usually introduced in the reinforcement learning and
$0<\gamma\leq 1$. We note that we can consider step1 in Figure.1 as a
reinforcement learning in which NN as an actor decides the rotating frame at
the next time step in each turn and the cost $l(t)$ is given depending on the
action. After that, because we now have the kick operator at the next time
step $K(t+\delta t)$, we repeatedly do the same procedure until the time step
reaches $t=T$, and calculate the total loss function as

$\displaystyle l=\sum_{t}l(t)\delta t+\epsilon||\sum_{t}K^{\prime}(t)\delta
t||,$ (7)

where $\mathrm{N_{t}}=T/\delta t$ is the total time step, and update the
parameters of NN by using the back propagation. The first term in Eq.(7)
trains the NN to decrease the time dependence of the dressed Hamiltonian, and
the second term in Eq.(7) trains the NN to get the time-
periodicity.111$||\sum_{t}K^{\prime}(t)||=0$ leads to the time-periodicity of
the kick operator $K(t=0)=K(t=T)$, which further leads to the time-periodicity
of $K^{\prime}(t)$ and $H_{r}(t)$. Finally, we do the backpropagation to the
NN and update the parameters in the NN. We note that the loss function in Eq.
(7) is zero as the minimum when the RNN outputs the exact RF. We also note, in
our framework, there are two hyper-parameters $\gamma$ and $\epsilon$.
$\epsilon$ decides how the time-periodicity is emphasized and $\gamma$ must be
smaller than one for the convergence of the learning. we set
$\epsilon=20.0,\gamma=0.95$ in this letter.

In step 1, we numerically derive the kick operator $K(t)$ in the matrix form.
However, of course, this numerical form cannot be said to be a “theoretical
analysis method” because it changes as the parameter or model changes. In step
2, we physicists interpret the results as the operator form. Once we get the
kick operator and the RF in the operator form, its construction method does
not change if we change the parameters and the model.

Although it is usually challenging to find the operator form from the matrix
form, we can get some hints. In the first place, because the width of the
input and output layer is limited by the memory of the computers, we have to
use the small system, and it is comparably easy to translate the matrix form
to the operator form in small systems. Moreover, we can get the numerical
results in the different parameters, and therefore, we can find the parameter
dependence of each matrix element of the kick operator. It tells us what kind
of operator constructs each matrix element. We believe these clues are enough
for physicists to construct the kick operator in the operator form and find
the construction methods.

After deriving the operator form, we can check whether the constructed RF can
be applied to other parameters or models. 222For example, after we derive the
construction method for the appropriate rotating frame in the two spin model
in Eq.(12), we apply it to the spin-1 model by substituting $H_{0}$ and $V(t)$
of the spin-1 model, and calculate the dressed Hamiltonian and its
eigenvalues. We check whether the time-dependence of the dressed Hamiltonian
becomes smaller than the bare Hamiltonian by some order. If not, it might mean
that, due to the symmetry of the two-spin model, some parts of the operator
form are omitted, and we should derive the construction method in a more
complex model with lower symmetry.

Next, using these procedures, we demonstrate that we “derive” the high-
frequency expansion method to construct the appropriate RF in a concrete
model.

model calculation – Here we introduce the concrete model and demonstrate the
concrete procedure. For simplicity, we analyze an interacting two-spin model
under the periodic driving, which reads,

$\displaystyle\hat{H}_{0}$ $\displaystyle=$
$\displaystyle\bm{h}\cdot\sum_{i=1,2}\hat{\bm{s}}_{i}+\sum_{\alpha=x,y,z}J_{\alpha}\hat{s}^{\alpha}_{1}\hat{s}^{\alpha}_{2},$
(8) $\displaystyle\hat{V}(t)$ $\displaystyle=$
$\displaystyle\bm{\xi}\cdot\sin(\Omega t)\sum_{i}\hat{\bm{s}}_{i},$ (9)

where $\hat{s}_{i}$ represents the quantum spin operator at site $i$, $\bm{h}$
represents a static magnetic fields, $J_{\alpha}$ represents an interaction
between the two spins, and $\bm{\xi}$ represents an AC magnetic fields. For
the simplicity, we set $\bm{h}=(0,0,h_{z})$, $\bm{J}=(J_{x},0,J_{z})$, and
$\bm{\xi}=(\xi,0,0)$.

This Hamiltonian can be described by a 4*4 matrix in the basis as

$\displaystyle H_{0}$ $\displaystyle=$ $\displaystyle-
h_{z}(\tau^{0}\otimes\sigma^{z}+\tau^{z}\otimes\sigma^{0})$ (10)
$\displaystyle-
J_{x}(\tau^{x}\otimes\sigma^{x})-J_{z}(\tau^{z}\otimes\sigma^{z})$
$\displaystyle V(t)$ $\displaystyle=$ $\displaystyle-\xi\sin(\Omega
t)(\tau^{0}\otimes\sigma^{x}+\tau^{x}\otimes\sigma^{0}),$ (11)

where $\tau^{\alpha},\sigma^{\alpha}$ is the Pauli matrix about the site index
and the spin index. We vectorize and input this matrix into the NN and perform
the learning.

Next, we show the concrete results of the two-spin model in step 1. We use the
NN, in which the size of the input layer and output layer is 20(4 for the
parameters of $\hat{H}(t)$ and 16 for $\hat{K}(t)$) and 16 (16 for
$\hat{K^{\prime}}(t)$). The NN also holds the two hidden layers. We use Adam
[52] as the optimization function and set the iteration number of the learning
to 8000, and the activation function is tanh. Figure 2 shows the hidden layer
width dependence of the learning dynamics. In the single iteration, the NN
learns the single cycle of the time-periodic dynamics. In the learning
dynamics, the loss function sometimes suddenly increases, and this may stem
from the exploding gradient problem in machine learning with an RNN[53, 54].
We can see that the narrower hidden layers get more stable learning dynamics.
We note that we have calculated for some initial parameters of NN and show the
best learning dynamics in this letter. We note that the wide hidden layer
often makes the learning not converge. (see Supplementary materials (SM) for
the details[55].) The wide hidden layer can also output a too complicated kick
operator to use or for us to understand in the operator form. Therefore, the
narrow hidden layer might be suitable for our proposed procedure. Figure 3
shows that, in the RF derived by the RNN, the time-dependence of the
eigenvalues of the dressed Hamiltonian becomes smaller than that of $H(t)$ by
$10^{-2}$ order.

Figure 2: loss function evolution in the learning step and its dependence on
the width of the hidden layer. Figure 3: Reduction of the time-dependence of
the eigenvalues of the dressed Hamiltonian. The left and right panel show the
eigenvalues of $H(t)$ and $H_{r}(t)$ in the RF derived by the RNN.

Figure 4: Resulting the kick opeartor and its parameter dependence. Figure
4(a) shows the dominant terms of the resulting kick operator. Figure 4(b)-(f)
show the parameter dependence of the amplitude of each dominant term. In order
to see the parameter dependence, the amplitude is normalized by the amplitude
of certain parameters.

Then, we show the resulting RF, which is derived by the RNN in figure 4(a). We
can find three dominant contributions, which are proportional to
(i)$(\tau^{0}\otimes\sigma^{x}+\tau^{x}\otimes\sigma^{0})$,
(ii)$(\tau^{y}\otimes\sigma^{z}+\tau^{z}\otimes\sigma^{y})$, and
(iii)$(\tau^{0}\otimes\sigma^{y}+\tau^{y}\otimes\sigma^{0})$. We also check
their dependence on the parameters. From figure 4(b)-(f), we can see that each
term (i)-(iii) is proportional to $\xi\Omega^{-1}$, $\xi J_{z}\Omega^{-2}$,
and $\xi h_{z}\Omega^{-2}$. We note that the slight deviation from the
proportional relationship should stem from the higher order expansions, and we
do not focus on them in this letter although we can analyze them. Next, we can
deduce the operator form of each term. First, let us consider the first term
(i). It is independent of the parameters of $\hat{H}_{0}$ and is proportional
to the drive strength, which means it can be described by $\hat{V}(t)$.
Moreover, it seems proportional to $(1-\cos(\Omega t))/\Omega$, while
$\hat{V}(t)$ is proportional to $\sin(\Omega t)$. Therefore, we can deduce its
operator form as $\int dt^{\prime}\hat{V}(t^{\prime})$.333One might think, for
example, $\frac{1}{\Omega}(1-\frac{1}{\Omega}\frac{d}{dt}\hat{V^{\prime}}(t))$
is also a candidate. However, when we think the driving proportional to the
step function about time, this kind of definition is not proper because the
time derivative diverges. Then, we analyze the second term (ii). Because it is
proportional to $\xi J_{z}$, it should be the product of $\hat{V}(t)$ and
$J_{z}\hat{s}^{z}_{1}\hat{s}^{z}_{2}$, and actually
$(\tau^{y}\otimes\sigma^{z}+\tau^{z}\otimes\sigma^{y})=i[(\tau^{0}\otimes\sigma^{x}+\tau^{x}\otimes\sigma^{0}),\tau^{z}\otimes\sigma^{z}]/2$.
Moreover, because it is proportional to $\Omega^{-2}\sin(\Omega t)$, it should
include the double integration of time. Therefore we can deduce the second
term can be written in the operator form as $-i\int
dt_{1}\int_{0}^{t_{1}}dt_{2}[\hat{V}(t_{2}),J_{z}\hat{s}^{z}_{1}\hat{s}^{z}_{2}]$.444One
might consider other candidate such as $-2iJ_{z}\int
dt_{1}\int_{0}^{t_{1}}dt_{2}\hat{V}(t_{2})\hat{s}^{z}_{1}\hat{s}^{z}_{2}$.
However, if it is, $-2iJ_{x}\int
dt_{1}\int_{0}^{t_{1}}dt_{2}\hat{V}(t_{2})\hat{s}^{x}_{1}\hat{s}^{x}_{2}$ term
also should appear in the kick operator while in fact there is no such a term.
$J_{x}$-independence of the dominant terms implies that the kick operator
should be written in the commutator form.  By performing the same procedure
with the other term, we can reach the proposal construction of the kick
operator as

$\displaystyle\hat{K}(t)=\int dt_{1}\Bigl{(}\hat{V}(t_{1})-i\int
dt_{2}\bigl{[}\hat{V}(t_{2}),\hat{H}_{0}\bigr{]}\Bigr{)}.$ (12)

Finally, we can check whether this construction method can be applied to the
other models and conclude that this is the appropriate RF. The dashed line in
Figure 4(a) describes the kick operator in Eq. (12), which highly matches the
RNN-derived kick operator.(see the SM for the evaluation of deducing the
appropriate frame.) In fact, this derived method corresponds to the Floquet-
Magnus expansion up to the second order. Interestingly, this RNN treatment
gives us the appropriate RF, not the exact RF, in which $H_{r}(t)$ is time-
independent, even though the loss function defined in Eq. (7) becomes the
minimum at the exact RF. It should stem from the representation capability
limit of the NN because the exact RF is usually a highly complex function of
the operators. However, this limit of NN is beneficial for physicists because
the limited NN should prefer the perturbative or simple RF, and it should be
easier for us to interpret and more useful. Therefore, the narrow hidden
layers should be better, while they should have enough width to express the
perturbative expression of RF. We note that we also tried to derive the
appropriate RF in the resonant regime where the energy scale of the system is
comparable to the frequency of the driving. Approaching to the resonant
regime, the learning dynamics become unstable and the RNN rarely achieve the
appropriate RF, and we cannot get the appropriate RF at the resonant regime.
(see the detail in the SM[55].) These results should imply that, in the
resonant regime, there is no simple or perturbative construction methods of
the appropriate RF.

Generalization of the procedure to derive the appropriate unitary
transformation. – Finally, we argue how the procedure we demonstrate in the
previous section can be applied to other general problems. In the previous
section, we have defined the loss function to train the NN to give us the RF
in which the time dependence of the dressed Hamiltonian is small. This idea
can be applied to find other appropriate frames or projections, which make us
able to solve the physical problem easily. We can define the loss function as
it becomes small in the frame which has desirable properties, and then, after
the learning (if the learning works well), NN gives us the desirable frame. As
we have demonstrated, we first calculate the appropriate frame or the
projection in the matrix form in a small system, analyze the parameter
dependence, and then, we physicists should be able to translate the operator
form and develop the construction method. If the translation is difficult, we
can also utilize another reinforcement learning techniques for finding the
construction method of the appropriate frame, such as the Monte Carlo tree
search algorithm.[58, 59, 60, 5, 61, 62, 63] In this method, we update the
agent’s policy to make the final operator form almost the same as the derived
appropriate frame and get the construction method described by the expression
tree. Although this method might be able to give us the operator form of the
appropriate frame automatically, we have to define the search area of the
operation previously. 555For the case of the rotating frame in periodically-
driven systems, the commutation of the static Hamiltonian and the driving
term, the time integration, and the summation of terms are necessary as
possible operations for the above method to derive the high-frequency
expansion. However, in general cases, we usually do not know what operations
are necessary to construct the appropriate frame. In such cases, we first
deduce the necessary operation from the parameter dependence of the
appropriate frame numerically derived at step 1 and apply the above methods to
derive how to combine the operation to get the appropriate frame. Therefore,
the deduction by the physicist should be necessary independent of whether we
use the reinforcement learning techniques.

Conclusion. – In this letter, we have proposed the method to construct the
appropriate RF with the aid of RNN and demonstrate the concrete process and
the results. By defining the loss function as it becomes small when the output
RF has the desirable property, which is the smallness of the time dependence
of the dressed Hamiltonian, we can get the appropriate RF in the matrix form.
In a small system, we can translate the results to the operator form and get
the analytical method to construct the appropriate RF. Interestingly, the
proposed method gives us the appropriate RF, not the exact RF. This property
should stem from the representation capability limit of the NN, and this limit
is beneficial for physicists to interpret and utilize the results. Although
some previous works[65, 66, 67, 68] solve the dynamics or equilibrium state
using RNN, our subject and goal differ from theirs. While they focus on
solving and getting the numerical results, we aim to derive the theoretical
analysis methods. We believe our proposed method is useful even in other
systems and this work paves a new way to utilize machine learning to develop
analytical methods in physics.

Acknowledgements.– YM deeply appriciate Takashi Mori and Kaoru Mizuta for
fruitiful discussion and Ken Mochizuki and Shuntaro Sumita for their valuable
comments on the manuscript. YM also thanks Irasutoya666 Irasutoya is the
japanese free illustration site at https://www.irasutoya.com/ for allowing me
to use the illustration of the physicist in Figure 1. We use Julia[70] and its
library Flux.jl[71] for all calculation, and the codes we have used are
uploaded in the Github[72]. This work is supported by RIKEN Special
Postdoctoral Researcher Program.

## References

  * Young  _et al._ [2018] T. Young, D. Hazarika, S. Poria, and E. Cambria, IEEE Computational Intelligence Magazine 13, 55 (2018).
  * Krizhevsky  _et al._ [2012] A. Krizhevsky, I. Sutskever, and G. E. Hinton, in  _Advances in Neural Information Processing Systems_, Vol. 25, edited by F. Pereira, C. Burges, L. Bottou, and K. Weinberger (Curran Associates, Inc., 2012).
  * He  _et al._ [2016] K. He, X. Zhang, S. Ren, and J. Sun, in  _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ (2016) pp. 770–778.
  * Badue  _et al._ [2019] C. Badue, R. Guidolini, R. V. Carneiro, P. Azevedo, V. B. Cardoso, A. Forechi, L. Jesus, R. Berriel, T. Paixão, F. Mutz, L. Veronese, T. Oliveira-Santos, and A. F. De Souza, “Self-driving cars: A survey,”  (2019).
  * Silver  _et al._ [2017] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y. Chen, T. Lillicrap, F. Hui, L. Sifre, G. van den Driessche, T. Graepel, and D. Hassabis, Nature 550, 354 (2017).
  * Wang [2016] L. Wang, Phys. Rev. B 94, 195105 (2016).
  * Carrasquilla and Melko [2017] J. Carrasquilla and R. G. Melko, Nature Physics 13, 431 (2017).
  * van Nieuwenburg  _et al._ [2017] E. van Nieuwenburg, Y.-H. Liu, and S. Huber, Nature Physics 13, 435 (2017).
  * Hu  _et al._ [2017] W. Hu, R. R. P. Singh, and R. T. Scalettar, Phys. Rev. E 95, 062122 (2017).
  * Yoshioka  _et al._ [2018] N. Yoshioka, Y. Akagi, and H. Katsura, Phys. Rev. B 97, 205110 (2018).
  * Liu and van Nieuwenburg [2018] Y.-H. Liu and E. P. L. van Nieuwenburg, Phys. Rev. Lett. 120, 176401 (2018).
  * Nomura  _et al._ [2017] Y. Nomura, A. S. Darmawan, Y. Yamaji, and M. Imada, Phys. Rev. B 96, 205152 (2017).
  * Carleo and Troyer [2017] G. Carleo and M. Troyer, Science 355, 602 (2017), https://www.science.org/doi/pdf/10.1126/science.aag2302 .
  * Carleo  _et al._ [2018] G. Carleo, Y. Nomura, and M. Imada, Nature Communications 9, 5322 (2018).
  * Yoshioka and Hamazaki [2019] N. Yoshioka and R. Hamazaki, Phys. Rev. B 99, 214306 (2019).
  * Brockherde  _et al._ [2017] F. Brockherde, L. Vogt, L. Li, M. E. Tuckerman, K. Burke, and K.-R. Müller, Nature Communications 8, 872 (2017).
  * Kirkpatrick  _et al._ [2021] J. Kirkpatrick, B. McMorrow, D. H. P. Turban, A. L. Gaunt, J. S. Spencer, A. G. D. G. Matthews, A. Obika, L. Thiry, M. Fortunato, D. Pfau, L. R. Castellanos, S. Petersen, A. W. R. Nelson, P. Kohli, P. Mori-Sánchez, D. Hassabis, and A. J. Cohen, Science 374, 1385 (2021), https://www.science.org/doi/pdf/10.1126/science.abj6511 .
  * Guo  _et al._ [2022] J. Guo, Z. Chen, Z. Liu, X. Li, Z. Xie, Z. Wang, and Y. Wang, Scientific Reports 12, 15326 (2022).
  * Pederson  _et al._ [2022] R. Pederson, B. Kalita, and K. Burke, Nature Reviews Physics 4, 357 (2022).
  * Tsukamoto  _et al._ [2022] M. Tsukamoto, S. Ito, K. Ogawa, Y. Ashida, K. Sasaki, and K. Kobayashi, Scientific Reports 12 (2022), 10.1038/s41598-022-18115-w.
  * Kamenev [2004] A. Kamenev, “Many-body theory of non-equilibrium systems,”  (2004).
  * Haken [1987] H. Haken,  _Self-Organizing Systems: The Emergence of Order_, edited by F. E. Yates, A. Garfinkel, D. O. Walter, and G. B. Yates (Springer US, Boston, MA, 1987) pp. 417–434.
  * Heisenberg [1928] W. Heisenberg, Zeitschrift für Physik 49, 619 (1928).
  * Wilson [1975] K. G. Wilson, Rev. Mod. Phys. 47, 773 (1975).
  * Lee  _et al._ [1953] T. D. Lee, F. E. Low, and D. Pines, Phys. Rev. 90, 297 (1953).
  * Schrieffer and Wolff [1966] J. R. Schrieffer and P. A. Wolff, Phys. Rev. 149, 491 (1966).
  * Silbey and Harris [1984] R. Silbey and R. A. Harris, The Journal of Chemical Physics 80, 2615 (1984), https://doi.org/10.1063/1.447055 .
  * Ashida  _et al._ [2018] Y. Ashida, T. Shi, M. C. Bañuls, J. I. Cirac, and E. Demler, Phys. Rev. Lett. 121, 026805 (2018).
  * Imbrie [2016] J. Z. Imbrie, Journal of Statistical Physics 163, 998 (2016).
  * Shi  _et al._ [2020] T. Shi, E. Demler, and J. I. Cirac, Phys. Rev. Lett. 125, 180602 (2020).
  * Ashida  _et al._ [2021] Y. Ashida, A. m. c. İmamoğlu, and E. Demler, Phys. Rev. Lett. 126, 153603 (2021).
  * Ashida  _et al._ [2022] Y. Ashida, T. Yokota, A. m. c. İmamoğlu, and E. Demler, Phys. Rev. Research 4, 023194 (2022).
  * Mori  _et al._ [2016] T. Mori, T. Kuwahara, and K. Saito, Phys. Rev. Lett. 116, 120401 (2016).
  * Kuwahara  _et al._ [2016] T. Kuwahara, T. Mori, and K. Saito, Annals of Physics 367, 96 (2016).
  * Mallayya  _et al._ [2019] K. Mallayya, M. Rigol, and W. De Roeck, Phys. Rev. X 9, 021027 (2019).
  * Bukov  _et al._ [2015] M. Bukov, L. D’Alessio, and A. Polkovnikov, Advances in Physics 64, 139 (2015), https://doi.org/10.1080/00018732.2015.1055918 .
  * Eckardt [2017] A. Eckardt, Rev. Mod. Phys. 89, 011004 (2017).
  * Oka and Kitamura [2019] T. Oka and S. Kitamura, Annual Review of Condensed Matter Physics 10, 387 (2019), https://doi.org/10.1146/annurev-conmatphys-031218-013423 .
  * Zenesini  _et al._ [2009] A. Zenesini, H. Lignier, D. Ciampini, O. Morsch, and E. Arimondo, Phys. Rev. Lett. 102, 100403 (2009).
  * Jotzu  _et al._ [2014] G. Jotzu, M. Messer, R. Desbuquois, M. Lebrat, T. Uehlinger, D. Greif, and T. Esslinger, Nature 515, 237 (2014).
  * Meinert  _et al._ [2016] F. Meinert, M. J. Mark, K. Lauber, A. J. Daley, and H.-C. Nägerl, Phys. Rev. Lett. 116, 205301 (2016).
  * McIver  _et al._ [2020] J. W. McIver, B. Schulte, F.-U. Stein, T. Matsuyama, G. Jotzu, G. Meier, and A. Cavalleri, Nature Physics 16, 38 (2020).
  * Ponte  _et al._ [2015] P. Ponte, Z. Papić, F. m. c. Huveneers, and D. A. Abanin, Phys. Rev. Lett. 114, 140401 (2015).
  * Mori [2022] T. Mori, Phys. Rev. Lett. 128, 050604 (2022).
  * Mikami  _et al._ [2016] T. Mikami, S. Kitamura, K. Yasuda, N. Tsuji, T. Oka, and H. Aoki, Phys. Rev. B 93, 144307 (2016).
  * Eckardt and Anisimovas [2015] A. Eckardt and E. Anisimovas, New Journal of Physics 17, 093039 (2015).
  * Rumelhart  _et al._ [1986] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, Nature 323, 533 (1986).
  * Elman [1990] J. L. Elman, Cognitive Science 14, 179 (1990).
  * Werbos [1988] P. J. Werbos, Neural Networks 1, 339 (1988).
  * Note [1] $||\sum@\slimits@_{t}K^{\prime}(t)||=0$ leads to the time-periodicity of the kick operator $K(t=0)=K(t=T)$, which further leads to the time-periodicity of $K^{\prime}(t)$ and $H_{r}(t)$.
  * Note [2] For example, after we derive the construction method for the appropriate rotating frame in the two spin model in Eq.(12), we apply it to the spin-1 model by substituting $H_{0}$ and $V(t)$ of the spin-1 model, and calculate the dressed Hamiltonian and its eigenvalues. We check whether the time-dependence of the dressed Hamiltonian becomes smaller than the bare Hamiltonian by some order. If not, it might mean that, due to the symmetry of the two-spin model, some parts of the operator form are omitted, and we should derive the construction method in a more complex model with lower symmetry.
  * Kingma and Ba [2014] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,”  (2014).
  * Bengio  _et al._ [1994] Y. Bengio, P. Simard, and P. Frasconi, IEEE Transactions on Neural Networks 5, 157 (1994).
  * Pascanu  _et al._ [2013] R. Pascanu, T. Mikolov, and Y. Bengio, in  _Proceedings of the 30th International Conference on Machine Learning_, Proceedings of Machine Learning Research, Vol. 28, edited by S. Dasgupta and D. McAllester (PMLR, Atlanta, Georgia, USA, 2013) pp. 1310–1318.
  * [55] See Supplemental Material http://link.aps.org/supplemental/ for (i) hidden layer width dependence of the learning dynamics; (ii) evaluation of deducing the operator form; (iii) derivation of the third-order Floquet-Magnus expansion; (iv) quest for the appropriate rotating frame under the resonant drive.
  * Note [3] One might think, for example, $\frac{1}{\Omega}(1-\frac{1}{\Omega}\frac{d}{dt}\hat{V^{\prime}}(t))$ is also a candidate. However, when we think the driving proportional to the step function about time, this kind of definition is not proper because the time derivative diverges.
  * Note [4] One might consider other candidate such as $-2iJ_{z}\intop\ilimits@dt_{1}\intop\ilimits@_{0}^{t_{1}}dt_{2}\hat{V}(t_{2})\hat{s}^{z}_{1}\hat{s}^{z}_{2}$. However, if it is, $-2iJ_{x}\intop\ilimits@dt_{1}\intop\ilimits@_{0}^{t_{1}}dt_{2}\hat{V}(t_{2})\hat{s}^{x}_{1}\hat{s}^{x}_{2}$ term also should appear in the kick operator while in fact there is no such a term. $J_{x}$-independence of the dominant terms implies that the kick operator should be written in the commutator form.
  * Coulom [2007] R. Coulom, in  _Computers and Games_ , edited by H. J. van den Herik, P. Ciancarini, and H. H. L. M. J. Donkers (Springer Berlin Heidelberg, Berlin, Heidelberg, 2007) pp. 72–83.
  * Hopcroft  _et al._ [2007] J. E. Hopcroft, R. Motwani, and J. D. Ullman,  _Introduction to automata theory, languages, and computation, 3rd Edition_ , Pearson international edition (Addison-Wesley, 2007).
  * Kusner  _et al._ [2017] M. J. Kusner, B. Paige, and J. M. Hernández-Lobato, in  _Proceedings of the 34th International Conference on Machine Learning_, Proceedings of Machine Learning Research, Vol. 70, edited by D. Precup and Y. W. Teh (PMLR, 2017) pp. 1945–1954.
  * Lample and Charton [2020] G. Lample and F. Charton, in  _International Conference on Learning Representations_ (2020).
  * Mundhenk  _et al._ [2021] T. Mundhenk, M. Landajuela, R. Glatt, C. P. Santiago, D. faissol, and B. K. Petersen, in  _Advances in Neural Information Processing Systems_, Vol. 34, edited by M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (Curran Associates, Inc., 2021) pp. 24912–24923.
  * Sun  _et al._ [2023] F. Sun, Y. Liu, J.-X. Wang, and H. Sun, in  _The Eleventh International Conference on Learning Representations_ (2023).
  * Note [5] For the case of the rotating frame in periodically-driven systems, the commutation of the static Hamiltonian and the driving term, the time integration, and the summation of terms are necessary as possible operations for the above method to derive the high-frequency expansion.
  * Flurin  _et al._ [2020] E. Flurin, L. S. Martin, S. Hacohen-Gourgy, and I. Siddiqi, Phys. Rev. X 10, 011006 (2020).
  * Hibat-Allah  _et al._ [2020] M. Hibat-Allah, M. Ganahl, L. E. Hayward, R. G. Melko, and J. Carrasquilla, Phys. Rev. Res. 2, 023358 (2020).
  * Lin  _et al._ [2022] K. Lin, J. Peng, C. Xu, F. L. Gu, and Z. Lan, Journal of Chemical Theory and Computation 18, 5837 (2022), pMID: 36184823, https://doi.org/10.1021/acs.jctc.2c00702 .
  * Wu  _et al._ [2022] D. Wu, R. Rossi, F. Vicentini, and G. Carleo, “From tensor network quantum states to tensorial recurrent neural networks,”  (2022).
  * Note [6] Irasutoya is the japanese free illustration site at https://www.irasutoya.com/.
  * Bezanson  _et al._ [2017] J. Bezanson, A. Edelman, S. Karpinski, and V. B. Shah, SIAM review 59, 65 (2017).
  * Innes [2018] M. Innes, Journal of Open Source Software 3, 602 (2018).
  * [72] Please check https://github.com/YoshihiroMichishita
/julia/tree/master/RNN-RF.

Supplemental Materials for
“Machine-learning-assisted construction of appropriate rotating frame”

##  S1 S1. hidden-layer-width dependence of the learning dynamics

In figure. 2 in the main text, we have shown only successful learning
dynamics. In this section, we show the learning dynamics of 5 batches, in
which the NN is randomly initialized and the width of the hidden layers is
each 2, 4, 8, 16, 64, and 256. Within these five batches, we can see that the
learning dynamics are most stable when the width of the hidden layer is 8. As
we have noted in the main text, narrow hidden layers are proper for the stable
learning dynamics while it rarely get the appropriate RF when it is too
narrow. In figure. S2, we also show the resulting kick operator and the
dressed Hamiltonian when the learning does not work well. We note that we
vectorize a 4*4 Hermitian matrix as

$\displaystyle
M=\begin{pmatrix}\mathrm{h1}&\mathrm{h2}+i\mathrm{h3}&\mathrm{h4}+i\mathrm{h5}&\mathrm{h6}+i\mathrm{h7}\\\
\mathrm{h2}-i\mathrm{h3}&\mathrm{h8}&\mathrm{h9}+i\mathrm{h10}&\mathrm{h11}+i\mathrm{h12}\\\
\mathrm{h4}-i\mathrm{h5}&\mathrm{h9}-i\mathrm{h10}&\mathrm{h13}&\mathrm{h14}+i\mathrm{h15}\\\
\mathrm{h6}-i\mathrm{h7}&\mathrm{h11}-i\mathrm{h12}&\mathrm{h14}-i\mathrm{h15}&\mathrm{h16}\end{pmatrix}.$

In figure S2, the time-dependence of each element is shown.

Figure S1: The hidden layer width dependence of the learning dynamics. We
represent $w$ as the width of the hidden layer, and $b$ as the batch index.
The parameters are $(\xi,\Omega,J_{z},J_{x},h_{z})=(0.2,20.0,1.0,0.7,0.5)$.

Figure S2: The results in $w=64,b=1$, where the learning ends when $l\sim
10^{-2}$. In such case, the resulting kick operator has many nonzero elements,
and the resulting dressed Hamiltonian does not get the time periodicity.

##  S2 S2. Evaluation of deducing the operator form

In this section, we discuss how to evaluate the deduce. Because we use the NN
as a function approximator, the important thing is not to derive the operator
form exactly reproducing the numerical results. We want the operator form,
which roughly reproduces the numerical results(i), can be written in a simple
form(ii), and has the desirable property(iii). We have already shown that the
“derived” operator form in Eq.(12) satisfies conditions (i) and (ii). Thus we
check condition (iii), which is the small time-dependence of the dressed
Hamiltonian. Figure S3 shows the maximum value of the dressed Hamiltonian
divided by that of the bare Hamiltonian and the maximum value of the
eigenvalues divided by that of the bare Hamiltonian. Both figures show that
the ”derived” operator form can realize a smaller time-dependence than the
bare Hamiltonian by $10^{-}2$ order. Therefore, we can conclude that the
derived construction of the rotating frame is appropriate enough. This
evaluation method of the derivation of the operator form is also applicable to
general cases. Figure S3 also shows that the RNN-derived rotating frame is
comparable to the third-order Floquet-Magnus expansions and better than it in
the small frequency region $\Omega\sim 10$. We note that we can also“derive”
the Floquet-Magnus expansion up to the third order from the RNN-derived kick
operator, and discuss it in the next section.

Figure S3: check the derived construction method reproduce the numerical
results by the RNN.
The left figure shows that the max value of the dressed Hamiltonian, and the
right shows that the max value of the eigenvalues of the dressed Hamiltonian.
Both figure shows that the second order perturbation with the Floquet-Magnus
expansion realize $10^{-2}$ order compared to the bare Hamiltonian.The
parameters are $(\xi,J_{z},J_{x},h_{z})=(0.2,1.0,0.7,0.5)$.

##  S3 S3. Derivation of the third-order Floquet-Magnus expansion

When we want a more complex but less time-dependent rotating frame
construction method, we focus on the sub-dominant terms in the RNN-derived
results. Figure S4 shows the difference between the RNN-derived Kick operator
and the second-order Floquet-Magnus expansion in Eq.(12). Because we already
have reproduced the dominant terms up to the $10^{-3}$ order by the second-
order Floquet-Magnus expansion, the rest sub-dominant terms are $10^{-4}$
order. Next, we summarize these sub-dominant terms to the three matrix forms
as
$(\tau^{z}\otimes\sigma^{x}+\tau^{x}\otimes\sigma^{z})$,$(\tau^{z}\otimes\sigma^{0}+\tau^{0}\otimes\sigma^{z})$
and $(\tau^{z}\otimes\sigma^{z})$, and check the parameter dependence of them.
Figure S5(a) shows the time-dependence of the dominant and sub-dominant terms
in the kick operator, and Figure S5(b-f) shows the parameter dependence of
their amplitude. We can see that the sub-dominant terms
($(\tau^{z}\otimes\sigma^{x}+\tau^{x}\otimes\sigma^{z})$,$(\tau^{z}\otimes\sigma^{0}+\tau^{0}\otimes\sigma^{z})$
and $(\tau^{z}\otimes\sigma^{z})$) are proportional to $\xi
h_{z}(2J_{z}-J_{x})\Omega^{-3}$, $\xi^{2}h_{z}\Omega^{-3}$, and
$\xi^{2}J_{z}\Omega^{-3}$.

Figure S4: the difference between the kick operator derived by the RNN and by
Eq.(12).

Figure S5: Resulting the kick opeartor and its parameter dependence. Figure
4(a) shows the sub-dominant terms of the resulting kick operator. Figure
4(b)-(f) show the parameter dependence of the amplitude of each dominant term.
In order to see the parameter dependence, the amplitude is normalized by the
amplitude of certain parameters.

By performing the same procedures as in the main text, we can derive the
operator form of the sub dominant terms, which are

$\displaystyle-\int^{t}dt_{1}\int^{t_{1}}dt_{2}\int^{t_{2}}dt_{3}\Biggl{(}\Bigl{[}[\hat{V}(t_{3}),\hat{H}_{0}],\hat{H}_{0}\Bigr{]}+\frac{1}{2}\Bigl{[}\hat{V}(t_{2}),[\hat{V}(t_{3}),H_{0}]\Bigr{]}\Biggr{)}.$
(S2)

This expression corresponds to the third order Floquet-Magnus expansion.

##  S4 S4. Quest for the appropriate rotating frame under the resonant drive

In this section, we show the results of the resonant regime calculated by our
proposed method. First, we check the learning dynamics and its dependence on
the hidden layer width. Figure S6 shows the learning dynamics at
$\Omega=4.0,w=2$ and $\Omega=2.0$. We can see that the learning dynamics
become unstable and rarely achieve the appropriate RF as the drive approaches
resonant, and we could not achieve the appropriate RF with a small loss.
Remembering that the RNN can achieve the appropriate RF independent of the
initial condition (independent of the batch index) in figure S1, these results
should imply that the loss function of the parameter space has a single
minimum with gentle gradient when there is a scale separation (the scale
separation between $\Omega$ and the energy scale of the system), while it has
some or many local minima with steep gradient when the scale separation is
smeared.

We also show the best of the resulting kick operator and the dressed
Hamiltonian in figure S7 and S8. In the middle regime at $\Omega=4$ in figure
S7, the RNN gives us the appropriate RF which can be constructed with the
high-frequency expansion (up to $1/\Omega^{3}$), while its learning dynamics
is unstable. In the resonant regime in figure S8, the resulting kick operator
seems different from the Floquet-Magnus expansion(see and compare with figure
S7), and the resulting dressed Hamiltonian does not get the small time
dependence and the time-periodicity. This failure of the quest for the
appropriate RF in the resonant regime might imply that there is no “implicit”
scale separation and a simple construction of the appropriate RF, or there are
many sharp potentials for the training and it is difficult to reach the
appropriate RF. The improvements in the RNN framework might lead to the
discovery of the appropriate RF in the resonant regime, which is left for
future works.

Figure S6: Hidden layer width dependence of learning dynamics in the resonant
regime.$w$ represents the width of the hidden layers and $b$ represents the
batch index.

Figure S7: Best results ($w=2,b=5$) in the middle regime at $\Omega=5$

Figure S8: Best results ($w=8,b=4$) in the resonant regime at $\Omega=2$