# A case study of Generative AI in MSX Sales Copilot: Improving seller
productivity with a real-time question-answering system for content
recommendation

###### Abstract

In this paper, we design a real-time question-answering system specifically
targeted for helping sellers get relevant material/documentation they can
share live with their customers or refer to during a call. Taking the Seismic
content repository as a relatively large scale example of a diverse dataset of
sales material, we demonstrate how LLM embeddings of sellers’ queries can be
matched with the relevant content. We achieve this by engineering prompts in
an elaborate fashion that makes use of the rich set of meta-features available
for documents and sellers. Using a bi-encoder with cross-encoder re-ranker
architecture, we show how the solution returns the most relevant content
recommendations in just a few seconds even for large datasets. Our recommender
system is deployed as an AML endpoint for real-time inferencing and has been
integrated into a Copilot interface that is now deployed in the production
version of the Dynamics CRM, known as MSX, used daily by Microsoft sellers.

Keywords: Copilot, Semantic search, Prompt engineering, Question-Answering,
Cross-encoder

Manpreet Singh Ravdeep Pasricha Nitish Singh Ravi Prasad Kondapalli Manoj R
Kiran R Laurent Boué

##  1 Introduction

Generative AI has caught the imagination of the world since ChatGPT was
released by Open AI towards the end of 2022. Leading this trend, Microsoft has
announced various Copilot initiatives with the first one being the GitHub
Copilot in June 2022 to assist users of Visual Studio in code auto-completion.
Subsequently, Microsoft 365 Copilot was announced in March 2023 to convert
user input into content in M365 apps, such as Word, Excel, PowerPoint,
Outlook, Teams. Around the same time, Microsoft announced the Dynamics 365
Copilot as the world’s first Copilot in both Customer Relationship Management
(CRM) and Enterprise Resource Planning (ERP) applications bringing next-
generation AI to every line of business. The Microsoft Sales Experience team
has internally customized the Dynamics 365 Copilot for its CRM system,
popularly known as MSX and released MSX Copilot in July 2023 to select
sellers.

The goal of the MSX Copilot is to improve seller productivity and it went live
with few skills in July 2023. A skill is a subject area or knowledge or task
awareness for the MSX Sales Copilot. The objective of these skills is to
assist the sellers prepare for a meeting with a customer more efficiently. For
example, a seller should be able to ask questions to the Copilot chatbot in
natural language about any of the topics covered by the skills and get
relevant answers to their questions.

We have built and deployed the first machine learning based skill for MSX
Copilot called “MSX Content Recommender” which is the topic of this paper.
Given a seller’s query, the purpose of this skill is to return relevant
technical documentation drawn from the Seismic content repository (seismic, ).

##  2 Related work

With the democratization of transformer models (vaswani2017attention, ) like
BERT (devlin2018bert, ), RoBERTa (liu2019roberta, ), Sentence-Bert
(reimers2019sentence, ) and, more recently, Large Language Models (LLMs) like
GPT (openai2023gpt4, ), most issues involving Natural Language
Processing/Understanding have been approached via the angle of text
embeddings.

In this work we focus on semantic search which consists of finding relevant
documents from an input query which is not based on keyword matching but
rather on contextual understanding. Because of these recent advancements in
LLMs many search and chatbot applications are moving from lexical-based search
to semantic search.

Unlike classical semantic search problems where one transforms both the user
query and the content of the documents into embeddings, we are instead
considering the situation where one only has access to meta-feature data about
the documents but not their actual content.

Our strategy consists in transforming the metadata about the documents into an
English-formed sentence as explained in Section 3. This way, we lift ourselves
back to the level of asymmetric semantic search. On one side, we have short
user queries and on the other side we have longer textual description of the
documents based on their metadata. In this context, we employ a 2-stage LLM-
based architecture to solve the search problem as explained in Section 4.
Next, we provide performance evaluation both in terms of latency as well as
(human) domain experts relevancy estimation in Section 5. Finally, we conclude
with a description of how our system has been integrated as a Copilot skill
currently live in the Microsoft Sales Experience (MSX) platform in Section 6.

##  3 Metadata prompt engineering

As mentioned in the previous section, we are considering the situation where
we do not have access to the content of the documents but only to their
metadata.

In a manner similar to tabular datasets, documents are characterized by a set
of features $\mathcal{F}=\\{f_{1},\cdots,f_{n}\\}$ and any specific document
$c_{k}$ takes on specific feature values
$\mathcal{F}(c_{k})=\\{f_{1}(c_{k}),\cdots,f_{n}(c_{k})\\}$.

As an example, one may think of a categorical feature $f_{i}=\texttt{format}$
that characterizes the format of a document. Its possible values for a
specific document $c_{k}$ may be
$f_{i}(c_{k})\in\\{\texttt{PDF},\texttt{URL},\dots\\}$ depending on the
document. One may also have numerical features $f_{i}$ such that
$f_{i}(c_{k})\in\mathcal{R}$. For example, $f_{i}\equiv\texttt{number of
views}$ so that $f_{i}(c_{k})$ counts how many times document $c_{k}$ has been
viewed in the past. In this case, we introduce a user-defined function $g$
that maps numerical features into different categories depending on the
numerical value $f_{i}(c_{k})$ and on the statistics of $f_{i}$. This way,
applying $g$ to $f_{i}(c_{k})$ transforms the numerical features into finite
sets of categories. A specific implementation of $g$ will be discussed in
Section 5.3.

If we define the function $g$ to be the identity for categorical features, we
can absorb both categorical and numerical features under the same notation so
that $g\circ f_{i}(c_{k})$ always corresponds to a string value associated
with feature $f_{i}$ regardless of whether $f_{i}$ is a categorical or
numerical feature.

Given a document $c_{k}$, we concatenate each one its features $f_{i}$ with
and their string-valued representations $g\circ f_{i}(c_{k})$ so that $c_{k}$
is now translated into a “prompt” defined as

$\text{Prompt}(c_{k})\coloneqq\bigoplus_{i=1}^{n}\Big{(}f_{i}\oplus\Big{[}g\circ
f_{i}(c_{k})\Big{]}\Big{)}$ (1)

where $\oplus$ stands for string concatenation. In practice we have considered
$n\approx 20$ features. An ablation study of including or not including
numerical features with a specific implementation of $g$ is presented in
Section 5

Since all documents and their metadata are available offline, we can apply our
prompt engineering function defined in Eq. (1) to all documents and obtain a
set of prompts

$\mathcal{P}=\\{\text{Prompt}(c_{1}),\cdots,\text{Prompt}(c_{n})\\}$ (2)

In other words, we have created a new representation of all the documents in
the corpus $\mathcal{C}$ as a set of textual features $\mathcal{P}$.

##  4 Architecture of the LLM-based recommender model

Figure 1: Illustration of the overall 2-stage architecture described in
Section 4.

Given a corpus of content $\mathcal{C}=\\{c_{1},\cdots,c_{n}\\}$ (where each
$c_{i}$ is a document) and a query $q$ entered into the Copilot interface, the
goal of the model is to respond with a list of relevant documents drawn from
$\mathcal{C}$.

Classically, one of the most popular metric to evaluate the relevance of
recommender models is the so-called Normalized Discounted Cumulative Gain
(NDCG). Unfortunately this requires to have labeled data for the ground truth
relevance score of the search results. Those are not available in our case and
we discuss in the conclusion our future plans in this regard. Instead we
propose to follow a completely unsupervised approach that relies on pre-
trained publicly available LLM models. We define a 2-stage system

  1. 1.

the first stage is responsible for quickly retrieving a short list of
documents that are potentially good matches to the user query. This is
implemented via a bi-encoder model (see Section 4.1).

  2. 2.

the second stage is responsible for re-ranking the short list produced in the
first stage to have more relevant documents appearing first. This is
implemented via a cross-encoder model (see Section 4.2).

As mentioned in the previous section, the problem we tackle in this work is
that of asymmetric semantic search where a short query is used to match
against longer text. This context of asymmetric question-answering was
previously shown to be well-suited for pre-trained publicly available sentence
transformers (reimers-2020-Curse_Dense_Retrieval, ; reimers2019sentence, ;
modelURL, ) which are trained on MS MARCO dataset (bajaj2016ms, ).

###  4.1 Bi-encoder document retrieval

Figure 2: Illustration of the bi-encoder retrieval phase where the embeddings
of the metadata prompts $\mathcal{P}$, pre-computed offline, are compared to
the input query $q$ via cosine similarity. Note how the two branches operate
independently of each other and only connect after the MSMarco DistillBERT
embeddings.

We apply the DistillBERT (sanh2019distilbert, ) (pre-trained on MSMarco
dataset) to all prompts in $\mathcal{P}$ and cache the resulting set of
embeddings (modelURL, ). Note that this operation may be done completely
offline. At this stage, the prompt embeddings are computed independently from
the query embeddings. This allows for quick retrieval of the top candidates
but neglects any kind of cross-interaction between the document prompts
$\mathcal{P}$ and the search query $q$. At time of inference (i.e. when a user
is invoking a search query), we pass the query $q$ into the same model to get
the query’s embedding representation. Then we use cosine similarity between
the pre-computed prompt embeddings and this input query embedding to get the
top-$K$ documents

$\\{c_{\text{cand}_{1}},\cdots,c_{\text{cand}_{K}}\\}\subset\mathcal{C}$

that are most closely related to the query as shown in Fig. 2. Those
candidates represent only a small subset of the original corpus $\mathcal{C}$
of possible documents.

###  4.2 Cross-encoder re-ranking

Once the top-$K$ documents have been shortlisted by the bi-encoder document
retrieval phase, we now create pairs of the input query with the metadata
prompts corresponding to the candidate documents

$\Big{[}\big{(}q,\text{Prompt}(c_{\text{cand}_{1}})\big{)},\cdots,\big{(}q,\text{Prompt}(c_{\text{cand}_{K}})\big{)}\Big{]}$

Those pairs are then passed together as a unit to the cross-encoder MSMarco
MiniLM model which returns a similarity score for each one of the $K$ pairs as
shown in Fig. 3. Finally, the top-5 ranked documents are returned to the user.

Internally the cross-encoder model uses an attention mechanism between input
and prompt to achieve better performance than the retrieval step. However this
performance comes at a cost of higher run times (reimers2019sentence, ).
Furthermore, the model cannot be run offline (as we had done previously for
the bi-encoder document retrieval phase) since the query $q$ cannot be known
in advance.

To strike a balance between getting accurate documents and low latency, we
experimented with different number of top-$K$ candidates shortlisted during
the retrieval phase. We observed that having many candidates $K\gtrsim 200$
had higher latency and that, on the other end, having few candidates
$K\lesssim 20$ would provide too small a dataset to provide relevant
documents. We found that $K=100$ was the best option for re-ranking in a
reasonable time while providing relevant documents.

Latency-wise, the main hyper-parameter of the cross-encoder model we have
tuned for performance is the batch size $b$ which encodes how many samples to
load at inference time 111Note that this batch size is at inference time which
is slightly unusual in the context of traditional ML models but makes sense in
the context of pre-trained LLM inference. It simply refers to the number of
candidates from the top-$K$ returned by the retriever can be loaded
simultaneously into the cross encoder.. We explore the impact of this
parameter in detail in Section 5.1.

Figure 3: Illustration of re-ranking phase using CrossEncoder where pair of
input query text and prompt text are passed to MSMarco MiniLM model which
produces a score based on similarity between the pair.

In summary, the overall 2-stage architecture is illustrated in Fig.1. Note
that such architectures have recently gained more interest due to their
ability to combine fast retrieval with powerful and accurate re-rankers
(retrieveAndReRank, ; zhang2023semantic, ; reimers2019sentence, ).

##  5 Performance evaluation

To test the performance of the recommender model, we created $31$ queries in
collaboration with domain experts from the Seismic catalog (seismic, ). These
evaluation queries are used both for latency and quality evaluation.

###  5.1 Inference latency

Figure 4: Median latency in milliseconds of different machines on the evaluation queries w.r.t. to batch size $b$. The latency times represent the overall latency which includes the time taken by bi-encoder to encode the user query to get the embeddings, computing cosine similarity against pre-computed document embeddings, shortlisting the top-100 candidate documents, passing them on to the cross-encoder to re-rank and finally return top-5 documents as described in Section 4 and Fig.1 (Results for Standard F4s V2 are omitted for the sake of clarity but the actual numbers are reported in Table 1.) | $b=1$ | $b=2$ | $b=4$ | $b=8$ | $b=16$ | $b=32$ | $b=64$
---|---|---|---|---|---|---|---
Standard F4s v2 | $4.17\pm 1.41$ | $4.89\pm 2.08$ | $5.85\pm 2.72$ | $7.14\pm 3.36$ | $8.03\pm 3.32$ | $8.84\pm 3.73$ | $10.22\pm 4.16$
Standard DS4 v2 | $1.53\pm 0.41$ | $1.49\pm 0.54$ | $1.58\pm 0.68$ | $1.81\pm 0.75$ | $1.96\pm 0.71$ | $2.11\pm 0.73$ | $2.34\pm 0.81$
Standard NV12s v3 | $0.64\pm 0.07$ | $0.54\pm 0.13$ | $0.55\pm 0.18$ | $0.60\pm 0.19$ | $0.66\pm 0.20$ | $0.74\pm 0.21$ | $0.83\pm 0.25$
Table 1: Average inference latency with standard deviation as captured on
different machines and for different values of the batch size $b$. (Times are
in seconds)

As we need to respond to user queries in real-time, we present in this section
a study of the inference latency of the model described in Section 4. Mostly
we focus on the impact of the batch size $b$ in the context of the time-
consuming cross-encoder phase of the model. For the purpose of this experiment
we used three different types of Virtual Machines on Azure:

  * •

Standard F4s v2 - 4 Cores, 8 GB RAM, 32 GB disk

  * •

Standard DS4 v2 - 8 Cores, 28 GB RAM, 56 GB disk

  * •

Standard NV12s v3 - 12 Cores, 112 GB RAM, 336 GB disk, 1 Nvidia Tesla M60 8 GB
vRAM

We show in Fig. 4 the median latency over the evaluation queries for different
values of $b$. In addition, Table 1 shows average latency with standard
deviation over the evaluation queries with different values of the batch size
$b$. We notice that:

  * •

For CPU machines, as we use the machine with more available cores the overall
latency is reduced. However, as expected using a machine with single GPU
performs significantly faster than the CPU machines we tested.

  * •

As the batch size increases, the overall inference time also increases as
shown in Fig. 4 (which shows the median value) and Table 1.

As mentioned previously, our model is expected to return responses to a query
in real time. So saving a few hundred milliseconds in latency is crucial. This
experiment shows that $b$ can have a significant impact on the latency. In
conclusion, it is best to choose $b=2$ or $b=4$ for the VMs available to us.

###  5.2 Human annotators for quantitative relevancy evaluation

We passed the evaluation queries to our recommender model and presented the
top-5 results for each query to four human annotators who were asked to rate
the quality of the responses on a scale from 0 to 5.

Given a query $q$, a score of 0 indicates that none of the responses are
relevant whereas a score of 5 indicates that all 5 responses are relevant to
the query. Clearly any score in between quantifies the proportion of responses
that are deemed relevant by the human annotators. This process of human
verification and assessment of the quality of the model yields 4 independent
scores ranging from $0$ to $5$ for all queries. We present a summarized
analysis in Table 2. This shows that most queries were judged as “relevant”.
In fact, the annotators also pointed out that those few queries that fell
under the category of “not relevant” were either too verbose or too generic so
that no correct documents could be picked out from the catalog anyway.

We also considered the average score of each human annotator which ranged from
2.74 to 3.26 to verify that no annotators were overly positive or negative in
their evaluation thereby confirming that the results did not suffer from bias.

Bucket | Score | Number
---|---|---
| Range | of Queries
Relevant | $[3.5-5]$ | $15/31$
Somewhat relevant | $[2-3.5)$ | $9/31$
Not relevant | $[0-2)$ | $7/31$
Table 2: The average scores of the human annotators are bucketed into 3
different classes depending on their numerical values. Most of the queries
which fell into the relevant bucket were short queries specifying what kind of
documents or products they were looking for. The queries which fell into Not
relevant bucket were verbose and generic in nature. The human annotators also
noticed that in some cases even if query was overly verbose but if it had the
right keywords our model was able to return some relevant contents.

###  5.3 Ablation study

#### Architecture

In this section, we demonstrate by example that the 2-stage architecture,
although more complex and costly than a simple 1-stage (i.e. retiever only
without cross-encoder), returns noticeably more relevant documents.

For example, consider the query

What are contents available for Dynamics 365? Give a PDF format document.

The retriever returns only $2$ PDF documents in its top-5 similarity items.
Although the query specifically asks for PDF documents, the remaining
documents are in OFT and PPTX formats. On the other hand, if we allow the
retriever to return more $K=100$ documents, we are bound to increase the
recall rate of PDF documents although all except 2 would still fall outside of
the top-5. This is where the second stage cross-encoder acts as a re-ranker
that tightly couples the query with those $K=100$ documents so that more
attention can be given to the PDF aspect of the query. In fact, the cross-
encoder successfully re-orders the $K=100$ candidates so that all top-5
documents are in the correct format.

In order to confirm and make a more quantitative comparison between the full
2-stage system (bi-encoder document retrieval + cross-encoder re-ranking) and
a simple 1-stage system (composed of the bi-encoder only), we have submitted
10 of the evaluation queries for human expert verification on the quality of
the top-5 results returned by both architectures. Unsurprisingly, this has
revealed that $90\%$ of the queries received more (or at least equally)
relevant recommendations using the full 2-stage architecture.

#### Numerical features prompt engineering

A large fraction of the metadata associated with the documents comes in the
form of numerical features.

As explained in Section 3, those features can also be integrated into the
prompt representation of the documents by introducing a function $g$ that
converts numerical values into different categories.

Given a numerical feature $f_{i}$ and its value $f_{i}(c_{k})\in\mathcal{R}$
for document $c_{k}$, we defined the mapping function $g$ as

$g\circ f_{i}(c_{k})=\begin{cases}\texttt{high}&\mbox{if
$\,\,\,f_{i}(c_{k})>p_{85}$}\\\ \texttt{medium}&\mbox{if
$\,\,\,p_{65}<f_{i}(c_{k})<p_{85}$}\\\ \texttt{low}&\mbox{if
$\,\,\,f_{i}(c_{k})<p_{65}$}\\\ \texttt{zero}&\mbox{if
$\,\,\,f_{i}(c_{k})=0$}\end{cases}$

where $p_{r}$ stands for the $r^{\text{th}}$ percentile of $f_{i}$ as
estimated over all the documents in $\mathcal{C}$.

Similarly to the PDF example in the paragraph above, our human annotators
confirmed that incorporating those numerical features did have a significant
impact on the relevance of the results. For example, the results returned in
response to a query specifically asking for documents with a “high” number of
views will often miss that detail in the top-5 results of the retrieval phase
but correct itself thanks to the second stage cross-encoder re-ranker.

##  6 Integration into Copilot

Our model has been successfully integrated into the MSX Sales Copilot. It is
being refreshed on a weekly basis to accommodate changes/latest updates in
meta-properties of the documents.

###  6.1 Technical details

Our recommendation model is one of the few skills that has been integrated
into MSX Copilot by using Semantic Kernel (semanticKernel, ). Semantic Kernel
is an open source SDK used to integrate LLMs (or AI services) into
applications. When a query is typed in the Copilot chat window, the planner in
Semantic Kernel is invoked. Planners are used to generate plans and invoke the
respective skills based on the context of the query. Skills (like our
recommendation models) are used to perform certain specific functions. Based
on the query and its context, the planner may decide to invoke our
recommendation skill. This skill then makes an API call to our model which is
deployed on Azure Machine Learning (AML) endpoint. Once the AML endpoint is
called, our recommendation model processes that query as described in Section
4 and returns the top-5 most relevant documents to the chat window as
illustrated in the following section. This entire process takes a few seconds
with the majority of it spent by the planner only.

###  6.2 How it looks

A snapshot of the actual User Interface in MSX is shown in Fig. 5. In blue, we
see the query from the seller. This along with business rules are sent to our
model. The purple outlined box is the response with top-5 documents returned
by our model. Each of these has a link to the document which the seller can
use to open it and share with the end customer using livesend links.

Figure 5: Example of the current UI integration of our model into the
production MSX Copilot.

##  7 Conclusion

The recommender model presented here is the first to be integrated as a skill
into the production MSX Copilot. The initial feedback from the sellers is
encouraging in the first two months since launch. Compared to their previous
experience where sellers had to rely on a sub-optimal filter-based search
system hosted externally on the Seismic website (as opposed to directly
embedded into the MSX interface), this new real-time question-answering
recommender system is seen as a tremendous improvement by the sellers’
community as they have reported in satisfaction surveys. For example, when
asked: “Is the content recommendation skill relevant to the seller’s daily
tasks?”, the received score was 4 out of 5. Similarly, when asked to rate the
relevancy of the recommended documents, sellers gave it a rating of 3.7 out of
5. Moving forward, we are continuously gathering their feedback for further
improvement of the model such as integrating more context based on the seller,
opportunity etc… From a technical perspective, the model innovated compared to
classical semantic search in the sense that we do not use the content of the
documents to recommend but only their metadata.

Our choice of $\approx 20$ features (see Section 3) is limited by the
available meta-features available for documents from the Seismic catalog
$\mathcal{C}$ itself. Generalizing to other corpus of content, it is possible
that more features would be available which would presumably help increase the
relevance of the recommendations. Nonetheless, there are other factors that
may limit the number of features one may consider. From an engineering
perspective the model needs to operate close to real-time with strict latency
requirements. Adding more features would translate into longer response times
and a quality/quantity trade-off on the number of features would need to be
found. Additionally, from a modelling perspective, the MS Marco LLM we used in
the current version of the model has a context length of 512 tokens. With 20
features, this gives each feature about 25 tokens from which a good English
description of the feature has to be generated. Adding more features would
reduce the average number of available tokens per feature highlighting the
need for another kind of trade-off here as well.

As programmatic access to the actual content of the Seismic documents
themselves is not currently available, we limited the current version of the
model to be based on meta-features about the documents only. For other corpus
where such access is available, introducing content as a new feature set will
raise new and challenging questions. For example, the documents in the Seismic
catalog are cluttered across multiple formats (PowerPoint presentations, PDFs,
Word documents…) and one will have to consolidate together multi-modal
features (text, images, audio recordings…) into unified embeddings. We reserve
this for a future version of the model once programmatic access to the content
of the documents is resolved.

Ultimately, our intentions are to build a content recommendation search and
question-answering system that sellers can fully leverage even in customized
contexts. For example, we are exploring how to make the recommendations
specific to the opportunities that sellers are invested in. Sales
opportunities are a crucial component in nurturing early customers and more
customization in this area is of potential high impact. Additionally, we
intend to incorporate the actual text/images of the documents into the
embeddings as well as enrich the query-side prompts with more features
specific to the seller asking a question. Finally, we wish to collect more
user feedback as a way of proxied labeled data so we can start incorporating
more supervision via metrics such as normalized Discounted Cumulative Gain
(nDCG) into the model.

##  8 Acknowledgements

We thank our colleagues in CX Data for their feedback and support. In
particular, we thank the CX Data ML Review board (Vijay Agneeswaran, Daniel
Yehdego, Ivan Barrientos) for reviewing the model and giving us useful inputs.
We thank the product managers from SPS Team (Binh “Binnie” Tran, Manish
Prabhakar) who helped us with their domain knowledge on the Seismic platform
and the content space. We thank the Copilot engineering team, especially Yash
Pal, for providing support in deploying this model to MSX Copilot platform.

## References

  * (1) Seismic, “Seismic.” https://seismic.com/.
  * (2) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017.
  * (3) J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018.
  * (4) Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692, 2019.
  * (5) N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” arXiv:1908.10084, 2019.
  * (6) OpenAI, “Gpt-4 technical report,” 2023.
  * (7) N. Reimers and I. Gurevych, “The curse of dense low-dimensional information retrieval for large index sizes,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), (Online), pp. 605–611, Association for Computational Linguistics, 8 2021.
  * (8) SentenceTransformers, “Pre-trained ms marco models.” https://www.sbert.net/docs/pretrained-models/msmarco-v3.html[Last accessed: ].
  * (9) P. Bajaj, D. Campos, N. Craswell, L. Deng, J. Gao, X. Liu, R. Majumder, A. McNamara, B. Mitra, T. Nguyen, et al., “Ms marco: A human generated machine reading comprehension dataset,” arXiv preprint arXiv:1611.09268, 2016.
  * (10) V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,” arXiv preprint arXiv:1910.01108, 2019.
  * (11) SentenceTransformers, “Retrieve & re-rank.” https://www.sbert.net/examples/applications/retrieve_rerank/README.html [Last accessed: ].
  * (12) C. Zhang, C. Song, S. Agarwal, H. Wu, X. Zhang, and J. J. Lu, “A semantic search framework for similar audit issue recommendation in financial industry,” in Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, pp. 1208–1211, 2023.
  * (13) Microsoft, “Semantic kernel.” https://learn.microsoft.com/en-us/semantic-kernel/overview/ [Last accessed: ].


# Weighted Hardy Type inequalities with Robin Boundary Conditions

Ismail Kombe and Abdullah Yener  Ismail Kombe, Department of Mechatronics
Engineering
Faculty of Engineering
İstanbul Commerce University
Küçükyalı, 34840, İstanbul, Türkiye<EMAIL_ADDRESS>Abdullah Yener,
Department of Mathematics
Faculty of Humanities and Social Sciences
İstanbul Commerce University
Beyoğlu, 34445, İstanbul, Türkiye<EMAIL_ADDRESS>
(Date: August 8, 2022)

###### Abstract.

In this paper we establish a general weighted Hardy type inequality for the
$p-$Laplace operator with Robin boundary condition. We provide various
concrete examples to illustrate our results for different weights.
Furthermore, we present some Heisenberg-Pauli-Weyl type inequalities with
boundary terms on balls with radius $R$ at the origin in $\mathbb{R}^{n}$.

###### Key words and phrases:

Hardy inequality, Boundary term, Robin boundary condition

##  1\. Introduction

The classical Hardy inequality states that

$\int_{\mathbb{R}^{n}}\left|\nabla\phi\left(x\right)\right|^{p}dx\geq\left|\frac{n-p}{p}\right|^{p}\int_{\mathbb{R}^{n}}\frac{\left|\phi\left(x\right)\right|^{p}}{\left|x\right|^{p}}dx,$
(1.1)

and holds for all $\phi\in C_{0}^{\infty}\left(\mathbb{R}^{n}\right)$ if
$1<p<n$, and for all $\phi\in
C_{0}^{\infty}\left(\mathbb{R}^{n}\backslash\left\\{0\right\\}\right)$ if
$p>n$. The constant $|(n-p)/p|^{p}$ in the right-hand side of 1.1 is sharp but
not achieved. This inequality plays an important role in many areas such as
analysis, geometry and mathematical physics. Owing to this, there is a vast
amount of literature on the Hardy type inequalities together with their
variations and generalizations.

For instance, V.G. Maz’ya [15] proved the following weighted integral
inequality

$\int_{\mathbb{R}^{n}}|x_{n}|^{p-1}|\nabla\phi\left(x\right)|^{p}dx\geq\frac{1}{(2p)^{p}}\int_{\mathbb{R}^{n}}\frac{|\phi\left(x\right)|^{p}}{\left(x_{n-1}^{2}+x_{n}^{2}\right)^{\frac{1}{2}}}dx,$
(1.2)

where $\phi\in C_{0}^{\infty}(\mathbb{R}^{n})$.

In the paper [5], E.B. Davies proved the following extension of Hardy’s
inequality for convex domains $\Omega\subset\mathbb{R}^{n},$ $n\geq 2,$

$\int_{\Omega}\left|\nabla\phi\left(x\right)\right|^{2}dx\geq\frac{1}{4}\int_{\Omega}\frac{\left|\phi\left(x\right)\right|^{2}}{\delta^{2}\left(x\right)}dx,$
(1.3)

where $\phi\in C_{0}^{\infty}\left(\Omega\right)$. Here

$\delta\left(x\right)=\mathrm{dist}\,\left(x,\partial\Omega\right)=\min_{y\in\partial\Omega}\left|x-y\right|$
(1.4)

is the distance function to the boundary $\partial\Omega$. Moreover the
constant $\frac{1}{4}$ is sharp and not achieved. We refer the interested
reader to the monographs [13], [5], [2], [10], [17] and the paper [9].

On the other hand, very little work has been done on Hardy type inequalities
with Robin boundary conditions. However, there has been some initiation in
this area of interest. In an interesting paper, H. Kovařik and A. Laptev [12]
proved, among other results, the following Hardy inequality for Laplace
operators with Robin boundary conditions,

$\displaystyle\int_{\Omega}\left|\nabla\phi\right|^{2}dx+\int_{\partial\Omega}\sigma\left|\phi\right|^{2}d\nu\geq\frac{1}{4}$
$\displaystyle\int_{\Omega}\left(\delta\left(x\right)+\frac{1}{2\sigma\left(p\left(x\right)\right)}\right)^{-2}\left|\phi\right|^{2}dx$
(1.5) $\displaystyle+$
$\displaystyle\frac{1}{4}\int_{\Omega}\left(R_{in}+\frac{1}{2\sigma\left(p\left(x\right)\right)}\right)^{-2}\left|\phi\right|^{2}dx$
$\displaystyle+$
$\displaystyle\frac{1}{2}\int_{\partial\Omega}\left(R_{in}+\frac{1}{2\sigma\left(y\right)}\right)^{-1}\left|\phi\right|^{2}d\nu,$

where $\Omega\subset\mathbb{R}^{n}$ is a bounded convex domain with smooth
boundary $\partial\Omega$, $\phi\in H^{1}(\Omega)$, $0\leq\sigma\in
L^{\infty}\left(\partial\Omega\right)$, $d\nu$ is the surface measure on
$\partial\Omega$ and $R_{in}:=\text{sup}\\{\delta(x):x\in\Omega\\}$. Here
$p\left(x\right)$ is the projection function $p:\Omega\setminus
S\to\partial\Omega$ defined by $p\left(x\right):=y\in\partial\Omega\
:\,\delta\left(x\right)=|x-y|,\quad x\in\Omega\setminus S$, where
$S\subset\Omega$ the subset of points in $\Omega$ for which there exist at
least two points $y_{1},y_{2}\in\partial\Omega$ so that the minimum in (1.4)
is achieved.

Later, T. Ekholm, H. Kovařik and A. Laptev [6] studied the best constant in a
Hardy inequality for the $p-$Laplace operator on convex domains with Robin
boundary conditions. Moreover, there has also been an interest regarding Hardy
type inequalities for functions which does not vanish on the boundary of a
given domain, see e.g. [1], [3], [4], [19] and [16].

In light of the results mentioned above, it is natural to investigate under
what conditions the following general weighted Hardy-type inequalities with
boundary terms are valid

$\displaystyle\int_{\Omega}a\left(x\right)\left|\nabla\phi\right|^{p}dx$
$\displaystyle\geq$
$\displaystyle\int_{\Omega}b\left(x\right)\left|\phi\right|^{p}dx+\int_{\partial\Omega}\beta\left(x\right)\left|\phi\right|^{p}d\nu.$
(1.6)

Within this frame work, the main objective of this article is to study the
general weighted Hardy type inequalities for the $p-$Laplace operators with
Robin boundary conditions. We should emphasize that our unifying method is
quite practical and constructive to obtain several weighted Hardy, Maz’ya and
Heisenberg-Pauli-Weyl type inequalities with boundary terms.

##  2\. Weighted Hardy Type Inequalities With Boundary Terms

In what follows, $\Omega$ represents smooth bounded domains contained in
$\mathbb{R}^{n},$ $\partial_{\nu}u$ denotes the exterior normal derivative of
$u$ at the boundary of $\Omega$ and $d\nu$ is the surface measure on
$\partial\Omega$. The generic point is
$x=\left(x_{1},\ldots,x_{n}\right)\in\mathbb{R}^{n}$ and
$r=\left|x\right|=\sqrt{x_{1}^{2}+\cdots+x_{n}^{2}}.$ We denote by
$B_{R}=\left\\{x\in\mathbb{R}^{n}:\left|x\right|<R\right\\}$ the open ball at
the origin with radius $R$ in $\mathbb{R}^{n}$. The boundary is the sphere
$\partial B_{R}=\left\\{x\in\mathbb{R}^{n}:\left|x\right|=R\right\\}$.

We are now ready to state the main result of this paper.

######  Theorem 2.1.

Let $\Omega$ be a smooth bounded domain contained in $\mathbb{R}^{n}.$ Let
$a\in C^{1}\left(\Omega\right)$ be a nonnegative function, $u\in
C^{\infty}\left(\Omega\right)$ be a positive function, $b\in
L_{\text{loc}}^{1}\left(\Omega\right)$ and $\beta\in
L_{\text{loc}}^{1}\left(\partial\Omega\right)$ such that

$-\nabla\cdot\left(a\left(x\right)\left|\nabla u\right|^{p-2}\nabla
u\right)\geq b\left(x\right)u^{p-1}\qquad\text{a.e. in }\Omega$ (2.1)

and

$a\left(x\right)\left|\nabla
u\right|^{p-2}\partial_{\nu}u=\beta\left(x\right)u^{p-1}\qquad\text{a.e. at
}\partial\Omega,$ (2.2)

where $\partial_{\nu}u$ denotes the exterior normal derivative of $u$ at the
boundary of $\Omega.$ There exists a positive constant $c_{p}=c\left(p\right)$
such that; if $p\geq 2,$ then

$\displaystyle\int_{\Omega}a\left(x\right)\left|\nabla\phi\right|^{p}dx$
$\displaystyle\geq$
$\displaystyle\int_{\Omega}b\left(x\right)\left|\phi\right|^{p}dx+c_{p}\int_{\Omega}a\left(x\right)\Big{|}\nabla\Big{(}\frac{\phi}{u}\Big{)}\Big{|}^{p}u^{p}dx$
(2.3)
$\displaystyle+\int_{\partial\Omega}\beta\left(x\right)\left|\phi\right|^{p}d\nu$

and if $1<p<2,$ then

$\displaystyle\int_{\Omega}a\left(x\right)\left|\nabla\phi\right|^{p}dx$
$\displaystyle\geq$
$\displaystyle\int_{\Omega}b\left(x\right)\left|\phi\right|^{p}dx+c_{p}\int_{\Omega}a\left(x\right)\frac{\left|\nabla\left(\frac{\phi}{u}\right)\right|^{2}u^{2}}{\left(\left|\frac{\phi}{u}\nabla
u\right|+\left|\nabla\left(\frac{\phi}{u}\right)\right|u\right)^{2-p}}dx$
(2.4)
$\displaystyle+\int_{\partial\Omega}\beta\left(x\right)\left|\phi\right|^{p}d\nu$

holds for all $\phi\in C^{\infty}\left(\Omega\right).$ Here, $d\nu$ is the
surface measure on $\partial\Omega$.

###### Proof.

For any $\phi\in C^{\infty}\left(\Omega\right)$ we now set
$\varphi:=\frac{\phi}{u},$ where $0<u\in C^{\infty}\left(\Omega\right).$ Thus,

$\left|\nabla\phi\right|^{p}=\left|\varphi\nabla u+u\nabla\varphi\right|^{p}.$

We now use the following convexity inequality:

$\left|\xi+\eta\right|^{p}\geq|\xi|^{p}+p|\xi|^{p-2}\xi\cdot\eta+c_{p}|\eta|^{p},\qquad\forall\xi,\eta\in\mathbb{R}^{n},\quad
p\geq 2,$

where $c_{p}$ is a positive constant depending only on $p,$ see [14]. As an
immediate consequence of the above inequality with the vectors
$\xi=\varphi\nabla u$ and $\eta=u\nabla\varphi,$ we obtain

$\left|\nabla\phi\right|^{p}\geq\left|\nabla
u\right|^{p}\left|\varphi\right|^{p}+u\left|\nabla u\right|^{p-2}\nabla
u\cdot\nabla\left(\left|\varphi\right|^{p}\right)+c_{p}\left|\nabla\varphi\right|^{p}u^{p}.$
(2.5)

Multiplying the inequality $\left(\ref{e4}\right)$ by $a\left(x\right)$ on
both sides, and then applying integration by parts over $\Omega$ for the
middle term, yields

$\displaystyle\int_{\Omega}a\left(x\right)\left|\nabla\phi\right|^{p}dx$
$\displaystyle\geq$ $\displaystyle\int_{\Omega}a\left(x\right)\left|\nabla
u\right|^{p}\left|\varphi\right|^{p}dx-\int_{\Omega}\nabla\cdot\left(a\left(x\right)u\left|\nabla
u\right|^{p-2}\nabla u\right)\left|\varphi\right|^{p}dx$
$\displaystyle+\int_{\partial\Omega}a\left(x\right)u\left|\nabla
u\right|^{p-2}\left|\varphi\right|^{p}\partial_{\nu}ud\nu+c_{p}\int_{\Omega}a\left(x\right)\left|\nabla\varphi\right|^{p}u^{p}dx$
$\displaystyle=$
$\displaystyle-\int_{\Omega}\nabla\cdot\left(a\left(x\right)\left|\nabla
u\right|^{p-2}\nabla u\right)u\left|\varphi\right|^{p}dx$
$\displaystyle+c_{p}\int_{\Omega}a\left(x\right)\left|\nabla\varphi\right|^{p}u^{p}dx+\int_{\partial\Omega}a\left(x\right)u\left|\nabla
u\right|^{p-2}\left|\varphi\right|^{p}\partial_{\nu}ud\nu.$

It therefore follows from $\left(\ref{de1}\right)$ and
$\left(\ref{de2}\right)$ that

$\int_{\Omega}a\left(x\right)\left|\nabla\phi\right|^{p}dx\geq\int_{\Omega}b\left(x\right)u^{p}\left|\varphi\right|^{p}dx+c_{p}\int_{\Omega}a\left(x\right)\left|\nabla\varphi\right|^{p}u^{p}dx+\int_{\partial\Omega}\beta\left(x\right)u^{p}\left|\varphi\right|^{p}d\nu.$

Finally, making the variable change $\varphi=\frac{\phi}{u}$ in the above
integrals, one has the desired inequality for $p\geq 2:$

$\int_{\Omega}a\left(x\right)\left|\nabla\phi\right|^{p}dx\geq\int_{\Omega}b\left(x\right)\left|\phi\right|^{p}dx+c_{p}\int_{\Omega}a\left(x\right)\Big{|}\nabla\Big{(}\frac{\phi}{u}\Big{)}\Big{|}^{p}u^{p}dx+\int_{\partial\Omega}\beta\left(x\right)\left|\phi\right|^{p}d\nu.$

The inequality $\left(\ref{e3}\right)$ can be proved in the same spirit, and
in this case we apply the following convexity inequality:

$\left|\xi+\eta\right|^{p}\geq\left|\xi\right|^{p}+p\left|\xi\right|^{p-2}\xi\cdot\eta+c_{p}\frac{\left|\eta\right|^{2}}{\left(\left|\xi\right|+\left|\eta\right|\right)^{2-p}},\qquad\forall\xi,\eta\in\mathbb{R}^{n},\quad
1<p<2,$

where $c_{p}=c\left(p\right)>0,$ see [14]. We shall omit the details. ∎

###  2.1. Applications of Theorem 2.1

As we stated earlier, our method is quite practical to construct various
weighted Hardy type inequalities including boundary terms on some bounded
domains in $\mathbb{R}^{n}$. To do this, we need to identify suitable model
functions $a$ and $u$ that satisfy the above hypotheses in the given domain.
Let us begin by considering the model functions

$a=\left|x\right|^{\alpha}\text{\quad
and\quad}u=\left|x\right|^{-\left(\frac{n+\alpha-p}{p}\right)}$

in Theorem $\ref{thm2.1}.$ After some computations, we readily get the
subsequent result:

######  Corollary 2.1.

Let $\alpha\in\mathbb{R}$ and $n+\alpha>p>1.$ Then the following inequality
holds:

$\displaystyle\int_{B_{R}}\left|x\right|^{\alpha}\left|\nabla\phi\right|^{p}dx$
$\displaystyle\geq$
$\displaystyle\left(\frac{n+\alpha-p}{p}\right)^{p}\int_{B_{R}}\left|x\right|^{\alpha-p}|\phi|^{p}dx$
$\displaystyle-\left(\frac{n+\alpha-p}{p}\right)^{p-1}R^{\alpha-p+1}\int_{\partial
B_{R}}|\phi|^{p}d\nu$

for all $\phi\in C^{\infty}(B_{R}).$

On the other hand, by applying Theorem 2.1 with the following pair

$a=\left|x\right|^{\alpha}\sinh^{\gamma}\left|x\right|\text{\quad
and\quad}u=\left|x\right|^{-\left(\frac{n+\alpha+\gamma-p}{p}\right)}$

and noting that $\left|x\right|\coth\left|x\right|\geq 1,$ we obtain the power
hyperbolic sine $L^{p}$ Hardy-type inequality with a boundary term.

######  Corollary 2.2.

Let $\alpha\in\mathbb{R},$ $\gamma\geq 0$ and $n+\alpha+\gamma>p>1.$ Then the
following inequality holds:

$\displaystyle\int_{B_{R}}\left|x\right|^{\alpha}\sinh^{\gamma}\left|x\right|\left|\nabla\phi\right|^{p}dx$
$\displaystyle\geq$
$\displaystyle\left(\frac{n+\alpha+\gamma-p}{p}\right)^{p}\int_{B_{R}}\left|x\right|^{\alpha-p}\sinh^{\gamma}\left|x\right||\phi|^{p}dx$
$\displaystyle-\left(\frac{n+\alpha+\gamma-p}{p}\right)^{p-1}\frac{\sinh^{\gamma}R}{R^{p-\alpha-1}}\int_{\partial
B_{R}}|\phi|^{p}d\nu$

for all $\phi\in C^{\infty}(B_{R}).$

Recall that the following weighted $L^{2}$ Hardy type inequalities in the
Euclidean setting were proved by Ghoussoub and Moradifam [11]: Let $s,t>0$ and
$\alpha,\gamma,m$ be real numbers.

  * •

If $\alpha\gamma>0$ and $m\leq\frac{n-2}{2},$ then for all $\varphi\in
C_{0}^{\infty}(\mathbb{R}^{n})$

$\int_{\mathbb{R}^{n}}\frac{\left(s+t\left|x\right|^{\alpha}\right)^{\gamma}}{\left|x\right|^{2m}}\left|\nabla\varphi\right|^{2}dx\geq\left(\frac{n-2m-2}{2}\right)^{2}\int_{\mathbb{R}^{n}}\frac{\left(s+t\left|x\right|^{\alpha}\right)^{\gamma}}{\left|x\right|^{2m+2}}\varphi^{2}dx.$
(2.6)

  * •

If $\alpha\gamma<0$ and $2m-\alpha\gamma\leq n-2,$ then for all $\varphi\in
C_{0}^{\infty}(\mathbb{R}^{n})$

$\int_{\mathbb{R}^{n}}\frac{\left(s+t\left|x\right|^{\alpha}\right)^{\gamma}}{\left|x\right|^{2m}}\left|\nabla\varphi\right|^{2}dx\geq\left(\frac{n+\alpha\beta-2m-2}{2}\right)^{2}\int_{\mathbb{R}^{n}}\frac{\left(s+t\left|x\right|^{\alpha}\right)^{\gamma}}{\left|x\right|^{2m+2}}\varphi^{2}dx.$
(2.7)

We now extend and improve the above inequalities $\left(\ref{m1}\right)$ and
$\left(\ref{m2}\right)$ to the $L^{p}$ case with a boundary term on the
$R-$ball in $\mathbb{R}^{n}$. In order to do so, we now take the pair as

$a=\frac{\left(s+t\left|x\right|^{\alpha}\right)^{\gamma}}{\left|x\right|^{pm}}\text{\quad
and\quad}u=\left|x\right|^{-\left(\frac{n-pm-p}{p}\right)}$

in Theorem $\ref{thm2.1}.$ This gives the following improvement of the
inequality $\left(\ref{m1}\right).$

######  Corollary 2.3.

Let $s,t>0$ and $\alpha,\gamma,m\in\mathbb{R}$. If $\alpha\gamma>0$ and
$1<p\leq n-pm,$ then for all $\phi\in C^{\infty}(B_{R})$ one has

$\displaystyle\int_{B_{R}}\frac{\left(s+t\left|x\right|^{\alpha}\right)^{\gamma}}{\left|x\right|^{pm}}\left|\nabla\phi\right|^{p}dx\geq$
$\displaystyle\
C_{n,p,m}^{p}\int_{B_{R}}\frac{\left(s+t\left|x\right|^{\alpha}\right)^{\gamma}}{\left|x\right|^{pm+p}}|\phi|^{p}dx$
$\displaystyle+C_{n,p,m}^{p-1}\alpha\gamma
t\int_{B_{R}}\frac{\left(s+t\left|x\right|^{\alpha}\right)^{\gamma-1}}{\left|x\right|^{pm+p-\alpha}}|\phi|^{p}dx$
$\displaystyle-
C_{n,p,m}^{p-1}\frac{\left(s+tR^{\alpha}\right)^{\gamma}}{R^{pm+p-1}}\int_{\partial
B_{R}}|\phi|^{p}d\nu,$

where $C_{n,p,m}=\left(\frac{n-pm-p}{p}\right).$

If we consider the units

$a=\frac{\left(s+t\left|x\right|^{\alpha}\right)^{\gamma}}{\left|x\right|^{pm}}\text{\quad
and\quad}u=\left|x\right|^{-\left(\frac{n+\alpha\gamma-pm-p}{p}\right)},$

then we immediately obtain the following improvement of the inequality
$\left(\ref{m2}\right).$

######  Corollary 2.4.

Let $s,t>0$ and $\alpha,\gamma,m\in\mathbb{R}$. If $\alpha\gamma<0$ and
$1<p\leq n+\alpha\gamma-pm,$ then for all $\phi\in C^{\infty}(B_{R})$ one has

$\displaystyle\int_{B_{R}}\frac{\left(s+t\left|x\right|^{\alpha}\right)^{\gamma}}{\left|x\right|^{pm}}\left|\nabla\phi\right|^{p}dx\geq$
$\displaystyle\
C_{n,p,m,\alpha,\gamma}^{p}\int_{B_{R}}\frac{\left(s+t\left|x\right|^{\alpha}\right)^{\gamma}}{\left|x\right|^{pm+p}}|\phi|^{p}dx$
$\displaystyle-C_{n,p,m,\alpha,\gamma}^{p-1}\alpha\gamma
s\int_{B_{R}}\frac{\left(s+t\left|x\right|^{\alpha}\right)^{\gamma-1}}{\left|x\right|^{pm+p}}|\phi|^{p}dx$
$\displaystyle-
C_{n,p,m,\alpha,\gamma}^{p-1}\frac{\left(s+tR^{\alpha}\right)^{\gamma}}{R^{pm+p-1}}\int_{\partial
B_{R}}|\phi|^{p}d\nu,$

where $C_{n,p,m,\alpha,\gamma}=\left(\frac{n+\alpha\gamma-pm-p}{p}\right).$

######  Remark 2.2.

Note that if $\alpha=0$ or $\gamma=0\ $in the above two inequalities, then
they reduce to Hardy type inequalities with usual weights. Hence, we are
interested in the case when $\alpha\gamma\neq 0.$

It is worth stressing here that, by considering the model functions

$a=\left(1+\left|x\right|^{\frac{p}{p-1}}\right)^{\alpha\left(p-1\right)}\text{
\quad and\quad}u=\left(1+\left|x\right|^{\frac{p}{p-1}}\right)^{1-\alpha}$

in Theorem $\ref{thm2.1},$ we obtain an improved version of one of the
Skrzypczak’s result in [18] with a boundary term on the $R-$ball in
$\mathbb{R}^{n}$.

######  Corollary 2.5.

Let $1<p<n$ and $\alpha>1.$ Then for all $\phi\in C^{\infty}(B_{R}),$ there
holds

$\displaystyle\int_{B_{R}}\left(1+\left|x\right|^{\frac{p}{p-1}}\right)^{\alpha\left(p-1\right)}\left|\nabla\phi\right|^{p}dx$
$\displaystyle\geq$ $\displaystyle
n\left(\frac{p\left(\alpha-1\right)}{p-1}\right)^{p-1}\int_{B_{R}}\left(1+\left|x\right|^{\frac{p}{p-1}}\right)^{\left(\alpha-1\right)\left(p-1\right)}|\phi|^{p}dx$
$\displaystyle-\left(\frac{p\left(\alpha-1\right)}{p-1}\right)^{p-1}R\left(1+R^{\frac{p}{p-1}}\right)^{\left(\alpha-1\right)\left(p-1\right)}\int_{\partial
B_{R}}|\phi|^{p}d\nu.$

Another consequence of Theorem 2.1 with the functions

$a=\left|x\right|^{\alpha}\text{\quad
and\quad}u=\left(1+\left|x\right|^{\frac{p}{p-1}}\right)^{-\left(\frac{n+\alpha-p}{p}\right)}$

leads us to the following linearized Sobolev inequality:

######  Corollary 2.6.

Let $\alpha\in\mathbb{R}$ and $n+\alpha>p>1.$ Then the inequality

$\displaystyle\int_{B_{R}}\left|x\right|^{\alpha}\left|\nabla\phi\right|^{p}dx$
$\displaystyle\geq$
$\displaystyle\left(\frac{n+\alpha-p}{p-1}\right)^{p-1}\left(n+\alpha\right)\int_{B_{R}}\frac{\left|x\right|^{\alpha}}{\left(1+\left|x\right|^{\frac{p}{p-1}}\right)^{p}}|\phi|^{p}dx$
$\displaystyle-\left(\frac{n+\alpha-p}{Rp-R}\right)^{p-1}\int_{\partial
B_{R}}|\phi|^{p}d\nu$

holds for every $\phi\in C^{\infty}(B_{R}).$

Even though the literature has mostly focused on power radial weights, we now
establish $L^{p}$ Hardy-type inequalities with nonradial weights with respect
to the inclusion of boundary terms on smooth or piecewisely smooth bounded
domains in $\mathbb{R}^{n}$. In order to do so, we now take the pair as

$a=\frac{e^{\alpha x_{1}}}{\alpha^{p-1}}\text{\quad and\quad}u=e^{\alpha
x_{1}}.$

This gives the following result.

######  Corollary 2.7.

Let $\alpha>0$ and $p>1.$ Then, for all $\phi\in
C^{\infty}\left(B_{R}\right),$ we have

$\int_{B_{R}}\frac{e^{\alpha
x_{1}}}{\alpha^{p-1}}\left|\nabla\phi\right|^{p}dx\geq-\alpha
p\int_{B_{R}}e^{\alpha x_{1}}|\phi|^{p}dx+\frac{1}{R}\int_{\partial
B_{R}}x_{1}e^{\alpha x_{1}}|\phi|^{p}d\nu.$

On the other hand, by specializing the functions as

$a=e^{\alpha\left(1-p\right)x_{1}}\text{\quad and\quad}u=e^{\alpha x_{1}},$

we have another weighted $L^{p}$ Hardy type inequality with a boundary term.

######  Corollary 2.8.

Let $\alpha\in\mathbb{R}$ and $p>1.$ Then, for all $\phi\in
C^{\infty}\left(B_{R}\right),$ we have

$\int_{B_{R}}e^{\alpha\left(1-p\right)x_{1}}\left|\nabla\phi\right|^{p}dx\geq\frac{\alpha\left|\alpha\right|^{p-2}}{R}\int_{\partial
B_{R}}x_{1}e^{\alpha\left(1-p\right)x_{1}}|\phi|^{p}d\nu.$

We now set the non-symmetric functions

$a=e^{-px_{1}}\text{\quad and\quad}u=e^{x_{1}}.$

This yields the following $L^{p}$ Hardy type inequality with non-symmetric
weights and a boundary remainder term.

######  Corollary 2.9.

For any $\phi\in C^{\infty}\left(B_{R}\right)$ and $p>1,$ one has

$\int_{B_{R}}\frac{\left|\nabla\phi\right|^{p}}{e^{px_{1}}}dx\geq\int_{B_{R}}\frac{|\phi|^{p}}{e^{px_{1}}}dx+\frac{1}{R}\int_{\partial
B_{R}}\frac{x_{1}}{e^{px_{1}}}|\phi|^{p}d\nu.$

Another immediate application of Theorem 2.1 with the following functions

$a\equiv 1\text{\quad and\quad}u=e^{x_{1}+\cdots+x_{n}}$

is the following inequality:

######  Corollary 2.10.

For any $\phi\in C^{\infty}\left(B_{R}\right)$ and $p>1,$ one has

$\int_{B_{R}}\left|\nabla\phi\right|^{p}dx\geq\left(1-p\right)n^{p/2}\int_{B_{R}}|\phi|^{p}dx+\frac{n^{\frac{p-2}{2}}}{R}\int_{\partial
B_{R}}\left(x_{1}+\cdots+x_{n}\right)|\phi|^{p}d\nu.$

Let us mention that when one consider Theorem 2.1 with the pair

$a=e^{x_{1}+\cdots+x_{n}}\text{\quad and\quad}u=e^{x_{1}+\cdots+x_{n}},$

one can obtain the following result including an exponential weight:

######  Corollary 2.11.

For any $\phi\in C^{\infty}\left(B_{R}\right)$ and $p>1,$ one has

$\displaystyle\int_{B_{R}}e^{x_{1}+\cdots+x_{n}}\left|\nabla\phi\right|^{p}dx$
$\displaystyle\geq$ $\displaystyle-
pn^{p/2}\int_{B_{R}}e^{x_{1}+\cdots+x_{n}}|\phi|^{p}dx$
$\displaystyle+\frac{n^{\frac{p-2}{2}}}{R}\int_{\partial
B_{R}}\left(x_{1}+\cdots+x_{n}\right)e^{x_{1}+\cdots+x_{n}}|\phi|^{p}d\nu.$

######  Remark 2.3.

Note that one can also apply Theorem 2.1 on some domains $\Omega$ in
$\mathbb{R}^{n}$ with piecewisely smooth boundary $\partial\Omega.$ The
following is the first corollary in this direction.

Suppose that

$a\equiv 1\text{\quad and\quad}u=x_{1}r,$

then we derive the following inequality:

######  Corollary 2.12.

Let $\Omega=\left\\{x\in B_{R}:x_{1}>0\right\\}.$ For all $\phi\in
C^{\infty}(\Omega),$ there holds

$\int_{\Omega}\left|\nabla\phi\right|^{2}dx\geq-\left(n+1\right)\int_{\Omega}\frac{\phi^{2}}{\left|x\right|^{2}}dx+\int_{\partial\Omega}\left(\frac{1}{R}-\frac{1}{x_{1}}\right)\phi^{2}d\nu.$

When we take the following pair

$a=\left(\frac{1}{x_{1}^{2}}+\frac{1}{x_{2}^{2}}+\cdots+\frac{1}{x_{n}^{2}}\right)^{\frac{2-p}{p}}\text{\quad
and\quad}u=\left(x_{1}x_{2}\ldots x_{n}\right)^{1/p}$

in Theorem $\ref{thm2.1},$ we have the subsequent weighted $L^{p}$ Hardy type
inequality with boundary terms.

######  Corollary 2.13.

Let $1<p<\infty\ $and $\Omega=\left\\{x\in B_{R}:x_{i}>0,\text{
}i=1,2,\ldots,n\right\\}.$ Then, for all $\phi\in
C^{\infty}\left(\Omega\right),$ we have

$\displaystyle\int_{\Omega}\left(\frac{1}{x_{1}^{2}}+\frac{1}{x_{2}^{2}}+\cdots+\frac{1}{x_{n}^{2}}\right)^{\frac{2-p}{2}}\left|\nabla\phi\right|^{p}dx$
$\displaystyle\geq$
$\displaystyle\frac{1}{p^{p}}\int_{\Omega}\left(\frac{1}{x_{1}^{2}}+\frac{1}{x_{2}^{2}}+\cdots+\frac{1}{x_{n}^{2}}\right)|\phi|^{p}dx$
$\displaystyle-\frac{1}{p^{p-1}}\int_{\partial\Omega}\left(\frac{1}{x_{1}^{2}}+\frac{1}{x_{2}^{2}}+\cdots+\frac{1}{x_{n}^{2}}\right)|\phi|^{p}d\nu$
$\displaystyle+\frac{n}{Rp^{p-1}}\int_{\partial\Omega}|\phi|^{p}d\nu.$

We now set the pair as

$a=x_{1}^{p-1}\text{\quad
and\quad}u=\left(\log\frac{R}{x_{1}}\right)^{\frac{p-1}{p}}.$

Hence, we derive the power logarithmic $L^{p}$ Hardy type inequality.

######  Corollary 2.14.

Let $p>1$ and $\Omega=\left\\{x\in B_{R}:x_{1}>0\right\\}.$ Then, for all
$\phi\in C^{\infty}\left(\Omega\right),$ we have

$\displaystyle\int_{\Omega}x_{1}^{p-1}\left|\nabla\phi\right|^{p}dx$
$\displaystyle\geq$
$\displaystyle\left(\frac{p-1}{p}\right)^{p}\int_{\Omega}\frac{|\phi|^{p}}{x_{1}\left(\log\frac{R}{x_{1}}\right)^{p}}dx$
$\displaystyle+\left(\frac{p-1}{p}\right)^{p-1}\frac{1}{R}\int_{\partial\Omega}\frac{R-x_{1}}{\left(\log\frac{R}{x_{1}}\right)^{p-1}}|\phi|^{p}d\nu.$

Inspired by the inequality of Maz’ya (1.2) and considering the functions

$a=x_{n}^{p-1}\text{\quad
and\quad}u=\left(x_{n-1}^{2}+x_{n}^{2}\right)^{-\frac{1}{2p}},$

we obtain the following Maz’ya type inequality with a boundary term.

######  Corollary 2.15.

Let $p>1$ and $\Omega=\left\\{x\in B_{R}:x_{n}>0\right\\}.$ Then, for all
$\phi\in C^{\infty}\left(\Omega\right),$ we have

$\displaystyle\int_{\Omega}x_{n}^{p-1}\left|\nabla\phi\right|^{p}dx$
$\displaystyle\geq$
$\displaystyle\frac{1}{p^{p}}\int_{\Omega}\frac{|\phi|^{p}}{(x_{n-1}^{2}+x_{n}^{2})^{p/2}}dx$
$\displaystyle-\frac{1}{Rp^{p-1}}\int_{\partial\Omega}\frac{x_{n}^{p-1}(x_{n-1}^{2}+x_{n}^{2}-Rx_{n})}{\left(x_{n-1}^{2}+x_{n}^{2}\right)^{\frac{p}{2}}}|\phi|^{p}d\nu.$

Another consequence of the Theorem 2.1 with the special functions

$a=e^{\alpha x_{1}^{2}x_{2}^{2}\dots x_{n}^{2}}\text{\quad
and\quad}u=e^{-\alpha x_{1}^{2}x_{2}^{2}\dots x_{n}^{2}}$

is the following inequality.

######  Corollary 2.16.

For all $\phi\in C^{\infty}\left(B_{R}\right)$ and $\alpha>0,$ we have

$\displaystyle\int_{B_{R}}e^{\alpha x_{1}^{2}x_{2}^{2}\dots
x_{n}^{2}}\left|\nabla\phi\right|^{2}dx\geq$ $\displaystyle
2\alpha\int_{B_{R}}e^{\alpha x_{1}^{2}x_{2}^{2}\dots
x_{n}^{2}}\left(x_{1}^{2}x_{2}^{2}\dots
x_{n}^{2}\right)\left(\frac{1}{x_{1}^{2}}+\cdots+\frac{1}{x_{n}^{2}}\right)|\phi|^{2}dx$
$\displaystyle-\frac{2\alpha n}{R}\int_{\partial B_{R}}e^{\alpha
x_{1}^{2}x_{2}^{2}\dots x_{n}^{2}}\left(x_{1}^{2}\dots
x_{n}^{2}\right)|\phi|^{2}d\nu.$

### Heisenberg-Pauli-Weyl Inequality

Finally, we turn our attention to the classical Heisenberg-Pauli-Weyl
inequality which states that

$\left(\int_{\mathbb{R}^{n}}|\nabla
f(x)|^{2}dx\right)\left(\int_{\mathbb{R}^{n}}|x|^{2}|f(x)|^{2}dx\right)\geq\frac{n^{2}}{4}\left(\int_{\mathbb{R}^{n}}|f(x)|^{2}dx\right)^{2}$
(2.8)

for all $f\in L^{2}(\mathbb{R}^{n})$. It has been studied in various contexts
and we refer to [8] and [7] for an overview of the history and the relevance
of this inequality.

It is worth mentioning here that Theorem 2.1 can be applied to obtain the
local Heisenberg-Pauli-Weyl-type inequalities with boundary terms. In order to
be more precise, we now take $p=2$ in Theorem 2.1 and choose the following
functions

$a\equiv 1\text{\quad and\quad}u=e^{-\alpha\left|x\right|^{2}},\quad\alpha>0.$

This allows the derivation of the inequality

$\int_{B_{R}}\left|\nabla\phi\right|^{2}dx\geq-4\alpha^{2}\int_{B_{R}}\left|x\right|^{2}\phi^{2}dx+2\alpha
n\int_{B_{R}}\phi^{2}dx-2\alpha R^{2}\int_{\partial B_{R}}\phi^{2}d\nu.$

Optimizing in $\alpha$, we find the local Heisenberg-Pauli-Weyl type
inequality with a boundary term.

######  Corollary 2.17.

For all $\phi\in C^{\infty}\left(B_{R}\right),$ we have

$\left(\int_{B_{R}}\left|\nabla\phi\right|^{2}dx\right)\left(\int_{B_{R}}\left|x\right|^{2}\phi^{2}dx\right)\geq\frac{1}{4}\left(n\int_{B_{R}}\phi^{2}dx-R^{2}\int_{\partial
B_{R}}\phi^{2}d\nu\right)^{2}.$

On the other hand, by setting the functions

$a\equiv 1\text{\quad and\quad}u=e^{-\alpha\left|x\right|},\quad\alpha>0,$

we shall deduce from Theorem 2.1 that

$\int_{B_{R}}\left|\nabla\phi\right|^{2}dx\geq-\alpha^{2}\int_{B_{R}}\phi^{2}dx+\alpha\left(\left(n-1\right)\int_{B_{R}}\frac{\phi^{2}}{\left|x\right|}dx-\int_{\partial
B_{R}}\phi^{2}d\nu\right).$

Hence, a very similar argument applies to obtain the following version of the
Heisenberg-Pauli-Weyl type inequality.

######  Corollary 2.18.

For every $\phi\in C^{\infty}(B_{R}),$ one has

$\left(\int_{B_{R}}\left|\nabla\phi\right|^{2}dx\right)\left(\int_{B_{R}}\phi^{2}dx\right)\geq\frac{1}{4}\left(\left(n-1\right)\int_{B_{R}}\frac{\phi^{2}}{\left|x\right|}dx-\int_{\partial
B_{R}}\phi^{2}d\nu\right)^{2}.$

Considering the functions

$a\equiv 1\text{\quad and\quad}u=e^{-\alpha x_{n}^{2}},\quad\alpha>0,$

as a consequence of Theorem 2.1, we readily have

$\int_{B_{R}}\left|\nabla\phi\right|^{2}dx\geq-4\alpha^{2}\int_{B_{R}}x_{n}^{2}|\phi|^{2}dx+2\alpha\left(\int_{B_{R}}|\phi|^{2}dx-\int_{\partial
B_{R}}x_{n}^{2}|\phi|^{2}d\nu\right).$

Arguing as above, we obtain the following inequality.

######  Corollary 2.19.

For every $\phi\in C^{\infty}(B_{R}),$ one has

$\left(\int_{B_{R}}\left|\nabla\phi\right|^{2}dx\right)\left(\int_{B_{R}}x_{n}^{2}|\phi|^{2}dx\right)\geq\frac{1}{4}\left(\int_{B_{R}}|\phi|^{2}dx-\int_{\partial
B_{R}}x_{n}^{2}|\phi|^{2}d\nu\right)^{2}.$

Finally, when we take the pair as

$a\equiv 1\text{\quad
and\quad}u=e^{-\alpha\left(x_{n-1}^{2}+x_{n}^{2}\right)},\quad\alpha>0,$

we get

$\displaystyle\int_{B_{R}}\left|\nabla\phi\right|^{2}dx\geq$
$\displaystyle-4\alpha^{2}\int_{B_{R}}\left(x_{n-1}^{2}+x_{n}^{2}\right)|\phi|^{2}dx+4\alpha\int_{B_{R}}|\phi|^{2}dx$
$\displaystyle-\frac{2\alpha}{R}\int_{\partial
B_{R}}\left(x_{n-1}^{2}+x_{n}^{2}\right)|\phi|^{2}d\nu.$

Hence, with a similar approach, we achieve the following inequality.

######  Corollary 2.20.

For all $\phi\in C^{\infty}\left(B_{R}\right),$ we have

$\left(\int_{B_{R}}\left|\nabla\phi\right|^{2}dx\right)\left(\int_{B_{R}}\left(x_{n-1}^{2}+x_{n}^{2}\right)|\phi|^{2}dx\right)\geq\frac{A}{4},$

where $A=\left(2\int_{B_{R}}|\phi|^{2}dx-\frac{1}{R}\int_{\partial
B_{R}}\left(x_{n-1}^{2}+x_{n}^{2}\right)|\phi|^{2}d\nu\right)^{2}$.

## References

  * [1] Adimurthi, Hardy-Sobolev inequality in $H^{1}\left(\Omega\right)$ and its applications, Comm. Contem. Math., 4 (2002), 409–434.
  * [2] A. A. Balinsky, W. D. Evans and R. T. Lewis, The Analysis and Geometry of Hardy’s Inequality, Springer, Heidelberg, 2015.
  * [3] E. Berchio, D. Cassani and F. Gazzola, Hardy-Rellich inequalities with boundary remainder terms and applications, Manuscripta Math. 131 (2010), no. 3-4, 427–458.
  * [4] C. Cowan, Optimal Hardy inequalities for general elliptic operators with improvements, Commun. Pure Appl. Anal. 9 (2010), no. 1, 109–140.
  * [5] E.B. Davies, A review of Hardy inequalities, in: The Maz’ya Anniversary Collection, vol. 2, in: Oper. Theory Adv. Appl., vol. 110, Birkhäuser, Basel, 1999.
  * [6] T. Ekholm, H. Kovařík and A. Laptev, Hardy inequalities for p-Laplacians with Robin boundary conditions, Nonlinear Anal. 128 (2015), 365–379.
  * [7] W. Erb, Uncertainty principles on Riemannian manifolds, Logos Berlin, 2011.
  * [8] G. B. Folland and A. Sitaram, The uncertainty principle: A mathematical survey, J. Fourier Anal. Appl., 3 (1997), 207–238.
  * [9] J.A. Goldstein, I. Kombe and A. Yener, A unified approach to weighted Hardy type inequalities on Carnot Groups, Discrete Contin. Dyn. Syst. 37 (2017), no.4, 2009–2021.
  * [10] N. Ghoussoub and A. Moradifam, Functional Inequalities: New Perspectives and New Applications, Mathematical Survey and Monographs series, AMS, 2013.
  * [11] N. Ghoussoub and A. Moradifam, Bessel potentials and optimal Hardy and Hardy-Rellich inequalities, Math. Ann. 349 (2011), 1-57.
  * [12] H. Kovařík and A. Laptev, Hardy inequalities for Robin Laplacians, J. Funct. Anal. 262 (2012), 4972–4985.
  * [13] Â A. Kufner and B. Opic, Hardy-type inequalities, Pitman Reserach Notes in Math. Series 219, Longman Sci. and Tech, Harlow, 1990\.
  * [14] P. Lindqvist, On the equation $\text{div}(|\nabla u|^{p-2}\nabla u)+\lambda|u|^{p-2}u=0$, Proc. Amer. Math. Soc. 109 (1990), 157–164.
  * [15] V.G. Mazâ€™ya, Sobolev Spaces, Springer, Berlin, 1985.
  * [16] Y. Pinchover and I. Versano, Optimal Hardy-weights for elliptic operators with mixed boundary conditions, arXiv:2103.13979.
  * [17] M. Ruzhansky and D. Suragan, Hardy inequalities on homogeneous groups, Progress in Math. Vol. 327, Birkhäuser, 2019.
  * [18] I. Skrzypczak, Hardy type inequalities derived from $\mathit{p-}$harmonic problems, Nonlinear Anal. 93 (2013), 30-50.
  * [19] Z.Q. Wang and M. Zhu, Hardy inequalities with boundary terms, Electron. J. Differential Equations 2003, No. 43.


# Finite-time Lyapunov exponents of deep neural networks

L. Storm  Department of Physics, University of Gothenburg, 41296 Gothenburg,
Sweden  H. Linander  Department of Mathematical Sciences, Chalmers Technical
University and University of Gothenburg, Sweden  J. Bec  MINES Paris, PSL
Research University, CNRS, Cemef, Sophia-Antipolis, France F-06900  Université
Côte d’Azur, Inria, CNRS, Cemef, Sophia-Antipolis, France, F-06900  K.
Gustavsson  Department of Physics, University of Gothenburg, 41296 Gothenburg,
Sweden  B. Mehlig  Department of Physics, University of Gothenburg, 41296
Gothenburg, Sweden

###### Abstract

We compute how small input perturbations affect the output of deep neural
networks, exploring an analogy between deep networks and dynamical systems,
where the growth or decay of local perturbations is characterised by finite-
time Lyapunov exponents. We show that the maximal exponent forms geometrical
structures in input space, akin to coherent structures in dynamical systems.
Ridges of large positive exponents divide input space into different regions
that the network associates with different classes. These ridges visualise the
geometry that deep networks construct in input space, shedding light on the
fundamental mechanisms underlying their learning capabilities.

Deep neural networks can be trained to model complex functional relationships
J Jumper et al. (2021). The expressivity of such neural networks – their
ability to unfold intricate data structures – increases exponentially as the
number of layers increases Poole  _et al._ (2016). However, deeper networks
are harder to train, due to the multiplicative growth or decay of signals as
they propagate through the network. This multiplicative amplification, also
known as the unstable-gradient problem Mehlig (2021), causes signals to either
explode or vanish in magnitude if the number of layers is too large. A second
important problem is that we lack insight into the learning mechanisms.
Although there is some intuition for shallow networks Minsky and Papert
(1969), there is still no general understanding of the principles that cause
some architectures to fail, while others work better.

For a common type of deep networks, the so-called multi-layer perceptrons
[Fig. 1(a)], we show that these two problems are closely related. We exploit
the fact that such networks are discrete dynamical systems; inputs
$\boldsymbol{x}^{(0)}$ are mapped iteratively through
$x_{i}^{(\ell)}=g(\sum_{j=1}^{N_{\ell}}w_{ij}^{(\ell)}x_{j}^{(\ell-1)}\\!-\theta_{i}^{(\ell)})$.
Here, $g(\cdot)$ is a non-linear activation function Mehlig (2021), the layer
index $\ell=0,\ldots,L\\!+\\!1$ plays the role of time, $L$ is the number of
hidden layers, $N_{\ell}$ is the number of neurons in layer $\ell$, and the
weights $w_{ij}^{(\ell)}$ and thresholds $\theta_{i}^{(\ell)}$ are parameters.
Sensitivity of $\boldsymbol{x}^{(\ell)}$ to small changes in the inputs
$\boldsymbol{x}^{(0)}=\boldsymbol{x}$ corresponds to exponentially growing
perturbations in a chaotic system with positive maximal Lyapunov exponent Ott
(2002); Cvitanović  _et al._ (2020)
$\lim_{\ell\to\infty}\lambda_{1}^{(\ell)}(\boldsymbol{x})$, with growth rate
$\lambda_{1}^{(\ell)}(\boldsymbol{x})=\ell^{-1}\log|\delta\boldsymbol{x}^{(\ell)}|/|\delta\boldsymbol{x}|$.
The multiplicative ergodic theorem Ott (2002) guarantees that
$\lambda_{1}^{(L)}(\boldsymbol{x})$ converges as $L\to\infty$ to a limit that
is independent of $\boldsymbol{x}$.

The standard way of initialising network parameters is to choose zero
thresholds and random weight matrices with independent Gaussian-distributed
elements with zero mean, and variance $\sigma_{w}^{2}$. In this case, the
Lyapunov exponents are determined by a product of random matrices Crisanti
_et al._ (1993), and in the mean-field limit of $N_{\ell}=N\to\infty$, one
finds $\lambda_{1}^{(L)}\sim\log(GN\sigma_{w}^{2})$ independent of
$\boldsymbol{x}$ and $L$. Here, the constant $G$ depends on the choice of
activation function Pennington  _et al._ (2017). This relation explains why
the initial weight variance should be chosen so that $GN\sigma_{w}^{2}=1$,
because then signals neither contract nor expand Pennington  _et al._ (2017);
Sutskever  _et al._ (2013); Glorot and Bengio (2010), stabilising the
learning. The maximal Lyapunov exponent also determines the success or failure
in predicting chaotic time series with recurrent networks Jaeger and Haas
(2004); Pathak  _et al._ (2018); Storm  _et al._ (2022) that use large
reservoirs of neurons with random weights. In that case, the mean-field limit
$N\to\infty$ works very well Storm  _et al._ (2022).

For finite $N$ and $L$, the maximal finite-time Lyapunov exponent (FTLE)
$\lambda_{1}^{(L)}(\boldsymbol{x})$ depends on the input $\boldsymbol{x}$.
Averaging over input patterns yields an estimate for the Lyapunov exponent
Vogt  _et al._ (2020), but even if the average
$\langle\lambda_{1}^{(L)}(\boldsymbol{x})\rangle$ over inputs vanishes, some
patterns may exhibit large positive exponents, causing the training to fail.

Moreover, the weights of a trained network are not random but should reflect
what the network has learned about the inputs. This raises the question: does
the maximal exponent form geometric structures in input space, just as in
dynamical systems where the ridges of high FTLE define Lagrangian coherent
structures Haller and Yuan (2000); Lucarini and Bódai (2017); Beneitez  _et
al._ (2020)? How does the variations of the maximal FTLE in input-space depend
on the number $L$ of layers of the network, and on its width $N$?

To answer these questions, we computed the maximal FTLEs for fully connected
deep neural networks with different widths and numbers of layers. For a simple
classification problem with two-dimensional inputs $\boldsymbol{x}$ divided
into two classes with targets $t(\boldsymbol{x})=\pm 1$ [Fig. 1(b)], we show
how the $\boldsymbol{x}$-dependence of the maximal FTLE changes when changing
$N$ and $L$. For narrow networks (small $N$), we find that the maximal FTLE
forms ridges of large exponents in the input plane, much like Lagrangian
coherent structures in high-dimensional dynamical systems Haller and Yuan
(2000); Lucarini and Bódai (2017); Beneitez  _et al._ (2020). These ridges
provide insight into the learning process, illustrating how the network learns
to change its output by order unity in response to a small shift of the input
pattern across the decision boundary. However, as the network width grows, we
see that the ridges disappear, suggesting a different learning mechanism.
Similar conclusions hold for a more complex classification problem using the
MNIST data set of hand-written digits, where FTLE structures in input space
explain variations in classification accuracy and predictive uncertainty.

Figure 1:  Classification with a fully connected feed-forward network. (a)
Layout with two input components $x_{1}^{(0)}$ and $x_{2}^{(0)}$, $L$ hidden
layers with $N=5$ neurons, and one output $x^{(L\\!+\\!1)}$ for
classification. (b) Two-dimensional input plane (schematic) for a
classification problem with a circular decision boundary that separates input
patterns with targets $t=+1$ ($\blacksquare$) from those with $t=-1$ ($\Box$,
green).

Finite-time Lyapunov exponents. Figure 1(a) shows a multi-layer perceptron
Mehlig (2021), a fully-connected feed-forward neural network with $L$ hidden
layers, $N_{0}$ input components, $N$ neurons per hidden layer with non-linear
activation functions, and $N_{L+1}$ output neurons. The network maps every
input $\boldsymbol{x}^{(0)}=\boldsymbol{x}$ to an output $x^{(L\\!+\\!1)}$.
Weights and thresholds are varied to minimise the output error
$[x^{(L\\!+\\!1)}-t(\boldsymbol{x})]^{2}$, so that the network predicts the
correct target $t(\boldsymbol{x})$ for each input $\boldsymbol{x}$. The
sensitivity of $\boldsymbol{x}^{(\ell)}$ to small changes
$\delta\boldsymbol{x}$ is determined by linearisation,

$\delta\boldsymbol{x}^{(\ell)}\\!=\\!\use@mathgroup\M@U\symAMSb{D}^{(\ell)}\use@mathgroup\M@U\symAMSb{W}^{(\ell)}\\!\cdot\\!\cdot\\!\cdot\,\use@mathgroup\M@U\symAMSb{D}^{(2)}\use@mathgroup\M@U\symAMSb{W}^{(2)}\use@mathgroup\M@U\symAMSb{D}^{(1)}\use@mathgroup\M@U\symAMSb{W}^{(1)}\delta{\boldsymbol{x}}\\!\equiv\\!{\use@mathgroup\M@U\symAMSb{J}}_{\ell}\delta{\boldsymbol{x}}\,.$
(1)

Here, $\use@mathgroup\M@U\symAMSb{W}^{(\ell)}$ are the weight matrices, and
$\mathbb{D}^{(\ell)}$ are diagonal matrices with elements
$D_{ij}^{(\ell)}=g^{\prime}(b_{i}^{(\ell)})\delta_{ij}$, with
$b_{i}^{(\ell)}=\sum_{j=1}^{N_{\ell}}w_{ij}^{(\ell)}x_{j}^{(\ell-1)}\\!-\theta_{i}^{(\ell)}$
and $g^{\prime}(b_{i}^{(\ell)})=\tfrac{\rm d}{{\rm
d}b}g(b)|_{b=b_{i}^{(\ell)}}$. The Jacobian
${\use@mathgroup\M@U\symAMSb{J}}_{\ell}(\boldsymbol{x})$ characterises the
growth or decay of small perturbations to $\boldsymbol{x}$ Ott (2002);
Cvitanović  _et al._ (2020). Its maximal singular value
$\Lambda_{1}^{(\ell)}(\boldsymbol{x})$ increases or decreases exponentially as
a function of $\ell$, with rate
$\lambda_{1}^{(\ell)}(\boldsymbol{x})\equiv\ell^{-1}\log\Lambda_{j}^{(\ell)}(\boldsymbol{x})$.
The singular values
$\Lambda_{1}^{(\ell)}(\boldsymbol{x})>\Lambda_{2}^{(\ell)}(\boldsymbol{x})>\ldots$
are the square roots of the non-negative eigenvalues of the right Cauchy-Green
tensor
$\use@mathgroup\M@U\symAMSb{J}_{\ell}^{\top}(\boldsymbol{x})\use@mathgroup\M@U\symAMSb{J}_{\ell}(\boldsymbol{x})$.
The maximal eigenvector of
$\use@mathgroup\M@U\symAMSb{J}_{\ell}^{\top}(\boldsymbol{x})\use@mathgroup\M@U\symAMSb{J}_{\ell}(\boldsymbol{x})$
determines the direction of maximal stretching, i.e. in which input direction
the output changes the most, starting from a given input $\boldsymbol{x}$.

FTLEs and Cauchy-Green tensors are used in solid mechanics to identify elastic
deformation patterns Truesdell and Noll (1965), and to find regions of
instability in plastic deformation Ren  _et al._ (2012) and crack initiation
Jin  _et al._ (2021). More generally, FTLEs help to characterise the
sensitivity of complex dynamics to initial conditions Vannitsem (2017);
Uthamacumaran (2021); Morozov  _et al._ (2020). In fluid mechanics, they
explain the alignment of particle transported by the fluid Ni  _et al._
(2014); Bezuglyy  _et al._ (2009), providing valuable insight into the
stretching and rotation of fluid elements over time and space Johnson  _et
al._ (2017). FTLEs allow to identify Lagrangian coherent structures Haller and
Yuan (2000); Lucarini and Bódai (2017); Beneitez  _et al._ (2020); strongly
repelling fluid-velocity structures that help to organise and understand flow
patterns Gibson  _et al._ (2008). These geometrical structures appear as
surfaces of large maximal FTLEs, orthogonal to the maximal stretching
direction.

In applying these methods to neural networks, one should recognise several
facts. First, in deep neural networks, the weights change from layer to layer.
Therefore the corresponding dynamical system is not autonomous. Second, the
number $N_{\ell}$ of neurons per layer may change as a function of $\ell$,
corresponding to a changing phase-space dimension. Third, the neural-network
weights are trained. This limits the exponential growth of the maximal
singular value, as we show below. Fourth, one can use different activation
functions, such as the piecewise linear ReLU function Mehlig (2021), or the
smooth tanh function Pennington  _et al._ (2017). Here we use
$g(b)=\mbox{tanh}(b)$, so that the network map is continuously differentiable
just like the dynamical systems for which Lagrangian coherent structures were
found and analysed.

Two-dimensional data set. To illustrate the geometric structures formed by the
maximal FTLE, we first consider a toy problem. The data set [Fig. 1(b)]
comprises $4\times 10^{4}$ input patterns, with 90% used for training, the
rest for testing. We trained fully connected feed-forward networks on this
data set by stochastic gradient descent, minimising the output error
$[x^{(L\\!+\\!1)}-t(\boldsymbol{x})]^{2}$. In this way we obtained
classification accuracies of at least 98%. We considered different network
layouts, changing the numbers of layers and hidden neurons per layer. The
weights were initialised as independent Gaussian random numbers with zero mean
and variance $\sigma_{w}^{2}\sim N^{-1}$, while the thresholds were initially
set to zero. After training, we computed the maximal FTLE in layer $L$ and the
associated stretching direction from Eq. (1) as described in Refs. Eckmann and
Ruelle (1985); Okushima (2003).

Figure 2:  Geometrical FTLE structures in input space for different widths $N$
and depths $L$ of fully-connected feed-forward neural networks trained on the
data set shown schematically in Fig. 1(b). Shown is the colour-coded magnitude
of $L\lambda_{1}^{(L)}(\boldsymbol{x})$, and the maximal stretching directions
(black lines). Figure 3:  (a) Classification error for a fully connected feed-
forward network with $L=2$ hidden layers with random weights (not trained),
and trained output weights, as a function of the number $N$ of hidden neurons
per layer (solid black line). Also shown is the classification error for the
fully trained network (dashed line). Both curves were obtained for the data
set shown schematically in Fig. 1(b). (b) Evolution of maximal-FTLE
distribution as a function of training time measured in epochs Mehlig (2021),
for a network with $L=8$ hidden layers with $N=50$ neurons per layer. The
weights were initialised with different variances, $\log
GN\sigma_{w}^{2}=-0.2$ (blue), $0$ (green), and $0.2$ (red).

The results are summarised in Figure 2, which shows maximal-FTLE fields for
trained networks with different layouts. First, we see that the ridges of
large positive $\lambda_{1}^{(L)}({\boldsymbol{x}})$ align with the decision
boundary between the two classes [Fig. 1(b)]. The ridges are most prominent
for small $N$ and large $L$. In this case, the network learns by grouping the
inputs into two different basins of attraction for $t=\pm 1$, separated by a
ridge of positive $\lambda_{1}^{(L)}({\boldsymbol{x}})$. A small shift of the
input across the decision boundary leads to a substantial change in the
output.

Second, the contrast increases as $L$ becomes larger, quantifying the
exponential expressivity of deep neural networks. For larger $L$, the network
can resolve smaller input distances $\delta\boldsymbol{x}$ because the
singular values increase/decrease exponentially from layer to layer. Comparing
networks of different depths, we find that
$L\lambda_{1}^{(L)}(\boldsymbol{x})$ saturates for large $L$, on the ridge.
This is a consequence of the training: the network learns to produce output
differences on the order of $\delta x^{(L+1)}\sim 1$, and to resolve input
differences $\delta\boldsymbol{x}$ on the scale of the mean distance between
neighbouring patterns over the decision boundary. Therefore, the saturation
value is larger when the number densities of input pattern is higher (not
shown). Even though $\Lambda_{1}^{(L)}(\boldsymbol{x})$ saturates,
$\Lambda_{2}^{(L)}(\boldsymbol{x})<1$ decreases exponentially as $L$ grows
(not shown), thereby causing
$\Lambda_{1}^{(L)}(\boldsymbol{x})/\Lambda_{2}^{(L)}(\boldsymbol{x})$ to
increase exponentially, as in dynamical systems Haller (2011).

Third, the ridges gradually disappear as the number $N$ of hidden neurons per
layer increases, because the maximal singular value of
$\use@mathgroup\M@U\symAMSb{J}_{L}(\boldsymbol{x})$ approaches a definite
$\boldsymbol{x}$-independent limit as $N\to\infty$ at fixed $L$. In the
infinite-width limit, training is equivalent to kernel regression with a
kernel that is independent of the inputs in the training data set Chen  _et
al._ (2020); Jacot  _et al._ (2018). But how can the network distinguish
inputs with different targets in this case, without ridges indicating decision
boundaries? One possibility is that the large number of hidden neurons allows
the network to embed the inputs into a high-dimensional space where they can
be separated thanks to the universal approximation theorem Kidger and Lyons
(2020). In this case, training only the output weights (and threshold) should
suffice. Figure 3(a) confirms this, as the classification error decreases with
increasing embedding dimension, for random hidden weights. We remark that the
classification error of the fully trained network is smaller than the error
with random hidden weights. This is not surprising, since different random
embeddings have different classification errors when the number of patterns
exceeds twice the embedding dimension Cover (1965); Mehlig (2021).

Fourth, Figure 2 also shows the maximal stretching directions. For large $L$
they become orthogonal to the ridges of large
$\lambda_{1}^{(L)}(\boldsymbol{x})$. This demonstrates that there is a
stringent analogy between the FTLE ridges of deep neural networks and
Lagrangian coherent structures. The stretching patterns appear to exhibit
singular points where the maximal stretching tends to zero Wilkinson  _et al._
(2009), reflecting topological constraints imposed on the direction field.

Fifth, one may wonder how the FTLE structures depend on weight initialisation.
When weights are initialised with a small variance, $\sigma_{w}\ll 1$, most
FTLEs are negative initially [blue in Fig. 3(b)]. This implies a slowing down
of the initial training (vanishing-gradient problem). To see this, consider
the fundamental forward-backward dichotomy of deep neural networks Mehlig
(2021): weight updates in the stochastic-gradient algorithm are given by
$\delta w_{mn}^{(\ell)}\propto\Delta_{m}^{(\ell)}x_{n}^{(\ell-1)}$, where

$[\boldsymbol{\Delta}^{(\ell)}]^{\sf T}=[\boldsymbol{\Delta}^{(L)}]^{\sf
T}\use@mathgroup\M@U\symAMSb{D}^{(L)}\use@mathgroup\M@U\symAMSb{W}^{(L)}\cdots\use@mathgroup\M@U\symAMSb{D}^{(\ell\\!+\\!1)}\use@mathgroup\M@U\symAMSb{W}^{(\ell\\!+\\!1)}\use@mathgroup\M@U\symAMSb{D}^{(\ell)}\,,$
(2)

and
$\Delta^{(L)}_{j}=g^{\prime}(b^{(L+1)})[x^{(L+1)}\\!-\\!t(\boldsymbol{x})]g^{\prime}(b_{j}^{(L)})w_{j}^{(L+1)}$.
It follows from Eq. (2) that negative FTLEs cause small weight increments
$\delta w_{mn}^{(\ell)}$. Conversely, when the maximal FTLE is positive and
too large, the weights grow rapidly, leading to training instabilities.
Remarkably, Figure 3(b) demonstrates a self-organising effect due to training:
the distributions of the maximal FTLE converge to centre around zero. This is
explained by the fact that the network learns by creating maximal-FTLE ridges
in input space: to accommodate positive and negative
$\lambda_{1}^{(L)}(\boldsymbol{x})$, the distribution centres around zero,
alleviating the unstable-gradient problem.

Figure 4:  Maximal-FTLE field for the MNIST data LeCun  _et al._ (2018). A
fully connected feed-forward network with $N=20$ neurons per hidden layer,
$L=16$ hidden layers, and a softmax layer with ten outputs was trained to a
classification accuracy of 98.88%. The maximal FTLE was calculated for each of
the $28^{2}$-dimensional inputs and projected to two dimensions (see text).
(a) Training data in the non-linear projection. For each input, the maximal
FTLE $\lambda_{1}^{(L)}$ is shown colour-coded (legend). The box contains 93%
of the recognised digits $0$. A threefold blow up of this box is also shown.
The line represents a sequence of adversarial attacks from $9$ to $4$ (see
text), with $\lambda_{1}^{(L)}(\boldsymbol{x})$ colour-coded. (b)
Classification error on the test set as a function of
$\lambda_{1}^{(L)}(\boldsymbol{x})$. (c) Predictive uncertainty $H$ (see text)
as a function of $\lambda_{1}^{(L)}(\boldsymbol{x})$.

MNIST data set. This data set consists of 60,000 images of handwritten digits
$0$ to $9$. Each grayscale image has $28\times 28$ pixels and was pre-
processed to facilitate machine learning LeCun  _et al._ (2018). Deep neural
networks can achieve high precision in classifying this data, with accuracies
of up to $99.77$% on a test set of 10,000 digits Ciregan  _et al._ (2012).

We determined the maximal-FTLE field for this data set for a network with
$L=16$ hidden layers, each containing $N=20$ neurons, and a standard softmax
layer with ten outputs Mehlig (2021). To visualise the geometrical structures
in the $28^{2}$-dimensional input space, we projected it to two dimensions as
follows. We added a bottleneck layer with two neurons to the fully trained
network, just before the softmax-output layer. We retrained only the weights
and thresholds of this additional layer and the output layer, keeping all
other hidden neurons unchanged. The local fields $b_{1}$ and $b_{2}$ of the
two bottleneck neurons are the coordinates of the two-dimensional
representation shown in Figure 4(a). We see that the input data separate into
ten distinct clusters corresponding to the ten digits. The maximal FTLEs at
the centre of these clusters are very small or even negative, indicating that
the output is not sensitive to small input changes. These regions are
delineated by areas with significantly larger positive FTLEs [see
$3\times$zoom in panel (a)]. Figure 2 leads us to expect that patterns with
large $\lambda_{1}^{(L)}(\boldsymbol{x})$ are located near the decision
boundaries in high-dimensional input space. This is verified by strong
correlations between $\lambda_{1}^{(L)}(\boldsymbol{x})$ and both the
classification error and the predictive uncertainty. Figure 4(b) shows that
the classification error on the test set is larger for inputs $\boldsymbol{x}$
with larger $\lambda_{1}^{(L)}(\boldsymbol{x})$. Figure 4(c) shows that large
values of $\lambda_{1}^{(L)}(\boldsymbol{x})$ correlate with high predictive
uncertainty, measured by the entropy $H$ of the posterior predictive
distribution Gawlikowski  _et al._ (2021). For softmax outputs, where
$x_{i}^{(L+1)}$ can be interpreted as probabilities, $H=-\sum_{i}\langle
x_{i}^{(L+1)}\rangle\log\langle x_{i}^{(L+1)}\rangle$, where
$\langle\cdot\rangle$ denotes an average over an ensemble of networks (here
with ten members) with the same layout but different weight initialisations
Lakshminarayanan  _et al._ (2017); Hoffmann and Elster (2021). These
observations confirm that ridges of maximal FTLEs localise the decision
boundaries.

Figure 4(a) also shows $\lambda_{1}^{(L)}(\boldsymbol{x})$ along a path
generated by an adversarial attack. The attack begins from a sample within the
cluster corresponding to the digit $9$ and aims to transform it into a digit
$4$ by making small perturbations to the input data Madry  _et al._ (2018)
toward class $4$. We see that the maximal FTLE is small at first, then
increases as the path approaches the decision boundary, and eventually
decreases again. This indicates that our conclusions regarding the
correlations between large maximal FTLEs and decision boundaries extend to
neighbourhoods of the MNIST training set that contain patterns the network has
not encountered during training.

Conclusions. We explored geometrical structures formed by the maximal FTLE in
input space, for deep neural networks trained on different classification
problems. We found that ridges of positive exponents define the decision
boundaries, for a two-dimensional toy classification problem, and for a high-
dimensional data set of hand-written digits. In the latter case, we projected
the high-dimensional input space to two dimensions, and found that the network
maps digits into distinct clusters surrounded by FTLE ridges at the decision
boundaries. This conclusion is supported by the fact that the locations of the
FTLE ridges correlate with low classification accuracy and high predictive
uncertainty.

The network layout determines how prominent the FTLE structures are. As the
number of layers increases, the ridges sharpen, emphasising their role in
learning and classification. However, as the number of hidden neurons per
layer tends to infinity, the FTLE structures disappear. In this limit, the
network separates the inputs by embedding them into a high-dimensional space,
rendering training of the hidden neurons unnecessary.

It is important to underscore that the two different ways to learn, by FTLE
ridges or embedding, result in qualitative differences regarding
classification errors and predictive uncertainties, and may also affect how
susceptible a network is to adversarial attacks. The geometrical method
presented here extends to other network architectures (such as convolutional
networks), and will help to visualise and understand the mechanisms that allow
such neural networks to learn.

###### Acknowledgements.

LS was supported by grants from the Knut and Alice Wallenberg (KAW) Foundation
(no. 2019.0079) and Vetenskapsrådet (VR), no. 2021-4452. JB received support
from UCA-JEDI Future Investments (grant no. ANR-15-IDEX-01). HL was supported
by a grant from the KAW Foundation. KG was supported by VR grant 2018-03974,
and BM by VR grant 2021-4452. Part of the the numerical computations for this
project were performed on resources provided by the Swedish National
Infrastructure for Computing (SNIC).

## References

  * J Jumper et al. (2021) J Jumper et al., “Highly accurate protein structure prediction with AlphaFold,” Nature 596, 583–589 (2021).
  * Poole  _et al._ (2016) B Poole, S Lahiri, M Raghu, J Sohl-Dickstein, and S Ganguli, “Exponential expressivity in deep neural networks through transient chaos,” Advances in neural information processing systems 29 (2016).
  * Mehlig (2021) B Mehlig,  _Machine learning with neural networks: an introduction for scientists and engineers_ (Cambridge University Press, 2021).
  * Minsky and Papert (1969) M Minsky and S Papert,  _Perceptrons. An Introduction to Computational Geometry_ (MIT Press, 1969).
  * Ott (2002) E Ott,  _Chaos in Dynamical Systems_, 2nd ed. (Cambridge University Press, 2002).
  * Cvitanović  _et al._ (2020) P Cvitanović, R Artuso, R Mainieri, G Tanner, and G Vattay,  _Chaos: Classical and Quantum_ (Niels Bohr Institute, Copenhagen, 2020).
  * Crisanti  _et al._ (1993) A Crisanti, A Vulpiani, and G Paladin,  _Products of random matrices in Statistical Physics_ (Springer, Berlin, 1993).
  * Pennington  _et al._ (2017) J Pennington, S Schoenholz, and S Ganguli, “Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice,” Advances in neural information processing systems 30 (2017).
  * Sutskever  _et al._ (2013) I Sutskever, J Martens, G Dahl, and GE Hinton, “On the importance of initialization and momentum in deep learning,” in  _Proceedings of the 30th International Conference on Machine Learning - Volume 28_ (2013) pp. III–1139–III–1147.
  * Glorot and Bengio (2010) X Glorot and Y Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in  _Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics_ , Proceedings of Machine Learning Research, Vol. 9, edited by YW Teh and M Titterington (JMLR Workshop and Conference Proceedings, Chia Laguna Resort, Sardinia, Italy, 2010) pp. 249–256.
  * Jaeger and Haas (2004) H Jaeger and H Haas, “Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication,” Science 304, 78–80 (2004).
  * Pathak  _et al._ (2018) J Pathak, B Hunt, M Girvan, Z Lu, and E Ott, “Model-free prediction of large spatiotemporally chaotic systems from data: A reservoir computing approach,” Physical Review Letters 120, 024102 (2018).
  * Storm  _et al._ (2022) L Storm, K Gustavsson, and B Mehlig, “Constraints on parameter choices for successful time-series prediction with echo-state networks,” Machine Learning: Science and Technology 3, 045021 (2022).
  * Vogt  _et al._ (2020) R Vogt, MP Touzel, E Shlizerman, and G Lajoie, “On Lyapunov exponents for RNNs: Understanding information propagation using dynamical systems tools,” (2020), arXiv:2006.14123 [cs.LG] .
  * Haller and Yuan (2000) G Haller and G Yuan, “Lagrangian coherent structures and mixing in two-dimensional turbulence,” Physica D 147, 352–370 (2000).
  * Lucarini and Bódai (2017) V Lucarini and T Bódai, “Edge states in the climate system: exploring global instabilities and critical transitions,” Nonlinearity 30, R32 (2017).
  * Beneitez  _et al._ (2020) M Beneitez, Y Duguet, P Schlatter, and DS Henningson, “Edge manifold as a Lagrangian coherent structure in a high-dimensional state space,” Phys. Rev. Res. 2, 033258 (2020).
  * Truesdell and Noll (1965) C Truesdell and W Noll, “The Non-Linear Field Theories of Mechanics,” in  _The Non-Linear Field Theories of Mechanics_ , Encyclopedia of Physics, Vol. III, edited by S. Flügge (Springer, Berlin, 1965) pp. 1–579.
  * Ren  _et al._ (2012) JL Ren, C Chen, ZY Liu, R Li, and G Wang, “Plastic dynamics transition between chaotic and self-organized critical states in a glassy metal via a multifractal intermediate,” Phys. Rev. B 86, 134303 (2012).
  * Jin  _et al._ (2021) X Jin, Y Chen, L Wang, H Han, and P Chen, “Failure prediction, monitoring and diagnosis methods for slewing bearings of large-scale wind turbine: A review,” Measurement 172, 108855 (2021).
  * Vannitsem (2017) S Vannitsem, “Predictability of large-scale atmospheric motions: Lyapunov exponents and error dynamics,” Chaos 27, 032101 (2017).
  * Uthamacumaran (2021) A Uthamacumaran, “A review of dynamical systems approaches for the detection of chaotic attractors in cancer networks,” Patterns 2, 100226 (2021).
  * Morozov  _et al._ (2020) A Morozov, K Abbott, K Cuddington, T Francis, G Gellner, A Hastings, Y-C Lai, S Petrovskii, K Scranton, and ML Zeeman, “Long transients in ecology: Theory and applications,” Phys. Life Rev. 32, 1–40 (2020).
  * Ni  _et al._ (2014) R Ni, NT Ouellette, and GA Voth, “Alignment of vorticity and rods with Lagrangian fluid stretching in turbulence,” Journal of Fluid Mechanics 743, R3 (2014).
  * Bezuglyy  _et al._ (2009) V Bezuglyy, M Wilkinson, and B Mehlig, “Poincare indices of rheoscopic visualisations,” Europhys. Lett. (2009).
  * Johnson  _et al._ (2017) PL Johnson, SS Hamilton, R Burns, and C Meneveau, “Analysis of geometrical and statistical features of Lagrangian stretching in turbulent channel flow using a database task-parallel particle tracking algorithm,” Phys. Rev. Fluids 2, 014605 (2017).
  * Gibson  _et al._ (2008) JF Gibson, J Halcrow, and P Cvitanovic, “Visualizing the geometry of state space in plane Couette flow,” Journal of Fluid Mechanics 611, 107–130 (2008).
  * Eckmann and Ruelle (1985) JP Eckmann and D Ruelle, “Ergodic theory of chaos and strange attractors,” Rev. Mod. Phys. 57, 617–656 (1985).
  * Okushima (2003) T Okushima, “New method for computing finite-time Lyapunov exponents,” Phys. Rev. Lett. 91, 254101 (2003).
  * Haller (2011) G Haller, “A variational theory of hyperbolic Lagrangian coherent structures,” Physica D 240, 574–598 (2011).
  * Chen  _et al._ (2020) S Chen, H He, and W Su, “Label-aware neural tangent kernel: Toward better generalization and local elasticity,” NIPS 33, 15847–15858 (2020).
  * Jacot  _et al._ (2018) A Jacot, F Gabriel, and C Hongler, “Neural tangent kernel: Convergence and generalization in neural networks,” NIPS 31 (2018).
  * Kidger and Lyons (2020) P Kidger and T Lyons, “Universal approximation with deep narrow networks,” (2020), arXiv:1905.08539 [cs.LG] .
  * Cover (1965) TM Cover, “Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition,” IEEE Trans. on electronic computers , 326 (1965).
  * Wilkinson  _et al._ (2009) M Wilkinson, V Bezuglyy, and B Mehlig, “Fingerprints of random flows?” Physics of Fluids 21 (2009), 043304.
  * LeCun  _et al._ (2018) Y LeCun, C Cortes, and CJC Burges, “The MNIST database of handwritten digits,”  yann.lecun.com/exdb/mnist (2018).
  * Ciregan  _et al._ (2012) D Ciregan, U Meier, and J Schmidhuber, “Multi-column deep neural networks for image classification,” in  _2012 IEEE Conference on Computer Vision and Pattern Recognition_ (2012) pp. 3642–3649.
  * Gawlikowski  _et al._ (2021) J Gawlikowski, CRN Tassi, M Ali, J Lee, M Humt, J Feng, A Kruspe, R Triebel, P Jung, R Roscher, _et al._ , “A survey of uncertainty in deep neural networks,” arXiv:2107.03342 (2021).
  * Lakshminarayanan  _et al._ (2017) B Lakshminarayanan, A Pritzel, and C Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles,” NIPS 30 (2017).
  * Hoffmann and Elster (2021) L Hoffmann and C Elster, “Deep ensembles from a Bayesian perspective,” arXiv:2105.13283 (2021).
  * Madry  _et al._ (2018) A Madry, A Makelov, L Schmidt, D Tsipras, and A Vladu, “Towards deep learning models resistant to adversarial attacks,” in  _International Conference on Learning Representations_ (2018).