# Effect of microstructure on fatigue properties of hyperelastic materials

Anna Stepashkina  Fuguang Chen  Lipeng Chen<EMAIL_ADDRESS>Research
Center for New Materials Computing, Zhejiang Lab, Hangzhou, China

###### Abstract

Homogenization is a technique for the analysis of complex materials by
replacing them with equivalent homogeneous materials that exhibit similar
properties. By constructing a three-dimensional (3D) porous material model and
employing homogenization technique, effective properties of the hydrogel pore
structure were calculated. It is found that the microstructure of hyperelastic
materials has considerable influence on their macroscopic mechanical
properties, pores with a radius of up to 65 microns at a small strain can
significantly reinforce material structure and improve its fatigue resistance.
This work highlights the potential of engineering pore structures for the
enhancement of mchanical properties and durability of hydrogels.

##  I Introduction

Due to their unique elastic properties that remain consistent regardless of
the rate of deformation, hyperelastic materials are widely used in the
production of biological materials, foams, rubbers [1, 2, 3, 4, 5, 6, 7, 8].
Some hyperelastic materials can be compressed by a strain of more than 70% [5,
7, 9]. Additionally, these materials often exhibit a Poisson ratio close to
0.5. When analyzing mechanical properties of hyperelatsic materials using
classical mechanics models, such as Cauchy’s theorem, one typically obtains
incorrect results. Hyperelastic materials can be viewed as a special case of
Cauchy elastic materials. In order to accurately describe deformation
properties of hyperelastic materials and calculate the stress-strain curve,
rheological mechanics models are deemed the most appropriate ones. These
hyperelastic rheological models work as types of constitutive models for ideal
elastic materials, with the stress-strain relation derived from a strain
energy density function

$\sigma=J^{-1}\frac{\delta{W_{s}}}{\delta{F}}F^{T}.$ (1)

Here, $J$ is the elastic volume ratio ($J=\mathrm{det}F$), $F$ is the
deformation gradient, $W_{s}$ is the strain energy density. The stress-strain
relation for hyperelastic materials is described by nonlinear mechanical
models, including nonlinear elastic and nonlinear viscoelastic models [10,
11]. The most famous models are neo-Hookean [10, 9, 12], Mooney-Rivlin [10,
13], Ogden [10, 13, 14], Storakers [15].

The neo-Hookean model [10, 9, 12] is a hyperelastic material model, which can
be viewed as an extension of Hooke’s law that can predict the nonlinear
stress-strain behavior of materials undergoing large deformations. In contrast
to linear elastic materials, the stress-strain relation of a neo-Hookean
material is initially linear and goes to plateau at certain strains. In neo-
Hookean model, one does not account for the dissipation of energy as heat when
straining the material, and perfect elasticity is assumed at all stages of
deformation.

Based on the statistical thermodynamics of cross-linked polymer chains, the
neo-Hookean model is usable for plastics and rubber-like substances [10, 9,
12]. Since polymer chains can move relative to each other when a stress is
applied, cross-linked polymers will behave in a neo-Hookean manner. However,
polymer chains will be stretched to a maximum point at which covalent cross
links will allow, leading to a dramatic increase in the elastic modulus of the
material. For neo-Hookean model, the energy density $W_{s}$ is defined as

$W_{s}=\frac{\mu}{2}(I-3)-\mu\mathrm{ln}J+\frac{\lambda}{2}\mathrm{ln}^{2}J,$
(2)

where $I$ is the deformation tensor trace, and $\mu$, $\lambda$ are Lame
parameters,

$\mu=\frac{E}{2(1+\nu)},$ (3) $\lambda=\frac{E\nu}{(1+\nu)(1-2\nu)}.$ (4)

Here, $\nu$ is the Poisson ratio, and $E$ is the Young’s modulus.

The neo-Hookean model has been superseded by the Mooney-Rivlin model [10],
which can describe the stress-strain relation with high accuracy. It works
well from small to medium strains [13]. The Mooney-Rivlin model has special
phenomenological parameters. For the one direction compression, $W_{s}$ can be
written as

$W_{s}=C_{10}(I_{1}-3)+C_{01}(I_{2}-3)+\frac{\kappa}{2}(J-1)^{2}.$ (5)

Here, $\kappa=\frac{E}{3(1-2\nu)}$ is the bulk modulus, $C_{10}$, $C_{01}$ are
Mooney-Rivlin parameters. These parameters can be obtained by fitting
experimental test data with the analytical expression describing the stress-
strain relation. $C_{10}$, $C_{01}$ can be calculated from the system of
equations

$\sigma_{i}=2(C_{10}+\frac{C_{01}}{\epsilon_{i}})(\epsilon_{i}-\frac{1}{\epsilon_{i}^{2}}).$
(6)

Here, $\sigma_{i}$ is the stress corresponding to the deformation
$\epsilon_{i}$.

In the Ogden model [10, 13, 14], the strain energy function $W_{s}$ is defined
as

$W_{s}=\frac{\mu_{\mathrm{mod}}}{\alpha}(\lambda_{1}^{\alpha}+\lambda_{2}^{\alpha}+\lambda_{3}^{\alpha}-3)+\frac{\mu_{\mathrm{mod}}}{2\alpha\beta}(J^{-\alpha\beta}-1)^{2},$
(7)

with

$\beta=\frac{\nu}{1-2\nu},$ (8) $\mu_{\mathrm{mod}}=0.5\frac{E}{1+\nu},$ (9)

where $\mu_{\mathrm{mod}}$ is the shear modulus, $\alpha$ is a constant which
can be determined from the equation

$\sigma_{i}=2\frac{\mu_{\mathrm{mod}}}{\alpha}(\lambda^{\alpha}_{i}-(\frac{1}{\sqrt{\lambda_{i}}})^{\alpha}).$
(10)

The Mooney-Rivlin model and Odgen model work well only from small to medium
deformation, for the large deformation one needs to employ the Storakers model
[15], which yields better results in the description of stress-strain
relation. The Storakers model is a modern version of previous phenomenological
models [10, 11, 9, 12, 13, 14], often used to model highly compressible
samples. Assuming that only elastic deformation is allowed, one can obtain
$W_{s}$ of an isotropic hyperelastic material

$W_{s}=\frac{2\mu_{\mathrm{mod}}}{\alpha}(\lambda_{1}^{\alpha}+\lambda_{2}^{\alpha}+\lambda_{3}^{\alpha}-3+\frac{1}{\beta}(J^{-\alpha\beta}-1)).$
(11)

Determining empirical parameters is indeed crucial when using rheological
models to describe the mechanical properties of materials. These parameters
enable us to construct stress-strain curves and analyze various applied
problems, such as fatigue behavior. However, in traditional rheological
models, one often neglects the microstructure of materials, which largely
restricts their application to materials with complex microstructures, such as
porous hyperelastic materials.

In this work, we proposed a novel approach using the homogenization technique
to predict the mechanical properties of porous hyperelastic materials.
Homogenized properties are used to model the mechanical behavior of composite
materials or heterogeneous structures. This significantly simplifies the
analysis and design processes while capturing the overall mechanical behavior
of materials. By including microstructures into the model, it is possible to
improve the predictive accuracy and develop more realistic rheological models
for hyperelatic materials.

##  II Mechanical properties of homogenized hyperelastic materials

Experimental studies of mechanical properties and microstructures of
hyperelastic materials, in particular, hydrogels and biological materials,
have shown that these materials have porous structures with pore sizes ranging
from 50 um to 100 um [2, 3, 5, 8, 16]. Following the research in Ref. [8], we
employ aforementioned methods to construct approximated models to calculate
the stress-strain curve of hydrogel. To this end, we need to obtain the
empirical parameters in the model, which can be determined by fitting the
model to a set of measurements. The method for determining empirical
parameters depends on the quality of the experiment, the range of
deformations, and the loading method. To find the accurate solution, it is
often necessary to carry out a large number of measurements.

For the neo-Hookean model, we can build the dependence directly since all
parameters are known. The optimal values of empirical parameters in other
models (the Mooney-Rivlin model, the Storakers model, the Odgen model) are
determined using the least squares method based on the experimental stress-
strain curve. Fig. 1 shows the stress-strain curve of hydrogel calculated by
neo-Hookean, Mooney-Rivlin, Odgen and Storakers models. These rheological
models are often based on certain simplifications and may not capture all the
complexities of material behavior, especially in the presence of pores or
other microstructural features. To accurately describe the effect of
pores/inclusions on mechanical properties, we build a homogenized model of the
material using the finite element method. Homogenization refers to the process
of calculating the effective properties of a material, which involves treating
the porous structure as a whole material.

Figure 1: the stress-strain curve for hydrogel

Constructing the microstructure of a material is a complex process that
requires careful design. The arrangement of pores in the material is chaotic,
and their shapes exhibit complex geometric structures [2, 3, 8]. In the
homogenization process, a cube with pores was considered as a periodic element
of the material. To a first approximation, the pores were spherical objects
located periodically,

  * •

one sphere in the center of the cube,

  * •

6 spheres in the center of the cube faces,

  * •

8 spheres at the vertices of the cube.

The cell structure is shown in Fig. 2 (A). The size of the cubic cell is 300
um, and the diameter of the sphere is calculated based on the pores volume
fraction 10% of the material (radius r $\approx$ 47 um).

Figure 2: A: microstructure model of hydrogel (pores volume fraction 10%); B:
deformation during pressing (pores volume fraction 10%, strain 20%), C:
fatigue (pores volume fraction 10%, strain 20%)

Calculating homogenized properties involves the simulation of averaged
properties of the material within a small volume that exhibits structural
periodicity. Instead of simulating the entire material, a small representative
volume element (RVE) is selected. The RVE is chosen to be small enough to
capture the local microstructure but large enough to represent the overall
behavior of the material. Homogenized properties were averaged by the
properties of each constituent phase within the RVE, which can be determined
by the system of equation with periodical boundary conditions,

$0=\nabla(F\cdot{S})^{T}+\textbf{F}\cdot{v}$ (12)

,

$F=I+\nabla\textbf{u}.$ (13)

Here, $F$ is the deformation gradient tensor, which describes how the body
deforms in response to applied forces or displacements. It relates the current
configuration of the body to its reference (undeformed) configuration. $S$ is
the second Piola-Kirchhoff stress tensor, which represents the stress state
within the body. It accounts for the internal forces within the material.
$\textbf{F}\cdot{v}$ represents the external body forces per unit volume
acting on the body. These forces include gravity, pressure, or other applied
loads. $\nabla\textbf{u}$ is the displacement gradient tensor, and $I$ is the
stretch tensor which represents the relative deformation or change in shape of
the body. Eq. 12 represents the equilibrium condition for a deformable body,
which states that the sum of internal and external forces acting on a small
volume element of the body is zero.

Upon solving the homogenization problem, one obtains Poisson’s ratio, Young’s
modulus and shear modulus for the porous hydrogel. The stress-strain curve
obtained by the homogenization method is displayed in Fig. 1, which increases
accuracy significantly as compared to empirical models.

##  III Fatigue analysis for homogenized material

Since the homogenization method yields results in good agreement with
experiment, we will systematically investigate how pores affect mechanical
properties of the material. As a control simulation, we will study the fatigue
of the material, which depends on factors such as the applied load,
microstructure, external conditions. We consider a closed system at
temperature 20°C and pressure 105 Pa. During operation, hydrogel materials are
usually subjected to cyclic loading, and we consider 4 compression modes,

  * •

0%-5%-0%,

  * •

0%-10%-0%,

  * •

0%-20%-0%,

  * •

0%-30%-0%.

The higher the deformation, the faster fatigue develops, but cycles to failure
can change nonlinearly. The microstructures, such as pore size and volume
fraction can affect significantly the fatigue characteristics of the material.
The coarse-grained structure increases fatigue, and reaching a certain
concentration of inclusions can increase cycles to failure. As a crucial
parameter for studying the influence of inclusion size, we will take the
volume fraction of pores, according to which the diameter of spherical
inclusions is calculated. We study the volume fraction of pores ranging from
5% to 35% (sphere diameter from $\approx$ 37 um to 72 um). The fatigue model
is sketched in Figs 2B, C.

Figure 3: Fatigue for different pores volume fractions.

The highest mechanical stress concentrations are observed at the boundaries of
spherical inclusions within the hyperelastic material, as illustrated in Fig
2B. This phenomenon can be attributed to the abrupt changes in material
properties at the interface between the inclusions and the surrounding matrix.
When an external load is applied to the material, the stress tends to
concentrate at the boundaries of the inclusions because of the mismatch in
stiffness or elasticity between two phases. The stress builds up at the
interfaces, leading to localized regions of high stress. This so-called stress
concentration effect has important implications on the mechanical behavior and
failure mechanism of the material. The regions with high stress concentrations
are more susceptible to the initiation and propagation of defects, such as
cracks or voids. These defects can act as stress concentrators, further
exacerbating the stress concentrations and potentially leading to premature
failure of the material.

The number of cycles to failure means the minimum value of cycles when
irreversible deformations occur. It leads to small defects, and further
exploitation of the material can result in further destruction, even total
destruction. By treating the hyperelatic material as a composite, the pores
can act as reinforcements, similar to the role of fibers in a fiber-reinforced
composite. The pores can hinder the propagation of cracks and defects, leading
to improved toughness and resistance to failure. In addition, the pores can
also facilitate the energy absorption and dissipation, which contributes to
the overall mechanical performance of the material.

Fig. 3 displays the fatigue behavior of hydrogel as a function of pores volume
fraction at strains of 5%, 10%, 20%, 30%. Under cyclic loading with a small
compressive strain of 5%, the number of compression cycles increases with the
pore volume fraction, leading to the reinforced structure. Upon further
increasing the pore volume fraction ($\geq$ 25%, pore diameter r $\geq$ 65
um), the fatigue characteristics of the material deteriorate sharply. For
cyclic loadings with a large value of compressive strain, one can also observe
the reinforced effect of pores, but not so significantly. It is found that the
minimum number of cycles at a compressive strain of 30% for a maximum pore
volume of 35% (r $\approx$ 37 um) is 2190.

##  IV Conclusion

By using the homogenization technique and building a 3D porous material model,
we derived effective properties of porous hydrogels. Homogenization involves
treating the complex microstructure as an equivalent homogeneous material,
allowing for simpler analysis while still capturing the overall behavior. This
approach enables us to calculate the effective elastic modulus, Poisson’s
ratio, and other mechanical properties of the hydrogel with pores. By
modelling the deformation and stress distribution within the hydrogel under
various loading conditions, we obtain highly accurate stress-strain curve.
Under cyclic loading, we also studied the fatigue behavior of the porous
hydrogel, which provides detailed information on the durability and lifespan
of the hydrogel under realistic service conditions. It is found that pores
with a radius of up to 65 um at a small compressive strain can significantly
strengthen the material structure and improve the fatigue performance.

The combination of homogenization technique, a detailed 3D microstructure
model and fatigue analysis offers a comprehensive understanding of the
mechanical response of hyperelastic materials like hydrogels. This knowledge
have important applications in fields such as biomedical engineering, soft
robotics, and tissue engineering, where mechanical properties and durability
of these materials play a critical role in their functionality and
reliability. Work in this direction is in progress.

## References

  * [1] Tiancheng Wang, Yanrui Xu, Ruolan Li, Haoping Wang, Yuege Xiong, and Xiaokun Wang.  Simulating hyperelastic materials with anisotropic stiffness models in a particle-based framework.  Computers & Graphics, 116:437–447, 2023.
  * [2] Eva Baur, Matteo Hirsch, and Esther Amstad.  Porous 3d printable hydrogels.  Advanced Materials Technologies, 8(9):2201763, 2023.
  * [3] Xia Zhong, Chengdong Ji, Andrew KL Chan, Sergei G Kazarian, Andrew Ruys, and Fariba Dehghani.  Fabrication of chitosan/poly ($\varepsilon$-caprolactone) composite hydrogels for tissue engineering applications.  Journal of Materials Science: Materials in Medicine, 22:279–288, 2011.
  * [4] Majid Shahzad, Ali Kamran, Muhammad Zeeshan Siddiqui, and Muhammad Farhan.  Mechanical characterization and fe modelling of a hyperelastic material.  Materials research, 18:918–924, 2015.
  * [5] Hossein B Khaniki, Mergen H Ghayesh, Rey Chin, and Marco Amabili.  Hyperelastic structures: A review on the mechanics and biomechanics.  International Journal of Non-Linear Mechanics, 148:104275, 2023.
  * [6] Tomasz, Katarzyna Kazimierska-Drobny, and Mariusz Kaczmarek.  Stiffness and pre-stretching estimation from indentation test of hyperelastic membrane.  International Journal of Mechanical Sciences, 274:109235, 2024.
  * [7] D Sutarya, IR Pamungkas, H Sukma, and DS A Sartono.  The effect of catalysts addition to silicone rubber as pressure transmitting medium in the pebble fuel production process.  In Journal of Physics: Conference Series, volume 2048, page 012017. IOP Publishing, 2021.
  * [8] Xihao Pan, Rui Li, Wenyue Li, Wei Sun, Yiyang Yan, Xiaochen Xiang, Jinghua Fang, Youguo Liao, Chang Xie, Xiaozhao Wang, et al.  Silk fibroin hydrogel adhesive enables sealed-tight reconstruction of meniscus tears.  Nature Communications, 15(1):2651, 2024.
  * [9] LRG Treloar.  The elasticity of a network of long-chain molecules—ii.  Transactions of the Faraday Society, 39:241–246, 1943.
  * [10] Beomkeun Kim, Seong Beom Lee, Jayone Lee, Sehyun Cho, Hyungmin Park, Sanghoon Yeom, and Sung Han Park.  A comparison among neo-hookean model, mooney-rivlin model, and ogden model for chloroprene rubber.  International Journal of Precision Engineering and Manufacturing, 13:759–764, 2012.
  * [11] Jorgen S Bergstrom.  Mechanics of solid polymers: theory and computational modeling.  William Andrew, 2015.
  * [12] Thomas J Pence and Kun Gou.  On compressible versions of the incompressible neo-hookean material.  Mathematics and Mechanics of Solids, 20(2):157–182, 2015.
  * [13] Stephen K Melly, Liwu Liu, Yanju Liu, and Jinsong Leng.  A review on material models for isotropic hyperelasticity.  International Journal of Mechanical System Dynamics, 1(1):71–88, 2021.
  * [14] Raymond W Ogden, Giuseppe Saccomandi, and Ivonne Sgura.  Fitting hyperelastic models to experimental data.  Computational Mechanics, 34:484–502, 2004.
  * [15] B Storåkers.  On material representation and constitutive branching in finite compressible elasticity.  Journal of the Mechanics and Physics of Solids, 34(2):125–145, 1986.
  * [16] Jonathan J Campbell, Anke Husmann, Robert D Hume, Christine J Watson, and Ruth E Cameron.  Development of three-dimensional collagen scaffolds with controlled architecture for cell migration studies using breast cancer cell lines.  Biomaterials, 114:34–43, 2017.


# Exact Mutual information for lognormal random variables

Maurycy Chwiłka, Jan Karbowski∗ Department of Mathematics, Informatics, and
Mechanics; Institute of Applied Mathematics and Mechanics, University of
Warsaw, ul. Banacha 2, 02-097 Warsaw, Poland

###### Abstract

Stochastic correlated observables with lognormal distribution are ubiquitous
in nature, and hence they deserve an exact information-theoretic
characterization. We derive a general analytical formula for mutual
information between vectors of lognormally distributed random variables, and
provide lower and upper bounds on its value. That formula and its bounds
involve determinants and traces of high dimensional covariance matrices of
these variables. Exact explicit forms of mutual information are calculated for
some special cases and types of correlations. As an example, we provide an
analytic formula for mutual information between neurons, relevant for neural
networks in the brain.

Keywords

mutual information, lognormal distribution, entropy, tridiagonal and block
matrices,
neuronal information.

##  I Introduction

Lognormal distribution is arguably the most common probability distribution
describing naturally occurring phenomena in nature, from earthquakes, to stock
market fluctuations, to brain activity [1, 2, 3, 4]. Some other examples are
included in Refs. [5, 6, 7]. Lognormal distribution has skewed shape and is
characterized by a long tail [1]. This means that the likelihood of huge size
stochastic events with this distribution is small, but still significantly
higher than for those described by short-tailed distributions, e.g., by
Gaussian density.

Two or more stochastic variables can be correlated, and one of the most useful
tools describing such correlation is mutual information (MI). MI is an
information-theoretic concept [8, 9] that measures average number of bits we
gain about behavior of one variable by observing variability of another,
correlated variable. Obviously, MI has a wide applicability in many different
areas of science, from information science [9, 8] and rapidly developing
machine and deep learning [10, 11], through different branches of physics (as
diverse as material science, stochastic thermodynamics, and cosmology) [13,
14, 15, 12], to molecular biology [16, 17] and neuroscience [18]. Despite this
huge popularity in using MI, it is important to stress that its exact
analytical form for nontrivial continuous stochastic variables is known only
for Gaussian distribution, e.g., [17, 18, 19]. To our knowledge, no analytical
formula exists for other continuous distributions. Finding MI in those other
cases generally requires heavy numerical approaches that are time consuming,
and which have their internal problems (e.g., sampling of continuous variables
into discrete bins) that can introduce errors [19]. Thus clearly, there is a
need for an exact analytical expressions for MI for other distributions,
different from short-tailed Gaussian.

In this paper, we derive an exact formula for mutual information between
vectors of random variables with lognormal distributions. We use that generic
formula to obtain exact explicit expressions in some special cases of
topological relationships between random variables, which can appear in many
applications. We also apply this formalism to derive MI for information
transfer between neurons in the neural networks of cerebral cortex.

##  II Preliminaries

###  II.1  Lognormal distribution

Let ${\bf z}=(z_{1},z_{2},\dots,z_{N})$ be a $N$ dimensional Gaussian random
variable with mean vector ${\bf\mu}=(\mu_{1},...,\mu_{N})$ and covariance
matrix $\Sigma$, which is positive definite (symmetric with real and positive
eigenvalues). We define a new multidimensional random variable ${\bf
x}=(x_{1},x_{2},\dots,x_{N})$, such that $x_{i}=e^{z_{i}}$ for every
$i=1,...,N$. Then ${\bf x}$ is lognormally distributed and has the following
probability density $\rho({\bf x})$

$\displaystyle\rho(\bm{x})=\frac{1}{\sqrt{(2\pi)^{N}\det{(\Sigma)}}\prod_{i=1}^{N}x_{i}}\exp{(-\frac{1}{2}(\ln{(\bm{x})}-\bm{\mu})^{T}\Sigma^{-1}(\ln{(\bm{x})}-\bm{\mu}))},$
(1)

with $\langle\ln(x_{i})\rangle=\mu_{i}$, and the covariance matrix
$\Sigma_{ij}=\langle\ln(x_{i})\ln(x_{j})\rangle-\langle\ln(x_{i})\rangle\langle\ln(x_{j})\rangle$,
where $\langle...\rangle$ denotes averaging with respect to the density in Eq.
(1). We denote the variance of $\ln x_{i}$ as
$\Sigma_{ii}=\langle\ln^{2}(x_{i})\rangle-\langle\ln(x_{i})\rangle^{2}\equiv\sigma_{i}^{2}$.
For $N=1$, the density in Eq. (1) reduces to
$\rho(x)=e^{-(\ln{(x)}-\mu)^{2}/2\sigma^{2}}/(\sqrt{2\pi\sigma^{2}}x)$.

The major moments of the lognormal distribution, including variance
$Var(x_{i})=\langle x_{i}^{2}\rangle-\langle x_{i}\rangle^{2}$, and covariance
$Cov(x_{i},x_{j})=\langle x_{i}x_{j}\rangle-\langle x_{i}\rangle\langle
x_{j}\rangle$, are related to the parameters $\mu_{i}$ and $\Sigma_{ij}$ as
[1]:

$\displaystyle\langle x_{i}\rangle=e^{\mu_{i}+\frac{1}{2}\sigma_{i}^{2}},$
$\displaystyle Var(x_{i})=(e^{\sigma_{i}^{2}}-1)\langle x_{i}\rangle^{2},$
$\displaystyle Cov(x_{i},x_{j})=(e^{\Sigma_{ij}}-1)\langle x_{i}\rangle\langle
x_{j}\rangle.$ (2)

###  II.2  Definition of mutual information

Mutual information MI between two lognormal random variables X and Y is
defined as [8]

$\mbox{MI}(X,Y)=h(X)+h(Y)-h(X,Y),$ (3)

where $h(X)$, and $h(Y)$ are differential entropies for X and Y variables,
i.e.

$h(X)=-\frac{1}{\ln{2}}\int\rho(\bm{x})\ln{(\rho(\bm{x}))}\,d\bm{x},$

and $h(X,Y)$ is the joint differential entropy given by

$h(X,Y)=-\frac{1}{\ln{2}}\int\rho(\bm{x},\bm{y})\ln{(\rho(\bm{x},\bm{y}))}\,d\bm{x}\,d\bm{y},$

where $\rho({\bf x})$ is as in Eq. (1) and $\rho({\bf x,y})$ is the joint
probability density of ${\bf x}$ and ${\bf y}$ (see below). MI(X,Y) quantifies
the amount of information we gain about X by observing Y, or vice versa.

##  III Results

###  III.1  Mutual information for lognormal stochastic vectors

First we determine entropy of $N$ dimensional variable ${\bf x}$. To achieve
this, we need the average of the argument of $\exp$ in Eq. (1). We have
$\langle[\ln(\bm{x})-\mu]^{T}\Sigma^{-1}[\ln(\bm{x})-\mu)]\rangle=\sum_{ij}(\Sigma^{-1})_{ij}\langle[\ln(x_{i})-\mu_{i}][\ln(x_{j})-\mu_{j}]\rangle=\sum_{ij}(\Sigma^{-1})_{ij}\Sigma_{ji}=\sum_{ij}(\Sigma^{-1}\Sigma)_{ii}=N$,
where we used the definition of covariance matrix appearing after Eq. (1).
After some additional straightforward steps we obtain the formula for entropy
of $N$ dimensional lognormal random variable ${\bf x}$:

$\displaystyle h({\bf x})$
$\displaystyle=-\frac{1}{\ln{2}}\langle\ln{(\rho(\bm{x}))}\rangle$
$\displaystyle=\frac{1}{\ln{2}}\Big{[}\frac{1}{2}\ln{((2\pi
e)^{N}\det{(\Sigma)})}+\sum_{i=1}^{N}\langle\ln{(x_{i})}\rangle\Big{]}.$ (4)

Now, we partition our dimensionality $N$ into two parts, $N=n+k$, and define
two new multidimensional lognormal random variables $X$ and $Y$ such that
${\bf x}=(X,Y)$, where $X=(x_{1},x_{2},\dots,x_{n})$ and
$Y=(y_{1},y_{2},\dots,y_{k})$, with $y_{i}\equiv x_{n+i}$. The lognormal
distribution of $X$ has the parameters ${\bf\mu_{X}}$ and $\Sigma_{X}$,
defined as ${\bf\mu_{X}}=(\langle\ln x_{1}\rangle,...,\langle\ln
x_{n}\rangle)$ and $n\times n$ matrix
$(\Sigma_{X})_{ij}=\langle\ln(x_{i})\ln(x_{j})\rangle-\langle\ln(x_{i})\rangle\langle\ln(x_{j})\rangle$.
The lognormal distribution of $Y$ has the parameters ${\bf\mu_{Y}}$ and
$\Sigma_{Y}$, defined as ${\bf\mu_{Y}}=(\langle\ln y_{1}\rangle,...,\langle\ln
y_{k}\rangle)$ and $k\times k$ matrix
$(\Sigma_{Y})_{ij}=\langle\ln(y_{i})\ln(y_{j})\rangle-\langle\ln(y_{i})\rangle\langle\ln(y_{j})\rangle$.
$N$ dimensional variable ${\bf x}$ is lognormally distributed with vector of
means $({\bf\mu_{X},\mu_{Y}})$ and $N\times N$ global covariance matrix
$\Sigma_{XY}$ dependent on $\Sigma_{X}$ and $\Sigma_{Y}$ matrices through the
block matrix

$\displaystyle\Sigma_{XY}=\begin{bmatrix}\Sigma_{X}&Cov_{XY}\\\
Cov_{XY}^{T}&\Sigma_{Y}\end{bmatrix},$ (5)

where $Cov_{XY}$ is $n\times k$ covariance matrix between variables $\ln X$
and $\ln Y$, i.e., $Cov_{XY}\equiv Cov(\ln X,\ln Y)$, and
$(Cov_{XY})_{ij}=\langle\ln(x_{i})\ln(y_{j})\rangle-\langle\ln(x_{i})\rangle\langle\ln(y_{j})\rangle$.

Consequently, from Eq. (4) it follows that we can write entropies for each of
the three lognormal variables $X$, $Y$, and ${\bf x}$ as

$\displaystyle h(X)=\frac{1}{\ln{2}}\Big{[}\frac{1}{2}\ln{((2\pi
e)^{n}\det{(\Sigma_{X})})}+\sum_{i=1}^{n}\langle\ln{(x_{i})}\rangle\Big{]}$
$\displaystyle h(Y)=\frac{1}{\ln{2}}\Big{[}\frac{1}{2}\ln{((2\pi
e)^{k}\det{(\Sigma_{Y})})}+\sum_{i=n+1}^{N}\langle\ln{(x_{i})}\rangle\Big{]}$
$\displaystyle h(X,Y)=\frac{1}{\ln{2}}\Big{[}\frac{1}{2}\ln{((2\pi
e)^{N}\det{(\Sigma_{XY})})}+\sum_{i=1}^{N}\langle\ln{(x_{i})}\rangle\Big{]}.$

Now, using Eq. 3, we can observe that

$\displaystyle\mbox{MI}(X,Y)=\frac{1}{2\ln
2}\ln{\Big{(}\frac{\det{(\Sigma_{X})}\det{(\Sigma_{Y})}}{\det{(\Sigma_{XY})}}\Big{)}}.$
(6)

This is the general formula for mutual information of two random
multidimensional lognormal variables, and it depends only on the matrices of
covariance of corresponding Gaussian random vectors with appropriate
parameters - both joint and marginal distributions. The formula (6) can be
rewritten in an equivalent form if we use a Schur’s decomposition for the
determinant of the block matrix $\Sigma_{XY}$, namely [20, 21]

$\displaystyle\det\Sigma_{XY}=\det(\Sigma_{X})\det(\Sigma_{Y}-Cov_{XY}^{T}\Sigma_{X}^{-1}Cov_{XY})$
$\displaystyle=\det(\Sigma_{Y})\det(\Sigma_{X}-Cov_{XY}\Sigma_{Y}^{-1}Cov_{XY}^{T}).$
(7)

Then the mutual information is

$\displaystyle\mbox{MI}(X,Y)=\frac{1}{\ln
4}\ln{\Big{(}\frac{\det{(\Sigma_{Y})}}{\det(\Sigma_{Y}-Cov_{XY}^{T}\Sigma_{X}^{-1}Cov_{XY})}\Big{)}}$
$\displaystyle=\frac{1}{\ln
4}\ln{\Big{(}\frac{\det{(\Sigma_{X})}}{\det(\Sigma_{X}-Cov_{XY}\Sigma_{Y}^{-1}Cov_{XY}^{T})}\Big{)}}$
$\displaystyle=\frac{1}{\ln
4}\ln{\Big{(}\frac{1}{\det(I-\Sigma_{X}^{-1}Cov_{XY}\Sigma_{Y}^{-1}Cov_{XY}^{T})}\Big{)}},$
(8)

where $I$ is the identity matrix, and the last equality follows from a known
identity, $\det(AB)=\det(A)\det(B)$, valid for two arbitrary square matrices
$A$ and $B$. Eqs. (6) and (8) are the basic exact results of this paper, which
allow us to obtain analytical expressions for mutual information in some
specific cases. Note that MI(X,Y) in Eq. (8) is nonzero only if there are
correlations between X and Y, i.e., when the covariance matrix $Cov_{XY}$ is
nonzero. Depending on the structure of matrices $\Sigma_{X}$, $\Sigma_{Y}$,
and $Cov_{XY}$, one can use either of the three expressions in Eq. (8),
whichever is easier, for finding MI(X,Y).

###  III.2  Lower and upper bounds for MI, and approximated formula

In this section we find the lower and upper bounds for mutual information. For
this we use the following inequalities linking determinant and trace of any
positive definite square matrix A (see the Appendix A for the proof):

$\displaystyle\mathrm{Tr}(I-A^{-1})\leq\ln(\det(A))\leq\mathrm{Tr}(A-I)$
$\displaystyle\mathrm{Tr}(I-A)\leq\ln\big{(}\frac{1}{\det(A)}\big{)}\leq\mathrm{Tr}(A^{-1}-I),$
(9)

where the last inequality can be obtained from the first by the substitution
$A\mapsto A^{-1}$. From this, we immediately obtain that

$\displaystyle\frac{1}{\ln
4}\mathrm{Tr}\Big{(}\Sigma_{X}^{-1}Cov_{XY}\Sigma_{Y}^{-1}Cov_{XY}^{T}\Big{)}\leq\mbox{MI}(X,Y)\leq\frac{1}{\ln
4}\mathrm{Tr}\Big{(}\big{[}I-\Sigma_{X}^{-1}Cov_{XY}\Sigma_{Y}^{-1}Cov_{XY}^{T}\big{]}^{-1}-I\Big{)}.$
(10)

Thus, MI is bounded by the relative magnitude of the covariance matrix
$Cov_{XY}$. In many instances the correlations between $\ln(X)$ and $\ln(Y)$
are weak, and hence that matrix is small (its entries are small).
Consequently, by writing $Cov_{XY}=\epsilon\overline{Cov}_{XY}$, with
$\epsilon\mapsto 0$, we can make use of an approximation [20],
$\det(I+\epsilon A)\approx 1+\epsilon\mathrm{Tr}(A)+O(\epsilon^{2})$, in the
last line of Eq. (8) for MI. This produces, after expanding the logarithm, the
approximate formula for MI as

$\displaystyle\mbox{MI}(X,Y)=\frac{\epsilon^{2}}{\ln
4}\mathrm{Tr}\Big{(}\Sigma_{X}^{-1}\overline{Cov}_{XY}\Sigma_{Y}^{-1}\overline{Cov}_{XY}^{T}\Big{)}$
$\displaystyle+O\Big{(}\epsilon^{4}\Big{)}.$ (11)

It is easy to note that this approximation corresponds to the lower bound on
MI in Eq. (10).

###  III.3  Exact analytical formulas for some cases and special types of
matrices

In this section we consider several specific cases, each illustrated by graphs
with nodes and arrows. An arrow between the nodes indicates a correlation
among particular random variables.

####  III.3.1 Case $n=k=1$.

In this case, $X=x_{1}$ and $Y=y_{1}=x_{2}$, and the global covariance matrix
appearing in Eq. (5) is $2\times 2$, and has the following form

$\displaystyle\Sigma_{XY}=\begin{bmatrix}\sigma_{X}^{2}&Cov_{XY}\\\
Cov_{XY}&\sigma_{Y}^{2}\end{bmatrix},$ (12)

where $\sigma_{X}^{2}=Var(\ln
X)=\langle\ln^{2}(X)\rangle-\langle\ln(X)\rangle^{2}$, and
$\sigma_{Y}^{2}=Var(\ln Y)=\langle\ln^{2}(Y)\rangle-\langle\ln(Y)\rangle^{2}$.
As a result

$\displaystyle\mbox{MI}(X,Y)$
$\displaystyle=\frac{1}{2\ln{2}}\ln{\Big{(}\frac{\sigma_{X}^{2}\sigma_{Y}^{2}}{\sigma_{X}^{2}\sigma_{Y}^{2}-Cov_{XY}^{2}}\Big{)}}$
$\displaystyle=\frac{1}{2\ln{2}}\ln{\Big{(}\frac{1}{1-c^{2}}\Big{)}},$ (13)

where $c$ denotes correlation coefficient between variables $\ln(X)$ and
$\ln(Y)$, defined as $c=Cov_{XY}/\sigma_{X}\sigma_{Y}$. The coefficient $c$
can be expressed by covariance of $X$ and $Y$, $Cov(X,Y)$ and their variances
$Var(X),Var(Y)$ as

$\displaystyle c=\frac{\ln[1+\frac{Cov(X,Y)}{\langle X\rangle\langle
Y\rangle}]}{\sqrt{\ln[1+\frac{Var(X)}{\langle
X\rangle^{2}}]\ln[1+\frac{Var(Y)}{\langle Y\rangle^{2}}]}},$ (14)

which follows from Eq. (2). Because of the Cauchy-Schwartz inequality for
covariance and variance, we have $Cov(X,Y)\leq\sqrt{Var(X)Var(Y)}$, and this
implies that $c$ is bounded from above by $c_{0}$, which is

$\displaystyle c_{0}=\frac{\ln[1+\frac{\sqrt{Var(X)Var(Y)}}{\langle
X\rangle\langle Y\rangle}]}{\sqrt{\ln[1+\frac{Var(X)}{\langle
X\rangle^{2}}]\ln[1+\frac{Var(Y)}{\langle Y\rangle^{2}}]}}.$ (15)

Consequently, we have an upper bound on MI(X,Y) in this case as

$\displaystyle\mbox{MI}(X,Y)$
$\displaystyle\leq\frac{1}{2\ln{2}}\ln\Big{(}\frac{1}{1-c_{0}^{2}}\Big{)}.$
(16)

This inequality may be useful in cases when we do not know covariances but we
do know variances of original data.

####  III.3.2 Case $n=k=2$.

In this case, $X=(x_{1},x_{2})$ and $Y=(y_{1},y_{2})$, and the covariance
matrix $\Sigma_{XY}$ appearing in Eq. (5) is $4\times 4$. We choose the
components of $\Sigma_{XY}$ with some degree of symmetry:
$(\Sigma_{X})_{ij}=\sigma_{x}^{2}\delta_{ij}+\gamma_{x}(1-\delta_{ij})$,
$(\Sigma_{Y})_{ij}=\sigma_{y}^{2}\delta_{ij}+\gamma_{y}(1-\delta_{ij})$, and
$(Cov_{XY})_{ij}=a\delta_{ij}+b(1-\delta_{ij})$, where $i,j=1,2$. Here, all
four variables $x_{1},x_{2},y_{1},y_{2}$ are mutually correlated (see the
graph below).

$X_{1}$$X_{2}$$Y_{1}$$Y_{2}$

Thus we have $\det(\Sigma_{Y})=\sigma_{y}^{4}-\gamma_{y}^{2}$ and
$\det(\Sigma_{X})=\sigma_{x}^{4}-\gamma_{x}^{2}$. Additionally, we obtain

$\displaystyle\det[\Sigma_{Y}-Cov_{XY}^{T}\Sigma_{X}^{-1}Cov_{XY}]$
$\displaystyle=\frac{[(\sigma_{x}^{2}-\gamma_{x})(\sigma_{y}^{2}-\gamma_{y})-(a-b)^{2}]}{\det\Sigma_{X}}$
$\displaystyle\times[(\sigma_{x}^{2}+\gamma_{x})(\sigma_{y}^{2}+\gamma_{y})-(a+b)^{2}]$

Using Eq. (8), this leads to mutual information as

$\displaystyle\mbox{MI}(X,Y)=\frac{-1}{\ln
4}\ln\Big{(}\Big{[}1-\frac{(a-b)^{2}}{(\sigma_{x}^{2}+\gamma_{x})(\sigma_{y}^{2}+\gamma_{y})}\Big{]}$
$\displaystyle\times\Big{[}1-\frac{(a+b)^{2}}{(\sigma_{x}^{2}-\gamma_{x})(\sigma_{y}^{2}-\gamma_{y})}\Big{]}\Big{)}.$
(17)

Eq. (17) suggests that MI decreases monotonically with increasing variances
$\sigma_{x}^{2}$, $\sigma_{y}^{2}$ of $\ln(X)$ and $\ln(Y)$. Moreover, MI has
a minimum as a function of correlations $\gamma_{x}$ between $x_{1},x_{2}$, if
$2\sigma_{x}^{2}(\sigma_{y}^{2}-\gamma_{y})\geq(a+b)^{2}$; see Fig. 1
(analogous applies to $\gamma_{y}$). MI also diverges for large intra-
correlations $\gamma_{x}$, $\gamma_{x}$, and for sufficiently strong inter-
correlations $a,b$ between $\ln(X)$ and $\ln(Y)$ variables (either positive or
negative; Fig. 1).

Figure 1: MI in Eq. (17) as a function of $\gamma_{x}$ and $a$. (A) solid
line: $\sigma_{x}^{2}=9$, $\sigma_{y}^{2}=9$, $a=2.4$, $b=1$, $\gamma_{y}=1$;
dotted line: $\sigma_{x}^{2}=9$, $\sigma_{y}^{2}=9$, $a=3$, $b=1.5$,
$\gamma_{y}=1$. (B) solid line: $\sigma_{x}^{2}=9$, $\sigma_{y}^{2}=9$,
$\gamma_{x}=1$, $b=0$, $\gamma_{y}=1$; dotted line: $\sigma_{x}^{2}=9$,
$\sigma_{y}^{2}=16$, $\gamma_{x}=1$, $b=8.4$, $\gamma_{y}=1$.

####  III.3.3 Case $n\geq 2$, $k=1$, with tridiagonal matrix for $\Sigma_{X}$.

In this case we take $\Sigma_{X}$ to be tridiagonal matrix, and choose
$(\Sigma_{X})_{ii}=\sigma_{x}^{2}$, $(\Sigma_{X})_{i,i\pm 1}=\gamma$, with all
other elements of $\Sigma_{X}$ to be zero. This means that correlated are only
those $x_{i},x_{j}$ that are nearest neighbors in the “indices space”. Matrix
$\Sigma_{Y}$ is just scalar, equal to $\sigma_{y}^{2}$. Covariance $Cov_{XY}$
is a vector, and we take it $(Cov_{XY})_{i}=a$. This configuration is depicted
by the graph below:

$X_{1}$$X_{2}$$X_{3}$$X_{n-1}$$X_{n}$…$Y$

The matrix $\Sigma_{X}$ is tridiagonal, and determinants of tridiagonal
matrices are known [22, 23]. The determinant of arbitrary tridiagonal $n\times
n$ matrix with all diagonal elements $t$ and all off-diagonal elements $s$ is
given by the function $\theta_{n}(t,s)$, which takes the form [22, 23]

$\displaystyle\theta_{n}(t,s)=\frac{\Big{[}\big{(}t+\sqrt{t^{2}-4s^{2}}\big{)}^{n+1}-\big{(}t-\sqrt{t^{2}-4s^{2}}\big{)}^{n+1}\Big{]}}{2^{n+1}\sqrt{t^{2}-4s^{2}}}.$

Hence, we have $\det(\Sigma_{X})=\theta_{n}(\sigma_{x}^{2},\gamma)$. Mutual
information is determined from the first line in Eq. (8), and is given by

$\displaystyle\mbox{MI}(X,Y)=\frac{1}{\ln
4}\ln\left(\frac{\sigma_{y}^{2}}{\sigma_{y}^{2}-Cov_{XY}^{T}\Sigma_{X}^{-1}Cov_{XY}}\right),$
(19)

where $Cov_{XY}^{T}\Sigma_{X}^{-1}Cov_{XY}$ can be found after tedious
calculations as (see the Appendix B)

$\displaystyle
Cov_{XY}^{T}\Sigma_{X}^{-1}Cov_{XY}=\frac{na^{2}}{(\sigma_{x}^{2}+2\gamma)}+\frac{\gamma
a^{2}}{2^{n}(\sigma_{x}^{2}+2\gamma)(\sigma_{x}^{4}-4\gamma^{2})\det(\Sigma_{X})}\Big{[}(-1)^{n+1}2^{n+1}\gamma^{n}(\sigma_{x}^{2}-2\gamma)$
$\displaystyle+(\sigma_{x}^{2}+\sqrt{\sigma_{x}^{4}-4\gamma^{2}})^{n}(\sigma_{x}^{2}-2\gamma+\sqrt{\sigma_{x}^{4}-4\gamma^{2}})+(\sigma_{x}^{2}-\sqrt{\sigma_{x}^{4}-4\gamma^{2}})^{n}(\sigma_{x}^{2}-2\gamma-\sqrt{\sigma_{x}^{4}-4\gamma^{2}})\Big{]}.$
(20)

The above complicated formula, simplifies in two extreme cases. In the case
when $n=2$, we obtain
$Cov_{XY}^{T}\Sigma_{X}^{-1}Cov_{XY}=2a^{2}/(\sigma_{x}^{2}+\gamma)$, and
hence mutual information takes the simple form $MI(X,Y)=\frac{1}{\ln
4}\ln\Big{(}\frac{\sigma_{y}^{2}}{\sigma_{y}^{2}-2a^{2}/(\sigma_{x}^{2}+\gamma)}\Big{)}$.
In the case of $n\gg 1$, the first term on the right in Eq. (20) dominates,
and we obtain $Cov_{XY}^{T}\Sigma_{X}^{-1}Cov_{XY}\approx
na^{2}/(\sigma_{x}^{2}+\gamma)+O(1)$, and MI is given by a similar expression
$MI(X,Y)\approx\frac{1}{\ln
4}\ln\Big{(}\frac{\sigma_{y}^{2}}{\sigma_{y}^{2}-na^{2}/(\sigma_{x}^{2}+\gamma)}\Big{)}$.
The latter formula implies that MI diverges for the number of units
$n\approx\sigma_{y}^{2}(\sigma_{x}^{2}+\gamma)/a^{2}$. In both cases, it is
easily seen that MI grows with correlations $a$ between variables $X$ and $Y$
(both positive and negative), but it decreases monotonically with increasing
the correlations $\gamma$ between $x_{i}$ variables. The latter implies that
negative correlations among $x_{i}$ carry more information than their positive
correlations. This is a different situation than in case number 2, where both
strong positive and negative correlations between $x_{i}$ enhance MI. As
expected, MI also decreases with increasing variances $\sigma_{x}^{2}$ and
$\sigma_{y}^{2}$. General dependence of mutual information on $\gamma$,
$\sigma_{x}^{2}$, and $n$, based on the exact Eqs. (19) and (20) is presented
in Fig. 2.

Figure 2: MI in Eq. (19) as a function of $\gamma$, $\sigma_{x}^{2}$, and $n$.
(A) solid line: $\sigma_{x}^{2}=10$, $\sigma_{y}^{2}=1$, $a=1$, $n=10$; dotted
line: $\sigma_{x}^{2}=9$, $\sigma_{y}^{2}=9$, $a=3$, $n=10$; dash-dotted line:
$\sigma_{x}^{2}=10$, $\sigma_{y}^{2}=10$, $a=3$, $n=10$. (B) solid line:
$\sigma_{y}^{2}=1$, $a=1$, $\gamma=\frac{1}{2}$, $n=10$; dotted line:
$\sigma_{y}^{2}=1$, $a=1.3$, $\gamma=\frac{1}{2}$, $n=10$. (C) points:
$\sigma_{x}^{2}=10$, $\sigma_{y}^{2}=1$, $a=1$, $\gamma=1$; x:
$\sigma_{x}^{2}=10$, $\sigma_{y}^{2}=1$, $a=1$, $\gamma=3$.

####  III.3.4 Case $n\geq 2$, $k=1$, with Kac-Murdock-Szegö matrix for
$\Sigma_{X}$.

The only difference with the point 3 above is that now the matrix $\Sigma_{X}$
has all nonzero elements, which are given by
$(\Sigma_{X})_{ij}=\sigma_{x}^{2}\gamma^{|i-j|}$, where $|\gamma|\leq 1$ [24].
This means that all $x_{i}$ are mutually correlated, but distant ones (with
remote indices) are gradually less correlated (see below).

$X_{1}$$X_{2}$$X_{3}$$X_{n-1}$$X_{n}$…$Y$

The inverse of the matrix $\Sigma_{X}$ is [24, 20]

$\displaystyle(\Sigma_{X}^{-1})_{ij}=\frac{\delta_{ij}[1+\gamma^{2}(1-\delta_{i1}-\delta_{in})]-\gamma(\delta_{i,j-1}+\delta_{i,j+1})}{\sigma_{x}^{2}(1-\gamma^{2})},$

which allows us to find

$\displaystyle
Cov_{XY}^{T}\Sigma_{X}^{-1}Cov_{XY}=\frac{a^{2}[n(1-\gamma)+2\gamma]}{\sigma_{x}^{2}(1+\gamma)}.$
(22)

Consequently, mutual information in this case is given by a simple explicit
formula:

$\displaystyle\mbox{MI}(X,Y)=\frac{1}{\ln
4}\ln\left(\frac{\sigma_{y}^{2}}{\sigma_{y}^{2}-\frac{a^{2}[n(1-\gamma)+2\gamma]}{\sigma_{x}^{2}(1+\gamma)}}\right).$
(23)

It is easily seen that MI grows monotonically with the number of units $n$ and
diverges for $n=\frac{[\sigma_{x}^{2}\sigma_{y}^{2}(1+\gamma)-2\gamma
a^{2}]}{(1-\gamma)a^{2}}$. The dependence of MI on other parameters is similar
to the case number 3 above. In particular, MI decreases monotonically with
increasing correlations $\gamma$ among $x_{i}$ variables from negative
($\gamma<0$) to positive ($\gamma>0$).

####  III.3.5 Case $n\geq 2$, $k\geq 2$, with non-symmetric and nondiagonal
$Cov_{XY}$.

Let us consider $n\times n$ matrix $\Sigma_{X}$ and $k\times k$ matrix
$\Sigma_{Y}$ to be both tridiagonal with elements
$(\Sigma_{X})_{ij}=\sigma_{x}^{2}\delta_{ij}+\gamma_{x}(\delta_{i,j-1}+\delta_{i,j+1})$,
and
$\Sigma_{Y}=\sigma_{y}^{2}\delta_{ij}+\gamma_{y}(\delta_{i,j-1}+\delta_{i,j+1})$.
The covariance $Cov_{XY}$ is $n\times k$ sparse matrix taken with one nonzero
element $(Cov_{XY})_{n1}=a$ and the rest elements are 0 (see the graph below).

$X_{1}$$X_{2}$$X_{3}$$X_{n-1}$$X_{n}$…$Y_{1}$$Y_{2}$$Y_{3}$$Y_{k-1}$$Y_{k}$…

This case is relevant for two groups of cascade networks of interacting
elements.

We use the last line in Eq. (8) for MI. In this configuration, it is easy to
show that
$(\Sigma_{X}^{-1}Cov_{XY}\Sigma_{Y}^{-1}Cov_{XY}^{T})_{ij}=a^{2}(\Sigma_{X}^{-1})_{in}(\Sigma_{Y}^{-1})_{11}$
for $j=n$ and 0 for $1\leq j\leq n-1$, for every $i$. This leads to the simple
form for the determinant
$\det(I-\Sigma_{X}^{-1}Cov_{XY}\Sigma_{Y}^{-1}Cov_{XY}^{T})=1-a^{2}(\Sigma_{X}^{-1})_{nn}(\Sigma_{Y}^{-1})_{11}$,
and consequently for MI

$\displaystyle\mbox{MI}(X,Y)=\frac{1}{\ln
4}\ln\Big{(}\frac{1}{1-a^{2}(\Sigma_{X}^{-1})_{nn}(\Sigma_{Y}^{-1})_{11}}\Big{)}.$
(24)

The matrix elements $(\Sigma_{X}^{-1})_{nn}$ and $(\Sigma_{Y}^{-1})_{11}$ are
given by the ratios of the function $\theta_{n}$ defined in Eq. (18), i.e.,
$(\Sigma_{X}^{-1})_{nn}=\theta_{n-1}(\sigma^{2}_{x},\gamma_{x})/\theta_{n}(\sigma^{2}_{x},\gamma_{x})$,
and
$(\Sigma_{Y}^{-1})_{11}=\theta_{k-1}(\sigma^{2}_{y},\gamma_{y})/\theta_{k}(\sigma^{2}_{y},\gamma_{y})$;
see Appendix B.

In the limits $n\gg 1$, $k\gg 1$, and for $\sigma_{x}^{2}>2\gamma_{x}$ and
$\sigma_{y}^{2}>2\gamma_{y}$, we obtain an approximated MI as

$\displaystyle\mbox{MI}(X,Y)_{n\gg 1}\approx\frac{-1}{\ln
4}\ln\Big{(}1-\frac{4a^{2}}{\big{(}\sigma_{x}^{2}+\sqrt{\sigma_{x}^{4}-4\gamma_{x}^{2}}\big{)}}$
$\displaystyle\times\big{(}\sigma_{y}^{2}+\sqrt{\sigma_{y}^{4}-4\gamma_{y}^{2}}\big{)}^{-1}\Big{)}.$

This equation shows that large variances of both $\sigma_{x}^{2}$ and
$\sigma_{y}^{2}$ have detrimental effect on MI. On the other hand, increasing
covariances among $\ln(X)$ ($|\gamma_{x}|$), and among $\ln(Y)$
($|\gamma_{y}|$) causes increase in MI.

####  III.3.6 Case $n=k\geq 2$ with symmetric and diagonal $Cov_{XY}$.

In this case, we take $\Sigma_{X}$ to be tridiagonal with elements
$(\Sigma_{X})_{ij}=\sigma_{x}^{2}\delta_{ij}+\gamma_{x}(\delta_{i,j-1}+\delta_{i,j+1})$,
and $\Sigma_{Y}$ to be diagonal with elements
$(\Sigma_{Y})_{ij}=\sigma_{y}^{2}\delta_{ij}$. Additionally, we choose the
covariance matrix $Cov_{XY}$ to be diagonal, $(Cov_{XY})_{ij}=a\delta_{ij}$.
This situation corresponds to the system depicted below

$X_{1}$$X_{2}$$X_{3}$$X_{n-1}$$X_{n}$…$Y_{1}$$Y_{2}$$Y_{3}$$Y_{n-1}$$Y_{n}$…

Here, there are correlations between nearest neighbors of variables $x_{i}$,
but there are no correlations between variables $y_{i}$. Additionally, $x_{i}$
correlates directly only with $y_{i}$.

Elements of the matrix of interest
$\Sigma_{X}-Cov_{XY}\Sigma_{Y}^{-1}Cov_{XY}^{T}$ appearing in Eq. (8) for
mutual information are
$(\Sigma_{X}-Cov_{XY}\Sigma_{Y}^{-1}Cov_{XY}^{T})_{ij}=(\sigma_{x}^{2}-\frac{a^{2}}{\sigma_{y}^{2}})\delta_{ij}+\gamma_{x}(\delta_{i,j-1}+\delta_{i,j+1})$.
Thus it is also the tridiagonal matrix, similar as $\Sigma_{X}$. Therefore,
the determinants of both $\Sigma_{X}$ and
$\Sigma_{X}-Cov_{XY}\Sigma_{Y}^{-1}Cov_{XY}^{T}$ are given by Eq. (18), i.e.,
$\det(\Sigma_{X})=\theta_{n}(\sigma^{2}_{x},\gamma_{x})$ and
$\det{(\Sigma_{X}-Cov_{XY}\Sigma_{Y}^{-1}Cov_{XY}^{T})}=\theta_{n}(\sigma_{x}^{2}-\frac{a^{2}}{\sigma_{y}^{2}},\gamma_{x})$.

Figure 3: MI in Eq. (24) as a function of $\gamma_{x}$, $a$, and $n$. (A)
solid line: $\sigma_{x}^{2}=10$, $\sigma_{y}^{2}=1$, $a=1$, $n=10$; dotted
line: $\sigma_{x}^{2}=9$, $\sigma_{y}^{2}=9$, $a=3$, $n=10$; dash-dotted line:
$\sigma_{x}^{2}=10$, $\sigma_{y}^{2}=10$, $a=3$, $n=10$. (B) solid line:
$\sigma_{x}^{2}=9$, $\sigma_{y}^{2}=9$, $\gamma_{x}=1$, $n=10$; dotted line:
$\sigma_{x}^{2}=9$, $\sigma_{y}^{2}=4$, $\gamma_{x}=1$, $n=10$. (C) points:
$\sigma_{x}^{2}=25$, $\sigma_{y}^{2}=1$, $a=1$, $\gamma_{x}=1$; x:
$\sigma_{x}^{2}=50$, $\sigma_{y}^{2}=1$, $a=2$, $\gamma_{x}=2$.

The corresponding mutual information is given by the first line in Eq. (8) as

$\displaystyle\mbox{MI}(X,Y)=\frac{1}{\ln
4}\ln\left(\frac{\theta_{n}(\sigma^{2}_{x},\gamma_{x})}{\theta_{n}(\sigma_{x}^{2}-\frac{a^{2}}{\sigma_{y}^{2}},\gamma_{x})}\right).$
(25)

The dependence of MI on $\gamma_{x}$, $a$, and $n$ is shown in Fig. 3. In a
special case $n=k=2$, we recover MI in the case number 2 (see Eq. (17)), but
with $b=\gamma_{y}=0$.

###  III.4  MI for neural activities as lognormal variables.

In this section we apply the exact formula for MI in Eq. (8) to neuroscience,
i.e., we derive mutual information relevant for information transfer in neural
networks of a mammalian brain. Specifically, we find MI between activities of
$n$ presynaptic neurons with firing rates $\vec{f}=(f_{1},f_{2},...,f_{n})$
and activity of a postsynaptic neuron with the firing rate $Y$. This situation
can be depicted by the analogical graph as in the case no. 4 considered above.
The simplest model relating $Y$ and $\vec{f}$ is [25, 26]

$\displaystyle Y=\sum_{i=1}^{n}w_{i}f_{i},$ (26)

where $\vec{w}=(w_{1},w_{2},...,w_{n})$ are synaptic weights, and we assume
that all of them are positive (all excitatory presynaptic neurons). We note
that in general the dependence of $Y$ on $\vec{f}$ could in principle be
nonlinear. We assume linearity, since it was shown that for a biophysically
motivated nonlinear class of neurons (so-called class 2), the dependence of
$Y$ on $\vec{f}$ can become linear by incorporating adaptation in neural
firing rates, which is often observed in brain networks [27, 28, 29].

Our major assumption is that both firing rates $\vec{f}$, $Y$, and synaptic
weights $\vec{w}$ have lognormal distributions, which is compatible with
experimental data for neurons and synapses in the mammalian cerebral cortex
[4]. It should be clearly said that postsynaptic activity $Y$, as it stands in
Eq. (26), has an unknown exact form of probability density. However, it has
been numerically verified by others [30, 31] that $Y$ is approximately
lognormally distributed, given lognormal distributions of $\vec{f}$ and
$\vec{w}$ (note that each product $w_{i}f_{i}$ is lognormally distributed).

The goal is to find mutual information $MI(\vec{f},Y)$ between vectors
$\vec{f}$ and $Y$. We note that although the mutual information between pre-
and post-synaptic neural activities has been found many times in the past [18,
19, 32], none of those approaches used lognormally distributed variables
(mostly Gaussian distribution was used, and that distribution can yield
negative of both $f_{i}$ and $w_{i}$, which is unphysical). We assume that
activities of presynaptic neurons (i.e., $\ln(f_{i})$) are weakly correlated,
and take for their covariance matrix $\Sigma_{f}$ the form given by Kac-
Murdock-Szegö matrix, i.e., $(\Sigma_{f})_{ij}=\sigma_{f}^{2}\gamma^{|i-j|}$,
with $|\gamma|\ll 1$, where $\sigma_{f}^{2}$ is the variance for all
$\ln(f_{i})$. The corresponding variance for all $\ln(w_{i})$ is denoted as
$\sigma_{w}^{2}$, and the variance for postsynaptic activity $\ln(Y)$ as
$\sigma_{Y}^{2}$. The latter depends on the parameters characterizing
distributions of $\vec{f}$ and $\vec{w}$ (see below). Additionally, we assume
that synaptic weights $w_{i}$ are not correlated between themselves, and that
vectors $\vec{f}$ and $\vec{w}$ are not correlated either. These two
assumptions are consistent with empirical observations, but are not essential
for the derivation (in principle they could be included). They make the final
formula look simpler. Using Eq. (2), we can write expressions for various
moments of $f_{i}$ and $w_{i}$, which are used later:

$\displaystyle\langle f_{i}\rangle=e^{\mu_{f}+\frac{1}{2}\sigma_{f}^{2}},$
$\displaystyle\langle w_{i}\rangle=e^{\mu_{w}+\frac{1}{2}\sigma_{w}^{2}},$
$\displaystyle\langle f_{i}^{2}\rangle=e^{2\mu_{f}+2\sigma_{f}^{2}},$
$\displaystyle\langle w_{i}^{2}\rangle=e^{2\mu_{w}+2\sigma_{w}^{2}},$
$\displaystyle\langle
f_{i}f_{j}\rangle=e^{2\mu_{f}+\sigma_{f}^{2}(1+c_{ij})},$ (27)

where $\mu_{f}=\langle\ln(f_{i})\rangle$, $\mu_{w}=\langle\ln(w_{i})\rangle$,
and $c_{ij}$ is the correlation coefficient between $\ln(f_{i})$ and
$\ln(f_{j})$, i.e.,
$c_{ij}\sigma_{f}^{2}=\langle(\ln(f_{i})-\mu_{f})(\ln(f_{j})-\mu_{f})\rangle=(\Sigma_{f})_{ij}$.
The latter means that $c_{ij}=\gamma^{|i-j|}$.

To determine MI we need the variance of $\ln(Y)$ and covariance matrix between
$\ln(Y)$ and $\ln(f_{i})$. This is accomplished by finding the first two
moments of $Y$ in terms of the parameters characterizing $\vec{f}$ and
$\vec{w}$, and match them to the first two moments of assumed lognormal form
of $Y$ (so-called Wilkinson method; [30, 31]). We have

$\displaystyle\langle Y\rangle=\sum_{i=1}^{n}\langle w_{i}\rangle\langle
f_{i}\rangle,$

and

$\displaystyle\langle Y^{2}\rangle=\sum_{i=1}^{n}\langle
w_{i}^{2}\rangle\langle
f_{i}^{2}\rangle+2\sum_{i=1}^{n-1}\sum_{j=i+1}^{n}\langle w_{i}\rangle\langle
w_{j}\rangle\langle f_{i}f_{j}\rangle,$

which after using Eqs. (27) yields

$\displaystyle\langle
Y\rangle=ne^{\mu_{f}+\mu_{w}}e^{(\sigma_{f}^{2}+\sigma_{w}^{2})/2},$
$\displaystyle\langle
Y^{2}\rangle=e^{2(\mu_{f}+\mu_{w})}e^{\sigma_{f}^{2}+\sigma_{w}^{2}}$
$\displaystyle\times\Big{(}ne^{\sigma_{f}^{2}+\sigma_{w}^{2}}+2\sum_{i=1}^{n-1}\sum_{j=i+1}^{n}e^{\sigma_{f}^{2}c_{ij}}\Big{)}.$
(28)

On the other hand, for lognormally distributed $Y$, we have $\langle
Y\rangle=e^{\mu_{Y}+\frac{1}{2}\sigma_{Y}^{2}}$, and $\langle
Y^{2}\rangle=e^{2\mu_{Y}+2\sigma_{Y}^{2}}$, where $\mu_{Y}$ and
$\sigma_{Y}^{2}$ are the mean and the variance of $\ln(Y)$. This allows us to
find $\sigma_{Y}^{2}$ in terms of the parameters for $\vec{f}$ and $\vec{w}$
as

$\displaystyle\sigma_{Y}^{2}=\ln\Big{(}\frac{1}{n}e^{(\sigma_{f}^{2}+\sigma_{w}^{2})}+\frac{2}{n^{2}}\sum_{i=1}^{n-1}\sum_{j=i+1}^{n}e^{\sigma_{f}^{2}c_{ij}}\Big{)}.$
(29)

To find the covariance $Cov_{Y\vec{f}}\equiv Cov(\ln(Y),\ln(\vec{f}))$ between
$\ln(Y)$ and $\ln(f_{i})$, we first determine the covariance $Cov(Y,f_{i})$
between $Y$ and $f_{i}$. We have $Cov(Y,f_{i})=\langle Yf_{i}\rangle-\langle
Y\rangle\langle f_{i}\rangle=\langle w_{i}\rangle\langle
f_{i}^{2}\rangle-\langle Y\rangle\langle f_{i}\rangle+\sum_{j\neq
i}^{n}\langle w_{j}\rangle\langle f_{i}f_{j}\rangle$. All these moments are
given above, and we obtain

$\displaystyle
Cov(Y,f_{i})=e^{m_{w}+\sigma_{w}^{2}/2}e^{2(m_{f}+\sigma_{f}^{2})}$
$\displaystyle\times\Big{(}1-e^{-\sigma_{f}^{2}}\big{[}n-\sum_{j\neq
i}^{n}e^{\sigma_{f}^{2}c_{ij}}\big{]}\Big{)},$ (30)

which after using Eq. (2) we obtain the covariance vector of interest

$\displaystyle(Cov_{Y\vec{f}})_{i}=\ln\Big{(}\frac{1}{n}\big{[}e^{\sigma_{f}^{2}}+\sum_{j\neq
i}^{n}e^{\sigma_{f}^{2}c_{ij}}\big{]}\Big{)}.$ (31)

The general formulas in Eqs. (29) and (31) can be expanded for weak
presynaptic neural correlations, i.e., $e^{\sigma_{f}^{2}c_{ij}}\approx
1+\sigma_{f}^{2}c_{ij}$ for $|\gamma|\ll 1$, with the help of $\sum_{j\neq
i}c_{ij}=\gamma\big{[}\delta_{i1}+\delta_{in}+2(1-\delta_{i1})(1-\delta_{in})\big{]}+O(\gamma^{2})$.
This leads to

$\displaystyle\sigma_{Y}^{2}=\ln\Big{(}1+\frac{1}{n}\big{[}e^{(\sigma_{f}^{2}+\sigma_{w}^{2})}-1\big{]}\Big{)}$
$\displaystyle+\frac{2\gamma\sigma_{f}^{2}(1-\frac{1}{n})^{2}}{1+\frac{1}{n}\big{[}e^{(\sigma_{f}^{2}+\sigma_{w}^{2})}-1\big{]}}+O(\gamma^{2}),$
(32)

and

$\displaystyle(Cov_{Y\vec{f}})_{i}=\ln\Big{(}1+\frac{1}{n}\big{[}e^{\sigma_{f}^{2}}-1\big{]}\Big{)}$
$\displaystyle+\frac{\gamma\sigma_{f}^{2}\big{[}\delta_{i1}+\delta_{in}+2(1-\delta_{i1})(1-\delta_{in})\big{]}}{n\big{[}1+\frac{1}{n}(e^{\sigma_{f}^{2}}-1)\big{]}}+O(\gamma^{2}).$
(33)

Note that the covariance vector $Cov_{Y\vec{f}}$ is finite even when there are
no correlations between presynaptic neural activities ($\gamma\mapsto 0$),
however in the limit $n\mapsto\infty$ it vanishes.

The last step is to find the value of
$Cov_{Y\vec{f}}^{T}\Sigma_{f}^{-1}Cov_{Y\vec{f}}$, using approximation to Eq.
(21) for small $\gamma$, i.e.,
$(\Sigma_{f}^{-1})_{ij}=\frac{1}{\sigma_{f}^{2}}\big{[}\delta_{ij}-\gamma(\delta_{i,j-1}+\delta_{i,j+1})\big{]}+O(\gamma^{2})$.
The result is

$\displaystyle
Cov_{Y\vec{f}}^{T}\Sigma_{f}^{-1}Cov_{Y\vec{f}}=\frac{n(1-2\gamma)}{\sigma_{f}^{2}}\ln^{2}\Big{(}1+\frac{1}{n}\big{[}e^{\sigma_{f}^{2}}-1\big{]}\Big{)}$
$\displaystyle+O(\gamma^{2}).$

This term also vanishes in the limit of very large number of presynaptic
neurons, $n\mapsto\infty$.

Mutual information between pre- and post-synaptic neuronal activities is given
by the first line of Eq. (8), with $\det(\Sigma_{Y})=\sigma_{Y}^{2}$, and
$\det\Big{(}\Sigma_{Y}-Cov_{Y\vec{f}}^{T}\Sigma_{f}^{-1}Cov_{Y\vec{f}}\Big{)}=\sigma_{Y}^{2}-Cov_{Y\vec{f}}^{T}\Sigma_{f}^{-1}Cov_{Y\vec{f}}$,
which are given by Eq. (32) and (34). MI takes the form

$\displaystyle\mbox{MI}(\vec{f},Y)=\frac{1}{\ln
4}\ln\Big{(}\frac{\ln\big{(}1+\frac{1}{n}[e^{(\sigma_{f}^{2}+\sigma_{w}^{2})}-1]\big{)}}{\big{[}\ln\big{(}1+\frac{1}{n}[e^{(\sigma_{f}^{2}+\sigma_{w}^{2})}-1]\big{)}-\frac{n}{\sigma_{f}^{2}}\ln^{2}\big{(}1+\frac{1}{n}\big{[}e^{\sigma_{f}^{2}}-1\big{]}\big{)}\big{]}}\Big{[}1-\gamma
g(n,\sigma_{f}^{2},\sigma_{w}^{2})+O(\gamma^{2})\Big{]}\Big{)},$ (35)

where the function $g(n,\sigma_{f}^{2},\sigma_{w}^{2})$ is defined as

$\displaystyle
g(n,\sigma_{f}^{2},\sigma_{w}^{2})=\frac{2n\ln^{2}\big{(}1+\frac{1}{n}[e^{\sigma_{f}^{2}}-1]\big{)}\Big{[}\frac{(1-1/n)^{2}}{\big{(}1+\frac{1}{n}[e^{\sigma_{f}^{2}+\sigma_{w}^{2}}-1]\big{)}}+\frac{1}{\sigma_{f}^{2}}\ln\big{(}1+\frac{1}{n}[e^{(\sigma_{f}^{2}+\sigma_{w}^{2})}-1]\big{)}\Big{]}}{\ln\big{(}1+\frac{1}{n}[e^{(\sigma_{f}^{2}+\sigma_{w}^{2})}-1]\big{)}\Big{[}\ln\big{(}1+\frac{1}{n}[e^{(\sigma_{f}^{2}+\sigma_{w}^{2})}-1]\big{)}-\frac{n}{\sigma_{f}^{2}}\ln^{2}\big{(}1+\frac{1}{n}\big{[}e^{\sigma_{f}^{2}}-1\big{]}\big{)}\Big{]}}.$
(36)

We observe that negative correlations between presynaptic neurons ($\gamma<0$)
always increase MI, while their positive correlations ($\gamma>0$) tend to
decrease MI. Also note that MI does not depend at all on $\mu_{f}$ and
$\mu_{w}$. For very noisy synapses, i.e., for $\sigma_{w}\mapsto\infty$, both
factors under the logarithm in Eq. (35) tend to 1 (function $g\mapsto 0$), and
hence mutual information $\mbox{MI}\mapsto 0$. In the limit when number of
presynaptic neurons $n\mapsto\infty$, Eq. (35) simplifies significantly, and
we obtain

$\displaystyle\mbox{MI}(\vec{f},Y)_{n\gg 1}\approx\frac{-1}{\ln
4}\ln\Big{(}\frac{\Big{[}1-\frac{(e^{\sigma_{f}^{2}}-1)^{2}}{\sigma_{f}^{2}[e^{(\sigma_{f}^{2}+\sigma_{w}^{2})}-1]}\Big{]}}{\big{[}1-\gamma
g(n,\sigma_{f}^{2},\sigma_{w}^{2})\big{]}}\Big{)},$ (37)

where

$\displaystyle
g(n,\sigma_{f}^{2},\sigma_{w}^{2})\approx\frac{2n(e^{\sigma_{f}^{2}}-1)^{2}}{(e^{(\sigma_{f}^{2}+\sigma_{w}^{2})}-1)\big{[}(e^{(\sigma_{f}^{2}+\sigma_{w}^{2})}-1)-\frac{(e^{\sigma_{f}^{2}}-1)^{2}}{\sigma_{f}^{2}}\big{]}}.$

Eqs. (37) and (38) imply that MI grows with increasing the variance of
presynaptic neurons $\sigma_{f}^{2}$. Additionally, for very weak
variabilities in presynaptic firing rates and synaptic weights, i.e., for
$\sigma_{f}^{2},\sigma_{w}^{2}\ll 1$, we obtain even simpler form
$\mbox{MI}(\vec{f},Y)\approx\frac{1}{\ln
4}\ln\Big{(}\big{[}1+\frac{\sigma_{f}^{2}}{\sigma_{w}^{2}}\big{]}\big{[}1-\frac{2n\gamma\sigma_{f}^{4}}{\sigma_{w}^{2}(\sigma_{f}^{2}+\sigma_{w}^{2})}\big{]}\Big{)}$,
where
$\frac{\sigma_{f}^{2}}{\sigma_{w}^{2}}\approx\frac{Var(f_{i})}{Var(w_{i})}\frac{\langle
w_{i}\rangle^{2}}{\langle f_{i}\rangle^{2}}$. Thus the ratio
$\frac{\sigma_{f}^{2}}{\sigma_{w}^{2}}$ can be interpreted as the signal to
noise ratio, with the variability in synaptic weights serving as the noise.

##  IV Conclusions

In this paper we derived analytical expression for mutual information of two
random correlated vectors of arbitrary dimensions, which are lognormally
distributed. This exact formula involves determinants of multidimensional
matrices composing the general covariance matrix. We also obtained closed-form
analytical expressions of MI for different types of the covariance matrix,
mostly sparse with a high degree of symmetry. We hope that these formulas may
be helpful in many practical applications. Moreover, we applied these results
to information transfer in neuroscience, and provided an analytical formula
for MI of a neuron receiving many correlated synaptic inputs that are
lognormally distributed.

##  V Acknowledgments

The work was supported by the Polish National Science Centre (NCN) grant
number 2021/41/B/ST3/04300 (JK).

∗ Corresponding author<EMAIL_ADDRESS>
##  Appendix A Proof of Eq. (9).

It is sufficient to prove the upper part of Eq. (9). This inequality is
equivalent to the following inequality involving eigenvalues $\lambda_{i}$ of
$n\times n$ positive definite square matrix A:

$\displaystyle\sum_{i=1}^{n}\Big{(}1-\lambda_{i}^{-1}\Big{)}\leq\ln\Big{(}\prod_{i=1}^{n}\lambda_{i}\Big{)}\leq\sum_{i=1}^{n}\Big{(}\lambda_{i}-1\Big{)}.$
(39)

By the definition of matrix A, all the eigenvalues must be real and positive.
The above inequalities follow from elementary inequalities for logarithm,
namely $\frac{x}{(x+1)}\leq\ln(1+x)\leq x$, for arbitrary $x\geq-1$. Making a
substitution $x=\lambda_{i}-1$, and summing all the terms from $i=1$ to $i=n$,
we obtain Eq. (A1).

##  Appendix B Inverse and determinant of tridiagonal matrix, and proof of Eq.
(20).

To compute $Cov_{XY}^{T}\Sigma_{X}^{-1}Cov_{XY}$ in Eq. (20), we need first to
find the inverse of the tridiagonal matrix $\Sigma_{X}$. Let us consider a
general $n\times n$ tridiagonal matrix $A$ with all diagonal elements equal to
$t$, and off-diagonal elements all equal to $s$. Inverse elements of $A$ are
given by [22, 33]

$\displaystyle(A^{-1})_{ij}=(-1)^{i+j}s^{j-i}\theta_{i-1}\phi_{j+1}/\theta_{n}$
(40)

for $i\leq j$, where $\theta_{n}=\det(A)$. Matrix $A^{-1}$ is symmetric, and
hence $(A^{-1})_{ij}=(A^{-1})_{ji}$. In Eq. (B1), $\theta_{i}$ and $\phi_{i}$
are functions of $t$ and $s$. Additionally $\theta_{i}$ satisfy the following
recurrence relations

$\displaystyle\theta_{i}=t\theta_{i-1}-s^{2}\theta_{i-2}$ (41)

with $\theta_{0}=1$, and $\theta_{1}=t$. Similarly, $\phi_{i}$ satisfy the
following recurrence relations

$\displaystyle\phi_{i}=t\phi_{i+1}-s^{2}\phi_{i+2}$ (42)

with $\phi_{n+1}=1$, and $\phi_{n}=t$.

Both of the recurrence relations, Eqs. (B2) and (B3), can be solved by a
standard substitution $\theta_{i}=r^{i}$, with unknown $r$. The solution for
$r$ is $r_{\pm}=\frac{1}{2}(t\pm\sqrt{t^{2}-4s})$. The specific solutions for
$\theta_{i}$ and $\phi_{i}$ are given by linear combinations of $r_{-}$ and
$r_{+}$, with coefficients dependent on the boundary conditions in Eqs. (B2)
and (B3). It is easy to show that

$\displaystyle\theta_{i}(t,s)=\frac{\Big{[}\big{(}t+\sqrt{t^{2}-4s^{2}}\big{)}^{i+1}-\big{(}t-\sqrt{t^{2}-4s^{2}}\big{)}^{i+1}\Big{]}}{2^{i+1}\sqrt{t^{2}-4s^{2}}}$

and $\phi_{i}=\theta_{n-i+1}$, for $i=1,2,...,n$. The functions
$\theta_{i}(t,s)$ can be interpreted as determinants of reduced $i\times i$
matrices generated from the original matrix $A$, with all diagonal elements
$t$ and all off-diagonal elements $s$.

Now we can determine $Cov_{XY}^{T}\Sigma_{X}^{-1}Cov_{XY}$ in Eq. (20). We
have

$\displaystyle\sum_{i,j=1}^{n}(Cov_{XY}^{T})_{i}(\Sigma_{X}^{-1})_{ij}(Cov_{XY})_{j}=a^{2}\sum_{i,j=1}^{n}(\Sigma_{X}^{-1})_{ij}$
$\displaystyle=a^{2}\Big{(}\sum_{i=1}^{n}(\Sigma_{X}^{-1})_{ii}+2\sum_{i=1}^{n-1}\sum_{j=i+1}^{n}(\Sigma_{X}^{-1})_{ij}\Big{)},$

where the symmetric matrix $\Sigma_{X}^{-1}$ has the following elements

$\displaystyle(\Sigma_{X}^{-1})_{ij}=\frac{(-1)^{i+j}\gamma^{j-i}}{\det(\Sigma_{X})}\theta_{i-1}\theta_{n-j},$
(44)

for $i\leq j$. These sums can be executed, and after lengthly computations one
can obtain Eq. (20) in the main text.

## References

  * [1] EL Crow, K Shimizu (Eds.). Lognormal Distributions: Theory and Applications. New York: Dekker (1988).
  * [2] D Sornette, L Knopoff. The paradox of the expected time until the next earthquake. Bull. Seismol. Soc. America 87, 789-798 (1997).
  * [3] F Black, M Scholes. The pricing of options and corporate liabilities. J. Polit. Economy 81, 637 (1973).
  * [4] G Buzsaki, K Mizuseki. The log-dynamic brain: how skewed distributions affect network operations. Nature Rev. Neurosci. 15, 264-278 (2014).
  * [5] A Tang, H Liu, G Liu, et al. Lognormal Distribution of Local Strain: A Universal Law of Plastic Deformation in Material. Phys. Rev. Lett. 124, 155501 (2020). (doi = 10.1103/PhysRevLett.124.155501).
  * [6] Corral A, F Udina, E Arcaute. Truncated lognormal distributions and scaling in the size of naturally defined population clusters. Phys. Rev. E 101, 042312 (2020). ( doi = 10.1103/PhysRevE.101.042312).
  * [7] P Sobkowicz, M Thelwall, K Buckley, et al. Lognormal distributions of user post lengths in Internet discussions - a consequence of the Weber-Fechner law? EPJ Data Science 2, (2013). (doi = 10.1140/epjds14).
  * [8] TM Cover, JA Thomas, Elements of information theory. Wiley and Sons (2006).
  * [9] CE Shannon. A mathematical theory of communication. Bell System Technical Journal 27, 379-423 (1948). (doi=10.1002/j.1538-7305.1948.tb01338.x)
  * [10] C Lin, S Xiong, Y Chen. Mutual information maximizing GAN inversion for real face with identity preservation. J. of Visual Commun. and Image Repres. 87, 103566 (2022). (doi = https://doi.org/10.1016/j.jvcir.2022.103566).
  * [11] M Gabrie, A Manoel, C Luneau, et al. Entropy and mutual information in models of deep neural networks. J. Stat. Mech. 124014 (2019).
  * [12] Seifert U. Stochastic thermodynamics, fluctuation theorems and molecular machines. Rep. Prog. Phys. 75, 126001 (2012).
  * [13] G De Tomasi, S Bera, JH Bardarson, F Pollmann. Quantum Mutual Information as a Probe for Many-Body Localization. Phys. Rev. Lett. 118, 016804 (2017). (doi = 10.1103/PhysRevLett.118.016804).
  * [14] N Iizuka, D Kabat. Mutual information in Hawking radiation. Phys. Rev. D 88, 084010 (2013). (doi = 10.1103/PhysRevD.88.084010).
  * [15] P Golub, A Antalik, P Beran, J Brabec. Mutual information prediction for strongly correlated systems. Chem. Phys. Lett. 813, 140297 (2023). (https://doi.org/10.1016/j.cplett.2023.140297).
  * [16] Q Wang, Y Xiao, D Meng. Identification of structural key genes of mutual information gene networks of brain tumor. Physica A 608, 128322 (2022). (https://doi.org/10.1016/j.physa.2022.128322).
  * [17] R Cheong, A Rhee, CJ Wang, I Nemenman, A Levchenko. Information trasduction capacity of noisy biochemical signaling networks. Science 334, 354-358 (2011).
  * [18] F Rieke, D Warland, R de Ruyter, W Bialek. Spikes: Exploring the neural code. Cambridge, MA: MIT Press (1999).
  * [19] L Paninski. Estimation of entropy and mutual information. Neural Comput. 15, 1191-1253.
  * [20] RA Horn, CR Johnson. Matrix Analysis. Cambridge Univ. Press (1991).
  * [21] JR Silvester. Determinants of block matrices. The Mathematical Gazette 501, 460-467 (2000).
  * [22] GY Hu, RF O’Connell. Analytical inversion of symmetric tridiagonal matrices. J. Physics A, 29, 1511 (1996).
  * [23] LG Molinari. Determinants of block tridiagonal matrices. Linear Algebra and its Applications 429, 2221–2226 (2008).
  * [24] M Kac, WL Murdock, G Szegö. On the eigenvalues of certain Hermitian forms. Arch. Ration. Mech. Anal. 2, 767-800 (1953).
  * [25] JA Hertz, AS Krogh, RG Palmer. Introduction to the Theory of Neural Computation. Westview Press, Santa Fe, NM (1991).
  * [26] P Dayan, LF Abbott. Theoretical Neuroscience. Cambridge, MA: MIT Press (2001).
  * [27] GB Ermentrout. Linearization of F-I curves by adaptation. Neural Comput. 10, 1721-1729 (1998).
  * [28] GB Ermentrout, DH Terman. Mathematical Foundations of Neuroscience. Springer: New York, NY (2010).
  * [29] J Karbowski. Energetics of stochastic BCM type synaptic plasticity and storing of accurate information. J. Comput. Neurosci. 49, 71-106 (2021).
  * [30] AA Abu-Dayya, NC Beaulieu. Outage probabilities in the presence of correlated lognormal interferers. IEEE Trans. Veh. Techn. 43, 164-173 (1994).
  * [31] NB Mehta, J Wu, AF Molisch, J Zhang. Approximating a sum of random variables with a lognormal. IEEE Trans. Wireless Comm. 6, 2690-2699 (2007).
  * [32] N Brunel, JP Nadal. Mutual information, Fisher information and population coding. Neural Comput. 10, 1731-1757 (1998).
  * [33] RA Usmani. Inverse of a tridiagonal Jacobi Matrix. Linear Algebra and its Applications 212-213, 413-414 (1994).