# A theory for the sparsity emerged in the Forward Forward algorithm

Yukun Yang
Institute of Theoretical Computer Science
Graz University of Technology
<EMAIL_ADDRESS>

(February 2023)

###### Abstract

This report explores the theory that explains the high sparsity phenomenon [4]
observed in the forward-forward algorithm [2]. The two theorems proposed
predict the sparsity changes of a single data point’s activation in two cases:
Theorem 1: Decrease the goodness of the whole batch. Theorem 2: Apply the
complete forward forward algorithm to decrease the goodness for negative data
and increase the goodness for positive data. The theory aligns well with the
experiments tested on the MNIST dataset.

##  1 Definition

###  1.1 Linear layer

Let $\bm{x}\in\mathbb{R}^{m}$ be the input vector to the fully-connected layer
with $\mathbf{W}\in\mathbb{R}^{n\times m}$ the weight matrix. The output
activation of the fully-connected layer with the ReLU activation is given by:

$\bm{h}=\text{ReLU}(\mathbf{W}\bm{x})\in\mathbb{R}^{n},{\rm~{}~{}~{}where~{}~{}~{}}\text{ReLU}(\bm{v})[p]=\left\\{\begin{aligned}
&\bm{v}[p],\text{~{}if~{}}\bm{v}[p]>0\\\
&0,\text{~{}otherwise}\end{aligned}\right.,$ (1)

where $\bm{v}[p]$ is the $p$th element of the vector $\bm{v}$. Further, we
define a mask function $m$ indicating if each dimension of $\bm{h}$ is
positive or been masked out by ReLU to zero:

$\bm{m}\in\mathbb{R}^{n}\text{~{}~{}~{},~{}~{}~{}}\bm{m}[p]=\left\\{\begin{aligned}
&1,\text{~{}if~{}}\bm{h}[p]>0\\\ &0,\text{~{}otherwise}\end{aligned}\right..$
(2)

This proof only discusses the ReLU activation.

###  1.2 Goodness

For a batch of input vectors
$\mathcal{X}=\\{\bm{x}_{1},\bm{x}_{2},\cdots,\bm{x}_{N}\\}$ with batch size
$=N$, the corresponding outputs are defined as
$\mathcal{H}=\\{\bm{h}_{1},\bm{h}_{2},\cdots,\bm{h}_{N}\\}$, where
$\bm{h}_{i}=\text{ReLU}(\mathbf{W}\bm{x}_{i})$. We define the goodness as the
summation of all hidden activations’ $l_{2}$ norm minus a fixed threshold
$\theta$, i.e.:

$G(\mathcal{H})=\sum_{i=1}^{N}\|\bm{h}_{i}\|_{2}^{2}-\theta.$ (3)

###  1.3 Sparsity

The sparsity $S$ of an activation vector $\bm{h}$ is defined as [3]:

$S(\bm{h})=\frac{\sqrt{n}-\frac{\|\bm{h}\|_{1}}{\|\bm{h}\|_{2}}}{\sqrt{n}-1}.$
(4)

The sparsity measure $S(\bm{h})\in[0,1]$. A higher value of $S(\bm{h})$
indicates a sparser vector $\bm{h}$.

###  1.4 Forward forward algorithm

We define two group of data: A batch of positive data
$\mathcal{X}_{+}=\\{\bm{x}_{+1},\bm{x}_{+2},\cdots,\bm{x}_{+N}\\}$ with batch
size $=N$, the corresponding outputs are defined as
$\mathcal{H}_{+}=\\{\bm{h}_{+1},\bm{h}_{+2},\cdots,\bm{h}_{+N}\\}$, where
$\bm{h}_{+i}=\text{ReLU}(\mathbf{W}\bm{x}_{+i})$; and a batch of negative data
$\mathcal{X}_{-}=\\{\bm{x}_{-1},\bm{x}_{-2},\cdots,\bm{x}_{-N}\\}$ with batch
size $=N$, the corresponding outputs are defined as
$\mathcal{H}_{-}=\\{\bm{h}_{-1},\bm{h}_{-2},\cdots,\bm{h}_{-N}\\}$, where
$\bm{h}_{-i}=\text{ReLU}(\mathbf{W}\bm{x}_{-i})$.

Then, define loss $L$ as:

$L=G(\mathcal{H}_{-})-G({\mathcal{H}_{+}})$ (5)

Minimizing $L$ is equivalent to increase positive data’s goodness while
decrease negative data’s goodness. In each iteration, we perform the
stochastic gradient descent to train the weights $W$ with a batch of positive
data and a batch of negative data.

##  2 Results

Assumptions:

  1. 1.

The width of the linear layer $n\rightarrow+\infty$

  2. 2.

Learning rate $\eta=\mathcal{O}(\frac{1}{n})$

  3. 3.

The pre-activation input $\mathbf{W}\bm{x}$ does not have zero element:
$\forall\bm{x}_{i}\in\mathbf{X},\forall p,(\mathbf{W}\bm{x}_{i})[p]\neq 0$.
(Note: what I assumed here is neither the input $\bm{x}$ nor the output
$\bm{h}$ which usually has many zero elements. The weighted sum
$\mathbf{W}\bm{x}$ can rarely equals to exact zero.)

  4. 4.

All inputs $\bm{x}_{i}\in\mathbf{X}$ are normalized to have a unit length:
$\forall i,~{}\|\bm{x}_{i}\|_{2}=1$

  5. 5.

Weights are initialized by a Gaussian distribution
$\mathbf{W}\sim\mathcal{N}(0,\frac{2}{n})$. The variance $\sigma=\frac{2}{n}$
is picked following the Kaiming initialization [1], which guarantees the
output activation $\bm{h}$ has the same scale of $l_{2}$ norm as compared to
the input $\bm{x}$.

  6. 6.

All inputs and all layers at initialization are iid, i.e. independent,
identically distributed, and thus, so are the weights and gradients.

  7. 7.

The non-linear activation function is ReLU.

It is not always guaranteed that all samples would becomes sparser in the
forward forward algorithm. After each weight adjustment, there are always
samples’ sparsity that are increased, unchanged, or decreased. This depends on
the property of each data point. A property that, if the majority of data
points satisfy, leads to a sparser activation for all in general.

This proof begins with a simpler scenario: We wish to know under what property
of a data point that decreasing the goodness of the whole batch would increase
the data point’s sparsity. This gives us Theorem 1.

######  Theorem 1.

If a data point $\bm{x}_{i}$ with its activation $\bm{h}_{i}$ satisfies:

$\frac{\|\bm{h}_{i}\|_{1}}{\|\bm{h}_{i}\|_{2}}<\frac{\sum_{k=1}^{N}\|\bm{h}_{k}\odot\bm{m}_{i}\|_{1}cos(\bm{x}_{k},\bm{x}_{i})}{\sum_{k=1}^{N}\|\bm{h}_{k}\odot\bm{m}_{i}\|_{2}cos(\bm{h}_{i},\bm{h}_{k}\odot\bm{m}_{i})cos(\bm{x}_{k},\bm{x}_{i})},$
(6)

then decreasing the goodness $G(\mathcal{H})$ following gradient descent would
increase $\bm{h}_{i}$’s sparsity $S(\bm{h}_{i})$.

In above, $\odot$ is the elementwise product. Importantly to know, the reverse
order of this theorem is also true. If decreasing the goodness
$G(\mathcal{H})$ following gradient descent would increase $\bm{h}_{i}$’s
sparsity $S(\bm{h}_{i})$, then increase the goodness $G(\mathcal{H})$
following gradient descent would decrease $S(\bm{h}_{i})$. We cannot always
increase sparsity no matter if we decrease or increase the goodness. The
phenomenon that both positive and negative samples have their activations
sparsity increased in the forward forward algorithm cannot be studied
separately. It must be studied as a co-effect of the co-existence of both
groups of data samples.

Therefore, under the same assumptions, we derived the following Theorem for
the complete setting of the forward forward algorithm, where there exist both
positive data $\mathcal{X}_{+}$ and negative data $\mathcal{X}_{-}$, and their
corresponding activations $\mathcal{H}_{+}$ and $\mathcal{H}_{-}$:

######  Theorem 2.

If a data point $\bm{x}_{i}$ (No matter if $\bm{x}_{i}\in\mathcal{X}_{-}$ or
$\bm{x}_{i}\in\mathcal{X}_{+}$) with its activation $\bm{h}_{i}$ satisfies :

$\|\bm{h}_{i}\|_{2}(A_{+}-A_{-})<\|\bm{h}_{i}\|_{1}(B_{+}-B_{-})$ (7)

where

$\displaystyle A_{+}$
$\displaystyle=\sum_{k=1}^{N}\|\bm{h}_{+k}\odot\bm{m}_{i}\|_{1}cos(\bm{x}_{+k},\bm{x}_{i})$
(8) $\displaystyle A_{-}$
$\displaystyle=\sum_{k=1}^{N}\|\bm{h}_{-k}\odot\bm{m}_{i}\|_{1}cos(\bm{x}_{-k},\bm{x}_{i})$
$\displaystyle B_{+}$
$\displaystyle=\sum_{k=1}^{N}\|\bm{h}_{+k}\odot\bm{m}_{i}\|_{2}cos(\bm{h}_{i},\bm{h}_{+k}\odot\bm{m}_{i})cos(\bm{x}_{+k},\bm{x}_{i})$
$\displaystyle B_{-}$
$\displaystyle=\sum_{k=1}^{N}\|\bm{h}_{-k}\odot\bm{m}_{i}\|_{2}cos(\bm{h}_{i},\bm{h}_{-k}\odot\bm{m}_{i})cos(\bm{x}_{-k},\bm{x}_{i}),$

then decreasing the loss $L=G(\mathcal{H}_{-})-G({\mathcal{H}_{+}})$ following
gradient descent would increase $\bm{h}_{i}$’s sparsity $S(\bm{h}_{i})$.

##  3 Discussion

The key findings are (Please refer to the experimental results in the later
sections for detail):

  * •

Most of the data points satisfy Theorem 1 at the beginning of training:
Decrease the goodness would increase their sparsity, and increase the goodness
would decrease their sparsity.

  * •

When consider the co-effects of both positive and negative data, we observed
that there are usually more than half data points satisfy Theorem 2 and are
becoming sparser in training. This result holds for both positive and negative
data.

To dig out more insights from them, I am considering the following directions
like:

  * •

Interpret the intuitive meaning of the theorems, and explain why they are
(Especially Theorem 2) satisfied by most of the data points in a dataset like
MNIST and CIFAR10 in general.

  * •

Consider principles of designing the negative data based on Theorem 2.

  * •

Simplify the theorems by bringing in more details from experiments. For
example, consider what if negative samples are generated by substituting
positive samples’ labels by randomly generated wrong labels, so that
$\bm{x}_{-}i$ differs from $\bm{x}_{+}i$ by just two pixels.

  * •

Modularize the theorems to be more general, such that the effects on the
sparsity after changing the goodness function, the normalization method, and
others can be better predicted/explained.

  * •

Consider the extended versions like on recurrent neural networks, and on
spiking neurons with temporal dynamics to guide neuromorphic applications.

##  4 Proof

Sparsity comparison

Naming $\bm{h}_{i}$ and $\bm{h}_{i}^{\prime}$ as the activations before and
after an weight update respectively, our strategy is to compare the value of
the sparsity measures $S(\bm{h}_{i})$ and $S(\bm{h}_{i}^{\prime})$:

$\text{(Left~{}hand~{}side)~{}~{}~{}}\frac{\sqrt{n}-\frac{\|\bm{h}_{i}\|_{1}}{\|\bm{h}_{i}\|_{2}}}{\sqrt{n}-1}\text{~{}~{}~{}v.s.~{}~{}~{}}\frac{\sqrt{n}-\frac{\|\bm{h}_{i}^{\prime}\|_{1}}{\|\bm{h}_{i}^{\prime}\|_{2}}}{\sqrt{n}-1}\text{~{}~{}~{}(Right~{}hand~{}side)}.$
(9)

We aim to demonstrate that the sparsity prior to the update, represented by
the left-hand side, is less than or equal to the sparsity following the
update, represented by the right-hand side, when $\bm{h}_{i}$ fulfills the
condition specified in both theorems. We start by simplifying this comparison
to its equivalent forms:

$\displaystyle\Longleftrightarrow\sqrt{n}-\frac{\|\bm{h}_{i}\|_{1}}{\|\bm{h}_{i}\|_{2}}$
$\displaystyle\text{~{}~{}~{}v.s.~{}~{}~{}}\sqrt{n}-\frac{\|\bm{h}_{i}^{\prime}\|_{1}}{\|\bm{h}_{i}^{\prime}\|_{2}},$
(10)
$\displaystyle\Longleftrightarrow\frac{\|\bm{h}_{i}^{\prime}\|_{1}}{\|\bm{h}_{i}^{\prime}\|_{2}}$
$\displaystyle\text{~{}~{}~{}v.s.~{}~{}~{}}\frac{\|\bm{h}_{i}\|_{1}}{\|\bm{h}_{i}\|_{2}},$
$\displaystyle\Longleftrightarrow\|\bm{h}_{i}^{\prime}\|_{1}\|\bm{h}_{i}\|_{2}$
$\displaystyle\text{~{}~{}~{}v.s.~{}~{}~{}}\|\bm{h}_{i}\|_{1}\|\bm{h}_{i}^{\prime}\|_{2}.$

Further, when we replace $|\bm{h}_{i}^{\prime}|_{1}$ with its delta changes
$\|\bm{h}_{i}\|_{1}+\Delta_{l_{1}}$, and likewise substitute
$\|\bm{h}_{i}^{\prime}\|_{2}$ with $\|\bm{h}_{i}\|_{2}+\Delta_{l_{2}}$, we
obtain the comparison as:

$\displaystyle\Longleftrightarrow(\|\bm{h}_{i}\|_{1}+\Delta_{l_{1}})\|\bm{h}_{i}\|_{2}$
$\displaystyle\text{~{}~{}~{}v.s.~{}~{}~{}}\|\bm{h}_{i}\|_{1}(\|\bm{h}_{i}\|_{2}+\Delta_{l_{2}}),$
(11) $\displaystyle\Longleftrightarrow\Delta_{l_{1}}\|\bm{h}_{i}\|_{2}$
$\displaystyle\text{~{}~{}~{}v.s.~{}~{}~{}}\|\bm{h}_{i}\|_{1}\Delta_{l_{2}}.$

Such a comparison requires an evaluation on the changes of $\bm{h}_{i}$’s
$l_{1}$ and $l_{2}$ norms: $\Delta{l_{1}}$ and $\Delta{l_{2}}$. Next, we will
exam how the update on weights $\mathbf{W}$ affect these two values.

###  4.1 Proof of Theorem 1

Weight update:

Recalling the goodness
$G(\mathcal{H})=\sum_{k=1}^{N}\|\bm{h}_{k}\|_{2}^{2}-\theta$, we have the
partial derivative of $G$ w.r.t the weight $\mathbf{W}$ as:

$\frac{\partial G}{\partial\mathbf{W}}=\sum_{k=1}^{N}\frac{\partial
G}{\partial\bm{h}_{k}}\frac{\partial\bm{h}_{k}}{\partial\mathbf{W}}=\sum_{k=1}^{N}2(\bm{h}_{k}\odot\bm{m}_{k})\bm{x}_{k}^{T}.$
(12)

The term $\bm{m}_{k}$ comes from the derivative of ReLU function, which is
defined as:

$\bm{m}_{k}[p]=\left\\{\begin{aligned} &1,\text{~{}if~{}}\bm{h}_{k}[p]>0\\\
&0,\text{~{}otherwise}\end{aligned}\right..$ (13)

We notice that:

$(\bm{h}_{k}\odot\bm{m}_{k})[p]=\left\\{\begin{aligned}
&\bm{h}_{k}[p],\text{~{}if~{}}\bm{h}_{k}[p]>0\\\
&0,\text{~{}otherwise}\end{aligned}\right.\text{~{}~{}~{}equals~{}~{}to~{}~{}~{}}\bm{h}_{k},$
(14)

so $\bm{m}_{k}$ can be absorbed into $\bm{h}_{k}$, and (49) can be further
simplified as:

$\frac{\partial
G}{\partial\mathbf{W}}=\sum_{k=1}^{N}2\bm{h}_{k}\bm{x}_{k}^{T}.$ (15)

Applying gradient descent on W with the step size $\eta$, we have the weight
change $\Delta\mathbf{W}$ as:

$\Delta\mathbf{W}=-2\eta\sum_{k=1}^{N}\bm{h}_{k}\bm{x}_{k}^{T}.$ (16)

Therefore, the activation $\bm{h}_{i}$ and its updated activation
$\bm{h}_{i}^{\prime}$ are:

$\bm{h}_{i}=\text{ReLU}\left(\mathbf{W}\bm{x}_{i}\right),~{}~{}~{}\text{and~{}~{}~{}}\bm{h}_{i}^{\prime}=\text{ReLU}\left[\left(\mathbf{W}-2\eta\sum_{k=1}^{N}\bm{h}_{k}\bm{x}_{k}^{T}\right)\bm{x}_{i}\right].$
(17)

Next, we compare the above two vectors, and measure the changes of its $l_{1}$
and $l_{2}$ norms separately.

Change of the $\bm{l_{1}}$ norm:

$\Delta_{l_{1}}=\|\bm{h}_{i}^{\prime}\|_{1}-\|\bm{h}_{i}\|_{1}.$ (18)

With ReLU activation, all elements in both $\bm{h}_{i}$ and
$\bm{h}_{i}^{\prime}$ are non-negative, so $\forall p,\bm{h}_{i}[p]\geq 0$ and
$\bm{h}_{i}^{\prime}[p]\geq 0$, and the $l_{1}$ norm is simply the summation
of all elements:

$\displaystyle\Delta_{l_{1}}$
$\displaystyle=\|\bm{h}_{i}^{\prime}\|_{1}-\|\bm{h}_{i}\|_{1}=\sum_{p=1}^{n}\bm{h}_{i}^{\prime}[p]-\sum_{p=1}^{n}\bm{h}_{i}[p],$
(19) $\displaystyle=\mathds{1}^{T}(\bm{h}_{i}^{\prime}-\bm{h}_{i}).$

In above, $\mathds{1}$ is an all-one vector with length $n$. The scalar
product between $\mathds{1}$ and any other vector $\bm{h}$ with the same
length results in a summation of all terms of $\bm{h}$:

$\mathds{1}^{T}\bm{h}=\sum_{p=1}^{n}h[p].$ (20)

Bringing in the value of both $\bm{h}_{i}^{\prime}$ and $\bm{h}_{i}$ into
(19), we get:

$\Delta_{l_{1}}=\mathds{1}^{T}\left[\text{ReLU}\left((\mathbf{W}-2\eta\sum_{k=1}^{N}\bm{h}_{k}\bm{x}_{k}^{T})\bm{x}_{i}\right)-\text{ReLU}\left(\mathbf{W}\bm{x}_{i}\right)\right].$
(21)

Next, we will simplify the above equation by proving the sign of each element
in ReLU before and after the weight update is the same.

With assumption 2, we notice the changes $\Delta$ inside the ReLU before and
after the weight update is an infinitesimal vector when $n\rightarrow+\infty$:

$\Delta=-2\eta\sum_{k=1}^{N}\bm{h}_{k}\bm{x}_{k}^{T}\bm{x}_{i}=\mathcal{O}(\frac{1}{n})\sum_{k=1}^{N}\bm{h}_{k}\bm{x}_{k}^{T}\bm{x}_{i},$
(22)

From assumption 4, we have
$\bm{x}_{k}^{T}\bm{x}_{i}=cos(\bm{x}_{k},\bm{x}_{i})\leq 1$, which does not
enlarge the magnitude of the resulting vector. Since the batch size $N$ is a
constant, ($n>>N$), $\sum_{k=1}^{N}\bm{h}_{k}$ is among the same magnitude as
any $\bm{h}_{k}$.

As compared to $\mathbf{W}\bm{x}_{i}$, we can always have a $\Delta$ that has
a smaller absolute value on every element by increasing $n$. Therefore, with
assumption 1, we can guarantee: For $\bm{h}_{i}^{\prime}$ and $\bm{h}_{i}$,
the signs of all elements in ReLU does not change after a weight update:

$\text{sign}\left[\left(\mathbf{W}-2\eta\sum_{k=1}^{N}\bm{h}_{k}\bm{x}_{k}^{T}\right)\bm{x}_{i}\right]=\text{sign}\left(\mathbf{W}\bm{x}_{i}\right),$
(23)

With this property, the subtraction between two ReLU terms can be discussed
separately based on the sign inside ReLU.

For the $p$-th element, if $(\mathbf{W}\bm{x}_{i}[p])>0$, we can remove ReLU
in (19) and have the change $\bm{h}_{i}^{\prime}[p]-\bm{h}_{i}[p]$ equals to
$\Delta[p]$. Whereas if $(\mathbf{W}\bm{x}_{i})[p]<0$, then we have both
$\bm{h}_{i}[p]=\bm{h}_{i}^{\prime}[p]=0$. So the update has no change on
$\bm{h}_{i}[p]$. As a conclusion, we get:

$\bm{h}_{i}^{\prime}[p]-\bm{h}_{i}[p]=\left\\{\begin{aligned}
&\Delta[p],\text{~{}if~{}}(\mathbf{W}\bm{x}_{i})[p]>0\\\
&0,\text{~{}if~{}}(\mathbf{W}\bm{x}_{i})[p]<0\end{aligned}\right..$ (24)

Here we intentionally omit the situation when $(\mathbf{W}\bm{x}_{i}[p])=0$ by
assuming this is not exist (assumption 3). This assumption is generally true
for the weighted sum of an input’s all dimension can hardly be zero if the
input $\bm{x}_{i}$ is not an all-zero vector. We will further verify this
through experiments.

Such an effect can be perfectly captured by masking out the negative part of
$\Delta$ by $\bm{m}_{i}$:

$\bm{h}_{i}^{\prime}-\bm{h}_{i}=-2\eta\sum_{k=1}^{N}\bm{h}_{k}\bm{x}_{k}^{T}\bm{x}_{i}\odot\bm{m}_{i}.$
(25)

Bring the above equation into (19), we have:

$\displaystyle\Delta_{l_{1}}$
$\displaystyle=\mathds{1}^{T}(-2\eta\sum_{k=1}^{N}\bm{h}_{k}\bm{x}_{k}^{T}\bm{x}_{i}\odot\bm{m}_{i}),$
(26)
$\displaystyle=-2\eta\sum_{k=1}^{N}cos(\bm{x}_{k},\bm{x}_{i})\mathds{1}^{T}(\bm{h}_{k}\odot\bm{m}_{i}).$

Since all elements in $(\bm{h}_{k}\odot\bm{m}_{i})$ are non-negative with the
ReLU activation, the summation of all elements is equivalent to calculating
its $l_{1}$ norm. We arrive at our final form of the $\Delta_{l_{1}}$ as:

$\Delta_{l_{1}}=-2\eta\sum_{k=1}^{N}cos(\bm{x}_{k},\bm{x}_{i})\|\bm{h}_{k}\odot\bm{m}_{i}\|_{1}.$
(27)

The above equation is an accurate measure of the changes on $\bm{h}_{i}$’s
$l_{1}$ norm before and after a weight update.

Change of the $\bm{l_{2}}$ norm:

$\Delta_{l_{2}}=\|\bm{h}_{i}^{\prime}\|_{2}-\|\bm{h}_{i}\|_{2}.$ (28)

Performing multi-dimension Taylor expansion, we get:

$\|\bm{h}_{i}^{\prime}\|_{2}=\|\bm{h}_{i}\|_{2}+(\bm{h}_{i}^{\prime}-\bm{h}_{i})^{T}\mathcal{D}(\|\bm{h}_{i}\|_{2})+\frac{1}{2!}(\bm{h}_{i}^{\prime}-\bm{h}_{i})^{T}\mathcal{D}^{2}(\|\bm{h}_{i}\|_{2})(\bm{h}_{i}^{\prime}-\bm{h}_{i})+\cdots,$
(29)

where $\mathcal{D}(\|\bm{h}_{i}\|_{2})$ is the gradient of the $l_{2}$ norm at
$\bm{h}_{i}$, and $\mathcal{D}^{2}(\|\bm{h}_{i}\|_{2})$ is the Hessian matrix.
The gradient is:

$\mathcal{D}(\|\bm{h}_{i}\|_{2})=(\frac{\partial\|\bm{h}_{i}\|_{2}}{\partial\bm{h}_{i}[1]},\frac{\partial\|\bm{h}_{i}\|_{2}}{\partial\bm{h}_{i}[2]},\cdots,\frac{\partial\|\bm{h}_{i}\|_{2}}{\partial\bm{h}_{i}[n]})^{T}.$
(30)

For each term in the above equation, we have:

$\frac{\partial\|\bm{h}_{i}\|_{2}}{\partial\bm{h}_{i}[p]}=\frac{\partial\sqrt{\sum_{q=1}^{n}\bm{h}_{i}[q]^{2}}}{\partial\bm{h}_{i}[p]}=\frac{2\bm{h}_{i}[p]}{2\sqrt{\sum_{q=1}^{n}\bm{h}_{i}[q]^{2}}}=\frac{\bm{h}_{i}[p]}{\|\bm{h}_{i}\|_{2}}.$
(31)

Bring (31) back into (30), we have the gradient as:

$\mathcal{D}(\|\bm{h}_{i}\|_{2})=\frac{1}{\|\bm{h}_{i}\|_{2}}\bm{h}_{i}.$ (32)

Represent the higher terms by their big-O notation:

$\frac{1}{2!}(\bm{h}_{i}^{\prime}-\bm{h}_{i})^{T}\mathcal{D}^{2}(\|\bm{h}_{i}\|_{2})(\bm{h}_{i}^{\prime}-\bm{h}_{i})+\cdots=\mathcal{O}(\|\bm{h}_{i}^{\prime}-\bm{h}_{i}\|_{2}^{2}).$
(33)

Bring (25) into the above equation, we get:

$\mathcal{O}(\|\bm{h}_{i}^{\prime}-\bm{h}_{i}\|_{2}^{2})=\mathcal{O}(\|-2\eta\sum_{k=1}^{N}\bm{h}_{k}\bm{x}_{k}^{T}\bm{x}_{i}\odot\bm{m}_{i}\|_{2}^{2}).$
(34)

In the above equation, we know $\eta=\mathcal{O}(\frac{1}{n})$ from assumption
2; and we know $\bm{x}_{k}^{T}\bm{x}_{i}=cos(\bm{x}_{k},\bm{x}_{i})<1$ from
assumption 4. Based on assumption 5, the output $\bm{h}$ and the input
$\bm{x}$ shall be in the same magnitude. Considering the dataset’s size $N$ as
a constant, we have
$\|\sum_{k=1}^{N}\bm{h}_{k}\odot\bm{m}_{i}\|_{2}^{2}=\mathcal{O}(1)$.
Combining them all, we have:

$\mathcal{O}(\|\bm{h}_{i}^{\prime}-\bm{h}_{i}\|_{2}^{2})=\mathcal{O}(\frac{1}{n^{2}}).$
(35)

Bringing (29), (32), (33) and (35) into (28), we get:

$\Delta_{l_{2}}=(\bm{h}_{i}^{\prime}-\bm{h}_{i})^{T}\frac{1}{\|\bm{h}_{i}\|_{2}}\bm{h}_{i}+\mathcal{O}(\frac{1}{n^{2}}).$
(36)

Further bring (25) into the above equation to substitute
$(\bm{h}_{i}^{\prime}-\bm{h}_{i})$:

$\displaystyle\Delta_{l_{2}}$
$\displaystyle=\left(-2\eta\sum_{k=1}^{N}\bm{h}_{k}\bm{x}_{k}^{T}\bm{x}_{i}\odot\bm{m}_{i}\right)^{T}\frac{1}{\|\bm{h}_{i}\|_{2}}\bm{h}_{i}+\mathcal{O}(\frac{1}{n^{2}}),$
(37)
$\displaystyle=-2\eta\frac{1}{\|\bm{h}_{i}\|_{2}}\left(\sum_{k=1}^{N}(\bm{h}_{k}\odot\bm{m}_{i})cos(\bm{x}_{k},\bm{x}_{i})\right)^{T}\bm{h}_{i}+\mathcal{O}(\frac{1}{n^{2}}).$

To further simplify the above equation, we substitute the scalar product
between the two vectors $(\bm{h}_{k}\odot\bm{m}_{i})$ and $\bm{h}_{i}$ as
their cosine similarity multiplies their $l_{2}$ norms:

$\displaystyle\Delta_{l_{2}}$
$\displaystyle=-2\eta\frac{1}{\|\bm{h}_{i}\|_{2}}\sum_{k=1}^{N}\|\bm{h}_{k}\odot\bm{m}_{i}\|_{2}\|\bm{h}_{i}\|_{2}cos(\bm{h}_{k}\odot\bm{m}_{i},\bm{h}_{i})cos(\bm{x}_{k},\bm{x}_{i})+\mathcal{O}(\frac{1}{n^{2}}),$
(38)
$\displaystyle=-2\eta\sum_{k=1}^{N}\|\bm{h}_{k}\odot\bm{m}_{i}\|_{2}cos(\bm{h}_{k}\odot\bm{m}_{i},\bm{h}_{i})cos(\bm{x}_{k},\bm{x}_{i})+\mathcal{O}(\frac{1}{n^{2}}).$

Now we have the change on both $l_{1}$ and $l_{2}$ norms. Next, we will bring
them back to the sparsity comparison.

Final step

Recall comparing the value of the sparsity measures $S(\bm{h}_{i})$ and
$S(\bm{h}_{i}^{\prime})$:

$\text{(Left~{}hand~{}side)~{}~{}~{}}\frac{\sqrt{n}-\frac{\|\bm{h}_{i}\|_{1}}{\|\bm{h}_{i}\|_{2}}}{\sqrt{n}-1}\text{~{}~{}~{}v.s.~{}~{}~{}}\frac{\sqrt{n}-\frac{\|\bm{h}_{i}^{\prime}\|_{1}}{\|\bm{h}_{i}^{\prime}\|_{2}}}{\sqrt{n}-1}\text{~{}~{}~{}(Right~{}hand~{}side)},$
(39)

is equivalent to compare

$\text{(Left~{}hand~{}side)~{}~{}~{}}\Delta_{l_{1}}\|\bm{h}_{i}\|_{2}\text{~{}~{}~{}v.s.~{}~{}~{}}\|\bm{h}_{i}\|_{1}\Delta_{l_{2}}\text{~{}~{}~{}(Right~{}hand~{}side)}.$
(40)

Bringing the calculated $\Delta_{l_{1}}$ in (27) and $\Delta_{l_{1}}$ in (38)
into the above comparison, we get:

$\displaystyle-2\eta\sum_{k=1}^{N}$ $\displaystyle
cos(\bm{x}_{k},\bm{x}_{i})\|\bm{h}_{k}\odot\bm{m}_{i}\|_{1}\|\bm{h}_{i}\|_{2}\text{~{}~{}~{}v.s.~{}~{}~{}}$
(41)
$\displaystyle\|\bm{h}_{i}\|_{1}\left[-2\eta\sum_{k=1}^{N}\|\bm{h}_{k}\odot\bm{m}_{i}\|_{2}cos(\bm{h}_{k}\odot\bm{m}_{i},\bm{h}_{i})cos(\bm{x}_{k},\bm{x}_{i})+\mathcal{O}(\frac{1}{n^{2}})\right].$

Next, we divide both sides by $(-2\eta)$. Since $(-2\eta)<0$, we need to swap
the two sides after dividing:

$\|\bm{h}_{i}\|_{1}\left[\sum_{k=1}^{N}\|\bm{h}_{k\bm{m}_{i}}\|_{2}cos(\bm{h}_{k\bm{m}_{i}},\bm{h}_{i})cos(\bm{x}_{k},\bm{x}_{i})+\mathcal{O}(\frac{1}{n})\right]\text{~{}~{}v.s.~{}~{}}\sum_{k=1}^{N}cos(\bm{x}_{k},\bm{x}_{i})\|\bm{h}_{k\bm{m}_{i}}\|_{1}\|\bm{h}_{i}\|_{2},$
(42)

where $\bm{h}_{k\bm{m}_{i}}=\bm{h}_{k}\odot\bm{m}_{i}$ for short. Since
$\eta=\mathcal{O}(\frac{1}{n})$, dividing $\mathcal{O}(\frac{1}{n^{2}})$ by
$-2\eta$ results in $\mathcal{O}(\frac{1}{n})$. Reformulate the above equation
gives us the final form:

$\frac{\|\bm{h}_{i}\|_{1}}{\|\bm{h}_{i}\|_{2}}\text{~{}~{}v.s.~{}~{}}\frac{\sum_{k=1}^{N}cos(\bm{x}_{k},\bm{x}_{i})\|\bm{h}_{k\bm{m}_{i}}\|_{1}}{\sum_{k=1}^{N}\|\bm{h}_{k\bm{m}_{i}}\|_{2}cos(\bm{h}_{k\bm{m}_{i}},\bm{h}_{i})cos(\bm{x}_{k},\bm{x}_{i})+\mathcal{O}(\frac{1}{n})}.$
(43)

We can omit $\mathcal{O}(\frac{1}{n})$ as $n\rightarrow+\infty$.

In conclusion, an activation $\bm{h}_{i}$ becomes sparser after weight update:

$\text{(Left~{}hand~{}side)~{}~{}~{}}\frac{\sqrt{n}-\frac{\|\bm{h}_{i}\|_{1}}{\|\bm{h}_{i}\|_{2}}}{\sqrt{n}-1}<\frac{\sqrt{n}-\frac{\|\bm{h}_{i}^{\prime}\|_{1}}{\|\bm{h}_{i}^{\prime}\|_{2}}}{\sqrt{n}-1}\text{~{}~{}~{}(Right~{}hand~{}side)},$
(44)

is equivalent to requiring:

$\frac{\|\bm{h}_{i}\|_{1}}{\|\bm{h}_{i}\|_{2}}<\frac{\sum_{k=1}^{N}cos(\bm{x}_{k},\bm{x}_{i})\|\bm{h}_{k\bm{m}_{i}}\|_{1}}{\sum_{k=1}^{N}\|\bm{h}_{k\bm{m}_{i}}\|_{2}cos(\bm{h}_{k\bm{m}_{i}},\bm{h}_{i})cos(\bm{x}_{k},\bm{x}_{i})}.$
(45)

[Q.E.D.]

###  4.2 Proof of Theorem 2

Weight update:

Recalling the goodness
$G(\mathcal{H})=\sum_{k=1}^{N}\|\bm{h}_{k}\|_{2}^{2}-\theta$, and the loss
$L=G(\mathcal{H}_{-})-G({\mathcal{H}_{+}})$ we have the partial derivative of
$L$ w.r.t the weight $\mathbf{W}$ as:

$\frac{\partial L}{\partial\mathbf{W}}=\sum_{k=1}^{N}\frac{\partial
L}{\partial\bm{h}_{-k}}\frac{\partial\bm{h}_{-k}}{\partial\mathbf{W}}-\sum_{k=1}^{N}\frac{\partial
L}{\partial\bm{h}_{+k}}\frac{\partial\bm{h}_{+k}}{\partial\mathbf{W}}=\sum_{k=1}^{N}2(\bm{h}_{-k}\odot\bm{m}_{-k})\bm{x}_{k}^{T}-\sum_{k=1}^{N}2(\bm{h}_{+k}\odot\bm{m}_{+k})\bm{x}_{k}^{T}.$
(46)

The term $\bm{m}_{k}$ comes from the derivative of ReLU function, which is
defined as:

$\bm{m}_{k}[p]=\left\\{\begin{aligned} &1,\text{~{}if~{}}\bm{h}_{k}[p]>0\\\
&0,\text{~{}otherwise}\end{aligned}\right..$ (47)

We notice that:

$(\bm{h}_{k}\odot\bm{m}_{k})[p]=\left\\{\begin{aligned}
&\bm{h}_{k}[p],\text{~{}if~{}}\bm{h}_{k}[p]>0\\\
&0,\text{~{}otherwise}\end{aligned}\right.\text{~{}~{}~{}equals~{}~{}to~{}~{}~{}}\bm{h}_{k},$
(48)

so $\bm{m}_{k}$ can be absorbed into $\bm{h}_{k}$, and (49) can be further
simplified as:

$\frac{\partial
G}{\partial\mathbf{W}}=\sum_{k=1}^{N}2\bm{h}_{-k}\bm{x}_{-k}^{T}-\sum_{k=1}^{N}2\bm{h}_{+k}\bm{x}_{+k}^{T}.$
(49)

Applying gradient descent on W with the step size $\eta$, we have the weight
change $\Delta\mathbf{W}$ as:

$\Delta\mathbf{W}=-2\eta\sum_{k=1}^{N}\left(\bm{h}_{-k}\bm{x}_{-k}^{T}-\bm{h}_{+k}\bm{x}_{+k}^{T}\right).$
(50)

Therefore, the activation $\bm{h}_{i}$ and its updated activation
$\bm{h}_{i}^{\prime}$ are:

$\bm{h}_{i}=\text{ReLU}\left(\mathbf{W}\bm{x}_{i}\right),~{}~{}~{}\text{and~{}~{}~{}}\bm{h}_{i}^{\prime}=\text{ReLU}\left[\left(\mathbf{W}-2\eta\sum_{k=1}^{N}\left(\bm{h}_{-k}\bm{x}_{-k}^{T}-\bm{h}_{+k}\bm{x}_{+k}^{T}\right)\right)\bm{x}_{i}\right].$
(51)

Next, we compare the above two vectors, and measure the changes of its $l_{1}$
and $l_{2}$ norms separately.

Change of the $\bm{l_{1}}$ norm:

Following the same logic as (25) in the previous section, we can get:

$\bm{h}_{i}^{\prime}-\bm{h}_{i}=-2\eta\sum_{k=1}^{N}\left(\bm{h}_{-k}\bm{x}_{-k}^{T}\bm{x}_{i}\odot\bm{m}_{i}-\bm{h}_{+k}\bm{x}_{+k}^{T}\bm{x}_{i}\odot\bm{m}_{i}\right).$
(52)

Bring this into (19), we get:

$\displaystyle\Delta_{l_{1}}$
$\displaystyle=\mathds{1}^{T}(\bm{h}_{i}^{\prime}-\bm{h}_{i}),$ (53)
$\displaystyle=\mathds{1}^{T}(-2\eta\sum_{k=1}^{N}\left(\bm{h}_{-k}\bm{x}_{-k}^{T}\bm{x}_{i}\odot\bm{m}_{i}-\bm{h}_{+k}\bm{x}_{+k}^{T}\bm{x}_{i}\odot\bm{m}_{i}\right)),$
$\displaystyle=-2\eta\sum_{k=1}^{N}\left(\|\bm{h}_{-k}\odot\bm{m}_{i}\|_{1}cos(\bm{x}_{-k},\bm{x}_{i})-\|\bm{h}_{+k}\odot\bm{m}_{i}\|_{1}cos(\bm{x}_{+k},\bm{x}_{i})\right),$
$\displaystyle=2\eta(A_{+}-A_{-}),$

where

$\displaystyle A_{+}$
$\displaystyle=\sum_{k=1}^{N}\|\bm{h}_{+k}\odot\bm{m}_{i}\|_{1}cos(\bm{x}_{+k},\bm{x}_{i}),$
(54) $\displaystyle A_{-}$
$\displaystyle=\sum_{k=1}^{N}\|\bm{h}_{-k}\odot\bm{m}_{i}\|_{1}cos(\bm{x}_{-k},\bm{x}_{i}).$

Change of the $\bm{l_{2}}$ norm:

Recall the local Taylor expansion gives us (36):

$\Delta_{l_{2}}=(\bm{h}_{i}^{\prime}-\bm{h}_{i})^{T}\frac{1}{\|\bm{h}_{i}\|_{2}}\bm{h}_{i}+\mathcal{O}(\frac{1}{n^{2}}).$
(55)

Bring the $(\bm{h}_{i}^{\prime}-\bm{h}_{i})$ in (52) into the above equation,
we get:

$\displaystyle\Delta_{l_{2}}$
$\displaystyle=\left(-2\eta\sum_{k=1}^{N}\left(\bm{h}_{-k}\bm{x}_{-k}^{T}\bm{x}_{i}\odot\bm{m}_{i}-\bm{h}_{+k}\bm{x}_{+k}^{T}\bm{x}_{i}\odot\bm{m}_{i}\right)\right)^{T}\frac{1}{\|\bm{h}_{i}\|_{2}}\bm{h}_{i}+\mathcal{O}(\frac{1}{n^{2}}),$
(56)
$\displaystyle=-2\eta\sum_{k=1}^{N}\Bigg{(}\|\bm{h}_{-k}\odot\bm{m}_{i}\|_{2}cos(\bm{x}_{-k},\bm{x}_{i})cos(\bm{h}_{-k}\odot\bm{m}_{i},\bm{h}_{i})$
$\displaystyle-\|\bm{h}_{+k}\odot\bm{m}_{i}\|_{2}cos(\bm{x}_{+k},\bm{x}_{i})cos(\bm{h}_{+k}\odot\bm{m}_{i},\bm{h}_{i})\Bigg{)}+\mathcal{O}(\frac{1}{n^{2}}),$
$\displaystyle=2\eta(B_{+}-B_{-})+\mathcal{O}(\frac{1}{n^{2}}),$

where

$\displaystyle B_{+}$
$\displaystyle=\sum_{k=1}^{N}\|\bm{h}_{+k}\odot\bm{m}_{i}\|_{2}cos(\bm{x}_{+k},\bm{x}_{i})cos(\bm{h}_{+k}\odot\bm{m}_{i},\bm{h}_{i}),$
(57) $\displaystyle B_{-}$
$\displaystyle=\sum_{k=1}^{N}\|\bm{h}_{-k}\odot\bm{m}_{i}\|_{2}cos(\bm{x}_{-k},\bm{x}_{i})cos(\bm{h}_{-k}\odot\bm{m}_{i},\bm{h}_{i}).$

Now we have the change on both $l_{1}$ and $l_{2}$ norms. Next, we will bring
them back to the sparsity comparison.

Final step

Recall comparing the value of the sparsity measures $S(\bm{h}_{i})$ and
$S(\bm{h}_{i}^{\prime})$:

$\text{(Left~{}hand~{}side)~{}~{}~{}}\frac{\sqrt{n}-\frac{\|\bm{h}_{i}\|_{1}}{\|\bm{h}_{i}\|_{2}}}{\sqrt{n}-1}\text{~{}~{}~{}v.s.~{}~{}~{}}\frac{\sqrt{n}-\frac{\|\bm{h}_{i}^{\prime}\|_{1}}{\|\bm{h}_{i}^{\prime}\|_{2}}}{\sqrt{n}-1}\text{~{}~{}~{}(Right~{}hand~{}side)},$
(58)

is equivalent to compare

$\text{(Left~{}hand~{}side)~{}~{}~{}}\Delta_{l_{1}}\|\bm{h}_{i}\|_{2}\text{~{}~{}~{}v.s.~{}~{}~{}}\|\bm{h}_{i}\|_{1}\Delta_{l_{2}}\text{~{}~{}~{}(Right~{}hand~{}side)}.$
(59)

Bringing the calculated $\Delta_{l_{1}}$ in (53) and $\Delta_{l_{1}}$ in (56)
into the above comparison, we get:

$2\eta(A_{+}-A_{-})\|\bm{h}_{i}\|_{2}\text{~{}~{}~{}v.s.~{}~{}~{}}\|\bm{h}_{i}\|_{1}\left(2\eta(B_{+}-B_{-})+\mathcal{O}(\frac{1}{n^{2}})\right).$
(60)

Next, we devide both sides by $2\eta$. Since $\eta=\mathcal{O}(\frac{1}{n})$,
dividing $\mathcal{O}(\frac{1}{n^{2}})$ by $2\eta$ results in
$\mathcal{O}(\frac{1}{n})$:

$\|\bm{h}_{i}\|_{2}(A_{+}-A_{-})\text{~{}~{}~{}v.s.~{}~{}~{}}\|\bm{h}_{i}\|_{1}\left((B_{+}-B_{-})+\mathcal{O}(\frac{1}{n})\right).$
(61)

We can omit $\mathcal{O}(\frac{1}{n})$ as $n\rightarrow+\infty$.

In conclusion, when requiring goodness decrease for negative samples and
goodness increase for positive samples, asking an activation $\bm{h}_{i}$
becomes sparser after weight update:

$\text{(Left~{}hand~{}side)~{}~{}~{}}\frac{\sqrt{n}-\frac{\|\bm{h}_{i}\|_{1}}{\|\bm{h}_{i}\|_{2}}}{\sqrt{n}-1}<\frac{\sqrt{n}-\frac{\|\bm{h}_{i}^{\prime}\|_{1}}{\|\bm{h}_{i}^{\prime}\|_{2}}}{\sqrt{n}-1}\text{~{}~{}~{}(Right~{}hand~{}side)},$
(62)

is equivalent to requiring:

$\|\bm{h}_{i}\|_{2}(A_{+}-A_{-})<\|\bm{h}_{i}\|_{1}(B_{+}-B_{-}),$ (63)

where

$\displaystyle A_{+}$
$\displaystyle=\sum_{k=1}^{N}\|\bm{h}_{+k}\odot\bm{m}_{i}\|_{1}cos(\bm{x}_{+k},\bm{x}_{i}),$
(64) $\displaystyle A_{-}$
$\displaystyle=\sum_{k=1}^{N}\|\bm{h}_{-k}\odot\bm{m}_{i}\|_{1}cos(\bm{x}_{-k},\bm{x}_{i}),$
$\displaystyle B_{+}$
$\displaystyle=\sum_{k=1}^{N}\|\bm{h}_{+k}\odot\bm{m}_{i}\|_{2}cos(\bm{h}_{i},\bm{h}_{+k}\odot\bm{m}_{i})cos(\bm{x}_{+k},\bm{x}_{i}),$
$\displaystyle B_{-}$
$\displaystyle=\sum_{k=1}^{N}\|\bm{h}_{-k}\odot\bm{m}_{i}\|_{2}cos(\bm{h}_{i},\bm{h}_{-k}\odot\bm{m}_{i})cos(\bm{x}_{-k},\bm{x}_{i}).$

[Q.E.D.]

##  5 Preliminary Experimental Results

###  5.1 Theorem 1

MNIST on a randomly initialized linear layer:

Here we test how a real-world dataset satisfies the derived property in
Theorem 1:

$\text{(Left~{}hand~{}side)~{}~{}~{}}\frac{\|\bm{h}_{i}\|_{1}}{\|\bm{h}_{i}\|_{2}}<\frac{\sum_{k=1}^{N}cos(\bm{x}_{k},\bm{x}_{i})\|\bm{h}_{k\bm{m}_{i}}\|_{1}}{\sum_{k=1}^{N}\|\bm{h}_{k\bm{m}_{i}}\|_{2}cos(\bm{h}_{k\bm{m}_{i}},\bm{h}_{i})cos(\bm{x}_{k},\bm{x}_{i})}\text{~{}~{}~{}(Right~{}hand~{}side)}.$
(65)

We define a linear layer with ReLU activation initialized by
$\mathbf{W}\sim\mathcal{N}(0,\frac{2}{n})$, and feed in the normalized MNIST
images $\mathcal{X}=\\{\bm{x}_{1},\bm{x}_{2},\cdots,\bm{x}_{N}\\}$ with batch
size $=N$. The corresponding outputs are
$\mathcal{H}=\\{\bm{h}_{1},\bm{h}_{2},\cdots,\bm{h}_{N}\\}$. For each image,
we check its values on the left hand side (LHS) and the right hand side (RHS)
of Theorem 1. The results after traversing the whole training set is shown in
the figure below:

Figure 1: (A) With a randomly initialized linear layer that has n=128 neurons,
we feed in MNIST images $\bm{x}$ to get their activations $\bm{h}$. Then, for
each sample, we bring $\bm{x}$ and $\bm{h}$ into the equation in Theorem 1 and
visualized the values of LHS (blue) and RHS (orange). Almost all images
(>99.98%) in the MNIST dataset satisfies LHS<RHS. (B) Keeping all other hyper-
parameters fixed, the larger the batch size, the higher ratio of input data
points satisfy the derived property.

Therefore, the derived property is satisfied in most of the cases. This is not
surprising. Because decreasing goodness means decreasing all activation’s
$l_{2}$ norm, which would naturally lead many dimensions reduced to zero with
the ReLU activation. Yet as training proceed, many input samples’ activations
would be totally silenced, making further sparser impossible. Therefore we may
observe an decreasing in the ratio of data samples that become sparser as
training proceed. This is tested in the next experiment:

MNIST while training

As training proceed, it becomes more and more challenge to push the activation
sparser. Which means the ratio of data points in a batch that satisfies
Theorem 1 shall keep decreasing as well.

Here, we train the linear layer on the MNIST dataset for two epochs (938
iterations with batch size N=128) to decrease the goodness of each batch. We
plot both the ratio of data points that become sparser over the batch size N
in each iteration after the weight update, and the ratio of data points that
satisfy Theorem 1 among the whole batch. The result is shown in the figure
below:

Figure 2: The blue line keeps track of the actual ratio of data points that
become sparser over the batch size N in each iteration after the weight
update. The orange line is the ratio of data points that satisfy Theorem 1 in
each iteration. (Loss is defined as the square sum of the activation. Hyper-
parameters: Batch size N=128, neuron number n=2000, the optimizer adopted is
the stochastic gradient descent (SGD) with learning rate = 0.001, momentum=0).

One can observe that the two lines follow the same decreasing trend as
training proceed. Although the two lines do not perfectly overlap, the result
still demonstrated that the proposed Theorem is a good indicator about the
change of sparseness.

###  5.2 Theorem 2

We run the single layer forward forward algorithm (FFA) on MNIST with positive
examples and negative examples constructed the same way as in the original FFA
paper: for the positive samples, the inputs are the combination of both the
images and their corresponding labels, while the negative samples’ inputs are
the combination of images with randomly generated wrong labels.

We keep track of the ratio in each batch that becomes sparser for both
positive and negative samples, together with the ratio in a batch that
satisfies Theorem 2. The result is shown below:

Figure 3: In both (A) and (B), the blue lines keep track of the actual ratio
of data points that become sparser over the batch size N in each iteration
after the weight update. The orange lines are the ratios of data points that
satisfy Theorem 2 in each iteration. (A) shows the two ratios for the positive
data samples, (B) shows the two ratios of the negative data samples. (We
trained on MNIST for 5 epochs, with batch size N=128 for both the positive and
negative samples, neuron number n=2000, the optimizer is SGD with learning
rate = 0.001, momentum=0.)

From the result, we observe: 1) The proposed Theorem 2 follows the same trend
as the actual result, which shows the correctness of the derived theorem. The
mismatch may need further investigation. 2) The sparser ratio is larger then
50% for most of the time for both positive and negative samples, which support
the experimental findings that both the positive and negative samples would
have sparser activations as trained by the forward forward algorithm.

## References

  * [1] Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun  “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification”  In _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 1026–1034
  * [2] Geoffrey Hinton  “The forward-forward algorithm: Some preliminary investigations”  In _arXiv preprint arXiv:2212.13345_ , 2022
  * [3] Patrik O Hoyer  “Non-negative matrix factorization with sparseness constraints.”  In _Journal of machine learning research_ 5.9, 2004
  * [4] Niccolò Tosato et al.  “Emergent representations in networks trained with the Forward-Forward algorithm”  In _arXiv preprint arXiv:2305.18353_ , 2023


# Constrained Hamiltonian dynamics for electrons in magnetic field and
additional forces besides the Lorentz force acting on electrons

Hiroyasu Koizumi  Division of Quantum Condensed Matter Physics, Center for
Computational Sciences, University of Tsukuba, Tsukuba, Ibaraki 305-8577,
Japan<EMAIL_ADDRESS>
###### Abstract

We consider the forces acting on electrons in magnetic field including the
constraints and a condition arising from quantum mechanics. The force is
calculated as the electron mass, $m_{e}$, multiplied by the total time-
derivative of the velocity field evaluated using the quantum mechanical many-
electron wave function. The velocity field includes a term of the Berry
connection from the many-body wave function; thereby, quantum mechanical
effects are included. It is shown that additional important forces besides the
Lorentz force exist; they include the gradient of the electron velocity field
kinetic energy, the gradient of the chemical potential, and the ‘force’ for
producing topologically protected loop currents. These additional forces are
shown to be important in superconductivity, electric current in metallic
wires, and charging of capacitors.

  * April, 2024

##  1 Introduction

Currently, the basic equations for classical electromagnetism are four
Maxwell’s equations and the Lorentz force acting on charged particles
expressed using the electric field ${\bf E}$ and magnetic field ${\bf B}$ [1,
2]. Although the above Maxwell’s equations are named after Maxwell, the
equations put forward by him were rather different [3]; actually, he also
derived the Lorentz force,

$\displaystyle{\bf F}_{\rm Lorentz}=-e({\bf E}+{\bf v}\times{\bf B})$ (1)

before Lorentz, in the course of obtaining the equation for the Faraday’s law
of induction,

$\displaystyle{\cal E}=-{d\over{dt}}\int_{S(t)}{\bf B}({\bf r},t)\cdot d{\bf
S}$ (2)

where ${\cal E}$ is the electromotive force, $S(t)$ is the surface, ${\bf r}$
is the spatial coordinate, and $t$ is time [4]. The time dependence in the
surface indicates the possible time-dependent movement of it.

He realized that the force expressed as a gradient of a scalar potential
$\varphi$ was not enough to explain Faraday’s law of induction, and the vector
potential ${\bf A}$ was introduced to go beyond the usual Newtonian
description of the dynamics. With ${\bf A}$ and $\varphi$, ${\bf E}$ and ${\bf
B}$ are given by

$\displaystyle{\bf E}$ $\displaystyle=$ $\displaystyle-\partial_{t}{\bf
A}-\nabla\varphi$ $\displaystyle{\bf B}$ $\displaystyle=$
$\displaystyle\nabla\times{\bf A}$ (3)

Then, the electromotive force is expressed as

$\displaystyle{\cal E}=-{d\over{dt}}\int_{C(t)}{\bf A}({\bf r},t)\cdot d{\bf
r}$ (4)

where $C(t)$ is the perimeter of $S(t)$.

Assuming the fluid dynamics of electric charge, the total time derivative of
the vector potential

$\displaystyle{d\over{dt}}{\bf A}({\bf r},t)={\partial\over{\partial t}}{\bf
A}({\bf r},t)+{\bf v}({\bf r},t)\cdot\nabla{\bf A}({\bf r},t)$ (5)

was considered with ${\bf v}({\bf r},t)$ being the velocity field of the
electric fluid. Then, he showed that ${\bf E}$ arises from the first term, and
${\bf B}$ from the second. Later, however, the vector potential was considered
merely a mathematical tool, since the fluid that mediates electromagnetic
force envisaged by Maxwell (often called ‘ether’) was not detected [5]. Now
all the forces acting on an electron is believed to be given by Eq. (1) [6].

Next, let us consider an asymmetry in the electromotive force generation in a
system composed of a magnet and a conductor in relative motion. One of the
motivations that led Einstein to develop the special theory of relativity was
this asymmetry [7, 8]. The asymmetry is following: If the conductor is at rest
and the magnet is moving, ${\bf E}$ is generated in the conductor; on the
other hand, when the magnet is at rest and the conductor is moving, ${\bf B}$
is the cause of the electromotive force. He showed that ${\bf E}$ and ${\bf
B}$ appeared in Eq. (1) were connected by the Lorentz transformation, thus,
the asymmetry is just the difference in the frame used to describe the same
phenomenon [7, 8]. At this point, it was believed that standard four equations
plus the Lorentz force expressed by ${\bf E}$ and ${\bf B}$ are the whole
basic equations for electromagnetic phenomena [2].

However, Feynman claimed that the asymmetry still remains [9]; if the
conductor is at rest and the magnet is moving, ${\bf E}$ calculated by the
Maxwell-Faraday equation

$\displaystyle\nabla\times{\bf E}=-\partial_{t}{\bf B}$ (6)

is responsible for the electromotive force generation; on the other hand, if
the magnet is at rest and the conductor is moving, the Lorentz force in Eq.
(1) with ${\bf B}$ is responsible for the electromotive force generation [9];
thus, two different equations are used. On the other hand, the Maxwell’s
derivation using ${\bf A}$ is a unified one, where two cases arise from the
single ${\bf A}$; further, the Einstein’s Lorentz transformation argument is
already included in this view since the four vector $({\varphi},{\bf A})$
satisfies the Lorentz transformation [10].

In quantum mechanics, ${\bf A}$ is a fundamental physical quantity describing
electromagnetism as manifested in the Aharonov-Bohm effect [11, 12] that was
confirmed by experiments [13, 14]. Further, the velocity field of electrons
exists as the expectation value of the velocity operator calculated by the
many-body wave function. Thus, the quantum mechanical situation is more like
the one Maxwell envisaged. Actually, the asymmetry pointed out by Feynman can
be lifted by considering the duality of a $U(1)$ phase on the wave function
that appears when electromagnetic field exists [15].

In quantum mechanics, an additional vector potential type object arises. They
are called the ‘Berry connection ’ [16]. In particular, the Berry connection
from many-body wave functions,

$\displaystyle\\!{\bf A}^{\rm MB}_{\Psi}({\bf r},t)\\!=\\!{{{\rm
Re}\left\\{\int d\sigma_{1}d{\bf x}_{2}\cdots d{\bf x}_{N}\Psi^{\ast}({\bf
r},\sigma_{1},\cdots,{\bf x}_{N},t)(-i\hbar\nabla)\Psi({\bf
r},\sigma_{1},\cdots,{\bf x}_{N},t)\right\\}}\over{\hbar\rho({\bf r},t)}}$ (7)

defined by the present author is important for the electromagnetic problems.
It has been argued to play crucial roles in superconductivity [17, 18]. Here,
‘$\rm{Re}$’ denotes the real part, $\Psi$ is the total wave function, ${\bf
x}_{i}$ collectively stands for the coordinate ${\bf r}_{i}$ and the spin
$\sigma_{i}$ of the $i$th electron, $N$ is the total number of electrons,
$-i\hbar\nabla$ is the Schrödinger’s momentum operator for the coordinate
vector ${\bf r}$, and $\rho$ is the number density calculated from $\Psi$.

In previous works [17, 18], we have shown that the velocity field is given by

$\displaystyle{\bf v}={e\over m_{e}}{\bf A}+{\hbar\over{m_{e}}}{\bf
A}_{\Psi}^{\rm MB}$ (8)

including ${\bf A}_{\Psi}^{\rm MB}$ [19]. Then, the force acting on an
electron is calculated by

$\displaystyle m_{e}{d\over{dt}}{\bf v}={e}{d\over{dt}}{\bf
A}+{\hbar}{d\over{dt}}{\bf A}_{\Psi}^{\rm MB}$ (9)

From the term $e{d\over{dt}}{\bf A}$, the Lorentz force arises.

The main purpose of the present work is to show new forces arising from Eq.
(9), and consider examples where they play important roles. In the due course,
the meaning of the gauge invariance in the electromagnetism is changed; in
classical theory, all basic equations for electromagnetism are expressed using
${\bf E}$ and ${\bf B}$; there exists a freedom in the choice (or gauge) of
${\bf A}$ and $\varphi$ for the same ${\bf E}$ and ${\bf B}$ in Eq. (3); thus,
it is required that when a problem of electromagnetism is solved using ${\bf
A}$ and $\varphi$, the result should not depend on the gauge used.

The above gauge invariance requirement is handled differently in the present
work. The velocity field in Eq. (8) indicates that ${\bf A}_{\Psi}^{\rm MB}$
must compensate the gauge ambiguity in ${\bf A}$ and make the sum ${e\over
m_{e}}{\bf A}+{\hbar\over{m_{e}}}{\bf A}_{\Psi}^{\rm MB}$ gauge invariant. In
this way, the gauge invariance requirement is now the requirement on ${\bf
A}_{\Psi}^{\rm MB}$. It is worth noting that a similar requirement arises in
the Nambu-Goldstone mode appearing in the standard theory of superconductivity
[20]. The requirement on ${\bf A}_{\Psi}^{\rm MB}$ can be achieved by imposing
two constrains: 1) the conservation of local charge; 2) the single-valuedness
of wave functions as functions of spatial coordinates. Actually this point was
already explained in our previous work [18]. It is also important to know that
${\bf A}_{\Psi}^{\rm MB}$ gives rise to topologically protected loop currents,
and supercurrent is generated by the velocity field in Eq. (8) if those loop
currents are stable [19]. This indicates ${\bf A}_{\Psi}^{\rm MB}$ replaces
the Nambu-Goldstone mode in the standard theory of superconductivity.

The constraints on ${\bf A}_{\Psi}^{\rm MB}$ are due to quantum mechanics. In
other words, quantum mechanics requires additional constrains compared with
classical mechanics. Then, additional forces arise from them. Such a
possibility was already considered by Dirac in the theory of generalized
Hamiltonian dynamics[21, 22]. He considered constrains expressed as functions
of canonical conjugate variables; however, the two constrains we have
mentioned are more complicated ones. We also add a condition in the
generalized Hamiltonian dynamics; it is the presence of chemical potential (or
Fermi energy). Metallic electric current is generated by the chemical
potential difference generated by an electric battery [23, 24, 25], and the
present formalism explains this current generation by including the term of
the gradient of the chemical potential as a force arises in Eq. (9).

The organization of the present work is as follows: In Section 2, two
constraints and a condition arising from quantum mechanics are explained. In
Section 3, the Newton equation for the generalized Hamiltonian for electrons
in electromagnetic field is presented and examined for some examples. Lastly,
in Section 4, we conclude the present work.

##  2 Hamiltonian dynamics with two constraints and a condition from quantum
mechanics

We will consider two constrains and a condition on the generalized Hamiltonian
dynamics required by quantum mechanics. Some of the constrains are already
considered in our previous work [17, 18]; however, we explain them succinctly,
below, for completeness.

###  2.1 Single-valuedness of the many-electron wave function as a function of
electron coordinates

In the Schrödinger’s representation, the operator for the momentum
$\hat{p}_{r}$ conjugate to the canonical coordinate $\hat{q}_{r}$ is given by

$\displaystyle\hat{p}_{r}=-i\hbar{{\partial}\over{\partial q_{r}}},\quad
r=1,\cdots,n$ (10)

This implicitly requires the existence of ket vectors

$\displaystyle\hat{q}_{r}|q_{1},\cdots,q_{r},\cdots,q_{n}\rangle={q}_{r}|q_{1},\cdots,q_{r},\cdots,q_{n}\rangle$
(11)

that are eigenfunctions for the coordinate operators $\hat{q}_{r}$ [26]. Since
$q_{r}$ is an eigenvalue, it needs to be uniquely specified. The wave function
for a physical state $|\psi\rangle$ is given by

$\displaystyle\psi(q_{1},\cdots,q_{r},\cdots,q_{n})=\langle
q_{1},\cdots,q_{r},\cdots,q_{n}|\psi\rangle$ (12)

in the bra-ket notation. Thus, the wave function is a single-valued function
of $q_{r}$ due to the fact that the value of $q_{r}$ is uniquely specified.

The above single-valuedness constraint needs special attention when electrons
perform spin-twisting itinerant motion. Since electrons are particle with spin
${1\over 2}$, they may perform spin-twisting itinerant motion. Besides, their
wave functions are spinors, sign-change may occur when the spin-twisting
motion is such that it moves along a loop in the coordinate space with
performing odd number of spin rotations. In such a case, sign-change of wave
functions occurs if the usual Schrödinger equation is solved by the energy
minimization requirement. However, this sign-change is rectified if we
introduce a $U(1)$ phase that compensates the sign change. Adding such a
$U(1)$ phase is allowed in quantum mechanics, as is shown below; however, it
is not well-known. Usually, only cases with non sign-change are considered,
thus, the $U(1)$ phase is a constant. However, when dealing with cases with
the sign-change, this $U(1)$ becomes an additional degree-of-freedom needs to
be handle.

Let us explain the appearance of the additional $U(1)$ phase, and how to
handle when the sign-change occurs. We start with the commutation relations in
quantum mechanics given by

$\displaystyle[\hat{q}_{r},\hat{q}_{s}]=0,\quad[\hat{p}_{r},\hat{p}_{s}]=0,\quad[\hat{q}_{r},\hat{p}_{s}]=i\hbar\delta_{rs}$
(13)

These relations are more fundamental than the Schrödinger’s representation in
Eq. (10). Thus, we may use the following $\hat{p}_{r}$’s

$\displaystyle\hat{p}_{r}=-i\hbar{{\partial}\over{\partial q_{r}}}+{{\partial
F}\over{\partial q_{r}}},\quad r=1,\cdots,n$ (14)

where $F$ is a function of $q_{1},\cdots,q_{n}$, and also of time $t$.

Dirac claimed that this extra ${{\partial F}\over{\partial q_{r}}}$ is,
however, can be removed by the following transformation of the wave function,

$\displaystyle\psi(q_{1},\cdots,q_{n})\rightarrow
e^{i\gamma}\psi(q_{1},\cdots,q_{n})$ (15)

with $\gamma$ related to $F$ by $F=\hbar\gamma+\mbox{constant}$. Thus, we can
always use $p_{r}$’s in Eq. (10) and deal with the wave function
$e^{i\gamma}\psi(q_{1},\cdots,q_{n})$ [26]. The wave function
$e^{i\gamma}\psi(q_{1},\cdots,q_{n})$ is taken up as a whole in the usual
treatment, thus, $e^{i\gamma}$ is not considered.

Contrary to the Dirac’s claim, there exist situations where this $e^{i\gamma}$
as an extra degree-of-freedom that can be used to fulfill the single-
valuedness constraint [17]. Note that this type of $U(1)$ phase was first
introduced by O’Brien in the context of the dynamical Jahn-Teller problem,
where the degenerate electronic wave function by symmetry behaves as a spinor
[27].

The relation $[p_{r},p_{s}]=0$ in Eq. (14) requires the following condition

$\displaystyle{\partial\over{\partial q_{s}}}{{\partial F}\over{\partial
q_{r}}}-{\partial\over{\partial q_{r}}}{{\partial F}\over{\partial q_{s}}}=0$
(16)

In the case where $e^{i\gamma}$ is multi-valued, this condition is violated at
some points. They are topological defects where the amplitude of
$\psi(q_{1},\cdots,q_{n})$ is zero. Due to this zeroness of the amplitude, we
can treat them as singularites. Such defects arise when electrons perform
spin-twisting itinerant motion with centers of the spin-twisting being the
defects.

We rewrite $e^{i\gamma}\psi(q_{1},\cdots,q_{n})$ in Eq. (15) as

$\displaystyle\Psi({\bf x}_{1},\cdots,{\bf x}_{N},t)=\Psi_{0}({\bf
x}_{1},\cdots,{\bf x}_{N},t)\exp\left(i\sum_{j=1}^{N}\int_{0}^{{\bf
r}_{j}}{\bf A}_{\Psi}^{\rm MB}({\bf r}^{\prime},t)\cdot d{\bf
r}^{\prime}\right)$ (17)

so that the relation to the Berry connection in Eq. (7) becomes apparent; we
can identify $\gamma$ as

$\displaystyle\gamma=\sum_{j=1}^{N}\int_{0}^{{\bf r}_{j}}{\bf A}^{\rm
MB}_{\Psi}({\bf r}^{\prime},t)\cdot d{\bf r}^{\prime}$ (18)

For convenience sake, we introduce angular variable $\chi$ with period $2\pi$,

$\displaystyle{{\chi({\bf r},t)}}=-2\int^{{\bf r}}_{0}{\bf A}_{\Psi}^{\rm
MB}({\bf r}^{\prime},t)\cdot d{\bf r}^{\prime}$ (19)

The multi-valued behavior of $\chi$ is characterized by the topological
integer, ‘winding number’, defined by

$\displaystyle w_{C}[\chi]={1\over{2\pi}}\oint_{C}\nabla\chi\cdot d{\bf r}$
(20)

The winding number $w_{C}[\chi]$ becomes non-zero integer when singularities
of $\chi$ exist within the loop $C$. The wave function $\Psi$ is now expressed
as

$\displaystyle\Psi({\bf x}_{1},\cdots,{\bf x}_{N},t)=\Psi_{0}({\bf
x}_{1},\cdots,{\bf x}_{N},t)\exp\left(-{i\over 2}\sum_{j=1}^{N}\int_{0}^{{\bf
r}_{j}}\chi({\bf r}^{\prime},t)\cdot d{\bf r}^{\prime}\right)$ (21)

The velocity field ${\bf v}$ is calculated as the expectation value of the
velocity operator,

$\displaystyle{\bf v}={e\over m_{e}}{\bf A}+{\hbar\over{m_{e}}}{\bf
A}_{\Psi}^{\rm MB}={e\over m_{e}}\left({\bf
A}-{\hbar\over{2e}}\nabla\chi\right)$ (22)

Now we shall consider the single-valuedness constraint, explicitly. It is
taken as the constraint on winding numbers of $\chi$ around loops. The loops
for calculating them are constructed in the following way: Let us consider
‘flattening’ of a 3D lattice; examples are depicted in Figs. 1 and 2. The
‘flattening’ is a procedure in which some faces of the 3D lattice are removed
so that there is no cell surrounded by faces after the removing. For the
flattened lattice (i.e., the lattice obtained by flattening), there exists a
path from outside to the center of every cell. For the flattened lattice, the
following equality holds,

$\displaystyle N_{b}=N_{p}+N_{v}-1$ (23)

where $N_{b}$ is the total number of bonds (or edges), $N_{p}$ is the total
number of plaques (or faces), and $N_{v}$ is the total number of vertices (or
sites). The above equality is the Euler’s relation for a 2D lattice; thus, we
call the above procedure as the ‘flattening’. The same flattening is
considered in our previous work, where the flattened lattices are 2D lattices
in the 2D plane [18].

Figure 1: An example for flattening a 3D lattice. The four shaded faces in the
3D lattice are removed. Figure 2: Another example of flattening of a 3D
lattice. The eight shaded faces in the 3D lattice are removed.

The singularities of $\chi$ form strings. Analogous string of singularities is
considered by Dirac [28]. The Dirac’s string has terminal points, but, the
strings here are those without terminal points. When a closed path $C$ is
taken around a string, we have the winding number given in Eq. (20). If this
number is odd, $e^{-{i\over 2}\chi}$ (the $U(1)$ phase factor appearing in Eq.
(21)) changes sign after the shifting of the coordinate once around $C$. We
assume at most one string enters each cell from a face of the cell and exits
from another face of the same cell. The winding numbers of $\chi$ for all the
plaques of the flattened lattice are given as externally supplied parameters.
In other words, they are boundary conditions. They are chosen so that the
single-valuedness of the total wave function is achieved. From the single-
valuedness constraint, totally $N_{p}$ equations are obtained.

###  2.2 Conservation of local charge constraint

In order to specify $\nabla\chi$ in the lattice, we need to know $N_{b}$
values of the difference of $\chi$’s between bonds. The relation in Eq. (23)
indicates that we need $N_{v}-1$ additional equations besides $N_{p}$
equations from the single-valuedness constraint. Actually, the conservation of
local charge constraint supplies the required number of equations since the
number of points where the conservation is considered is $N_{v}$ with the
total charge conserved; $N_{v}$ equations are redundant, and $N_{v}-1$
equations are independent. With the two constrains, we can uniquely obtain the
discrete version of $\nabla\chi$.

Let us calculate the total time-derivative of the velocity field ${\bf v}$,

$\displaystyle{{d{\bf v}}\over{dt}}$ $\displaystyle=$
$\displaystyle\partial_{t}{\bf v}+({\bf v}\cdot\nabla){\bf v}$
$\displaystyle=$ $\displaystyle\partial_{t}{\bf v}+{1\over 2}\nabla v^{2}-{\bf
v}\times(\nabla\times{\bf v})$ $\displaystyle=$
$\displaystyle{e\over{m_{e}}}\partial_{t}{\bf
A}-{\hbar\over{2m_{e}}}\partial_{t}(\nabla\chi)+{1\over 2}\nabla v^{2}-{e\over
m_{e}}{\bf v}\times{\bf B}+{\hbar\over{2m_{e}}}{\bf
v}\times\nabla\times(\nabla\chi)$

where ${\bf B}=\nabla\times{\bf A}$ is use. The obtained $\nabla\chi$ can be
used in the discretized version of the above formula.

###  2.3 The existence of chemical potential

In order to deal with electric current conduction in metallic wires, we
include the chemical potential. The presence of the chemical potential is a
quantum effect arising from the existence of the Fermi energy in metals.

First, we use the relation ${\bf E}=-\partial_{t}{\bf A}-\nabla\varphi$ in Eq.
(LABEL:eq13a),

$\displaystyle{{d{\bf v}}\over{dt}}$ $\displaystyle=$
$\displaystyle{{-e}\over{m_{e}}}{\bf
E}+{{-e}\over{m_{e}}}\nabla\left(\varphi+{\hbar\over{2e}}\partial_{t}\chi\right)+{1\over
2}\nabla v^{2}-{e\over m_{e}}{\bf v}\times{\bf B}$
$\displaystyle+{\hbar\over{2m_{e}}}{\bf
v}\times\nabla\times(\nabla\chi)+{\hbar\over{2m_{e}}}(\nabla\partial_{t}-\partial_{t}\nabla)\chi$

Next, we consider the sum, ${\bf A}-{\hbar\over{2e}}\nabla\chi$, in ${\bf v}$.
This sum is gauge invariant, which also implies its time-component partner
${\varphi}+{\hbar\over{2e}}\partial_{t}\chi$ is gauge invariant. Taking into
account the fact that the chemical potential $\mu$ appears in a similar manner
as the scalar potential $\varphi$ in the electron Hamiltonian, and also the
fact that the chemical potential is gauge invariant, we obtain the following
relation,

$\displaystyle\mu=e\left(\varphi+{\hbar\over{2e}}\partial_{t}\chi\right)$ (26)

With this identification, we can include the condition that the chemical
potential exists in a metal. Then, Eq. (LABEL:eq24) is further modified as

$\displaystyle{{d{\bf v}}\over{dt}}$ $\displaystyle=$
$\displaystyle{{-e}\over{m_{e}}}{\bf E}-{{1}\over{m_{e}}}\nabla\mu+{1\over
2}\nabla v^{2}+{{-e}\over m_{e}}{\bf v}\times{\bf B}$ (27)
$\displaystyle+{\hbar\over{2m_{e}}}{\bf
v}\times\nabla\times(\nabla\chi)+{\hbar\over{2m_{e}}}(\nabla\partial_{t}-\partial_{t}\nabla)\chi$

##  3 Newton equation for the generalized Hamiltonian for electrons in
electromagnetic field

###  3.1 Classical single electron case

The classical single electron case is obtained by omitting terms with $\chi$,
and also terms arising from many-body effects, $-{{1}\over{m_{e}}}\nabla\mu$
and ${1\over 2}\nabla v^{2}$, in Eq. (27). The result is

$\displaystyle m_{e}{{d{\bf v}}\over{dt}}={\bf F}_{\rm Lorentz}$ (28)

This is the formula in the usual classical electromagnetic theory.

###  3.2 Classical many electron case

In this case, the Newton equation is obtained by omitting terms with $\chi$ in
Eq. (27). The result is

$\displaystyle m_{e}{{d{\bf v}}\over{dt}}={\bf F}_{\rm
Lorentz}-\nabla\mu+\nabla{m_{e}\over 2}v^{2}$ (29)

The presence of the force $-\nabla\mu$ changes the explanation for the energy
flow occurring in a system with a metallic wire and an electrical battery, and
also for that occurring in charging of a capacitor. We consider these cases,
below.

####  3.2.1 Steady straight current in a metallic wire connected to an
electric battery.

The battery connected to a metallic wire generates the chemical potential
difference [23, 24, 25]. In the steady straight current case, we have $\nabla
v^{2}=0$ in the current flow direction. The current generates ${\bf B}$ but no
force acting on the conduction electrons is generated. We consider the steady
flow of electrons with the compensating back ground positive charge, which
yields ${\bf E}=0$. We also add the term arising from resistivity using the
relaxation approximation (this term is absent in Eq. (27)). Then, the Newton
equation for this case is given by

$\displaystyle m_{e}{{d{\bf v}}\over{dt}}=-\nabla\mu-m_{e}{1\over\tau}{\bf
v}=0$ (30)

where $\tau$ is the relaxation time.

In the usual explanation for the energy flow, $-e{\bf E}$ is used instead of
$-\nabla\mu$ appearing in Eq. (30). Then, non-zero Poynting vector arises; it
gives rise to the energy flow of radiation entering entering from the outside
of the wire [29]. This explanation is odd since the source of the energy is
chemical reactions in the battery. The above equation explains that the energy
flow occurs through the wire due to the generation of the chemical potential
difference by the battery.

####  3.2.2 Charging of a capacitor by an electric battery.

The charging of a capacitor generates electric field between the electrodes of
the capacitor. In the usual explanation for the energy flow, ${\bf E}$
developed between the electrodes and the magnetic field generated by the
displacement current arising from the time-dependence of ${\bf E}$ between the
electrodes generates ${\bf B}$. By these ${\bf E}$ and ${\bf B}$, non-zero
Poynting vector is produced, giving rise to the energy flow by radiation that
enters from outside.Thus, this energy flow supplies the energy stored in the
electric field of the capacitor [29]. This is odd, since the energy is
supplied by chemical reactions in the battery that produce the chemical
potential difference of the two electrodes.

We can explain the energy flow in this case as follows. The vector potential
in this case is calculated using the formula

$\displaystyle{\bf A}({\bf r},t)={\mu_{0}\over{4\pi}}\int
d^{3}r^{\prime}{{{\bf j}({\bf r}^{\prime},t-|{\bf r}-{\bf
r}^{\prime}|/c)}\over{|{\bf r}-{\bf r}^{\prime}|}}$ (31)

where $c$ is the speed of light; thus, only the true current ${\bf j}$
contributes to the generation of ${\bf B}$ and the displacement current
generation of ${\bf B}$ is absent. Thus, the energy flow by the radiation does
not occur. The electric field energy stored in the capacitor is supplied by
the battery that generates the chemical potential difference between the
electrodes of the capacitor.

###  3.3 Quantum many electron case

When dealing with cases where ${\bf A}$ appears explicitly as in the Meissner
current in superconductors, Eq. (LABEL:eq13a) must be used. In this case, the
Newton equation is given by

$\displaystyle m_{e}{{d{\bf v}}\over{dt}}={e}\partial_{t}{\bf
A}-{\hbar\over{2}}\partial_{t}(\nabla\chi)+{m_{e}\over 2}\nabla v^{2}-{e}{\bf
v}\times{\bf B}+{\hbar\over{2}}{\bf v}\times\nabla\times(\nabla\chi)$ (32)

We examine some examples that can be explained by forces in the above
equation.

####  3.3.1 ${\hbar\over{2}}{\bf v}\times\nabla\times(\nabla\chi)$.

With this term, the effective magnetic field becomes,

$\displaystyle{\bf B}\rightarrow{\bf
B}-{\hbar\over{2e}}\nabla\times(\nabla\chi)$ (33)

as infered in Eq. (32). This combination of terms appears in the magnetic
field equation for superconductors in the presence of vortices [30]. Using the
Ampère’s circuital law

$\displaystyle\nabla\times{\bf B}=\mu_{0}{\bf j}$ (34)

where $\mu_{0}$ is the permeability of free space, the current density

$\displaystyle{\bf j}=-en_{s}{\bf v}$ (35)

where $n_{s}$ is the superconducting electron density, and the velocity field
in Eq. (22), we have

$\displaystyle\nabla\times(\nabla\times{\bf B})=-{{e^{2}n_{s}\mu_{0}}\over
m_{e}}\left[{\bf B}-{\hbar\over{2e}}\nabla\times(\nabla\chi)\right]$ (36)

Note that, the replacement in Eq. (33) appears in the right-hand-side of the
equation. This equation is simplified as

$\displaystyle\nabla^{2}{\bf B}={1\over{\lambda_{L}^{2}}}{\bf
B}-{\Phi_{0}\over{\lambda_{L}^{2}}}{1\over{2\pi}}\nabla\times(\nabla\chi)$
(37)

where $\Phi_{0}={{h}\over{2e}}$ is the flux quantum, and
$\lambda_{L}=\sqrt{m_{e}\over{\mu_{0}n_{s}e^{2}}}$ is the London penetration
depth.

Figure 3: Schematic pictures for the relation between flow and vorticity. a:
Magnetic field ${\bf B}$ is given as the vorticity of the vector potential
${\bf A}$ as ${\bf B}=\nabla\times{\bf A}$ in the Maxwell’s view of the
magnetic field [31]. b: Vorticity arising as a delta-function like magnetic
field from $\nabla\chi$. Here, the situation is considered where $\chi$
creates a winding number one vortex line centered at ${\bf r}$ along $z$-axis.
There exists a string of singularities of the wave function along the vortex
line. Note that such an object was considered by Dirac [28].

Maxwell interpreted that ${\bf B}$ is the vorticity of the flow ${\bf A}$ [31]
(Fig. 3a). Likewise, we can consider the vorticity from the current
$-{\hbar\over{2e}}\nabla\chi$. It is expressed using a delta-function as will
be explained, below. Let us consider the case where the winding number is one,
$w_{C}[\chi]=1$, and string of singularities exist along $z$-axis (Fig. 3b).
In this case, the following relation holds

$\displaystyle{1\over{2\pi}}\nabla\times(\nabla\chi)=\delta^{(2)}({\bf r}){\bf
e}_{z}$ (38)

where $\delta^{(2)}({\bf r})$ is a two-dimensional ($xy$-plane) delta
function, and ${\bf e}_{z}$ is the unit vector in the $z$-direction. Thus, Eq.
(37) becomes the formula first put forward by Abrikosov [30, 32]. It is
notable that a similar string of singularities was envisaged by Dirac [28]. In
the present case, those strings provide vorticity that produce loop currents.

####  3.3.2 ${m_{e}\over 2}\nabla v^{2}$.

Figure 4: A situation for the normal-superconducting phase transition in a
magnetic field. The superconductor phase exists $x\leq 0$, and the normal
phase exists in $x>0$. The magnetic field in the normal phase region is ${\bf
B}=B_{0}{\bf e}_{z}$. The Lorentz force $-e{\bf v}\times{\bf B}$ and the
gradient of the kinetic energy force ${m_{e}\over 2}\nabla v^{2}$ balance.

In a type I superconductor, a reversible superconducting-normal phase
transition in a magnetic field occurs. We have discussed this before [18],
however, we reproduce it succinctly, below, for the convenience of the later
discussion. The situation we consider is depicted in Fig. 4. The magnetic
field in the superconductor is given by

$\displaystyle{\bf B}=B_{0}{\bf e}_{z}e^{x/\lambda_{L}}$ (39)

Using the Ampère’s circuital law, the current density is calculated as

$\displaystyle{\bf j}=-{1\over{\mu_{0}\lambda_{L}}}B_{0}{\bf
e}_{y}e^{x/\lambda_{L}}$ (40)

Then, the velocity field is obtained as

$\displaystyle{\bf v}={1\over{n_{s}e\mu_{0}\lambda_{L}}}B_{0}{\bf
e}_{y}e^{x/\lambda_{L}}$ (41)

From Eqs. (39) and (41), the following relation is obtained

$\displaystyle{m_{e}\over 2}\partial_{x}v^{2}=e({\bf v}\times{\bf
B})_{x}={{B_{0}^{2}}\over{n_{s}\mu_{0}\lambda_{L}}}e^{{2x}\over{\lambda_{L}}}$
(42)

This indicates that the inclusion of this term explains the force balance,
correctly, which is not possible by the standard theory that only includes the
Lorentz force.

Figure 5: The force balance of electrons for the centripetal, Lorentz, and
gradient of the kinetic energy forces in the equatorial plane normal to the
angular velocity ${\bm{\omega}}$ of a rotating spherical superconductor. The
correct centripetal force is given by $-e{\bf v}\times{\bf B}+{m_{e}\over
2}\nabla v^{2}=-{e\over 2}{\bf v}\times{\bf B}$.

A similar force balance appears in the London magnetic field problem of a
rotating superconductor. In a rotating superconductor, the velocity potential
is given by

$\displaystyle{\bf v}={\bm{\omega}}\times{\bf r}$ (43)

where ${\bm{\omega}}$ is the angular velocity of the rotation. Then, from Eqs.
(22) and (43), we have

$\displaystyle\nabla\times({\bm{\omega}}\times{\bf
r})=\nabla\times\left[{e\over m_{e}}\left({\bf
A}-{\hbar\over{2e}}\nabla\chi\right)\right]$ (44)

Neglecting delta-function type contributions that might arise from
$\nabla\times(\nabla\chi)$, we have

$\displaystyle{\bf B}={{2m_{e}}\over e}{\bm{\omega}}$ (45)

This is the London magnetic field associated with the London moment
phenomenon.

The correct centripetal force for the electron cannot be obtained by the
standard theory [33, 34]. However, the correct one is obtained by including
the gradient of the kinetic energy term as shown below. Let us consider a
sphere of a superconductor, and consider the equatorial plane whose normal is
the parallel to ${\bm{\omega}}$ (see Fig. 5). The centripetal force in this
plane is in the direction toward the center, and its magnitude is given by

$\displaystyle{{m_{e}v^{2}}\over r}={{m_{e}\omega^{2}r^{2}}\over
r}={m_{e}\omega^{2}r}$ (46)

The Lorentz force $-e{\bf v}\times{\bf B}$ is in the same direction as the
centripetal force, and its magnitude is

$\displaystyle e(r\omega){{2m_{e}}\over e}{\omega}={{2m_{e}}\omega^{2}r}$ (47)

The force ${m_{e}\over 2}\nabla v^{2}$ is in the opposite direction to the
centripetal force; its magnitude is given by

$\displaystyle{m_{e}\over 2}{d\over{dr}}(r\omega)^{2}={m_{e}\omega^{2}r}$ (48)

From Eqs. (46)-(48), the force balance

$\displaystyle\mbox{Centripetal force}=-e{\bf v}\times{\bf B}+{m_{e}\over
2}\nabla v^{2}$ (49)

is obtained.

####  3.3.3  $-{\hbar\over{2}}\partial_{t}(\nabla\chi)$.

The reversible conversion between magnetic flux and the supercurrent occurs in
the reversible superconducting-normal phase transition in a type I
superconductor problem [35]. In this process, the appearance of an electric
field due to the time-variation of the magnetic field must be avoided. This
means that the conventional Maxwell-Faraday equation $\nabla\times{\bf
E}=-\partial_{t}{\bf B}$ cannot be used. In the present formalism, this
equation is not a basic equation. Instead, we consider the relation in Eq.
(4).

Actually, ${\cal E}$ given by

$\displaystyle{\cal E}={1\over{-e}}\int_{C}{{d(m_{e}{\bf v})}\over{dt}}\cdot
d{\bf r}$ (50)

Thus, ${\cal E}$ is regarded as the time-dependent change of the circulation

$\displaystyle\partial_{t}\int_{C}{\bf v}\cdot d{\bf r}$ (51)

multiplied by $-m_{e}/e$. Here, the case is assumed wheres the time-dependence
of the $C(t)$ is absent. If we consider the circulation generated by
$\nabla\chi$, the term $-{\hbar\over{2}}\partial_{t}(\nabla\chi)$ can be
considered as the force for generating topologically protected loop currents.
In this case, we can write ${\cal E}$ as

$\displaystyle{\cal E}={1\over{e}}\int_{C}{{\hbar}\over
2}\partial_{t}(\nabla\chi)\cdot d{\bf r}$ (52)

On the other hand, ${e}\partial_{t}{\bf A}$ may be viewed as the force for
generating magnetic flux. Thus, the condition for the reversible conversion
between magnetic flux and the topologically protected loop current is
expressed by the following equality

$\displaystyle-\oint_{C}\partial_{t}{\bf A}({\bf r},t)\cdot d{\bf
r}=\oint_{C}{{\hbar}\over{2e}}\partial_{t}\nabla\chi\cdot d{\bf r}$ (53)

which may be simplified as

$\displaystyle\partial_{t}{\bf A}=-{{\hbar}\over{2e}}\partial_{t}\nabla\chi$
(54)

This is the condition we have obtained in our previous work [35].

##  4 Concluding remarks

Figure 6: Schematic pictures of the current in a rotating superconducting
sphere that generates the London magnetic field. a: The velocity field for
electrons ${\bf v}$ is generated by the rotation. Initially, the magnetic
field is zero, ${\bf B}=0$; due to the gradient of the kinetic energy force
$\nabla{m_{e}\over 2}{\bf v}^{2}$, electrons move outward direction to the
surface. As a consequence the velocity of the underlying positive cores
exceeds that of electrons, generating current ${\bf j}$. The generated current
produces the Lorenz force that acts on electrons moving outward; eventually,
the velocity of the electrons becomes that expected by the whole system
rotation with ${\bm{\omega}}$, except in the surface region where outward
motion of electrons is impossible. b: The surface current is denoted by ${\bf
j}_{\rm surf}$, which is generated due to the slow-down of the velocity of
surface electrons by ${\bf v}_{\rm surf}$. Here, the slow-down velocity means
it is slower than this amount from the velocity of the whole system rotation
with ${\bm{\omega}}$.

Here, we give a possible scenario how the London moment is generated from
${\bm{\omega}}=0$ initial state. The explanation of this process has not been
successful by the standard London theory that includes only the Lorentz force.
When a spherical superconductor starts rotating from ${\bm{\omega}}=0$, first,
the magnetic field is absent; then, electrons move outward radial direction
due to the force ${{m_{e}}\over 2}\nabla{v}^{2}$ since the velocity of inner
electrons is slower than the outer ones. This electron motion generates a
magnetic field by the current creation due to the difference in the velocity
of electrons and that of the underlying positive ion cores; the positive ion
cores move with velocity by the ${\bm{\omega}}$ rotation, however, that of the
negative electrons moved from inner area is slower (Fig. 6).

When the magnetic field is generated, the motion of the electrons is
accelerated by the Lorentz force produced by the generated magnetic field and
the outward radial direction velocity of electrons; eventually, electrons move
with the rotating sphere with the same velocity as the underlying ion cores.
This change of the electron velocity does not occur in the surface region,
since the outward radial direction motion is impossible there; therefore the
electron velocity remains slower than that of the positive ion cores; this
slower electron motion gives rise to the surface current that produces the
London magnetic field given by Eq. (45).

It is also important to note that the outward radial direction motion of
electrons will cause the electron density spatial variation that will generate
nonzero gradient of the chemical potential. Thus, the back flow of electrons
occurs. As a consequence loop currents will be generated. Then, the final
surface current should be considered as a collection of loop currents as
depicted in Fig. 7. The bulk current is zero due to the cancelation.

Figure 7: A schematic snap shot of the velocity field of electrons that is
added to the velocity from the whole system rotation in Fig. 6b. It is assumed
that it is a collection of topologically protected loop currents produced by
$\nabla\chi$, and they move with the whole system rotation; other
contributions for electric current from electrons are canceled out by the
electric current contribution from the background positive ion cores in a
corse-grained average sence. The produced current in the bulk is zero due to
the cancelation of currents by loop currents. Only the surface current
remains.

The present theory indicates that the stability of $\nabla\chi$ changes the
behavior of the system. If $\nabla\chi$ is not stable, the velocity in Eq.
(22) will be zero in average, meaning

$\displaystyle\langle\oint_{C}{\bf A}\cdot d{\bf
r}\rangle-{\hbar\over{2e}}\langle\oint_{C}\nabla\chi\cdot d{\bf r}\rangle=0$
(55)

due to the fluctuation of the winding numbers for $\chi$, where
$\langle\cdots\rangle$ denotes time-average, and $C$ is any of spacial loops
in the system. In this situation, ${\bf A}$ also fluctuates, and its explicit
form is not well-defined. In this situation, ${\bf A}$ should be considered
merely as a mathematical tool for the calculation. However, if it is stable,
the velocity field in Eq. (22) becomes meaningful with well-defined ${\bf A}$
supplemented with the compensation of the gauge ambiguity by $\nabla\chi$. In
this case, persistent loop currents will flow [18, 19]. The electric current
generated by such a velocity field should be persistent, and the Meissner
effect will be realized. The current for this case is not the linear response
to external perturbations. Thus, it cannot be explained by the linear response
theory [36].

## References

  * [1] Lorentz H 1952 Electromagnetic phenomena in a system moving with any velocity less than that of light The principle of relativity: A collection of original memoirs on the special and general theory of relativity ed Perrett W and Jeffery G B (Dover)
  * [2] Feynman R P, Leighton R B and Sands M 1963 The Feynman Lectures on Physics vol II (Reading: Addison-Wesley) pp 18–1
  * [3] Maxwell J 1954 A Treatise on Electricity and Magnetism 3rd ed (Dover)
  * [4] Yaghjian A D 2020 Progress In Electromagnetics Research M 93 35–42
  * [5] Lorentz H A 1952 Michelson’s interference experiment The principle of relativity: A collection of original memoirs on the special and general theory of relativity ed Perrett W and Jeffery G B (Dover)
  * [6] Feynman R P, Leighton R B and Sands M 1963 The Feynman Lectures on Physics vol II (Reading: Addison-Wesley) pp 15–4
  * [7] Einstein A 1905 Ann. Phys. 322 891–921
  * [8] Einstein A 1951 On the electrodynamics of moving bodies The principle of relativity: A collection of original memoirs on the special and general theory of relativity ed Perrett W and Jeffery G B (Dover)
  * [9] Feynman R P, Leighton R B and Sands M 1963 The Feynman Lectures on Physics vol II (Reading: Addison-Wesley) pp 17–1
  * [10] Feynman R P, Leighton R B and Sands M 1963 The Feynman Lectures on Physics vol II (Reading: Addison-Wesley) pp 26–1
  * [11] Aharonov Y and Bohm D 1959 Phys. Rev. 115 167
  * [12] Feynman R P, Leighton R B and Sands M 1963 The Feynman Lectures on Physics vol II (Reading: Addison-Wesley) pp 15–5
  * [13] Tonomura A, Matsuda T, Suzuki R, Fukuhara A, Osakabe N, Umezaki H, Endo J, Shinagawa K, Sugita Y and Fujiwara H 1982 Phys. Rev. Lett. 48 1443
  * [14] Tonomura A, Osakabe N, Matsuda T, Kawasaki T, Endo J, Yano S and Yamada H 1986 Phys. Rev. Lett. 56(8) 792–795 URL https://link.aps.org/doi/10.1103/PhysRevLett.56.792
  * [15] Koizumi H 2017 J. Supercond. Nov. Magn. 30 3345–3349
  * [16] Berry M V 1984 Proc. Roy. Soc. London Ser. A 391 45
  * [17] Koizumi H 2022 Physics Letters A 450 128367 ISSN 0375-9601 URL https://www.sciencedirect.com/science/article/pii/S0375960122004492
  * [18] Koizumi H 2023 Journal of Physics A: Mathematical and Theoretical 56 455303
  * [19] Koizumi H 2023 Journal of Physics A: Mathematical and Theoretical 56 185301 URL https://dx.doi.org/10.1088/1751-8121/acc858
  * [20] Nambu Y 1960 Phys. Rev. 117 648
  * [21] Dirac P A M 1958 Proc. R. Soc. Lond. A246 326–332
  * [22] Dirac P A M 2001 Generalized Hamiltonian dynamics (Dover)
  * [23] Landauer R 1957 IBM J. Res. Dev. 1 223
  * [24] Büttiker M, Imry Y, Landauer R and Pinhas S 1985 Phys. Rev. B 31(10) 6207–6215 URL https://link.aps.org/doi/10.1103/PhysRevB.31.6207
  * [25] Datta S 1995 Electronic Transports in Mesoscopic Systems (Cambridge Univ. Press)
  * [26] Dirac P 1958 Principles of Quantum Mechanics (Oxford: Oxford Univ. Press) Section 22 4th ed
  * [27] O’Brien M C M 1964 Proc. Roy. Soc. Ser. A 281 323–339
  * [28] Dirac P 1931 Proc. Roy. Soc. London 133 60
  * [29] Feynman R P, Leighton R B and Sands M 1963 The Feynman Lectures on Physics vol II (Reading: Addison-Wesley) pp 27–5
  * [30] Abrikosov A A 1957 Sov. Phys. JETP 5 1174
  * [31] Maxwell J C 1861 Phil. Mag. Ser. 4 21 281–291
  * [32] Tinkham M 1996 Introduction to superconductivity 2nd ed (MacGraw-Hill)
  * [33] Hirsch J E 2003 Phys. Lett. A 315 474–479
  * [34] Hirsch J E 2013 Physica Scripta 89 015806 URL https://doi.org/10.1088/0031-8949/89/01/015806
  * [35] Koizumi H 2024 Physica Scripta 99 015952
  * [36] Kubo R 1957 Journal of the Physical Society of Japan 12 570–586 URL https://doi.org/10.1143/JPSJ.12.570