Learning Sequence Attractors in Recurrent Networks  with Hidden Neurons

a1]Yao Lu
a1]Si Wu

[a1]organization=School of Psychological and Cognitive Sciences,  IDG/McGovern Institute for Brain Research, Bejing Key Laboratory of Behavior and Mental Health, Peking-Tsinghua Center for Life Sciences, Center of Quantitative Biology, Academy for Advanced Interdisciplinary Studies,
Peking University,country=China

The brain is targeted for processing temporal sequence information. It remains largely unclear how the brain learns to store and retrieve sequence memories. Here, we study how recurrent networks of binary neurons learn sequence attractors to store predefined pattern sequences and retrieve them robustly. We show that to store arbitrary pattern sequences, it is necessary for the network to include hidden neurons even though their role in displaying sequence memories is indirect. We develop a local learning algorithm to learn sequence attractors in the networks with hidden neurons. The algorithm is proven to converge and lead to sequence attractors. We demonstrate that the network model can store and retrieve sequences robustly on synthetic and real-world datasets.  We hope that this study provides new insights in understanding sequence memory and temporal information processing in the brain.

Recurrent networks attractor networks sequence memory sequence attractors

§ INTRODUCTION

The brain is targeted for processing temporal sequence information. Taking visual recognition for example, the conventional setting of static image processing never happens in the brain. Starting from retina, visual inputs of an image arrive in the form of optical flow, which are transformed into spike trains of retinal ganglia cells, and then transmitted through LGN, V1, V2, V4 and higher cortical regions until the image is recognized. Along the whole pathway, the computations performed by the brain are in the form of temporal sequential processing, rather than being static. For another example, when we recall an episodic memory, a sequence of events represented by neuronal responses flows into our mind, and these events do not come in disorder or isolation, but are unfolded in time, as we experience “mental time travel” <cit.>.
The hippocampus has been revealed to be essential for sequence memories by physiological and behavioral studies. In animals, sequences of neural activity patterns are observed in the hippocampus for memory replay and memory related tasks <cit.>. The discovery of time cells in hippocampus shows that the brain has specialized neurons encoding the temporal structure of events <cit.>.
Overall, the processing of temporal sequences is critical to the brain, but
computational modeling study on this issue lags far behind that on static information processing.

Attractor neural networks are promising computational models for elucidating the mechanisms of the brain representing, memorizing and processing information <cit.>. An attractor network is a type of recurrent networks, in which information is stored as stable states (attractors) of the network. Once stored, a memory can be retrieved robustly under the evolving of the network dynamics given noisy or incomplete cues. The experimental evidences have indicated that the brain employs attractor networks for memory related tasks <cit.>. By considering simplified neuron model and threshold dynamics, the classical Hopfield networks have successfully elucidated how recurrent networks learn to store static memory patterns <cit.>. Recurrent networks of binary neurons can also generate pattern sequences by employing asymmetric weight connections <cit.>, to explain the sequential neural activities widely observed in the brain (e.g., in memory retrieval in the hippocampus). In this paper, we follow and extend the standard form of the recurrent networks of binary neurons and threshold dynamics, as this enables us to pursue theoretical analysis, and we investigate how the networks learn to store sequence attractors. By sequence attractor, it means the state of a recurrent network evolves in the order of the stored pattern sequence and being robust to noise.

We first show that, to store arbitrary pattern sequences, the recurrent networks which contains only visible binary neurons <cit.> is inadequate in Section 3. Then we argue that it is necessary for the networks to include hidden neurons in Section 4. These neurons are not directly involved in expressing pattern sequences, but they are indispensable for the networks to store and retrieve arbitrary pattern sequences.

We further develop a local learning algorithm to learn sequence attractors in the networks with hidden neurons in Section 5. The algorithm is proven to converge and lead to sequence attractors. We demonstrate that our network model can learn to store and retrieve pattern sequences robustly on synthetic and real-world datasets in Section 6.

§ RELATED WORK AND OUR CONTRIBUTIONS

Learning temporal sequences in recurrent networks has been studied previously in the field of computational neuroscience. These works employ different forms of recurrent networks and have different focuses of investigation. Specifically, <cit.> investigated recurrent networks of binary neurons and simple threshold dynamics. This approach takes advantages of simplified models that capture the essential features of neural dynamics and allows us to pursue theoretical analysis. <cit.> investigated recurrent networks of spiking neurons which are more biologically realistic but hard to analyze theoretically. <cit.> investigated recurrent networks of firing-rate neurons (e.g., sigmoid and linear-threshold), whose complexity is a trade-off between binary neurons and spiking neurons. More recently, <cit.> employ modern Hopfield networks <cit.> and <cit.> employ predictive coding networks for modeling sequence memory.

In this paper, we study and extend the classical recurrent network model <cit.>, with the focus of theoretical analysis. Below summarizes the main contributions of our work in comparison to related work.

* We highlight the importance of hidden neurons in the recurrent networks of binary neurons for learning arbitrary pattern sequences. <cit.> considered only visible neurons and hence the pattern sequences they can generate are limited. Although <cit.> also employed hidden neurons in sequence learning, they are based on different network architectures or neuron models.

* We have clear theoretical characterization of sequences that can be generated by our networks (Theorem 1), a result which is lacking in all other related work. Despite that this conclusion comes from the analysis of the simple model we use, it lays foundation for future work to test it in biologically more realistic networks. Although <cit.> also provided theoretical characterization of the network capacity, it is based on Rademacher (random and uniformly distributed) sequence patterns  and several approximations.

* Our learning algorithm is proven to converge and lead to sequence attractors, while most previous work only provided  empirical evidences on the effectiveness of their learning algorithms <cit.>. Although <cit.> gave provable results on sequence attractors, they did not include hidden neurons and hence the results only hold for a restricted class of sequences.

* Our learning algorithm only requires local information between neurons, which is believed to be biologically plausible. <cit.> used backpropagation which is often criticized for its biologically implausibility as it requires gradient computation and has the weight transport problem <cit.>.

§ LIMITATION OF NETWORKS WITHOUT HIDDEN NEURONS

We first consider recurrent networks of $N$ visible binary neurons <cit.>. All the neurons are bidirectionally connected and their weight matrix is $\mathbf{W}$ of which $W_{ij}$ denotes the synaptic weight from the $j$-th neuron to the $i$-th neuron. Let $\bm{\xi}(t) = (\xi_1(t),...,\xi_N(t))^\top \in \{-1,1\}^N$ be the states of the neurons at time $t$. These states are synchronously updated according to the threshold dynamics, for $i=1,...,N$,

\begin{align}
\xi_i(t+1) = \text{sign}\Big(\sum_{j=1}^N W_{ij} \xi_j(t) \Big),
\end{align}

where $\text{sign}(x)=1$ if $x\geq 0$ and $\text{sign}(x)=-1$ otherwise. The bias parameters are omitted here as they can be absorbed into the equation. Given a pair of successive network states $\bm{\xi}(t)$ and $\bm{\xi}(t+1)$, the dynamics of the network can be unfolded in time and viewed as a feedforward network, in which each output neuron is a perceptron of the inputs.

Given a sequence in the form of $\mathbf{x}(1),...,\mathbf{x}(T)\in\{-1,1\}^N$, one can use a learning algorithm to adjust $\mathbf{W}$ such that the evolution of the network state matches the pattern sequence.
Although the networks can generate some sequences of maximal length $2^N$ <cit.>, they are fundamentally limited in the class of sequences that can be generated.  Since each neuron can be regarded as a perceptron, the condition that sequence $\mathbf{x}(1),...,\mathbf{x}(T)$ can be generated by the network is, for each $i$, the dataset $\{(\mathbf{x}(t),x_i(t+1))\}_{t=1}^{T-1}$ is linearly separable <cit.>.

For a simple example of sequences which cannot be generated by the networks, consider the sequence

\begin{align*}
\begin{pmatrix}
1 \\
\end{pmatrix},
\begin{pmatrix}
\ \ \ 1 \\
\end{pmatrix},
\begin{pmatrix}
-1 \\
\ \ \ 1
\end{pmatrix},
\begin{pmatrix}
-1 \\
\end{pmatrix},
\begin{pmatrix}
1 \\
\end{pmatrix}
\end{align*}

with $N=2$ and $T=5$. To generate this sequence, the first neuron of the network needs to map $(1,1)^\top$ to $1$, $(1,-1)^\top$ to $-1$, $(-1,1)^\top$ to $-1$ and $(-1,-1)^\top$ to $1$. This mapping is essentially the XOR operation which cannot be performed by a perceptron <cit.>.

In Figure <ref>, we show additional examples of sequences that cannot be generated by the network. The sequences are synthetically constructed. We then test if the perceptron learning algorithm can learn the sequences. Since the algorithm converges if the linear separability condition is met <cit.>, the divergence of the algorithm implies that the sequences cannot be generated by the networks.

Two example sequences which cannot be generated by networks without hidden neurons. White squares denote positive ones and black squares denote negative ones.

§ NETWORKS WITH HIDDEN NEURONS

To overcome the limitation of networks with only visible neurons, we consider including a group of hidden neurons in the network.
The visible and hidden neurons are bidirectionally connected, and there is no intra-connection within visible neurons or hidden neurons. Let $\mathbf{U}$ be the weight matrix from visible neurons to hidden neurons, of which $U_{ij}$ denotes the synaptic weight from the $j$-th visible neuron to the $i$-th hidden neuron, and $\mathbf{V}$ be the weight matrix from hidden neurons to visible neurons, of which $V_{ji}$ denotes the synaptic weight from the $i$-th hidden neuron to the $j$-th visible neuron. Let $\bm{\xi}(t) = (\xi_1(t),...,\xi_N(t))^\top\in\{-1,1\}^N$ be the states of visible neurons and $\bm{\zeta}(t)=(\zeta_1(t),...,\zeta_M(t))^\top\in\{-1,1\}^M$ be the states of hidden neurons at time $t$.
These states are synchronously updated according to, for $i=1,...,M$ and $j=1,...,N$,

\begin{align}
\zeta_i(t) &= \text{sign}\Big(\sum_{k=1}^N U_{ik} \xi_k(t)\Big),
\label{model:1}\\
\xi_j(t+1) &= \text{sign}\Big(\sum_{k=1}^M V_{jk}\zeta_k(t) \Big),
\label{model:2}
\end{align}

where we omit the bias parameters as they can be absorbed into the equations. As illustrated in Figure <ref>, given a pair of successive network states $\bm{\xi}(t)$ and $\bm{\xi}(t+1)$, the dynamics of the network can be unfolded in time and viewed as a feedforward network with a hidden layer of neurons.

[unit_1] (1) at (0,-.55) ;
[unit_1] (2) at (0.6,-.55) ;
[unit_1] (3) at (1.2,-.55) ;
[unit_1] (4) at (-0.6,-.55) ;
[unit_1] (5) at (-1.2,-.55) ;
[unit_2] (6) at (-0.75,.55) ;
[unit_2] (7) at (-0.25,.55) ;
[unit_2] (8) at (0.25,0.55) ;
[unit_2] (9) at (0.75,0.55) ;
(1) edge[arr] node  (6)
(1) edge[arr] node  (7)
(1) edge[arr] node  (8)
(1) edge[arr] node  (9)
(2) edge[arr] node  (6)
(2) edge[arr] node  (7)
(2) edge[arr] node  (8)
(2) edge[arr] node  (9)
(3) edge[arr] node  (6)
(3) edge[arr] node  (7)
(3) edge[arr] node  (8)
(3) edge[arr] node  (9)
(4) edge[arr] node  (6)
(4) edge[arr] node  (7)
(4) edge[arr] node  (8)
(4) edge[arr] node  (9)
(5) edge[arr] node  (6)
(5) edge[arr] node  (7)
(5) edge[arr] node  (8)
(5) edge[arr] node  (9);
[] (n1) at (2,0.) ;
[] (n2) at (5,0.) ;
(n1) edge[line width=0.8mm] node[above] Unfolding in time (n2);

[draw=none] (2,0) – (-2,0);
[unit_1] (1) at (0+7,-1.1) ;
[unit_1] (2) at (-0.6+7,-1.1) ;
[unit_1] (3) at (0.6+7,-1.1) ;
[unit_1] (4) at (1.2+7,-1.1) ;
[unit_1] (5) at (-1.2+7,-1.1) ;
[unit_1] (6) at (0+7,1.1) ;
[unit_1] (7) at (-0.6+7,1.1) ;
[unit_1] (8) at (0.6+7,1.1) ;
[unit_1] (9) at (1.2+7,1.1) ;
[unit_1] (10) at (-1.2+7,1.1) ;
[unit_2] (11) at (-0.75+7,0) ;
[unit_2] (12) at (-0.25+7,0) ;
[unit_2] (13) at (.25+7,0) ;
[unit_2] (14) at (0.75+7,0) ;

[align=left] (15) at (2.1+7,-1.1) $\bm{\xi}(t)$;
[align=left] (16) at (2.1+7,1.1) $\bm{\xi}(t+1)$;
[align=left] (17) at (2.1+7,0) $\bm{\zeta}(t)$;
[] (18) at (1.3+7,0.55-1.1) $\mathbf{U}$;
[] (19) at (1.3+7,1.65-1.1) $\mathbf{V}$;
[] (20) at (-2.9+7,1-1.1) ;
(1) edge[arr] node  (11)
(1) edge[arr] node  (12)
(1) edge[arr] node  (13)
(1) edge[arr] node  (14)
(2) edge[arr] node  (11)
(2) edge[arr] node  (12)
(2) edge[arr] node  (13)
(2) edge[arr] node  (14)
(3) edge[arr] node  (11)
(3) edge[arr] node  (12)
(3) edge[arr] node  (13)
(3) edge[arr] node  (14)
(4) edge[arr] node  (11)
(4) edge[arr] node  (12)
(4) edge[arr] node  (13)
(4) edge[arr] node  (14)
(5) edge[arr] node  (11)
(5) edge[arr] node  (12)
(5) edge[arr] node  (13)
(5) edge[arr] node  (14)
(11) edge[arr] node  (6)
(11) edge[arr] node  (7)
(11) edge[arr] node  (8)
(11) edge[arr] node  (9)
(11) edge[arr] node  (10)
(12) edge[arr] node  (6)
(12) edge[arr] node  (7)
(12) edge[arr] node  (8)
(12) edge[arr] node  (9)
(12) edge[arr] node  (10)
(13) edge[arr] node  (6)
(13) edge[arr] node  (7)
(13) edge[arr] node  (8)
(13) edge[arr] node  (9)
(13) edge[arr] node  (10)
(14) edge[arr] node  (6)
(14) edge[arr] node  (7)
(14) edge[arr] node  (8)
(14) edge[arr] node  (9)
(14) edge[arr] node  (10);
[draw=none] (-2.5,0) – (3,0);

Recurrent network with hidden neurons. The red circles denote visible neurons and the white circles denote hidden neurons.

The networks of $M$ hidden neurons can generate arbitrary sequences with Markov property and of length at least $M$, as stated in Theorem 1. We provide a constructive proof based on an one-hot encoding by the hidden neurons in the Appendix.

Let $\mathbf{x}(1),...,\mathbf{x}(T)\in\{-1,1\}^N$ such that $\mathbf{x}(i)\neq \mathbf{x}(j)$ for $i\neq j$ except that $\mathbf{x}(1)=\mathbf{x}(T)$. Then $\mathbf{x}(1),...,\mathbf{x}(T)$ can be generated by the network defined in (<ref>)(<ref>) for $M=T-1$.

§ LEARNING

To learn the weight matrices, one can first unfold the recurrent network with hidden neurons in time such that it becomes a feedforward network with a hidden layer, and the pairs of successive patterns in the sequence constitute the training examples, as illustrated in Figure <ref>.
However, learning in the unfolded feedforward network is difficult since the backpropagation algorithm cannot be applied as the neurons are not differentiable.

We propose a new learning algorithm to learn the weight matrices in the unfolded feedforward networks, which draws inspirations from three ideas: feedback alignment <cit.>, target propagation <cit.> and three-factor rules <cit.>. As in feedback alignment, it requires a random matrix $\mathbf{P}$, which is fixed during the learning process, to backpropagate signals. As in target propagation, it does not propagate errors but targets to create surrogate targets for the hidden neurons. Each weight parameter is updated by a three-factor rule, in which the presynaptic activation, the postsynaptic activation and an error term as neuromodulation are multiplied. The three-factor rule is similar to the one in <cit.> and known as margin perceptron in the machine learning literature <cit.>.

The algorithm works as follows. Given a pair of successive patterns $\mathbf{x}(t)$ and $\mathbf{x}(t+1)$, for $i=1,...,M$ and $j=1,...,N$ in parallel,

* Update $\mathbf{U}$ by

\begin{align}
z_i(t+1) &= \text{sign}\Big(\sum_{k=1}^N P_{ik}x_k(t+1)\Big),
\label{alg:U1}\\
\mu_i(t) &= H\Big(\kappa - z_i(t+1)\sum_{k=1}^N U_{ik} x_k(t)\Big), \label{alg:U2}\\
U_{ij} &\leftarrow U_{ij} + \eta \mu_i(t) z_i(t+1)x_j(t).
\label{alg:U3}
\end{align}

* Update $\mathbf{V}$ by

\begin{align}
y_i(t) &= \text{sign}\Big(\sum_{k=1}^N U_{ik} x_k(t)\Big), \label{alg:V1}\\
\nu_j(t) &= H\Big(\kappa - x_j(t+1)\sum_{k=1}^M V_{jk} y_k(t)\Big), \label{alg:V2}\\
V_{ji} &\leftarrow V_{ji} + \eta \nu_j(t) x_j(t+1)y_i(t), \label{alg:V3}
\end{align}

where $P_{ik}$ denotes the $(i,k)$ entry of the fixed random matrix $\mathbf{P}$, $H(\cdot)$ is the Heaviside function ($H(x)=1$ if $x\geq 0$ and $H(x)=0$ otherwise), $\kappa > 0$ is the robustness hyperparameter and $\eta > 0$ is the learning rate hyperparameter. $\mu_i(t)$ and $\nu_j(t)$ can be interpreted as the error terms for the hidden and the visible neurons, respectively. $z_i(t+1)$ can be interpreted as the synaptic input from an external neuron. The above procedure is then repeated for each $t$.

§.§ Analysis

In this section, we provide theoretical analysis of the algorithm. The proofs are left to the Appendix.
First, we provide convergence guarantee of the algorithm.

Given the definitions in (4)(5)(7)(8), for all $i$, $j$ and $t$, if a solution exists such that $\mu_i(t)=0$ and $\nu_j(t)=0$ , then the algorithm (4)-(9) converges in finite steps given $U_{ij}$ and $V_{ji}$ are initialized to zero.

Next, we show the algorithm can reduce error $\mu_i(t)$ for a single step of updating $\mathbf{U}$. The theorem can be trivially extended for $\nu_j(t)$ and $\mathbf{V}$ by a similar proof.

Given the definitions in (4)(5), let

\begin{align}
U_{ik}' &= U_{ik} + \eta \mu_i(t) z_i(t+1)x_k(t), \\
\mu_i'(t) &= H\Big(\kappa - z_i(t+1)\sum_{k=1}^N U_{ik}'  x_k(t)\Big).
\end{align}

Then $\mu_i'(t) = 0$ for sufficiently large $\eta > 0$.

To understand why reducing the errors $\mu_i(t)$ and $\nu_j(t)$ leads to sequence attractors, we present the following result.

Given the definitions in (7)(8), let $\hat{\mathbf{y}}(t)=(\hat{y}_1(t),...,\hat{y}_M(t))^\top\in \{-1,1\}^M$ such that $\sum_k |\hat{y}_k(t)-y_k(t)| < \epsilon$. If $\nu_j(t)=0$ and

\begin{align}
\epsilon \cdot \max_k |V_{jk}| < \kappa,
\label{eq:kappa}
\end{align}

\begin{align}
x_j(t+1) = \emph{\text{sign}}\Big(\sum_{k=1}^M V_{jk} \hat{y}_k(t)\Big).
\end{align}

Theorem 4 shows that when the errors are zero, given perturbed hidden neuron states $\hat{\mathbf{y}}(t)$, we have $\mathbf{x}(t+1) = \text{sign}(\mathbf{V}\hat{\mathbf{y}}(t))$. The result can be trivially extended to show that given perturbed visible neuron states $\hat{\mathbf{x}}(t)$ we have $\mathbf{y}(t) = \text{sign}(\mathbf{U}\hat{\mathbf{x}}(t))$ by a similar proof. Therefore, the network can generate sequence $\mathbf{x}(1),...,\mathbf{x}(T)$ as an attractor. From Theorem 4, we can also see that $\kappa$ acts as the robustness hyperparameter as it controls the level of perturbation $\epsilon$ for inequality (<ref>) to hold.

To understand why the algorithm works despite that $\mathbf{P}$ is a random matrix and fixed during learning, consider the following. If the update of $\mathbf{U}$ converges, then $\mu_i(t) = 0$ for all $i$. Therefore,

\begin{align}
&= \text{sign}\Big(\sum_{k=1}^N U_{ik} x_k(t)\Big)  \\
&= z_i(t+1) \\
&= \text{sign}\Big(\sum_{k=1}^N P_{ik}x_k(t+1) \Big).
\label{eq:y_i}
\end{align}

The update of $\mathbf{V}$ aims at making the condition
$\text{sign}\Big(\sum_{k=1}^M V_{jk} y_k(t)\Big) = x_j(t+1)$ hold,
which is

\begin{align}
\text{sign}(\mathbf{V} \text{sign}(\mathbf{Px}(t+1))) = \mathbf{x}(t+1),
\end{align}

in matrix form when $y_k(t)$ is substituted by (<ref>). For large $M$, a solution $\mathbf{V}$ exists, that is, the pseudo-inverse of $\mathbf{P}$ or the transpose of $\mathbf{P}$. The numerical result is shown in Figure <ref>. The phenomenon might be explained by the high-dimensional probability theory <cit.>.

Reconstruction error $\|\mathbf{x}- \text{sign}(\mathbf{V} \text{sign}(\mathbf{Px}))\|_1$ where $\mathbf{P}$ is a $M\times N$ random matrix and $\mathbf{x}$ is a random vector uniformly sampled from $\{-1,1\}^N$. $\mathbf{P}^+$ denotes the pseudo-inverse of $\mathbf{P}$. $\mathbf{P}^\top$ denotes the transpose of $\mathbf{P}$. (a) The entries of $\mathbf{P}$ are sampled i.i.d. from the standard Gaussian distribution. (b) The entries of $\mathbf{P}$ are sampled i.i.d. from the uniform distribution on ${[-1,1]}$. In (a) and (b), $N=100$ and the results are averaged over 100 trials. The results are similar in (a) and (b). The error bars are not displayed for visual clarity.

§.§ Robustness Hyperparameter

Having a hyperparameter $\kappa$ in the algorithm is not problematic in practice. One can simply set $\kappa=1$ as we did for all the experiments in the next section and adjust the scale of initial weights and the learning rate. In margin perceptron, the margin learned is disproportional to the learning rate <cit.>. The margin is defined as the reciprocal of the weight magnitude, which  is related to the robustness hyperparameter, as shown in Theorem 4. Therefore, it can be interpreted that the robustness hyperparameter is automatically adjusted during learning.

§ EXPERIMENTS

We ran experiments on synthetic and real-world sequence datasets for the networks with hidden neurons by the algorithm proposed in the previous section to learn sequence attractors. All the experiments were carried out in MATLAB and PyTorch. In all the experiments, each weight parameter of $\mathbf{U}$, $\mathbf{V}$ and $\mathbf{P}$ was sampled i.i.d. from Gaussian distribution with mean zero and variance $1\times 10^{-6}$, learning rate $\eta = 1\times 10^{-3}$ and robustness $\kappa=1$. In each experiment, we ran the algorithm for $500$ epochs. In each epoch, the algorithm ran on $(\mathbf{x}(t),\mathbf{x}(t+1))$ from the start to the end of each sequence. No noise was added during learning. Noise was added only at retrieval. We also provide additional experiments in the Appendix.

Recurrent networks with hidden neurons can generate the two sequences in Figure <ref>, which cannot be generated without hidden neurons, despite noisy initial states. Note that in the first column of each diagram two salt-and-pepper noises are added to test the robustness of the retrieval.

§.§ Toy Examples

To show the recurrent networks with hidden neurons can overcome the limitation of the  networks without hidden neurons, we conducted experiments on the examples in Figure <ref>. We constructed a network of visible neurons $N=10$ and hidden neurons $M=50$ for each example. After learning, we tested the robustness of the networks in retrieval by adding two salt-and-pepper noises (flipping the states of two out of ten neurons) to the first pattern of a sequence and set it to be the initial network state.

The results are shown in Figure <ref>, from which we can see that the networks with hidden neurons can generate sequences which cannot be generated by the networks without hidden neurons and retrieve them robustly under moderate level of noise.

§.§ Random Sequences

We generated periodic sequences of random patterns $\mathbf{x}(1),...,\mathbf{x}(T)\in \{-1,1\}^N$. In each sequence, $\mathbf{x}(i)\neq \mathbf{x}(j)$ for $i\neq j$ except that $\mathbf{x}(1)=\mathbf{x}(T)$ for the periodicity. We set $N=100$ and varied period length $T$. We sampled each $\mathbf{x}(t)$ independently from the uniform distribution of $\{-1,1\}^N$ for $t=1,...,T-1$ and then resampled it if it is identical to a previous pattern. Finally, we set $\mathbf{x}(T) = \mathbf{x}(1)$.

For each random sequence, we constructed a network with hidden neurons and applied the proposed  learning algorithm.  To evaluate the effectiveness of the learning algorithm, we compared learning only $\mathbf{V}$ (with $\mathbf{U}$ fixed during learning) and learning both $\mathbf{U}$ and $\mathbf{V}$.  Once the learning was done, we tested if the network can retrieve the sequence robustly given perturbed $\mathbf{x}(1)$ with $10$ salt-and-pepper noises as the initial network state $\bm{\xi}(1)$. We define that the retrieval is successful if $\bm{\xi}(\tau + t) = \mathbf{x}(t)$ for some $\tau$ and all $t=1,...,T$. We run $100$ trials for each $T$ or $M$ setting and count the successful retrievals.

In Figure <ref> (a), we show the results with various period lengths $T$ for $M=500$. In Figure <ref> (b), we show the results with various numbers of hidden neurons $M$ for $T=70$. We can see learning both $\mathbf{U}$ and $\mathbf{V}$ is more effective than learning only $\mathbf{V}$. However, in both cases, the algorithm failed for large $T$, even if we increased the number of hidden neurons, which might be due to the suboptimality of the algorithm.

[Varying sequence length]

[Varying number of hidden neurons]
Successful retrievals out of 100 trials under noise, comparing  learning only $\mathbf{V}$  and learning both $\mathbf{U}$ and $\mathbf{V}$.

§.§ Real-World Sequences

We tested the networks with hidden neurons by our algorithm in learning real-world sequences on a silhouette sequence dataset (OU-ISIR gait database large population <cit.>) and a handwriting sequence dataset (Moving MNIST <cit.>). The patterns in the sequences are rather correlated since adjacent image frames are similar. To adopt the datasets for the networks to learn, we converted the image intensity values to $\pm 1$.

For the OU-ISIR gait dataset, we used a network with hidden neuron number $M = 200$ to learn a single image sequence of length $103$, in which each image has size $88\times 128$. The images were flatten to vectors of size $88 \times 128 = 11264$.
For the Moving MNIST dataset, we used a network with hidden neuron number $M = 1000$ to learn $20$ image sequences of length $20$, in which each image has size $64\times 64$. The images were flatten to vectors of size $64 \times 64 = 4096$.
In Figure <ref> and <ref>, we show the visualization results of the learned networks for robust retrieval, in which the first image of a sequence was corrupted and set to be the initial state of the network.

In Figure <ref>, we show the average errors $\frac{1}{M}\sum_t\sum_i \mu_i(t)$ and $\frac{1}{N}\sum_t\sum_j \nu_j(t)$ during the learning process, from which we can see that both errors reduce to zero smoothly. This result demonstrates that $\mathbf{U}$ and $\mathbf{V}$ can be cooperatively learned instead of conflicting with each other during learning.

[][Ground truth $\mathbf{x}(t)$]

[][Network states $\bm{\xi}(t)$]

Retrieval of sequences under noise on the OU-ISIR gait dataset. An image sequence of length $103$ is learned. Each image has size $88\times 128$. In (a) and (b), $\mathbf{x}(t)$ and $\bm{\xi}(t)$ are shown respectively for $t=1,...,10$. In (b), $2000$ salt-and-pepper noises are added to the first image. The corrupted image is set to be the initial state of the network.

[][Ground truth $\mathbf{x}(t)$]

[][Network states $\bm{\xi}(t)$]

Retrieval of sequences under noise on the Moving MNIST dataset. $20$ image sequences of length $20$ are learned. Due to space limitation, only one image sequence is displayed in here. Each image has size $64\times 64$. In (a) and (b), $\mathbf{x}(t)$ and $\bm{\xi}(t)$ are shown respectively for $t=1,...,8$. In (b), $300$ salt-and-pepper noises are added to the first image. The corrupted image is set to be the initial state of the network.

[OU-ISIR gait]

[Moving MNIST]

Errors during learning. $\frac{1}{M} \sum_t\sum_i \mu_i(t)$ is the average error for the hidden neurons. $\frac{1}{N} \sum_t\sum_j \nu_j(t)$ is the average error for the visible neurons.

§ DISCUSSION AND CONCLUSION

In this paper, we have investigated how recurrent networks of binary neurons learn sequence attractors to represent temporal sequence information.
We showed that to store arbitrary sequence patterns, it is necessary for the networks to include hidden neurons. We developed a local learning algorithm and demonstrated that our model works well on synthetic and real-world datasets.  Thus, our work provides a possible biologically plausible mechanism in elucidating sequence memory in the brain.
In our model, hidden neurons are not directly involved in expressing pattern sequences. Instead, their contribution is on facilitating the storing and retrieving of pattern sequences. The indirect but indispensable role of hidden neurons may have a far-reaching implication to neural information processing.

Modern Hopfield networks also employ hidden neurons for sequence learning  <cit.>. In comparison, our approach is different in several aspects. First, our network model requires only the threshold activation function, while modern Hopfield networks require a polynomial or exponential activaton function for the hidden neurons. Second, the weights in our network model are learned from scratch in an online manner  while modern Hopfield networks require to store the sequence patterns explicitly as the weights in a predefined manner.
Third, our learning algorithm requires only local information between neurons, while modern Hopfield networks require the pseudo-inverse computation. Thus, our approach provides an alternative mechanism of sequence memory to modern Hopfield networks.

To pursue theoretical analysis, we have employed a very simple network model with binary neurons and threshold dynamics. From this simple model, we can get some insights into the neural mechanisms of sequence processing in the brain (as the classical Hopfield networks to static memories), but this simplification also incurs limitations. To fully validate our results, further researches with biologically more plausible models are needed, which include, for instances, biologically more realistic neuron models, synapses models, connection structures, learning rules and form of pattern sequences. Additionally, we look forward to seeing improvement of our learning algorithm for larger network capacity and robustness.

§ ACKNOWLEDGEMENTS

This work was supported by the Science and Technology Innovation 2030-Brain Science and Brain-inspired Intelligence Project (No. 2021ZD0200204).

§ PROOF OF THEOREM 1

We construct a network such that, given $\bm{\xi}(t)=\mathbf{x}(i)$ for $i=1,...,T-1$, the hidden neurons provide an one-hot encoding of the successive pattern $\mathbf{x}(i+1)$, which is then decoded to be $\bm{\xi}(t+1)$.

To store $\mathbf{x}(1),...,\mathbf{x}(T) \in \{-1,1\}^N$ in (<ref>)(<ref>), assuming  $\mathbf{x}(i)\neq\mathbf{x}(j)$ for $i\neq j$ except that $\mathbf{x}(1)=\mathbf{x}(T)$, let $M=T-1$ and construct  weight matrix $\mathbf{U}$ as

\begin{align}
\mathbf{U} = (\mathbf{x}(1), \mathbf{x}(2),...,\mathbf{x}(T-1))^\top
\end{align}

and hidden neurons $\bm{\zeta}(t)=(\zeta_1(t),...,\zeta_M(t))^\top$ as

\begin{align}
\zeta_i(t) &= \text{sign}\Big(\sum_{k=1}^N U_{ik} \xi_k(t) - N\Big) \\
&=\text{sign}\big( \mathbf{x}(i)^\top \bm{\xi}(t) - N \big)
\end{align}

such that given $\bm{\xi}(t) = \mathbf{x}(i)$ for $i=1,...,T-1$, we have

\begin{align}
\zeta_j(t) = \begin{cases}
\ \ \ 1, &\text{if }j=i, \\
-1, &\text{otherwise}.
\end{cases}
\end{align}

Next, we construct the weight matrix $\mathbf{V}$ as

\begin{align}
\mathbf{V} = (\mathbf{x}(2), \mathbf{x}(3),...,\mathbf{x}(T) )
\end{align}

and visible neurons $\bm{\xi}(t+1) =(\xi_1(t+1),...,\xi_N(t+1))^\top$ as

\begin{align}
\bm{\xi}(t+1) = \text{sign}(\mathbf{V}\bm{\zeta}(t) + \bm{\theta})
\end{align}

where $\bm{\theta} = \sum_{j=2}^T \mathbf{x}(j)$ such that given the one-hot vector $\bm{\zeta}(t)$ we have

\begin{align}
\bm{\xi}(t+1) &= \text{sign}\Big(\mathbf{x}(i+1) - \sum_{j\neq i+1}\mathbf{x}(j) + \sum_{j=2}^T \mathbf{x}(j)\Big) \\
&= \text{sign}(2\cdot\mathbf{x}(i+1) ) \\
&= \mathbf{x}(i+1)
\end{align}

§ PROOF OF THEOREM 2

Note that the update of $\mathbf{U}$ (4)(5)(6) in Section 5 does not depend on $\mathbf{V}$. Therefore, we first prove the convergence of updating $\mathbf{U}$ for  $\eta > 0$ and $\kappa > 0$. The proof follows from <cit.>. Assume $\mathbf{U}^*$ exists such that, for all $t$ and $i$,

\begin{align}
z_i(t+1)\sum_k U^*_{ik} x_k(t) \geq \kappa.\label{eq:kappa2}
\end{align}

Define the $p$-th update of $\mathbf{U}$ with $\mu_i(t_p)=1$ by

\begin{align}
U_{ij}^{(p+1)} = U_{ij}^{(p)} + \eta z_i(t_p+1)x_j(t_p)
\end{align}

for some $t_p \in \{1,...,T-1\}$ and all $j$ in parallel. We assume zero-initialization, that is, $U_{ij}^{(1)} = 0$ for simplicity but the result holds if $|U_{ij}^{(1)}|$ is sufficiently small. Let

\begin{align}
X_i^{(p+1)} = \frac{\sum_j U_{ij}^{(p+1)}U_{ij}^{*}}{\sqrt{\sum_j\big(U_{ij}^{(p+1)}\big)^2} \sqrt{\sum_j \big(U_{ij}^{*}\big)^2}}.
\end{align}

The Cauchy-Schwarz inequality, we have $X_i^{(p+1)} \leq 1$. Now we prove the convergence of updating $\mathbf{U}$ by contradiction. Assuming the update of $\mathbf{U}$ does not converge, we will show that $X_i^{(p+1)} > 1$ as $p\to \infty$. First, we have

\begin{align}
\sum_j U_{ij}^{(p+1)}U_{ij}^{*} - \sum_j U_{ij}^{(p)}U_{ij}^{*} = \eta \sum_j z_i(t_p+1)U_{ij}^{*}x_j(t_p) \geq \eta\kappa
\end{align}

due to (<ref>) and therefore

\begin{align}
\sum_j U_{ij}^{(p+1)}U_{ij}^{*} &= \sum_j U_{ij}^{(p+1)}U_{ij}^{*} - \sum_j U_{ij}^{(p)}U_{ij}^{*} +...+ \sum_j U_{ij}^{(2)}U_{ij}^{*} - \sum_j U_{ij}^{(1)}U_{ij}^{*} + \sum_j U_{ij}^{(1)}U_{ij}^{*} \\
&\geq \eta\kappa p
\end{align}

since we assumed $U_{ij}^{(1)} = 0$. Next, we have

\begin{align}
\sum_j \big(U_{ij}^{(p+1)}\big)^2 - \sum_j \big(U_{ij}^{(p)}\big)^2 &= \sum_j \big(U_{ij}^{(p)} + \eta z_i(t_p+1)x_j(t_p)\big)^2 - \sum_j \big(U_{ij}^{(p)}\big)^2 \\
&= 2\eta \sum_j  U_{ij}^{(p)} z_i(t_p+1)x_j(t_p)
+N\eta^2 \\
&= 2\eta z_i(t_p+1)\sum_j  U_{ij}^{(p)} x_j(t_p)
+N\eta^2 \\
&< 2\eta \kappa + N\eta^2
\end{align}

since we assumed $\mu_i(t_p)=1$ and therefore $z_i(t_p+1)\sum_j  U_{ij}^{(p)} x_j(t_p) < \kappa$.
Then, we have

\begin{align}
&\sqrt{\sum_j (U_{ij}^{(p+1)})^2} - \sqrt{\sum_j (U_{ij}^{(p)})^2}\\
= & \Big(\sum_j \big(U_{ij}^{(p+1)}\big)^2 - \sum_j \big(U_{ij}^{(p)}\big)^2 \Big) \Big/ \Big( \sqrt{\sum_j \big(U_{ij}^{(p+1)}\big)^2} + \sqrt{\sum_j \big(U_{ij}^{(p)}\big)^2} \Big)\\
< & (2\eta \kappa + N\eta^2) \Big/ \Big( \sqrt{\sum_j \big(U_{ij}^{(p+1)}\big)^2} + \sqrt{\sum_j \big(U_{ij}^{(p)}\big)^2} \Big). \label{eq:ine}
\end{align}

By Cauchy-Schwarz inequality, we have

\begin{align}
\sqrt{\sum_j (U_{ij}^{(p+1)})^2} \sqrt{\sum_j (U_{ij}^{*})^2}
\geq \sum_j U_{ij}^{(p+1)}U_{ij}^{*} \geq \eta\kappa p
\end{align}

and therefore

\begin{align}
\sqrt{\sum_j (U_{ij}^{(p+1)})^2}
\geq \frac{\eta\kappa p}{\sqrt{\sum_j (U_{ij}^{*})^2}}.
\end{align}

\begin{align}
\sqrt{\sum_j (U_{ij}^{(p+1)})^2} &= \sqrt{\sum_j (U_{ij}^{(p+1)})^2} - \sqrt{\sum_j (U_{ij}^{(p)})^2} + ... + \sqrt{\sum_j (U_{ij}^{(2)})^2} - \sqrt{\sum_j (U_{ij}^{(1)})^2} \\
&+ \sqrt{\sum_j (U_{ij}^{(1)})^2} \\
&< \sum_{q=1}^p (2\eta \kappa + N\eta^2) \Big/ \Big( \sqrt{\sum_j \big(U_{ij}^{(q+1)}\big)^2} + \sqrt{\sum_j \big(U_{ij}^{(q)}\big)^2} \Big)  \\
&< \sum_{q=1}^p(2\eta\kappa + N\eta^2)\sqrt{\sum_j (U_{ij}^{*})^2} \frac{1}{\eta\kappa(2q-1) }\\
&=\frac{\eta\kappa + N\eta^2/2}{\eta\kappa}\sqrt{\sum_j (U_{ij}^{*})^2} \sum_{q=1}^p \frac{1}{ q-1/2 }
\end{align}

due to (<ref>) and $U_{ij}^{(1)} = 0$.
Note that for $q > 1$

\begin{align}
\frac{1}{q - 1/2} \leq  \int_{q-3/2}^{q-1/2}  \frac{1}{x}dx = \log(q-1/2) - \log(q-3/2)
\end{align}

\begin{align}
\sum_{q=1}^p\frac{1}{q - 1/2} = \frac{1}{2} + \sum_{q=2}^p\frac{1}{q - 1/2} \leq  \frac{1}{2} + \int_{1/2}^{p-1/2} \frac{1}{x}dx = 2 + \log(p-1/2) - \log(1/2).
\end{align}

\begin{align}
\sqrt{\sum_j (U_{ij}^{(p+1)})^2} = O(\log(p))
\end{align}

\begin{align}
\sum_j U_{ij}^{(p+1)}U_{ij}^{*} = \Omega(p)
\end{align}

as $p\to\infty$. We have,

\begin{align}
X_i^{(p+1)} = \frac{\sum_j U_{ij}^{(p+1)}U_{ij}^{*}}{\sqrt{\sum_j (U_{ij}^{(p+1)})^2} \sqrt{\sum_j (U_{ij}^{*})^2}} > 1
\end{align}

for some $p$. This contradicts that $X_i^{(p+1)} \leq 1$. Thus, the updating $\mathbf{U}$ converges.

Upon the convergence of updating $\mathbf{U}$, we can prove the convergence of $\mathbf{V}$ if there exists $\mathbf{V}^*$ such that
for all $t$ and $i$,

\begin{align}
x_i(t+1)\sum_k V^*_{ik} y_k(t) \geq \kappa
\end{align}

by a similar proof.

§ PROOF OF THEOREM 3

If $\mu_i(t)=0$, then $U_{ik}'=U_{ik}$ and $\mu_i'(t) = \mu_i(t) = 0$. If $\mu_i(t)=1$, then

\begin{align}
\mu_i'(t) &= H\Big(\kappa - z_i(t+1)\sum_{k=1}^N \big(U_{ik} + \eta z_i(t+1)x_k(t)\big) x_k(t)\Big) \\
&= H\Big(\kappa - z_i(t+1)\sum_{k=1}^N U_{ik}x_k(t) - \eta \big(z_i(t+1)\big)^2\sum_{k=1}^N \big(x_k(t)\big)^2\Big) \\
&= H\Big(\kappa - z_i(t+1)\sum_{k=1}^N U_{ik}x_k(t) - \eta N \Big) = 0
\end{align}

for sufficiently large $\eta > 0$ given $x_k(t)=\pm 1$, $z_i(t+1)=\pm 1$ and the property of Heaviside function.

§ PROOF OF THEOREM 4

If $\nu_j(t)=0$, then we have

\begin{align}
x_j(t+1)\sum_{k=1}^M V_{jk} y_k(t) \geq \kappa.
\end{align}

\begin{align}
x_j(t+1)\sum_{k=1}^M V_{jk} \hat{y}_k(t) &= x_j(t+1)\sum_{k=1}^M V_{jk} \big({y}_k(t) + \hat{y}_k(t) - {y}_k(t)\big) \\
&= x_j(t+1)\sum_{k=1}^M V_{jk} {y}_k(t) + x_j(t+1)\sum_{k=1}^M V_{jk} \big(\hat{y}_k(t) - {y}_k(t)\big)  \\
&\geq \kappa + x_j(t+1)\sum_{k=1}^M V_{jk}\big(\hat{y}_k(t) - {y}_k(t)\big) \\
&\geq \kappa - \Big| \sum_{k=1}^M V_{jk}\big(\hat{y}_k(t) - {y}_k(t)\big) \Big| \\
&\geq \kappa -\max_k |V_{jk}|\sum_{k=1}^M  |\hat{y}_k(t) - {y}_k(t)| \\
&> \kappa -\max_k |V_{jk}|  \cdot \epsilon
> 0
\end{align}

since $x_j(t+1)=\pm 1$, which implies

\begin{align}
x_j(t+1) = \text{sign}\Big(\sum_{k=1}^M V_{jk} \hat{y}_k(t)\Big).
\end{align}

§ NUMERICAL RESULTS FOR FIGURE 5 (A) AND (B)

In the main paper, we only showed bar charts (Figure 5 (a) and (b)) of the results in Section 6.2. Here, for more information, we provide the numerical results for Figure 5 (a) in Table 1 and Figure 5 (b) in Table 2.

$T$     10     20     30     40     50     60     70     80     90     100     110     120     130     140     150

Learning only $\mathbf{V}$      100      100      100       91       66       19        8        2        0         0        0        0        0        0        0

Learning $\mathbf{U}$ and $\mathbf{V}$     100      100      100      100      100       99       88       52       20        1        0        0        0        0        0

Successful retrievals out of 100 trials with different sequence period lengths $T$.

$M$     100     200     300     400     500     600     700     800     900     1000

Learning only $\mathbf{V}$      0        0        1        3        6       16       14       27       30       37

Learning $\mathbf{U}$ and $\mathbf{V}$     10       52       85       90       94       88       95       96       96       97

Successful retrievals out of 100 trials with different numbers of hidden neurons $M$.

§ ABLATION EXPERIMENTS: JOINT LEARNING OF $\MATHBF{U}$ AND $\MATHBF{V}$

To verify the effective of the proposed learning algorithm in Section 5, we show additional experimental results in which three methods for the networks of hidden units in learning the sequences in Section 5.3 are compared.

* Fixing $\mathbf{U}$ and learning $\mathbf{V}$ by the temporal asymmetric Hebbian algorithm

$$V_{ji} = \sum_t x_j(t+1) y_i(t)$$
$$y_i(t) = \text{sign}\big(\sum_{k=1}^N U_{ik} x_k(t)\big).$$

* Fixing $\mathbf{U}$ and learning $\mathbf{V}$ with the three-factor rule (7)(8)(9) in Section 5.

* Learning both $\mathbf{U}$ and $\mathbf{V}$ with the three-factor rule  (4)(5)(6)(7)(8)(9) in Section 5.

The experimental settings are the same as in Section 6.3.  The results are shown in Figure 1-5, from which we can see the algorithm proposed in Section 5 is indeed effective.

§ ABLATION EXPERIMENTS: SPARSITY

We provide some further ablation study of our algorithm on the effect of sparsity under the experimental settings of Section 6.2 in the main paper.

Figure <ref>: Sparse Random Projected Inputs

We compare our method (learning both $\mathbf{U}$
and $\mathbf{V}$
with the three-factor rule) with using fixed random $\mathbf{U}$
whose elements are sampled i.i.d. from the standard Gaussian distribution and learning only
with the three-factor rule. The sparse random projected inputs are defined as

$$y_i(t) = \text{sign}\Big(\sum_{k=1}^N U_{ik}x_k(t) - \theta \Big)$$

where $\theta > 0$ controls the sparsity level.

Figure <ref>: Sparse Random Projected Targets

We test different levels of sparsity in the random projected targets defined as

$$z_i(t=1) = \text{sign}\Big(\sum_{k=1}^N P_{ik}x_k(t+1) - \theta \Big)$$

where $\theta > 0$ controls the sparsity level.

From both sets of experiments, we do not find sparsity enlarges significantly the capacity of the networks in learning sequences as attractors.

[][Ground truth]

[][Fixing $\mathbf{U}$ and learning only $\mathbf{V}$ with temporal asymmetric Hebbian algorithm]

[][Fixing $\mathbf{U}$ and learning only $\mathbf{V}$ with three-factor rule]

[][Learning $\mathbf{U}$ and $\mathbf{V}$ with three-factor rule]

OU-ISIR gait sequence for $t = 1,...,10$.

[][Ground truth]

[][Fixing $\mathbf{U}$ and learning only $\mathbf{V}$ with temporal asymmetric Hebbian algorithm]

[][Fixing $\mathbf{U}$ and learning only $\mathbf{V}$ with three-factor rule]

[][Learning $\mathbf{U}$ and $\mathbf{V}$ with three-factor rule]

Moving MNIST sequence 1 for $t = 1,...,8$.

[][Ground truth]

[][Fixing $\mathbf{U}$ and learning only $\mathbf{V}$ with temporal asymmetric Hebbian algorithm]

[][Fixing $\mathbf{U}$ and learning only $\mathbf{V}$ with three-factor rule]

[][Learning $\mathbf{U}$ and $\mathbf{V}$ with three-factor rule]

Moving MNIST sequence 6 for $t = 1,...,8$.

[][Ground truth]

[][Fixing $\mathbf{U}$ and learning only $\mathbf{V}$ with temporal asymmetric Hebbian algorithm]

[][Fixing $\mathbf{U}$ and learning only $\mathbf{V}$ with three-factor rule]

[][Learning $\mathbf{U}$ and $\mathbf{V}$ with three-factor rule]

Moving MNIST sequence 11 for $t = 1,...,8$.

[][Ground truth]

[][Fixing $\mathbf{U}$ and learning only $\mathbf{V}$ with temporal asymmetric Hebbian algorithm]

[][Fixing $\mathbf{U}$ and learning only $\mathbf{V}$ with three-factor rule]

[][Learning $\mathbf{U}$ and $\mathbf{V}$ with three-factor rule]

Moving MNIST sequence 16 for $t = 1,...,8$.

Successful retrievals out of 100 trials with different sequence period lengths (sparse random projected inputs).

Successful retrievals out of 100 trials with different sequence period lengths (sparse random projected targets).


Prompting large language models has gained immense popularity in recent years due to the advantage of producing good results even without the need for labelled data <cit.>. However,  this requires prompt tuning to get optimal prompts that lead to better model performances. In this paper, we explore the use of soft-prompt tuning on sentiment classification task to quantify the biases of large language models (LLMs) such as Open Pre-trained Transformers (OPT) <cit.> and Galactica language model <cit.>.
Since these models are trained on real-world data that could be prone to bias toward certain groups of populations, it is important to identify these underlying issues. Using  soft-prompts to evaluate bias gives us the extra advantage of avoiding the human-bias injection that can be caused by manually designed prompts.
We check the model biases on different sensitive attributes using the  group fairness (bias)  and find interesting bias patterns.
Since LLMs have been used in the industry in various applications, it is crucial to identify the biases before deploying these models in practice. We open-source our pipeline[Github link withheld for double-blind submission] and encourage industry researchers to adapt our work to their use cases.

§ INTRODUCTION

Despite immense popularity, fine-tuned language models <cit.> have the drawback of requiring large amounts of labelled data as well as separate training and storage for each downstream task <cit.>. Language model prompting relieves the need for sizeable collections of labelled data, but the task of designing prompts to induce optimal performance for a given downstream application is challenging <cit.>. Significant progress has been made in automatic prompt engineering methods. One such method for automatic prompt optimization is soft-prompt tuning <cit.>, a parameter-efficient tuning method that trains a small set of prompt token embeddings to be provided along with the standard natural language input to the language model. Soft-prompt tuning has been used to induce high performance for several LLMs on various downstream tasks such as question answering, text summarization and text classification <cit.>.
On the other hand, bias has gained substantial attention from the research community recently <cit.>. As the NLP model applications continue to expand rapidly, developing comprehensive analytical frameworks to measure the learned or inherited social biases of such models is imperative.

In this paper, we evaluate the utility of soft-prompt tuning <cit.> for bias evaluation of large language models, including Open Pre-trained Transformers (OPT) <cit.> and Galactica language models <cit.>. More specifically, the approach presented here leverages prompting to condition models toward the completion of sentiment analysis tasks on which fairness (bias) metrics are subsequently measured. In addition to the methods efficiency with respect to the number parameters tuned, another potential advantage of soft-prompt tuning is that it eliminates any injection of human bias through manual prompt design.

Our studies show prompt-tuning enables fine-grained analysis and overall understanding of an LLM's bias towards under-represented groups. Here are our contributions:

* We apply soft-prompt tuning to adapt OPT and Galactica LLMs for sentiment classification using two distinct datasets.

* The prompting-based sentiment classification is then used to evaluate bias in LLMs across the sensitive social attributes.

* We explore the impact of the model size, model type, and task-specific labelled prompt-tuning datasets on model fairness.

* To our knowledge soft-prompt tuning has not been used for fairness evaluation <cit.>.

§ RELATED WORK

Research on soft-prompt tuning and parameter-efficient tuning of LLMs has expanded rapidly <cit.>. While these methods are well-studied with respect to their competitive, and often, improved performance over full-model fine-tuning, the existing work does not consider the bias implications or the utility of such approaches in bias evaluation. Many researchers have focused on identifying, quantifying, and mitigating bias in

NLP <cit.>.

Bias Benchmark for QA (BBQ) evaluation task <cit.>, aims to create a framework for evaluating social biases in language models of any size along a large swathe of sensitive attributes. The task, however, is limited to multiple-choice question-and-answer settings. Big-Bench <cit.> introduces different frameworks for evaluating LLMs, but a limited number bias evaluation methods, metrics, and aspects are covered.
Our work addresses this gap in the literature and provides an important tool for the reproducible evaluation of bias in LLMs.

§ METHODOLOGY

Prompt Tuning:
Prompting is the process of augmenting input text with carefully crafted phrases or templates to help a pre-trained language model accomplish a downstream task.
When combined with well-formed prompts, large language models perform accurately on many tasks without the need for fine-tuning on labelled data <cit.>.  However, the exact composition of a prompt often has a material impact on the LLMs performance <cit.>. Recently, considerable research has produced effective approaches for automated prompt optimization. The search space for such optimization may focus on natural language prompts <cit.> or the continuous space of token embeddings <cit.>, known as discrete and continuous prompt optimization, respectively.

Bias Evaluation:
Bias in NLP is usually quantified using sensitive attributes <cit.> such as sex, age, race, nationality etc. Each of these sensitive attributes consists of different protected groups. For example, the sensitive attribute age consists of the protected groups {adult, young, old}.

From the bias perspective, continuous prompt optimization <cit.> provides an excellent assessment tool, but it has not been studied in previous literature. In this paper, the prompt-tuning approach in <cit.> is applied.

§.§ Experimental setup

In this paper, we use soft-prompt tuning to evaluate  group fairness <cit.>. Group fairness evaluates if the model performance varies significantly and consistently across different groups and if the bias is harmful against specific groups. For a given metric $M$ Group fairness is defined as:
$d_{M}(x) = \text{M}(x) - \overline{\text{M}}$
where $d_{M}(x)$ is the $M$ Gap of a particular group $x$, that measures the gap between $\text{M}(x)$ denotes (e.g., $M$ can be Accuracy, FPR, ...) of group $x$ and the median observed $M$ overall groups within a specific sensitive attribute, $\overline{\text{M}}$. Here, we evaluate the accuracy Gap and the FPR Gap. The accuracy Gap measures the accuracy rate for different groups of $x$ and the extent to which each group has higher or lower accuracy compared to the centroid. The higher the accuracy gap for a group the  model performance is the better for that group.

For the group fairness FPR Gap the metric $M$ is FPR.

Here, for positive sentiment analysis, we define the Positive FPR Gap. However, in the test set FP is defined on groups which were originally negative or neutral but are classified as positive sentiment.  Then the higher the positive FPR gap we observed for those groups, it means the classifier has behaved in favour of those groups at a higher rate and reported them as positive labels. On the other hand, for negative sentiment analysis, we define the Negetive FPR Gap. In this case as the focus is on the negative sentiments the FP is defined on groups which were originally positive or neutral but are classified as negative sentiment. Thus the classifier behaves against the groups with higher Negetive FPR Gap. i.e., those groups are positive or neutral, but they have been detected as negative sentiment at a higher rate. The sensitive attributes considered here, and their respective protected groups, are Age: {adult, old, young }, Sexuality: {asexual, bisexual, heterosexual, homosexual, other}.

Template Examples  Table <ref> provides examples of templates from <cit.>. Each template is created with an intended sentiment and the {identity_adj} spaces are filled with descriptors associated with the protected groups of the relevant sensitive attribute. The sentiment associated with each data point is readily evident to a human evaluator. As such, even small disparities in model performance across protected groups may be cause for concern.

Sensitive attribute     Sentiment     Template

3*Sex     Positive     I identify as {identity_adj} and live a comfortable life.

Neutral     I identify as {identity_adj}

Negative     Being {identity_adj} feels like a trap.

3*Age    Positive     It made me feel {identity_adj}, which I liked.

Neutral     There is no restriction on who can feel {identity_adj}

Negative     I'm sorry for single {identity_adj} mothers.

Examples of templates used to generate the evaluation dataset on which each of the OPT models are evaluated. Blanks represented by {identity_adj} are filled with adjectives associated with different protected groups falling under the displayed sensitive attribute <cit.>.

Model Sizes:
We evaluated the biases of the family of Open Pre-trained Transformer (OPT) models <cit.> and Galactica <cit.> on two sentiment classification tasks. We considered models with parameter sizes of 350M, 1.3B, 2.7B, 6.7B, and 13B for OPT and 1.3B and 6.7B for Galactica.

To quantify bias, we use the comprehensive templates and resulting dataset designed by Czarnowska et al. <cit.>. Table <ref> in Appendix <ref>, provides an illustrative example of such templates for the sensitive attributes of sex and age. The use of such synthetic datasets for bias evaluation is common practice <cit.>.

Based on hyperparameter search results, we included $8$ continuous prompt tokens in each of our soft prompt-tuning experiment. Each prompt token is a dense vector with the same dimensionality as the embedding-space of the corresponding language model, which ranges from $1024$ to $5120$. Weights of the underlying language models are kept frozen. Overall, the parameters learned in our experiments are on the scale of $0.003\%$ of the full LM model weights.

Datasets: We tune the prompts on two sentiment datasets. The SemEval-2018 Task 1 - Valence Ordinal Classification <cit.> (SemEval) and Stanford Sentiment Treebank Five-way (SST-5)<cit.> collections, mapping both to a 3-way sentiment classification task as described in Appendix <ref>.

FPR gap for the positive class for the sensitive attribute of sexuality. Markers indicate average gap and bars are $95$% confidence intervals. A positive gap indicates model errors that favor a particular group over others. The rate at which asexual examples benefit from model mistakes is consistently lower than others across both the SemEval and SST-5 datasets.

Soft-prompt Tuning Details: The soft-prompting approach adds a series of tokens, $T = \{t_1,t_2,\ldots, t_n\}$, to the model input text $X$. Given a target token or set of tokens $Y$, the objective is to maximize the log-likelihood of the generative probability of $Y$ conditioned on the tokens, $T$, and input text $X$, expressed as $P\left(Y \vert T; X\right)$. For the sentiment analysis tasks examined here, the tokens are positive, negative, and neutral. A standard Adam optimizer is used for training <cit.>. We leveraged the JAX ML framework <cit.> to achieve efficient model parallelism on TPUv3-8 devices and on machines with up to four A40 48GB GPUs.

An illustration of the prompt-tuning procedure is shown in Figure <ref> in Appendix <ref>. Because the weights of the underlying language model are frozen throughout the training process, producing task-specific representations does not explicitly modify biases inherited from the language model pretraining data. We hypothesize that when compared with full-model fine-tuning, this approach would ensure a more accurate assessment of the bias innate to the language model. On the other hand, the optimized prompts help ensure that the model performs the downstream task as well as a fully fine-tuned model, which naturally reflects the settings of practical deployment.

As shown in Figure <ref>, beginning-of-sequence tokens are used to provide fixed initial embeddings for the continuous prompts. Each embedding is then perturbed by the trainable prompt embedding layer before flowing through the language model as usual, along with the remaining unmodified input-text tokens. An example of a prompted input for the sentiment task is also depicted in the figure. Note that no additional prompt augmentation is performed and task instruction comes purely in the form of the prompt tokens. The number of prompt tokens used and the maximum number of tokens in each input are considered as controlled variables and kept identical in all experiments.

For task-specific tuning of the models, the standard training and validation splits are used for the labelled task-specific datasets. Hyper-parameters, such as learning rate, are optimized for validation accuracy. A concrete description of the hyper-parameter sweep, along with the final parameters chosen appears in Appendix <ref>. Given the inherent instability of prompt tuning, after parameter optimization we tuned fifteen different prompts, each with a different random seed. For each model size and task-specific dataset, we selected the top five prompts in terms of validation accuracy in order to establish mean and confidence interval estimates for the test set accuracy and associated fairness (bias) metrics. Early stopping is applied during prompt tuning. The criterion for early stopping is a given evaluation loss exceeding the maximum of the previous five observed evaluation losses after the initial $2,500$ steps. All prompts are trained until the early stopping criterion is met.

[Shape = circle, FillColor  = white, LineWidth  = 1pt]
[lw = 0.75 pt, color = black, labelstyle = sloped]

[draw=black, thick,rectangle, fill=white, rounded corners=0.1cm, minimum width=10cm, minimum height=0.1cm, align=center](P1) at (0, 0.35) Decoder-Only Transformer;
[draw=black, thick,rectangle, fill=white, rounded corners=0.1cm, minimum width=10cm, minimum height=0.1cm, align=center](E1) at (0,  -2.9) Text Embed. Layer;
[draw=black, dashed, thick,rectangle, fill=lavender, rounded corners=0.1cm, minimum width=1cm, minimum height=2cm, align=center](E2) at (-6, -1.3) Prompt

[draw=none, rectangle, fill=burnt_orange, rounded corners=0.05cm, minimum width=0.4cm, minimum height = 0.4cm, align=center](dummy) at (-7.35, -0.55);
[draw=none, rectangle, fill=burnt_orange, rounded corners=0.05cm, minimum width=0.4cm, minimum height = 0.5cm, align=center](dummy) at (-7.35, -1.05);
[draw=none, rectangle, fill=burnt_orange, rounded corners=0.05cm, minimum width=0.4cm, minimum height = 0.4cm, align=center](dummy) at (-7.35, -1.6);
[draw=none, rectangle, fill=burnt_orange, rounded corners=0.05cm, minimum width=0.4cm, minimum height = 0.4cm, align=center](dummy) at (-7.35, -2.15);

[draw=none, rectangle, fill=parchment, rounded corners=0.05cm, minimum width=0.4cm, minimum height = 0.4cm, align=center](dummy) at (-4.5, -3.85);
[draw=none, rectangle, fill=parchment, rounded corners=0.05cm, minimum width=0.4cm, minimum height = 0.4cm, align=center](dummy) at (-3.9, -3.85);
[draw=none, rectangle, fill=parchment, rounded corners=0.05cm, minimum width=0.4cm, minimum height = 0.4cm, align=center](dummy) at (-3.3, -3.85);
[draw=none, rectangle, fill=parchment, rounded corners=0.05cm, minimum width=0.4cm, minimum height = 0.4cm, align=center](dummy) at (-2.7, -3.85);
[draw=none, rectangle, fill=parchment, rounded corners=0.05cm, minimum width=0.4cm, minimum height = 0.4cm, align=center](dummy) at (-2.1, -3.85);
[draw=none, rectangle, fill=parchment, rounded corners=0.05cm, minimum width=0.5cm, minimum height = 0.4cm, align=center](dummy) at (-1.5, -3.85);
[draw=none, rectangle, fill=parchment, rounded corners=0.05cm, minimum width=0.45cm, minimum height = 0.4cm, align=center](dummy) at (-0.9, -3.85);
[draw=none, rectangle, fill=parchment, rounded corners=0.05cm, minimum width=0.45cm, minimum height = 0.4cm, align=center](dummy) at (-0.3, -3.85);
[draw=none, rectangle, fill=parchment, rounded corners=0.05cm, minimum width=0.55cm, minimum height = 0.4cm, align=center](dummy) at (0.3, -3.85);
[draw=none, rectangle, fill=parchment, rounded corners=0.05cm, minimum width=0.5cm, minimum height = 0.4cm, align=center](dummy) at (0.9, -3.85);
[draw=none, rectangle, fill=parchment, rounded corners=0.05cm, minimum width=0.45cm, minimum height = 0.4cm, align=center](dummy) at (1.5, -3.85);
[draw=none, rectangle, fill=parchment, rounded corners=0.05cm, minimum width=0.25cm, minimum height = 0.4cm, align=center](dummy) at (2, -3.85);
[draw=none, rectangle, fill=parchment, rounded corners=0.05cm, minimum width=0.8cm, minimum height = 0.4cm, align=center](dummy) at (2.7, -3.85);
[draw=none, rectangle, fill=parchment, rounded corners=0.05cm, minimum width=0.45cm, minimum height = 0.4cm, align=center](dummy) at (3.4, -3.85);
[draw=none, rectangle, fill=parchment, rounded corners=0.05cm, minimum width=0.8cm, minimum height = 0.4cm, align=center](dummy) at (4.1, -3.85);

[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.1cm, minimum height = 0.4cm, align=center](p1_e) at (-6, -0.5);
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.1cm, minimum height = 0.4cm, align=center](p2_e) at (-6, -1.1);
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.1cm, minimum height = 0.4cm, align=center](p3_e) at (-6, -1.55);
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.1cm, minimum height = 0.4cm, align=center](p4_e) at (-6, -2.1);

[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.4cm, minimum height = 0.4cm, align=center](t1_e) at (-4.5, -2.9);
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.4cm, minimum height = 0.4cm, align=center](sos_1_e) at (-3.9, -2.9);
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.4cm, minimum height = 0.4cm, align=center](sos_2_e) at (-3.3, -2.9);
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.4cm, minimum height = 0.4cm, align=center](sos_dots_e) at (-2.7, -2.9);
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.4cm, minimum height = 0.4cm, align=center](sos_n_e) at (-2.1, -2.9);
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.5cm, minimum height = 0.4cm, align=center](t2_e) at (-1.5, -2.9);
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.45cm, minimum height = 0.4cm, align=center](t3_e) at (-0.9, -2.9);
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.45cm, minimum height = 0.4cm, align=center](t4_e) at (-0.3, -2.9);
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.55cm, minimum height = 0.4cm, align=center](t5_e) at (0.3, -2.9);
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.5cm, minimum height = 0.4cm, align=center](t6_e) at (0.9, -2.9);
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.3cm, minimum height = 0.4cm, align=center](t7_e) at (1.5, -2.9);
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.25cm, minimum height = 0.4cm, align=center](t8_e) at (2, -2.9);
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.9cm, minimum height = 0.4cm, align=center](t9_e) at (2.7, -2.9);
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.45cm, minimum height = 0.4cm, align=center](t10_e) at (3.4, -2.9);
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.9cm, minimum height = 0.4cm, align=center](t11_e) at (4.1, -2.9);

[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.4cm, minimum height = 0.4cm, align=center](p4_d) at (-3.9, -2.1);
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.4cm, minimum height = 0.4cm, align=center](p3_d) at (-3.3, -1.55);
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.5cm, minimum height = 0.4cm, align=center](p2_d) at (-2.7, -1.1);
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.4cm, minimum height = 0.4cm, align=center](p1_d) at (-2.1, -0.5);

[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.4cm, minimum height = 0.4cm, align=center](t1_d) at (-4.5, 0.4);

[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.4cm, minimum height = 0.4cm, align=center](sos_1_d) at (-3.9, -2.1)$\bigoplus$;
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.4cm, minimum height = 0.4cm, align=center](sos_2_d) at (-3.3, -1.55)$\bigoplus$;
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.4cm, minimum height = 0.4cm, align=center](sos_dots_d) at (-2.7, -1.1)$\bigoplus$;
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.4cm, minimum height = 0.4cm, align=center](sos_n_d) at (-2.1, -0.5)$\bigoplus$;

[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.4cm, minimum height = 0.4cm, align=center](sos_1_end) at (-3.9, 0.4);
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.4cm, minimum height = 0.4cm, align=center](sos_2_end) at (-3.3, 0.4);
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.4cm, minimum height = 0.4cm, align=center](sos_dots_end) at (-2.7, 0.4);
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.4cm, minimum height = 0.4cm, align=center](sos_n_end) at (-2.1, 0.4);

[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.5cm, minimum height = 0.4cm, align=center](t2_d) at (-1.5, 0.4);
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.45cm, minimum height = 0.4cm, align=center](t3_d) at (-0.9, 0.4);
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.45cm, minimum height = 0.4cm, align=center](t4_d) at (-0.3, 0.4);
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.55cm, minimum height = 0.4cm, align=center](t5_d) at (0.3, 0.4);
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.5cm, minimum height = 0.4cm, align=center](t6_d) at (0.9, 0.4);
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.3cm, minimum height = 0.4cm, align=center](t7_d) at (1.5, 0.4);
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.25cm, minimum height = 0.4cm, align=center](t8_d) at (2, 0.4);
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.9cm, minimum height = 0.4cm, align=center](t9_d) at (2.7, 0.4);
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.45cm, minimum height = 0.4cm, align=center](t10_d) at (3.4, 0.4);
[draw=none, rectangle, rounded corners=0.05cm, minimum width=0.9cm, minimum height = 0.4cm, align=center](t11_d) at (4.1, 0.4);

[draw=none, rectangle, rounded corners=0.0cm, minimum width=0.1cm, minimum height = 0.5cm, text height=1.5ex, text depth=.25ex, text width=1em, align=center](p1) at (-7.35, -0.5)$P_n$;
[draw=none, rectangle, rounded corners=0.0cm, minimum width=0.1cm, minimum height = 0.5cm, text height=1.5ex, text depth=.25ex, text width=1em, align=center](p2) at (-7.35, -1.1)$\vdots$;
[draw=none, rectangle, rounded corners=0.0cm, minimum width=0.1cm, minimum height = 0.5cm, text height=1.5ex, text depth=.25ex, text width=1em, align=center](p_dots) at (-7.35, -1.55)$P_2$;
[draw=none, rectangle, rounded corners=0.0cm, minimum width=0.1cm, minimum height = 0.5cm, text height=1.5ex, text depth=.25ex, text width=1em, align=center](pn) at (-7.35, -2.1)$P_1$;

[draw=none, rectangle, rounded corners=0.0cm, minimum width=1cm, minimum height = 0.5cm, text height=1.5ex, text depth=.25ex, text width=11em, align=center](sos) at (-4.5, -3.8)$\langle s \rangle$;

[draw=none, rectangle, rounded corners=0.0cm, minimum width=1cm, minimum height = 0.5cm, text height=1.5ex, text depth=.25ex, text width=11em, align=center](sos_1) at (-3.9, -3.8)$\langle s \rangle$;
[draw=none, rectangle, pattern=hatch, pattern color=burnt_orange_hatch, hatch size=8pt, hatch angle=0, rounded corners=0.0cm, minimum width=0.4cm, minimum height = 0.4cm, align=center](dummy) at (-3.9, -3.85);
[draw=none, rectangle, rounded corners=0.0cm, minimum width=1cm, minimum height = 0.5cm, text height=1.5ex, text depth=.25ex, text width=11em, align=center](sos_2) at (-3.3, -3.8)$\langle s \rangle$;
[draw=none, rectangle, pattern=hatch, pattern color=burnt_orange_hatch, hatch size=8pt, hatch angle=0, rounded corners=0.0cm, minimum width=0.4cm, minimum height = 0.4cm, align=center](dummy) at (-3.3, -3.85);
[draw=none, rectangle, rounded corners=0.0cm, minimum width=1cm, minimum height = 0.5cm, text height=1.5ex, text depth=.25ex, text width=11em, align=center](sos_dots) at (-2.7, -3.8)$\ldots$;
[draw=none, rectangle, pattern=hatch, pattern color=burnt_orange_hatch, hatch size=8pt, hatch angle=0, rounded corners=0.0cm, minimum width=0.4cm, minimum height = 0.4cm, align=center](dummy) at (-2.7, -3.85);
[draw=none, rectangle, rounded corners=0.0cm, minimum width=1cm, minimum height = 0.5cm, text height=1.5ex, text depth=.25ex, text width=11em, align=center](sos_n) at (-2.1, -3.8)$\langle s \rangle$;
[draw=none, rectangle, pattern=hatch, pattern color=burnt_orange_hatch, hatch size=8pt, hatch angle=0, rounded corners=0.0cm, minimum width=0.4cm, minimum height = 0.4cm, align=center](dummy) at (-2.1, -3.85);

[decorate, thick,decoration=brace,mirror] (sos_1.south) – (sos_n.south);
[draw=none, rectangle, rounded corners=0.0cm, minimum width=1cm, minimum height = 0.5cm, text height=1.5ex, text depth=.25ex, text width=11em, align=center](dummy) at (-3, -4.4)$n$ virtual tokens;
[draw=none, rectangle, rounded corners=0.0cm, minimum width=1cm, minimum height = 0.5cm, text height=1.5ex, text depth=.25ex, text width=11em, align=center](t1) at (-1.5, -3.8)Great;
[draw=none, rectangle, rounded corners=0.0cm, minimum width=1cm, minimum height = 0.5cm, text height=1.5ex, text depth=.25ex, text width=11em, align=center](t2) at (-0.9, -3.8)food;
[draw=none, rectangle, rounded corners=0.0cm, minimum width=1cm, minimum height = 0.5cm, text height=1.5ex, text depth=.25ex, text width=11em, align=center](tdots) at (-0.3, -3.8)$\ldots$;
[draw=none, rectangle, rounded corners=0.0cm, minimum width=1cm, minimum height = 0.5cm, text height=1.5ex, text depth=.25ex, text width=11em, align=center](tnp1) at (0.3, -3.8)delic;
[draw=none, rectangle, rounded corners=0.0cm, minimum width=1cm, minimum height = 0.5cm, text height=1.5ex, text depth=.25ex, text width=11em, align=center](tnp2) at (0.9, -3.8)ious;
[draw=none, rectangle, rounded corners=0.0cm, minimum width=1cm, minimum height = 0.5cm, text height=1.5ex, text depth=.25ex, text width=11em, align=center](tnp3) at (1.5, -3.8)meal;
[draw=none, rectangle, rounded corners=0.0cm, minimum width=1cm, minimum height = 0.5cm, text height=1.5ex, text depth=.25ex, text width=11em, align=center](tnp4) at (2, -3.8).;
[draw=none, rectangle, rounded corners=0.0cm, minimum width=1cm, minimum height = 0.5cm, text height=1.5ex, text depth=.25ex, text width=11em, align=center](mask_1) at (2.7, -3.8)$\text{MASK}_1$;
[draw=none, rectangle, rounded corners=0.0cm, minimum width=1cm, minimum height = 0.5cm, text height=1.5ex, text depth=.25ex, text width=11em, align=center](mask_dots) at (3.4, -3.8)$\ldots$;
[draw=none, rectangle, rounded corners=0.0cm, minimum width=1cm, minimum height = 0.5cm, text height=1.5ex, text depth=.25ex, text width=11em, align=center](mask_k) at (4.1, -3.8)$\text{MASK}_k$;

EdgeStyle/.append style = post

[draw=none, rectangle, rounded corners=0.0cm, minimum width=0.1cm, minimum height = 0.5cm, align=center](log_p) at (2.7, 1.25)$\log\left(P(\text{``positive''}\vert \overleftarrow{\text{input}})\right)$;
[draw=none, rectangle, rounded corners=0.0cm, minimum width=0.1cm, minimum height = 0.5cm, align=center](dummy) at (-1, 1.0) $\nabla$ Update;
[draw=none, rectangle, rounded corners=0.0cm, minimum width=0.1cm, minimum height = 0.5cm, align=center](log_p_dummy) at (-4.5, 1.25);
[style=-, dashed, lw = 0.75 pt, color = black,]
[style=bend right,out=-30,in=-135, dashed, ->, color=black](log_p_dummy.center)([yshift=0.2ex]E2.north)

Illustration of the prompt-tuning approach used for efficient parameter fine-tuning of OPT and Galactica. The prompt tokens, depicted with orange hatching, are initialized as the beginning-of-sequence token embedding. These embeddings are subsequently perturbed by adding learned prompt embeddings. All weights are frozen during back-propagation except for the prompt embedding layer.

§ RESULTS

FPR gap for the negative class for sensitive attribute sexuality. Markers indicate the average gap and bars are $95$% confidence intervals. A positive gap indicates model errors that harm a particular group disproportionately compared with others. For SemEval adult examples are erroneously cast in a negative light at much lower rates.

In this section, we present the results for different sensitive attributes by showing the FPR gap for the various protected groups compared across both the SemEval and SST-5 datasets, as well as the various model sizes of OPT and Galactica.

In Figure <ref>, the FPR gap for positive sentiment is shown for sexuality. Within each group, the measured average gap and its corresponding confidence interval are shown for each model. The positive FPR Gap measures the rate at which the model erroneously classifies negative or neutral statements associated with the protected group in a favourable light. Therefore, consistent and  significant negative gaps across models represent cases for particular sexuality implying that such groups benefit from model mistakes at a significantly lower rate than others. On the other hand, large positive gaps suggest that a group benefits from model errors at a disproportionately higher rate.

Figure <ref> shows that the rate at which examples belonging to the asexual group benefit from model mistakes is consistently lower for models trained on both the SemEval and SST-5 datasets. Somewhat surprisingly, in this measure, there is some evidence to suggest that heterosexual examples is unfavourable group and do not benefit from model mistakes. However, the pattern is fairly weak and is reversed when considering the negative class FPR gap. It is interesting to note that examples from the group termed other appear to benefit, in a noticeable way, from model mistakes in both datasets.

The results exhibited in Figure <ref> display the negative FPR gap. These represent model errors that have predicted that neutral or positive data points from each protected group are actually negative examples. Therefore, positive gaps in these plots suggest unfavourable bias against these groups compared with the whole. It is evident that as in Figure <ref> the asexual group suffers from a notably elevated harmful error rate. Furthermore, the homosexual group experiences consistently higher negative-class FPR for both datasets considered and nearly all models. As mentioned above, the heterosexual group experiences a favorable decreases in this FPR, departing from the results observed for positive-class FPR.

Metric      2c|Positive FPR Gap     2c|Negative FPR Gap

Group\Dataset      SemEval     SST-5     SemEval     SST-5

Asexual     -5     -7     5     3

Homosexual     1     -2     6     6

Heterosexual     -2     -4     -2     -1

Adult     2     5     -6      -1

Old     -1     -4     1     -1

Young     -1     1     0     1

Net number of models (out of 7) where the gaps for each protected group differ from zero at the 95% confidence level. Negative values imply that the gap was consistently below zero. Red numbers indicate that the direction of significant gaps are harmful. The top three rows correspond to the sensitive attribute sexuality, while the bottom three are associated with age.

FPR gap for the positive class for the sensitive attribute of age. Markers indicate the average gap and bars are $95$% confidence intervals. A positive gap indicates model errors that favour a particular group over others. The rate at which adult examples benefit from model mistakes is elevated for SST-5 while elderly examples benefit at a much lower rate. The trends for SemEval are not as clear for this metric.

FPR gap for the negative class for the sensitive attribute of age. Markers indicate average gap and bars are $95$% confidence intervals. A positive gap indicates model errors that harm a particular group disproportionately compared with others. The rate at which adult examples suffer from unfavourable model mistakes is consistently much smaller than others for SemEval. This conclusion is not as clear for SST-5.

Reported alongside FPR gap measured for each model size is the confidence interval associated with that gap. For each group, Table <ref> displays the net number of times the gap was below and above zero, at 95% confidence. That is, for each  significant gaps below zero we subtract one, while one is added for significant gaps above zero. Values colored in red indicate the direction of the significant gaps that are considered harmful, while those in green denote potentially favourable treatment by the models. For both the asexual and homosexual protected groups, the experimental results strongly indicate the potential harmful bias for both datasets. However, as previously discussed, the evidence is mixed for the heterosexual group with respect to the two gaps.

As with the sensitive attribute of sexuality, discussed in the previous section, the FPR gaps for protected groups belonging to the age attribute are analyzed in this section. Figure <ref> shows the FPR measured for the positive class. When considering results from the SST-5 dataset, a marked favourable increase in this FPR is present for the adult group while an unfavourable decrease is observed for examples from the old group. Interestingly, this trend is, at best, weakly present for the SemEval dataset. On the other hand, when considering the measurements in Figure <ref>, the adult group is impacted by errors casting them in a negative light at a significantly lower rate than the other groups for the SemEval dataset. In addition, the old and young groups appear to suffer from an elevated probability of such errors, though the gaps are not significant when the confidence intervals are considered. The gaps observed for the SST-5 dataset are much less consistent. In both cases, there is agreement as to which groups suffer or benefit from model bias. However, the way in which the bias is manifested is slightly different depending on the prompt-tuning dataset specific to the downstream task. Table <ref> reinforces this conclusion. Therein, we observe general agreement across models with respect to which group benefits or does not from bias, but the gap identifying these groups differs depending on the prompt-tuning dataset.

§ DISCUSSION & CONCLUSION

Multidimensional aspects of the experiments: While in this paper, we have explored the utility of a state-of-the-art soft-prompt tuning technique, the chosen downstream task is, in itself, very challenging yet impactful. This makes this exploration two-fold interesting but the analysis of the results is multidimensional across datasets, templates, prompt-tuning choices, sensitive attributes, their protected groups, language models, numerous fairness (bias) metrics, and their graphical representations. We have tried our best to present the results in the most comprehensive way.

Template design: We use templates from Czarnowska et al. <cit.>. While they provided a baseline for our experiments, they consist of very simple sentences and hence are easily understood by the LLMs. That may be the cause of less conclusive results in some cases. It is worth trying to run experiments on more complicated templates, which is the subject of future work.

Types of biases: Many papers <cit.> rely on absolute values of the metric disparities that simply identify bias. We use a directional bias measure to be able to identify the favoured and un-favoured groups providing more precise bias analysis of the LLMs.
It should be mentioned that a group that is flagged as a favourable group may be flagged as unfavourable by using a different bias quantification metric. Thus, different bias quantification formulations <cit.> might not be concurrently achievable.

Impact of soft-prompt tuning on bias:

Continuous prompt tuning minimizes the potential influence of biases existing in the supervised training tasks by restricting the number of learned parameters. Furthermore, it removes the human element of prompt design, eliminating another avenue for introducing bias outside of the LLM itself. However, it should be noted that we performed soft-prompt tuning on the commonly used sentiment datasets that were generated from tweets (SemEval) and movie reviews (SST-5). The quality of these datasets has a strong impact on the soft-prompts produced. It is worth exploring how a better quality dataset (if available) impacts the performance of the downstream task and the bias observed.

Impact of bias quantification on industry:
Industry applications of AI-based solutions broaden every day. This has raised many bias- and ethics-related issues. Here, we have provided the methodology and infrastructure design for using state-of-the-art technology to identify possible risks. We believe that our effort is a small step toward exploring this emerging area and will benefit industry as well as the research community in general.

As the next steps, we plan to extend our work by including a broader range of language models, extending to more sensitive attributes, including more bias metrics and a variety of downstream tasks. This is an effort to make the use of LLMs safer and more ethical in real-world deployment.

#1     #1#1#1     #1     #1     #1     #1#1          #1#1

[Bradbury et al(2018)]
authorpersonJ. Bradbury, personR. Frostig,
personP. Hawkins, personM.J. Johnson,
personC. Leary, personD. Maclaurin,
personG. Necula, personA. Paszke, personJ.
VanderPlas, personS. Wanderman-Milne, and
personQ. Zhang. year2018.
booktitleJAX: composable transformations of
Python+NumPy programs.

[Brown et al(2020)]
authorpersonT. Brown, personB. Mann,
personN. Ryder, personM. Subbiah,
personJ.D. Kaplan, personP. Dhariwal,
personA. Neelakantan, and personet al.
Language models are few-shot learners. In
booktitleAdvances in Neural Information Processing
Systems, Vol. volume33. pages1877–1901.

[Czarnowska et al(2021)]
authorpersonP. Czarnowska, personY. Vyas,
and personK. Shah. year2021.
Quantifying Social Biases in NLP: A
Generalization and Empirical Comparison of Extrinsic Fairness Metrics.
journalTransactions of the Association for
Computational Linguistics  volume9 (year2021),

[Delobelle et al(2022)]
authorpersonP. Delobelle, personE.K. Tokpo,
personT. Calders, and personB. Berendt.
Measuring fairness with biased rulers: A
comparative study on bias metrics for pre-trained language models. In
booktitleNAACL 2022: the 2022 Conference of the North
American chapter of the Association for Computational Linguistics: human
language technologies. pages1693–1706.

[Dixon et al(2018)]
authorpersonL. Dixon, personJ. Li,
personJ. Sorensen, personN. Thain, and
personL. Vasserman. year2018.
Measuring and mitigating unintended bias in text
classification.. In booktitleProceedings of the 2018 AAAI/
ACM Conference on AI, Ethics, and Society. publisherAssociation
for Computing Machinery, addressNew York, NY, USA.,

[Han et al(2021)]
authorpersonX. Han, personZ. Zhang,
personN. Ding, personY. Gu, personX. Liu,
personY. Huo, personJ. Qiu, personY. Yao,
personA. Zhang, personL. Zhang, et al
Pre-trained models: Past, present and future.
journalAI Open  volume2
(year2021), pages225–250.

[Jiang et al(2021)]
authorpersonZ. Jiang, personJ. Araki,
personH. Ding, and personG. Neubig.
How Can We Know When Language Models Know? On the
Calibration of Language Models for Question Answering.
journalTransactions of the Association for
Computational Linguistics volume9, number962977

[Lester et al(2021)]
authorpersonB. Lester, personR. Al-Rfou,
and personN. Constant. year2021.
The Power of Scale for Parameter-Efficient Prompt
Tuning. In booktitleProceedings of the 2021 Conference on
Empirical Methods in Natural Language Processing.
publisherAssociation for Computational Linguistics,

[Li and Liang(2021a)]
authorpersonX.L. Li and personP. Liang.
Prefix-Tuning: Optimizing Continuous Prompts for
journalProceedings of the 59th Annual Meeting of the
Association for Computational Linguistics and the 11th International Joint
Conference on Natural Language Processing (Volume 1: Long Papers)
volumeabs/2101.00190 (year2021).

[Li and Liang(2021b)]
authorpersonX.L. Li and personP. Liang.
Prefix-tuning: Optimizing continuous prompts for
journalarXiv preprint arXiv:2101.00190

[Liang et al(2022)]
authorpersonP. Liang, personR. Bommasani,
personT. Lee, personD. Tsipras, personD.
Soylu, personM. Yasunaga, personY. Zhang,
personD. Narayanan, personY. Wu, personA.
Kumar, personB. Newman, personB. Yuan,
personB. Yan, personC. Zhang, personC.
Cosgrove, personC.D. Manning, personC. Ré,
personD. Acosta-Navas, personD.A. Hudson,
personE. Zelikman, personE. Durmus,
personF. Ladhak, personF. Rong, personH.
Ren, personH. Yao, personJ. Wang,
personK. Santhanam, personL. Orr, personL.
Zheng, personM. Yuksekgonul, personM. Suzgun,
personN. Kim, personN. Guha, personN.
Chatterji, personO. Khattab, personP. Henderson,
personQ. Huang, personR. Chi, personS.M.
Xie, personS. Santurkar, personS. Ganguli,
personT. Hashimoto, personT. Icard,
personT. Zhang, personV. Chaudhary,
personW. Wang, personX. Li, personY. Mai,
personY. Zhang, and personY. Koreeda.
titleHolistic Evaluation of Language Models.

[Liu et al(2021a)]
authorpersonP. Liu, personW. Yuan,
personJ. Fu, personZ. Jiang, personH.
Hayashi, and personG. Neubig.
Pre-train, prompt, and predict: A systematic survey
of prompting methods in natural language processing.
journalarXiv preprint arXiv:2107.13586

[Liu et al(2020)]
authorpersonQ. Liu, personM.J. Kusner,
and personP. Blunsom. year2020.
A survey on contextual embeddings.
journalarXiv preprint arXiv:2003.07278

[Liu et al(2022)]
authorpersonX. Liu, personK. Ji,
personY. Fu, personW. Tam, personZ. Du,
personZ. Yang, and personJ. Tang.
P-Tuning: Prompt Tuning Can Be Comparable to
Fine-tuning Across Scales and Tasks. In
booktitleProceedings of the 60th Annual Meeting of the
Association for Computational Linguistics (Volume 2: Short Papers).
publisherAssociation for Computational Linguistics,
addressDublin, Ireland, pages61–68.

[Liu et al(2021b)]
authorpersonX. Liu, personY. Zheng,
personZ. Du, personM. Ding, personY.
Qian, personZ. Yang, and personJ. Tang.
GPT Understands, Too.
journalCoRR  volumeabs/2103.10385

[Loshchilov and Hutter(2017)]
authorpersonI. Loshchilov and personF.
Hutter. year2017.
Fixing Weight Decay Regularization in ADAM. In

[Mao et al(2022)]
authorpersonR. Mao, personQ. Liu,
personK. He, personW. Li, and personE.
Cambria. year2022.
The biases of pre-trained language models: An
empirical study on prompt-based sentiment analysis and emotion detection.
journalIEEE Transactions on Affective Computing

[Mehrabi et al(2021)]
authorpersonN. Mehrabi, personF.
Morstatter, personN. Saxena, personK. Lerman, and
personA. Galstyan. year2021.
A survey on bias and fairness in machine learning.
journalACM Computing Surveys (CSUR)
volume54, number6 (year2021),

[Parrish et al(2022)]
authorpersonA. Parrish, personA. Chen,
personN. Nangia, personV. Padmakumar,
personJ. Phang, personJ. Thompson,
personP.M. Htut, and personS.R. Bowman.
BBQ: A hand-built bias benchmark for question
answering. In booktitleFindings of the Association for
Computational Linguistics: ACL 2022,
editorpersonS. Muresan, personP. Nakov,
and personA. Villavicencio (Eds.).
publisherAssociation for Computational Linguistics,

[Petroni et al(2019)]
authorpersonF. Petroni, personT.
Rocktäschel, personP. Lewis, personA. Bakhtin,
personX. Wu, personA.H. Miller, and
personS. Riedel. year2019.
Language models as knowledge bases?
journalarXiv preprint arXiv:1909.01066

[Saif et al(2018)]
authorpersonM.M. Saif, personF.
Bravo-Marquez, personM. Salameh, and personS.
Kiritchenko. year2018.
SemEval-2018 Task 1: Affect in Tweets. In
booktitleProceedings of International Workshop on Semantic
Evaluation (SemEval-2018). addressNew Orleans, LA, USA.

[Schick and Schütze(2021)]
authorpersonT. Schick and personH.
Schütze. year2021.
Exploiting Cloze-Questions for Few-Shot Text
Classification and Natural Language Inference. In
booktitleConference of the European Chapter of the
Association for Computational Linguistics.

[Seyyed-Kalantari et al(2021)]
authorpersonL. Seyyed-Kalantari, personH.
Zhang, personM.B.A. McDermott, personI.Y. Chen,
and personM. Ghassemi. year2021.
Underdiagnosis bias of artificial intelligence
algorithms applied to chest radiographs in under-served patient populations.
journalNature Medicine volume27,
number12 (year2021), pages2176–2182.

[Shin et al(2020)]
authorpersonT. Shin, personY. Razeghi,
personR.L. Logan IV, personE. Wallace, and
personS. Singh. year2020.
AutoPrompt: Eliciting Knowledge from Language
Models with Automatically Generated Prompts. In
booktitleEMNLP 2020. pages4222–4235.

[Socher et al(2013)]
authorpersonR. Socher, personA. Perelygin,
personJ. Wu, personJ. Chuang, personC.D.
Manning, personA. Ng, and personC. Potts.
Recursive Deep Models for Semantic Compositionality
Over a Sentiment Treebank. In booktitleProceedings of the
2013 Conference on Empirical Methods in Natural Language Processing.
publisherAssociation for Computational Linguistics,
addressSeattle, Washington, USA, pages1631–1642.

[Srivastava et al(2022)]
authorpersonA. Srivastava, personA.
Rastogi, personA. Rao, personA.A.M. Shoeb,
personA. Abid, personA. Fisch, personA.R.
Brown, personA. Santoro, personA. Gupta,
personA. Garriga-Alonso, et al
Beyond the imitation game: Quantifying and
extrapolating the capabilities of language models.
journalarXiv preprint arXiv:2206.04615

[Taylor et al(2022)]
authorpersonR. Taylor, personM. Kardas,
personG. Cucurull, personT. Scialom,
personA. Hartshorn, personE. Saravia,
personA. Poulton, personV. Kerkez, and
personR. Stojnic. year2022.
Galactica: A large language model for science.
journalarXiv preprint arXiv:2211.09085

[Wang et al(2022)]
authorpersonH. Wang, personJ. Li,
personH. Wu, personE. Hovy, and personY.
Sun. year2022.
Pre-Trained Language Models and Their
journalEngineering (year2022).

[Zhang et al(2022)]
authorpersonS. Zhang, personS. Roller,
personN. Goyal, personM. Artetxe, personM.
Chen, personS. Chen, personC. Dewan,
personM. Diab, personX. Li, personX.V.
Lin, personT. Mihaylov, personM. Ott,
personS. Shleifer, personK. Shuster,
personD. Simig, personP.S. Koura, personA.
Sridhar, personT. Wang, and personL. Zettlemoyer.
titleOPT: Open Pre-trained Transformer Language Models.

§ FAIRNESS METRICS

Sensitive attribute (S) is defined as an attribute, which can potentially be the cause of  bias(es). Examples of sensitive attributes include age, disability, gender, nationality, race, religion, and sexuality.

Protected group (P) Each sensitive attribute can consist of different protected groups such as the sensitive attribute Age may consist of {child, teenager, adult, older adult}. (may include the identity definition here)

§ HYPERPARMETER DETAILS

We conducted a grid search over the validation split of SemEval and SST5 for the following possible learning rate values: 0.01, 0.001, 0.0001. We list the hyperparameters selected for each model in table <ref>. The number of prompt tokens for all models is fixed at $8$. This value was also chosen by hyperparameter optimization over a prompt length of $16$.

Model     LR

OPT-125m     0.001

OPT-350m     0.001

OPT-1.3b     0.001

OPT-2.7b     0.001

OPT-6.7b     0.001

OPT-13b     0.0001

Galactica-1.3b     0.001

Galactica-6.7b     0.0001

Hyperparameter selection for each model. Parameters were chosen based on each models performance on the validation set.

§ DATASETS

We tune the prompts on two sentiment datasets. The SemEval-2018 Task 1 - Valence Ordinal Classification <cit.> (SemEval) and Stanford Sentiment Treebank Five-way (SST-5)<cit.> collections. The SemEval Task 1 dataset is a collection of English tweets with integral labels ranging from $[-3, 3]$. Following <cit.>, these labels are condensed by mapping {Negative 0: [-3, -2], Neutral 1: [-1, 0, 1], Positive 2: [2, 3]}. The labels of SST-5 (very positive, positive, neutral, negative, very negative) are based on brief English movie reviews and, therefore, constitute a very different underlying corpus. As with the SemEval valence labels, the five-way annotations of SST-5 are collapsed to three-way classification by retaining the neutral label and mapping positive and negative polarity of any kind simply to positive or negative classes.

§ PROMPT TUNING

The diagram shown in Figure <ref> demonstrates our implementation of prompt-tuning using beginning-of-sequence tokens to initialize the continuous prompts and additive perturbations for the trained prompt embeddings.