# Turning graphene into a lab for noncommutativity

Alfredo Iorio<EMAIL_ADDRESS>Faculty of Mathematics and Physics,
Charles University, V Holešovičkách 2, 18000 Prague 8, Czech Republic  Boris
Ivetić<EMAIL_ADDRESS>Wöhlergasse 6, 1100 Vienna, Austria  Pablo Pais
<EMAIL_ADDRESS>Instituto de Ciencias Físicas y Matemáticas, Universidad
Austral de Chile, Casilla 567, 5090000 Valdivia, Chile  Faculty of Mathematics
and Physics, Charles University, V Holešovičkách 2, 18000 Prague 8, Czech
Republic

###### Abstract

It was recently shown that taking into account the granular structure of
graphene lattice, the Dirac-like dynamics of its quasiparticles resists beyond
the lowest energy approximation. This can be described in terms of new phase-
space variables, $(\vec{X},\vec{P})$, that enjoy generalized Heisenberg
algebras. In this letter, we add to that picture the important case of
noncommuting $\vec{X}$, for which $[X^{i},X^{j}]=\mathrm{i}\,\theta^{ij}$ and
we find that $\theta^{ij}=\ell^{2}\,\epsilon^{ij}$, with $\ell$ the lattice
spacing. We close by giving both the general recipe and a possible specific
kinematic setup for the practical implementation of this approach to test
noncommutative theories in tabletop analog experiments on graphene.

##  I Introduction

In this letter, in the same spirit of [1], that is, regarding graphene as an
analog of certain quantum gravity (QG) scenarios, we consider the possibility
that coordinates enjoy canonical noncommutativity (CNC)

$[X_{i},X_{j}]=\mathrm{i}\,\theta_{ij},$ (1)

with $\theta_{ij}$ a constant antisymmetric tensor of dimension $L^{2}$ and
$i,j,...=1,2$.

With analogs we can test the behaviour of one physical system, by doing
experiments on another physical system [2]. This becomes particularly
important when we want to test theories describing phenomena that otherwise
lie far beyond our experimental capabilities [3].

The research on gravity analogs can be seen as originating from the seminal
work of Unruh [4]. He proposed searching for experimental signatures of the
Unruh [5] and the Hawking effect [6], in a fluid-dynamical analog. These days,
we can reproduce many different aspects of fundamental physics in analog
systems, in Bose–Einstein condensates [7], in Weyl semimetals [8], in graphene
[9, 10, 11, 12, 13], and on many more platforms [14].

Our interest here is on graphene as an analog of high-energy fundamental
physics, based on the fact that its low-energy quasiparticle excitations [15]
are massless Dirac pseudo-relativistic fermions (the matter fields $\psi$),
propagating in a carbon two-dimensional honeycomb lattice. We shall focus on a
regime that goes slightly beyond the lowest energy regime just recalled, where
nonlinear effects start playing a role, and the modified structure of space-
time of QG may be mimicked.

As it was first recognised in [16, 17], the effective high-energy momenta in
combination with the standard (low-energy) spatial coordinates obey a modified
Heisenberg algebra, characterised by deformed commutation relations between
position and momenta, which is characteristic of certain QG models involving a
minimal length (MLM) [18, *AliDasVagenas2, *AliDasVagenas3, *AliDasVagenas4,
*AliDasVagenas5].

A recent paper [1] continued on this, allowing the spatial coordinates at high
energy to be modified as well. One crucial result there is that, in two
important cases with GUPs, such high-energy coordinates simply coincide with
the low-energy coordinates. In other cases, such coordinates have complicated
expressions regarding the standard, measurable phase space variables. In all
cases, though, coordinates commute. Here, we show that noncommuting
coordinates are indeed possible in a canonical form, putting on more firm
grounds what was first speculated in [17].

CNC is one of the earliest models of noncommutativity, appearing independently
in various high-energy physics theories [23, 24, 25, 26,
*iorio2008noncommcomment]. For instance, in [23], it is argued that the
uncertainty relations descending from (1) may reflect constraints on the
localization of a particle due to classical gravity when the distance becomes
less then the Schwarzschild radius. The idea there is that there is a value of
the wavelength of the photon, $\lambda_{\gamma}$, used to measure the position
of a particle, that coincides with the Schwarzschild radius, $R_{\gamma}$,
associated with that energy, $E_{\gamma}=\hbar\omega_{\gamma}$. When such a
length scale is reached, one cannot go beyond it. Hence, there is a minimal
length, which gives a minimal spatial uncertainty. In [23], and usually, such
length is Planck’s, but there are also alternatives [28]. In this letter, as
we shall see in a moment, we do not need such arguments as the minimal length
is there regardless of any gravitational radius (on this, see also [28]).

In CNC models, the algebra is usually given by (1) and the two standard
commutation relations, between momenta and momenta and between positions and
momenta. In the case of graphene, however, in addition to (1), the commutation
relation between positions and momenta changes as well to

$[X_{i},P_{j}]=\mathrm{i}\,\hbar F_{ij}(P),$ (2)

where $F_{ij}(P)$ is an antisymmetric tensor that shares many features with
those appearing in MLMs. This leads to a novel type of noncommutativity, which
is a mix of CNC and MLM.

The letter is organized as follows. Section II is dedicated to recalling the
main reasons why graphene is a QG analog and setting the notation. Section III
is where we obtain noncommuting coordinates. Section IV is where we explain,
in general terms, how to use graphene as a tabletop laboratory to test
noncommutative theories, while in Section V, we focus on a specific kinematic
arrangement. Section VI is dedicated to our conclusions.

##  II The set-up

The dispersion relations of graphene can be approximated by [1]

$\displaystyle E_{\pm}=\eta\left(\pm|{\cal F}|-0.15|{\cal F}|^{2}\right)$ (3)

where $\eta\simeq-$2.8\text{\,}\mathrm{eV}$$ is the nearest neighbour hopping
energy,

${\cal
F}({\vec{k}})=\sum_{m=1}^{3}e^{(\mathrm{i}/\hbar){\vec{k}}\cdot{\vec{s}^{(1)}}_{m}}=e^{-\mathrm{i}\ell
k_{2}/\hbar}\left[1+2e^{\mathrm{i}\frac{3}{2}\ell
k_{2}/\hbar}\cos\left(\frac{\sqrt{3}}{2\hbar}\ell k_{1}\right)\right]\,,$ (4)

$\ell$ is the lattice spacing, the three vectors ${\vec{s}^{(1)}}_{m}$ connect
nearest neighbouring lattice sites [16] (all vectors are spatial two-
dimensional), and the zero of the energy is set at the Dirac point, i.e.
$E_{\pm}|_{k_{D}}=0$.

Expanding around the Dirac points, see [15], at the leading order in the
parameter $\ell|\vec{p}|/\hbar$ (the long-wavelength limit), one gets

$E_{\pm}=\pm v_{F}|\vec{p}|\,,$ (5)

where $\vec{p}$ is a small momentum and $v_{F}\equiv 3/2\,\eta\ell/\hbar\simeq
c/300$, is the Fermi velocity. Therefore, the physics of the conductivity
quasiparticles of graphene is governed, at order $O(|\vec{p}|)$, by the
standard Dirac linear Hamiltonian

$H(p)=v_{F}\,\sum_{p}\,\psi^{\dagger}_{p}\,\not{p}\,\psi_{p}\,,$ (6)

that, through

${\rm det}(H-E\mathbb{1})=0\,,$ (7)

gives (5) as dispersion relation. Our convention is111There are many
conventions related to different choices for the pairs of inequivalent Dirac
points and various arrangements of the $a$ and $b$ operators to form the
spinor $\psi$. See Appendix B of [29] for more details.
$\psi_{p}^{\dagger}=(b_{p}^{*},a_{p}^{*})$, where $a_{p}$ and $b_{p}$ are the
annihilation operators for the $L_{A}$ and $L_{B}$ sublattices [15].

Once one gets away from the Dirac points, higher-order momenta enter, and the
dispersion relation is no longer linear. It has been recognised in [16] that
one can maintain a linear dispersion relation, provided one redefines the
momenta. The relation (3) defines the auxiliary quantity

${\vec{P}_{0}}\equiv-\frac{\hbar}{\ell}\,\left({\rm Re}{\cal F},{\rm Im}{\cal
F}\right)\,,$ (8)

that was dubbed supermomentum in [1].

The dispersion relation (3) can be obtained from the secular equation (7),
written for the following Hamiltonian given in terms of the supermomenta
$\vec{P_{0}}$

$H(P_{0})=V_{F}\sum_{\vec{k}}\psi^{\dagger}_{\vec{k}}\left(\not\\!P_{0}-A\;\not\\!P_{0}\not\\!P_{0}\right)\psi_{\vec{k}}\,,$
(9)

where $V_{F}=\eta\ell/\hbar=2/3\,v_{F}$,
$\not{P}_{0}\equiv\vec{\sigma}\cdot{\vec{P}_{0}}$, $\vec{\sigma}$ are the
usual Pauli matrices and $A\simeq 0.15\ell/\hbar$.

Noticeably, this is the same Hamiltonian obtained in the phenomenology of QG,
when generalizing the Dirac Hamiltonian to accommodate a GUP with a minimal
fundamental length [18, *AliDasVagenas2, *AliDasVagenas3, *AliDasVagenas4,
*AliDasVagenas5]. Here such a fundamental length is clearly given by the
carbon-to-carbon distance, $\ell$.

In [18, *AliDasVagenas2, *AliDasVagenas3, *AliDasVagenas4, *AliDasVagenas5],
the authors introduce two different “layers” describing the given physical
system. Namely, “high energy” and “low energy” phase-space variables that,
adjusting our notation to theirs, we may call $(\vec{X},\vec{P})$ and
$(\vec{X}_{0},\vec{P}_{0})$, respectively. Here (and there), the definition of
the “high-energy momenta” descends directly from (9), that in turn is
associated with the dispersion relations (3)

${\vec{P}}\equiv{\vec{P}_{0}}(1-A|\vec{P_{0}}|)\,.$ (10)

This we call the _hypermomentum_.

With this definition, we arrive again at the standard Dirac linear Hamiltonian

$H(P)=V_{F}\,\sum_{k}\,\psi^{\dagger}_{k}\,\not{P}\,\psi_{k}\;,$ (11)

with dispersion relation

$E_{\pm}=\pm V_{F}|\vec{P}|\;.$ (12)

Notice that, to obtain $H(P_{0})$ from $H(P)$, one should first use the
expansion (10), and then use the customary Dirac prescription
$|\vec{P_{0}}|\to\not{P_{0}}$

$\displaystyle H(P(P_{0}))$ $\displaystyle=$ $\displaystyle
V_{F}\,\sum_{k}\,\psi^{\dagger}_{k}\,\vec{\sigma}\cdot\vec{P}\,\psi_{k}$ (13)
$\displaystyle=$ $\displaystyle
V_{F}\,\sum_{k}\,\psi^{\dagger}_{k}\vec{\sigma}\cdot\left(\vec{P_{0}}(1-A|\vec{P_{0}}|)\right)\,\psi_{k}$
$\displaystyle\to$ $\displaystyle
V_{F}\,\sum_{k}\,\psi^{\dagger}_{k}\left(\vec{\sigma}\cdot\vec{P_{0}}-A(\vec{\sigma}\cdot\vec{P_{0}})\,(\vec{\sigma}\cdot\vec{P_{0}})\right)\,\psi_{k}$
$\displaystyle=$ $\displaystyle
V_{F}\,\sum_{k}\,\psi^{\dagger}_{k}\left(\not{P_{0}}-A|\vec{P_{0}}|^{2}\right)\,\psi_{k}=H(P_{0})\,,$

where, in the second term of the last line, we used
$\sigma_{i}\sigma_{j}=\delta_{ij}\mathbb{1}+i\epsilon_{ijk}\sigma_{k}$ and the
symmetry of $P^{i}_{0}P^{j}_{0}$.

For all the above, see [1]. There it is shown that, in this set-up, we do know
supermomenta and hypermomenta in terms of the measurable momenta $\vec{p}$

$P_{0}^{i}(p)\,\,\,\,\mbox{and}\,\,\,\,P^{i}(P_{0}(p))\,,$ (14)

and one needs to find supercoordinates and hypercoordinates in terms of
measurable phase-space variables $(\vec{x},\vec{p})$

$X_{0}^{i}(x,p)\,\,\,\,\mbox{and}\,\,\,\,X^{i}(X_{0}(x,p),P_{0}(p))\,,$ (15)

where, for the latter, we can take as an experimental fact that the phase-
space variables are canonical

$[x^{i},p^{j}]=\mathrm{i}\hbar\delta^{ij},\ \ \
[x^{i},x^{j}]=0=[p^{i},p^{j}]\;.$ (16)

Henceforth, we shall always have

$[P_{0}^{i}(p),P_{0}^{j}(p)]=0=[P^{i}(p),P^{j}(p)]\,,$ (17)

whereas a great variety of choices are allowed for the associated generalized
coordinates.

##  III Noncommuting Hypercoordinates

Upon introduction of the hypermomenta $P^{i}$ in equation 10,
$P^{i}=P_{0}^{i}(1-A\,|\vec{P_{0}}|)$, which, in turn, can be expressed in
terms of standard $p^{i}$ as [1]

$\displaystyle P^{1}$
$\displaystyle=p^{1}(1-A\,|\vec{p}|)+\ell\left(\frac{A\,(p^{1})^{2}\left((p^{1})^{2}-3(p^{2})^{2}\right)}{4p}+\frac{1}{4}(1-A\,|\vec{p}|)\left((p^{2})^{2}-(p^{1})^{2}\right)\right)-\frac{1}{8}\,\ell^{2}\,|\vec{p}|^{2}\,p^{1}\;,$
(18) $\displaystyle P^{2}$
$\displaystyle=p^{2}(1-A\,|\vec{p}|)+\frac{\ell\,p^{2}\left(-A(p^{1})^{3}-5Ap^{1}(p^{2})^{2}+2|\vec{p}|p^{1}\right)}{4|\vec{p}|}-\frac{1}{8}\,\ell^{2}\,|\vec{p}|^{2}p^{2}\;,$

the question is to find hypercoordinates $X^{i}(x,p)$ that complement $P^{i}$
in a way that closes the Heisenberg algebra. As discussed at length in [1],
unlike the momenta $P^{i}(p)$, whose functional dependence on the “true”
(laboratory) momenta $p^{i}$ is dictated by the dispersion relation, the
functional form of the hypercoordinates on the laboratory phase-space
variables is not uniquely fixed by the algebraic requirements alone. Two
possible definitions were explored in [1]. Here we consider another, more
general, case:

$[X^{i},P^{j}]=\mathrm{i}\hbar\,F^{i\,j}(\vec{P})\;,\ \
[X^{i},X^{j}]=\mathrm{i}G^{i\,j}(\vec{P})\;,\ \ [P^{i},P^{j}]=0\;,$ (19)

where

$F^{i\,j}(\vec{P})\equiv
f^{i\,k}(\vec{P})\,\left((1-A\,|\vec{P}|-A^{2}\,|\vec{P}|^{2})\delta^{kj}-A\,\frac{P^{k}\,P^{j}}{|\vec{P}|}\,(1+A\,|\vec{P}|)\right)\;,$
(20)

with

$f^{ij}(\vec{P})=\delta^{ij}+\frac{1}{2}\,\ell\,(1+A|\vec{P}|)\left(\begin{array}[]{cc}-P^{1}&P^{2}\\\
P^{2}&P^{1}\\\
\end{array}\right)-\frac{1}{2}\,\ell^{2}\left(\begin{array}[]{cc}(P^{1})^{2}&P^{1}P^{2}\\\
P^{1}P^{2}&(P^{2})^{2}\\\ \end{array}\right)\;,$ (21)

being $G^{ij}$ an arbitrary antisymmetric function of $\vec{P}$. Besides zero,
the simplest choice for $G^{ij}$ is

$G^{ij}=L^{2}\,\epsilon^{ij}\,,$ (22)

where $\epsilon^{ij}$ is the totally antisymmetric tensor in two dimensions
and $L$ is a length scale. Since these calculations are $O(\ell^{2})$, length
scales of higher orders could not be appreciated, henceforth

$L(\ell)=a\,\ell\;,$ (23)

where $a$ is a $O(1)$ numerical constant that we set to $1$, for simplicity.

Note that this is the only possibility compatible with our requirements, apart
from the choice of [1], $G^{ij}=0$. If one wants a result proportional to
$|\vec{P}|$, this requires a factor of $\ell^{3}$ to compensate for the
physical dimension, which is bigger than $O(\ell^{2})$. If we include
$\ell/|\vec{P}|$, we do have the correct physical dimension, but $G^{ij}$
would not have a reasonable limit for $|\vec{P}|\to 0$.

Therefore we have

$[X^{i},X^{j}]=\mathrm{i}\,\theta^{ij}\,,$ (24)

where

$\theta^{ij}=\,\ell^{2}\,\epsilon^{ij}\,.$ (25)

This is the CNC we were looking for, realized in terms of the commutative
phase-space variables, $(\vec{x},\vec{p})$, by
$X_{1}=x_{1}-1/(2\hbar)\,\ell^{2}\,p_{2}$ and
$X_{2}=x_{2}+1/(2\hbar)\,\ell^{2}\,p_{1}$, i.e.,

$X^{i}=x^{i}-\frac{1}{2\hbar}\,\theta_{ij}p^{j}\,.$ (26)

This realization is known in the literature as the Bopp’s shift, see, e.g.,
[30].

##  IV Operational recipe for the analog noncommutativity

We now want to give the operational recipe to use graphene to test
noncommutative theories in an analog lab. Notice that this scheme could also
be applied to other analog systems.

First, we have to bear in mind that we only have experimental access to the
measurable “lab” variable $(\vec{x},\vec{p})$. The system is described by a
Hamiltonian $H(p)$, written in terms of $(\vec{x},\vec{p})$. This Hamiltonian
is very complicated, as it must include higher orders; see [1, 16] and (27)
later here. However, this is not an issue because we have access to
experiments. Hence, “nature makes the computations”, so it should be possible
to compute everything, at least numerically. To this, $H(p)$ corresponds to
$H(P)$ in (11), where the variables are the noncommutative $\vec{X}$.

With these, the operational recipe is

  * •

Measure observables in the lab, such as the spectrum, $E_{n}(x,p)$. We might
do so for the free system as a test and then apply a suitably engineered
potential, $V(x,p)$. Such potential should correspond, through $x_{i}(X,P)$
and $p_{i}(P)$ (see (26) and (18)), to a wanted potential, $V(X)$, in the
“target noncommutative system”.

  * •

Rewrite the outcomes of that measurement in the noncommuting variables, e.g.,
$E_{n}[x(X,P),p(X,P)]$, for the example of the spectrum.

  * •

Perform a calculation using the Hamiltonian $H(P)$, first for the free case
and then adding the potential $V(X)$ to it. The free case is only for setting
up/verifying the correspondence. We want to study the interacting case to test
the model based on noncommuting $\vec{X}$. We then obtain the observables,
e.g., $E_{n}(X,P)$. Here, we are entirely theoretical/mathematical.

  * •

Compare the measured observables with the theoretical prediction based on the
noncommutative theory. E.g., $E_{n}[x(X,P),p(X,P)]$, obtained experimentally,
with $E_{n}(X,P)$, obtained from the noncommutative theory.

Notice that we shall have two sets of data written in the $(\vec{X},\vec{P})$
variables: one set is obtained by rewriting the results of the experiments and
calculations done with the setting $H(p)$, $(\vec{x},\vec{p})$. The other set
is obtained by performing the (noncommutative) calculations within the setting
$H(P)$, $(\vec{X},\vec{P})$.

The above recipe can be sketched in the following diagram

$(\vec{x},\vec{p});H(x,p)$$E_{n}(H(x,p))$$(\vec{X},\vec{P});H(X,P)$$E_{n}(H(X,P))$experimentNC
analogcomparisoncalculation

For instance, for the free case (no external potential), a typical Hamiltonian
that departs from the linear approximation, given in lab variables
$(\vec{x},\vec{p})$, is [1]

$\displaystyle H$ $\displaystyle=$ $\displaystyle
v_{F}\sum_{\vec{p}}\psi^{\dagger}_{\vec{p}}\Bigg{[}\sigma_{1}\left(p_{1}-\frac{\ell}{4}(p^{2}_{1}-p^{2}_{2})-\frac{\ell^{2}}{8}p_{1}(p^{2}_{1}+p^{2}_{2})\right)+\sigma_{2}\left(p_{2}+\frac{\ell}{2}p_{1}p_{2}-\frac{\ell^{2}}{8}p_{2}(p^{2}_{1}+p^{2}_{2})\right)$
(27) $\displaystyle-$
$\displaystyle\frac{3}{2}A\left((p^{2}_{1}+p^{2}_{2})-\frac{\ell}{2}p_{1}^{3}+\frac{3\ell}{2}p_{1}p^{2}_{2}\right)\Bigg{]}\psi_{\vec{p}}\;.$

To this corresponds the free Hamiltonian (11),
$H(P)=V_{F}\,\sum_{k}\,\psi^{\dagger}_{k}\,\not{P}\,\psi_{k}$, written in
terms of the hypervariables.

##  V Applying the recipe in a specific setup

The construction of physical theories pertaining to the deformed commutator in
(24) has a long and rich tradition, see, e.g., [25, 31, 24, 32], among many
other references. Such noncommutativity relies on the mapping between algebras
of functions of the commuting coordinates (standard) and functions of
noncommuting coordinates. The quintessential ingredient is the commutator (24)
itself [25].

In this section, we shall provide an example of a possible application of the
recipe. Given the generality of the results of this letter, we shall not do
that by focusing on a full-fledged quantum electrodynamic calculation. The
latter would need to face the subtleties of having to deal with massless
particles, an issue that points to the Klein paradox, somehow faced in
graphene with deformed algebras already, albeit in a commutative context [33].
Future work on the noncommutative Klein paradox, within the approach presented
here, has been planned [34].

What we shall do, instead, is to focus on kinematics, which is the most
straightforward way to test the novel noncommutativity introduced in this
work, Eq. (24). This does not require the application of the full dynamics of
the deformed field theory to describe the processes. We only need to recognize
that position operators represent generators of infinitesimal displacements on
the momentum space. Since they do not commute, this also implies that finite
displacements of the momenta will only commute when collinear, but in general,
they do not commute. The action of the displacement of electrons’ momentum
represents the absorption or emission of a photon. With this in mind, the
example we single out here will convey general information common to many
dynamical theories.

The noncommutativity then can be observed experimentally in the following way.
Let us assume an electron with a fixed initial value of the momentum,
$\vec{P}$, absorbs two non-collinear photons momenta $\vec{Q}$ and
$\vec{Q^{\prime}}$, in succession. In the commutative case, the final value of
electron momentum, $\vec{K}$, does not depend on the order in which the two
photons are absorbed. In the noncommutative case, the ordering will make a
difference due to the noncommutativity of the momenta addition law, which is a
direct consequence of (24). Thus, the two final momenta of the electron
$\vec{K}$ and $\vec{K^{\prime}}$ are generally different for the processes
depicted in the figure below.

$\vec{P}$$\vec{Q}$$\vec{Q^{\prime}}$$\vec{K}$$\vec{P}$$\vec{Q^{\prime}}$$\vec{Q}$$\vec{K^{\prime}}$
Figure 1: An electron of the initial momentum $\vec{P}$ absorbs two photons of
the momenta $\vec{Q}$ and $\vec{Q^{\prime}}$ in the opposite order in diagrams
on the left and the right. The resulting momenta of the electron are, in
general, different in the two cases, $\vec{K}\neq\vec{K^{\prime}}$.

The experimental setting required to achieve this is the following. One needs
two sets of electrons prepared in the same initial state of the definite
momenta $\vec{P}$, which are exposed to two consecutive short monochromatic
non-collinear electromagnetic pulses. The interaction of light with the
$\pi$-electrons of graphene is well-studied these days, a noticeable example
being the theoretical and experimental studies on the laser-graphene
interaction [35], see also [36].

The order of the shooting of photons on the two sets must be as in Fig. 1.
Finally, one measures the difference between the momenta of the electrons in
the two final sets.

Now, the general form of the momentum addition law is not familiar for the
noncommutativity defined by relation (24). One can, however, on purely
dimensional grounds, determine the leading order form

$\vec{Q}\oplus\vec{P}=(1+a_{1}\theta_{jk}P_{j}Q_{k})\vec{P}+(1+a_{2}\theta_{jk}P_{j}Q_{k})\vec{Q}\,.$
(28)

The symbol $\oplus$ represents the fact that the momentum conservation law
gets deformed; we follow a convention where the momentum on the left displaces
the momentum on the right; that is, the electron of the momentum $\vec{P}$
absorbs (gets displaced by) the photon of the momentum $\vec{Q}$.222The
development of the field theory of identical particles, such as scalar
$\phi^{n}$ theory, on the noncommutative background, suffers from
insurmountable ambiguities related to their indistinguishability. This,
however, does not represent a physical constraint since no known elementary
particles emit or absorb like particles. The difference in the coefficients
$a_{1}$ and $a_{2}$ is the measure of noncommutativity of the momenta addition
law. For the two processes described above, we have then,

$\displaystyle\vec{K}$
$\displaystyle=\vec{Q^{\prime}}\oplus(\vec{Q}\oplus\vec{P})\;,$

and

$\displaystyle\vec{K}^{\prime}$
$\displaystyle=\vec{Q}\oplus(\vec{Q^{\prime}}\oplus\vec{P})\;.$

After some algebra, we can express the difference $\vec{K^{\prime}}-\vec{K}$
in a convenient expression that, in vectorial notation, reads

$\vec{K^{\prime}}-\vec{K}=\ell^{2}\,(Q^{\prime}_{1}\,Q_{2}-Q_{1}\,Q^{\prime}_{2})\,\left(3a_{1}\vec{P}+(a_{1}+a_{2})(\vec{Q}+\vec{Q^{\prime}})\right)\;.$
(29)

It is clear from Eq. (29) that the difference is zero when $\vec{Q}$ and
$\vec{Q^{\prime}}$ are collinear and that the leading order correction in the
length parameter is $\ell^{2}$.

Now, the crucial part of the story: Suppose that the light interacting with
graphene can probe nonlinear regimes of the dispersion relations for the
$\pi$-electrons, as discussed in [36]. This nonlinearity, within the framework
given here, see also [1], can be transferred to the new phase space variables,
$(\vec{X},\vec{P})$, with $\vec{X}$ given by (26), and noncommuting, as
opposed to the phase space variables, $(\vec{x},\vec{p})$, which we actually
measure in the lab. Suppose the results of the above-described experiments are
now written in terms of $(\vec{X},\vec{P})$. In that case, we can check
whether the predictions of the theoretical arguments given above, based on
noncommutativity, indeed occur!

It is vital to notice two facts. First, if we use $(\vec{x},\vec{p})$, we
shall only see some nonlinear effects, and no noncommutativity can be there.
Second, even in the new variables, $(\vec{X},\vec{P})$, it is by no means
trivial that we see the noncommutativity. We may as well not see any
noncommutativity at all, even in the new variables! It is then a real test of
noncommutativity that we are performing by a clever change of phase space
coordinates, in part dictated by the detailed form of the dispersion relations
of the material, in part dictated by the presence of a length scale, given by
the lattice spacing, $\ell$.

##  VI Conclusions and outlook

We discussed a conceptually simple realization of a phase space whose
variables close a deformed Heisenberg algebra

$[X^{i},P^{j}]=\mathrm{i}\hbar\,F^{i\,j}(\vec{P})\;,\ \
[X^{i},X^{j}]=\mathrm{i}\theta^{ij}\;,\ \ [P^{i},P^{j}]=0\,.$ (30)

Deformations of the Heisenberg algebra are usually studied in the formalism of
Hopf algebras and noncommutative geometry [37, 38]. The deformation given
above has not been studied yet in this context. It would be interesting, from
a mathematical point of view, to investigate the elements of the
noncommutative geometry, such as co-product, star product, and the deformation
of the energy-momentum conservation (the momenta addition rule). From a
physical point of view, the most interesting question is that of the structure
of space-time following from the full algebra (30), given that the first
commutator, while keeping the other two trivial, gives rise to a minimal
length [18], whereas the second commutator, while keeping the remaining two
trivial, leads to the uncertainty compatible with the constraints on the
position measurement that are imposed by the (classical) gravity [23, 39].

Albeit powerful and elegant, though, the mathematical formalism alone cannot
resolve the inherent ambiguities when trying to define dynamics that would
correspond to the given particular deformation. Given the results of this
letter, graphene could be used not only to confirm theoretical models, but to
guide the construction of the physical theory itself, by providing
experimental constraints. Indeed, the Dirac-like dynamics resists beyond the
lowest energy approximation of the conductivity electrons of graphene, hence a
“quantum-gravity-corrected” emergent field theory is there at our disposal.
One must simply use new phase-space variables $(\vec{X},\vec{P})$ that enjoy
the generalized Heisenberg algebra (30), where noncommuting coordinates,
$\vec{X}$, obeying $[X^{i},X^{j}]=\mathrm{i}\,\ell^{2}\,\epsilon^{ij}$, are
very natural.

On the one hand, the effect here is enormous compared to the one that might
take place at the fundamental level, $\Delta X^{1}\,\Delta
X^{2}\geq\ell^{2}/2\sim 10^{50}\,\,\ell_{Planck}^{2}$. On the other hand,
there is no need for a magnetic length, $\ell_{B}=\sqrt{\hbar c/eB}$, as often
required for noncommutativity in similar set-ups [31], nor do we need
localization processes that create micro black holes [23].

As such, our findings are very general, macroscopic, and easy to obtain.
Hence, they offer a reliable and robust way to test noncommutative theories in
tabletop analog experiments. Here we propose both the general recipe for the
practical implementation of this approach and a specific arrangament that
illustrates how to apply that recipe.

## Acknowledgements

We are indebted to Salvatore Mignemi for the many discussions on
noncommutativity and related topics. A. I. and P. P. gladly acknowledge
support from Charles University Research Center (UNCE 24/SCI/016). P. P. is
also supported by Fondo Nacional de Desarrollo Científico y Tecnológico–Chile
(Fondecyt Grant No. 3200725).

## References

  * Iorio  _et al._ [2022] A. Iorio, B. Ivetić, S. Mignemi, and P. Pais, Physical Review D 106, 116011 (2022).
  * Feynman  _et al._ [1963] R. Feynman, R. Leighton, and M. Sands,  _The Feynman Lectures on Physics_, Addison-Wesley world student series No. v. 2 (Addison-Wesley, 1963).
  * Iorio [2019] A. Iorio, Journal of Physics: Conference Series 1275, 012013 (2019).
  * Unruh [1981] W. G. Unruh, Phys. Rev. Lett. 46, 1351 (1981).
  * Unruh [1976] W. G. Unruh, Phys. Rev. D 14, 870 (1976).
  * Hawking [1975] S. W. Hawking, Communications in Mathematical Physics 43, 199 (1975).
  * Muñoz de Nova  _et al._ [2019] J. R. Muñoz de Nova, K. Golubkov, V. I. Kolobov, and J. Steinhauer, Nature 569, 688 (2019).
  * Gooth  _et al._ [2017] J. Gooth, A. C. Niemann, T. Meng, A. G. Grushin, K. Landsteiner, B. Gotsmann, F. Menges, M. Schmidt, C. Shekhar, V. Süß, R. Hühne, B. Rellinghaus, C. Felser, B. Yan, and K. Nielsch, Nature 547, 324 (2017).
  * Iorio [2011] A. Iorio, Ann. Phys. 326, 1334 (2011).
  * Iorio [2102] A. Iorio, Eur. Phys. J. Plus 127, 156 (2102).
  * Iorio and Lambiase [2012] A. Iorio and G. Lambiase, Physics Letters B 716, 334 (2012).
  * Iorio and Lambiase [2014] A. Iorio and G. Lambiase, Phys. Rev. D 90, 025006 (2014).
  * Iorio [2015] A. Iorio, International Journal of Modern Physics D 24, 1530013 (2015), https://doi.org/10.1142/S021827181530013X .
  * Barceló  _et al._ [2005] C. Barceló, S. Liberati, and M. Visser, Living Reviews in Relativity 8, 12 (2005).
  * Castro Neto  _et al._ [2009] A. H. Castro Neto, F. Guinea, N. M. R. Peres, K. S. Novoselov, and A. K. Geim, Rev. Mod. Phys. 81, 109 (2009).
  * Iorio  _et al._ [2018] A. Iorio, P. Pais, I. A. Elmashad, A. F. Ali, M. Faizal, and L. I. Abou-Salem, International Journal of Modern Physics D 27, 1850080 (2018), https://doi.org/10.1142/S0218271818500803 .
  * Iorio and Pais [2019] A. Iorio and P. Pais, J. Phys. Conf. Ser. 1275, 012061 (2019), arXiv:1902.00116 [hep-th] .
  * Das and Vagenas [2008] S. Das and E. C. Vagenas, Phys. Rev. Lett. 101, 221301 (2008).
  * Ali  _et al._ [2011a] A. F. Ali, S. Das, and E. C. Vagenas, Phys. Rev. D84, 044013 (2011a), arXiv:1107.3164 [hep-th] .
  * Ali  _et al._ [2009] A. F. Ali, S. Das, and E. C. Vagenas, Phys. Lett. B678, 497 (2009).
  * Das  _et al._ [2010] S. Das, E. C. Vagenas, and A. F. Ali, Phys. Lett. B690, 407 (2010).
  * Ali  _et al._ [2011b] A. F. Ali, S. Das, and E. C. Vagenas, Phys. Rev. D 84, 044013 (2011b).
  * Doplicher  _et al._ [1995] S. Doplicher, K. Fredenhagen, and J. E. Roberts, Communications in Mathematical Physics 172, 187 (1995).
  * Seiberg and Witten [1999] N. Seiberg and E. Witten, Journal of High Energy Physics 1999, 032 (1999).
  * Madore  _et al._ [2000] J. Madore, S. Schraml, P. Schupp, and J. Wess, Eur. Phys. J. C 16, 161 (2000), arXiv:hep-th/0001203 .
  * Iorio and Sỳkora [2002] A. Iorio and T. Sỳkora, International Journal of Modern Physics A 17, 2369 (2002).
  * Iorio [2008] A. Iorio, Physical Review D 77, 048701 (2008).
  * Iorio  _et al._ [2020] A. Iorio, G. Lambiase, P. Pais, and F. Scardigli, Physical Review D 101, 105002 (2020).
  * Iorio and Pais [2018] A. Iorio and P. Pais, Annals of Physics 398, 265 (2018).
  * Curtright  _et al._ [1998] T. Curtright, D. Fairlie, and C. K. Zachos, Phys. Rev. D 58, 025002 (1998), arXiv:hep-th/9711183 .
  * Jackiw [2002] R. Jackiw, Nucl. Phys. B Proc. Suppl. 108, 30 (2002), arXiv:hep-th/0110057 .
  * Iorio [2007] A. Iorio, Journal of Physics: Conference Series 67, 012008 (2007).
  * Shah  _et al._ [2022] N. A. Shah, A. Contreras-Astorga, F. m. c. Fillion-Gourdeau, M. A. H. Ahsan, S. MacLean, and M. Faizal, Phys. Rev. B 105, L161401 (2022).
  * [34] A. Iorio, B. Ivetić, and P. Pais, Work in progress .
  * Higuchi  _et al._ [2017] T. Higuchi, C. Heide, K. Ullmann, H. B. Weber, and P. Hommelhoff, Nature 550, 224 (2017).
  * Ciappina  _et al._ [2020] M. F. Ciappina, A. Iorio, P. Pais, and A. Zampeli, Phys. Rev. D 101, 036021 (2020).
  * Majid [2000] S. Majid,  _Foundations of Quantum Group Theory_ (Cambridge University Press, 2000).
  * Connes [1994] A. Connes,  _Noncommutative Geometry_ (Academic Press, 1994).
  * Mead [1964] C. A. Mead, Phys. Rev. 135, B849 (1964).


# Leveraging Large Language Models to Measure
Gender Bias in Gendered Languages

Erik Derner1, Sara Sansalvador de la Fuente1, Yoan Gutiérrez2,
Paloma Moreda2, Nuria Oliver1
1 ELLIS Alicante, Alicante, Spain 2 University of Alicante, Alicante, Spain
Correspondence<EMAIL_ADDRESS>
###### Abstract

Gender bias in text corpora used in various natural language processing (NLP)
contexts, such as for training large language models (LLMs), can lead to the
perpetuation and amplification of societal inequalities. This is particularly
pronounced in gendered languages like Spanish or French, where grammatical
structures inherently encode gender, making the bias analysis more
challenging. Existing methods designed for English are inadequate for this
task due to the intrinsic linguistic differences between English and gendered
languages. This paper introduces a novel methodology that leverages the
contextual understanding capabilities of LLMs to quantitatively analyze gender
representation in Spanish corpora. By utilizing LLMs to identify and classify
gendered nouns and pronouns in relation to their reference to human entities,
our approach provides a nuanced analysis of gender biases. We empirically
validate our method on four widely-used benchmark datasets, uncovering
significant gender disparities with a male-to-female ratio ranging from 4:1 to
6:1. These findings demonstrate the value of our methodology for bias
quantification in gendered languages and suggest its application in NLP,
contributing to the development of more equitable language technologies.

Leveraging Large Language Models to Measure
Gender Bias in Gendered Languages

Erik Derner1, Sara Sansalvador de la Fuente1, Yoan Gutiérrez2, Paloma Moreda2,
Nuria Oliver1 1 ELLIS Alicante, Alicante, Spain 2 University of Alicante,
Alicante, Spain Correspondence<EMAIL_ADDRESS>

##  1 Introduction

Figure 1: Overview of the proposed method for gender bias detection and
measurement in gendered languages using LLMs.

In recent years, the presence of social biases in machine learning models
(Barocas et al., 2019) has gained significant attention due to their potential
to perpetuate and amplify existing inequalities, impacting areas of great
consequence in people’s lives, such as hiring practices (Raghavan et al.,
2020), law enforcement (Babuta and Oswald, 2019), healthcare (Panch et al.,
2019), and everyday digital interactions. Among various forms of bias, gender
bias is particularly concerning because it affects roughly half of the global
population and has pervasive effects across different sectors of society. This
concern is amplified in the area of natural language processing (NLP), where
biases present in training corpora can propagate through machine learning
models and impact a broad range of applications.

In this work, we consider bias in text corpora used to train large language
models (LLMs). The main motivation for studying bias in training data of LLMs
stems from the lifecycle of these models. From training to deployment, biases
in the training data can not only be perpetuated but even amplified by the
models (Kotek et al., 2023). This data often comes from diverse sources,
including books, websites, social media, and other digital texts. If these
sources contain biases – which they frequently do, reflecting societal
prejudices and stereotypes – the models learn these biases as part of their
training process. Hence, the first stage – data curation – becomes of
paramount importance. It involves selecting, cleaning, and annotating the
training data to minimize biases as much as possible. Effective data curation
requires a nuanced understanding of the types of biases that can exist and the
methods to mitigate them, including removing or balancing biased content,
ensuring diverse and representative data samples, and incorporating fairness
and ethical guidelines in the data selection process (Gallegos et al., 2024).

However, existing methods for evaluating bias in text, such as gender polarity
(Dhamala et al., 2021) and co-occurrences (Bordia and Bowman, 2019), have been
primarily developed for English. These approaches often rely on the neutrality
of the English language in terms of grammatical gender, allowing for
relatively straightforward identification and categorization of gendered
expressions. Nonetheless, such an approach is not directly applicable to
gendered languages like Spanish, where nouns, articles, adjectives and other
word forms reflect gender, making the detection and analysis of gender bias a
much harder problem to solve.

Given the limitations of existing methods and the prevalence of gendered
languages in the world, the focus of this paper is to develop a novel and
robust method to detect gender bias in corpora used to train LLMs in Spanish,
a representative gendered language spoken by over 500 million people across
diverse cultures and regions in the world. In Spanish, every noun carries a
gender marker but only the words that refer to human beings are reflective of
a potential gender bias.

We propose a novel methodology that leverages state-of-the-art LLMs to detect
and analyze gender bias in Spanish text corpora. Unlike traditional methods
that rely on part-of-speech tagging or specific machine learning models to
detect person references, our approach benefits from the contextual
understanding capabilities of LLMs. By prompting LLMs to identify and classify
nouns and pronouns by their reference to persons and their grammatical gender,
we introduce a method that respects the linguistic structure of Spanish while
providing the necessary depth of analysis.

We first provide an overview of the most relevant literature on biases in NLP
systems in English and non-English languages, followed by a detailed
description of the proposed methodology. We empirically evaluate the proposed
method on several benchmark datasets and uncover significant levels of gender
imbalance with a male:female ratio ranging between 4:1 and 6:1, depending on
the dataset. Interestingly, the English translations of the same corpora
exhibit a male:female gender ratio of 1:1 to 3.5:1. Our results illustrate the
importance of developing novel methods to detect gender bias in text corpora
in gendered languages, such as Spanish. We conclude with a discussion,
limitations, ethical implications, and lines of future work.

##  2 Related Work

#### Biases in NLP systems

There is a growing body of literature on biases in NLP systems, which has been
summarized in a number of surveys (Blodgett et al., 2020; Stańczak and
Augenstein, 2021; Bansal, 2022). In particular, Blodgett et al. (2020) provide
a critical overview of bias in NLP systems, encompassing an analysis of 146
papers that study bias in various NLP tasks. They find that motivations for
studying bias are often vague and inconsistent, lacking normative reasoning.
For example, studies on embedding spaces (Bolukbasi et al., 2016; Caliskan et
al., 2017; Gonen and Goldberg, 2019) illustrate biases but often do not
critically engage with what constitutes bias. Similarly, works on language
modeling (Lu et al., 2020; Bordia and Bowman, 2019), coreference resolution
(Rudinger et al., 2018; Zhao et al., 2018), machine translation (Vanmassenhove
et al., 2018; Stanovsky et al., 2019), and sentiment analysis (Kiritchenko and
Mohammad, 2018) highlight bias in NLP but vary widely in their
conceptualizations and methodologies. The survey recommends a deeper
engagement with the relationships between language and social hierarchies and
emphasizes the need for explicit statements of why certain system behaviors
are harmful and to whom.

Recent advances in addressing biases in LLMs include various techniques
focusing on detecting biases in the embeddings, prompt-based interactions, and
contextual adaptations (Gallegos et al., 2024). However, we are not aware of
any works aimed at the detection of bias in the training corpora, which is our
focus.

Bias quantification approaches proposed to date have addressed three main
areas: metrics for bias evaluation, datasets for bias evaluation, and methods
leveraging generated text (Gallegos et al., 2024). Embedding-based metrics,
such as the Sentence Encoder Association Test (May et al., 2019) and
Contextualized Embedding Association Test (Guo and Caliskan, 2021), extend
traditional bias metrics to contextualized embeddings. Probability-based
metrics like pseudo-log-likelihood scores (Salazar et al., 2020) evaluate the
likelihood of stereotypical versus anti-stereotypical sentences, while prompt-
based metrics, including co-occurrence bias scores and sentiment analysis,
measure bias in the generated text by comparing the distribution of terms
associated with different social groups (Bordia and Bowman, 2019; Dhamala et
al., 2021). These techniques provide comprehensive frameworks for assessing
bias, ensuring a more nuanced and accurate understanding of how biases
manifest in LLM outputs.

#### Biases in languages other than English

From a language perspective, most existing research on bias in NLP has focused
on English. For instance, Dhamala et al. (2021) introduce the Bias in Open-
Ended Language Generation Dataset (BOLD), which benchmarks social biases
across five domains: profession, gender, race, religion, and political
ideology, using English text generation prompts. Similarly, Webster et al.
(2018) address gender bias in coreference resolution with the Gendered
Ambiguous Pronouns (GAP) corpus, a gender-balanced dataset of ambiguous
pronoun-name pairs.

Efforts to evaluate bias in non-English languages include the CrowS-pairs
dataset, initially designed for English by Nangia et al. (2020), which
provides sentence pairs to evaluate nine types of bias. Recognizing the need
for multilingual approaches, Névéol et al. (2022) extended CrowS-pairs to
French, creating 1,677 sentence pairs to cover stereotypes across ten types of
bias, including gender and age. This extension highlighted the prevalence of
biases in language models across different cultural contexts.

Other studies have explored gender bias in both English and German, revealing
that methods effective for detecting bias in English may not apply as well to
other languages (Bartl et al., 2020). Along those lines, a broader survey by
Ramesh et al. (2023) emphasizes the need for bias evaluation and mitigation in
language models beyond English, highlighting the challenges of adapting bias
detection methods to multilingual contexts. This growing body of research
underscores the importance of developing and refining techniques to evaluate
and mitigate biases in various languages, ensuring the equitable performance
of NLP systems globally.

#### Our contributions

While there is a body of literature on the impact of biases on various machine
learning algorithms or studies related to stereotypes represented in datasets,
to the best of our knowledge, there are no studies that have analyzed how to
measure gender bias in texts written in gendered languages from the point of
view of the prevalence of words referring to each gender. Addressing this gap
is the focus of our research.

Our core contribution can be contextualized in a pyramid of methodologies to
automatically detect gender bias in text corpora. Traditional word-counting
methods would be located at the base of the pyramid. Unfortunately, their
simplicity prevents their applicability to gendered languages like Spanish.
Above this, we would find conventional NLP techniques, such as part-of-speech
tagging. However, these methods might not reliably distinguish between
gendered references of people versus animals or inanimate objects. At the apex
of our methodological pyramid, we position our proposed LLM-based approach
that not only identifies gendered words but also classifies them according to
their role in conveying gender-related information within the text.
Ultimately, we envision the use of these types of AI tools to automate bias
evaluation with a comprehensive understanding of the underlying linguistic and
social contexts.

This paper aims to shed light on the inherent challenges posed by gendered
languages and propose a robust framework for detecting and measuring gender
bias in Spanish text corpora. The insights derived from this study are
expected to guide the development of fairer and more equitable language
technologies, particularly in gendered languages.

##  3 Methodology

This section presents our methodology for assessing gender bias in language
corpora. First, we describe a gender polarity method from the literature that
has been proposed to measure gender-specific terms in English texts. We build
upon this approach to propose a novel gender bias quantification method
leveraging the LLMs’ natural language comprehension power to accommodate the
complexities of gendered languages, using Spanish as a case study.

###  3.1 Gender Polarity

Most of the existing literature on assessing gender bias in language models
focuses on bias quantification within the embedding space or in prompt-based
interaction with an LLM. However, the scope of this paper is to measure bias
in the LLM training data. The most relevant approach is presented in Dhamala
et al. (2021), which introduces gender polarity as a method to quantify the
presence of gender-specific language in a given text. The authors propose two
metrics to evaluate gender polarity. The first one, _unigram matching_ ,
involves a straightforward count of gender-specific tokens (words) from a
predefined list. The tokens are defined as follows:

  * •

Male tokens: he, him, his, himself, man, men, he’s, boy, boys.

  * •

Female tokens: she, her, hers, herself, woman, women, she’s, girl, girls.

The second metric proposed in Dhamala et al. (2021) employs word embeddings to
assess the proximity of words to a gendered vector space. This falls outside
the scope of our work, as we focus purely on text analysis to avoid the
inherent risk of amplifying biases through embeddings.

While these metrics were designed to evaluate text generation models in
prompt-based interactions, specifically on the BOLD dataset (Dhamala et al.,
2021), we propose extending the application of _unigram matching_ , further
referred to as the _gender polarity_ , to quantify gender bias in larger text
corpora. This approach facilitates more comprehensive assessments of gender
representation in linguistic datasets.

In a given text, the number of male tokens (denoted as $G_{M}$) and the number
of female tokens ($G_{F}$) are counted. The gender bias in the text can then
be expressed as the ratio $G_{M}:G_{F}$. The gender polarity method was
designed for English texts, where gender differentiation in language usage is
mostly captured through distinct pronouns and other gender-specific tokens.
Nonetheless, this approach cannot be directly applied to gendered languages,
such as Spanish, by merely translating the token lists to the target language.
In gendered languages, specific parts of speech, such as nouns or adjectives,
have inherent gender properties that often require different forms based on
the gender of the subjects or objects they describe. This leads to a more
complex interaction between grammatical gender and semantic content,
necessitating a tailored method capable of accounting for the intrinsic gender
agreement rules in such languages.

###  3.2 Measuring Gender Bias in Gendered Languages

We close the gap in bias quantification for gendered languages by proposing a
method that takes inspiration from the gender polarity analysis and
accommodates the specific grammatical and semantic features in gendered
languages. We empirically evaluate the method on Spanish as an example of a
gendered language.

The following pipeline summarizes the main steps that compose the proposed
framework:

1\. Identification of all nouns and pronouns in a given text, which is needed
to consider all potentially gendered language elements, as these are the
primary carriers of gender information in Spanish.

2\. Classification of each identified noun and pronoun with respect to whether
it refers to a person (P) or not (N) to focus on human-related references,
which reflect gender representation.

3\. Determination of the grammatical gender (masculine, M; feminine, F) of
each identified word, as it directly correlates to the perceived or intended
gender of the person being referenced.

Note that we have consciously decided not to include adjectives, as they adapt
their form to agree with the noun they modify, leading to redundancy in our
analysis if those nouns are already accounted for. Furthermore, adjectives
alone do not necessarily indicate the gender of the person being described
unless paired directly with specific gendered nouns or pronouns, rendering the
classification in step 2 impossible in many cases. Therefore, including
adjectives could potentially negatively impact our method’s accuracy by
introducing elements more reflective of grammatical structure rather than
gender representation.

This structured approach enables a comprehensive analysis of gender
representation in text, differentiating between words that refer to people and
other types of words, which is crucial for gendered languages.

###  3.3 Gender Bias Quantification Using LLMs

Implementing the pipeline described in the previous section using classical
NLP methods would typically involve a combination of tools, leveraging part-
of-speech tagging for step 1 and gender classification based, for instance, on
word endings and dictionary lookups for step 3. However, accurately
determining whether a noun or pronoun refers to a person (step 2) would
require more sophisticated semantic analysis tools.

Given these challenges, we propose to leverage LLMs for their proficiency in
understanding natural language nuances and context. LLMs can seamlessly
integrate all steps of the proposed pipeline. After a process of manual
interactive prompt-tuning, we designed a prompt that instructs the LLM to
perform noun and pronoun identification, determine if these refer to human
beings, and classify their grammatical gender, all in a single query. This
approach utilizes the LLM’s ability to parse and interpret complex language
structures and perform multiple tasks simultaneously.

The English translation of our prompt, originally crafted in Spanish (see
Appendix A), is as follows:

_< EXAMPLES>_
_Sentence: <SENTENCE>_
_Identify all nouns and pronouns in the given sentence. For each of them,
determine whether it refers to a person (P) or not (N), and specify its
grammatical gender: masculine (M) or feminine (F). Exclude surnames. Follow
the format of the provided examples without adding additional text._

We adopt a few-shot prompting strategy, showing the LLM a number of example
sentences along with their analyses to prime it for the task, see Figure 1.
This methodology enhances the model’s performance by providing clear examples
of the desired output format and analysis procedure. Surnames are explicitly
excluded from the analysis because they do not convey gender information in
Spanish.

To analyze the gender representation in a given dataset, it is parsed sentence
by sentence and each sentence is evaluated using the aforementioned prompt. We
introduce the notation $L_{p,g}$, which denotes the number of words identified
by our LLM-based approach, where $p\in\\{P,N\\}$ and $g\in\\{M,F\\}$ are
binary variables that capture whether the word refers to a person ($P$) or not
($N$) and the word’s grammatical gender ($M$ for masculine, $F$ for feminine),
respectively. Analogously to the gender polarity method described in Section
3.1, the gender bias in the given text is expressed by the ratio
$L_{P,M}:L_{P,F}$.

Advanced state-of-the-art LLMs are used to perform this task, specifically
various versions of OpenAI’s ChatGPT. These models have demonstrated
significant capabilities in solving complex tasks involving nuanced language
understanding and have been trained on diverse linguistic datasets. Given the
intricacies of gendered languages and the complexity of the task, more
advanced models from the GPT-4 family offer substantial advantages in accuracy
and reliability over smaller or older model architectures.

##  4 Experiments

This section describes the experimental setup used to validate our methodology
and to assess the extent of gender bias across various benchmark datasets. We
utilize bilingual corpora to report the results using our LLM-based approach
to quantify gender bias in Spanish alongside the gender polarity method for
English texts.

Table 1: Performance of different LLMs on the validation dataset. The values correspond to the means and standard deviations calculated from five repetitions of the evaluation for each model. Model | Accuracy (%) | Precision (%) | Recall (%) | F-score (%)
---|---|---|---|---
gpt-4-turbo-preview | $85.68\pm 0.93$ | $87.51\pm 0.49$ | $86.58\pm 0.90$ | $87.04\pm 0.61$
gpt-4o | $87.57\pm 1.21$ | $80.45\pm 1.35$ | $89.31\pm 1.19$ | $84.65\pm 1.26$
gpt-4-turbo | $\mathbf{89.40\pm 0.98}$ | $\mathbf{89.53\pm 0.56}$ | $\mathbf{90.96\pm 0.72}$ | $\mathbf{90.24\pm 0.55}$

###  4.1 Datasets

We evaluate the proposed method on several parallel datasets from the OPUS
Machine Translation project dataset collection111The website specifies that
the data provided on the platform are believed to be free to redistribute.
(Tiedemann, 2012) that includes aligned translations between Spanish and
English. We focus on parallel datasets to facilitate the comparison of the
gender polarity method for English with our LLM-based gender bias
quantification method for Spanish. The datasets were chosen based on their
extended use in machine translation and NLP model training in general. In
particular, we evaluated our approach on the Europarl, CCAligned, Global
Voices, and WMT-News datasets.

#### Europarl:

The Europarl dataset (Koehn, 2005) is a multilingual corpus extracted from the
proceedings of the European Parliament. It is widely used for research in the
NLP community, particularly for tasks such as machine translation, language
modeling, and cross-lingual studies. It includes transcripts in 21 European
languages, including Spanish and English. The texts cover parliamentary
debates and speeches, offering a diverse range of topics and formal language
styles. We use the bilingual Spanish-English dataset in version v7, covering
the period from April 1996 to November 2011, which consists of 1,965,734
sentences for each language.

#### CCAligned:

This dataset (El-Kishky et al., 2020) is a large-scale multilingual corpus of
billions of sentences derived from web-crawled data. Specifically, the corpus
used in our experiments was created from 68 Common Crawl snapshots up until
March 2020 (v1). It is designed to support research in tasks involving machine
translation and cross-lingual studies. The bilingual Spanish-English corpus
contains 15,250,346 sentences for each language.

#### Global Voices:

The Global Voices dataset (Nguyen and Daumé III, 2019) is a multilingual
corpus collected from the Global Voices website, which features news articles
and stories written by a global network of authors and translated by
volunteers into multiple languages. This dataset, consisting of 359,002
sentence pairs in Spanish and English, is broadly used for research in machine
translation, multilingual text analysis, and other NLP tasks. We used the
version v2018q4.

#### WMT-News:

The WMT-News dataset is a collection of parallel corpora specifically curated
for machine translation and related NLP tasks. It is associated with the
annual Conference on Machine Translation (WMT, originally Workshop on Machine
Translation), which organizes shared tasks to benchmark and advance the state
of the art in machine translation. The version we used, v2019, contains 14,522
Spanish-English sentence pairs.

We created two representative subsets of 1,000 randomly selected sentence
pairs from each of these datasets. The choice of this sample subset size was
motivated by the lower bound calculated using the formula from Daniel and
Cross (2018): $n=z^{2}p(1-p)/e^{2}$, where $z=2.576$ for a confidence level
$\alpha=0.99$, margin of error $e=0.05$, and population proportion $p=0.5$
(worst-case scenario), yielding $n=664$. Note that this sample size is aligned
with the sample size used in various NLP studies (Kreutzer et al., 2022;
Ranathunga et al., 2024). Using two subsets of each dataset helps to balance
achieving a certain level of generalization of the results and managing the
computational costs involved.

###  4.2 Validation

To first validate the proposed method, the author team created a manually
annotated dataset consisting of 100 sentences in Spanish extracted from the
Europarl dataset. This validation dataset serves as the ground truth for
assessing and comparing the performance of different LLMs and their variants,
and it also allows for efficiently measurable prompt engineering.

We evaluated the following LLMs available through the OpenAI API: gpt-4-turbo-
preview (gpt-4-0125-preview), gpt-4o (gpt-4o-2024-05-13), and gpt-4-turbo
(gpt-4-turbo-2024-04-09). For each of them, we executed five runs of our
method with the same (default) configuration. For few-shot prompting, we used
five sentence-analysis pairs listed in Appendix B. The budget spent on these
experiments was approximately 10 USD.

To evaluate the correctness of the LLM output, we employed a case-insensitive
comparison of the identified words and the (mis)match of the two attributes
($p$ and $g$, see Section 3.3) w.r.t. the ground truth. We report the number
of words that were correctly identified and correctly classified in both
attributes ($n_{c}$), correctly identified but incorrectly classified in at
least one attribute ($n_{i}$), missed (not identified) by the method
($n_{m}$), and extra words that do not appear in the ground truth but were
returned by the method ($n_{e}$). Using these values, we define the following
performance metrics:

  * •

Accuracy: $A=n_{c}/(n_{c}+n_{i}+n_{m})$,

  * •

Precision: $P=n_{c}/(n_{c}+n_{i}+n_{e})$,

  * •

Recall: $R=n_{c}/(n_{c}+n_{m})$,

  * •

F-score: $F=2PR/(P+R)$.

As reflected in Table 1, gpt-4-turbo performed the best across all metrics.
Hence, we selected gpt-4-turbo as the LLM of choice for our empirical study.
In addition, we tested four open-source models from the
Ollama222https://github.com/ollama/ollama project that can be run for
inference locally: Llama 2 Uncensored (7B), Mistral (7B), Neural Chat (7B),
and Llama 2 (13B). Unfortunately, all of them were found unsuitable due to
their poor performance. This task appears to be too complex for these smaller
models, as they did not even output the analysis in the specified format in
the vast majority of cases.

###  4.3 Results

First, Table 2 summarizes the results of applying the gender polarity method
described in Section 3.1 to the English sentences in the sample subsets of the
four benchmark datasets listed in Section 4.1.

Table 2: Evaluation of two representative samples for each of the four benchmark datasets in English using the gender polarity method. The reported results follow the notation introduced in Section 3.1. The gender bias (male:female) is reported in the last column. Dataset | $G_{M}$ | $G_{F}$ | $G_{M}:G_{F}$
---|---|---|---
Europarl 1 | 32 | 23 | 1.39 : 1
Europarl 2 | 38 | 26 | 1.46 : 1
CCAligned 1 | 16 | 15 | 1.07 : 1
CCAligned 2 | 15 | 14 | 1.07 : 1
Global Voices 1 | 136 | 95 | 1.43 : 1
Global Voices 2 | 129 | 90 | 1.43 : 1
WMT-News 1 | 200 | 65 | 3.08 : 1
WMT-News 2 | 248 | 72 | 3.44 : 1

To evaluate the Spanish version of the datasets, we employed the gpt-4-turbo
model (gpt-4-turbo-2024-04-09) with default configuration. For few-shot
prompting, we used five sentence-analysis pairs listed in Appendix B. The
results of the proposed method (see Section 3.3) on two representative samples
of the four benchmark datasets are summarized in Table 3. The budget spent on
these experiments was approximately 70 USD.

Table 3: Evaluation of two representative samples for each of the four benchmark datasets in Spanish using the proposed LLM-based method. The reported results follow the notation introduced in Section 3.3, and the asterisk denotes any value of the binary variable. The gender bias (male:female) is reported in the last column. Dataset | $L_{*,M}$ | $L_{*,F}$ | $L_{N,*}$ | $L_{P,*}$ | $L_{P,M}$ | $L_{P,F}$ | $L_{P,M}:L_{P,F}$
---|---|---|---|---|---|---|---
Europarl 1 | 3531 | 3131 | 5989 | 677 | 541 | 136 | 3.98 : 1
Europarl 2 | 3400 | 3096 | 5765 | 736 | 587 | 149 | 3.94 : 1
CCAligned 1 | 2218 | 1478 | 3388 | 307 | 246 | 61 | 4.03 : 1
CCAligned 2 | 2184 | 1510 | 3385 | 310 | 254 | 56 | 4.54 : 1
Global Voices 1 | 3205 | 2350 | 4495 | 1063 | 869 | 194 | 4.48 : 1
Global Voices 2 | 3237 | 2292 | 4513 | 1019 | 830 | 189 | 4.39 : 1
WMT-News 1 | 3576 | 2489 | 5140 | 929 | 797 | 132 | 6.04 : 1
WMT-News 2 | 3710 | 2514 | 5223 | 1001 | 840 | 161 | 5.22 : 1

While the male:female ratio varies significantly across datasets, all datasets
are biased towards masculine references in both languages. The overall
agreement of the results for the two subsets of each dataset suggests that the
subsets are well-representative. The results indicate that the gender bias is
substantially more pronounced in Spanish than in English, although this
comparison needs to be treated with caution as the methods are different.

Interestingly, in Spanish, the lowest male:female disparity is found in the
Europarl dataset, which makes intuitive sense as the European Parliament is a
political institution that strives for gender equality and representation
across its member states, reflecting broader societal efforts towards balanced
gender representation in government and public discourse. Conversely, the
largest male:female disparity is identified in the WMT-News dataset,
suggesting that news reporting and journalism – especially in certain regions
or topics – may still be male-dominated, reflecting broader societal trends
and biases in media representation and participation.

These insights highlight the importance of using diverse data sources to train
and evaluate NLP models to ensure that they do not perpetuate existing biases.
The results confirm the utility of our LLM-based method in capturing and
quantifying gender biases effectively.

##  5 Discussion

The results of our study have significant implications for the field of
natural language processing, particularly in the understanding and mitigation
of gender bias in gendered languages. By employing state-of-the-art LLMs to
analyze Spanish corpora, our research highlights the extent to which gender
bias can permeate linguistic datasets and subsequently influence machine
learning models trained on these datasets. Below, we summarize the main
findings of our research.

#### 1\. LLMs are a promising tool to identify gender bias in gendered corpora

There is a lack of methods to automatically detect gender disparity in text
corpora in gendered languages, such as Spanish. To the best of our knowledge,
we are the first to propose and empirically demonstrate the value of LLMs,
especially the larger and more advanced versions like gpt-4-turbo, as a tool
to effectively identify and classify gendered language within a complex
linguistic framework like Spanish. This capability is crucial for developing
fairer NLP applications, as it allows for a deeper understanding of how gender
is represented in text and how these representations can lead to biased
outcomes.

#### 2\. Gender bias pervades commonly used corpora in Spanish

We identify a significant male:female gender disparity, ranging between 4:1
(Europarl dataset) and 5–6:1 (WMT-News dataset), in commonly used benchmark
datasets in Spanish. The Europarl dataset exhibits the smallest gender
disparity, which is aligned with the institution’s commitment to gender
equality and balanced representation. This finding suggests that legislative
efforts and policies aided at promoting gender balance can significantly
influence the language and content of official records. However, the number of
references to males is four times the number of references to females,
indicating that more efforts are needed to achieve representative gender
equality. Conversely, we identify the largest gender disparity in the WMT-News
dataset, which suggests that news reporting and journalism remain male-driven,
reflecting broader societal trends and editorial biases in the media industry.

The prevalence of masculine w.r.t. feminine references across different
datasets underscores the pervasive nature of gender bias in text corpora and
the importance of identifying and mitigating such biases to create more
balanced datasets, which in turn would enable more equitable models.

#### 3\. Disparities in gender representation in English vs. Spanish

The differences in gender ratio between the English and Spanish versions of
our datasets further underline the linguistic and cultural variances affecting
gender representation. While both languages exhibited biases, the Spanish
datasets generally showed a higher degree of male dominance. This may be
partly due to the grammatical structure of Spanish, which includes gendered
nouns and more explicit gender marking, potentially amplifying the visibility
and perpetuation of gender biases. The English datasets, while also biased,
tended to have lower ratios of male:female references, possibly due to a
combination of linguistic factors and different societal norms influencing the
text. Interestingly, in the English corpora, the CCAligned samples exhibit the
lowest gender disparity. Conversely and aligned with the findings in the
Spanish case, the WMT-News samples have the largest male:female ratio. These
results highlight the need for considering both linguistic and cultural
contexts when analyzing and mitigating biases in multilingual and
multicultural corpora.

##  6 Conclusion

We have presented a novel methodology for measuring gender bias in Spanish
text corpora using large language models. Through experiments with four
benchmark datasets, we have demonstrated the efficacy of this approach, which
leverages the capabilities of LLMs to differentiate between gendered
references in a nuanced manner.

Our findings reveal significant gender disparities that range between 4:1 and
6:1 in the male-to-female ratio, depending on the dataset, highlighting the
need for continuous monitoring and correction of biases in linguistic data
sources. The methodology developed herein not only contributes to the field of
bias research in NLP but also sets a precedent for future studies aiming to
understand and rectify gender bias in languages other than English, broadening
the scope of bias detection and mitigation strategies to a global context.
Ensuring that language technologies do not perpetuate or even amplify existing
social biases is essential.

In future work, we aim to further refine our approach and explore the
integration of additional linguistic and cultural nuances that may affect the
representation of gender in text. Moreover, extending this framework to other
gendered languages could provide a more comprehensive global perspective on
gender bias in NLP. Continued advancements in model capabilities and a growing
understanding of bias will undoubtedly enhance the fairness and accuracy of
language-based AI systems.

## Acknowledgments

The work of E.D., S.S., and N.O. on this paper was supported by a nominal
grant received at the ELLIS Unit Alicante Foundation from the Regional
Government of Valencia in Spain (Convenio Singular signed with Generalitat
Valenciana, Conselleria de Innovación, Industria, Comercio y Turismo,
Dirección General de Innovación). E.D. is also supported by Intel Corporation.

The work of Y.G. and P.M. on this paper was partially funded by the University
of Alicante, the Spanish Ministry of Science and Innovation, the Generalitat
Valenciana, and the European Regional Development Fund (ERDF) through the
following funding: As main, VIVES: “Pla de Tecnologies de la Llengua per al
valencià” project (2022/TL22/00215334) from the Projecte Estratègic per a la
Recuperació i Transformació Econòmica (PERTE) and the following supportive
projects: NL4DISMIS (CIPROM/2021/21) with funds granted by the Generalitat
Valenciana (Conselleria d’Educacio, Investigacio, Cultura i Esport), COOLANG
(PID2021-122263OB-C22), and CORTEX (PID2021-123956OB-I00) with funds granted
by MCIN/AEI/10.13039/501100011033 and, as appropriate, by “ERDF A way of
making Europe”, by the “European Union” or by the “European Union
NextGenerationEU/PRTR”.

## Limitations

While we believe that our study provides valuable insights into the
measurement of gender bias in gendered languages using LLMs, we have
identified several limitations.

Inclusion of epicene words: In our analysis, we included epicene words – nouns
that have the same form, including grammatical gender, for male and for female
referents. For example, ‘la persona’ (person) is grammatically feminine but
can refer to individuals of any gender. This inherent ambiguity can lead to
inaccuracies in gender classification. In the data analyzed in this work,
epicene words represent 5.8 % of all identified person-referencing words. It
is noteworthy that 73.7 % of epicene words were counted as feminine (see
Appendix C), such that excluding the epicene words would make the gender
ratios even more biased towards the masculine gender.

Contextuality of the analysis: Our methodology focuses primarily on the
occurrence and classification of gendered nouns and pronouns without deeply
analyzing the context in which these words appear. This approach provides a
quantitative measure of gender representation but lacks the nuance of
qualitative analysis. Understanding the context could reveal more about the
nature of gender bias, such as whether certain roles or characteristics are
predominantly associated with a specific gender. Future work should
incorporate contextual analysis to provide a richer and more comprehensive
understanding of gender bias in texts.

Binary gender: The current framework represents gender as a binary variable –
masculine and feminine – and does not account for non-binary or other gender
identities. This limitation reflects the grammatical structure of Spanish but
is not aligned with the broader spectrum of gender identities present in the
society. As language evolves and becomes more inclusive, future studies should
consider expanding the classification system to encompass a wider range of
gender identities, ensuring that NLP models are sensitive to and
representative of all individuals.

Budget implications: Using commercial, state-of-the-art LLMs in research can
have a non-negligible financial cost. While we selected representative subsets
from widely used benchmark datasets, a more extensive analysis could
potentially provide more robust and generalizable findings at a cost.

LLM dependence and model variability: Our method heavily relies on the
capabilities of advanced LLMs, specifically gpt-4-turbo. The performance and
accuracy of our bias detection framework are directly tied to the model’s
proficiency in understanding and interpreting language. Different versions or
configurations of LLMs might yield varying results, and smaller open-source
models that allow for local inference currently lack the performance needed
for such complex tasks. Additionally, the use of proprietary models may impact
the reproducibility and transparency of our findings.

Cultural and linguistic variations: Gender representation in language can be
influenced by cultural and regional variations, which our study does not
explicitly address. The datasets used, while diverse, may not fully capture
the nuances of different Spanish-speaking communities. Future research should
consider the impact of cultural and regional differences on gender bias to
develop more universally applicable methodologies.

By addressing these limitations in future work, we aim to refine our
methodology, incorporate a broader range of linguistic and cultural nuances,
and enhance the fairness and accuracy of gender bias detection in NLP systems.

## Ethics Statement

We aim to promote fairness and inclusivity by identifying and quantifying
biases in text corpora used to train LLMs. We have adhered to ethical
standards and code of conduct by ensuring transparency, reproducibility, and
validating our methodology against manually annotated data. The datasets used
for evaluation are publicly available benchmark datasets. While our work
highlights significant gender disparities, we recognize the limitations of
focusing on grammar-based gender classification and the reliance on specific
LLMs.

We are committed to ethical AI use, advocating for continuous improvement in
bias detection and mitigation techniques. Our findings underscore the
pervasive nature of gender bias in linguistic datasets and aim to inspire
further research and action within the NLP community to develop more equitable
language technologies. By raising awareness and providing robust
methodologies, we contribute to the broader discourse on AI ethics and the
pursuit of technologies that uphold human dignity and equality.

## References

  * Babuta and Oswald (2019) Alexander Babuta and Marion Oswald. 2019.  Data analytics and algorithmic bias in policing.  _RUSI Briefing Paper_.
  * Bansal (2022) Rajas Bansal. 2022.  A survey on bias and fairness in natural language processing.  _arXiv preprint arXiv:2204.09591_.
  * Barocas et al. (2019) Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2019.  _Fairness and Machine Learning: Limitations and Opportunities_.  The MIT Press.
  * Bartl et al. (2020) Marion Bartl, Malvina Nissim, and Albert Gatt. 2020.  Unmasking contextual stereotypes: Measuring and mitigating BERT’s gender bias.  In _Proceedings of the Second Workshop on Gender Bias in Natural Language Processing_ , pages 1–16, Barcelona, Spain (Online). Association for Computational Linguistics.
  * Blodgett et al. (2020) Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020.  Language (technology) is power: A critical survey of “bias” in NLP.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5454–5476.
  * Bolukbasi et al. (2016) Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016.  Man is to computer programmer as woman is to homemaker? Debiasing word embeddings.  _Advances in Neural Information Processing Systems_ , 29.
  * Bordia and Bowman (2019) Shikha Bordia and Samuel Bowman. 2019.  Identifying and reducing gender bias in word-level language models.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop_ , pages 7–15.
  * Caliskan et al. (2017) Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017.  Semantics derived automatically from language corpora contain human-like biases.  _Science_ , 356(6334):183–186.
  * Daniel and Cross (2018) Wayne W Daniel and Chad L Cross. 2018.  _Biostatistics: A Foundation for Analysis in the Health Sciences_.  Wiley.
  * Dhamala et al. (2021) Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021.  BOLD: Dataset and metrics for measuring biases in open-ended language generation.  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , pages 862–872.
  * El-Kishky et al. (2020) Ahmed El-Kishky, Vishrav Chaudhary, Francisco Guzmán, and Philipp Koehn. 2020\.  CCAligned: A massive collection of cross-lingual web-document pairs.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5960–5969.
  * Gallegos et al. (2024) Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. 2024.  Bias and fairness in large language models: A survey.  _Computational Linguistics_ , pages 1–79.
  * Gonen and Goldberg (2019) Hila Gonen and Yoav Goldberg. 2019.  Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 609–614.
  * Guo and Caliskan (2021) Wei Guo and Aylin Caliskan. 2021.  Detecting emergent intersectional biases: Contextualized word embeddings contain a distribution of human-like biases.  In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_ , pages 122–133.
  * Kiritchenko and Mohammad (2018) Svetlana Kiritchenko and Saif Mohammad. 2018.  Examining gender and race bias in two hundred sentiment analysis systems.  In _Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics_ , pages 43–53.
  * Koehn (2005) Philipp Koehn. 2005.  Europarl: A parallel corpus for statistical machine translation.  In _Proceedings of Machine Translation Summit X: Papers_ , pages 79–86.
  * Kotek et al. (2023) Hadas Kotek, Rikker Dockum, and David Sun. 2023.  Gender bias and stereotypes in large language models.  In _Proceedings of The ACM Collective Intelligence Conference_ , CI ’23, pages 12–24, New York, NY, USA. Association for Computing Machinery.
  * Kreutzer et al. (2022) Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Auguste Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, et al. 2022.  Quality at a glance: An audit of web-crawled multilingual datasets.  _Transactions of the Association for Computational Linguistics_ , 10:50–72.
  * Lu et al. (2020) Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. 2020\.  Gender bias in neural natural language processing.  _Logic, Language, and Security: Essays Dedicated to Andre Scedrov on the Occasion of His 65th Birthday_ , pages 189–202.
  * May et al. (2019) Chandler May, Alex Wang, Shikha Bordia, Samuel Bowman, and Rachel Rudinger. 2019\.  On measuring social biases in sentence encoders.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 622–628.
  * Nangia et al. (2020) Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020.  CrowS-pairs: A challenge dataset for measuring social biases in masked language models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1953–1967. Association for Computational Linguistics.
  * Névéol et al. (2022) Aurélie Névéol, Yoann Dupont, Julien Bezançon, and Karën Fort. 2022.  French CrowS-pairs: Extending a challenge dataset for measuring social bias in masked language models to a language other than English.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8521–8531.
  * Nguyen and Daumé III (2019) Khanh Nguyen and Hal Daumé III. 2019.  Global Voices: Crossing borders in automatic news summarization.  In _Proceedings of the 2nd Workshop on New Frontiers in Summarization_ , pages 90–97.
  * Panch et al. (2019) Trishan Panch, Heather Mattie, and Rifat Atun. 2019.  Artificial intelligence and algorithmic bias: Implications for health systems.  _Journal of Global Health_ , 9(2).
  * Raghavan et al. (2020) Manish Raghavan, Solon Barocas, Jon Kleinberg, and Karen Levy. 2020.  Mitigating bias in algorithmic hiring: Evaluating claims and practices.  In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ , pages 469–481.
  * Ramesh et al. (2023) Krithika Ramesh, Sunayana Sitaram, and Monojit Choudhury. 2023.  Fairness in language models beyond English: Gaps and challenges.  In _Findings of the Association for Computational Linguistics: EACL 2023_ , pages 2106–2119, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Ranathunga et al. (2024) Surangika Ranathunga, Nisansa de Silva, Menan Velayuthan, Aloka Fernando, and Charitha Rathnayake. 2024.  Quality does matter: A detailed look at the quality and utility of web-mined parallel corpora.  In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 860–880.
  * Rudinger et al. (2018) Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018.  Gender bias in coreference resolution.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 8–14.
  * Salazar et al. (2020) Julian Salazar, Davis Liang, Toan Q Nguyen, and Katrin Kirchhoff. 2020.  Masked language model scoring.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2699–2712.
  * Stańczak and Augenstein (2021) Karolina Stańczak and Isabelle Augenstein. 2021.  A survey on gender bias in natural language processing.  _arXiv preprint arXiv:2112.14168_.
  * Stanovsky et al. (2019) Gabriel Stanovsky, Noah A Smith, and Luke Zettlemoyer. 2019.  Evaluating gender bias in machine translation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1679–1684.
  * Tiedemann (2012) Jörg Tiedemann. 2012.  Parallel data, tools and interfaces in opus.  In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)_ , pages 2214–2218.
  * Vanmassenhove et al. (2018) Eva Vanmassenhove, Christian Hardmeier, and Andy Way. 2018.  Getting gender right in neural machine translation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3003–3008.
  * Webster et al. (2018) Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018.  Mind the GAP: A balanced corpus of gendered ambiguous pronouns.  _Transactions of the Association for Computational Linguistics_ , 6:605–617.
  * Zhao et al. (2018) Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018\.  Gender bias in coreference resolution: Evaluation and debiasing methods.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 15–20.

##  Appendix A Appendix: Prompt Formulation

Through manual interactive testing, we crafted the following prompt in
Spanish, which is used in all experiments with the proposed LLM-based method
reported in this paper:

_< EXAMPLES>_
_Frase: <SENTENCE>_
_Instrucciones: Identifica todos los sustantivos y pronombres en la frase
proporcionada. Para cada uno, determina si se refiere a un ser humano (S) o no
(N), y especifica su género gramatical: masculino (M) o femenino (F). Excluye
los apellidos. Sigue el formato de los ejemplos proporcionados sin añadir
texto adicional._

The placeholder <EXAMPLES> is replaced with priming examples, listed in Table
5. Each of them is prepended with ‘Ejemplo #:’ (Spanish for ‘example’), where
# is replaced with the example index. The placeholder <SENTENCE> is replaced
with the sentence to be analyzed.

##  Appendix B Appendix: Few-Shot Prompting Examples

Through interactive experimenting with the LLMs, and following the common best
practices, we concluded that it is beneficial to employ the few-shot prompting
technique. We selected five sentences from the Europarl dataset (see Section
4.1) and provided the ground truth analysis (created manually) to prime the
LLM for the bias quantification task, see Table 5.

##  Appendix C Appendix: Epicene Words

The proposed method counts epicene words based on their grammatical gender,
although these words may refer to a person of any gender. Table 4 lists
epicene words identified across all datasets analyzed in this work. Epicene
words represent 5.8 % of all identified words referring to a person. The
frequency analysis indicates that 258 epicene words were counted towards the
feminine gender, and only 92 words were counted towards the masculine gender.

Table 4: Epicene words and their frequencies, identified across all datasets evaluated in this work using the proposed LLM-based method. Note that the word ‘miembro’ appears twice because it can be identified as feminine in specific contexts (indicated by the article ‘la’), although it generally has the masculine grammatical gender. Word | $p$ | $g$ | Frequency
---|---|---|---
personas | $P$ | $F$ | 149
miembros | $P$ | $M$ | 63
gente | $P$ | $F$ | 54
persona | $P$ | $F$ | 34
miembro | $P$ | $M$ | 20
víctimas | $P$ | $F$ | 14
individuo | $P$ | $M$ | 7
víctima | $P$ | $F$ | 5
miembro | $P$ | $F$ | 2
individuos | $P$ | $M$ | 2
Table 5: Few-shot prompting examples used in the experiments. Note that although we indicate person-referencing words with ‘P’ in the paper, the list of words employed in the prompt in Spanish uses ‘S’ instead. Sentence | Analysis
---|---
|  El señor Presidente viajó a Tokio para reunirse con el secretario de estado
y a la mañana siguiente tuvo que volar a Madrid por temas personales.
---
| señor – S, M
---
Presidente – S, M
Tokio – N, M
secretario – S, M
estado – N, M
mañana – N, F
Madrid – N, M
temas – N, M
|  Mi colega Sr. Allan Hofmann se dirigió a los ciudadanos de Madrid,
recordándoles que son personas con derechos y responsabilidades.
---
| colega – S, M
---
Sr. – S, M
Allan – S, M
ciudadanos – S, M
Madrid – N, M
personas – S, F
derechos – N, M
responsabilidades – N, F
|  El señor Presidente de la comisión de educación se reunió con los
estudiantes en Tokio, donde el distinguido Sir Ben Smith compartió su visión
sobre el futuro de la enseñanza.
---
| señor – S, M
---
Presidente – S, M
comisión – N, F
educación – N, F
estudiantes – S, M
Tokio – N, M
Sir – S, M
Ben – S, M
visión – N, F
futuro – N, M
enseñanza – N, F
|  El Sr. Johnson, un respetado colega de la ciudadanía británica, ha vivido
en Londres durante más de dos décadas, donde trabaja incansablemente para
mejorar la comunidad local.
---
| Sr. – S, M
---
colega – S, M
ciudadanía – N, F
Londres – N, M
décadas – N, F
comunidad – N, F
|  Encontré en Europa no solo destinos turísticos, sino un hogar temporal
donde me sentí ciudadana del mundo, abrazando la diversidad y la riqueza
cultural que esta tierra ofrece.
---
| Europa – N, F
---
destinos – N, M
hogar – N, M
ciudadana – S, F
mundo – N, M
diversidad – N, F
riqueza – N, F
tierra – N, F