# Comment on “Single-mode excited entangled coherent states”

Hong-chun Yuan1and Li-yun Hu1,2
1Department of Physics, Shanghai Jiao Tong University, Shanghai 200030, China
2 College of Physics and Communication Electronics, Jiangxi Normal University,
Nanchang 330022, China Corresponding author<EMAIL_ADDRESS>
###### Abstract

In Xu and Kuang (J. Phys. A: Math. Gen. 39 (2006) L191), the authors claim
that, for single-mode excited entangled coherent states
$\left|\Psi_{\pm}(\alpha,m)\right\rangle$, “the photon excitations lead to the
decrease of the concurrence in the strong field regime of
$\left|\alpha\right|^{2}$ and the concurrence tends to zero when
$\left|\alpha\right|^{2}\rightarrow\infty$”. This is wrong.

PACS number: 03.65.Ud; 03.67.Hk

In the recent paper [1], single-mode excited entangled coherent states
(SMEECSs) $\left|\Psi_{\pm}(\alpha,m)\right\rangle$ are introduced and their
entanglement characteristics and the influence of photon excitations on
quantum entanglement are also investigated. They claim that “the photon
excitations lead to the decrease of the concurrence in the strong field regime
of $\left|\alpha\right|^{2}$ and the concurrence tends to zero when
$\left|\alpha\right|^{2}\rightarrow\infty$”. Unfortunately, however, this
conclusion is wrong.

First, we recall the entangled coherent states (ECS) [2]

$\left|\Psi_{\pm}(\alpha,0)\right\rangle=N_{\pm}(\alpha,0)(\left|\alpha,\alpha\right\rangle\pm\left|-\alpha,-\alpha\right\rangle),$
(1)

where
$\left|\alpha,\alpha\right\rangle\equiv\left|\alpha\right\rangle_{a}\otimes\left|\alpha\right\rangle_{b}$
with $\left|\alpha\right\rangle_{a}$ and $\left|\alpha\right\rangle_{b}$ being
the usual coherent state in $a$ and $b$ modes, respectively, and

$\left(N_{\pm}(\alpha,0)\right)^{-2}=2\left[1\pm\exp\left(-4\left|\alpha\right|^{2}\right)\right],$
(2)

is the normalization constants.

The SMEECSs are obtained throught actions of a creation operator of a single-
mode optical field on the ECSs, which are expressed as

$\left|\Psi_{\pm}(\alpha,m)\right\rangle=N_{\pm}(\alpha,m)a^{\dagger
m}(\left|\alpha,\alpha\right\rangle\pm\left|-\alpha,-\alpha\right\rangle),$
(3)

where without any loss of generality we consider $m$-photon excitations of the
mode $a$ in the ECS and $N_{\pm}(\alpha,m)$ represents the normalization
factor. Using the identity of operator [3]

$a^{n}a^{\dagger m}=\left(-i\right)^{n+m}\colon
H_{m,n}\left(ia^{\dagger},ia\right)\colon$ (4)

where the symbol $\colon\colon$represents the normal ordering for Bosonic
operators $\left(a^{\dagger},a\right)$, and $H_{m,n}(\eta,\eta^{\ast})$ is the
two-variable Hermite polynomial[4],

$H_{m,n}(\eta,\eta^{\ast})=\sum_{l=0}^{\min(m,n)}\frac{\left(-1\right)^{l}n!m!}{l!\left(m-l\right)!\left(n-l\right)!}\eta^{m-l}\eta^{\ast
n-l},$ (5)

we can easily obtain

$\left\langle\alpha\right|a^{m}a^{\dagger
m}\left|\alpha\right\rangle=m!L_{m}(-\left|\alpha\right|^{2}),\text{ \
}\left\langle\alpha\right|a^{m}a^{\dagger
m}\left|-\alpha\right\rangle=m!e^{-2\left|\alpha\right|^{2}}L_{m}\left(\left|\alpha\right|^{2}\right),$
(6)

and directly calculate the normalization factor

$\left[N_{\pm}(\alpha,m)\right]^{-2}=2m!\left[L_{m}\left(-\left|\alpha\right|^{2}\right)\pm
e^{-4\left|\alpha\right|^{2}}L_{m}\left(\left|\alpha\right|^{2}\right)\right],$
(7)

where $L_{m}(x)$ is the $m-$order Laguerre polynomial defined by[5]

$L_{m}(x)=\sum_{l=0}^{m}\frac{(-1)^{l}m!x^{l}}{\left(l!\right)^{2}(m-l)!}.$
(8)

It is quite clear that when $m=0$, Eq.(3) reduces to the usual ECSs in Eq.(1).
Eq.(7) is valid for any integer $m$ (including the case of $m=0$) which is
different from Eq.(4) of Ref.[1].

Next, we calculate the concurrence for the SMEECSs. Noticing that the photon-
added coherent states (PACSs) $\left|\alpha,m\right\rangle$ is defined by[5]

$\left|\alpha,m\right\rangle=\frac{a^{\dagger
m}\left|\alpha\right\rangle}{\sqrt{m!L_{m}\left(-\left|\alpha\right|^{2}\right)}},$
(9)

thus the SMEECSs $\left|\Psi_{\pm}(\alpha,m)\right\rangle$ in terms of the
PACSs can be rewritten as

$\left|\Psi_{\pm}(\alpha,m)\right\rangle=M_{\pm}(\alpha,m)(\left|\alpha,m\right\rangle\otimes\left|\alpha\right\rangle\pm\left|-\alpha,m\right\rangle\otimes\left|-\alpha\right\rangle),$
(10)

where the normalization constant $M_{\pm}(\alpha,m)$ is determined by

$\left[M_{\pm}(\alpha,m)\right]^{2}=\frac{L_{m}\left(-\left|\alpha\right|^{2}\right)}{2\left[L_{m}\left(-\left|\alpha\right|^{2}\right)\pm
e^{-4\left|\alpha\right|^{2}}L_{m}\left(\left|\alpha\right|^{2}\right)\right]}.$
(11)

Following the approach of Ref.[6] and considering Eqs.(6) and (9), for the
SMEECSs $\left|\Psi_{\pm}(\alpha,m)\right\rangle,$ the concurrence can be
calculated as

$C_{\pm}(\alpha,m)=\frac{\sqrt{\left(1-p_{1}^{2}\right)\left(1-p_{2}^{2}\right)}}{1\pm
p_{1}p_{2}},$ (12)

where

$p_{1}=\left\langle\alpha,m\right|\left.-\alpha,m\right\rangle=\frac{\exp(-2\left|\alpha\right|^{2})L_{m}\left(\left|\alpha\right|^{2}\right)}{L_{m}\left(-\left|\alpha\right|^{2}\right)},$
(13)

and

$p_{2}=\left\langle\alpha\right|\left.-\alpha\right\rangle=\exp(-2\left|\alpha\right|^{2}).$
(14)

Then submitting Eqs.(13) and (14) into Eq.(12) we see that

$C_{\pm}(\alpha,m)=\frac{\left[\left(L_{m}^{2}\left(-\left|\alpha\right|^{2}\right)-e^{-4\left|\alpha\right|^{2}}L_{m}^{2}\left(\left|\alpha\right|^{2}\right)\right)\left(1-e^{-4\left|\alpha\right|^{2}}\right)\right]^{1/2}}{L_{m}\left(-\left|\alpha\right|^{2}\right)\pm
e^{-4\left|\alpha\right|^{2}}L_{m}\left(\left|\alpha\right|^{2}\right)},$ (15)

which is another expression different from Eqs.(23) and (24) in Ref.[1]. In
particular, when $m=0$, Eq.(15) becomes

$C_{+}(\alpha,0)=\frac{1-e^{-4\left|\alpha\right|^{2}}}{1+e^{-4\left|\alpha\right|^{2}}},C_{-}(\alpha,0)=1.$
(16)

It implies that the concurrence $C_{+}(\alpha,0)$ of ECS
$\left|\Psi_{+}(\alpha,0)\right\rangle$ increases with the values of
$\left|\alpha\right|^{2};$ while $C_{-}(\alpha,0)$ is indenpent of
$\left|\alpha\right|^{2}\ $and $\left|\Psi_{-}(\alpha,0)\right\rangle$ is a
maximally entangled state.

In order to see clearly the influence of the concurrence with parameter $m$,
the concurrences $C$ for the state $\left|\Psi_{\pm}(\alpha,m)\right\rangle$
as a function of $\left|\alpha\right|^{2}$ are shown in Figs.1 and 2. It is
shown that $C_{\pm}(\alpha,m)$ increases with the increase of
$\left|\alpha\right|^{2}$ for given parameter $m$. Especially, the concurrence
$C_{\pm}(\alpha,m)$ tend to unit for the larger
$\left|\alpha\right|^{2}.$These conclusions are completely different from
those of Ref.[1].

## References

  * [1] Xu L and Kuang L M 2006 J. Phys. A: Math. Gen. 39 L191
  * [2] Sanders B C 1992 Phys. Rev. A 45 6811; Luis A 2001 Phys. Rev. A 64 054102
  * [3] Hu L Y and Fan H Y 2009 Phys. Scr. 79 035004
  * [4] Wünsche A 2000 J. Phys. A: Math. Gen. 33 1603; Hu L Y and Fan H Y 2009 Chin. Phys. B 18 0902
  * [5] Agarwal G S and Tara K 1991 Phys. Rev. A 43 492
  * [6] Wang X G 2002 J. Phys. A: Math. Gen. 35 165; Rungta P et al 2001 Phys. Rev. A 64 042315

Figure 1: Concurrence of entanglement of
$\left|\phi_{+}(\alpha,m)\right\rangle$ as a function of
$\left|\alpha\right|^{2}$ for the different photon excitations with $m$
$=0$(solid line), $m=1$(dashed line), $m=3$(dotted line), $m=5$(dash-dotted
line), and $m=20$(dash-dash-dotted line), respectively. Figure 2: Concurrence
of entanglement of $\left|\phi_{-}(\alpha,m)\right\rangle$ as a function of
$\left|\alpha\right|^{2}$ for the different photon excitations with
$m=0$(solid line), $m=3$(dashed line), $m=5$(dotted line), $m=10$(dash-dotted
line), and $m=20$(dash-dash-dotted line), respectively.


&+\sum_{|\beta|=p-2}\begin{pmatrix}p\\\beta\end{pmatrix}\theta^{\beta^2}u^\beta\sum_{l=1}^n\theta_1^{2\beta_1+1}\theta_i^{2\beta_i+1}d_i\frac{\partial u_1}{\partial x_l}\frac{\partial u_i}{\partial x_l}\nonumber\\
&=\sum_{|\beta|=p-2}\begin{pmatrix}p\\\beta\end{pmatrix}\theta^{\beta^2}u^\beta\sum_{l=1}^n\theta_1^{2\beta_1+1}\theta_i^{2\beta_i+1}(d_1+d_i)\frac{\partial u_1}{\partial x_l}\frac{\partial u_i}{\partial x_l}.
\end{align}

The result follows by combining (<ref>), (<ref>), (<ref>), (<ref>), (<ref>) and (<ref>).

Acknowledgement. This work is partially supported by NAWI Graz, and the International Research Training Group IGDK 1754 “Optimization and Numerical Analysis for Partial Differential Equations with Nonsmooth Structures”, funded by the German Research Council (DFG) project number 188264188/GRK1754 and the Austrian Science Fund (FWF) under grant number W 1244-N18.

[1]
Björn Augner and Dieter Bothe.
Analysis of some heterogeneous catalysis models with fast sorption
and fast surface chemistry.
arXiv preprint arXiv:2006.12098, 2020.

[2]
Amal Alphonse, Charles M Elliott, and Joana Terra.
A coupled ligand-receptor bulk-surface system on a moving domain:
Well posedness, regularity, and convergence to equilibrium.
SIAM Journal on Mathematical Analysis, 50(2):1544–1592, 2018.

[3]
Salem Abdelmalek and Said Kouachi.
Proof of existence of global solutions for m-component
reaction–diffusion systems with mixed boundary conditions via the lyapunov
functional method.
Journal of Physics A: Mathematical and Theoretical,
40(41):12335, 2007.

[4]
Nicholas D Alikakos.
$L^p$ bounds of solutions of reaction-diffusion equations.
Communications in Partial Differential Equations,
4(8):827–868, 1979.

[5]
Nicholas D Alikakos.
Regularity and asymptotic behavior for the second order parabolic
equation with nonlinear boundary conditions in ${L}^p$.
Journal of Differential Equations, 39(3):311–344, 1981.

[6]
Henri Berestycki, Anne-Charline Coulon, Jean-Michel Roquejoffre, and Luca
The effect of a line with nonlocal diffusion on Fisher-KPP
Mathematical Models and Methods in Applied Sciences,
25(13):2519–2562, 2015.

[7]
Johannes Borgqvist, Adam Malik, Carl Lundholm, Anders Logg, Philip Gerlee, and
Marija Cvijovic.
Cell polarisation in a bulk-surface model can be driven by both
classic and non-classic Turing instability.
bioRxiv, 2020.

[8]
Henri Berestycki, Jean-Michel Roquejoffre, and Luca Rossi.
The influence of a line with fast diffusion on Fisher-KPP
Journal of mathematical biology, 66(4-5):743–766, 2013.

[9]
José A Cañizo, Laurent Desvillettes, and Klemens Fellner.
Improved duality estimates and applications to reaction-diffusion
Communications in Partial Differential Equations,
39(6):1185–1204, 2014.

[10]
M Cristina Caputo, Thierry Goudon, and Alexis F Vasseur.
Solutions of the $4$-species quadratic reaction-diffusion system are
bounded and $C^\infty$-smooth, in any space dimension.
Analysis and PDE, 12(7):1773–1804, 2019.

[11]
Karoline Disser.
Global existence, uniqueness and stability for nonlinear dissipative
bulk-interface interaction systems.
Journal of Differential Equations, 2020.

[12]
Julian Fischer.
Global existence of renormalized solutions to entropy-dissipating
reaction–diffusion systems.
Archive for Rational Mechanics and Analysis, 218(1):553–587,

[13]
Klemens Fellner, Evangelos Latos, and Bao Quoc Tang.
Well-posedness and exponential equilibration of a volume-surface
reaction–diffusion system with nonlinear boundary coupling.
Annales de l'Institut Henri Poincaré C, Analyse non
linéaire, 35(3):643–673, 2018.

[14]
Klemens Fellner, Jeff Morgan, and Bao Quoc Tang.
Global classical solutions to quadratic systems with mass control in
arbitrary dimensions.
Annales de l'Institut Henri Poincaré C, Analyse non
linéaire, 37(2):281–307, 2020.

[15]
Klemens Fellner, Stefan Rosenberger, and Bao Quoc Tang.
Quasi-steady-state approximation and numerical simulation for a
volume-surface reaction-diffusion system.
Communications in Mathematical Sciences, 14(6):1553–1580,

[16]
Annegret Glitzky and Alexander Mielke.
A gradient structure for systems coupling reaction–diffusion effects
in bulk and interfaces.
Zeitschrift für angewandte Mathematik und Physik,
64(1):29–52, 2013.

[17]
Pilar García-Peñarrubia, Juan J Gálvez, and Jesús Gálvez.
Mathematical modelling and computational study of two-dimensional and
three-dimensional dynamics of receptor–ligand interactions in signalling
response mechanisms.
Journal of Mathematical Biology, 69(3):553–582, 2014.

[18]
Pierre Grisvard.
Elliptic problems in nonsmooth domains.
SIAM, 2011.

[19]
Stephan Hausberg and Matthias Röger.
Well-posedness and fast-diffusion limit for a bulk–surface
reaction–diffusion system.
Nonlinear Differential Equations and Applications NoDEA,
25(3):17, 2018.

[20]
Yong-Il Kwon and Jeffrey J Derby.
Modeling the coupled effects of interfacial and bulk phenomena during
solution crystal growth.
Journal of crystal growth, 230(1-2):328–335, 2001.

[21]
Said Kouachi.
Existence of global solutions to reaction-diffusion systems via a
lyapunov functional.
Electronic Journal of Differential Equations (EJDE)[electronic
only], 2001:Paper–No, 2001.

[22]
Damien Lamberton.
Equations d'évolution linéaires associées à des
semi-groupes de contractions dans les espaces lp.
Journal of Functional Analysis, 72(2):252–262, 1987.

[23]
Olga A Ladyženskaja, Vsevolod Alekseevich Solonnikov, and Nina N
Linear and quasi-linear equations of parabolic type, volume 23.
American Mathematical Soc., 1988.

[24]
Alessandra Lunardi.
Interpolation theory, volume 16.
Springer, 2018.

[25]
Alexander Mielke.
Thermomechanical modeling of energy-reaction-diffusion systems,
including bulk-interface interactions.
Discrete and Continuous Dynamical Systems - S, 6:479, 2013.

[26]
Jeff Morgan and Vandana Sharma.
Martin’s problem for volume-surface reaction-diffusion systems.
In Contributions to Partial Differential Equations and
Applications, pages 359–370. Springer, 2019.

[27]
Jeff Morgan and Bao Quoc Tang.
Boundedness for reaction–diffusion systems with lyapunov functions
and intermediate sum conditions.
Nonlinearity, 33(7):3105, 2020.

[28]
Simon Malham and Jack X Xin.
Global solutions to a reactive boussinesq system with front data on
an infinite domain.
Communications in mathematical physics, 193(2):287–316, 1998.

[29]
Robin Nittka.
Inhomogeneous parabolic neumann problems.
Czechoslovak Mathematical Journal, 64(3):703–742, 2014.

[30]
Michel Pierre.
Global existence in reaction-diffusion systems with control of mass:
a survey.
Milan Journal of Mathematics, 78(2):417–455, 2010.

[31]
Jan Prüss.
Maximal regularity for abstract parabolic problems with inhomogeneous
boundary data in $L_p$-spaces.
Mathematica Bohemica, 127(2):311–327, 2002.

[32]
Michel Pierre and Didier Schmitt.
Blowup in reaction-diffusion systems with dissipation of mass.
SIAM review, 42(1):93–106, 2000.

[33]
Michel Pierre, Takashi Suzuki, and Haruki Umakoshi.
Global-in-time behavior of weak solutions to reaction–diffusion
systems with inhomogeneous dirichlet boundary condition.
Nonlinear Analysis, 159:393–407, 2017.

[34]
Lucas M Stolerman, Michael Getz, SG Smith, Michael Holst, and Padmini
Stability analysis of a bulk-surface reaction model for
membrane-protein clustering.
to appear in Bulletin of Mathematical Biology, 2020.

[35]
Vandana Sharma.
Global existence and uniform estimates of solution to reaction
diffusion systems with mass transport type boundary conditions.
arXiv, pages arXiv–2005, 2020.

[36]
Vandana Sharma and Jeff Morgan.
Global existence of solutions to reaction-diffusion systems with mass
transport type boundary conditions.
SIAM Journal on Mathematical Analysis, 48(6):4202–4240, 2016.

[37]
Vandana Sharma and Jeff Morgan.
Uniform bounds for solutions to volume-surface reaction diffusion
Differential and Integral Equations, 30(5/6):423–442, 2017.

[38]
Ondřej Souček, Vít Orava, Josef Málek, and Dieter Bothe.
A continuum model of heterogeneous catalysis: Thermodynamic framework
for multicomponent bulk and surface phenomena coupled by sorption.
International Journal of Engineering Science, 138:82–117,

[39]
Philippe Souplet.
Global existence for reaction–diffusion systems with dissipation of
mass and quadratic growth.
Journal of Evolution Equations, 18(4):1713–1720, 2018.

[40]
Andrew Yeckel and Jeffrey J Derby.
Computer modelling of bulk crystal growth.
Bulk Crystal Growth of Electronic, Optical, and Optoelectronic
Materials, pages 73–119, 2005.


# The Double-$Q$ Ground State with Topological Charge Stripes in the Skyrmion
Candidate GdRu2Si2

G. D. A. Wood<EMAIL_ADDRESS>Department of Physics, University of
Warwick, Coventry, CV4 7AL, United Kingdom  D. D. Khalyavin  ISIS Facility,
STFC Rutherford Appleton Laboratory, Harwell Science and Innovation Campus,
Oxfordshire OX11 0QX, United Kingdom  D. A. Mayoh  Department of Physics,
University of Warwick, Coventry, CV4 7AL, United Kingdom  J. Bouaziz  Peter
Grünberg Institut and Institute for Advanced Simulation, Forschungszentrum
Jülich & JARA, D-52425 Jülich, Germany  A. E. Hall  Department of Physics,
University of Warwick, Coventry, CV4 7AL, United Kingdom  S. J. R. Holt
Faculty of Engineering and Physical Sciences, University of Southampton,
Southampton SO17 1BJ, United Kingdom  Max Planck Institute for the Structure
and Dynamics of Matter, Luruper Chaussee 149, 22761 Hamburg, Germany  F.
Orlandi  ISIS Facility, STFC Rutherford Appleton Laboratory, Harwell Science
and Innovation Campus, Oxfordshire OX11 0QX, United Kingdom  P. Manuel  ISIS
Facility, STFC Rutherford Appleton Laboratory, Harwell Science and Innovation
Campus, Oxfordshire OX11 0QX, United Kingdom  S. Blügel  Peter Grünberg
Institut and Institute for Advanced Simulation, Forschungszentrum Jülich &
JARA, D-52425 Jülich, Germany  J. B. Staunton  Department of Physics,
University of Warwick, Coventry, CV4 7AL, United Kingdom  O. A. Petrenko
Department of Physics, University of Warwick, Coventry, CV4 7AL, United
Kingdom  M. R. Lees  Department of Physics, University of Warwick, Coventry,
CV4 7AL, United Kingdom  G. Balakrishnan<EMAIL_ADDRESS>Department of Physics, University of Warwick, Coventry, CV4 7AL, United
Kingdom

###### Abstract

GdRu2Si2 is a centrosymmetric magnet in which a skyrmion lattice has recently
been discovered. Here, we investigate the magnetic structure of the zero field
ground state using neutron diffraction on single crystal and polycrystalline
^160GdRu2Si2. In addition to observing the principal propagation vectors
$\mathbf{q}_{1}$ and $\mathbf{q}_{2}$, we discover higher order magnetic
satellites, notably $\mathbf{q}_{1}+2\mathbf{q}_{2}$. The appearance of these
satellites are explained within the framework of a new double-$Q$ constant-
moment solution. Using powder diffraction we implement a quantitative
refinement of this model. This structure, which contains vortexlike motifs, is
shown to have a one-dimensional topological charge density.

††preprint: APS/123-QED

Skyrmions and merons are two-dimensional (2D) vortexlike spin textures which
have integer and half-integer topological charge respectively [1]. These non-
collinear spin textures exhibit a range of emergent phenomena such as the
topological Hall effect and these properties are anticipated to find use in
low-power spintronics [2, 3]. Magnetic skyrmion research has principally been
focused on noncentrosymmetric materials such as MnSi and Cu2OSeO3, where the
Dzyaloshinskii-Moriya interaction plays a key role in the stabilization of
skyrmions in a magnetic field [4, 5].

In contrast, the interactions stabilizing skyrmions in centrosymmetric
materials are varied and in general less well understood. Notably though,
experiments have determined the formation of skyrmions in a set of
centrosymmetric Gd/Eu-based intermetallics: Gd2PdSi3 [6], Gd3Ru4Al12 [7],
EuAl4 [8], EuGa2Al2 [9], and of focus to this letter GdRu2Si2 [10, 11, 12].
Eu^2+ and Gd^3+ ions have long been established as prototypical Heisenberg
spin systems ($S=7/2,L=0$), these ions are therefore an important inclusion in
this class of materials, although not essential [13]. In Gd2PdSi3 and
Gd3Ru4Al12 geometrical frustration is expected to be a key consideration for
the formation of the skyrmion lattices, whereas for GdRu2Si2, EuAl4 and
EuGa2Al2, which crystallize in the $I4/mmm$ space group, this cannot be the
case.

For a consistent theory of the magnetism in GdRu2Si2, it is necessary to
establish the magnetic structures across the phase diagram shown in Fig. 1.
Resonant elastic x-ray scattering (REXS) studies reported two incommensurate
propagation vectors, denoted as $\mathbf{q}_{1}$ and $\mathbf{q}_{2}$ along
the a and b axes respectively [10, 12]. Initially, Phase 1 was suggested to be
a single-$Q$ helix state and Phase 2 as the square skyrmion lattice, composed
of two superimposed orthogonal helices [10]. An ab initio study predicted that
the incommensurate helical state was due to Fermi surface nesting, reproducing
propagation vectors and a critical temperature ($T_{\mathrm{c}}$) in close
accordance with experiments. Moreover, that study predicted that the skyrmion
lattice could be formed with just first-order Ruderman–Kittel–Kasuya–Yosida
(RKKY) interactions, magnetic anisotropy and Zeeman coupling [14].
Subsequently, a spectroscopic scanning tunneling microscopy experiment
identified double-$Q$ incommensurate ordering in Phase 1, detecting
$\mathbf{q}_{1}+\mathbf{q}_{2}$ and $2\mathbf{q}_{1,2}$ distortions [11]. The
identification of incommensurate electron density coincident with the magnetic
order strengthens arguments for RKKY exchange [14], as opposed to alternative
mechanisms which had previously been suggested [15]. Following the indications
of 2D ordering across the phase diagram, phases 1 and 3 were assigned as meron
lattices with Phase 1 constructed from a helix on $\mathbf{q}_{1}$ and a spin-
density wave (SDW) along $\mathbf{q}_{2}$. To stabilize this ground state,
biquadratic exchange terms are required as shown in the REXS/Monte Carlo
annealing study [12]. No microscopic description has been given for Phase 4,
however, the phase boundaries are detectable using magnetometry. The lack of
information regarding this phase is probably due to the magnetism being weak
with increased disorder in proximity to $T_{\mathrm{c}}$ ($45$ K) and is a
topic of further study.

Figure 1:  (a) Crystal structure of GdRu2Si2, space group: $I4/mmm$. (b) H-T
phase diagram of GdRu2Si2 with $H\parallel c$. Phase boundaries have been
established using a combination of longitudinal resistivity
($\rho_{\mathrm{xx}}$), DC magnetometry ($M_{\mathrm{dc}}$) and AC
susceptibility ($\chi_{\mathrm{ac}}$). This phase diagram is consistent with
previous determinations [16]. Phases 1 and 3 have been nominally designated as
meron lattices and Phase 2 the skyrmion lattice, with no current
interpretation of Phase 4 [12].

In this Letter, the zero field ground state of GdRu2Si2 is examined using
time-of-flight neutron diffraction. Within the $\left(h,k,0\right)$
diffraction plane, magnetic satellites of the form $\mathbf{q}_{1}$,
$\mathbf{q}_{2}$ and $\mathbf{q}_{1}+2\mathbf{q}_{2}$ are found. Furthermore,
the principal propagation vectors $\mathbf{q}_{1}$ and $\mathbf{q}_{2}$ are
shown to be inequivalent in magnitude. These aspects taken together, allow the
analytical construction of a double-$Q$ constant-moment solution, which is
entropically favorable. This model is used in the quantitative refinement of
the powder diffraction, giving a complete magnetic structure solution. The
topological charge density (TCD) of this phase is particularized and shown to
have one-dimensional (1D) topological charge stripes. Finally the implications
of this new magnetic structure on the Hamiltonian are discussed.

To reduce neutron absorption isotopically enriched 160Gd was used.
Polycrystalline 160GdRu2Si2 was prepared by arc melting stoichiometric
quantities of the constituent materials. A single crystal of isotopically-
enriched 160GdRu2Si2, 17 mm in length and 2.3 mm in diameter was grown by the
floating zone method.

Polycrystalline and single crystal neutron diffraction measurements were taken
on the WISH diffractometer at the ISIS Neutron and Muon Source UK [17]. The
measurements were taken in Phase 1 in zero magnetic field and at
$1.5~{}$\mathrm{K}$$ to maximize the magnetic scattering signal.
Polycrystalline 160GdRu2Si2 was ground into a fine powder and placed in a
vanadium can. The single crystal was aligned so that $\left(h,k,0\right)$ was
the principal scattering plane.

Figure 2:  Single crystal neutron diffraction data in the $(h,k,0)$ plane
showing magnetic satellites surrounding the $\left(1,1,0\right)$ nuclear peak
at $1.5~{}$\mathrm{K}$$. Intensity has been scaled so that the nuclear peak
and sets of incommensurate satellites of the form: $\mathbf{q}_{1}$,
$\mathbf{q}_{2}$ and $\mathbf{q}_{1}+2\mathbf{q}_{2}$ (circled in black), can
all be simultaneously observed. The arcs are Al powder diffraction lines from
the sample holder.

Magnetic satellites surrounding the $\left(1,1,0\right)$ nuclear peak are
shown in Fig 2. Attention was given to this peak, in part as it was convenient
to access the sets of magnetic satellites, and also the small scattering
vector ensured that the magnetic form factor was large. The magnetic
diffraction shown is due to two orthogonal magnetic domains, however this is
only clear in subsequent measurements (Fig. 3) which resolve the inequivalence
between the $\mathbf{q}_{1}$ and $\mathbf{q}_{2}$ propagation vectors. The two
domains are denoted with $A$ and $B$ subscripts respectively. The set of
satellites observed is $\left(1\pm q_{1},1,0\right)_{A}$, $\left(1,1\pm
q_{2},0\right)_{A}$, $\left(1\pm q_{1},1\pm 2q_{2},0\right)_{A}$,
$\left(1,1\pm q_{1},0\right)_{B}$, $\left(1\pm q_{2},1,0\right)_{B}$ and
$\left(1\pm 2q_{2},1\pm q_{1},0\right)_{B}$ . See Fig. S1 for a schematic of
how each magnetic domain contributes to the diffraction plane [18]. Whilst the
existence of $\mathbf{q}_{1}$ and $\mathbf{q}_{2}$ satellites had been well
established this is the first instance in which
$\mathbf{q}_{1}+2\mathbf{q}_{2}$ satellites have been observed in a
diffraction experiment. We note that surrounding other nuclear peaks,
satellites of the form $\mathbf{q}_{1}+\mathbf{q}_{2}$ were infrequently
observed (see Fig. S2 for an example [18]). The presence of
$\mathbf{q}_{1}+\mathbf{q}_{2}$ and $\mathbf{q}_{1}+2\mathbf{q}_{2}$
satellites are independent proof of the double-$Q$ magnetic structure, coming
from the evaluation of the possible free-energy coupling terms [19]. The
associated distortions ($\mathbf{q}_{1}+\mathbf{q}_{2}$ and
$\mathbf{q}_{1}+2\mathbf{q}_{2}$) are produced by the two principal arms of
the star ($\mathbf{q}_{1}$ and $\mathbf{q}_{2}$) being coupled together in the
ground state magnetic structure. Moreover, the translation and time reversal
symmetries require a third power invariant for the
$\mathbf{q}_{1}+\mathbf{q}_{2}$ and a fourth power invariant for the
$\mathbf{q}_{1}+2\mathbf{q}_{2}$ distortions. This in turn implies that the
former is of structural origin and the latter is magnetic.

Additionally, the existence of these higher-order satellites indicates that
the magnetic structure may have a constant-moment solution, whereas the
magnetic structure solutions described in this material hitherto have varying
spin-density. Strongly motivating this notion, the saturation magnetization of
GdRu2Si2 has been shown to be $7~{}\mu_{\mathrm{B}}$ [16, 10]. This
substantiates that the magnetism is formed of the localized f-electron moments
of Gd^3+ which interact via valence electrons at the Fermi surface. This model
is inconsistent with a system of varying spin-density, as this would incur a
significant entropic cost due to disorder and increased kinetic energy, which
is not expected in the zero field ground state. Indeed, established SDW
systems ordinarily have particular reasons for how the SDW is stabilized. In
itinerant magnets this is due to wave-vector nesting in the Fermi surface
[20], or in insulators frustrated exchange interactions can generate these
unusual states [21, 22, 23]. However, GdRu2Si2 is in neither of these
categories of materials.

Before the constant-moment solution can be constructed, it is necessary to
establish the inequivalence in magnitude between $\mathbf{q}_{1}$ and
$\mathbf{q}_{2}$ [12]. This is shown in Fig. 3, which shows measurements of
satellites surrounding the Brillouin zone center: $\left(0+q_{1},0,0\right)$
and $\left(0+q_{2},0,0\right)$. These satellites are observed in the same
region of reciprocal space due to the ground state forming in two mutually
orthogonal domains, reducing the symmetry of the parent tetragonal structure.
By the Curie principle, this induces (so far undetected) nuclear distortions
with identical domain formation.

Figure 3:  Single crystal neutron diffraction data showing the splitting
between inequivalent $\mathbf{q}_{1}$ and $\mathbf{q}_{2}$ at
$1.5~{}$\mathrm{K}$$. The peaks have been fit with a pseudo-Voigt function.
These measurements are of the $\left(0+q_{1},0,0\right)$ and
$\left(0+q_{2},0,0\right)$ satellites.

Having established the $\mathbf{q}_{1}+2\mathbf{q}_{2}$ satellites and the
inequivalence between $\mathbf{q}_{1}$ and $\mathbf{q}_{2}$, the constant-
moment solution can be fully described. The ground state of GdRu2Si2 has been
previously proposed to comprise a helix and an orthogonal SDW propagating
along $\mathbf{q}_{1}$ and $\mathbf{q}_{2}$, respectively [12]. The
construction of a constant-moment solution includes the same helix and SDW
with the addition of a second helix, coplanar with the first, but propagating
along $\mathbf{q}_{1}+2\mathbf{q}_{2}$ as an ansatz. The magnetic moment
$\mathbf{m_{r}}$, at a position vector $\mathbf{r}$ is then given by,

$\mathbf{m_{r}}=\begin{bmatrix}0\\\ m_{1}\cos(k_{1})\\\
m_{1}\sin(k_{1})\end{bmatrix}+\begin{bmatrix}m_{2}\sin(k_{2})\\\ 0\\\
0\end{bmatrix}+\begin{bmatrix}0\\\ m_{3}\cos(k_{1}+2k_{2})\\\
m_{3}\sin(k_{1}+2k_{2})\end{bmatrix}.$ (1)

Here, $k_{1,2}=-2\pi\bm{q_{1,2}\cdot r}$ and $m_{1,2,3}$ are scalar
amplitudes. Note that Eq. 1 holds for any choice of two propagation vectors,
and that these planar waves can be defined on an arbitrary orthonormal basis.
For a constant-moment solution $\mathbf{m_{r}}\cdot\mathbf{m_{r}}$ is
evaluated,

$\mathbf{m_{r}}\cdot\mathbf{m_{r}}=m_{1}^{2}+m_{3}^{2}+4m_{1}m_{3}\cos^{2}(k_{2})-2m_{1}m_{3}+m_{2}^{2}\sin^{2}(k_{2}),$
(2)

with the condition $m_{3}=m_{2}^{2}/4m_{1}$. Hence a constant-moment solution
can be provided in the form of Eq. 1 with an appropriate moment condition.

Figure 4:  Rietveld refinement of GdRu2Si2 at $1.5~{}$\mathrm{K}$$. Red points
are the data, the black curve is the model and the blue line is the difference
curve. Green and purple bars indicate the nuclear and magnetic Bragg peaks,
respectively. This pattern is from the high-resolution bank of the WISH
diffractometer, where $\mathbf{q}_{1}$ and $\mathbf{q}_{2}$ can be resolved,
as highlighted in the inset. $R_{wp}=15.3\%$ and $R_{exp}=8.73\%$ and
$GOF=1.75$.

Quantitative evaluation of the above model was implemented by analyzing the
powder neutron diffraction (Fig. 4) using fullprof [24]. The most appropriate
choice of basis was found to be $\mathbf{\hat{a}}=[1,0,0]$,
$\mathbf{\hat{b}}=[0,1,0]$, and $\mathbf{\hat{c}}=[0,0,1]$, with
$\mathbf{q}_{1}=[0.22540(2),0,0]$ and $\mathbf{q}_{2}=[0,0.22014(3),0]$. The
scalar magnitudes were found to be: $m_{1}=(6.49\pm 0.10)~{}\mu_{\mathrm{B}}$,
$m_{2}=(5.44\pm 0.16)~{}\mu_{\mathrm{B}}$ and $m_{3}=(1.14\pm
0.09)~{}\mu_{\mathrm{B}}$. Note that in the powder data there were
insufficient statistics to independently fit the
$\mathbf{q}_{1}+2\mathbf{q}_{2}$ satellite peaks, and therefore the value of
$m_{3}$ is implicit through the derived moment condition. From Eq. 2 this
gives a total moment size of $\left|\mathbf{m_{r}}\right|=(7.6\pm
0.1)~{}\mu_{\mathrm{B}}$. The discrepancy between the measured moment size and
the $7~{}\mu_{\mathrm{B}}$ of Gd^3+ is caused by a systematic error,
attributed to the high levels of absorption in the sample, which is heavily
correlated to the scale factor and in turn the refined moment size. This
magnetic structure is illustrated in Fig. 5, where periodic vortexlike motifs
are observed.

The topological charge structure is now examined. Our analysis is based in the
continuous field limit, but we note that in the more discrete limit of a small
unit cell such as in this system, the notion of TCD is less clear. TCD is
given by,

$\Omega=\mathbf{\hat{m_{r}}}\cdot\frac{\partial\mathbf{\hat{m_{r}}}}{\partial
x}\times\frac{\partial\mathbf{\hat{m_{r}}}}{\partial y},$ (3)

where $\mathbf{\hat{m_{r}}}=\mathbf{m_{r}}/\left|\mathbf{m_{r}}\right|$.
Substituting Eq. 1 into Eq. 3 yields,

$\Omega=\frac{(2\pi)^{2}4q_{1}q_{2}m_{1}m_{2}}{4m_{1}^{2}+m_{2}^{2}}\cos(-2\pi
q_{2}y).$ (4)

Here the Cartesian axes are associated with the $\mathbf{\hat{a}}$,
$\mathbf{\hat{b}}$ and $\mathbf{\hat{c}}$ directions. Hence the TCD is locally
non-trivial, 1D, and oscillates in the same direction as the SDW component of
the magnetic structure. We refer to this defining feature as topological
charge stripes. Additionally, this establishes that the spin texture of this
ground state cannot be compatible with a periodic meron or skyrmion lattice.
This is since these spin textures would have a 2D periodic TCD structure, over
which a 2D integral to evaluate the topological charge can be well defined
with periodic boundary conditions. With the TCD structure given in Eq. 4, a 2D
integral to evaluate the topological charge over this scalar field would be
arbitrary given there is only periodicity in a single direction. We expect
that these topological charge stripes may produce novel transport phenomena,
in which there may be significant anisotropy perpendicular and parallel to the
propagation of the TCD due to the topological Hall effect.

Figure 5:  A model of the magnetic structure, shown over $8\times 8$ unit
cells. The out-of-plane component of the magnetic moment $m_{c}$ is shown with
the blue-red color scale. The topological charge density is represented with
the black-white color scale, showing the topological charge stripes.

A first principles analysis of the Fourier transform of the magnetic
interactions in GdRu2Si2 assuming fourfold symmetry (without magneto-striction
effects) reveals the presence of peaks at four characteristic wave vectors
$\mathbf{q}_{1}=(\pm q_{\text{max}},0,0)$ and $\mathbf{q}_{2}=(0,\pm
q_{\text{max}},0)$ [14]. This indicates the possibility of stabilizing
single-$Q$ states at multiple wave vectors. The inclusion of multiple-spin
interactions leads to the stabilization of multiple-$Q$ states [25].
Particularly, the biquadratic interactions play an important role in the
stabilization of anisotropic double-$Q$ states [12]. We postulate that the
biquadratic interactions together with other more complex isotropic multiple-
spin interactions, such as three-site four spin interactions [26] and ring
exchange [27] can generate the complex magnetic structure reported here [28].
These higher-order spin interactions derive from the feedback effect on the
valence electrons from the development of the complex magnetic order of the Gd
local moments [29].

In conclusion, the zero field ground state of GdRu2Si2 has been studied using
time-of-flight neutron diffraction. In addition to observing the principal
magnetic satellites $\mathbf{q}_{1}$ and $\mathbf{q}_{2}$ we discover
satellites of the form $\mathbf{q}_{1}+2\mathbf{q}_{2}$. These magnetic
satellites allow the analytical construction and quantitative refinement of a
constant-moment solution, which is entropically favorable. The principles upon
which the constant-moment solution is constructed are expected to be a novel
aid in determining the complex nature of multi-$Q$ magnetic structures. The
TCD of this magnetic structure is shown to be locally non-trivial, 1D, and it
is not consistent with either meron or skyrmion lattices. We speculate that
these topological charge stripes may give rise to novel magneto-electronic
coupling, such as producing electronic nematicity. These results establish
that GdRu2Si2 has a wealth of topologically non-trivial spin textures and is
therefore an ideal setting in which phase transitions between distinct
topological structures can be experimentally probed. An interpretation of the
in-field phases and particularly Phase 4 remains a topic of further study.

###### Acknowledgements.

We would like to acknowledge Tom Orton and Patrick Ruddy for their technical
support. This Letter was financially supported by two Engineering and Physical
Sciences Research Council grants: EP/T005963/1, and the UK Skyrmion Project
Grant, EP/N032128/1. This Letter was supported by U.K. Research and Innovation
and Science and Technology Facilities Council through the provision of beam
time at the ISIS Neutron and Muon Source [30], as well as partial funding for
the ^160Gd isotope used in the neutron experiment.

## References

  * Lancaster [2019] T. Lancaster, Skyrmions in magnetic materials, Contemp. Phys. 60, 246 (2019).
  * Neubauer  _et al._ [2009] A. Neubauer, C. Pfleiderer, B. Binz, A. Rosch, R. Ritz, P. Niklowitz, and P. Böni, Topological Hall effect in the A phase of MnSi, Phys. Rev. Lett. 102, 186602 (2009).
  * Fert  _et al._ [2017] A. Fert, N. Reyren, and V. Cros, Magnetic skyrmions: advances in physics and potential applications, Nat. Rev. Mater. 2, 1 (2017).
  * Mühlbauer  _et al._ [2009] S. Mühlbauer, B. Binz, F. Jonietz, C. Pfleiderer, A. Rosch, A. Neubauer, R. Georgii, and P. Böni, Skyrmion lattice in a chiral magnet, Science 323, 915 (2009).
  * Seki  _et al._ [2012] S. Seki, X. Yu, S. Ishiwata, and Y. Tokura, Observation of skyrmions in a multiferroic material, Science 336, 198 (2012).
  * Kurumaji  _et al._ [2019] T. Kurumaji, T. Nakajima, M. Hirschberger, A. Kikkawa, Y. Yamasaki, H. Sagayama, H. Nakao, Y. Taguchi, T.-H. Arima, and Y. Tokura, Skyrmion lattice with a giant topological hall effect in a frustrated triangular-lattice magnet, Science 365, 914 (2019).
  * Hirschberger  _et al._ [2019] M. Hirschberger, T. Nakajima, S. Gao, L. Peng, A. Kikkawa, T. Kurumaji, M. Kriener, Y. Yamasaki, H. Sagayama, H. Nakao, _et al._ , Skyrmion phase and competing magnetic orders on a breathing kagomé lattice, Nat. Commun. 10, 5831 (2019).
  * Takagi  _et al._ [2022] R. Takagi, N. Matsuyama, V. Ukleev, L. Yu, J. S. White, S. Francoual, J. R. Mardegan, S. Hayami, H. Saito, K. Kaneko, _et al._ , Square and rhombic lattices of magnetic skyrmions in a centrosymmetric binary compound, Nat. Commun. 13, 1 (2022).
  * Moya  _et al._ [2022] J. M. Moya, S. Lei, E. M. Clements, C. S. Kengle, S. Sun, K. Allen, Q. Li, Y. Y. Peng, A. A. Husain, M. Mitrano, M. J. Krogstad, R. Osborn, A. B. Puthirath, S. Chi, L. Debeer-Schmitt, J. Gaudet, P. Abbamonte, J. W. Lynn, and E. Morosan, Incommensurate magnetic orders and topological hall effect in the square-net centrosymmetric EuGa2Al2 system, Phys. Rev. Mater. 6, 074201 (2022).
  * Khanh  _et al._ [2020] N. D. Khanh, T. Nakajima, X. Yu, S. Gao, K. Shibata, M. Hirschberger, Y. Yamasaki, H. Sagayama, H. Nakao, L. Peng, _et al._ , Nanometric square skyrmion lattice in a centrosymmetric tetragonal magnet, Nat. Nanotechnol. 15, 444 (2020).
  * Yasui  _et al._ [2020] Y. Yasui, C. J. Butler, N. D. Khanh, S. Hayami, T. Nomoto, T. Hanaguri, Y. Motome, R. Arita, T.-H. Arima, Y. Tokura, _et al._ , Imaging the coupling between itinerant electrons and localised moments in the centrosymmetric skyrmion magnet GdRu2Si2, Nat. Commun. 11, 5925 (2020).
  * Khanh  _et al._ [2022] N. D. Khanh, T. Nakajima, S. Hayami, S. Gao, Y. Yamasaki, H. Sagayama, H. Nakao, R. Takagi, Y. Motome, Y. Tokura, T.-H. Arima, and S. Seki, Zoology of multiple-Q spin textures in a centrosymmetric tetragonal magnet with itinerant electrons, Adv. Sci. 9, 2105452 (2022).
  * Hou  _et al._ [2021] Z. Hou, L. Li, C. Liu, X. Gao, Z. Ma, G. Zhou, Y. Peng, M. Yan, X.-X. Zhang, and J. Liu, Emergence of room temperature stable skyrmionic bubbles in the rare earth based REMn2Ge2 (RE = Ce, Pr, and Nd) magnets, Mater. Today Phys. 17, 100341 (2021).
  * Bouaziz  _et al._ [2022] J. Bouaziz, E. Mendive-Tapia, S. Blügel, and J. B. Staunton, Fermi-surface origin of skyrmion lattices in centrosymmetric rare-earth intermetallics, Phys. Rev. Lett. 128, 157206 (2022).
  * Nomoto  _et al._ [2020] T. Nomoto, T. Koretsune, and R. Arita, Formation mechanism of the helical $\mathit{Q}$ structure in Gd-based skyrmion materials, Phys. Rev. Lett. 125, 117204 (2020).
  * Garnier  _et al._ [1996] A. Garnier, D. Gignoux, D. Schmitt, and T. Shigeoka, Giant magnetic anisotropy in tetragonal GdRu2Ge2 and GdRu2Si2, Physica B 222, 80 (1996).
  * Chapon  _et al._ [2011] L. C. Chapon, P. Manuel, P. G. Radaelli, C. Benson, L. Perrott, S. Ansell, N. J. Rhodes, D. Raspino, D. Duxbury, E. Spill, _et al._ , WISH: The new powder and single crystal magnetic diffractometer on the second target station, Neutron News 22, 22 (2011).
  * [18] See Supplemental Material at [link to Supplemental] for S1: Schematic of how the two magnetic domains contribute to the diffraction plane. S2: Single crystal neutron diffraction data in the $(h,k,0)$ plane showing satellites surrounding the $(-1,1,0)$ nuclear peak.
  * [19] Y. A. Izyumov and V. N. Syromyatnikov,  _Phase transitions and crystal symmetry_ (Kluwer Academic Publishers 1990).
  * Fawcett [1988] E. Fawcett, Spin-density-wave antiferromagnetism in chromium, Rev. Mod. Phys. 60, 209 (1988).
  * Kimura  _et al._ [2008] S. Kimura, M. Matsuda, T. Masuda, S. Hondo, K. Kaneko, N. Metoki, M. Hagiwara, T. Takeuchi, K. Okunishi, Z. He, K. Kindo, T. Taniyama, and M. Itoh, Longitudinal spin density wave order in a quasi-1D ising-like quantum antiferromagnet, Phys. Rev. Lett. 101, 207201 (2008).
  * Canévet  _et al._ [2013] E. Canévet, B. Grenier, M. Klanjšek, C. Berthier, M. Horvatić, V. Simonet, and P. Lejay, Field-induced magnetic behavior in quasi-one-dimensional ising-like antiferromagnet BaCo2V2O8: A single-crystal neutron diffraction study, Phys. Rev. B 87, 054408 (2013).
  * Facheris  _et al._ [2022] L. Facheris, K. Y. Povarov, S. D. Nabi, D. G. Mazzone, J. Lass, B. Roessli, E. Ressouche, Z. Yan, S. Gvasaliya, and A. Zheludev, Spin density wave versus fractional magnetization plateau in a triangular antiferromagnet, Phys. Rev. Lett. 129, 087201 (2022).
  * Rodríguez-Carvajal [1993] J. Rodríguez-Carvajal, Recent advances in magnetic structure determination by neutron powder diffraction, Physica B 192, 55 (1993).
  * Hayami and Motome [2021] S. Hayami and Y. Motome, Topological spin crystals by itinerant frustration, J. Phys.: Condens. Matter 33, 443001 (2021).
  * Hoffmann and Blügel [2020] M. Hoffmann and S. Blügel, Systematic derivation of realistic spin models for beyond-Heisenberg solids, Phys. Rev. B 101, 024418 (2020).
  * Heinze  _et al._ [2011] S. Heinze, K. Von Bergmann, M. Menzel, J. Brede, A. Kubetzka, R. Wiesendanger, G. Bihlmayer, and S. Blügel, Spontaneous atomic-scale magnetic skyrmion lattice in two dimensions, Nat. Phys. 7, 713 (2011).
  * Sharma  _et al._ [2022] V. Sharma, Z. Wang, and C. D. Batista, Machine learning assisted derivation of effective low energy models for metallic magnets, arXiv preprint arXiv:2212.09796 (2022).
  * Mendive-Tapia and Staunton [2017] E. Mendive-Tapia and J. B. Staunton, Theory of magnetic ordering in the heavy rare earths: Ab initio electronic origin of pair- and four-spin interactions, Phys. Rev. Lett. 118, 197202 (2017).
  * Balakrishnan  _et al._ [2022] G. Balakrishnan, G. D. A. Wood, P. Manuel, A. E. Hall, F. Orlandi, D. D. Khalyavin, D. A. Mayoh, O. A. Petrenko, and M. R. Lees, The skyrmion lattice and other non-collinear spin structures in GdRu2Si2, STFC ISIS Facility 10.5286/ISIS.E.RB2210259 (2022).


# M2SA: Multimodal and Multilingual Model for Sentiment Analysis of Tweets

###### Abstract

In recent years, multimodal natural language processing, aimed at learning
from diverse data types, has garnered significant attention. However, there
needs to be more clarity when it comes to analysing multimodal tasks in multi-
lingual contexts. While prior studies on sentiment analysis of tweets have
predominantly focused on the English language, this paper addresses this gap
by transforming an existing textual Twitter sentiment dataset into a
multimodal format through a straightforward curation process. Our work opens
up new avenues for sentiment-related research within the research community.
Additionally, we conduct baseline experiments utilising this augmented dataset
and report the findings. Notably, our evaluations reveal that when comparing
unimodal and multimodal configurations, using a sentiment-tuned large language
model as a text encoder performs exceptionally well.

Keywords: sentiment analysis, multilingual, multimodal

M2SA: Multimodal and Multilingual Model for Sentiment Analysis of Tweets

Gaurish Thakkar1, Sherzod Hakimov2, Marko Tadić1
---
1Faculty of Humanities and Social Sciences, University of Zagreb
2Computational Linguistics, University of Potsdam
{gthakkar<EMAIL_ADDRESS><EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

Social media platforms serve as conduits for the dissemination of information.
Tweets have emerged as a trendy medium through which individuals communicate
and express their ideas and opinions. Twitter (aka X) is widely used by
researchers as a prominent social media platform for engaging in micro-
blogging and fostering interactions. Sentiment analysis (Pang and Lee, 2005)
is a well-studied topic in natural language processing. The topic has received
consideration in both unimodal and multimodal contexts. The proliferation of
social media platforms, including Twitter and YouTube, has led to a common
practice of assessing content using several modalities (You et al., 2016; Yu
et al., 2020). This approach offers additional context through spoken, non-
verbal, and auditory aspects. The primary focus in many domains of natural
language processing (NLP) often revolves around higher-resourced languages.
However, the challenge of processing lower-resourced languages remains
unresolved.

The process of annotating supervised datasets for natural language processing
(NLP) tasks is a labour-intensive endeavour requiring significant investment
of time, financial resources, and effort. Recently, several shared tasks,
including SemEval (Nakov et al., 2013a; Ghosh et al., 2015), have introduced
tasks aimed at identifying the polarity of tweets, categorising them into
predetermined classes. All of the datasets for the shared tasks are
accompanied by labels considered the gold standard. Another point to take into
account here is that previous approaches (Raffel et al., 2019; Xie et al.,
2020; Cliche, 2017) focused on text-only, while posts shared on social media
sometimes include images, videos, etc. Approaches incorporating multimodal
information (Poria et al., 2016b; Cheema et al., 2021; Poria et al., 2016a)
for the classification of sentiment are predominantly focused on the English
language.

This paper presents a straightforward approach for enhancing pre-existing
publicly accessible datasets to conduct multimodal (image & text) sentiment
analysis on Twitter called M2SA (Multimodal Multilingual Sentiment Analysis).
We have collected existing datasets in 21 languages where each annotated post
includes both text and image with the annotated labels being either positive,
negative, or neutral. We then trained a multimodal model that combines image
and text embedding features to classify the target labels.

Our contributions are as follows:

  * •

We engage in curating, enriching, and analysing pre-existing Twitter sentiment
datasets in 21 different languages.

  * •

The pre-trained model architectures use a fusion of textual information and
visual features, utilising large language models for text encoding and image
encoding.

  * •

The study examines the effects of utilising machine translation instances in
the context of lower-resourced languages.

All resources (pre-trained models, datasets) and the source code are shared
publicly111https://github.com/cleopatra-itn/M2SA-multimodal-multilingual-
sentiment-analysis. The subsequent sections of the paper are structured in the
following manner: Section 2 provides an overview of the existing literature
and research in the field. Section 3 provides a comprehensive overview of the
processes involved in data collection, enrichment, and the statistical
characteristics of the dataset. The methodology for classification is outlined
in Section 4. The experimental setup and results are outlined in Sections 5
and 6. The paper concludes in Section 7.

## 2\. Related Work

Poria et al. (2016b) presented a framework that uses CNNs to extract features
from multimodal data’s visual and textual modalities. The visual features are
extracted using a CNN model that has been previously trained, such as VGG16 or
ResNet-50. A CNN model trained on a massive corpus of text data is used to
extract the textual features. The combined extracted features from the visual
and textual modalities are then fed into an MKL classifier. The MKL classifier
discovers the optimal combination of kernels for distinguishing between
distinct emotions or sentiments. Poria et al. (2016a) used both feature and
decision-level fusion methods to merge affective information extracted from
multiple modalities. Cheema et al. (2021) evaluated various embedding features
from both text and visual content. Huang et al. (2023) proposed a new
framework for multimodal sentiment analysis in realistic environments, with
two main components: a module for multimodal word refinement and a module for
cross-modal hierarchical fusion. Baecchi et al. (2016) employed a strategy
that uses a skip-gram neural network to extract features from the text mode.
Image-specific features are extracted using a denoising autoencoder (Vincent
et al., 2010) neural network. The denoising autoencoder network is taught to
reconstruct an image from its corrupted version. The extracted features from
the text and image modalities are then concatenated and fed to an SVM
classifier. In addition to considering the modelling strategies for the
sentiment analysis task, it is essential to identify the available
benchmarking datasets. English contains a substantial quantity of multimodal
datasets on sentiment and emotion analysis (Go et al., 2009; Mohammad et al.,
2018). While the TweetEval (Barbieri et al., 2020) examines the application of
large language models to seven tasks in Twitter, including emotion, emoji,
irony sentiment, and others, the test set is monolingual. The paper authored
by Garg et al. (2022) provides a comprehensive exposition on diverse
multimodal datasets, encompassing the domain of multimodal sentiment analysis.

## 3\. Multimodal Multilingual Sentiment Analysis (M2SA)

Figure 1: The dataset’s distribution across different languages.

According to our investigation, numerous datasets are available for conducting
both unimodal and multimodal sentiment analysis. The conversion of unimodal
datasets, particularly those derived from Twitter, to a multimodal format has
been limited. The fundamental hypothesis underlying our utilisation of the
unimodal dataset posits that, given its gold annotation, the Twitter dataset
can be linked to an image that has not been previously examined or employed in
the context of multimodal sentiment classification. Thus, we present our
contribution in this field called M2SA (Multimodal Multilingual Sentiment
Analysis).

### 3.1. Data Collection

An initial step in initiating the enrichment process involves conducting a
manual search of pre-existing Twitter sentiment datasets. We do not target any
other social media datasets to process them uniformly and keep them from a
single source. This search is conducted through the utilisation of search
engines and data repositories such as HuggingFace
Datasets222https://huggingface.co/datasets, European Language
Grid333https://live.european-language-grid.eu, and
GitHub444https://github.com/. To retrieve the dataset, a search is conducted
using specific keywords such as twitter sentiment analysis dataset, social
media sentiment analysis dataset, and twitter sentiment shared tasks. Next,
the compiled list of datasets undergoes the process of querying tweet
information using the Twitter API. The JSON format is used to store the text
and images associated with each individual tweet in a dataset. The initially
collected datasets are then subjected to manual checking to exclude tasks
unrelated to sentiment analysis. Lastly, label transformations were applied on
class labels to convert them from a five-class to a three-class format in
cases where they did not initially possess three distinct sentiment
categories: positive, negative, and neutral. The preliminary investigation
yielded approximately 100 datasets in multiple languages. However, the final
version of our dataset consisted of only 56 distinct datasets, encompassing 21
different languages. This reduction in the number of datasets was primarily
due to the absence of tweet IDs linked to the corresponding text in most
datasets. Table 1 presents a comprehensive overview of various languages and
their respective datasets that were collected.

### 3.2. Preprocessing

Preprocessing social media texts is imperative due to their inherent
informality and noise. The preprocessing steps are delineated as follows:

  * •

Removal of all black and white images.

  * •

Tweet normalisation for USERs, URLs and HASHTAGS i.e., replace @ElonMusk →
<user_1>…, URLs → <URL_1>, #tweet → <hashtag>tweet<hashtag/>

  * •

Filtering of tweets with text content less than five characters, not
accounting for USER and URL tags.

  * •

Deduplication is performed using tweet IDs.

  * •

Checking if the same tweet ID has more than one label assigned and employing a
majority vote when needed.

  * •

Filtering of tweets with corrupted or no images or with images of less than
200 × 200 pixels size.

  * •

Checking the language tag in the tweet JSON and see if it matches the target
language.

  * •

Translation of English tweets for lower-resourced languages using the
NLLB555https://huggingface.co/facebook/nllb-200-3.3B machine translation (MT)
model.

The complete preprocessed dataset is structured according to a schema that can
be described as follows:

  * •

tweetid: unique identifier for the tweet.

  * •

normalised-text: text obtained after applying preprocessing steps.

  * •

language: the language of the text.

  * •

translated-text: text in the target language obtained using the NLLB model.

  * •

image-paths: list of images associated with the tweet.

  * •

label: POSITIVE|NEGATIVE|NEUTRAL

### 3.3. Dataset

Lang |  Dataset name
---|---
ar |  SemEval-2017
ar |  TM-Senti@ar
bg |  Twitter-15@Bulgarian
bs |  Twitter-15@Bosnian
da |  AngryTweets
de |  xLiMe@German, Twitter-15@German, TM-Senti@de
en |  SemEval-2013-task2, SemEval-2015, SemEval-2016
en |  CB COLING2014 vanzo
en |  CB IJCOL2015 ENG castellucci
en |  RETWEET
es |  xLiMe@spanish
es |  Copres14
es |  mavis@tweets
es |  Twitter-15@Spanish
es |  JOSA corpus
es |  TASS 2018, 2019, 2020
es |  TASS 2012, 2013, 2014, 2015
fr |  DEFT 2015
hr |  InfoCoV-Senti-Cro-CoV-Twitter
hr |  Twitter-15@Croatian
hu |  Twitter-15@Hungarian
it |  CB IJCOL2015 ITA castellucci
it |  xLiMe@Italian
it |  sentipolc16
it |  TM-Senti@it
lv |  Latvian tweet corpus
mt |  Malta-Budget-2018, 2019, 2020
pl |  Twitter-15@Polish
pt |  Twitter-15@Portuguese
pt |  Brazilian tweet@tweets
ru |  Twitter-15@Russian
sq |  Twitter-15@Albanian
sr |  doiserbian@tweet
sr |  Twitter-15@Serbian
sv |  Twitter-15@Swedish
tr |  BounTi Turkish
zh |  TM-Senti@zh-ids
Table 1:  Languages and their corresponding dataset names

Figure 1 illustrates the comprehensive distribution of datasets across
different classes, encompassing 21 languages. The final dataset consists of
143K data points.

The dataset contains the following languages: Arabic-ar (Nakov et al., 2013b;
Yin et al., 2021), Bulgarian-bg (Mozetič et al., 2016), Bosnian-bs (Mozetič et
al., 2016), Danish-da (Pauli et al., 2021), German-de (Rei et al., 2016),
English-en (Nakov et al., 2013c; Ghosh et al., 2015; Nakov et al., 2013d, a;
Vanzo et al., 2014; Castellucci et al., 2015; Tayebi Arasteh et al., 2021),
Spanish-es (Adrián, 2016-05-25; Santamaría et al., 2022; Agüero-Torales et
al., 2021; Villena-Román and Garcıa-Morera, 2013; Román et al., 2015; Vilares
et al., 2015; Montejo-Ráez and Díaz-Galiano, 2016; Cámara et al., 2018; Díaz-
Galiano et al., 2019; García-Vegaa et al., 2020), French-fr (Vukotić et al.,
2015), Croatian-hr (Babić et al., 2021), Hungarian-hu (Mozetič et al., 2016),
Italian-it (Moctezuma et al., 2016) , Maltese-mt (Cortis and Davis, 2021),
Polish-pl (Mozetič et al., 2016), Portuguese-pt (Patrick et al., 2022),
Russian-ru (Mozetič et al., 2016), Serbian-sr (Ljajić and Marovac, 2018),
Swedish-sv (Mozetič et al., 2016), Turkish-tr (Mutlu and Özgür, 2022),
Chinese-zh (Yin et al., 2021), Latvian-lv (Muischnek and Müürisep, 2018) and
Albanian-sq (Mozetič et al., 2016). Languages with more data points, such as
German, Spanish, English, Italian, Arabic, and Polish, possess dataset
instances exceeding 10,000, whereas other languages exhibit 5,000 or fewer
instances of text and images. The dataset exhibits an average token count
ranging from 4.25 to 5.94 words, separating each token by a space. One
observable pattern is that tweets classified as positive tend to be more
likely to be accompanied by images than other categories. The diagram also
indicates an imbalance in the datasets across the languages.

## 4\. Methodology

Figure 2: Model architecture of the M2SA.

### 4.1. Problem Definition

In the task of unimodal sentiment analysis, the model receives a sequence
$X_{m}$ as input, where $m$ represents the length of the sequence. The model
then produces a single class as output, which belongs to a closed set
consisting of positive, negative, or neutral sentiments. In the context of
multimodal sentiment analysis, the model receives input from multiple
modalities denoted as $X^{1}_{m}\ldots X^{2}_{n}$, and the output is
equivalent to that of unimodal sentiment analysis. The objective of the models
is to extract features from the input vectors and acquire the ability to
classify sentiment accurately.

The model architecture of the overall sentiment classification system is
depicted in Figure 2. We examine distinct computational scenarios,
encompassing the analysis of textual data alone and the integration of both
textual and visual information, to classify the sentiment expressed in tweets.
In the context of unimodal textual experiments, the models employed include
Multilingual-BERT (Devlin et al., 2019a), XLM-RoBERTa (Conneau et al., 2020a),
and XLMR-SM, a fine-tuned model specifically designed for sentiment analysis.
In the context of multimodal systems, pre-trained vision models (CLIP and
DINOv2) are employed as feature extractors. The combined textual and visual
features are modelled using a concatenation operator.

In the context of language processing, datasets pertaining to a specific
language are regarded as a cohesive entity. The dataset containing train,
validation, and test sets is utilised directly within their respective sets.
In this scenario, if there is no distinct set available, we will partition it
into train (85%), test (10%), and validation (5%) sets manually.

Given that the text has already undergone processing, no additional processing
has been applied to it. The input text undergoes tokenisation, during which it
is padded and truncated according to the maximum length supported by the
language models. The ’input id’ and ’attention mask’ are passed into the
Language Model (LLM) to extract textual features for each instance in the
dataset. Regarding image modality, the images undergo preprocessing through an
image preprocessor linked to the corresponding vision models. The image
preprocessor’s output is subsequently inputted into the vision encoder. The
concatenation of the output from the text and vision encoder is subsequently
projected onto a linear layer, which is then followed by a softmax layer for
the purpose of classification. The models undergo separate fine-tuning
processes using a combined dataset comprising samples from multiple languages.
Furthermore, given the limited amount of data available for languages such as
Latvian and Albanian, we opt to employ translation techniques to convert the
existing text from another dataset into these languages with fewer than 10,000
tweets. Prior to the translation process, a language detection procedure using
an existing model666papluca/xlm-roberta-base-language-detection is executed.
This process aims to accurately identify the source text, as the dataset
contains text from various languages. Machine translation models rely on
providing source and target language codes to perform translations
effectively. In the context of the dataset, a language detection process is
conducted to classify the instances into their respective source languages.
Subsequently, the machine translation pipeline receives each grouped set along
with the corresponding source and target language codes. The subsequent
subsections discuss the model architectures and pertinent details associated
with the models.

### 4.2. Text Encoders

To evaluate the model’s efficacy in the absence of visual features, we
conducted a standard fine-tuning process utilising the Transformer model’s
output. Specifically, we employed contextualised sentence embedding, which
consists of a 768-element vector. The vector is subsequently fed into a fully
connected (FC) layer consisting of three neurons, which is accompanied by a
softmax layer for classification. The subsequent text models were employed as
encoders for textual data.

#### 4.2.1. Multilingual-BERT (M-BERT)

BERT (Devlin et al., 2019b) is a bidirectional transformer pre-trained with
the masked language modelling (MLM) and next sentence prediction (NSP)
objectives on the top 104 languages with the largest Wikipedia. This model is
chosen due to its multilingual nature.

#### 4.2.2. XLM-RoBERTa (XLM-R)

The XLM-R model (Conneau et al., 2020b) is a large multilingual language model
trained on 2.5 TB of filtered Common Crawl data containing 100 languages. The
model was trained with the Masked Language Modelling (MLM) objective, with 15%
of the input words masked. The model has been shown to perform really well on
downstream tasks when fine-tuned for supervised tasks. XLM-R can understand
the input’s language solely based on the input IDs without having to use
language tensors. This model has been proven to improve the M-BERT scores in
various tasks.

#### 4.2.3. XLM-RoBERTa-Sentiment-Multilingual (XLMR-SM)

The XLMR-SM model (Antypas et al., 2022) is a fine-tuned version of XLM-T
(Barbieri et al., 2022) on the tweet sentiment multilingual dataset (all),
which consists of text from the following languages: Arabic, English, French,
German, Hindi, Italian, Portuguese, and Spanish. The XLM-T model has been pre-
trained on approximately 198 million multilingual tweets. We introduce this
model to study the effect of the presence of sentiment in the pre-trained
encoder. Since this model is based on XLM-R, which is trained on tweets and
fine-tuned on sentiment datasets, it should perform better at the
classification task.

### 4.3. Vision Encoders

The vision encoders divide an image into fixed-size segments and turn them
into a sequence that the model can interpret. The encoder analyses the links
between these image patches to capture the image’s overall meaning, much like
transformers do with text. The visual features obtained from the vision models
are combined with those obtained from the Transformers text models. The
combined output from the encoders is projected into the latest shared space
and fine-tuned on a supervised dataset. We employed the following vision
encoder models:

#### 4.3.1. CLIP

The CLIP model, as described in the paper by Radford et al. (2021), is a
multimodal framework that combines visual and linguistic information. The CLIP
model utilises a transformer architecture, specifically the Vision Transformer
(ViT), to extract visual features. Additionally, it employs a causal language
model to acquire text features. Consequently, the textual and visual
attributes are subsequently mapped onto a latent space with equivalent
dimensionality. Computing the similarity score involves calculating the dot
product between the projected image and text features.

#### 4.3.2. DINOv2

The DINOv2 (Oquab et al., 2023) model is a self-supervised learning approach
that builds upon the DINO framework proposed by Caron et al. (2021). The
dataset utilised for pre-training purposes is meticulously curated to
encompass a diverse range of images sourced from various domains and
platforms, including but not limited to natural images, social media images,
and product images. This ensures that the acquired features can be applied to
diverse practical scenarios.

## 5\. Experimental Setup

In this section, we provide details about the implementation and
configurations that we used to train the model architecture.

### 5.1. Implementation

The neural network’s implementation is founded on the PyTorch library. The
pre-trained models in the HuggingFace model hub are utilised through direct
API calls. All monolingual models employed a batch size of 8 and a learning
rate of 3$e^{-5}$. All experiments were conducted using an NVIDIA V100 GPU
with a memory capacity of 16GB. The translation module employed the
NLLB-200-3.3B model (Costa-jussà et al., 2022), which encompasses all the
languages in the dataset that are considered lower-resourced. All multilingual
models employed a learning rate of 5$e^{-5}$.

### 5.2. Model Configurations

We used the following configurations to train the model architecture and
evaluate the results:

  * •

Unimodal vs. Multimodal: First, we experiment with training the unimodal model
by using only the text. In another configuration, we train the model using
tweets’ image and text content. Such a model considers both modalities and
predicts the sentiment label jointly.

  * •

Original data vs. Inclusion of translations: In one configuration, we used
only the extracted tweets as input for the text encoder to train the model. As
shown in Figure 1, not all languages within the curated dataset possess many
instances that can be utilised for training purposes. Therefore, the original
tweets are machine-translated from English into the target language, and we
combine the original text with the translations to train the models for lower-
resourced languages.

  * •

Monolingual vs. Multilingual: In the monolingual setting, we train separate
models for each language using data only from the respective language (either
the original data or the addition of translations). In the multilingual
setting, the data for all languages are merged, and we train a single model
for all languages.

## 6\. Evaluation

In this section, we analyse the outcomes produced by the aforementioned
configurations. Additionally, we proceed to analyse the obtained results. The
training and early-stopping of the train set are conducted based on the loss
observed on the validation set. Final scoring is performed using the test set.
The experiments were conducted with five different random seeds (42, 123, 777,
2020, 31337), and the resulting macro F1 scores were reported.

### 6.1. Results

Lang | M | X | M+C | X-SM | M+D | X-SM+C | X-SM | M+C | X-SM+C
---|---|---|---|---|---|---|---|---|---
| monolingual | multilingual
ar | 57.3 | 64.6 | 53.6 | 66.5 | 25.1 | 69.1 | 41.0 | 61.3 | 72.7
bg | 51.9 | 38.0 | 53.7 | 63.1 | 11.1 | 60.5 | 53.5 | 57.8 | 60.8
bs | 62.4 | 57.0 | 60.5 | 64.4 | 35.4 | 66.5 | 40.3 | 63.1 | 67.9
da | 48.8 | 34.5 | 46.9 | 66.9 | 21.9 | 59.1 | 55.1 | 57.8 | 75.2
de | 68.7 | 89.1 | 69.4 | 90.1 | 10.7 | 89.6 | 56.3 | 75.3 | 92.9
en | 34.1 | 18.8 | 30.4 | 36.2 | 6.6 | 33.0 | 64.2 | 52.2 | 53.7
es | 46.5 | 22.6 | 36.9 | 51.6 | 8.0 | 46.4 | 61.4 | 49.4 | 59.6
fr | 51.1 | 40.2 | 50.9 | 64.5 | 18.5 | 64.9 | 65.8 | 41.0 | 51.5
hr | 58.5 | 28.7 | 56.4 | 64.6 | 25.7 | 55.9 | 40.5 | 57.7 | 63.4
hu | 50.9 | 43.1 | 50.5 | 62.5 | 17.8 | 66.3 | 47.3 | 56.1 | 63.7
it | 40.3 | 29.8 | 24.0 | 55.8 | 4.4 | 60.2 | 54.4 | 56.6 | 63.1
mt | 60.3 | 60.3 | 60.0 | 68.3 | 11.9 | 62.0 | 35.9 | 44.0 | 56.8
pl | 67.8 | 45.3 | 46.2 | 68.7 | 12.7 | 69.5 | 51.2 | 63.8 | 72.3
pt | 67.2 | 48.1 | 51.8 | 64.3 | 29.5 | 74.6 | 48.3 | 52.8 | 61.8
ru | 65.5 | 43.9 | 70.6 | 73.1 | 27.1 | 75.3 | 64.9 | 65.7 | 82.3
sr | 42.6 | 23.4 | 38.1 | 49.7 | 21.6 | 43.8 | 48.7 | 49.9 | 65.3
sv | 68.2 | 43.0 | 59.2 | 73.1 | 28.7 | 73.3 | 54.5 | 66.0 | 80.2
tr | 45.9 | 32.1 | 44.4 | 49.6 | 11.6 | 49.4 | 47.9 | 41.3 | 47.8
zh | 57.6 | 98.9 | 64.9 | 99.0 | 26.3 | 98.4 | 43.9 | 68.7 | 98.4
lv | 22.6 | 19.0 | 24.8 | 22.0 | 21.5 | 18.1 | 76.8 | 52.4 | 61.6
sq | 20.7 | 20.7 | 20.5 | 20.5 | 7.8 | 20.5 | 33.7 | 43.5 | 45.4
bg_mt | 26.1 | 23.5 | 25.8 | 23.5 | 9.1 | 29.4 |  |  |
bs_mt | 17.3 | 19.0 | 15.6 | 18.5 | 9.1 | 20.6 |  |  |
da_mt | 20.7 | 20.7 | 20.7 | 24.5 | 15.0 | 24.7 |  |  |
fr_mt | 23.1 | 23.1 | 23.1 | 25.8 | 13.6 | 23.4 |  |  |
hr_mt | 34.0 | 25.4 | 28.9 | 34.9 | 16.5 | 46.9 |  |  |
hu_mt | 28.7 | 21.2 | 22.8 | 28.6 | 10.3 | 28 |  |  |
mt_mt | 30.1 | 18.6 | 20.9 | 43.8 | 12.0 | 26.3 |  |  |
pt_mt | 16.4 | 8.7 | 10.5 | 22.9 | 23.4 | 21.9 |  |  |
ru_mt | 41.3 | 17.8 | 28.8 | 46.9 | 23.7 | 45.6 |  |  |
sr_mt | 18.8 | 18.8 | 18.6 | 25.5 | 17.8 | 23.0 |  |  |
sv_mt | 31.7 | 17.3 | 24.3 | 54.6 | 19.8 | 34.7 |  |  |
tr_mt | 33.7 | 30.8 | 32.5 | 31.5 | 13.8 | 30.8 |  |  |
zh_mt | 38.2 | 66.7 | 38.0 | 78.1 | 25.3 | 85.4 |  |  |
Table 2:  F1 comparison of models using visual and textual features. M:
M-BERT, C: CLIP, X: XLM-Roberta, X-SM: XLM-RoBERTa-Sentiment-Multilingual, D:
DINOv2. {lang}_mt: it refers to the model that uses data from original tweets
and their translations for that specific lower-resourced language. The value
included within a cell containing model headers signifies the model’s
performance on the test set for the specific language indicated by the lang
column. Monolingual training involves the use of data from a single language,
whereas multilingual training involves the incorporation of training data from
multiple languages. The best result for each language is highlighted in bold.

The results (F1-score) for the model configurations are given in Table 2.

Unimodal vs. Multimodal: In terms of using textual features to train unimodal
models, we can observe that, on average, textual features from XLM-RoBERTa-
Sentiment-Multilingual yielded higher F1-scores than Multilingual-BERT or XLM-
RoBERTa-base. When we combine both modalities to train multimodal models, we
can observe that the combination of XLM-RoBERTa-Sentiment-Multilingual with
CLIP (X-SM+C) demonstrated superior performance compared to other multimodal
models. The unimodal models of the Bulgarian, Danish, German, Croatian,
Maltese, and Chinese languages exhibit superior performance compared to their
multimodal counterparts. In contrast, the multimodal model demonstrated
superior performance for the remaining higher-resourced languages.

Original data vs Inclusion of translated text: In the context of lower-
resourced languages, the utilisation of machine-translated instances sourced
from higher-resourced languages, such as English, did not yield significant
performance improvements. In the context of the Chinese language, including
translated instances resulted in a decline in the overall performance. We
hypothesise that, in contrast to product and movie reviews, which encompass
comprehensive contextual information as a cohesive entity, one single tweet
lacks the wider contextual frame. Consequently, the translation of the
original language is of lower quality and results in a modification of the
overall meaning.

Monolingual vs Multilingual: When compared with monolingual models’ results,
on average, training a single model for all languages yielded the best
performance for 17 languages, where results for Croatian, Hungarian, Maltese,
Portuguese are higher with monolingual models. It suggests that providing a
single model instead of 21 language-specific models is adequate for many
languages of interest in this paper. Regarding modality for multilingual
configuration, the combination of XLM-RoBERTa-Sentiment-Multilingual with CLIP
(X-SM+C) yielded the best performance across many languages. Thus, we can
confirm that the model trained with the configuration of multimodal and
multilingual achieved the best score for the sentiment analysis of tweets that
include both text and image content.

Figure 3 displays (on right) the average F1-scores for each language and for
each combination of pre-trained models (on left). In the first subplot, it is
evident that X-SM+C exhibits superior performance across all languages, with
XLM-RoBERTa-Sentiment-Multilingual (X-SM) following closely behind. These
findings also suggest the significance of pre-trained models, particularly
those that are highly specialised or domain-specific in the context of
sentiment tasks. In the second subplot, we observe that languages such as
Chinese, Russian, Swedish, and German have overall better scores on all the
trained models.

Figure 3: The (left) plot illustrates the averaged F1-score across various
models. The (right) plot illustrates the averaged F1-score across languages.

### 6.2. Error Analysis

A manual inspection was conducted on the predictions generated by the best
performing unimodal and multimodal model. The errors observed in the model can
be classified into the following categories:

Missing Context: The tweets exhibited a level of ambiguity that required the
application of external knowledge about the world in order to determine the
polarity of the messages. Given that tweets often only capture a fragment of a
larger conversation and lack the necessary background context, it can be
argued that these tweets require additional information beyond the presented
text in order to accurately classify their content. The majority of incorrect
predictions for unimodal and multimodal can be placed in this category.

Disputable: It is important to note that not all labels present in numerous
datasets can be regarded as definitive ground truth, particularly in the case
of (Mozetič et al., 2016), which has been previously identified as having
noise and exhibiting low inter-rater agreement (Rasooli et al., 2018). It is
our contention that the identification of these instances with noisy labels
should be accomplished through the utilisation of established frameworks such
as Northcutt et al. (2021). This observation suggests that there remains
significant potential for enhancement and validates the efficacy of the
collaborative assessment of multimodal data.

Figurative Language: Although the multimodal features help in majority of the
case, the models cannot comprehend cases such as sarcasm. In this case, the
textual model predicts a neutral class while the multimodal model predicts a
positive class, despite the fact that the original class from the dataset is
negative.

In Figure 4, we show a few examples from X-SM+C where the multilingual model
predicts the correct label and the unimodal makes an incorrect classification.
In the example (c), the tweet contains the text "Wishing Prince George a very
Happy Birthday! Mum & Dad may not be looking forward to the terrible
<number>’s, but we are!" is classified as negative by the text model, but the
multimodal multimodal multilingual model correctly predicts it as positive.

Figure 4: Examples from multilingual multimodal model predicts the correct
label and text-only model fail

## 7\. Conclusion

This paper presents the model architecture trained on the dataset extracted
from various sources for multimodal sentiment classification in a multilingual
context. To achieve the objective, this study employed a straightforward
methodology to enhance an existing unimodal dataset from Twitter, transforming
it into a multimodal one. Numerous models have been trained utilising textual
data and a combination of textual and visual modalities. The primary
conclusion drawn from this study is that incorporating sentiment knowledge
into transformers-based models enhances the accuracy of tweet sentiment
classification. The efficacy of the same model settings varies across
different languages. Training a single model for all languages multilingual
and multimodal data yielded the best performance across many languages. In our
prospective endeavours, we intend to utilise tweets devoid of images that
underwent filtration during the preprocessing phase. We aim to augment the
existing dataset by incorporating additional languages. One potential avenue
for advancing research is using translated datasets derived from languages
other than the target language.

## Limitations

The performance of pre-trained models in highly specialised or domain-specific
tasks may be limited due to the broad coverage of topics in their training
data. The pre-trained models learn from the data they are trained on, which
can result in the introduction of any inherent biases in the training data.
This bias can affect model outputs, particularly when the data do not
represent all demographic, cultural, or social groups. The sentiment datasets
used contain a bias towards a particular topic, which was incorporated by the
annotators when the datasets were labelled.

## Acknowledgements

This work was partially funded by the EU Horizon 2020 Research and Innovation
Programme under the Marie Sklodowska-Curie grant agreement no. 812997
(CLEOPATRA ITN). This work was partially funded from the European Union’s
Horizon Europe Research and Innovation Programme under Grant Agreement No
101070631 and from the UK Research and Innovation (UKRI) under the UK
government’s Horizon Europe funding guarantee (Grant No 10039436).

## 8\. Bibliographical References

##

  * Adrián (2016-05-25) Cerón-Guzmán Jhon Adrián. 2016-05-25.  A sentiment analysis model of spanish tweets. case study: Colombia 2014 presidential election.
  * Agüero-Torales et al. (2021) Marvin Agüero-Torales, David Vilares, and Antonio López-Herrera. 2021.  On the logistical difficulties and findings of jopara sentiment analysis.  In _Proceedings of the Fifth Workshop on Computational Approaches to Linguistic Code-Switching_ , pages 95–102, Online. Association for Computational Linguistics.
  * Antypas et al. (2022) Dimosthenis Antypas, Asahi Ushio, Jose Camacho-Collados, Leonardo Neves, Vitor Silva, and Francesco Barbieri. 2022.  Twitter Topic Classification.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Babić et al. (2021) Karlo Babić, Milan Petrović, Slobodan Beliga, Sanda Martinčić-Ipšić, Mihaela Matešić, and Ana Meštrović. 2021.  Characterisation of covid-19-related tweets in the croatian language: Framework based on the cro-cov-csebert model.  _Applied Sciences_ , 11(21).
  * Baecchi et al. (2016) Claudio Baecchi, Tiberio Uricchio, Marco Bertini, and Alberto Del Bimbo. 2016.  A multimodal feature learning approach for sentiment analysis of social network multimedia.  _Multimedia Tools and Applications_ , 75:2507–2525.
  * Barbieri et al. (2020) Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa Anke, and Leonardo Neves. 2020.  TweetEval: Unified benchmark and comparative evaluation for tweet classification.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1644–1650, Online. Association for Computational Linguistics.
  * Barbieri et al. (2022) Francesco Barbieri, Luis Espinosa Anke, and Jose Camacho-Collados. 2022.  XLM-T: Multilingual language models in Twitter for sentiment analysis and beyond.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 258–266, Marseille, France. European Language Resources Association.
  * Cámara et al. (2018) Eugenio Martínez Cámara, Yudivián Almeida-Cruz, Manuel Carlos Díaz-Galiano, Suilan Estévez-Velarde, Miguel Ángel García Cumbreras, Manuel García Vega, Yoan Gutiérrez, Arturo Montejo-Ráez, Andrés Montoyo, Rafael Muñoz, Alejandro Piad-Morffis, and Julio Villena-Román. 2018.  Overview of TASS 2018: Opinions, health and emotions.  In _Proceedings of TASS 2018: Workshop on Semantic Analysis at SEPLN, TASS@SEPLN 2018, co-located with 34nd SEPLN Conference (SEPLN 2018), Sevilla, Spain, September 18th, 2018_ , volume 2172 of _CEUR Workshop Proceedings_ , pages 13–27. CEUR-WS.org.
  * Caron et al. (2021) Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. 2021.  Emerging properties in self-supervised vision transformers.  In _2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021_ , pages 9630–9640. IEEE.
  * Castellucci et al. (2015) Giuseppe Castellucci, Andrea Vanzo, Danilo Croce, and Roberto Basili. 2015.  Context-aware models for twitter sentiment analysis.  _IJCoL vol. 1, n. 1 december 2015: Emerging Topics at the First Italian Conference on Computational Linguistics_ , page 69.
  * Cheema et al. (2021) Gullal S. Cheema, Sherzod Hakimov, Eric Müller-Budack, and Ralph Ewerth. 2021.  A fair and comprehensive comparison of multimodal tweet sentiment analysis methods.  In _MMPT@ICMR2021: Proceedings of the 2021 Workshop on Multi-Modal Pre-Training for Multimedia Understanding, Taipei, Taiwan, August 21, 2021_ , pages 37–45. ACM.
  * Cliche (2017) Mathieu Cliche. 2017.  Bb_twtr at semeval-2017 task 4: Twitter sentiment analysis with cnns and lstms.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 573–580.
  * Conneau et al. (2020a) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020a.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Conneau et al. (2020b) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020b.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Cortis and Davis (2021) Keith Cortis and Brian Davis. 2021.  A Dataset of Multidimensional and Multilingual Social Opinions for Malta’s Annual Government Budget.
  * Costa-jussà et al. (2022) Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022.  No language left behind: Scaling human-centered machine translation.  _ArXiv preprint_ , abs/2207.04672.
  * Devlin et al. (2019a) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019a.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Devlin et al. (2019b) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019b.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Díaz-Galiano et al. (2019) Manuel Carlos Díaz-Galiano, Manuel García Vega, Edgar Casasola, Luis Chiruzzo, Miguel Ángel García Cumbreras, Eugenio Martínez Cámara, Daniela Moctezuma, Arturo Montejo-Ráez, Marco Antonio Sobrevilla Cabezudo, Eric Sadit Tellez, et al. 2019.  Overview of tass 2019: One more further for the global spanish sentiment analysis corpus.  In _IberLEF@ SEPLN_ , pages 550–560.
  * García-Vegaa et al. (2020) Manuel García-Vegaa, Manuel Carlos Díaz-Galianoa, Miguel Á García-Cumbrerasa, Flor Miriam Plaza del Arcoa, Arturo Montejo-Ráeza, Salud María Jiménez-Zafraa, Eugenio Martínez Cámarab, César Antonio Aguilarc, Marco Antonio, Sobrevilla Cabezudod, et al. 2020.  Overview of tass 2020: Introducing emotion detection.  In _Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2020) co-located with 36th Conference of the Spanish Society for Natural Language Processing (SEPLN 2020), Málaga, Spain_ , pages 163–170.
  * Garg et al. (2022) Muskan Garg, Seema Wazarkar, Muskaan Singh, and Ondřej Bojar. 2022.  Multimodality for NLP-centered applications: Resources, advances and frontiers.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 6837–6847, Marseille, France. European Language Resources Association.
  * Ghosh et al. (2015) Aniruddha Ghosh, Guofu Li, Tony Veale, Paolo Rosso, Ekaterina Shutova, John Barnden, and Antonio Reyes. 2015.  SemEval-2015 task 11: Sentiment analysis of figurative language in Twitter.  In _Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)_ , pages 470–478, Denver, Colorado. Association for Computational Linguistics.
  * Go et al. (2009) Alec Go, Richa Bhayani, Lei Huang, and Riloff Ellen. 2009.  Twitter sentiment analysis.  _arXiv preprint arXiv:0903.4294_.
  * Huang et al. (2023) Ju Huang, Pengtao Lu, Shuifa Sun, and Fangyi Wang. 2023.  Multimodal sentiment analysis in realistic environments based on cross-modal hierarchical fusion network.  _Electronics_ , 12(16):3504.
  * Ljajić and Marovac (2018) Adela Ljajić and Ulfeta Marovac. 2018.  Improving sentiment analysis for twitter data by handling negation rules in the serbian language.  _Computer Science and Information Systems_ , 16:13–13.
  * Moctezuma et al. (2016) Daniela Moctezuma, Circuito Tecnopolo Norte No, Eric S Tellez, Mario Graff, Sabino Miranda-Jiménez, and Circuito Tecnopolo Sur. 2016.  On the performance of b4msa on sentipolc’16.  In _of the Final Workshop 7 December 2016, Naples_ , page 200.
  * Mohammad et al. (2018) Saif Mohammad, Svetlana Kiritchenko, Mohammad Salameh, and John Paul Dredze. 2018\.  Multimodal twitter sentiment analysis.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3377–3389.
  * Montejo-Ráez and Díaz-Galiano (2016) Arturo Montejo-Ráez and Manuel Carlos Díaz-Galiano. 2016.  Participación de sinai en tass 2016.  In _TASS@ SEPLN_ , pages 41–45.
  * Mozetič et al. (2016) Igor Mozetič, Miha Grčar, and Jasmina Smailović. 2016.  Twitter sentiment for 15 european languages.  Slovenian language resource repository CLARIN.SI.
  * Muischnek and Müürisep (2018) K Muischnek and K Müürisep. 2018.  Latvian tweet corpus and investigation of sentiment analysis for latvian.  In _Human Language Technologies–The Baltic Perspective: Proceedings of the Eighth International Conference Baltic HLT 2018_ , volume 307, page 112. IOS Press.
  * Mutlu and Özgür (2022) Mustafa Melih Mutlu and Arzucan Özgür. 2022.  A dataset and BERT-based models for targeted sentiment analysis on Turkish texts.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop_ , pages 467–472, Dublin, Ireland. Association for Computational Linguistics.
  * Nakov et al. (2013a) Preslav Nakov, Sara Rosenthal, Zornitsa Kozareva, Veselin Stoyanov, Alan Ritter, and Theresa Wilson. 2013a.  SemEval-2013 task 2: Sentiment analysis in Twitter.  In _Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013)_ , pages 312–320, Atlanta, Georgia, USA. Association for Computational Linguistics.
  * Nakov et al. (2013b) Preslav Nakov, Sara Rosenthal, Zornitsa Kozareva, Veselin Stoyanov, Alan Ritter, and Theresa Wilson. 2013b.  SemEval-2013 task 2: Sentiment analysis in Twitter.  In _Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013)_ , pages 312–320, Atlanta, Georgia, USA. Association for Computational Linguistics.
  * Nakov et al. (2013c) Preslav Nakov, Sara Rosenthal, Zornitsa Kozareva, Veselin Stoyanov, Alan Ritter, and Theresa Wilson. 2013c.  SemEval-2013 task 2: Sentiment analysis in Twitter.  In _Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013)_ , pages 312–320, Atlanta, Georgia, USA. Association for Computational Linguistics.
  * Nakov et al. (2013d) Preslav Nakov, Sara Rosenthal, Zornitsa Kozareva, Veselin Stoyanov, Alan Ritter, and Theresa Wilson. 2013d.  SemEval-2013 task 2: Sentiment analysis in Twitter.  In _Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013)_ , pages 312–320, Atlanta, Georgia, USA. Association for Computational Linguistics.
  * Northcutt et al. (2021) Curtis Northcutt, Anish Athalye, and Jonas Mueller. 2021.  Pervasive label errors in test sets destabilize machine learning benchmarks.  In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks_ , volume 1. Curran.
  * Oquab et al. (2023) Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. 2023.  Dinov2: Learning robust visual features without supervision.  _ArXiv preprint_ , abs/2304.07193.
  * Pang and Lee (2005) Bo Pang and Lillian Lee. 2005.  Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales.  In _Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05)_ , pages 115–124, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Patrick et al. (2022) Hallan Patrick, Fernando Paulo Belfo, and António Trigo. 2022.  Brazilian tweets classified for sentiment analysis.
  * Pauli et al. (2021) Amalie Brogaard Pauli, Maria Barrett, Ophélie Lacroix, and Rasmus Hvingelby. 2021.  DaNLP: An open-source toolkit for Danish natural language processing.  In _Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa)_ , pages 460–466, Reykjavik, Iceland (Online). Linköping University Electronic Press, Sweden.
  * Poria et al. (2016a) Soujanya Poria, Erik Cambria, Newton Howard, Guang-Bin Huang, and Amir Hussain. 2016a.  Fusing audio, visual and textual clues for sentiment analysis from multimodal content.  _Neurocomputing_ , 174:50–59.
  * Poria et al. (2016b) Soujanya Poria, Iti Chaturvedi, Erik Cambria, and Amir Hussain. 2016b.  Convolutional mkl based multimodal emotion recognition and sentiment analysis.  In _2016 IEEE 16th International Conference on Data Mining (ICDM)_ , pages 439–448.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning transferable visual models from natural language supervision.  In _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 8748–8763. PMLR.
  * Raffel et al. (2019) Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21:140:1–140:67.
  * Rasooli et al. (2018) Mohammad Sadegh Rasooli, Noura Farra, Axinia Radeva, Tao Yu, and Kathleen McKeown. 2018.  Cross-lingual sentiment transfer with limited resources.  _Machine Translation_ , 32:143–165.
  * Rei et al. (2016) Luis Rei, Simon Krek, and Dunja Mladenić. 2016.  xLiMe twitter corpus XTC 1.0.1.  Slovenian language resource repository CLARIN.SI.
  * Román et al. (2015) Julio Villena Román, Eugenio Martínez Cámara, Janine García Morera, and Salud M Jiménez Zafra. 2015.  Tass 2014-the challenge of aspect-based sentiment analysis.  _Procesamiento del Lenguaje Natural_ , 54:61–68.
  * Santamaría et al. (2022) Lucia Prieto Santamaría, Juan Manuel Tuñas, Diego Fernández Peces-Barba, Almudena Jaramillo, Manuel Cotarelo, Ernestina Menasalvas, Antonio Conejo Fernández, Amalia Arce, Angel Gil de Miguel, and Alejandro Rodríguez González. 2022.  Influenza and measles-mmr: two case study of the trend and impact of vaccine-related twitter posts in spanish during 2015-2018.  _Human Vaccines & Immunotherapeutics_, 18(1):1–16.  PMID: 33662222.
  * Tayebi Arasteh et al. (2021) Soroosh Tayebi Arasteh, Mehrpad Monajem, Vincent Christlein, Philipp Heinrich, Anguelos Nicolaou, Hamidreza Naderi Boldaji, Mahshad Lotfinia, and Stefan Evert. 2021.  How will your tweet be received? predicting the sentiment polarity of tweet replies.  In _Proceedings of the 2021 IEEE 15th International Conference on Semantic Computing (ICSC)_ , pages 370–373, Laguna Hills, CA, USA.
  * Vanzo et al. (2014) Andrea Vanzo, Danilo Croce, and Roberto Basili. 2014.  A context-based model for sentiment analysis in Twitter.  In _Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers_ , pages 2345–2354, Dublin, Ireland. Dublin City University and Association for Computational Linguistics.
  * Vilares et al. (2015) David Vilares, Yerai Doval, Miguel A Alonso, and Carlos Gómez-Rodríguez. 2015.  Lys at tass 2015: Deep learning experiments for sentiment analysis on spanish tweets.  In _TASS@ SEPLN_ , pages 47–52.
  * Villena-Román and Garcıa-Morera (2013) Julio Villena-Román and Janine Garcıa-Morera. 2013.  Tass 2013—workshop on sentiment analysis at sepln 2013: An overview.  In _Proceedings of the TASS workshop at SEPLN_ , pages 112–125.
  * Vincent et al. (2010) Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. 2010.  Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion.  _J. Mach. Learn. Res._ , 11:3371–3408.
  * Vukotić et al. (2015) Vedran Vukotić, Vincent Claveau, and Christian Raymond. 2015.  Irisa at deft 2015: supervised and unsupervised methods in sentiment analysis.  In _DeFT, Défi Fouille de Texte, joint à la conférence TALN 2015_.
  * Xie et al. (2020) Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. 2020.  Unsupervised data augmentation for consistency training.  _Advances in neural information processing systems_ , 33:6256–6268.
  * Yin et al. (2021) Wenjie Yin, Rabab Alkhalifa, and Arkaitz Zubiaga. 2021.  TM-Senti.
  * You et al. (2016) Quanzeng You, Liangliang Cao, Hailin Jin, and Jiebo Luo. 2016.  Robust visual-textual sentiment analysis: When attention meets tree-structured recursive neural networks.  In _Proceedings of the 2016 ACM Conference on Multimedia Conference, MM 2016, Amsterdam, The Netherlands, October 15-19, 2016_ , pages 1008–1017.
  * Yu et al. (2020) Wenmeng Yu, Hua Xu, Fanyang Meng, Yilin Zhu, Yixiao Ma, Jiele Wu, Jiyun Zou, and Kaicheng Yang. 2020.  CH-SIMS: A Chinese multimodal sentiment analysis dataset with fine-grained annotation of modality.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3718–3727, Online. Association for Computational Linguistics.