11institutetext: Departamento de Física, Universidad de Santiago de Chile
(USACH), Av. Victor Jara 3493, Estación Central, Chile
11email<EMAIL_ADDRESS>22institutetext: Center for
Interdisciplinary Research in Astrophysics and Space Sciences (CIRAS),
Universidad de Santiago de Chile, Chile 33institutetext: Núcleo de
Astroquímica, Facultad de Ingeniería, Universidad Autónoma de Chile, Av. Pedro
de Valdivia 425, Providencia, Santiago, Chile 44institutetext: Department of
Physics and Astronomy, Haverford College, 370 Lancaster Avenue, Haverford, PA,
19041, USA 55institutetext: Centre for Astrophysics and Supercomputing,
Swinburne University of Technology, John St, Hawthorn, VIC 3122, Australia
66institutetext: Department of Astronomy and Cornell Center for Astrophysics
and Planetary Science, Cornell University, Ithaca, New York, 14853, USA

# VLA proper motion constraints on the origin, age, and potential magnetar
future of PSR J1734$-$3333

C. M. Espinoza 1122 M. Vidal-Navarro 33 W. C. G. Ho 44 A. Deller 55 S.
Chatterjee 66

###### Abstract

Context. The characteristic age of PSR J1734$-$3333 estimated from its current
spin down rate implies that it is a young pulsar ($\tau_{c}<10$ kyr). But the
time derivative of its spin down rate differs markedly from that assumed for
normal radio pulsars, meaning its actual age is uncertain. G354.8$-$0.8 is a
supernova remnant (SNR) whose centre is located 21′ away of the pulsar, and
with a morphology that suggests an association with the pulsar.

Aims. We want to assess the likelihood of the association between PSR
J1734$-$3333 and G354.8$-$0.8 or other nearby supernova remnants
quantitatively, with the objective of shedding light on the real age of this
pulsar.

Methods. Observations with the Karl G. Jansky Very Large Array were carried
out in 2015 and 2019 that allow precise astrometric measurements and
consequently a proper motion estimate for the pulsar.

Results. The proper motion was found to be $\mu_{\alpha}=10\pm 10$ mas yr-1
and $\mu_{\delta}=-29\pm 11$ mas yr-1 (error bars are $1$-$\sigma$). Though
marginal, this detection rules out the association with G354.8$-$0.8 because
it means the pulsar is not moving away from the centre of the SNR. No SNR
consistent with the measured proper motion and an age $\sim\tau_{c}$ could be
found. We also present the first measurement of the spectral index for this
pulsar, $\alpha=-1.1\pm 0.3$, measured between $1.5$ and $3.0$ GHz.

Conclusions. The SNR produced by the birth supernova of PSR J1734$-$3333 could
have already faded to undetectable brightness, for which estimates suggest
timescales of $10$-$100$ kyr. This and other considerations lead us to
conclude that the pulsar is possibly older than $45$-$100$ kyr. PSR
J1734$-$3333 is a pulsar with rotational properties that place it between
standard radio pulsars and magnetars, and we interpret our result in the
context of a possible future life as a magnetar for this pulsar.

###### Key Words.:

stars: neutron – pulsars: general – pulsars: individual: PSR J1734$-$3333 –
proper motions – ISM: supernova remnants

##  1 Introduction

The age of a neutron star is difficult to measure, as for many other
astronomical sources. The most robust way to do it is by identifying the birth
supernova of the neutron star. However, this can be done precisely only for a
very small number of objects, as 5-10 supernovae have historically been
observed in our galaxy (Stephenson & Green 2005), and neutron stars are faint
sources –practically undetectable at distances beyond the Magellanic Clouds.
The explosions, however, leave imprints in the interstellar medium that can
remain visible at radio wavelengths for $10-100$ kyr (Sarbadhicary et al.
2017), thereby allowing the association of pulsars with supernova remnants
(SNRs). However, pulsars are rarely found at the centre of SNRs (Frail et al.
1994), as most are expelled like bullets during the explosions possibly due to
asymmetries in the process (e.g. Socrates et al. 2005). The transverse
velocities of pulsars (based on proper motion and distance estimates) are
particularly large, with a mean close to $310$ km s-1 (Hobbs et al. 2005),
which is at least 10 times larger than the average velocities for stars in the
solar neighbourhood (e.g. Gaia Collaboration et al. 2018), and with some
measured velocities ranging as high as $1000$ km s-1 (Chatterjee et al. 2005;
Deller et al. 2019).

Thus, associations between SNRs and pulsars are not always straightforward to
make (e.g. see the chapter on young pulsars in Lyne & Graham-Smith 2012). The
farther the pulsar is from the explosion site, the higher the possibility that
pulsar and SNR are unrelated. In order to confirm an association, it could be
necessary to account for up to $100$ kyr of evolution of the SNR (that we
assume as the maximum possible age of a SNR), and movement across the Galaxy
of the pulsar (e.g. Suzuki et al. 2021). In some situations, proper motion
measurements for the pulsars can shed light on the matter. For an association
to be secure, the pulsar must be moving away from where the explosion took
place (usually adopted as the centre of the SNR), and the time necessary to
move the pulsar to its current position must match the age of the system. If
such time coincided with an independent age measurement of the SNR or the
pulsar, or both, then the association would be firmly confirmed. But this is
rarely possible, as SNR and pulsar ages are hard to obtain.

In this paper we present interferometric observations of PSR J1734$-$3333
designed to measure its proper motion, evaluate its possible connection with
the shell-type SNR G354.8$-$0.8 (Whiteoak & Green 1996; Green 2014), and, if
possible, constrain the age of the pulsar. This association was first proposed
by Manchester et al. (2002) based on the proximity of the two objects and the
fact that the SNR exhibits a teardrop shape oriented towards the pulsar. But,
given that the evidence is circumstantial and that the pulsar is 21′ away of
the centre of the SNR, the association was not considered secure (Manchester
et al. 2002). No estimates for the distance of SNR G$354.8$$-$$0.8$ are
available. While confirming this association is in itself important, PSR
J1734$-$3333 possesses some other interesting properties, like a decreasing
characteristic age and a potential future as a magnetar, which add relevance
to the proper motion measurement that we present.

###  1.1 The age and braking index of PSR J1734$-$3333

Another common way to estimate the age of a pulsar is by measuring its period
$P$ and period time derivative $\dot{P}$, assume a spin down law and calculate
the time required to reach the current value of $P$. Typically, the rotation
is believed to be slowed down by magnetic dipole radiation, which implies
$\dot{P}\,\propto\,P^{-1}$ for a constant dipole magnitude. If additionally
the initial period $P_{0}$ is assumed to have been significantly shorter than
today’s value, that is if $\left(P_{0}/P\right)^{2}\ll 1$, then the age is
given by $\tau_{c}=P/2\dot{P}$, which is known as the characteristic age (e.g.
Manchester & Taylor 1977; Lyne & Graham-Smith 2012). However, the caveats
involved can sometimes be enough to make this age definition unreliable.

PSR J1734$-$3333 rotates with a typical period $P=1.169$ s, but slows down
with the fourth largest period derivative amongst pulsars ($\dot{P}=2.3\times
10^{-12}$, Morris et al. 2002; Manchester et al. 2005). These measurements
imply the particularly young characteristic age $\tau_{c}=8.1$ kyr. However, a
measurement of the second derivative of the spin period indicates that the
rotation of this pulsar may not be driven by constant magnetic dipole braking.

The braking index ($n$) is used to characterise the long-term rotational
evolution of pulsars, and is defined from the relation
$\dot{P}\,\propto\,P^{2-n}$ (e.g. Manchester & Taylor 1977; Lyne & Graham-
Smith 2012). Pulsars evolving purely under the effects of magnetic dipole
braking are expected to exhibit a braking index of $3$. However, measurements
point to either $n<3$ in very young pulsars like PSR J1734$-$3333 or to
$10\leq n\leq 100$, for cases possibly affected by timing noise or glitch
recoveries (Parthasarathy et al. 2020). To measure the braking index it is
necessary to detect the period second derivative, which is a fairly small
quantity, hence difficult to detect (Espinoza et al. 2017). In the case of PSR
J1734$-$3333, two independent measurements indicate that $n$ is in the range
$0.7$-$1.4$, one of the lowest braking indices ever measured (Espinoza et al.
2011; Lower et al. 2021). Such low braking index carries some consequences,
amongst which is that the characteristic age is decreasing (if $n<1$), or
increasing very slowly with time (if $n\gtrsim 1$, Lyne 2004). Therefore,
$\tau_{c}$ is a highly unreliable proxy for the age of this pulsar, which
implies that PSR J1734$-$3333 could be substantially older than $8.1$ kyr.

###  1.2 PSR J1734$-$3333 and the magnetars

Both $P$ and $\dot{P}$ measurements enable the inference of the amplitude of
the effective dipole magnetic field responsible for the slow down, should
magnetic dipole radiation be the dominant braking mechanism (e.g. Manchester &
Taylor 1977). It corresponds to $B=3.2\times 10^{19}(P\dot{P})^{1/2}$, where
standard assumptions regarding the orientation of the dipole and the size of
the star have been made, and for PSR J1734$-$3333 amounts to $B=5.2\times
10^{13}$ G, which is the fourth largest magnetic field strength known amongst
pulsars (The ATNF Pulsar Catalogue, Manchester et al.
2005)111http://www.atnf.csiro.au/research/pulsar/psrcat, and higher than the
inferred values for at least three of the approximately 30 known magnetars
(The McGill Online Magnetar Catalog, Olausen & Kaspi
2014)222http://www.physics.mcgill.ca/ pulsar/magnetar/main.html. Magnetars are
a class of neutron stars, which emit in the X-rays and are identified by their
luminous X-ray and soft $\gamma$-ray outburst episodes that usually require
more power than is available in the rotation of the star. They rotate with
periods in the range $2$-$12$ s, which is slower than most pulsars, and
slowdown quicker than all pulsars (with the notable exception of three sources
with relatively low-$\dot{P}$ values, Rea et al. 2010, 2012, 2014)). Both the
high $\dot{P}$ values and the high energy emission of magnetars are believed
to be driven and powered by the decay of very intense magnetic fields of
$10^{14-15}$ G (e.g. Woods & Thompson 2006; Mereghetti et al. 2015; Kaspi &
Beloborodov 2017). This is in opposition to most other, rotation powered
pulsars that rotate faster, have lower dipole field strengths, exhibit modest
and stable X-ray emission (if any), and are observed mostly in radio
wavelengths.

PSR J1734$-$3333 is a normal radio pulsar with standard X-ray emission
properties, unlike the magnetars (though there is a debate on whether the
pulsar is hotter than others of similar $\tau_{c}$, see Olausen et al. 2013;
Keane et al. 2013). However, its very low braking index implies that its
rotational properties are evolving quickly towards those of the magnetars. In
$\sim 30$ kyr, if the braking index remained low, the pulsar will rotate with
a period of $8$ s and the dipole magnetic field strength $B$ inferred from the
pulsar spin-down would be comparable to that for magnetars (Espinoza et al.
2011).

The separation between magnetars and rotation powered pulsars has indeed
become less clear as new neutron stars have been discovered (e.g. Hu et al.
2020) and observations of previously known pulsars accumulate. High-$B$
rotation powered pulsars have been seen to turn into magnetars for a few
months (see the X-ray outbursts of PSRs J1846$-$0258 and J1119$-$6127, Gavriil
et al. 2008; Kuiper & Hermsen 2009; Archibald et al. 2016) and some magnetars
have been seen to awake in the radio band during or after outbursts (e.g.
Camilo et al. 2006, 2007; Eatough et al. 2013). Therefore, it is possible that
some high-$B$ pulsars are dormant magnetars, which any day could display full
magnetar high-energy behaviour. Note that there is not a single rotation
powered pulsar known with the rotational properties of magnetars: large
periods and period derivatives. Thus PSR J1734$-$3333, with its high
$\dot{P}$, could be an evolutionary link between the two classes of pulsars.
Collectively, these facts make its age, proper motion, and relationship with
SNR G354.8$-$0.8 particularly relevant. Evolutionary links between neutron
star families are one possible solution to explain the overabundance of
neutron stars in the Galaxy with respect to the rate of supernova explosions
(Keane & Kramer 2008).

##  2 Data

We used the VLA on two epochs separated by approximately $4.2$ yr to image the
field of PSR J1734$-$3333 and constrain its proper motion.

###  2.1 Observations

The first observations were carried out on May 24${}^{\textrm{th}}$, 2015,
using the BnA VLA configuration. This configuration was chosen due to the low
declination ($-33^{\circ}$) of the pulsar, which at observatory site
translates to a maximum elevation of $\sim 20^{\circ}$. On this opportunity
the observations were done on a single pointing using both the L (1–2 GHz) and
S (2–4 GHz) bands. The second observation was carried out only at the S band,
on August 21${}^{\textrm{st}}$ 2019\. After 2015 the BnA configuration was not
offered anymore, hence we used the A configuration for the second observation.
Table 1 lists a summary of the observations.

For both observation epochs, the primary calibrator was 3C286 (flux density
and bandpass), and the secondary calibrator was J1713-3226 (for phase). The
data were calibrated following standard procedures, using the VLA calibration
pipeline that is built on the Common Astronomy Software Applications (CASA
McMullin et al. 2007) versions 4.4.2 and 5.4.2. After an initial calibration
run, we inspected manually the resulting visibilities and calibration tables
in order to search for problematic data (either RFI or instrumental). RFI was
present in the S-band data of both observations of both epochs. After the
removal of the problematic visibilities, we reran the calibration pipeline and
iterated the entire process until no signs of bad data were detected. The
flagged data in S-band accounts for $\sim\\!20\%$ and $\sim\\!14\%$ of the on-
time source for the 2015 and 2019 data, respectively.

Table 1: Log of observations.

Date | Array(a) | Frequency(b) | Synthesized beam(c) | $\sigma$
---|---|---|---|---
|  | (GHz) | (arcsec) | ($\mu$Jy bm-1)
24-May-2015 | BnA | 1.5 | (5.3 x 2.1 / -48) | 100
24-May-2015 | BnA | 3.0 | (2.5 x 1.4 / -44) | 8.6
2-May-2019 | A | 3.0 | (1.5 x 0.5 / 9.5) | 7.9
333 $a$$a$footnotetext: VLA configuration. $b$$b$footnotetext: Centre
frequencies for each band that is made up of $2048\times 1$ MHz channels.
$c$$c$footnotetext: Briggs-weighted beam in arcsec, in the form (BMAJ x BMIN /
BPA), where BMAJ and BMIN are the full-width major and minor axis, and BPA is
the beam parallactic angle (PA) in degrees East of North.

Table 2: Source parameters from the 2015 map obtained with the CASA routine
imfit.

ID | Flux Density | S/N | Dist. PSR | RA | DEC | 1-$\sigma$ RA uncertainty | 1-$\sigma$ DEC uncertainty
---|---|---|---|---|---|---|---
| ($\mu{\rm Jy}$) |  | (arcsec) | J2000 | J2000 | (mas) | (mas)
3 | $94.4\pm 8.5$ | $11.1$ | $554.7$ | 17h34m31${}^{\rm s}\\!.$0096 | -33∘42′32${}^{\prime\prime}\\!\\!.$3247 | $93.0$ | $103.8$
4 | $143.3\pm 8.3$ | $17.3$ | $389.1$ | 17h34m19${}^{\rm s}\\!.$2379 | -33∘39′37${}^{\prime\prime}\\!\\!.$0915 | $69.9$ | $73.0$
5 | $257.4\pm 8.2$ | $31.2$ | $349.7$ | 17h34m33${}^{\rm s}\\!.$1721 | -33∘39′00${}^{\prime\prime}\\!\\!.$8382 | $37.3$ | $30.6$
7 | $129.5\pm 8.7$ | $14.9$ | $706.1$ | 17h33m36${}^{\rm s}\\!.$6996 | -33∘38′44${}^{\prime\prime}\\!\\!.$3096 | $86.6$ | $99.4$
8 | $173.6\pm 8.1$ | $21.3$ | $364.9$ | 17h34m04${}^{\rm s}\\!.$9805 | -33∘37′21${}^{\prime\prime}\\!\\!.$1721 | $57.7$ | $59.0$
9 | $400.5\pm 8.6$ | $46.5$ | $533.6$ | 17h35m05${}^{\rm s}\\!.$4970 | -33∘37′08${}^{\prime\prime}\\!\\!.$2245 | $24.8$ | $24.0$
10 | $51.4\pm 7.1$ | $7.3$ | $273.4$ | 17h34m41${}^{\rm s}\\!.$7640 | -33∘36′40${}^{\prime\prime}\\!\\!.$5500 | $272.7$ | $169.7$
11 | $82.5\pm 7.5$ | $11.0$ | $196.2$ | 17h34m29${}^{\rm s}\\!.$3675 | -33∘36′33${}^{\prime\prime}\\!\\!.$7612 | $76.2$ | $83.5$
12 | $109.6\pm 8.3$ | $13.1$ | $568.8$ | 17h35m09${}^{\rm s}\\!.$9687 | -33∘36′23${}^{\prime\prime}\\!\\!.$9391 | $99.2$ | $95.4$
13 | $68.0\pm 8.6$ | $7.9$ | $227.1$ | 17h34m38${}^{\rm s}\\!.$9762 | -33∘36′09${}^{\prime\prime}\\!\\!.$7422 | $169.6$ | $91.6$
14 | $152.3\pm 8.1$ | $18.7$ | $340.0$ | 17h34m00${}^{\rm s}\\!.$2694 | -33∘34′29${}^{\prime\prime}\\!\\!.$4742 | $50.7$ | $61.7$
17p | $343.7\pm 8.3$ | $41.5$ | $2.8$ | 17h34m27${}^{\rm s}\\!.$0591 | -33∘33′21${}^{\prime\prime}\\!\\!.$9157 | $25.8$ | $25.9$
18 | $88.6\pm 7.9$ | $11.2$ | $89.8$ | 17h34m34${}^{\rm s}\\!.$0219 | -33∘33′08${}^{\prime\prime}\\!\\!.$2368 | $91.5$ | $110.7$
19 | $111.6\pm 7.9$ | $14.1$ | $127.8$ | 17h34m19${}^{\rm s}\\!.$3574 | -33∘31′53${}^{\prime\prime}\\!\\!.$7939 | $81.2$ | $76.5$
20 | $149.2\pm 8.1$ | $18.3$ | $95.9$ | 17h34m23${}^{\rm s}\\!.$7535 | -33∘31′52${}^{\prime\prime}\\!\\!.$5611 | $55.4$ | $78.1$
21 | $323.2\pm 8.3$ | $39.1$ | $316.6$ | 17h34m50${}^{\rm s}\\!.$7384 | -33∘31′33${}^{\prime\prime}\\!\\!.$1535 | $26.3$ | $27.8$
22 | $56.4\pm 7.2$ | $7.8$ | $119.5$ | 17h34m31${}^{\rm s}\\!.$0282 | -33∘31′32${}^{\prime\prime}\\!\\!.$1939 | $100.3$ | $104.0$
25 | $73.0\pm 7.7$ | $9.4$ | $457.4$ | 17h33m51${}^{\rm s}\\!.$5883 | -33∘31′20${}^{\prime\prime}\\!\\!.$5649 | $163.2$ | $104.0$
30 | $120.8\pm 9.1$ | $13.3$ | $648.8$ | 17h33m38${}^{\rm s}\\!.$8158 | -33∘29′16${}^{\prime\prime}\\!\\!.$3236 | $106.3$ | $79.9$
31 | $461.4\pm 8.1$ | $56.9$ | $262.9$ | 17h34m32${}^{\rm s}\\!.$8634 | -33∘29′07${}^{\prime\prime}\\!\\!.$8686 | $18.9$ | $19.1$
33 | $107.0\pm 8.3$ | $12.9$ | $337.7$ | 17h34m38${}^{\rm s}\\!.$3613 | -33∘28′14${}^{\prime\prime}\\!\\!.$2692 | $83.5$ | $75.8$
35 | $134.5\pm 8.6$ | $15.6$ | $491.0$ | 17h34m23${}^{\rm s}\\!.$9488 | -33∘25′10${}^{\prime\prime}\\!\\!.$4081 | $76.0$ | $80.2$

444 $p$$p$footnotetext: This is PSR J1734$-$3333.

Table 3: Source parameters from the 2019 map obtained with the CASA routine
imfit.

ID | Flux Density | S/N | Dist. PSR | RA | DEC | 1-$\sigma$ RA uncertainty | 1-$\sigma$ DEC uncertainty
---|---|---|---|---|---|---|---
| ($\mu{\rm Jy}$) |  | (arcsec) | J2000 | J2000 | (mas) | (mas)
3 | $71.4\pm 8.2$ | $8.7$ | $554.5$ | 17h34m31${}^{\rm s}\\!.$0092 | -33∘42′32${}^{\prime\prime}\\!\\!.$1128 | $25.5$ | $128.6$
4 | $121.0\pm 8.1$ | $15.0$ | $388.8$ | 17h34m19${}^{\rm s}\\!.$2163 | -33∘39′36${}^{\prime\prime}\\!\\!.$7490 | $14.2$ | $85.8$
5 | $202.2\pm 7.8$ | $26.0$ | $349.6$ | 17h34m33${}^{\rm s}\\!.$1663 | -33∘39′00${}^{\prime\prime}\\!\\!.$7171 | $8.8$ | $47.9$
7 | $68.0\pm 9.6$ | $7.1$ | $706.0$ | 17h33m36${}^{\rm s}\\!.$7158 | -33∘38′44${}^{\prime\prime}\\!\\!.$4313 | $58.3$ | $143.1$
8 | $104.2\pm 8.8$ | $11.9$ | $364.9$ | 17h34m04${}^{\rm s}\\!.$9742 | -33∘37′21${}^{\prime\prime}\\!\\!.$0350 | $21.3$ | $98.7$
9 | $212.0\pm 9.3$ | $22.8$ | $533.5$ | 17h35m05${}^{\rm s}\\!.$4935 | -33∘37′08${}^{\prime\prime}\\!\\!.$1699 | $12.2$ | $45.4$
10 | $43.4\pm 7.7$ | $5.7$ | $273.3$ | 17h34m41${}^{\rm s}\\!.$7537 | -33∘36′40${}^{\prime\prime}\\!\\!.$6089 | $46.7$ | $161.0$
11 | $79.5\pm 8.2$ | $9.7$ | $196.0$ | 17h34m29${}^{\rm s}\\!.$3628 | -33∘36′33${}^{\prime\prime}\\!\\!.$5833 | $26.3$ | $95.1$
12 | $61.5\pm 9.5$ | $6.5$ | $568.7$ | 17h35m09${}^{\rm s}\\!.$9587 | -33∘36′23${}^{\prime\prime}\\!\\!.$9360 | $52.9$ | $150.8$
13 | $42.9\pm 9.0$ | $4.8$ | $226.8$ | 17h34m38${}^{\rm s}\\!.$9721 | -33∘36′09${}^{\prime\prime}\\!\\!.$2950 | $49.8$ | $163.9$
14 | $115.2\pm 8.8$ | $13.1$ | $340.0$ | 17h34m00${}^{\rm s}\\!.$2709 | -33∘34′29${}^{\prime\prime}\\!\\!.$4341 | $18.6$ | $77.8$
17p | $299.9\pm 7.6$ | $39.4$ | $2.8$ | 17h34m27${}^{\rm s}\\!.$0561 | -33∘33′21${}^{\prime\prime}\\!\\!.$9422 | $6.1$ | $31.0$
18 | $81.5\pm 7.8$ | $10.5$ | $90.0$ | 17h34m34${}^{\rm s}\\!.$0394 | -33∘33′08${}^{\prime\prime}\\!\\!.$5021 | $19.6$ | $108.9$
19 | $83.1\pm 8.5$ | $9.8$ | $128.0$ | 17h34m19${}^{\rm s}\\!.$3521 | -33∘31′53${}^{\prime\prime}\\!\\!.$5162 | $32.9$ | $110.5$
20 | $137.5\pm 7.4$ | $18.6$ | $95.9$ | 17h34m23${}^{\rm s}\\!.$7530 | -33∘31′52${}^{\prime\prime}\\!\\!.$5326 | $12.7$ | $66.4$
21 | $248.9\pm 8.5$ | $29.1$ | $316.6$ | 17h34m50${}^{\rm s}\\!.$7281 | -33∘31′32${}^{\prime\prime}\\!\\!.$9706 | $9.7$ | $35.0$
22 | $35.9\pm 7.4$ | $4.8$ | $119.7$ | 17h34m31${}^{\rm s}\\!.$0308 | -33∘31′32${}^{\prime\prime}\\!\\!.$0615 | $91.8$ | $251.3$
25 | $44.7\pm 9.7$ | $4.6$ | $457.5$ | 17h33m51${}^{\rm s}\\!.$5749 | -33∘31′20${}^{\prime\prime}\\!\\!.$5771 | $70.9$ | $186.3$
30 | $42.8\pm 8.8$ | $4.8$ | $648.9$ | 17h33m38${}^{\rm s}\\!.$8149 | -33∘29′16${}^{\prime\prime}\\!\\!.$0670 | $82.9$ | $288.1$
31 | $384.0\pm 7.7$ | $49.9$ | $262.9$ | 17h34m32${}^{\rm s}\\!.$8613 | -33∘29′07${}^{\prime\prime}\\!\\!.$8548 | $4.6$ | $23.6$
33 | $58.0\pm 7.2$ | $8.0$ | $337.7$ | 17h34m38${}^{\rm s}\\!.$3459 | -33∘28′14${}^{\prime\prime}\\!\\!.$1778 | $50.5$ | $212.3$
35 | $122.9\pm 8.1$ | $15.1$ | $491.0$ | 17h34m23${}^{\rm s}\\!.$9361 | -33∘25′10${}^{\prime\prime}\\!\\!.$4301 | $19.3$ | $73.5$

555 $p$$p$footnotetext: This is PSR J1734$-$3333.

After calibration, we used the tclean task from CASA (version 5.8) to image
the measurement sets for each epoch. We tested several weighting and imaging
parameters and we settled for the ones that provided better sensitivity and
lower level of imaging artefacts. We chose the Briggs weighting scheme (Briggs
et al. 1999) with robust=0.5 in order to find a balance between good signal-
to-noise ratio and a narrow synthesised beam. Also, we used the option
nterms=2, which allows the tclean task to model the spectral indices of the
sources. We created images using a large grid of $13122\times 13122$ square
pixels, each with a side of 0.1 arcsec. This image size is roughly 1.5 times
the primary beam at S-band, motivated by the intention of also including in
the analysis sources that lie farther away from the position of the pulsar.
Because of the large image size, we had to use the wide-field imaging
parameters gridder=wproject and wprojectplanes=-1.

Due to the different array configurations for each epoch, the 2019 data have a
more elongated synthesised beam (see Table 1). Figure 1 shows the 2015 S-band
image for the pulsar and for the three brighter sources in the field. Contours
of the 2019 S-band observation are overlaid.

Figure 1: Clean S-band images for the four brighter sources in the 2015
observation. Overlaid contours for the 2015 (red) and the 2019 (blue)
observations are shown too. The pulsar is source 17 (top left).

###  2.2 Source detection and astrometry

After we produced cleaned images for each epoch, we used an automatic source
detection software (the DAOStarFinder routine from the photutils
package666https://photutils.readthedocs.io/en/stable/index.html) to identify
all the sources that could be used to reference the position of the pulsar.
This procedure yielded a total of 41 sources that were detected in both
epochs. From these sources, a number of them were complex, i.e. they were
resolved in the 2019 data but unresolved in the 2015 maps (due to the
difference in beam sizes) while others were resolved and showed extended
emission in both epochs. We kept only the ones that were consistent with being
unresolved point sources in both epochs. The final number of sources that we
used to measure their positions was $N=21$.

We fitted for the position of the selected sources using the imfit task from
CASA. The task fits a 2-D Gaussian for each source where the free parameters
are the location of the centroid, both semi-axis and its position angle. In
tables 2 and 3 we list the parameters for the sources, including the pulsar,
for both epochs. In these tables, the ID number is the index of the source
from the original 41 sources detected by DAOStarFinder. We also list the flux
density of each source, its signal-to-noise ratio (S/N) based on the rms noise
of the maps, the distance to the coordinates of the pulsar, as previously
published in Espinoza et al. (2011, RA(J2000)=17h34m26.9s,
DEC(J2000)=-33d33m20s) and the RA-DEC coordinates with their respective
1-$\sigma$ uncertainty as calculated by imfit. The listed flux densities have
not been corrected by the primary beam attenuation. In these tables, the
pulsar corresponds to ID=17.

##  3 Proper motion estimate

At each epoch, we measure the position of the pulsar and the point-like
background sources listed in Tables 2 and 3. We assume that the proper motion
of all these background sources is zero, and use that constraint to measure a
single offset, $\Delta C$, between the two observations. The ensemble of
background sources (or reference sources) is characterised via its centroid,
which is the average of their positions at a given epoch:
$C=\left(\langle\alpha\rangle,\langle\delta\rangle\right)$. Thus the offset of
the centroid is $\Delta C=C_{2019}-C_{2015}$. This quantity is used to
minimise the net translation that this ensemble of sources may present between
the two observations. The effective displacement of the pulsar between the two
epochs is the position difference between the 2019 and 2015 observations,
$\Delta p$, corrected by $\Delta C$:

$D=\Delta p-\Delta C\quad.$ (1)

The uncertainty of $D$ was calculated by propagating the uncertainties of
$\Delta p$ and $\Delta C$. To calculate the uncertainty of $\Delta p$, the
error bars reported by imfit (Tables 2 and 3) were added in quadrature. The
uncertainty of $\Delta C$ was calculated by adding the uncertainty of
$C_{2015}$ and $C_{2019}$ in quadrature, where each of them was measured using
a bootstrap method (Efron & Tibshirani 1991). In this method, several
realisations (500,000 in this case) of the calculation of $\Delta C$ were done
using samples of reference sources which were variations of the original
sample of $N$ sources (in this case the $N=21$ sources in Tables 2 and 3).
Each varied sample was generated by selecting $N$ reference sources randomly
from the original list of $N$ sources; and note that any source could be
selected more than once in a given realisation. The standard deviation of all
the offsets was defined as the uncertainty of the centroid offset calculated
from the original sample.

Using the 21 reference sources in Tables 2 and 3 we obtained a displacement
for the pulsar in RA of $0\aas@@fstack{\prime\prime}02\pm
0\aas@@fstack{\prime\prime}04$, and a displacement in DEC of
$-0\aas@@fstack{\prime\prime}12\pm 0\aas@@fstack{\prime\prime}05$ (Table 4).
Figure 2 plots the offset in RA and DEC for the sources listed in Tables 2 and
3. The offsets between epochs of some individual sources can be significantly
different to the centroid’s offset, and sometimes even larger than the offset
of the pulsar. For example, source 18 has particularly large residuals, as can
be seen in the top panel of Fig. 2, probably due to its own proper motion. In
order to test the effect of this large shift in position of individual
sources, we experimented by removing the reference sources with large
residuals and calculated everything again. The results obtained for most
combinations were consistent (considering uncertainties) with the results
described above: a marginal detection of the pulsar displacement in DEC (of
negative sign) and a non-detection of the pulsar displacement in RA. As an
example, in Table 4 and Fig. 2 (bottom) we present the results obtained when
using only the 17 sources with offsets that lie within the standard deviation
of the entire sample (that is, are between the dotted lines in the top plots
in Fig. 2). The difference in the resulting proper motion estimate when using
both samples is consistent with zero. These results indicate that both the
reference frame and the measured proper motion for the pulsar are robust.

Table 4: Offsets and displacements for two sets of reference sources.

Parameter | RA | DEC
---|---|---
$\Delta p$ (″) | $-0.04\pm 0.03$ | $-0.03\pm 0.04$
Using all the reference sources
$\Delta C$ (″) | $-0.06\pm 0.03$ | $0.10\pm 0.02$
$D$ (″) | $0.02\pm 0.04$ | $-0.12\pm 0.05$
Proper motion (mas yr-1) | $5\pm 10$ | $-29\pm 11$
Using 17 reference sources(a)
$\Delta C$ (″) | $-0.09\pm 0.03$ | $0.10\pm 0.02$
$D$ (″) | $0.04\pm 0.04$ | $-0.12\pm 0.05$
Proper motion (mas yr-1) | $10\pm 10$ | $-29\pm 11$
777The offset of the pulsar is $\Delta p$. The offset of the reference frame
($\Delta C$) and pulsar effective displacement ($D$) are given for two sets of
reference sources.
$a$$a$footnotetext: Having rejected the sources with ID 4, 7, 13, and 18.

Figure 2: Offsets between the positions measured in 2015 and 2019 of the
sources in Tables 2 and 3. The pulsar is source 17 and its offset ($\Delta p$)
is plotted with a blue square. The centroid’s offset and the standard
deviation of the reference sources around this value are shown with horizontal
dotted lines. The grey area in each plot is the uncertainty of the centroid’s
offset, as calculated using a bootstrap method. Top: Results for RA and DEC
obtained for 21 reference sources. Bottom: Results obtained for the 17
reference sources that present offsets consistent with the centroid’s offset.
The rejected sources are plotted with smaller red symbols and were not used to
measure the centroid (see Table 4).

Distributions of the proper motion components $\mu_{\alpha}$ and
$\mu_{\delta}$, for the analysis that uses 17 reference sources, are shown in
Fig. 3. Each value comes from a measurement of $D$ performed via a particular
realisation of the frame of reference (using the bootstrap process described
above) and the pulsar offset $\Delta p$. For $\Delta p$ we drew values from
normal distributions centred at the RA and DEC offset components presented in
Table 4 and with dispersions equal to the quoted uncertainties. Note that the
distribution for $\mu_{\delta}$ is wider mainly because of the orientation of
the elongated beam shape in the 2019 observation, which covers a longer range
in DEC than in RA (Fig. 1). The proper motions were calculated for a
difference in time of $4.2433$ yr between observations. Our results are
consistent with a motion of the pulsar between both epochs only in DEC, which
implies a proper motion $\mu_{\delta}=-29\pm 11$ mas yr-1. We acknowledge that
this is a marginal detection, as the displacement of the pulsar in DEC is
similar to the standard deviation of the other sources displacements (the
horizontal dotted lines in Fig. 2).

Figure 3: Pulsar proper motion values obtained from realisations of both the
centroid of 17 reference sources (from the bootstrap method) and the pulsar
offset $\Delta p$ (using a Gaussian distribution centred at the measured value
and with a dispersion given by its uncertainty, Table 4).

##  4 A complicated association with G354.8$-$0.8

If PSR J1734$-$3333 and SNR G354.8$-$0.8 were associated, the proper motion of
the pulsar should be $-172$ mas yr-1 in RA and $64$ mas yr-1 in DEC, assuming
an age of $8.1$ kyr (the characteristic age of the pulsar) and that it has
travelled with constant velocity since the explosion. From our results, we can
write the $2$-$\sigma$ limits $-10<\mu_{\alpha}<30$ mas yr-1 (Fig. 3, Table
4), which is inconsistent with the expected $-172$ mas yr-1 at high
significance. In declination, our detection $\mu_{\delta}=-29\pm 11$ mas yr-1
is also significantly inconsistent with the expectation, and implies movement
of the pulsar but towards the SNR. These results therefore indicate that the
pulsar and G354.8–0.8 are not related, meaning that the original SNR that was
born with the pulsar might have faded and the system may be older than
$10$-$100$ kyr.

We note that $\mu_{\delta}=-29\pm 11$ mas yr-1 implies a transverse velocity
of $630\pm 210$ km s-1 (at a distance $d_{4.5}=4.5$ kpc, as estimated from the
dispersion measure and the YMW16 electron density model, Yao et al. 2017),
which is faster than the average transverse speed amongst pulsars but still a
possible value. The distance to the pulsar is however rather uncertain, as it
depends on the model used for the electron density column. The NE2001 model
(Cordes & Lazio 2002) indicates that the distance is $d_{6.1}=6.1$ kpc. This
distance would imply a speed of $870\pm 290$ km s-1, which lies at the very
high end of the distribution of pulsar transverse velocities (Hobbs et al.
2005).

However, the possibility that the marginal detection of $\mu_{\delta}$ is a
statistical fluke cannot be completely neglected. Hence we also consider the
possibility that the proper motion of the pulsar is below our detection
capabilities, and consider a $2$-$\sigma$ upper limit for the total proper
motion $\mu<28$ mas yr-1, obtained by doubling the calculated uncertainties in
RA and DEC. Given this proper motion constraint, if pulsar and SNR were indeed
associated, the pulsar would need at least $45$ kyr to cover the distance
determined by the $21$′ offset with the SNR centre. If the system was older
than $45$ kyr, as required by this upper limit scenario and for a true SNR-
pulsar association, the pulsar transverse velocity would be less than
$590\,d/d_{4.5}$ km s-1, where $d$ is the assumed distance to the pulsar. This
is however an unlikely scenario given the results of our observations, which
suggest that the pulsar moves in a direction $130^{\circ}$ away of what the
association with the SNR requires (with a measured position angle error of
$16^{\circ}$, this is a $8$-$\sigma$ distance in position angle).

###  4.1 Another SNR as candidate?

Figure 4 shows the known SNRs that lie in the vicinity of PSR J1734$-$3333\.
Their positions were taken from the Green (2019) catalogue of supernova
remnants. We also searched for SNRs detected at lower radio wavelengths, hence
potentially older than those detected at shorter wavelengths (Roy & Bhatnagar
2006; Hurley-Walker et al. 2019), but found none near the pulsar. In Fig. 4,
regions that show where the pulsar was in the past are plotted, based on the
proper motion we obtained. Just by considering the direction of the proper
motion, it is tempting to consider either G355.6$-$00.0 or G355.4+00.7 as
candidates to the associated SNR for the pulsar. In terms of the direction,
this is plausible given that $\mu_{\alpha}$ is essentially unconstrained by
our data. However, the large separation between the PSR and both SNRs makes
the association unlikely, as it would require an age for the SNRs greater than
100 kyr, and it is unlikely that SNRs remain detectable in the radio for this
long (Sarbadhicary et al. 2017). Moreover, G355.6+00.0 is believed to lie at a
distance of around 12 kpc, albeit with large uncertainties (Case &
Bhattacharya 1998; Minami et al. 2013), which is much larger than what it is
estimated for the pulsar. For this remnant, the age is estimated to be $\sim
20$ kyr by Minami et al. (2013) and $9\pm 1.7$ kyr by Leahy et al. (2020),
considerably less than what is required to be associated with PSR
J1734$-$3333.

Figure 4: Supernova remnants in the vicinity of PSR J1734$-$3333\. The black
circles represent SNRs from the Green (2019) catalogue and SNR G354.8$-$0.8 is
in green. The diameter of each circle is given by the size of the SNR listed
in the catalogue. The blue dot shows the current location of the pulsar, and
the horizontal dashed lines show where the pulsar would have been at each
labeled time in the past considering a proper motion $\mu_{\delta}=-29$ mas
yr-1. The shaded regions mark the possible location of the pulsar in the past
considering also a proper motion in RA, for which we use the $1$-$\sigma$
range $\mu_{\alpha}\in[0,20]$ mas yr-1 (dark shading), and the $2$-$\sigma$
range $\mu_{\alpha}\in[-10,30]$ mas yr-1 (light shading).

##  5 Radio spectral index

Besides the astrometry analysis of the sources using the S-band of the VLA, we
also observed the field in L-band during the 2015 epoch. This allows for the
measurement of the spectral index of the pulsar.

The spectral index for pulsars is defined from a relation between the radio
flux density $S$ and the observing frequency $f$, given by $S\propto
f^{\alpha}$. Using the 2015 observation in the L band, we measure a flux
density for PSR J1734$-$3333 of $710\pm 70\,\mu$Jy, centred at $1.5$ GHz.
Combining this with the S-band ($3.0$ GHz) 2015 measurement of $344\pm
8\,\mu$Jy (Table 2) we obtain a spectral index $\alpha=-1.1\pm 0.3$. The flux
density of the pulsar in S-band does not change significantly between 2015 and
2019, so this result is the same if the S-band measurement from 2019 was used
instead. A spectral index of $-1.1$ is higher but still consistent with values
measured for other normal pulsars, which exhibit a mean between $-1.8$ and
$-1.6$ (Toscano et al. 1998; Maron et al. 2000; Jankowski et al. 2018). Thus
the spectrum of PSR J1734$-$3333 is somewhat flatter than the spectra of most
other radio pulsars. Magnetars, on the other hand, when detected in the radio
exhibit flat radio spectra (in general $\alpha<-0.8$ for the magnetars, e.g.
Huang et al. 2021).

##  6 Discusion

Interferometric observations of PSR J1734$-$3333 were performed on two epochs
separated by $4.2$ yr. Astrometric measurements and comparison of both images
allow a marginal detection of the proper motion, $\mu_{\alpha}=10\pm 10$ mas
yr-1 and $\mu_{\delta}=-29\pm 11$ mas yr-1, which indicates that the pulsar
and SNR G354.8$-$0.8 are not related. We also consider a scenario in which
this detection is not real and assume an upper limit on the total proper
motion of $\mu<28$ mas yr-1. Both scenarios suggest that the pulsar is older
than $45$-$100$ kyr, which can be compared to its characteristic age of only
$8.1$ kyr.

The spectral index that we measure is flatter than what is found for most
pulsars and approaches those measured for the six magnetars detected at radio
wavelengths. Thus not only the rotational properties but also some aspects of
the radio emission of PSR J1734$-$3333 resemble magnetar properties. If PSR
J1734$-$3333 displays one day magnetar-like high energy behaviour, then it
will be evident that other magnetars also had a past-life as radio pulsars and
are much older than what has normally been assumed. In the likely scenario
that magnetars are powered by intense magnetic fields –and assuming that spin-
down based measurements can reliably be used to estimate $B$ in this type of
objects, what physical mechanism could make high-$B$ pulsars evolve into even-
higher-$B$ magnetars?

While the X-ray properties of this pulsar are unremarkable in general, the
temperature measured from those observations suggests that it may be hotter
than other pulsars of similar $\tau_{c}$, though not as hot as the magnetars
(Olausen et al. 2013; Keane et al. 2013). If the pulsar was older than $45$
kyr, then its temperature would be particularly high for its age, thereby
supporting the idea that the magnetic field could be evolving quickly in this
pulsar (the amplitude of its surface magnetic dipole component increasing,
Muslimov & Page 1996; Espinoza et al. 2011; Ho 2011, 2015) and generating
additional heat (e.g. Kaminker et al. 2006; Ho et al. 2012; De Grandis et al.
2020). Such evolution could be caused by the emergence of a field that was
buried by accretion from a short-lived fallback disc after the supernova
explosion (Chevalier 1989; Bernal et al. 2013). There are calculations that
relate pulsar proper motions with the rate of magnetic field growth, which in
turn is related to the braking index (Guneydas & Eksi 2013). For PSR
J1734$-$3333 this model could accommodate transverse speeds of up to $\sim
500$ km s-1, which is somewhat slower than our measurement. An alternative
scenario is that the spin evolution of the pulsar has been driven by accretion
from a permanent fallback disc, for which models predict a pulsar age in a
similar range to the one we obtain (Çalişkan et al. 2013; Liu et al. 2014). In
this model magnetars are not powered by the decay of very powerful magnetic
fields but by accretion (Çalışkan & Ertan 2012).

More timing observations to monitor the braking index and more sensitive X-ray
studies will help to assess the likelihood of the above scenarios and
understand the future of PSR J1734$-$3333.

##  7 Acknowledgements

C.M.E acknowledges support by ANID FONDECYT/Regular grants 1171421 and
1211964. M.V-N. acknowledges support by ANID FONDECYT grant 11191205.

## References

  * Archibald et al. (2016) Archibald, R. F., Kaspi, V. M., Tendulkar, S. P., & Scholz, P. 2016, ApJ, 829, L21
  * Bernal et al. (2013) Bernal, C. G., Page, D., & Lee, W. H. 2013, ApJ, 770, 106
  * Briggs et al. (1999) Briggs, D. S., Schwab, F. R., & Sramek, R. A. 1999, in Astronomical Society of the Pacific Conference Series, Vol. 180, Synthesis Imaging in Radio Astronomy II, ed. G. B. Taylor, C. L. Carilli, & R. A. Perley, 127
  * Camilo et al. (2007) Camilo, F., Ransom, S. M., Halpern, J. P., & Reynolds, J. 2007, ApJ, 666, L93
  * Camilo et al. (2006) Camilo, F., Ransom, S. M., Halpern, J. P., et al. 2006, Nature, 442, 892
  * Case & Bhattacharya (1998) Case, G. L. & Bhattacharya, D. 1998, ApJ, 504, 761
  * Çalışkan & Ertan (2012) Çalışkan, Ş. & Ertan, Ü. 2012, ApJ, 758, 98
  * Çalişkan et al. (2013) Çalişkan, Ş., Ertan, Ü., Alpar, M. A., Trümper, J. E., & Kylafis, N. D. 2013, MNRAS, 431, 1136
  * Chatterjee et al. (2005) Chatterjee, S., Vlemmings, W. H. T., Brisken, W. F., et al. 2005, ApJ, 630, L61
  * Chevalier (1989) Chevalier, R. A. 1989, ApJ, 346, 847
  * Cordes & Lazio (2002) Cordes, J. M. & Lazio, T. J. W. 2002, arXiv e-prints, arXiv:astro-ph/0207156
  * De Grandis et al. (2020) De Grandis, D., Turolla, R., Wood, T. S., et al. 2020, ApJ, 903, 40
  * Deller et al. (2019) Deller, A. T., Goss, W. M., Brisken, W. F., et al. 2019, ApJ, 875, 100
  * Eatough et al. (2013) Eatough, R. P., Falcke, H., Karuppusamy, R., et al. 2013, Nature, 501, 391
  * Efron & Tibshirani (1991) Efron, B. & Tibshirani, R. 1991, Science, 253, 390
  * Espinoza et al. (2011) Espinoza, C. M., Lyne, A. G., Kramer, M., Manchester, R. N., & Kaspi, V. M. 2011, ApJ, 741, L13
  * Espinoza et al. (2017) Espinoza, C. M., Lyne, A. G., & Stappers, B. W. 2017, MNRAS, 466, 147
  * Frail et al. (1994) Frail, D. A., Goss, W. M., & Whiteoak, J. B. Z. 1994, ApJ, 437, 781
  * Gaia Collaboration et al. (2018) Gaia Collaboration, Katz, D., Antoja, T., et al. 2018, A&A, 616, A11
  * Gavriil et al. (2008) Gavriil, F. P., Gonzalez, M. E., Gotthelf, E. V., et al. 2008, Science, 319, 1802
  * Green (2014) Green, D. A. 2014, Bulletin of the Astronomical Society of India, 42, 47
  * Green (2019) Green, D. A. 2019, Journal of Astrophysics and Astronomy, 40, 36
  * Guneydas & Eksi (2013) Guneydas, A. & Eksi, K. Y. 2013, MNRAS, 430, L59
  * Ho (2011) Ho, W. C. G. 2011, MNRAS, 414, 2567
  * Ho (2015) Ho, W. C. G. 2015, MNRAS, 452, 845
  * Ho et al. (2012) Ho, W. C. G., Glampedakis, K., & Andersson, N. 2012, MNRAS, 422, 2632
  * Hobbs et al. (2005) Hobbs, G., Lorimer, D. R., Lyne, A. G., & Kramer, M. 2005, MNRAS, 360, 974
  * Hu et al. (2020) Hu, C.-P., Begiçarslan, B., Güver, T., et al. 2020, ApJ, 902, 1
  * Huang et al. (2021) Huang, Z.-P., Yan, Z., Shen, Z.-Q., et al. 2021, MNRAS, 505, 1311
  * Hurley-Walker et al. (2019) Hurley-Walker, N., Filipović, M. D., Gaensler, B. M., et al. 2019, PASA, 36, e045
  * Jankowski et al. (2018) Jankowski, F., van Straten, W., Keane, E. F., et al. 2018, MNRAS, 473, 4436
  * Kaminker et al. (2006) Kaminker, A. D., Yakovlev, D. G., Potekhin, A. Y., et al. 2006, MNRAS, 371, 477
  * Kaspi & Beloborodov (2017) Kaspi, V. M. & Beloborodov, A. M. 2017, ARA&A, 55, 261
  * Keane & Kramer (2008) Keane, E. F. & Kramer, M. 2008, MNRAS, 391, 2009
  * Keane et al. (2013) Keane, E. F., McLaughlin, M. A., Kramer, M., et al. 2013, ApJ, 764, 180
  * Kuiper & Hermsen (2009) Kuiper, L. & Hermsen, W. 2009, A&A, 501, 1031
  * Leahy et al. (2020) Leahy, D. A., Ranasinghe, S., & Gelowitz, M. 2020, ApJS, 248, 16
  * Liu et al. (2014) Liu, X.-W., Xu, R.-X., Qiao, G.-J., Han, J.-L., & Tong, H. 2014, Research in Astronomy and Astrophysics, 14, 85
  * Lower et al. (2021) Lower, M. E., Johnston, S., Dunn, L., et al. 2021, MNRAS, 508, 3251
  * Lyne & Graham-Smith (2012) Lyne, A. & Graham-Smith, F. 2012, Pulsar Astronomy
  * Lyne (2004) Lyne, A. G. 2004, in Young Neutron Stars and Their Environments, ed. F. Camilo & B. M. Gaensler, Vol. 218, 257
  * Manchester et al. (2002) Manchester, R. N., Bell, J. F., Camilo, F., et al. 2002, in Astronomical Society of the Pacific Conference Series, Vol. 271, Neutron Stars in Supernova Remnants, ed. P. O. Slane & B. M. Gaensler, 31
  * Manchester et al. (2005) Manchester, R. N., Hobbs, G. B., Teoh, A., & Hobbs, M. 2005, AJ, 129, 1993
  * Manchester & Taylor (1977) Manchester, R. N. & Taylor, J. H. 1977, Pulsars
  * Maron et al. (2000) Maron, O., Kijak, J., Kramer, M., & Wielebinski, R. 2000, A&AS, 147, 195
  * McMullin et al. (2007) McMullin, J. P., Waters, B., Schiebel, D., Young, W., & Golap, K. 2007, in Astronomical Society of the Pacific Conference Series, Vol. 376, Astronomical Data Analysis Software and Systems XVI, ed. R. A. Shaw, F. Hill, & D. J. Bell, 127
  * Mereghetti et al. (2015) Mereghetti, S., Pons, J. A., & Melatos, A. 2015, Space Sci. Rev., 191, 315
  * Minami et al. (2013) Minami, S., Ota, N., Yamauchi, S., & Koyama, K. 2013, PASJ, 65, 99
  * Morris et al. (2002) Morris, D. J., Hobbs, G., Lyne, A. G., et al. 2002, MNRAS, 335, 275
  * Muslimov & Page (1996) Muslimov, A. & Page, D. 1996, ApJ, 458, 347
  * Olausen & Kaspi (2014) Olausen, S. A. & Kaspi, V. M. 2014, ApJS, 212, 6
  * Olausen et al. (2013) Olausen, S. A., Zhu, W. W., Vogel, J. K., et al. 2013, ApJ, 764, 1
  * Parthasarathy et al. (2020) Parthasarathy, A., Johnston, S., Shannon, R. M., et al. 2020, MNRAS, 494, 2012
  * Rea et al. (2010) Rea, N., Esposito, P., Turolla, R., et al. 2010, Science, 330, 944
  * Rea et al. (2012) Rea, N., Israel, G. L., Esposito, P., et al. 2012, ApJ, 754, 27
  * Rea et al. (2014) Rea, N., Viganò, D., Israel, G. L., Pons, J. A., & Torres, D. F. 2014, ApJ, 781, L17
  * Roy & Bhatnagar (2006) Roy, S. & Bhatnagar, S. 2006, in Journal of Physics Conference Series, Vol. 54, Journal of Physics Conference Series, 152–155
  * Sarbadhicary et al. (2017) Sarbadhicary, S. K., Badenes, C., Chomiuk, L., Caprioli, D., & Huizenga, D. 2017, MNRAS, 464, 2326
  * Socrates et al. (2005) Socrates, A., Blaes, O., Hungerford, A., & Fryer, C. L. 2005, ApJ, 632, 531
  * Stephenson & Green (2005) Stephenson, F. R. & Green, D. A. 2005, Journal for the History of Astronomy, 36, 217
  * Suzuki et al. (2021) Suzuki, H., Bamba, A., & Shibata, S. 2021, ApJ, 914, 103
  * Toscano et al. (1998) Toscano, M., Bailes, M., Manchester, R. N., & Sandhu, J. S. 1998, ApJ, 506, 863
  * Whiteoak & Green (1996) Whiteoak, J. B. Z. & Green, A. J. 1996, A&AS, 118, 329
  * Woods & Thompson (2006) Woods, P. M. & Thompson, C. 2006, Soft gamma repeaters and anomalous X-ray pulsars: magnetar candidates, Vol. 39, 547–586
  * Yao et al. (2017) Yao, J. M., Manchester, R. N., & Wang, N. 2017, ApJ, 835, 29


# Spanish and LLM Benchmarks: is MMLU Lost in Translation?

Irene Plaza, Nina Melero, Cristina del Pozo, Javier Conde and Pedro Reviriego
ETSI de Telecomunicación
Universidad Politécnica de Madrid
28040 Madrid, Spain
Marina Mayor-Rocher
Facultad de Filosofía y Letras
Universidad Autónoma de Madrid
28049 Madrid, Spain
María Grandury
SomosNLP
24402, Ponferrada, Spain

###### Abstract

The evaluation of Large Language Models (LLMs) is a key element in their
continuous improvement process and many benchmarks have been developed to
assess the performance of LLMs in different tasks and topics. As LLMs become
adopted worldwide, evaluating them in languages other than English is
increasingly important. However, most LLM benchmarks are simply translated
using an automated tool and then run in the target language. This means that
the results depend not only on the LLM performance in that language but also
on the quality of the translation. In this paper, we consider the case of the
well-known Massive Multitask Language Understanding (MMLU) benchmark. Selected
categories of the benchmark are translated into Spanish using Azure Translator
and ChatGPT4 and run on ChatGPT4. Next, the results are processed to identify
the test items that produce different answers in Spanish and English. Those
are then analyzed manually to understand if the automatic translation caused
the change. The results show that a significant fraction of the failing items
can be attributed to mistakes in the translation of the benchmark. These
results make a strong case for improving benchmarks in languages other than
English by at least revising the translations of the items and preferably by
adapting the tests to the target language by experts.

_Keywords_ LLM $\cdot$ Evaluation $\cdot$ Benchmarks $\cdot$ Spanish

##  1 Introduction

Large Language Models are becoming a fundamental block in modern computing
systems enabling new applications and facilitating the interaction with users
Minaee et al., (2024). However, LLMs have limitations and their performance
has to be well understood before using them on a given application Zhao et
al., (2023). This has motivated the development of a large number of LLM
evaluation benchmarks that test the knowledge that models have of many
different topics and how well they can perform tasks such as logic reasoning
or problem-solving Guo et al., (2023). Most of these benchmarks are designed
so that the LLM responses can be processed automatically thus enabling testing
at scale with thousands of questions. This is commonly achieved by using
multiple-choice tests.

There are LLM benchmarks to evaluate a wide range of tasks and topics. For
example, there are tests to evaluate the capabilities of LLMs to solve common
sense reasoning problems Zellers et al., (2019) or to answer mathematical
questions Hendrycks et al., 2021b . To provide a more comprehensive
evaluation, some benchmarks evaluate several tasks, for example, the Multitask
Language Understanding (MMLU) test Hendrycks et al., 2021a  evaluates 57
different topics and other benchmarks increase the number of tasks and topics
to more than 200 Srivastava et al., (2022). The speed and energy dissipation
of LLMs are also important factors that are commonly evaluated in terms of the
number of tokens generated per second, the memory used or the energy per
token111https://huggingface.co/spaces/optimum/llm-perf-leaderboard, or with
more user-centric metrics like the time and energy needed to complete a given
task Conde et al., (2024). As LLMs become pervasive and used in almost any
domain and application, more benchmarks will be developed each having possibly
thousands or even hundreds of thousands of questions.

Another dimension of LLM evaluation is their performance in languages other
than English as in fact most users are native speakers of other languages.
Most benchmarks are written in English with questions taken in many cases from
different exams, such as university, high school or professional tests. The
simplest approach is to translate these same tests into other languages and
use them for multilingual evaluation. This clearly introduces a cultural bias,
especially when questions are related to subjects such as history, geography,
art or general culture. Ideally, specific tests should be developed or at
least adapted for each language.

However, this is not the only problem. To be able to evaluate LLMs in many
languages, and given the large number of questions of the benchmarks, the
standard procedure is to translate the English test to the target language
using automatic translation tools, for example, in the Okapi project Lai et
al., (2023) three benchmarks from the Open LLM Leaderboard Beeching et al.,
(2023) are translated using ChatGPT while in the evaluation of GPT4, the tests
were translated using Azure Translator Achiam et al., (2023). This implies
that the benchmarks in languages other than English are not only measuring the
performance of the LLM but also of the translation tool as the quality of the
translation can clearly impact the results.

In this work we perform an initial analysis of the impact of automatic
translation on one of the most widely used LLM benchmarks, the Multitask
Language Understanding (MMLU) test Hendrycks et al., 2021a  for one of the
most commonly used and chosen as a second language to learn, Spanish. The
analysis shows that automatic translation induces errors in the LLM answers
and thus distorts the benchmark’s results. Based on these findings, potential
solutions to this problem are also briefly discussed.

The rest of the work is organized as follows, in section 2 the methodology
used in our analysis is presented, followed by the results in section 3 and a
discussion of their implications and potential solutions in section 4. The
paper ends with the conclusion in section 5.

##  2 Methodology

This section discusses the methodology used in our analysis, first, the tests
used in the evaluation and tools selected are discussed to then describe the
evaluation procedure.

###  2.1 Tests and Tools

To evaluate the impact of automatic translation on the benchmarks, we have
selected three categories from the MMLU test Hendrycks et al., 2021a :
Miscellaneous, Philosophy, and US foreign policy with 783, 311, and 100
questions respectively. The first one covers a wide range of topics that can
be affected by translation, while the second one focuses on well-known content
that is universal. Finally, the last category focuses on US-related questions
that may also be prone to translation errors. Therefore, the three categories
can provide insights into the potential limitations of automatic translation.

In terms of translation tools, we consider two: Azure Translator and ChatGPT.
The reasoning behind our choices is that Azure Translator was the tool used to
evaluate MMLU for languages other than English in GPT4 Achiam et al., (2023).
Therefore, our findings would be applicable to GPT4 evaluation results. On the
other hand, it is interesting to check how well an LLM performs when
translating tests that will be used to evaluate the same LLM. Thus we
translate the questions with ChatGPT4 and then use them to evaluate it. This
enables us to check whether using the same tool for translation and testing
introduces any bias in the evaluation, for example, better performance may be
achieved when the same tool is used for both.

Finally, the LLM used to answer the questions, both in English and Spanish, is
GPT4, which is among the best-performing LLMs on the MMLU benchmark to date.
The rationale is that this model will produce the largest number of correct
answers in the absence of translation errors and thus will be the most
effective in detecting discrepancies due to translation errors. For example, a
model that already produces wrong answers for 50% of the questions in English
will not detect translation errors on that 50%. In the same way, it is
possible that right answers in English are incorrectly translated into Spanish
but answered correctly by the LLM.

###  2.2 Evaluation Procedure

The overall procedure is illustrated in Figure 1 and has the following steps:

  1. 1.

Translate the questions from the selected categories into Spanish using the
chosen tools: Azure Translator and ChatGPT4.

  2. 2.

Run the same MMLU tests on both the original and the Spanish translated
versions and log the answers and their log probabilities.

  3. 3.

Identify the questions in which the answers are correct in English and wrong
in Spanish.

  4. 4.

Analyze them manually. Two experts revise each question independently to avoid
mistakes or biases in the evaluation and the questions are divided into two
groups: correct and having translation errors. Revise and correct the
translation errors.

  5. 5.

Run the corrected questions again on the LLM. The number of correct answers is
logged and reported to estimate the impact of translation errors on the
benchmark results.

The procedure is designed to evaluate the impact of the errors in the
automatic translation on the test results. This enables us to focus only on
the questions that are answered correctly in English and wrongly in any of the
two Spanish translations, significantly reducing the effort needed for the
manual check of the translation. This is relevant as it enables the
methodology to scale to larger datasets with a reasonable effort and cost as
each question is checked and corrected by two experts.

Figure 1: Diagram of the evaluation methodology used.

###  2.3 Limitations

The proposed methodology has a number of limitations. Firstly, it cannot
detect errors in translation that do not affect the answer, i.e., if there is
a mistake in a translation but the LLM answer is correct it will not be
detected. This can be due to the translation error not affecting the LLM’s
ability to answer but also due to randomness in the responses. Secondly, the
fact that the LLM answers correctly to the revised translation may also be due
to randomness in some cases.

Finally, there are also limitations related to the number of questions tested
and the tools used. The evaluation was limited to approximately 1200 questions
and only two automatic translation tools to keep the effort of manual
revisions acceptable. To get a better estimate of the impact of automatic
translations, more questions taken from different benchmarks and using several
translation tools should be used. The results also depend on the prompts used
to ask the LLM, the LLM used, and its configuration parameters.

##  3 Results and Analysis

As per the proposed methodology, first, the questions are translated into
Spanish with Azure Translator and ChatGPT. The translations as well as the
rest of the results presented in the following, are available in a public
repository222https://zenodo.org/records/11314109. Next, the three versions of
the selected MMLU questions are run on ChatGPT4 and the results in terms of
the number and percentage of incorrect answers are summarized in Table 1. It
can be observed that there is a performance loss when the tests are run in
Spanish in all three categories and for both Azure and ChatGPT translations.
The relative loss is significantly larger in Philosophy and US foreign policy.

Table 1: Number (percentage) of incorrect answers for ChatGPT4. Category | English | Spanish (Azure) | Spanish (ChatGPT)
---|---|---|---
Miscellaneous | 47 (6%) | 65 (8.3%) | 66 (8.56%)
Philosophy | 43 (13.9%) | 66 (21.29%) | 57 (18.4%)
US foreign policy | 3 (3%) | 10 (10%) | 11 (11%)

The third step is to identify the questions that were correctly answered in
English but failed in Spanish. The number of such questions is given in Table
2. For US foreign policy, adding those to the failures in English gives the
number of failures in Spanish. This means that there are no correct answers in
Spanish for items that failed in English. Instead, for the other two
categories, the addition is larger than the number of failures in Spanish,
which means that there are a few correct answers in Spanish for items that
failed in English or that failed in both languages. Comparing the
translations, the one done with Azure Translator has more questions correctly
answered in English but failed in Spanish in total, and in two of the three
categories. This may be due to ChatGPT understanding better its own
translations.

Table 2: Number of correct answers in English and wrong in Spanish for ChatGPT4. Category | Spanish (Azure) | Spanish (ChatGPT)
---|---|---
Miscellaneous | 33 | 28
Philosophy | 29 | 21
US foreign policy | 7 | 8

In the fourth step, the questions with correct answers in English and wrong in
Spanish were analyzed manually to check the translations done by Azure
Translator and ChatGPT. There are clear examples of errors which can be
attributed to the translation. In the Miscellaneous category in the question
below, the correct answer is John Constable. For some reason, Azure Translator
translates the name to “Juan Alguacil” (in boldface) but not the others and
ChatGPT4 selects an incorrect answer. Instead, the answer is correct for the
ChatGPT translation as it does not translate John Constable.

[lq=8cm,language=english, correction, corsymb=] [br=1]Original question:
Which of the following artists is an English landscape painter who created a
poetic sense of changing atmospheric effects by using tiny applications of
local colour? Please answer:John Constable, William Morris, William Hogarth,
Joshua Reynolds [br=1]ChatGPT translated question:
¿Cuál de los siguientes artistas es un pintor paisajista inglés que creó un
sentido poético de los cambios en los efectos atmosféricos utilizando pequeñas
aplicaciones de color local? Por favor responda:John Constable, William
Morris, William Hogarth, Joshua Reynolds [br=4]Azure Translator question:
¿Cuál de los siguientes artistas es un pintor paisajista inglés que creó un
sentido poético de los efectos atmosféricos cambiantes mediante el uso de
pequeñas aplicaciones de color local? Por favor, responda:Juan Alguacil,
William Morris, William Hogarth, Joshua Reynolds

Similarly in the US foreign policy category, we have the following question
that is wrongly answered in Spanish for both Azure Translator and ChatGPT
translations. In this case, the issue is that the expression “American
multiplication table” is translated literally into Spanish and loses its
meaning of the large population growth in the US Sexton, (2018). Actually,
there is no translation for this expression, which points to a fundamental
limitation of translating benchmarks: there may be questions that cannot be
translated.

[lq=8cm,language=english, correction, corsymb=] [br=1]Original question:
What was meant by the term “American multiplication table”? Please
answer:Increase in the US population, Increase in US finances, Increase in US
military capability, Increase in US international influence [br=4]ChatGPT
translated question:
¿Qué se entendía por el término “tabla de multiplicación americana”? Por favor
responda:Aumento en la población de EE.UU, Aumento de las finanzas de EE.UU,
Aumento en la capacidad militar de EE.UU, Aumento en la influencia
internacional de EE.UU [br=4]Azure Translator question:
¿Qué se entendía por el término “tabla de multiplicación americana”? Por
favor, responda:Aumento en la población de EE.UU, Aumento de las finanzas de
EE.UU, Aumento en la capacidad militar de EE.UU, Aumento de la influencia
internacional de Estados Unidos

In addition to checking the translations to identify and correct mistakes, an
analysis of those mistakes was made to try to find patterns in the errors. The
conclusion was that the translation errors observed can be grouped into the
following main categories:

  1. 1.

Translation of a proper name: “John Constable” translated into “Juan
Alguacil”, or “Stephen King” as “Esteban Rey” which seems to demonstrate that
the model does not identify proper names as entities.

  2. 2.

Incorrect translation of a technical term: “Furrow opener” translated into
“abre surcos” instead of “arado”, “lieutenant general” as “general de
teniente” instead of “teniente general”.

  3. 3.

A term is not translated: “Wednesday child” translated into “El hijo de
Wednesday”, “Cicero” not translated into “Cicerón”.

  4. 4.

Cultural adaptation was needed in the translation: “The paper chase”
translated into “La persecución del papel”, which is not consistent with the
name used in Spain; and measures (length, currencies) are not adapted.

  5. 5.

Change of meaning: "grand” is incorrectly translated into Spanish as “big”,
“older” as “old” instead of as “great”, “rules make take into account…” as
“rules may take into account…”.

  6. 6.

Grammatical errors: the words in the question are feminine and the ones used
in the options appear as masculine or vice versa; and the text sometimes
contains ungrammatical sequences caused by literal translations (“how many
pence make a pound?” translated as “¿cuántos peniques hacen una libra?”.

Table 3 shows the results of the analysis done on the questions that had
correct answers in English and wrong in Spanish with the initial translation.
First, the number of questions in which the manual checking found issues in
the automatic translation is given. It can be observed that for Azure
translator the majority of the questions have errors in the translation while
for ChatGPT the numbers are lower but still significant.

As per the proposed methodology, in step 5, the incorrectly translated
questions were translated manually and rerun on ChatGPT4. The results in terms
of questions that are incorrectly answered after fixing the translation are
also shown in 3. It can be seen that a significant fraction of the questions
have the right answer after manually fixing the translation. In the case of
the Azure translations, the percentage is above 34% for all three categories
and exceeds 63% for Miscellaneous. In the case of ChatGPT, the number of
questions corrected is again lower in total and lower in two of the three
categories with values ranging from 9% to 37%. These results clearly
illustrate how the use of automatic translations can impact the results of
benchmarks ran in languages other than English.

Table 3: Analysis of the questions that had correct answers in English and wrong in Spanish for ChatGPT4. Category | Translation | Initial | Translation issues | After translation revision | Percentage corrected
---|---|---|---|---|---
Miscellaneous | Azure | 33 | 29 | 12 | 63.64%
Miscellaneous | ChatGPT | 28 | 15 | 17 | 32.14%
Philosophy | Azure | 29 | 25 | 19 | 34.48%
Philosophy | ChatGPT | 21 | 4 | 19 | 9.52%
US foreign policy | Azure | 7 | 5 | 4 | 42.86%
US foreign policy | ChatGPT | 8 | 6 | 5 | 37.5%

##  4 Discussion

The findings of our initial evaluation suggest that the use of automatic
translation of the questions in LLM benchmarks causes deviations in the
results due to errors in the translation. To eliminate those deviations and
ensure that the results are not contaminated by translation errors, the
translations should be revised by experts and, ideally, the tests should be
adapted to the language being evaluated. However, given the number of
questions on the LLM benchmarks and the number of languages evaluated, this
requires a large effort that calls for coordinated action from the community.

In fact, there are efforts in this direction such as the validation of the
Spanish Okapi benchmarks as part of the #Somos600M Project Grandury, (2024).
This community annotation effort uses open-source frameworks and is performed
by native Spanish-speaking volunteers. During the first two months, more than
60 persons participated and together covered one-third of the total number of
dataset items, which shows how time-consuming it is to manually validate and
correct these translations.

In the absence of manually adapted or checked benchmarks, our methodology
enables a fast evaluation of the impact of translation errors with limited
manual checking. Further refinements can be introduced to detect translation
errors. For example, after translating into Spanish we could translate back
into English the questions with different answers in both languages and run
the questions again. When the answer to this English translation is different
from the one of the original English question, the translation is likely to be
the culprit of the error. In general, developing strategies to identify these
issues automatically would be very helpful to understand the impact of
automatic translation and also to correct the translation errors.

In this work, we have focused on Spanish which is one of the most widely used
languages in the world and also is typically in the top five languages with
more data on the LLM training datasets. It would be interesting to study the
impact of translation errors on other languages. For languages with fewer data
and speakers, we would expect a larger number of translation errors but also a
larger number of genuine errors in the LLM answers. Therefore, the relative
impact of translation errors on the benchmark results compared to Spanish can
be either larger or smaller.

##  5 Conclusion and future work

In this work, we have analyzed the limitations of using automatic translation
of English benchmarks to evaluate LLMs in other languages. In more detail,
three categories from the MMLU benchmark have been translated into Spanish and
run on ChatGPT4. As previously mentioned, it would be interesting to test
other categories from the MMLU benchmark and other models. Then, the test
items for which the LLM answers are different in English and Spanish have been
identified and analyzed manually to understand if the differences can be
attributed to the translation. The results show that a significant fraction of
the differences are due to errors in the translation of the questions. These
findings highlight the need to improve non-English LLM benchmarks by at least
ensuring that the translations are correct and ideally by adapting the
questions to the target language and culture.

The development of language-specific or at least language-adapted benchmarks
should be a priority to provide better evaluation tools for multilingual LLMs.
To achieve that goal, open initiatives are needed to coordinate the efforts of
the community to develop for example language specific LLM leaderboards.

## References

  * Achiam et al., (2023) Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. (2023).  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774.
  * Beeching et al., (2023) Beeching, E., Fourrier, C., Habib, N., Han, S., Lambert, N., Rajani, N., Sanseviero, O., Tunstall, L., and Wolf, T. (2023).  Open llm leaderboard.  https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard.
  * Conde et al., (2024) Conde, J., González, M., Reviriego, P., Gao, Z., Liu, S., and Lombardi, F. (2024).  Speed and conversational large language models (llms): Not all is about tokens per second.  Computer, 57(8):in press.
  * Grandury, (2024) Grandury, M. (2024).  The #somos600m project: Generating nlp resources that represent the diversity of the languages from latam, the caribbean, and spain.  In North American Chapter of the Association for Computational Linguistics Conference: LatinX in AI (LXAI) Research Workshop.
  * Guo et al., (2023) Guo, Z., Jin, R., Liu, C., Huang, Y., Shi, D., Yu, L., Liu, Y., Li, J., Xiong, B., Xiong, D., et al. (2023).  Evaluating large language models: A comprehensive survey.  arXiv preprint arXiv:2310.19736.
  * (6) Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2021a).  Measuring massive multitask language understanding.  In International Conference on Learning Representations.
  * (7) Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. (2021b).  Measuring mathematical problem solving with the math dataset.  arXiv preprint arXiv:2103.03874.
  * Lai et al., (2023) Lai, V. D., Van Nguyen, C., Ngo, N. T., Nguyen, T., Dernoncourt, F., Rossi, R. A., and Nguyen, T. H. (2023).  Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback.  arXiv preprint arXiv:2307.16039.
  * Minaee et al., (2024) Minaee, S., Mikolov, T., Nikzad, N., Chenaghlu, M., Socher, R., Amatriain, X., and Gao, J. (2024).  Large language models: A survey.  arXiv preprint arXiv:2402.06196.
  * Sexton, (2018) Sexton, J. (2018).  A Nation Forged by Crisis: A New American History.  Hachette UK.
  * Srivastava et al., (2022) Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022).  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  arXiv preprint arXiv:2206.04615.
  * Zellers et al., (2019) Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. (2019).  Hellaswag: Can a machine really finish your sentence?  arXiv preprint arXiv:1905.07830.
  * Zhao et al., (2023) Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023).  A survey of large language models.  arXiv preprint arXiv:2303.18223.