# Chiral Meissner effect in time-reversal invariant Weyl superconductors

Vira Shyta  Institute for Theoretical Solid State Physics, IFW Dresden,
Helmholtzstr. 20, 01069 Dresden, Germany  Jeroen van den Brink  Institute for
Theoretical Solid State Physics, IFW Dresden, Helmholtzstr. 20, 01069 Dresden,
Germany  Institute for Theoretical Physics and Würzburg-Dresden Cluster of
Excellence ct.qmat, TU Dresden, 01069 Dresden, Germany  Flavio S. Nogueira
Institute for Theoretical Solid State Physics, IFW Dresden, Helmholtzstr. 20,
01069 Dresden, Germany

###### Abstract

Weyl semimetals are characterised by pairs of topologically protected nodes at
which electronic bands cross and electrons attain a definite chirality. Due to
the chiral anomaly, the non-conservation of charges with given chirality, the
axion term appears in their effective electromagnetic action. We determine how
this affects the properties of time-reversal invariant Weyl superconductors
(SCs) in the London regime. For type II SCs we show that axion coupling
generates magnetic $B$-fields transverse to a vortex. Above a critical axion
coupling vortices become unstable and a transition into a type I SC follows.
In this regime an applied $B$-field not only decays inside the SC within the
London penetration depth, but the axion coupling generates an additional
perpendicular field. Consequently the $B$-field inside the superconductor
progressively rotates away from the applied one when going into the bulk. At a
critical coupling the Meissner state breaks down. The novel chiral SC state
that then emerges has a periodically divergent susceptibility, at which the
winding of $B$ inside the superconductor jumps. Thus the axion coupling leaves
crisp experimentally observable signatures in Weyl SCs.

Introduction– Experimentally superconductivity has been reported in a number
of Weyl semimetals, both at ambient [1, 2, 3, 4, 5, 6] and high pressures [7,
8]. The topological nature of Weyl semimetals [9, 10, 11, 12, 13, 14, 15]
gives hope that Majorana zero modes bounded to vortices [16, 17] may be
detected in the future. Another recent experimental development in the field
was the recently found superconductivity in the time-reversal invariant (TRI)
Weyl semimetal PtBi2 [18, 5, 6], where the superconducting state seems to
occur only on the surface of the material. Accordingly, a Berezinskii-
Kosterlitz-Thouless phase transition [19, 20] was reported to occur [5].

As the low-energy electromagnetic response of superconductors (SCs) is
governed by the London equations, the question arises how the presence of Weyl
nodes modifies the electromagnetic properties of these Weyl superconductors,
in particular as to their Meissner effect for a type I and magnetic vortices
for a type II SC. Here we consider the London electrodynamics of TRI Weyl
superconductors [9, 10, 11, 12, 13, 14, 15], which in the case of Weyl
semimetals originates from the axion action,

$S_{a}=\frac{\alpha}{4\pi^{2}}\int dt\int
d^{3}r\vartheta(t,\bm{r})\bm{E}\cdot\bm{B},$ (1)

where $\alpha$ is the fine-structure constant and the axion field is assumed
to have the explicit form $\vartheta(t,\bm{r})=\bm{b}\cdot\bm{r}-b_{0}t$. Here
$\bm{b}$ and $b_{0}$ represent the separation between Weyl nodes in momentum
and energy, respectively [14, 15]. Specifically, we will be interested here in
the case where TRI holds, which leads to a net $\bm{b}=0$ due to the presence
of time reversed Weyl node pairs. A typical physical consequence of the axion
response is the chiral magnetic effect (CME) [21, 22, 23], which implies that
the current density contains a contribution $\bm{j}_{\rm
CME}=-a\bm{B}/(4\pi)$, where $a$ is the axion coupling constant, related to
$b_{0}$. Here we uncover a number of novel electrodynamic features that follow
from the interplay between the axion-induced CME and superconductivity in Weyl
systems.

Within the London theory the superconducting current is given by $\bm{j}_{\rm
SC}=q\rho_{s}({\mbox{\boldmath$\nabla$}}\theta-q\bm{A})$, where $\rho_{s}$ is
the superfluid stiffness, $q=2e$ is the charge, $\theta$ is the phase of the
order parameter and $\bm{A}$ the vector potential. Therefore, in a Weyl
superconductor with TRI the total current density is given by
$\bm{j}=\bm{j}_{\rm SC}+\bm{j}_{\rm CME}$.

As we will see, while the magnetic field expulsion from a superconductor is
ensured by its current being proportional to $\bm{A}$, the contribution from
CME, which is linear in $\bm{B}$, leads to a rotation in the magnetic field
screening. The chiral behavior of the Meissner effect may be understood by
first considering the non-superconducting phase. In this case
${\mbox{\boldmath$\nabla$}}\times\bm{B}=-a\bm{B}$ and we see that
$\nabla^{2}\bm{B}+a^{2}\bm{B}=0$, which yields spatially rotating magnetic
field profiles. When the system becomes superconducting, the Meissner
screening twists as a response to the rotation induced by the CME.

We also find that at a critical axion coupling $a_{c}$ the Meissner state
breaks down and the magnetic field starts to rotate periodically inside the
entire SC. The number of windings of the field inside the SC is quantized and
transitions between plateaus associated with a divergence in magnetic
susceptibility. Apart from this we establish how the axion coupling manifests
in the vortex properties of type II Weyl SCs. Since the vortex appears as a
response to the external magnetic field, the magnetic field inside of the
vortex is expected to be directed along the vortex line. However, in a Weyl
superconductor another, transverse component of the magnetic field is induced.
Due to the competition between the axion term and superconductivity a
transition occurs from this chiral vortex state to a conventional one without
vortices at a critical coupling $a_{c}$.

Axion London electrodynamics — Accounting for the chiral anomaly in Weyl
systems, the following Lagrangian governs the electromagnetic proporties of
TRI Weyl superconductors in the London regime

$\displaystyle\mathcal{L}$ $\displaystyle=$
$\displaystyle\frac{\epsilon}{8\pi}\bm{E}^{2}-\frac{1}{8\pi}\bm{B}^{2}+\frac{\rho_{s}}{2}\left[\left(\partial_{t}\theta-q\phi\right)^{2}-({\mbox{\boldmath$\nabla$}}\theta-q\bm{A})^{2}\right]$
(2) $\displaystyle-$
$\displaystyle\frac{q^{2}}{8\pi^{2}}b_{0}\varepsilon_{0ijk}A_{i}\partial_{j}A_{k},$

with units such that $\hbar=c=1$. The most important equation for us following
from the above Lagrangian is,

${\mbox{\boldmath$\nabla$}}\times\bm{B}=4\pi\bm{j}_{\rm
SC}+\epsilon\partial_{t}\bm{E}-a\bm{B},$ (3)

where $a=q^{2}b_{0}/\pi$. In the static regime that we consider here Eq. (3)
becomes a generalized London equation having the current density of the form
mentioned above, namely, one where the total current includes the CME
contribution, $\bm{j}_{\rm CME}=-a\bm{B}/(4\pi)$.

Vortex in type II Weyl SC — We first consider the fate of a magnetic vortex in
presence of an axion coupling. Due to the CME current, the analysis here
differs significantly from previous discussions on the subject based on the
Witten effect [24], where the field of the vortex induces a fractional charge
at the interface between an SC and a topological insulator [25], as well as a
fractional angular momentum [26, 27]. The vortex axion physics discussed below
does not involve the electric field and is intrinsic to TRI Weyl
superconductors, so proximity to a topological material needs not to be
assumed. Taking the curl of Eq. (3) we obtain in the static regime,

$-\nabla^{2}\bm{B}+a{\mbox{\boldmath$\nabla$}}\times\bm{B}+M^{2}\bm{B}=\frac{M^{2}\Phi_{0}}{2\pi}{\mbox{\boldmath$\Omega$}},$
(4)

where $M^{2}=4\pi q^{2}\rho_{s}$ represents the inverse square of the London
penetration depth $\lambda$ (in London theory without axion coupling
$\lambda=1/M$), $\Phi_{0}=2\pi/q$ is the elementary flux quantum, and
${\mbox{\boldmath$\Omega$}}={\mbox{\boldmath$\nabla$}}\times{\mbox{\boldmath$\nabla$}}\theta$
is the vorticity (recall that the curl of a gradient vanishes everywhere,
except there where topological defects like vortices exist [28]).

For an infinite system the exact solution is obtained by performing a Fourier
transform, which leads to

$\displaystyle B_{i}(p)=\frac{2\pi
M^{2}\Phi_{0}\delta(p_{z}){\bar{p}}^{2}}{{\bar{p}}^{4}-a^{2}p^{2}}\left(\delta_{iz}+\frac{ia}{{\bar{p}}^{2}}\varepsilon_{izk}p_{k}\right),$
(5)

where ${\bar{p}}^{2}=p^{2}+M^{2}$ and yields in real space
$\bm{B}(\bm{r})=B_{\varphi}(r)\hat{{\mbox{\boldmath$\varphi$}}}+B_{z}(r)\hat{\bm{z}}$,
with

$B_{\varphi}(r)=\frac{M^{2}\Phi_{0}}{2\pi\sqrt{a^{2}-a_{c}^{2}}}\sum_{\sigma=\pm}\sigma
M_{\sigma}K_{1}(M_{\sigma}r),$ (6)
$B_{z}(r)=\frac{M^{2}\Phi_{0}}{2\pi\sqrt{a_{c}^{2}-a^{2}}}\sum_{\sigma=\pm}M_{\sigma}K_{0}(M_{\sigma}r),$
(7)

where $K_{\alpha}(x)$ are modified Bessel functions of the second kind, and
$2M_{\pm}=\sqrt{a_{c}^{2}-a^{2}}\pm ia,$ where $a_{c}=2M$. Equations (6) and
(7) reduce to the well known London solution when $a=0$, yielding a magnetic
field parallel to the $z$-axis. The axion contribution generates a
$\varphi$-component of the magnetic field and, as a consequence, a component
of the current parallel to the vortex is generated. The total current
screening the vortex is thus encircling it in a helical manner, with a
handedness determined by the sign of $a$. This solution is well defined for
$a<a_{c}$.

In Fig. 1 the magnetic induction components corresponding to the vortex
solution of Eqs. (6) and (7) are displayed for different values of $a$. We
note the fields start to develop more spatial structure with increasing $a$,
so the Meissner effect around the vortex is not complete, with a spatially
damped oscillatory behavior emerging. For values of $a$ close to the critical
value $a_{c}$ the oscillations become much stronger, as shown in panels (c)
and (d) of Fig. 1. For $a\geq a_{c}$ the argument of the Bessel functions
appearing in Eqs. (6) and (7) become purely imaginary and the vortex solution
breaks down. As a result, the system must transition into a type I regime. As
at the SC phase transition the penetration depth $\lambda$ diverges and thus
$M$ tends to zero, for any finite (and possibly small) $a$, the regime $a\geq
a_{c}$ is always realized close to the SC phase transition.

(a)  (b)

(c)  (d)

(e)  (f)

Figure 1: Magnetic induction profiles for different values of the axion
coupling $a$ [panels (a) and (b) for the field components $B_{\varphi}$ and
$B_{z}$, respectively]. Fields are plotted in units of $M^{2}\Phi_{0}/(2\pi)$
against the radial coordinate $r$ in units of $M$. Panels (c) and (d) show the
field components for $a=1.99a_{/}2$, corresponding to a situation where $a$ is
very close to the critical value $a_{c}=2M$ for which the vortex solution
ceases to exist. We note that as $a$ approaches $a_{c}$ the field profiles
start to become more oscillatory. The onset of these spatial oscillations is
illustrated by the three-dimensional plots for $B_{z}$ in panels (e) and (f)
for $a=3a_{c}/2$ and $a=1.99a_{c}/2$, respectively.

The vortex solution can also be obtained exactly in a finite slab of thickness
$L$, with the vortex line perpendicular to the surface, as discussed in the
Supplemental Material. In this case it is interesting to consider the external
magnetic field explicitly for the case when the vortex solution does not
exist, where $a\geq a_{c}$. Here we can see better the perfect diamagnetic
character of the phase. Actually, since for this geometry the magnetic field
is perpendicular to the surface, continuity of the normal component implies
that the CME current vanishes, in which case the usual London equation
follows. The only possible solution henceforth is one of a vanishing magnetic
induction.

Meissner effect in type I Weyl SC — Compared to the situation of an external
magnetic field is applied along the surface normal of a type I London SC,
where perfect diamagnetism is unaltered by the CME, the more interesting case
is the one with the field applied parallel to the surface, rendering a finite
London penetration depth. Here, due to the CME, we find a crucial difference
relative the usual London electrodynamics: the Meissner screening works
differently, since application of an external magnetic field generates
additional components for the magnetic induction, as the differential
equations for the field components are coupled via the axion term. As a first
example, let us consider a semi-infinite superconductor located in the region
$x>0$ in the presence of an applied magnetic field $\bm{B}_{\rm ap}=B_{\rm
ap}\hat{\bm{y}}$. For this simple geometry one obtains the coupled equations

$\displaystyle-\partial_{x}^{2}B_{y}+M^{2}B_{y}-a\partial_{x}B_{z}=0,$ (8)
$\displaystyle-\partial_{x}^{2}B_{z}+M^{2}B_{z}+a\partial_{x}B_{y}=0$ (9)

with the boundary conditions, $B_{y}(x=0)=B_{ap}$,
$B_{y}(x\rightarrow\infty)=0$, $B_{z}(x=0)=0$, $B_{z}(x\rightarrow\infty)=0$.
As with the vortex solution, there are two distinct regimes to consider:
$a<a_{c}$ and $a>a_{c}$. The former yields the solution
$\bm{B}(x)=B_{ap}e^{-(x/2)\sqrt{a_{c}^{2}-a^{2}}}\hat{\bm{u}}(x)$ in terms of
the unit vector,

$\hat{\bm{u}}(x)=\cos\left(ax/2\right)\hat{\bm{y}}+\sin\left(ax/2\right)\hat{\bm{z}},$
(10)

and one observes that the field inside the SC rotates with respect to the
applied one, a chiral oscillatory feature as was found for the vortex. Thus,
applying a magnetic field in the $y$-direction does not only lead to a
Meissner effect with an exponentially decaying $y$-component of the field, but
also generates a similarly decaying field along the $z$-direction as a
consequence of the axion coupling. The corresponding field profiles are shown
in panels (a) and (b) of Fig. 2 for exemplary values of $a$ below and above
$a_{c}$. Note that the effective penetration depth renormalized by $a$ is
given by $\lambda=1/\sqrt{M^{2}-a^{2}/4}$ and is larger than the
unrenormalized London penetration depth $1/M$. Clearly also this effective
penetration depth diverges for $a=a_{c}$, showing that indeed this value of
$a$ corresponds to a critical point.

(a) $a=a_{c}/2$ (b) $a=5a_{c}/2$

(c) $a=0.4a_{c}$ (d) $a=3a_{c}/4$

(e) $a=0.95a_{c}$ (f) $a=1.1a_{c}$

Figure 2: Panel (a): Magnetic field components of a semi-infinite
superconductor located at $x>0$ and in the presence of an applied magnetic
field, $\bm{B}_{\rm ap}=B_{\rm ap}\hat{\bm{y}}$. Panel (b): Absence of
Meissner effect for $a=5a_{c}/2$ in a semi-infinite TRI Weyl superconductor.
From panels (c-e) we have $a<a_{c}$, corresponding to $a=0.4a_{c}$,
$a=3a_{c}/4$, and $a=0.95a_{c}$, respectively. We can see once more the onset
of spatial oscillations in the magnetic induction as $a$ increases. In panel
(f) $a=1.1a_{c}$, which is a little bit above its critical value. In this case
the axion coupling completely dominates over Meissner screening.

For $a\geq a_{c}$ there is no solution that fulfils the boundary condition
$B_{z}(x\rightarrow\infty)=0$. Thus, instead of demanding
$B_{y}(x\rightarrow\infty)=0$ and $B_{z}(x\rightarrow\infty)=0$, we only
enforce the boundary conditions at the surface $B_{y}(x=0)=B_{ap}$,
$B_{z}(x=0)=0$, and require the solutions to be real, which yields

$\bm{B}(x)=B_{ap}\cos(\sqrt{a_{c}^{2}-a^{2}}~{}x/2)\hat{\bm{u}}(x)$ (11)

Remarkably, the magnetic field inside of the sample exhibits a purely
oscillatory behavior and as illustrated in Fig. 2-(b) for $a=5a_{c}/2$, there
is no Meissner effect in the sense that the magnetic field is eventually
screened inside the bulk of the SC. Instead, it rotates around the surface
normal modulated by the field magnitude as it penetrates all the way into the
bulk. This brings to the fore once more the important role of the critical
value $a_{c}=2M$ of the axion coupling $a$ in modifying the nature of the
Meissner effect. To investigate how it precisely signals a phase transition we
compute the magnetic susceptibility as a response to the applied field. This
requires us to determine the average the magnetic induction over the system,
which cannot be done easily in a convergent manner in semi-infinite system and
it is more convenient to consider a finite slab geometry with two surfaces
such that $|x|<L/2={\bar{L}}$. We obtain,

$\displaystyle\bm{B}(x)=\frac{B_{ap}}{\sin(\bar{L}\sqrt{a^{2}-a_{c}^{2}})}$
$\displaystyle\times\sum_{\sigma=\pm}\sigma\sin\left[\frac{\sqrt{a^{2}-a_{c}^{2}}}{2}\left(x+\sigma\bar{L}\right)\right]\hat{\bm{u}}(x-\sigma\bar{L}),$
(12)

which has the advantage to hold any value of $a$. Figure 2 shows the magnetic
induction profiles corresponding to Eq. (Chiral Meissner effect in time-
reversal invariant Weyl superconductors) for increasing values of $a$ up to
slightly above $a_{c}$. The axion coupling causes a spatial rotation in the
Meissner screening, causing it to disappear for $a>a_{c}$ when the
oscillations are stronger. Remarkably, for $a$ significantly larger than
$a_{c}$ the oscillation amplitude can get larger than the applied field.

In order to elucidate the behavior for $a>a_{c}$, we calculate the diamagnetic
susceptibility $\chi$ from the spatial average of the magnetic induction. From
the expressions it is clear that axion-induced field component $B_{z}$
averages to zero for any $a$. For the component parallel to the applied field
we obtain the diamagnetic susceptibility

$\chi=\frac{\sqrt{a^{2}-a_{c}^{2}}}{LM^{2}\sin({\bar{L}}\sqrt{a^{2}-a_{c}^{2}})}\left[\cos({\bar{L}}\sqrt{a^{2}-a_{c}^{2}})-\cos(a{\bar{L}})\right]$
(13)

which for $a\to a_{c}$ goes to $\frac{2}{M^{2}L^{2}}[1-\cos(ML)].$ For $ML\gg
1$, corresponding to a large slab thickness compared to the London penetration
depth, $\chi$ vanishes for all $a\leq a_{c}$. For $a>a_{c}$, on the other
hand, the susceptibility diverges for

$a^{2}=a_{c}^{2}+\left(\frac{2\pi n}{L}\right)^{2}~{}~{}~{}{\rm
with}~{}~{}~{}~{}n\in\mathbb{N}.$ (14)

Thus, at quantized values of the axion coupling the system becomes unstable.
At these values the winding of the field inside the SC changes by unity. It is
interesting to note that this leads to a situation reminiscent of the Little-
Parks effect [29] in superconducting cylinders subjected to a parallel
magnetic field, where the persistent current suppresses the cylinder’s
superconductivity. Although in our case no such geometry is involved, the
currents generated by the CME affect the SC state of the slab, and
particularly so when the length scale associated to the axion coupling is
comparable to the London penetration depth.

Conclusions and outlook — As the axion term affects the properties of Weyl
superconductors in a rather nontrivial manner, several distinct experimentally
testable predictions follow from our results. This is clear for the currents
parallel to magnetic vortices and the magnetic fields perpendicular to them
induced by the axion coupling. In future work it will be interesting to
establish how this affects the vortex lattice and its stability. The vortex
becoming unstable at the critical axion coupling implies a transition from a
type II to a type I superconducting state. Such a transition between type I
and II SC in the same material is known as type 1.5 superconductivity for
multiband systems [30, 31]. As close to the SC phase transition the London
penetration depth diverges the critical axion coupling vanishes there. Thus
for any given axion coupling intrinsic to the Weyl SC material, close enough
to the SC transition the system automatically enters the strong coupling
regime where the vortex state becomes unstable and a type 1.5 regime may
ensue.

In type I superconductors the axion induced magnetic field component
perpendicular to the applied field and parallel to the surface may be explored
by surface sensitive probes, e.g. the magneto-optic Kerr effect. It is
interesting to note that the rotating $B$-field inside the SC also causes
vortices close to the surface to cant with respect to the applied field, which
results in magnetic stray fields outside the SC [32]. This is a rather
intricate consequence of the emergent field components transverse to the flux
line, as simple addition of a mirror vortex cannot fulfil the boundary
conditions on the surface.

For strong coupling the axion renormalization of the London penetration depth
may be probed experimentally, whereas the breaking down of the Meissner state
is associated with a periodically divergent susceptibility, at which the
winding of $\bm{B}$ inside the superconductor jumps, allowing in principle for
rather direct observation.

We thank Volodymyr Kravchuk for stimulating discussions. We acknowledge
financial support by the Deutsche Forschungsgemeinschaft (DFG, German Research
Foundation), through SFB 1143 project A5 and the Würzburg-Dresden Cluster of
Excellence on Complexity and Topology in Quantum Matter-ct.qmat (EXC 2147,
Project Id No. 390858490).

While preparing this manuscript we became aware of the work of M. Stålhammar
et al. [33], who report semi-infinite slab and cylinder solutions similar to
our results.

## References

  * Butch  _et al._ [2011] N. P. Butch, P. Syers, K. Kirshenbaum, A. P. Hope, and J. Paglione, “Superconductivity in the topological semimetal yptbi,” Phys. Rev. B 84, 220504 (2011).
  * Qi  _et al._ [2016] Y. Qi, P. G. Naumov, M. N. Ali, C. R. Rajamathi, W. Schnelle, O. Barkalov, M. Hanfland, S.-C. Wu, C. Shekhar, Y. Sun, V. Süß, M. Schmidt, U. Schwarz, E. Pippel, P. Werner, R. Hillebrand, T. Förster, E. Kampert, S. Parkin, R. J. Cava, C. Felser, B. Yan, and S. A. Medvedev, “Superconductivity in weyl semimetal candidate mote2,” Nature Communications 7, 11038 (2016).
  * Noh  _et al._ [2017] H.-J. Noh, J. Jeong, E.-J. Cho, K. Kim, B. I. Min, and B.-G. Park, “Experimental realization of type-ii dirac fermions in a ${\mathrm{pdte}}_{2}$ superconductor,” Phys. Rev. Lett. 119, 016401 (2017).
  * Leng  _et al._ [2017] H. Leng, C. Paulsen, Y. K. Huang, and A. de Visser, “Type-i superconductivity in the dirac semimetal ${\mathrm{pdte}}_{2}$,” Phys. Rev. B 96, 220506 (2017).
  * Veyrat  _et al._ [2023] A. Veyrat, V. Labracherie, D. L. Bashlakov, F. Caglieris, J. I. Facio, G. Shipunov, T. Charvin, R. Acharya, Y. Naidyuk, R. Giraud, J. van den Brink, B. Büchner, C. Hess, S. Aswartham, and J. Dufouleur, “Berezinskii–kosterlitz–thouless transition in the type-i weyl semimetal ptbi2,” Nano Letters 23, 1229 (2023).
  * Kuibarov  _et al._ [2023] A. Kuibarov, O. Suvorov, R. Vocaturo, A. Fedorov, R. Lou, L. Merkwitz, V. Voroshnin, J. I. Facio, K. Koepernik, A. Yaresko, G. Shipunov, S. Aswartham, J. van den Brink, B. Büchner, and S. Borisenko, “Superconducting arcs,” (2023), arXiv:2305.02900 [cond-mat.supr-con].
  * Kang  _et al._ [2015] D. Kang, Y. Zhou, W. Yi, C. Yang, J. Guo, Y. Shi, S. Zhang, Z. Wang, C. Zhang, S. Jiang, A. Li, K. Yang, Q. Wu, G. Zhang, L. Sun, and Z. Zhao, “Superconductivity emerging from a suppressed large magnetoresistant state in tungsten ditelluride,” Nature Communications 6, 7804 (2015).
  * Pan  _et al._ [2015] X.-C. Pan, X. Chen, H. Liu, Y. Feng, Z. Wei, Y. Zhou, Z. Chi, L. Pi, F. Yen, F. Song, X. Wan, Z. Yang, B. Wang, G. Wang, and Y. Zhang, “Pressure-driven dome-shaped superconductivity and electronic structural evolution in tungsten ditelluride,” Nature Communications 6, 7805 (2015).
  * Burkov and Balents [2011] A. A. Burkov and L. Balents, “Weyl semimetal in a topological insulator multilayer,” Phys. Rev. Lett. 107, 127205 (2011).
  * Zyuzin  _et al._ [2012] A. A. Zyuzin, S. Wu, and A. A. Burkov, “Weyl semimetal with broken time reversal and inversion symmetries,” Phys. Rev. B 85, 165110 (2012).
  * Zyuzin and Burkov [2012] A. A. Zyuzin and A. A. Burkov, “Topological response in weyl semimetals and the chiral anomaly,” Phys. Rev. B 86, 115133 (2012).
  * Grushin [2012] A. G. Grushin, “Consequences of a condensed matter realization of lorentz-violating qed in weyl semi-metals,” Phys. Rev. D 86, 045001 (2012).
  * Hosur  _et al._ [2014] P. Hosur, X. Dai, Z. Fang, and X.-L. Qi, “Time-reversal-invariant topological superconductivity in doped weyl semimetals,” Phys. Rev. B 90, 045130 (2014).
  * Yan and Felser [2017] B. Yan and C. Felser, “Topological materials: Weyl semimetals,” Annual Review of Condensed Matter Physics 8, 337 (2017).
  * Burkov [2018] A. Burkov, “Weyl metals,” Annual Review of Condensed Matter Physics 9, 359 (2018).
  * Kobayashi and Sato [2015] S. Kobayashi and M. Sato, “Topological superconductivity in dirac semimetals,” Phys. Rev. Lett. 115, 187001 (2015).
  * Yan  _et al._ [2020] Z. Yan, Z. Wu, and W. Huang, “Vortex end majorana zero modes in superconducting dirac and weyl semimetals,” Phys. Rev. Lett. 124, 257001 (2020).
  * Shipunov  _et al._ [2020] G. Shipunov, I. Kovalchuk, B. R. Piening, V. Labracherie, A. Veyrat, D. Wolf, A. Lubk, S. Subakti, R. Giraud, J. Dufouleur, S. Shokri, F. Caglieris, C. Hess, D. V. Efremov, B. Büchner, and S. Aswartham, “Polymorphic ${\mathrm{ptbi}}_{2}$: Growth, structure, and superconducting properties,” Phys. Rev. Materials 4, 124202 (2020).
  * Berezinskii [1971] V. Berezinskii, “Destruction of long-range order in one-dimensional and two-dimensional systems having a continuous symmetry group i. classical systems,” Sov. Phys. JETP 32, 493 (1971).
  * Kosterlitz and Thouless [1973] J. M. Kosterlitz and D. J. Thouless, “Ordering, metastability and phase transitions in two-dimensional systems,” Journal of Physics C: Solid State Physics 6, 1181 (1973).
  * Kharzeev [2014] D. E. Kharzeev, “The chiral magnetic effect and anomaly-induced transport,” Progress in Particle and Nuclear Physics 75, 133 (2014).
  * Burkov [2015] A. A. Burkov, “Chiral anomaly and transport in weyl metals,” Journal of Physics: Condensed Matter 27, 113201 (2015).
  * Armitage  _et al._ [2018] N. P. Armitage, E. J. Mele, and A. Vishwanath, “Weyl and dirac semimetals in three-dimensional solids,” Rev. Mod. Phys. 90, 015001 (2018).
  * Witten [1979] E. Witten, “Dyons of charge $e\theta/2\pi$,” Physics Letters B 86, 283 (1979).
  * Nogueira  _et al._ [2016] F. S. Nogueira, Z. Nussinov, and J. van den Brink, “Josephson currents induced by the witten effect,” Phys. Rev. Lett. 117, 167002 (2016).
  * Nogueira  _et al._ [2018] F. S. Nogueira, Z. Nussinov, and J. van den Brink, “Fractional angular momentum at topological insulator interfaces,” Phys. Rev. Lett. 121, 227001 (2018).
  * Nogueira and van den Brink [2022] F. S. Nogueira and J. van den Brink, “Absence of induced magnetic monopoles in maxwellian magnetoelectrics,” Phys. Rev. Res. 4, 013074 (2022).
  * Kleinert [1989] H. Kleinert,  _Gauge Fields in Condensed Matter: Vol. 1: Superflow and Vortex Lines (Disorder Fields, Phase Transitions) Vol. 2: Stresses and Defects (Differential Geometry, Crystal Melting)_ (World Scientific, 1989).
  * Little and Parks [1962] W. A. Little and R. D. Parks, “Observation of quantum periodicity in the transition temperature of a superconducting cylinder,” Phys. Rev. Lett. 9, 9 (1962).
  * Babaev and Speight [2005] E. Babaev and M. Speight, “Semi-meissner state and neither type-i nor type-ii superconductivity in multicomponent superconductors,” Phys. Rev. B 72, 180502 (2005).
  * Moshchalkov  _et al._ [2009] V. Moshchalkov, M. Menghini, T. Nishio, Q. H. Chen, A. V. Silhanek, V. H. Dao, L. F. Chibotaru, N. D. Zhigadlo, and J. Karpinski, “Type-1.5 superconductivity,” Phys. Rev. Lett. 102, 117001 (2009).
  * [32] V. Shyta, J. van den Brink, and F. Nogueira, “to be published,” .
  * Stålhammar  _et al._ [2023] M. Stålhammar, D. Rudneva, T. H. Hansson, and F. Wilczek, “Emergent chern-simons interactions in 3+1 dimensions,” (2023), arXiv:2309.10025 [cond-mat.mes-hall].

##  Appendix A Supplemental Material: vortex solution in a slab geometry

The vortex solution for a slab defined in the region $|z|<L/2$ is more easily
obtained by considering the differential equations for the vector potential.
The London equation for the vector potential reads,

$-\nabla^{2}\bm{A}+M^{2}\bm{A}=\frac{M^{2}\Phi_{0}}{2\pi}\frac{\hat{{\mbox{\boldmath$\varphi$}}}}{r}-a{\mbox{\boldmath$\nabla$}}\times\bm{A}.$
(15)

Based on the infinite vortex solution, we consider the Ansatz,

$\displaystyle A_{r}(r,z)$ $\displaystyle=$ $\displaystyle
a\frac{M^{2}\Phi_{0}}{2\pi}\int
dp\frac{J_{1}(pr)}{(p^{2}+M^{2})^{2}-a^{2}p^{2}}\frac{\partial\beta(p,z)}{\partial
z},$ (16) $\displaystyle A_{\varphi}(r,z)$ $\displaystyle=$
$\displaystyle\frac{M^{2}\Phi_{0}}{2\pi}\int
dp\frac{J_{1}(pr)(p^{2}+M^{2})}{(p^{2}+M^{2})^{2}-a^{2}p^{2}}\beta(p,z),$ (17)
$\displaystyle A_{z}(r,z)$ $\displaystyle=$
$\displaystyle-a\frac{M^{2}\Phi_{0}}{2\pi}\int
dp\frac{pJ_{0}(pr)}{(p^{2}+M^{2})^{2}-a^{2}p^{2}}\beta(p,z),$ (18)
$\displaystyle B_{r}$ $\displaystyle=$
$\displaystyle-\frac{M^{2}\Phi_{0}}{2\pi}\int
dp\frac{\left(p^{2}+M^{2}\right)J_{1}(pr)}{\left(p^{2}+M^{2}\right)^{2}-a^{2}p^{2}}\frac{\partial\beta(p,z)}{\partial
z},$ (19) $\displaystyle B_{\varphi}$ $\displaystyle=$
$\displaystyle-\frac{aM^{2}\Phi_{0}}{2\pi}\int
dp\frac{J_{1}(pr)\left[p^{2}\beta-\frac{\partial^{2}\beta(p,z)}{\partial
z^{2}}\right]}{\left(p^{2}+m^{2}\right)^{2}-a^{2}p^{2}},$ (20) $\displaystyle
B_{z}$ $\displaystyle=$ $\displaystyle\frac{M^{2}\Phi_{0}}{2\pi}\int
dp\frac{p\left(p^{2}+M^{2}\right)J_{0}(pr)}{\left(p^{2}+M^{2}\right)^{2}-a^{2}p^{2}}\beta(p,z),$
(21)

where the function $\beta(p,z)$ is determined by application of the boundary
conditions, which imposes the continuity of $\bm{A}$ and its derivatives with
respect to $z$ at the interfaces $z=\pm L/2$. With this we obtain,

$\displaystyle\beta(p,z)$
$\displaystyle=\left\\{\begin{array}[]{l}e^{p\left(\frac{L}{2}-z\right)}\frac{\left[\tau_{1}\sinh\left(\frac{L\tau_{1}}{2}\right)\cos\left(\frac{L\tau_{2}}{2}\right)-\tau_{2}\cosh\left(\frac{L\tau_{1}}{2}\right)\sin\left(\frac{L\tau_{2}}{2}\right)\right]}{p\cosh\left(\frac{L\tau_{1}}{2}\right)\cos\left(\frac{L\tau_{2}}{2}\right)+\tau_{1}\sinh\left(\frac{L\tau_{1}}{2}\right)\cos\left(\frac{L\tau_{2}}{2}\right)-\tau_{2}\cosh\left(\frac{L\tau_{1}}{2}\right)\sin\left(\frac{L\tau_{2}}{2}\right)}\\\
1-\frac{p\cosh\left(\tau_{1}z\right)\cos\left(\tau_{2}z\right)}{p\cosh\left(\frac{L\tau_{1}}{2}\right)\cos\left(\frac{L\tau_{2}}{2}\right)+\tau_{1}\sinh\left(\frac{L\tau_{1}}{2}\right)\cos\left(\frac{L\tau_{2}}{2}\right)-\tau_{2}\cosh\left(\frac{L\tau_{1}}{2}\right)\sin\left(\frac{L\tau_{2}}{2}\right)}\\\
e^{p\left(\frac{L}{2}+z\right)}\frac{\left[\tau_{1}\sinh\left(\frac{L\tau_{1}}{2}\right)\cos\left(\frac{L\tau_{2}}{2}\right)-\tau_{2}\cosh\left(\frac{L\tau_{1}}{2}\right)\sin\left(\frac{L\tau_{2}}{2}\right)\right]}{p\cosh\left(\frac{L\tau_{1}}{2}\right)\cos\left(\frac{L\tau_{2}}{2}\right)+\tau_{1}\sinh\left(\frac{L\tau_{1}}{2}\right)\cos\left(\frac{L\tau_{2}}{2}\right)-\tau_{2}\cosh\left(\frac{L\tau_{1}}{2}\right)\sin\left(\frac{L\tau_{2}}{2}\right)},\end{array}\right.$
(22)

where we defined,

$\tau_{1}^{2}(p)=\frac{1}{2}\left(\sqrt{\left(p^{2}+M^{2}\right)^{2}-a^{2}p^{2}}+p^{2}+M^{2}-\frac{a^{2}}{2}\right),$
(23)
$\tau_{2}^{2}(p)=\frac{1}{2}\left(\sqrt{\left(p^{2}+M^{2}\right)^{2}-a^{2}p^{2}}-\left(p^{2}+M^{2}-\frac{a^{2}}{2}\right)\right).$
(24)

We note that,

$\tau_{1}(0)=\sqrt{4M^{2}-a^{2}}/2,~{}~{}~{}~{}~{}~{}~{}~{}\tau_{2}(0)=ia/2,$
(25)

and thus we obtain the complex mass scales $M_{\pm}=\tau_{1}(0)\pm\tau_{2}(0)$
that appear in the infinite vortex solution in the main text.

Up to linear order in $a$,

$\displaystyle B_{\varphi}$ $\displaystyle=$
$\displaystyle-\frac{aM^{2}\Phi_{0}}{2\pi}\int
dp\frac{J_{1}(pr)\left[p^{2}\beta_{0}(p,z)-\frac{\partial^{2}\beta_{0}(p,z)}{\partial
z^{2}}\right]}{\left(p^{2}+M^{2}\right)^{2}}$ (26) $\displaystyle B_{r}$
$\displaystyle=$ $\displaystyle-\frac{M^{2}\Phi_{0}}{2\pi}\int
dp\frac{J_{1}(pr)}{\left(p^{2}+M^{2}\right)}\frac{\partial\beta_{0}(p,z)}{\partial
z}$ (27) $\displaystyle B_{z}$ $\displaystyle=$
$\displaystyle\frac{M^{2}\Phi_{0}}{2\pi}\int
dp\frac{pJ_{0}(pr)}{\left(p^{2}+M^{2}\right)}\beta_{0}(p,z),$ (28)

where the notation $\beta_{0}(p,z)$ refers to the function $\beta(p,z)$ with
$a=0$,

$\beta_{0}(p,z)=\left\\{\begin{array}[]{l}\frac{\tau
e^{p\left(\frac{L}{2}-z\right)}}{\tau+p\operatorname{coth}\left(\frac{\tau
L}{2}\right)},\quad z>\frac{L}{2}\\\ 1-\frac{p\cosh\left(\tau
z\right)}{p\cosh\left(\frac{\tau L}{2}\right)+\tau\sinh\left(\frac{\tau
L}{2}\right)},\quad-\frac{L}{2}<z<\frac{L}{2}\\\ \frac{\tau
e^{p\left(\frac{L}{2}+z\right)}}{\tau+p\operatorname{coth}\left(\frac{\tau
L}{2}\right)},\quad z<-\frac{L}{2},\end{array}\right.$ (29)

$\tau=\sqrt{p^{2}+M^{2}}$.


longtable

# Reproducibility of COVID-19 pre-prints††thanks: Preprint: This present
manuscript is a version of a preprint by the authors (Collins and Alexander
2021). The text, tables and figures of this manuscript overlap with those of
the preprint version by the authors. Acknowledgements: Two anonymous reviewers
and the editor provided extensive comments that substantially improved this
paper and we are grateful to them. We thank Amy Farrow, Jessica Gronsbell,
Monica Alexander, and Thomas William Rosenthal for helpful comments. Funding
statement: Collins thanks CANSSI Ontario and Monica Alexander for financial
support. Data accessibility statement: Code and data are available at:
https://github.com/anniecollins/reproducibility_markers_in_covid19_preprints.
Author contributions: Collins had the original idea and wrote the first draft
of the paper. Collins and Alexander obtained and created the datasets. Collins
and Alexander analyzed the data. All authors interpreted the data and
contributed writing the paper and approved the final version.

Annie Collins111University of Toronto. Rohan Alexander222University of
Toronto<EMAIL_ADDRESS>
(13 January 2022)

###### Abstract

To examine the reproducibility of COVID-19 research, we create a dataset of
pre-prints posted to arXiv, bioRxiv, and medRxiv between 28 January 2020 and
30 June 2021 that are related to COVID-19. We extract the text from these pre-
prints and parse them looking for keyword markers signaling the availability
of the data and code underpinning the pre-print. For the pre-prints that are
in our sample, we are unable to find markers of either open data or open code
for 75 per cent of those on arXiv, 67 per cent of those on bioRxiv, and 79 per
cent of those on medRxiv.

##  1 Introduction

Scientists use open repositories of papers to disseminate their research more
quickly than is possible in traditional journals or conference proceedings,
and to obtain feedback on their work prior to publication. These repositories,
such as arXiv, bioRxiv, and medRxiv, are a critical component of scientific
communication and a lot of research builds on the pre-prints posted there.
Pre-print repositories have been especially important during the 2019 novel
coronavirus (COVID-19) pandemic and the changes it has imposed on the
scientific community (Else 2020). The centrality of pre-prints to science
means that it is important that the results that are posted are credible.
These repositories are not peer-reviewed, and, in general, anyone with
appropriate academic credentials can submit a pre-print.

Neither peer-review nor credentials are a panacea nor a guarantee of quality.
And the gate-keeping and slow publication times of traditional journals mean
pre-print repositories are important. But it is important that scientists
impose standards on themselves, and arguably repositories have a role to play
here. Following Weissgerber et al. (2021), we examine pre-prints about
COVID-19 posted to arXiv, bioRxiv, and medRxiv from 28 January 2020 through to
30 June 2021. By way of background, each of these three repositories has a
different focus: arXiv is general although it has especially high rates of
usage from fields like mathematics, physics, and computer science, bioRxiv
focuses on biological sciences, and medRxiv focuses on health sciences.

We search for markers of open science as indicators of reproducibility,
specifically open data and open code. The definition of reproducibility tends
to vary by context and academic field (Barba 2018). For the purposes of this
paper, we define reproducibility to mean the ability for different researchers
to achieve the same results given the same data and computational methods as
the original source. This contrasts with replicability, which we define as the
ability for different researchers to achieve consistent results by conducting
the full data collection and analysis process in lieu of reusing original
data. These definitions match that of National Academies of Sciences and
Medicine (2019) and Cacioppo et al. (2015). What constitutes open code or open
data is complicated and discipline specific. The details of the oddpub
approach are available in Riedel, Kip, and Bobrov (2020). The general criteria
are that specific mention should be made of where the data and code are
located, and that data should be as close to raw as possible. Data and code
must also be freely accessible to anyone (no request, application,
registration process, or affiliation required).

We find that of the papers sampled, approximately 75 per cent of papers from
arXiv, 67 per cent of papers from bioRxiv, and 79 per cent of papers from
medRxiv contain neither open data nor open code markers. A summary of our main
results is contained in Figure 1. Examining trends over time, we find that the
proportion of pre-prints containing open data or code markers has fluctuated
but shown no obvious trend throughout the pandemic. We also find that the
presence of open data or open code markers seems to have little association
with a pre-print’s subsequent publication, and the subset of sampled pre-
prints that have been published contains approximately the same proportion of
papers with these markers.

Figure 1: Summary of process and main results

The remainder of this paper is structured as follows: in Section 2 we discuss
the process of constructing our dataset through retrieving pre-prints from the
arXiv, bioRxiv, and medRxiv repositories and mining them for open data and
open code markers. In Section 3, we present the results and key findings of
this process. Finally, in Section 4 we discuss the implications of these
findings in the broader context of reproducibility and science during the
COVID-19 pandemic, as well as next steps to expand on our findings and
questions raised in the research process.

##  2 Methodology

###  2.1 Pre-print metadata

Our primary dataset consists of pre-print metadata extracted from the arXiv,
bioRxiv, and medRxiv pre-print repositories via their respective Application
Programming Interfaces (APIs). This metadata varies by repository, but
generally includes: title, abstract, author(s), date created, research field,
DOI, version number, corresponding author, corresponding author’s
institutional affiliation, published DOI (if the pre-print has since been
published in a peer-reviewed journal), and download link. The data collection
process was conducted separately for COVID-19 and pre-COVID-19 papers.

For COVID-19-related pre-prints, we first created a local copy of each
repository containing all metadata for pre-prints posted between 1 January,
2020, and 30 June, 2021. We classified individual pre-prints as
“COVID-19-related” based on whether they contained one or more of the
following terms in their title or abstract (case insensitive): “COVID-19,”
“COVID 19,” “corona virus,” “coronavirus,” “coronavirus-2,” “SARS-CoV-2,”
“SARSCoV-2,” and “2019-nCoV.” We then randomly sampled pre-prints for further
analysis.

For pre-COVID-19 pre-prints, we created a local copy of each repository
containing all metadata for pre-prints posted between 1 January, 2019, and
December 31, 2019. Since medRxiv was launched in June 2019, we used all pre-
print data from the latter half of 2019. We then randomly sampled 1,200 pre-
prints from each repository’s dataset for analysis, except for medRxiv for
which only 913 pre-prints were available over this time.

###  2.2 Open data and code detection

We checked our sampled pre-prints for open data and code markers using the
Open Data Detection in Publications (ODDPub) text mining algorithm (Riedel,
Kip, and Bobrov 2020) within the oddpub R package (Riedel 2019)
(RRID:SCR_018385) . This required downloading each pre-print as a PDF and then
converting the PDFs to text files. We then conducted the open data and open
code detection procedure, which involved searching for keywords and other
markers of open data and open code availability. This was conducted using the
open_data_search() function from the oddpub package. In the validation
conducted by the authors of the package, the ODDPub algorithm had a
sensitivity of 0.73 and a specificity of 1.00 for open code detection, and a
sensitivity of 0.73 and a specificity of 0.97 for open data detection compared
with manual screening (Riedel, Kip, and Bobrov 2020). Since the ODDPub
algorithm was developed specifically for biomedical publications, we conducted
our own validation process for its performance on arXiv pre-prints. We found
that the ODDPub algorithm performed with a sensitivity of 0.60 and a
specificity of 0.98 for open code detection, and a sensitivity of 0.67 and a
specificity of 0.98 for open data detection compared with manual screening.
Details of our validation procedure are contained in Appendix A. Our work was
conducted using the statistical programming language R (R Core Team 2020)
(RRID:SCR_001905).

The result of this process is a dataset indicating the presence of open data
or open code markers in each pre-print (with a logical vector for each marker,
followed by the relevant open data or open code statements where applicable).
Our final dataset was formed by joining this output with the original sample
metadata, typically using the DOI or the unique file name, to form a dataset
including all original metadata for each pre-print alongside its open data and
open code status and markers.

##  3 Results

###  3.1 Pre-pandemic pre-prints

To examine the influence of the COVID-19 pandemic on open science practices
during the pandemic, we analyzed pre-prints posted between January and
December 2019 from each of the four repositories in question. Since medRxiv
was founded in June 2019, all pre-prints posted in the latter half of 2019
were analyzed (a total of 913). For all other repositories, a random sample of
1,200 was taken from all non-COVID-19-related pre-prints posted in the
relevant date range.

Figure 2: Number of pre-prints posted per month in 2019 to arXiv (N =
150,018), bioRxiv (N = 31,752), medRxiv (N = 913)

Between June and December 2019, the number of pre-prints posted to medRxiv
monthly saw an overall increase, which may be expected as the repository
gained recognition and popularity in the medical research community (Figure
2). The number of pre-prints posted monthly to bioRxiv also saw a slight
overall increase throughout 2019, while the number of those posted to arXiv
fluctuated throughout the year (Figure 2). Due to its relative immaturity at
the beginning of the COVID-19 pandemic, a significant portion of medRxiv’s
overall usage has been dedicated to COVID-19-related research. In total,
21,647 pre-prints were posted to medRxiv between June 2019 and 30 June, 2021,
13,194 of which (approximately 61 per cent) relate to COVID-19.

Of the analyzed pre-prints from 2019, 93 per cent of those posted to arXiv, 63
per cent of those posted to bioRxiv, and 75 per cent of those posted to
medRxiv showed no indication of open data or open code.

Table 1: Counts and proportion of published COVID-19 pre-prints in each repository, 2019 Repositiory | Published pre-prints | Proportion published
---|---|---
arXiv | 62,045 | 0.41
bioRxiv | 20,445 | 0.64
medRxiv | 555 | 0.61
Table 2: Counts and proportions of open data markers by whether the pre-print was published, 2019 Status | Both | Neither | Open code | Open data | Proportion with neither
---|---|---|---|---|---
arXiv
Unpublished | 11 | 645 | 44 | 3 | 0.92
Published | 6 | 467 | 15 | 9 | 0.94
bioRxiv
Unpublished | 38 | 264 | 34 | 87 | 0.62
Published | 78 | 488 | 64 | 147 | 0.63
medRxiv
Unpublished | 19 | 277 | 19 | 43 | 0.77
Published | 38 | 409 | 36 | 72 | 0.74

Examining publication rates for pre-pandemic papers, we observe that 41 per
cent of pre-prints posted to arXiv, 64 per cent of pre-prints posted to
bioRxiv, and 61 per cent of pre-prints posted to medRxiv during 2019 were
eventually peer reviewed and published (Table 1). When disaggregated by open
data and code status, we find that published and unpublished pre-prints
contain open data and code markers in similar proportions (Table 2).

###  3.2 All pre-prints related to COVID-19

The number of pre-prints posted per month increased in the first half of 2020
across all repositories, reaching a maximum sometime between April and June
(depending on repository) and subsequently decreasing. The number of pre-
prints posted monthly since August 2020 has remained reasonably steady, with
the exception of medRxiv, which experienced an increase to nearly 1,000 pre-
prints posted in March 2021 (Figure 3). For context, COVID-19 was declared a
pandemic by the World Health Organization (WHO) on March 11, 2020, at which
point the number of cases globally had just surpassed 118,000 (primarily in
east Asia) and the virus had been reported in 114 countries (World Health
Organization 2020).

Figure 3: Number of pre-prints related to COVID-19 posted to arXiv (N =
4,250), bioRxiv (N = 4,146), and medRxiv (N = 13,194) per month

###  3.3 Open data and code

From the collection of all pre-prints related to COVID-19, we randomly sampled
3,500 pre-prints to analyze, stratified by repository. This sample is broken
down as follows: 1,500 from medRxiv, 1,000 from arXiv, and 1,000 from bioRxiv.
Broadly, we are unable to find markers of either open data or open code for
2,606 pre-prints or approximately 74 per cent of our sample (Appendix B Table
10). Of the remaining pre-prints, 7 per cent contained open code markers only,
10 per cent contained open data markers only, and 8 per cent included markers
of both open data and open code.

When differentiated by repository, we observe that open data and code markers
were absent from 75 per cent of the sampled arXiv pre-prints, 67 per cent of
the sampled bioRxiv pre-prints, 79 per cent of the sampled medRxiv pre-prints.
The distribution of the remaining portion of pre-prints also varies by
repository (Appendix B Table 10). Notably, 28 per cent of sampled pre-prints
from bioRxiv contained open data markers and 22 per cent of sampled arXiv pre-
prints contained markers of open code, the highest proportions of any
repository for each type of marker. Our results are similar to McGuinness and
Sheppard (2021), who focus on medRxiv and find that 23 per cent describe open
data.

The distribution of total sampled pre-prints and sampled pre-prints with open
data or code markers roughly follows that of COVID-19-related pre-prints
posted in general (Figure 4). The proportion of pre-prints with open data or
code has fluctuated over time but shows no consistent overall increase or
decrease throughout the course of the pandemic, nor in conjunction with
increases or decreases in the total number of pre-prints posted to any given
repository. In our datasets, very few (if any) pre-prints were sampled for the
month of January 2020. None of these pre-prints contained open data or open
code markers, thus the 0 per cent rate of open data and code for this month
across all repositories should be considered an outlier.

It is also important to note that pre-prints posted during the early months of
the pandemic were likely using, and reusing, publicly available data sources
due to an inability to collect original data within a short timeframe.
Additionally, oddpub does not consider ‘[t]he reuse of data/code previously
published by other researchers’ (Riedel, Kip, and Bobrov 2020). A different
definition of open data could enable pre-prints that reuse publicly available
data to be considered as having their data available for reproducibility
purposes.

The proportion of bioRxiv and medRxiv pre-prints lacking both open data and
open code are approximately four per cent higher than the corresponding
proportions of 2019 pre-prints, suggesting that the analyzed pre-prints from
2019 may contain an overall higher prevalence of open data and code markers
than pre-prints concerning COVID-19. Specifically, we found that open data
availability in medRxiv pre-prints was significantly associated with a pre-
pandemic registration date ($\chi$2 = 4.8508, p < 0.005), as was open code
availability for bioRxiv pre-prints ($\chi$2 = 14.491, p < 0.005). This would
suggest that open data and code practices may have suffered in the context of
COVID-19, or that it may be something that is backfilled after posting.

On the other hand, the analyzed arXiv COVID-19-related pre-prints contain a
higher proportion of open data and code markers overall than their 2019
counterparts, with an increase of 18 per cent in sampled arXiv papers. For
these repositories, the presence of both open data and open code markers was
significantly associated with registration during the pandemic, suggesting
that pre-prints related to COVID-19 in arXiv may have more consistently
adhered to open science practices than their pre-pandemic counterparts
($\chi$2 = 93.124, arXiv open data; $\chi$2 = 106.88, arXiv open code; $\chi$2
= 12.303).

Figure 4: Number of sampled pre-prints posted in 2019, compared with those
related to COVID-19: from arXiv (n in 2019 = 1,000; n related to COVID =
1,200), bioRxiv (n in 2019 = 1,000; n related to COVID n = 1,200), and medRxiv
(n in 2019 = 1,500; n related to COVID n = 913) distinguished by presence of
open data or code markers

###  3.4 Publication status

The proportion of pre-prints that have been published varies by repository
(Table 3). Notably, of all COVID-19-related pre-prints in our dataset,
approximately 30 per cent of those posted to bioRxiv and nearly one-third of
those posted to medRxiv were published. This is high in comparison to the
proportion from arXiv, and although this might suggest that COVID-19-related
pre-prints in biomedical fields have received greater attention overall than
pre-prints from other fields, our results in Section 3.1 suggest that this
pattern pre-dates the pandemic.

Table 3: Counts and proportion of published COVID-19 pre-prints in each repository Repository | Published pre-prints | Proportion published
---|---|---
arXiv | 717 | 0.17
bioRxiv | 1,230 | 0.30
medRxiv | 4,230 | 0.32

In Table 4 we disaggregate sampled pre-prints by whether there is an
indication of publication. We find that the proportion of pre-prints with open
data or code markers among those that have been published is roughly the same
as pre-prints that have not been published, differing by only a few percentage
points.

There is limited literature examining the relationship between data and code
availability in manuscripts between the pre-print and publication stages.
McGuinness and Sheppard (2021) examine differences in data availability
statements between medRxiv pre-prints and their published counterparts. They
find that data availability was maintained for most of their sample, varying
by journal data sharing policy with greater improvements in openness among
manuscripts published in journals mandating data sharing. While limited to
medRxiv, the results of McGuinness and Sheppard (2021) align with our own work
and provide initial evidence to suggest that data availability is generally
maintained or improved between the pre-print and publication stages.

Table 4: Counts and proportions of open data markers by whether the pre-print was published Status | Both | Neither | Open code | Open data | Proportion with neither
---|---|---|---|---|---
arXiv
Unpublished | 79 | 622 | 100 | 27 | 0.75
Published | 20 | 130 | 15 | 7 | 0.76
bioRxiv
Unpublished | 55 | 471 | 29 | 145 | 0.67
Published | 20 | 203 | 15 | 62 | 0.68
medRxiv
Unpublished | 10 | 112 | 8 | 8 | 0.81
Published | 113 | 1,068 | 81 | 100 | 0.78

Our dataset likely imperfectly characterizes publication and does not have the
publication details for some papers that were published. And even if it were a
perfect record, there is a publication lag (estimated at an average of around
60 days for COVID-19-related pre-prints, although that varies by discipline)
that may especially skew the results for pre-prints in the latter portion of
our sample (Kwon 2020).

##  4 Discussion

###  4.1 On the role of transparency and reproducibility

Transparency and reproducibility are hallmarks of quality scientific research
due to their relationship with independent verification (Stodden 2020). Open
data and open code contribute to both by allowing the scientific community to
more easily verify the authenticity of purported scientific discoveries and
their supporting evidence. Data sharing also allows others to reuse other
researchers’ data sets for further analysis or to supplement their own data,
contributing to new insights within their field of study.

These factors are especially important in cases where scientific research may
quickly and directly impact clinical practice or public policy, such as
research on the COVID-19 pandemic. Among many other impacts on the research
landscape, COVID-19 has increased the popularity of pre-prints from both a
production and consumption standpoint. The number of COVID-19 pre-prints
posted to medRxiv increased in the early stages of the pandemic, while non-
COVID-19 pre-print numbers were largely as expected. The same trends were
apparent in abstracts accessed by medRxiv users, where COVID-19 pre-print
abstracts were viewed over 15 times more than non-COVID-19 pre-print abstracts
(Fraser et al. 2021). For these reasons, it is important to examine open
science standards and reproducibility within pre-print repositories.

Open data is generally accepted to be beneficial to the scientific process and
to a paper’s reproducibility potential, hence it is concerning that around 75
per cent of pre-prints in our sample contained no open data markers. This
concern is slightly mitigated by recognition of challenges in working with
biomedical data compared with data in other fields, notably privacy and ethics
concerns when working with personal data (Floca 2014). The COVID-19 pandemic
has seen open science initiatives, as evidenced by the creation of open data
repositories such as the dashboard maintained by the Center for Systems
Science and Engineering at Johns Hopkins University (Dong, Du, and Gardner
2020) or the large number of publishers who removed paywalls from published
COVID-19 research (Gill 2020). While the intention at the start of the
pandemic was that there would be ‘clear statements regarding the availability
of underlying data’ (Wellcome Trust 2020) some retractions of work have been
based on ‘unreliable or nonexistent data’ (Teixeira da Silva, Bornemann-
Cimenti, and Tsigaris 2021).

Open code as an open science marker is context and field-dependent; for
instance, not all biomedical research papers will rely on computational
methods for their analyses. However, in pre-prints where code comprises a
large portion of the methodology or results, posting it openly to repositories
like GitHub contributes to a pre-print’s potential reproducibility. This is
important when computational methods are used to form predictions about
emerging situations with limited data or laboratory research, which was the
case for modelling studies in the early days of the COVID-19 pandemic. We also
see growing concern over the quality and consequences of this sort of
research, with bioRxiv no longer allowing purely computational work (Kwon
2020).

The other concern is the adverse selection issue caused by meeting the open
science aims of sharing code and data. Authors that share their data and code
open their work up to criticism. If authors who make their data and code
available make similar mistakes to authors who choose to not publish their
data and code, it is more likely that the mistake would not be noticed in the
case where data and code were not published. The current system is biased
against those who follow best practice. McGuinness and Sheppard (2021)
advocate for ‘(s)trict editorial policies that mandate data sharing,’ and
other changed norms are needed.

###  4.2 The role of pre-print repositories

There has been a large amount of research on COVID-19 (Teixeira da Silva,
Tsigaris, and Erfanmanesh 2021). Many concerns have arisen from the rate at
which COVID-19 research has been posted and consumed through pre-print
repositories, particularly in the early stages of the pandemic (Raynaud et al.
2020). Rushed scientific research has the potential to skip (or at least place
less importance on) open science practices, so it may be reasonable to expect
a decrease in open data or code markers in the initial few months of the
pandemic. We found little relationship between date posted and likelihood of
having open data or code markers, with the proportion of pre-prints containing
these markers fluctuating from month to month. This suggests that open science
practices are more influenced by other factors, perhaps including training,
publication bias, or the nature of the pre-print itself. On the other hand, we
do not see an overall long-term increase in either open data or open code
markers throughout our period of analysis, which we may have expected in the
context of the open science movements the pandemic has fostered. Although not
pre-print specific, Else (2020) found that overall research output has
fluctuated between different fields and topics (namely modelling disease
spread, public health, diagnostics and testing, mental health, and hospital
mortality) throughout different stages of the pandemic, which may account for
some of the fluctuation and overall lack of noticeable trend in our sample.

To emphasize the ongoing need for open data and code in modelling a pandemic,
we consider two high profile epidemiological models that emerged in early
2020. Modelling was conducted by Imperial College London (ICL) (Ferguson et
al. 2020) and the Institute for Health Metrics and Evaluation (IHME) at the
University of Washington (Murray 2020), and both were initially posted to pre-
print repositories. The ICL model went on to become the most cited pre-print
as of December 2020 (Else 2020), and both had significant influence over
policy and public health decisions worldwide (Adam 2020). An independent
review of these two models by Jin et al. (2020) found that while code and data
were openly available for both, only the ICL model was reproducible due to
limited transparency on the underlying methodology of the IHME model. The
open-source nature of these models was fundamental to reproduction attempts
and is an example of the need for open data and code in COVID-19 research,
particularly as pre-prints influence public decision-making.

In the context of the above factors, it was encouraging to find in our
analysis that the proportion of pre-prints with open data or code posted to
arXiv increased from 7 per cent pre-pandemic to 25 per cent for
COVID-19-related pre-prints. This pattern, however, was not observed among the
analyzed bioRxiv and medRxiv pre-prints, and may just reflect the nature of
COVID-19 pre-prints. With many pre-prints from these repositories still
pertaining to epidemiological modelling, one might hope that they should
universally be subject to the same analysis as conducted by Jin et al. (2020)
as for the examples above, which is made possible by the availability of
relevant code and data. Our analysis suggests a need for future investigation
and potential overall improvement in open science standards for these types of
pre-prints (subject to the data and code considerations already discussed).
This need is again emphasized by the new-found speed at which pre-prints may
gain public, media, and political attention in the context of the pandemic,
particularly those from medRxiv and bioRxiv. One further concern is raised by
Teixeira da Silva (2020), who shows that there are pre-prints on those two
pre-print servers—medRxiv and bioRxiv—that were withdrawn or retracted with
relatively little information about the underlying reason, after gaining
substantial media attention.

###  4.3 The importance of open data and open code

Beyond pre-prints, COVID-19 has influenced publication and peer review
processes, with timelines for COVID-19 papers being expedited at the expense
of longer waits for other scientific research (Else 2020). It is important
that open data and code standards be maintained in published work as well. In
our sample, published pre-prints contain open data or code markers in similar
proportions to their unpublished counterparts, a pattern that was present for
pre-prints related to COVID-19 and those posted in 2019. This appears
initially to alleviate some concerns over the relationship between open data
and publication bias, that is, the potential that journals have favored novel
yet less transparent or reproducible papers over those with null results but a
high standard of open science practices. However, publication bias is complex,
and this result should be approached with caution. Concerns have already been
raised through systemic reviews of COVID-19 publications (Raynaud et al.
2020), and oversights in data accessibility have led to high profile
retractions of publications in the past; for example, papers from _The Lancet_
and the _New England Journal of Medicine_ which were withdrawn due to concerns
over the private nature of their underlying dataset (Ledford and Noorden
2020). Cabanac, Oikonomidi, and Boutron (2021) show that not all pre-prints
are linked to their subsequent peer-reviewed publication, which may further
bias our results. Additionally, there is the potential for bias due to older
pre-prints having had more time to be published than newer pre-prints. And
Oikonomidi et al. (2020) and Bero et al. (2021) show that differences between
updated versions of the same pre-print can be substantial; again, this is
something that we do not account for and could bias our results.

In all fields of science, increasing access to data and code used for pre-
printed or published research is a step in the direction of more transparent,
reproducible, and reliable research. The COVID-19 pandemic has created a
novel, constantly changing scientific culture that should be navigated with
care to uphold standards of scientific practice for both the research
community and the safety of the public. Our analysis shows that there is room
for improvement in the areas of open data and code availability within
COVID-19 pre-print papers on arXiv, bioRxiv, and medRxiv

There is demand for timely research and high frequency results because the
pandemic rapidly evolves. Pre-prints are efficient in this role because there
is no time spent on peer review. They also allow lesser-known researchers to
better disperse their research because of the possibility that fast-tracked
peer review may be biased towards established researchers. While there is a
clear need for pre-prints, the point remains that they do not go through the
peer review process. This question of quality and validity is particularly
pertinent in the COVID-19 context because poorly validated results and false
information may spread quickly and have real effects. We are not saying that
peer review implies that a paper is of a high-quality; we are instead saying
that the provision of code and data alongside the pre-print goes some way to
allowing others to trust the findings of pre-prints, even though they have not
been peer-reviewed. One way this could be encouraged would be for all pre-
print repositories to have authors characterize the extent to which they have
adopted open science practices as part of their submission, in the same way
that is done in SocArXiv. Although those pre-prints that do not adopt these
practices should not be rejected from pre-print repositories, greater clarity
around this would be useful and might move the state-of-the-art forward.

###  4.4 Weaknesses and next steps

Future work would expand our analysis to consider the geographic distribution
of research and the potential influence of different practices and policies
concerning open science. This is important because the epicenter of the
pandemic changed throughout the pandemic, which may have implications for our
time-based analysis.

A logical next step would be to extend this analysis to additional pre-print
servers. We have begun considering samples of pre-pandemic and
COVID-19-related pre-prints posted to SocArXiv, a social sciences pre-print
server hosted by the Center for Open Science. We validated the ODDPub
algorithm against the presence of data links provided by pre-print authors
upon submission (available in the pre-print metadata drawn from the Open
Science Framework API) and found that the algorithm performed with 52 per cent
sensitivity on the 2019 sample and 29 per cent sensitivity for
COVID-19-related pre-prints. The high rate of false negatives for open data
detection is concerning, and it was decided that the ODDPub algorithm is not
suitable for use on pre-prints from this server without modification. A more
generalized (or perhaps field-specific) algorithm would be necessary for
analysis of open data and code availability in SocArXiv and other more
specialized servers. Details of this validation are available in Appendix C.

We recognize that factors beyond open data and code play a large role in the
reproducibility of scientific research. Not all pre-prints providing open data
or code will be reproducible. Factors such as data documentation,
methodological reporting, software choice, and many others all play a role in
the reproduction process and should be regarded with just as much gravity when
disseminating results.

An important weakness is the potential presence of false negatives in
indicators of publication in our dataset. Abdill and Blekhman (2019) estimate
that the false-negative rate may be as high as 37.5 per cent for data pulled
from the bioRxiv API, meaning analysis of published papers may represent only
a fraction of those that have been published. It is unclear to what extent
this is the case for other repositories or what bias may exist in the subset
of pre-prints for which publication was detected, because it is likely that
this process relies on title-based text matching (Abdill and Blekhman 2019).
It is also likely that some of our more recent sampled pre-prints will be
published in the future which we could not account for at the time of our data
collection.

Our paper depends on search responses from the various repositories, which are
based on our selection of keywords. Our selection of keywords is not
exhaustive, for instance, perhaps ‘the pandemic’ could result in additional
papers. Future work could make this keyword approach more systematic, for
instance following King, Lam, and Roberts (2017).

We also recognize that this analysis relies heavily on text-based analysis
which was not verified directly in most cases and may lead to higher levels of
uncertainty. The oddpub package was built to analyze biomedical publications
and it may be that some of the differences that we find between repositories
are due to this. We also note that the ODDPub algorithm is relatively narrow
in its definition of “open,” excluding data that is available via registration
or in some other restricted form. Considering a broader definition of
openness, either through using a less restrictive algorithm or through manual
verification, would likely produce different results particularly for pre-
prints using clinical data. Future work could take smaller sub-samples to
validate factors like publication status, paper topic, and open code and data
status, beyond the approaches we used here.

##  Appendix A ODDPub algorithm performance on arXiv pre-prints

We verified the accuracy of the ODDPub algorithm on a subset of our analyzed
pre-prints from 2019 from arXiv. We took a simple random sample of 100 papers.
In the original validation process, the annotators stratified by detection
status prior to sampling to ensure relatively high representation of papers
where open data or code was detected. Since the major concern for our manual
verification is potential false negatives, this skewed representation was
unnecessary. Open data and code status were verified first via the “Code &
Data” tab on each pre-print’s page on the arXiv website, then by checking for
an explicit data availability section within the pre-print PDF, and finally by
manually checking the body of the paper using keyword searches. Results were
recorded manually in Excel. This mimics the procedure outlined for the
original validation of ODDPub (Riedel, Kip, and Bobrov 2020).

Many of the pre-prints in arXiv did not use data or code, namely those from
pure mathematics and physics. There were also several that reused other
publicly or privately available data sets, and regardless of whether or not
they were shared alongside the paper, these do not count as open data
according to the standards outlined by the original authors of the ODDPub
algorithm (Riedel, Kip, and Bobrov 2020). Algorithmic performance is specified
in Table 6 and Table 8.

Table 5: ODDPub predictions for open data compared with manual check, arXiv sample Predicted | Data available | No data available
---|---|---
Open data detected | 2 | 2
No open data detected | 1 | 95
Table 6: ODDPub prediction accuracy, open data, arXiv sample Metric | Value
---|---
Accuracy | 0.97
Sensitivity | 0.67
Specificity | 0.98
Table 7: ODDPub predictions for open code compared with manual check, arXiv sample Predicted | Code available | No code available
---|---|---
Open code detected | 3 | 2
No open code detected | 2 | 93
Table 8: ODDPub prediction accuracy, open code, arXiv sample Metric | Value
---|---
Accuracy | 0.96
Sensitivity | 0.60
Specificity | 0.98

##  Appendix B Supporting tables

Table 9: Count and proportions of open data and code markers by pre-print repository in 2019 sample Markers | Count | Proportion of total
---|---|---
Total
Neither | 2,550 | 0.77
Open code | 212 | 0.06
Open data | 361 | 0.11
Both | 190 | 0.06
arXiv
Neither | 1,112 | 0.93
Open code | 59 | 0.05
Open data | 12 | 0.01
Both | 17 | 0.01
bioRxiv
Neither | 752 | 0.63
Open code | 98 | 0.08
Open data | 234 | 0.20
Both | 116 | 0.10
medRxiv
Neither | 686 | 0.75
Open code | 55 | 0.06
Open data | 115 | 0.13
Both | 57 | 0.06
Table 10: Count and proportions of open data and code markers by pre-print repository Markers | Count | Proportion of total
---|---|---
Total
Neither | 2,606 | 0.74
Open code | 248 | 0.07
Open data | 349 | 0.10
Both | 297 | 0.08
arXiv
Neither | 752 | 0.75
Open code | 115 | 0.12
Open data | 34 | 0.03
Both | 99 | 0.10
bioRxiv
Neither | 674 | 0.67
Open code | 44 | 0.04
Open data | 207 | 0.21
Both | 75 | 0.07
medRxiv
Neither | 1,180 | 0.79
Open code | 89 | 0.06
Open data | 108 | 0.07
Both | 123 | 0.08
Table 11: Number and proportion of all preprints in each category, for each preprint server Markers | Count | Proportion of total
---|---|---
arXiv
Both | 99 | 0.10
Neither | 752 | 0.75
Open Code | 115 | 0.12
Open Data | 34 | 0.03
bioRxiv
Both | 75 | 0.07
Neither | 674 | 0.67
Open Code | 44 | 0.04
Open Data | 207 | 0.21
medRxiv
Both | 123 | 0.08
Neither | 1,180 | 0.79
Open Code | 89 | 0.06
Open Data | 108 | 0.07

##  Appendix C ODDPub algorithm performance on SocArXiv pre-prints

SocArXiv allows authors to input a link to their data source/repository upon
submission of a pre-print. This link can then be accessed via the API
metadata. The presence of a data link was used as an indicator that a pre-
print provides open data for the purposes of validating the ODDPub algorithm.
When available, a data link is stored under the variable name
“attributes.data_links.” The data was manipulated using functions from the R
package tidyverse (Wickham et al. 2019) to create a binary variable indicating
data availability or lack thereof. We assume “attributes.data_links” to
indicate the true availability of data for the purposes of validating the
ODDPub algorithm. It is possible, however, that some authors failed to
indicate their data availability in the proper field upon posting to SocArXiv,
and thus some of the false positive may in fact be true positives.

Against the data availability indicated by pre-print authors in our 2019
sample, the ODDPub algorithm performed with an accuracy of 93 percent, a
sensitivity of 52 percent, and a specificity of 94 percent. In our 2020 and
2021 sample, the algorithm performed with an accuracy of 79 percent, a
sensitivity of 29 percent, and a specificity of 92 percent. Specific
predictions are broken down in Table 12 and Table 14.

It is unclear the precise inclusion criteria for data submitted to the data
link field. It is possible that some of the links provided lead to data sets
that are publicly available for reuse, which would not constitute “open data”
by the ODDPub algorithm’s definition, in which case the accuracy could
potentially be higher in reality than 93 percent and 79 percent in the samples
considered.

Table 12: ODDPub predictions for open data compared with data links provided by authors, 2019 SocArXiv sample ODDPub algorithm | Data linked | No data linked
---|---|---
Open data detected | 11 | 72
No open data detected | 10 | 1107
Table 13: ODDPub prediction accuracy, open data, 2019 SocArXiv sample Metric | Value
---|---
Accuracy | 0.93
Sensitivity | 0.52
Specificity | 0.94
Table 14: ODDPub predictions for open data compared with data links provided by authors, COVID-19-related SocArXiv pre-prints sample ODDPub algorithm | Data linked | No data linked
---|---|---
Open data detected | 28 | 31
No open data detected | 69 | 350
Table 15: ODDPub prediction accuracy, open data, COVID-19-related SocArXiv pre-prints sample Metric | Value
---|---
Accuracy | 0.79
Sensitivity | 0.29
Specificity | 0.92

## References

Abdill, Richard J, and Ran Blekhman. 2019. “Meta-Research: Tracking the
Popularity and Outcomes of All bioRxiv Preprints.” Edited by Emma Pewsey,
Peter Rodgers, and Casey S Greene. _eLife_ 8 (April): e45133.
https://doi.org/10.7554/eLife.45133.

Adam, David. 2020. “Special Report: The Simulations Driving the World’s
Response to COVID-19.” _Nature_ 580: 316–18.
https://doi.org/10.1038/d41586-020-01003-6.

Barba, Lorena A. 2018. “Terminologies for Reproducible Research.”
http://arxiv.org/abs/1802.03311.

Bero, Lisa, Rosa Lawrence, Louis Leslie, Kellia Chiu, Sally McDonald, Matthew
J Page, Quinn Grundy, et al. 2021. “Cross-Sectional Study of Preprints and
Final Journal Publications from COVID-19 Studies: Discrepancies in Results
Reporting and Spin in Interpretation.” _BMJ Open_ 11 (7).
https://doi.org/10.1136/bmjopen-2021-051821.

Cabanac, Guillaume, Theodora Oikonomidi, and Isabelle Boutron. 2021. “Day-to-
Day Discovery of Preprint–Publication Links.” _Scientometrics_ 126 (6):
5285–5304.

Cacioppo, John T, Robert M Kaplan, Jon A Krosnick, James L Olds, and Heather
Dean. 2015. “Social, Behavioral, and Economic Sciences Perspectives on Robust
and Reliable Science.” _Report of the Subcommittee on Replicability in Science
Advisory Committee to the National Science Foundation Directorate for Social,
Behavioral, and Economic Sciences_.

Collins, Annie, and Rohan Alexander. 2021. “Reproducibility of COVID-19 Pre-
Prints.” http://arxiv.org/abs/2107.10724.

Dong, Ensheng, Hongru Du, and Lauren Gardner. 2020. “An Interactive Web-Based
Dashboard to Track COVID-19 in Real Time.” _The Lancet Infectious Diseases_
20: 533–34. https://doi.org/10.1016/S1473-3099(20)30120-1.

Else, Holly. 2020. “How a Torrent of COVID Science Changed Research Publishing
— in Seven Charts.” _Nature_ 588: 553.
https://doi.org/10.1038/d41586-020-03564-y.

Ferguson, Neil M, Daniel Laydon, Gemma Nedjati-Gilani, Natsuko Imai, Kylie
Ainslie, Marc Baguelin, Sangeeta Bhatia, et al. 2020. “Report 9: Impact of
Non-Pharmaceutical Interventions (NPIs) to Reduce COVID-19 Mortality and
Healthcare Demand.” https://doi.org/10.25561/77482.

Floca, Ralf. 2014. “Challenges of Open Data in Medical Research.” In _Opening
Science: The Evolving Guide on How the Internet Is Changing Research,
Collaboration and Scholarly Publishing_ , edited by Sönke Bartling and Sascha
Friesike, 297–307. Cham: Springer International Publishing.
https://doi.org/10.1007/978-3-319-00026-8_22.

Fraser, Nicholas, Liam Brierley, Gautam Dey, Jessica K Polka, Máté Pálfy,
Federico Nanni, and Jonathon Alexis Coates. 2021. “Preprinting the COVID-19
Pandemic.” _bioRxiv_. https://doi.org/10.1101/2020.05.22.111294.

Gill, David. 2020. “Immediate Free Access to Research: The Scholarly Response
to COVID-19.” https://www.lib.sfu.ca/help/publish/scholarly-
publishing/radical-access/scholarly-covid19.

Jin, Jin, Neha Agarwala, Prosenjit Kundu, Yi Wang, Ruzhang Zhao, and Nilanjan
Chatterjee. 2020. “Transparency, Reproducibility, and Validation of COVID-19
Projection Models.” https://www.jhsph.edu/covid-19/articles/transparency-
reproducibility-and-validation-of-covid-19-projection-models.html.

King, Gary, Patrick Lam, and Margaret E Roberts. 2017. “Computer-Assisted
Keyword and Document Set Discovery from Unstructured Text.” _American Journal
of Political Science_ 61 (4): 971–88.

Kwon, Diana. 2020. “How Swamped Preprint Servers Are Blocking Bad Coronavirus
Research.” _Nature_ 580 (May): 130–31.
https://doi.org/10.1038/d41586-020-01394-6.

Ledford, Heidi, and Richard Van Noorden. 2020. “High-Profile Coronavirus
Retractions Raise Concerns about Data Oversight.” _Nature_ 582: 160.
https://doi.org/10.1038/d41586-020-01695-w.

McGuinness, Luke A, and Athena L Sheppard. 2021. “A Descriptive Analysis of
the Data Availability Statements Accompanying medRxiv Preprints and a
Comparison with Their Published Counterparts.” _PloS One_ 16 (5): e0250887.

Murray, Christopher JL. 2020. “Forecasting the Impact of the First Wave of the
COVID-19 Pandemic on Hospital Demand and Deaths for the USA and European
Economic Area Countries.” _medRxiv_.
https://doi.org/10.1101/2020.04.21.20074732.

National Academies of Sciences, Engineering, and Medicine. 2019.
_Reproducibility and Replicability in Science_. Washington, DC: The National
Academies Press. https://doi.org/10.17226/25303.

Oikonomidi, Theodora, Isabelle Boutron, Olivier Pierre, Guillaume Cabanac, and
Philippe Ravaud. 2020. “Changes in Evidence for Studies Assessing
Interventions for COVID-19 Reported in Preprints: Meta-Research Study.” _BMC
Medicine_ 18 (1): 1–10.

R Core Team. 2020. _R: A Language and Environment for Statistical Computing_.
Vienna, Austria: R Foundation for Statistical Computing.
https://www.R-project.org/.

Raynaud, Marc, Huanxi Zhang, Kevin Louis, Valentin Goutaudier, Jiali Wang,
Quentin Dubourg, Yongcheng Wei, et al. 2020. “COVID-19-Related Medical
Research: A Meta-Research and Critical Appraisal.” _BMC Medical Research
Methodology_ 21\. https://doi.org/10.1186/s12874-020-01190-w.

Riedel, Nico. 2019. _Oddpub: Detection of Open Data & Open Code Statements in
Biomedical Publications_. https://github.com/quest-bih/oddpub.

Riedel, Nico, Miriam Kip, and Evgeny Bobrov. 2020. “ODDPub – a Text-Mining
Algorithm to Detect Data Sharing in Biomedical Publications.” _Data Science
Journal_ 19 (1): 42. https://doi.org/10.5334/dsj-2020-042.

Stodden, Victoria. 2020. “Theme Editor’s Introduction to Reproducibility and
Replicability in Science.” _Harvard Data Science Review_ 2 (4).
https://doi.org/10.1162/99608f92.c46a02d4.

Teixeira da Silva, Jaime A. 2020. “Silently Withdrawn or Retracted Preprints
Related to Covid-19 Are a Scholarly Threat and a Potential Public Health Risk:
Theoretical Arguments and Suggested Recommendations.” _Online Information
Review_.

Teixeira da Silva, Jaime A, Helmar Bornemann-Cimenti, and Panagiotis Tsigaris.
2021. “Optimizing Peer Review to Minimize the Risk of Retracting
COVID-19-Related Literature.” _Medicine, Health Care and Philosophy_ 24 (1):
21–26.

Teixeira da Silva, Jaime A, Panagiotis Tsigaris, and Mohammadamin Erfanmanesh.
2021. “Publishing Volumes in Major Databases Related to Covid-19.”
_Scientometrics_ 126 (1): 831–42.

Weissgerber, Tracey, Nico Riedel, Halil Kilicoglu, Cyril Labbé, Peter Eckmann,
Gerben Ter Riet, Jennifer Byrne, et al. 2021. “Automated Screening of COVID-19
Preprints: Can We Help Authors to Improve Transparency and Reproducibility?”
_Nature Medicine_ 27 (1): 6–7.

Wellcome Trust. 2020. _Sharing Research Data and Findings Relevant to the
Novel Coronavirus (COVID-19) Outbreak_. Vienna, Austria: Wellcome Trust.
https://wellcome.org/coronavirus-covid-19/open-data.

Wickham, Hadley, Mara Averick, Jennifer Bryan, Winston Chang, Lucy D’Agostino
McGowan, Romain François, Garrett Grolemund, et al. 2019. “Welcome to the
tidyverse.” _Journal of Open Source Software_ 4 (43): 1686.
https://doi.org/10.21105/joss.01686.

World Health Organization. 2020. “WHO Director-General’s Opening Remarks at
the Media Briefing on COVID-19 - 11 March 2020.” https://www.who.int/director-
general/speeches/detail/who-director-general-s-opening-remarks-at-the-media-
briefing-on-covid-19---11-march-2020.