# Algorithmic Randomness and Probabilistic Laws

Jeffrey A. Barrett111Department of Logic and Philosophy of Science, University
of California, Irvine, Irvine, CA 92697-5100. Email<EMAIL_ADDRESS>and
Eddy Keming Chen Department of Philosophy, University of California, San
Diego, 9500 Gilman Dr, La Jolla, CA 92093-0119. Email<EMAIL_ADDRESS>
###### Abstract

We consider two ways one might use algorithmic randomness to characterize a
probabilistic law. The first is a _generative $\mbox{chance}^{\star}$ law_.
Such laws involve a nonstandard notion of chance. The second is a
_$\mbox{probabilistic}^{\star}$ constraining law_. Such laws impose relative
frequency and randomness constraints that every physically possible world must
satisfy. While each notion has virtues, we argue that the latter has
advantages over the former. It supports a unified governing account of non-
Humean laws and provides independently motivated solutions to issues in the
Humean best-system account. On both notions, we have a much tighter connection
between probabilistic laws and their corresponding sets of possible worlds.
Certain histories permitted by traditional probabilistic laws are ruled out as
physically impossible. As a result, such laws avoid one variety of empirical
underdetermination, but the approach reveals other varieties of
underdetermination that are typically overlooked.

###### Contents

  1. 1 Introduction
  2. 2 Randomness constraints
  3. 3 Alternative algorithmic notions
  4. 4 $\mbox{Probabilities}^{\star}$, $\mbox{chances}^{\star}$, and laws
  5. 5 Discussion
  6. 6 Conclusion

##  1 Introduction

Probabilistic laws, as they are usually understood, involve a variety of
underdetermination. This is illustrated by a simple example.

Consider repeated tosses of a coin that produce an infinite $\omega$-sequence
of results $\langle r_{1},r_{2},\ldots\rangle$, where $r_{i}$ is the result of
the $i$th coin toss. Each such possible sequence of tosses gives the history
of events of a possible world. Let $\Omega^{L}$ be the set of all such worlds
that accord with a law $L$.

Now consider the probabilistic law $L$:

$L$:

Each element in the $\omega$-sequence of coin tosses $\langle
r_{1},r_{2},\ldots\rangle$ is determined independently and with an unbiased
probability of heads and tails.

One might think of $L$ as descriptive of a fundamentally random process,
something like starting with a sequence of spin-$1/2$ particles each in a
eigenstate of $z$-spin, then measuring their $x$-spins in turn.

As probabilistic laws are typically understood, $\Omega^{L}$ is the set of all
$\omega$-sequences. That is, $L$ does not rule out any world. A world
compatible with $L$ might exhibit _any_ limiting relative frequency or no
limiting relative frequency at all. As a result, even the full history of a
world will fail to determine $L$ in an continuous cardinality of cases.

And since $\Omega^{L}$ is compatible with every probabilistic law with heads
and tails as possible outcomes with positive probability on each toss, even
the full set of worlds compatible with $L$ does nothing to determine $L$ over
any other probabilistic law.

This sort of underdetermination is closely related to a corresponding sort of
empirical coherence.222See Barrett (1996), (1999) and (2020) for a
presentation and discussions of empirical coherence. A physical law is
empirically coherent, in the sense we are interested in here, only if it is
always in principle possible for one to have empirical support for the law if
the law is in fact true.333Throughout this paper unless specified otherwise,
when we say “$L$ is the true law” or “the law $L$ is true,” we mean not just
$L$ is true but also $L$ is the law. This is compatible with the non-Humean
perspective where laws govern and the Humean perspective where laws form the
optimal description of the mosaic.  If a law is empirically incoherent, then
it may be impossible to learn that the law is true with even complete
evidence. The law $L$ is empirically incoherent in this sense as there are
$\omega$-sequences that might occur if $L$ is true that would provide no
empirical evidence whatsoever for accepting $L$. In such worlds one would
never have any empirical support for accepting the correct probabilistic law
even with full evidence. Indeed, since worlds compatible with $L$ might
exhibit any limiting relative frequency, there is a continuous cardinality of
such worlds.

One might get a tighter fit between probabilistic laws and empirical evidence
by appealing to a stronger conception of probability and a correspondingly
stronger variety of probabilistic laws. Consider the law $L^{\star}$:

$L^{\star}$:

The $\omega$-sequence of coin tosses $\langle r_{1},r_{2},\ldots\rangle$ is
_random_ with unbiased relative frequencies of heads and tails.

Here being _random_ or not is a property of the full $\omega$-sequence. It
remains then to say what it might mean for a sequence to be random.

The notions of randomness we will consider here are algorithmic. They are
defined in terms of statistical tests that determine whether a full
$\omega$-sequence exhibits any specifiable pattern. What matters at present is
that each sequence will either pass or fail the test for being random.

$\Omega^{L^{\star}}$$\Omega^{L}$$M^{L}$ Figure 1: $\Omega^{L}$, the set of
worlds compatible with law $L$, is the set of all $\omega$-sequences of coin
toss results. $\Omega^{L^{\star}}$, the set of worlds compatible with law
$L^{\star}$, is a proper subset of $\Omega^{L}$. All members of
$\Omega^{L^{\star}}$ exhibit the random pattern and relative frequencies
stipulated by $L^{\star}$. $M^{L}$, the relative complement of
$\Omega^{L^{\star}}$ in $\Omega^{L}$, is the set of ‘maverick worlds,’ i.e.
those that are usually regarded as compatible with $L$ but lack the random
pattern or relative frequencies.

While $L$ is compatible with all $\omega$-sequences of results, $L^{\star}$ is
not. Let $\Omega^{L^{\star}}$ be the set of all worlds that accord with the
law $L^{\star}$. All worlds in $\Omega^{L^{\star}}$ exhibit the random
unbiased sequences stipulated by $L^{\star}$ and hence, unlike $\Omega^{L}$,
is a proper subset of the set of all possible $\omega$-sequences.
Specifically, $\Omega^{L^{\star}}$ contains no _maverick worlds_ , worlds
where the results exhibit a specifiable pattern or fail to exhibit the right
relative frequencies or fail to exhibit any relative frequencies at all. (See
Figure 1.)

If $L^{\star}$ is true, then _any_ physically possible world fully determines
$L^{\star}$. Further, no special probabilistic background assumptions or
priors regarding what world one inhabits are required for successful
inquiry.444Both the Principle Principle and Cournot’s Principle are sometimes
used for this purpose. See Diaconis and Skyrms (2018, 66–7) for a brief
discussion of the latter. We return to this point in §5. A non-dogmatic
inquirer in any physically possible world might determine the truth of
$L^{\star}$ by simply conditioning on the results of coin tosses. Indeed,
$L^{\star}$ is empirically coherent in the strong sense that, with complete
evidence, one will surely learn it up to an equivalence class of
computationally indistinguishable laws, something we will discuss later. A
probabilistic law like $L^{\star}$ is much like a deterministic law in that
the law is fully determined by the evidence.

In the present paper, we consider how one might understand a $\star$-law like
$L^{\star}$ as a _generative $\mbox{chance}^{\star}$ law_ or as a
_$\mbox{probabilistic}^{\star}$ constraining law_. We will then discuss the
costs and benefits of such an approach. While the two notions are closely
related, we will argue that thinking of $L^{\star}$ as a
$\mbox{probabilistic}^{\star}$ constraining law that governs the full sequence
of coin-tosses has a number of salient virtues. They are relevant to
discussions about the metaphysics of laws. As we explain in §4, the notion of
a $\mbox{probabilistic}^{\star}$ constraining law removes a major obstacle for
developing a unified non-Humean account of governing laws, according to which
laws govern by constraining physical possibilities. Such a notion also
provides independently motivated solutions to the issues of the Big Bad Bug
and the definition of _fit_ in the Humean best-system account of laws.

##  2 Randomness constraints

In order to characterize a $\star$-law, one needs a test of randomness for
$\omega$-sequences. A random sequence of tosses with an unbiased coin should
exhibit an even relative frequency of heads and tails in the limit. But this,
of course, is not sufficient. The limiting relative frequency of an
alternating sequence of heads and tails will be $1/2$ for heads and tails, but
this sequence is clearly not random.

There are three further conditions that one should want an unbiased random
sequence to satisfy: a random sequence should be generic, patternless, and not
allow for the success of a fair betting strategy.555See Li and Paul Vitányi
(2008) and Dasgupta (2011) for introductions to algorithmic complexity and
randomness. See also Barrett and Huttegger (2021) for a discussion of these
notions and how they relate to each other. The present section follows part of
that discussion. See Eagle (2021) for an introduction to some of the
philosophical issues involving randomness. These three conditions are closely
related. The core idea is that an $\omega$-sequence should count as random
only if it exhibits no finitely specifiable regularity that characterizes the
sequence and might consequently be used to make predictions better than
chance.

Algorithmic tests are helpful in characterizing what it might mean for an
infinite sequence to be patternless. As a first try, one might take an
$\omega$-sequence to be patternless, and hence random, if and only if there is
no finite-length algorithm that produces the sequence.666One might think of an
algorithm as a program in a Turing-computable language and the length of the
algorithm as the length of the program. Different languages will differ in the
length they assign to an abstract algorithm by no more than the length of a
program that translates between the two languages. If there is such an
algorithm for an $\omega$-sequence of coin-tosses, then the algorithm
expresses a regularity, something that one might even think of as a
deterministic law, that characterizes the sequence. But that an
$\omega$-sequence cannot be represented by a finite algorithm is again not
sufficient for it to be random in the sense we are interested in here.

Consider an infinite sequence that consists of a repeated three-block pattern
of one thousand heads followed by one thousand tails followed by one thousand
random and unbiased heads and tails. The relative frequency of heads and tails
in the full sequence is unbiased. And since there are an infinite number of
random blocks, such a sequence cannot be represented by a finite-length
algorithm. But the sequence is clearly not random. A good Bayesian inquirer
might quickly learn to bet on heads a thousand times, then bet on tails a
thousand times, then bet anything at all a thousand times, then repeat the
pattern. If so, she will enjoy unbounded wealth in the limit.

One way to put the problem is that there is no bound on the amount that a
finite initial segment of this sequence might be compressed. One might write a
very short program that takes advantage of the regularity of the blocks of
heads and tails, then write a program that outputs an initial segment by
alternating that short routine with a routine that just lists each random
block. In this way, one might eventually shorten the algorithmic
representation of finite initial segments of the sequence by more than any
constant $c$. This observation provides the key idea behind _Martin-Löf
randomness_.

An $\omega$-sequence is Martin-Löf random if and only if there is a constant
$c$ such that all finite initial segments are $c$-incompressible by a prefix-
free Turing machine.777An initial segment is $c$-incompressible if and only if
it is not representable by an algorithm that is $c$ shorter than the initial
segment. A prefix-free Turing machine is a universal Turing machine that is
self-delimiting and hence can read its input in one direction without knowing
what, if anything, comes next. Such a machine provides an even playing field.
See Li and Paul Vitányi (2008). This definition also satisfies the two other
desiderata for a suitable notion of randomness. If a sequence is Martin-Löf
random, then there is no fair betting strategy that generates unbounded
wealth. And since measure one of infinite-length sequences are Martin-Löf
random in unbiased Lebesgue measure, it meshes well with the intuition that
random sequences are generic.

One might also define what it means for a sequence to be Martin-Löf random by
considering the set of statistical tests that such a sequence will pass. A
Martin-Löf test is a sequence $\\{U_{n}\\}_{n\in\omega}$ of uniformly
$\Sigma^{0}_{1}$ classes such that $\mu(U_{n})\leq 2^{-n}$ for all $n$, where
$\mu$ is the unbiased Lebesgue measure over the sequences. Being uniformly
$\Sigma^{0}_{1}$ means that there is a single constructive specification of
the sequence of classes. A constructive specification is one that can be
represented by a ordinary algorithm.888See Barrett and Huttegger (2021) for
further details. The idea is that each sequence $\\{U_{n}\\}_{n\in\omega}$ of
uniformly $\Sigma^{0}_{1}$ classes corresponds to a way that a sequence might
be special and thus fail an associated statistical test of randomness. A
sequence passes a particular Martin-Löf test if it is not special in the
specified sense.

Let $2^{\omega}$ be the set of all $\omega$-length sequences (infinite-length
sequences indexed by $\omega$). A class $C\subset 2^{\omega}$ is Martin-Löf
null if there is a Martin-Löf test$\\{U_{n}\\}_{n\in\omega}$ such that
$C\subseteq\bigcap_{n}U_{n}$. A sequence $S\in 2^{\omega}$ is _Martin-Löf
random_ if and only if $\\{S\\}$ is not Martin-Löf null. That is, a sequence
$S$ is Martin-Löf random if and only if it passes every Martin-Löf test. And
again, a sequence has this property if and only if there is a constant $c$
such that all finite initial segments are $c$-incompressible by a prefix-free
Turing machine.

One might use the notion of Martin-Löf randomness to specify the law
$L^{\star}$ as a constraint on the set of physically possible worlds:

$L^{\star}_{ML}$:

The $\omega$-sequence of coin tosses $\langle r_{1},r_{2},\ldots\rangle$ is
_Martin-Löf random_ with unbiased relative frequencies of heads and tails.

Here _all_ of the worlds in $\Omega^{L^{\star}_{ML}}$ are random with well-
defined unbiased relative frequencies. As a result, a non-dogmatic inquirer
will surely infer unbiased relative frequencies in the limit. And inasmuch as
all initial segments of her data will be $c$-incompressible, she will have as
good of evidence as possible that the data are patternless and are hence
randomly distributed.999A $\mbox{probabilistic}^{\star}$ law need not presume
a fundamental direction of time. In order to determine an initial segment, the
definition of Martin-Löf randomness seems to presuppose an initial time and a
temporal direction. However, one can generalize the notion by requiring that
the ordered-sequence of coin tosses be Martin-Löf random for any specified
temporal direction and for any toss one regards as the “initial” toss.

##  3 Alternative algorithmic notions

Martin-Löf randomness is not the only way that one might characterize a
probabilistic coin-toss law. There are other algorithmic notions of randomness
to choose from. Schnorr randomness is a closely-related notion with many of
the same virtues.

A Schnorr test is a Martin-Löf test where the measures $\mu(U_{n})$ are
themselves uniformly computable. A class $C\subset 2^{\omega}$ is Schnorr null
if there is a Schnorr test $\\{U_{n}\\}_{n\in\omega}$ such that
$C\subseteq\bigcap_{n}U_{n}$. And a sequence $S\in 2^{\omega}$ is _Schnorr
random_ if and only if $\\{S\\}$ is not Schnorr null.

Schnorr randomness has similar virtues to Martin-Löf randomness. Initial
segments are patternless in a strong sense, there is a natural sense in which
there is no fair betting strategy, and measure one of infinite-length
sequences, including all those that are Martin-Löf random, are Schnorr random.
(See Figure 2.) And as with Martin-Löf randomness, one might use the notion of
Schnorr randomness to specify a probabilistic constraining law:

$L^{\star}_{S}$:

The $\omega$-sequence of coin tosses $\langle r_{1},r_{2},\ldots\rangle$ is
_Schnorr random_ with unbiased relative frequencies of heads and tails.

One might also consider an associated notion of chance here as well. A
$\mbox{chance}^{\star}_{S}$ process behaves just like an ordinary chance
process except that it can never produce an infinite sequence that fails to be
Schnorr random with well-defined relative frequencies.

While Martin-Löf randomness provides a particularly natural notion of
randomness, Schnorr randomness also has conceptual virtues.101010See Downey
and Griffiths (2002) for details regarding the properties of Schnorr
randomness and Downey and Hirschfeldt (2010) for a description and comparison
of Martin-Löf and Schnorr randomness. So there is a choice to make, but it is
arguably a choice without empirical consequences.

$\Omega^{L^{\star}_{ML}}$$\Omega^{L^{\star}_{S}}$$\Omega^{L}$ Figure 2:
$L^{\star}_{S}$, which is formulated with Schnorr randomness, is compatible
with more worlds than $L^{\star}_{ML}$, which employs Martin-Löf randomness.
However, the two are empirically indistinguishable, if one is limited to
Turing-strength computation.

Since there are sequences that are Schnorr random but not Martin-Löf random,
$L^{\star}_{ML}$ and $L^{\star}_{S}$ are different laws. That said, they are
in a strong sense empirically equivalent since there is no effective procedure
that would determine whether a particular sequence is Martin-Löf random or
Schnorr random but not Martin-Löf random.111111See Barrett and Huttegger
(2021) for a proof. The notion of effective procedure here, as elsewhere in
the paper when not explicitly stipulated to be otherwise, is the standard
Church-Turing one. Hence if one is limited to Turing-strength computation, one
would never be able to distinguish between $L^{\star}_{ML}$ and
$L^{\star}_{S}$ no matter what empirical evidence one had.

The upshot is that moving from a standard probabilistic law to a probabilistic
constraining law eliminates one variety of empirical underdetermination, but
it reveals two others. First, insofar as one expects a sequence of coin tosses
governed by a traditional probabilistic law $L$ to be such that one can detect
no discernible pattern, one should expect law $L^{\star}$ to be empirically
indistinguishable from $L$. And insofar as one is limited to Turing-strength
computations, one will be unable to distinguish between different versions of
$L^{\star}$ like $L^{\star}_{ML}$ and $L^{\star}_{S}$. Since Martin-Löf
randomness has the sort of properties we want and as it is arguably the
standard algorithmic notion (Dasgupta 2011), we shall understand $L^{\star}$
as $L^{\star}_{ML}$.

##  4 $\mbox{Probabilities}^{\star}$, $\mbox{chances}^{\star}$, and laws

What kind of physical law is $L^{\star}$? And how does it govern the world? We
suggest that one might think of $L^{\star}$ as a _generative
$\mbox{chance}^{\star}$ law_ or as a _$\mbox{probabilistic}^{\star}$
constraining law_. We will start with the first.

As a _generative $\mbox{chance}^{\star}$ law_, $L^{\star}$ tells us that each
toss is generated by unbiased $\mbox{chances}^{\star}$, where a
$\mbox{chance}^{\star}$ process behaves just like an ordinary chance process
except that it can _never_ produce an infinite sequence that fails to be
Martin-Löf random or fails to exhibit well-defined relative
frequencies.121212Alternatively, one might consider a similar algorithmic
notion of chance but without requiring there be well-defined relative
frequencies. Here we are assuming well-defined relative frequencies so that an
agent might infer the law given full empirical evidence by conditioning on the
results of coin tosses as she goes. The cost of this further constraint is
modest since we are already requiring the full sequence to be Martin-Löf
random. As a result, a $\mbox{chance}^{\star}$ process involves a subtle
violation of independence. The sequence of tosses will pass every finitely
specifiable test for statistical independence, but since the full sequence
must exhibit the property of being Martin-Löf random with unbiased relative
frequencies, a $\mbox{chance}^{\star}$ process is _holistically_ constrained.
The constraint is not felt on any finite set of tosses, nor is it discoverable
by effective means, but it does require that a relationship hold between the
full sequence of tosses that is generated by the process in the limit. This
interdependence between outcomes may be incompatible with the usual intuitions
behind wanting a generative law. It may also be incompatible with how causal
explanation works more generally.

Given this, $L^{\star}$ is more naturally regarded as a law that governs by
constraining the entire history of the world—in this case, the full
$\omega$-sequence of outcomes. It tells us which sequences of outcomes are
physically possible, namely those that satisfy the frequency constraint and
the randomness constraint imposed by the law. This understanding of
probabilistic laws and their governance also meshes well with Chen and
Goldstein’s (2022) minimal primitivism account (MinP), according to which laws
are certain primitive facts that govern the world by constraining the physical
possibilities of the entire spacetime and its contents.131313Three notes about
the literature. (1) The present account fleshes out one of the interpretive
options of probabilistic laws discussed by Chen and Goldstein (2022, §3.3.3,
Option 4). (2) In one respect, the present account is similar to John T.
Roberts’s nomic frequentism (2009), as they both employ frequency constraints.
However, Roberts does not appeal to algorithmic randomness. On nomic
frequentism, non-random sequences (such as the alternating heads-tails
sequence) are still physically possible. Inasmuch as any non-random sequence
is regarded as evidence against the probabilistic law and in favor of a
deterministic law, it would be better to exclude such sequences from physical
possibilities. (3) Adlam (2022) presents an account of laws of nature as
constraints that is similar to Chen and Goldstein (2022) and contains a
helpful discussion of Roberts’s nomic frequentism.

Understood this way, $L^{\star}$ addresses problems encountered by both non-
Humean and Humean accounts of laws. We will start with the former.

On non-Humean governing accounts of laws, there is a puzzle concerning
precisely how probabilistic laws govern. According to the standard view,
probabilistic laws do not rule out any world. Instead, a probabilistic law
such as $L$ merely assigns some number between zero and one to every
(measurable) subsets in the space of all $\omega$-sequences. This raises a
puzzle: what do these numbers between zero and one represent in physical
reality?

Some non-Humeans appeal to gradable notions such as "propensities" (Maudlin
2007, p.20) or "probabilities of necessitation" (Armstrong 1983, p.172).
Suppose a probabilistic law assigns a $0.2$ probability to the next outcome
being heads. On the propensities view, the chance setup has a $0.2$ propensity
to bring about a heads-outcome in the next toss. On the probabilities of
necessitation view, the current state of affairs necessitates the state of
affairs of a heads-outcome to $0.2$ probability. But while one might make
sense of non-gradable notions of physical possibility and impossibility,
gradable notions such as propensities and degrees of necessitation are less
clear. This seems undesirable.141414See Chen and Goldstein (2022, sections 2
and 3.3.3) for discussions.

In contrast, probabilistic laws, such as $L^{\star}$, can be viewed as a
special class of constraining laws. They constrain what is physically possible
by ruling out certain sequences of outcomes, namely the maverick worlds. A
sequence is physically impossible just in case it fails either the frequency
constraint or the randomness constraint imposed by the law. This allows us to
do away with gradable notions such as propensities or probabilities of
necessitation altogether. In their place, we require only non-gradable notions
of physical possibilities and impossibilities.

Consider MinP as a non-Humean example. We can now employ a single primitive
relation, namely _constraining_ , to understand how both probabilistic laws
and non-probabilistic laws relate to the world. Both types of laws govern by
constraining what is physically possible, thereby ruling out what is
physically impossible. The way that $L^{\star}$ constrains the world is not so
different from that of $F=ma$. $L^{\star}$ constrains the physical
possibilities to be all and only the non-maverick worlds. $F=ma$ constrains
the physical possibilities to be all and only the solutions of $F=ma$. In this
way, $L^{\star}$ removes a major obstacle to a unified understanding of
probabilistic and non-probabilistic laws.

Humeans may also find it useful to adopt $L^{\star}$ for the sort of work we
have been discussing. First, it is relevant to the issue of the Big Bad Bug
(Lewis 1986, pp.xiv-xvi). Lewis notices that the original version of the
Principal Principle and Humean supervenience lead to a contradiction. There
are certain histories of the Humean mosaic, called _undermining histories_ ,
that are assigned, according to the Principal Principle, non-zero probability,
conditionalized on some probabilistic theory $T$ being the best system.
However, they are also assigned, according to Humean supervenience, zero
probability, because $T$ would not be the best system had any of its
undermining histories been actual.

Now, consider the sort of history that would count as undermining. An
undermining history either has the wrong limiting frequencies or no limiting
frequencies or exhibit patterns that can be summarized by a simpler system,
such as a deterministic law in the case of the alternating heads-tails
sequence. Undermining histories, then, are exactly the histories of maverick
worlds, as they lack the frequency or randomness patterns exemplified by
typical sequences of the standard probabilistic law.

Understanding probabilistic laws as $\star$-laws rules out maverick worlds as
physically impossible. If maverick worlds are physically impossible, then
there are no physically possible undermining histories that can be used to
derive the contradiction, and the Big Bad Bug is eliminated. Inasmuch as
restricting to $\star$-laws is also motivated by considerations of
underdetermination and empirical coherence, a Humean may find this solution
particularly natural.

While $L^{\star}$ is naturally interpreted as a constraining law that is well-
suited for non-Humean governing accounts such as MinP, Humeans need not
interpret the constraint as something that exists over and above the mosaic.
They are free to translate the present account into their preferred language
by regarding $L^{\star}$ as a new type of Humean best-system law. They might
then use it to define a new notion of Humean physical possibilities
($\Omega^{L^{\star}}_{Humean}$) and regard both as supervenient on the Humean
mosaic.151515Hoefer (2019, pp.156-158) suggests that, in his preferred
solution to the Big Bad Bug, we should conditionalize on the non-occurrence of
an undermining history. Thinking of the probabilistic law as $L^{\star}$
provides a principled reason, namely that the undermining histories are
physically impossible.

Second, Humeans who understand probabilistic laws as $\star$-laws can also
avoid appealing to _fit_ as a criterion in the best-system analysis of
probabilistic laws, which allows them to bypass difficulties with how to
characterize this notion.161616For helpful discussions, see Elga (2004). Given
an $\omega$-sequence, there is much underdetermination among probabilistic
laws such as $L$, and one needs something like _fit_ to choose the winning
best system. This is because the standard way of understanding
_informativeness_ as the quantity of worlds being excluded does not
distinguish among probabilistic laws like $L$. In contrast, there is
significantly less underdetermination among probabilistic laws like
$L^{\star}$. If we consider a spectrum of different probabilistic statements
like $L^{\star}$ that differ, say, in their specifications of the relative
frequencies in the $\omega$-sequence, then at most one of them is compatible
with the $\omega$-sequence, and thus at most one of them is an axiom in the
best system of that $\omega$-sequence. The best system analysis of a
probabilistic law, such as $L^{\star}$, is much like that of a non-
probabilistic law, such as $F=G\frac{m_{1}m_{2}}{r^{2}}$. If we consider a
spectrum of different versions of the Newtonian gravitational law that differ
in the value of the gravitational constant $G$, then at most one of them is
true of the mosaic, and thus at most one of them is an axiom of the best
system of the mosaic. Given any Humean mosaic, one needs criteria such as
simplicity and informativeness, but one does not need the statistical
criterion of fit, to determine the best system (if there is one). Hence, the
usual problems associated with fit would not arise for such Humeans.171717One
might wonder that there is now an analogous problem of underdetermination
associated with the choice between, say, $L^{\star}_{ML}$ and $L^{\star}_{S}$.
Given this, do Humeans still need something like _fit_? It is not exactly
analogous. The choice between them is not a problem for an idealized observer
with the full sequence and sufficient computational power.

##  5 Discussion

We have shown how a $\star$-law may be thought of as either a generative
$\mbox{chance}^{\star}$ law or a $\mbox{probabilistic}^{\star}$ constraining
law, where the notions of $\mbox{chance}^{\star}$ and
$\mbox{probability}^{\star}$ are subtly different from traditional chance or
probability. Two differences are particularly salient.

The first concerns independence. The results of coin tosses on $L^{\star}$
satisfy every computable test for independence and will hence appear to be
statistically independent. One might say that the results are
$\mbox{probabilistically}^{\star}$ independent. But inasmuch as some sequences
are impossible, there is also a sense in which the results of tosses in this
full $\omega$-sequence are interdependent. To understand $L^{\star}$ as a
generative $\mbox{chance}^{\star}$ law, one would need to allow for a holistic
causal structure that guarantees random sequences with unbiased relative
frequencies in the limit. Depending on one’s commitments regarding causal
explanation, this may lead one to favor understanding $L^{\star}$ as a
$\mbox{probabilistic}^{\star}$ constraining law. If one does decide to gives
up on a generative $\mbox{chance}^{\star}$ law, one is, as we have just seen,
left with a useful option for both proponents of governing-law accounts and
Humeans.

The second difference is that $\mbox{chance}^{\star}$ and
$\mbox{probability}^{\star}$ depend on a choice of a particular standard of
algorithmic randomness. We saw this in the distinction between Martin-Löf and
Schnorr randomness. But if one is limited to Turing-strength computations,
there is no way to distinguish between $L^{\star}_{ML}$ and $L^{\star}_{S}$ on
the basis of empirical evidence alone. The result is a computational sort of
empirical underdetermination.181818See Barrett and Huttegger (2022) for a
discussion of this point. Since as one can only learn a $\star$-law up to an
equivalence class of computationally indistinguishable laws, one might take
the law $L^{\star}$ to be any law in this class.

The possibility of $\star$-laws reveals a further variety of
underdetermination. Inasmuch as one expects the sequence of tosses one gets on
a traditional probabilistic law $L$ to be patternless, one expects $L$ to be
computationally indistinguishable from both $L^{\star}_{ML}$ and
$L^{\star}_{S}$. Of course, these varieties of empirical underdetermination
are not new, they have just gone unnoticed.

That said, $\star$-laws also help to eliminate some forms of empirical
underdetermination. If $L^{\star}$ is true as either a generative or
constraining law, then if it is among the laws that one takes seriously, then,
unlike traditional probabilistic laws but very much like deterministic laws
like $F=ma$ and $F=G\frac{m_{1}m_{2}}{r^{2}}$, one will surely learn it on
complete evidence in _every_ physically possible world.

In contrast, if $L$ is the true law, there will be a continuous cardinality of
maverick worlds such that, if one were to inhabit any of them, one could never
learn $L$ from the results of the coin tosses. On the usual approach to
thinking about laws, one needs special background assumptions to overcome this
difficulty. Specifically, one needs to argue that inhabiting a maverick world
of the true law is sufficiently unlikely or atypical that one has rational
justification for ignoring the possibility.191919Such an argument might appeal
to background assumptions like the Principal Principle or Cournot’s Principle.
While a discussion of the status of such principles is a topic for another
occasion, note that in the present case _every_ world is a maverick world
according to a continuous cardinality of traditional probabilistic laws. As a
result, given sigma additivity, one needs to argue that one is justified in
simply assigning probability zero to almost all probabilistic laws since they
regard the actual world as a maverick world. This point should be
uncontroversial, but it is worth emphasizing. While such assumptions may be
warranted given one’s other commitments, they are not required for empirical
coherence if one restricts one’s hypotheses to $\star$-laws.

In summary, $\star$-laws provide a way of understanding probabilistic laws as
constraints on possible worlds. This helps to clarify how probabilistic laws
might govern. For non-Humeans, they provide a unified way of thinking about
laws as governing by constraints. And for Humeans, they provide a principled
way that they might ignore undermining histories. Specifically, the new notion
of probabilistic law provides a better and independently motivated way to deal
with the Big Bad Bug.

##  6 Conclusion

We have used algorithmic randomness to characterize two types of probabilistic
laws: a _generative $\mbox{chance}^{\star}$ law_, and a
_$\mbox{probabilistic}^{\star}$ constraining law_. We have argued that
$\star$-laws provide a novel way of understanding probabilities and chances,
and help to address one variety of empirical underdetermination, but they also
reveal other varieties that have been underappreciated. For all we know, our
world might be characterized by a traditional probabilistic law or a
$\star$-law.

In our view, the notion of a $\mbox{probabilistic}^{\star}$ constraining law
has advantages over that of a generative $\mbox{chance}^{\star}$ law. It
meshes well with the holistic character of the randomness and relative
frequency constraints, directly supports a unified governing account of non-
Humean laws, and provides independently motivated solutions to issues in the
Humean best-system account. We suggest that both notions are worthy of study
and may lead to new ideas concerning the nature of laws.

## Acknowledgement

We are grateful for helpful discussions with Emily Adlam, Gordon Belot, Craig
Callender, Eugene Chua, Kenny Easwaran, Sheldon Goldstein, Alan Hájek, Simon
Saunders, Charles Sebens, Shelly Yiran Shi, Brian Skyrms, Jason Turner, and
Nino Zanghì.

Bibliography

Adlam, Emily. (2022) “Laws of Nature as Constraints,” _Foundations of Physics_
, 52, 28. https://arxiv.org/abs/2109.13836

Armstrong, David M. (1983) _What is a Law of Nature?_ Cambridge: Cambridge
University Press.

Barrett, Jeffrey A. (2020) _The Conceptual Foundations of Quantum Mechanics_ ,
Oxford: Oxford University Press.

Barrett, Jeffrey A. (1999) _The Quantum Mechanics of Minds and Worlds_ ,
Oxford: Oxford University Press.

Barrett, Jeffrey A. (1996) “Empirical Adequacy and the Availability of
Reliable Records in Quantum Mechanics,” _Philosophy of Science_ 63(1): 49–64.

Barrett, Jeffrey A. and Simon Huttegger (2021) “Quantum Randomness and
Underdetermination,” _Philosophy of Science_ 87(3).
https://doi.org/10.1086/708712

Chen, Eddy Keming and Sheldon Goldstein (2022) “Governing without a
Fundamental Direction of Time: Minimal Primitivism about Laws of Nature,” in
Yemima Ben-Menahem (ed.), _Rethinking the Concept of Law of Nature_ ,
Springer, pp.21-64. https://arxiv.org/abs/2109.09226

Dasgupta, Abhijit (2011) “Mathematical Foundations of Randomness,” in Prasanta
Bandyopadhyay and Malcolm Forster (eds.), _Philosophy of Statistics (Handbook
of the Philosophy of Science: Volume 7)_ , Amsterdam: Elsevier, pp. 641–710.
http://dasgupab.faculty.udmercy.edu/Dasgupta-JSfinal.pdf

Diaconis, Persi and Brian Skyrms (2018) _Ten Great Ideas about Chance_ ,
Princeton and Oxford: Princeton University Press.

Eagle, Antony, (2021) “Chance versus Randomness,” The Stanford Encyclopedia of
Philosophy (Spring 2021 Edition), Edward N. Zalta (ed.),

https://plato.stanford.edu/archives/spr2021/entries/chance-randomness

Elga, Adam (2004) “Infinitesimal Chances and the Laws of Nature,”
_Australasian Journal of Philosophy_ , 82(1), pp.67-76.

Hoefer, Carl (2019) _Chance in the World: A Humean Guide to Objective Chance_
, New York: Oxford University Press.

Lewis, David (1986) _Philosophical Papers, Volume II_ , Oxford: Oxford
University Press.

Maudlin, Tim (2007) _The Metaphysics Within Physics_ , New York: Oxford
University Press.

Roberts, John T. (2009) “Laws about Frequencies,” preprint, http://philsci-
archive.pitt.edu/505


# An Empirical Study of Flaky Tests in JavaScript

Negar Hashemi1, Amjed Tahir2 and Shawn Rasheed3  Massey University
Palmerston North, New Zealand
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Flaky tests (tests with non-deterministic outcomes) can be problematic for
testing efficiency and software reliability. Flaky tests in test suites can
also significantly delay software releases. There have been several studies
that attempt to quantify the impact of test flakiness in different programming
languages (e.g., Java and Python) and application domains (e.g., mobile and
GUI-based). In this paper, we conduct an empirical study of the state of flaky
tests in JavaScript. We investigate two aspects of flaky tests in JavaScript
projects: the main causes of flaky tests in these projects and common fixing
strategies. By analysing 452 commits from large, top-scoring JavaScript
projects from GitHub, we found that flakiness caused by concurrency-related
issues (e.g., async wait, race conditions or deadlocks) is the most dominant
reason for test flakiness. The other top causes of flaky tests are operating
system-specific (e.g., features that work on specific OS or OS versions) and
network stability (e.g., internet availability or bad socket management). In
terms of how flaky tests are dealt with, the majority of those flaky tests
($>$80%) are fixed to eliminate flaky behaviour and developers sometimes skip,
quarantine or remove flaky tests.

###### Index Terms:

Flaky Tests, Test Bugs, JavaScript

##  I Introduction

Test flakiness is a significant issue that affects the quality of software
products. The impact of test flakiness is most apparent in regression testing,
as regression tests rely on tests having deterministic outcomes to ensure that
code changes do not break existing functionality. Flaky tests, which have non-
deterministic outcomes due to factors such as concurrency, randomness, shared
state and reliance on external resources, can make tests unreliable for this
purpose. Such flaky behaviour is problematic as it leads to intermittent
breaks in builds, uncertainty in choosing corrective measures for failing
tests and bug fixes. This can have a negative impact on development, product
quality, and delivery [1, 2, 3]. Flakiness also impacts testing-related
activities such as test suite minimization [4] and test parallelization [5],
as well as techniques that rely on testing such as fault localization [6] or
program repair [7].

Flaky tests are not only problematic, but they are also quite common in large
codebases. For example, it was reported that around 16% of tests at Google are
flaky, and 1 in 7 of the tests occasionally fail in a way that is not caused
by changes to the code or tests [8], making it a real challenge for automated
testing [9]. Vahabzadeh et al. [10] found that 21% of the false alarms in
Apache projects are caused by flaky tests. It was also shown that around 13%
of failed builds in CI pipelines are due to flaky tests [11].

With the increased attention given to flaky tests in research and practice,
there have been a number of studies that investigated different aspects of
flaky tests, such as detection or elimination techniques. Previous studies on
test flakiness and its causes largely focus on specific sources of test
flakiness, such as order-dependency [12], concurrency [13] or UI-specific
flakiness [14, 15]. There have been some empirical studies that investigated
flakiness in the context of specific programming languages. These studies aim
to understand the prevalence of test flakiness, underlying causes and
strategies employed by developers to respond to them. The first empirical
study by Luo et al. [16] focusses on flakiness fixing commits mined from
Apache projects (mostly written in Java). A similar approach was followed for
Android applications in Thorve et al. [17]. A study on flakiness in Python
applications [18] examines existing tests (by running the tests themselves)
rather than mining commits that fix flaky tests.

To the best of our knowledge, there are no studies that specifically
investigate flaky test causes (beyond UI-causes [15]) in JavaScript projects,
despite its popularity and extensive use. JavaScript has been the most
commonly used programming language for several
years111https://insights.stackoverflow.com/survey/2021#technology-most-
popular-technologies. It is also popular in various application domains,
including web (server-side and client-side), mobile, and the Internet of
Things (IoT) [19, 20]. In addition, given the nature of the language (handling
HTTP requests, running on browsers), JavaScript is known to be one of the
languages that uses asynchronous APIs extensively, with an execution model
susceptible to races. Asynchronization and other concurrency bugs are known
sources of flakiness in other languages, in particular Java and Python.

In this paper, we present the first extensive empirical study of flaky tests
in JavaScript. In particular, the goal of this paper is to investigate how
prevalent flaky tests are in JavaScript projects, and to understand the causes
and impact of flaky tests in these projects. To further investigate flaky
tests, we collect and analyse 452 commits from popular JavaScript projects on
GitHub. For each flaky test, we inspect the commit messages, the pull
requests, and the changes to the code to understand the cause of flakiness.
Also, we identify the main strategies that developers follow to deal with
flaky tests.

The remainder of this paper is structured as follows: we present related work
on flaky tests in Section II. Our research questions and methodology are
explained in Section III. We present our detailed results and answer our
research questions, followed by a discussion of the results in Section IV.
Finally, we present our conclusion in Section VI.

##  II Related Work

Several studies published in recent years investigated the main causes and
impact of flaky tests in both open-source and proprietary software. In a study
of the maintenance of the regression test suites in GitHub projects [11], it
was observed that about 13% of the test failures are due to flaky tests. A
survey with developers on the precipitation of flaky tests by Ahmad et al.
[21] identified 23 factors that are perceived to affect test flakiness
(including technical and organisational factors). Similarly, Eck et al. [22]
studied developers’ precipitation when it comes to flaky tests (including
Mozilla projects’ developers) to examine the nature and the origin of 200
flaky tests that had been fixed by the same developers. It was found that
flakiness is perceived as a significant issue by the vast majority of
developers surveyed.

Lam et al. [23] conducted a large-scale longitudinal study of flaky tests to
determine when those tests become flaky, and what changes cause them to become
so. They found that 75% of the flaky tests (184 out of 245) are flaky when
added, and 85% of flaky tests can be detected when detectors are run on newly
added or directly modified tests. Vahabzadeh et al. [10] studied the types of
bugs in test code and observed that flaky tests and semantic bugs constitute
the dominant cause of tests provoking false alarms in the test code.

There have been a few empirical studies on flaky tests in different
programming languages and application domains, but none we could find that
studied flakiness, as an issue in itself, specifically in JavaScript. We
discuss a number of those studies below, and provide a summary of flaky test
causes in Table I.

Luo et al.[16] reported the first extensive empirical study of flaky tests,
categorizing causes and fixing strategies of flaky tests by studying 201
commits of fixes in open-source Apache projects. The root causes of flakiness
were split into ten main categories: async wait, concurrency, test order
dependency, resource leak, network, time, IO, randomness, floating point
operations, and unordered collections.

Gruber et al. [24] analysed projects from the Python Package
Index222https://pypi.org/ to study the cause of the detected flaky tests, and
found that order dependency is the main cause of flakiness in Python projects.
The study identified infrastructure flakiness as a new type of test flakiness
that has not been reported previously. Lam et al. [25] studied the lifecycle
of flaky tests in six large-scale projects at Microsoft, and found that
asynchronous calls are the leading cause of flaky tests in those projects.
They also proposed an automated solution, called Flakiness and Time Balancer
(FaTB), to reduce the frequency of flaky-test failures caused by asynchronous
calls.

The impact of flaky tests has also been studied in different application
domains. A study of flaky tests in Android apps [17] identified two new causes
to those discussed in [16], namely: program logic and UI. Dutta et al. [26]
studied flaky tests in applications that use probabilistic programming or
machine learning frameworks, finding that randomness, as expected, is the
biggest cause of flakiness in those applications. Romano et al. [15] analysed
235 flaky UI tests from 62 web and Android projects, and identified four
categories of causes for UI-based flaky tests. Moran et al. [27] focused on
detecting the root cause of flakiness in web applications by analysing test
execution under different combinations of the environmental factors that may
trigger flakiness.

There have also been a number of tools that have been targeted to detect
certain types of flaky tests such as DeFlaker [28], RootFinder [29],
iFixFlakies [30], SHAKER [31] and FlakeScanner [32].

TABLE I: Causes of flaky tests as identified in prior studies Luo et al.[16] | Thorve et al.[17] |  Dutta et al. [26] | Gruber et al.[24]
---|---|---|---
Async Wait | Concurrency | Algorithmic Non-determinism | Infrastructure
Concurrency | Dependency | Floating-point Computations | Test Order Dependency
Test Order Dependency | Program Logic | Incorrect/Flaky API Usage | Network
Resource Leak | Network | Unsynced Seeds | Randomness
Network | UI | Concurrency | IO
Time |  | Hardware | Time
IO |  | Other | Async Wait
Randomness |  |  | Concurrency
Floating Point Operations |  |  | Resource leak
Unordered Collections |  |  | Test Case Timeout
|  |  | Unordered Collections

##  III Study Methodology

The main goal of this study is to investigate causes and strategies followed
to deal with flaky tests in JavaScript. In this paper we answer the following
two research questions:

RQ1 What are the main causes of flaky tests in JavaScript projects?

RQ2 What are the strategies followed when dealing with flaky tests in
JavaScript projects?

Our target is to analyse commits that are believed to refer to tests with
flaky behaviour, rather than exposing flakiness by compiling and analysing the
programs themselves. We conducted the study in two phases. We first collect
commits (including commit messages and source code changes) from open-source
JavaScript projects and then manually analyse the causes of flakiness as noted
in these commits. We also analyse the strategies followed when those flaky
tests were fixed.

###  III-A Dataset

We constructed a dataset of JavaScript projects obtained from GitHub. We
targeted only popular projects based on the number of stars of each project
(based on the star rating on GitHub). GitHub’s star feature allows users to
mark their interest or helpful repositories on GitHub. In a way, it is a
metric that shows how much interest a project has drawn. We found that the
number of stars is highly correlated with the number of forks and
contributors, indicating popularity.

To search through the repositories, we used a GitHub
API333https://docs.github.com/en/rest/reference/search to search through all
publicly available GitHub repositories and filter results of the commits for
analysis. We first extract the top 40 starred JavaScript projects on GitHub.
We choose the top 40 projects because they provide a large enough sample
(number of commits) that is deemed suitable for our analysis. We compared the
number of commits we retrieved by the ones in similar previous studies: Luo et
al. [16] retrieved 486 commits in total, but analysed 201 commits that fix
flaky tests; while Thorve et al. [17] retrieved 77 commits. Romano et al. [15]
analysed 235 flaky UI test. In comparison, we extracted a total of 316,948
commits and ended up with 735 flaky tests’ related commits.

TABLE II: Projects from the top 40 most starred JavaScript projects on GitHub with commit messages containing at least one of the search keywords Repositories | #Stars | #Commits |  JavaScript File |  #Flakiness Related Commits | #Contributor | Age |  Description
---|---|---|---|---|---|---|---
freeCodeCamp/freeCodeCamp | 320k | 28253 |  47.4% |  4 | 4287 | Oct. 2014 |  Corpus of educational JavaScript projects
vuejs/vue | 188k | 3200 |  97.6% |  2 | 388 | Dec. 2013 |  JavaScript framework for building UI
facebook/react | 174k | 14448 |  95.7% |  22 | 1495 | July 2013 |  JavaScript library for building user interfaces
twbs/bootstrap | 153k | 21165 |  41.4% |  5 | 1231 | Oct. 2011 |  Web development JavaScript framework
electron/electron | 96.6k | 25574 |  6.5% |  48 | 1069 | July 2013 |  Desktop apps framework
nodejs/node | 81.6k | 34487 |  61.2% |  467 | 3012 | May 2009 |  Cross-platform JavaScript runtime environment
denoland/deno | 77.7k | 6310 |  23.6% |  3 | 635 | May 2018 |  A JavaScript and TypeScript runtime environment
mrdoob/three.js | 74.6k | 38172 |  54.2% |  1 | 1482 | Apr. 2010 |  JavaScript 3D Library
mui-org/material-ui | 70.9k | 17626 |  58.2% |  19 | 2278 | Nov. 2014 |  A framework to build React apps
storybookjs/storybook | 64.5k | 35707 |  9% |  14 | 1365 | April 2016 |  UI components development tool
atom/atom | 56k | 38404 |  88.3% |  68 | 488 | Dec. 2013 |  Cross-platform text editor
jquery/jquery | 55.3k | 6536 |  93.6% |  9 | 276 | Jan. 2006 |  JavaScript Library
chartjs/Chart.js | 54.7k | 4043 |  98.3% |  1 | 380 | June 2014 |  Data visualization library
ElemeFE/element | 50.8k | 4500 |  26.7% |  1 | 556 | Sep. 2016 |  A Vue.js-based UI Toolkit for Web
lodash/lodash | 50.6k | 8005 |  100% |  2 | 310 | April 2012 |  A modern JavaScript utility library
moment/moment | 46k | 3961 |  99.7% |  8 | 590 | Jan. 2015 |  A JavaScript date library
meteor/meteor | 42.6k | 24211 |  92.3% |  50 | 465 | Jan. 2012 |  JavaScript web framework
yarnpkg/yarn | 40.1k | 2346 |  98.7% |  11 | 521 | June 2016 |  JavaScript package manager
Total |  | 316948 |  |  735 | 20828 |  |

###  III-B Mining Process

Figure 1: Mining flaky tests’ commits from GitHub’s JavaScript projects

Our mining process is shown in Fig. 1. By using GitHub API, we searched
through 40 repositories for the following keywords: “flaky”, “flakey”,
“flakiness”, “intermit”, “fragile”, “brittle”, “Intermittent”, and “non-
deterministic test”. We found 18 repositories to have at least one commit with
one of the above keywords we used. This resulted in a total of 735 commits. We
conducted this search on May 8th, 2021. Table II shows statistics from the
projects we analysed.

Out of those 735 commits, we found that 254 of them are related to languages
other than JavaScript, so we excluded those as we are interested only in
flakiness that is related to JavaScript. This left us with a total of 481
commits that are relevant to our analysis.

###  III-C Manual Analysis of Commits

We manually analysed all 481 commits that we included. For each commit, we
analysed the commit message, any code changes in the files in the repository
as well as the associated issues (if any) in the issue tracker. We started our
process by manually filtering out commits related to languages other than
JavaScript (out of 735 commits, we found 481 to be JavaScript-related). Two of
the authors then individually and separately classified all 481 commits based
on the cause of flakiness and then compared their classification results. To
inform our classification, we checked the associated issues (if any), pull
requests and changes in the test and CUT. When there were classification
disagreements between the two authors, a third co-author was then involved to
resolve conflicts. The third author analysed a total of 139 commits (where the
two authors did not agree or were not sure). Disagreements between all authors
were settled through discussions until a 100% agreement was reached.

We categorize each of these commits into one of the following :

  * •

Relevant: commits that are directly related to flaky tests.

  * •

Irrelevant: commits that contain one of the keywords, but it is not related to
flakiness.

  * •

Duplicated: identical commits.

Of those 481 commits, we found that 11 commits were found to be irrelevant
(not related to test flakiness), and 18 commits were duplicated, thus those
were excluded. This resulted in a total of 452 commits that we considered
relevant, which we manually classified. We provide a full replication package
(including scripts and data) online at:
https://doi.org/10.5281/zenodo.6757825.

##  IV Results

###  IV-A RQ1: What are the main causes of flaky tests in JavaScript projects?

To answer RQ1, we manually analysed and categorised all commits based on the
relevant causes of flakiness (based on the commit message or the change in the
files associated with the commit). We used the list of causes noted in
previous empirical studies on test flakiness as our baseline (see Table I). We
then classified each commit based on the list of flakiness causes. If the
cause of flakiness is new or not in the list, we then add it as a new cause
and continue with our classification.

During this process, we were not able to categorise or determine the cause of
flakiness in 94 commits. Hence, those were categorised as “unsure” or “hard to
categorise” and we continued to analyse the remaining 358 commits.

As it turns out, most of the causes we identified fit well into one or more of
the categories noted in previous studies (i.e., [16, 17, 26, 24]). However,
some of the causes we identified resulted in new categories that best reflect
the nature of flakiness and thus those have been categorised separately.

Table III shows the results of the top 10 causes we found in the JavaScript
projects we analysed. We present the results of our categories below, together
with examples of each cause from the projects we analysed.

TABLE III: Overall results of flaky test categories (causes) in JavaScript

Cause of Flakiness | # of Commits | %
---|---|---
Concurrency | 74 | 20.7%
Async Wait | 70 | 19.6%
OS | 66 | 18.4%
Network | 45 | 12.6%
Platform | 37 | 10.3%
UI | 21 | 5.9%
Hardware | 17 | 4.7%
Time | 12 | 3.4%
Resource Leak | 10 | 2.8%
Other | 13 | 3.6%
Total | 365* |

  * *

The total number of commits is 358 as there are 7 commits related to 2 causes
of flakiness.

Concurrency We classify a commit in this category when a test is flaky due to
any concurrency-related issues in the test or code under test (CUT) such as
event races, atomicity violations or deadlocks [33]. We found that 74 of the
flaky tests belong to this category. For example, in this
snippet444https://tinyurl.com/2s32r7xk from Node.js, using setImmediate in
line 6 could lead to race conditions, which lead to non-deterministic test
outcomes.

⬇

1onst keepOpen = setTimeout(() => {

2@@ -20,7 +20,7 @@ const timer = setInterval(() => {

3 timer._onTimeout = () => {

4 throw new Error(’Unrefd interval fired after being cleared.’);

5 };

6 setImmediate(() => clearTimeout(keepOpen));

7 }

8}, 1);

Another example of a concurrency related flaky tests is the following example
from Meteor555https://tinyurl.com/ysde5t37. This test is flaky because the
template rendered callbacks get called after flush time, but not if the
template got destroyed. If the client managed to respond to the server
rejecting the method before the client’s flush cycle, the rendered callback
would never fire.

⬇

1testAsyncMulti(’spacebars - template - defer in rendered callbacks’,
[function (test, expect) {

2 var tmpl = Template.spacebars_template_test_defer_in_rendered;

3 var coll = new Meteor.Collection("test-defer-in-rendered--client-only");

4 tmpl.items = function () {

5 return coll.find();

6 };

7 var subtmpl = Template.spacebars_template_test_defer_
in_rendered_subtemplate;

8 subtmpl.rendered = expect(function () {

9 Meteor.defer(function () {

10 });

11 });

12 var div = renderToDiv(tmpl);

13 Meteor._suppress_log(1);

14 coll.insert({});

15}]);

Async Wait A commit is labelled to be in this category when a test makes an
asynchronous call and does not wait properly for the result of the call to
become available before using it. Async Wait is in fact a subcategory of
concurrency, but we classify it separately here mainly because it represents
nearly half (48%) of the concurrency-related flaky tests. In the projects we
analysed, there are 70 commits in total that fits into the async wait
category, such as the following example from
Electron666https://tinyurl.com/2p8zr5k4:

⬇

1 it(’should clear the navigation history’, async () => {

2 loadWebView(webview, {

3 nodeintegration: ’on’,

4 src: ‘file://${fixtures}/pages/history.html‘

5 })

6const event = await waitForEvent(webview, ’ipc-message’)

In this test, the IPC message was sent before it was fully loaded, so the
history did not contain everything it should have, and cannot clear all the
pages, leading to non-deterministic outcomes.

In the following test777https://tinyurl.com/2p8ebjbx from FreeCodeCamp, using
cy.contains(’Go to next challenge’).click(); in line 10 and then not waiting
for the result before visiting the next page can cause flakiness.

⬇

1describe(’project submission’, () => {

2 it(’Should be possible to submit Python projects’, () => {

3 const { superBlock, block, challenges } = projects;

4 challenges.forEach(challenge => {

5 cy.visit(‘/learn/${superBlock}/${block}/${challenge}‘);

6 cy.get(’#dynamic-front-end-form’)

7 .get(’#solution’)

8 .type(’https://repl.it/@camperbot/python-project#main.py’);

9 cy.contains("I’ve completed this challenge").click();

10 cy.contains(’Go to next challenge’).click();

Operating Systems (OS) A commit is classified to be in the OS category when
the test fails due to a run in a specific OS or OS version, while passing on
others. It can happen because of a test dependency on specific OS features or
environmental settings. We classify OS as a separate category of root causes
from the platform (discussed later) because it represents a large percentage
of flaky tests by itself. In total, we found that 66 commits belong to the OS
category. The following test888https://tinyurl.com/533ycmsc from Node.js is a
good example of a flaky test in this category.

⬇

1if (process.platform === ’darwin’) {

2 setTimeout(function() {

3 fs.writeFileSync(filepathOne, ’world’);

4 }, 100);

5 } else {

6 fs.writeFileSync(filepathOne, ’world’);

7}

In OS X, events for fs.watch() might only start showing up after a delay. To
work around that, there is a timer in the test that delays the writing of the
file by 100ms, but in some cases this might not be enough. So the event never
fired, and the test times out.

Another example of an OS flaky test from
Angular999https://tinyurl.com/2p9d7rxk shows a test that resulted in a
ECONNREFUSED error (i.e., refused connection) for only Windows, but it works
well for other OS.

⬇

1function launchChromeAndRunLighthouse(url, flags, config) {

2 const launcher = new ChromeLauncher({autoSelectChrome: true});

3 return launcher.run().

4 then(() => lighthouse(url, flags, config)).

5 then(results => launcher.kill().then(() => results)).

6 catch(err => launcher.kill().then(() => { throw err; }, () => { throw err;
}));

7}

Network A commit is in this category when a test fails due to remote
connection failures (e.g., lost internet connection when accessing an external
URL) or local bad socket management. There are 45 commits that fit into the
network category. One example we see from this category is the following test
from Node.js101010https://tinyurl.com/yc6c97m6, which is using port ‘0’ for an
IPv6-only operation and assuming that the OS would supply a port that was also
available in IPv4. However, the CI results seem to indicate that a port can be
supplied that is in use by IPv4 but available to IPv6, resulting in the test
failing.

⬇

1 net.createServer().listen({

2 host,

3 port: 0,

4 ipv6Only: true,

5 }, common.mustCall());

Another example from React111111https://tinyurl.com/yfprp6tx shows a test that
uses absolute URLs, which can end up in flakiness.

⬇

1var __dirname = __filename.split(’/’).reverse().slice(1).reverse().join(’/’);

2window.ReactWebWorker_URL = __dirname + ’/../src/test/worker.js’ + cacheBust;

3document.write(’<script src="’ + __dirname + ’/../build/jasmine.js’ +
cacheBust + ’"><\/script>’);

4document.write(’<script src="’ + __dirname + ’/../build/react.js’ + cacheBust
+ ’"><\/script>’);

5document.write(’<script src="’ + __dirname + ’/../build/react-test.js’ +
cacheBust + ’"><\/script>’);

6document.write(’<script src="’ + __dirname + ’/../node_modules/jasmine-
tapreporter/src/tapreporter.js’ + cacheBust + ’"><\/script>’);

7document.write(’<script src="’ + __dirname + ’/../test/the-files-to-
test.generated.js’ + cacheBust + ’"><\/script>’);

8document.write(’<script src="’ + __dirname + ’/../test/jasmine-execute.js’ +
cacheBust + ’"><\/script>’);

Platform The commits in this category include all flaky tests related to
platform, e.g., test environment, browsers. We excluded OS related flaky tests
here as those, although are platform related, are a large enough subcategory
that we discussed separately earlier in this section. We identified 37 commits
in total from the platform category. The following
example121212https://tinyurl.com/3xms6muw from Electon shows a test that, due
to some specifications, failed on one CI, but passed on another. The test
times out regularly on Travis CI but pass when built on Jenkins.

⬇

1it(’can be manually resized with setSize even when attribute is present’,
done => {

2 w = new BrowserWindow({show: false, width: 200, height: 200})

3 w.loadURL(’file://’ + fixtures + ’/pages/webview-no-guest-resize.html’)

Similarly, the following test131313https://tinyurl.com/348bzef5 from Node.js
is flaky due to reliance on platform timing in lines 3, 5, 8, 9, 11, and 18.

⬇

1const common = require(’../common’);

2const fs = require(’fs’);

3const platformTimeout = common.platformTimeout;

4const t1 = setInterval(() => {

5 common.busyLoop(platformTimeout(12));

6}, platformTimeout(10));

7const t2 = setInterval(() => {

8 common.busyLoop(platformTimeout(15));

9}, platformTimeout(10));

10const t3 =

11 setTimeout(common.mustNotCall(’eventloop blocked!’), platformTimeout(200));

12setTimeout(function() {

13 fs.stat(’/dev/nonexistent’, () => {

14 clearInterval(t1);

15 clearInterval(t2);

16 clearTimeout(t3);

17 });

18}, platformTimeout(50));

UI The commits in this category include all flaky tests related to UI
features, e.g., flakiness due to different windows display, the blinking
cursor, or the highlight decoration. There are 21 tests that belong to the UI
category. The following example141414https://tinyurl.com/yckkhh4a from Atom
shows a test aims to verify that the cursor blinks when the editor is focused
and the cursors are not moving. It expects the blinking cursor to start in the
visible state, and then transition to the invisible state. But the initial
state of the cursor is not important, since the cursor toggles between the
visible and the invisible state.

⬇

1await component.getNextUpdatePromise()

2 const [cursor1, cursor2] = element.querySelectorAll(’.cursor’)

3 expect(getComputedStyle(cursor1).opacity).toBe(’1’)

4 expect(getComputedStyle(cursor2).opacity).toBe(’1’)

5 await conditionPromise(() =>

6 getComputedStyle(cursor1).opacity === ’0’ &&
getComputedStyle(cursor2).opacity === ’0’

7 )

8 await conditionPromise(() =>

9 getComputedStyle(cursor1).opacity === ’1’ &&
getComputedStyle(cursor2).opacity === ’1’

10 )

11 await conditionPromise(() =>

12 getComputedStyle(cursor1).opacity === ’0’ &&
getComputedStyle(cursor2).opacity === ’0’

13 )

Also, the following Atom test151515https://tinyurl.com/mvdvm9vf is flaky due
to UI related issues. Resize events are unreliable and may not be emitted
right away. This could cause the test code to wait for an update promise that
was unrelated to the resize event (e.g., cursor blinking).

⬇

1 setEditorHeightInLines(component, 13);

2 await setEditorWidthInCharacters(component, 50);

3 expect(component.getRenderedStartRow()).toBe(0);

4 expect(component.getRenderedEndRow()).toBe(13);

Hardware This category combines all flakiness in test outcomes that are
resulted from hardware-dependencies. We found a total of 17 commits related to
the hardware category. For example, in the following
example161616https://tinyurl.com/mttkmtx5 from Node.js, the test runs in
Raspberry Pi. The number of clients is set to 100, which has caused some flaky
behaviour when run in Raspberry Pi.

⬇

1var responses = 0;

2var N = 10;

3var M = 10;

4server.listen(common.PORT, function() {

5 for (var i = 0; i < N; i++) {

6 setTimeout(function() {

7 for (var j = 0; j < M; j++) {

8 http.get({ port: common.PORT, path: ’/’ }, function(res) {

9 console.log(’%d %d’, responses, res.statusCode);

10 if (++responses == N * M) {

11 console.error(’Received all responses, closing server’);

12 server.close();

Another example from this category is the following test from Node.js
test171717https://tinyurl.com/2rfs42zs, which, when looping rapidly and making
new connections (in line 5) it can cause theRaspberry Pi 2 Model to
malfunction.

⬇

1const server = net.createServer(function listener(c) {

2 connections.push(c);

3}).listen(common.PORT, function makeConnections() {

4 for (var i = 0; i < NUM; i++) {

5 net.connect(common.PORT, function connected() {

6 clientConnected(this);

7 });

Time The commits in this category have flakiness that comes from time related
issues, e.g., relying on the system’s time. We classify 12 commits in total
into the time category. The following
example181818https://tinyurl.com/354h3uf4 shows a flaky test that is the
result of relying on the system’s time with the use of Date.now() function.

⬇

1const now = Date.now();

2while (now + 10 >= Date.now());

The following snippet191919https://tinyurl.com/mvfmf8rz from Moment can cause
flakiness because of checking the equality of times that can be different by a
millisecond.

⬇

1test.expect(7);

2test.equal(moment.utc().valueOf(), moment().valueOf(), "Calling moment.utc()
should default to the current time");

Resource Leak We classify a commit in this category in cases where the CUT did
not properly manage resources, e.g., memory allocations issues, database
connections, loss of connection, or not enough space. There are 10 commits
from the resource leak category. The following
example202020https://tinyurl.com/2s44nnua from Node.js shows a flaky test due
to EMFILE, which means that a process is trying to open too many files.

⬇

1if (accumulated.includes(’Error:’) && !finished) {

2 assert(

3 accumulated.includes(’ENOSPC: System limit for number ’ +

4 ’of file watchers reached’),

5 accumulated);

The following snippet212121https://tinyurl.com/4yycbc73 from Node.js shows a
test that is not performing proper clean-up and so it would fail if run more
than one time on the same machine.

⬇

1function test_up_multiple(cb) {

2 console.error(’test_up_multiple’);

3 if (skipSymlinks) {

4 console.log(’skipping symlink test (no privs)’);

5 return runNextTest();

6 }

7 fs.mkdirSync(tmp(’a’), 0755);

8 fs.mkdirSync(tmp(’a/b’), 0755);

9 fs.symlinkSync(’..’, tmp(’a/d’), ’dir’);

10@@ -432,6 +443,7 @@ function test_up_multiple(cb) {

11 if (er) throw er;

12 assert.equal(abedabed_real, real);

13 cb();

14 });

15 });

16}

Other Known Causes Other remaining causes (13 commits) are related to several
categories. The remaining causes are (each provided with an example from the
projects we analysed): IO222222https://tinyurl.com/2zv3acya, test order
dependency232323https://tinyurl.com/39nnnt85, floating point
operations242424https://tinyurl.com/2p999bjw,
randomness252525https://tinyurl.com/3utzckbb, and implementation
dependency262626https://tinyurl.com/yc2vtvzz.

In addition, we categorized 7 commits in total into more than one cause. For
those commits, we believe that there are multiple causes that may have an
equal effect on the flaky behaviour. For example, the following
test272727https://tinyurl.com/2p8hwj7w from Node.js falsely assumed that
closing the client (which also currently destroys the socket rather than
shutting down the connection) would still leave enough time for the server
side to receive the stream error.

⬇

1server.on(’stream’, common.mustCall((stream) => {

2 stream.on(’error’, common.mustCall(() => {

3 stream.on(’close’, common.mustCall(() => {

4 server.close();

5 }));

6 req = client.request();

7 req.resume();

8 req.on(’error’, common.mustCall(() => {

9 req.on(’close’, common.mustCall(() => {

10 client.close();

Both the network and the wait for the connection issues have the same impact
on the test being flaky. Hence, we categorized this test under network and
async wait categories.

RQ1 findings: The top four causes of test flakiness in JavaScript projects are
concurrency (21%), async wait (20%), OS (18%) and network (13%). Other causes
of flaky tests we found include platform, UI, hardware, time and resource
leak.

###  IV-B RQ2: What are the common strategies followed when dealing with flaky
tests in JavaScript projects?

When developers face a flaky test, they usually fix it by changing the test
code, the CUT, or both. We found that a total of 350 commits to fix flakiness
by changing the test code, 3 commits changed CUT, and 5 commits made changes
to both. Although those changes aim to fix the flaky tests, they do not
necessarily completely get rid of the flaky behaviour, but instead they
improve/reduce such behaviour. In other words, the changes sometimes decrease
the chance of the flaky behaviour, but do not eliminate it.

For each commit, we identify the main strategies that developers use to deal
with flaky tests. We categorise those into one of five different strategies,
as follows. A visual summary of the distribution of commits based on the
followed strategy is shown in Fig. 2.

Figure 2: Distribution of Response Strategies

Fix: This strategy aims to fix any flaky test that has been detected/reported
(after reproducing and confirming the flaky behaviour). We found that 295
commits to present an immediate fix of the noted flaky test. This is not
surprising as we mainly mined commits with known flaky tests (that is, flaky
tests that have already been detected by the developers, and thus it is very
likely to have been actioned). The following
example282828https://tinyurl.com/mttkmtx5 from Node.js shows a flaky test from
the _Hardware_ category (the test failed due to set numbers of clients in
Raspberry Pi). The CI results292929https://tinyurl.com/2zz6fmvs show that it
is fixed by reducing the number of clients from 100 to 16.

⬇

1var N = 10;

2var M = 10;

3var N = 4;

4var M = 4;

5server.listen(common.PORT, function() {

6 for (var i = 0; i < N; i++) {

7 setTimeout(function() {

8 for (var j = 0; j < M; j++) {

9 http.get({ port: common.PORT, path: ’/’ }, function(res) {

Improve: In this strategy, the test code or the CUT are changed to decrease
the chance of flakiness or to make the code more traceable to fix it later
(which makes it technical debt in nature). There are 23 commits that in this
category. The following example303030https://tinyurl.com/nwmexy27 from Atom
shows an added delay to the tests to make them less flaky, but they did not
fix the root problem (an explanation is provided in the comment section in the
following listing).

⬇

1// In Windows64, in some situations nsfw (the currently default watcher)

2// does not trigger the change events if they happen just after start
watching,

3// so we need to wait some time. More info:
https://github.com/atom/atom/issues/19442

4await timeoutPromise(300);

Skip/ignore/disable: This strategy provides an option to developers to skip or
ignore the test that is flaky. In some cases, especially when the developer is
fully aware of the implications of those flaky tests, they may decide to
disable those flaky tests and continue with the build as planned. We found 26
commits in total that belong to this category. For example, in the following
code313131https://tinyurl.com/bdzewfc7 from Electron, there is a condition to
skip the test for “win32” platform or “arm64” architecture.

⬇

1 if describe(process.platform !== ’win32’ || process.arch !== ’arm64’)(’did-
change-theme-color event’, () => {

The developer noted the following in a pull
request323232https://tinyurl.com/3epjpe5b:

> “…there are a couple of tests that are consistently flaky on arm (both Linux
> and Windows variants). This PR disables those flakes so that we can get our
> ARM CI to the point where it can be relied on for PR validation instead of
> maintainers ignoring it.”

Quarantine: In this strategy, developers isolate flaky tests from other
healthy tests by keeping them in a quarantined area (e.g., different test
suite or development branch) in order to diagnose and then fix those tests.
This is a common strategy used in practice (e.g., [8, 34, 35]). There are 8
commits that belong to this category. For example, the
code333333https://tinyurl.com/4ybhbhj7 from Node.js move a portion of the test
to a separate file as the test is flaky on CentOS. The developer then noted
that:

> “[test] has been flaky on CentOS. This allows us to .. eliminate the cause
> of the flakiness without affecting other unrelated portions of the test.”

Remove: The strategy suggests that all flaky tests should be removed from the
test suite. We found 6 commits that remove the test to completely eliminate
the flaky behaviour. For example, there is a
commit343434https://tinyurl.com/3uzke4rf from Node.js shows a tests that was
completely removed the test from the test suit due to being flaky. The
developer noted the following:

> “[the test] is supposed to test an internal debug feature, but what it
> effectively ends up testing, is the exact lifecycle of different kinds of
> internal handles … making the test fail intermittently … It’s not a good
> test, delete it.”

Fig. 3 demonstrates the percentages of strategies followed based on the cause
of flakiness. The _Fix_ strategy represents the largest portion of each
category. The most commits under _Improve_ category are related to _UI_ ,
_hardware_ , and _time_ causes. Similarly, as expected, most _Skip_ actions
are taken to deal with _OS_ and _Platform_ -dependent tests. The reason is
that the tests are conditional in nature (will run in specific OSs or
platforms), so that the test can run without any non-determinism. Also, the
largest portion of _Remove_ is related to _UI_ and _Platform_ categories. We
list the most common change patterns used to fix the top four causes
(concurrency, async wait, OS and network) in Table IV.

Figure 3: Distribution of Flakiness Causes on Strategies (the number of commits are shown in parentheses) TABLE IV: Common Changes for Four Top Categories Category | Common Change | Description
---|---|---
Concurrency | waitForEvent |  In some cases, the test is flaky because of confliciting operations in different threads. By making the test wait and adding locks, they can prevent race conditions and synchronizing operations.
increase timeout
setTimeout
setInterval
sleep
add lock
Async Wait | waitForEvent |  Most of the time, developers add time to the test to deal with async wait related test flakiness. For doing this, they usually increase time out or wait for special events.
async
reordering code
Await
setTimeout
setInterval
increase timeout
timeoutPromise
OS | add condition |  In most of the cases, developers check the type and version and run/skip tests on specific ones.
async |  Sometimes, the test is flaky because of the way the OS manages resources as it can take more time to run than expected.
setTimeout
Network | common.port |  In most cases, network related flakiness is caused by bad socket or port management, or lost internet connection.
socket.setTimeout
socket.destory
socket.end()
client.shutdown()

RQ2 findings: We found that 82% of flaky tests are fixed (to eliminate the
flaky behaviour), 7% improved (reduces flakiness), 7% skipped or disabled, 2%
quarantined (for a later fix), and 2% completely removed.

###  IV-C Implications

Our study has implications for both JavaScript testers (who write test code
that might be flaky) and designers of flaky test detection and management
tools. Our results show that JavaScript projects, as with projects in other
languages studied to date, are prone to flaky tests, and that common causes of
flaky tests in programs written using other languages (in particular Java and
Python) are also prevalent in JavaScript. Similarly to prior studies [16, 17],
we observed _Concurrency_ and _Async Wait_ as top causes for flakiness in
JavaScript projects. Unlike in other empirical studies, _Platform_ is also a
major cause of flakiness in JavaScript. This includes all scenarios where the
test runs in a different environment, browser, or OS. The implication from
this observation is that implicit platform dependence (e.g. specific OS) must
be made explicit in the test code. Having automated ways of detecting platform
related flaky tests would be of great value to developers. Any methods that
can enhance developers’ understanding of the requirements and limitations of
the different platforms used (on-the-fly) can potentially reduce the number of
flaky tests caused by platform dependency.

The results of Concurrency and Async Wait being the most dominant cause of
flakiness is expected given that JavaScript applications are highly
asynchronous. Unlike previous studies [16, 24], we observed a small number of
flaky tests that were caused by _Test Order Dependency_. The platform or
testing frameworks are unlikely to be the cause for this low number, since
testing frameworks for JavasScript (e.g. Jest353535https://jestjs.io/,
Mocha363636https://mochajs.org/) have support for parallelising and ordering
test execution and support for shared state across tests. It is possible that
implicit shared state is more likely in object-oriented languages such as Java
with static fields. Note that nearly 47% of shared state in order-dependent
tests [36] are external (e.g. files, database).

In addition, current detection tools focus mainly on languages like Java,
Python and Ruby. Searching through the different resources and the recent
literature surveys on flaky tests [36, 37], we found only one tool, NodeRacer
[38], that aims to manifest test flakiness causes by event races in JavaScript
programs. There is related work on detecting other concurrency issues in
JavaScript programs such as NRace [39] or atomicity violations, NodeAV [40].
There is, generally, a lack of research on tools or techniques to deal with
non-deterministic tests in JavaScript projects. More work is needed to provide
detection tools that target other categories of causes for test flakiness in
JavaScript, such as order dependent, OS and network flakiness.

##  V Threats to Validity

Missing flaky tests related commits: We used a keyword-based approach to
locate commits that are flakiness related. To provide an extended coverage, we
used an extended list of 7 flaky test-related keywords in our search string
(flaky, intermittent, fragile, brittle, flakey, non-deterministic test and
intermit). It is still possible that we could miss either 1) commits that used
keywords other than the one we identified in our research, or 2) flaky tests
that have no associated commits (only reported in the issue tracker but have
not been reviewed or actioned yet).

Generalisation of the findings: The study considers commits from JavaScript
projects that are publicly available on GitHub. The observations noted in this
study, in terms of the categories of flaky tests, can be limited to the
selected projects and may not be generalizable to other JavaScript programs.
We mitigated this by selecting popular and highly starred projects that are
managed by big communities and represent various types of applications
(server, frontend/backend frameworks, web applications, graphics, etc.). Thus,
our sample is somehow representative of real-world programs. Including
additional projects will surely supplement our findings, and this may lead to
more generalised conclusions.

Manual classification of causes: A large part of the study depends on manual
analysis of the data (commit messages and source code changes), which could
affect the construct validity due to potential personal oversight and bias. In
order to reduce potential false positives, all identified commits were
independently classified by two authors (all commits were separately
classified by the two of the authors), and when there were disagreements about
a certain commit, we introduced a conflict resolution step where a third
author was then involved to also independently classify the disagreed on
commits. We followed that with a rigorous discussion of the causes between all
authors until we reached an agreement (100% agreement level). There were still
issues with a number of commits that all authors agreed that the cause of
flakiness is unknown or cannot be identified - those have been classified as
“hard to classify” and then excluded from the analysis.

##  VI Conclusion

In this paper, we investigate the presence of flaky tests in JavaScript
projects. We first categorise flaky test-related commits based on the cause of
flakiness, and then investigate the common strategies followed to deal with
test flakiness. Our study shows that the common causes of flaky tests in
JavaScript are not different from those noted in other languages - in
particular, Java [16] and Python [24]. We found that 70% of flaky test commits
in JavaScript are caused by one of the following: Concurrency, Async Wait, OS
or Network. Unlike in previous studies [16, 24], which identify test-order
dependency as one of the key causes of flakiness, we found only very few flaky
test commits that are caused by test-order-dependency. In terms of fixing
strategies, we note that the majority of flaky tests (82%) are fixed to
eliminate flaky behaviour completely. A smaller percentage of those flaky
tests are either skipped (ignored), quarantined (to be fixed later, becoming a
technical debt) or completely removed from the test suite. These results can
help future research on flaky tests, in particular JavaScript tools designers
in building tools that help to detect and remove flaky tests from test suites.

## Acknowledgment

This work was partially funded by a SfTI National Science Challenge grant No.
MAUX2004.

## References

  * [1] M. Fowler, “Eradicating non-determinism in tests,” 2011, https://martinfowler.com/articles/nonDeterminism.html.
  * [2] A. Sandhu, “How to fix flaky tests,” 2015. [Online]. Available: https://tech.justeattakeaway.com/2015/03/30/how-to-fix-flaky-tests/
  * [3] J. Palmer, “Test flakiness – methods for identifying and dealing with flaky tests,” 2019. [Online]. Available: https://engineering.atspotify.com/2019/11/18/test-flakiness-methods-for-identifying-and-dealing-with-flaky-tests/
  * [4] M. Machalica, A. Samylkin, M. Porth, and S. Chandra, “Predictive test selection,” in _2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)_. IEEE, 2019, pp. 91–100.
  * [5] W. Lam, A. Shi, R. Oei, S. Zhang, M. D. Ernst, and T. Xie, “Dependent-test-aware regression testing techniques,” in _Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis_ , 2020, pp. 298–311.
  * [6] B. Vancsics, T. Gergely, and Á. Beszédes, “Simulating the effect of test flakiness on fault localization effectiveness,” in _2020 IEEE Workshop on Validation, Analysis and Evolution of Software Tests (VST)_ , 2020, pp. 28–35.
  * [7] Y. Qin, S. Wang, K. Liu, X. Mao, and T. F. Bissyandé, “On the impact of flaky tests in automated program repair,” in _2021 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)_ , 2021, pp. 295–306.
  * [8] J. Micco. Flaky tests at google and how we mitigate them. [Online]. Available: https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html
  * [9] G. Pirocanac. Test flakiness – one of the main challenges of automated testing. [Online]. Available: https://testing.googleblog.com/2020/12/test-flakiness-one-of-main-challenges.html
  * [10] A. Vahabzadeh, A. M. Fard, and A. Mesbah, “An empirical study of bugs in test code,” in _2015 IEEE international conference on software maintenance and evolution (ICSME)_. IEEE, 2015, pp. 101–110.
  * [11] A. Labuschagne, L. Inozemtseva, and R. Holmes, “Measuring the cost of regression testing in practice: A study of java projects using continuous integration,” in _Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering_ , 2017, pp. 821–830.
  * [12] A. Gambi, J. Bell, and A. Zeller, “Practical test dependency detection,” in _2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST)_. IEEE, 2018, pp. 1–11.
  * [13] Z. Dong, A. Tiwari, X. L. Yu, and A. Roychoudhury, “Concurrency-related flaky test detection in android apps,” 2020.
  * [14] A. M. Memon and M. B. Cohen, “Automated testing of gui applications: models, tools, and controlling flakiness,” in _2013 35th International Conference on Software Engineering (ICSE)_. IEEE, 2013, pp. 1479–1480.
  * [15] A. Romano, Z. Song, S. Grandhi, W. Yang, and W. Wang, “An empirical analysis of ui-based flaky tests,” in _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_. IEEE, 2021, pp. 1585–1597.
  * [16] Q. Luo, F. Hariri, L. Eloussi, and D. Marinov, “An empirical analysis of flaky tests,” in _Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering_ , 2014, pp. 643–653.
  * [17] S. Thorve, C. Sreshtha, and N. Meng, “An empirical study of flaky tests in android apps,” in _2018 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. IEEE, 2018, pp. 534–538.
  * [18] A. Sjöbom, “Studying test flakiness in python projects: Original findings for machine learning,” 2019.
  * [19] S. Technologies, “Why is javascript so popular?” 2018. [Online]. Available: https://web.archive.org/web/20191208013733/https://www.simplytechnologies.net/blog/2018/4/11/why-is-javascript-so-popular
  * [20] I. Lima, J. Silva, B. Miranda, G. Pinto, and M. d’Amorim, “Exposing bugs in javascript engines through test transplantation and differential testing,” _Software Quality Journal_ , vol. 29, no. 1, pp. 129–158, 2021.
  * [21] A. Ahmad, O. Leifler, and K. Sandahl, “Empirical analysis of practitioners’ perceptions of test flakiness factors,” _Software Testing, Verification and Reliability_ , vol. 31, no. 8, p. e1791, 2021.
  * [22] M. Eck, F. Palomba, M. Castelluccio, and A. Bacchelli, “Understanding flaky tests: The developer’s perspective,” in _Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2019, pp. 830–840.
  * [23] W. Lam, S. Winter, A. Wei, T. Xie, D. Marinov, and J. Bell, “A large-scale longitudinal study of flaky tests,” _Proc. ACM Program. Lang._ , vol. 4, 2020\. [Online]. Available: https://doi.org/10.1145/3428270
  * [24] M. Gruber, S. Lukasczyk, F. Kroiß, and G. Fraser, “An empirical study of flaky tests in python,” in _2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST)_. IEEE, 2021, pp. 148–158.
  * [25] W. Lam, K. Muşlu, H. Sajnani, and S. Thummalapenta, “A study on the lifecycle of flaky tests,” in _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_ , 2020, pp. 1471–1482.
  * [26] S. Dutta, A. Shi, R. Choudhary, Z. Zhang, A. Jain, and S. Misailovic, “Detecting flaky tests in probabilistic and machine learning applications,” in _Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis_. Association for Computing Machinery, 2020, pp. 211–224.
  * [27] J. Morán Barbón, C. Augusto Alonso, A. Bertolino, C. A. Riva Álvarez, P. J. Tuya González _et al._ , “Flakyloc: flakiness localization for reliable test suites in web applications,” _Journal of Web Engineering, 2_ , 2020.
  * [28] J. Bell, O. Legunsen, M. Hilton, L. Eloussi, T. Yung, and D. Marinov, “Deflaker: Automatically detecting flaky tests,” in _2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE)_. IEEE, 2018, pp. 433–444.
  * [29] W. Lam, P. Godefroid, S. Nath, A. Santhiar, and S. Thummalapenta, “Root causing flaky tests in a large-scale industrial setting,” in _Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis_ , 2019.
  * [30] A. Shi, W. Lam, R. Oei, T. Xie, and D. Marinov, “ifixflakies: A framework for automatically fixing order-dependent flaky tests,” in _Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2019, pp. 545–555.
  * [31] D. Silva, L. Teixeira, and M. d’Amorim, “Shake it! detecting flaky tests caused by concurrency with shaker,” in _2020 IEEE International Conference on Software Maintenance and Evolution (ICSME)_ , 2020, pp. 301–311.
  * [32] Z. Dong, A. Tiwari, X. L. Yu, and A. Roychoudhury, “Flaky test detection in android via event order exploration,” in _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2021, pp. 367–378.
  * [33] J. Wang, W. Dou, Y. Gao, C. Gao, F. Qin, K. Yin, and J. Wei, “A comprehensive study on real world concurrency bugs in node. js,” in _2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 2017, pp. 520–531.
  * [34] E. Zhu, “Solving flaky tests in rspec,” 2019. [Online]. Available: https://flexport.engineering/solving-flaky-tests-in-rspec-9ceadedeaf0e
  * [35] R. Agarwal, “Handling Flaky Unit Tests in Java,” Jun. 2021. [Online]. Available: https://eng.uber.com/handling-flaky-tests-java/
  * [36] O. Parry, G. M. Kapfhammer, M. Hilton, and P. McMinn, “A survey of flaky tests,” _ACM Transactions on Software Engineering and Methodology (TOSEM)_ , vol. 31, no. 1, pp. 1–74, 2021.
  * [37] B. Zolfaghari, R. M. Parizi, G. Srivastava, and Y. Hailemariam, “Root causing, detecting, and fixing flaky tests: State of the art and future roadmap,” _Software: Practice and Experience_ , vol. 51, no. 5, pp. 851–867, 2021.
  * [38] A. T. Endo and A. Møller, “Noderacer: Event race detection for node. js applications,” in _2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)_. IEEE, 2020, pp. 120–130.
  * [39] X. Chang, W. Dou, J. Wei, T. Huang, J. Xie, Y. Deng, J. Yang, and J. Yang, “Race detection for event-driven node. js applications,” in _2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 2021, pp. 480–491.
  * [40] X. Chang, W. Dou, Y. Gao, J. Wang, J. Wei, and T. Huang, “Detecting atomicity violations for event-driven node. js applications,” in _2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)_. IEEE, 2019, pp. 631–642.