# Treatment of Epistemic Uncertainty in Conjunction Analysis with Dempster-
Shafer Theory

L. Sanchez 111PhD Candidate, Aerospace Centre of Excellence, luis.sanchez-
<EMAIL_ADDRESS>and M. Vasile222Professor at University of
Strathclyde and Director of Aerospace Centre of Excellence,
<EMAIL_ADDRESS>Aerospace Centre of Excellence. University
of Strathclyde, Glasgow, G1 1XJ, United Kingdom  S. Sanvido333Space Debris
Engineer. IMS Space Consultancy GmbH<EMAIL_ADDRESS>Space IMS
Space Consultancy GmbH, 64297, Darmstadt, Germany  K. Mertz 444Spaces Debris
Engineer, Space Debris Office, ESA/ESOC<EMAIL_ADDRESS>Space Debris
Office, European Space Operations Center (ESOC), European Space Agency (ESA),
Darmstadt, 64293, Germany  C. Taillan555Space Surveillance Engineer. Space
Security, Safety and Sustainability Office, CNES<EMAIL_ADDRESS>Space Security, Safety and Sustainability Office, Centre National d’Etudes
Spatiales (CNES), Toulouse, 31401, France

###### Abstract

The paper presents an approach to the modelling of epistemic uncertainty in
Conjunction Data Messages (CDM) and the classification of conjunction events
according to the confidence in the probability of collision. The approach
proposed in this paper is based on Dempster-Shafer Theory (DSt) of evidence
and starts from the assumption that the observed CDMs are drawn from a family
of unknown distributions. The Dvoretzky–Kiefer–Wolfowitz (DKW) inequality is
used to construct robust bounds on such a family of unknown distributions
starting from a time series of CDMs. A DSt structure is then derived from the
probability boxes constructed with DKW inequality. The DSt structure
encapsulates the uncertainty in the CDMs at every point along the time series
and allows the computation of the belief and plausibility in the realisation
of a given probability of collision. The methodology proposed in this paper is
tested on a number of real events and compared against existing practices in
the European and French Space Agencies. We will show that the classification
system proposed in this paper is more conservative than the approach taken by
the European Space Agency but provides an added quantification of uncertainty
in the probability of collision.

## Abstract

Keywords: Space Traffic Management, Conjunction Data Message, Epistemic
Uncertainty, Dempster-Shafer theory of evidence, Conjunction Assessment,
Decision-making.

AI

    Artificial Intelligence
Bel

    Belief
bpa

    basic probability assignment
CAM

    Collision Avoidance Manoeuvre
CARA

    Conjunction Assessment Risk Analysis
CDF

    Cumulative Distribution Function
CDM

    Conjunction Data Message
CNES

    Centre National d’Etudes Spatiales
DKW

    Dvoretzky–Kiefer–Wolfowitz
DoU

    Degree of Uncertainty
DSt

    Dempster-Shafer theory of evidence
eCDF

    empirical Cumulative Distribution Function
ESA

    European Space Agency
ESOC

    European Space Operations Centre
FE

    Focal Element
FN

    False Negative
FP

    False Positive
FPR

    False Positive Rate
HBR

    Hard-Body Radius
IDSS

    Intelligent Decision Support System
JAC

    Java for Assessment of Conjunctions
KS

    Kolmogorov-Smirnov
LEO

    Low Earth Orbit
ML

    Machine Learning
mWSM

    modified Weighted Sum Method
Pl

    Plausibility
PoC

    Probability of Collision
ROC

    Receiver Operating Characteristic
SEM

    Space Environment Management
SDO

    Space Debris Office
sPoC

    scaled Probability of Collision
STM

    Space Traffic Management
TCA

    Time of Closest Approach
TN

    True Negative
TOPSIS

    Technique for Order of Preference by Similarity to Ideal Solution
TP

    True Positive
TPR

    True Positive Rate
WPM

    Weighted Product Method
WSM

    Weighted Sum Method

##  1 Introduction

The close encounter of two space objects, also known as a conjunction between
a chaser and a target, can lead to a collision if the relative position of the
two objects is not properly controlled. The Probability of Collision (PoC) to
happen depends on the probability that each of the two objects occupies a
given position in space. This probability can be derived from the knowledge of
the orbit of the two objects and the associated uncertainty.

It is customary to assume that the distribution of possible positions of the
two objects at the time of closest encounter follows a multivariate Gaussian
with a given mean and covariance matrix, see [1, 2]. This assumption is
limited by three sources of uncertainty: the uncertainty in the dynamic model
used to propagate the orbit from the last available observation to the time of
closest approach, the uncertainty in the actual distribution at the time of
closest approach, and the uncertainty in the last observed state before
closest approach. We argue that all three forms of uncertainty are epistemic
in nature since they derive from a lack of knowledge of the model,
distribution and error in the observations.

The information on a given close encounter is generally available in the form
of a Conjunction Data Message (CDM), which contains the means and covariances
of the two objects at Time of Closest Approach (TCA), see [3]. Thus, in this
paper, we start from the assumption that the mean and covariance in each CDM
are affected by epistemic uncertainty, which is reflected in an uncertainty in
the correct value of the PoC.

The general attempt to compensate for the uncertainty in the CDMs is to
improve the realism of the covariance matrix by improving its propagation,
[4], or by some form of updating of the dynamic model, [5]. These approaches
are all very valuable but require direct access to the post-observation data.
Other methods based solely on the available CDMs tried to predict the next
CDMs using machine learning starting from an available time series, see [6, 7,
8, 9], or increased the last covariance under the assumption that the series
of CDMs should follow a given distribution, [10, 11]. This last approach does
not modify the mean value or miss distance.

So far, only a limited number of authors have directly addressed epistemic
uncertainty in conjunction analysis, see for example [12, 13, 14, 15]. In [16,
17, 18] the authors proposed a robust approach to conjunction analysis and
collision avoidance planning based on Dempster-Shafer theory of evidence
(DSt). DSt allows making decisions informed by the degree of confidence in the
correctness of a value rather then by the value itself, [19]. However, the
available information to build the frame of discernment that is needed in DSt
is often limited in a sequence of CDMs. CDMs contain little information on the
three forms of uncertainty listed above and essentially only provide
covariance and mean value of the miss distance. Thus, one key question is how
to translate the time series of CDMs into the frame of discernment used in
DSt. The underlying assumption in this work is that the CDMs are observables
drawn from an unknown family of distributions defined within some bounds.
Without uncertainty, one would be able to exactly predict the next CDMs as the
mean and covariance would only depend on observations with a known
distribution and there would be no uncertainty in the propagation model and
distribution at TCA. Furthermore, we assumed that the CDMs computed from
observations acquired close to the TCA were less affected by model and
distribution uncertainty. This is reasonable as the propagation time is
shorter and thus both nonlinearities and model errors have a lower impact on
the propagation of the distribution of the possible states.

The paper introduces a methodology, based on the Dvoretzky–Kiefer–Wolfowitz
(DKW) inequality, [20], to derive a DSt structure capturing the epistemic
uncertainty in a given sequence of CDMs. From the DSt structures, one can
compute the Belief (Bel) and Plausibility (Pl) that the value of the PoC is
correct and an upper and lower bound on its value. The paper then proposes a
classification system that exploits the use of Bel and Pl to differentiate
between events that are uncertain from events that can lead to a collision.
The overall methodology is tested on a number of real conjunction scenarios
with known sequences of CDMs and compared against current practices in the
European Space Agency (ESA) and Centre National d’Etudes Spatiales (CNES).

The rest of the paper is structured as follows. Section 2 briefly introduces a
methodology previously presented by the authors to deal with epistemic
uncertainty for risk assessment in space encounters. Section 3 extends this
methodology to deal with sequence of CDMs. In Section 4, some numerical cases
are presented showing the operation of the proposed method and comparing the
approach with the procedure followed by real operators. Finally, Section 5
concludes the paper with the final remarks and future work.

##  2 Conjunction Analysis with Dempster-Shafer Structures

This section briefly introduces the basic idea of DSt applied to Conjunction
Assessment Risk Analysis (CARA). It also includes the DSt-based conjunction
classification system already introduced by the authors in previous works.
More details on DSt can be found in [21], and more details on its application
to space conjunction assessment can be found in [16, 17, 18].

In this paper, we consider only fast encounters between two objects: object 1
and object 2. Under the typical modelling assumptions of fast encounters, see
[22], the PoC can be defined as:

$PoC=\frac{1}{2\pi\sqrt{\|\mathbf{\Sigma}\|}}\int\limits_{\mathcal{B}((0,0),R)}e^{-\frac{1}{2}\left((\mathbf{b}-\bm{\mu})^{T}\mathbf{\Sigma}^{-1}(\mathbf{b}-\bm{\mu})\right)}d\xi
d\zeta$ (1)

where, without loss of generality, object 2 is at the centre of the coordinate
system of the impact plane at the time of closest approach (TCA),
$\mathbf{b}=[\xi,\zeta]^{T}$ is the position vector of object 1 with respect
to object 2 projected onto the impact plane, $\bm{\Sigma}$ is the $2\times 2$
combined covariance matrix of the position of the two objects in the impact
plane ($\bm{\Sigma}=\bm{\Sigma}_{1}+\bm{\Sigma}_{2}$, with $\bm{\Sigma}_{1}$
and $\bm{\Sigma}_{2}$ the individual covariance matrices of object 1 and 2
respectively) and $\bm{\mu}=[\mu_{\xi},\mu_{\zeta}]^{T}$ is the expected
position vector of object 1 with respect to object 2 projected onto the impact
plane. In the remainder of the paper $\bm{\mu}$ is called miss distance. The
integration region ${\mathcal{B}((0,0),R)}$, or Hard-Body Radius (HBR), is a
disk with radius $R$ centred at the origin of the impact plane.

When the covariance $\bm{\Sigma}$ and miss distance $\bm{\mu}$ are not
precisely known the $PoC$ is affected by a degree of uncertainty. This lack of
knowledge translates into an epistemic uncertainty in the exact value of
$\bm{\Sigma}$ and $\bm{\mu}$. The epistemic uncertainty in covariance
$\bm{\Sigma}$ and miss distance $\bm{\mu}$ can come from incertitude in the
sources of information, from poor knowledge of the measurements or propagation
model or from an approximation of the actual distribution on the impact plane
at TCA. As shown in [16] and [17], this epistemic uncertainty can be modelled
with DSt.

The idea proposed in [16], was to use DSt to compute the level of confidence
in the correctness of the value of the PoC, given the available evidence on
the sources of information. Each component of the combined covariance matrix
in the impact plane,
$[\sigma^{2}_{\xi},\sigma^{2}_{\zeta},\sigma_{\xi\zeta}]$, was modelled with
one or more intervals and so was the miss distance $[\mu_{\xi},\mu_{\zeta}]$.
A basic probability assignment (bpa) was then associated with each interval.
The intervals and the associated bpa can be derived, for example, directly
from the raw observations, [15, 23], or from a time series of CDMs, [3], as
explained later in this paper. Note that in the case in which raw observation
data are available, one could directly compute the confidence on the miss
distance, see [15]. However, in the following we will consider the CDMs as the
observable quantities and the PoC, computed from the CDMs, to be the quantity
of interest.

Given the intervals and associated bpa, one can compute the cross-product of
all the intervals under the assumption of epistemic independence. Each product
of intervals with non-zero bpa constitutes a Focal Element (FE), $\gamma_{i}$,
whose joint bpa is the product of the bpas of the individual intervals. When
computing the PoC, each FE defines a family of bi-variate Gaussian
distributions on the impact plane. In the following, the collection of all
focal elements forms the uncertainty space U, and the uncertain parameter
vector is
$\mathbf{u}=[\mu_{\xi},\mu_{\zeta},\sigma^{2}_{\xi},\sigma^{2}_{\zeta},\sigma_{\xi\zeta}]^{T}$
so that $\mathbf{u}\in U$.

Given the set $\Phi=\\{PoC|PoC\geq PoC_{0}\\}$ and $\Omega=\\{\mathbf{u}\in
U|PoC(\mathbf{u})\in\Phi\\}$ the Pl and Bel that the $PoC$ is larger than a
given threshold $PoC_{0}$ given the available evidence are:

$Bel(\Omega)=\sum_{\gamma_{i}\subset\Omega}{bpa(\gamma_{i})}$ (2a)
$Pl(\Omega)=\sum_{\gamma_{i}\cap\Omega\neq\emptyset}{bpa(\gamma_{i})}$ (2b)

For different values $PoC_{0}$, Eqs. 2a and 2b define two curves (see the
example in Fig. 1). The area between the curves, $A_{Pl,Bel}$, in logarithmic
scale, is:

$A_{Pl,Bel}=\int_{\log(\underline{PoC})}^{0}Pl(\Omega)\,d(\log(PoC))-\int_{\log(\underline{PoC})}^{0}Bel(\Omega)\,d(\log(PoC))$
(3)

$Bel(\Omega)$ is a lower bound on the probability that $PoC\geq PoC_{0}$. Its
value is computed by adding up all the FEs fully supporting the hypothesis
$PoC\geq PoC_{0}$. $Pl(\Omega)$ is an upper bound on the probability that
$PoC\geq PoC_{0}$. Its value is computed by adding up all the FEs only
partially supporting the hypothesis $PoC\geq PoC_{0}$. The area $A_{Pl,Bel}$
quantifies the amount of uncertainty on the probability that $PoC\geq
PoC_{0}$, i.e. if no epistemic uncertainty is present, both curves would
reduced to the same Cumulative Distribution Function (CDF). Thus, for a given
value of $PoC_{0}$, a large value of Pl associated with a small value of
$A_{Pl,Bel}$ suggests that there is a lot of support to the hypothesis
$PoC\geq PoC_{0}$ given the available information. On the contrary a large
value of Pl associated to a large value of $A_{Pl,Bel}$ suggests that the
hypothesis $PoC\geq PoC_{0}$ is very plausible to be true but with a high
degree of uncertainty.

Figure 1: Support to the value of $PoC$ being greater than a given value:
$Bel$ -black solid line; $Pl$ \- black dashed line. The dotted purple line
represents a possible $PoC_{0}$.

In [16], a DSt-based classification system was proposed to decide whether, for
a given conjunction event, a Collision Avoidance Manoeuvre (CAM) was required
or not. In this paper, we propose a revised version of the classification
approach proposed [16]. A given conjunction event is classified according to:
i) the value of the Pl at $PoC=PoC_{0}$ or $Pl(PoC_{0})$, ii) the time of
closest approach $t2TCA$ and iii) the area $A_{Pl,Bel}$. We introduced five
thresholds: two time thresholds indicating the proximity of the event, $T_{1}$
and $T_{2}$, the maximum admissible $PoC$, or $PoC_{0}$, the level of Pl,
$Pl_{0}$, above which there is sufficient support to the hypothesis $PoC\geq
PoC_{0}$, and the value of area $A_{0}$, above which the information is
considered to be uncertain. Three of the five thresholds, $T_{1}$, $T_{2}$ and
$PoC_{0}$, are decided by the operators and depend on operational constraints,
the other two $Pl_{0}$ and $A_{0}$ need to be tuned under evidence-based
criteria, as it will be explained in the remainder of the paper.

We then introduce the following six classes, see Table 1, each defined by a
combination of $Pl(PoC_{0})$, $t2TCA$ and $A_{Pl,Bel}$:

Table 1: Conjunction risk assessment evidence-based classification criterion. Time to TCA |  $\mathbf{Pl}$ at $\mathbf{PoC_{0}}$ | Area between curves | Class
---|---|---|---
$t2TCA\leq T_{1}$ | $Pl(PoC_{0})<Pl_{0}$ | - | 5
$Pl(PoC_{0})\geq Pl_{0}$ | $A_{Pl,Bel}<A_{0}$ | 1
$A_{Pl,Bel}\geq A_{0}$ | 0
$T_{1}<t2TCA$ $t2TCA\leq T_{2}$ | $Pl(PoC_{0})<Pl_{0}$ | - | 4
$Pl(PoC_{0})\geq Pl_{0}$ | $A_{Pl,Bel}<A_{0}$ | 2
$A_{Pl,Bel}\geq A_{0}$ | 3
$t2TCA>T_{2}$ | - | - | 3

  * •

Class 0: there is enough evidence supporting $PoC\geq PoC_{0}$ but is
accompanied by a high degree of uncertainty and no time to acquire new
measurements, due to the proximity of the event, hence a CAM is recommended.

  * •

Class 1: there is full support to the hypothesis $PoC\geq PoC_{0}$, with
limited uncertainty, and short $t2TCA$, hence a CAM is required.

  * •

Class 2: there is full support to the hypothesis $PoC\geq PoC_{0}$, with
limited uncertainty, preparing a CAM is recommended, but a CAM is not executed
yet due to the available time before the encounter.

  * •

Class 3: there is enough evidence supporting $PoC\geq PoC_{0}$ but is
accompanied by a high degree of uncertainty with sufficient time to acquire
new measurements, hence more measurements should be acquired.

  * •

Class 4: there is insufficient evidence supporting $PoC\geq PoC_{0}$ and
sufficient time to acquire new measurements.

  * •

Class 5: no action is implemented, since $t2TCA$ is too short and there is
insufficient evidence supporting $PoC\geq PoC_{0}$.

Note that for $t2TCA>T_{2}$ all events are classified as Class 3 because the
required action is to acquire more measurements. Also, it has to be noted that
the level of confidence that one has in the computed value of the $PoC$
depends only on $Pl_{0}$. If $Pl_{0}$ is set to zero it means that one accepts
even a single piece of partial evidence that $PoC\geq PoC_{0}$ to escalate the
Class from 5, to 0 or 1, or from 4 to 2 or 3.

##  3 Modelling Epistemic Uncertainty in Conjunction Data Messages

The use of DSt to model epistemic uncertainty does not require any assumption
on the probability of an event and also captures rare events with low
probability. On the other hand with no direct information on measurements and
dynamic model, one can only rely on the CDMs to define the FEs and associated
probability masses.

This section presents a methodology to associate one of the six classes
introduced in the previous section to a given sequence of CDMs. The first step
is to derive the FEs from the time series of miss distances and covariance
matrices in the CDMs. In accordance with DSt, we make no prior assumption on
the underlying distribution of the CDMs and, instead, we consider that each
CDM is drawn from an unknown set of probability distributions. The assumption
is that the value of the uncertain vector $\mathbf{u}$ in each CDMs is a
sample drawn from the set of unknown distributions. We make use of the DKW
inequality, [20], to build an upper and lower bound to the set starting from
the empirical Cumulative Distribution Function (eCDF) derived from the
sequence of CDMs.

Given a sequence of CDMs and the eCDF of each of the components of the
uncertain vector $\mathbf{u}$, the DKW inequality defines the following upper
and a lower bounds

$F_{n}(x)-\sqrt{\frac{\ln{\frac{2}{\delta}}}{2n}}\leq\mathcal{F}(x)\leq
F_{n}(x)+\sqrt{\frac{\ln{\frac{2}{\delta}}}{2n}}$ (4)

around the eCDF $F_{n}(x)$ (dashed green lines in Fig. 2(b)), given $n$ CDMs
and the confidence level $1-\delta$ that the exact distribution
$\mathcal{F}(x)\in F_{n}(x)\pm\varepsilon$, where
$\varepsilon=\sqrt{\frac{\ln{\frac{2}{\delta}}}{2n}}$.

Note that expression Eq. 4 implies that for an infinite number of observations
$\mathcal{F}(x)=F_{n}(x)$. However, in the following, we will show that in
real sequences not all CDMs follow the same distribution. Convergence to a
single distribution is, therefore, plausible for a single sequence with
consistent measurements and propagation model. Furthermore, $F_{n}(x)$ would
converge to a delta function if each observation returned the same mean and
covariance and the propagation model would not introduce any variability or
nonlinearity.

From the confidence region defined by the DKW bands, it is possible to build a
probability box, or p-box, [24, 25, 26], for each of the components of
$\mathbf{u}$. A p-box is a set of all CDFs compatible with the data, that is,
the bounded region containing all distributions from where the set of samples
may have been drawn, [25]. The upper and lower bounds of the p-box are
monotonic non-decreasing functions, ranging from 0 and 1, so that
$\underline{\mathcal{F}}(x)\leq\mathcal{F}(x)\leq\overline{\mathcal{F}}(x)$,
with $\underline{\mathcal{F}}(x)$ and $\overline{\mathcal{F}}(x)$ the upper
and lower bounds of the p-box for a given variable $x$, [24].

In this work, the p-box bounds are computed from the CDF of a weighted sum of
univariate Gaussians, each one centred at one of the samples. More formally
the assumption is that $\mathcal{F}(x)$ can be approximated by:

$\mathcal{F}(x)\sim\mathcal{P}(x)=\int_{-\inf}^{\inf}\sum_{i}^{n}w_{i}\mathcal{N}(x_{i},\sigma_{i};x)\,dx,$
(5)

with $x_{i}$ the realisations of the uncertain variable $x$, $w_{i}$ a weight
associated with each sample, and $\sigma_{i}$ the variance of the Gaussian
distribution associated with the ith-sample. See Fig. 2(a) for an illustrative
example. Implicitly, it implies that each sample presents some uncertainty
which is modelled with a Gaussian distribution (grey lines in Fig. 2(a)). This
distribution represents the confidence in the sample’s value. By doing so, we
admit that when we observe a sequence of CDM we cannot tell from which exact
distribution that sequence is drawn. This is consistent with the available
sequences of real CDMs and the approach adopted by CNES to model the
uncertainty in the covariance realism (see section 4.2.2).

In order to define the limits of the p-box, the two free parameters on each
Gaussian distribution on the weighted sum, $w_{i}$ and $\sigma_{i}$, must be
computed by solving the optimisation problems:

$\left\\{\begin{array}[]{l}\overline{\mathcal{P}}(x)=\max_{w_{i},\sigma_{i}}\mathcal{P}(x;w_{i},\sigma_{i})\\\
\underline{\mathcal{P}}(x)=\min_{w_{i},\sigma_{i}}\mathcal{P}(x;w_{i},\sigma_{i})\end{array}\right.s.t.\left\\{\begin{array}[]{l}\overline{\mathcal{P}}(x)\leq\min(1,F_{n}(x)+\varepsilon)\\\
\underline{\mathcal{P}}(x)\geq\max(0,F_{n}(x)-\epsilon)\end{array}\right.\text{,}$
(6)

where $\overline{\mathcal{P}}(x),\underline{\mathcal{P}}(x)$ are the upper and
lower bounds of the p-box, respectively (red dashed-pointed line in Fig.
2(c)). An approximation to
$\overline{\mathcal{P}}(x),\underline{\mathcal{P}}(x)$ can be computed by
finding the values of $w_{i}$ and $\sigma_{i}$ in Eq. 5 that best fit the
upper and lower DKW bands:

$\left\\{\begin{array}[]{l}\overline{\mathcal{P}}(x)\approx\overline{P}(x)=\text{fit}_{w_{i},\sigma_{i}}(F_{n}(x)+\epsilon)\\\
\underline{\mathcal{P}}(x)\approx\underline{P}(x)=\text{fit}_{w_{i},\sigma_{i}}(F_{n}(x)-\epsilon)\end{array}\right.\text{.}$
(7)

Eq. 7 gives the upper and lower bounds on the probability of realising a
particular value of the uncertain vector $\mathbf{u}$ but the definition of a
set of intervals for each component of $\mathbf{u}$ requires first the
definition of the range of each component. Eq. 5 suggests that each p-box has
infinite support. However, this would lead to an inconvenient infinite range
for variance and miss distance. Instead, in the following we define the more
practical interval $[\underline{x},\overline{x}]$ such that:

$\int_{\underline{x}}^{\infty}w_{1}\mathcal{N}(x_{1},\sigma_{1};x)\,dx=0.99,\;\;\int_{-\infty}^{\overline{x}}w_{n}\mathcal{N}(x_{n},\sigma_{i};x)\,dx=0.99.$
(8)

It is important to note that the assumption is that the miss distance and each
component of the covariance can be treated independently. This is generally
not the case, however, the independence assumption in this paper leads to a
more conservative set of focal elements that cover the space of realisations
of the uncertainty vector. Although this can lead to over-conservative
decisions, it is deemed to be acceptable in the case of high-risk events with
little available information.

(a)

(b)

(c)

(d)

Figure 2: Example of intervals derivation form the eCDF. (a) eCDF (solid
blue), individual sample’s Gaussian pdf distributions (solid grey), pdf of the
sum of Gaussian distributions for the eCDF fit (solid orange) (b) eCDF (solid
blue), DKW bands (dashed green), fitted eCDF with weighted sum of Gaussian
distributions (dashed-pointed orange). (c) eCDF (solid blue), DKW bands
(dashed green), p-box optimising the weighted sum of Gaussian distributions
(dashed-pointed red), 1% and 99% percentiles (vertical pointed black lines).
(d) eCDF (solid blue), p-box (dashed-pointed red), 1 $\alpha$-cut 2 intervals’
Pl and Bel (dashed blue), 7 $\alpha$-cuts 8 intervals’ Pl and Bel (dashed
black). Dotted thin horizontal lines for the $\alpha$-cuts: light blue at 0.5
for the 2 intervals partition, grey lines spaced 0.125 for the 8 intervals
partition.

###  3.1 Scaling of the CDMs

The approach described in previous sections assumes that every CDM has the
same relative importance and no additional source of information is available
to qualify each individual CDM. However, as the $t2TCA$ decreases, so does the
effect of the uncertainty on the true shape of the distribution on the impact
plane and the effect of model uncertainty in the propagation. Fig. 3(a) shows
the normalised determinant of multiple sequences of covariance matrices taken
from the database of the ESA’s Collision Avoidance Kelvins Challenge, [27, 8].
The database contains 13,152 sequences of CDMs of some of the Low Earth Orbit
(LEO) satellites monitored by the ESA Space Debris Office (SDO). The figure
shows that one can fit the simple exponential law
$y^{\prime}=e^{-3t^{\prime}}$ to the magnitude of the determinant (red line in
the figure). However, one cannot simply trust later CDMs due to large
uncertainty in each individual sequence. Thus, we propose the following fit
for each individual sequence:

$y^{\prime}=Ce^{At^{\prime}}+B\hskip 15.0ptA,B,C\geq 0,$ (9a)
$y^{\prime}=\frac{\|\mathbf{\Sigma}\|}{\max_{CDMs}(\|\mathbf{\Sigma}\|)}$ (9b)
$t^{\prime}=\frac{(1-\max_{CDMs}(t2TCA))}{(\min_{CDMs}(t2TCA)-max_{CDMs}{t2TCA})}$
(9c)

Once the parameters $A$,$B$ and $C$ are fitted to the samples from a given
sequence, the following weight is associated with each CDM in that sequence:

$w_{CDM_{i}}=\frac{1}{y^{\prime}(t2TCA_{CDM_{i}})}$ (10)

The weight is applied to each sample in the eCDF used to compute the DKW
bounds: the probability mass associated with each sample is re-scaled by a
factor $w_{CDM_{i}}$. See Fig. 4 where the eCDF of $\mu_{\xi}$ for an example
with 5 observations is shown both with samples equally weighted (dashed red)
or having applied the weighting law described above (blue).

(a)

(b)

Figure 3: Fitting law: (a) $y^{\prime}=e^{-3t^{\prime}}$ (thick red line) and
the dimensionless covariance determinant for a number of sequences of CDMs
(thinner lines), (b) Fitted law (dashed-pointed red) of a single CDM sequence
(dashed-pointed black).

This approach results in a scaling of the probability mass associated with the
CDMs but still allows the quantification of highly uncertain CDMs since there
is no filtering process. The reason is that, with no information on trusted
sources or individual CDMs, one cannot make any assumption on which CDM is
more credible.

Figure 4: eCDF for $\mu_{\xi}$ weighing the samples (blue) and with samples
equally weighted (dashed red).

###  3.2 $\alpha$-cuts and DSt Structures

Once a p-box is defined, the intervals for each component of $\mathbf{u}$ are
derived from a series of equally spaced $\alpha$-cuts, light blue and grey
dotted horizontal thin lines in Fig. 2(d). Each $\alpha$-cut creates interval,
[28, 29]:

$[x_{\alpha},x^{\alpha}]=\left\\{x\,|\,\mathcal{F}(x)\geq\alpha\right\\}\text{.}$
(11)

The intersection with the upper bounds in the p-box defines the lower limit of
the interval, and the intersections with the lower bound define the upper
limit of the interval. The number of intervals is equal to the number of cuts
plus one, and the bpa associated with each interval, assuming the cuts are
evenly spaced, is equal to the inverse of the number of cuts. The intervals
and their bpa will define an envelope around the p-box (blue and black dashed
lines in Fig. 2(d)). The greater the number of $\alpha$-cuts, the closer the
envelope will be to the p-box, but the more computationally expensive is the
computation of Bel and Pl. From the intervals associated with each component
of $\mathbf{u}$ one can compute the FE $\gamma_{i}$ and their associated
$bpa(\gamma_{i})$ by performing the Cartesian product of all the intervals and
associated bpas. Once the FE and bpas are computed, the Pl, Bel of $PoC\geq
PoC_{0}$ are computed with Eq. 2 (see Fig. 5) and the conjunction event is
classified according to Table 1.

Figure 5: Plausibility and Belief of $PoC\geq PoC_{0}$. Black: 1 $\alpha$-cut
(two intervals) per variable, 32 FEs. Blue: 7 $\alpha$-cut (eight intervals)
per variable, 32768 FEs. Solid lines: belief. Dashed lines: plausibility.
Dotted purple vertical line: $PoC_{0}$.

Even in this case, we implicitly maintained the assumption that variables are
independent, although it is not true that the components of the miss distance
and of the covariance are all independent. Approaches to address dependencies
already exist in the literature, see [30], and will be considered in future
works. The independence assumption has two implications: i) the uncertainty
space $U$ is an outer approximation of the space of all distributions of
$\mathbf{u}$ and ii) some focal elements might not contain any sample of
$\mathbf{u}$. The combination of the two generally leads to over-conservative
results. Thus, in order to partially recover the interdependence between
uncertain quantities, yet coherent with DSt, a $bpa=0$ is assigned to all
empty FEs and their bpa, coming from the Cartesian product, is evenly
distributed to the rest of FEs so that $\sum_{i}bpa(\gamma_{i})=1$.

##  4 Numerical experiments

In this section, some numerical tests are presented. The aim is to show the
applicability of the methodology presented in previous sections and compare
its outcome to the decisions made in past real cases by actual satellite
operators: European Space Operations Centre (ESOC) and CNES.

###  4.1 Parameter Tuning

The methodology proposed in this paper requires the prior definition of the
values of two thresholds: $Pl_{0}$ and $A_{0}$. These two thresholds should be
tuned by analysing a large dataset of conjunction events with known outcomes.
However, in every database of CDMs available to the authors, the number of
provable Class 1 and 2 conjunctions is very small or zero.

Since $A_{0}$ does not affect Class 4 and 5, which depend only on $Pl_{0}$,
but influences the number of True Positives (actual collisions) and False
Positives (no-collisions believed to be collisions), one can define $Pl_{0}$
first and then use $A_{0}$ to quantify the degree of uncertainty in the class
associated to an event. According to the classification in Table 1, the
expected outcome is that low values of $Pl_{0}$ would increase the number of
events classified as Class 1 or 2, reducing, at the same time, the number of
False Negatives (collisions believed to be no-collisions) and increasing the
amount of TPs. If this is combined with high values of $A_{0}$, the chances of
detecting all high-risk events are high, but at the cost of increasing the
number of FPs. If instead, $A_{0}$ is low, more events will be classified as
uncertain (Class 0 and 3). On the contrary, a higher value of $Pl_{0}$ would
reduce the false alerts, FPs, but at the risk of increasing the number of FNs.

This paper used the DSt structure to set a value for $Pl_{0}$. If there is at
least one FE supporting $PoC>PoC_{0}$, it means that there exists at least one
piece of evidence suggesting that the PoC can be correct. This piece of
evidence may correspond to an extreme event with low probability. Following
this idea, we propose the value $Pl_{0}=\min_{i}(bpa(\gamma_{i}))$. This
implies that even a PoC that corresponds to a rare event in the generation of
a CDM is considered to be plausible. The value of $A_{0}$ is selected by
balancing the number of TPs and FPs. The idea is to try to reduce the number
of FPs by reclassifying them as uncertain cases and presenting the level of
such uncertainty to the operator. A low value of $A_{0}$ implies that the
operator accepts very little uncertainty in the sequence of CDM, which reduces
the number of FPs but potentially classifies some TPs as uncertain. On the
other hand, a greater value of $A_{0}$ implies that the operator is very
conservative and accepts to treat a number of FPs as TPs. Thus, the decision
to execute a CAM is related to the confidence of the operator in the quality
of the CDMs. For highly uncertain sequences of CDMs, a low $A_{0}$ is
recommended, but if the quality of the CDMs is high, a higher $A_{0}$ should
be used.

In the following, rather than selecting the value of the area threshold
$A_{0}$, we select the value of the normalised area
$A^{*}_{0}=\\{0,0.05,0.1,0.15,...,0.95,1\\}$, where $A^{*}_{0}$ is the
fraction of the maximum possible area between the $Bel$ and $Pl$ curves, that
is, when Bel drops to zero at the minimum value of PoC, $\underline{PoC}$, and
Pl remains equal to one until $PoC=1$. In this tuning exercise the area is
computed by taking the lower limit $\underline{PoC}=10^{-30}$ for the PoC as
this is the lowest value computed from all the sequences of CDMs in our
database. For all the first four tests in this paper, we will use a value of
$A_{0}^{*}=0.1$ that allows one to clearly differentiate Event 1 from Events 3
and 4 in the following section. In the last test, we will present the
sensitivity of the number of recommended CAMs to the value of $A_{0}^{*}$.

###  4.2 Comparison Against SDO and CNES

The results in this section will show a comparison between the CARA performed
with the proposed evidence-based method and the decisions made by real
operators in a selected number of real cases. The two operators considered in
this study are the ESA’s SDO and CNES. Each of them has a different approach
to conjunction analysis. Four real conjunction events are analysed and the
different operational approaches are compared.

For all examples the values of the thresholds are reported in in Table 2.

Table 2: Threshold values. Threshold | Units | Value
---|---|---
$T_{1}$ | days | 3
$T_{2}$ | days | 5
$PoC_{0}$ | - | $10^{-4}$
$Pl_{0}$ | - | $1/\\#FE$
$A^{*}_{0}$ | - | $0.1$
$\underline{PoC}$ | - | $10^{-30}$

The evolution of the normalised area gap between the $Pl$ and $Bel$ curves, or
$A_{Pl,Brel}$, over time, for all four cases can be found in Figure 6, where
$A^{*}_{Pl,Bel}=A_{Pl,Bel}/max(A_{Pl,Bel})$ is the normalised area between
curves, $A_{Pl,Bel}$, defined in Eq. 3. The Figure confirms that an
$A^{*}_{0}=0.1$ is appropriate to differentiate between cases like Event 1
from cases like Event 3 and 4. All four cases are presented in more detail in
the following subsections.

Figure 6: Evolution of the normalised $A_{Pl,Bel}$ over time, for Events 1 to
4.

####  4.2.1 Space Debris Office Conjunction Risk Assessment

The approach followed by the ESA SDO is probability-based, relying mainly on
the value of the PoC computed with the information from the CDMs, or the PoC
included in the CDM. The following quote may summarise the conjunction risk
assessment process adopted by the SDO: For a given close approach the last
obtained CDM, including the computed risk, can be assumed to be the best
knowledge we have about the potential collision and the state of the two
objects in question. In most cases, the Space Debris Office will alarm control
teams and start thinking about a potential avoidance manoeuvre 2 days prior to
the close approach in order to avoid the risk of collision, to then make a
final decision 1 day prior, [27]. Nevertheless, each mission monitored by the
SDO has specific operational constraints (i.e. the time needed to prepare and
execute the manoeuvre) and will have its own risk and time thresholds,
$PoC_{0}$ and $T_{1}$. The time threshold $T_{1}$ is generally 2 or 3 days
away from TCA. At that point the mission team is informed about the possible
collision, and a final decision is usually made (when possible) 1 day from
TCA, [27]. The risk threshold $PoC_{0}$ is determined statistically based on
the overall collision risk and the annual frequency of close approaches,
trading off the ignored risk and the avoided risk by selecting the risk
threshold at the cost of an expected number of annual manoeuvres, see [1].
Generally, for missions in the LEO regime, a threshold of $PoC_{0}=10^{-4}$
leads to a risk reduction of around 90% at the expense of 1 to 3 manoeuvres
per year, with current levels of traffic. However, a lower threshold, around
$10^{-5}$, may be considered to ensure sufficient time to prepare a collision
avoidance manoeuvre in the case of escalated events, [1].

Following this approach, the SDO escalates an event when the PoC of the last
CDM is bigger than the threshold. Escalating an event means that further and
more detailed analyses are required. If the risk is still above the threshold
at the decision time, a CAM is designed in cooperation with the mission team,
whose final decision will be made based on the value of PoC included in the
last CDM received before the go/no-go decision time. More detailed information
on the CARA process of the SDO can be found in [1]. For the first three events
in this subsection, only CDMss from the MiniCat database were considered.

##### Event #1

This event represents a high-risk scenario provided by the ESA SDO. The
uncertain geometry in the impact plane, with the whole sequence of CDMs and
the PoC evolution are displayed in Fig. 7. Events with PoC above the threshold
for times to TCA greater than $T_{1}$ make the event escalate, that is, they
are further analysed and possible alerts to the mission’s team can be
triggered, while high-risk CDMs received in the last 72 hours trigger a CAM
procedure.

(a)

(b)

Figure 7: CDM information for example in Event #1: High-risk event. (a)
Uncertain ellipses in the sequence of CDMs. Green ellipses correspond to
earlier CDMs, and red ellipses to later CDMs. (b) Evolution of the PoC in the
CDMs with the time to the TCA. Blue solid line: PoC; orange dashed line: PoC
threshold.

From Fig. 7(b), one can see that the PoC remains high along the whole
sequence. Even if at the beginning it was below the threshold, its proximity
to $PoC_{0}$ along with the upward trend made the operator escalate the event.
The PoC threshold was violated within the last few days before TCA, which led
to a CAM execution to reduce the risk of the event.

We applied our evidence-based methodology to this case by following the
approach presented in Section 3. The DKW bands were computed assuming a
confidence interval $\delta=0.5$. The CDMs were weighted according to the
exponential law in Eq. 9. Fig. 8 shows the fitting law after having received
all the CDMs (red) along with the value of the combined covariance matrix
determinant, for the whole sequence (black). For the fitting law in Fig. 8(b),
the value of the dimensionless parameters in Eq. 9a after having received the
whole sequence are: $A=1.0752,B=0.9811,C=0.001716$. Note that the value of the
parameters varies with the number of CDMs received to better fit the
covariance determinant evolution up to that time.

(a)

(b)

Figure 8: Fitting law to weight the CDMs after having received the whole
sequence in Event #1: High-risk event. (a) Solid black line: value of the
determinant from the CDMs, dashed red line: fitting law of the covariance
matrix determinant. (b) Weight of the CDMs as a function of the time to the
TCA.

We repeated the same analysis with different numbers of $\alpha$-cuts per
uncertain variable: $\\#\alpha\text{-cuts}=\left\\{1,2,3,4,5,7\right\\}$.
These cuts led to a number of intervals per variable equal to
$\\#\text{intervals}=\left\\{2,3,4,5,6,8\right\\}$, which translated into a
number of FEs $\\#\text{FE}=\left\\{32,243,1024,3125,7776,16807\right\\}$,
respectively. The Pl and Bel curves for the PoC, for each number of cuts, is
presented in Fig. 9, after having received the whole sequence of CDMs.

Figure 9: Pl and Bel of the PoC after having received the whole sequence of
CDMs Event #1: High-risk event. Solid vertical grey line: maximum PoC in the
sequence, dashed vertical grey line: PoC of last CDM, pointed purple line: PoC
threshold. For the rest of the colours: Belief in solid lines and Plausibility
in dashed lines. Black: 1 $\alpha$-cut per variable (2 intervals per variable,
32 FEs), blue: 2 $\alpha$-cuts, red: 3 $\alpha$-cuts, green: 4 $\alpha$-cuts,
purple: 5 $\alpha$-cuts, yellow: 7$\alpha$-cuts.

Fig. 9 shows that, although the increasing number of $\alpha$-cuts provides a
more refined set of curves, their shape and values varies only slightly. In
this case, the Bel and Pl curves overlap for most values of PoC except for a
small interval around the $PoC_{0}$, as it could be expected both, from the
uncertainty geometry in Fig. 7(a) and the values of the PoC in Fig. 7(b).
Since the information in the CDM is coherent across the whole sequence, the
gap between the Pl and Bel curves is small.

Fig. 10 shows the classification, purple solid line, as a function of the time
to the TCA from the last received CDM. The figure shows also the PoC directly
computed from the CDM.

Initially, the event is classified as Class 4 and rapidly falls to Class 5,
since there is little evidence supporting a higher PoC. However, at 2.5 days
from TCA, the PoC consistently grows above the threshold. Given the little
uncertainty in the sequence of CDMs the event is reclassified as Class 1 and a
CAM is recommended.

This is the same decision finally taken by the SDO. As seen in Fig. 9, the
support for a high value of PoC is high and the gap between the curves (level
of uncertainty) is very small. Thus, the recommended action in the last days
prior to the encounter would be to implement a manoeuvre to reduce the risk of
a collision.

Figure 10: Collision risk assessment for Event #1: High-risk event. Solid
narrow lines: evidence-based classification with different number of
$\alpha$-cuts: $\\#\alpha\text{-cuts}=\left\\{1,2,3,4,5,7\right\\}$ (note that
they overlap each other, so only $\\#\alpha\text{-cuts}=7$ is visible in
solind purple). Crossed-solid line: PoC in the CDMs used by SDO for
assessment. Horizontal thick lines: evidence approach safety bands: green, low
risk-uncertain boundary; yellow, uncertain-high risk boundary; red, mid term
high risk-long term high risk boundary. Dashed black line: Risk threshold
(overlapping evidence-based high-risk boundary). Vertical black line: decision
time threshold $T_{1}$.

##### Event #2

A similar analysis was done for the Low-risk conjunction event illustrated in
Fig. 11, also provided by the ESA SDO. Opposite to the previous event, in this
case, the PoC remains well below the threshold, so no alert is required to be
triggered and no CAM is required to be designed or executed.

(a)

(b)

Figure 11: CDM information for example in Event #2: Low-risk event. (a)
Uncertain ellipses in the sequence of CDMs. Green ellipses correspond to
earlier CDMs, and red ellipses to later CDMs. (b) Evolution of the PoC in the
CDMs with the time to the TCA. Blue solid line: PoC; orange dashed line: PoC
threshold.

The evidence-based analysis was performed using the same parameters as before:
$\delta=0.5$ for the DKW bands, with a different number of $\alpha$-cuts:
$\\#\alpha\text{-cuts}=\left\\{1,2,3,4,5,7\right\\}$ per variable. The final
set of CDMs was weighted with the exponential fitting law Eq. 9a using the
following parameters: $A=0.6049,B=5.0896,C=0.4518$. The fitting law (red) and
the combined covariance matrix determinant in the CDMs (black) appear in Fig.
12. Note the convergence in the second half of the sequence.

(a)

(b)

Figure 12: Fitting law to weight the CDMs after having received the whole
sequence in Event #2: Low-risk event. (a) Solid black line: value of the
determinant from the CDMs, dashed red line: fitting law of the covariance
matrix determinant. (b) Weight of the CDMs as a function of the time to the
TCA.

In Fig. 13, the corresponding Pl and Bel curves on the value of PoC after
having received all the CDMs of the event are shown. Again, increasing the
number of $\alpha$-cuts makes the curves smoother and shows a converging
trend, but does not change the overall confidence in the value PoC. The
maximum value of PoC with some supporting evidence is well below the
threshold, indicating that the event can be deemed to be safe. However, the
left-most part of the Bel and Pl curves shows a significant gap. This can be
explained by the fact that the ellipses are not too different from each other
(Fig. 11(a)) and they tend to converge to a single ellipse for the later CDMs,
as shown in Fig. 11(b). Thus, the initial information content in each CDMtends
to support lower values of PoC, which explains the lower value of Bel on the
left of the graph. However, due to the concentration of information around the
later CDMs, the big drop both in Pl and Bel occurs at PoC$\sim 10^{-7}$.

Figure 13: Pl and Bel of the PoC after having received the whole sequence of
CDMs Event #2: Low-risk event. Solid vertical grey line: maximum PoC in the
sequence, dashed vertical grey line: PoC of last CDM, pointed purple line: PoC
threshold. For the rest of the colours: Belief in solid lines and Plausibility
in dashed lines. Black: 1 $\alpha$-cut per variable (2 intervals per variable,
32 FEs), blue: 2 $\alpha$-cuts, red: 3 $\alpha$-cuts, green: 4 $\alpha$-cuts,
purple: 5 $\alpha$-cuts, yellow: 7$\alpha$-cuts.

Finally, the conjunction assessment for the whole sequence is shown in Fig.
14. This event displays a greater uncertainty with respect to the previous
scenario, but values of the PoC greater than $10^{-7}$ have no supporting
evidence and $Pl=Bel=0$. Thus, the event is initially classified as Class 4
($t2TCA>T_{1}$) and then dropped to Class 5 ($t2TCA\leq T_{1}$) for the whole
sequence, meaning that no further action should be taken by the operator. This
is the same decision made by the SDO.

Figure 14: Collision risk assessment for Event #2: Low-risk event. Solid
narrow lines: evidence-based classification with different number of
$\alpha$-cuts: $\\#\alpha\text{-cuts}=\left\\{1,2,3,4,5,7\right\\}$ (note that
they overlap each other, so only $\\#\alpha\text{-cuts}=7$ is visible in solid
purple). Crossed-solid line: PoC in the CDMs used by SDO for assessment.
Horizontal thick lines: evidence approach safety bands: green, low risk-
uncertain boundary; yellow, uncertain-high risk boundary; red, mid term high
risk-long term high risk boundary. Dashed black line: Risk threshold
(overlapping evidence-based high-risk boundary). Vertical black line: decision
time threshold.

##### Event #3

This last event is affected by a significant level of uncertainty. The
encounter geometry and the evolution of the PoC in the CDMs are shown in Fig.
15. Despite the initial high risk, with values of PoC close to the threshold,
the final decision of the SDO was not to take any further action. This
decision was dictated by the later values of the PoC, that were all
consistently lower than the initial ones, and considerably below $PoC_{0}$.

(a)

(b)

Figure 15: CDM information for example in Event #3: Uncertain event. (a)
Uncertain ellipses in the sequence of CDMs. Green ellipses correspond to
earlier CDMs, and red ellipses to later CDMs. (b) Evolution of the PoC in the
CDMs with the time to the TCA. Blue solid line: PoC; orange dashed line: PoC
threshold, vertical dashed grey line: TCA.

The evidence-based analysis was performed with the same parameters as before:
$\delta=0.5$ for the DKW bands. The exponential fitting law Eq. 9a to weight
the CDMs, after having received the whole sequence, had the following
parameters $A=0.7917,B=7.1471,C=0.1858$ and is shown in Fig. 16 (red line)
along with the covariance matrix determinant (black line).

(a)

(b)

Figure 16: Fitting law to weight the CDMs after having received the whole
sequence in Event #3: Uncertain event. (a) Solid black line: value of the
determinant from the CDMs, dashed red line: fitting law of the covariance
matrix determinant. Vertical dashed grey line: TCA. (b) Weight of the CDMs as
a function of the time to the TCA. Vertical dashed grey line: TCA.

The Pl and Bel curves for the PoC were computed for different $\alpha$-cuts:
$\\#\alpha\text{-cuts}=\left\\{1,2,3,4,5,7\right\\}$. The curves are shown in
Fig. 17. In this case, there is a significant gap between Pl and Bel for all
the values of PoC for which $Pl>0$. This uncertainty (or level of disagreement
between CDMs) can be seen in Fig. 15(a), which shows the variety of the
uncertainty ellipses from the beginning of the sequence to the last CDMs. In
this case the supporting evidence that a value of $PoC>PoC_{0}$ is plausible
does not go to zero but the gap between the Pl and Bel curves suggests that a
further analysis is required although the value of Pl is low and Bel is zero.

Figure 17: Pl and Bel of the PoC after having received the whole sequence of
CDMs Event #3: Uncertain event. Solid vertical grey line: maximum PoC in the
sequence, dashed vertical grey line: PoC of last CDM, pointed purple line: PoC
threshold. For the rest of the colours: Belief in solid lines and Plausibility
in dashed lines. Black: 1 $\alpha$-cut per variable (2 intervals per variable,
32 FEs), blue: 2 $\alpha$-cuts, red: 3 $\alpha$-cuts, green: 4 $\alpha$-cuts,
purple: 5 $\alpha$-cuts, yellow: 7$\alpha$-cuts.

Fig. 18 shows the result of the classification: the event starts at Class 2,
given the potential high risk suggested by the initial CDMs but quickly drops
to Class 3 ($t2TCA>T_{1}$) because of the level of uncertainty and is finally
classified as Class 0 (for $t2TCA\leq T_{1}$). In this case, our approach
would suggest a further analysis due to the non-zero plausibility of a high
PoC and a high difference between Pl and Bel, while the decision made by the
SDO was to take no further action. The more prudent recommendation coming from
our classification system would lead to a further inspection of the Pl curve
with the realisation that the supporting evidence is small, albeit not zero.

Figure 18: Collision risk assessment for Event #3: Uncertain event. Solid
narrow lines: evidence-based classification with different number of
$\alpha$-cuts: $\\#\alpha\text{-cuts}=\left\\{1,2,3,4,5,7\right\\}$ (note that
they overlap each other, so only $\\#\alpha\text{-cuts}=7$ is visible for all
$t2TCA$ in solid purple; $\\#\alpha\text{-cuts}=1$ in solid blue and
$\\#\alpha\text{-cuts}=2$ in solid orange are visible at one $t2TCA$ each).
Crossed-solid line: PoC in the CDMs used by SDO for assessment. Horizontal
thick lines: evidence approach safety bands: green, low risk-uncertain
boundary; yellow, uncertain-high risk boundary; red, mid term high risk-long
term high risk boundary. Dashed black line: Risk threshold (overlapping
evidence-based high-risk boundary). Vertical black line: decision time
threshold $T_{1}$. Vertical dashed grey line: TCA.

####  4.2.2 CNES Conjunction Risk Assessment

In order to compensate for the possible lack of realism of the covariance
matrix at TCA, CNES re-scales both the covariance matrix of the primary and
secondary body with two factors, respectively $k_{p}\in K_{P}$ and $k_{s}\in
K_{S}$. A scaled PoC, called scaled Probability of Collision (sPoC), is
obtained by solving the following PoC maximisation problem (see [10, 11]):

$\left\\{\begin{array}[]{l}sPoC=\max_{k_{p}\in K_{P},k_{s}\in
K_{S}}PoC(\Sigma)\\\ \text{s.t.}\hskip
10.0pt\mathbf{\Sigma}=k_{p}^{2}\mathbf{\Sigma}_{p}+k_{s}^{2}\mathbf{\Sigma}_{s}\end{array}\right.,$
(12)

where $\mathbf{\Sigma}_{p}$ and $\mathbf{\Sigma}_{s}$ are, respectively, the
primary and secondary covariance matrices in a given CDM associated to the
conjunction event under consideration.

The two sets $K_{P}$ and $K_{S}$ are derived, for each sequence of CDMs, under
the assumption that CDMs are samples drawn from an underlying distribution,
and the last CDM contains the most reliable estimation of the position of the
two objects. Thus, by using the last CDMs as a reference, it is possible to
compute the Mahalanobis distance of all previous CDMs from the last one. If
one assumes that the uncertainty in position is Gaussian, the Mahalanobis
distance should follow a $X^{2}$ distribution with 3 degrees of freedom. By
performing a Kolmogorov-Smirnov (KS) test between the distribution of the
computed Mahalanobis distances and the theoretical one, and setting a desired
level of realism, one can define the sets $K_{P}$ and $K_{S}$. More details
can be found in [31].

CNES decision-making is based on both the value of sPoC and a number of
geometric considerations. Events with values of $sPoC>5\cdot 10^{-4}$ are
classified as High-Interest Event, the more risky classification level (red
level). For values of $10^{-4}<sPoC<5\cdot 10^{-4}$, the event is classified
as an Interest Event, the second level of risk (orange level). If the value of
the $sPoC$ is below those thresholds, caution geometric criteria are applied:
miss distance below 1 km or radial distance below 200 m. Note that these
threshold values are the default ones and may differ from mission to mission.
If the CDMs are received 4-5 days before the encounter or earlier, no alerts
are raised independently of the value of sPoC, although the event is placed
under study if some of the above criteria are violated. For later CDMs, alerts
may be raised according to the value of sPoC. Finally, if the high risk
continues after the decision time (usually 2 days before the encounter), a
final decision is made before the TCA.

In the following, we will test our approach on a real close encounter faced by
CNES and compare our classification against the one of CNES.

##### Event #4

This scenario presents a high-risk collision case for a real close encounter
where CNES had to implement a manoeuvre to reduce the risk.

Fig. 19(a) shows the geometry of the event, where the earlier CDMs (green
ellipses) suggested a low PoC, while later CDMs (red and amber ellipses)
suggest a high PoC. Fig. 19(b) shows the PoC and the sPoC. The latter is above
the threshold $10^{-4}$ from the start and progressively increases while the
PoC displays a large variability till about a day before TCA. CNES classified
the event as High-Interest Event, meaning that careful monitoring was
required, starting from the 12th CDM (2.96 days before the TCA). The final
decision to perform a manoeuvre was taken 30 hours before the encounter. Note
that the CDM received about a 1.2 days from TCA indicates a PoC$<10^{-5}$,
well below the risk threshold, while the sPoC indicates a risk above
$10^{-3}$, which aligns better with the last three CDMs received between the
decision time and the CAM execution time).

(a)

(b)

Figure 19: CDM information for example in Event #4. (a) Uncertain ellipses in
the sequence of CDMs. Green ellipses correspond to earlier CDMs, and red
ellipses to later CDMs. (b) Evolution of the PoC in the CDMs with the time to
the TCA. Blue solid line: PoC; dashed-dotted line: sPoC; orange dashed line:
PoC threshold.

The evidence-based analysis was performed following the same approach as for
the SDO cases, with $\\#intervals=\left\\{2,3,4,5,6,8\right\\}$ intervals per
variable and CDM weighted according to the exponential law in Fig. 20.

(a)

(b)

Figure 20: Fitting law to weight the CDMs after having received the whole
sequence in Event #4. (a) Solid black line: value of the determinant from the
CDMs, dashed red line: fitting law of the covariance matrix determinant. (b)
Weight of the CDMs as a function of the time to the TCA.

The Pl and Bel corresponding to the whole sequence of CDM are shown in Fig.
21, and the classification sequence for different numbers of intervals is
shown in Fig. 22. In Fig. 21 one can see that $Pl(PoC_{0})$ is nearly 1, and
$Pl(sPoC)>0$ along the whole time series. In fact, $Pl=0$ at $PoC\sim
10^{-2}$, while $\max(sPoC)=5\cdot 10^{-3}$. However, the gap between the Pl
and Bel curves is very high, indicating a degree of uncertainty in the
sequence of CDMs. This is due to the variability in the CDMs. Thus the event
is classified as Class 0.

Although this event is placed in the same class as Event 3, the supporting
evidence is quite different. Event 4 has a $Pl\approx 1$ and Bel different
from zero at $PoC_{0}$ while Event 3 has $Bel=0$ and $Pl<0.2$ at $PoC_{0}$.
This means that, although in this paper we opted for a very conservative
classification of the events such that both Events 3 and 4 fall in the same
uncertainty class, a simple analysis of the Bel and Pl curves would suggests
that the available evidence for Event 4 supports a high probability of
collision, up to $10^{-2}$ in fact, while for Event 3 the supporting evidence
at $PoC_{0}$ is quite low.

Figure 21: Pl and Bel of the PoC after having received the whole sequence of
CDMs Event #4. Solid vertical grey line: maximum PoC in the sequence, dashed
vertical grey line: PoC of last CDM, pointed purple line: PoC threshold. For
the rest of the colours: Belief in solid lines and Plausibility in dashed
lines. Black: 1 $\alpha$-cut per variable (2 intervals per variable, 32 FEs),
blue: 2 $\alpha$-cuts, red: 3 $\alpha$-cuts, green: 4 $\alpha$-cuts, purple: 5
$\alpha$-cuts, yellow: 7$\alpha$-cuts. Figure 22: Collision risk assessment
for Event #4. Solid narrow lines: evidence-based classification with different
number of $\alpha$-cuts: $\\#\alpha\text{-cuts}=\left\\{1,2,3,4,5,7\right\\}$
(note that they overlap each other, so only $\\#\alpha\text{-cuts}=7$ is
visible in solid purple). Crossed-solid line: PoC in the CDMs used by SDO for
assessment. Horizontal thick lines: evidence approach safety bands: green, low
risk-uncertain boundary; yellow, uncertain-high risk boundary; red, mid-term
high risk-long term high risk boundary. Dashed black line: Risk threshold
(overlapping evidence-based high-risk boundary). Vertical black line: decision
time threshold $T_{1}$.

###  4.3 Statistical Analysis of CAM Executions

After having compared the proposed evidence-based conjunction assessment
approach against real operations on specific cases, in this section we compare
the number CAMs that our evidence-based approach would recommend over a large
number of real conjunctions experienced by a single mission.

The selected mission is the ESA SWARM-A satellite, orbiting in the LEO regime
(circular polar orbit of 87.7 deg at 511 km of altitude), dedicated to
studying the Earth’s magnetic field as part of a constellation of three
satellites. The mission thresholds to trigger conjunction alerts are
$PoC_{0}=10^{-4}$ and $T_{1}=72$ hours. Thus, any satellite with a PoC above
the threshold in the last 3 days would escalate and would require further
analysis, and eventually, a possible CAM design or execution. Nevertheless,
encounters presenting a higher risk or an increasing trend before $T_{1}$ may
be escalated if the operator considers that there is a potential risk for the
mission. Finally, the go/no-go decision is subject to operational constraints:
the time required to design a CAM after receiving the triggering manoeuvre,
the possibility to upload and check the design manoeuvre and the ground
station availability.

The database of CDMs includes alerts from 2015 to 2022, with a total of 36,072
events. Overall, most of the events in the database did not represent a threat
to the satellite, with only 20 representing escalated events. As explained
before, an escalated event is an encounter where the PoC, or the PoC trend,
suggests that the conjunction may be high risk. From those escalated events,
only 2 required a CAM to be executed.

The evidence-based analysis was performed with the same thresholds as the
previous study cases (Table 2): $PoC_{0}=10^{-4}$, $T_{1}=3$ days, $T_{2}=5$
days, $Pl_{0}=1/243$, $A^{*}_{0}=0.1$, with $\underline{PoC}=10^{-30}$, and
$A_{0}=3$. The DKW bands were obtained assuming a confidence interval of
$\delta=0.5$. As shown before, a higher number of $\alpha$-cuts would refine
the Pl and Bel curves, providing closer curves that better represent the
actual epistemic uncertainty. However, this is at the expense of increasing
the computational cost and with limited impact on the final classification.
Thus 2 $\alpha$-cuts (3 intervals) per variable, with a total of 243 FEs per
analysis were used.

Since the evidence-based analysis lacks the real information available in the
actual operation of the satellite that may have affected the operator decision
(for example, the ground station availability or the mission constraints), the
statistics were computed at four decision times: $T_{d}=3$ days to the TCA,
corresponding with the mission time threshold, $T_{1}$; $T_{d}=2$ days to the
TCA, allowing for more data to arrive; $T_{d}=1$ day to the encounter, the
usual go-no go decision time in ESA’s missions, [1]; and the epoch of the last
CDM in the sequence, $T_{d}=0$. For simplicity, we assume that there is no
operational constraint that prevents or modifies the final decision and all
information is, thus, available.

Table 3 includes the results from the analysis, compared with the actual
statistics provided by the SDO. It is important to bear in mind the
differences between the approaches. An event classified as Class 3 or Class 0
(labelled as Uncertain), with the evidence-based approach, would not
correspond, necessarily, to an escalated event, since the meaning is
different: while an escalated event assumes a certain level of risk, a Class 0
or 3, suggests a degree of uncertainty that requires further investigation
before making a final decision. This further investigation might be simply
limited to an inspection of the Bel and Pl curves as in cases 3 and 4 above or
might require additional observations. On the other hand, for all Class 1
events, the recommendation is to perform a CAM.

Table 3: Results from the statistical analysis on the SWARM-A mission, with the SDO approach and the evidence-based approach. Threshold: $PoC_{0}=10^{-4}$,$T_{1}=3$ days, $T_{2}=5$ days, $Pl_{0}=1/243$. Partition with 2 $\alpha$-cuts per variable. Upper tier: $A^{*}_{0}=0.1$ ($A_{0}=3$); middle tier: $A^{*}_{0}=0.5$ ($A_{0}=15$); lower tier: $A^{*}_{0}=0.8$ ($A_{0}=24$). SDO | Evidence-based |
---|---|---
# events |  | $A^{*}_{0}$ | # events | $T_{d}=3$ | $T_{d}=2$ | $T_{d}=2$ | $T_{d}=0$
Total | 36,072 |  | Total | 24,296 | 27,918 | 32,108 | 36,072
Escalated | 20 | 0.1 | Unc. | 120 | 130 | 172 | 293
CAM | 2 |  | CAM | 1 | 2 | 3 | 2
|  | 0.5 | Unc. | 102 | 98 | 107 | 154
|  |  | CAM | 19 | 34 | 68 | 141
|  | 0.8 | Unc. | 95 | 83 | 77 | 75
|  |  | CAM | 26 | 49 | 98 | 220

From the upper tier in Table 3 (with $A^{*}_{0}=0.1$), one can observe that:
i) the total number of events increases with the delay in the decision time
because more CDMs are available for a decision; ii) the number of manoeuvres
proposed by the evidence-based approach is similar to the number of CAMs
proposed by the SDO operators; iii) the evidence-based classification system
found many more uncertain cases than the SDO. The Table shows also the number
of CAMs and uncertain events for $A^{*}_{0}$ equal to 0.5 and 0.8. As
expected, an increase in the values of $A^{*}_{0}$ increases the number of
CAMs and reduces the number of uncertain cases.

Even if the $Pl_{0}$ threshold is quite low, the number of events escalating
to Class 1 remains small. Thus, in this test case, the system is robust enough
to remove false negatives without introducing false alerts. Also, the number
of CAMs remains roughly constant independently of the decision time
(especially, for the selected default value of $A^{*}_{0}=0.1$). On the other
hand, the number of Class 0 events is between 6 and 15 times higher than the
number of escalated events proposed by ESOC. It is here where the evidence-
based system differentiates from the probabilistic approach used by ESOC.
Class 0 events are those with $Pl(PoC_{0})>Pl_{0}$, but are still deemed
uncertain because $A_{Pl,Bel}>A_{0}$. $Pl$ captures all realisations, within
each Focal Element, that correspond to extreme cases, extreme low or extreme
high $PoC$, compatible with the observed sequence of CDMs. Hence, a large
$A_{Pl,Bel}$ with high $Pl$ signifies that there is the evidence that a high
$PoC$ event can occur but is uncertain. As in the case of Event 3, many of
these cases display a low Pl and zero Bel. Others present conflicting CDM,
that cannot be resolved without further observations, or a high Pl for high
PoC values, as in Event 4 but with a low Bel. An example can be seen in Figure
23. The evolution of the combined covariance shows a radical rotation of
nearly 90 degrees at -4 days from TCA. The evolution of the PoC does not
provide any evidence that the covariance had a step change, but remains close
to the threshold limit. The evidence-based approach, instead. shows quite some
uncertainty and maintains a high $Pl$ till the end of the sequence, suggesting
that the event cannot be discarded and requires further analysis.

Note that the percentage of events in this category increases when delaying
the decision. This indicates a growing disagreement among CDMs in the sequence
as the time approaches TCA, an aspect usually overlooked by probabilistic-
based approaches.

(a)

(b)

(c)

Figure 23: Conjunction event with conflicting CDMs: a) evolution of the
relative position distribution on the impact place, b) evolution of the PoC,
c) Pl and Bel curves of the whole CDMs sequence.

##  5 Conclusions

This work presented a methodology to model and quantify the epistemic
uncertainty in a sequence of CDMs, and exploit this quantification to make
robust decisions about conjunction events. The method was tested against real
operations on a number of real scenarios. The key working assumption was that
the value of the miss distance and covariance matrix in each CDM were drawn
from a set of unknown distributions. The DKW inequality was used to build
bounds on this set and derive a set of focal elements, with associated
probability mass supporting a given value of the probability of collision.

The collection of focal elements was used to compute the Pl and Bel on a given
value of the PoC. The Pl at $PoC_{0}$, or $Pl_{0}$ was proposed as a further
criterion to make a decision on the actual severity of a conjunction event,
while the difference between Pl and Bel, or $A_{Pl,Bel}$, was proposed as a
measure of the uncertainty in the quantification of the PoC.

It was found that when the set of CDMs contains coherent information over the
whole time series, the proposed classification system suggests the same
decisions normally made by the ESA SDO. When the sequence of CDMs presents a
higher degree of variability or a degree of inconsistency the proposed
evidence-based approach recommends more conservative decisions compared to the
SDO but also provides the operator with a quantification of the related
uncertainty.

A comparison with the approach used at CNES, based on the concept of sPoC,
showed that the proposed evidence-based approach returns decisions that are
less conservative but, at the same time, provides a higher level of
information on the uncertainty in the decision. By comparing the ESA and CNES
uncertain cases, it was also found that a further inspection of the Pl and Bel
curves offers a way to disambiguate the events as the different evolution of
PoC over time is reflected in a lower or higher value of Pl and Bel.

Finally, a statistical analysis on a database of real encounters of an ESA
mission showed that the number of recommended CAMs is similar but the
evidence-based approach tends to detect a higher number of uncertain cases
that require further analysis.

Although in our analysis no operational constraints were considered, the
number of detected uncertain cases suggests that relying only on the last CDM
may be too optimistic while the scaled PoC approach might be too pessimistic
without a further uncertainty quantification. In relation to the uncertain
cases, different situations can be found which may lead the operator to take
different actions. Further analysis on the treatment of these scenarios should
be taken and a threshold tuning analysis using virtual datasets or a mixed
dataset of real and virtual CDMs may help with this task. The approach
proposed in this paper assumes that no additional information on the CDMs is
available nor that information on the uncertainty in the propagation model or
individual observations can be used. However, if additional information was
available one could improve the quantification of uncertainty of each CDMs and
build better defined p-boxes with tighter bounds.

Future work will need to consider the correlation and interdependence among
variables during the construction of the focal elements and build a more
refined model. Furthermore, the current databases of real CDMs do not
represent a controlled set of events, because the actual outcome is unknown. A
representative synthetic database would greatly help in improving the
classification system. Last but not least, machine learning can be used to
directly classify events from the time series of CDMs. This approach
represents an extension of what was already proposed by the authors and would
improve on current efforts to predict the last CDM with machine learning as it
would embed a quantification of uncertainty in the prediction.

## Acknowledgements

This work was funded by the European Space Agency, through the Open Space
Innovation Platform (OSIP), "Idea I-2019-01650: Artificial Intelligence for
Space Traffic Management".

The authors would like to thank CNES-Toulouse and the DOA/SME/SE Office for
the opportunity to research with them and for sharing really valuable
information with us. More specifically, we would like to thank François
Laporte for the insightful discussions during the approach development. The
authors would like to thank the ESA’s Space Debris Office at ESOC for
providing both very useful data and feedback on this work during the research
stays at their facilities.

## References

  * Merz et al. [2017] Merz, K., Braun, V., Benjamin Bastida, V., Flohrer, T., Funke, Q., Krag, H., and Lemmens, S., “Current collision avoidance service by ESA’s Space Debris Office,” _7 ${}^{\text{th}}$ European Conference on Space Debris_, ESA/ESOC, Darmstadt, Germany, 2017.
  * Newman et al. [2019] Newman, L. K., Mashiku, A. K., Hejduk, M. D., Johnson, M. D., and Rosa, J. D., “NASA Conjunction Assessment Risk Analysis (CARA) updated requirements architecture,” _AAS/AIAA Astrodynamics Specialist Conference_ , Portland, Maine, US, 2019.
  * CCSDS [2013] CCSDS, “Recommended Standard: CCSDS 508.0-B-1. Recommendation for space data system standards. Conjunction data message,” Tech. rep., CCSDS, Washington, DC, USA, Jun. 2013.  https://public.ccsds.org/Pubs/508x0b1e2s.pdf.
  * Aristoff et al. [2014] Aristoff, J. M., Horwood, J. T., Singh, N., and Poore, A. B., “Nonlinear uncertainty propagation in orbital elements and transformation to Cartesian space without loss of realism,” _AAS/AIAA Astrodynamics Specialist Conference_ , San Diego, CA, US, 2014.
  * Cano et al. [2023] Cano, A., Pastor, A., Escobar, D., Míguez, J., and Sanjurjo-Rivo, M., “Covariance determination for improving uncertainty realism in orbit determination and propagation,” _Advances in Space Research. Space Environment Management and Space Sustainability_ , Vol. 72, No. 7, 2023, pp. 2759–2777.  DOI: https://doi.org/10.1016/j.asr.2022.08.001.
  * Pinto et al. [2020] Pinto, F., Acciarini, G., Metz, S., Boufelja, S., Kaczmarek, S., Merz, K., Martínez-Heras, J., Letizia, F., Bridges, C., and Baydin, A., “Towards automated satellite conjunction management with bayesian deep learning,” _AI for Earth Sciences Workshop at NeurIPS_ , 2020.  URL https://nips.cc/virtual/2020/public/workshop_16105.html.
  * Acciarini et al. [2021] Acciarini, G., Pinto, F., Letizia, F., Martínez-Heras, J. A., Merz, K., Bridges, C. P., and Güneş Baydin, A., “Kessler: a machine learning library for spacecraft collision avoidance,” _8 ${}^{\text{th}}$ European Conference on Space Debris_, ESA/ESOC, Darmstadt, Germany, 2021.
  * Uriot et al. [2022] Uriot, T., Izzo, D., Simões, L. F., Abay, R., Einecke, N., Rebhan, S., Martinez-Heras, J. A., Letizia, F., Siminski, J., and Merz, K., “Spacecraft collision avoidance challenge: design and results of a machine learning competition,” _Astrodynamics_ , Vol. 6, No. 2, 2022, pp. 121–140.  DOI: https://doi.org/10.1007/s42064-021-0101-5.
  * Caldas et al. [2023] Caldas, F., Soares, C., Nunes, C., and Guimarães, M., “Conjunction Data Messages for space collision behave as a Poisson process,” _31 ${}^{\text{st}}$ European Signal Processing Conference (EUSIPCO)_, Hesinki, Finland, 2023.
  * Laporte [2014a] Laporte, F., “JAC Software, dedicated to the analysis of conjunction messages,” _SpaceOps 2014 Conference_ , Pasadena, CA, US, 2014a.  DOI: https://doi.org/10.2514/6.2014-1774.
  * Laporte [2014b] Laporte, F., “JAC Software, solving conjunction assessment issues,” _Proceedings of the Advanced Maui Optical and Space Surveillance Technologies Conference (AMOS)_ , Maui, Hawaii, US, 2014b.
  * Tardioli and Vasile [2015] Tardioli, C., and Vasile, M., “Collision and re-entry analysis under aleatory and epistemic uncertainty,” _Advances in Astronautical Sciences_ , Vol. 156, 2015, pp. 4205 – 4220.
  * Delande et al. [2018] Delande, E., Houssineau, J., and Jah, M., “A New Representation of uncertainty for data fusion in SSA Detection and Tracking Problems,” _2018 21st International Conference on Information Fusion (FUSION)_ , Cambridge, United Kingdom, 2018.  DOI: https://doi.org/10.23919/ICIF.2018.8455540.
  * Balch et al. [2019] Balch, M., Martin, R., and Ferson, S., “Satellite conjunction analysis and the false confidence theorem,” _Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences_ , Vol. 475, No. 20180565, 2019\.  DOI: https://doi.org/10.1098/rspa.2018.0565.
  * Greco et al. [2021] Greco, C., Sánchez, L., and Vasile, M., “A robust Bayesian agent for optimal collision avoidance manoeuvre planning,” _8 ${}^{\text{th}}$ European Conference on Space Debris_, ESA/ESOC, Darmstadt, Germany, 2021.
  * Sánchez and Vasile [2021] Sánchez, L., and Vasile, M., “On the use of machine learning and evidence theory to improve collision risk management,” _Acta Astronautica_ , Vol. 181, 2021, pp. 694–706.  DOI: https://doi.org/10.1016/j.actaastro.2020.08.004.
  * Sánchez and Vasile [2022] Sánchez, L., and Vasile, M., “Intelligent agent for decision-making support and collision avoidance manoeuvre design on space traffic management,” _Advances in Space Research. In press_ , 2022.  DOI: https://doi.org/10.1016/j.asr.2022.09.023.
  * Sánchez et al. [2022] Sánchez, L., Stevenson, E., Vasile, M., Rodríguez-Fernández, V., and Camacho, D., “An intelligent system for robust decision-making in the all-vs-all conjunction screening problem,” _3 ${}^{\text{rd}}$ IAA Conference on Space Situational Awareness (ICSSA)_, Tres Cantos, Madrid, Spain, 2022.
  * Helton et al. [2005] Helton, J. C., Oberkampf, W. L., and Johnson, J. D., “Competing failure risk analysis using evidence theory,” _Risk Analysis_ , Vol. 25, No. 4, 2005, pp. 973–995.  DOI: https://doi.org/10.1111/j.1539-6924.2005.00644.x.
  * Dvoretzky et al. [1956] Dvoretzky, A., Kiefer, J., and Wolfowitz, J., “Asymptotic minimax character of the sample distribution function and of the classical multinomial estimator,” _Annals of Mathematical Statistics_ , Vol. 27, No. 3, 1956, pp. 642–669.  DOI: https://doi.org/10.1214/aoms/1177728174.
  * Shafer [1976] Shafer, G., _A Mathematical theory of evidence_ , 1st ed., Princeton University Press, Princeton, NJ, 1976.  ISBN: 9780691100425.
  * Serra et al. [2016] Serra, R., Arzelier, D., Joldes, M., Lasserre, J. B., Rondepierre, A., and Salvy, B., “Fast and accurate computation of orbital collision probability for short-term encounters,” _Journal of Guidance, Control, and Dynamics_ , Vol. 39, 2016, pp. 1–13.  DOI: https://doi.org/10.2514/1.G001353.
  * Greco and Vasile [2021] Greco, C., and Vasile, M., “Robust Bayesian particle filter for space object tracking under severe uncertainty,” _Journal of Guidance, Control, and Dynamics_ , Vol. 45, No. 3, 2021, pp. 481–498.  DOI: https://doi.org/10.2514/1.G006157.
  * Ferson et al. [2023] Ferson, S., Kreinovich, V., Ginzburg, L., Sentz, K., and Myers, D. S., “Constructing probability boxes and Dempster-Shafer structures,” Tech. rep., Sandia National Lab., Albuquerque, NM, United States, 2023.  DOI: https://doi.org/10.2172/809606.
  * Ferson et al. [2007] Ferson, S., Kreinovich, V., Hajagos, J., Oberkampf, W., and Ginzburg, L., _Experimental uncertainty estimation and statistics for data having interval uncertainty_ , Princeton University Press, Sandia National Laboratories (SNL), Albuquerque, NM, and Livermore, CA (United States), 2007, Chap. 4: Descriptive statistics for interval data, pp. 28–87.  DOI: https://doi.org/10.2172/910198.
  * He et al. [2017] He, Y., Mirzargar, M., and Kirby, R. M., “An efficient reliability analysis approach for structure based on probability and probability box models,” _Structural and Multidisciplinary Optimization_ , Vol. 56, 2017, pp. 167–181.  DOI: https://doi.org/10.1007/s00158-017-1659-7.
  * ESA [2019] ESA, “European Space Agency: Kelvins collision avoidance challenge,” https://kelvins.esa.int/collision-avoidance-challenge/home/, 2019.
  * He et al. [2015] He, Y., Mirzargar, M., and Kirby, R. M., “Mixed aleatory and epistemic uncertainty quantification using fuzzy set theory,” _International Journal of Approximate Reasoning_ , Vol. 66, 2015, pp. 1–15.  DOI: https://doi.org/10.1016/j.ijar.2015.07.002.
  * Chojnacki et al. [2007] Chojnacki, E., Baccou, J., and Destercke, D., “Numerical sensitivity and efficiency in the treatment of epistemic and aleatory uncertainty,” _5 th International Conference on Sensitivity Analysis of Model Output_, Budapest, Hungary, 2007.  DOI: https://doi.org/10.1016/j.ijar.2015.07.002.
  * Ferson et al. [2004] Ferson, S., Nelsen, R. B., Hajagos, J., Berleant, D. J., Zhang, J., Tucker, W. T., Ginzburg, L. R., and Oberkampf, W. L., “Dependence in probabilistic modeling, Dempster-Shafer theory, and probability bounds analysis,” Tech. rep., Sandia National Lab., United States, Oct. 2004.  DOI: https://doi.org/10.2172/919189.
  * Stroe et al. [2021] Stroe, I. F., Stanculescu, A. D., Ilioaica, P. B., Blaj, C. F., Nita, M. A., Butu, A. F., Escobar, D., Tirado, J., Bija, B., and Saez, D., “AUTOCA autonomous collision avoidance system,” _8 ${}^{\text{th}}$ European Conference on Space Debris_, ESA/ESOC, Darmstadt, Germany, 2021.

  *[PoC]: Probability of Collision
  *[CDM]: Conjunction Data Message
  *[TCA]: Time of Closest Approach
  *[CDMs]: Conjunction Data Message
  *[DSt]: Dempster-Shafer theory of evidence
  *[DKW]: Dvoretzky–Kiefer–Wolfowitz
  *[Bel]: Belief
  *[Pl]: Plausibility
  *[ESA]: European Space Agency
  *[CNES]: Centre National d’Etudes Spatiales
  *[CARA]: Conjunction Assessment Risk Analysis
  *[HBR]: Hard-Body Radius
  *[bpa]: basic probability assignment
  *[FE]: Focal Element
  *[bpas]: basic probability assignment
  *[FEs]: Focal Element
  *[CDF]: Cumulative Distribution Function
  *[CAM]: Collision Avoidance Manoeuvre
  *[eCDF]: empirical Cumulative Distribution Function
  *[CDFs]: Cumulative Distribution Function
  *[LEO]: Low Earth Orbit
  *[SDO]: Space Debris Office
  *[ESOC]: European Space Operations Centre
  *[TPs]: True Positive
  *[FPs]: False Positive
  *[FNs]: False Negative
  *[CAMs]: Collision Avoidance Manoeuvre
  *[sPoC]: scaled Probability of Collision
  *[KS]: Kolmogorov-Smirnov