Dark matter searches and energy accumulation and release in materials

Sergey Pereverzev 1$\star$

1 Lawrence Livermore National Laboratory, California, USA

*<EMAIL_ADDRESS>
14th International Conference on Identification of Dark Matter
Vienna, Austria, 18-22 July 2022 10.21468/SciPostPhysProc.?

## Abstract

Efforts to identify dark matter by detecting nuclear recoils produced by dark
matter particles reveal low-energy backgrounds of unknown origin in different
types of detectors. In many cases, energy accumulation and delayed burst-like
releases of stored energy could provide an explanation. These dynamics follow
Prigogine‘s ideas on systems with energy and the general Self-Organized
Criticality scenario. We compare these models with properties of excess
backgrounds in cryogenic solid-state detectors, relaxation processes in
glasses and crystals, our observations of delayed luminescence in NaI(Tl), and
make predictions for more phenomena present in these systems and in
superconducting photon detectors and qubits. Experiments to create accurate
phenomenological models are needed.

##  1 Introduction

A plausible resolution of the dark matter problem would be the direct
detection of dark matter particles. Expectations are that rare nuclear recoils
produced by these particles can be observed in sensitive detectors operating
underground at low background radiation conditions. Low-energy recoils also
can be caused by atmospheric and solar neutrinos- an effect thought to put a
sensitivity limit on terrestrial searches for low-energy recoils caused by
dark matter particles. On the other hand, as low-energy sensitivity improves,
many detectors start to see a large number of not-well-identified low-energy
events. Noble liquids dual-phase detectors can detect sub-keV recoils where
very few electrons and photons are produced [1]. Recent progress in solid-
state low-temperature detectors allows the detection of events with energy
deposition below 10 eV, see for example background spectra for the MINER
detectors in [2]. The spectra of events observed in many solid-state detectors
[2, 3] and Noble Liquid detectors (some review is given in [4]) often rise
sharply towards low energies, and the number of these low-energy events is
larger than expected neutrino or dark matter interactions. These backgrounds
rise with ionization load and are larger for detectors operating above ground.
In solid-state detectors [2,3], background rises with mechanical stress in the
detector material. We can assume that we are dealing with rare, condensed
matter or chemical events leading to small energy releases inside our
detectors. There are various ways energy can be pumped into and stored inside
materials. For materials and systems out of the thermal equilibrium, where
interactions of excitations, defects, and other objects or sub-systems bearing
the excess energy are present, avalanche-like energy relaxation events
(energy-release events) and other complex dynamic phenomena can take place.
Thus, we hypothesize that stored energy relaxation events can mimic rare low-
energy interactions with particles.

Systems with energy flow were first studied by Ilia Prigogine and later by
self-organized criticality theory. Effects of this type remain insufficiently
investigated and could be unfamiliar to the dark matter community. We provide
a brief introduction and examples.

In this paper, we will discuss solid-state detectors. We present our
experimental result on energy accumulation and release as delayed luminescence
in NaI(Tl), make predictions of new effects in low-temperature solid-state
detectors, and suggest experiments to look for relaxation avalanches at much
smaller energies – down to 10 meV – in superconducting IR photon (quantum)
sensors.

Possible mechanisms of energy or charge accumulation and releases in Noble
Liquid dual-phase detectors are discussed in [4]. Accumulation of unextracted
electrons on the liquid-gas interface can lead not only to the production of
parasitic signals, but possibly to Wigner crystallization of surface-bound
electrons, and suppression of electron extraction for small signals produced
below the liquid surface. This discussion will be published elsewhere [5].

##  2 Excess low-energy backgrounds

The presence of low-energy ionization events with the production of very few
electrons (1, 2, 3, 4,…8) and the number of events rising sharply to a single
electron limit was reported in many noble-liquid dual-phase detectors deployed
for dark matter particles searches starting with XENON 10 experiment [6], than
XENON 100 [7], XENON 1T [8], and Dark Side 50 (50 kg liquid Argon target) [9]
detectors (see also Fig.2 in [4]). These ionization events cannot be caused by
thermal fluctuations- ionization energies of Xe and Ar are too high. Low-
energy particles cannot penetrate deep inside detectors because of self-
shielding by pure Xe (Ar) liquid, and low-angle Compton scattering of gammas
necessary required observation of larger-angle scattering events at a rate
that is not present. On the other hand, the energy required for ionization can
be already present in the detector in form of meta-stable excited molecules,
chemical radicals, or trapped ions [4]. If energy or trapped charge releases
can take the form of small avalanches – then the resulting event spectra could
be of observed type – as we discuss below.

Energy can accumulate in the materials due to residual and cosmogenic
radioactivity. Not all energy is transformed into heat and luminescence
immediately following ionization events- defects, trapped ions, chemical
radicals, and other long-leaved excitations can be accumulating in the
materials; an increase of temperature after exposure to ionizing radiation can
lead to release of accumulated energy in the form of thermally-stimulated
luminescence and other related phenomena [10]. David Nygren has suggested that
this stored energy is responsible for low-energy background events in the
NaI(Tl) scintillators [11]. Detection of several luminescence photons during
the sub-µs interval is considered an event in NaI(Tl) scintillator, and pulse-
shape discrimination is used to select events resembling luminescence pulses
produced by particles in DAMA-LIBRA [12] and similar experiments (see, for
example, [13]).

In low-temperature solid-state detectors [2,3], researchers are looking for
events that are spikes of the temperature of the target crystal, detection of
hot phonon bursts by superconducting transition edge sensors placed at the
target crystal surface, detection of a small current pulse when an electric
field is applied to the target crystal, detection luminescence bursts with
photon detectors surrounding target crystal- and combinations of these effects
as additional criteria to select events produced by low-energy interactions
with particles. Importantly, for many detectors, starting at energies below 1
keV, the obtained energy spectra rise sharply towards the detector thresholds.
[2]. There is a growing amount of evidence that at least part of these events
is produced by the thermo-mechanical stress in target crystals [3].

We will discuss below that mechanical stress, exposure to ionising radiation,
environmental changes as temperature, pressure, electric and magnetic fields,
exposure to light, IR, microwaves, sounds and vibrations can deposit excess
energy into materials at low temperatures. This leads to hypothesis that
presence of excess energy in material or detector and interactions between
energy-bearing states can lead to correlations in delayed energy releases and
excess low-energy backgrounds.

##  3 Prigogine’s systems with energy flow; self-organized criticality

For a system with energy flow and away from thermal equilibrium, Ilia
Prigogine [14] postulated several general principles, like the formation of
dissipative systems (this can be a pattern of convection or sequence of
chemical reactions), transitions from havoc to order, and generation of
complexity. Prigogine ideas are important for understanding the origin of life
and the functioning of the live cell. Still, they are applicable to non-
organic systems where internal interactions can lead to correlations in energy
dissipation processes, self-organization, self-reproduction of structures,
etc. Unfortunately, for many important detector applications, interactions
inside materials are not known in sufficient detail to build ab initio
theoretical models.

Self-Organized Criticality theory (SOC) [15,16] analyzed the results of
computer dynamics simulation for large multi-particle systems with known
interactions between particles. For the system where avalanche-like relaxation
takes place (a sand pile when more material is added to the top is an example)
some common features can be observed: the spectrum of relaxational events
(avalanches) is decreasing with event energy polynomially (not exponentially),
so catastrophic events are possible; power noise spectrum is close to 1/f
(pink nose), pumping energy into the system by small quanta can result in
large relaxation events, and reinforcement of relaxation on small scale can
lead to suppression of large relaxation events (placing the sand pile on a
vibrating plate). Several examples are known in condensed matter physics where
the experimental statistic is close to model calculation predictions: crack
formation in materials under stress [17] and production of quantized vortexes
in superconductors when the critical current density is reached [18].

Thus, when we see the spectrum of background events rising toward low
energies, or a noise power spectrum close to 1/f is present, we can suspect
that a SOC-like dynamic is present. Then we need to ask if energy can be
stored in our material or system, what are the sources of this energy, and if
energy-bearing states can interact. In experiments, we can try to increase or
decrease the energy influx into our system or try quenching energy-bearing
states. Pumping energy into the system by small quanta and looking for the
appearance of “up-conversion-like events” is another possibility. The
spontaneous transition of qubits into an excited state or the appearance of a
hot photon in a cold superconducting resonator could be examples of actual up-
conversion events when energy is pumped into the materials by small (sub-gap)
quanta.

##  4 Excess energy in glasses

At temperatures below the glass transition, amorphous (disordered) materials
are out of thermodynamic equilibrium, and relaxation toward a lower energy
state (like the formation of small crystals in a glass) can continue for
hundreds of years. Relaxation processes in the glass state (response to force
or other stimuli) became long and dependent on the internal state of the
material, i.e., history-dependent. Relaxation can involve changing internal
stress and other parameters, like dielectric constant; both applications of
force or electric field can cause long relaxation processes. When the force or
electric field is applied and then removed, glass can be brought into a state
with higher internal energy, i.e., energy can be pumped into the system with
glass-like relaxation properties by application of electric or magnetic
fields, stress, etc.

At low temperatures, many subsystems in materials demonstrate complex and
history-dependent relaxation properties. As examples, we can name charges and
spins localized on boundaries and interphases in SQUIDs [19], the motion of
charges in dielectric substrate probed by single-electron transistors on the
surface [20], magnetic moments of impurities in superconductors forming spin
glass [21].

Glass properties become more complex at Ultra-Low Temperatures (below mK);
organic and non-organic glasses start to demonstrate the memory effect-
dielectric constant “remember” the electric field in which material was cooled
to low temperatures [22].

A dominant theoretical approach to glasses is based on a tunneling two-level
systems model developed in the 1970s [23,24]. The TLS model is also widely
used to describe noise and decoherence in superconducting sensors and qubits
[25]. The TLS model postulates the existence of objects in material that are
multi-particle configurations with two closely arranged minimal energy
states/configurations such that tunneling in between these
states/configurations is possible. Each TLS is described with two parameters:
energy differences between the two adjacent minima and tunneling
coupling/probability. Two-level systems were assumed to be non-interactive
[23,24], and the universality of properties of different glasses originate in
the universality of TLS parameters distribution in materials. The TLS models
successfully describe many properties of glasses, but, as Anttony Leggett
points out [26], the TLS model may not be the only model suited to describe
these properties. The microscopic models for TLS with required parameters are
still debated (see discussion in [25]).

In Prigogine’s approach interactions between energy-bearing states lead to new
emerging phenomena; interactions are the necessary conditions (but not
sufficient) for avalanches in SOC model. Authors of [22] also highlighted the
"importance of interactions between active defects in glasses." The above
discussion demonstrates that the description of glasses and decoherence with
TLS may be incomplete- it likely is missing phenomena emerging from
interactions between energy-bearing states. Observation of energy up-
conversion effects when energy is pumped into the material by small quanta
would be important demonstration of the role of internal interactions.

##  5 Delayed luminescence in NaI(Tl)

The Saint-Gobain company, the main manufacturer of NaI(Tl) crystal
scintillators, has warned on web site and in other documentation that mild
exposure of crystal to UV light may lead to the appearance of low rate (few
per second) scintillation pulses resembling irradiation with few keV energy
electrons, and these pulses disappear by itself in several hours or days. This
looks like an up-conversion effect we expect to see for the SOC-like dynamics.
As no other information is provided by the company, we tried to reproduce
these experiments [27]. We also expected [5] that after-luminescence can be
suppressed by exposure to red or IR light, as such exposure suppresses
thermally stimulated luminescence in alkali-metal halides [28]. We have found
[28] that exposure to UV light results in delayed luminescence response
lasting several days, and that exposure to energetic electrons from a Co-60
source leads to delayed luminescence lasting several hours, see Fig.1. We also
confirmed that single energetic particle events and muons are causing a slight
increase in background photon emission; to see this effect one needs to do
averaging of response (traces) for many energetic particles. The delayed
luminescence signals we observed after UV exposure was mostly random photon
emission processes. Saint-Gobain researchers were not discussing the existence
of this random flux of uncorrelated single photons and how particle-like
pulsed were selected. Right after exposure to UV light delayed luminescence in
our experiments was too intense to be separated onto well-defined events
(photon bursts). As intensity became close to equilibrium background photon
emission, more data is required to check that photon bursts present in excess
of what one can expect from a random coincidence in the number of photons
detected during a short time interval. We used coincidences of photon
detection by right and left PMTs to trigger data acquisition and applied a
pulse-shape discrimination analysis similar to the analysis used by the DAMA-
LIBRA experiments [12]. The leakage into the “particle-like” domain was
dependent on the left-right coincidence interval, the length of the photon
counting window, and a choice of “separate event” criteria, i.e., the absence
of photons for some time before and after the event. Though these cuts
strongly reduce the number of “candidate particle-like events”, we can see
leakage into the “particle domain” [27]. A more detailed analysis with an
exact replication of DAMA-LIBRA algorithms is required to quantitatively
characterize this leakage under different conditions.

(a)

(b)

Figure 1: The delayed light intensity in NaI(Tl) [27] was monitored over time
following UV exposure (a, solid black line) and Co-60 irradiation (a, dashed
blue line). Three hours after the UV exposure, the crystal was exposed to red
light (a, solid red line); and a pronounced decrease in the afterglow
intensity was observed. The shaded green box corresponds to the red light
irradiation time. Control runs were performed, in which exposure with the UV
or red light sources was omitted, but the same procedure was used for source
placement as in the runs with exposure. This check confirmed that the
experimental procedure had a negligible contribution to the observed trend (a,
dashed lines). We also observed delayed light following large energy
depositions in the crystal (b). Note that we omit the data point at the zero
time, which is the time of the large ionization event (b).

We have demonstrated (see Fig.1) that exposure to red light can strongly
suppress delayed luminescence signals. Delayed luminescence intensity and
decay time are dependent on the type of energetic particles (see also [29]);
paper [30] demonstrates that the decay time of delayed luminescence after
exposure to 60Co strongly depends on temperature. This suggests that other
environmental factors, including pressure, electric and magnetic fields,
mechanical stress, changes in the microwave or RF backgrounds, or vibrations
level also can affect delayed and, possibly, fast luminescence responses.

Unfortunately, DAMA-LIBRA and other experiments trying to check /replicate
observed yearly modulation in the rate of particle-like low-energy events are
not collecting information about the intensity of random background photon
emission. Depth of modulation (if present) and the relative phases to
modulations of muon flux, solar neutrino flux, and the DAMA-LIBRA background
oscillations would be of interest. In-phase variation of random photon
emission with the DAMA-LIBRA oscillations could indicate the presence of a
single environmental factor affecting both signals. Low-energy Solar neutrino
and/or low-mass dark matter particles can have insufficient energy to produce
luminescence photons. Still, it can produce hot phonons with energies
sufficient to trigger the release of stored energy. A.K Drukier even suggested
detection on low-energy nuclear recoils based on the release of energy stored
in micro-explosives [31]. Thus, analysis of all possibilities for modulations
can be complex. Our current model for the immediate and delayed luminescence
responses could be insufficiently accurate: it could be missing photon
bunching and/or modulations by environmental effects (most likely temperature
[31]).

##  6 Phenomenology comparison

Low-temperature solid-state detectors use different single-crystal sensors
like Si, Ge, SiO2, CaWO4, Zn, etc., cooled to 10-50 mK temperatures.

Microscopic mechanisms of small inelastic deformation (flow) of single-crystal
samples were extensively studied (especially Si and Ge) at room temperatures
under different load conditions, including inhomogeneous loading and micro-
indentations, see, for example, [32,33]. The flow demonstrates small steps
consisting of transformations in microscopic volumes of material. These
transformations are changes in crystallographic structure, chemical
transformations, the appearance of the twin boundaries, sliding plains,
appearance, and the motion of dislocations. These transformations are
dissipating events and should result in heat and hot phonon productions. It is
also known that the appearance and motion of defects, dislocations, etc., in
dielectrics, semiconductors, and metal samples can be accompanied by emission
of photons and electrons (surface electrons) from the sample [34,35]. These
relaxation processes can continue after the force that causes deformation is
removed. This has a strong similarity with relaxation processes in glasses.
Importantly, the long relaxation process in glasses can be caused not only by
inelastic deformation (flow) but by other, more “gentle” actions. An example
is level-burning experiments in organic glasses at low temperatures [36]; here
laser pulse is producing a notch transparency window in a glass, and longtime
observation ( year) of increasing absorption in this notch transparency window
at low temperatures was required to demonstrate not exponential but polynomial
relaxation time dependence. The non-interacting TLS model predicts that
relaxation is exponential.

We know that energy accumulates in solids during exposure to ionizing
radiation- and we have effects of thermally stimulated luminescence, electron
emission from the surface, and thermally stimulated conductivity in irradiated
samples. Releases of stored energy due to thermal activation at ambient
temperature and due to tunneling should also lead to delayed luminescence,
surface electron emission, and production of current carriers, or
quasiparticles. We see this as delayed luminescence in NaI(Tl). Relative
numbers of different energy-bearing states produced by inelastic deformation,
(thermomechanical stress), or by specific types of ionizing radiation will
differ. On the other hand, there are energy transfer processes between
different energy-storing states and configurations, states bearing excessive
energy and states responsible for photon and electron emission, and
quasiparticle production. We observed that exposure to a red light suppresses
both TSL and delayed luminescence, which indicates that states responsible for
the energy storage in both cases are identical or have many similarities.

The intensity of TSL and TSEE in Xe crystal can decrease in order of magnitude
when the crystal is annealed before irradiation [37]. One can expect similar
effects of defects on TSL, TSEE, and on delayed luminescence, electron, and
quasiparticle emission in other materials.

Authors of [3] introduce the term “micro fracturing” to explain the increase
in particle-like events and quasiparticle production with increased
thermomechanical stress in low-temperature solid-state detectors. We can state
this differently: relaxation of mechanical stress includes step-like small
transformation at room temperature and at low (10-50 mK) temperature.

The next questions are if relaxation processes can still contain step-like
transitions in two limiting cases. One case is when the stored energy is
decreasing, and the system is nearing an equilibrium state. To some extent,
this is equivalent to a small number of defects produced by ionizing radiation
or by UV light. The other case is lower temperatures and the presence of low-
energy “excitations” - as mentioned above, many systems in materials
demonstrate glass-like properties at low temperatures. Our tentative answers
are “yes” in both cases. These answers follow Prigogine’s ideas and SOC
theory. It could be possible to check these predictions experimentally.

One possibility is to check for photon bunching in NaI(Tl) delayed
luminescence after exposure to UV light when the material is closing to the
equilibrium (low background luminescence) state, we have discussed this above.
In low-temperature solid-state detectors, one can look for delayed signals
appearing after exposure to ionizing radiation, UV or visible light. We expect
that bursts of photon, phonon, and quasiparticle emission could be present.
These experiments are better to try on samples with low mechanical stress
after long storage at low temperatures in a low radioactivity underground
laboratory. Low-temperature experiments with NaI(Tl) where both luminescence
and temperature spikes are recorded are of high interest as they could
demonstrate an absence of keV scale heat production when small photon emission
bursts are detected.

##  7 Testing for SOC-like dynamics in superconducting devices

Superconducting nanowire single photon detector (SNSPD) is a new rapidly
developing IR photon sensors technology demonstrating in comparison with other
superconducting photon sensors technologies the fastest photon counting rate,
the lowest dark counts, along with high energy sensitivity. Absorption of an
IR photon in a superconducting nanowire carrying small DC results in breaking
superconductivity in a small section of the nanowire and heat production in
this normal section by the applied current. The normal section of the nanowire
wire starts to grow, and a voltage peak appears across the nanowire.
Importantly, there is no energy dissipation inside the superconducting
nanowire by the small readout current while the detector is waiting for a
photon, and thermo-mechanical stress energy is small and likely relaxation
time after cooling down is fast coming to stop for thin (several nm) and
narrow nanowire. This means that energy pumping into nanowire and substrate
vanishes when this detector is waiting for a photon arrival, so SOC-like
dynamic should be suppressed in these devices (we discussed this idea in
[38]). The low dark count rate allows us to look for SOC-like dynamics and
effective energy up-conversion effects in these devices: one can expose the
SNSPD to a low-intensity flux of microwave photons with photon energy below
the superconducting gap in the nanowire. Then photons cannot break Cooper
pairs in nanowires and can only produce low-energy excitations in a substrate
or sub-gap excitations in the superconductor. An increase in the dark counts
would indicate energy up-conversion-like events in the device, and observation
of dark count dependence on the frequency of microwave radiation can provide
some spectroscopy of low-energy states in the device materials which are
involved in SOC-like dynamics. One can change electron concentration in a
nanowire by applying an electric field perpendicular to the substrate, and
this will affect the superconducting transition temperature and critical
current. By lowering the temperature and critical current simultaneously, one
can try to increase the energy sensitivity of SNSPD. This also will make tests
for SOC-like dynamics more sensitive. Photon detection at longer wavelengths
is interesting for space microwave astronomy and searches for axions.

##  8 Discussion

We compared properties of excess low-energy backgrounds in dark matter
detectors, relaxation processes in glasses, and Prigogine’s ideas about
systems with energy flow. We came to the hypothesis that processes of energy
accumulation by “interacting excitations” in materials and releases of this
stored energy can be present in all detectors and cause events of burst-like
emission of photons, phonons, or quasiparticles, emission of surface
electrons, etc. At the same time, our current knowledge of these processes and
internal interactions are insufficient for model-independent dark matter
searches: we may not be able to exclude beyond a reasonable doubt these
processes as an explanation for observed low-energy events in our detectors.

In other words, a slight shift of paradigm is required. In addition to nuclear
and particle physics effects that produce the dominant part of backgrounds in
dark matter particle searches, we need to pay attention also to possible non-
equilibrium thermodynamic effects. Intuitive expectations based on equilibrium
thermodynamics are not working well here: we are custom to energy transfer
cascades from high-energy excitations to low-energy excitations, but here we
can find cascades producing large energy events or excitations from small
energy excitations.

We have insufficient knowledge of excitations in materials and their
interactions for ab initio calculations/modeling dynamics of complex systems
away from thermal equilibrium. Moreover, we expect to find new effects based
on the comparison of phenomenology we see in different systems. Looking for
these new effects will help to build more accurate phenomenological models and
chart boundaries where simplified models can work. A correct understanding of
material processes should help to select better materials and readout
techniques to improve detectors’ sensitivity for elastic coherent neutrino
scattering and dark matter particles, or/and improve limitations that direct
detection experiments put on dark matter particle models.

It is commonly accepted that muon or other energetic particle can cause
correlated errors in multi-qubit superconducting quantum processors. Excess
backgrounds events described in [2,3] have sufficient energy to cause quantum
errors and correlated quantum errors: these events emit energetic phonons and
photons, and relaxation avalanches could be non-local. Moreover, microwave
pulses used to control qubits are depositing energy into materials, thus
potentially leading to delayer relaxation events and possibly up-conversion
events. Probabilities of such events could be studied experimentally for
different materials and device designs to improve coherence time of QI
devises.

Our general conclusion is that searches for low-energy interactions with
neutrinos or hypothetical dark matter particles could be more efficient if we
in parallel acquire and analyze data on energy accumulation and release
effects in materials and devices. Studies of non-equilibrium thermodynamics
effects using tools of particle physics, condensed matter, and QI techniques
would benefit all these disciplines. This invites wider collaboration between
HEP and condensed matter scientists, and we argue that joint research programs
between funding agencies (like HEP and BES divisions of the DOE) are required.

## Acknowledgements

This project is supported by the U.S. Department of Energy (DOE) Office of
Science/High Energy Physics under Work Proposal Number SCW1508 awarded to
Lawrence Livermore National Laboratory (LLNL). LLNL is operated by Lawrence
Livermore National Security, LLC, for the DOE, National Nuclear Security
Administration (NNSA) under Contract DE-AC52-07NA27344. LLNL-PROC-840806

## References

  * [1] B.G. Lenardo, J. Xu, S. Pereverzev, O.A. Akindele, D. Naim, J. Kingston, A. Bernstein, K. Kazkaz, M. Tripathi, C. Awe, L. Li, J. Runge, S. Hedges, P. An, P. S. Barbeau, Low-energy physics reach of xenon detectors for nuclear-recoil-based dark matter and neutrino experiments, Phys. Rev. Lett. 123, 231106 (2019), https://doi.org/10.1103/PhysRevLett.123.231106.
  * [2] Adari, P., at al., EXCESS workshop: Descriptions of rising low-energy spectra, (2022), https://doi.org/10.48550/arXiv.2208.02790.
  * [3] Biekert, A., at al., Stress Induced Source of Phonon Bursts and Quasiparticle Poisoning, (2022), https://doi.org/10.48550/arXiv.2208.02790.
  * [4] S. Pereverzev, Detecting low-energy interactions and the effects of energy accumulation in materials, Phys. Rev. D 105, 063002 (2022), https://doi.org/10.1103/PhysRevD.105.063002.
  * [5] S. Pereverzev, What surfaces in operation of noble liquid dark matter detectors, submitted to proceedings of the conference Light Emission in Noble Elements (LIDINE 2022), https://doi.org/10.48550/arXiv.2212.12969 (2022).
  * [6] P. Sorensen et al., Lowering the low-energy threshold of xenon detectors, Proc. Sci., IDM2010 (2011) 017 https://doi.org/10.48550/arXiv.1011.6439 .
  * [7] E. Aprile, et al., Low-mass dark matter search using ionization signals in XENON100, Phys. Rev. D. 94, 092001 (2016), https://doi.org/10.1103/PhysRevD.94.092001.
  * [8] E. Aprile, et al., “Light Dark Matter Search with Ionization Signals in XENON1, Phys. Rev. Lett. 123, 251801 (2019), https://doi.org/10.1103/PhysRevLett.123.251801.
  * [9] P. Agnes, I. F.M. Albuquerque, T. Alexander, A. K. Alton, G. R. Araujo, D.M. Asner, M. Ave et al., Low-Mass Dark Matter Search with the DarkSide-Experimen, Phys. Rev. Lett. 121, , 081307 (2018), https://doi.org/10.1103/PhysRevLett.121.081307.
  * [10] Reuven Chen and Stephen McKeeve, book: Theory of Thermo-luminescence and Related Phenomena, World Scientific Publishing Co., Singapore (1997) , ISBN 9810222955, https://doi.org/10.1142/2781.
  * [11] David Nugren, A testable conventional hypothesis for the DAMA-LIBRA annual modulation, https://doi.org/10.48550/arXiv.1102.0815 (2011).
  * [12] R. Bernabei et al., The DAMA/LIBRA apparatus, Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 592(3), 297-315 (2008), https://doi.org/10.1016/j.nima.2008.04.082
  * [13] Adhikari, G., et al., Induced annual modulation signature in COSINE-100 data by DAMA/LIBRA analysis method, (2022) https://doi.org/10.48550/arXiv.2208.05158.
  * [14] Ilya Prigogine, Time, structure and fluctuations", Science 201, 777-785 (1978), 10.1126/science.201.4358.777.
  * [15] Per Back, Chao Tang, and Kurt Weisenfeld, Self-Organized Criticality: An Explanation of 1/f Noise, Phys. Rev. Lett., 59, 381, (1987) https://doi.org/10.1103/PhysRevLett.59.381
  * [16] Can Chen and Per Back,, Self-Organized Criticality, Scientific American, 264, 46-53, (1991)
  * [17] G. Caldarelli, F. D. Di Tolla, and A. Petri, Self-organization and annealing disorder in a fracturing process, Phys. Rev. Lett., 77, 2503 (2996) https://doi.org/10.1103/PhysRevLett.77.2503.
  * [18] V. K. Vlasko-Vlasov, U. Welp, V. Metlushko, and G. V. Crabtree, Experimental test of the self-organized criticality of vortices in superconductors, Phys. Rev. B, 69, 140504 (2004) https://doi.org/10.1103/PhysRevB.69.140504.
  * [19] S. K. Choi, D. H. Lee, S. G. Louie, and J. Clarke, Localization of metal- induced gap states at the metal-insulator interface: origin of flux noise in SQUIDs and superconducting qubits, Phys. Rev. Lett., 103, 197001 (2009) https://doi.org/10.1103/PhysRevLett.103.197001.
  * [20] A. Pourkabirian, M. V. Gustafsson, G. Johansson, J. Clarke, and P. Delsing, Nonequilibrium Probing of Two-Level Charge Fluctuators Using the Step Response of a Single-Electron Transistor, Phys. Rev. Lett., 113, 256801 (2014) https://doi.org/10.1103/PhysRevLett.113.256801.
  * [21] V.M. Galitski, A.I. Larkin, Spin glass versus superconductivity, Phys. Rev. B, 66, 064526 (2002), https://doi.org/10.1103/PhysRevB.66.064526.
  * [22] S. Rogge, D. Natelson, and D. Osheroff, Evidence for the importance of interactions between active defects in glasses, Phys. Rev. Lett., 76, 3136 (1996), https://doi.org/10.1103/PhysRevLett.76.3136.
  * [23] P. W. Anderson, B. I. Halperin, C. M. Varma, Anomalous low-temperature thermal properties of glasses and spin glasses, The Philosophical Magazine: A Journal of Theoretical Experimental and Applied Physics 25, 1-9 (1971), https://doi.org/10.1080/14786437208229210.
  * [24] W. A. Phillips, Two-level states in glasses, Rep. Prog. Phys., 50, 165723 (1987), 10.1088/0034-4885/50/12/003.
  * [25] C. Muller, J.H. Cole, J. Lisenfeld, Towards understanding two-level-systems in amorphous solids: Insights from quantum circuits, Reports on Progress in Physics, 82, 124501 (2019), 10.1088/1361-6633/ab3a7e.
  * [26] A.J. Leggett, D.C. Vural, "Tunneling two-level systems" model of the low-temperature properties of glasses: are "smoking gun" test possible?, The Journal of Physical Chemistry B, 117, 12966-12971 (2013), https://doi.org/10.1021/jp402222g .
  * [27] F. Sutanto, J. Xu, S. Pereverzev, A. Bernstein, Production and suppression of delayed light in NaI(Tl), (2022), https://doi.org/10.48550/arXiv.2208.09974 .
  * [28] M. P. Fontana, H. Blume, and W. J. van Sciver, Properties of exciton states in NaI I. The intrinsic photoluminescence, Phys. Status Solid, 29, 158 (1968), https://doi.org/10.1002/pssb.19680290117 .
  * [29] J C Robertson and J G Lynch, The Luminescent Decay of Various Crystals for Particles of Different Ionization Density, Proc. Phys. Soc., 77, 751 (1961), 10.1088/0370-1328/77/3/326 .
  * [30] C. R. Emighand, L. R. Megill, he Long-Lived Phosphorescent Components of Thallium-Activated Sodium Iodide, Phys. Rev., 93, 1160 (1954), https://doi.org/10.1103/PhysRev.93.1190 .
  * [31] . A.K. Drukier, R. L. Fagaly and R. Bielski, Towards a new class of detectors for dark matter and neutrinos, International Journal of Modern Physics A 29, 1443006 (2014), 10.1142/9789813226609_0036 .
  * [32] Liangchi Zhang, Irena Zarudi, Towards a deeper understanding of plastic deformation in mono-crystalline silicon , International Journal Mechanical Sciences, 43, 1985-1996 (2001), https://doi.org/10.1016/S0020-7403(01)00024-8 .
  * [33] Bradby, J.E., Williams, J.S., Wong-Leung, J., et al. Mechanical deformation in silicon by micro-indentation, Journal of Materials Research, 43, 15 (2001), https://doi.org/10.1557/JMR.2001.0209 .
  * [34] W.D. Von Voss and F.R. Brotze, Electron emission from plastically strained aluminum , J. of Applied Physics, 30, 1639 (1959), https://doi.org/10.1063/1.1735027 .
  * [35] W.J. Baxter, A study of plastic deformation by exoelectron emission , Vacuum, 22, 571-575 (1972), https://doi.org/10.1016/0042-207X(72)90028-0 .
  * [36] Kharlamov, B.M., Studies of relaxation processes in organic glasses at low temperatures using the hole-burning spectroscopy technique: Verification of applicability limits of the model of two-level systems, Optics and Spectroscopy, 91, 461–46 (2001), https://doi.org/10.1134/1.1405229 .
  * [37] E. V. Savchenko, I. V. Khyzhniy, G. B. Gumenchuk, A. N. Ponomaryov, and V. E. Bondubey, Relaxation emission of electrons and photons from rare-gas solids: Correlation and competition between TSL and TSEE, Phys. Status Solidi C, 4, 1088 (2007), https://doi.org/https://doi.org/10.1002/pssc.200673809 .
  * [38] Sergey Pereverzev, Gianpaolo Carosi, Viacheslav Li, “Superconducting Nanowire Single-Photon Detectors and effect of accumulation and unsteady releases of excess energy in materials (2022), https://doi.org/10.48550/arXiv.2204.01919.


#  On-orbit No-contact Anomaly Debug Procedure for the CuPID Cubesat
Root cause investigation

Emil A. Atz
Brian M. Walsh
Connor J. O’Brien
Boston University
Center for Space Physics
Mechanical Engineering

###### Abstract

The CuPID CubeSat Observatory was a 6U cubesat launched into low-Earth orbit
with a ride-share opportunity in Fall 2021. The mission was supported by
NASA’s Heliophysics division and motivated scientifically with the objective
to image X-rays produced in the magnetosphere. After launch, the team was
unable to communicate with the spacecraft. This document presents the testing
and analysis of attempts to contact the spacecraft and investigation to the
likely cause of failure with the radio system.

Keywords: Cubesat, no-contact, anomaly, on-orbit test

## Introduction

### Mission summary

The Cusp Plasma Imaging Detector (CuPID) CubeSat Observatory was developed by
a multi-disciplinary team at Boston University, NASA Goddard Space Flight
Center, Johns Hopkins University, Drexel University, Merrimack College, and
the University of Alaska, Fairbanks. The project was supported through NASA’s
Heliophysics Technology and Instrument Development for Science (H-TIDeS)
program. The mission was motivated scientifically to study temporal and
spatial features of magnetic reconnection by imaging plasma density structures
in soft X-rays in the magnetospheric cusps. CuPID was funded in 2016 for an
estimated launch in 2020, though delay in launch manifest and the CoVID-19
pandemic delayed launch until 2021.

The launch as a ride-share with the LandSat-9 mission was supported through
NASA’s CubeSat Launch Initiative (CSLI). The spacecraft was delivered to the
launch vehicle in July 2021 and launched on September 27, 2021 into a nearly
sun-synchronous orbit (97.6∘ inclination) at 550 km altitude.

Figure 1: Two students install CuPID into the ESPA (EELV Secondary Payload
Adapter) ring CubeSat deployer during integration with launch vehicle
components.

### Document overview

With CubeSats of today being a system of systems, we first describe the CuPID
mission and critical elements of the satellite important to the no-contact
investigation. Then, the anomaly testing sections detail the possible issues
inside these flight critical elements and the steps taken to test, and remedy
them where possible.

## CuPID Mission Design

To meet its science objectives, the CuPID cubesat carried two co-aligned
science instruments with fixed geometries. At high latitudes, the three-axis
stabilized spacecraft adjusted its attitude to point the science instruments
radially outward (zenith) and image the magnetospheric cusps. More about the
mission design can be found in Walsh et al, 2021 [1]. The instruments included
a wide field-of-view X-ray telescope, micro-dosimeter suite and body mounted
magnetometer utilized for attitude determination and magnetic field
monitoring. More details on the instruments and calibrations are found in Atz
et al, 2022 [2].

Figure 2: Top: CuPID’s avionics system and enclosure on workbench without
wiring harness. The magnetometer is mounted on the backside of the diving
board. The batteries supply the EPS which distributes power to the rest of the
sub-systems. The flight computer receives data from the GPS and ADACS to
control the spacecraft pointing. Ground commanding and telemetry is
transmitted through the UHF radio. Bottom: CuPID mounted to a vibration table
during environmental testing. Elements to note include CuPID’s largest solar
panel with 22 cells, the stowed radio antenna, and covered star tracker
aperture leading to the avionics inside.

### Mission Critical Hardware and Software

CuPID used an avionics system developed by Adcole Maryland Aerospace (AMA)
that incorporated a number of components built by the company and some
integrated from other vendors. The avionics enclosure and sub-systems are
shown in Figure 2. The avionics included Adcole Maryland Aerospace’s MAI401
for attitude determination and control (ADACS - Attitude Determination and
Control System). The ADACS was designed to determine spacecraft attitude with
a magnetometer, six coarse sun sensors, and a star tracker, and provide
pointing and stabilization with three reaction wheels and three magnetorquer
rods.

The avionics used an Innoflight CFC-300 flight computer. CuPID’s software,
written in C++ with a modular design, was compiled for the Linux system
running on the CFC-300. The modular flight code was divided into a number of
modules controlling different elements of the spacecraft functionality. A
Process Monitor module was responsible for starting the satellite and the
other software modules, and the Rules module read current spacecraft data and
managed issues when they arose. Some of the benefits and challenges of the
Rules module are discussed in a later section. The CFC-300 used NAND flash
memory that was divided into three partitions. First was a non-volatile
portion that contained the Linux operating system. The other two sections were
regularly read from and written to, and contained the modular flight software
in a smaller partition and generated spacecraft and science data in a larger
partition.

The communication system was built around matching Astronautical Development
Lithium-2 (Astro-Dev LI-2) radios on the spacecraft and in the ground station
communicating at 400.5 MHz. At 400.5 MHz, in the space research ultra high
frequency (UHF) band, CuPID was not allowed to send beacon packets. CuPID’s
antenna was a 1/4 wavelength deployable dipole antenna. The antenna was
deployed by a burn-wire and resistor system that operated after ejection from
the rocket.

In addition to the development of a flight unit, a complete replica of the
flight unit called the flat-sat, was used for software and operations testing.
See Figure 3. Additionally, the flat-sat could be used for debug purposes
after launch of the flight unit.

### Significant Mission Impacts

The CuPID mission was significantly affected by the onset of the CoVID-19
pandemic. In December 2019, hardware elements of the flight unit were
integrated at GSFC, and system level testing began. The CuPID team was
notified the spacecraft was manifested on a launch scheduled for one year
later, December 2020. When GSFC closed its gates in March 2020 due to the
pandemic, CuPID was locked inside and still had a delivery date of September
2020, though later that month, the CuPID team was notified the launch was
delayed until March 2021.

Through some political navigation, CuPID was extracted from GSFC and brought
to BU where testing could more easily be conducted with special health and
safety protocols. Testing facilities that were at GSFC had to be built at BU,
and supply chain challenges and lead times forced the team to make compromises
in certain testing hardware, such as an X-ray beam line.

During the summer of 2020, AMA was sold and CuPID’s avionics support was
terminated. The engineers working on CuPID moved to other companies. The BU
CuPID team maintained a strong relationship with one of the lead engineers,
and without this connection, CuPID would have had a significantly steeper
climb to launch.

Testing of the integrated instruments and avionics were conducted daily.
Flight software was iterated and tested on the flatsat, and instrument
calibration was conducted on the flight unit. In November 2020, one month
before planned delivery, another launch delay was announced, moving launch to
September 2021.

Figure 3: CuPID’s flatsat used to test mission operations and software
changes. An orbit simulation was input to the ADACS. The flatsat and ADACS
responded accordingly. The operator monitored flight-like telemetry on the
computer output and commanded the satellite through the radio.

## Anomaly Testing and Results

The following sections outline some of the challenges the CuPID team navigated
with the small satellite and investigated while attempting to contact CuPID.
Each section leads with a synopsis of the content.

### Early TLE’s

Summary: Multiple sources supplied multiple TLE’s shortly after deployment
which posed challenges in tracking multiple objects in the sky. As a secondary
payload, information regarding the deployment and tracking of the satellite
came with large error bars.
Solution/Result: Wider ground antenna beam width allows for more sky coverage,
and given enough time, TLE’s mature and are more precise.

Shortly after launch, when contact had not been made with the spacecraft after
the first several ground-passes, the team began anomaly investigation. The
first challenge the team encountered was identifying the proper TLE 111Two
Line Element which corresponded to CuPID. Multiple sources provided different
TLE’s.

Within hours of launch and from the launch provider, the team was to get an
estimated TLE from models based on the launch vehicle position. The delivery
of this TLE was delayed one day by the launch provider. A second TLE was
expected to be delivered from models run by GSFC. Another TLE was generated by
the US Space Force North American Aerospace Defense Command (NORAD) which uses
radar tracking of all objects orbiting Earth. In addition, not all objects
ejected from the Centaur upper stage were claimed, therefore six possible
positions of CuPID were identified.

Figure 4 shows the SatPC 222SatPC was the software used to track TLEs and
control the ground station antenna pointing. Link to software:
https://www.amsat.org map of the possible CuPID locations shortly after
launch. Although all are relatively in track, the sequence of them proved
challenging for ground pass operations because there were multiple satellites
in the sky, at different locations.

Figure 4: Computer ground track of potential TLE’s for CuPID two days after
launch.

A ground pass with multiple TLE’s to track will have one satellite inside the
RF beam, and multiple outside of it, leaving limited time to track each TLE.
CuPID’s ground station used two two yagi antennae to produce a beam width of
about 30∘ and RF gain of 15 dBi. One antenna has a beam of about 60∘ and RF
gain of 11.5 dBi.

One antenna was removed to open the beam width, at the cost of RF power. This
wider beam width allowed for greater inaccuracies in pointing and TLE
knowledge. After a few days, the TLE’s matured from accurate radar
measurements. In addition, some of the objects were claimed by other
institutions. CuPID’s TLE was well known after a week but did not respond to
commanding.

Should CuPID have had a beacon (discussed later) narrowing the TLE options
would have been easier. With a satellite beacon, the ground station could
listen to each TLE for a signal, and the maximum signal would designate the
correct TLE.

### B-dot control

Summary: Magnetometer/magnetorquer control in the ADACS could have a mismatch
in the hardware directions and controller causing excess spin rates of CuPID.
Solution/Result: Ground testing showed the directions were correct, though the
polarity of the magnetorquers could not be confirmed, and testing of the
complete B-dot control process showed inconclusive results. Ground imaging of
CuPID on orbit were also inconclusive.

After deployment from the rocket, or after any power cycle or reset, the
spacecraft ADACS was programmed to enter a rate nulling mode where it used a
B-dot333B-dot is a shorthand writing of the derivative of B, the rate of
change of magnetic field magnitude control to decrease the rate of spin of the
spacecraft. It would remain in this mode until the spacecraft was commanded
from the ground.

If, rather than nulling the spin rate, the controller spun the spacecraft to
higher spin rates, it was anticipated the team would not be able to
communicate with the spacecraft. The communication system required spin rates
less than 10∘/s to successfully receive commands from the ground. Another
cubesat mission using the same version of the ADACS hardware reported their
spacecraft spinning up to 1200∘/s in one axis when in B-dot mode through
monitoring beacon packets [3].

One possible cause of such a failure is mislabeling of the polarity of the
axes on the magnetometer or magnetorquers. Another cubesat group using the
same MAI401 hardware did find some elements mislabeled in their system during
ground testing. In the case of CuPID, the polarity of the magnetorquers was
confirmed with ground testing of the flight unit prior to launch and on the
flat-sat after launch. A second possible cause of this failure could be issues
in the firmware that performs the B-dot algorithm on the ADACS unit. The
algorithm requires input from magnetometer measurements and results in output
of magnetorquer actions. A coordinate system issue in the firmware could not
be tested with the available tools and still remains a possibility. The B-dot
control of the flight unit CuPID was tested on an air bearing setup, though
results were inconclusive because of the total mass and friction of the air
bearing assembly.

Some resources exist to image orbiting bodies which can provide information on
the behavior of a spacecraft, such as spin rate or cross section, even if the
spacecraft is not responding. Optical imaging was explored to monitor CuPID’s
spin rate, but the size of the 6U spacecraft, with no deployable solar panels
to expand the cross-section, produced a small signal to noise, and ultimately
spin rates were not reported back to the CuPID team.

### Memory corruption

Summary: CFC-300 data partition became corrupted twice on the flight unit
spacecraft due to a power down during write moment on drive.
Solution/Result: Power down sequence would un-mount partition before powering
down the system.

Over the course of months of power cycling and daily testing, the partition
for data storage, but not flight software became corrupt two times. Through
direct commanding, of the CFC-300 computer, the corrupt data partition was
erased and reset. Flight software was not disrupted. Additional abilities were
developed to allow the flight software and a ground operator command the
CFC-300 in case this happened on orbit.

The cause of the corruption was determined to be an electrical power anomaly
during a moment of data writing. The power anomaly was found to be caused
during a operator defined power down of the spacecraft at an unfortunate, but
precise moment. The slow power drop during a write action would cause the
corruption. This problem was remedied with a software defined un-mounting of
the partition of the hard drive before powering down.

Although the problem was corrected during manual shut down sequences,
unexpected resets could still cause this problem. On orbit, unplanned power
events can be common, and could cause the hard drive to become corrupt.
Although the flight software partition never corrupted during ground testing,
it is still a possibility on orbit.

### Spacecraft reset timing

Summary: Rules in the software could reset the spacecraft in a periodic
fashion that would synchronize with ground passes.
Solution/Result: Ground passes were conducted from Fairbanks, AK, with a
mobile ground station to change the location, and increase the number of
passes in a day.

The Rules module included a few software defined rules to control the
spacecraft and maintain it’s state of health.

  1. 1.

Every 12 hours without ground commands- Entire spacecraft reboot

  2. 2.

Every 6 hours- radio reset

  3. 3.

Battery below 60% state of charge - shut down everything except the power
monitoring system until power returns

Should the satellite receive no successful commands from the ground in 12
hours, the flight computer would force the electrical power system to reset,
which then power cycled the flight computer and restarted the entire
satellite. Spacecraft startup took about two minutes. The 12 hour reboot was
included as a precaution in case an issue on orbit required a reboot to
maintain radio link.

Figure 5: The mobile ground station during testing at Poker Flat in Jan. 2022.
TOP: Tripod, rotator head and antennae. BOTTOM: Radio, RF amplifier, power
supplies and laptop setup inside.

The 6 hour radio reset was included after ground testing proved the radio
required frequent resets. The radio showed instances of poor packet
management, and at times was locked transmitting noise. In these instances, no
commands could enter the radio via RF, and the radio was consuming battery
power while transmitting. The software defined radio reboot would cut these
issues short should they occur. The reset only lasted 10 seconds.

A rule regarding the battery state of charge was included to protect the
battery health. CuPID utilized two body mounted solar panels, one on each of
the 6U faces of the satellite. Software would power off the satellite below a
60% state of charge, and would power it back on as the charge increased. It
was determined that in a tumbling attitude and only using the limited power
consumption after startup, CuPID was likely power negative. Modeling showed
this power cycling could occur once per day.

Although the concept of these rules are common on satellites, the scheduling
provided an unfortunate mismatch for CuPID’s orbit. CuPID’s orbit at 550 km
has a period of almost exactly 90 minutes, and because it is sun-synchronous,
ground passes were at nearly the same time each day. Each day of ground passes
consisted of four total ground passes, with two iterations of two sequential
orbits (e.g. 10AM, 11:30AM, and 10PM, 11:30PM). There could be a condition
where the satellite would be powered off during many sequential ground passes
because the 90 minute orbit period is a factor of the 6 hour radio reset and
12 hour spacecraft reboot. This could have been prevented by selecting a reset
interval that was not an interval of the orbital period.

In order to decrease the likelihood that this was the cause of no-contact,
multiple sequential orbit ground passes needed to be conducted. At higher
latitudes, the 97.6∘ orbit inclination allowed for up to 8 sequential orbit
passes, although some were at elevations below 10∘. The team developed a
mobile ground station with a small antenna rotator head, tripod and some power
supplies, see Figure 5. The rotator was a Yaesu G5500 controlled by an EA4TX
ARS-USB rotator controller. This mobile station was brought to Poker Flat
Research Range outside of Fairbanks, AK in January, 2022 for six days of
testing. Seven consecutive passes were conducted on two out of the six days of
operation from Poker Flat Research Range. Although multiple sequential passes
were conducted, CuPID did not respond to commanding.

### Antenna deployment

Summary: The antenna was designed to be deployed with a burn wire mechanism,
and if not deployed, the antenna gain was limited to 1.
Solution/Result: Ground testing showed confidence in the mechanism, though the
issue on orbit is still a possibility and cannot be confirmed.

The deployment of the antenna is an unlikely cause of no signal from CuPID.
The deployment mechanism was successfully tested after a vibration environment
four times (risk reduction of the chassis, low level vibe, requirement vibe,
increased requirement vibe) and performed nominally each time. The stowed
antenna can be seen in Figure 2 and Figure 6A. The deployment software and
mechanism from boot-up was tested multiple times without failure.

A less significant but possible software issue regarding the antenna
deployment, was that the software would only attempt to deploy the antenna
once at initial start up after deployment. After the automatic software
attempt was made to burn the wire, a file was written that told the software
the antenna was deployed. Any later startup, would then read this file and
know not to deploy the antenna. There was no automatic method to read a failed
antenna deploy and re-attempt. Link budgets showed that contact could still be
made, though margins were closer because the deployed antenna gain was 2.15
dBi verses stowed gain at 1 dBi. Ground testing (although higher power than on
orbit) proved a stowed antenna would still communicate. Commands from the
ground could attempt to re-deploy the antenna. A better design of the antenna
deployment system would include automatic repeated attempts to deploy the
antenna should the satellite sense it failed.

### Radio performance

Summary: The LI-2 radio likely had a mismatch in firmware and hardware
components, which manifested in poor performance.
Solution/Result: Ground testing illuminated the poor performance, though the
cause was not determined until days before delivery. Further ground testing
could have dis-qualified the radio for flight.

CuPID’s communication system was based around a matched pair of Astro-Dev LI-2
radios. One radio was configured for the CubeSat, the other for the ground
station. Licensed to communicate at 400.5 MHz, the ground radio transmitted at
9,600 bits/s and received at 38.4 kbits/s. The spacecraft transmitted and
received the opposite baud rates. The spacecraft’s 1/4 wavelength deployable
antenna (Figure 6A) was deployed by a burn wire resistor. The spacecraft
flight computer was configured to communicate with the radio at 115,200 baud.

Figure 6: CuPID’s spacecraft radio (left) and ground radio (right) systems. A:
CuPID with stowed antenna. B: CuPID’s LI-2 radio and board inside avionics. C:
Ground radio system. D: Ground antenna.

Discussions with the radio manufacturer days before the delivery of CuPID
illuminated a possible cause of failure. The radio’s default interface baud
rate was 9,600, whereas the flight computer was configured to communicate with
the radio at 115,200. On the ground, the radio was configured, and stored to
communicate at 115,200 baud. Should the radio reset to default values (for
example, from a single event upset), the flight computer had no means to
determine and reset the baud rate to re-establish the interface.

Testing of the radio on the ground consistently showed an offset to the
transmit frequency. Although configured at 400.5 MHz, the radio transmitted at
400.531 MHz. This was originally presented to the radio manufacturer in March
of 2020, but was not answered until the discussion days before delivery of
CuPID. The frequency offset was described as a signature of a mismatch between
the firmware and hardware on the radio due to a version control setback from
supply chain issues during manufacturing.

Licensed in the space research band, CuPID was not allowed to beacon data.
Instead, CuPID’s communication system required a command to enter the
spacecraft before data was transmitted from it. This means that a significant
signal must reach the spacecraft radio to be decoded and acted upon. The input
signal must be above the noise floor of the radio. An increase in the ground
station power output would increase the power input to the spacecraft radio.
An RF amplifier was added into the ground station (Toptek PA-100-05) that
output up to 100 W to the ground station antenna (antenna gain of about 15
dBi). Assuming a loss of about 5 dB in transmission, 100 W from the amplifier
resulted in an effective isotropic radiated power (EIRP) of 60 dBm, or 1000 W.
This emitted power was 20 times the designed and expected link budget. This
still did not result in response from CuPID. The power amplifier was also used
in passes at Fairbanks, AK.

The team also performed ground passes from NASA Wallops Flight Facility (WFF)
with a 30 m dish antenna. The antenna has a gain of 36 dBi. CuPID’s ground
radio input 1.75 W, resulting in an EIRP of 63.43 dBm or 2203 W (assuming 5
dBm of losses). Although a sizeable emission, this still did not produce a
response from CuPID. Seven ground passes over 4 days were conducted from the
WFF ground station.

Discussions with the IT-SPINS CubeSat team from Montana State University (MSU)
showed they too had problems with an Astro-Dev LI-2 radio procured near the
same time as CuPID. Not only did IT-SPINS B-dot control in the MAI-401 ADACS
accelerate the spin rate of the satellite, but their LI-2 would not take
commands, although they did have a radio beacon transmitting state of health
data [3].

CuPID had spare radios that were purchased during development. Two radios were
sent to MSU for characterization. The noise floor of the radios was found to
be unstable, varying between -130 dB and -90 dB when tested in an anechoic
chamber with a known noise floor at -123 dB. This could be an artifact of the
unproven mismatch between hardware and firmware in the radio. A noise floor of
-90 dB is significantly high. Should this be real, the Gain/Noise factor of
the radio would be at least 40 dB lower than the designed value.

In terms of link budget, one can make estimates from a best case scenario
ground pass. For a high elevation pass ($>$70∘) where the free space path loss
was a minimum, and from WFF where the EIRP was maximized, link budget
estimates with the as measured radio performance suggest that uplink margins
were not favorable. The power into the radio would be near -104 dB, which is
lower than a noise floor at -90 dB. Estimates for the bit error rate (BER)
calculated from the noise temperature and received figure of merit show that
link could be made, but margin was within a few dB.

Three things that can further decrease the BER during a pass include
spacecraft orientation, elevation and occupied bandwidth.

There are a few spacecraft orientations which decrease CuPID’s antenna gain.
The ideal gain was 2.15 dBi. If CuPID was spinning, it is likely this value
was decreased, therefore decreasing the BER margin. Additionally, if CuPID’s
antenna never deployed, the receiver gain was reduced to 1.

At 70∘ elevation, the free space path loss is -139.75 dBm, whereas at 10∘
elevation, the free space path loss is -151 dBm. This decreases the BER margin
by 12 dB.

Occupied bandwidth was measured at -3 dB from center frequency peak. The
frequency offset of the radio (about 30 kHz) would require a wider occupied
bandwidth. CuPID’s FCC license reports the bandwidth at -40 dB and -60 dB, at
22.9 kHz and 216.9 kHz respectively. At a -40 dB bandwidth, the occupied
bandwidth was 43.44 dB. This is 10.43 dB larger than the bandwidth at 3 dB,
and would decrease the BER margin accordingly.

These situations all decrease the BER margin, and all are likely included in
some capacity. Even with the strongest signal and best BER margin, from WFF,
the margin would likely be consumed by these additional taxes on the uplink
system.

## Improvements

The development and operations of CuPID led to multiple lessons learned that
are useful to future missions. Here, three improvements to the CuPID mission
are discussed.

First, a radio should be tested beyond what would be expected on orbit.
CuPID’s radio was assumed functional from the manufacturer, and issues during
development were attributed to the development team’s software. In reality,
the radio performance was far below what was expected. Although the radio had
functionality on the ground, issues such as the low data throughput, and
frequency offset should raise flags to test the radio further. The noise floor
of the radio should be confirmed before launch. In addition, the default
values of the radio should be coded into a software check for the flight
computer to try should commands not be received.

Beaconing is a communication feature that CuPID’s radio license was not
permitted to use. Beaconing, is a radio frequency (RF) communication method
where a satellite emits packets of information at known period. The Federal
Communication Commission (FCC) does not allow beaconing outside of the amateur
frequency band (UHF 420.0 to 450.0 MHz). CuPID, at 400.5 MHz was not allowed
to beacon. For a mostly student built satellite, a radio beacon should be an
essential component.

A great benefit of the beacon, is that it allows the amateur radio community
to listen to the satellite, but not de-code any data unless the process is
shared with them. The SatNOGS Network444https://network.satnogs.org/, is a
community of amateur radio users who use the network to track satellites and
upload their ground pass observations. The network has ground stations all
over the world, and data can be accessed online. Combining the SatNOGS Network
with a beaconing satellite would have allowed CuPID to get nearly continuous
monitoring. Additionally, a beacon would allow a mission in early operations
to quickly locate which TLE corresponds to a team’s satellite. By instead
listening to each TLE for the beacon, one could determine their TLE out of
multiple options based on signal strength.

A third improvement to CuPID would have included more automatic rules in
flight software that execute if the satellite does not receive commands from
the ground. Although a simple reset could fix some issues, commands that
attempt to re-deploy the antenna, or re-establish communication with the radio
could have been implemented.

## Current Cubesat Economy

The small satellite industry continues to rapidly evolve and grow,
particularly in the commercial and government markets. Launches of small
satellites have increased near exponentially [4]. A feature of this growth is
rapid commercial realignments including mergers and acquisitions. In some
cases this can have significant implications for customers and be disruptive
on schedule and program development. In the case of CuPID, the avionics
developer was acquired and the branch responsible for the avionics development
dissolved prior to the end of the program, causing a rapid re-alignment of
development activities within the CuPID team. Stability should be an element
of consideration in addition to performance and cost when evaluating vendors.

## Conclusion

On-orbit anomaly testing of the CuPID CubeSat led to a few important lessons
learned. These are some critical elements for developing a CubeSat:

  1. 1.

Having a flat-sat, or replica of the spacecraft electronics that a builder
interacts with like it is the flight unit is indispensable for ground testing.

  2. 2.

Test a flight radio intensely - confirm noise floor, etc.

  3. 3.

Put licensed radio frequency in a frequency band that allows beaconing. Early
operations are easier if a satellite automatically transmits.

  4. 4.

If using NAND memory, include software to un-mount the memory before powering
off the satellite to prevent corruption of the drive partition.

  5. 5.

Include and test software fail-safes that are acted upon in the case of no-
contact from the ground. (E.g. automatic re-attempt of antenna deploy, reset
of radio baud rate)

The CuPID cubesat was built with the objectives to develop a small form-factor
and low resource wide field-of-view X-ray telescope and to use this instrument
to understand macro-scale spatial and temporal properties of magnetopause
reconnection. Ground characterization allowed demonstration of the performance
of the miniaturized X-ray telescope. After the spacecraft was launched into
orbit, an anomaly in-flight prevented communication from the ground-station
with the spacecraft. Further anomaly testing spanned a number of spacecraft
sub-systems and found irregular and temporally variable performance of the
radio as the most likely cause of failed communication.

## References

  * [1] BM Walsh, MR Collier, E Atz, L Billingsley, JM Broll, HK Connor, D Chornay, T Cragwell, N Dobson, S Eckert, et al.  The cusp plasma imaging detector (cupid) cubesat observatory: Mission overview.  Journal of Geophysical Research: Space Physics, 126(4):e2020JA029015, 2021.
  * [2] Emil Atz, Brian Walsh, Connor O’Brien, Michael Collier, Ariel Berman, Lisa Billingsley, J Bernard Blake, Jeffery Broll, Dennis Chornay, William Crain, et al.  The cusp plasma imaging detector (cupid) cubesat observatory: Instrumentation.  Review of Scientific Instruments, 93(6):064504, 2022.
  * [3] Sarah Dolan.  Building and operating cubesats-successes and struggles: Montana state’s experience with firebird and it-spins.  Presented at the 2022 CubeSat Developers Workshop, 2022.
  * [4] Thyrso Villela, Cesar A Costa, Alessandra M Brandão, Fernando T Bueno, and Rodrigo Leonardi.  Towards the thousandth cubesat: A statistical overview.  International Journal of Aerospace Engineering, 2019, 2019.