# Programmable EM Sensor Array for Golden-Model Free Run-time Trojan Detection
and Localization

Hanqiu Wang  ECE Department, University of Florida
Gainesville, FL 32611 USA
<EMAIL_ADDRESS>Max Panoff  ECE Department, University of Florida
Gainesville, FL 32611 USA
<EMAIL_ADDRESS>Zihao Zhan  ECE Department, University of Florida
Gainesville, FL 32611 USA
<EMAIL_ADDRESS>Shuo Wang  ECE Department, University of Florida
Gainesville, FL 32611 USA
<EMAIL_ADDRESS>Christophe Bobda  ECE Department, University of Florida
Gainesville, FL 32611 USA
<EMAIL_ADDRESS>Domenic Forte  ECE Department, University of Florida
Gainesville, FL 32611 USA
<EMAIL_ADDRESS>
###### Abstract

Side-channel analysis has been proven effective at detecting hardware Trojans
in integrated circuits (ICs). However, most detection techniques rely on large
external probes and antennas for data collection and require a long
measurement time to detect Trojans. Such limitations make these techniques
impractical for run-time deployment and ineffective in detecting small Trojans
with subtle side-channel signatures. To overcome these challenges, we propose
a Programmable Sensor Array (PSA) for run-time hardware Trojan detection,
localization, and identification. PSA is a tampering-resilient integrated on-
chip magnetic field sensor array that can be re-programmed to change the
sensors’ shape, size, and location. Using PSA, EM side-channel measurement
results collected from sensors at different locations on an IC can be analyzed
to localize and identify the Trojan. The PSA has better performance than
conventional external magnetic probes and state-of-the-art on-chip single-coil
magnetic field sensors. We fabricated an AES-128 test chip with four AES
Hardware Trojans. They were successfully detected, located, and identified
with the proposed on-chip PSA within 10 milliseconds using our proposed cross-
domain analysis.

###### Index Terms:

EM sensor, side-channel, Hardware Trojan

##  I Introduction

Hardware Trojans (HTs) are malicious modifications inserted into ICs by
adversaries during the fabrication or design process. These alterations can
disrupt system functionality, causing security and privacy issues such as
privacy leakage, denial-of-service and privilege escalation, etc. As the
reliance on third-party designs and manufacturing continues to grow, the
threat posed by HTs becomes more alarming. Over the years, researchers have
discovered that HTs often produce detectable anomalies through side-channels
like power consumption and electromagnetic (EM) emanations. The anomalies in
side-channels can be detected after the chip is fabricated, either during the
test phase or in run time, but often require costly external sensing setups
and a golden model, a Trojan-free circuit used as a reference. This makes
deploying some of these approaches in run time not applicable. Moreover, many
side-channel oriented techniques suffer from low resolution of the collected
traces.

As a result, on-chip sensors have gained attraction as a preferred solution.
These sensors can intrinsically collect traces from EM side-channels through
magnetic field coupling at run time with higher resolutions, thus ruling out
the need for external measurement equipment (e.g., external probes, moving
stages, etc.). Previous research by Jiaji et al. [1] used a single coil
covering the entire chip area, which has low signal-to-noise ratio (SNR)
issues. Consequently, this approach couldn’t detect small HTs with few gates
and required over 10,000 measurements to detect larger HTs. Its long mean time
to detect (MTTD) limits its practical effectiveness.

In this paper, we propose a Programmable Sensor Array (PSA) to address the
limitations of Jiaji’s work. Our improved design, shown in Figure 1(a),
consists of a crossbar-like structure. The PSA can be programmed to different
sizes and shapes, allowing it to cover or uncover specific parts of the chip
for better SNR and HT localization. At each crossing point in the wire grid, a
transmission gate (T-gate) controls the node’s connectivity. This allows the
sensor array to have different shapes, locations, and sizes. The size of a
single sensor within the PSA can also be programmed to approximately match the
size of a HT, ensuring the highest magnetic field emanations from HTs are
captured. The improved SNR of PSA enables almost instant runtime detection,
reducing MTTD significantly. The contributions of this paper can be summarized
as follows:

  * •

A novel on-chip PSA for EM side-channel measurement with big advantages over
existing techniques was proposed, developed and validated.

  * •

A silicon demonstration of the proposed PSA to detect and localize active HTs
on an AES design implemented with TSMC 65nm technology.

  * •

Significant SNR improvements with the proposed PSA compared to prior work.

  * •

Experimental validation of the PSA’s capability to detect, locate, and
identify digital HTs within 10 ms with the proposed novel cross-domain
analysis technique.

(a)

(b)

(c)

Figure 1: (a)3D structure of on-chip PSA where top-level metals form the coils
and transmission gate switches in the active layer control the size and shape
at each intersection. (Only one MOSFET is shown for simplicity) (b)An example
of PSA topology forming a 2-turn coil. The red dots denote the locations where
switches are on. (c)T-gate layout.

##  II Threat Model and Prior Work

###  II-A Run-time Verification Versus Test-phase Verification

The distinction between run-time and test-phase verifications hinges primarily
on the approach to HT detection. During the test phase, efforts are
concentrated on the detection of HTs that can be intentionally triggered. The
effectiveness of HT detection is evaluated based on the accuracy of feature
and trace categorizations as well as the number of attempts needed to activate
the Trojan. Most research focuses on developing algorithms to successfully
trigger HTs within the minimum amount of time [2] [3]. Conversely, during run
time, potential adversaries must first activate an HT before it can be
detected if the whole batch of ICs is infected with HT, which is also the
assumption of this paper. The primary metric is the duration between the
Trojan’s activation and its detection. This encompasses recognizing the
system’s activation and discerning the triggering conditions, often referred
to as the Mean Time to Detect (MTTD) [4].

To reduce the hardware overhead during the evaluations, lightweight
measurement and data processing components should be used. Bai et al.
developed RASCv2 board, a compact 2cm x 2cm board that can replace an
oscilloscope for side-channel analysis to detect malware [5][6]. The board
consists of two analog-to-digital converters (ADCs) for sampling EM and power
side-channel traces, an EM antenna for signal capture, and an FPGA for data
processing. The data processed by the FPGA are monitored by a security house
via an onboard Bluetooth module, making run-time side-channel verification
feasible. In this paper, we will integrate the concept of employing the RASC
board (manufactured at a trusted facility) for data processing with the PSA’s
on-chip measurement capabilities to detect HTs in a real-time and trusted
manner.

###  II-B EM Side-channel Data Collection Methods

There are several existing techniques to detect and analyze EM side-channel
leakage. Traditionally, external EM probes and oscilloscopes are used to
collect EM side-channel emanation traces over IC packages. The captured
signals will be analyzed in either the time or frequency domains to identify
HTs [7, 8, 9]. To improve the quality of the collected traces and reduce
overhead, He et al. introduced a single-winding on-chip EM sensor covering the
whole chip to replace external probes [1]. In [9], Nguyen et al. introduced an
EM backscattering-based Trojan detection approach. A transmitter antenna
injects carrier signals into the IC, and the reflection signals that have been
modulated with the activities of HTs are subsequently captured with a receiver
antenna to identify HTs. This technique can detect impedance variations
resulting from HT activities, even if these HTs have very small current
consumption. Compared with our proposed PSA detailed in Section III, existing
measurement techniques have problems of low SNR, limited spatial resolution,
complex implementations, etc. These issues result in the limitations of HT
detection rates, HT localization, and runtime deployment. A comprehensive
comparison will be presented in Section III-B.

##  III Programmable Sensor Array (PSA)

One of the main challenges of on-chip EM sensors is the balance among sensing
area, accuracy, and cost. A single large sensing coil covering the entire chip
is relatively easy to design and implement; however, it could be inaccurate
due to magnetic flux self-cancellation111Magnetic flux forming a small loop
within a large sensor induces no voltage on it.. Multiple smaller coils on the
same layer may improve the overall resolution but would have smaller signal
magnitudes and coverage. Using coils across multiplemetal layers can avoid
these issues, but with extra cost. To mitigate these problems, we propose the
PSA technique that enables the customization of sensor size and position on
the chip at run time.

###  III-A PSA Topology and Structure

PSA utilizes a wire grid that spans two metal layers, with a switch positioned
at every intersection. Each switch is essentially a transmission gate
(T-gate), depicted in Figure 1(c), comprising a single PMOS and NMOS gate
connected in parallel. The shape, size, and location of a sensing array can be
programmed at run time by controlling specific switches, as shown in Figure
1(b). The size and location of PSA can be programmed via T-gates by connecting
or disconnecting different cross-points on the wire grid or lattice. The
lattice, comprising several rectangular sensing regions, can be used as part
of the sensing array within the PSA.

Adjusting the shape and size of the PSA enables the PSA to circumvent the
self-cancellation issue highlighted in [1], with many advantages. For
instance, it facilitates the localization of any detected HTs by reshaping the
sensing array. It also eliminates sensing boundary limitations or the need for
significant overlap when employing multiple coils.

Implementing such side-channel sensors on chip will raise concerns on PSA’s
ability to execute side-channel attacks (SCA). Indeed, PSA itself does not
protect from SCAs. But if adversaries can physically access the chip and
conduct SCAs, it may be vulnerable to SCA regardless of the presence of PSA.
Additionally, raw EM traces are not transmitted over any communication
channels. Only processed data should be sent, and this can help prevent SCAs.

###  III-B PSA Comparison with Prior Work

To highlight the improvement of the PSA-based HT detection technique over
prior works, we made a comprehensive comparison in Table I. This comparison
considers five criteria: HT detection accuracy, spatial resolution of the
measurement, required number of traces, SNR, and feasibility for run-time
deployment.

The approaches relying on traces collected from conventional external probes
[7, 8] and on-chip single coils [1] fail to detect small HT with few gates
(implemented as T3 in our chip in Table II). While Nguyen et al. [9] reports a
100% detection rate, their approach shares a common limitation with others:
the inability to pinpoint the exact location of the HT on the chip. On the
other hand, our PSA approach not only ensures a 100% detection rate but also
stands out with the enhanced spatial resolution, providing the distinct
capability of precisely identifying the HTs’ physical location.

Due to low SNR, prior works often rely on statistical analysis, requiring
numerous measurements to detect Trojans. For instance, Xiaolong et al. [7] and
Jiaji et al. [1] compare the Euclidean distance between traces or explore the
Euclidean distance distributions. Nguyen et al. [9] use Principal Component
Analysis and K-means algorithm to categorize the collected spectra. These
approaches require 100 to more than 10,000 measurements. In contrast, the EM
side-channel traces from PSA exhibit sufficiently high SNR with an apparent
difference between HT-inactive and HT-active traces. Therefore, fewer than ten
measurements are needed, leading to a significantly shorter MTTD and faster
Trojan detection.

For implementation complexity, unlike prior works necessitating cumbersome
measurement devices [7, 8, 9], the PSA approach does not need external probes
so it has high feasibility for run-time deployment. Fujimoto et al. also
exploited the on-chip power noise measurement(OCM) to execute Correlational
Power Analysis with high SNR [10, 11], it is also possible to use such OCM to
detect HT, but that requires further investigation.

TABLE I: Comparison of EM side-channel data collection methods. Features |  External Probe [7, 8] |  Nguyen [9] |  On-chip Single Coil [1] |  PSA (proposed)
---|---|---|---|---
HT Detection rate |  Low |  High |  Low |  High
HT Localization |  No |  No |  No |  Yes
Measurement# |  $>$10,000 |  100 |  $>$10,000 |  $<$10
SNR |  14.3dB |  N/A |  30.5dB |  41.0dB
Run-time analysis |  No |  No |  Yes |  Yes

##  IV PSA’s Tamper-resilience

Hardware Trojan risk primarily originates from Third Party Intellectual
Property (3PIP) vendors and malicious foundries. Our PSA approach is
tampering-resilient and can effectively reduce the risks under both attacking
scenarios. The effectiveness of PSA in detecting and locating HTs will be
proved with the experimental results of Subsection VI-D. This section
introduces a few case studies demonstrating PSA’s resistance to tampering from
either source.

###  IV-A Case 1: Malicious 3PIP

For malicious 3PIP attacks, as PSA is implemented on the two topmost metal
layers after synthesizing the Register Transfer Level (RTL) code and is
isolated from the main circuit, it would not interact with any 3PIP.
Therefore, any pre-silicon modifications, including all potential 3PIP
attacks, made prior to the layout phase will not compromise the PSA.

###  IV-B Case 2: Malicious Foundries

When dealing with malicious foundries, the PSA itself may be modified.
However, tampering with such a mixed digital and analog structure is
challenging. The adversary must modify additional circuitry, increasing the
attack’s complexity, thus lowering the attack’s success rate. Additionally,
any modifications that disable the PSA will trigger alarms during the test
phase, as the PSA will return testing values.

Even if the attacker successfully completes the modifications, designers can
easily detect them by reverse-engineering the two topmost metal layers. They
are typically the thickest metal layers and are relatively easy to be reverse-
engineered. Alternatively, designers can outsource the fabrication of the two
topmost metal layers to other trusted foundries, a split-manufacturing method
similar to those discussed in [12]. This provides an additional layer of
security, making the PSA a robust defense against HT attacks.

##  V PSA implementation on A Test Chip

###  V-A Integrating PSA on AES-128 Test Chip

To assess the performance of PSA in both test and run-time phases we
implemented and fabricated the PSA on the two top metal layers of an Advanced
Encryption Standard (AES128) encryption design using TSMC65nm technology. The
proposed PSA was situated on layers Metal 7 and Metal 8 of the chip, covering
the entire chip. It is a lattice including 36 horizontal wires, 36 vertical
wires, and 1296 switches. The lattice wire measures 16$\mu$m in length and
1$\mu$m in width. Frequency sweeping is used to determine the optimal length
and width that maximize the signal magnitude in the desired frequency range of
10MHz-100MHz.

For this test chip, the entire area was uniformly divided into 16 square
sensing areas or sensors. Each sensor shares 33% of its area with adjacent
sensors to ensure adequate circuitry sampling near the borders between any two
sensors.

Main Circuit The underlying main circuit is an AES-128-LUT core, integrated
with an RS232 UART communication module [13]. The clock frequency was set to
33MHz. The layout of the routing wires of the test chip, along with the
indices of the PSA sensors, is illustrated in Figure 2.

Figure 2: Sensor deployment, IO pin assignment, and Amoeba module view on the
AES128 test chip.

HT Design The design incorporates four distinct HTs, modified from Trust-Hub
[14] [15]. T1 is an amplitude modulation radio carrier Trojan capable of
emitting an electromagnetic (EM) wave at a frequency of 750KHz. T2 is a chain
of inverters connected to a key wire to amplify its leakage current. If T2 is
implanted, attackers could recover the key via power analysis. T3 is a Code
Division Multiple Access (CDMA) channel Trojan designed to leak the key, and
T4 is a simple denial-of-service Trojan that elevates power consumption,
potentially causing the IC to overheat. The four Trojans employ digital
standard cells and each has unique triggering conditions. The 4 HT’s locations
are shown in the Amoeba view in Figure 2.

IO Pin Assignment The test chip uses a QFN 6mm x 6mm package and has 8 IO pins
on each side. The IO pins’ names are shown in Figure 2. The PSA uses the 8 IO
pins on the right side. 2 IO pins are needed to measure the induced
differential output voltage of one sensor, so we have 4 output channels, from
sensor1+/- to sensor4+/-. The 4 sensors on each row use the channel on the
same row. For example, sensor 0,1,5,6 in the figure uses sensor1 channel. The
PSA control signals use 4 of 8 IO pins on the bottom side. They were decoded
into gate signals for T-gates with the fully combinational decoder in the
figure.

HT Triggering Condition T1 is activated periodically when a counter reaches
21’h1F_FFFF under the 33MHz clock. T2 is triggered when the first four bytes
of the plaintext are 16’hAAAA. T3 and T4 are the always-on HTs, and we added
external enable signals as triggers in experiments. The gate number counts and
area ratios of these four HTs are shown in Table II.

TABLE II: Trojan gates count and percentage Circuit | Overall | T1 | T2 | T3 | T4
---|---|---|---|---|---
Standard Cell Number | 28806 | 1881 | 2132 | 329 | 2181
Percentage | 100 | 6.52 | 7.40 | 1.14 | 7.57

We assess the performance of the PSA in detecting and locating HTs by
activating and deactivating sensors during the execution of AES encryption. As
depicted in Figure 2, the green box represents the area of a 6-turn-coil
sensor, where most HT circuits are implemented. The orange arcs represent the
size of a circle-shape external EM probe. The main circuit primarily falls
under sensors 2, 3, 4, 7, 8, 9, 10, 11, and 14. Of these sensors, sensor 10
offers the most coverage of both Trojan payloads and triggers.

###  V-B T-gate Design and PSA Implementation Cost

Due to the absence of the T-gates that fulfill our requirements in the
standard cell library, we designed our own customized T-gates, which are
illustrated within a 3.2µm x 4µm customized cell layout in Figure 1(c). A
single finger of the NMOS and PMOS devices in the design exhibit an aspect
ratio of 500nm/60nm and 610nm/60nm respectively to have a similar current
conducting capability. For enhanced performance, each of these MOSFETs
incorporates 10 fingers. To ensure a low turn-on resistance, the layout
features two PMOS and two NMOS devices, effectively creating a pair of T-gates
connected in parallel. As a result, this T-gate configuration achieves a
resistance of approximately 34 Ohms.

For power overhead, the dynamic power consumption of PSA sensors is negligible
and largely contributed by the leakage power from the transistors. Therefore,
PSA’s impact on the overall power profile is minimal.

For area and resource allocation, the T-gates used in PSA account for an
additional 5% of the total chip area, with the PSA located on metal layers M7
and M8. Despite this, the design effectively preserves routing capacity by
aligning PSA wires parallel to main circuit wires, reducing top-layer routing
capacity by just 6.25%.

In contrast to Jiaji’s single-coil structure [1], which utilizes 100% of the
top layer’s routing capacity, PSA proves to be more efficient.

##  VI Evaluation on Test Chip

###  VI-A Test and Data Collection Platform Setup

The test chip was mounted on a PCB board with voltage-level shifters and open-
loop OP-AMPs. The output of each output channel of the PSA is amplified by a
THS4504D OP-AMP with 50dB DC gain and 200MHz UGB, aligning well with our
target frequency range from DC to 120MHz. A 33MHz crystal oscillator is used
to generate the clock signal. The evaluation is conducted while the test chip
executes AES-128 encryption. It receives plaintext from and sends ciphertext
to a laptop through serial communications. During this operation, an
oscilloscope or a spectrum analyzer triggered by the rising edge of the clock
signal captures the amplified PSA output and generates the related frequency
spectrum for further analysis. Additionally, we use the zero-span mode of the
spectrum analyzer to measure the time-domain signal of the PSA’s output at a
desired single frequency.

For run-time deployment, a system integrated on the PCB board can replace the
oscilloscopes, spectrum analyzers, and computers to measure and analyze side-
channel data.

###  VI-B Signal-to-Noise Ratio (SNR) Measurement

To demonstrate the improved measurement quality offered by the PSA sensor
array, we conducted SNR measurements and compared the results with those from
an external Langer-EMV LF1 probe. We employ He’s SNR measurement approach as
detailed in [1]. Noise traces are first collected from the powered-up chip
without any encryption activity. Signal traces are then gathered while the
chip performs AES encryption. The SNR is calculated based on the root mean
square (RMS) voltage ratio of these two data sets, as demonstrated in Equation
(1).

$SNR=20log\left(\frac{Vrms_{signal}}{Vrms_{noise}}\right)$ (1)

The SNR of the PSA is 41.0 dB, surpassing both the external probe’s 14.3 dB
and the single-coil on-chip sensor’s 30.5 dB as reported in [1]. In Figure 3,
the green spectrum represents the difference in dB between the spectra from
the sensor array and the external probe, demonstrating that the spectrum from
the PSA can be up to 55 dB higher than that from an external EM probe.
Overall, the proposed PSA structure offers significantly superior SNR results
in comparison to either external probes or single-coil on-chip sensors.

The best state-of-the-art external probe is the ICR HH100-6 set with a
diameter of 100 $\mu$m manufactured by the Langer EMV-Technik. Based on the
manufacturers documents, the SNR of ICR probe is approximately 34dB below 120
MHz, so the best probe is still worse than our proposed PSA.

Figure 3: Spectrum magnitude comparison between those from the PSA and an
external EM probe.

###  VI-C PSA Performance under Different Supply Voltages and Different
Ambient Temperatures

We have conducted comprehensive evaluations of PSA’s performance under varying
operational conditions to confirm its suitability for runtime deployment
across a wide range of applications. We performed simulations and experiments
at different supply voltages, spanning from 0.8V to 1.2V, which covers the
voltage supply voltage range for TSMC 65nm chips. Additionally, we carried out
simulations under different ambient temperatures, ranging from
$-40\text{\,}\mathrm{\SIUnitSymbolCelsius}$ to
$125\text{\,}\mathrm{\SIUnitSymbolCelsius}$.

####  VI-C1 Voltage

Theoretically, a supply voltage increase results in a reduction of the turn-on
resistance of T-gates. In the Virtuoso simulation, there was only a 4dB drop
in the impedance of a single PSA sensor when the voltage supply was raised
from 0.8V to 1.2V. We also made an experiment to measure the current response
of the PSA by adding a 70mV frequency sweeping chirp signal to one sensor of
the PSA and varying the supply voltage from 0.8V to 1.25V. The current does
not change significantly, which agrees with the simulation results.

####  VI-C2 Temperature

Theoretically, an ambient temperature change only slightly affects the
impedance of the PSA. Our simulations demonstrate that the impedance remains
relatively stable, fluctuating within a range of 4dB.

###  VI-D Runtime Cross-Domain Analysis

(a) T1 active, Sensor 10

(b) T2 active, Sensor 10

(c) T3 active, Sensor 10

(d) T4 active, Sensor 10

(e) T1 active, Sensor 0

Figure 4: Frequency response captured by sensors 10 and 0 for different HTs:
red and blue colors represent the Trojan active and inactive cases,
respectively.

For each of the 16 sensors, EM traces are recorded under five scenarios: when
HTs T1, T2, T3, and T4 are individually activated and in the absence of any
active HT. For always-on HTs, we use external enable signals to activate them.
Each trace spans a frequency band from DC to 120MHz, populated with 2000
sample points. We averaged five collected traces to derive the spectrum shown
in Figure 4. Notably, when examining the output from Sensor 10, as shown in
Figure 4(a), 4(b), 4(c), and 4(d), two prominent frequency components at 48MHz
and 84MHz, the sideband frequency components of the 1st and 3rd order clock
harmonics, show up in the spectrum when HTs are active. We also record the
traces from Sensor 0 as shown in Figure 4(e), where no HTs are implemented
beneath the sensor. There is hardly any spectrum difference that can
distinguish HT’s activities. This result further proves that our proposed PSA
has the high spatial resolution needed to locate HTs.

In the experiments, we only need fewer than ten traces collected to detect a
HT, resulting in less than 10 ms MTTD, a significant improvement over one
single on-chip coil.

After identifying prominent frequency components related to HT’s activities,
we further switch back to the time domain by employing the zero-span mode of
the spectrum analyzer to examine the time-domain waveforms of the two
frequency components. Zero-span mode allows us to analyze the pattern of a
time-domain signal at a desired fixed frequency. The time-domain waveforms at
48MHz when T1, T2, T3 and T4 are active are shown in Figure 5(a), 5(b), 5(c),
and 5(d). It is shown that even if different Trojans leaked their information
at the same frequency, the difference in their time-domain signals at 48MHz
can still clearly differentiate different Trojans. This is because different
HTs result in different modulation patterns for clock frequency, and the
information on modulation patterns is included in the time-domain waveforms of
the side band frequency components. Therefore, by identifying the unique
patterns in the time-domain signals, we can successfully classify all 4 HTs
without full supervision.

(a) T1 active

(b) T2 active

(c) T3 active

(d) T4 active

Figure 5: Time-domain signals of the identified prominent frequency components
collected from the PSA sensor 10 were recovered with zero-span mode to
differentiate different HTs successfully.

##  VII Conclusion

In this paper, we proposed an integrated on-chip EM sensor array, PSA, that
can be programmed to change its shape, size, and location. The PSA was
implemented and tested in a 65nm technology node. The proposed PSA has a lower
overhead than existing state-of-the-art on-chip coil sensor designs with much
higher SNR and the capability of locating HTs. Furthermore, the MTTD during
the runtime verification has been significantly reduced to less than 10 ms.
The proposed cross-domain analysis can identify prominent frequency components
that include the activity information of different Trojans in the frequency
domain and differentiate different Trojans by examining the time-domain
signals of these prominent frequency components by switching to the time
domain.

##  VIII Acknoledgement

This work was supported by the Office of Naval Research under Award Number
N00014-19-1-2405.

## References

  * [1] J. He, X. Guo, H. Ma, Y. Liu, Y. Zhao, and Y. Jin, “Runtime trust evaluation and hardware trojan detection using on-chip em sensors,” in _2020 57th ACM/IEEE Design Automation Conference (DAC)_ , 2020, pp. 1–6.
  * [2] R. S. Chakraborty, F. Wolff, S. Paul, C. Papachristou, and S. Bhunia, “Mero: A statistical approach for hardware trojan detection,” in _International Workshop on Cryptographic Hardware and Embedded Systems_. Springer, 2009, pp. 396–410.
  * [3] Y. Huang, S. Bhunia, and P. Mishra, “Mers: statistical test generation for side-channel analysis based trojan detection,” in _Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security_ , 2016, pp. 130–141.
  * [4] D. Forte, C. Bao, and A. Srivastava, “Temperature tracking: An innovative run-time approach for hardware trojan detection,” in _2013 IEEE/ACM International Conference on Computer-Aided Design (ICCAD)_ , 2013, pp. 532–539.
  * [5] Y. Bai, A. Stern, J. Park, M. Tehranipoor, and D. Forte, “Rascv2: Enabling remote access to side-channels for mission critical and iot systems,” _ACM Transactions on Design Automation of Electronic Systems (TODAES)_ , vol. 27, no. 6, pp. 1–25, 2022.
  * [6] Y. Bai, J. Park, M. Tehranipoor, and D. Forte, “Real-time instruction-level verification of remote iot/cps devices via side channels,” _Discover Internet of Things_ , vol. 2, no. 1, p. 1, 2022.
  * [7] J. He, Y. Zhao, X. Guo, and Y. Jin, “Hardware trojan detection through chip-free electromagnetic side-channel statistical analysis,” _IEEE Transactions on Very Large Scale Integration (VLSI) Systems_ , vol. 25, no. 10, pp. 2939–2948, 2017.
  * [8] S. Faezi, R. Yasaei, and M. A. Al Faruque, “Htnet: Transfer learning for golden chip-free hardware trojan detection,” in _2021 Design, Automation & Test in Europe Conference & Exhibition (DATE)_, 2021, pp. 1484–1489.
  * [9] L. N. Nguyen, B. B. Yilmaz, M. Prvulovic, and A. Zajic, “A novel golden-chip-free clustering technique using backscattering side channel for hardware trojan detection,” in _2020 IEEE International Symposium on Hardware Oriented Security and Trust (HOST)_ , 2020, pp. 1–12.
  * [10] D. Fujimoto, M. Nagata, S. Bhasin, and J.-L. Danger, “A novel methodology for testing hardware security and trust exploiting on-chip power noise measurement,” in _The 20th Asia and South Pacific Design Automation Conference_ , 2015, pp. 749–754.
  * [11] D. Fujimoto, N. Miura, M. Nagata, Y. Hayashi, N. Homma, Y. Hori, T. Katashita, K. Sakiyama, T.-H. Le, J. Bringer, P. Bazargan-Sabet, and J.-L. Danger, “On-chip power noise measurements of cryptographic vlsi circuits and interpretation for side-channel analysis,” in _2013 International Symposium on Electromagnetic Compatibility_ , 2013, pp. 405–410.
  * [12] J. Rajendran, O. Sinanoglu, and R. Karri, “Is split manufacturing secure?” in _2013 Design, Automation & Test in Europe Conference & Exhibition (DATE)_, 2013, pp. 1259–1264.
  * [13] S. Morioka and A. Satoh, “An optimized s-box circuit architecture for low power aes design,” in _International Workshop on Cryptographic Hardware and Embedded Systems_. Springer, 2002, pp. 172–186.
  * [14] B. Shakya, T. He, H. Salmani, D. Forte, S. Bhunia, and M. Tehranipoor, “Benchmarking of hardware trojans and maliciously affected circuits,” _Journal of Hardware and Systems Security_ , vol. 1, no. 1, pp. 85–102, 2017\.
  * [15] H. Salmani, M. Tehranipoor, and R. Karri, “On design vulnerability analysis and trust benchmarks development,” in _2013 IEEE 31st International Conference on Computer Design (ICCD)_ , 2013, pp. 471–474.


on-shell observables in the one-body sector also control _off-shell_ graviton
exchange processes in the two-body sector, where the binary dynamics is
described by NRGR. By including diagrams with insertions of the multipole
operators $I_{ab}$, $J_{ab}$, it becomes possible to include the effects of
horizon dynamics in the EFT while retaining a worldline description of the
binary constituents.

For example, single graviton exchange between two black holes generates a
tidal friction ($T$-odd) term in the two-body equations of motion associated
with the excitation of horizon modes, leading to a flux of energy across the
event horizon. This is encoded in the box diagram contribution to the in-in
effective action of the binary

$i\Delta\Gamma[x_{A},\eta;{\tilde{x}}_{A},\eta]\supset\includegraphics[width=187.89912pt,valign={c}]{zosobox}+(1\leftrightarrow
2),$ (128)

plus a similar magnetic contribution. Given the form of the correlators
$\langle I^{ab}I^{cd}\rangle$, $\langle J^{ab}J^{cd}\rangle$, this is a non-
local in time quantity. The frictional force on the particle trajectories in
the black hole binary, associated with the absorption of gravitational energy
by the horizons, follow from variation of the effective action,
${\vec{F}}_{1}=-\left.{\vec{F}}_{2}=\delta\Delta\Gamma/\delta{\vec{x}}_{1}\right|_{{\tilde{x}}_{A}=x_{A}}$.

The variation of Eq. (128), with respect to the particle trajectories is
causal, in the sense that it depends only the retarded Green’s function,

$G_{R}^{ab,cd}(\tau)=-i\theta(\tau)\langle[I^{ab}(\tau),I^{cd}(0)]\rangle,$
(129)

of the multipole operators on the black hole worldline. This again is a
consequence of the in-in Feynman rules, together with the operator identity
Eq. (115). Note that the retarded Green’s function is not technically the same
as the Wightman function in Eq. (126) . They are, however, related by a
dispersion relation, which in frequency space reads

$G^{R}(\omega)=i\int_{-\infty}^{\infty}{d\omega^{\prime}\over
2\pi}{A_{+}(\omega^{\prime})-A_{+}(-\omega^{\prime})\over\omega-\omega^{\prime}-i0^{+}},$
(130)

where we have suppressed the tensor structure, which is identical to the one
in Eq. (126).

The dispersion relation implies that the imaginary part of $G^{R}(\omega)$,
responsible for dissipation, is controlled by the Wightman function
$A_{+}(\omega)$ which we obtained by matching the microscopic theory. On the
other hand, the real part of the dispersion relation

$\mbox{Re}G^{R}(\omega)=\mbox{Pr}\int_{0}^{\infty}{d\omega^{\prime}\omega^{\prime}\over\pi}{A_{+}(\omega^{\prime})-A_{+}(-\omega^{\prime})\over\omega^{2}-\omega^{\prime
2}},$ (131)

involves arbitrarily large frequency scales. Evaluation of the RHS becomes
impossible while remaining in the regime of validity of the EFT. In any case,
in the EFT, the real part of the retarded response also receives a
contribution from local counterterms on the black hole’s worldline, e.g. from
those in Eq. (16), so matching the physical response of the black hole
requires knowledge of other observables beyond the graybody factor
$\sigma_{abs}(\omega)$. One would also need an analytic expression for the
elastic graviton scattering amplitude, in the $\omega\rightarrow 0$ limit.

We can get around this by noting that, in light of Eq. (26), we can think of
the static Love number as the $\omega\rightarrow 0$ limit of the AC response
function $G^{R}(\omega)$. Eq. (26) is actually a special case of a more
general linear response relation Goldberger:2005cd ; Chakrabarti:2013lua ;
Goldberger:2020wbx , between the induced moment $\langle I_{ab}\rangle$ on the
surface of the black hole, (which, e.g. determines the long distance
quadrupolar gravitational field as seen by asymptotic observers) and the input
$E_{ab}(x(\tau))$, a slowly varying background field in which the black hole
propagates999Classical general relativity approaches to the motion of black
holes in curved spacetime backgrounds are reviewed in Poisson:2011nh .

$\displaystyle\langle I_{ab}(\omega)\rangle=\int_{-\infty}^{\infty}d\tau
e^{i\omega\tau}\langle I_{ab}(\tau)\rangle$ $\displaystyle=$
$\displaystyle{2\over 3}{r_{s}^{5}\over
G_{N}}k^{E}_{\ell=2}E_{ab}(\omega)+G_{R}^{ab,cd}(\omega)E_{cd}(\omega)$ (132)
$\displaystyle\equiv$ $\displaystyle{2\over 3}{r_{s}^{5}\over
G_{N}}\kappa^{E}_{\ell=2}(\omega)E_{ab}(\omega),$

where $E_{ab}(\omega)=\int d\tau e^{i\omega\tau}E_{ab}(x(\tau))$ and we have
defined, generalizing Eq. (26), the AC Love number of the $d=4$ Schwarzschild
black hole, $\kappa^{E}_{\ell=2}(\omega)$ on the second line of this equation.
The static Love number is then the $\omega\rightarrow 0$ limit of this
quantity,

$\kappa^{E}_{\ell=2}(0)=k^{E}_{\ell=2}+3G_{N}r_{s}^{-5}\int_{0}^{\infty}{d\omega\over
2\pi}{A_{+}(\omega)\over\omega}.$ (133)

Calculations in full general relativity Binnington:2009bb ; Damour:2009va ,
and in the EFT Kol:2011vg , indicate that for Schwarzschild black holes, the
quadrupolar static response vanishes, i.e.

$\kappa^{E}_{\ell=2}(0)=0,$ (134)

at least in $d=4$ spacetime dimensions. This pattern seems to extend to the
static Love response at $\ell>2$ Pani:2015hfa .

If $\kappa^{E}_{\ell=2}(0)=0$, we can intepret Eq. (133) as a sum rule
Rothstein:2014sra ; GnR  enforcing a cancellation between the bare Love number
$k^{E}_{\ell=2}$ and the radiative corrections from the degrees of freedom
localized on the worldline. Given that the latter quantity receives
contributions from UV modes of arbitrarily large frequency, this suggests that
the vanishing of the static response requires a fine tuning of parameters that
is unnatural (in the ‘t Hooft tHooft:1979rat  sense) from the point of view of
the EFT GnR ; Rothstein:2014sra ; Porto:2016zng . The question of wether this
apparent tuning of parameters has a resolution in terms of some hidden
symmetries of the EFT degrees of freedom has been a subject of recent active
study, see Charalambous:2021mea ; Charalambous:2021kcz ; Hui:2021vcv ;
Hui:2022vbh .

Given that the static response is zero, we now have enough information to fix
the low frequency AC Love number,

$\kappa^{E}_{\ell=2}(\omega)\approx-{i\pi\over 15}r_{s}|{\omega}|+\cdots.$
(135)

Inserting this into the box diagram, the instantaneous force on each black
hole is local in time, but not derivable from the variation of a local action,

${\vec{F}}_{1}=-{\vec{F}}_{2}=-{32\over
5}{G_{N}^{7}(m_{1}m_{2})^{2}(m_{1}^{4}+m_{2}^{4})\over|{\vec{x}}_{1}-{\vec{x}}_{2}|}\left[{\vec{v}}_{12}+{2({\vec{v}}_{12}\cdot{\vec{x}}_{12}){\vec{x}}_{12}\over|{\vec{x}}_{12}|^{2}}\right],$
(136)

where ${\vec{x}}_{12}$ and ${\vec{v}}_{12}$ are the relative displacement and
velocity of the binary. This is a 6.5PN correction to the equations of motion,
analogous to the Burke-Thorne radiation damping force from the previous
section. It gives a 4PN correction to the mechanical energy loss of the binary
${\dot{E}}=\sum_{A}{\vec{v}}_{A}\cdot{\vec{F}}_{A}$.

In the case of spinning objects, the correlator is no longer determined by a
single form factor $A^{E,B}_{+}(\omega)$ as more tensor structures, involving
the spin vector of the object, can appear. For non-zero spin, the EFT has been
extended to slowly spinning black holes Porto:2007qi  , to more general
spinning sources in ref. Endlich:2016jgc , and generalized to rapidly spinning
(close to extremal) Kerr black holes Goldberger:2020fot . As in the non-
spinning case, the full causal response is determined by matching to the black
hole absorption rates in staro ; Page:1976df  as well as results of
LeTiec:2020spy ; Chia:2020yla ; LeTiec:2020bos , to fix the local counterterms
(the conservative part of the tidal response) on the worldline Lagrangian to
zero.

For instance, the energy loss in the spinning case, for arbitrary spins

$\left.{dE\over dt}\right|_{h}={8\over
5}{G_{N}^{5}m_{1}^{2}m_{2}\over|{\vec{x}}_{1}-{\vec{x}}_{2}|}(m_{1}+m_{2})\left[1+3\chi_{1}^{2}-{15\over
4}\chi_{1}^{2}\left({{\vec{s}}_{1}\cdot({\vec{x}_{1}}-{\vec{x}}_{2})\over|{\vec{x}}_{1}-{\vec{x}}_{2}|}\right)^{2}\right]{\vec{S}}_{1}\cdot{\vec{L}}+(1\leftrightarrow
2),$ (137)

at leading PN order Goldberger:2020fot
$({\vec{s}}_{A}={\vec{S}}_{A}/|{\vec{S}}_{A}|$). For nearly maximally rotating
black holes, with $\chi=|{\vec{S}}|/G_{N}M\mathrel{\raise
1.29167pt\hbox{$<$\kern-7.5pt\lower 4.30554pt\hbox{$\sim$}}}1,$ Eq. (137) gets
enhanced, by the superradiant effect zeldovich ; staro , to 2.5PN order
compared to 4PN in the Schwarzschild case, $\chi=0$. Notice that the energy
flux can have either sign depending on the relative orientation between the
black hole spin and the orbital angular momentum ${\vec{L}}$. In particular,
it is possible to extract rotational energy from the black holes as in the
Penrose process Penrose:1971uk . Eq. (137) generalizes to arbitrary orbits and
spin orientations earlier results obtained by classical techniques in
Alvi:2001mx ; Chatziioannou:2012gq ; Chatziioannou:2016kem .

#### Quantum effects

Because the  formalism is explicitly quantum mechanical, it is capable of
describing processes involving quantum black holes Hawking:1974sw ;
Hawking:1976ra  interacting with other particles or fields. In particular, it
can be used Goldberger:2019sya ; Goldberger:2020geb  to take into account the
effects of Hawking radiation on scattering observables.

The EFT description assumes that black holes behave according to the standard
quantum mechanical rules, with unitary time evolution and a complete Hilbert
space of microstates. It is valid in the window of momentum transfers $q$
defined by

$t^{-1}_{Page}\ll q\ll 1/G_{N}M_{BH}\ll m_{Pl},$ (138)

where, e.g. in the Schwarzschild case, the black hole evaporation scale is of
order the Page time Page:1976df , $t_{Page}\sim G_{N}^{2}M_{BH}^{3}$. The
upper bound ensures that the worldline description of the black hole is
reliable, while the lower bound allows us to treat black holes as
approximately long-lived asymptotic states in the $S$-matrix. Since
$G_{N}M_{BH}\gg 1/m_{PL}$, the black holes can be regarded as being
semiclassical. Because the time scales are short compared to $t_{Page}$, the
processes we will consider are not sensitive to the apparent loss of unitarity
in black hole evaporation Hawking:1976ra .

To construct the EFT, we have to determine how the structure of multipole
operator correlation functions is modified by the presence of a Hawking
thermal spectrum of radiation emission from the black hole. As in the
classical case, we match to the simplest possible observables that depend, on
the EFT side, on the multipole correlators. It is convenient in particular to
calculate the on-shell transition probabilities $p(m\rightarrow n)$ for a Kerr
black hole to emit $n$ gravitons out to future ${\cal I}^{+}$, given that $m$
particles of the same energy $\omega$ and angular momentum quantum number
$\ell$ are incident on the black hole in the far past from ${\cal I}^{-}$ .
Explicit results101010The results of refs. Bekenstein:1977mv ;
Panangaden:1977pc  are for free scalar fields propagating in the black hole
background, but they generalize naturally to higher spin $s>0$ fields by
simply replacing the scalar transmission coefficients by their higher spin
version found in refs. staro ; Page:1976df . are available for this
observable, in the limit of free quantum field theory in the black hole
background Bekenstein:1977mv ; Panangaden:1977pc  ,

$\displaystyle p_{\ell}(m\rightarrow n)$ $\displaystyle=$
$\displaystyle{(1-x)x^{n}(1-|R_{\ell}|^{2})^{n+m}\over(1-x|R_{\ell}|^{2})^{n+m+1}}$
$\displaystyle\times\sum_{k=0}^{\min(n,m)}{(n+m-k)!\over
k!(n-k)!(m-k)!}\left[{(x^{-1}|R_{\ell}|^{2}-1)(1-x|R_{\ell}|^{2})\over(1-|R_{\ell}|^{2})^{2}}\right]^{k},$

where $x=\exp[-\beta_{H}\omega]$, $|R_{\ell}(\omega)|^{2}$ is the Boltzmann
factor for the Kerr black hole and $|R_{\ell}(\omega)|^{2}$ are the partial
wave reflection coefficients obtained in staro ; Page:1976df .

In , one finds that to leading order in $\omega/m_{Pl}\ll 1$, the
$m\rightarrow n$-particle probabilities are controlled by $n+m$-point Wightman
correlation functions of the multipole operators in Eq. (57). Given the
structure of the full theory result Bekenstein:1977mv ; Panangaden:1977pc ,
these higher points correlators factorize into suitable products of two-point
functions, up to non-Gaussianities suppressed by $\omega^{2}/m^{2}_{Pl}\ll 1$.
Somewhat surprisingly, one finds that at $r_{s}\omega\ll 1$ the effects of
Hawking radiation are not Planck suppressed at the level of the Wightman
functions. Instead, they become _enhanced_ in the limit
$\hbar\omega/T_{H}=4\pi r_{s}\omega\ll 1$ where the EFT is valid, a
consequence of the high temperature behavior of the Planck distribution.

Despite this enhancement at the level of the Wightman functions, in the full
theory the effects of Hawking radiation cancel in the free field _retarded_
correlators. Up to corrections suppressed by $1/m_{Pl}$, these take the same
form in the Unruh state Unruh:1976db  that describes an evaporating black hole
or the Boulware state Boulware:1974dm  where the black hole does not emit
radiation. Consequently, finite size effects associated with the quantized
nature of the black hole horizon, e.g. in the observables discussed in the
previous section, are suppressed by at least one power of
$\omega^{2}/m_{PL}^{2}\ll 1$. Thus, if one assumes that black holes evolve
according to the usual rules of quantum mechanics, the results of
Goldberger:2019sya  imply a no-go theorem on the possibility of detecting
black microstate “hair” or other possible exotic signatures of quantum
behavior in binary black hole mergers at LIGO/VIRGO or any other foreseeable
experiment.

On the other hand, the emission of Hawking radiation does modify observables
that depend on the Wightman functions directly. While not of phenomenological
important, an example Goldberger:2020geb  of formal interest is the inelastic
scattering of elementary particles incident on a semiclassical black hole,
mediated by the exchange of _virtual_ (off-shell) Hawking gravitons.

For illustration, consider a scalar particle $\phi$ with mass in the range
$k_{B}T_{H}\ll m_{\phi}\ll M_{BH}$. In this window, direct $s$-channel
production of $\phi$-particle Hawking pairs is exponentially (Boltzmann)
suppressed, so that the scattering process

$\phi(p)+\mbox{BH}\rightarrow\phi(p^{\prime})+{\mbox{BH}}^{\prime}$ (140)

proceeds instead through off-shell graviton exchange. One finds the result
Goldberger:2020geb

$\displaystyle{d^{3}\sigma\over dq^{2}d(q\cdot v)}$ $\displaystyle\approx$
$\displaystyle{7G_{N}r_{s}^{5}\over 270\pi[(v\cdot
p)^{2}-m_{\phi}^{2}]}\left[(v\cdot p)^{4}-m_{\phi}^{2}(v\cdot
p)^{2}\left(1-{12\over 7}{(v\cdot q)^{2}\over q^{2}}\right)\right.$ (141)
$\displaystyle\left.+{1\over 7}m^{4}\left(1-3{(v\cdot q)^{2}\over
q^{2}}+6{(v\cdot q)^{4}\over q^{4}}\right)\right],$

for the differential cross section in the black hole rest frame
$v^{\mu}=(1,{\vec{0}})$, as a function of the momentum transfer
$q^{\mu}=p^{\mu}-p^{\prime\mu}$. The point of this result is that, in generic
regions of phase space, the integrated cross section scales as $\sim
q^{2}/m^{2}_{Pl}$ relative to the leading order classical gravitational
scattering between point masses $M$, $m_{\phi}$. It is parametrically of the
same size as the sort of ${\cal O}(\hbar)$ corrections from graviton vacuum
polarization loops that appear in scattering processes involving elementary
particles, e.g. Donoghue:1993eb ; Donoghue:1994dn ; Bjerrum-Bohr:2002gqz . Eq.
(141) can be therefore be regarded as a specific realization of a
qualitatively new phenomenon in low energy quantum gravity, associated with
the exchange of virtual Hawking gravitons and tractable (calculable) by the
methods of effective field theory applied to gravity.

##  5 Acknowledgments

This chapter is partially based on lectures delivered at the Pauli Center
workshop _From Scattering Amplitudes to Gravitational-Wave Predictions for
Compact Binaries_ held at the University of Zurich and the ETH Zurich on July
4-15, 2022. I thank the organizers V. Del Duca, H. Ita, and L. Mayer for the
invitation to present this subject, and the students for many stimulating
questions. The research presented here is partially supported by the US
Department of Energy under grant DE-SC00-17660.

## References

  * (1) J. Aasi et al. [LIGO Scientific], “Advanced LIGO,” Class. Quant. Grav. 32, 074001 (2015) [arXiv:1411.4547 [gr-qc]]; F. Acernese et al. [VIRGO], “Advanced Virgo: a second-generation interferometric gravitational wave detector,” Class. Quant. Grav. 32, no.2, 024001 (2015) [arXiv:1408.3978 [gr-qc]]; T. Akutsu et al. [KAGRA], “Overview of KAGRA: Calibration, detector characterization, physical environmental monitors, and the geophysics interferometer,” PTEP 2021, no.5, 05A102 (2021) [arXiv:2009.09305 [gr-qc]].
  * (2) B. P. Abbott et al. [LIGO Scientific and Virgo], “Observation of Gravitational Waves from a Binary Black Hole Merger,” Phys. Rev. Lett. 116, no.6, 061102 (2016) [arXiv:1602.03837 [gr-qc]].
  * (3) L. F. Abbott, “The Background Field Method Beyond One Loop,” Nucl. Phys. B 185, 189-203 (1981)
  * (4) G. L. Almeida, S. Foffa and R. Sturani, “Gravitational multipole renormalization,” Phys. Rev. D 104, no.8, 084095 (2021) [arXiv:2107.02634 [gr-qc]].
  * (5) K. Alvi, “Energy and angular momentum flow into a black hole in a binary,” Phys. Rev. D 64, 104020 (2001) [arXiv:gr-qc/0107080 [gr-qc]].
  * (6) R. L. Arnowitt, S. Deser and C. W. Misner, “The Dynamics of general relativity,” Gen. Rel. Grav. 40, 1997-2027 (2008) [arXiv:gr-qc/0405109 [gr-qc]].
  * (7) H. Asada and T. Futamase, “Propagation of gravitational waves from slow motion sources in Coulomb type potential,” Phys. Rev. D 56, R6062-R6066 (1997) [arXiv:gr-qc/9711009 [gr-qc]].
  * (8) J. D. Bekenstein and A. Meisels, “Einstein $A$ and $B$ Coefficients for a Black Hole,” Phys. Rev. D 15, 2775-2781 (1977)
  * (9) M. Beneke and V. A. Smirnov, “Asymptotic expansion of Feynman integrals near threshold,” Nucl. Phys. B 522, 321-344 (1998) [arXiv:hep-ph/9711391 [hep-ph]].
  * (10) T. Binnington and E. Poisson, “Relativistic theory of tidal Love numbers,” Phys. Rev. D 80, 084018 (2009) [arXiv:0906.1366 [gr-qc]].
  * (11) N. E. J. Bjerrum-Bohr, J. F. Donoghue and B. R. Holstein, “Quantum gravitational corrections to the nonrelativistic scattering potential of two masses,” Phys. Rev. D 67, 084033 (2003) [erratum: Phys. Rev. D 71, 069903 (2005)] [arXiv:hep-th/0211072 [hep-th]].
  * (12) For a review of classical approaches to the PN expansion, as well as more complete references, see L. Blanchet, “Gravitational Radiation from Post-Newtonian Sources and Inspiralling Compact Binaries,” Living Rev. Rel. 17, 2 (2014) [arXiv:1310.1528 [gr-qc]].
  * (13) D. G. Boulware, “Quantum Field Theory in Schwarzschild and Rindler Spaces,” Phys. Rev. D 11, 1404 (1975)
  * (14) A. Buonanno, M. Khalil, D. O’Connell, R. Roiban, M. P. Solon and M. Zeng, “Snowmass White Paper: Gravitational Waves and Scattering Amplitudes,” [arXiv:2204.05194 [hep-th]].
  * (15) C. P. Burgess, “Quantum gravity in everyday life: General relativity as an effective field theory,” Living Rev. Rel. 7, 5-56 (2004) [arXiv:gr-qc/0311082 [gr-qc]].
  * (16) W. L. Burke and K. S. Thorne, in “Relativity”, edited by M. Carmeli, S. I. Fickler, and L. Witten, pp. 209-228, Plenum, New York, 1970.
  * (17) H. B. Callen and T. A. Welton, “Irreversibility and generalized noise,” Phys. Rev. 83, 34-40 (1951)
  * (18) E. Calzetta and B. L. Hu, Phys. Rev. D 35, 495 (1987).
  * (19) E. A. Calzetta and B. L. B. Hu, Cambridge University Press, 2022, ISBN 978-1-00-929003-6, 978-1-00-928998-6, 978-1-00-929002-9, 978-0-511-42147-1, 978-0-521-64168-5.
  * (20) W. E. Caswell and G. P. Lepage, “Effective Lagrangians for Bound State Problems in QED, QCD, and Other Field Theories,” Phys. Lett. B 167, 437-442 (1986)
  * (21) S. Chakrabarti, T. Delsate and J. Steinhoff, “New perspectives on neutron star and black hole spectroscopy and dynamic tides,” [arXiv:1304.2228 [gr-qc]].
  * (22) P. Charalambous, S. Dubovsky and M. M. Ivanov, “On the Vanishing of Love Numbers for Kerr Black Holes,” JHEP 05, 038 (2021) [arXiv:2102.08917 [hep-th]].
  * (23) P. Charalambous, S. Dubovsky and M. M. Ivanov, “Hidden Symmetry of Vanishing Love Numbers,” Phys. Rev. Lett. 127, no.10, 101101 (2021) [arXiv:2103.01234 [hep-th]].
  * (24) K. Chatziioannou, E. Poisson and N. Yunes, “Tidal heating and torquing of a Kerr black hole to next-to-leading order in the tidal coupling,” Phys. Rev. D 87, no.4, 044022 (2013) [arXiv:1211.1686 [gr-qc]].
  * (25) K. Chatziioannou, E. Poisson and N. Yunes, “Improved next-to-leading order tidal heating and torquing of a Kerr black hole,” Phys. Rev. D 94, no.8, 084043 (2016) [arXiv:1608.02899 [gr-qc]].
  * (26) H. S. Chia, “Tidal deformation and dissipation of rotating black holes,” Phys. Rev. D 104, no.2, 024013 (2021) [arXiv:2010.07300 [gr-qc]].
  * (27) T. Damour and O. M. Lecian, “On the gravitational polarizability of black holes,” Phys. Rev. D 80, 044017 (2009) [arXiv:0906.3003 [gr-qc]].
  * (28) P. D. D’Eath, “Dynamics of a small black hole in a background universe,” Phys. Rev. D 11, 1387-1403 (1975)
  * (29) L. V. Delacrétaz, S. Endlich, A. Monin, R. Penco and F. Riva, “(Re-)Inventing the Relativistic Wheel: Gravity, Cosets, and Spinning Objects,” JHEP 11, 008 (2014) [arXiv:1405.7384 [hep-th]].
  * (30) B. S. DeWitt, “Quantum Theory of Gravity. 1. The Canonical Theory,” Phys. Rev. 160, 1113-1148 (1967)
  * (31) B. S. DeWitt, “Quantum Theory of Gravity. 2. The Manifestly Covariant Theory,” Phys. Rev. 162, 1195-1239 (1967)
  * (32) B. S. DeWitt, “Quantum Theory of Gravity. 3. Applications of the Covariant Theory,” Phys. Rev. 162, 1239-1256 (1967)
  * (33) J. F. Donoghue, “Leading quantum correction to the Newtonian potential,” Phys. Rev. Lett. 72, 2996-2999 (1994) [arXiv:gr-qc/9310024 [gr-qc]].
  * (34) J. F. Donoghue, “General relativity as an effective field theory: The leading quantum corrections,” Phys. Rev. D 50, 3874-3888 (1994) [arXiv:gr-qc/9405057 [gr-qc]].
  * (35) J. F. Donoghue, “Introduction to the effective field theory description of gravity,” [arXiv:gr-qc/9512024 [gr-qc]].
  * (36) A. Einstein, Annalen Phys. 49, no.7, 769-822 (1916)
  * (37) S. Endlich and R. Penco, “A Modern Approach to Superradiance,” JHEP 05, 052 (2017) [arXiv:1609.06723 [hep-th]].
  * (38) E. E. Flanagan and T. Hinderer, “Constraining neutron star tidal Love numbers with gravitational wave detectors,” Phys. Rev. D 77, 021502 (2008) [arXiv:0709.1915 [astro-ph]].
  * (39) R. P. Feynman, “Quantum theory of gravitation,” Acta Phys. Polon. 24, 697-722 (1963)
  * (40) S. Foffa and R. Sturani, “Effective field theory calculation of conservative binary dynamics at third post-Newtonian order,” Phys. Rev. D 84, 044031 (2011) [arXiv:1104.1122 [gr-qc]].
  * (41) S. Foffa and R. Sturani, “Dynamics of the gravitational two-body problem at fourth post-Newtonian order and at quadratic order in the Newton constant,” Phys. Rev. D 87, no.6, 064011 (2013) [arXiv:1206.7087 [gr-qc]].
  * (42) S. Foffa and R. Sturani, “Effective field theory methods to model compact binaries,” Class. Quant. Grav. 31, no.4, 043001 (2014) [arXiv:1309.3474 [gr-qc]].
  * (43) S. Foffa, “Gravitating binaries at 5PN in the post-Minkowskian approximation,” Phys. Rev. D 89, no.2, 024019 (2014) [arXiv:1309.3956 [gr-qc]].
  * (44) S. Foffa, P. Mastrolia, R. Sturani and C. Sturm, “Effective field theory approach to the gravitational two-body dynamics, at fourth post-Newtonian order and quintic in the Newton constant,” Phys. Rev. D 95, no.10, 104009 (2017) [arXiv:1612.00482 [gr-qc]].
  * (45) R. Fujita, “Gravitational radiation for extreme mass ratio inspirals to the 14th post-Newtonian order,” Prog. Theor. Phys. 127, 583-590 (2012) [arXiv:1104.5615 [gr-qc]].
  * (46) C. R. Galley and M. Tiglio, “Radiation reaction and gravitational waves in the effective field theory approach,” Phys. Rev. D 79, 124027 (2009) [arXiv:0903.1122 [gr-qc]].
  * (47) C. R. Galley, A. K. Leibovich, R. A. Porto and A. Ross, “Tail effect in gravitational radiation reaction: Time nonlocality and renormalization group evolution,” Phys. Rev. D 93, 124010 (2016) [arXiv:1511.07379 [gr-qc]].
  * (48) H. Georgi, “On-shell effective field theory,” Nucl. Phys. B 361, 339-350 (1991)
  * (49) J. B. Gilmore and A. Ross, “Effective field theory calculation of second post-Newtonian binary dynamics,” Phys. Rev. D 78, 124021 (2008) [arXiv:0810.1328 [gr-qc]].
  * (50) W. D. Goldberger and M. B. Wise, “Renormalization group flows for brane couplings,” Phys. Rev. D 65, 025011 (2002) [arXiv:hep-th/0104170 [hep-th]].
  * (51) W. D. Goldberger and I. Z. Rothstein, “An Effective field theory of gravity for extended objects,” Phys. Rev. D 73, 104029 (2006) [arXiv:hep-th/0409156 [hep-th]].
  * (52) W. D. Goldberger and I. Z. Rothstein, “Dissipative effects in the worldline approach to black hole dynamics,” Phys. Rev. D 73, 104030 (2006) [arXiv:hep-th/0511133 [hep-th]].
  * (53) W. D. Goldberger and I. Z. Rothstein, “Towers of Gravitational Theories,” Gen. Rel. Grav. 38, 1537-1546 (2006) [arXiv:hep-th/0605238 [hep-th]].
  * (54) W. D. Goldberger, “Les Houches lectures on effective field theories and gravitational radiation,” [arXiv:hep-ph/0701129 [hep-ph]].
  * (55) W. D. Goldberger and A. Ross, “Gravitational radiative corrections from effective field theory,” Phys. Rev. D 81, 124015 (2010) [arXiv:0912.4254 [gr-qc]].
  * (56) W. D. Goldberger, A. Ross and I. Z. Rothstein, “Black hole mass dynamics and renormalization group evolution,” Phys. Rev. D 89, no.12, 124033 (2014) [arXiv:1211.6095 [hep-th]].
  * (57) W. D. Goldberger and I. Z. Rothstein, “An Effective Field Theory of Quantum Mechanical Black Hole Horizons,” JHEP 04, 056 (2020) [arXiv:1912.13435 [hep-th]].
  * (58) W. D. Goldberger and I. Z. Rothstein, “Virtual Hawking Radiation,” Phys. Rev. Lett. 125, no.21, 211301 (2020) [arXiv:2007.00726 [hep-th]].
  * (59) W. D. Goldberger and I. Z. Rothstein, “Horizon radiation reaction forces,” JHEP 10, 026 (2020) [arXiv:2007.00731 [hep-th]].
  * (60) W. D. Goldberger, J. Li and I. Z. Rothstein, “Non-conservative effects on spinning black holes from world-line effective field theory,” JHEP 06, 053 (2021) [arXiv:2012.14869 [hep-th]].
  * (61) W. D. Goldberger, “Effective field theories of gravity and compact binary dynamics: A Snowmass 2021 whitepaper,” [arXiv:2206.14249 [hep-th]].
  * (62) W. D. Goldberger and I. Z. Rothstein, unpublished.
  * (63) M. H. Goroff and A. Sagnotti, “The Ultraviolet Behavior of Einstein Gravity,” Nucl. Phys. B 266 (1986), 709-736
  * (64) B. Grinstein and I. Z. Rothstein, “Effective field theory and matching in nonrelativistic gauge theories,” Phys. Rev. D 57, 78-82 (1998) [arXiv:hep-ph/9703298 [hep-ph]].
  * (65) S. N. Gupta, “Quantization of Einstein’s gravitational field: general treatment,” Proc. Phys. Soc. A 65, 608-619 (1952)
  * (66) R. O. Hansen, “Multipole moments of stationary space-times,” J. Math. Phys. 15, 46-52 (1974)
  * (67) A. J. Hanson and T. Regge, “The Relativistic Spherical Top,” Annals Phys. 87, 498 (1974)
  * (68) S. W. Hawking, “Particle Creation by Black Holes,” Commun. Math. Phys. 43, 199 (1975) Erratum: [Commun. Math. Phys. 46, 206 (1976)].
  * (69) S. W. Hawking, “Breakdown of Predictability in Gravitational Collapse,” Phys. Rev. D 14, 2460-2473 (1976)
  * (70) T. Hinderer, “Tidal Love numbers of neutron stars,” Astrophys. J. 677, 1216-1220 (2008) [arXiv:0711.2420 [astro-ph]].
  * (71) T. Hinderer, B. D. Lackey, R. N. Lang and J. S. Read, “Tidal deformability of neutron stars with realistic equations of state and their gravitational wave signatures in binary inspiral,” Phys. Rev. D 81, 123016 (2010) [arXiv:0911.3535 [astro-ph.HE]].
  * (72) L. Hui, A. Joyce, R. Penco, L. Santoni and A. R. Solomon, “Ladder symmetries of black holes. Implications for love numbers and no-hair theorems,” JCAP 01, no.01, 032 (2022) [arXiv:2105.01069 [hep-th]].
  * (73) L. Hui, A. Joyce, R. Penco, L. Santoni and A. R. Solomon, “Near-Zone Symmetries of Kerr Black Holes,” [arXiv:2203.08832 [hep-th]].
  * (74) L. V. Keldysh, “Diagram technique for nonequilibrium processes,” Zh. Eksp. Teor. Fiz. 47, 1515-1527 (1964)
  * (75) I. B. Khriplovich and A. A. Pomeransky, “Tail of gravitational radiation and Coulomb final state interaction,” Phys. Lett. A 252, 17-19 (1999) [arXiv:gr-qc/9712040 [gr-qc]].
  * (76) B. Kol and M. Smolkin, “Classical Effective Field Theory and Caged Black Holes,” Phys. Rev. D 77, 064033 (2008) [arXiv:0712.2822 [hep-th]].
  * (77) B. Kol and M. Smolkin, “Non-Relativistic Gravitation: From Newton to Einstein and Back,” Class. Quant. Grav. 25, 145011 (2008) [arXiv:0712.4116 [hep-th]].
  * (78) B. Kol, M. Levi and M. Smolkin, “Comparing space+time decompositions in the post-Newtonian limit,” Class. Quant. Grav. 28, 145021 (2011) [arXiv:1011.6024 [gr-qc]].
  * (79) B. Kol and M. Smolkin, “Black hole stereotyping: Induced gravito-static polarization,” JHEP 02, 010 (2012) [arXiv:1110.3764 [hep-th]].
  * (80) D. A. Kosower, B. Maybee and D. O’Connell, JHEP 02, 137 (2019) [arXiv:1811.10950 [hep-th]].
  * (81) For a review, see L. Lehner and F. Pretorius, “Numerical Relativity and Astrophysics,” Ann. Rev. Astron. Astrophys. 52, 661-694 (2014) [arXiv:1405.4840 [astro-ph.HE]].
  * (82) A. Le Tiec and M. Casals, “Spinning Black Holes Fall in Love,” Phys. Rev. Lett. 126, no.13, 131102 (2021) [arXiv:2007.00214 [gr-qc]].
  * (83) A. Le Tiec, M. Casals and E. Franzin, “Tidal Love Numbers of Kerr Black Holes,” Phys. Rev. D 103, no.8, 084021 (2021) [arXiv:2010.15795 [gr-qc]].
  * (84) M. Levi, “Effective Field Theories of Post-Newtonian Gravity: A comprehensive review,” Rept. Prog. Phys. 83, no.7, 075901 (2020) [arXiv:1807.01699 [hep-th]].
  * (85) M. E. Luke, A. V. Manohar and I. Z. Rothstein, “Renormalization group scaling in nonrelativistic QCD,” Phys. Rev. D 61, 074025 (2000) [arXiv:hep-ph/9910209 [hep-ph]].
  * (86) M. Maggiore, Oxford University Press, 2007, ISBN 978-0-19-857074-5, 978-0-19-852074-0
  * (87) B. Modrekiladze, “Gravitational Wave Signals from Finite Size Effects in Spinning Binary Inspirals Including Parity Violating Constituents,” [arXiv:2204.00028 [hep-th]].
  * (88) D. N. Page, “Particle Emission Rates from a Black Hole: Massless Particles from an Uncharged, Nonrotating Hole,” Phys. Rev. D 13, 198-206 (1976)
  * (89) P. Panangaden and R. M. Wald, “Probability Distribution for Radiation from a Black Hole in the Presence of Incoming Radiation,” Phys. Rev. D 16, 929-932 (1977)
  * (90) P. Pani, L. Gualtieri, A. Maselli and V. Ferrari, “Tidal deformations of a spinning compact object,” Phys. Rev. D 92, no.2, 024010 (2015) [arXiv:1503.07365 [gr-qc]].
  * (91) R. Penrose, “Gravitational collapse: The role of general relativity,” Riv. Nuovo Cim. 1, 252-276 (1969)
  * (92) R. Penrose and R. M. Floyd, “Extraction of rotational energy from a black hole,” Nature 229, 177-179 (1971)
  * (93) E. Poisson and M. Sasaki, “Gravitational radiation from a particle in circular orbit around a black hole. 5: Black hole absorption and tail corrections,” Phys. Rev. D 51, 5753-5767 (1995) [arXiv:gr-qc/9412027 [gr-qc]].
  * (94) E. Poisson, “Absorption of mass and angular momentum by a black hole: Time-domain formalisms for gravitational perturbations, and the small-hole / slow-motion approximation,” Phys. Rev. D 70, 084044 (2004) [arXiv:gr-qc/0407050 [gr-qc]].
  * (95) E. Poisson, A. Pound and I. Vega, Living Rev. Rel. 14, 7 (2011) doi:10.12942/lrr-2011-7 [arXiv:1102.0529 [gr-qc]].
  * (96) R. A. Porto, “Post-Newtonian corrections to the motion of spinning bodies in NRGR,” Phys. Rev. D 73, 104031 (2006) [arXiv:gr-qc/0511061 [gr-qc]].
  * (97) R. A. Porto, “Absorption effects due to spin in the worldline approach to black hole dynamics,” Phys. Rev. D 77, 064026 (2008) [arXiv:0710.5150 [hep-th]].
  * (98) R. A. Porto, A. Ross and I. Z. Rothstein, “Spin induced multipole moments for the gravitational wave amplitude from binary inspirals to 2.5 Post-Newtonian order,” JCAP 09, 028 (2012) [arXiv:1203.2962 [gr-qc]].
  * (99) R. A. Porto, “The effective field theorist’s approach to gravitational dynamics,” Phys. Rept. 633, 1-104 (2016) [arXiv:1601.04914 [hep-th]].
  * (100) R. A. Porto, “The Tune of Love and the Nature(ness) of Spacetime,” Fortsch. Phys. 64, no.10, 723-729 (2016) [arXiv:1606.08895 [gr-qc]].
  * (101) A. Ross, “Multipole expansion at the level of the action,” Phys. Rev. D 85, 125033 (2012) [arXiv:1202.4750 [gr-qc]].
  * (102) I. Z. Rothstein, “Progress in effective field theory approach to the binary inspiral problem,” Gen. Rel. Grav. 46, 1726 (2014)
  * (103) For a review, see M. Sasaki and H. Tagoshi, “Analytic black hole perturbation approach to gravitational radiation,” Living Rev. Rel. 6, 6 (2003) [arXiv:gr-qc/0306120 [gr-qc]].
  * (104) G. Schäfer and P. Jaranowski, “Hamiltonian formulation of general relativity and post-Newtonian dynamics of compact binaries,” Living Rev. Rel. 21, no.1, 7 (2018) [arXiv:1805.07240 [gr-qc]].
  * (105) J. S. Schwinger, “Brownian motion of a quantum oscillator,” J. Math. Phys. 2, 407-432 (1961)
  * (106) V. A. Smirnov,“Feynman integral calculus,” Springer-Verlag Berlin Heidelberg 2006, ISBN 978-3-540-30610-8,978-3-642-06789-1,978-3-540-30611-5.
  * (107) A. Sommerfeld, “Über die Beugung und Bremsung der Elektronen,” Annalen Phys. 403, no.3, 257-330 (1931)
  * (108) A. A. Starobinskii and S. M. Churilov, Zh. Eksp. Teor. Fiz. 65, 3 (1973).
  * (109) H. Tagoshi, S. Mano and E. Takasugi, “PostNewtonian expansion of gravitational waves from a particle in circular orbits around a rotating black hole: Effects of black hole absorption,” Prog. Theor. Phys. 98, 829-850 (1997) [arXiv:gr-qc/9711072 [gr-qc]].
  * (110) G. ’t Hooft and M. J. G. Veltman, “One loop divergencies in the theory of gravitation,” Ann. Inst. H. Poincare Phys. Theor. A 20, 69 (1974).
  * (111) G. ’t Hooft, NATO Sci. Ser. B 59, 135-157 (1980) doi:10.1007/978-1-4684-7571-5_9 Copy to ClipboardDownload
  * (112) W. G. Unruh, “Notes on black hole evaporation,” Phys. Rev. D 14, 870 (1976)
  * (113) S. Weinberg, “Photons and Gravitons in $S$-Matrix Theory: Derivation of Charge Conservation and Equality of Gravitational and Inertial Mass,” Phys. Rev. 135, B1049-B1056 (1964)
  * (114) S. Weinberg, “Photons and gravitons in perturbation theory: Derivation of Maxwell’s and Einstein’s equations,” Phys. Rev. 138, B988-B1002 (1965)
  * (115) S. Weinberg, “Infrared photons and gravitons,” Phys. Rev. 140, B516-B524 (1965)
  * (116) S. Weinberg, “Gravitation and Cosmology: Principles and Applications of the General Theory of Relativity,” John Wiley and Sons, 1972, ISBN 978-0-471-92567-5, 978-0-471-92567-5
  * (117) S. Weinberg, “Quantum contributions to cosmological correlations,” Phys. Rev. D 72, 043514 (2005) [arXiv:hep-th/0506236 [hep-th]].
  * (118) S. Weinzierl, “Feynman Integrals,” [arXiv:2201.03593 [hep-th]].
  * (119) Ya. B. Zel’dovich, Zh. Eksp. Teor. Fiz. Pis’ma 14, 270 (1971) [JETP Letters 14, 180 (1971)].
  * (120) A. Zimmerman, “An Introduction to Gravitational Wave Data Analysis.” Talk given at the workshop _High-Precision Gravitational Waves_ , Kavli Insitute for Theoretical Physics, Santa Barbara, California. May 13 2022. https://online.kitp.ucsb.edu/online/gwaves22/zimmerman/


# ColibriES: A Milliwatts RISC-V Based Embedded System Leveraging Neuromorphic
and Neural Networks Hardware Accelerators for Low-Latency Closed-loop Control
Applications

Georg Rutishauser1, Robin Hunziker1, Alfio Di Mauro1, Sizhen Bian1, Luca
Benini12, Michele Magno1  1Departement Informationstechnologie und
Elektrotechnik, ETH Zürich, Switzerland 2Dipartimento di Ingegneria
dell’Energia Elettrica e dell’Informazione, Università di Bologna, Bologna,
Italy

###### Abstract

End-to-end event-based computation has the potential to push the envelope in
latency and energy efficiency for edge AI applications. Unfortunately, event-
based sensors (e.g., DVS cameras) and neuromorphic spike-based processors
(e.g., Loihi) have been designed in a decoupled fashion, thereby missing major
streamlining opportunities. This paper presents ColibriES, the first-ever
neuromorphic hardware embedded system platform with dedicated event-sensor
interfaces and full processing pipelines. ColibriES includes event and frame
interfaces and data processing, aiming at efficient and long-life embedded
systems in edge scenarios. ColibriES is based on the Kraken system-on-chip and
contains a heterogeneous parallel ultra-low power (PULP) processor, frame-
based and event-based camera interfaces, and two hardware accelerators for the
computation of both event-based spiking neural networks and frame-based
ternary convolutional neural networks. This paper explores and accurately
evaluates the performance of event data processing on the example of gesture
recognition on ColibriES, as the first step of full-system evaluation. In our
experiments, we demonstrate a chip energy consumption of 7.7 $\mathrm{mJ}$ and
latency of 164.5 $\mathrm{ms}$ of each inference with the DVS Gesture event
data set as an example for closed-loop data processing, showcasing the
potential of ColibriES for battery-powered applications such as wearable
devices and UAVs that require low-latency closed-loop control.

##  I Introduction

Neuromorphic computing is a promising paradigm for low latency, energy-
efficient artificial intelligence [1] for a wide range of embedded
applications [2]. Complementing this novel processing paradigm, event-based
sensors have the potential to unlock the full potential of neuromorphic
hardware for applications such as intelligent sensing and ultra-fast robotics
[3]. On the other hand, frame-based processing using deep neural networks
(DNNs) is currently offering impressive performance and is being heavily tuned
for low-power, and energy-efficient embedded platforms [4].

Combining the advantages of event-based image sensors, neuromorphic
processing, traditional image sensors, and deep neural networks yields hybrid
hardware and software systems that can outperform the current state of the art
in low-latency, energy-efficient systems like unmanned vehicles. For instance,
optical flow estimation is one of the most used methods for autonomous
navigation. Improving its speed and accuracy will improve the entire
autonomous vehicles field [5]. Our work aims at building the first hybrid
system combining frame information with DNN-based processing and event
information with SNN-based processing for autonomous vehicles and robots [6].

###  I-A Motivation

TABLE I: Comparison of ColibriES to recent work

| [7] | [8] | [9] | [10] | [11] | ColibriES (ours)
---|---|---|---|---|---|---
Platform | Loihi | TrueNorth | Loihi | Loihi | TrueNorth | Kraken (SNE)
Application a | KWS | KWS | GR | GR | GR | GR
Input Data | MFCC d | Audio | DVS + sEMG Evts. d | DVS Evts. | DVS Evts. | DVS Evts.
Dataset | Custom | TIDIGITS | Custom | DVS128 | DVS128 | DVS128
# of Classes | 2 | 11 | 5 | 11 | 11 | 11
Network b | 2L SMLP | 7L SCNN | 6L SCNN + 4L SMLP | 5L SCNN | 16L SCNN | 7L SCNN
Accuracy | $95.9\%$ |  $90.8\%$-$95.1\%$ e | $96\%$ | $90.5\%$ |  $86.5\%$-$94.6\%$ e | $83\%$
End-to-End? | ✗ | ✓ | ✗ | ✗ | ✓ | ✓
$P_{proc,inf}$ ($\mathrm{mW}$) b | 110 | 14.4 (85)-38.6 (91) e | 141.9 | N/A | 88.5 - 178.8 | 35.6
$P_{proc,idle}$ ($\mathrm{mW}$) b | 29.2 | 12.1 (82.7) - 30.3 (82.7) e | 29.2 [7] | 29.2 [7] | 68.8 - 134.4 f | 17.7
$E_{proc,inf}$ ($\mathrm{mJ}$) d | 0.371 | 15.9 (93.6) - 42.5 (100.1) g | 5.9 | N/A | 29.8 | 7.7

  * a

KWS: Keyword Spotting, GR: Gesture Recognition

  * b

$\bm{N}$L SCNN: $N$-layer spiking convolutional neural network, $\bm{N}$L
SMLP: $N$-layer spiking multi-layer perceptron

  * c

$P_{proc,\\{inf,idle\\}}$: Processor chip/full system chip power during
inference/while idle

  * d

$E_{proc,inf}$: Processor chip/full system energy per inference. Energy
normalized to 6 inf/s for GR.

  * e

Preprocessing not included in energy measurements.

  * f

Lowest-power and highest-accuracy implementations. Numbers in parentheses are
energy/power omit scaling for resource usage.

  * g

We assume the preprocessing and classification pipeline needs to run for the
entire length of a sample ($1104\text{\,}\mathrm{ms}$).

Most current systems using visual sensing to perform intelligent tasks rely on
frame-based cameras [12]. Conventional video sensors record the entire image
at a fixed rate and resolution. These vision systems use sequences of RGB
frames, which are heavily redundant and inefficient to process. Event-based
cameras (silicon retinas) have introduced a new vision paradigm by mimicking
the biological retina. Instead of measuring the intensity of every pixel at
fixed time intervals, they report events of significant pixel intensity
changes. Each such event is represented by its position, the sign of change,
and timestamp in microsecond resolution. Because of their asynchronous
operation principle, they are a natural match for spiking neural networks
(SNNs) [13] rather than DNNs.

State-of-the-art DNN approaches in machine learning provide excellent results
for vision tasks with standard cameras. In contrast, asynchronous event
sequences require special handling, and SNNs can take advantage of this
asynchronicity, offering very low detection latency in the order of
microseconds [14]. However, SNN-based algorithms are still evolving, and
current SNNs are still not comparable in terms of accuracy to DNN-based
approaches.

Another aspect limiting the employment of energy-proportional event-based
cameras in real application scenarios requiring ultra-low-power devices is the
high power consumption spent at such sensors’ interfaces. Event cameras often
expose non-standard communication protocols, and data acquisition is commonly
carried out with a high-power field-programmable gate array (FPGA), and a USB
interface [15].

Neuromorphic processors execute SNNs on large-scale integrated circuits [16]
by mimicking the natural biological structures of the nervous system,
implementing neurons and synapses directly in silicon. A typical neuromorphic
processor can simulate hundreds of thousands or even millions of spiking
neurons in real time. However, to achieve truly power-efficient real-time
operation, such platforms should also be equipped with microcontroller
subsystems capable of performing actuation and closed-loop control. Even
advanced prototypes, e.g., the Loihi Kapoho Bay board, do not present such
features as they are generally conceived as research and evaluation testbeds.
Kraken, a novel RISC-V-based system-on-chip (SoC) addressing this challenge,
has been recently proposed [17]. Kraken includes a heterogeneous parallel
ultra-low power (PULP) processor [18] and two hardware accelerators for both
spiking neural networks and frame-based convolutional neural networks. A key
feature of the Kraken SoC is that both RGB camera and event-based camera
interfaces are present, as well as the general-purpose input output pins
useful for direct control of motors and other devices. Kraken can deliver up
to 2TOP/s/W 2bit SIMD on the RISC-V cluster and 1.1TSynOP/s/W on the
neuromorphic accelerator, which makes it suitable for embedded battery-
operated devices requiring low-latency closed-loop control. Table I compares
our proposed ColibriES system with previous works implementing advanced tasks
on neuromorphic processors. The only published works implementing true end-to-
end neuromorphic applications are based on IBM’s TrueNorth platform [19], a
purely neuromorphic processor that lacks the advanced compute and control
capabilities enabled by the Kraken SoC, relying solely on SNNs for all
processing tasks, which in turn restricts the range of tasks that can be
solved on the platform.

###  I-B Contribution

This paper presents ColibriES, the first-ever neuromorphic truly embedded
evaluation platform hosting the Kraken SoC. The ColibriES board was designed
to accurately measure the complete processing pipeline’s power consumption,
energy, and latency. ColibriES can connect to both event- and frame-based
cameras that directly interface with the Kraken SoC. The platform is designed
for battery-powered applications requiring low-latency closed-loop control
feedback, such as wearable electronics, unmanned aerial vehicles (UAVs), and
other applications that impose stringent requirements on the computational and
energy resources available to perform the processing. Figure 1 shows the high-
level architecture of the proposed platform and provides an overview of how
the system could be used as a high-speed closed-loop controller. Evaluations
of power consumption, functionality, and energy efficiency are presented with
experimental measurements, demonstrating the versatility and low power of
ColibriES.

Figure 1: Architecture of the embedded platform developed as evaluation board
hosting the novel Kraken SoC, which includes PWM pins for high-speed control.

The rest of this paper is organized as follows: Section II describes the
proposed ColibriES platform with a particular focus on Kraken’s architectural
features. Section III shows the experimental results, and section IV concludes
the paper.

##  II System Architecture

In this section, we present the heart of ColibriES, the heterogeneous Kraken
SoC, and the evaluation board used in our experiments.

###  II-A Kraken SoC

Figure 2: Kraken SoC block diagram

The Kraken SoC has a heterogeneous architecture composed of three main
subsystems. Fig. 2 shows a block diagram representation of the Kraken chip.
The first subsystem, the fabric controller (FC), is built around a 32-bit
RISC-V core which acts as the central programmable control unit for the whole
SoC. The FC hosts the main interconnection busses to the main L2 memory and
the advanced peripheral bus (APB), which controls all the SoC peripherals. The
FC domain also contains a full set of on-chip peripherals such as a RISC-V
compliant debug unit, four programmable timer-counters and a power management
unit.

The second subsystem is a RISC-V-based general-purpose accelerator called the
cluster. The cluster domain hosts eight RISC-V cores enhanced with specialized
instruction set architecture (ISA) extensions like hardware loops, multiply
and accumulate, and vectorial instructions for low-precision machine learning
(ML) workloads. Each RISC-V core of the cluster accelerator is equipped with a
private floating point unit (FPU) and implements the full 32-bit RISC-V
floating-point ISA extension. The cluster also features a
$128\text{\,}\mathrm{K}$b L1 tightly coupled data memory (TCDM). The L1 memory
can serve memory requests from all cluster cores in parallel with single-cycle
access latency and a low average contention rate (<10% even on the most data-
intensive kernels). Fast event management, parallel thread dispatching, and
synchronization are supported by a dedicated hardware block (HW Sync),
enabling very fine-grained parallelism and high energy efficiency in parallel
workloads. The cluster can be clock-gated at single-core granularity, reducing
the dynamic power consumption while waiting for core synchronization.

The third domain, the external hardware processing engine (EHWPE), hosts two
accelerators. One is the sparse neural engine (SNE) neuromorphic accelerator
presented in [17], and the other is CUTIE, a ternary weight neural network
accelerator presented in [20]. The EHWPE domain operates on two independent
clock domains. One clock is generated by the FC frequency locked loop (FLL),
which is used to clock the interface logic at the boundary between the
accelerators and the main system interconnect. The other clock is generated by
a dedicated EHWPE FLL, which is used to clock the accelerator’s computing
engines. Like many peripherals of the SoC, the accelerators are programmed via
memory-mapped register interfaces. The FC domain hosts a vast set of IO
peripherals commonly adopted on microcontroller-like SoCs. The chip features
76 digital pads, 44 of which can be used to arbitrarily map any pin of the
peripherals mentioned above to a chip pad in an all-to-all crossbar
configuration, providing high IO mapping flexibility. The pin mapping is
configurable via a memory-mapped register interface connected to the APB. Each
peripheral can generate interrupts depending on data transmission events; such
events can then be routed to the interrupt controller of the RISC-V core.

An autonomous IO subsystem, the µDMA[21], hosts all the IO peripherals. This
system can be programmed to autonomously orchestrate data transfers from the
peripherals to the L2 memory and vice versa, freeing the RISC-V core from
micro-managing such data transfers. Moreover, Kraken’s µDMA is equipped with a
dedicated hardware interface for the ultra-low power dynamic visual sensor
(DVS) presented in [22]. Such an interface was presented in [23] and allows
streaming events directly from the event camera to Kraken.

To carefully control the chip’s overall power consumption and enable a
versatile application-specific power management scheme, Kraken is subdivided
into three independent power domains. Such power domain partitioning allows to
switch-off of unused parts of the system that the running application might
not require. The power-up/down of the controllable domains can be performed at
run time. Figure 2 also shows the chip power domain partitioning, SoC,
Cluster, and EHWPE, respectively. The FC domain is always on because it hosts
essential components needed for SoC management tasks such as the boot
procedure, clock generator initialization, or accelerator reset and
programming.

###  II-B Kraken evaluation board

The Kraken evaluation board integrates the Kraken chip with the external
components needed to implement complete applications. A USB-C connector is
used for JTAG and UART data transfer and supplies a $5\text{\,}\mathrm{V}$
rail from which all other power supplies are derived. Kraken’s three power
domains are supplied by individually runtime-configurable buck converters,
allowing for application-controlled dynamic voltage and frequency scaling
(DVFS). Connectivity to the DVS camera is provided through an 80-pin low-
profile board-to-board connector, which also supplies the camera board with
the required $1.2\text{\,}\mathrm{V}$ and $5\text{\,}\mathrm{V}$ rails. Off-
chip memory is present in the form of a combined HyperFlash/HyperRAM chip and
a quad-SPI flash memory chip, which can also be used to store application code
for standalone booting. Arduino headers and a camera parallel interface (CPI)
ribbon connector provide additional connectivity, with level shifters between
every off-chip connector and Kraken’s I/O pins. An annotated photograph of the
board is shown in Fig. 3.

Figure 3: Kraken evaluation board

##  III Evaluation

TABLE II: DVS Gesture Network Parameters | Type | Size | Feature Size | Features | Stride
---|---|---|---|---|---
0 | Input | 128x128x2 | - | - | -
1 | Pool | 32x32x2 | 4x4x1 | 2 | 4
2 | Conv | 32x32x16 | 3x3x2 | 16 | 1
3 | Pool | 16x16x16 | 2x2x1 | 16 | 2
4 | Conv | 16x16x32 | 3x3x1 | 32 | 1
5 | Pool | 8x8x32 | 2x2x1 | 32 | 2
6 | Full | 512 | 2048 | 512 | -
7 | Full | 11 | 512 | 11 | -

In this section, we evaluate the end-to-end performance of ColibriES when
executing SNN inference as the first step of the full-system evaluation. We
used the IBM DVS Gesture event data set [24], which contains 11 hand gestures
from 29 subjects under three illumination conditions from a DVS128 event
camera. Six subjects’ data are used for testing and the remaining for
training. The SNN used for our experiments is reported in Table II. The
network is composed of two convolutional layers and two fully-connected
layers. To train the network, we used a supervised learning approach based on
back-propagation, derived from the one presented in [25], which has been
implemented in PyTorch. In this framework, the Neuron dynamic of the leaky-
integrate and fire (LIF) neurons implemented in SNE is accurately modeled to
closely reflect the hardware implementation and minimize the mismatch between
the accuracy achieved during training and the one achieved on the real
hardware platform.

We break down the closed loop into three distinct parts: data acquisition on
the FC through the dedicated DVS interface, data processing on the engines,
which includes a spike preprocessing step in the cluster and a spike train
inference step in the SNE, and actuators control using PWM signals. For
advanced processing tasks, neural networks that exceed SNE’s output neuron
capacity are executed on the accelerator in a tiled way, and the SNE is used
in a time-domain-multiplexing fashion. The preprocessing step performed on the
cluster is necessary to assemble a single input event stream from multiple
output tiles and create the tiled input streams for the tiles of the
successive layer. Note that the main system clock of Kraken can vary from a
few $\mathrm{kHz}$ to $200\text{\,}\mathrm{MHz}$, and the typical operating
frequency is $50\text{\,}\mathrm{MHz}$. In typical operating conditions, the
latency of changing an output actuation signal like a PWM signal is a few
system clock cycles, i.e., below $1\text{\,}\mathrm{\SIUnitSymbolMicro s}$.
Therefore, we considered this latency negligible compared to data acquisition
and processing. Table III reports the energy and latency metrics of the
ColibriES system for the gesture recognition task execution. Using a 300
milliseconds window input, Kraken requires 164.5 $\mathrm{ms}$ and 7.7
$\mathrm{mJ}$ to perform a DVS-to-label prediction, meeting real-time
processing constraints. The inference execution on SNE takes only 32
$\mathrm{ms}$ and 1.4 $\mathrm{mJ}$.

TABLE III: Performance metrics of ColibriES on DVS-HandGesture Dataset with
accuracy of 83%

Proc. Stage | Time (ms) | Power ($\mathrm{mW}$)a | Energy ($\mathrm{mJ}$)c
---|---|---|---
Idle | Active
Data Acquisition (FC) | 1.5 | 3.5 | 3.8 | 0.006
Preprocessing (Cluster)c, | 131 | 6.5 | 34 | 4.6
SNN Inference (SNE) | 32 | 7.7 | 44 | 1.4
Total | 164.5 | 17.7 | 35.6 c | 7.7

  * •

All power numbers measured at $V_{DD}=$0.65\text{\,}\mathrm{V}$$

  * a

Idle power measured with respective components clock-gated.

  * b

Total energy is computed as the sum of active energy contributions and idle
energy of inactive components.

  * c

Average total power consumption during inference

##  IV Conclusion

This paper presented ColibriES, an energy-efficient heterogeneous platform
including an event-based camera interface and an RGB interface, together with
the control features of a standard microcontroller. The platform core is the
Kraken system-on-chip, a PULP processor with eight RISC-V cores and two
dedicated hardware accelerators for machine learning: a neuromorphic processor
and a frame base machine learning accelerator. As the first step of system
evaluation on ColibriES, this paper reports the evaluation of end-to-end
gesture recognition from event camera data with the dedicated DVS interface
and the neuromorphic core. Thanks to the direct access to the stream of events
from the event camera, ColibriES can achieve real-time event processing by
executing SNN inference on sensor data. Experimental results show that our
platform can perform low-latency tasks under stringent power constraints, and
it is suitable for battery-powered operation and low-latency applications. On
the IBM DVS Gesture event data set, ColibriES achieves real-time 164ms
processing on 300ms time window samples, consuming only 7.7mJ of total energy
and meeting the severe latency and energy constraints of emerging high-speed
closed-loop control applications, such as autonomous drones and wearable
control embedded systems, while keeping the power in the mW range.

## Acknowledgment

The authors would like to thank armasuisse Science & Technology for funding
this research.

## References

  * [1] C. I. Nwakanma, J.-W. Kim, J.-M. Lee, and D.-S. Kim, “Edge ai prospect using the neuroedge computing system: Introducing a novel neuromorphic technology,” _ICT Express_ , vol. 7, no. 2, pp. 152–157, 2021.
  * [2] A. Vitale, E. Donati, R. Germann, and M. Magno, “Neuromorphic edge computing for biomedical applications: Gesture classification using emg signals,” _IEEE Sensors Journal_ , 2022.
  * [3] A. Lele, Y. Fang, J. Ting, and A. Raychowdhury, “An end-to-end spiking neural network platform for edge robotics: From event-cameras to central pattern generation,” _IEEE Transactions on Cognitive and Developmental Systems_ , 2021.
  * [4] P. K. Huynh, M. L. Varshika, A. Paul, M. Isik, A. Balaji, and A. Das, “Implementing spiking neural networks on neuromorphic architectures: A review,” _arXiv preprint arXiv:2202.08897_ , 2022.
  * [5] Y. Akbari, N. Almaadeed, S. Al-Maadeed, and O. Elharrouss, “Applications, databases and open computer vision research from drone videos and images: a survey,” _Artificial Intelligence Review_ , vol. 54, no. 5, pp. 3887–3938, 2021.
  * [6] M. L. Varshika, A. Balaji, F. Corradi, A. Das, J. Stuijt, and F. Catthoor, “Design of many-core big little $\mu$brains for energy-efficient embedded neuromorphic computing,” in _2022 Design, Automation & Test in Europe Conference & Exhibition (DATE)_. IEEE, 2022, pp. 1011–1016.
  * [7] P. Blouw, X. Choo, E. Hunsberger, and C. Eliasmith, “Benchmarking keyword spotting efficiency on neuromorphic hardware,” _ACM International Conference Proceeding Series_ , 2019.
  * [8] W.-y. Tsai, D. R. Barch, A. S. Cassidy, M. V. DeBole, A. Andreopoulos, B. L. Jackson, M. D. Flickner, J. V. Arthur, D. S. Modha, J. Sampson, and V. Narayanan, “Always-On Speech Recognition Using TrueNorth, a Reconfigurable, Neurosynaptic Processor,” _IEEE Transactions on Computers_ , vol. 66, no. 6, pp. 996–1007, jun 2017. [Online]. Available: http://ieeexplore.ieee.org/document/7750640/
  * [9] E. Ceolini, C. Frenkel, S. B. Shrestha, G. Taverni, L. Khacef, M. Payvand, and E. Donati, “Hand-Gesture Recognition Based on EMG and Event-Based Camera Sensor Fusion: A Benchmark in Neuromorphic Computing,” _Frontiers in Neuroscience_ , vol. 14, no. August, pp. 1–15, aug 2020. [Online]. Available: https://www.frontiersin.org/article/10.3389/fnins.2020.00637/full
  * [10] R. Massa, A. Marchisio, M. Martina, and M. Shafique, “An Efficient Spiking Neural Network for Recognizing Gestures with a DVS Camera on the Loihi Neuromorphic Processor,” _Proceedings of the International Joint Conference on Neural Networks_ , no. July, pp. 1–9, 2020.
  * [11] A. Amir, B. Taba, D. Berg, T. Melano, J. McKinstry, C. Di Nolfo, T. Nayak, A. Andreopoulos, G. Garreau, M. Mendoza, J. Kusnitz, M. Debole, S. Esser, T. Delbruck, M. Flickner, and D. Modha, “A Low Power, Fully Event-Based Gesture Recognition System,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , vol. 27, no. 36. IEEE, jul 2017, pp. 7388–7397. [Online]. Available: http://upecen.edu.pe/ebooks/Administraci{\’{o}}n/Managingtoimprovepublicservices.JeanHartleyet.al..pdf#page=194https://ieeexplore.ieee.org/document/8100264/
  * [12] J. Chai, H. Zeng, A. Li, and E. W. Ngai, “Deep learning in computer vision: A critical review of emerging techniques and application scenarios,” _Machine Learning with Applications_ , vol. 6, p. 100134, 2021.
  * [13] L. Steffen, D. Reichard, J. Weinland, J. Kaiser, A. Roennau, and R. Dillmann, “Neuromorphic stereo vision: A survey of bio-inspired sensors and algorithms,” _Frontiers in neurorobotics_ , vol. 13, p. 28, 2019.
  * [14] C. Pfeiffer, S. Wengeler, A. Loquercio, and D. Scaramuzza, “Visual attention prediction improves performance of autonomous drone racing agents,” _Plos one_ , vol. 17, no. 3, p. e0264471, 2022.
  * [15] P. D. Schiavone, D. Rossi, A. Di Mauro, F. K. Gürkaynak, T. Saxe, M. Wang, K. C. Yap, and L. Benini, “Arnold: An efpga-augmented risc-v soc for flexible and low-power iot end nodes,” _IEEE Transactions on Very Large Scale Integration (VLSI) Systems_ , vol. 29, no. 4, pp. 677–690, 2021.
  * [16] Y. Qiu, C. Xiao, L. Peng, J. Wang, Z. Kang, S. Li, and L. Wang, “A novel ring-based small-world noc for neuromorphic processor,” in _2021 IEEE 32nd International Conference on Application-specific Systems, Architectures and Processors (ASAP)_. IEEE, 2021, pp. 234–241.
  * [17] A. Di Mauro, A. S. Prasad, Z. Huang, M. Spallanzani, F. Conti, and L. Benini, “Sne: an energy-proportional digital accelerator for sparse event-based convolutions,” in _2022 Design, Automation & Test in Europe Conference & Exhibition (DATE)_. IEEE, 2022, pp. 825–830.
  * [18] X. Wang, M. Magno, L. Cavigelli, and L. Benini, “Fann-on-mcu: An open-source toolkit for energy-efficient neural network inference at the edge of the internet of things,” _IEEE Internet of Things Journal_ , vol. 7, no. 5, pp. 4403–4417, 2020.
  * [19] M. V. Debole, B. Taba, A. Amir, F. Akopyan, A. Andreopoulos, W. P. Risk, J. Kusnitz, C. O. Otero, T. K. Nayak, R. Appuswamy, P. J. Carlson, A. S. Cassidy, P. Datta, S. K. Esser, G. J. Garreau, K. L. Holland, S. Lekuch, M. Mastro, J. Mckinstry, C. Di Nolfo, J. Sawada, B. Paulovicks, K. Schleupen, B. G. Shaw, J. L. Klamo, M. D. Flickner, J. V. Arthur, and D. S. Modha, “TrueNorth: Accelerating From Zero to 64 Million Neurons in 10 Years,” _Computer_ , vol. 52, no. 5, pp. 20–29, 2019.
  * [20] M. Scherer, A. Di Mauro, G. Rutishauser, T. Fischer, and L. Benini, “A 1036 top/s/w, 12.2 mw, 2.72 μj/inference all digital tnn accelerator in 22 nm fdx technology for tinyml applications,” in _2022 IEEE Symposium in Low-Power and High-Speed Chips (COOL CHIPS)_ , 2022, pp. 1–3.
  * [21] A. Pullini, D. Rossi, I. Loi, G. Tagliavini, and L. Benini, “Mr. wolf: An energy-precision scalable parallel ultra low power soc for iot edge processing,” _IEEE Journal of Solid-State Circuits_ , vol. 54, no. 7, pp. 1970–1981, 2019.
  * [22] C. Li, L. Longinotti, F. Corradi, and T. Delbruck, “A 132 by 104 10$\mu$m-pixel 250$\mu$w 1kefps dynamic vision sensor with pixel-parallel noise and spatial redundancy suppression,” in _2019 Symposium on VLSI Circuits_. IEEE, 2019, pp. C216–C217.
  * [23] A. Di Mauro, M. Scherer, J. F. Mas, B. Bougenot, M. Magno, and L. Benini, “Flydvs: An event-driven wireless ultra-low power visual sensor node,” in _2021 Design, Automation & Test in Europe Conference & Exhibition (DATE)_, 2021, pp. 1851–1854.
  * [24] A. Amir, B. Taba, D. Berg, T. Melano, J. McKinstry, C. Di Nolfo, T. Nayak, A. Andreopoulos, G. Garreau, M. Mendoza _et al._ , “A low power, fully event-based gesture recognition system,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 7243–7252.
  * [25] Y. Wu, L. Deng, G. Li, J. Zhu, and L. Shi, “Spatio-temporal backpropagation for training high-performance spiking neural networks,” _Frontiers in neuroscience_ , vol. 12, p. 331, 2018.