# Cross-Modal Object Tracking: Modality-Aware Representations and
A Unified Benchmark

Chenglong Li1, Tianhao Zhu2, Lei Liu2, Xiaonan Si2, Zilin Fan2, Sulan Zhai3

###### Abstract

In many visual systems, visual tracking often bases on RGB image sequences, in
which some targets are invalid in low-light conditions, and tracking
performance is thus affected significantly. Introducing other modalities such
as depth and infrared data is an effective way to handle imaging limitations
of individual sources, but multi-modal imaging platforms usually require
elaborate designs and cannot be applied in many real-world applications at
present. Near-infrared (NIR) imaging becomes an essential part of many
surveillance cameras, whose imaging is switchable between RGB and NIR based on
the light intensity. These two modalities are heterogeneous with very
different visual properties and thus bring big challenges for visual tracking.
However, existing works have not studied this challenging problem. In this
work, we address the cross-modal object tracking problem and contribute a new
video dataset, including $654$ cross-modal image sequences with over $481$K
frames in total, and the average video length is more than $735$ frames. To
promote the research and development of cross-modal object tracking, we
propose a new algorithm, which learns the modality-aware target representation
to mitigate the appearance gap between RGB and NIR modalities in the tracking
process. It is plug-and-play and could thus be flexibly embedded into
different tracking frameworks. Extensive experiments on the dataset are
conducted, and we demonstrate the effectiveness of the proposed algorithm in
two representative tracking frameworks against $17$ state-of-the-art tracking
methods. We will release the dataset for free academic usage, dataset download
link and code will be released soon.

## Introduction

Visual tracking is an important problem in the field of computer vision and
plays a critical role in many visual systems, such as visual surveillance,
intelligent transportation, and robotics. However, existing tracking methods
often base on RGB image sequences which are sensitive to illumination
variations, and some targets are thus invalid in low-light conditions. In such
scenarios, the tracking performance of existing methods might degrade
significantly.

Some works introduce other modalities such as depth and infrared data to
overcome imaging limitations of RGB source (Song and Xiao 2013; Li et al.
2016, 2019b). However, multi-modal imaging platforms usually require elaborate
design and cannot be applied in many real-world applications at present. For
example, the depth sensors can provide valuable additional depth information
to improve tracking results by robust occlusion and model drift handling, but
suffer from the limited range (e.g., $4$-$5$ meters at most) and indoor
environment (Song and Xiao 2013; Li et al. 2016). Thermal sensors are usually
independent of RGB ones and their visual properties are very different.
Therefore, a lot of efforts are needed in platform design and frames alignment
(Li et al. 2016, 2019b).

Figure 1: (a) Illustration of heterogeneous properties between RGB and NIR
modalities. The visual camera changes RGB imaging to NIR when the light
intensity becomes low from normal, and vice versa. (b) A comparison of our
approach with state-of-the-art trackers in facing the challenge of modality
switch, including DiMP (Bhat et al. 2019), SiamRPN++ (Li et al. 2019a) and RT-
MDNet (Jung et al. 2018). The results show that our method handles this
challenge well but the others trackers fail when the appearance of the target
varies significantly caused by modality switch.

Near-infrared (NIR) imaging becomes an essential part of many surveillance
cameras, whose imaging is switchable between RGB and NIR based on the light
intensity, as shown in Fig. 1(a). This kind of imaging system well handles
imaging limitations of RGB source in low-light conditions while avoiding the
imaging and platform problems introduced by existing multi-modal visual
systems. From Fig. 1(b) we can also observe that these two modalities are
heterogeneous with very different visual properties and the appearance of the
target object is thus totally different in different modalities. Such an
appearance gap brings big challenges for visual tracking, and existing
tracking works have not studied this challenging problem.

In this work, we address the problem of cross-modal object tracking and aim to
answer the following two questions. How to design a suitable algorithm, which
could mitigate the appearance gap between RGB and NIR modalities and flexibly
embedded into different tracking frameworks, for robust cross-modal object
tracking? How to create a video benchmark dataset for the promotion of
research and development of cross-modal object tracking?

First, we propose a Modality-Aware cross-Modal Object Tracking algorithm
(MArMOT), which learns the modality-aware target representations to mitigate
the appearance gap between RGB and NIR modalities in the tracking process.
MArMOT is plug-and-play and could thus be flexibly embedded into different
tracking frameworks. MArMOT includes two parallel CNN branches to learn
modality-specific target representations using different sets of training
samples. Besides, we do not know which modality appears in the tracking
process. Thus, we design a ensemble module to adaptively incorporate effective
features from both branches with any modality as input. In this way, the
appearance gap between RGB and NIR modalities can be well addressed.

Second, to build a unified benchmark dataset, we collect $654$ cross-modal
object tracking sequences. The total number of video frames reaches over
$481$K, and the average video length and the maximum length of one sequence
are more than $735$ and $1.6$K frames. This dataset contains most of the real-
world challenges in cross-modal object tracking task. Most importantly, it
contains more challenges in adverse environmental conditions, as shown in Fig.
1(b), which easily triggers modality switch and significantly declines the
capability of visual trackers.

The major contributions of this work can be summarized as follows. First, we
introduce a new task called cross-modal object tracking that is very
challenging but practical in many visual systems. Second, we propose a novel
algorithm to mitigate the appearance gap of target object between different
modalities for robust cross-modal object tracking, and integrate it into two
typical tracking frameworks for effectiveness and generality validations.
Third, we develop a three-stage learning algorithm to train the proposed
tracking networks efficiently and effectively. Fourth, we create a unified
benchmark dataset which contains most of the real-world challenges in cross-
modal object tracking. Finally, we carry out an extensive experiment to
demonstrate the effectiveness of the proposed approaches against the state-of-
the-art trackers and clarify the research room on the cross-modal object
tracking.

## MArMOT Trackers

In this section, we first introduce the proposed Modality-Aware cross-Modal
Object Tracking model (MArMOT), and then the tracking architectures with
MArMOT including how embed the proposed plug-and-play MarMOT into two typical
tracking frameworks. At last, the three-stage learning algorithm and the
tracking details are provided.

Figure 2: Details of MArMOT. The BN+ReLU layers after each Conv layer and the
ReLU layers after each FC layer are omitted for clarity. Herein, $\oplus$ and
$\otimes$ denote the operations of element-wise addition and multiplication
respectively. F_RGB and F_NIR denote the output features of RGB and NIR after
two parallel modality-aware branches respectively. GAP indicates the global
average pooling.

### MArMOT Model

In the task of cross-modal object tracking, two modalities are heterogeneous
with very different visual properties and thus bring big challenges for visual
tracking. To solve this problem, we propose a new MArMOT which learns the
modality-aware target representations to mitigate the appearance gap between
RGB and NIR modalities in tracking process. Note that MArMOT is plug-and-play
and can thus be flexibly embedded into different tracking frameworks.

MArMOT includes two parallel modality-aware branches to learn modality-
specific target representations using different sets of training samples.
Besides, we do not know which modality appears in the tracking process. Thus,
we design an ensemble module to adaptively incorporate effective features from
both branches with any modality as input. In this way, the appearance gap
between RGB and NIR modalities can be well addressed, as shown in Fig. 2.

Modality-Aware Branch

. Two parallel modality-aware branches are followed by backbone network, and
used for learning modality-specific representations of the target in different
modalities. As for the architecture of each branch, we use the inception-like
network (Szegedy et al. 2016) for the effective and efficient computation. The
details can be seen in Fig. 2(b). In each branch, the first $1\times 1$
convolutional layer is used to capture modality-specific representations. Then
it is divided into two flows by using another two $1\times 1$ convolutional
layers with half channels to decrease the dimensionality of the input feature
and fed into two types of $3\times 3$ convolution to increase the adaptability
of the network to targets of different scales. Their outputs are concatenated
together as the modality-specific representation.

Ensemble Layer

. Due to the particularity of cross-modal object tracking, we design two
parallel modality-aware branches to capture the modality-specific
representations. However, in tracking process, we only have one modality as
the input in each frame and do not know which modality is presented. To handle
this problem, we design an ensemble layer to adaptively integrate features
outputted from two branches given one modal input. By this way, we can obtain
the effective features no matter which modality is input. Specifically, we
utilize the SKNet (Li et al. 2019d) to fuse the features of the two parallel
branches by weighting them using the normalized weights, and thus achieve
adaptive fusion of these two branch features. The detailed design can be found
in Fig. 2(c).

Figure 3: Visualization of target features in a sequence. (a) Projected
features of the baseline tracker. (b) Projected features of our MArMOT with
RT-MDNet as baseline. Herein, the blue circles and yellow triangles represent
target features of RGB and NIR modalities respectively. From the results we
can see that the heterogeneous gap of the target object between different
modalities is mitigated to some extent.

To visually demonstrate the effectiveness of our method, we present the
features of an example obtained by the baseline tracker RT-MDNet (Jung et al.
2018) and by our tracker MArMOTRT-MDNet after being projected to the 2D space
via the t-SNE algorithm (Van der Maaten and Hinton 2008), respectively, as
shown in Fig. 3. It can be found that the gap between target features of RGB
and NIR can be eliminated well after the introduction of the proposed
algorithm.

Figure 4: Visualization of the tracking architectures with MArMOT. (a) and (b)
show the detailed structures of MArMOT combining with RT-MDNet and DiMP-50.

### Tracking Architectures with MArMOT

We embed the proposed plug-and-play MArMOT model into two tracking frameworks,
i.e., RT-MDNet (Jung et al. 2018) and DiMP-$50$ (Bhat et al. 2019), named
MArMOTRT-MDNet and MArMOTDiMP-50 respectively, to verify the effectiveness and
generalization of MArMOT. The overall tracking frameworks are shown in Fig. 4.

For each tracking framework, we first use the backbone network to extract deep
feature representations of the target, then embed the proposed MArMOT model to
mitigate the appearance gap of the target representations between different
modalities, and finally send it to the classification branch and regression
branch of target localization. Specifically, on the tracking framework of RT-
MDNet, we use the first three layers of VGG-M to capture modality-shared
features of the target. Then, we insert our MArMOT model after the third layer
to achieve modality-aware feature representation learning. More details are
are shown in Fig. 4(a). As for the DiMP-$50$ tracking framework, the input
features of IoU predictor (regression branch) and model predictor
(classification branch) are not the same layer. Therefore, we insert the
MArMOT model after the third and fourth blocks of ResNet$50$ for IoU predictor
and model predictor. For more details, the tracking framework is shown in Fig.
4(b).

Figure 5: The visualization of three-stage training method. The parameters
learned in each stage are shown in orange.

### Three-stage Learning Algorithm

There are two problems in training the entire tracking frameworks. First, the
loss of a training sample with any modality will be backwardly propagated to
two modality-aware branches. Thus, there is no guarantee that the two
modality-aware branches will learn the corresponding modality-specific
representation of the target. Second, the modality information is available in
training stage but unavailable in testing stage. Therefore, we need to train
an ensemble layer to simulate the modal agnostic situation in tracking
process. To handle these two problems, we design an effective three-stage
training algorithm.

  * •

Stage I : Fine-tune the parameters of the baseline network on our dataset.
Note that our dataset is the first cross-modal tracking dataset. To adapt the
tracker to the cross-modal scenario, we first need to fine-tune the parameters
of the baseline network pre-trained on other large-scale datasets on our
training set. The learning rate of the network parameters is set to one-tenth
of the default learning rate of the baseline network, and the number of
iterations remains the same.

  * •

Stage II : Train two parallel modality-aware branches. To enable two parallel
modality-aware branches to learn the modality-specific representations of the
target in different modalities, we first divide the training set into two
subsets according to modality type, and use the corresponding sub-datasets to
learn the parameters of the corresponding modality-aware branches. Since the
baseline network has been adapted to the cross-modal tracking task in the
first stage, thus, in this stage, we only learn the parameters of the two
modality-aware branches, and the rest of the parameters are fixed (except for
the parameters of the fc$6$ layer based on the RT-MDNet (Jung et al. 2018)
framework). The initial learning rate is set to $1e-6$ and $1e-4$, and the
number of iterations is set to $50$ and $1000$, respectively.

  * •

Stage III : Train ensemble layer and fine-tune the parameters of the baseline
network on our dataset again. After the first two stages of training, the
baseline network has been able to adapt to the tracking of cross-modal
scenarios, and the two parallel modality-aware branches have also learned the
modality-specific representations of the target in different modalities. Since
which modality in each frame is unknown in the tracking process, the deep
features extracted by the backbone need to be sent to two parallel modality-
aware branches to extract the corresponding modality-specific representations.
To simulate the modality-unknowable situation in the tracking process, we
train the ensemble layer at this stage to perform weighted fusion of the
features of the two branches, and fine-tune the parameters of the network to
adapt to the situation after embedding the proposed MArMOT. At this stage, we
only learn the parameters of the ensemble layer and fine-tune the parameters
of the baseline network except the backbone. The learning rate of the ensemble
layer is the same as the modality-aware branch of the second stage, and the
learning rate of the tracker is the same as the first stage, and the number of
iterations is set to the same as the second stage.

Fig. 5 shows more details, and the learned part is indicated in the orange
color.

### Online Tracking

The tracking processes and parameter settings of our trackers during online
tracking are the most same as the baseline trackers. The only difference is
that the deep features extracted by the backbone networks (VGG-M in MArMOTRT-
MDNet and ResNet$50$ in MArMOTDiMP-50) are sent to the proposed MArMOT model
to mitigate the appearance gap of the target representations under different
modalities. The outputs of MArMOT are as the inputs of the classifiers
(fc$4-$fc$6$ in MArMOTRT-MDNet and model predictor in MArMOTDiMP-50) and the
regressor (IoU predictor module in MArMOTDiMP-50). The details of tracking
processes of our trackers are shown in Fig. 4.

## CMOTB Benchmark

Large-scale dataset are crucial in cross-modal object tracking because they
are not only useful for training deep trackers, but also for evaluating
different tracking algorithms. To this end, we provide a large-scale cross-
modal object tracking benchmark, called CMOTB. In this section, we introduce
CMOTB with detailed analysis.

### Data Collection and Annotation

Large-scale collection.

Current object tracking field lacks cross-modal video data, and we introduce
our CMOTB benchmark. Our goal is to provide large-scale and high-diverse
cross-modal object tracking benchmark for real-world scenarios and challenges.
To this end, we use hand-held cameras to capture video data in a large range
of scenes and background complexities. Unlike traditional visual tracking
data, we need to consider the variations of light intensity that trigger
modality switch in data creation. Therefore, we carefully select some
environmental conditions to simulate real-world applications such as visual
surveillance, intelligent transportation and self-driving systems. Fig. 1
shows a typical example of CMOTB dataset, and we can see that the imaging is
switching with several times between RGB and NIR modalities. By this way, we
collect $654$ cross-modal image sequences with over $481$K frames in total and
the average video length of is more than $735$ frames.

Table 1: The details of our CMOTB Dataset

Benchmark | Video |  | Min
---
frames
| Mean
---
frames
| Max
---
frames
| Total
---
frames
CMOTB (train) | 438 | 85 | 750 | 2037 | 328K
CMOTB (test) | 216 | 101 | 712 | 1838 | 153K

We present the detailed information of CMOTB in Table 1. Note that there is no
other cross-modal object tracking dataset, therefore, we divide CMOTB into
training set and testing set for facilitating the training of deep trackers
for cross-modal object tracking.

High-quality dense annotation.

We use a minimum bounding box to represent object states including position
and scale, and annotate each frame for training and evaluation set. Since the
labeling process is time-consuming and labor-intensive, we design an auxiliary
labeling tool based on the ViTBAT (Biresaw et al. 2016). The tool allows the
labeling of their states manually or semi-automatically through a simple and
friendly user interface in a time-efficient manner. The generated bounding
boxes are accurate in most situations. However, when the object undergoes
drastic appearance variations, the generated bounding boxes might be not quite
accurate. For these bounding boxes, we manually adjust them carefully.

To ensure high-quality annotations, we train $4$ professional annotators to
learn consistent annotation standards. Moreover, we let professional checkers
to carry out a frame-by-frame inspection to prevent wrong and inaccurate
marking. Due to the special challenges brought by modality switch, some
objects are sometimes temporarily invisible, which may result in losing a few
frames or more than a dozen frames. For such scenarios, we will keep ground
truths unchanged of target object until it is visible.

Table 2: Distribution of the number of modality switch.

Modality switch | Once | Twice | Three times | More than three times
---|---|---|---|---
Number of sequences | 410 | 182 | 47 | 15

Table 3: Descriptions of 11 different attributes in CMOTB.

Attribute |  Definition
---|---
SV |  Scale Variation - The ratio of bounding box is outside the rage $[0.5,2]$.
BC |  Background Clutter - The background has the similar appearance as the target.
ARC |  Aspect Ratio Change - The ratio of bounding box aspect ratio is outside the rage $[0.5,2]$.
SO |  Similar Object - There are objects of similar shape or same type near the target.
FM |  Fast Motion - The motion of the target is larger than the size of its bounding box.
IPR |  In-Plane Rotation - The target rotates out of the image plane.
OV |  Out-of-View - The target completely leaves the video frame.
PO |  Partial Occlusion - The target is partially occluded.
MA |  Modality Adaptation - Frame content has high intensity due to imaging adaptation to the environment in switching.
FO |  Full Occlusion - The target is fully occluded.
MB |  Motion Blurred - The target region is blurred due to target or camera motion.

Table 4: Distribution of attribute-based sequences on CMOTB testing set.

Attribute | SV | BC | ARC | SO | FM | IPR | OV | PO | MA | FO | MB
---|---|---|---|---|---|---|---|---|---|---|---
Number | 38 | 61 | 23 | 72 | 27 | 134 | 31 | 115 | 97 | 47 | 53

### Attributes

Existing multi-modal tracking datasets, e.g., RGBD (Song and Xiao 2013) and
RGBT (Li et al. 2016, 2019b), include two-modal data in each frame, while our
dataset has only one modality in each frame but might occur modality switch.
This is the major difference from existing multi-modal tracking datasets. The
modality switch means that the imaging is changed from one modality to another
one caused by light intensity variation. In such scenarios, the appearance of
target object usually varies significantly so that trackers are easily failed.
Note that the number of modality switch in a sequence is a key factor in
affecting trackers. Therefore, we take the switch times in data creation and
report the data distribution on switch times in Table 2.

According the modality switch, a new attribute, i.e., modality adaptation, is
thus introduced in CMOTB. The modality adaptation means that some frames have
high intensity due to imaging adaptation to the environment in modality
switch. It does not always occur when imaging is switch, and thus we take it
as an attribute. To enable attribute-based performance analysis of trackers,
we annotate each sequence with several attributes from the total $11$
attributes, including Scale Variation (SV), Aspect Ratio Change (ARC), Fast
Motion (FM), Out-of-View (OV), Modality Adaptation (MA), Motion Blur (MB),
Background Clutter (BC), Similar Object(SO), In-Plane Rotation (IPR), Partial
Occlusion (PO) and Full Occlusion (FO). The attributions are defined in Table
3, and Table 4 shows the video distribution of attributes on the testing set.

### Statistics

CMOTB consists of $654$ video sequences, which cover most of the challenges in
real-world scenarios. According to the holdout method (Kohavi et al. 1995), we
randomly split the testing and training sets of our dataset with the ratio of
$1:2$. And We have counted the distribution of attributes on testing set in
Table 4. The total number of frames of CMOTB reaches $481$K, and the average
length of our video sequence and the maximum number of frames reach $735$ and
$2037$ frames respectively. More details are shown in Table 1.

### Discussion

Differences from relevant tasks

. We discuss the differences of our new task from the task of multi-modal
visual object tracking. Existing work usually introduce thermal infrared or
depth information to achieve multi-modal visual object tracking, called RGBT
tracking (Li et al. 2019b, 2020) and RGBD tracking (Song and Xiao 2013).
Comparing with multi-modal visual object tracking, our task has the following
differences and advantages. First, our task is more practical. Many visual
cameras have equipped with NIR imaging, but RGBT or RGBD tracking requires two
cameras. Second, our task is more cost-effective. Thermal cameras are usually
very expensive and depth sensors have limited imaging range and environment,
but our task only relies on surveillance cameras and thus does not have these
limitations. Finally, the multi-modal data in our task do not have any
alignment error. Both RGBT and RGBD tracking tasks involve two cameras and the
alignment cross different modalities is needed, while our imaging system only
includes one camera whose imaging is switchable between RGB and NIR
modalities.

## Experiment

### Evaluated algorithms

We evaluate $17$ most advanced and representative trackers on our benchmark.
These trackers cover mainstream tracking algorithms from $2016$ to $2020$, and
they are MDNet (Nam and Han 2016), RT-MDNet (Jung et al. 2018), SiamFC
(Bertinetto et al. 2016), SPLT (Yan et al. 2019), GradNet (Li et al. 2019c),
TACT (Choi, Kwon, and Lee 2020), SiamMask (Wang et al. 2019), VITAL (Song et
al. 2018), GlobalTrack (Huang, Zhao, and Huang 2020), SiamRPN++ (Li et al.
2019a), ATOM (Danelljan et al. 2019), DiMP-$50$(Bhat et al. 2019), SiamBAN
(Chen et al. 2020), SiamDW (Zhang and Peng 2019), LTMU (Dai et al. 2020),
Ocean (Zhang et al. 2020) and DaSiamRPN (Zhu et al. 2018). It should be noted
that DiMP-$50$ and RT-MDNet are two representative tracking frameworks based
on regression and classification, respectively. Thus, to demonstrate the
effectiveness of the proposed method, we embed MArMOT into these two
frameworks, named MArMOTDiMP-50 and MArMOTRT-MDNet, respectively. It is worth
noting that all algorithms are evaluated on our testing set using the model
provided by authors.

### Evaluation metrics

To evaluate the performance of different trackers, we employ the widely used
tracking evaluation metrics including precision rate (PR), normalized
precision rate (NPR) and success rate (SR) (Muller et al. 2018) for
quantitative performance evaluation. PR is the percentage of frames whose
distance of the estimated bounding boxes with the ground truth is below a
predefined threshold, to rank the trackers, we set the distance threshold to
$20$ pixels to compute the representative PR. However, since the PR is very
sensitive to the target size, thus, we normalize the PR on the size of the
ground truth to calculate the normalized precision rate, distance threshold is
also set to $20$ pixels to compute the representative NPR. SR is the
percentage of frames where the overlap rate between the estimated bounding
boxes and the ground truth are greater than a threshold, we set the overlap
ratio to $0.5$ and use the area under curve of SR plots to compute two kinds
of representative SR scores respectively, denoting SR-I score and SR-II score
for the clarity.

Figure 6:  Tracking curves of $17$ trackers on the CMOTB testing dataset. In
(c), we show two kinds of representative SR scores in the legends, and the
left and right ones are SR-I and SR-II respectively.

### Overall Performance

We present the tracking performance in terms of precision plots, normalized
precision plots and success plots in Fig. 6, and the representative scores are
shown in the legends.

Regression based deep trackers. Regression-based trackers such as DiMP-$50$,
LTMU, SiamRPN++, SiamBAN, ATOM, SiamMask, have achieved high performance while
running at real-time speed. They are usually offline trained to learn a
powerful regressor from large-scale datasets to locate the target. However,
their performance is limited in cross-modal object tracking due to the
existence of large heterogeneous gap between RGB and NIR modalities, as shown
in Fig. 6. To verify the effectiveness of the proposed method, we insert the
proposed MArMOT into the DiMP-$50$ tracking framework, namely MArMOTDiMP-50,
and train the entire framework through the proposed multi-stage training
method. Our MArMOTDiMP-50 outperforms the baseline tracker DiMP-$50$ with
$11.2\%$/$9.7\%$/$11.8\%$/$9.2\%$ gains in PR/NPR/SR-I/SR-II, and has
excellent performance gains compared with all comparison methods.

Classification based deep trackers. Classification based deep trackers such as
MDNet, RT-MDNet and VITAL, usually employ online learning to train binary
classifiers using positive and negative samples, and thus have a good
generalization ability. To verify the effectiveness and generalization of the
proposed method, we also insert MArMOT into the RT-MDNet framework, namely
MArMOTRT-MDNet, and train the entire framework through the proposed multi-
stage training method. And our MArMOTRT-MDNet outperforms the baseline tracker
RT-MDNet with $15.8\%$/$16.6\%$/$21.4\%$/$15.5\%$ gains in PR/NPR/SR-I/SR-II.
We can find that although the performance of the baseline tracker is very low,
but it surpasses the performance of all classification-based tracking
frameworks after introducing our proposed model, which proves the
effectiveness of our proposed method.

Table 5: SR-I scores of different trackers on 11 challenging attributes using
the CMOTB testing sets. And speed comparisons of different trackers. Herein,
the best and second best results are indicated by red and blue fonts.

Trackers | SV | BC | ARC | SO | FM | IPR | OV | PO | MA | FO | MB | FPS
---|---|---|---|---|---|---|---|---|---|---|---|---
MArMOTDiMP-50 | 0.806 | 0.708 | 0.788 | 0.688 | 0.752 | 0.727 | 0.831 | 0.702 | 0.749 | 0.697 | 0.654 | 25
DiMP-50 | 0.693 | 0.610 | 0.663 | 0.561 | 0.616 | 0.599 | 0.717 | 0.580 | 0.616 | 0.597 | 0.552 | 32
LTMU | 0.710 | 0.601 | 0.658 | 0.582 | 0.517 | 0.601 | 0.742 | 0.569 | 0.633 | 0.584 | 0.511 | 3
SiamRPN++ | 0.634 | 0.568 | 0.642 | 0.602 | 0.525 | 0.587 | 0.652 | 0.569 | 0.597 | 0.530 | 0.507 | 21
MArMOTRT-MDNet | 0.617 | 0.576 | 0.521 | 0.567 | 0.502 | 0.557 | 0.680 | 0.552 | 0.596 | 0.448 | 0.576 | 24
SiamBAN | 0.626 | 0.587 | 0.626 | 0.564 | 0.505 | 0.551 | 0.631 | 0.521 | 0.563 | 0.505 | 0.504 | 29
ATOM | 0.628 | 0.526 | 0.650 | 0.525 | 0.550 | 0.577 | 0.633 | 0.523 | 0.583 | 0.508 | 0.470 | 28
SiamMask | 0.591 | 0.538 | 0.613 | 0.533 | 0.550 | 0.522 | 0.612 | 0.486 | 0.542 | 0.457 | 0.438 | 41
VITAL | 0.600 | 0.525 | 0.515 | 0.522 | 0.487 | 0.508 | 0.585 | 0.499 | 0.505 | 0.474 | 0.456 | 0.3
GlobalTrack | 0.506 | 0.531 | 0.559 | 0.458 | 0.427 | 0.515 | 0.602 | 0.482 | 0.558 | 0.533 | 0.496 | 1
DasiamRPN | 0.577 | 0.522 | 0.549 | 0.538 | 0.540 | 0.511 | 0.594 | 0.489 | 0.507 | 0.482 | 0.473 | 144
MDNet | 0.531 | 0.477 | 0.494 | 0.480 | 0.457 | 0.473 | 0.540 | 0.460 | 0.457 | 0.468 | 0.457 | 1
Ocean | 0.518 | 0.444 | 0.578 | 0.446 | 0.502 | 0.460 | 0.539 | 0.448 | 0.480 | 0.452 | 0.424 | 42
TACT | 0.457 | 0.451 | 0.531 | 0.377 | 0.344 | 0.438 | 0.559 | 0.428 | 0.497 | 0.411 | 0.372 | 29
GradNet | 0.476 | 0.430 | 0.424 | 0.429 | 0.424 | 0.437 | 0.441 | 0.407 | 0.419 | 0.384 | 0.375 | 68
SiamFC | 0.524 | 0.376 | 0.376 | 0.448 | 0.473 | 0.410 | 0.487 | 0.405 | 0.377 | 0.377 | 0.394 | 44
SPLT | 0.442 | 0.387 | 0.498 | 0.421 | 0.390 | 0.419 | 0.506 | 0.387 | 0.431 | 0.411 | 0.361 | 13
SiamDW | 0.326 | 0.397 | 0.456 | 0.376 | 0.354 | 0.372 | 0.481 | 0.374 | 0.394 | 0.314 | 0.392 | 22
RT-MDNet | 0.386 | 0.347 | 0.400 | 0.415 | 0.476 | 0.365 | 0.392 | 0.351 | 0.336 | 0.362 | 0.354 | 29

Runtime analysis. To verify the influence of the proposed method on the
tracking efficiency, we carried out an efficiency analysis of the tracker
without and with the proposed MArMOT in DiMP-$50$ and RT-MDNet tracking
frameworks. The experiments are run on two platforms with Intel(R) Xeon(R)
Silver $4210$ CPU ($32$G RAM), GeForce RTX $3090$ GPU and Intel(R) Xeon(R)
Silver $4210$ CPU ($32$G RAM), GeForce RTX $1080$Ti GPU for DiMP-$50$ and RT-
MDNet respectively. The last column of Table 5 reports the running speeds of
these trackers, which demonstrates that the tracking speeds decrease slightly
with an additional MArMOT.

### Attribute-based Performance

To analyze performance under different challenges faced by existing trackers,
we evaluate our algorithm against $17$ tracking algorithms on $11$ attributes,
as shown in Table 5. It can be seen from the table that our method
MArMOTDiMP-50 achieves the best results in all attributes compared with all
other algorithms. In addition, after adding our module to the two frameworks,
it can also be concluded that every attribute has an excellent performance
improvement compared with the baseline method, which proves the effectiveness
and generalization ability of our method.

Table 6:  Comparison of deep trackers re-trained on CMOTB dataset, where *
indicates the tracker is re-trained using CMOTB training dataset, and
$\uparrow$ indicates the performance improvements with re-training over their
own baselines.

Trackers | PR | NPR | SR-I | SR-II
---|---|---|---|---
DiMP-50* | 0.642↑6.9% | 0.630↑6.4% | 0.710↑8.2% | 0.599↑6.3%
RT-MDNet* | 0.517↑11.0% | 0.523↑11.4% | 0.534↑14.7% | 0.444↑10.9%
LTMU* | 0.561↑2.3% | 0.563↑2.2% | 0.628↑1.6% | 0.526↑0.1%
SiamRPN++* | 0.574↑1.8% | 0.552↑1.3% | 0.612↑1.1% | 0.503↑0.1%
SiamMask* | 0.556↑3.2% | 0.529↑2.2% | 0.571↑2.6% | 0.486↑3.0%
MDNet* | 0.534↑3.9% | 0.526↑3.1% | 0.547↑4.4% | 0.461↑3.1%
GlobalTrack* | 0.495↑3.1% | 0.482↑2.0% | 0.555↑1.8% | 0.477↑2.4%
MArMOTDiMP-50 | 0.685 | 0.663 | 0.746 | 0.628
MArMOTRT-MDNet | 0.565 | 0.575 | 0.601 | 0.490

### Training Dataset Validation

We select seven representative trackers including DiMP-$50$, RT-MDNet, LTMU,
SiamRPN++, SiamMask, MDNet and GlobalTrack to demonstrate the effectiveness of
our training dataset in the training of deep models. The results are shown in
Table 6, which shows that all the re-trained deep trackers achieve obvious
improvements and verify the necessity of proposing this dataset for the study
of cross-modal object tracking. In addition, after adding our proposed model
to the DiMP-$50$ and RT-MDNet frameworks, we can see that the performance has
been further improved, i.e., $4.3\%$/$3.3\%$/$3.6\%$/$2.9\%$ and
$4.8\%$/$5.2\%$/$6.7\%$/$4.6\%$ gains on the PR/NPR/SR-I/SR-II respectively,
which proves the effectiveness of MArMOT.

### Analysis of MArMOT

Table 7: Tracking results of trackers on the synthetic RGBT dataset, where *
indicates the tracker is re-trained using synthetic RGBT training dataset.

Trackers | PR | NPR | SR-I | SR-II
---|---|---|---|---
DiMP-50-RGBT | 0.681 | 0.468 | 0.523 | 0.446
DiMP-50*-RGBT | 0.757 | 0.507 | 0.602 | 0.502
MArMOTDiMP-50-RGBT | 0.767 | 0.534 | 0.639 | 0.529

### Evaluation on Synthetic Data

To further validate the effectiveness of our MArMOT model, we construct a
synthetic RGB-thermal cross-modal dataset from existing RGBT datasets,
including GTOT (Li et al. 2016) and RGBT234 (Li et al. 2019b). In order to
simulate cross-modal tracking tasks more accurately, each frame in synthetic
videos is generated by selecting one modality from corresponding RGB and
thermal images, according to the challenge labels of illumination various and
thermal crossover. Specifically, if there is no illumination various in the
first frame, each sequence always starts with the RGB modality, and the
modality switch is only performed when the challenge of illumination various
or thermal crossover occurs. In addition, if there are too many modality
switch (i.e., the number of modality-switch is greater than 5), the sequence
is considered to be too challenging and the sequence is discarded; if there is
no modality switch in the entire sequence, then part of 1/4 to 1/2 in current
sequence will switch to another modality.

For this synthetic dataset, we select the RGBT234 dataset as the training set
and GTOT dataset as the testing set, and retrain the entire network for the
cross-modal RGBT tracking task with proposed three-stage training method. The
experimental results are shown in Table 7.

From the results we can see that our MArMOT model can well handle appearance
gap between RGB and thermal modalities in tracking process, and thus further
prove the generalization and effectiveness of our method in handling the
different cross-modal tracking task.

Table 8: Comparison of several variants of our MArMOT, where * indicates the
tracker is re-trained using CMOTB training dataset.

Trackers | PR | NPR | SR-I | SR-II
---|---|---|---|---
DiMP-50 | 0.573 | 0.566 | 0.628 | 0.536
DiMP-50* | 0.642 | 0.630 | 0.710 | 0.599
MArMOTDiMP-50-one-stage | 0.662 | 0.646 | 0.726 | 0.612
MArMOTDiMP-50 | 0.685 | 0.663 | 0.746 | 0.628

### Effectiveness of Modality-aware Representations

To verify the effectiveness of the proposed modality-aware representations and
proposed three-stage training method, we implement the variant trackers, named
MArMOTDiMP-50-one-stage, by using one-stage training method to train the
entire networks together on the CMOTB dataset.

The results are shown in Table 8. The experimental results show that proposed
three-stage training method outperforms the one-stage training method with
$2.3\%$/$1.7\%$/$2.0\%$/$1.6\%$ gains in PR/NPR/SR-I/SR-II, which can prove
that the proposed three-stage learning method is benefit to modality-aware
branches to learn corresponding modality-specific target representations. In
addition, we can also find that the performance of the one-stage training
method is still better than DiMP-50*, which can verify the effectiveness of
the proposed MArMOT model for mining cross-modal information.

## Conclusion

We provide a large-scale cross-modal object tracking benchmark with high-
quality dense bounding box annotations. And we also propose a simple yet
effective method based on a modality-aware feature learning algorithm for
cross-modal object tracking purpose. Extensive experiments on the dataset
demonstrate the effectiveness of the proposed method against state-of-the-art
trackers. By releasing this dataset, we believe that it will help the research
and developments of cross-modal object tracking. In the future, we will study
more effective tracking algorithms to solve the cross-modal tracking problem
and extend the dataset to cover more real-world scenarios.

## References

  * Bertinetto et al. (2016) Bertinetto, L.; Valmadre, J.; Henriques, J. F.; Vedaldi, A.; and Torr, P. H. 2016\.  Fully-convolutional siamese networks for object tracking.  In _Proceedings of the European Conference on Computer Vision_ , 850–865. Springer.
  * Bhat et al. (2019) Bhat, G.; Danelljan, M.; Gool, L. V.; and Timofte, R. 2019.  Learning discriminative model prediction for tracking.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 6182–6191.
  * Biresaw et al. (2016) Biresaw, T. A.; Nawaz, T.; Ferryman, J.; and Dell, A. I. 2016.  Vitbat: Video tracking and behavior annotation tool.  In _Proceedings of the IEEE International Conference on Advanced Video and Signal Based Surveillance_ , 295–301. IEEE.
  * Chen et al. (2020) Chen, Z.; Zhong, B.; Li, G.; Zhang, S.; and Ji, R. 2020.  Siamese box adaptive network for visual tracking.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 6668–6677.
  * Choi, Kwon, and Lee (2020) Choi, J.; Kwon, J.; and Lee, K. M. 2020.  Visual tracking by tridentalign and context embedding.  In _Proceedings of the Asian Conference on Computer Vision_.
  * Dai et al. (2020) Dai, K.; Zhang, Y.; Wang, D.; Li, J.; Lu, H.; and Yang, X. 2020.  High-performance long-term tracking with meta-updater.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 6298–6307.
  * Danelljan et al. (2019) Danelljan, M.; Bhat, G.; Khan, F. S.; and Felsberg, M. 2019.  Atom: Accurate tracking by overlap maximization.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 4660–4669.
  * Huang, Zhao, and Huang (2020) Huang, L.; Zhao, X.; and Huang, K. 2020.  Globaltrack: A simple and strong baseline for long-term tracking.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, 11037–11044.
  * Jung et al. (2018) Jung, I.; Son, J.; Baek, M.; and Han, B. 2018.  Real-time mdnet.  In _Proceedings of the European Conference on Computer Vision_ , 83–98.
  * Kohavi et al. (1995) Kohavi, R.; et al. 1995.  A study of cross-validation and bootstrap for accuracy estimation and model selection.  In _Proceedings of the International Joint Conference on Artificial Intelligence_ , volume 14, 1137–1145. Montreal, Canada.
  * Li et al. (2019a) Li, B.; Wu, W.; Wang, Q.; Zhang, F.; Xing, J.; and Yan, J. 2019a.  Siamrpn++: Evolution of siamese visual tracking with very deep networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 4282–4291.
  * Li et al. (2016) Li, C.; Cheng, H.; Hu, S.; Liu, X.; Tang, J.; and Lin, L. 2016.  Learning collaborative sparse representation for grayscale-thermal tracking.  _IEEE Transactions on Image Processing_ , 25(12): 5743–5756.
  * Li et al. (2019b) Li, C.; Liang, X.; Lu, Y.; Zhao, N.; and Tang, J. 2019b.  RGB-T object tracking: benchmark and baseline.  _Pattern Recognition_ , 96: 106977.
  * Li et al. (2020) Li, C.; Liu, L.; Lu, A.; Ji, Q.; and Tang, J. 2020.  Challenge-aware rgbt tracking.  In _Proceedings of the European Conference on Computer Vision_ , 222–237. Springer.
  * Li et al. (2019c) Li, P.; Chen, B.; Ouyang, W.; Wang, D.; Yang, X.; and Lu, H. 2019c.  GradNet: Gradient-guided network for visual object tracking.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 6162–6171.
  * Li et al. (2019d) Li, X.; Wang, W.; Hu, X.; and Yang, J. 2019d.  Selective kernel networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 510–519.
  * Muller et al. (2018) Muller, M.; Bibi, A.; Giancola, S.; Alsubaihi, S.; and Ghanem, B. 2018.  Trackingnet: A large-scale dataset and benchmark for object tracking in the wild.  In _Proceedings of the European Conference on Computer Vision_ , 300–317.
  * Nam and Han (2016) Nam, H.; and Han, B. 2016.  Learning multi-domain convolutional neural networks for visual tracking.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 4293–4302.
  * Song and Xiao (2013) Song, S.; and Xiao, J. 2013.  Tracking revisited using RGBD camera: Unified benchmark and baselines.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 233–240.
  * Song et al. (2018) Song, Y.; Ma, C.; Wu, X.; Gong, L.; Bao, L.; Zuo, W.; Shen, C.; Lau, R. W.; and Yang, M.-H. 2018.  Vital: Visual tracking via adversarial learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 8990–8999.
  * Szegedy et al. (2016) Szegedy, C.; Ioffe, S.; Vanhoucke, V.; and Alemi, A. 2016.  Inception-v4, inception-resnet and the impact of residual connections on learning.  _arXiv preprint arXiv:1602.07261_.
  * Van der Maaten and Hinton (2008) Van der Maaten, L.; and Hinton, G. 2008.  Visualizing data using t-SNE.  _Journal of Machine Learning Research_ , 9(11).
  * Wang et al. (2019) Wang, Q.; Zhang, L.; Bertinetto, L.; Hu, W.; and Torr, P. H. 2019.  Fast online object tracking and segmentation: A unifying approach.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 1328–1338.
  * Yan et al. (2019) Yan, B.; Zhao, H.; Wang, D.; Lu, H.; and Yang, X. 2019.  ’Skimming-Perusal’Tracking: A framework for real-time and robust long-term tracking.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 2385–2393.
  * Zhang and Peng (2019) Zhang, Z.; and Peng, H. 2019.  Deeper and wider siamese networks for real-time visual tracking.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 4591–4600.
  * Zhang et al. (2020) Zhang, Z.; Peng, H.; Fu, J.; Li, B.; and Hu, W. 2020.  Ocean: Object-aware anchor-free tracking.  In _Proceedings of the European Conference on Computer Vision_ , 771–787. Springer.
  * Zhu et al. (2018) Zhu, Z.; Wang, Q.; Li, B.; Wu, W.; Yan, J.; and Hu, W. 2018.  Distractor-aware siamese networks for visual object tracking.  In _Proceedings of the European Conference on Computer Vision_ , 101–117.


Simulation of Muon Tomography Projections to Image the Pyramids of Giza

Presentation of a simulation of geometric tracks of muon trajectories through
a pyramid to a two-plane detector used in one sided muon tomography

Author: Mira Liu
Advisor: Patrick La Riviere, PhD

Submitted as a rotation report presented for the fulfillment of University of
Chicago Medical Physics MPHY 41800 Research in Advanced Tomographic Imaging
with Dr. La Riviere.

Work presented here is not peer-reviewed and is not submitted for independent
publication. It is part of a project proposed by an interdisciplinary group of
researchers from the Oriental Institute, the University of Virginia, Fermilab,
the Geophysics Department of Cairo University, and the University of Chicago
Medical Physics collaborating to produce high-resolution tomographic images.
Work presented here contributed to published paper111Bross, Alan David, Dukes,
E. C., Ehrlich, Ralf, Fernandez, Eric, Dukes, Sophie, Gobashy, Mohamed,
Jamieson, Ishbel, La Riviere, Patrick J., Liu, Mira, Marouard, Gregory,
Moeller, Nadine, Pla-Dalmau, Anna, Rubinov, Paul, Shohoud, Omar, Vargas,
Phillip, and Welch, Tabitha. 2022. ”Tomographic Muon Imaging of the Great
Pyramid of Giza”. United States. https://doi.org/10.31526/jais.2022.280.
available here: https://www.osti.gov/biblio/1844785

Graduate Program in Medical Physics
University of Chicago
Chicago, IL
April 4, 2020

##  1 Abstract

Purpose:  A geometric simulation of a possible two-plane detector was
developed to test the abilities of the detector to generate high-resolution
images of the Great Pyramid using muon tomography.

Methods and Materials: Trajectory range, angular resolution, and acceptance of
the detector were calculated with a simulation. Trajectories and the
corresponding sinogram space covered were simulated first with one detector in
one location, and then two moving detectors on adjacent sides of the pyramid.
The resolution at the center slice of the pyramid was calculated using the
angular resolution of the detector.

Results: The simulation returned trajectory range encompassing the pyramid and
peak angular resolution of $.0004sr$. Sinogram space covered by one position
was inadequate, however two moving detectors on adjacent sides of the pyramid
cover a significant portion. Resolution at the center of the pyramid is
roughly $3m$.

Conclusions: The simulation provides a way to calculate the detector positions
needed to cover an adequate amount of sinogram space for high-resolution
cosmic-ray tomographic reconstruction of the Great Pyramids.

Key Words: high-resolution muon tomography, one-sided tomography, sinogram
simulation, detector simulation

##  2 Introduction

Knowledge of the Great Pyramid of Giza is currently limited to knowledge from
optical instruments. The question of how these structures differ, and reasons
why, call for detailed, high-resolution imaging of those internal core
structures by other means. In 2017 a new void was detected by the Scan
Pyramids team through cosmic-ray muon tomography, albeit with relatively poor
resolution [1]. A project has since been proposed by an interdisciplinary
group of researchers from the Oriental Institute, the University of Virginia,
Fermilab, the Geophysics Department of Cairo University, and the University of
Chicago Medical Physics collaborating to produce high-resolution tomographic
images. These would reconstruct the interior structure of the Great Pyramid of
Giza with one-sided cosmic-ray muon tomography[2]. The Medical Physics aspect
involves exploring the ability of the proposal to meet the required
tomographic sampling necessary for a high-resolution 3D reconstruction of the
pyramid. This rotation project focused on building a simulation to study the
geometric dimensions of the proposed detector and the Great Pyramid, angular
resolution and acceptance of the detector, resolution of detection, and the
range of sinogram space able to be covered.

##  3 Methods and Materials

The dimensions of the proposed detectors and the dimensions of the external
shape of the Great Pyramid were taken from a previous GEANT4 monte carlo
simulation of cosmic-ray muon tomography[3]. A numerical simulation was
written to show that the widest trajectory angles subtended the entire Great
Pyramid if the far detector side was located 25m from the edge of the pyramid.
Given the pixel dimensions of the detector, the angular resolution and
acceptance of the detectors were calculated using the definitions provided by
Lesparre et. al.[4] From this, a simulation was written to translate all
potential trajectories of the detector to sinogram space covered to test the
ability of high-resolution tomographic reconstruction. This simulation can be
used to determine placement and movement of the detectors at the base of the
pyramid. Lastly, the simulation was extended to know the resolution at the
furthest depth of the pyramid given the angular resolution of the pixels. As
this is a purely geometric simulation, the cosmic ray spectrum, probability of
interaction, energy of incident muon, scattering, density, and attenuation
were not included. These sections are explored in detail in the subsections
below.222All code was written in Python3 and source code developed is
available at:
https://github.com/miramliu/MuonTomography.git

###  3.1 Geometric Dimensions

The proposed detector is built within a shipping container, of length $9.6m$,
height $2.4m$, and depth $2m$[3]. A front panel and a back panel of
scintillator modules allow hits to be connected to define a muon’s trajectory
through the detector. On each panel $2\times 2cm^{2}$ pixels are arranged such
that there are $120$ pixels vertically, and $480$ pixels horizontally. This
creates a grid of $240\times 480=115200$ ij-pixels (i representing the
horizontal location, j representing the vertical location), each of which have
a trajectory through the matching $115200$ ij-pixels $D=2m$ away on the
opposite panel. This generates $1.3\times 10^{10}$ paths between the two
panels. Trajectories of muons are calculated by reconstructing the vector
connecting the pixel in the front array and back array knowing the distance
between the two pixels $m=\Delta i$ and $n=\Delta j$ with $\vec{v}=[\Delta
i,\Delta j]$ as shown in Figure7. The trajectories with the greatest angle
provide the limit of the angular range. The pyramid with dimensions of depth
and width $230.33m$ and height $138.7m$ must fit within this angular range to
be imaged in its entirety.

###  3.2 Angular Resolution and Acceptance

Given the dimensions and number of the detector pixels, the angular resolution
and acceptance can be calculated. Using the definition from Lesparre et.
al.[4] the angular resolution for each discrete direction of sight of the
detector is equal to the solid angle covering all trajectories able to hit a
given pair of pixels of that direction of sight. This angular resolution was
simulated with and without tilt correction. This angular resolution and the
area of the detector with this angular resolution were used to calculate the
acceptance.

The direction of sight $r_{mn}$ is the direction of the vector passing through
the center of two pixels. For pairs of pixels with the same horizontal and
vertical displacement ($m=\Delta i,n=\Delta j$), the direction of sight is
equal. The angular resolution for this particular direction of sight is given
by the solid angle between these two pixels
$\delta\Omega=A/r^{2}=4cm^{2}/r^{2}$, as shown in Figure 7. Correcting for
tilt, for each direction of sight $r_{mn}$ the angular resolution is given by
the solid angle $\delta\Omega=\frac{A\cos\theta\cos\Psi}{r^{2}}$. This takes
into account how the area covered by the solid angle formed by the connection
of the far corners of offset pixels ($m=\Delta i\neq 0,n=\Delta j\neq 0$ is
not equal to the area of the pixels, as shown in Figure 7.

With the solid angle as a function of the discrete directions of sight
available from the detector, the number of pixel pairs that have that discrete
direction of sight can be converted to the detection area available per solid
angle. This is done with Equation 23 from Lesparre et. al.[4],
$\tau(r_{mn})=S(r_{mn})\times\delta\Omega(r_{mn})$ representing Acceptance =
Detection Area $\times$ Angular Resolution for each discrete direction of
sight $r_{mn}$ with $m=\Delta i,n=\Delta j$. Doing this for every $r_{mn}$
returns the acceptance $\tau(r_{mn})$ of the detector.

###  3.3 Sinogram Space

Looking at one level of pixels, meaning all pixels of a row with a constant j
$(\Delta j=0)$, the trajectories measurable through a 2d slice of an $xy$
layer of the pyramid at the height corresponding to $j$ can be calculated. The
pyramid coordinate system is constructed with the origin at the center, $x$
parallel to the $i$-axis, $y$ parallel to the depth of the detector, and $z$
normal to the surface of the earth. From this single horizontal slice the
angle of a trajectory is given by $\psi=\tan^{-}1(dy/dx)$ where $dy$ is equal
to the $2m$ for all pixels due to the distance between the two detector
plates, and $dx=\Delta i$. Converting from detector $ij$ space to pyramid
$xyz$ space we can form projection space $\xi$ and $\eta$ with $\xi$ as the
axis rotated by the azimuthal angle $\phi=\psi-\frac{\pi}{2}$[5] around the
$z$ axis. In projection space the angle of rotation $\phi$ as well as the
intersection of the trajectory with the $\xi$ axis allows calculation of that
trajectory’s location in a sinogram. A plot of projection space of one
trajectory superimposed on the pyramid and pyramid coordinate system is shown
in Figure 7.

With the left bottom corner of a panel being the origin of $ij$ space, and the
angle of rotation of $\eta$ being $\psi=\phi+\pi/2$, $\xi=x\cos\psi+y\sin\psi$
and $\eta=-x\sin\psi+y\cos\psi$ with $\psi$ being the angle of the muon
trajectory, i.e. the projection. The origin is shifted from the detector
coordinate basis to the pyramid’s coordinate basis with $x=(i-(960/2))/100$
and $y=-230.33/2+25$ both now in units of meters. With these conversions, for
every possible muon trajectory representing a different angle and position of
a projection, the sinogram space covered by the detector can be calculated. A
plot of a subset of projection angles is shown in Figure 7 as example of the
range of projection angles for one row of the detector at one position.

Finally, as the amount of sinogram space covered by the detector in one
position can be calculated, the detector can be moved and rotated in pyramid
space and the amount of sinogram space covered by different positions, and
multiple detectors, can be shown. As the pyramid is stationary the time of
projection images captured can be ignored.

###  3.4 Resolution

Lastly, the solid angle associated with each pixel pair was projected forward
to its resolution at the center of the pyramid. The distance travelled and the
resulting resolution is dependent on which vertical slice of the pyramid is
imaged, with the center being the worst resolution assuming 360 degree
rotation of the detector about the pyramid. For simplicity the lowest row of
pixels was chosen, $j=0$, as it has the most relevant angle range, and that
row was paired with every pixel on the front detector. This was done for one
position of the detector with significant overlap of solid angles for
different pairs of pixels with the same direction of sight.

##  4 Results

###  4.1 Geometric Dimensions

The geometric simulation of the maximum range of a stationary detector placed
at $x=0$ and $25m$ from the base of the pyramid contained trajectories that
encompassed the furthest edges of the pyramid. This is shown in Figure 7. The
normalized direction of sight for every eighth pixel on the far panel with
every eighth pixel on the front panel is shown in Figure 7 with the detector
facing up.

###  4.2 Angular Resolution and Acceptance

The angular resolution, i.e. solid angle, associated with a certain direction
of sight, is plotted as a function of $\theta$ and $\Psi$ without and with
tilt correction as shown in Figures 13 and 13. These values are as expected,
as Lesparre et. al. finds a peak angular resolution of $.015sr$ with
$5x5cm^{2}$ pixels and a $D=80cm$, because $\frac{(5\times
5)}{(80/2)^{2})}=.015sr$[4]. Similarly, with the proposed detector with
$2\times 2cm^{2}$ pixels and a $D=200cm$, the peak angular resolution is
$\frac{(2\times 2)}{(200/2)^{2})}=.0004sr$, which agrees with the simulation.
Acceptance, i.e. detector area times angular resolution for each direction of
sight, is shown in Figure 13. The values calculated in Lespare et. al. are not
matched by our detector. The peak acceptance calculated by hand for a
$16\times 16$ pixel grid with pixel size $25cm^{2}$ with peak angular
resolution of $.015sr$ is equal to $16\times 5^{2}\times.015=100$. However the
figure presented by Lesparre et. al. has a peak of $200cm^{2}sr$. The plot
shown in Figure 13 follows the calculation done above, rather than the plot
shown in the Lesparre paper.

###  4.3 Sinogram Space

The points of sinogram space covered by the detector at one position are shown
in Figure 13. This is reasonable as the width of the detector is 9.6m, while
the width of the pyramid is 230m. Therefore at $\phi=0$, completely forward
direction of sight, the total range of $\xi_{0}$ covered is equal to the width
of the detector. When a detector is moved in $20m$ increments along the
$x$-axis of the pyramid at a constant $25m$ from the base, and another
detector is moved along the $y$-axis of the pyramid, a majority of sinogram
space is covered as shown in Figure 13.

###  4.4 Resolution

Lastly, the projections and the resulting angular resolution are shown as a
grid at the center slice of the pyramid in Figure 13. There is a resolution of
roughly $3\times 3m^{2}$ which leads to an oversampling at every $2cm$ pixel.

##  5 Discussion

This simulation showed the medical physics side of muon tomography and the
possibility of high-resolution reconstruction using muon trajectories as
tomographic line integrals. It also studied the angular resolution and
acceptance of a current version of the detector, which can be changed based on
resolution demands. The geometric simulation can also be used to test possible
movements of detectors around the pyramid and the resulting sinogram space
covered. The simulation shows that with detectors moving along two sides of
the pyramid, a majority of sinogram space can be covered allowing possible
reconstruction. The sampling resolution of sinogram space wanted for high-
resolution reconstruction determines the step sizes between detector
placement. The resolution at the center of the pyramid is roughly $3\times
3m^{2}$, leading to oversampling.

The main limitation to the simulation is that it is purely geometric, does not
take into account possible scattering, energy attenuation, the muon spectrum,
timing, or cosmic ray statistics. Future work involving statistical
reconstruction involves taking into account coulomb scattering and scatter
based on material properties as discussed by Schultz et. al [6] and Hanson et.
al.[7]. Specifically the energy loss incurred by muons of initial energy
$E_{0}>E_{min}$ travelling through the pyramid can be used to determine the
opacity along the light of sight. That is,
$E_{0}-E_{f}=-\int_{0}^{L}l^{\prime}\frac{dE}{dx}dl^{\prime}=\int_{0}^{L}\rho(l^{\prime})(\frac{dT}{\rho
dx})_{E^{\prime},l^{\prime}}dl^{\prime}$ as notated by Hanson et. al.[7]. This
means that given the path length $L$, muons of a certain incident energy
$E_{0}$ lose an amount of energy per density, and that knowing that path
length one can know the opacity the muon travelled through to yield a final
energy at the detector of $E_{f}$. The muon spectrum measured by the detector
of open air could return a good estimate of $E_{0}$, and the change in the
resulting spectrum per muon trajectory through the pyramid can be used to
calculate the opacity of that trajectory.

This simulation provides a way to calculate the positions needed to cover an
adequate amount of sinogram space for high-resolution tomographic
reconstruction, as well as a way to calculate the path length $L$ through the
pyramid as a function of trajectory. In future work the energy attenuation as
a function of opacity, comparison to open air, and scattering should be
included.

##  6 Conclusion

This simulation of geometric tracks of muon trajectories through a pyramid to
a detector used in one-sided muon tomography showed the ability to fill
sinogram space with a series of positioned detectors along the basis of the
pyramid. It also showed the angular resolution and acceptance of the detector,
and the resolution at the center of the pyramid. This suggests that using muon
attenuation to calculate opacity of every trajectory one can reconstruct a
high-resolution image of the pyramids using one sided tomography.

##  7 Acknowledgments

Supported, in part, by the U.S. National Institute of Biomedical Imaging and
Bioengineering of the National Institutes of Health under grant number T32
EB002103.

## References

  * [1] K. Morishima, M. Kuno, A. Nishio and al  “Discovery of a big void in Khufu’s Pyramid by observation of cosmic-ray muons.”  In _Nature_ 552, 2017, pp. 386–390  DOI: 10.1038/nature24647
  * [2] A.D. Bross et al.  “Exploring the Great Pyramid – EGP: High-Resolution Muon Tomography of the Great Pyramid” Unpublished paper, 2018
  * [3] R. Ehrlich  “Pyramid Simulation” Unpublished presentation, 2019
  * [4] N. Lesparre et al.  “Geophysical Muon Imaging: Feasibility and Limits”  In _Geophysical Journal International_ 183, 2010, pp. 1348–1361  DOI: https://doi.org/10.1111/j.1365-246X.2010.04790.x
  * [5] J.L. Prince and J. Links  “Medical Imaging Signals and Systems, 1st Edition”  Prentice Hall, 2005
  * [6] L.J. Schultz et al.  “Statistical Reconstruction for Cosmic Ray Muon Tomography”  In _IEEE Transactions on Image Processing_ 16, 2007, pp. 1985–1993  DOI: 10.1109/TIP.2007.901239
  * [7] K.M. Hanson et al.  “Computed Tomography using Proton Energy Loss”  In _Physics in Medicine and Biology_ 26, 1981, pp. 965–983  DOI: https://doi.org/10.1088/0031-9155/26/6/001

Figure 1: A diagram of the shipping container detector proposed using the $ij$
coordinate system. A trajectory and its vector is shown.

Figure 2: A diagram of the un-tilted solid angle and calculation used in the
code.

Figure 3: A diagram of the tilted solid angle and the calculation used in the
code.

Figure 4: A diagram of projection space $\eta\xi$ from one incident muon with
trajectory $\vec{v}$, angle $\psi$, traveling through the bottom slice of the
pyramid shown in black, with the pyramid coordinate system $xy$ for reference.

Figure 5: A diagram of the projection angles captures by every sixteenth pixel
in a single row paired with every sixteenth pixel in the front panel of the
same row. If every pixel is used the ’fan’ is filled in.

Figure 6: A plot of the range of trajectories with respect to the pyramid. If
placed at the middle and $25m$ from the base of the pyramid, the entire
pyramid falls within the trajectory range.

Figure 7: A plot of the directions of sight as unit vectors using every eighth
pixel.

Figure 8: A plot of the angular resolution of the detector in $sr$ per angle
of direction of sight. The effect of tilt on the solid angle is not taken into
account.

Figure 9: A plot of the angular resolution of the detector in $sr$ per angle
of direction of sight. The effect of tilt on the solid angle is taken into
account.

Figure 10: A plot of the acceptance of the detector in
$cm^{2}\leavevmode\nobreak\ sr$ per angle of direction of sight . The effect
of tilt on the solid angle is taken into account.

Figure 11: A plot of the points in sinogram space covered by the detector
located at the center of the $x$-axis and $25m$ from the base of the pyramid.

Figure 12: A plot of the points in sinogram space covered by a detector moved
along the entire range of the pyramid along both the $x$ and $y$ axis eleven
times ($\leavevmode\nobreak\ 20m$ steps). In both cases the detector is
pointed perpendicular to the axis it is travelling along, meaning it is facing
the pyramid.

Figure 13: A plot of the resolution at the center slice of the pyramid for
every eighth pixel for reference.