Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

NeSy’20/21 @ IJCLR: 15th International Workshop on Neural-Symbolic Learning
and Reasoning, October 25–27, 2021

[orcid=0000-0003-4429-3311<EMAIL_ADDRESS>url=https://www.linkedin.com/in/jyhong0304/, ]

[orcid=0000-0002-7073-6932<EMAIL_ADDRESS>url=https://isearch.asu.edu/profile/1995237, ]

# An Insect-Inspired Randomly, Weighted Neural Network with Random Fourier
Features For Neuro-Symbolic Relational Learning

Jinyung Hong  Theodore P. Pavlic  School of Computing and Augmented
Intelligence, Arizona State University, Tempe, AZ 85281, USA  School of
Sustainability, Arizona State University, Tempe, AZ 85281, USA  School of
Complex Adaptive Systems, Arizona State University, Tempe, AZ 85281, USA
School of Life Sciences, Arizona State University, Tempe, AZ 85281, USA

(2021)

###### Abstract

The computer-science field of Knowledge Representation and Reasoning (KRR)
aims to understand, reason, and interpret knowledge as efficiently as human
beings do. Because many logical formalisms and reasoning methods in the area
have shown the capability of higher-order learning, such as abstract concept
learning, integrating artificial neural networks (ANNs) with KRR methods for
learning complex and practical tasks has received much attention. For example,
Neural Tensor Networks (NTNs) are neural-network models capable of
transforming symbolic representations into vector spaces where reasoning can
be performed through matrix computation; when used in Logic Tensor Networks
(LTNs), they are able to embed first-order logic symbols such as constants,
facts, and rules into real-valued tensors. The integration of KRR and ANN
suggests a potential avenue for bringing biological inspiration from
neuroscience into KRR. However, higher-order learning is not exclusive to
human brains. Insects, such as fruit flies and honey bees, can solve simple
associative learning tasks and learn abstract concepts such as “sameness” and
“difference,” which is viewed as a higher-order cognitive function and
typically thought to depend on top-down neocortical processing. Empirical
research with fruit flies strongly supports that a randomized representational
architecture is used in olfactory processing in insect brains. Based on these
results, we propose a Randomly Weighted Feature Network (RWFN) that
incorporates randomly drawn, untrained weights in a encoder that uses an
adapted linear model as a decoder. The randomized projections between input
neurons and higher-order processing centers in the input brain is mimicked in
RWFN by a single-hidden-layer neural network that specially structures latent
representations in the hidden layer using random Fourier features that better
represent complex relationships between inputs using kernel approximation.
Because of this special representation, RWFNs can effectively learn the degree
of relationship among inputs by training only a linear decoder model. We
compare the performance of RWFNs to LTNs for Semantic Image Interpretation
(SII) tasks that have been used as a representative example of how LTNs
utilize reasoning over first-order logic to surpass the performance of solely
data-driven methods. We demonstrate that compared to LTNs, RWFNs can achieve
better or similar performance for both object classification and detection of
the _part-of_ relations between objects in SII tasks while using much far
fewer learnable parameters (1:62 ratio) and a faster learning process (1:2
ratio of running speed). Furthermore, we show that because the randomized
weights do not depend on the data, several decoders can share a single
randomized encoder, giving RWFNs a unique economy of spatial scale for
simultaneous classification tasks.

###### keywords:

insect neuroscience model architecture randomization neuro-symbolic computing

##  1 Introduction

The human brain has an extraordinary ability to memorize and learn new things
to solve a variety of problems with difficulty ranging from trivial to
complex. To understand the cognitive architecture of the brain, research on
producing a wiring diagram of the connections among all neurons, called
_Connectomics_ [1], has focused not only on the human brain [2] but also on
the brains of insects [3, 4], and such research has influenced the development
of machine learning and Artificial Intelligence (AI) [5]. However, far more is
known about the function of coarse-grained, high-level structures in the brain
than the neuron-scale layout of important brain regions. Similarly, the high
degrees of freedom in artificial neural networks (ANN) has provided an
opportunity for the introduction of Knowledge Representation and Reasoning
(KRR) to constructively constrain ANN architectures and training methods. In
particular, combining KRR techniques with ANNs promises to enhance the high
performance of modern AI with explainability and interpretability, which is
necessary for generalized human insight and increased trustworthiness.

Several recent studies across statistical relational learning (SRL), neural-
symbolic computing, knowledge completion, and approximate inference [6, 7, 8,
9] have shown that neural networks can be integrated with logical systems to
perform robust learning and effective inference while also providing increased
interpretability from symbolic knowledge extraction. These neural-network
knowledge representation approaches use _relational embedding_ to represents
relational predicates in a neural network [10, 11, 12, 13]. For example,
Neural Tensor Networks (NTNs) are structured to encode the degree of
association among pairs of entities in the form of tensor operations on real-
valued vectors [12]. These NTNs have been synthesized with neural symbolic
integration [7] in the development of Logic Tensor Networks (LTNs) [14], which
can extend the power of NTNs to reason over first-order many-valued logic
[15].

Although KRR aims to lift the reasoning ability of computers to that of
humans, such higher-order learning and reasoning capabilities are not unique
to humans. Insect neuroscience has shown that insects show sophisticated and
complex behaviors even though they possess miniature central nervous systems
compared to the human brain [16]. For example, attention-like processes have
been demonstrated in in fruit flies and honey bees [17, 18], and concept
learning has been shown in bees [19]. Specifically, it has been shown that the
honey bee brain contains high levels of cognitive sophistication so that it
can learn relational concepts such as “same,” “different,” “larger than,”
“better than,” among others, and researchers continue to study the
neurobiological mechanisms and computational models underlying these
capabilities [20]. Just as KRR is now being used to better shape ANN’s for
more sophisticated reasoning and increased interpretability, the architectures
demonstrated in the honey bee brain may provide insights into how to augment
ANN’s with higher-order reasoning abilities akin to those demonstrated in
insects.

In this paper, we propose Randomly Weighted Feature Networks (RWFNs), an
insect-brain-inspired single-hidden-layer neural network for relational
embedding that incorporates randomly drawn, untrained weights in its encoder
with a trained linear model as a decoder. Our approach is mainly motivated by
neural circuits in the insect brain centered around the Mushroom Body (MB).
The MB, analogous to the neocortex in humans, is a vital region of the insect
brain supporting concept learning because it is responsible for stimulus
identification, categorization, and element learning [21, 22, 23]. We can
model the MB as a neural network model with three layers: Input Neurons (INs)
– Kenyon Cells (KCs) – mushroom body Extrinsic Neurons (ENs). One of the
remarkable properties of MB is that the connections between INs and KCs are
relatively random and sparse [24]. To mimic this characteristic, we used a
random weight matrix to transform the input between the input and hidden
layers to generate the latent representation of the relationship between real-
valued input entities. By doing so, the learning process involves only the
training of the weights between the hidden and output layers, which is simple
and fast. In contrast, a conventional LTN would incorporate an NTN specially
trained to capture logical relationships present in data, which requires more
learning parameters and a more complex learning process.

Our method is also influenced by random Fourier features [25], a kernel
approximation method that overcomes the issues of conventional kernel machines
or kernel methods [26]. Kernel methods are one of the most powerful and
theoretically grounded approaches for nonlinear statistical learning problems,
including classification, regression, clustering, and others [27, 28, 29].
However, the main issue of the kernel method is the lack of scalability for
large datasets and a slow training process [25, 30]. Random Fourier features
can address these issues by approximating the kernel function by using an
explicit feature mapping that projects the input data into a randomized
feature space and by applying faster linear models to learn. Interestingly,
the random Fourier features model can also be viewed as a class of a single-
hidden-layer neural network model with a fixed weight between the input and
hidden layers. Thus, we leveraged this to substitute the tensor operations in
conventional NTNs that model the linear interactions between entities and
utilized the projection from input into another space as another feature
representation in the hidden layer of our model to learn relationships.

Thus, our proposed model is an insect-inspired single-hidden-layer network
with latent representation derived by the integration of the input
transformation between INs and KCs and random Fourier features, and it only
requires training of a linear decoder. By applying the model to solve the
Semantic Image Interpretation (SII) tasks, we show that a trained linear
decoder in RWFNs can effectively capture the likelihood of _part-of_
relationships at a level of performance exceeding that of traditional LTNs,
even with far fewer parameters and a faster learning process. To the best of
our knowledge, this is the first research to integrate both insect
neuroscience and neuro-symbolic approaches for reasoning under uncertainty and
for learning in the presence of data and rich knowledge. Furthermore, because
the encoder weights in our model do not depend upon the data, the single
encoder can be shared among several decoders, each trained for a different
classifier, giving RWFNs an economy of spatial scale in our model applications
where several classifiers need to be used simultaneously.

##  2 Related Work and Background

#### Insect Neuroscience

The MB in the insect brain receives processed olfactory, visual, and
mechanosensory stimuli [31] and is viewed as the critical region responsible
for multimodal associative learning [21]. In the fruit fly, thousands of
Kenyon Cells (KCs) in the MB each receive a set of random $\sim$7 inputs from
INs [24, 32], and this is similarly true for honey bees [33]. A simplified
neural circuit modeling the MB is a neural network with three layers
consisting of: i) INs that provide olfactory, visual, and mechanosensory
inputs, ii) KCs generating the sparse-encoding of sensory stimuli, and iii)
ENs for activating several different behavioral responses [34]. In particular,
INs receive various inputs from Antennal Lobe (AL) glomeruli, Medulla, and
Lobula optic neuropils [34]. For simplicity, we focus on the olfactory pathway
between glomeruli in the AL and KCs in the MB [34, 33]. The insect olfactory
neural circuit has a divergence–convergence structure where $\sim$800 AL
glomeruli form a coded feature vector that expands into a spare representation
across $\sim$170,000 KCs, and these are decoded by $\sim$400 ENs that actuate
motor pathways based on this information-processing pipeline [33]. This
general divergence–convergence structure applies equally as well to honey bees
and fruit flies [35]; therefore, for modeling the architecture of our methods
for brevity, we interchangeably leverage the neural circuits of olfactory
nervous systems between these insects.

In this paper, the input transformation of odorant representation in the AL to
the higher-order representation across the KCs in the MB is shown in the
projection between the input and hidden layers of our model, and this
representation plays the critical role in learning the relationships presented
on the input.

#### Random Fourier Features

Kernel machines, e.g., Support Vector Machines (SVMs) [36, 37], have received
significant attention due to their capability for function approximation and
excellent performance of detecting decision boundaries with enough training
data. These methods use transformations, as with a lifting function $\phi$,
that help to better discriminate among different inputs. Given dataset vector
inputs $\textbf{x},\textbf{y}\in\mathbb{R}^{d}$, the kernel function
$k(\textbf{x},\textbf{y})=\langle\phi(\textbf{x}),\phi(\textbf{y})\rangle$
represents the similarity (i.e., inner product) between x and y in the
transformed space. However, because of the potential complexity of the
transformation $\phi$, learning the kernel function $k$ may require
significant computational and storage costs.

Random Fourier features [25], instead, provide a data transformation that
permits using a far less expensive approximation of the kernel function. For
each vector input $\textbf{x}\in\mathbb{R}^{d}$, the technique applies a
randomized feature function $\textbf{z}:\mathbb{R}^{d}\to\mathbb{R}^{D}$
(generally, $D\gg d$ with sample size $N\gg D$) that maps x to evaluations of
$D$ random Fourier basis from the Fourier transform of kernel $k$. In this
transformed space, kernel evaluations can be approximated by linear
operations, as in:

$k(\textbf{x},\textbf{y})=\langle\phi(\textbf{x}),\phi(\textbf{y})\rangle\approx\textbf{z}(\textbf{x})^{\top}\textbf{z}(\textbf{y})$
(1)

Thus, by transforming the input with z, fast linear learning methods can be
leveraged to approximate the evaluations of nonlinear kernel machines.

In this paper, we use random Fourier features as latent representations that
reduce the complexity of learning relations among real-valued entities. As
described in Section 1, adaptable NTNs within LTNs have been used to encode
relationships among real-valued entities. We replace the adaptable NTNs with
random Fourier features that have high expressiveness with low decoding
overhead. Details and intuitions will be given in Section 3.

#### Logic Tensor Networks (LTNs)

The RWFNs we propose are meant to improve upon LTNs for statistical relational
learning tasks. LTNs integrate learning based on NTNs [12] with reasoning
using first-order, many-valued logic [15], all implemented in TensorFlow [14].
Here, we briefly introduce LTN syntax and semantics for use in mapping logical
symbols to numerical values and learning reasoning relations among real-valued
vectors using the logical formulas.

Although a first-order-logic (FOL) language $\mathcal{L}$ and its signature
are defined by consisting of three disjoint sets – i) $\mathcal{C}$
(constants), ii) $\mathcal{F}$ (functions) and iii) $\mathcal{P}$ (predicate)
– we ignore function symbols $\mathcal{F}$ because they are not used in SII
tasks that we focus on here. For any predicate symbol $s$, $\alpha(s)$ can be
described as its _arity_ , and logical formulas in $\mathcal{L}$ enable the
description of relational knowledge. The objects being reasoned over with FOL
are mapped to an interpretation domain $\subseteq\mathbb{R}^{n}$ so that every
object is associated with an $n$-dimensional vector of real numbers.
Intuitively, this $n$-tuple indicates $n$ numerical features of an object.
Thus, predicates are interpreted as fuzzy relations on real vectors. With this
numerical background, we can now define the numerical _grounding_ of FOL with
the following semantics; this grounding is necessary for NTNs to reason over
logical statements.

Let $n\in\mathbb{N}$. An $n$-grounding, or simply grounding, $\mathcal{G}$ for
a FOL $\mathcal{L}$ is a function defined on the signature of $\mathcal{L}$
satisfying the following conditions:

$\displaystyle\mathcal{G}(c)\in\mathbb{R}^{n}\;\text{for every constant
symbol}\;c\in\mathcal{C}$
$\displaystyle\mathcal{G}(P)\in\mathbb{R}^{n\cdot\alpha(f)}\to[0,1]\;\text{
for predicate sym.~{}}\;P\in\mathcal{P}$

Given a grounding $\mathcal{G}$, the semantics of closed terms and atomic
formulas is defined as follows:

$\displaystyle\mathcal{G}(P(t_{1},\dots,t_{m}))\triangleq\mathcal{G}(P)(\mathcal{G}(t_{1}),\dots,\mathcal{G}(t_{m}))$

The semantics for connectives, such as
$\mathcal{G}(\neg\phi),\mathcal{G}(\phi\land\psi),\mathcal{G}(\phi\lor\psi)$,
and $\mathcal{G}(\phi\rightarrow\psi)$, can be computed by following the fuzzy
logic such as the Lukasiewicz $t$-norm [15].

A partial grounding $\hat{\mathcal{G}}$ can be defined on a subset of the
signature of $\mathcal{L}$. A grounding $\mathcal{G}$ is said to be a
completion of $\hat{\mathcal{G}}$ if $\mathcal{G}$ is a grounding for
$\mathcal{L}$ and coincides with $\hat{\mathcal{G}}$ on the symbols where
$\hat{\mathcal{G}}$ is defined. Let _GT_ be a grounded theory which is a pair
$\langle\mathcal{K},\hat{\mathcal{G}}\rangle$ with a set $\mathcal{K}$ of
closed formulas and a partial grounding $\hat{\mathcal{G}}$. A grounding
$\mathcal{G}$ satisfies a _GT_ $\langle\mathcal{K},\hat{\mathcal{G}}\rangle$
if $\mathcal{G}$ completes $\hat{\mathcal{G}}$ and $\mathcal{G}(\phi)=1$ for
all $\phi\in\mathcal{K}$. A _GT_ $\langle\mathcal{K},\hat{\mathcal{G}}\rangle$
is satisfiable if there exists a grounding $\mathcal{G}$ that satisfies
$\langle\mathcal{K},\hat{\mathcal{G}}\rangle$. In other words, deciding the
satisfiability of $\langle\mathcal{K},\hat{\mathcal{G}}\rangle$ amounts to
searching for a grounding $\mathcal{G}$ such that all the formulas of
$\mathcal{K}$ are mapped to 1. If a _GT_ is not satisfiable, the best possible
satisfaction that we can reach with a grounding is of our interest.

Grounding $\mathcal{G}^{*}$ captures the implicit correlation between
quantitative features of objects and their categorical/relational properties.
The grounding of an $m$-ary predicate $P$, namely $\mathcal{G}(P)$, is defined
as a generalization of the NTN [12], as a function from $\mathbb{R}^{mn}$ to
$[0,1]$, as follows:

$\mathcal{G}_{LTN}(P)(\textbf{v})=\sigma(u_{P}^{\top}\texttt{f}(\textbf{v}^{\top}W_{P}^{[1:k]}\textbf{v}+V_{P}\textbf{v}+b_{P}))$
(2)

where
$\textbf{v}=\langle\textbf{v}_{1}^{\top},\dots,\textbf{v}_{m}^{\top}\rangle^{\top}$
is the $mn$-ary vector obtained by concatenating each $\textbf{{v}}_{i}$.
$\sigma$ is the sigmoid function and f is the hyperbolic tangent ($\tanh$).
The parameters for $P$ are: $W_{P}^{[1:k]}$, a 3-D tensor in
$\mathbb{R}^{k\times mn\times mn},V_{P}\in\mathbb{R}^{k\times
mn},b_{P}\in\mathbb{R}^{k}$ and $u_{P}\in\mathbb{R}^{k}$. Because our RWFN
model can be used to ground a predicate as $\mathcal{G}_{RWFN}(P)$, we can
directly compare the performance of RWFNs for the SII tasks with LTNs.

##  3 Randomly Weighted Feature Networks (RWFNs)

In this section, we introduce the details of Randomly Weighted Feature
Networks (RWFNs). The underlying intuition behind the development of this
model can be found in Appendix A.

###  3.1 Model Architecture

(a) Visualization of the structure of the Randomly Weighted Feature Network.
In the depicted case, the input vector v constitutes of two entities,
$e_{1},e_{2}\in\mathbb{R}^{3}$ and it shows to learn a binary relation between
them $(e_{1},R,e_{2})$, such as (Cat, hasPart, Tail).

(b) Visualization of the structure of the Randomly Weighted Feature Network
with weight sharing. In the case of learning each classifier from the class
$\mathcal{C}_{1}$ to the class $\mathcal{C}_{i}$, RWFNs allow us to use the
same encoder to extract features from each data from the class
$\mathcal{C}_{1}$ to the class $\mathcal{C}_{i}$.

Figure 1: The architectures of RWFNs and RWFNs with weight sharing

Let the input vector v be
$[\textbf{v}_{1}^{\top},\dots,\textbf{v}_{m}^{\top}]^{\top}$, the $mn$-ary
vector where $m$ is arity and $n$ is the input dimension. We first define the
two kinds of latent representations: i) the input transformation between AL
glomeruli and KCs inspired by the insect brain, and ii) the transformed input
using a randomized feature mapping $\textbf{z}(\cdot)$ in random Fourier
features.

For the bio-inspired representation, we select $N_{in}\in[1,mn)$ indices of
the input at random without replacement for each hidden node111$N_{in}<mn$ to
prevent hidden-unit outputs becoming trivially 0 by Eq. (3). In our setting,
$N_{in}=7$.. In other words, the output $\bar{v}_{j}$ of each hidden node
$j\in\\{1,...,B\\}$ is a weighted combination of all $mn$ inputs where only
$N_{in}<mn$ inputs have $w_{j,i}=1$ and all other inputs have $w_{j,i}=0$. So
the inputs are effectively gated by the weights on each hidden node and the
weight matrix $\textbf{W}\in\mathbb{R}^{mn\times B}$ in this computation is
random, binary, and sparse.

Once the $j$th hidden node has produced weighted sum $\bar{v}_{j}$, the post-
processing step to produce intermediate output is performed in the hidden
layer. Mimicking Eq. (6) with $C=1$, the $j$th intermediate output
$\hat{x}_{j}$ is:

$\hat{v}_{j}=\bar{v}_{j}-\mu,\;\;\mu=\frac{1}{B}\sum_{i=1}^{B}\bar{v}_{i}$ (3)

where $B$ is the number of hidden units. Therefore, the sparse output of the
$j$-th KC node can be defined as $h_{j}^{(1)}=g(\hat{v}_{j})$ where $g$ is the
ReLU function [38] that allows the model to produce sparse hidden output,
which is more biologically plausible. By doing so, we define the output vector
as $\textbf{h}_{1}=[h_{1}^{(1)},\dots,h_{B}^{(1)}]^{\top}$.

On the other hand, to generate random Fourier features, we used a randomized
feature function $\textbf{z}(\cdot)$ in [25, 39], we can project the input as
follows:

$\textbf{h}_{2}=\textbf{z}(\textbf{v})=\frac{\sqrt{2}}{\sqrt{B}}\cos(\textbf{R}^{\top}\textbf{v}+\textbf{b})$
(4)

where $\textbf{R}\sim Normal^{mn\times B}(0,1)$ and $\textbf{b}\sim
Uniform^{B}(0,2\pi)$, which is Gaussian kernel approximation. Consequently,
the output vector $\textbf{h}_{2}$ can be considered as another latent
representation of relationship among input. Rahimi et al. [25], Sutherland and
Schneider [39], and Liu et al. [30] provide theoretical derivations of kernel
approximation and comparative analyses of various kinds of random Fourier
features.

Finally, using the above two latent representations, our RWFNs can be defined
as a function from $\mathbb{R}^{mn}$ to $[0,1]$:

$\mathcal{G}_{RWFN}(P)(\textbf{{v}})=\sigma\left(\boldsymbol{\beta}^{\top}\textbf{h}\right)\\\
=\sigma\left(\boldsymbol{\beta}^{\top}\texttt{f}\left(\begin{bmatrix}\textbf{h}_{1}\\\
\textbf{h}_{2}\end{bmatrix}\right)\right)$ (5)

where h is the final hidden representation obtained by applying the hyperbolic
tangent ($\tanh$) function f to the concatenation of $\textbf{h}_{1}$ and
$\textbf{h}_{2}$, and $\sigma$ is the sigmoid function; the $\tanh$ function
was used for the numeric stabilization. Because our model requires to adapt
only $\boldsymbol{\beta}\in\mathbb{R}^{2B}$, it possess a faster learning
process with fewer parameters compared to LTNs. Fig. 1(a) shows a
visualization of the structure of our model.

###  3.2 RWFNs with Weight Sharing

In the insect brain, extrinsic neurons from the MB are processed by several
small, downstream neuropils that ultimately lead to decision-making outcomes,
such as muscle actuation. If we view these small neuropils as decoding the
complex representations in the MB, then different decoders responsible for
different decisions all use information sourced from the same randomized
representations in the MB. The MB can be viewed as a generalized encoder that
is not tailored for a particular task; consequently, it provides a shared
resource to reduce the complexity of these downstream neuropils.

Because the weights of the randomized encoder of an RWFN are independent of
the training data, they can also serve as a shared resource for multiple
relatively simple (i.e., linear) downstream decoders trained for different
classifiers. We refer to this property as _weight sharing_. Fig. 1(b) shows a
visualization of the structure of our model applied with weight sharing to the
learning of $i$ different classifiers. The large solid box surrounds a single
encoder that serves as a common feature extractor for all classifiers.

The entities in the original definition of RWFNs in Fig. 1(a) become the
placeholder to be injected by the input for each classifier. Instead of
generating the randomized encoder for each classifier, each classifier uses
the same encoder, and training only requires learning the weights of that
classifier’s highly simple linear decoder. This approach increases reusability
and cost efficiency in a way beyond what is possible with LTNs, which must
train all encoder and decoder networks separately for each classifier.

##  4 Experimental Evaluation

To evaluate the performance of our proposed RWFNs over LTNs, we employ both
for SII tasks, which extract structured semantic descriptions from images.
Very few SRL applications have been applied to SII tasks because of the high
complexity involved with image learning. Donadello et al. [40] define two main
tasks of SII as: (i) the classification of bounding boxes, and (ii) the
detection of the _part-of_ relation between any two bounding boxes. They
demonstrated that LTNs can successfully improve the performance of solely
data-driven approaches, including the state-of-the art Fast Region-based
Convolutional Neural Networks (Fast R-CNN) [41]. Our experiments are conducted
by comparing the performance of two tasks of SII between RWFNs and LTNs. These
tasks are well defined in first-order logic, and the codes implemented in
TensorFlow framework have been provided and can be used to compare the
performance of LTNs with RWFNs.

###  4.1 Methods

Here, we provide details of our experimental comparison of RWFNs and LTNs. We
utilize the formalization of SII in first-order logic from Donadello et al.
[40]. For brevity, we describe: (i) the difference of the ground theories
between RWFNs and LTNs, (ii) the data set used in the experiments (Appendix
B), and (iii) the RWFN and LTN hyperparameters used in the experiments
(Appendix B). We omit other formalization details of the SII tasks that can be
found elsewhere [40].

#### Defining the Grounded Theories for RWFNs and LTNs

A set of bounding boxes of images correctly labelled with the classes that
they belong to and pairs of bounding boxes that properly labelled with the
_part-of_ relation were provided. These datasets can be considered as a
training set, and a grounded theory
$\mathcal{T}_{\text{LTN}}\triangleq\langle\mathcal{K},\hat{\mathcal{G}}_{LTN}\rangle$
can be constructed. In particular, $\mathcal{K}$ contains: (i) the set of
closed literals $C_{i}(b)$ and $\texttt{partOf}(b,b^{\prime})$ for every
bounding box $b$ labelled with $C_{i}$ and for every pair of bounding boxes
$\langle b,b^{\prime}\rangle$ connected by the partOf relation, and (ii) the
set of the mereological constraints for the _part-of_ relation, including
asymmetric constraints, lists of several parts of an object, or restrictions
that whole objects cannot be part of other objects and every part object
cannot be divided further into parts. Furthermore, the partial grounding
$\hat{\mathcal{G}}_{LTN}$ is defined on all bounding boxes of all the images
in the training set where both $class(C_{i},b)$ and the bounding box
coordinates are computed by the Fast R-CNN object detector.
$\hat{\mathcal{G}}$ is not defined for the predicate symbols in $\mathcal{P}$
and is to be learned.

A grounded theory
$\mathcal{T}_{\text{RWTN}}\triangleq\langle\mathcal{K},\hat{\mathcal{G}}_{RWFN}\rangle$
where a partial grounding $\hat{\mathcal{G}}_{RWFN}$ can be described for
predicates using Eq. (5). Thus, we can easily compare the performance between
$\hat{\mathcal{G}}_{RWFN}$ (Eq. (5)) and $\hat{\mathcal{G}}_{LTN}$ (Eq. (2)).

###  4.2 Results

Our experiments mainly focus on the comparison of the performance between our
model and LTN, but figures also include results with Fast-RCNN [41] for type
classification and the inclusion ratio $ir$ baseline in the _part-of_
detection task. If $ir$ is greater than a given threshold $th$ (in our
experiments, $th=0.7$), then the bounding boxes are said to be in the partOf
relation. Every bounding box $b$ is classified into $C\in\mathcal{P}_{1}$ if
$\mathcal{G}(C(b))>th$.

(a) RWFNs achieve similar performance for object type classification compared
to LTNs, achieving an Area Under the Curve (AUC) of 0.772 (compared to 0.770).

(b) RWTNs outperform LTNs on the detection of _part-of_ relations, achieving
AUC of 0.647 (compared to 0.613).

Figure 2: Precision–recall curves for indoor objects type classification and
the partOf relation between objects.

Results for indoor objects are shown in Fig. 2 where AUC is the area under the
precision–recall curve. The results show that, for the _part-of_ relation and
object types classification, RWFNs achieve better performance than LTNs.
However, there is some variance in the results because of the stochastic
nature of the experiments. Consequently, we carried out five such experiments
for each task, for which the sample averages and 95% confidence intervals are
shown in Table 1. These results confirm that our model can achieve similar
performance as LTNs for object-task classification and superior performance
for detection of _part-of_ relations.

In Table 1, we only included AUC numbers for RWFNs with weight sharing (third
column) for object-type classification because _part-of_ relations only
require a single classifier. The performance of RWFNs with weight sharing for
the object-type classification task (which requires 11 classifiers for indoor
objects, 23 for vehicles, and 26 for animals) shows only a marginal gap in
performance compared to other models, which demonstrates the effectiveness and
efficiency of the approach of using a single shared encoder in RWFNs with
weight sharing.

Table 1: AUC of T1 (object type classification) and T2 (detection of _part-of_ relation) for LTN, RWFN, and RWFN with weight sharing across label groups. $\text{MEAN}_{\pm 2\times\text{SD}}$ for all models. Best performances shown in bold. Label-Task | LTN | RWFN | RWFN w/ W.S
---|---|---|---
Indoor-T1 | $.769_{\pm.0314}$ | $.770_{\pm.0092}$ | $\textbf{.773}_{\pm.028}$
Indoor-T2 | $.619_{\pm.082}$ | $\textbf{.648}_{\pm.0621}$ | —
Vehicle-T1 | $.709_{\pm.0289}$ | $\textbf{.711}_{\pm.0162}$ | $.706_{\pm.0111}$
Vehicle-T2 | $.576_{\pm.0355}$ | $\textbf{.613}_{\pm.0489}$ | —
Animal-T1 | $\textbf{.701}_{\pm.024}$ | $.700_{\pm.024}$ | $.697_{\pm.0237}$
Animal-T2 | $.640_{\pm.0783}$ | $\textbf{.661}_{\pm.0364}$ | —

As summarized in Appendix C, we also conducted ablation studies to assess the
degree to which the AL–MB input transformation and the random Fourier features
each contribute to the performance of the model. We have also included, in
Appendix D, a detailed comparison of performance among LTNs, RWFNs, and RWFNs
with weight sharing. Specifically, we compare LTNs and RWFNs in terms of
numbers of learnable parameters and running times; we also compare RWFNs with
and without weight sharing in terms of space complexity.

##  5 Conclusion and Future Work

In this paper, we introduced Randomly Weighted Feature Networks, which
incorporate the insect-brain-inspired neuronal feature representation and
unique random features derived by random Fourier features. The RWFN encoder
acts as a generalized feature extractor with greater relational expressiveness
and a learning model with relatively simpler structure. We demonstrated how
insights from the insect nervous system can be applied to the fields of
neural-symbolic computing and knowledge representation and reasoning for
relational learning.

Our work can be advanced in several ways. For one, RWFNs can be applied to
other variants of SII problems proposed by Donadello and Serafini [42], and
performance between our model and LTN for zero-shot learning in SII tasks can
be compared. In addition, we plan to extend application of RWFNs to tasks that
need to extract structural knowledge from not only images but also text, such
as visual question-answering challenges. Furthermore, we will investigate how
other methods from neuroscience for exploring biologically-plausible learning
algorithms might be applicable to our model. Finally, we will extend RWFNs to
include a recurrent part for representing dynamic features of time-series
data, similar to reservoir computing [43, 44, 45]; this approach may allow for
extracting time-varying relational knowledge necessary for developing a
framework for data-driven reasoning over temporal logic.

###### Acknowledgements.

This work was supported in part by NSF SES-1735579.

## References

  * Seung [2012] S. Seung, Connectome: How the brain’s wiring makes us who we are, HMH, 2012.
  * Sporns et al. [2005] O. Sporns, G. Tononi, R. Kötter,  The human connectome: a structural description of the human brain,  PLoS Comput Biol 1 (2005) e42.
  * Eichler et al. [2017] K. Eichler, F. Li, A. Litwin-Kumar, Y. Park, I. Andrade, C. M. Schneider-Mizell, T. Saumweber, A. Huser, C. Eschbach, B. Gerber, et al.,  The complete connectome of a learning and memory centre in an insect brain,  Nature 548 (2017) 175–182.
  * Takemura et al. [2017] S.-y. Takemura, Y. Aso, T. Hige, A. Wong, Z. Lu, C. S. Xu, P. K. Rivlin, H. Hess, T. Zhao, T. Parag, et al.,  A connectome of a learning and memory center in the adult drosophila brain,  Elife 6 (2017) e26975.
  * Hassabis et al. [2017] D. Hassabis, D. Kumaran, C. Summerfield, M. Botvinick,  Neuroscience-inspired artificial intelligence,  Neuron 95 (2017) 245–258.
  * Koller et al. [2007] D. Koller, N. Friedman, S. Džeroski, C. Sutton, A. McCallum, A. Pfeffer, P. Abbeel, M.-F. Wong, D. Heckerman, C. Meek, et al., Introduction to statistical relational learning, MIT press, 2007.
  * Garcez et al. [2008] A. S. Garcez, L. C. Lamb, D. M. Gabbay, Neural-symbolic cognitive reasoning, Springer Science & Business Media, 2008\.
  * Pearl [2014] J. Pearl, Probabilistic reasoning in intelligent systems: networks of plausible inference, Elsevier, 2014.
  * Nickel et al. [2015] M. Nickel, K. Murphy, V. Tresp, E. Gabrilovich,  A review of relational machine learning for knowledge graphs,  Proceedings of the IEEE 104 (2015) 11–33.
  * Sutskever and Hinton [2009] I. Sutskever, G. E. Hinton,  Using matrices to model symbolic relationship,  in: Advances in neural information processing systems, 2009, pp. 1593–1600.
  * Bordes et al. [2011] A. Bordes, J. Weston, R. Collobert, Y. Bengio,  Learning structured embeddings of knowledge bases,  in: Twenty-Fifth AAAI Conference on Artificial Intelligence, 2011.
  * Socher et al. [2013] R. Socher, D. Chen, C. D. Manning, A. Ng,  Reasoning with neural tensor networks for knowledge base completion,  in: Advances in neural information processing systems, 2013, pp. 926–934.
  * Santoro et al. [2017] A. Santoro, D. Raposo, D. G. Barrett, M. Malinowski, R. Pascanu, P. Battaglia, T. Lillicrap,  A simple neural network module for relational reasoning,  in: Advances in neural information processing systems, 2017, pp. 4967–4976.
  * Serafini and Garcez [2016] L. Serafini, A. d. Garcez,  Logic tensor networks: Deep learning and logical reasoning from data and knowledge,  arXiv preprint arXiv:1606.04422 (2016).
  * Bergmann [2008] M. Bergmann, An introduction to many-valued and fuzzy logic: semantics, algebras, and derivation systems, Cambridge University Press, 2008.
  * Avarguès-Weber et al. [2011] A. Avarguès-Weber, N. Deisig, M. Giurfa,  Visual cognition in social insects,  Annual review of entomology 56 (2011) 423–443.
  * van Swinderen and Greenspan [2003] B. van Swinderen, R. J. Greenspan,  Salience modulates 20–30 hz brain activity in drosophila,  Nature neuroscience 6 (2003) 579–586.
  * Spaethe et al. [2006] J. Spaethe, J. Tautz, L. Chittka,  Do honeybees detect colour targets using serial or parallel visual search?,  Journal of Experimental Biology 209 (2006) 987–993.
  * Avarguès-Weber et al. [2012] A. Avarguès-Weber, A. G. Dyer, M. Combe, M. Giurfa,  Simultaneous mastering of two abstract concepts by the miniature brain of bees,  Proceedings of the National Academy of Sciences 109 (2012) 7481–7486.
  * Avarguès-Weber and Giurfa [2013] A. Avarguès-Weber, M. Giurfa,  Conceptual learning by miniature brains,  Proceedings of the Royal Society B: Biological Sciences 280 (2013) 20131907\.
  * Menzel [2001] R. Menzel,  Searching for the memory trace in a mini-brain, the honeybee,  Learning & memory 8 (2001) 53–62.
  * Galizia [2014] C. G. Galizia,  Olfactory coding in the insect brain: data and conjectures,  European Journal of Neuroscience 39 (2014) 1784–1795.
  * Bazhenov et al. [2013] M. Bazhenov, R. Huerta, B. H. Smith,  A computational framework for understanding decision making through integration of basic learning rules,  Journal of Neuroscience 33 (2013) 5686–5697.
  * Caron et al. [2013] S. J. Caron, V. Ruta, L. Abbott, R. Axel,  Random convergence of olfactory inputs in the drosophila mushroom body,  Nature 497 (2013) 113–117.
  * Rahimi et al. [2007] A. Rahimi, B. Recht, et al.,  Random features for large-scale kernel machines.,  in: NIPS, volume 3, Citeseer, 2007, p. 5.
  * Smola and Schölkopf [1998] A. J. Smola, B. Schölkopf, Learning with kernels, volume 4, Citeseer, 1998.
  * Zhu and Hastie [2005] J. Zhu, T. Hastie,  Kernel logistic regression and the import vector machine,  Journal of Computational and Graphical Statistics 14 (2005) 185–205.
  * Drucker et al. [1997] H. Drucker, C. J. Burges, L. Kaufman, A. Smola, V. Vapnik, et al.,  Support vector regression machines,  Advances in neural information processing systems 9 (1997) 155–161.
  * Dhillon et al. [2004] I. S. Dhillon, Y. Guan, B. Kulis,  Kernel k-means: spectral clustering and normalized cuts,  in: Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, 2004, pp. 551–556.
  * Liu et al. [2020] F. Liu, X. Huang, Y. Chen, J. A. Suykens,  Random features for kernel approximation: A survey in algorithms, theory, and beyond,  arXiv preprint arXiv:2004.11154 (2020).
  * Mobbs [1982] P. Mobbs,  The brain of the honeybee apis mellifera. i. the connections and spatial organization of the mushroom bodies,  Philosophical Transactions of the Royal Society of London. B, Biological Sciences 298 (1982) 309–354.
  * Inada et al. [2017] K. Inada, Y. Tsuchimoto, H. Kazama,  Origins of cell-type-specific olfactory processing in the drosophila mushroom body circuit,  Neuron 95 (2017) 357–367.
  * Peng and Chittka [2017] F. Peng, L. Chittka,  A simple computational model of the bee mushroom body can explain seemingly complex forms of olfactory learning and memory,  Current Biology 27 (2017) 224–230.
  * Cope et al. [2018] A. J. Cope, E. Vasilaki, D. Minors, C. Sabo, J. A. Marshall, A. B. Barron,  Abstract concept learning in a simple neural network inspired by the insect brain,  PLoS computational biology 14 (2018) e1006435.
  * Endo et al. [2020] K. Endo, Y. Tsuchimoto, H. Kazama,  Synthesis of conserved odor object representations in a random, divergent-convergent network,  Neuron 108 (2020) 367–381.
  * Boser et al. [1992] B. E. Boser, I. M. Guyon, V. N. Vapnik,  A training algorithm for optimal margin classifiers,  in: Proceedings of the fifth annual workshop on Computational learning theory, 1992, pp. 144–152.
  * Cortes and Vapnik [1995] C. Cortes, V. Vapnik,  Support-vector networks,  Machine learning 20 (1995) 273–297.
  * Glorot et al. [2011] X. Glorot, A. Bordes, Y. Bengio,  Deep sparse rectifier neural networks,  in: Proceedings of the fourteenth international conference on artificial intelligence and statistics, JMLR Workshop and Conference Proceedings, 2011, pp. 315–323.
  * Sutherland and Schneider [2015] D. J. Sutherland, J. Schneider,  On the error of random fourier features,  arXiv preprint arXiv:1506.02785 (2015).
  * Donadello et al. [2017] I. Donadello, L. Serafini, A. D. Garcez,  Logic tensor networks for semantic image interpretation,  arXiv preprint arXiv:1705.08968 (2017).
  * Girshick [2015] R. Girshick,  Fast r-cnn,  in: Proceedings of the IEEE international conference on computer vision, 2015, pp. 1440–1448.
  * Donadello and Serafini [2019] I. Donadello, L. Serafini,  Compensating supervision incompleteness with prior knowledge in semantic image interpretation,  in: 2019 International Joint Conference on Neural Networks (IJCNN), IEEE, 2019, pp. 1–8.
  * Ferreira and Ludermir [2009] A. A. Ferreira, T. B. Ludermir,  Genetic algorithm for reservoir computing optimization,  in: 2009 International Joint Conference on Neural Networks, IEEE, 2009, pp. 811–815.
  * Sun et al. [2017] X. Sun, T. Li, Q. Li, Y. Huang, Y. Li,  Deep belief echo-state network and its application to time series prediction,  Knowledge-Based Systems 130 (2017) 17–29.
  * Wang et al. [2019] X. Wang, Y. Jin, K. Hao,  Echo state networks regulated by local intrinsic plasticity rules for regression,  Neurocomputing 351 (2019) 111–122.
  * Dasgupta et al. [2017] S. Dasgupta, C. F. Stevens, S. Navlakha,  A neural algorithm for a fundamental computing problem,  Science 358 (2017) 793–796.
  * Aso et al. [2009] Y. Aso, K. Grübel, S. Busch, A. B. Friedrich, I. Siwanowicz, H. Tanimoto,  The mushroom body of adult drosophila characterized by gal4 drivers,  Journal of neurogenetics 23 (2009) 156–172.
  * Chen et al. [2014] X. Chen, R. Mottaghi, X. Liu, S. Fidler, R. Urtasun, A. Yuille,  Detect what you can: Detecting and representing objects using holistic models and body parts,  in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 1971–1978.
  * Tieleman and Hinton [2012] T. Tieleman, G. Hinton, Lecture 6.5—RmsProp: Divide the gradient by a running average of its recent magnitude, COURSERA: Neural Networks for Machine Learning, 2012.
  * Akiba et al. [2019] T. Akiba, S. Sano, T. Yanase, T. Ohta, M. Koyama,  Optuna: A next-generation hyperparameter optimization framework,  in: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, 2019, pp. 2623–2631.

##  Appendix A The Intuitions of RWFNs

For the bio-inspired representation in our model, we concentrated on
implementing: (i) how to build random connections between the AL glomeruli
(input layer) and the KCs (hidden layer), and (ii) how to guarantee hidden-
layer sparsity to best differentiate one odor stimulus from another. We used a
sparse, binary, and random matrix to define an arbitrary set of inputs for
each KC in the model with inspiration from Caron et al. [24], Peng and Chittka
[33], and Dasgupta et al. [46]. In particular, in biological models of the
insect brain and the AL–MB interface, the firing rates from 7 randomly
selected glomeruli are passed and summed to each KC [24, 32]. Furthermore,
Endo et al. [35] developed a computational model of sparsity of the KCs’
output activity based on global inhibition the average KC input. In their
model, KCs output an intermediate result subject to global inhibition from the
average glomerular input to all KCs. The last KC activity is then produced by
thresholding the inhibited output through a ramp function, which is
functionally equivalent to the Rectified Linear Unit (ReLU) activation
function [38]. Thus, the output of the $j$th KC, $KC_{out_{j}}$, in the
computational model was described as:

$KC_{out_{j}}=\phi\left(KC_{in_{j}}-C\frac{1}{N_{KC}}\sum_{j}^{N_{KC}}KC_{in_{j}}\right)$
(6)

where $KC_{in_{j}}$ indicates the weighted sum of input from 7 random indices
of the input vector, $\phi$ is the ReLU, $C$ is the strength of global
inhibition, and $N_{KC}$ is the total number of KCs. The parameters $C=1.0$
and $N_{KC}=2000$ were chosen so as to match the values best calibrated to
real KC responses [35, 47]. With this KC representation, Endo et al. [35]
trained a linear decoder to successfully classify ’group’ from ’non-group’
odors. Similarly, we make use of Eq. (6) and train a linear model for learning
latent relationships among input.

For another hidden representation using random Fourier features in our model,
based on Eq. (1), we can define a decision function $f(\textbf{x})$ given a
dataset including $N$ data samples $\textbf{x},\textbf{y}\in\mathbb{R}^{d}$
and a randomized feature mapping $\textbf{z}:\mathbb{R}^{d}\to\mathbb{R}^{D}$
as follows:

$\begin{split}f(\textbf{x})&=\sum_{n=1}^{N}\alpha_{n}k(\textbf{x}_{n},\textbf{x})=\sum_{n=1}^{N}\alpha_{n}\langle\phi(\textbf{x}_{n}),\phi(\textbf{x})\rangle\\\
&\approx\sum_{n=1}^{N}\alpha_{n}\textbf{z}(\textbf{x}_{n})^{\top}\textbf{z}(\textbf{x})=\boldsymbol{\beta}^{\top}\textbf{z}(\textbf{x})\\\
\end{split}$ (7)

This indicates that if $\textbf{z}(\cdot)$ can approximate $\phi(\cdot)$ well,
we can simply map our data using $\textbf{z}(\cdot)$ and then use a linear
model to learn because both $\boldsymbol{\beta}$ and $\textbf{z}(\cdot)$ in
the above equation are $D$-vectors. Therefore, the task that we will describe
in the next section is how to find a random projection function
$\textbf{z}(\cdot)$ that can approximate the corresponding nonlinear kernel
machine appropriately.

The reason why we leverage the random Fourier feature is conciseness and
efficiency of computing linear interactions among input, which can be replaced
with the bilinear model in Eq. (2). In Eq. (2), the bilinear tensor was used
to compute the relation, which seems intuitive because each slice of the
tensor serves as being responsible for one type of relation. However, this
computation requires high computational cost with large number of parameters.
In contrast, the random Fourier features in Eq (7) can do the similar task
with a much faster learning process and fewer number of parameters.

Considering how Eq. (6) and Eq. (7) can be used in our model, the hidden
representation can be expressed with the concatenation of $KC_{out}$ and
$\textbf{z}(\cdot)$.

##  Appendix B Details of Experiments

#### Hardware specification of the server

The hardware specification of the server that we used to experiment is as
follows:

  * •

CPU: Intel® CoreTM i7-6950X CPU @ 3.00GHz (up to 3.50 GHz)

  * •

RAM: 128 GB (DDR4 2400MHz)

  * •

GPU: NVIDIA GeForce Titan Xp GP102 (Pascal architecture, 3840 CUDA Cores @ 1.6
GHz, 384 bit bus width, 12 GB GDDR G5X memory)

#### Source codes

All source codes, trained models, and figures in this paper are available at
https://github.com/jyhong0304/SII.

#### Datasets

The PASCAL-Part-dataset [48] and ontologies (WordNet) are chosen for the
_part-of_ relation. The PASCAL-Part-dataset contains 10103 images with
bounding boxes. They are annotated with object-types and the part-of relation
defined between pairs of bounding boxes. There are three main groups in
labels—animals, vehicles, and indoor objects—with their corresponding parts
and “part-of” label. There are 59 labels (20 labels for whole objects and 39
labels for parts). The images were then split into a training set with 80% of
the images and a test set with 20% of the images, maintaining the same
proportion of the number of bounding boxes for each label. Given a set of
bounding boxes detected by an object detector (Fast-RCNN), the task of object
classification is to assign to each bounding box an object type. The task of
_part-of_ detection is to decide, given two bounding boxes, if the object
contained in the first is a part of the object contained in the second.

#### Hyperparameter Setting

To compare the performance between RWFNs and LTNs, we trained two models
separately. For LTN, we configure the experimental environment following
Donadello et al. [40]. The LTNs were configured with a tensor of $k=6$ layers.
For RWFN, the number of hidden nodes $B=200$ for a classifier for object type
classification. In addition, we set the number of hidden nodes of a classifier
for _part-of_ detection as twice as large as the number of hidden nodes $B$ of
a classifier for object classification, which is 400. This is because the
dimension of inputs that the classifier for detecting the _part-of_ relation
is twice as large as the input space required for a classifier for objection
categorization. Referring to Donadello et al. [40], both models make use of a
regularization parameter $\lambda=10^{-10}$, Lukasiewicz’s $t$-norm
($\mu(a,b)=\operatorname{max}(0,a+b-1)$), and the harmonic mean as an
aggregation operator. We ran 1000 training epochs of the RMSProp [49] learning
algorithm available in TensorFlow for each model.

#### Hyperparameter Searching for RWFNs

To find out the best number of hidden nodes $B$, we used the Optuna framework
[50] with 500 iterations in the range of $[64,512]$. The Optuna framework
allows us to dynamically construct the parameter search space because we can
formulate hyperparameter optimization as the maximization/minimization process
of an objective function that takes a set of hyperparameters as input and
returns a validation score. In our case, the validation score returned was the
test AUC values. Furthermore, it provides efficient sampling methods, such as
relational sampling that exploits the correlations among the parameters.

##  Appendix C Ablation Studies

Table 2 shows the results of ablation studies. In order to show how much two
hidden representations – the input transformation between AL–MB and random
Fourier features – contribute to the performance of our model, we built two
separate RWFN models: one using the AL–MB input transformation only and
another using random Fourier features only. Then, we performed five
experiments and averaged AUCs of each model for object classification and
_part-of_ detection. The number of hyperparameter $\beta$ for each model was
set to the same as the number of hyperparameter for the original RWFN, which
is 200.

For the object-type classification and _part-of_ detection tasks using Indoor
label, the random Fourier features outperform the AL–MB input transformation.
On the other hand, the AL–MB input transformation for _part-of_ detection
tasks using Vehicle and Animal labels show better performance than the random
Fourier features. Therefore, these ablation studies show that the model
architecture of RWFNs in Eq. (5) can fully utilize both hidden representations
and contribute to their good performance shown in Table. 1 by compensating for
each other.

Table 2: AUC of T1 (object type classification) and T2 (detection of _part-of_ relation) for the AL–MB representation (AL–MB) and random Fourier features (RFF) across label groups. $\text{MEAN}_{\pm 2\times\text{SD}}$ for all models. The best performance is displayed in bold. Label-Task | AL–MB | RFF
---|---|---
Indoor-T1 | $.743_{\pm.021}$ | $\textbf{.766}_{\pm.012}$
Indoor-T2 | $.525_{\pm.102}$ | $\textbf{.641}_{\pm.010}$
Vehicle-T1 | $.710_{\pm.017}$ | $\textbf{.715}_{\pm.009}$
Vehicle-T2 | $\textbf{.612}_{\pm.027}$ | $.572_{\pm.079}$
Animal-T1 | $.705_{\pm.017}$ | $\textbf{.709}_{\pm.013}$
Animal-T2 | $\textbf{.664}_{\pm.069}$ | $.646_{\pm.020}$

##  Appendix D Performance Analysis

#### Relative Complexity of RWFNs and LTNs

To better appreciate the relative performance of RWFNs and LTNs, we can
compare the number of parameters for grounding a unary predicate for each
model. The dimension of the input in the dataset for both RWFNs and LTNs is
$n=64$. As shown in Eq. (2), the parameters to learn in LTNs are
$\\{u_{P}\in\mathbb{R}^{k},W_{P}^{[1:k]}\in\mathbb{R}^{n\times n\times
k},V_{P}\in\mathbb{R}^{k\times n},b_{P}\in\mathbb{R}^{k}\\}$, where $k=6$
following the configuration of the LTNs. Thus, the number of parameters in
LTNs is $(n^{2}+n+2)\cdot k=(64^{2}+64+2)\cdot 6=24972$. On the other hand, in
Eq. (3) and Eq. (4), the number of parameters in RWFNs are
$\\{\textbf{W}\in\mathbb{R}^{n\times B},\textbf{R}\in\mathbb{R}^{n\times
B},\textbf{b}\in\mathbb{R}^{B},\boldsymbol{\beta}\in\mathbb{R}^{2B}\\}$, where
$B=200$ following the configuration of the RWFNs. Therefore, the number of
parameters in RWFNs is $(2n+3)\cdot B=(2\cdot 64+3)\cdot 200=26200$. Although
our method requires more space complexity compared to LTNs ($26200>24972$),
the parameters $\\{\textbf{W},\textbf{R},\textbf{b}\\}$ in RWFNs are randomly
drawn and fixed weights. Thus, it is also necessary to compare the number of
learnable parameters across the two models.

All of the above parameters of LTNs must be adaptable, whereas the parameters
to learn in RWFNs for object type classification are only
$\boldsymbol{\beta}\in\mathbb{R}^{2B}$. Thus, the number of learnable
parameters is $400$, which is much smaller than that of LTNs. This means that
the ratio of the two numbers of parameters to learn is about $400:24972\approx
1:62$. Consequently, non-adaptable parameters in RWFNs can have significant
power to represent the latent relationship among objects so that the model can
efficiently extract relational knowledge even though using fewer adaptable
parameters. Furthermore, the number of LTN parameters heavily depends on the
number of features, whereas RWFNs are independent of the number of features.
In principle, this could allow the learning process in our model to be
accelerated if the feature representation from the encoder model is pre-
processed and stored.

#### Running Time

Figure 3: The comparison of running time, including data configuration time,
training time (sec) for LTN and RWFN

Fig. 3 depicts the comparison of running time, including data configuration
time and training time for LTNs and RWFNs. The running time of RWFNs is
roughly as half of that of LTNs. This is because the number of learnable
parameters in RWFNs is far smaller than LTNs, and RWFNs have linear models to
learn, which is much simpler compared to models used in LTNs.

#### Space Complexity of RWFNs with Weight Sharing

Weight sharing is a unique feature of RWFNs, which can greatly reduce
necessary space complexity when multiple classifiers are used simultaneously.
In the depicted case of learning $i$ classifiers in Fig. 1(b), the space
complexity for RWFNs without using weight sharing is $(2\cdot n\cdot B+3\cdot
B)\cdot i=(2\cdot n+3)\cdot B\cdot i\approx O(i\cdot B\cdot n)$. However, with
weight sharing, RWFNs can achieve much better space complexity, which is
$2\cdot n\cdot B+B+2\cdot B\cdot i\approx O(B\cdot n)$ because $B\cdot
i<B\cdot n$ for the experiments conducted in the SII task, but it can also be
$O(i\cdot B)$ for the different task. This indicates that the one of the
factors that highly influence to the space complexity of the original RWFNs
can be negligible when using weight sharing, which makes the model cost
efficient and economical.


# The Space Coronagraph Optical Bench (SCoOB): 5. End-to-end simulations of
polarization aberrations

Ramya M Anche  Steward Observatory, University of Arizona, 933N Cherry Avenue,
Tucson, Arizona, 85721, USA  Kyle J. Van Gorkom  Steward Observatory,
University of Arizona, 933N Cherry Avenue, Tucson, Arizona, 85721, USA  Jaren
N. Ashcraft  Steward Observatory, University of Arizona, 933N Cherry Avenue,
Tucson, Arizona, 85721, USA  James C. Wyant College of Optical Sciences,
University of Arizona, 933N Cherry Avenue, Tucson, Arizona, 85721, USA  Ewan
Douglas  Steward Observatory, University of Arizona, 933N Cherry Avenue,
Tucson, Arizona, 85721, USA  Emory L Jenkins  Steward Observatory, University
of Arizona, 933N Cherry Avenue, Tucson, Arizona, 85721, USA  James C. Wyant
College of Optical Sciences, University of Arizona, 933N Cherry Avenue,
Tucson, Arizona, 85721, USA  Sebastiaan Y. Haffert  Leiden University, Niels
Bohrweg 2, 2333 CA Leiden  Maxwell A. Millar-Blanchaer  Department of Physics,
University of California, Santa Barbara, CA, 93106, USA

###### Abstract

Polarization aberrations originating from the telescope and high-contrast
imaging instrument optics introduce polarization-dependent speckles and
associated errors in the image plane, affecting the measured exoplanet signal.
Understanding this effect is critical for future space-based high-contrast
imaging instruments that aim to image the Earth analogs with $10^{-10}$ raw
contrast and characterize their atmospheres. We present end-to-end modeling of
the polarization aberrations for a high-contrast imaging testbed, SCoOB. We
use a vector vortex coronagraph (VVC) as the focal plane mask, incorporate
polarization filtering, and estimate the peak contrast in the dark hole region
3-10 $\lambda/D$. The dominant polarization aberrations in the system are
retardance defocus and tilt due to the OAPs and fold mirrors. Although the
mean contrast in the dark hole region remains unaffected by the polarization
aberrations, we see brighter speckles limiting the contrast to 1$\times
10^{-9}$ at 1-2 $\lambda/D$. We extend the simulations using the measured
retardance maps for the VVC and find that the mean contrast in SCoOB is more
sensitive to retardance errors of the VVC and the QWP than the polarization
aberrations.

###### keywords:

Polarization aberrations, high-contrast imaging, Vector Vortex Coronagraphs,
space-based telescopes

##  1 INTRODUCTION

High-contrast imaging is a technique to directly image the exoplanets by
suppressing the starlight using coronagraphs, wavefront sensing & control
systems. The current spaced-based high-contrast instruments on board HST [1]
and JWST [2] can achieve a planet-to-star contrast $\sim$ 10-5-10-6, and in
the near future, the coronagraph instrument on board the Roman Space telescope
aims to achieve a raw contrast of 10-8 [3, 4, 5]. The Habitable World
Observatory (HWO), as recommended by the Astro2020 Decadal survey, consisting
of a UV/O/IR telescope, aims to search for habitable biosignatures in planets
around many stars with a direct imaging instrument achieving a planet-to-star
contrast of 10-10.

Various technologies, such as coronagraphic architectures and wavefront
control algorithms used on the space-based high-contrast imaging instruments,
are usually tested on high-contrast imaging testbeds (Ex: HCIT, DST [6], HiCAT
[7]) in a vacuum environment that serves as an experimental platform for
simulating a space-like environment with dynamic aberrations created by
thermal and mechanical vibrations of the telescope and pointing errors. The
Space Coronagraph Optical Bench (SCoOB) at the University of Arizona Space
Astrophysics Lab is one such testbed using a Vector Vortex Coronagraph (VVC)
[8, 9, 10] to achieve $4\times 10^{-9}$ contrast at 630nm in 2% bandpass[11,
12, 13].

Among the various aberrations arising from optical surfaces, thermal changes,
jitter, and detector noise, high-contrast imaging instruments also suffer from
polarization-dependent phase and amplitude aberrations such as retardance
piston, defocus, and tilt generated by the telescope and instrument optics and
coating. As these low-order aberrations are incoherent, the deformable mirror
cannot correct for these aberrations; as a result, they reach the focal plane
as ghosts, affecting the achievable contrast [14]. Thus, end-to-end modeling
and simulations of the telescope and the instrument using the Polarization Ray
Tracing (PRT) algorithm are essential to understand the magnitude of these
aberrations and their effect on the contrast, which is routinely done for
high-contrast imaging systems [15, 16, 17, 18, 19]. The PRT algorithm and
description of polarization aberrations has been very well developed in the
last two decades and hence will not be described here in detail [20, 14, 21,
22] and is also available as an open-source Python package called Poke [23].

In this article, we present the simulations of polarization aberrations for
the high-contrast imaging testbed SCoOB using the Zemax OpticStudio API ray
trace and performing PRT in a Python environment and diffraction simulations
through a coronagraph using HCiPy[24]. The paper is divided as follows: The
optical design of SCoOB is provided in Section 2 for which the Jones and
Mueller Pupils are shown in Section 3. We discuss the retardance and the
leakage terms for the Vector Vortex Coronagraph masks and their effect on the
contrast for a perfect simulated VVC and the VVC masks manufactured by Beam
Engineering in Section 4. The results are summarized in Section 5.

##  2 SCoOB optical design

The SCoOB optical layout, shown in Figure 1, consists of 8 Off-Axis Parabolas
(OAP), two folded flat mirrors, a Fast Steering Mirror (FSM), and a Deformable
mirror (DM). Detailed information about the optical prescription, wavefront
sensing, control algorithms, and dark hole digging procedure can be found in
Ashcraft et al. [23] and Van Gorkom et al. [11]. SCoOB utilizes a Vector
vortex coronagraph (VVC) as the focal plane mask and a Lyot Stop for starlight
suppression. The input and output set of crossed circular polarizers are
positioned in the collimated space after OAP0 and OAP6 to eliminate the
leakage term from the VVC (explained in upcoming sections). The current
performance of the testbed is a contrast of 4$\times 10^{-9}$ in a 2% bandpass
and 2$\times 10^{-8}$ in a 10% bandpass at 630 nm[12].

Figure 1: The Optical layout of the Space Coronagraph Optical Bench, which is
in the TVAC chamber, consists of eight OAPs, two-fold mirrors, a Fast steering
mirror (FSM), a deformable mirror (DM), VVC as the focal plane mask and a Lyot
stop. The final beam at the focal plane after OAP7 is F/22.

##  3 Jones pupil and Mueller pupil

We trace a grid of 256$\times$256 rays across the pupil through the optical
layout of SCoOB using Zemax OpticStudio API ray trace. We do not include VVC,
Lyot stop, and polarizers in the ray trace, as they would be incorporated
later in the HCIPy simulations. The polarization aberrations in any optical
system depend on: 1. The radius of curvature/ Angle of Incidence (AOI) 2.
coatings on the surface. Using the ray trace, we obtain the AOI on all the
mirror surfaces as shown in Figure 2. The maximum AOI $\sim$ 12° is seen on
OAP1 and OAP2, but the AOI varies linearly on the optical surfaces within 1° ,
indicating the dominant polarization aberration will be
retardance/diattenuation tilt.

Figure 2: AOI on all the mirrors in SCoOB is shown on the normalized pupil coordinates (px, py). Surface | Avg reflection/transmission | Avg throughput
---|---|---
7 OAPs (Al+185nm $\rm MgF_{2}$) | 0.93 | 0.6469
Polarizer -Input | 0.99 | 0.3202
QWP-Input | 0.99 | 0.3169
FSM (Ag+10nm $\rm Si_{3}N_{4}$) | 0.99 | 0.3138
DM (Bare Gold) | 0.93 | 0.2918
2 Fold Flats (Al+10nm SiO) | 0.95 | 0.2630
Polarizer -Output | 0.99 | 0.2607
QWP-Output | 0.99 | 0.2581
Table 1: The average reflection/transmission from all the optical components
on SCoOB as estimated from the PRT.

The coating on all the OAPs is protected Aluminum (Al+185nm MgF2), FSM is
protected silver (Ag+10 nm Si3N4), the DM is bare gold, and folded flat
mirrors have protected aluminum (Al+10nm SiO). We estimate the Fresnel
reflection coefficients and average throughput for all the mirrors in SCoOB as
shown in Table 1 using the PRT routine in Python. The transmission for the
polarizer and QWPs are obtained from the transmission curve from High
Precision Linear Polarizer and Achromatic QWP.

###  3.1 Jones pupil

The Jones matrices are estimated at the exit pupil at 630nm using the PRT for
each input ray with $X$\- and $Y-$ polarization as shown in Figure 3. The
amplitude Axx and Ayy vary over the pupil $\sim$ 0.02% and $\sim$ 0.06%, the
crosstalk components Axy and Ayx show an amplitude of 0.8%. The phases $\phi
xx$ and $\phi yy$ are dominated mainly by tilt and piston and vary 0.02
($\sim$ 2nm) and 0.06 ($\sim$ 6nm) radians over the pupil. The DM will be able
to correct for the common aberration between $\phi xx$ and $\phi yy$; hence,
the difference of these aberrations are considered in the HCIPy simulations
explained in the following section.

Figure 3: Jones pupil for SCoOB estimated at the exit pupil location (shown in
Figure 1) for R band. Axx and Ayy show the amplitudes for X and Y polarized
light, respectively, and Axy and Ayx are the cross-coupled components.
Similarly, $\phi xx$ and $\phi yy$ show the phase in radians for $X$ and $Y$
polarized light, and $\phi xy$ and $\phi yx$ are the cross-coupled components.
Although the amplitude terms correspond to the diattenuation aberrations, they
do not affect coronagraphic performance. The difference between the $\phi xx$
and $\phi yy$ corresponds to polarization-dependent phase aberrations that
cannot be corrected by the DM and thus affect the achievable contrast.

###  3.2 Mueller pupil

The 4$\times$4 Mueller matrix can be estimated by the Kronecker product of the
Jones pupils shown in Figure 3 [25, 26]. The Mueller matrix for SCoOB at the
exit pupil is shown in Figure 4. All the elements of the Mueller pupil display
linear variation due to the dominant aberrations, diattenuation, and
retardance tilt from the OAPs and Fold mirrors. The terms M10, M20, and M30
correspond to the linear diattenuation otherwise known as instrumental
polarization $\sim$ 2.4%. The lower 3$\times$3 corresponds to the retardance
matrix, which shows $<$ 0.05% of linear retardance and 16% of
crosstalk/circular retardance. When averaged over the pupil, the Mueller
matrix will be a diagonal matrix with non-zero M10, M01, M23, and M32 terms
corresponding to the aberrations.

As we have circular polarizers on the input side, the Mueller matrix is
multiplied by the Stokes vector for left circular polarization [1,0,0,1] to
estimate the output polarization (columns 1 and 4). As the VVC is a half-wave
retarder, the input polarization is flipped to the right circular and
propagates through the right circular polarizer (RCP) on the output end,
whereas the leakage from the VVC is blocked by the RCP.

To understand the polarization aberrations better, we constructed a
polarimeter around SCoOB with a polarization state generator before the input
circular polarizers and polarization state analyzer at the exit pupil and
measured the Mueller matrix at 630nm. The data were reduced using an open-
source polarimetry package called Katsu. The measurement details and resulting
Mueller matrix are provided in Ashcraft et. al, 2024 [13].

Figure 4: The normalized Mueller pupil for SCoOB obtained from the Jones
pupil.The terms M10, M20, and M30 correspond to the linear diattenuation
otherwise known as instrumental polarization $\sim$ 2.4%. The lower 3$\times$3
corresponds to the retardance matrix which shows $<$ 0.05% of linear
retardance and 16% of crosstalk/circular retardance

##  4 Effect on contrast

To estimate the effect of polarization aberrations on the contrast, we perform
high-contrast imaging simulations using HCIPy [27], a physical optics-based
simulation framework. In our simulations, we propagate the wavefront defined
by the complex electric fields along the optical path of SCoOB. The
unpolarized light from the laser is propagated through the input circular
polarizers, followed by the Jones pupils up to the VVC, and finally through
the output circular polarizers. These simulations are performed for four
scenarios:

  * •

Perfect VVC + circular polarizers with no polarization aberrations

  * •

Perfect VVC + circular polarizers with polarization aberrations

  * •

Errors on VVC + circular polarizers with polarization aberrations

  * •

VVC from BeamCo + errors on circular polarizers with polarization aberrations.

###  4.1 Perfect VVC and circular polarizers

In Figure 5, the contrast versus focal plane distance is shown for the
scenario of a perfect VVC and circular polarizers. The left panel represents
the case without polarization aberrations (dominated by numerical errors in
the representation of the VVC), while the right panel displays the contrast
with polarization aberrations from the Jones pupil. The mean contrast in the
dark hole region (3-10 $\lambda$/D) is estimated to be 8.54$\times 10^{-12}$
without polarization aberrations and 1.545$\times 10^{-11}$ with polarization
aberrations. Although the mean contrast in the dark hole region is well below
the expected contrast floor for SCoOB, brighter speckles are observed from the
polarization aberrations at inner working angles of approximately 1-2
$\lambda$/D, which limits the contrast to 1$\times 10^{-9}$.

Figure 5: The contrast vs the focal plane distance is shown for a) perfect VVC
and circular polarizers without polarization aberrations (left panel) and b)
perfect VVC and circular polarizers with polarization aberrations (right
panel). The mean contrast estimated in the dark hole region (3-10 $\lambda$/D)
is not affected by the polarization aberrations; however, we see brighter
speckles $\sim$ 1$\times 10^{-9}$ near 1-3 $\lambda$/D.

###  4.2 Global errors on retardance of QWP, VVC, and leakage from LP

We extend the high contrast simulations by introducing errors on QWP, VVC, and
the leakage from the linear polarizer. The VVC is an achromatic half-wave
retarder and is expected to have a retardance of $\pi$ in the desired
bandpass. However, it’s impossible to manufacture a perfect achromatic half-
wave retarder as it’s made of many layers of liquid crystal molecules.
Similarly, the QWP’s retardance and the linear polarizer’s extinction ratio
may vary due to manufacturing/alignment errors. We incorporate these errors in
our simulations by changing the retardance of VVC and QWP by 2-4 degrees and
leakage from linear polarizer to be $1\times 10^{-5}$ \- $1\times 10^{-3}$ to
estimate their effect on the contrast. The retardance errors on the VVC will
result in a leakage that has to be filtered by the circular analyzer after the
VVC ( QWP and Pol after OAP6 in Figure 1); thus the global retardance errors
on the QWPs will result in imperfect filtering by the QWPs affecting the
contrast.

Figure 6 shows the contrast versus focal plane distance for the global
retardance errors on the VVC, QWP, and leakage on the LP. The figures on the
top panel show that the global retardance error on the VVC does not affect the
contrast when there are no retardance errors on QWP and extinction of LP $\sim
10^{-5}$. The figures on the middle panel correspond to the imperfect
filtering by the circular polarizers with retardance on the QWP be 92°changes
the contrast by an order of magnitude with global retardance error on VVC
increased by 2°. The speckle residuals increase $\sim$ $10^{-6}$ for IWA $<$
3$\lambda/D$. The last panel corresponds to the case of QWP retardance of 94°
, which will result in a mean contrast of $10^{-9}$, an order of magnitude
lower than the perfect case. The speckle residuals are seen to increase in the
dark hole region due to imperfect retardance of VVC and imperfect filtering by
the crossed circular polarizers. These simulations show that, to achieve a
contrast $10^{=10}$, it is very crucial to have perfect retardance of 90°for
the QWP, a high extinction of linear polarizer, and accurate alignment of the
crossed circular polarizers.

Figure 6: The contrast vs focal plane distance is shown for different cases of
global retardance errors on QWP and VVC and leakage from the Linear Polarizer.
The retardance errors on the VVC is important only in the case of imperfect
filtering by the crossed circular polarizers.

###  4.3 VVC masks from BeamCo

Two VVC focal plane masks were fabricated by Beam Engineering Co. (BEAM Co.)
for use in the SCoOB testbed. These are charge-6 VVCs intended to operate over
a 20% bandwidth centered at 635nm. The VVCs are mounted on Ohara S-BAL35R
substrates, manufactured by Rainbow Research Optics, a non-browning radiation-
resistant glass with an index of refraction of 1.59. The AR-coated substrates
achieve R $<0.5\%$ from 400-700nm. The units were measured by BEAM Co. via
Mueller matrix microscopy prior to acceptance. The linear retardance within a
1.3mm-diameter centered on the VVC singularity is plotted in Figure 8. In the
first round of units (2022:0303, 2022:0305, 2022:0404), a
20$\mathrm{\SIUnitSymbolMicro m}$-diameter opaque dot was placed over the
location of the vortex singularity. In the recent round of units, we procured
one VVC without an opaque dot (2024:0$\mathrm{\SIUnitSymbolMicro m}$) and two
with 30 and 45$\mathrm{\SIUnitSymbolMicro m}$-diameter dots.

The retardance maps for the recent set of VVCs that were received in 2024 are
shown in Figure 7 where the retardance closer to the center in the region $<$
3$\lambda/D$ is $<$ 176° and increases to 178-180° in the dark hole region.
The wavelength dependence of the averaged retardance is shown in Figure 8 at
different radial separations for the retardance maps in Figure 7. The mask
with the 45$\mathrm{\SIUnitSymbolMicro m}$ diameter has the retardance
178-180°over the wavelength 500-700nm and is expected to have lower leakage
and better performance in contrast. The imperfect retardance values on the VVC
give rise to a leakage term, ending on the focal plane if not filtered using
the crossed circular polarizers. We estimate the leakage term originating from
retardance errors for each VVC as explained in [28] as shown in Figure 9. The
leakage in the region of the dark hole is in the order of $10^{-3}$-$10^{-4}$,
which will be filtered using the crossed circular polarizers.

We then incorporate these VVC masks in our high contrast simulations to
estimate the contrast vs focal plane distance for each VVC. As shown in Figure
10, we obtain a mean contrast of 2.03$\times 10^{-9}$ for the VVC with no
opaque spot, 8.18$\times 10^{-10}$ for the VVC with opaque spot of 30$\mu$m
and 5.4$\times 10^{-10}$ for the VVC with opaque spot of 45$\mu$m,
respectively. SCoOB has achieved a contrast of 4$\times 10^{-9}$ in 2%
bandpass, comparable to the mean contrast estimated in our simulations.

Figure 7: 2D Linear retardance maps measured using Mueller matrix microscope
by Beam Engineering Co. for the most recent set of VVC masks without and with
an opaque dot of diameter 30 and 45 $\mu$m.  Figure 8: Linear retardance as a
function of wavelength for the three Beam Engineering Co. VVC units delivered
to UArizona in 2024. The retardance is reported as a mean in binned radial
separations in an f/48 beam at $\lambda_{0}$=635nm. Data from Beam Engineering
Co. Figure 9: Leakage term originating from the imperfect retardance for the
VVCs which gets filtered using the crossed circular polarizers. Figure 10: The
contrast vs focal plane distance shown for the three VVCs manufactured by Beam
Engineering Co. with perfect filtering by the crossed circular polarizers.

##  5 Summary and conclusions

Polarization aberrations from the optics of the coronagraph instrument will
give rise to polarized speckles in the image plane, affecting the achievable
contrast.

  * •

We have performed end-to-end polarization aberrations for a coronagraph
testbed SCoOB that aims to achieve a contrast better than $10^{-8}$ using VVCs
as the focal plane mask.

  * •

We performed polarization ray tracing to obtain Jones pupils that show the
dominant polarization aberration is retardance tilt. These Jones pupils are
converted to Mueller pupils to compare with the measured values.

  * •

The polarization aberrations included with an ideal VVC do not limit our mean
contrast in the dark hole region; however, they give rise to bright speckles
near the IWA $<3\lambda/D$.

  * •

The simulations incorporate global retardance errors on the VVC and QWP and
leakage of the linear polarizer to estimate the achievable contrast for the
case of an imperfect VVC with imperfect filtering. We find that the mean
contrast degrades by an order of magnitude with imperfect filtering, while the
retardance errors on the VVC with perfect filtering do not affect the contrast
performance.

  * •

As a final step, we include the measured retardance maps from the VVCs
manufactured by BEAM Co. and estimate the mean contrast obtained for each of
the masks on the order of 1$\times 10^{-9}$.

We conclude that these set of simulations demonstrate that polarization
aberrations do not affect the contrast performance for SCoOB, which aims to
obtain a contrast better than $10^{-8}$ in the dark hole region of
3-10$\lambda/D$; however, it severely degrades the contrast by order of
magnitude in the IWA $<$ 3$\lambda/D$ where the future HWO observatory aims to
image Earth-like planets directly.

###### Acknowledgements.

Portions of this research were supported by funding from the Technology
Research Initiative Fund (TRIF) of the Arizona Board of Regents and by
generous anonymous philanthropic donations to the Steward Observatory of the
College of Science at the University of Arizona.

## References

  * [1] Debes, J. H., Ren, B., and Schneider, G., “Pushing the limits of the coronagraphic occulters on hubble space telescope/space telescope imaging spectrograph,” Journal of Astronomical Telescopes, Instruments, and Systems 5(3), 035003–035003 (2019).
  * [2] Kammerer, J., Girard, J., Carter, A. L., Perrin, M. D., Cooper, R., Thatte, D., Vandal, T., Leisenring, J., Wang, J., Balmer, W. O., et al., “Performance of near-infrared high-contrast imaging methods with jwst from commissioning,” in [Space Telescopes and Instrumentation 2022: Optical, Infrared, and Millimeter Wave ], 12180, 1369–1388, SPIE (2022).
  * [3] Poberezhskiy, I., Heydorff, K., Luchik, T., Zhao, F., Cady, E., Bedrosian, G., Colavita, M., Creager, B., Goullioud, R., Groff, T., et al., “Roman coronagraph instrument: engineering overview and status,” Space Telescopes and Instrumentation 2022: Optical, Infrared, and Millimeter Wave 12180, 672–695 (2022).
  * [4] Poberezhskiy, I., Luchik, T., Zhao, F., Frerking, M., Basinger, S., Cady, E., Colavita, M. M., Creager, B., Fathpour, N., Goullioud, R., et al., “Roman space telescope coronagraph: engineering design and operating concept,” in [Space Telescopes and Instrumentation 2020: Optical, Infrared, and Millimeter Wave ], 11443, 314–335, SPIE (2021).
  * [5] Kasdin, N. J., Bailey, V. P., Mennesson, B., Zellem, R. T., Ygouf, M., Rhodes, J., Luchik, T., Zhao, F., Riggs, A. E., Seo, B.-J., et al., “The nancy grace roman space telescope coronagraph instrument (cgi) technology demonstration,” in [Space Telescopes and Instrumentation 2020: Optical, Infrared, and Millimeter Wave ], 11443, 300–313, SPIE (2020).
  * [6] Meeker, S. R., Noyes, M., Tang, H., Ruane, G., Prada, C. M., Bendek, E., Baxter, W., Crill, B., Riggs, A. E., Poon, P. K., et al., “The twin decadal survey testbeds in the high contrast imaging testbed facility at nasa’s jet propulsion laboratory,” in [Techniques and Instrumentation for Detection of Exoplanets X ], 11823, 322–334, SPIE (2021).
  * [7] N’Diaye, M., Choquet, E., Pueyo, L., Elliot, E., Perrin, M. D., Wallace, J. K., Groff, T., Carlotti, A., Mawet, D., Sheckells, M., et al., “High-contrast imager for complex aperture telescopes (hicat): 1. testbed design,” in [Techniques and Instrumentation for Detection of Exoplanets VI ], 8864, 558–567, SPIE (2013).
  * [8] Mawet, D., Serabyn, E., Liewer, K., Burruss, R., Hickey, J., and Shemo, D., “The vector vortex coronagraph: laboratory results and first light at palomar observatory,” The Astrophysical Journal 709(1), 53 (2009).
  * [9] Mawet, D., Pueyo, L., Moody, D., Krist, J., and Serabyn, E., “The vector vortex coronagraph: sensitivity to central obscuration, low-order aberrations, chromaticism, and polarization,” in [Modern Technologies in Space-and Ground-based Telescopes and Instrumentation ], 7739, 378–390, SPIE (2010).
  * [10] Murakami, N., Hamaguchi, S., Sakamoto, M., Fukumoto, R., Ise, A., Oka, K., Baba, N., and Tamura, M., “Design and laboratory demonstration of an achromatic vector vortex coronagraph,” Optics express 21(6), 7400–7410 (2013).
  * [11] Van Gorkom, K., Douglas, E. S., Ashcraft, J. N., Haffert, S., Kim, D., Choi, H., Anche, R. M., Males, J. R., Milani, K., Derby, K., Harrison, L., and Durney, O., “The space coronagraph optical bench (scoob): 2. wavefront sensing and control in a vacuum-compatible coronagraph testbed for spaceborne high-contrast imaging technology,” (2022).
  * [12] Van Gorkom, K., Douglas, E. S., Milani, K., Ashcraft, J. N., Anche, R. M., Jenkins, E., Ingraham, P., Haffert, S., Kim, D., Choi, H., and Durney, O., “The Space Coronagraph Optical Bench (SCoOB): 4. vacuum performance of a high contrast imaging testbed ,” in [Space Telescopes and Instrumentation 2024: Optical, Infrared, and Millimeter Wave ], 13092, SPIE (2024).
  * [13] Ashcraft, J. N., Douglas, E. S., Anche, R. M., Van Gorkom, K., Millar-Blanchaer, M. A., Melby, W., and Jenkins, E., “The space coronagraph optical bench (SCoOB): 3. Mueller matrix polarimetry of a coronagraphic exit pupil,” in [Space Telescopes and Instrumentation 2024: Optical, Infrared, and Millimeter Wave ], 13092, SPIE (2024).
  * [14] Breckinridge, J. B., Lam, W. S. T., and Chipman, R. A., “Polarization aberrations in astronomical telescopes: the point spread function,” Publications of the Astronomical Society of the Pacific 127(951), 445 (2015).
  * [15] Mendillo, C. B., Howe, G. A., Hewawasam, K., Martel, J., Cook, T. A., and Chakrabarti, S., “Polarization aberration analysis for the picture-c exoplanetary coronagraph,” Journal of Astronomical Telescopes, Instruments, and Systems 5(2), 025003 (2019).
  * [16] Millar-Blanchaer, M. A., Anche, R. M., Nguyen, M. M., and Maire, J., “The polarization aberrations of the gemini telescope as seen by the gemini planet imager,” in [Ground-based and Airborne Instrumentation for Astronomy IX ], 12184, 1278–1288, SPIE (2022).
  * [17] Schmid, H. M., Bazzon, A., Roelfsema, R., Mouillet, D., Milli, J., Menard, F., Gisler, D., Hunziker, S., Pragt, J., Dominik, C., et al., “Sphere/zimpol high resolution polarimetric imager-i. system overview, psf parameters, coronagraphy, and polarimetry,” Astronomy & Astrophysics 619, A9 (2018).
  * [18] Anche, R. M., Ashcraft, J. N., Haffert, S. Y., Millar-Blanchaer, M. A., Douglas, E. S., Snik, F., Williams, G., van Holstein, R. G., Doelman, D., Van Gorkom, K., et al., “Polarization aberrations in next-generation giant segmented mirror telescopes (gsmts) i. effect on the coronagraphic performance,” arXiv preprint arXiv:2304.02079 (2023).
  * [19] Anche, R. M., Haffert, S. Y., Ashcraft, J. N., Milani, K., Van Gorkom, K., Derby, K., Douglas, E. S., and Millar-Blanchaer, M. A., “Estimation of polarization aberrations and their effect on the coronagraphic performance for future space telescopes,” in [Techniques and Instrumentation for Detection of Exoplanets XI ], 12680, 248–260, SPIE (2023).
  * [20] Chipman, R. A., Lam, W.-S. T., and Young, G., [Polarized light and optical systems ], CRC press (2018).
  * [21] Chipman, R. A., “Mechanics of polarization ray tracing,” Optical Engineering 34(6), 1636–1646 (1995).
  * [22] Breckinridge, J., Kupinski, M., Davis, J., Daugherty, B., and Chipman, R., “Terrestrial exoplanet coronagraph image quality: study of polarization aberrations in habex and luvoir update,” in [Space Telescopes and Instrumentation 2018: Optical, Infrared, and Millimeter Wave ], 10698, 429–445, SPIE (2018).
  * [23] Ashcraft, J. N., Choi, H., Douglas, E. S., Derby, K., Van Gorkom, K., Kim, D., Anche, R., Carter, A., Durney, O., Haffert, S., et al., “The space coronagraph optical bench (scoob): 1. design and assembly of a vacuum-compatible coronagraph testbed for spaceborne high-contrast imaging technology,” in [Space Telescopes and Instrumentation 2022: Optical, Infrared, and Millimeter Wave ], 12180, 2064–2077, SPIE (2022).
  * [24] Por, E. H., Haffert, S. Y., Radhakrishnan, V. M., Doelman, D. S., Van Kooten, M., and Bos, S. P., “High Contrast Imaging for Python (HCIPy): an open-source adaptive optics and coronagraph simulator,” in [Adaptive Optics Systems VI ], Proc. SPIE 10703 (2018).
  * [25] Chipman, R. A., Lam, W. S. T., and Breckinridge, J., “Polarization aberration in astronomical telescopes,” in [Polarization Science and Remote Sensing VII ], 9613, 96130H, International Society for Optics and Photonics (2015).
  * [26] Fujiwara, H., [Spectroscopic ellipsometry: principles and applications ], John Wiley & Sons (2007).
  * [27] Por, E. H., Haffert, S. Y., Radhakrishnan, V. M., Doelman, D. S., van Kooten, M., and Bos, S. P., “High contrast imaging for python (hcipy): an open-source adaptive optics and coronagraph simulator,” in [Adaptive Optics Systems VI ], 10703, 1112–1125, SPIE (2018).
  * [28] Ruane, G., Serabyn, E., Prada, C. M., Baxter, W., Bendek, E., Mawet, D., and Riggs, A. E., “Experimental analysis of the achromatic performance of a vector vortex coronagraph,” in [Space Telescopes and Instrumentation 2020: Optical, Infrared, and Millimeter Wave ], 11443, 504–516, SPIE (2020).