# Paying Attention to Activation Maps in Camera Pose Regression

Yoli Shavit  Ron Ferens
Bar-Ilan University, Israel  Yosi Keller

###### Abstract

Camera pose regression methods apply a single forward pass to the query image
to estimate the camera pose. As such, they offer a fast and light-weight
alternative to traditional localization schemes based on image retrieval. Pose
regression approaches simultaneously learn two regression tasks, aiming to
jointly estimate the camera position and orientation using a single embedding
vector computed by a convolutional backbone. We propose an attention-based
approach for pose regression, where the convolutional activation maps are used
as sequential inputs. Transformers are applied to encode the sequential
activation maps as latent vectors, used for camera pose regression. This
allows us to pay attention to spatially-varying deep features. Using two
Transformer heads, we separately focus on the features for camera position and
orientation, based on how informative they are per task. Our proposed approach
is shown to compare favorably to contemporary pose regressors schemes and
achieves state-of-the-art accuracy across multiple outdoor and indoor
benchmarks. In particular, to the best of our knowledge, our approach is the
only method to attain sub-meter average accuracy across outdoor scenes. We
make our code publicly available from: here.

##  1 Introduction

Estimating the camera position and orientation using a captured image is an
essential task in various contemporary computer vision applications, ranging
from augmented reality to autonomous driving to navigation in closed
environments such as malls, museums, airports and amusement parks, to name a
few. Many of these applications require a fast and light-weight solution that
is deployable on low-end devices, such as cellphones.

Recent state-of-the-art pipelines for visual pose estimation follow a
hierarchical paradigm [28, 32, 26], where given a query image, an image
retrieval (IR) method is first applied to fetch candidate images from a large,
geo-referenced image database. Local features are extracted from the query and
geo-referenced images and then matched. The resulting 2D-2D matches are
transformed into 2D-3D correspondences using depth and/or a 3D model obtained
with methods such as Light Detection and Ranging (LiDAR) and Structure from
Motion (SfM). Finally, the pose (or a set of poses) is estimated from 2D-3D
matches using Perspective-n-Point (PnP) and RANSAC algorithms [11]. While
recent pipelines implementing the aforementioned approach achieve remarkable
accuracy, they entail significant difficulties. First, such systems have to be
implemented in a client-server architecture, due to the large image datasets
that cannot be stored on thin clients (cellphone etc.). Second, a client-
server approach requires an online data connection which is inapplicable for
standalone localization applications. Last, the localization typically
requires $50-2000ms$ processing time, depending on the method used, the size
of the database and the number of matches [26]. In contrast, absolute
regression-based approaches estimate the camera pose with a single forward
pass, using the query image as an input to a trained regressor. The end device
has to store only the regressor parameters.

As convolutional neural networks (CNNs) were shown to be efficient and
accurate regressors in multiple domains, they were also applied to camera pose
regression, starting with the seminal work of Kendall et al. [16]. The
proposed architecture consisted of a convolutional backbone and a multilayer
perceptron (MLP) head, separately regressing the position and orientation. The
appealing 5 $ms$ runtime and simplicity (a single component instead of a heavy
pipeline) paved the way to a new research paradigm for camera pose estimation.
Numerous methods, soon to follow, aimed at maintaining the low runtime and
memory requirements, while improving the accuracy and generalization of the
original method [16, 14, 15, 20, 21, 36, 38, 31, 37, 6].

Different camera pose regressors suggested different backbones [20, 31], loss
formulations [15] and MLP architectures [38, 21] as well as additional
manipulations of the output [36, 37]. Common to all these methods, is that
they apply the pose regression using a single global image encoding computed
by the backbone CNN.

In this work, we propose a novel attention-based approach that considers an
activation map as a sequence of local spatial embeddings. Specifically, we
employ Transformers [35], shown to provide an efficient computational means
for analyzing spatially varying image activations as sequences [7]. This
allows for an adaptive analysis of the spatial image content, as images
typically contain a mixture of localization-informative and uninformative
image cues. Thus, we cast the camera pose regression as two sequence-to-one
problems, where the intermediate activation maps are input to a pair of
translation-aware and rotation-aware Transformer encoders (Fig.
LABEL:fig:teaser). The output of each encoder is used to regress the
corresponding localization attribute. The gist of our approach is that each
Transformer Encoder captures different informative image cues for either
position or orientation estimation. We demonstrate this by overlaying the
attention weights as heatsmaps (Figs. LABEL:fig:teaser and 2). The largest
attention values of the Positional Encoder are around corners, which are a
commonly used localization cue, as in the SIFT detector [18]. Similarly, the
Orientational Encoder emphasizes elongated edges that are informative visual
cues for orientation estimation. Other image features that are less location-
informative show weak attention activations. By adaptively detecting and
weighting the informative spatial activations, Encoders improve the
localization accuracy.

In summary, our contributions are as follows:

  * •

We propose a novel attention-based approach to camera pose regression using
dual encoders, one per regression task.

  * •

Our approach is experimentally shown to compare favourably with state-of-the-
art regression-based schemes, when applied to outdoor and indoor localization
datasets.

##  2 Related Work

###  2.1 Image-Based Camera Pose Estimation

A common approach to image-based pose estimation applies large-scale image
retrieval followed by image matching and PnP-RANSAC [32, 26, 22, 10]. Each
query image to be localized, is encoded using a CNN trained for IR, and a
relatively small set of nearest neighbors is retrieved for image-to-image
matching. Local feature matching can be applied using brute force methods [32]
or another learned component [27]. In order to convert image matches to the
world plane, spatial information needs to be available at inference time. The
pose can then be estimated using PnP-RANSAC from 2D-3D correspondences. In
this work we study a faster and lighter alternative to this pipelined
approach, namely camera pose regression.

Camera pose regression was first suggested by [16]. The authors proposed to
follow the success of CNN backbones in extracting visual features from images
and apply them for regressing the pose directly from the input image.
Specifically, an MLP head was attached to a GoogLeNet backbone, regressing the
position and orientation of the camera (replacing the classifier heads in the
original architecture). This novel approach, albeit far less accurate than
localization pipelines, enabled performing pose estimation with a single
forward pass in only a few milliseconds.

To the best of our knowledge, the paradigm of regressing the camera pose from
the final output of a CNN backbone was adopted by all regressors to date [30].
Variations to the architecture focused on alternatives to the original
proposed CNN backbone [20, 21, 38, 31] and on deeper, branching architectures
for the MLP head [38, 21]. Other works tried to address overfitting by
averaging over predictions from models with randomly dropped activations [14]
or by reducing the dimensionality of the global image encoding with Long-
Short-Term-Memory (LSTM) layers [36]. Multi-modality fusion (for example, with
inertial sensors) was also suggested as a means to improve accuracy [5].
Recently, Wang et al. suggested to use attention in order to guide the
regression process [37]. The authors applied dot product self-attention with
SoftMax on the final output of the CNN backbone and updated it with the new
attention-based representation (through summation), before regressing the pose
with an MLP head. While our work also uses attention, we propose a novel
paradigm, where instead of directly regressing the pose from a single global
embedding (with/without attention-guided weights) multi-resolution activations
maps are processed as a sequence of features using Transformer encoders.

In terms of the learning process, one of the main challenges is to
appropriately weight the losses associated with the position and orientation
predictions. Kendall et al. suggested to learn the parameters controlling the
tradeoff between the losses to gain better performance [15]. This formulation
was adopted by many pose regressors, however it still requires manually tuning
the parameters’ initialization for different datasets [34]. In a recent work
[31], the authors trained the model separately for position and orientation in
order to reduce the need of additional parameters, while achieving comparable
accuracy. Alternative representations for the orientation were also proposed
to gain better balance and stability of the pose loss [38, 5]. As part of
studying the performance of pose regressors, Sattler et al. proposed an IR-
based baseline for camera pose regression [29]. With this baseline, pose
predictions are made by taking a weighted average of poses from a set of
nearest neighbors. The weighting is computed during inference time based on
the distances between the IR encoding of the query and its neighbors. This
baseline was used as an empirical illustration for some of the limitations of
current pose regressors since no regressor was able to consistently surpass it
on either outdoor or indoor datasets (we use the same datasets for evaluating
our work).

Brachmann and Rother introduced a novel class of pose regression schemes [3,
4], by training a CNN to estimate the 3D locations corresponding to the pixels
in the query image. This establishes 2D-3D correspondences that are utilized
by a differentiable PnP-RANSAC to estimate the camera pose, in both training
and testing phases. Such approaches achieve state-of-the-art accuracy.
However, since PnP-RANSAC is applied at inference time, they are significantly
slower compared to purely regression-based approaches. A different, yet
related body of works, estimate the absolute camera pose using the relative
motion between the query image and a reference image, for which the ground
truth pose is known. The learning thus focuses on regressing the relative pose
given a pair of images [2, 17, 9]. These methods can generalize better since
the model is no longer confined to the absolute reference scene, but require
the availability of a pose-labelled database of anchors during inference time.
Combining relative and absolute regression was also shown to achieve
impressive accuracy [34, 25].

###  2.2 Attention and Transformers in Image Analysis

Transformers were introduced by Vaswani et al. [35] as a novel formulation of
attention-based sequence encoding and analysis. Attention mechanisms [1] are
neural network layers that aggregate information from the entire input
sequence. Transformers introduced attention layers, that scan through each
element of a sequence and update it by aggregating information from the whole
sequence. Attention-based approaches were shown to outperform RNNs in encoding
long sequences, and were applied in multiple recent works in natural language
processing (NLP) and computer vision [8, 24]. In particular, in the context of
image analysis, the use of attention relates to the bilinear pooling approach
of Zhang et al.[39], and its multiple extensions [12]. These schemes compute
the inner products between all CNN activations, and were shown to improve
fine-grained recognition significantly. Bilinear pooling and attention schemes
allow to implement ‘a needle in a haystack’ approaches, where each entry in
the CNN activations map is adaptively weighted. Thus, allowing to numerically
emphasize the contribution of the task-informative image locations, in
contrast to the visual clutter. For instance, consider the attention weights
visualization in Fig. LABEL:fig:teaser, where we highlight the position and
orientation of informative image locations. In this work, we utilize self-
attention where the inner products are computed between the different entries
of each particular activation map. This is implemented by an Transformer
encoder, in contrast to using a Transformer decoder where the inner products
between the input data and an additional set of vectors, denoted as queries,
are computed.

##  3 Transformer Encoder for Pose Regression

In this work we propose to localize a camera by estimating
$\mathbf{p}=<\mathbf{x},\mathbf{q}>$, where $\mathbf{x}\in\mathbb{R}^{3}$ is
the position of the camera in the world and $\mathbf{q}\in\mathbb{R}^{4}$ is
the quaternion encoding its 3D orientation. Thus, the problem of camera pose
regression is to jointly learn two regression tasks: regressing the camera
position $\mathbf{x}$ and regressing the camera orientation $\mathbf{q}$. We
model each task as a sequence-to-one problem, where the input is a sequential
representation of a learned activation map and the output is an estimate of
the corresponding pose component (either $\mathbf{x}$ or $\mathbf{q}$).
Following the success of Transformers in learning sequence-to-one and
sequence-to-sequence problems in NLP [24] and more recently in Computer Vision
[7], we propose to use Transformer encoders for aggregating sequential
representations into a single latent vector. Specifically, we employ
positional and orientational encoders that learn to emphasize the visual cues
for position and orientation inference, respectively. An MLP can then regress
the resulting encoder output, replacing the classifier head used in standard
sequence-to-one architectures [8]. We denote our model TransPoseNet as it
shifts from the conventional pose regression paradigm to a Transformer-based
model. An overview of the proposed scheme is shown in Fig. LABEL:fig:teaser,
where the image is initially processed by a CNN backbone to compute activation
maps that are analyzed by the dual encoders and corresponding regression
heads.

###  3.1 Network Architecture

TransPoseNet is composed of a shared convolutional backbone and two branches
for separately regressing the position and orientation of the camera. Each
branch consists of a separate Transformer encoder and an MLP head. Using
separate activation maps and Transformers allows us to attend to the different
features at different resolutions depending on the particular learned task.

Convolutional Backbone. The activation maps of convolutional backbones capture
latent visual features at different resolutions. Given an image
$\mathbf{I}\in\mathbb{R}^{H\times W\times C}$, we sample the backbone at two
different resolutions and take an activation map per regression task:
$M_{\mathbf{x}}$ and $M_{\mathbf{q}}$ respectively.

Sequential Representations of Activation Maps. We follow [7] and convert an
activation map $\mathbf{M}\in\mathbb{R}^{H_{m}\times W_{m}\times C_{m}}$ to a
sequence $\widehat{\mathbf{M}}\in\mathbb{R}^{{H_{m}}\cdot W_{m}\times C_{t}}$
using a $1\times 1$ convolution (projecting to dimension $C_{t}$) followed by
flattening . We further append a learned orientation/position token
$\mathbf{t}\in\mathbb{R}^{C_{t}}$ to the sequence of map features, analogous
to the class token in sequence-to-one NLP classification Transformer
architectures [8]. Hence, the input to the encoder is given by:

$E_{in}=[\mathbf{t},\widehat{\mathbf{M}}]\in\mathbb{R}^{({H_{m}}\cdot
W_{m}+1)\times C_{t}}.$ (1)

Positional Encoding. In order to preserve the spatial information of each
location in the map, and assign a particular positional encoding to the token
$\mathbf{t}$, the spatial positions are encoded using positional encoding as
in [7]. We train two one-dimensional encodings for the $X$ and $Y$ axes
separately, to reduce the number of learnt positional parameters. Thus, for an
activation map $\mathbf{M}$ we define the sets of positional embedding vectors
$\mathbf{E}_{x}\in\mathbb{R}^{\left(W_{m}+1\right)\times C_{t}/2}$ and
$\mathbf{E}_{y}\in\mathbb{R}^{\left(H_{m}+1\right)\times C_{t}/2}$, such that
a spatial position $\left(i,j\right),$ $i\in 1..H_{m}$, $j\in 1..W_{m}$, is
encoded by the concatenating of the corresponding embedding vector:

$\mathbf{E}_{pos}^{i,j}=\begin{bmatrix}\mathbf{E}_{x}^{j}\\\
\mathbf{E}_{y}^{i}\end{bmatrix}\in\mathbb{R}^{C_{t}}.$ (2)

The pose token is assigned with position $(0,0)$ and the embedding vectors
$(\mathbf{E}_{x}^{0},\mathbf{E}_{y}^{0})$, as these vectors will only be
associated with that token. Similar to the activation map $\mathbf{M}$, the
positional embeddings are reshaped as sequence
$\widehat{\mathbf{E}}\in\mathbb{R}^{({H_{m}}\cdot W_{m}+1)\times C_{t}}$ and
passed to the encoder.

Transformer Encoder. We use a standard Transformer encoder architecture [8]
consisting of $n$ identical blocks. Each block contains a self multi-head
attention (MHA) module and an MLP with two layers and gelu non-linearity. The
input is passed through a LayerNorm [35] before each module (MHA/MLP) and
added back to the output with residual connections [35] and dropout. As in [7]
we add the position encoding to the input before each layer and normalize the
output of the final layer with an additional LN pass. The encoder’s output
$\mathbf{t}^{\prime}\in\mathbb{R}^{C_{t}}$ at the position of the special
token $t$, represents a global and context aware summarization of the local
features of the input activation map.

MLP Head. an MLP head with one hidden layer and gelu non-linearity regresses
the target vector ($\mathbf{x}$ or $\mathbf{q}$) from $\mathbf{t^{\prime}}$.

###  3.2 Camera Pose Loss

Camera pose regressors are optimized to minimize the deviation between the
ground truth pose $\mathbf{p}_{0}=<\mathbf{x}_{0},\mathbf{q}_{0}>$ and the
predicted pose $\mathbf{p}=<\mathbf{x},\mathbf{q}>$. The position loss
$L_{\mathbf{x}}$ and the orientation loss $L_{\mathbf{q}}$ are measured by the
Euclidean distance between the ground truth and the estimation:

$L_{\mathbf{x}}=||\mathbf{x}_{0}-\mathbf{x}||_{2}$ (3)
$L_{\mathbf{q}}=\left\|\mathbf{q_{0}}-\frac{\mathbf{q}}{||\mathbf{q}||}\right\|_{2}$
(4)

where $\mathbf{q}$ is normalized to unit vector in order to map to a valid
rotation matrix. In order to balance between the two losses, while considering
their scale differences, Kendall et al. suggested to learn their relative
weight by modelling the uncertainty of each task [15]:

$L_{\mathbf{p}}=L_{\mathbf{x}}\exp(-s_{\mathbf{x}})+s_{\mathbf{x}}+L_{\mathbf{q}}\exp(-s_{\mathbf{q}})+s_{\mathbf{q}},$
(5)

where $s_{\mathbf{x}}$ and $s_{\mathbf{q}}$ are learned parameters. Recently,
Shavit and Ferens showed the advantage of separately learning each task on its
own [31]. Here, we follow a combined approach, where we first train the entire
model to minimize Eq. 5 and then fine-tune each MLP head solely with its
respective loss (Eqs. 3 and 4).

###  3.3 Implementation Details

Our localization approach is implemented using EfficientNet-B0 [33] as our
convolutional backbone, previously shown to focus on informative visual
features for image classification. The EfficientNet backbone has several
reduction levels, each corresponding to activation maps with decreasing
resolution and increasing depth. We use an open source implementation of
EfficientNet [19] and use the activation maps from two endpoints (reduction
levels): $m_{rdct4}\in\mathbb{R}^{14\times 14\times 112}$ and
$m_{rdct3}\in\mathbb{R}^{28\times 28\times 40}$. We use $m_{rdct4}$ and
$m_{rdct3}$ as inputs for the position and orientation branches, respectively.
We linearly project each activation map to a common depth dimension
$C_{t}=256$, and learn positional encoding of the same depth. For the
Transformer encoder we use a standard implementation with six blocks and a
dropout of $p=0.1$. Each block contains an MHA layer with four heads and an
MLP preserving the input dimension $C_{t}$. The hidden layer of the MLP
regressor heads expands the dimension from $256$ to $1024$ before regressing
the output vector with a fully connected layer.

##  4 Experimental Results

Table 1: Comparative analysis of pose regressors on the Cambridge Landmarks dataset (outdoor localization). We report the median position/orientation error in meters/degrees for each method. Best results are highlighted in bold. Method | K. College | Old Hospital | Shop Facade | St. Mary |
---|---|---|---|---|---
IR Baseline [29] | 1.48/4.45 | 2.68/4.63 | 0.90/4.32 | 1.62/6.06 |
PoseNet [16] | 1.92/5.40 | 2.31/5.38 | 1.46/8.08 | 2.65/8.48 |
BayesianPN [14] | 1.74/4.06 | 2.57/5.14 | 1.25/7.54 | 2.11/8.38 |
LSTM-PN [36] | 0.99/3.65 | 1.51/4.29 | 1.18/7.44 | 1.52/6.68 |
SVS-Pose [21] | 1.06/2.81 | 1.50/4.03 | 0.63/5.73 | 2.11/8.11 |
GPoseNet [6] | 1.61/2.29 | 2.62/3.89 | 1.14/5.73 | 2.93/6.46 |
PoseNet-Learnable [15] | 0.99/1.06 | 2.17/2.94 | 1.05/3.97 | 1.49/3.43 |
GeoPoseNet [15] | 0.88/1.04 | 3.20/3.29 | 0.88/3.78 | 1.57/3.32 |
MapNet [5] | 1.07/1.89 | 1.94/3.91 | 1.49/4.22 | 2.00/4.53 |
IRPNet [31] | 1.18/2.19 | 1.87/3.38 | 0.72/3.47 | 1.87/4.94 |
TransPoseNet (Ours) |  0.60/2.43 |  1.45/3.08 | 0.55/3.49 |  1.09/4.99 |

Our proposed scheme focuses on improving the accuracy of absolute pose
regression. As such, we evaluate it on multiple contemporary datasets used for
benchmarking camera pose regressors, and compare the results to recent state-
of-the-art regression-based absolute localization methods. Other classes of
localization schemes, described in section 2.1, which utilize additional data
at inference time (localization pipelines [32, 26, 22, 10] and relative pose
regression [2, 17, 9]) or that are an order of magnitude slower (3D-based
scene coordinate regression [3, 4]) are not considered for this analysis.

Datasets. The Cambridge Landmarks [16] dataset depicts an urban environment
and represents outdoor localization tasks with a spatial extent of $\sim
900-5500m^{2}$. We use four of its six scenes for comparative evaluation, as
the other two scenes are not commonly benchmarked. The 7Scenes [13] dataset
includes seven small-scale indoor scenes, depicting a spatial extent of $\sim
1-10m^{2}$. Both datasets present various localization challenges such as
occlusions, reflections, motion blur, lighting conditions, repetitive textures
and variations in view point and trajectory.

Training. We optimize our model using Adam, with $\beta_{1}=0.9$,
$\beta_{2}=0.999$ and $\epsilon=10^{-10}$ and a batch size of $8$. At the
first step of the training we minimize the loss defined in Eq. 5. Loss
parameters are initialized as in [34]. We use a learning rate of
$\lambda=10^{-4}$ and decrease it by a factor of $10$ every $100$ ($200$)
epochs for indoor (outdoor) localization for up to $300$ ($600$) epochs. We
apply a weight decay of $10^{-4}$ and train the encoders with a dropout
$p=0.1$. At the second training stage, we fine-tune each MLP head separately
with its respective loss (Eqs. 3 and 4), while freezing all other weights in
the network. This allows us to achieve better performance without needing to
trade-off between the two objectives. Since the position is dominant in
localizing large scale scenes, we fine-tune the orientation head with a latent
position prior from the position Transformer encoder (changing the orientation
MLP to take a concatenated input). For indoor localization, we train both
heads independently and use a higher learning rate and a higher weight decay
($10^{-3}$ and $10^{-2}$, respectively). In order to allow our model to better
generalize, we follow the augmentation procedure used by [16]. At train time
we rescale the image so that its smaller edge is resized to $256$ pixels and
take a random $224\times 224$ crop. We then randomly jitter the brightness,
contrast and saturation. At test time we take the center crop after rescaling
with no further augmentations. All models were trained on a single NVIDIA
Tesla V100 GPU using the PyTorch framework [23].

###  4.1 Comparative Analysis of Camera Pose Regressors

Table 2: Localization results for the Cambridge Landmarks dataset. We report the average position/orientation errors in meters/degrees and the respective rankings. Best results are highlighted in bold. The final ranking is set according to the average rank. Method |  | Average
---
[m/deg]
Ranks |  | Final
---
Rank
PoseNet [16] | 2.09/6.84 | 10/10 | 10
BayesianPN [14] | 1.92/6.28 | 8/9 | 9
LSTM-PN [36] | 1.30/5.52 | 2/8 | 5
SVS-Pose [21] | 1.33/5.17 | 3/7 | 5
GPoseNet [6] | 2.08/4.59 | 9/6 | 8
PoseNet-Learnable [15] | 1.43/2.85 | 5/1 | 2
GeoPoseNet [15] | 1.63/2.86 | 6/2 | 3
MapNet [5] | 1.63/3.64 | 6/5 | 7
IRRNet [31] | 1.42/3.45 | 4/4 | 3
TransPoseNet (Ours) |  0.91/3.47 |  1/3 | 1
Table 3: Comparative analysis of pose regressors on the 7Scenes dataset (indoor localization). We report the median position/orientation error in meters/degrees for each method. Best results are highlighted in bold. Method | Chess | Fire | Heads | Office | Pumpkin | Kitchen | Stairs
---|---|---|---|---|---|---|---
IR Baseline [29] | 0.18/10.0 | 0.33/12.4 | 0.15/14.3 | 0.25/10.1 | 0.26/9.42 | 0.27/11.1 |  0.24/14.7
PoseNet [16] | 0.32/8.12 | 0.47/14.4 | 0.29/12.0 | 0.48/7.68 | 0.47/8.42 | 0.59/8.64 | 0.47/13.8
BayesianPN [14] | 0.37/7.24 | 0.43/13.7 | 0.31/12.0 | 0.48/8.04 | 0.61/7.08 | 0.58/7.54 | 0.48/13.1
LSTM-PN [36] | 0.24/5.77 | 0.34/11.9 | 0.21/13.7 | 0.30/8.08 | 0.33/7.0 | 0.37/8.83 | 0.40/13.7
GPoseNet [6] | 0.20/7.11 | 0.38/12.3 | 0.21/13.8 | 0.28/8.83 | 0.37/6.94 | 0.35/8.15 | 0.37/12.5
PoseNet-Learnable [15] | 0.14/4.50 | 0.27/11.8 | 0.18/12.1 | 0.20/5.77 | 0.25/4.82 | 0.24/5.52 | 0.37/10.6
GeoPoseNet [15] | 0.13/4.48 | 0.27/11.3 | 0.17/13.0 | 0.19/5.55 | 0.26/4.75 | 0.23/5.35 | 0.35/12.4
MapNet [5] | 0.08/3.25 | 0.27/11.7 | 0.18/13.3 | 0.17/5.15 | 0.22/4.02 | 0.23/4.93 | 0.30/12.1
IRPNet [31] | 0.13/5.64 | 0.25/9.67 | 0.15/13.1 | 0.24/6.33 | 0.22/5.78 | 0.30/7.29 | 0.34/11.6
AttLoc [37] | 0.10/4.07 | 0.25/11.4 | 0.16/11.8 |  0.17/5.34 | 0.21/4.37 | 0.23/5.42 | 0.26/10.5
TransPoseNet (Ours) |  0.08/5.68 |  0.24/10.6 |  0.13/12.7 |  0.17/6.34 |  0.17/5.6 |  0.19/6.75 | 0.30/7.02
Table 4: Localization results for the 7Scenes dataset. We report the average position/orientation errors in meters/degrees and the respective rankings. Best results are highlighted in bold. The final ranking is set according to the average rank. Method |  | Average
---
[m/deg]
Ranks |  | Final
---
Rank
PoseNet [16] | 0.44/10.4 | $9/10$ | $10$
BayesianPN [14] | 0.47/9.81 | $10/7$ | $9$
LSTM-PN [36] | 0.31/9.86 | $7/8$ | $7$
GPoseNet [6] | 0.31/9.95 | $7/9$ | $8$
PoseNet-Learnable [15] | 0.24/7.87 | $6/4$ | $5$
GeoPoseNet [15] | 0.23/8.12 | $4/5$ | $4$
MapNet [5] | 0.21/7.78 | $3/2$ | $3$
IRPNet [31] | 0.23/8.49 | $4/6$ | $5$
AttLoc [37] | 0.20/7.56 |  $2/$1 | 1
TransPoseNet (Ours) |  0.18/7.78 |  1/$2$ | 1

Figure 1: Visual comparison of position (a) and orientation (b) errors of
different pose regressors, with respect to the IR baseline reported by [29].
The results are reported for the Cambridge Landmarks dataset.

Outdoor Localization. We report the median position and orientation errors on
four outdoor scenes from the Cambridge Landmarks dataset. Table 1 shows the
localization results of our method (TransPoseNet) compared to all other pose
regressors reporting on this benchmark using a single input image. We also
include the results of an IR baseline recently proposed as a reference for
evaluating the limitations of absolute pose regression methods [29].
TransPoseNet yields the lowest position error across all scenes and provides
the lowest orientation error when evaluated on the Shop Facade scene. It also
achieves the highest number of state-of-the-art results across the dataset.
When summarizing the results (Table 2), TransPoseNet is the only regressor
achieving sub-meter accuracy. It ranks first in terms of average position
error and third in orientation. In addition, when considering the total rank
(average of position and orientation ranks), TransPoseNet again ranks first.

We also consider an IR baseline, recently shown to present a lower error bound
on the performance of current pose regressors. Figures 1a and 1b compare the
position and orientation errors of different pose regressors with respect to
the IR baseline results (indicated in black). TransPoseNet is the only pose
regressor whose localization error is consistently below the IR error bar,
across both position and orientation. To the best of our knowledge,
TransPoseNet is the only pose regressor to-date to achieve this result.

Indoor Localization. Table 3 shows the results obtained when evaluating
TransPoseNet on the 7Scenes dataset. As in outdoor localization we compare its
performance to all pose regressors reporting on this benchmark with a single
image input. TransPoseNet achieves the best positional error in six out of the
seven scenes and the highest number of state-of-the-art results overall.
Similar to outdoor localization, TransPoseNet ranks favorably in terms of
position and orientation errors, achieving first and second place respectively
and ranking first in total (Table 4). Another attention-based method [37] also
achieves the first place in total ranking, but with fewer state-of-the-art
results (two versus seven achieved with TransPoseNet). When comparing the
performance to the IR baseline (Table 3), TransPoseNet has 14 out of 15
results under the bar, more than any other pose regressor to-date.

###  4.2 Paying Attention to Local Features

A key element in our proposed method is the use of one encoder per task
(position and orientation) for processing activation maps. The motivation
behind this design is to allow each encoder to attend to different features in
the image. We assume that the network will focus on features such as corners
and blobs to estimate the relative translation, and on edges and lines for
estimating the relative rotation. Figure 2 shows attention heatmaps for three
query images taken from different scenes in the Cambridge Landmarks and the
7Scenes datasets. Each row depicts the query image and the upsampled attention
weights taken from the last layers of the position and orientation encoders
(second and third column, respectively). The position encoder tends to assign
higher weights to features that are intuitively more distinctive for coarser
localization and position estimation. In contrast, the orientation encoder
focuses on lines and long edges that are informative for estimating the
rotational motion. For example, consider the triplet in the third row, the
position encoder focuses on the windows at the image’s center of mass, while
the orientation encoder attends to the elongated spires. The observed
differences between the two encoders reinforces our dual-head model
architecture. The relative motion emerging from attention-driven local
features in the image, with respect to a memorized map, can be summarized with
context-aware global absolute image descriptors used for regressing the
position and orientation.

Figure 2: Heatmap visualization of attention weights from the position and
orientation encoders (column (b) and (c)). Each row shows the query image
(column (a)) along with the respecting attentional heatmaps.

###  4.3 Ablation Study

We study the importance and impact of two aspects of our model design: the
attention mechanism and the resolution and depth of the activation maps. In
each evaluation, we started from the network used in Section 4.1 and modified
a single algorithmic component or hyper-parameter. All ablations were
conducted using the Shop Facade scene.

Table 5: Ablation study of CNN backbones evaluated on the Shop Facade scene. We report the median position and orientation errors. Both EfficientNet-based CNNs outperform the Resnet50 CNN. Using the EfficientNetB1 backbone degrades the accuracy due to overfitting. Backbone | Position | Orientation
---|---|---
| [meters] | [degrees]
Resnet50 | 0.81 |  4.76
EfficientNetB0 | 0.65 | 3.63
EfficientNetB1 | 0.69 | 4.03

Backbone. For studying the choice of backbone, we considered the Resnet50,
EfficientNetB0 and EfficientNetB1 CNNs, where both EfficientNet variants use
significantly less parameters than Resnet50. As shown in Table 5, both
EfficientNet variants significantly outperforms Resnet50. It is unclear from
our simulations, whether this is due to less overfitting or to their improved
architecture. The smaller EfficientNetB0 backbone provides the best results.

The Role of Attention and Transformers. To study the parameters of the
Transformer encoders, we trained our scheme with a varying number of
Transformer encoder layers. On the hand, the more layers are used, the better
the learning capacity. On the other hand, an over-parameterized Transformer
encoder might lead to overfitting. The results reported in Table 6 show a
‘sweet spot’ when using six layers. We also varied the Transformer encoders’
inner embedding dimension to verify our choice. The results in Table 7 show
that, similar to varying the number of layers, there exists a ’sweet spot’
when using 256 for the encoder latent dimension.

Table 6: Ablations of the Transformer Encoder configuration evaluated on the Shop Facade scene. We report the median position and orientation errors. Using six layers optimizes the accuracy and avoids overfitting. #Encoder Layers | Position | Orientation
---|---|---
| [meters] | [degrees]
2 | 0.92 | 4.61
4 | 0.93 | 4.2
6 | 0.65 | 3.63
8 | 0.89 | 3.75
Table 7: Ablations of model’s embedding dimensionality, evaluated on the Shop Facade scene. We report the median position and orientation errors. Encoder Dimension | Position | Orientation
---|---|---
| [meters] | [degrees]
64 | 0.79 | 4.35
128 | 0.81 | 3.99
256 | 0.65 | 3.63
512 | 0.68 | 4.18

Localizing with Coarse and Fine Activation Maps. Activation maps from
different endpoints of a CNN backbone capture different features in the input
image. Deeper reduction levels cover growing areas in the figure with an
increased depth. As orientation and position estimation may require different
feature complexity, we train our model with different combinations of
$m_{rdct3}\in\mathbb{R}^{28\times 28\times 40}$ and
$m_{rdct4}\in\mathbb{R}^{14\times 14\times 112}$ from the EfficientNet
backbone, as inputs for the position and orientation encoders (without fine-
tuning). Table 8 shows the results of four combinations evaluated on the Shop
Facade scene from the Cambridge Landmarks dataset. We either provide
$m_{rdct4}$/$m_{rdct3}$ to both encoder heads or use a combination (providing
$m_{rdct3}$ to one encoder and $m_{rdct4}$ to the other). Assigning the
position encoder with coarser activation maps yields better accuracy. An
opposite trend is observed for activation maps used by the orientation
encoder.

Table 8: Ablations of activation maps evaluated on the Shop Facade scene. We provide different combinations of $m_{rdct3}\in\mathbb{R}^{28\times 28\times 40}$ and $m_{rdct4}\in\mathbb{R}^{14\times 14\times 112}$ to the encoder heads and report the median position and orientation errors. | Reduction
---
Position/Orientation
| Position
---
[meters]
| Orientation
---
[degrees]
$m_{rdct3}$/$m_{rdct3}$ | 1.08 | 3.60
$m_{rdct3}$/$m_{rdct4}$ | 0.99 | 4.32
$\mathbf{m}_{\mathbf{rdct4}}$/$\mathbf{m}_{\mathbf{rdct3}}$ | 0.65 | 3.63
$m_{rdct4}$/$m_{rdct4}$ | 0.81 | 4.05

##  5 Conclusion

We presented a novel paradigm for camera pose regression which uses sequential
representations of multi-scale activation maps instead of a single global
descriptor from a CNN backbone. By applying dual multi-head attention
encoders, our model is able to focus on different local features depending on
the learned task. Our method ranks first on both outdoor and indoor benchmarks
and achieves a more consistent performance with respect to recent baselines.

## References

  * [1] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [2] Vassileios Balntas, Shuda Li, and Victor Prisacariu.  Relocnet: Continuous metric learning relocalisation using neural nets.  In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
  * [3] E. Brachmann, A. Krull, S. Nowozin, J. Shotton, F. Michel, S. Gumhold, and C. Rother.  Dsac â€” differentiable ransac for camera localization.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2492–2500, Los Alamitos, CA, USA, jul 2017. IEEE Computer Society.
  * [4] E. Brachmann and C. Rother.  Learning less is more - 6d camera localization via 3d surface regression.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4654–4662, 2018.
  * [5] Samarth Brahmbhatt, Jinwei Gu, Kihwan Kim, James Hays, and Jan Kautz.  Geometry-aware learning of maps for camera localization.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [6] Ming Cai, Chunhua Shen, and Ian Reid.  A hybrid probabilistic model for camera relocalization.  2019\.
  * [7] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision – ECCV 2020, pages 213–229, Cham, 2020. Springer International Publishing.
  * [8] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  * [9] Mingyu Ding, Zhe Wang, Jiankai Sun, Jianping Shi, and Ping Luo.  Camnet: Coarse-to-fine retrieval for camera re-localization.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  * [10] M. Dusmanu, I. Rocco, T. Pajdla, M. Pollefeys, J. Sivic, A. Torii, and T. Sattler.  D2-net: A trainable cnn for joint description and detection of local features.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8084–8093, 2019.
  * [11] Martin A. Fischler and Robert C. Bolles.  Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography.  Commun. ACM, 24(6):381–395, June 1981.
  * [12] Y. Gao, O. Beijbom, N. Zhang, and T. Darrell.  Compact bilinear pooling.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 317–326, 2016.
  * [13] B. Glocker, S. Izadi, J. Shotton, and A. Criminisi.  Real-time rgb-d camera relocalization.  In 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pages 173–179, 2013.
  * [14] Alex Kendall and Roberto Cipolla.  Modelling uncertainty in deep learning for camera relocalization.  In Proceedings of the International Conference on Robotics and Automation (ICRA), 2016.
  * [15] A. Kendall and R. Cipolla.  Geometric loss functions for camera pose regression with deep learning.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6555–6564, 2017.
  * [16] A. Kendall, M. Grimes, and R. Cipolla.  Posenet: A convolutional network for real-time 6-dof camera relocalization.  In 2015 IEEE International Conference on Computer Vision (ICCV), pages 2938–2946, 2015.
  * [17] Z. Laskar, I. Melekhov, S. Kalia, and J. Kannala.  Camera relocalization by computing pairwise relative poses using convolutional neural network.  In 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), pages 920–929, 2017.
  * [18] David G. Lowe.  Distinctive image features from scale-invariant keypoints.  Int. J. Comput. Vision, 60(2):91–110, Nov. 2004.
  * [19] Luke Melaskyriazi.  efficientnet-pytorch.  https://pypi.org/project/efficientnet-pytorch/, 2019.
  * [20] Iaroslav Melekhov, Juha Ylioinas, Juho Kannala, and Esa Rahtu.  Image-based localization using hourglass networks.  In 2017 IEEE International Conference on Computer Vision Workshops, ICCV Workshops 2017, Venice, Italy, October 22-29, 2017, pages 870–877. IEEE Computer Society, 2017.
  * [21] Tayyab Naseer and W. Burgard.  Deep regression for monocular camera-based 6-dof global localization in outdoor environments.  2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1525–1530, 2017.
  * [22] H. Noh, A. Araujo, J. Sim, T. Weyand, and B. Han.  Large-scale image retrieval with attentive deep local features.  In 2017 IEEE International Conference on Computer Vision (ICCV), pages 3476–3485, 2017.
  * [23] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. Alche-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32, pages 8026–8037. Curran Associates, Inc., 2019.
  * [24] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  2019\.
  * [25] N. Radwan, A. Valada, and W. Burgard.  Vlocnet++: Deep multitask learning for semantic visual localization and odometry.  IEEE Robotics and Automation Letters, 3(4):4407–4414, 2018.
  * [26] P. Sarlin, C. Cadena, R. Siegwart, and M. Dymczyk.  From coarse to fine: Robust hierarchical localization at large scale.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12708–12717, 2019.
  * [27] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superglue: Learning feature matching with graph neural networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4938–4947, 2020.
  * [28] T. Sattler, B. Leibe, and L. Kobbelt.  Efficient effective prioritized matching for large-scale image-based localization.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(9):1744–1756, 2017.
  * [29] T. Sattler, Q. Zhou, M. Pollefeys, and L. Leal-Taixé.  Understanding the limitations of cnn-based absolute camera pose regression.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3297–3307, 2019.
  * [30] Yoli Shavit and Ron Ferens.  Introduction to camera pose estimation with deep learning, 2019.
  * [31] Yoli Shavit and Ron Ferens.  Do we really need scene-specific pose encoders.  In To Appear in 2021 IEEE International Conference on Pattern Recognition (ICPR), 2021.
  * [32] H. Taira, M. Okutomi, T. Sattler, M. Cimpoi, M. Pollefeys, J. Sivic, T. Pajdla, and A. Torii.  Inloc: Indoor visual localization with dense matching and view synthesis.  IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–1, 2019.
  * [33] Mingxing Tan and Quoc Le.  EfficientNet: Rethinking model scaling for convolutional neural networks.  volume 97 of Proceedings of Machine Learning Research, pages 6105–6114, Long Beach, California, USA, 09–15 Jun 2019. PMLR.
  * [34] Abhinav Valada, Noha Radwan, and Wolfram Burgard.  Deep auxiliary learning for visual localization and odometry.  ICRA, pages 6939–6946, 2018.
  * [35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30, pages 5998–6008. Curran Associates, Inc., 2017\.
  * [36] F. Walch, C. Hazirbas, L. Leal-Taixé, T. Sattler, S. Hilsenbeck, and D. Cremers.  Image-based localization using lstms for structured feature correlation.  In 2017 IEEE International Conference on Computer Vision (ICCV), pages 627–637, 2017.
  * [37] Bing Wang, Changhao Chen, Chris Xiaoxuan Lu, Peijun Zhao, Niki Trigoni, and Andrew Markham.  Atloc: Attention guided camera localization.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10393–10401, 2020.
  * [38] J. Wu, L. Ma, and X. Hu.  Delving deeper into convolutional neural networks for camera relocalization.  In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 5644–5651, 2017.
  * [39] X. Zhang, H. Xiong, W. Zhou, W. Lin, and Q. Tian.  Picking deep filter responses for fine-grained image recognition.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1134–1142, 2016.


# Evaluating hardware differences for crowdsourcing and traditional recruiting
methods
††thanks: This work was supported by the EPSRC [EP/S023917/1].

Paul-David Joshua Zuercher  Department of Engineering
University of Cambridge
Cambridge, United Kingdom
<EMAIL_ADDRESS>
###### Abstract

The most frequently used method to collect research data online is
crowdsouring and its use continues to grow rapidly. This report investigates
for the first time whether researchers also have to expect significantly
different hardware performance when deploying to Amazon Mechanical Turk
(MTurk). This is assessed by collecting basic hardware parameters (Operating
System, GPU, and used browser) from Amazon Mechanical Turk (MTurk) and a
traditional recruitment method (i.e., snowballing). The significant hardware
differences between crowdsourcing participants (MTurk) and snowball recruiting
are reported including relevant descriptive statistics for assessing hardware
performance of 3D web applications. The report suggests that hardware
differences need to be considered to obtain valid results if the designed
experiment application requires graphical intense computations and relies on a
coherent user experience of MTurk and more established recruitment strategies
(i.e. snowballing).

###### Index Terms:

Crowdsourcing, Hardware, Mechanical Turk

##  I Introduction

#### Motivation

The most frequently used method to collect research data online is
crowdsouring and its use continues to grow rapidly [1]. Crowdsourcing is a
form of recruitment in which a task requiring human intelligence is offered to
an undefined network of people - also called ”crowd” [2]. Reasons for
recruiting using crowdsourcing, through platforms such as Amazon Mechanical
Turk (MTurk), include better scalability, flexibility of research design, and
affordability to perform experiments than other forms of recruiting [1].
Besides these benefits, a major challenge of crowdsourcing research data is
adapting the experiment to the crowdsourcing environment to obtain sound and
valid research results [1].

Hardware differences can impact experimental results that rely on a coherent
user experience because better or worse user experience is impaired if the
hardware is not performant enough [3]. For less hardware demanding 2D web
applications, such as like surveys, potential hardware differences are less
likely to affect the perceived user experience because almost all devices used
should suffice the hardware requirements. For more complex 3D experiments
applications (for example 3D game-like environments) the hardware performance
requirements are higher, and significant differences in available hardware
performance can lead to differences in user experience between the targeted
group and the crowdsourcing group [3]. Therefore, the investigated hardware
differences are relevant to all studies that rely on a coherent user
experience of crowdsourcing participants and more traditional forms of
recruitment (i.e., snowballing) using 3D web applications, since significant
hardware differences can lead to a significantly different user experience
[3].

#### Research question and hypothesis

This report closes the research gap of missing hardware performance data by
addressing the question: How hardware demanding can Amazon Mechanical Turk
(MTurk)experiments be in comparison to traditional recruitment options? Since
no prior research could be found that deemed the hardware differences
significant, the null hypothesis is any difference in hardware performance
between crowdsourcing and snowball recruitment is only due to chance. The
proposed alternative hypothesis is that the hardware performance of MTurk is
different to hardware performance of participants recruited using snowballing.

#### Choice of dataset

This research question and hypothesis is answered based on an online survey of
hardware components collated from (1) MTurk and (2) using snowballing (based
on social media posts and by distributing over the research institute mailing
list). The performance of the obtained qualitative data of GPU manufacturer
and model is quantified in terms of performance using the benchmark results of
[4] and compared for significant differences between the recruitment channels.

##  II Related Work

In the following appropriate methods and techniques for (1) crowdsourcing, (2)
analysing hardware performance, and (3) acquiring relevant hardware
information are reviewed.

###  II-A Crowdsourcing with hardware demanding applications

The biggest crowdsourcing platform is Amazon Mechanical Turk (MTurk). On MTurk
the main actors are the requesters and workers [5]. Requesters pay a fee to
create Human intelligence tasks (HITs)which are defined as any task that is
simple to solve with human intelligence [1]. Workers can choose from and
submit solutions for the list of Human intelligence tasks (HITs) [5]. If the
requester accepts the worker’s contribution, the worker gets a fixed payment
from the requester via MTurk [5]. In the last decades, more and more
researchers have used MTurk to perform experiments [1]. Crowdsourcing
experiments can follow common research guidelines for planning, implementing
and reporting experiments to ensure the validity of crowdsourcing research
[1].

To the author’s knowledge, no previous research has investigated the hardware
performance demographic of MTurk workers. Despite not directly analysing and
reporting hardware, previous approaches for managing hardware in crowdsourcing
environments can be classified into two groups. Firstly, researchers created
participant pools that suffice the application’s hardware requirements by
filtering out participants with insufficient hardware [6]. Secondly,
researchers reduced the application’s hardware requirements and create
coherent test environments by outsourcing demanding computations to an
externally hosted server [7, 8, 9]. In the second approach, the computation is
separated into browser- and server-based rendering. Thus, reducing the
hardware requirements is achieved by outsourcing the computational intensive
graphical operations to remote machines (servers) and whilst only rendering
the resulting graphics in the browser which is locally less computational
demanding [9].

The performance of browser-based rendering can be analysed based on the
hardware and software used in the system, while remote rendering requires
collecting additional information such as an sufficient connection between the
local device to the remote device [7, 9]. However, for both browser- and
server-based rendering it is important to understand how demanding the local
computations can be to ensure that (1) only allow participants with sufficient
hardware performance and (2) operations exceeding the user’s hardware
capabilities are externalised to a sufficiently powerful server.

###  II-B Analysing hardware performance

In general, the applications execution depends on the system that operates the
hardware [5]. Since online crowdsourcing experiments using MTurk cannot
legally ask participants to run any application outside of the web-browser,
this report’s analysis is restricted to web-applications. Web-applications are
applications that run inside a web browser application on an operating system.
Thus, the execution of a web-application is restricted by the (1) browser
executing the web application, (2) operating system executing the web browser
and (3) underlying computing hardware [10].

Determining the expected web application performance can be done either by (a)
inferring the performance capabilities based on collected hardware information
(indirect) or (b) running benchmarks on a machine (direct) [11]. However,
running these hardware performance tests directly can put a strain on the
computer, takes time and moreover the results can suffer from high variance if
the participant performs background tasks [12]. Therefore, it is more
advisable to only collect the hardware data in remote experiments and infer
the performance from the specifications indirectly.

After having obtained the hardware component information from the users, the
hardware performance capability can be estimated. The estimation of the
hardware performance can either be done by analysing the technological
specifications or by back-referencing application specific benchmark results
[5]. Analysing the performance based on technological specifications is
complex because small performance on one sub-component can reduce the overall
systems’ performance significantly depending on the application leading to
many details that need to be investigated [5]. Therefore, it is often
preferable to determine the performance of a system based on benchmark results
if the approximate application scenario is already known.

###  II-C Background literature for acquiring similar datasets.

To survey the hardware information, one could (a) ask the participants to
enter these information manually or (b) try to detect the information
automatically. Letting the participants enter the information might lead to a
self-selection bias since some of the participants might not know the
information and the information would be subject to human error. The non-
manual alternative, is to obtain the information using Application Programming
Interfaces (APIs)[13]. Since MTurk only allows distributing web applications,
information retrieval methods are restricted by the Web-API of the browsers
[13]. Research on collection of hardware information is of particular interest
of privacy researchers in the area of ”fingerprinting” [13]. The objective of
this research area is obtaining as much information as possible about a system
to create a digital fingerprint of the system or user that allows
identification or classification of the individual [13]. While the goal when
collecting hardware information to estimate performance is not to identify the
user, their methods allow obtaining (1) browser information, (2) operating
system information, and (3) hardware information [13].

##  III Method (design and implementation)

The hardware performance is assessed by first collecting the hardware
information of participants and secondly, assigning the participant’s hardware
to a performance score based on the selected benchmark. Finally, appropriate
tests from the area of statistical inference are used to evaluate the
project’s hypothesis.

###  III-A Experiment design

The experiment uses two treatment groups to sample data using a web
application. The web application handles the consent and hardware information
collection from the participants. The website is shared via two different
links to differentiate data from crowdsourcing and traditional recruitment
participants. The crowdsourcing link is distributed as a HIT on MTurk and a
request to participate on the study through a social media post. The website
requires the participant to agree to the participant information sheet and
forwards them to the data collection page. On the data collection page,
demographic data is surveyed from the participant and the hardware information
is automatically detected. Subsequently, the website sends the encrypted data
to a University owned Qualtrics server using the Qualtrics Application
Programming Interface (API). Subsequently, the participant obtains a payment
for participating in the study.

###  III-B Acquisition of hardware components and benchmark

The browser information and operating system information is obtained via the
user-agent header that is sent with each http-request. The
WEBGL_debug_renderer_info from the WebGL Web-API is used for obtaining the
hardware specifications. This data requires further processing as it includes
the concatenated vendor information such as the brand and model name, and the
WebGL renderer.

An open benchmark dataset is used as a comparative metric for analysing the
overall performance of a system. Subsequently, a dataset that contains the
benchmark results is obtained. Datasets were searched via the google research
dataset search utility111Google dataset search with query: ”GPU benchmark 3D”.
and a google search 222Google search with the keywords: ”GPU benchmark 3D
download dataset” (first page only).. Google database search did not yield any
appropriate datasets for assessing 3D GPU performance. The few results were
either hardware specifications or mobile device benchmarks. However, a Google
search delivered eight datasets. These datasets were analysed based on (1)
open research compliance and (2) covering all GPUs required based on obtained
survey data [5].

TABLE I: Overview of considered datasets with criteria. Name | Open research | GPU Coverage
---|---|---
3DMark.com | no | -
PassMark | no | 200 (insufficient)
GFXBench | no | 245 (insufficient)
Blender Benchmark | yes, open | 248 (insufficient)
UL Benchmarks | no | 199 (insufficient)
UserBenchmarks | transparent and free | 1201 (sufficient)

UserBenchmark’s [4] dataset is used for evaluating the GPU performance, due to
their transparent method for analysing hardware performance and sufficiently
extensive collection size of benchmarks (compare to Table I). Their
transparency qualities are a publicly available dataset and an open formula
for calculating the benchmark metric. Their benchmarking score combines the
average FPS with higher than average and lower than average fluctuations from
multiple games (called Effective Frames per second (EFPS), see Equation 1)
[4].

$\displaystyle\begin{split}EFPS=0.35*avg+1.69/8*(&\text{0.1\% lowest FPS
average}\\\ &+\text{0.1\% lowest FPS max}\\\ &+\text{1\% lowest FPS
average}\\\ &+\text{1\% lowest FPS max})\end{split}$ (1)

###  III-C Data analysis

The hardware performance is reported in form of qualitative data but also
quantitatively. Qualitative information include used Operating systems,
browsers, most commonly used GPUs that can qualify future research and
policies. These qualitative information are complemented by quantitative
descriptive statistics and the evaluation of the hypothesis. Both qualitative
and quantitative information are synthesised to answer the research question.

R (v3.6.5) is used to test whether the data supports rejecting absence of
hardware performance differences between MTurk and snowball recruiting. For
this an appropriate test procedure and significance level must be defined. As
convention, the significance level is set to $\alpha=0.05$. In terms of
testing procedure, one distinguishes between parametric tests that assume the
validity of modeling the population distribution with a specific set of
parameters and non-parametric tests that do not make such assumptions. Whether
the distribution is sufficiently modeled using normal distribution parameters
is tested with the Shapiro-Wilk test, and the necessary homoscedasticity is
tested using the Levene’s test. If the data is both normal and homoscedastic,
the Mann-Whitney U-test is used to evaluate the hypothesis. Otherwise,
Wilcoxon’s Rank-Sum test is used.

##  IV Results

Firstly, the data is analysed in terms of distribution and relevant
descriptive statistics. The dataset contains the $357$ samples from MTurk
(n=179) and snowball (n=178) recruiting. The benchmark score’s distribution
spans between $0.65$ and $236$ Effective Frames per second (EFPS) and
indicates that most participants (68.0%) reach a performance of less than 12.5
EFPS (compare to Figure 1). The mean of the score is 24.24 and the median is
5.93. The overall operating systems running the browser were Windows 10
(88.0%), Windows 8 (1.7%), Windows 7 (3.4%), MacOS 10 (4.8%) and MacOS 11
(2.2%). Additionally, the browsers running the web applications were Chrome
(75.4%), Firefox (24.4%), and Safari (0.3%).

Figure 1: The relative frequency distribution of EFPS (compare to [4]) per
treatment group. The benchmark score density is plotted layered for Amazon
Mechanical Turk (MTurk)and snowball recruitment (private). The bin-width is
set to 6.25 as calculated by the Freedman–Diaconis rule.

Secondly, the hypothesis tests are executed. Levene’s test indicates
homoscedasticity, $F=17.466$, $p<0.001$. The Shapiro-Wilk test indicated that
the data is non-normally distributed, $W(356)=0.615$, $p<0.001$. This is
confirmed by visual analysis (compare to Figure 2). Therefore, the Wilcoxon’s
test is used to assess the hypothesis. The test indicates that the hardware
performance of MTurk participants ($Mdn=7.16$) is significantly different to
those recruited using snowballing ($Mdn=5.03$), $W=19078$, $p=0.001$.

Figure 2: The QQ-plot of the data plot the normal distribution against the
Benchmark dataset scores. The visual analysis suggests non-normal distribution
because the data is not homogeneous with the normal distribution.

##  V Discussion and conclusion

The data collection was limited by the certain available hardware information
through Web-API. Thus, only allowing the hardware performance investigation
based on the main graphical processing unit (GPU). While this provides a good
estimate of the expected system’s capabilities, a computer system’s
performance is influenced by more than the GPU [11]. Especially the interplay
between CPU and GPU can significantly reduce the performance that would be
possible with the GPU [5]. Future research could investigate more advanced
concepts such as inferring CPU models based on statistical modelling. Further
research could analyse study-specific interactions of collectable data (such
as the interaction of browser, graphics engine and GPU) on performance.
However, since the used benchmark covers multiple configurations these
interaction effects are assumed to be at lest mitigated.

Moreover, while the crowdsourcing data is expected to be adequately
representative because of the large sample size [6], the snowball recruitment
was sampled in a Euro-centric and academia over-representative social network.
Thus, results are expected to vary when the samples are drawn from different
populations. Nevertheless, is the data sampling representative for many other
European researchers and the methodology, novel data, and considerations
reported are still integral for researchers to ensure quality and validity of
experiments.

The results validate rejection of the null hypothesis in favor of the
alternative that hardware performance in crowdsourcing and traditional
(snowball) recruiting differs significantly ($p=0.01$). Indicating that
applications have significantly different hardware conditions on MTurk than on
traditional recruiting. Since the hardware on MTurk is significantly
different, it might be necessary to filter participants or adapt the
application to the crowdsourcing environment to obtain valid results [1].

However, the data also shows that there are vast intra-group differences. Both
MTurk and traditional recruiting data is not normally distributed and has a
high variance of performance scores. This distribution suggests that the
application’s performance shouldn’t be tailored based on the average hardware
performance of the either population. The high spread of performances suggests
the average performance is an insufficient metrics because the frames per
second of lower than average performing hardware can still negatively affect
experimental results. Many of the reported GPUs have EFPS that would severely
impede the user experience (under 30 FPS) [3]. Thus, it would be more
appropriate to tailor the application to a performance baseline that allows
decent performance (e.g. above 30 FPS) on all targeted devices.

Ensuring appropriate application performance is possible by increasing
application efficiency or by increasing the threshold of hardware requirements
participants must suffice. Determining the application baseline is now
possible based on the reported data, by investigating the distribution of
devices with their respective EFPS scores. If it is not possible to lower the
application’s requirements sufficiently by increasing software efficiency,
externalising rendering to a server can ensure valid research results, given
sufficient bandwidth to participants.

## References

  * [1] H. Aguinis, I. Villamor, and R. S. Ramani, “Mturk research: Review and recommendations,” _Journal of Management_ , vol. 47, no. 4, pp. 823–837, 2021.
  * [2] E. Estellés-Arolas and F. González-Ladrón-de Guevara, “Towards an integrated crowdsourcing definition,” _Journal of Information science_ , vol. 38, no. 2, pp. 189–200, 2012.
  * [3] J. Y. Chen and J. E. Thropp, “Review of low frame rate effects on human performance,” _IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans_ , vol. 37, no. 6, pp. 1063–1076, 2007.
  * [4] Userbenchmark.com. (2022) Userbenchmark - effective fps methodology. https://www.userbenchmark.com/Faq/What-are-effective-frames-per-second-EFps/112 [last accessed: 2022-07-05].
  * [5] G. Paolacci, J. Chandler, and P. G. Ipeirotis, “Running experiments on amazon mechanical turk,” _Judgment and Decision making_ , vol. 5, no. 5, pp. 411–419, 2010.
  * [6] L. Mohan, P. Raman, V. Choppella, and Y. R. Reddy, “A crowdsourcing approach for quality enhancement of elearning systems.” in _ISEC_ , 2017, pp. 188–194.
  * [7] E. Dolstra, R. Vliegendhart, and J. Pouwelse, “Crowdsourcing gui tests,” in _2013 IEEE Sixth International Conference on Software Testing, Verification and Validation_. IEEE, 2013, pp. 332–341.
  * [8] E. Y. Kan, W. K. Chan, and T. Tse, “Crowdadaptor: A crowd sourcing approach toward adaptive energy-efficient configurations of virtual machines hosting mobile applications,” in _2014 IEEE 38th Annual Computer Software and Applications Conference_. IEEE, 2014, pp. 493–502.
  * [9] A. Evans, M. Romeo, A. Bahrehmand, J. Agenjo, and J. Blat, “3d graphics on the web: A survey,” _Computers & graphics_, vol. 41, pp. 43–61, 2014.
  * [10] T. Glander, A. Moreno, M. Aristizabal, J. Congote, J. Posada, A. Garcia-Alonso, and O. Ruiz, “Reweb3d: Enabling desktop 3d applications to run in the web,” in _Proceedings of the 18th International Conference on 3D Web Technology_ , 2013, pp. 147–155.
  * [11] M. Gries, “Methods for evaluating and covering the design space during early design development,” _Integration_ , vol. 38, no. 2, pp. 131–183, 2004.
  * [12] J. v. Kistowski, J. A. Arnold, K. Huppler, K.-D. Lange, J. L. Henning, and P. Cao, “How to build a benchmark,” in _Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering_ , 2015, pp. 333–336.
  * [13] A. ElBanna and N. Abdelbaki, “Browsers fingerprinting motives, methods, and countermeasures,” in _2018 International Conference on Computer, Information and Telecommunication Systems (CITS)_. IEEE, 2018, pp. 1–5.