# Learning Better Keypoints for Multi-Object 6DoF Pose Estimation

Yangzheng Wu, Michael Greenspan
RCVLab, Dept. of Electrical and Computer Engineering,
Queen’s University, Kingston, Ontario, Canada
{ y.wu<EMAIL_ADDRESS>
###### Abstract

We investigate the impact of pre-defined keypoints for pose estimation, and
found that accuracy and efficiency can be improved by training a graph network
to select a set of disperse keypoints with similarly distributed votes. These
votes, learned by a regression network to accumulate evidence for the keypoint
locations, can be regressed more accurately compared to previous heuristic
keypoint algorithms. The proposed KeyGNet, supervised by a combined loss
measuring both Wassserstein distance and dispersion, learns the color and
geometry features of the target objects to estimate optimal keypoint
locations. Experiments demonstrate the keypoints selected by KeyGNet improved
the accuracy for all evaluation metrics of all seven datasets tested, for
three keypoint voting methods. The challenging Occlusion LINEMOD dataset
notably improved ADD(S) by $+16.4\%$ on PVN3D, and all core BOP datasets
showed an AR improvement for all objects, of between $+1\%$ and $+21.5\%$.
There was also a notable increase in performance when transitioning from
single object to multiple object training using KeyGNet keypoints, essentially
eliminating the SISO-MIMO gap for Occlusion LINEMOD.

Figure 1: Keypoints sampled by FPS (blue dot), BBox (orange triangle), and
KeyGNet (green star). Keypoints selected by FPS and BBox are based on the
geometry of individual objects, whereas KeyGNet keypoints are learned from the
color and geometry of all dataset objects.

##  1 Introduction

Estimating the pose of objects in a scene is a fundamental problem in computer
vision [20, 50, 30], which enables a number of important applications such as
robot grasping operations [25] and augmented reality [20]. The most basic
formulation, called Six Degree-of-Freedom Pose Estimation (6DoF PE), recovers
the 3DoF translation and 3DoF rotation parameters of an object that has
undergone a rigid transformation. The research community has expanded its
efforts to address more general variations, such as allowing deformable
transformations [26, 27] and recent one shot training scenarios [38, 41, 7]
which estimate poses without groundtruth values on novel (never-seen and
unknown) objects, albeit with a performance reduction. Nevertheless, 6DoF PE
remains an active area of investigation, with the efficient pose estimation of
multiple objects being a particular focus recently [42].

There are two main approaches to solve 6DoF PE, the first being where the pose
is directly regressed by the network [50, 37, 24], with the network’s output
represented as either rotation angles and translation offsets [24], a
transformation matrix [52], or quaternions [50]. Regression methods are
relatively efficient and often implemented as end-to-end trainable
architectures [50, 24, 37]. While early versions of the direct regression
approach tended to lack accuracy [50], this has been improved upon recently
[52].

The second approach are keypoint-based methods [18, 36, 49, 48], which are the
main alternative to direct regression methods. For the 6DoF PE problem,
keypoints are defined as a set of 3D coordinates, expressed within an object-
centric coordinate reference frame. Keypoint-based methods are typically two-
stage processes, starting with a keypoint regression network that first
estimates keypoint locations within an image. The second stage then uses
model-to-image keypoint correspondences to estimate pose, often making use of
classical methods such as RANSAC or voting-based approaches to increase
robustness [4, 22, 12]. Keypoint methods have been shown to be among the most
accurate solutions to 6DoF PE, with recent works continuing to evolve and
advance this approach [51, 8, 19].

Despite the success of keypoint methods, the selection of the pre-defined
object-centric keypoint locations has been overwhelmingly based on one of two
approaches: Farthest Point Sampling (FPS) [36, 18] of points on the object
surface, or selection of a subset of the corners of the object bounding box
(BBox) [48, 49]. Both of these approaches generate keypoints purely based on
the 3D surface geometry of the set of objects, and ignore other appearance
information, such as color. Both approaches are also heuristic, with their
main objective being to produce keypoints that are geometrically dispersed,
and which fall on [36, 18], or close to [49, 48], the objects’ surfaces. Their
main constraint is that the keypoints are sufficient in number (i.e. $\geq 3$)
and distributed in a way (i.e. non-collinearly) so that object transformations
can be recovered from the downstream model-to-image point correspondences,
e.g. using PnP [12] or a suitable alternative [22].

In this paper, we cast attention to the generation of the pre-defined
keypoints themselves. We show that a data-driven generation of the initial
object-centric keypoint set can serve to improve the accuracy and the
efficiency of existing keypoint-based 6DoF PE methods. A graph network is
trained to optimize a disperse set of keypoints with similarly distributed
votes for keypoint voting 6DoF PE methods [36, 18, 48, 49]. The network that
encodes the geometry and color information is supervised in a manner that
regularizes the learning process, by considering both the distribution of
votes to each keypoint, as well as the dispersion (i.e. geometrical
sparseness) among them. Specifically, one loss term considers the Wasserstein
similarity of the histograms of voters for each keypoint, and a separate loss
term enforces keypoint dispersion. Our main contributions are:

  * •

We introduce a novel loss function and a graph convolutional network to learn
to generate a set of keypoints for a given set of objects.

  * •

We experimentally demonstrate that our generated keypoint sets improve
performance of existing keypoint-based 6DoF PE methods. The method not only
increases accuracy on networks trained on single objects, but also reduces the
performance gap between single and mutliple object scenarios. When using the
learned keypoint locations, the training time for the 6DoF PE methods is also
reduced.

To our knowledge, this is the first work that learns the location of pre-
defined keypoints for 6DoF PE, rather than generating them heuristically. The
performance improvement is in some cases significant, increasing accuracy by
between $1\%$ and $21.5\%$, and reducing training convergence time by between
$6h$ and $11h$. Our code will be released to the public, upon acceptance of
this paper.

##  2 Related Work

Keypoint Extraction Methods, which identify keypoint locations in a scene,
have been explored by the community to solve a variety of computer vision-
related problems such as Simultaneous Localization And Mapping (SLAM) [13,
44], Neural Radiance Field (NeRF) [10], and Non-Rigid Structure from Motion
(NrSFM) [26, 27]. The main objective of these methods is to improve the
effective detection of keypoints for better overall performance. Some of these
methods [8, 51] employ a trainable network to estimate keypoints with
confidence scores in order to provide redundancies for the consecutive
downstream processes. Others [49, 6] apply logical or geometric constraints to
the keypoints during training. More recent methods [16, 43] embed keypoints
into the latent space of a network so that these latent keypoints can also be
trainable (supervised by the loss) making the overall structure end-to-end.

Whereas in the above-described methods, the keypoints are not fixed with
respect to some world coordinate reference frame, keypoints have also been
pre-defined at fixed locations for problems such as facial recognition [9, 1]
and human pose estimation [6, 39], for which skeleton joints appear as an
obvious choice for keypoint locations to facilitate modeling human motion.
Keypoint selection for some other problems, however, such as 6DoF PE, is not
as apparent.

Most keypoint optimization methods [11, 3] focus on fine-tuning the post-
processing ignoring the impact of the keypoint selection process. Very few
methods [40, 5] address such issues, and find that the overall performance can
be improved by altering pre-defined keypoints selection.

Keypoint-based 6DoF PE Methods [36, 18, 49, 48] exhibit relatively good
accuracy compared to viewpoint-based methods [28, 45, 34] or direct regression
[50, 32, 24] methods. Some keypoint-based methods [33, 32, 35] generate
hypotheses of the pre-defined keypoint locations by training a network. The
hypotheses are in the form of probability heat maps, and are often filtered by
a mask, estimated by a detection network in order to remove background pixels.
These masks are typically noisy due to the occlusion of scenes in 6DoF PE, and
performance can heavily rely on the robustness of the final least square
fitting algorithms.

Keypoint voting-based methods [36, 18, 17, 49, 48], however, can better
accommodate noise. The networks cast votes, typically one per pixel, to
accumulate evidence for each keypoint. Voting adds redundancy to the detection
mask, and due to the highly redundant and independent nature of voting, the
resulting keypoints can be more precise, leading to better overall pose
estimation.

(a) FPS, Wass. dist.=19.4M

(b) BBox, Wass. dist.=36.8M

(c) BBox, Wass. dist.=6.4M

Figure 2:  Histograms of Wasserstein distances to LINEMOD driller model
surface points, for 3 sets of 3 keypoints: a) FPS keypoints, b) BBox keypoints
with maximum Wasserstein distance, c) BBox keypoints with minimum Wasserstein
distance.

Distribution Similarity Measures are widely used in ML to quantify the
uncertainty of distributions, inspired by information theory. There are a
variety such metrics including KL Divergence (KL Div), JS Divergence (JS Div),
and Wasserstein Distance [23]. Cross-entropy [31] is widely used for
classification tasks. It simplifies the similarity measure by removing the
relative entropy of groundtruth in KL Div, which is consistent during
training.

WGAN [2] justified that Wasserstein distance can optimize GAN training by
transforming classification into a regression problem, so that a linear
gradient can be created compared to a traditional GAN. Unlike KL Div and JS
Div, Wasserstein distance is a symmetric metric. They also demonstrate that
Wasserstein distance can still be measured when two distributions do not
overlap, which most other metrics cannot. This often happens in GAN training,
and can occur in keypoint voting-based 6DoF PE when a regression network
estimates votes for multiple keypoints.

##  3 Method

###  3.1 Motivation

Existing keypoint methods [36, 18, 49, 48] use regression networks to estimate
a quantity that geometrically relates each image pixel (and/or point), to each
keypoint. A variety of such quantities have been explored in the literature,
including the offset [18], direction vector [36], and radial distance [49, 48]
between points. Once estimated, these quantities are used to cast votes in an
accumulator space, the collection of which allows for the robust estimate of
the keypoint locations in the scene.

Fig. 2 illustrates three histograms of the radial distance quantity [49, 48]
from each point on the surface of an object’s CAD model (the LINEMOD driller),
to each of three sets of three keypoints. Each keypoint set was chosen using a
different keypoint selection method. Each histogram bin represents the number
of votes for that bin value that results using a voting scheme. Fig. 2(a)
shows the histogram of this distribution with keypoints selected using
Farthest Point Sampling (FPS). Fig. 2(b) shows the histogram when the three
keypoints are chosen from the eight corners of the bounding box which maximize
the Wasserstein distance. Fig. 2(c) shows the histogram of those three
bounding box keypoints with minimal Wasserstein distance.

It can be seen that both FPS and maximum Wasserstein BBox keypoints have
distributions with a greater variance among the three keypoints, compared to
those selected by minimum Wasserstein distance wherein the distributions of
votes are more similar. As the majority of methods [18, 36, 48] train a single
network for all keypoints, a larger variance between the quantities regressed
for each keypoint results in a scenario similar to class imbalance [15] in a
classification network. In contrast, a reduced variance of the regression
quantities, as in Fig. 2(c), can allow for better convergence of the network,
resulting in both more accurate estimates and faster training, as demonstrated
in Section 4.

We repeated this test, and found that the lower variation for the minimum
Wasserstein keypoints occurred consistently for all LINEMOD objects. This
preliminary test motivated us to further explore keypoint selection, and
ultimately develop a network structure to learn keypoints that result in
similarly distributed votes.

Figure 3: Overview of Keypoint Graph Network (KeyGNet). The network is trained
to estimate optimized dispersed keypoint locations with similarly distributed
votes on training segments for all objects in a dataset.

###  3.2 Keypoint Graph Network

The preliminary distribution similarity test of keypoint votes motivated us to
design a network that learns keypoint locations for a set of objects. The
network, shown in Fig. 3, is trained to maximize the similarity of the
regression quantities between surface points and keypoints. It is based on a
Graph Convolutional Network whose nodes are all points on the visible (non-
occluded) surface of the object of interest in each scene of the training
data.

Our loss function comprises two terms, the first of which enforces the
keypoint votes to distribute similarly for each keypoint. Wasserstein distance
was chosen as the similarty measure, as it has been shown to outperform other
popular similarity measures in learned networks [2, 14]. The second keypoint
dispersion term increases the separation of keypoints, which serves to improve
the accuracy of final transformation estimation [49]. Our network is trained
using all objects in a dataset, resulting in a single set of keypoints for all
objects, which makes the method appropriate for Multiple Instance Multiple
Object (MIMO) tasks [42].

Let $K\\!=\\!\left\\{k_{j}\right\\}_{1}^{N_{K}}$ be a set of keypoints within
an object-centric frame, and let $S\\!=\\!\left\\{p_{i}\right\\}_{1}^{N_{S}}$
be a point cloud comprising the non-occluded, visible surface of an object of
interest in a scene. For each point $p_{i}$ in $S$, a _vote_ $v_{i}^{j}$ is
defined as the regression quantity between $p_{i}$ and keypoint $k_{j}$. For
example, if radial voting is used [49], then vote $v_{i}^{j}$ will be a scalar
distance, whereas for offset voting [18], $v_{i}^{j}$ will be a 3D
displacement, etc.

The task is to estimate a keypoint set
$K^{*}\\!\\!=\\!\\!\left\\{k^{*}_{j}\right\\}_{1}^{N_{K}}$ that satisfies two
conditions. The first condition is that votes
$V^{j}\\!=\\!\\{v_{i}^{j}\\}\;\forall\;p_{i}\\!\in\\!S$ all share a similar
distribution:

$V^{j}\sim V^{i}\ \;\;\forall\;\;i,j\in[1\ldots N_{K}].$ (1)

The second condition is that the keypoints $k^{*}_{j}$ are geometrically
dispersed, so that they lie not too close to each other:

$\arg\max\sum_{i=1}^{N_{K}}\sum_{j=i+1}^{N_{K}}|\\!|k^{*}_{i}-k^{*}_{j}|\\!|.\
$ (2)

Our keypoint graph network (_KeyGNet_) builds on edgeconv [47], in which
graphs are dynamically computed in a neighborhood determined using k-nearest
neighbors within a hierarchy of Voronoi regions of increasing radii. The
convolution kernel operates on the edges (vertices) between points. The
network classification structure is modified by reshaping the output from
$3\\!\times\\!N_{K}$ classes into $N_{K}$ normalized keypoints. At training,
the network input is the superset $P$ of all surfaces $S$ in a training
dataset, i.e. $P\\!=\\!\\{S_{1},S_{2}...S_{N_{P}}\\}$. The output is the set
of optimized keypoints, $K^{*}\\!\\!=\\!\\!\\{k^{*}_{j}\\}_{1}^{N_{K}}$, which
can then be used for both training and inference by any keypoint-based 6DoF PE
method.

To supervise KeyGNet training, we utilize a combined loss with two terms. The
first term is a Wasserstein loss $\mathcal{L}_{wass}$, which adapts the work
of WassGAN-GP [14] involving both a critic loss and gradient penalty. The
$\mathcal{L}_{wass}$ between two sets of keypoint votes $V^{i}$ and $V^{j}$
is:

$\begin{split}\mathcal{L}_{wass}=&\mathbb{E}_{V^{i}}[D({V^{i}})]-\mathbb{E}_{V^{j}}[D({V^{j}})]\\\
&+\lambda\;\mathbb{E}_{V}[(\left|\\!\left|\nabla_{V}D({V})\right|\\!\right|-1)^{2}]\end{split}$
(3)

where $D(V^{i})$ and $D(V^{j})$ are histogram distributions of votes $V^{i}$
and $V^{j}$ for respective keypoints $k^{*}_{i}$ and $k^{*}_{j}$, $D(V)$ is
the joint distribution of $V^{i}$ and $V^{j}$, $\nabla_{V}$ is a gradient
calculated from $V$, and $\lambda$ is a gradient penalty hyperparameter.
Instead of applying the gradient penalty only when the generator learns to
imitate groundtruth (as in WassGAN-GP [14]), we apply it to all votes. In this
way, the distributions of votes are therefore trained to be similar to each
other, rather than similar to the groundtruth.

The second term is a dispersion loss $\mathcal{L}_{dis}$, which is inspired by
the FPS algorithm:

$\mathcal{L}_{dis}=e^{-\gamma\left|\\!\left|k_{i}-k_{j}\right|\\!\right|}$ (4)

where $\left|\\!\left|k_{i}-k_{j}\right|\\!\right|$ is the distance separating
keypoints $k_{i}$ and $k_{j}$. This term reduces the loss when keypoints stay
farther separated, with the value of hyperparameter $\gamma$ chosen so that
$\mathcal{L}_{dis}\in(0,1]$. The combined loss will then be:

$\mathcal{L}=\alpha\mathcal{L}_{wass}+\beta\mathcal{L}_{dis}$ (5)

which is calculated for all pairs of keypoints. The relative values of
$\alpha$ and $\beta$ can evolve as training proceeds, to initially confer a
greater emphasis on the $\mathcal{L}_{wass}$ term.

In summary, we train KeyGNet to learn a set of dispersed keypoints with
similarly distributed votes, by adapting the keypoint locations to the
objects’ surface geometries. The result is a single set of keypoints defined
for all objects within a dataset, which can then be used within keypoint-based
6DoF PE methods.

##  4 Experiments

###  4.1 Implementation Details

The proposed KeyGNet is trained to generate a set of optimized keypoints,
which we tested on a variety of state-of-the-art PE methods [49, 36, 18]. The
input segments of KeyGNet are the non-occluded visible surfaces of the objects
of interest within a scene. These segments are colored point clouds, derived
from training images by applying the camera intrinsics and groundtruth (GT)
object pose values. Layer normalization instead of batch normalization is used
in order to apply the gradient penalty in $\mathcal{L}_{wass}$ [14]. We train
KeyGNet with a batch size of 32 fully paralleled on six RTX6000 GPUs. The
optimization used is SGD, with the initial learning rate set to
$lr\\!=\\!1e\\!-\\!3$, decaying by a scale of $0.1$ every 50 epochs.

We test the optimized keypoints on three keypoint-based 6DoF PE voting
methods, RCVPose [49], PVNet [36] and PVN3D [18]. These methods use three
distinct types of voting, i.e. radial [49], vector [36], and offset [18]
respectively. As each PE method uses a different voting scheme, the $D()$
quantities in the Wasserstein loss of Eq. 3 will be different for each,
leading to distinct keypoint sets for each method, even when using the same
dataset. These keypoint sets are trained for each PE method, for all objects
in each dataset.

We use publicly available implementations of the above three PE networks, as
provided by the authors of the original works. These networks are mostly
unaltered, the only change being to the output shape to accommodate the shift
from SISO (Single Instance Single Object) to MIMO. The SISO structures are
converted to MIMO by simply training a single regression network for all
keypoints of all dataset objects. For each training run of each PE method, all
factors remained the same including the loss function, hyperparameters,
network depth, and number of keypoints.

###  4.2 Datasets & Evaluation Metrics

We test our optimized keypoints generated by KeyGNet on all Benchmark for 6D
Object Pose Estimation (BOP) [21] core datasets. BOP is a benchmark of 6DoF PE
providing uniform structures and evaluation metrics for twelve 6DoF PE
datasets in various application fields, among which seven are considered to be
core. The BOP metrics evaluate the performance on various aspects including
visibility, symmetry, and projection when an estimated pose and a GT pose are
applied. The average recall (AR) of Visible Surface Discrepancy (VSD), Maximum
Symmetry-Aware Surface Distance (MSSD), and Maximum Symmetry-Aware Projection
Distance (MSPD) were tested, as well as the overall AR.

We also tested on the original metrics of LINEMOD (LM), Occlusion LINEMOD
(LMO) [20], and YCB Video (YCB-V) in order to compare against other recent
leading non-keypoint-based methods. ADD(S) was introduced with the LM dataset,
and evaluates the average distance (for asymmetric objects) and the minimum
distance (for symmetric objects) between estimated and GT poses. The estimate
is considered correct when the average distance between the CAD model
transformed by the estimated and GT poses is within a certain threshold (e.g.
$10\%$ of the object diameter). PoseCNN [50] proposed ADD(S) AUC, which was
designed to be robust by evaluating ADD(S) distance over a series of
thresholds. The accuracy ratio is then plotted in 2D space and connected by a
curve. The overall score is the ratio of the area under the curve versus total
area.

###  4.3 Results

Figure 4: Visualization of poses estimated by KeyGNet keypoints. The
overlapped dots are projected point clouds sampled from the object CAD models,
applied with the estimated poses.

Dataset PVNet PVN3D RCVPose SISO MIMO SISO MIMO SISO MIMO FPS KGN FPS KGN FPS
KGN FPS KGN BBox KGN BBox KGN LMO 61.3 64.8 (+3.5) 52.7 64.8 (+12.1) 66.4 69.9
(+3.5) 54.8 68.7 (+13.9) 73.6 76.7 (+3.1) 64.8 76.4 (+11.6) YCB-V 77.4 79.8
(+2.4) 68.2 78.9 (+10.7) 77.8 82.6 (+4.8) 69.3 81.9 (+12.6) 85.2 88.7 (+3.5)
79.8 88.2 (+8.4) TLESS 67.7 70.1 (+2.4) 62.5 69.7 (+7.2) 67.3 70.3 (+3) 63.2
70.3 (+7.1) 71.5 75.4 (+3.9) 64.3 74.9 (+10.6) TUDL 91.6 93.1 (+1.5) 85.7 93.1
(+7.4) 90.1 93.4 (+3.3) 87.2 92.9 (+5.7) 97.8 98.8 (+1) 90.2 98.2 (+8) IC-BIN
70.6 73.2 (+2.6) 65.7 73.2 (+7.5) 70.4 76.6 (+6.2) 67.2 76.1 (+8.9) 74.0 76.6
(+2.6) 69.7 76.2 (+6.5) ITODO 48.0 49.4 (+1.4) 27.8 48.0 (+20.2) 49.5 53.9
(+9.8) 32.4 53.9 (+21.5) 54.7 58.1 (+3.4) 46.5 58.1 (+11.6) HB 82.5 85.0
(+2.5) 70.9 84.7 (+13.8) 82.8 87.6 (+4.8) 73.5 87.2 (+13.7) 87.3 89.7 (+2.4)
76.2 89.7 (+13.5) Average 71.3 73.6 (+2.3) 61.9 73.2 (+11.3) 72.0 76.3 (+4.3)
63.9 75.9 (+12) 77 80.6 (+3.6) 70.2 80.2 (+10)

Table 1:  Comparison of heuristic (FPS, BBox) vs. KeyGNet (KGN) performance
evaluated by BOP AR. Keypoints selected by KeyGNet improve the evaluation
metrics for all seven BOP core datasets.

PE Mode Keypoint LMO YCB-V Train Method Method ADD(S) ADD(S) AUC Time PVNet
SISO FPS 40.8 73.4 16h KGN 48.0 (+7.2) 79.1 (+5.7) 10h (-6h) MIMO FPS 31.7
65.1 22h KGN 47.9 (+16.2) 78.1 (+13) 12h (-10h) PVN3D  SISO FPS 63.2 92.3 17h
KGN 70.5 (+7.3) 95.7 (+3.4) 10h (-7h) MIMO FPS 53.9 86.5 23h KGN 70.3 (+16.4)
94.6 (+8.1) 12h (-10h) RCVPose  SISO BBox 71.1 95.9 16h KGN 75.5 (+4.4) 97.6
(+1.7) 09h (-7h) MIMO BBox 62 92.5 22h KGN 75.6 (+13.6) 96.6 (+4.1) 11h (-11h)

Table 2:  Comparison of heuristic (FPS, BBox) vs. KeyGNet (KGN) performance
evaluated by ADD(S). KeyGNet improves ADD(S) for both LMO and YCB-V datasets.

The impact of KeyGNet keypoints for the three 6DoF PE methods are shown in
Table 1 and Table 2. In all cases, keypoints selected by KeyGNet boosts the
performance, whether in SISO mode or MIMO mode, for all three methods, on all
seven BOP core datasets. Specifically, in MIMO mode, the ADD(S) of PVN3D is
improved by $16.4\%$ on LMO, the ADD(S) AUC of PVNet is improved by $13\%$ on
YCB-Video, and the BOP AR of PVN3D is improved by $12\%$ on average on all
seven BOP core datasets. These results indicate that 6DoF PE performance can
be impacted and improved, just by altering the location of the pre-defined
keypoints. Use of the KeyGNet keypoints not only improved the metrics on
average over all objects in each dataset, but they also improved the metrics
for each individual object (see Supplementary Material Table.S.2, Table.S.3,
and Table.S.4). The observed improvement is likely due primarily to similarly
distributed votes being easier to learn by the regression network, compared to
other distributions, as indicated in Fig. 2. A secondary contributing factor
is keypoint dispersion, as supported by Sec. 5.4.

##  5 Ablation and Hyperparameter Tuning Experiments

###  5.1 SISO vs MIMO

BOP introduced the two terms SISO (Single Instance of a Single Object) and
MIMO (Multiple Instances of Multiple different Objects) to distinguish between
varying levels of challenge in solving 6DoF PE for different types of scenes.
The majority of previous keypoint-based methods [49, 48, 18, 36] were designed
to address the less challenging SISO case by only processing a single object
at a time, which allows the training of unique network parameters for each
distinct object. Motivated in part by the most recent edition of the BOP
Challenge competition [21], several recent methods [17, 29] address the MIMO
case, with multiple object categories trained within a single network.

Networks trained for MIMO are typically less accurate than those trained for
SISO, likely due to a degradation of accuracy of the regression network. In
contrast, our method can effectively handle the MIMO case, with little or no
degradation in accuracy. One single set of optimized keypoints returned by
KeyGNet for all dataset objects serves to simplify the learning process of the
subsequent 6DoF PE, reducing the SISO-MIMO performance gap.

LMO PVNet PVN3D RCVPose object FPS KGN FPS KGN BBox KGN ape -7.6 -0.3 -8.2
-0.3 -4.2 -0.4 can -12.2 0 -11.2 0 -2.9 0 cat -7.1 0 -7.6 -0.3 -5.7 -0.2
driller -8.2 0 -10.3 0 -1.3 +0.4 duck -11.3 -0.3 -9.7 -0.4 -7.2 +0.4 eggbox
-12.2 0 -10.2 0 -2.1 +0.2 glue -6.7 0 -9.8 -0.4 -6.3 +0.2 holepuncher -7.3 0
-7.2 0 -2.1 0 average -9.1 -0.1 -9.3 -0.2 -6.3 +0.1

Table 3: SISO-MIMO performance gap on LMO. The change in ADD(S) when
converting from SISO to MIMO, for heuristic (FPS or BBox) and KeyGNet (KGN)
keypoint selection. The ADD(S) change is small for MIMO training using KGN
keypoints.

PVNet | PVN3D | RCVPose
---|---|---
FPS | KGN | FPS | KGN | BBox | KGN
-6.3 | -1.0 | -6.7 | -1.2 | -3.4 | -1.0
Table 4: SISO-MIMO performance gap on YCB-V. The average change in ADD(S) for
all objects, when converting from SISO to MIMO, for keypoints sampled
heuristically (FPS or BBox) and KeyGNet (KGN). There is a relatively small
change in ADD(S) AUC when the PE network is trained simultaneously on multiple
objects using KGN keypoints.

To illustrate this, we conduct an experiment comparing the performance change
when a single network is first trained individually for each dataset object
(SISO), and is then trained simultaneously for all objects (MIMO). In both
training scenarios, the network parameters were initialized randomly, from a
normal distribution with a mean of 0 and a standard deviation of 1.

The results are shown for each LMO object in Table 3, and averaged for all
YCB-V objects in Table 4. (The full results for each YCB-V object is listed in
the Supplementary Material in Table S.3.) For each of the three PE methods,
the change in ADD(S) when training for single vs. multiple objects is listed,
when using either heuristic keypoint selection (FPS or BBox) or KeyGNet (KGN)
keypoints. As can be seen, the performance gap is significantly reduced when
the network is converted from SISO to MIMO. While degradation existed for all
objects and all three PE methods using heuristic keypoints, no degradation
resulted for most LMO objects using KeyGNet keypoints, with a much smaller
average degradation for YCB-V. Interestingly, RCVPose even demonstrates a
small improvement of $+0.1$ on average in LMO. This experiment again shows
that the distribution of votes has a vital impact on the performance of the
network, closely correlated to overall 6DoF PE performance.

###  5.2 Distribution Similarity Losses of KeyGNet

There are a variety of distribution similarity metrics such as KL Divergence,
JS Divergence, and cross entropy, which have been discussed in Sec. 2. The
distribution of the comparison of votes in KeyGNet is similar to the generator
supervision in WGAN, which used Wasserstein distance. We conduct an experiment
to compare different distribution similarity losses against
$\mathcal{L}_{wass}$ defined in Eq. 3.

LMO | PVNet | PVN3D | RCVPose
---|---|---|---
object | $\mathcal{L}_{kl}$ | $\mathcal{L}_{js}$ | $\mathcal{L}_{ce}$ | $\mathcal{L}_{wass}$ | $\mathcal{L}_{kl}$ | $\mathcal{L}_{js}$ | $\mathcal{L}_{ce}$ | $\mathcal{L}_{wass}$ | $\mathcal{L}_{kl}$ | $\mathcal{L}_{js}$ | $\mathcal{L}_{ce}$ | $\mathcal{L}_{wass}$
ape | +20.2 | +22.0 | +20.3 | +22.6 | +20.3 | +22.5 | +21.4 | +22.9 | +9.6 | +9.6 | +8.6 | +11.7
can | +11.8 | +12.8 | +12.3 | +13.8 | +11.5 | +12.6 | +12.4 | +14.1 | +13.4 | +15.4 | +13.0 | +15.6
cat | +14.9 | +15.8 | +15.1 | +16.9 | +15.2 | +14.5 | +14.8 | +17.1 | +12.6 | +13.2 | +13.1 | +14.1
driller | +13.1 | +14.2 | +13.2 | +15.0 | +13.8 | +13.2 | +10.2 | +15.4 | +9.0 | +10.2 | +8.0 | +11.1
duck | +15.8 | +17.2 | +15.9 | +18.0 | +16.1 | +16.8 | +16.7 | +18.0 | +21.1 | +22.6 | +21.8 | +23.1
eggbox | +9.9 | +10.9 | +10.0 | +11.9 | +9.1 | +10.0 | +10.4 | +12.2 | +9.6 | +10.6 | +8.6 | +11.9
glue | +16.1 | +17.3 | +16.8 | +18.2 | +16.6 | +16.3 | +15.9 | +18.2 | +6.6 | +8.2 | +5.4 | +8.7
holepuncher | +10.7 | +12.5 | +11.2 | +13.3 | +11.2 | +12.6 | +12.0 | +13.7 | +9.7 | +11.7 | +9.1 | +12.7
average | +14.1 | +15.3 | +14.3 | +16.2 | +14.2 | +14.8 | +14.2 | +16.4 | +11.5 | +12.7 | +11.0 | +13.6
Table 5: ADD(S) Improvements of KeyGNet Distribution Similarity Losses tested
on LMO in MIMO mode. keypoints selected by $\mathcal{L}_{wass}$ lead to more
improvements on all three PE methods compared to the other losses.
$\mathcal{L}_{js}$ improves $+1.2\%$ more than $\mathcal{L}_{kl}$ and
$\mathcal{L}_{ce}$ on average.

We train KeyGNet by applying losses based on different metrics (KL Div Loss
$\mathcal{L}_{kl}$, JS Div Loss $\mathcal{L}_{js}$, Cross Entropy Loss
$\mathcal{L}_{ce}$) on LMO, on all three 6DoF PE methods. The results are
shown in Table 5. $\mathcal{L}_{wass}$ improves the performance the most
compared to the other losses, scoring highest for each individual object for
each PE method. The second best performance was $\mathcal{L}_{js}$, followed
by the $\mathcal{L}_{kl}$ and $\mathcal{L}_{ce}$ which had similar
improvements. The result of this experiment confirmed the WGAN conclusion, and
our assumption made in Sec. 2 that, even when the distribution of keypoint
votes has no overlap, $\mathcal{L}_{wass}$ can create relatively linear
gradients compared to other similarity measures.

###  5.3 Number of Keypoints

Figure 5: ADD(S) vs. number of KeyGNet keypoints, LMO dataset. PVNet and PVN3D
improve for more than 3, and saturate at 8 keypoints. RCVPose saturates at 3
keypoints.

PE methods use various numbers of keypoints, to balance time and accuracy.
Some [18, 36, 46] argue that more keypoints provide redundancy to the least
square fitting algorithm ultimately used in the final transformation
estimation, whereas others [49, 48] use as few as three keypoints to ease the
estimation task of the backbone network.

We conduct an experiment testing the impact of the number of KeyGNet keypoints
on network accuracy. Fig. 5 shows the ADD(S) trend of three PE methods with
optimized keypoints training with the optimized keypoints in MIMO mode, using
all LMO objects. PVNet [36] and PVN3D [18] exhibited a slight overall
improvement in ADD(s) as the number of keypoints increased, which saturated
when there are more than eight keypoints. The ADD(S) of RCVPose [49] stayed
fairly constant, independent of an increase in the number of keypoints beyond
three. In subsequent experiments, we chose the number of keypoints that
optimized performance for each PE method, i.e. 8 keypoints for PVNet and
PVN3D, and 3 for RCVPose.

###  5.4 Loss Component Weights

Figure 6: Impact of weighting loss components of Eq. 5. KeyGNet converges
either slowly ($\alpha\\!=\\!0.7,\beta\\!=\\!0.3$) or less accurately
($\alpha\\!=\\!0.3,\beta\\!=\\!0.7$) with fixed weights. Dynamic weight
scheduling swaps weight values at epoch 50, resulting in both fast and
accurate training.

The KGNet loss function (Eq. 5) comprises two terms. $\mathcal{L}_{wass}$
supervises the distribution of votes associated with each keypoint whereas
$\mathcal{L}_{dis}$ keeps the keypoints dispersed. These two components can
compete with each other simply because those keypoints with a similar
distribution of votes tend to cluster in a local neighborhood. Training with
an equal weight of both components can therefore cause the network to converge
to a local minima, with closely clustered keypoints, which therefore are less
effective at the ultimate PE goal of transformation estimation [49].

To address this, we experiment with different combinations of $\alpha$ and
$\beta$, which are the weighting factors of the respective loss terms. The
smoothed loss curve for training RCVPose on LMO for 3 keypoints is plotted in
Fig. 6. This plot shows that KeyGNet converges faster when $\alpha\\!=\\!0.7$
and $\beta\\!=\\!0.3$, whereas it is more accurate but converges slower when
$\alpha\\!\\!=\\!\\!0.3$ and $\beta\\!\\!=\\!\\!0.7$. We therefore train the
network using the plotted dynamic schedule, which weights $\mathcal{L}_{wass}$
more heavily ($\alpha\\!=\\!0.7$) initially, and shifts to weigh
$\mathcal{L}_{dis}$ more heavily ($\beta\\!=\\!0.7$) at epoch 50. This
schedule has the benefit of converging both efficiently and accurately.

###  5.5 Initial Input of KeyGNet

SISO MIMO LMO object FPS-KGN BBox-KGN KGN ape 65 64.9 65 can 96 96 96.4 cat
57.5 57.9 58 driller 79.9 80.2 82.1 duck 64.7 65.2 65.6 eggbox 80.7 80.7 82.2
glue 73.9 74.2 75.1 holepuncher 79.8 80.7 81.2 average 74.7 75 75.6

Table 6: ADD(S) for different initial KeyGNet inputs, tested on LMO using
RCVPose. Segment initialization (KGN) is slightly more accurate than heuristic
keypoint initialization.

KeyGNet inputs at training all points that fall on the visible surface of
objects within a scene (segments). An alternative input could be a smaller
number of keypoints intially sampled by FPS or BBox. Instead of all surface
points on the object segments, these heuristically sampled keypoints could
alternately serve as the initial input to KeyGNet.

To explore the impact of initialization, we conduct an experiment to compare
performance with keypoints optimized on different KeyGNet inputs. One KeyGNet
is trained and evaluated in SISO mode with the input of heuristically selected
keypoints, with KGN optimiztion (FPS-KGN and BBox-KGN). An alternate KeyGNet
is trained on segments, and estimates keypoints for all object categories in
MIMO mode (KGN). As shown in Table 6, KeyGNet trained with object segment
input data is slightly more accurate than those with heuristically defined
keypoints as input. In our subsequent experiments, we trained on segments,
with one set of keypoints for all objects.

##  6 Conclusion

In summary, we proposed KeyGNet to select a set of pre-defined dispersed
keypoints with optimal similarly distributed votes for MIMO keypoint voting-
based 6DoF PE. The keypoints were selected by training a graph network to make
the histograms of votes share similar distributions, so that the regression
network can estimate votes more accurately. Our keypoints improved the
performance on all seven BOP core datasets among all three SOTA methods
tested, and reduced the SISO-MIMO degradation of these methods. Training time
was also reduced when using KeyGNet keypoints.

## References

  * [1] Brandon Amos, Bartosz Ludwiczuk, and Mahadev Satyanarayanan.  Openface: A general-purpose face recognition library with mobile applications.  Technical report, CMU-CS-16-118, CMU School of Computer Science, 2016\.
  * [2] Martin Arjovsky, Soumith Chintala, and Léon Bottou.  Wasserstein generative adversarial networks.  In International conference on machine learning, pages 214–223. PMLR, 2017.
  * [3] Axel Barroso-Laguna, Edgar Riba, Daniel Ponsa, and Krystian Mikolajczyk.  Key. net: Keypoint detection by handcrafted and learned cnn filters.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 5836–5844, 2019.
  * [4] Paul J Besl and Neil D McKay.  Method for registration of 3-d shapes.  In Sensor fusion IV: control paradigms and data structures, volume 1611, pages 586–606. International Society for Optics and Photonics, 1992\.
  * [5] Michael Bosse and Robert Zlot.  Keypoint design and evaluation for place recognition in 2d lidar maps.  Robotics and Autonomous Systems, 57(12):1211–1224, 2009.
  * [6] Z. Cao, G. Hidalgo Martinez, T. Simon, S. Wei, and Y. A. Sheikh.  Openpose: Realtime multi-person 2d pose estimation using part affinity fields.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019\.
  * [7] Hanzhi Chen, Fabian Manhardt, Nassir Navab, and Benjamin Busam.  Texpose: Neural texture learning for self-supervised 6d object pose estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4841–4852, 2023.
  * [8] Hansheng Chen, Pichao Wang, Fan Wang, Wei Tian, Lu Xiong, and Hao Li.  Epro-pnp: Generalized end-to-end probabilistic perspective-n-points for monocular object pose estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2781–2790, 2022.
  * [9] Savina Colaco and Dong Seog Han.  Facial keypoint detection with convolutional neural networks.  In 2020 International Conference on Artificial Intelligence in Information and Communication (ICAIIC), pages 671–674. IEEE, 2020.
  * [10] Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan.  Depth-supervised nerf: Fewer views and faster training for free.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12882–12891, 2022.
  * [11] Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian.  Centernet: Keypoint triplets for object detection.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 6569–6578, 2019.
  * [12] Martin A Fischler and Robert C Bolles.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography.  Communications of the ACM, 24(6):381–395, 1981.
  * [13] Ruben Gomez-Ojeda, Francisco-Angel Moreno, David Zuniga-Noël, Davide Scaramuzza, and Javier Gonzalez-Jimenez.  Pl-slam: A stereo slam system through the combination of points and line segments.  IEEE Transactions on Robotics, 35(3):734–746, 2019.
  * [14] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville.  Improved training of wasserstein gans.  Advances in neural information processing systems, 30, 2017.
  * [15] Haibo He and Edwardo A Garcia.  Learning from imbalanced data.  IEEE Transactions on knowledge and data engineering, 21(9):1263–1284, 2009.
  * [16] Xingzhe He, Bastian Wandt, and Helge Rhodin.  Latentkeypointgan: Controlling gans via latent keypoints.  arXiv preprint arXiv:2103.15812, 2021.
  * [17] Yisheng He, Haibin Huang, Haoqiang Fan, Qifeng Chen, and Jian Sun.  Ffb6d: A full flow bidirectional fusion network for 6d pose estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3003–3013, 2021.
  * [18] Yisheng He, Wei Sun, Haibin Huang, Jianran Liu, Haoqiang Fan, and Jian Sun.  Pvn3d: A deep point-wise 3d keypoints voting network for 6dof pose estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [19] Yisheng He, Yao Wang, Haoqiang Fan, Jian Sun, and Qifeng Chen.  Fs6d: Few-shot 6d pose estimation of novel objects.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6814–6824, 2022.
  * [20] Stefan Hinterstoisser, Vincent Lepetit, Slobodan Ilic, Stefan Holzer, Gary Bradski, Kurt Konolige, and Nassir Navab.  Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes.  In Asian conference on computer vision, pages 548–562. Springer, 2012.
  * [21] Tomáš Hodaň, Martin Sundermeyer, Bertram Drost, Yann Labbé, Eric Brachmann, Frank Michel, Carsten Rother, and Jiří Matas.  Bop challenge 2020 on 6d object localization.  In European Conference on Computer Vision, pages 577–594. Springer, 2020.
  * [22] Berthold KP Horn, Hugh M Hilden, and Shahriar Negahdaripour.  Closed-form solution of absolute orientation using orthonormal matrices.  JOSA A, 5(7):1127–1135, 1988.
  * [23] Leonid V Kantorovich.  Mathematical methods of organizing and planning production.  Management science, 6(4):366–422, 1960.
  * [24] Wadim Kehl, Fabian Manhardt, Federico Tombari, Slobodan Ilic, and Nassir Navab.  Ssd-6d: Making rgb-based 3d detection and 6d pose estimation great again.  In Proceedings of the IEEE international conference on computer vision, pages 1521–1529, 2017.
  * [25] Kilian Kleeberger, Christian Landgraf, and Marco F Huber.  Large-scale 6d object pose estimation dataset for industrial bin-picking.  In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2573–2578. IEEE, 2019.
  * [26] Suryansh Kumar, Yuchao Dai, and Hongdong Li.  Multi-body non-rigid structure-from-motion.  In 2016 Fourth International Conference on 3D Vision (3DV), pages 148–156. IEEE, 2016.
  * [27] Suryansh Kumar and Luc Van Gool.  Organic priors in non-rigid structure from motion.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part II, pages 71–88. Springer, 2022.
  * [28] Yann Labbé, Justin Carpentier, Mathieu Aubry, and Josef Sivic.  Cosypose: Consistent multi-view multi-object 6d pose estimation.  In European Conference on Computer Vision, pages 574–591. Springer, 2020.
  * [29] Alan Li and Angela P Schoellig.  Multi-view keypoints for reliable 6d object pose estimation.  arXiv preprint arXiv:2303.16833, 2023.
  * [30] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  * [31] Shie Mannor, Dori Peleg, and Reuven Rubinstein.  The cross entropy method for classification.  In Proceedings of the 22nd international conference on Machine learning, pages 561–568, 2005.
  * [32] Markus Oberweger, Mahdi Rad, and Vincent Lepetit.  Making deep heatmaps robust to partial occlusions for 3d object pose estimation.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 119–134, 2018.
  * [33] Kiru Park, Timothy Patten, and Markus Vincze.  Pix2pose: Pixel-wise coordinate regression of objects for 6d pose estimation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7668–7677, 2019.
  * [34] Kiru Park, Timothy Patten, and Markus Vincze.  Neural object learning for 6d pose estimation using a few cluttered images.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision – ECCV 2020, pages 656–673, Cham, 2020. Springer International Publishing.
  * [35] Georgios Pavlakos, Xiaowei Zhou, Aaron Chan, Konstantinos G Derpanis, and Kostas Daniilidis.  6-dof object pose from semantic keypoints.  In 2017 IEEE international conference on robotics and automation (ICRA), pages 2011–2018. IEEE, 2017.
  * [36] Sida Peng, Yuan Liu, Qixing Huang, Xiaowei Zhou, and Hujun Bao.  Pvnet: Pixel-wise voting network for 6dof pose estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4561–4570, 2019.
  * [37] Mahdi Rad and Vincent Lepetit.  Bb8: A scalable, accurate, robust to partial occlusion method for predicting the 3d poses of challenging objects without using depth.  In Proceedings of the IEEE International Conference on Computer Vision, pages 3828–3836, 2017.
  * [38] Ivan Shugurov, Fu Li, Benjamin Busam, and Slobodan Ilic.  Osop: a multi-stage one shot object pose estimation framework.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6835–6844, 2022.
  * [39] Iqbal Singh, Xiaodan Zhu, and Michael Greenspan.  Multi-modal fusion with observation points for skeleton action recognition.  In 2020 IEEE International Conference on Image Processing (ICIP), pages 1781–1785. IEEE, 2020.
  * [40] Haowen Sun, Taiyong Wang, and Enlin Yu.  A dynamic keypoint selection network for 6dof pose estimation.  Image and Vision Computing, 118:104372, 2022.
  * [41] Jiaming Sun, Zihao Wang, Siyu Zhang, Xingyi He, Hongcheng Zhao, Guofeng Zhang, and Xiaowei Zhou.  Onepose: One-shot object pose estimation without cad models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6825–6834, 2022.
  * [42] Martin Sundermeyer, Tomáš Hodaň, Yann Labbe, Gu Wang, Eric Brachmann, Bertram Drost, Carsten Rother, and Jiří Matas.  Bop challenge 2022 on detection, segmentation and pose estimation of specific rigid objects.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2784–2793, 2023.
  * [43] Supasorn Suwajanakorn, Noah Snavely, Jonathan J Tompson, and Mohammad Norouzi.  Discovery of latent 3d keypoints via end-to-end geometric reasoning.  Advances in neural information processing systems, 31, 2018.
  * [44] Jiexiong Tang, Ludvig Ericson, John Folkesson, and Patric Jensfelt.  Gcnv2: Efficient correspondence prediction for real-time slam.  IEEE Robotics and Automation Letters, 4(4):3505–3512, 2019.
  * [45] Gu Wang, Fabian Manhardt, Jianzhun Shao, Xiangyang Ji, Nassir Navab, and Federico Tombari.  Self6d: Self-supervised monocular 6d object pose estimation.  In European Conference on Computer Vision, pages 108–125. Springer, 2020.
  * [46] Gu Wang, Fabian Manhardt, Federico Tombari, and Xiangyang Ji.  Gdr-net: Geometry-guided direct regression network for monocular 6d object pose estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16611–16621, 2021.
  * [47] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon.  Dynamic graph cnn for learning on point clouds.  Acm Transactions On Graphics (tog), 38(5):1–12, 2019.
  * [48] Y. Wu, A. Javaheri, M. Zand, and M. Greenspan.  Keypoint cascade voting for point cloud based 6dof pose estimation.  In 2022 International Conference on 3D Vision (3DV), pages 176–186, Los Alamitos, CA, USA, sep 2022. IEEE Computer Society.
  * [49] Yangzheng Wu, Mohsen Zand, Ali Etemad, and Michael Greenspan.  Vote from the center: 6 dof pose estimation in rgb-d images by radial keypoint voting.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part X, pages 335–352. Springer, 2022.
  * [50] Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox.  Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes.  2018\.
  * [51] Heng Yang and Marco Pavone.  Object pose estimation with statistical guarantees: Conformal keypoint detection and geometric uncertainty propagation.  arXiv preprint arXiv:2303.12246, 2023.
  * [52] Linfang Zheng, Chen Wang, Yinghan Sun, Esha Dasgupta, Hua Chen, Ales Leonardis, Wei Zhang, and Hyung Jin Chang.  Hs-pose: Hybrid scope feature extraction for category-level object pose estimation.  arXiv preprint arXiv:2303.15743, 2023.

## Supplementary

LMO | Random | KGN
---|---|---
Object | BBox | Sphere
ape | 53.7 | 55.2 | 65
can | 80.8 | 83.2 | 96.4
cat | 44.1 | 47.3 | 58
driller | 70.6 | 73.4 | 82.1
duck | 42.1 | 48.2 | 65.6
eggbox | 70.1 | 74.3 | 82.2
glue | 66.2 | 67.3 | 75.1
holepuncher | 68.5 | 72.5 | 81.2
average | 62 | 65.2 | 75.6
Table 7:  ADD(S) of RCVPose [49] on LMO using keypoints selected randomly
(BBox, Sphere) vs. with KeyGNet (KGN). The randomly selected keypoints use
RANSAC to minimize the Wasserstein distance measure.

##  Appendix A Overview

We document here some additional results and one more ablation study. The per
category KeyGNet results of LMO and YCB-V datasets evaluated by ADD(S) and
ADD(S) AUC in both SISO and MIMO modes on all three methods tested are shown
in Table. 8, Table. 9, and Table. 10. KeyGNet keypoints improved the
perfomance for all objects, in all datasets, among all three methods tested.
The BOP $AR$ (Average Recall) of Visible Surface Discrepancy ($AR_{VSD}$),
Maximum Symmetry-Aware Surface Distance ($AR_{MSSD}$), Maximum Symmetry-Aware
Projection Distance ($AR_{MSPD}$), and the overall average are reported in
Table. 11. All these metrics are improved in all six core datasets when the
KeyGNet keypoints are used. Last but not least, the SISO-MIMO gap is reduced
by using KeyGNet keypoints for all objects in YCB-V, as shown in Table. 12.

##  Appendix B Classical Distance Measure vs. KeyGNet

Instead of training a network, keypoints can be selected by measuring the
Wasserstein distance directlyon a collection of sets of keypoints. We conduct
a test by comparing the trained KeyGNet with a classical RANSAC [12] style
algorithm. The collection of initial keypoint sets are selected either
relatively randomly in a region centered at the bounding box’s corners, or
completely randomly in a sphere within the object reference frame of the CAD
model. The Wasserstein distances and the dispersion scores are then calculated
for each set of the keypoints. The algorithm repeats for $N$ times and the
keypoints with the minimum Wasserstein distances and the maximum dispersion
scores are selected.

We test the keypoints using RCVPose on LMO and compare the ADD(S) metric with
KeyGNet. The results are shown in Table. 7. It can be seen that he keypoints
selected with an initial location of bounding box corners are $3.2\%$ on
average worse than those selected with completely random initial locations.
This is possibly due to the restrictions caused by the initial input locations
of BBox corners. The learned KeyGNet keypoints have the best performance for
all objects in LMO, boosting the ADD(S) by $13.6\%$ and $10.4\%$ compared to
those randomly selected.

SISO MIMO LMO PVNet PVN3D RCVPose PVNet PVN3D RCVPose object FPS KGN FPS KGN
BBox KGN FPS KGN FPS KGN BBox KGN ape 15.8 21.2 33.9 40.2 61.3 65.4 8.2 20.9
25.7 39.9 57.1 65 can 63.3 74.2 88.6 93.7 93 96.4 51.1 74.2 77.4 93.7 90.1
96.4 cat 16.7 22.3 39.1 49.2 51.2 58.2 9.6 22.3 31.5 48.9 45.5 58 driller 65.7
76.6 78.4 88.3 78.8 81.7 57.5 76.6 68.1 88.3 77.5 82.1 duck 25.2 30.2 41.9
47.6 53.4 65.2 14.1 29.9 32.2 47.2 46.2 65.6 eggbox 50.2 57.8 80.9 85.2 82.3
82 38 57.8 70.7 85.2 80.2 82.2 glue 49.6 59.7 68.1 77.2 72.9 74.9 42.9 59.7
58.3 76.8 66.6 75.1 holepuncher 39.7 42.3 74.7 82.3 75.8 81.2 32.4 42.3 67.5
82.3 73.7 81.2 average 40.8 48 (+7.2) 63.2 70.5 (+7.3) 71.1 75.5 (+4.4) 31.7
47.9 (+16.2) 53.9 70.3 64.8 75.6 (+10.8)

Table 8: LMO Results: The ADD(S) AUC comparison of three keypoint voting-based methods (PVNet, PVN3D, RCVPose) using initially defined keypoints (FPS, BBox) and optimized keypoints (KGN) generated by KeyGNet. YCB | PVNet | PVN3D | RCVPose
---|---|---|---
object | FPS | KGN | FPS | KGN | BBox | KGN
002_master_chef_can | 54.6 | 69.5 | 75.3 | 84.8 | 92.1 | 95.1
003_cracker_box | 66.2 | 78.8 | 87.0 | 96.5 | 94.3 | 96.4
004_sugar_box | 66.3 | 76.1 | 90.9 | 96.5 | 94.2 | 97.7
005_tomato_soupcan | 62.2 | 74.7 | 81.5 | 92.8 | 91.5 | 96.7
006_mustard_bottle | 67.6 | 82.4 | 89.3 | 97.7 | 94.2 | 96.9
007Auna_fish_can | 64.9 | 77.0 | 87.0 | 95.7 | 94.2 | 96.4
008_pudding_box | 76.6 | 85.2 | 90.3 | 97.5 | 95.4 | 97.0
009_gelatin_box | 71.4 | 88.8 | 90.6 | 97.5 | 92.6 | 97.4
010_potted_meat_can | 69.5 | 84.1 | 82.5 | 93.7 | 88.1 | 93.9
011_banana | 67.9 | 76.8 | 90.9 | 97.2 | 94.9 | 97.5
019_pitcher_base | 67.8 | 76.9 | 88.1 | 97.5 | 93.1 | 97.9
021_bleach_cleanser | 70.2 | 74.9 | 92.2 | 96.6 | 95.4 | 98.4
024_bowl∗ | 66.9 | 78.3 | 87.3 | 95.6 | 91.0 | 97.3
025_mug | 71.5 | 79.8 | 91.8 | 96.5 | 94.2 | 96.9
035_power_drill | 67.6 | 81.7 | 89.2 | 96.9 | 93.7 | 97.5
036_wood_block∗ | 57.4 | 85.2 | 82.9 | 92.8 | 89.7 | 93.3
037_scissors | 64.2 | 80.4 | 83.2 | 91.5 | 92.3 | 95.9
040_large_marker | 65.5 | 81.9 | 84.2 | 88.0 | 86.5 | 95.6
051_large_clamp∗ | 55.7 | 66.5 | 84.4 | 88.5 | 90.5 | 97.6
052_extra_large_clamp∗ | 52.6 | 61.4 | 77.8 | 94.1 | 92.5 | 96.0
061_loam_brick∗ | 60.9 | 79.6 | 89.4 | 97.8 | 92.2 | 97.2
average | 65.1 | 78.1 | 86.5 | 94.6 | 92.5 | 96.6
Table 9: YCB-V Results. The ADD(S) AUC comparison of three keypoint voting-based methods (PVNet, PVN3D, RCVPose) in MIMO mode using initially defined keypoints (FPS, BBox) and optimized keypoints (KGN) generated by KeyGNet.  YCB | PVNet | PVN3D | RCVPose
---|---|---|---
object | FPS | KGN | FPS | KGN | BBox | KGN
002_master_chef_can | 60.2 | 70 | 79.3 | 85.2 | 94.7 | 96.2
003_cracker_box | 70.7 | 79.4 | 91.5 | 96.7 | 96.4 | 97.4
004_sugar_box | 73.2 | 76.6 | 96.9 | 97.3 | 97.6 | 98.7
005_tomato_soupcan | 67.7 | 75.1 | 89.0 | 93.2 | 95.4 | 97.6
006_mustard_bottle | 76.5 | 83 | 97.9 | 98.2 | 97.7 | 98.2
007Auna_fish_can | 71.3 | 77.2 | 90.7 | 96.3 | 96.7 | 97.4
008_pudding_box | 80.1 | 85.4 | 97.1 | 98.1 | 97.4 | 97.9
009_gelatin_box | 81.2 | 89.1 | 98.3 | 98.3 | 97.9 | 98.3
010_potted_meat_can | 76.9 | 84.6 | 87.9 | 94.2 | 92.6 | 95.3
011_banana | 73.2 | 77.6 | 96.0 | 97.6 | 97.2 | 98.4
019_pitcher_base | 74.3 | 77.4 | 96.9 | 98.0 | 96.7 | 99.2
021_bleach_cleanser | 70.9 | 75.4 | 95.9 | 97.3 | 98.4 | 99.3
024_bowl∗ | 69.7 | 79 | 92.8 | 96.4 | 95.3 | 98.2
025_mug | 75.3 | 80.6 | 96.0 | 97.1 | 97.1 | 98
035_power_drill | 74.3 | 82 | 95.7 | 97.2 | 96.9 | 98.3
036_wood_block∗ | 70.2 | 85.8 | 91.1 | 93.2 | 90.7 | 94.3
037_scissors | 66.4 | 81 | 87.2 | 92.1 | 94.9 | 97.2
040_large_marker | 67.3 | 82.4 | 91.6 | 94.3 | 93.2 | 96.3
051_large_clamp∗ | 66.2 | 72.2 | 95.6 | 96.2 | 96.2 | 98.3
052_extra_large_clamp∗ | 63.4 | 66.9 | 90.5 | 94.7 | 95.1 | 97.2
061_loam_brick∗ | 70.2 | 80.3 | 98.2 | 98.4 | 96.6 | 98.2
average | 73.4 | 79.1 | 92.3 | 95.7 | 95.9 | 97.6
Table 10: YCB-V Results. The ADD(S) AUC comparison of three keypoint voting-based methods (PVNet, PVN3D, RCVPose) in SISO mode using initially defined keypoints (FPS, BBox) and optimized keypoints (KGN) generated by KeyGNet.  Metric | Dataset | PVNet | PVN3D | RCVPose
---|---|---|---|---
FPS | KGN | FPS | KGN | BBox | KGN
$AR_{VSD}$ | LMO | 48.2 | 52.4 | 70.6 | 72.7 | 72.5 | 76.9
YCB-V | 78.2 | 82.7 | 76.9 | 83.6 | 84.4 | 88.3
TLESS | 65.7 | 67.2 | 68.3 | 72.2 | 70.8 | 75.3
TUDL | 90.5 | 92.5 | 87.3 | 91.9 | 98.0 | 99.4
IC-BIN | 70.6 | 72.6 | 67.2 | 73.9 | 74.1 | 80.4
ITODO | 42.4 | 44.7 | 43.2 | 47.7 | 50.7 | 50.8
HB | 77.1 | 78.7 | 78.4 | 84.4 | 82.5 | 85.9
$AR_{MSSD}$ | LMO | 66.4 | 70.2 | 62.5 | 69.3 | 73.4 | 73.6
YCB-V | 77.3 | 79.8 | 79.9 | 82.9 | 86.3 | 89.6
TLESS | 70.2 | 72.0 | 64.3 | 66.5 | 72.3 | 73.3
TUDL | 90.6 | 91.8 | 91.9 | 93.1 | 97.5 | 98.5
IC-BIN | 69.0 | 71.5 | 72.1 | 78.4 | 73.9 | 73.8
ITODO | 51.2 | 51.3 | 52.6 | 58.3 | 57.2 | 64.2
HB | 85.5 | 90.2 | 85.3 | 88.2 | 89.0 | 90.4
$AR_{MSPD}$ | LMO | 69.2 | 71.7 | 66.0 | 67.7 | 74.9 | 79.7
YCB-V | 76.7 | 77.0 | 76.5 | 81.2 | 84.9 | 88.1
TLESS | 67.3 | 71.1 | 69.3 | 72.1 | 71.5 | 77.7
TUDL | 93.7 | 94.9 | 91.2 | 95.1 | 97.8 | 98.6
IC-BIN | 72.3 | 75.4 | 71.8 | 77.6 | 73.9 | 75.6
ITODO | 50.3 | 52.3 | 52.7 | 55.6 | 56.2 | 59.4
HB | 84.8 | 86.0 | 84.7 | 90.2 | 90.5 | 93.0
$AR_{average}$ | LMO | 61.3 | 64.8 | 66.4 | 69.9 | 73.6 | 76.7
YCB-V | 77.4 | 79.8 | 77.8 | 82.6 | 85.2 | 88.7
TLESS | 67.7 | 70.1 | 67.3 | 70.3 | 71.5 | 75.4
TUDL | 91.6 | 93.1 | 90.1 | 93.4 | 97.8 | 98.8
IC-BIN | 70.6 | 73.2 | 70.4 | 76.6 | 74.0 | 76.6
ITODO | 48.0 | 49.4 | 49.5 | 53.9 | 54.7 | 58.1
HB | 82.5 | 85.0 | 82.8 | 87.6 | 87.3 | 89.7
Overall Average | 71.3 | 73.6 | 72.0 | 76.3 | 77 | 80.6
Table 11: BOP Core Dataset Results. The Average Recall ($AR$) of Visible Surface Discrepancy ($AR_{VSD}$), Maximum Symmetry-Aware Surface Distance ($AR_{MSSD}$), Maximum Symmetry-Aware Projection Distance ($AR_{MSPD}$), and the overall average for all six BOP core datasets are reported for three methods by using keypoints selected by the original method (FPS/BBox) and KeyGNet (KGN).  YCB-V | PVNet | PVN3D | RCVPose
---|---|---|---
object | FPS | KGN | FPS | KGN | BBox | KGN
002_master_chef_can | -5.6 | -0.5 | -4.0 | -0.4 | -2.6 | -1.1
003_cracker_box | -4.5 | -0.6 | -4.5 | -0.2 | -2.1 | -1.0
004_sugar_box | -6.9 | -0.5 | -6.0 | -0.8 | -3.4 | -1.0
005_tomato_soupcan | -5.5 | -0.4 | -7.5 | -0.4 | -3.9 | -0.9
006_mustard_bottle | -8.9 | -0.6 | -8.6 | -0.5 | -3.5 | -1.3
007Auna_fish_can | -6.4 | -0.2 | -3.7 | -0.6 | -2.5 | -1.0
008_pudding_box | -3.5 | -0.2 | -6.8 | -0.6 | -2.0 | -0.9
009_gelatin_box | -9.8 | -0.3 | -7.7 | -0.8 | -5.3 | -0.9
010_potted_meat_can | -7.4 | -0.5 | -5.4 | -0.5 | -4.5 | -1.4
011_banana | -5.3 | -0.8 | -5.1 | -0.4 | -2.3 | -0.9
019_pitcher_base | -6.5 | -0.5 | -8.8 | -0.5 | -3.6 | -1.3
021_bleach_cleanser | -0.7 | -0.5 | -3.7 | -0.7 | -3.0 | -0.9
024_bowl∗ | -2.8 | -0.7 | -5.5 | -0.8 | -4.3 | -0.9
025_mug | -3.8 | -0.8 | -4.2 | -0.6 | -2.9 | -1.1
035_power_drill | -6.7 | -0.3 | -6.5 | -0.3 | -3.2 | -0.8
036_wood_block∗ | -12.8 | -0.6 | -8.2 | -0.4 | -1.0 | -1.0
037_scissors | -2.2 | -0.6 | -4.0 | -0.6 | -2.6 | -1.3
040_large_marker | -1.8 | -0.5 | -7.4 | -6.3 | -6.7 | -0.7
051_large_clamp∗ | -10.5 | -5.7 | -11.2 | -7.7 | -5.7 | -0.7
052_extra_large_clamp∗ | -10.8 | -5.5 | -12.7 | -0.6 | -2.6 | -1.2
061_loam_brick∗ | -9.3 | -0.7 | -8.8 | -0.6 | -4.4 | -1.0
average | -6.3 | -1.0 | -6.7 | -1.2 | -3.4 | -1.0
Table 12: SISO-MIMO performance gap on YCB-V. The change in ADD(S) when
converting from SISO to MIMO, for keypoints sampled heuristically (FPS or
BBox) and KeyGNet (KGN). There is a relatively small change in ADD(S) AUC when
the PE network is trained simultaneously on multiple objects using KGN
keypoints.


# Estimation of Soiling Losses in Unlabeled PV Data ††thanks: This material is
based on work supported by the U.S. Department of Energy’s Office of Energy
Efficiency and Renewable Energy (EERE) under the Solar Energy Technologies
Award Number 38529.

Bennet Meyers1,2 1 SLAC National Accelerator Laboratory, Menlo Park, CA,
94025, USA
2 Stanford University, Stanford, CA, 94305, USA

###### Abstract

We provide a methodology for estimating the losses due to soiling for
photovoltaic (PV) systems. We focus this work on estimating the losses from
historical power production data that are unlabeled, i.e. power measurements
with time stamps, but no other information such as site configuration or
meteorological data. We present a validation of this approach on a small fleet
of typical rooftop PV systems. The proposed method differs from prior work in
that the construction of a performance index is not required to analyze
soiling loss. This approach is appropriate for analyzing the soiling losses in
field production data from fleets of distributed rooftop systems and is highly
automatic, allowing for scaling to large fleets of heterogeneous PV systems.

###### Index Terms:

photovoltaic systems, solar energy, distributed power generation, energy
informatics, machine learning, statistical learning, unsupervised learning,
soiling, unlabeled data

##  I Introduction

Soiling can cause significant energy yield reduction in photovoltaic (PV)
systems, as high as $-1\%$/day, but the effects are quite variable and
impacted by many factors from system geometry to local climate conditions to
nearby industry and agriculture [1]. Quantifying the losses due to soiling is
important for understanding and mitigating this effect, which in turn improves
the overall reliability and dependability of PV as an energy generation
source. This importance is seen in the dedicated subarea for soiling at this
conference, as well as the strong emphasis on soiling at other events such as
the yearly PV Reliability Workshop hosted by NREL. Largely absent from this
conversation, however, is quantification of the impacts of soiling in large
fleets of heterogenous, distributed PV systems, which comprised over 40% of
the installed capacity in 2020 [2]. The reason for this is a technical one: it
is very difficult to analyze soiling trends without a reliable reference, and
the unlabeled nature of distributed PV data make generating this reference
difficult or impossible.

In this work, we present a methodology, based on recent research on machine
learning for signal processing, that extracts estimates of system soiling
losses from _unlabeled_ production data. This work eliminates the requirement
of constructing a performance index to analyze soiling loss and is highly
automated, thus enabling the large-scale analysis of fleets of thousands of
heterogenous systems. The proposed approach is based on an implementation of
the _signal decomposition_ (SD) framework [3].

Our approach takes unlabeled PV power generation measurements as an input and
returns an estimate of the soiling loss over time, given as a percent loss
relative to the unsoiled performance. This trend may be used to calculate
secondary statistics such as the total energy loss or seasonal loss patterns.
We validate this method on synthetic data, labeled data from a soiling test
site, and on representative unlabled data. The algorithm is available as a
module in the Solar Data Tools package [4, 5]. This approach is uniquely
suited to the analysis of fleet-scale PV systems, where it can be difficult or
impossible to get suitable reference data for normalization.

##  II Related work and contributions

We are not the first to propose a method for estimating soiling losses from PV
production data [6, 7, 8, 9]. As described in [8], it has been determined that
a combined model of degradation, soiling, and seasonal bias outperforms
estimates of each loss term separately.

In this paper, we build upon previous work in a few ways. First, we propose a
clear unified signal model (implemented as an SD problem) to describe the
underlying components, rather than invoking iterative heuristics. Second, the
use of the extensible SD framework allows for the expression of component
classes that are designed specifically for this application Third, our
approach estimates soiling losses from unlabeled system power generation time
series data, without the need to construct a performance index (PI). As
discussed in the introduction, this unlocks the analysis of around 40% of the
installed PV capacity in the United States. An example of a PI constructed
from data labels and an unlabeled daily energy signal are shown in figure 1.
The proposed method allows for the analysis of _both_ unlabeled and labeled
data. Labeled data will provide more accurate results, when available, but
reasonable estimates may be obtained when such labels are unavailable.

$\displaystyle{0.8}$$\displaystyle{0.9}$$\displaystyle{1.0}$fraction
ofexpected energy [1]performance
index$\displaystyle{0}$$\displaystyle{100}$$\displaystyle{200}$$\displaystyle{300}$$\displaystyle{400}$$\displaystyle{500}$$\displaystyle{600}$day$\displaystyle{5}$$\displaystyle{10}$daily
energy [kWh]daily energy

Figure 1: A comparison of labeled data (top) and unlabeled data (bottom) from
a single PV system in a desert environment.

##  III Methods

We construct an SD problem [3] that models the decomposition of measured daily
PV system energy or normalized energy (i.e., a performance index, “PI”) into a
number of components, one of which represents the soiling loss in the system.
This approach can be thought of as an _unsupervised_ machine learning (ML)
method for finding structure in time series data, similar to model-based
clustering methods like Gaussian mixture models [10, §14.3.7]. Unlike
supervised ML, there is no “training” of the method; we simply design the
mathematical optimization problem, input the data for analysis, and receive
the soiling estimate. In this section, we describe the data preparation, SD
formulation, and validation procedure. Finally, we briefly describe a
procedure for utilizing the estimate of system soiling to “correct” the
measured power data.

###  III-A Data preparation

The proposed method of signal decomposition operates on a discrete daily time
series representing raw or normalized daily system energy production, which we
refer to as “unlabeled” and “labeled” data respectively. The purpose of
generating a performance index is to remove known sources or variation in
data, particularly due to available irradiance and operating temperature, and
is a typical analytical approach for PV performance analysis, but requires
additional knowledge about the system and its operating environment beyond
real power production.

The raw data may be any measurements of PV system power or energy production
indexed in time. This may be 1-minute measurements of instantaneous power,
15-minute interval averaged power, or daily energy production. If starting
from high-frequency power measurements, one simply integrates to get the daily
energy production. If one wishes to construct a PI, you then normalize by
expected daily energy at this time.

When starting from sub-daily measurements, any prefiltering step may be
applied, and rejected days can have their values replaced with NaN values
(which we represent as $?$ in our notation). The SD framework optimally
handles missing data points, so there is no need to use corrupted or
untrustworthy data nor replace such data with interpolated values.

In this paper, we validate on both synthetic and real data. As described in
§III-C, the synthetic data is already a daily time-series (a normalized PI),
so no preprocessing is required. For the real data, which has a 5-minute
measurement interval, we use the data cleaning and filtering tools provided in
Solar Data Tools [4, 5], and replace days that do not pass the quality check
with $?$ values. After obtaining a representation of daily energy production
(possibly with missing values), the signal is scaled so the 95${}^{\text{th}}$
percentile is equal to 1. This is our input to the signal decomposition
problem (SD problem). We do not construct a performance index on the real data
and instead analyze the raw energy signal.

###  III-B SD problem formulation

Utilizing the notation of signal decomposition defined in [3], we say that our
data for the SD problem is a signal $y\in\left({\mbox{\bf
R}}\cup\\{?\\}\right)^{T\times p}$ with length $T$ equal to the number of days
in the data set and measurement dimension $p=1$. In this case, because the
measurement dimension is equal to 1, $y$ may also be thought of as a column
vector of length $T$. We model the signal $y$ as the composition of $K=4$
components, $x^{1}$ to $x^{4}$, the sum of which must be equal to the signal
$y$ at the entries that do not contain missing values, or in other words,

$y_{t}=x_{t}^{1}+x_{t}^{2}+x_{t}^{3}+x_{t}^{4},\mbox{ for }t\in\mathcal{K},$

where $\mathcal{K}$ is the set of time indices that do not contain missing
values (the “known” set). The four components are defined in the SD model by
their cost functions $\phi_{k}(x^{k})$ for $k=1,\ldots,4$. (We drop the
superscript $k$ on $x$ to keep the notation lighter when not distinguishing
between particular components.) As we will see, the last component, $x^{4}$,
will represent the soiling signal which we wish to estimate from the data. The
other three components represent other processes which impact the energy
production of the system.

#### Component definitions

The first component represents the residual of the model, and it is taken to
be the quantile cost function [11, 12],

$\phi_{1}(x)=\mathbf{quant}_{\tau}(x)=\sum_{t=1}^{T}(1/2)\left\lvert
x_{t}\right\rvert+(\tau-1/2)x_{t},$

where $\tau\in(0,1)$ is a parameter. When $\tau<0.5$, positive residuals are
prefered to negative residuals, and vice versa when $\tau>0.5$. When working
with raw energy data, we set $\tau=0.85$, which strongly prefers negative
residuals, accounting for the fact that we have not normalized for weather
effects, and clouds tend to reduce rather than increase the system energy
product. When operating on normalized, PI data (such as the synthetic data set
in this paper), we set $\tau=0.5$ since we expect the deviations from the
expected output to be symmetric, or at the very least more symmetric then when
no normalization is performed.

The second component is a seasonal term, which is smooth and periodic each
year,

$\phi_{2}(x)=\left\\{\begin{array}[]{ll}\lambda_{2}\|D_{2}x\|_{2}^{2}&x_{t}=x_{t+Y}\mbox{,
for }t=1,\ldots T-Y\\\ \infty&\mbox{otherwise,}\end{array}\right.$

where $D_{2}\in{\mbox{\bf R}}^{(T-2)\times T}$ is the second-order discrete
difference operator. (See, for example, [13, §6.4] for information on
difference matrices.) $\lambda_{2}$ is a weighting parameter, and $Y=365$ is
the period of the component. The normalization of the energy signal affects
the expected amplitude of this component. That is, we would expect raw energy
data to have a larger seasonal component than normalized data. The component
definition presented here covers both cases well and is not sensitive to the
amplitude of the component.

The third component represents the bulk, long-term degradation rate, and is
given by

$\phi_{3}(x)=\left\\{\begin{array}[]{ll}0&x_{0}=0\mbox{ and }x_{t}=mt+b\mbox{
for }t=1,\ldots,T\\\ \infty&\mbox{otherwise,}\end{array}\right.$

for some values of $m$ and $b$. This just constraints the component to be
linear with respect to time with an initial value equal to zero. We have
chosen a linear degradation model due to its popularity in the literature, but
we note that other trend models could be employed within the SD framework,
e.g., a smooth, monotonically decreasing signal. Note that the third component
does not include a weight parameter, as the penalty function only takes on
values of zero or infinity.

The fourth and final component represents what we are interested in measuring,
the soiling losses in the system. This cost is defined as,

$\phi_{4}(x)=\left\\{\begin{array}[]{ll}\ell_{4a}(x)+\ell_{4b}(x)+\ell_{4c}(x)&x\preceq
0\\\ \infty&\mbox{otherwise,}\end{array}\right.$

where the summed functions are

$\displaystyle\ell_{4a}$ $\displaystyle=$
$\displaystyle\lambda_{4a}\|D_{2}x\|_{1}$ $\displaystyle\ell_{4b}$
$\displaystyle=$ $\displaystyle\lambda_{4b}\sum_{t=0}^{T}(-x_{t})$
$\displaystyle\ell_{4c}$ $\displaystyle=$
$\displaystyle\lambda_{4c}\mathbf{quant}_{\tau}(D_{1}x)$

with parameters $\lambda_{4a}$, $\lambda_{4b}$, and $\lambda_{4c}$. $D_{2}$ is
again the second-order discrete difference operator, and $D_{1}$, similarly,
is the first-order difference. The quantile cost parameter $\tau$ is taken to
be $0.9$ here. This cost is a composite of simpler functions, which combine to
select for signals with the following characteristics:

  * •

non-positive (soiling can only reduce the system power)

  * •

sparse in second-differences (i.e., piecewise linear)

  * •

with values “close” to zero (is sum-absolute sense)

  * •

a preference for more values with a negative slope than a positive one (i.e.,
a tendency towards slow degradation and quick recovery).

Component cost $\phi_{4}$ demonstrates the _extensibility_ of the SD
framework. We are able to build up a complex cost function from smaller units
and design it in a way to capture domain knowledge about the component.

#### SD parameters

The SD problem formulating includes four parameters, $\lambda_{2}$,
$\lambda_{4a}$, $\lambda_{4b}$, and $\lambda_{4c}$. These parameters are
tunable, and different values can greatly effect the characteristics and
quality of the resulting decomposition. A deep discussion on the role of
parameters in SD problems can be found in [3, §2.6]. While a method is
provided in [3, §2.7] for selecting optimal parameter values, we find that in
this context it makes more sense to rely on the practical experience of the
analyst. In other words, we have found values that work well in many cases,
and a small amount of hand-tuning is accepted in other cases. A description of
these parameters and their default values are given in table I.

TABLE I: SD problem parameters param. | value | description
---|---|---
$\lambda_{2}$ | $5\text{\times}{10}^{2}$ | stiffness of seasonal baseline
$\lambda_{4a}$ | $2$ | effects number of soiling component breakpoints
$\lambda_{4b}$ | $3\text{\times}{10}^{-2}$ | penalizes large values of soiling component
$\lambda_{4c}$ | $2\text{\times}{10}^{-1}$ | encourages asymmetric rates in soiling component

#### Solution method

The SD problem is convex (inequality-constrained quadratic program [14, §4.4])
and of modest size (around 2.1k variables for 3-year data set to around 15k
variables for a 10-year data set), so we simply use CVXPY [15, 16] and the
commercial Mosek solver [17], which is sufficient for research purposes. An
implementation of the algorithm described in [3] would allow for the removal
of the dependence on Mosek and is an area for future work.

###  III-C Validation

The most desirable method of validation for an unsupervised machine learning
algorithm such as the methods described in this paper would be access to real
PV system production data that has been hand-labeled with soiling trends.
Because this is difficult to obtain or generate, we take a multi-modal
approach to validation in this paper, with three different approaches to
validation, described below, ordered by how well labeled the data source is.

#### Synthetic data

We follow the methods published in [8] to evaluate the performance of the
algorithm on synthetic data that represents normalized energy productions.
This approach generates random realizations of PI signals, with components
drawn from pre-defined statistical models. The synthetic data model includes a
‘seasonal’ term which represents the seasonal variation in performance.
Normalization with a performance index is typically expected to lower this
seasonal variation, but it does not fully remove it. The noise term in the
model is Gaussian white noise, representing the assumption that a PI signal
normalized for weather phenomenon. We therefore set $\tau=0.5$ in $\phi_{1}$
of the SD formulation to reflect the expectation of symmetric residuals.

Because the synthetic model explicitly models the system soiling losses, we
are able to directly assess the ability of the SD soiling algorithm to
estimate the hidden soiling signal. We select _mean-absolute error_ (MAE) as a
summary error metric for comparing the known synthetic soiling loss to the SD
estimate, which is preferred over _root-mean-square error_ when the errors are
not expected to be normally distributed [18, §6.1.2]. Because analysts often
want to be able to estimate soiling _rates_ on a PV system, in addition to
understanding the total energy loss, we calculate the MAE on both the soiling
loss component and the first-order difference [19] of the loss, which we call
the soiling rate. Finally, we note a small error in estimate of a cleaning
event (i.e., one day before or after the true event) is of small consequence
to the analyst but will result in very large error values, especially for the
analysis of soiling rates. Therefore, we introduce a third summary statistic
which is the MAE of _filtered_ soiling rate, which simply selects for time
periods when both the synthetic soiling component and the SD estimate agree
that the instantanous soiling rate is negative, i.e., neither component is
currently in a cleaning event. This final metric provides useful insight into
the ability of the algorithm to accurately estimate the rate of soiling loss
between cleaning events.

#### Labeled production data

Labeled soiling data was presented in [20] and analyzed for soiling trends. We
leverage these published results to validate the SD soiling algorithm on the
_unlabeled_ power production data. For this power, we ignore the reference
system, treating the test system as an unlabeled data source, and estimate the
soiling losses of the test system using the SD formulation. Then we compare
results of the proposed method to the results from [20]. We calculate the same
three error metrics for this data set, as described previously for the
synthetic data.

#### Unlabeled production data

We demonstrate the application of the SD soiling method on a selection of 50
PV systems, with only access to measured real power. There is no known soiling
trend to compare to in this case. Instead, this represents what we see as a
typical use case, and we show how outlier sites may be identified.

###  III-D Soiling correction for downstream analysis

We briefly note that the estimate of soiling loss may be used to “correct” for
soiling for other analysis. For example, we note in two other papers submitted
to this conference [21, 22], that it is beneficial to account for soiling in
production PV data prior to analyzing shade losses. We provide a simple
procedure for doing this correction here.

The soiling loss component is a daily signal representing the fractional loss
in daily energy from soiling, typically between 0 and 1. Therefore, the
expected power output of the system in the absence of soiling would be the
measured power divided by the instantaneous soiling loss. The Solar Data Tools
implementation of the methods described in this paper includes a feature for
automatically performing this correction.

##  IV Results

###  IV-A Synthetic data

Following the scenario generation procedure defined in [8, §II-D], we define
six generative model configurations, with different levels of soiling,
seasonality, and noise. These scenarios are briefly described in II. Two
characteristic examples of the synthetic PI signals are shown in figure 2, and
the SD estimates of the soiling trends are compared to the true values in
figure 3. The error metric for these two examples is given in table III. The
soiling rate were on average around -0.001 for the first example and -0.0005
for the second example.

$\displaystyle{0.8}$$\displaystyle{0.9}$$\displaystyle{1.0}$synthetic PI
example
120102011201220132014201520162017201820192020$\displaystyle{0.85}$$\displaystyle{0.90}$$\displaystyle{0.95}$$\displaystyle{1.00}$synthetic
PI example 2

Figure 2: Two typical synthetic soiling signals generated by NREL software.
The top signal is drawn from scenario 1 and the bottom from scenario 2.

$\displaystyle{0.75}$$\displaystyle{0.80}$$\displaystyle{0.85}$$\displaystyle{0.90}$$\displaystyle{0.95}$$\displaystyle{1.00}$soiling
component, synthetic example 1syntheticSD
estimate20102011201220132014201520162017201820192020$\displaystyle{0.90}$$\displaystyle{0.95}$$\displaystyle{1.00}$soiling
component, synthetic example 2syntheticSD estimate

Figure 3: Comparison of the actual and estimated soiling components in the two synthetic examples shown in figure 2. TABLE II: Synthetic data scenarios number | name | description
---|---|---
1 | normal | soiling rates ($sr$) uniform in $[0,0.003]$
2 | M soil, H season |  $sr\in[0,0.001]$, double seasonal amplitude
3 | M Soil, H noise |  $sr\in[0,0.001]$, double noise amplitude
4 | seasonal cleaning |  $sr\in[0,0.005]$, cleaned seasonally
5 | M soil | $sr\in[0,0.005]$
6 | L soil | $sr\in[0,0.001]$
TABLE III: Error metrics for two synthetic soiling examples | loss MAE | rate MAE | filtered rate MAE
---|---|---|---
Ex. 1 | 0.008698 | 0.002257 | 0.000379
Ex. 2 | 0.005366 | 0.000919 | 0.000202

We sample the 6 scenarios 10 times each, and we solve the associated SD
problem for each of the 60 realizations. Finally, we calculate the three error
metrics for each realization. The distribution of loss MAE is given in figure
4; the distribution of rate MAE is given in figure 5, and the distribution of
the filtered rate MAE is given in figure 6.

$\displaystyle{0.002}$$\displaystyle{0.004}$$\displaystyle{0.006}$loss
MAE$\displaystyle{0}$$\displaystyle{2}$$\displaystyle{4}$$\displaystyle{6}$$\displaystyle{8}$$\displaystyle{10}$$\displaystyle{12}$CountnamenormalM
soil, H seasonM soil, H noiseseasonal cleaningM soilL soil

Figure 4: Distribution of loss MAE for the 60 realizations, labeled by
scenario.

$\displaystyle{0.0000}$$\displaystyle{0.0005}$$\displaystyle{0.0010}$$\displaystyle{0.0015}$$\displaystyle{0.0020}$$\displaystyle{0.0025}$rate
MAE$\displaystyle{0.0}$$\displaystyle{2.5}$$\displaystyle{5.0}$$\displaystyle{7.5}$$\displaystyle{10.0}$$\displaystyle{12.5}$$\displaystyle{15.0}$$\displaystyle{17.5}$$\displaystyle{20.0}$CountnamenormalM
soil, H seasonM soil, H noiseseasonal cleaningM soilL soil

Figure 5: Distribution of rate MAE for the 60 realizations, labeled by
scenario.

$\displaystyle{0.0001}$$\displaystyle{0.0002}$$\displaystyle{0.0003}$$\displaystyle{0.0004}$filtered
rate
MAE$\displaystyle{0}$$\displaystyle{2}$$\displaystyle{4}$$\displaystyle{6}$$\displaystyle{8}$$\displaystyle{10}$CountnamenormalM
soil, H seasonM soil, H noiseseasonal cleaningM soilL soil

Figure 6: Distribution of the filtered rate MAE for the 60 realizations,
labeled by scenario. This metric accurately captures how closely the SD method
estimated the loss rates during soiling periods.

###  IV-B Labeled data

The labeled performance index and corresponding unlabeled energy signal was
previously shown in figure 1. In figure 7, these signals are overlaid with the
“denoised” SD signal estimate, i.e., the sum of the estimated components
excluding the first residual term. Note how the SD model for the PI assumes
symmetric residuals ($\tau=0.5$) while the model for the energy signal assumes
highly asymmetric residuals ($\tau=0.85$). Figure 8 shows the comparison
between the soiling loss components estimated from the PI and from the
unlabeled energy signal. Taking the loss component derived from the PI signal
as groundtruth, we then calculate the three error metrics for the energy-
derived soiling estimate. The loss MAE is 0.042558; the rate MAE is 0.004734,
and the filtered rate MAE is 0.001756.

We find that the analysis of the unlabeled energy produces an estimate of
soiling losses that agrees well with the PI-derived estimate. We observe that
the unlabled analysis does worst around days 125–200, which was particularly
rainy and cloudy. This lack of clear sky baseline during this period is seen
to negatively impact the soiling estimate. However, with additional years of
data, this may be improved due to the seasonal structure in the second
component.

$\displaystyle{0.8}$$\displaystyle{0.9}$$\displaystyle{1.0}$fraction
ofexpected energy [1]performance indexPISD
model$\displaystyle{0}$$\displaystyle{100}$$\displaystyle{200}$$\displaystyle{300}$$\displaystyle{400}$$\displaystyle{500}$$\displaystyle{600}$day$\displaystyle{0.5}$$\displaystyle{1.0}$daily
energy [kWh]daily energyenergySD model

Figure 7: PI and energy signals for the labeled soiling test system, with the
denoised SD estimates overlaid in orange.

$\displaystyle{0}$$\displaystyle{100}$$\displaystyle{200}$$\displaystyle{300}$$\displaystyle{400}$$\displaystyle{500}$$\displaystyle{600}$day$\displaystyle{-0.2}$$\displaystyle{-0.1}$$\displaystyle{0.0}$fractionalsoiling
loss [1]PI derivedenergy derived

Figure 8: Comparison of the estimated soiling loss component from the PI
signal and the unlabeled energy signal.

###  IV-C Unlabeled data

In [21], we present a preliminary shade loss analysis of an unlabeled rooftop
PV data set. As described in that manuscript, the shade analysis depends on
first estimating and correcting the soiling losses in the signal. A view of
the soiling signal decomposition for one system in this data set is shown in
figure 9, and the soiling component in isolation is shown in figure 10.

$\displaystyle{0}$$\displaystyle{250}$$\displaystyle{500}$$\displaystyle{750}$$\displaystyle{1000}$$\displaystyle{1250}$day
number$\displaystyle{0.00}$$\displaystyle{0.25}$$\displaystyle{0.50}$$\displaystyle{0.75}$$\displaystyle{1.00}$norm.
daily energySD seasonal + deg componentsSD seasonal + deg + soiling components
Figure 9: An illustration of the soiling decomposition results for the
unlabeled rooftop PV data discussed in [21]. The soiling trend is estimated to
be the difference between the orange and green trends.
$\displaystyle{0}$$\displaystyle{250}$$\displaystyle{500}$$\displaystyle{750}$$\displaystyle{1000}$$\displaystyle{1250}$day
number$\displaystyle{-10}$$\displaystyle{0}$soiling loss [%] Figure 10: The
isolated soiling trend for the unlabeled data analysis.

##  V Conclusions

We present a methodology, based on the signal decomposition (SD) framework,
for estimating soiling losses PV system production data, i.e., time series
measurements of generated real power or energy, typically over multiple years.
Unique to this work is the capability to estimate soiling losses in raw,
unlabeled power/energy data, rather than requiring a performance index. By
utilizing the extensibility of the SD, we are able to design a signal
decomposition model that is bespoke to the problem of estimating soiling
losses. This results in a robust model for the soiling loss component,
$x^{4}$, as well as adjustable residual component, $x^{1}$, that is able to
model both unlabeled ($\tau=0.85$) and labeled ($\tau=0.5$) data. The ability
to analyze unlabled PV data potentially unlocks huge potential in the form of
fleet-scale datasets of hetergenous, distributed PV systems, which typically
have internet-connected power electronics which generate time series of real
power, but lack correlated meteorological measurements and possibly accurate
system models. A software implementation is available in the Solar Data Tools
package [4, 5] and a demonstration notebook of the code usage is available
online [23].

## Acknowledgments

The author would like to thank Stephen Boyd, Justin Luke, Elsa Kam-Lum, Mayank
Malik, and the entire GISMo Team at SLAC National Accelerator Laboratory for
their input and feedback on this work. I also recognize the Seaborn plotting
package for Python, which made the figures possible [24].

## References

  * [1] K. Ilse, L. Micheli, B. W. Figgis, K. Lange, D. Daßler, H. Hanifi, F. Wolfertstetter, V. Naumann, C. Hagendorf, R. Gottschalg, and J. Bagdahn, “Techno-economic assessment of soiling losses and mitigation strategies for solar power generation,” _Joule_ , vol. 3, no. 10, pp. 2303–2321, 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S2542435119304222
  * [2] M. Davis, C. Smith, B. White, R. Goldstein, X. Sun, M. Cox, G. Curtin, R. Manghani, S. Rumery, C. Silver, and J. Baca, _U.S. Solar market insight executive summary, 2020 year in review_. Wood Mackenzie and SEIA, 2021.
  * [3] B. Meyers and S. Boyd, “Signal decomposition using masked proximal operators,” pp. 1–60, feb 2022. [Online]. Available: http://arxiv.org/abs/2202.09338
  * [4] B. Meyers, E. Apostolaki-Iosifidou, and L. Schelhas, “Solar data tools: Automatic solar data processing pipeline,” in _2020 47th IEEE Photovoltaic Specialists Conference (PVSC)_ , 2020, pp. 0655–0656.
  * [5] B. Meyers, “solar-data-tools,” may 2022. [Online]. Available: http://dx.doi.org/10.5281/zenodo.6450368
  * [6] M. Deceglie, L. Micheli, and M. Muller, “Quantifying soiling loss directly from pv yield,” _IEEE Journal of Photovoltaics_ , vol. 8, no. 2, pp. 547–551, 2018.
  * [7] Å. Skomedal, H. Haug, and E. S. Marstein, “Endogenous soiling rate determination and detection of cleaning events in utility-scale pv plants,” _IEEE Journal of Photovoltaics_ , vol. 9, no. 3, pp. 858–863, 2019.
  * [8] A. Skomedal and M. Deceglie, “Combined Estimation of Degradation and Soiling Losses in Photovoltaic Systems,” _IEEE Journal of Photovoltaics_ , vol. 10, no. 6, pp. 1788–1796, nov 2020. [Online]. Available: https://ieeexplore.ieee.org/document/9186286/
  * [9] L. Micheli, M. Theristis, A. Livera, J. Stein, G. Georghiou, M. Muller, F. Almonacid, and E. Fernandez, “Improved PV soiling extraction through the detection of cleanings and change points,” _IEEE Journal of Photovoltaics_ , vol. 11, no. 2, pp. 519–526, 2021.
  * [10] T. Hastie, R. Tibshirani, and J. Friedman, _The Elements of Statistical Learning_ , ser. Springer Series in Statistics. New York, NY: Springer New York, dec 2009. [Online]. Available: http://ieeexplore.ieee.org/document/6727256/ http://link.springer.com/10.1007/978-0-387-84858-7
  * [11] R. Koenker and G. Bassett, “Regression quantiles,” _Econometrica_ , vol. 46, no. 1, p. 33, jan 1978. [Online]. Available: https://www.jstor.org/stable/1913643 https://www.jstor.org/stable/1913643?origin=crossref
  * [12] R. Koenker and K. F. Hallock, “Quantile regression,” _Journal of Economic Perspectives_ , vol. 15, no. 4, pp. 143–156, nov 2001. [Online]. Available: https://pubs.aeaweb.org/doi/10.1257/jep.15.4.143
  * [13] S. Boyd and L. Vandenberghe, _Introduction to Applied Linear Algebra_ , 2018\.
  * [14] ——, _Convex optimization_. Cambridge University Press, 2009.
  * [15] S. Diamond and S. Boyd, “CVXPY: A Python-embedded modeling language for convex optimization,” _Journal of Machine Learning Research_ , vol. 17, no. 83, pp. 1–5, 2016.
  * [16] A. Agrawal, R. Verschueren, S. Diamond, and S. Boyd, “A rewriting system for convex optimization problems,” _Journal of Control and Decision_ , vol. 5, no. 1, pp. 42–60, 2018.
  * [17] E. D. Andersen and K. D. Andersen, “The Mosek Interior Point Optimizer for Linear Programming: An Implementation of the Homogeneous Algorithm,” in _High performance optimization_ , 2000, pp. 197–232. [Online]. Available: http://link.springer.com/10.1007/978-1-4757-3216-0_8
  * [18] S. Boyd and L. Vandenberghe, “Convex Optimization,” in _Cambridge University Press_ , 2004.
  * [19] “Numpy.diff,” _NumPy v1.22 Manual_. [Online]. Available: https://numpy.org/doc/stable/reference/generated/numpy.diff.html
  * [20] E. Kam-Lum, B. E. Meyers, D. Cosme, B. Aissa, and G. Scabbia, “Soiling Rate Determination from Referenced Systems in Desert Climate using PVInsight Soiling Algorithm,” in _2021 IEEE 48th Photovoltaic Specialists Conference (PVSC)_ , no. July 2019. IEEE, jun 2021, pp. 2552–2554. [Online]. Available: https://ieeexplore.ieee.org/document/9518459/
  * [21] B. Meyers and D. F. Florez Rodriguez, “Estimation of shade losses in unlabeled PV data,” _Submitted to PVSC49_ , June 2022.
  * [22] D. F. Florez Rodriguez and B. Meyers, “Solar panel power simulation for shade detection,” _Submitted to PVSC49_ , June 2022.
  * [23] B. Meyers, “Soiling analysis demonstration,” _solar-data-tool GitHub repository_ , May 2022. [Online]. Available: https://github.com/slacgismo/solar-data-tools/blob/shade-dev/notebooks/Soiling_analysis_demonstration.ipynb
  * [24] M. L. Waskom, “seaborn: statistical data visualization,” _Journal of Open Source Software_ , vol. 6, no. 60, p. 3021, 2021. [Online]. Available: https://doi.org/10.21105/joss.03021