11institutetext: University of Birmingham, UK 22institutetext: UNIST, Korea
33institutetext: SUSTech, China

# S2Contact: Graph-based Network for 3D Hand-Object Contact Estimation with
Semi-Supervised Learning

Tze Ho Elden Tse Equal contribution11 Zhongqun Zhang††footnotemark:  11 Kwang
In Kim 22 Ales̆ Leonardis 11 Feng Zheng 33 Hyung Jin Chang 11

###### Abstract

Despite the recent efforts in accurate 3D annotations in hand and object
datasets, there still exist gaps in 3D hand and object reconstructions.
Existing works leverage contact maps to refine inaccurate hand-object pose
estimations and generate grasps given object models. However, they require
explicit 3D supervision which is seldom available and therefore, are limited
to constrained settings, e.g., where thermal cameras observe residual heat
left on manipulated objects. In this paper, we propose a novel semi-supervised
framework that allows us to learn contact from monocular images. Specifically,
we leverage visual and geometric consistency constraints in large-scale
datasets for generating pseudo-labels in semi-supervised learning and propose
an efficient graph-based network to infer contact. Our semi-supervised
learning framework achieves a favourable improvement over the existing
supervised learning methods trained on data with ‘limited’ annotations.
Notably, our proposed model is able to achieve superior results with less than
half the network parameters and memory access cost when compared with the
commonly-used PointNet-based approach. We show benefits from using a contact
map that rules hand-object interactions to produce more accurate
reconstructions. We further demonstrate that training with pseudo-labels can
extend contact map estimations to out-of-domain objects and generalise better
across multiple datasets. Project page is
available.111https://eldentse.github.io/s2contact/

##  1 Introduction

Understanding hand-object interactions have been an active area of study in
recent years [18, 16, 17, 3, 49, 26, 20, 62, 31, 51]. Besides common practical
applications in augmented and virtual reality [15, 54, 36, 51, 50, 66], it is
a key ingredient to advanced human-computer interaction [52] and imitation
learning in robotics [64]. In this paper, as illustrated in Figure 1, we
tackle the problem of 3D reconstruction of the hand and manipulated object
with the focus on contact map estimation.

Figure 1:  Overview of our semi-supervised learning framework, S2Contact. (a)
The model is pre-trained on a small annotated dataset. (b) Then, it is
deployed on unlabelled datasets to collect pseudo-labels. The pseudo-labels
are filtered with confidence-based on visual and geometric consistencies. Upon
predicting the contact map, the hand and object poses are jointly optimised to
achieve target contact via a contact model [12].

Previous works in hand-object interactions typically formulate this as a joint
hand and object pose estimation problem. Along with the development of data
collection and annotation methods, more accurate 3D annotations for real
datasets [11, 14, 4] are available for learning-based methods [10, 49].
Despite the efforts, there still exist gaps between hand-object pose
estimation and contact as ground-truth in datasets are not perfect. Recent
works attempt to address this problem with interaction constraints (attraction
and repulsion) under an optimisation framework [17, 3, 62]. However, inferred
poses continue to exhibit sufficient error to cause unrealistic hand-object
contact, making downstream tasks challenging [12]. In addition, annotations
under constrained laboratory environments rely on strong priors such as
limited hand motion which prevents the trained model from generalising to
novel scenes and out-of-domain objects.

To address the problem of hand-object contact modelling, Brahmbhatt et al. [1]
used thermal cameras observing the heat transfer from hand to object after the
grasp to capture detailed ground-truth contact. Their follow-up work
contributed a large grasp dataset (_ContactPose_) with contact maps and hand-
object pose annotations. Recent works are able to leverage contact maps to
refine inaccurate hand-object pose estimations [12] and generate grasps given
object model [20]. Therefore, the ability to generate an accurate contact map
is one of the key elements to reasoning physical contact. However, the number
of annotated objects is incomparable to manipulated objects in real life and
insufficient to cover a wide range of human intents. Furthermore, obtaining
annotations for contact maps is non-trivial as it requires thermal sensors
during data collection.

To enable the wider adoption of contact maps, we propose a unified framework
that leverages existing hand-object datasets for generating pseudo-labels in
semi-supervised learning. Specifically, we propose to exploit the visual and
geometric consistencies of contact maps in hand-object interactions. This is
built upon the idea that the poses of the hands and objects are highly-
correlated where the 3D pose of the hand often indicates the orientation of
the manipulated object. We further extend this by enforcing our contact
consistency loss for the contact maps across a video.

As the input to contact map estimator are in the form of point clouds, recent
related works [12, 20] typically follow a PointNet-based architectures [38,
39]. This achieves permutation invariance of points by operating on each point
independently and subsequently applying a symmetric function to accumulate
features [55]. However, the network performances are limited as points are
treated independently at a local scale to maintain permutation invariance. To
overcome this fundamental limitation, many recent approaches adopt graph
convolutional networks (GCN) [9, 25] and achieve state-of-the-art performances
in 3D representation learning on point clouds for classification, part
segmentation and semantic segmentation [55, 28, 30]. The ability to capture
local geometric structures while maintaining permutation invariance is
particularly important for estimating contact maps. However, it comes at the
cost of high computation and memory usage for constructing a local
neighbourhood with $K$-nearest neighbour ($K$-NN) search on point clouds at
each training epoch. For this reason, we design a graph-based neural network
that demonstrates superior results with less than half the learning parameters
and faster convergence.

Our contributions are three-fold:

  * •

We propose a novel semi-supervised learning framework that combines pseudo-
label with consistency training. Experimental results demonstrate the
effectiveness of this training strategy.

  * •

We propose a novel graph-based network for processing hand-object point
clouds, which is at least two times more efficient than PointNet-based
architecture for estimating contact between hand and object.

  * •

We conduct comprehensive experiments on three commonly-used hand-object
datasets. Experiments show that our proposed framework S2Contact outperforms
recent semi-supervised methods.

##  2 Related work

Our work tackles the problem of hand and object reconstruction from monocular
RGB videos, exploiting geometric and visual consistencies on contact maps for
semi-supervised learning. To the best of our knowledge, we are the first to
apply such consistencies on hand-object scenarios. We first review the
literature on _hand-object reconstruction_. Then, we review _point cloud
analysis_ with the focus of graph-based methods. Finally, we provide a brief
review on _semi-supervised learning in 3D hand-object pose estimation_.

###  2.1 Hand-object reconstruction

Previous works mainly tackle 3D pose estimations on hands [42, 67, 35, 44, 60,
60, 47] and objects [27, 29, 58, 5, 6] separately. Joint reconstruction of
hands and objects has been receiving increasing attention [18, 16, 3, 17].
Hasson et al. [18] introduces an end-to-end model to regress MANO hand
parameters jointly with object mesh vertices deformed from a sphere and
incorporates contact losses which encourages contact surfaces and penalises
penetrations between hand and object. A line of works [49, 10, 16, 3, 17, 62,
12, 19] assume known object models and regress a 6DoF object pose instead.
Other works focus on grasp synthesis [8, 21, 46, 20]. In contrast, our method
is in line with recent optimisation-based approaches for modelling 3D hand-
object contact. ContactOpt [12] proposes a contact map estimation network and
a contact model to produce realistic hand-object interaction. ContactPose
dataset [2] is unique in capturing ground-truth thermal contact maps. However,
3D contact labels are seldom available and limited to constrained labratory
settings. In this work, we treat contact maps as our primary learning target
and leverage unannotated datasets.

###  2.2 Point cloud analysis

Since point cloud data is irregular and unordered, early works tend to project
the original point clouds to intermediate voxels [33] or images [63], i.e.
translating into a well-explored 2D image problem. As information loss caused
by projection degrades the representational quality, PointNet [38] is proposed
to directly process unordered point sets and PointNet++ [39] is extends on
local point representation in multi-scale. As PointNet++ [39] can be view as
the generic point cloud analysis network framework, the research focus has
been shifted to generating better regional points representation. Methods can
be divided into convolution [57, 59], graph [55, 28, 30] and attention [13,
65] -based. Graph-based methods. GCNs have been gaining much attention in the
last few years. This is due to two reasons: 1) the rapid increase of non-
Euclidean data in real-world applications and 2) the limited performance of
convolutional neural networks when dealing with such data. As the unstructured
nature of point clouds poses a representational challenge in the community,
graph-based methods treat points as nodes of a graph and formulate edges
according to their spatial/feature relationships. MoNet [34] defines the
convolution as Gaussian mixture models in a local pseudo-coordinate system.
3D-GCN [30] proposes a deformable kernels which has shift and scale-invariant
properties for point cloud processing. DGCNN [55] proposes to gather nearest
neighbouring points in feature space and follow by the EdgeConv operators for
feature extraction. The EdgeConv operator dynamically computes node adjacency
at each graph layer using the distance between point features. In this paper,
we propose a computationally efficient network for contact map estimation
which requires less than half the parameters of PointNet [38] and GPU memory
of DGCNN [55].

###  2.3 Semi-supervised learning in 3D hand-object pose estimation

Learning from both labelled and unlabelled data simultaneously has recently
attracted growing interest in 3D hand pose estimation [61, 43, 23, 7, 48].
They typically focus on training models with a small amount of labelled data
as well as a relatively larger amount of unlabelled data. After training on
human-annotated datasets, pseudo-labelling and consistency training can be
used to train further and a teacher-student network with exponential moving
average (EMA) strategy [53] is common to accelerate the training. For
instance, So-HandNet [7] leverages the consistency between the recovered hand
point cloud and the original hand point cloud for semi-supervised training.
SemiHand [61] is the first to combine pseudo-labelling and consistency
learning for hand pose estimation. Liu et al. [31] is the only prior work on
3D hand-object pose estimation with semi-supervised learning. They proposed
spatial and temporal constraints for selecting the pseudo-labels from videos.
However, they are limited to pseudo hand labels and did not account for
physical contact with manipulated objects. In contrast, our work is the first
to explore pseudo-labelling for 3D hand-object contact map with geometric and
visual consistency constraints.

##  3 Methodology

Given a noisy estimate of hand and object meshes from an image-based
algorithm, we seek to learn a hand-object contact region estimator by
exploiting real-world hand and object video datasets without contact ground-
truths. Figure 1 shows an overview of our approach. In the following section,
we describe our learned contact map estimation network (GCN-Contact) in
Section 3.1 and our newly proposing semi-supervised training pipeline
(S2Contact) in Section 3.2 that utilise a teacher-student mutual learning
framework.

Figure 2: Framework of GCN-Contact. The network takes hand-object point clouds
$\mathbf{P}=(\mathbf{P}_{hand},\mathbf{P}_{obj})$ as input and perform $K$-NN
search separately on 3D position $\mathbf{P}_{pos}$ and point features
$\mathbf{P}_{F}$, i.e. $\mathbf{P}=\\{\mathbf{P}_{pos},\mathbf{P}_{F}\\}$.
Different dilation factors $d,d^{\prime}$ are used to enlarge the receptive
field for graph convolution $f(\cdot)$. Finally, features are concatenated and
pass to MLP to predict contact map.

###  3.1 GCN-Contact: 3D hand-object contact estimation

As pose estimates from an image-based algorithm can be potentially inaccurate,
GCN-Contact learns to infer contact maps
$\mathbf{C}=(\mathbf{C}_{hand},\mathbf{C}_{obj})$ from hand and object point
clouds $\mathbf{P}=(\mathbf{P}_{hand},\mathbf{P}_{obj})$. We adopted the
differential MANO [41] model from [18]. It maps pose
($\boldsymbol{\theta}\in\mathbb{R}^{51}$) and shape
($\boldsymbol{\beta}\in\mathbb{R}^{10}$) parameters to a mesh with $N=778$
vertices. Pose parameters ($\boldsymbol{\theta}$) consists of $45$ DoF (i.e.
$3$ DoF for each of the $15$ finger joints) plus $6$ DoF for rotation and
translation of the wrist joint. Shape parameters ($\boldsymbol{\beta}$) are
fixed for a given person. We sample $2048$ points randomly from object model
to form object point cloud. Following [12], we include $F$-dimensional point
features for each point: binary per-point feature indicating whether the point
belongs to the hand or object, distances from hand to object and surface
normal information. With network input
$\mathbf{P}=(\mathbf{P}_{hand},\mathbf{P}_{obj})$ where
$\mathbf{P}_{hand}\in\mathbb{R}^{778\times F}$ and
$\mathbf{P}_{obj}\in\mathbb{R}^{2048\times F}$, GCN-Contact can be trained to
infer discrete contact representation
($\mathbf{C}=(\mathbf{C}_{hand},\mathbf{C}_{obj})\in[0,1]$) [2] using binary
cross-entropy loss. Similarly to [12], the contact value range $[0,1]$ is
evenly split into $10$ bins and the training loss is weighted to account for
class imbalance.

#### Revisiting PointNet-based methods.

Recent contact map estimators are based on PointNet [20] and PoinetNet++ [12].
PointNet [38] directly processes unordered point sets using shared multi-layer
perceptron (MLP) networks. PointNet++ [39] learns hierarchical features by
stacking multiple learning stages and recursively capturing local geometric
structures. At each learning stage, farthest point sampling (FPS) algorithm is
used to re-sample a fixed number of points and $K$ neighbours are obtained
from ball query’s local neighbourhood for each sampled point to capture local
structures. The kernel operation of PointNet++ for point
$p_{i}\in\mathbb{R}^{F}$ with $F$-dimensional features can be described as:

$\displaystyle\dot{p_{i}}=\sigma\big{(}\Phi(p_{j}|j\in\mathcal{N}(i))\big{)},$
(1)

where the updated point $\dot{p_{i}}$ is formed by max-pooling function
$\sigma(\cdot)$ and PointNet as the basic building block for local feature
extractor $\Phi(\cdot)$ around point neighbourhood $\mathcal{N}(i)$ of point
$p_{i}$. The kernel of the point convolution can be implemented with MLPs.
However, MLPs are unnecessarily performed on the neighbourhood features which
causes a considerable amount of latency in PointNet++ [40]. This motivates us
to employ advanced local feature extractors such as convolution [57, 59],
graph [55, 28, 30] or self-attention mechanisms [13, 65].

#### Local geometric information.

While contact map estimation can take advantage of detailed local geometric
information, they usually suffer from two major limitations. First, the
computational complexity is largely increased with delicate extractors which
leads to low inference latency. For instance, in graph-based methods,
neighbourhood information gathering modules are placed for better modelling of
the locality on point clouds. This is commonly established by $K$-nearest
neighbour ($K$-NN) search which increases the computational cost quadratically
with the number of points and even further for dynamic feature representation
[55]. For reference on ModelNet$40$ point cloud classification task [40], the
inference speed of PointNet [38] is $41$ times faster than DGCNN [55]. Second,
Liu et al.’s investigation on local aggregation operators reveals that
advanced local feature extractors make surprisingly similar contributions to
the network performance under the same network input [32]. For these reasons,
we are encouraged to develop a computationally efficient design while
maintaining comparable accuracy for learning contact map estimation.

#### Proposed method.

To overcome the aforementioned limitations, we present a simple yet effective
graph-based network for contact map estimation. We use EdgeConv [55] to
generate edge features that describe the relationships between a point and its
neighbours:

$\displaystyle\Phi(p_{i},p_{j})=\text{ReLU}\big{(}\text{MLP}(p_{j}-p_{i},p_{i})\big{)},\;\;j\in\mathcal{N}(i),$
(2)

where neighbourhood $\mathcal{N}(i)$ is obtained by $K$-NN search around the
point $p_{i}$. As shown in Figure 2, we only compute $K$-NN search once at
each network pass to improve computational complexity and reduce memory usage.
In addition, we apply dilation on the $K$-NN results to increase the receptive
field without loss of resolution. To better construct local regions when hand
and object are perturbed, we propose to perform $K$-NN search on 3D position
and point features separately. Note that [12] perform ball query on $0.1-0.2m$
radius and [55] combine both position and features. Finally, we take
inspirations from the Inception model [45] in which they extract multi-scale
information by using different kernel sizes in different paths of the
architecture. Similarly, we process spatial information at various dilation
factors and then aggregates. The experiment demonstrates the effectiveness of
our proposed method and is able to achieve constant memory access cost
regardless of the size of dilation factor $d$ (see Table 1).

###  3.2 S2Contact: Semi-supervised training pipeline

Figure 3:  S2Contact pipeline. We adopt our proposed graph-based network GCN-
Contact as backbone. We utilise a teacher-student mutual learning framework
which is composed of a learnable student and an EMA teacher. The student
network is trained with labelled data
$\left\\{\mathbf{x}^{l},\mathbf{y}^{l}\right\\}$. For unlabelled data
$\mathbf{x}^{u}$, the student network takes pseudo contact labels
$\widetilde{\mathbf{C}}^{u}$ from its EMA teacher and compares with its
predictions $\widehat{\mathbf{C}}$. Please note that pseudo contact labels
$\widetilde{\mathbf{C}}^{u}$ is a subset of $\widetilde{\mathbf{C}}$, i.e.
$\widetilde{\mathbf{C}}^{u}\in\widetilde{\mathbf{C}}$. (a) refers to contact
consistency constraint for consistency training. To improve the quality of
pseudo-label, we adopt a confidence-based filtering mechanism to geometrically
(b) and visually (c) filter out predictions that violate contact constraints.

Collecting ground-truth contact annotation for hand-object dataset can be both
challenging and time-consuming. To alleviate this, we introduce a semi-
supervised learning framework to learn 3D hand-object contact estimation by
leveraging large-scale unlabelled videos. As shown in Figure 1, our proposed
framework relies on two training stages: 1) pre-training stage where the model
is pre-trained on the existing labelled data [2]; 2) semi-supervised stage
where the model is trained by the pseudo-labels from unlabelled hand-object
datasets [4, 14, 11]. As pseudo-labels are often noisy, we propose confidence-
based filtering with geometric and visual consistency constraints to improve
the quality of pseudo-labels.

#### Pre-training.

As good initial contact estimate enables semi-supervision, we pre-train our
graph-based contact estimator using a small labelled dataset
$\left\\{\mathbf{x}^{l},\mathbf{y}^{l}\right\\}$. We followed [12] and
optimise hand-object poses to achieve target contact. Upon convergence, we
clone the network to create a pair of student-teacher networks.

#### Pseudo-label generation.

To maintain a reliable performance margin over the student network throughout
the training, we adopt an EMA teacher which is commonly used in semi-
supervised learning. The output of the student network is the predicted
contact map $\widehat{\mathbf{C}}$. The teacher network generates pseudo-
labels which includes pre-filter contact map $\widetilde{\mathbf{C}}$ and
refined hand-object pose $\widetilde{P}_{ref}$. As it is crucial for the
teacher network to generate high-quality pseudo-labels under a semi-supervised
framework, we propose a confidence-based filtering mechanism that leverages
geometric and visual consistency constraints.

#### Contact consistency constraint for consistency training.

We propose a contact consistency loss to encourage robust and stable
predictions for unlabelled data $\mathbf{x}^{u}$. As shown in Figure 3 (a), we
first apply stochastic transformations $\mathcal{T}$ which includes flipping,
rotation and scaling on the input hand-object point clouds $\mathbf{x}^{u}$
for the student network. The predictions of the student network
$\widehat{\mathbf{y}}^{u}_{s}\in\widehat{\mathbf{C}}$ are compared with the
teacher predictions $\widetilde{\mathbf{y}}^{u}_{t}\in\widetilde{\mathbf{C}}$
processed by the same transformation $\mathcal{T}$ using contact consistency
loss:

$\displaystyle\begin{split}\mathcal{L}_{cont}(\mathbf{x}^{u})&=\|\Omega(\mathcal{T}(\mathbf{x}^{u}))-(\mathcal{T}(\Omega(\mathbf{x}^{u})))\|_{1}\\\
&=\|\widehat{\mathbf{y}}^{u}_{s}-\widetilde{\mathbf{y}}^{u}_{t}\|_{1},\end{split}$
(3)

where $\Omega(\cdot)$ represents the predicted contact map.

#### Geometric consistency constraint for pseudo-labels filtering.

As shown in Figure 3 (b), we propose a geometric consistency constraint to the
hand and object pseudo pose label $\widetilde{P}_{ref}$. Concretely, we allow
the Chamfer distance $\mathcal{L}_{Cham}$ between hand and object meshes to be
less than threshold $t_{dist}$:

$\displaystyle\mathcal{L}_{Cham}(\widetilde{\mathbf{v}}_{hand},\widetilde{\mathbf{v}}_{obj})=$
$\displaystyle\frac{1}{\left|\widetilde{\mathbf{v}}_{obj}\right|}\sum_{x\in\widetilde{\mathbf{v}}_{obj}}d_{\widetilde{\mathbf{v}}_{hand}}(x)+\frac{1}{\left|\widetilde{\mathbf{v}}_{hand}\right|}\sum_{y\in\widetilde{\mathbf{v}}_{hand}}d_{\widetilde{\mathbf{v}}_{obj}}(y),$
(4)

where $\widetilde{\mathbf{v}}_{hand}$ and $\widetilde{\mathbf{v}}_{obj}$
refers to hand and object point sets,
$d_{\widetilde{\mathbf{v}}_{hand}}(x)=\min_{y\in\widetilde{\mathbf{v}}_{hand}}\allowbreak\left\|x-y\right\|_{2}^{2}$,
and
$d_{\widetilde{\mathbf{v}}_{obj}}(y)=\min_{x\in\widetilde{\mathbf{v}}_{obj}}\left\|x-y\right\|_{2}^{2}$.
Similarly for interpenetration, we use
$\mathcal{L}_{SDF}(\widetilde{\mathbf{v}}_{obj})=\sum_{hand,obj}\sum_{i}\Psi_{h}\left(\widetilde{\mathbf{v}}_{obj}^{i}\right)\leq
t_{pen}$ to ensure object is being manipulated by hand. $\Psi_{h}$ is the
Signed Distance Field (SDF) from the hand mesh (i.e. ,
$\Psi_{h}(\widetilde{\mathbf{v}}_{obj})=-\min\left(\operatorname{SDF}\left(\widetilde{\mathbf{v}}_{obj}\right),0\right)$)
to detect object penetrations.

#### Visual consistency constraint for pseudo-labels filtering.

We observed that geometric consistency is insufficient to correct hand grasp
(see Table 5). To address this, we propose a visual consistency constraint to
filter out the pseudo-labels whose rendered hand-object image
$\widetilde{I}_{ho}$ does not match the input image. We first use a renderer
[22] to render the hand-object image from the refined pose
$\widetilde{P}_{ref}$ and obtain the hand-object segment of the input image
$I$ by applying the segmentation mask $M_{gt}$. Then, the structural
similarity (SSIM) [56] between two images can be computed. We keep pseudo-
labels when $\mathcal{L}_{SSIM}\leq t_{SSIM}$:

$\mathcal{L}_{SSIM}(I,M_{gt},\widetilde{I}_{ho})=1-SSIM\left(I\odot
M_{gt},\widetilde{I}_{ho}\right),$ (5)

where $\odot$ denotes element-wise multiplication.

#### Self-training with pseudo-labels.

After filtering pseudo-labels, our model is trained with the union set of the
human-annotated dataset and the remaining pseudo-labels. The total loss
$\mathcal{L}_{semi}$ can be described as:

$\mathcal{L}_{semi}(\widehat{\mathbf{C}},\mathbf{y}^{l},\widetilde{\mathbf{C}}^{u},\mathbf{x}^{u})=\mathcal{L}_{sup}\left(\widehat{\mathbf{C}},\mathbf{y}^{l}\right)+\mathcal{L}_{unsup}\left(\widehat{\mathbf{C}},\widetilde{\mathbf{C}}^{u}\right)+\lambda_{c}\mathcal{L}_{cont}\left(\mathbf{x}^{u}\right),$
(6)

where $\mathcal{L}_{sup}$ is a supervised contact loss, $\mathcal{L}_{unsup}$
is a unsupervised contact loss with pseudo-labels and $\lambda_{c}$ is a
hyperparameter. Note that $\mathcal{L}_{sup}$ (see Figure 2) and
$\mathcal{L}_{unsup}$ (see Figure 3) are both binary cross-entropy loss.

##  4 Experiments

#### Implementation details.

We implement our method in PyTorch [37]. All experiments are run on an Intel
i9-CPU @ 3.50GHZ, 16 GB RAM, and one NVIDIA RTX 3090 GPU. For pseudo-labels
filtering, $t_{dist}=0.7$, $t_{pen}=6$ and $t_{SSIM}=0.25$ are the constant
thresholds and stochastic transformations includes flipping ($\pm 20\%$),
rotation ($\pm 180^{\circ}$) and scaling ($\pm 20\%$). We train all parts of
the network simultaneously with Adam optimiser [24] at a learning rate
$10^{-3}$ for 100 epochs. We empirically fixed $K=10,d=4$ to produce the best
results.

#### Datasets and evaluation metrics.

_ContactPose_ is the first dataset [2] of hand-object contact paired with hand
pose, object pose and RGB-D images. It contains 2,306 unique grasps of $25$
household objects grasped with $2$ functional intents by $50$ participants,
and more than $2.9$M RGB-D grasp images. For fair comparisons with ContactOpt
[12], we follow their _Perturbed ContactPose_ dataset where hand meshes are
modified by additional noise to MANO parameters. This results in 22,624
training and 1,416 testing grasps. _DexYCB_ is a recent real dataset for
capturing hand grasping of objects [4]. It consists of a total of 582,000
image frames on 20 objects from the YCB-Video dataset [58]. We present results
on their default official dataset split settings. _HO-3D_ [14] is similar to
_DexYCB_ where it consists of 78,000 images frames on 10 objects. We present
results on the official dataset split (version 2). The hand mesh error is
reported after procrustes alignment and in $mm$.

  * •

_Hand error:_ We report the mean end-point error ($mm$) over 21 joints and
mesh error in $mm$.

  * •

_Object error:_ We report the percentage of average object 3D vertices error
within $10\%$ of object diameter (ADD-$0.1$D).

  * •

_Hand-object interaction:_ We report the intersection volume ($cm^{3}$) and
contact coverage ($\%$). Intersection volume is obtained by voxelising the
hand and object using a voxel size of $0.5cm$. Contact coverage refers to the
percentage of hand points between $\pm 2mm\%$ of the object surface [12].

#### Baseline.

For refining image-based pose estimates, we use the baseline pose estimation
network from Hasson et al. [16] and retrain it on the training split of the
respecting dataset. We filter out frames where the minimum distance between
the ground truth hand and object surfaces is greater than 2 $mm$. We also use
the contact estimation network DeepContact from Grady et al. [12] which takes
ground-truth object class and pose. For semi-supervised learning, we use the
baseline method from Liu et al. [31], a semi-supervised learning pipeline for
3D hand-object pose estimation from large-scale hand-object interaction
videos.

###  4.1 Comparative results

#### Refining small and large inaccuracies.

We use _ContactPose_ to evaluate GCN-Contact for refining poses with small
(_ContactPose_) and large (_Perturbed ContactPose_) inaccuracies. Table 1
shows the results for both cases. For _Perturbed ContactPose_ , the mean end-
point error over $21$ joints is $82.947mm$ before refinement. This is aimed at
testing the ability to improve hand poses with large errors. In contrast,
_ContactPose_ is used to evaluate $mm$-scale refinement. As shown, our method
consistently outperforms baseline and DGCNN [55]. We attribute the performance
gain to multi-scale feature aggregation with dilation. Qualitative comparison
with ContactOpt [12] is provided in Figure 4.

Table 1: Hand error rates ($mm$) on _Perturbed ContactPose_ and _ContactPose_
datasets.

| Baseline | DGCNN [55] | Ours
---|---|---|---
|  joint $\downarrow$ |  mesh $\downarrow$ |  joint $\downarrow$ |  mesh $\downarrow$ |  joint $\downarrow$ |  mesh $\downarrow$
_Perturbed ContactPose_ |  32.988 |  33.147 |  32.592 |  32.762 |  29.442 |  30.635
_ContactPose_ |  8.880 |  8.769 |  8.767 |  32.988 |  5.878 |  5.765

Figure 4: Qualitative comparison with ContactOpt [12] on _ContactPose_. We
observed that penetrations across hand and object (can be seen in (b)) is
likely to be caused by contact predictions appearing on both object surfaces.
Our model, trained only on _ContactPose_ , shows robustness to various hand
poses and objects.

#### Refining Image-based pose estimates.

We evaluate S2Contact in refining poses from an image-based pose estimator. We
use the baseline image-based pose estimation network from Hasson et al. [16]
and retrain it on the training split of the respecting dataset. Unlike [12],
we do not rely on ground-truth object class and pose. In particular, we
compare with the current state-of-the-art [31] which is also a semi-supervised
framework for 3D hand-object pose estimation. Liu et al. [31] proposes
spatial-temporal consistency in large-scale hand-object videos to generate
pseudo-labels for hand. In contrast, we leverage physical contact and visual
consistency constraints to generate pseudo contact labels which can be
optimised jointly with hand and object poses. As shown in Table 2, our method
outperforms [31] by $11.5\%$ in average object ADD-$0.1$D score. Besides, we
also compare with our baseline contact model ContactOpt [12]. As shown in
Table 2, we are able to further improve hand error by $1mm$ and $0.8mm$ over
joints and mesh. In addition to hand-object pose performance, our method is
able to better reconstruct hand and object with less intersection volume and
higher contact coverage. The above demonstrates that our method provides a
more practical alternative to alleviate the reliance on heavy dataset
annotation in hand-object. In addition, we provide qualitative comparison on
_HO-3D_ in Figure 5. We also report the cross-dataset generalisation
performance of our model on _DexYCB_ in Table 3. We select three objects (i.e.
, mustard bottle, potted meat can and bleach cleanser), to be consistent with
_HO-3D_. As shown, our method consistently shown improvements across all
metrics.

Table 2:  Error rates on _HO-3D_. Note that Liu et al. [31] is the current
state-of-the-art semi-supervised method. ave, inter and cover refers to
average, intersection volume and contact coverage, respectively.

| Hand error | Object ADD-0.1D($\uparrow$) | Contact
---|---|---|---
Methods |  joint $\downarrow$ |  mesh $\downarrow$ |  bottle |  can |  bleach |  avg |  cover $\uparrow$ | inter $\downarrow$
Initial Pose [16] |  11.1 |  11.0 |  - |  - |  - |  74.5 |  4.4 | 15.3$\pm$21.1
ContactOpt [12] |  9.7 |  9.7 |  - |  - |  - |  75.5 |  14.7 | 6.0$\pm$6.7
Liu et al. [31] |  9.9 |  9.5 |  69.6 |  53.2 |  86.9 |  69.9 |  - | -
Ours |  8.7 |  8.9 |  79.1 |  71.8 |  93.3 |  81.4 |  19.2 | 3.5$\pm$1.8

Table 3:  Error rates of the cross-dataset generalisation performance on
_DexYCB_.

| Hand error | Object ADD-0.1D($\uparrow$) | Contact
---|---|---|---
models |  joint $\downarrow$ |  mesh $\downarrow$ |  bottle |  can |  bleach |  avg |  cover $\uparrow$ | inter $\downarrow$
w/o semi-supervised |  13.0 |  13.7 |  76.9 |  46.2 |  68.4 |  63.8 |  4.1 | 16.0$\pm$12.3
semi-supervised |  11.8 |  12.1 |  83.6 |  53.7 |  74.2 |  70.5 |  9.3 | 10.5$\pm$7.9

Figure 5: Qualitative comparison with Initial Pose [31] on _HO-3D_. (a) We
observed that the reconstructions of hand and object are more likely to be
distant in 3D space due to depth and scale ambiguity with only RGB image as an
input. (b) In contrast, when without semi-supervised learning, there exists
more interpenetration or incorrect grasps between hand and object. (c) As
shown, together with visual cues and contact map estimations, our method is
able correct the above failure cases and generate more physically-plausible
reconstructions.

###  4.2 Ablation study

#### Number of $K$ neighbours.

Table 4(a) shows the results of varying number of $K$ neighbours without
dilation. As shown, increasing $K$ improves immediately at $K=10$ but does not
gain performance further.

#### Size of dilation factor $d$.

As shown above that performance saturates at $K=10$, we now fix $K$ and vary
the size of dilation factor $d$ in Table 4(b). We find that the combination
$K=10,d=4$ produce the best performance and do not improve by further
increasing $d$. This demonstrates the effectiveness of increasing the
receptive field for contact map estimation.

#### Combining $K$-NN computation.

To further study the effect of separately computing $K$-NN, we experiment with
combined $K$-NN computation with $d=4$ in Table 4(c). It can be seen that the
performance exceed the lower bound of (a) and similar to DGCNN’s performance
in Table 1. This is expected as this is similar to static EdgeConv [55] with
dilation. This shows that separate $K$-NN is crucial for this framework.

Table 4:  Performances of different GCN-Contact design choices on
_ContactPose_ and _HO-3D_. semi refers to semi-supervised learning. We
experiment on (a) number of $K$ neighbours without dilation, (b) size of
dilation factor $d$ with $K=10$ and (c) combining $K$-NN computation (denoted
with *) with $d=4$. Full results are in supplementary.

|  | _ContactPose_ |  _HO-3D_ w/o semi
---|---|---|---
|  | Hand error | Hand error |  Object error
| models |  joint $\downarrow$ |  mesh $\downarrow$ |  joint $\downarrow$ |  mesh $\downarrow$ |  add-$0.1$d ($\uparrow$)
| $K=5$ |  8.252 |  8.134 |  12.69 |  12.86 |  66.33
(a) | $K=10$ |  6.691 |  6.562 |  11.81 |  11.91 |  68.71
| $K=15$ |  6.715 |  6.617 |  11.85 |  11.92 |  68.70
| $d=2$ |  5.959 |  5.865 |  10.91 |  10.86 |  70.25
(b) | $d=4$ |  5.878 |  5.765 |  9.92 |  9.79 |  72.81
| $d=6$ |  5.911 |  5.805 |  9.95 |  9.79 |  72.81
| $K^{*}=5$ |  8.451 |  8.369 |  12.91 |  12.86 |  68.86
(c) | $K^{*}=10$ |  8.359 |  8.251 |  11.55 |  11.97 |  69.10
| $K^{*}=25$ |  8.369 |  8.286 |  11.57 |  11.97 |  69.06

#### Impact of our components.

We study the impact of semi-supervised learning on _HO-3D_. Since the hand
model (MANO) is consistent across datasets, the contact estimator can easily
transfer hand contact to new datasets without re-training. However, it is
insufficient to adapt to unlabelled dataset due to diverse object geometries.
Therefore, we propose a semi-supervised learning method to generate high-
quality pseudo-labels. As shown in Table 5, our method enables performance
boost on both hand and object. The hand joint error is $8.74mm$ while it is
$9.92mm$ without semi-supervised training. Also, the average object ADD-$0.1$D
has a significant $8.56\%$ improvement under S2Contact.

Table 5 shows a quantitative comparison of S2Contact with various filtering
constraints disabled demonstrating that constraints from both visual and
geometry domains are essential for faithful training. We also observed that
disabling $\mathcal{L}_{cont}$ can easily lead to unstable training and
$5.45\%$ performance degradation in object error. In contrast, geometric
consistencies ($\mathcal{L}_{Cham}$ and $\mathcal{L}_{SDF}$) have a comparably
smaller impact on hand and object pose. Despite that they account for less
than $2\%$ performance drop to object error, geometric consistencies are
important for contact (i.e. , more than $5\%$ for contact coverage). The
remaining factor, measuring visual similarity, has a more significant impact.
Disabling visual consistency constraint $\mathcal{L}_{SSIM}$ results in hand
joint error and object error increase by $1mm$ and $8.04\%$, respectively. We
validate that the combination of our pseudo-label filtering constraints are
critical for generating high-quality pseudo-labels and improving hand-object
pose estimation performance. Finally, we provide qualitative examples on out-
of-domain objects in supplementary.

Table 5:  Performances of different filtering constraints under semi-
supervised learning on _HO-3D_. semi refers to semi-supervised learning.

| Hand error |  Object error | Contact
---|---|---|---
models |  joint $\downarrow$ |  mesh $\downarrow$ |  add-0.1d ($\uparrow$) |  cover $\uparrow$ |  inter $\downarrow$
w/o semi |  9.92 |  9.79 |  72.81 |  12.1 |  8.3$\pm$10.5
w/ semi |  8.74 |  8.86 |  81.37 |  19.2 |  3.5$\pm$1.8
w/o $\mathcal{L}_{Cham}$ |  8.53 |  8.48 |  80.11 |  13.1 |  6.9$\pm$6.2
w/o $\mathcal{L}_{SDF}$ |  8.61 |  8.59 |  80.90 |  14.7 |  5.5$\pm$6.0
w/o $\mathcal{L}_{cont}$ |  9.28 |  9.19 |  75.92 |  13.9 |  4.8$\pm$3.1
w/o $\mathcal{L}_{SSIM}$ |  9.71 |  9.57 |  73.33 |  16.3 |  3.7$\pm$2.6

#### Computational analysis.

We report the model parameters and GPU memory cost in Table 4 of supplementary
material. For fair comparisons, all models are tested using a batch size of
$64$. As shown, our model has $2.4$X less the number of learnable model
parameters and $2$X less the GPU memory cost when compared to baseline and
DGCNN [55], respectively. We alleviate the need to keep a high density of
points across the network (DGCNN) while gaining performance.

##  5 Conclusion

In this paper, we have proposed a novel semi-supervised learning framework
which enables learning contact with monocular videos. The main idea behind
this study was to demonstrate that this can successfully be achieved with
visual and geometric consistency constraints for pseudo-label generation. We
designed an efficient graph-based network for inferring contact maps and shown
benefits of combining visual cues and contact consistency constraints to
produce more physically-plausible reconstructions. In the future, we would
like to explore more consistencies over time and or multiple views to further
improve the accuracy.

Acknowledgements. This research was supported by the MSIT (Ministry of Science
and ICT), Korea, under the ITRC (Information Technology Research Center)
support program (IITP–2022–2020–0–01789) supervised by the IITP (Institute of
Information & Communications Technology Planning & Evaluation) and the
Baskerville Tier 2 HPC service (https://www.baskerville.ac.uk/) funded by the
Engineering and Physical Sciences Research Council (EPSRC) and UKRI through
the World Class Labs scheme (EP/T022221/1) and the Digital Research
Infrastructure programme (EP/W032244/1) operated by Advanced Research
Computing at the University of Birmingham. KIK was supported by the National
Research Foundation of Korea (NRF) grant (No. 2021R1A2C2012195) and IITP
grants (IITP–2021–0–02068 and IITP–2020–0–01336). ZQZ was supported by China
Scholarship Council (CSC) Grant No. 202208060266. AL was supported in part by
the EPSRC (grant number EP/S032487/1). FZ was supported by the National
Natural Science Foundation of China under Grant No. 61972188 and 62122035.

## References

  * [1] Brahmbhatt, S., Ham, C., Kemp, C.C., Hays, J.: ContactDB: Analyzing and predicting grasp contact via thermal imaging. In: CVPR (2019)
  * [2] Brahmbhatt, S., Tang, C., Twigg, C.D., Kemp, C.C., Hays, J.: ContactPose: A dataset of grasps with object contact and hand pose. In: ECCV (2020)
  * [3] Cao, Z., Radosavovic, I., Kanazawa, A., Malik, J.: Reconstructing hand-object interactions in the wild. In: ICCV (2021)
  * [4] Chao, Y.W., Yang, W., Xiang, Y., Molchanov, P., Handa, A., Tremblay, J., Narang, Y.S., Van Wyk, K., Iqbal, U., Birchfield, S., et al.: DexYCB: A benchmark for capturing hand grasping of objects. In: CVPR (2021)
  * [5] Chen, W., Jia, X., Chang, H.J., Duan, J., Leonardis, A.: G2L-Net: Global to local network for real-time 6D pose estimation with embedding vector features. In: CVPR (2020)
  * [6] Chen, W., Jia, X., Chang, H.J., Duan, J., Shen, L., Leonardis, A.: FS-Net: Fast shape-based network for category-level 6D object pose estimation with decoupled rotation mechanism. In: CVPR (2021)
  * [7] Chen, Y., Tu, Z., Ge, L., Zhang, D., Chen, R., Yuan, J.: SO-HandNet: Self-organizing network for 3D hand pose estimation with semi-supervised learning. In: CVPR (2019)
  * [8] Corona, E., Pumarola, A., Alenya, G., Moreno-Noguer, F., Rogez, G.: GanHand: Predicting human grasp affordances in multi-object scenes. In: CVPR (2020)
  * [9] Defferrard, M., Bresson, X., Vandergheynst, P.: Convolutional neural networks on graphs with fast localized spectral filtering. In: NeurIPS (2016)
  * [10] Doosti, B., Naha, S., Mirbagheri, M., Crandall, D.J.: HOPE-Net: A graph-based model for hand-object pose estimation. In: CVPR (2020)
  * [11] Garcia-Hernando, G., Yuan, S., Baek, S., Kim, T.K.: First-person hand action benchmark with RGB-D videos and 3D hand pose annotations. In: CVPR (2018)
  * [12] Grady, P., Tang, C., Twigg, C.D., Vo, M., Brahmbhatt, S., Kemp, C.C.: ContactOpt: Optimizing contact to improve grasps. In: CVPR (2021)
  * [13] Guo, M.H., Cai, J.X., Liu, Z.N., Mu, T.J., Martin, R.R., Hu, S.M.: PCT: Point cloud transformer. Computational Visual Media (2021)
  * [14] Hampali, S., Rad, M., Oberweger, M., Lepetit, V.: Honnotate: A method for 3D annotation of hand and object poses. In: CVPR (2020)
  * [15] Han, S., Liu, B., Cabezas, R., Twigg, C.D., Zhang, P., Petkau, J., Yu, T.H., Tai, C.J., Akbay, M., Wang, Z., et al.: MEgATrack: Monochrome egocentric articulated hand-tracking for virtual reality. In: SIGGRAPH (2020)
  * [16] Hasson, Y., Tekin, B., Bogo, F., Laptev, I., Pollefeys, M., Schmid, C.: Leveraging photometric consistency over time for sparsely supervised hand-object reconstruction. In: CVPR (2020)
  * [17] Hasson, Y., Varol, G., Laptev, I., Schmid, C.: Towards unconstrained joint hand-object reconstruction from RGB videos. In: 3DV (2021)
  * [18] Hasson, Y., Varol, G., Tzionas, D., Kalevatykh, I., Black, M.J., Laptev, I., Schmid, C.: Learning joint reconstruction of hands and manipulated objects. In: CVPR (2019)
  * [19] Huang, L., Tan, J., Meng, J., Liu, J., Yuan, J.: HOT-Net: Non-autoregressive transformer for 3D hand-object pose estimation. In: ACM MM (2020)
  * [20] Jiang, H., Liu, S., Wang, J., Wang, X.: Hand-object contact consistency reasoning for human grasps generation. In: ICCV (2021)
  * [21] Karunratanakul, K., Yang, J., Zhang, Y., Black, M.J., Muandet, K., Tang, S.: Grasping field: Learning implicit representations for human grasps. In: 3DV (2020)
  * [22] Kato, H., Ushiku, Y., Harada, T.: Neural 3D mesh renderer. In: CVPR (2018)
  * [23] Kaviani, S., Rahimi, A., Hartley, R.: Semi-Supervised 3D hand shape and pose estimation with label propagation. arXiv preprint arXiv:2111.15199 (2021)
  * [24] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: ICLR (2015)
  * [25] Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. In: ICLR (2017)
  * [26] Kwon, T., Tekin, B., Stühmer, J., Bogo, F., Pollefeys, M.: H2O: Two hands manipulating objects for first person interaction recognition. In: ICCV (2021)
  * [27] Labbé, Y., Carpentier, J., Aubry, M., Sivic, J.: CosyPose: Consistent multi-view multi-object 6D pose estimation. In: ECCV (2020)
  * [28] Li, G., Muller, M., Thabet, A., Ghanem, B.: DeepGNSs: Can GCNs go as deep as CNNs? In: ICCV (2019)
  * [29] Li, Y., Wang, G., Ji, X., Xiang, Y., Fox, D.: DeepIM: Deep iterative matching for 6D pose estimation. In: ECCV (2018)
  * [30] Lin, Z.H., Huang, S.Y., Wang, Y.C.F.: Convolution in the cloud: Learning deformable kernels in 3D graph convolution networks for point cloud analysis. In: CVPR (2020)
  * [31] Liu, S., Jiang, H., Xu, J., Liu, S., Wang, X.: Semi-supervised 3D hand-object poses estimation with interactions in time. In: CVPR (2021)
  * [32] Liu, Z., Hu, H., Cao, Y., Zhang, Z., Tong, X.: A closer look at local aggregation operators in point cloud analysis. In: ECCV (2020)
  * [33] Maturana, D., Scherer, S.: VoxNet: A 3D convolutional neural network for real-time object recognition. In: IROS (2015)
  * [34] Monti, F., Boscaini, D., Masci, J., Rodola, E., Svoboda, J., Bronstein, M.M.: Geometric deep learning on graphs and manifolds using mixture model CNNs. In: CVPR (2017)
  * [35] Mueller, F., Bernard, F., Sotnychenko, O., Mehta, D., Sridhar, S., Casas, D., Theobalt, C.: GANerated hands for real-time 3D hand tracking from monocular RGB. In: CVPR (2018)
  * [36] Mueller, F., Davis, M., Bernard, F., Sotnychenko, O., Verschoor, M., Otaduy, M.A., Casas, D., Theobalt, C.: Real-time pose and shape reconstruction of two interacting hands with a single depth camera. In: SIGGRAPH (2019)
  * [37] Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., Devito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A.: Automatic Differentiation in Pytorch. In: NeurIPS (2017)
  * [38] Qi, C.R., Su, H., Mo, K., Guibas, L.J.: PointNet: Deep learning on point sets for 3D classification and segmentation. In: CVPR (2017)
  * [39] Qi, C.R., Yi, L., Su, H., Guibas, L.J.: PointNet++: Deep hierarchical feature learning on point sets in a metric space. In: NeurIPS (2017)
  * [40] Qian, G., Hammoud, H., Li, G., Thabet, A., Ghanem, B.: ASSANet: An anisotropic separable set abstraction for efficient point cloud representation learning. NeurIPS (2021)
  * [41] Romero, J., Tzionas, D., Black, M.J.: Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions on Graphics (ToG) (2017)
  * [42] Simon, T., Joo, H., Matthews, I., Sheikh, Y.: Hand keypoint detection in single images using multiview bootstrapping. In: CVPR (2017)
  * [43] Spurr, A., Molchanov, P., Iqbal, U., Kautz, J., Hilliges, O.: Adversarial motion modelling helps semi-supervised hand pose estimation. arXiv preprint arXiv:2106.05954 (2021)
  * [44] Spurr, A., Song, J., Park, S., Hilliges, O.: Cross-modal deep variational hand pose estimation. In: CVPR (2018)
  * [45] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: CVPR (2015)
  * [46] Taheri, O., Ghorbani, N., Black, M.J., Tzionas, D.: GRAB: A dataset of whole-body human grasping of objects. In: ECCV (2020)
  * [47] Tang, D., Chang, H.J., Tejani, A., Kim, T.K.: Latent regression forest: Structured estimation of 3D articulated hand posture. In: CVPR (2014)
  * [48] Tang, D., Yu, T.H., Kim, T.K.: Real-time articulated hand pose estimation using semi-supervised transductive regression forests. In: ICCV (2013)
  * [49] Tekin, B., Bogo, F., Pollefeys, M.: H+O: Unified egocentric recognition of 3D hand-object poses and interactions. In: CVPR (2019)
  * [50] Tse, T.H.E., De Martini, D., Marchegiani, L.: No need to scream: Robust sound-based speaker localisation in challenging scenarios. In: ICSR (2019)
  * [51] Tse, T.H.E., Kim, K.I., Leonardis, A., Chang, H.J.: Collaborative learning for hand and object reconstruction with attention-guided graph convolution. In: CVPR (2022)
  * [52] Ueda, E., Matsumoto, Y., Imai, M., Ogasawara, T.: A hand-pose estimation for vision-based human interfaces. IEEE Transactions on Industrial Electronics (2003)
  * [53] Wang, H., Cong, Y., Litany, O., Gao, Y., Guibas, L.J.: 3DIoUMatch: Leveraging IoU prediction for semi-supervised 3D object detection. In: CVPR (2021)
  * [54] Wang, J., Mueller, F., Bernard, F., Sorli, S., Sotnychenko, O., Qian, N., Otaduy, M.A., Casas, D., Theobalt, C.: RGB2Hands: real-time tracking of 3D hand interactions from monocular RGB video. In: SIGGRAPH (2020)
  * [55] Wang, Y., Sun, Y., Liu, Z., Sarma, S.E., Bronstein, M.M., Solomon, J.M.: Dynamic graph CNN for learning on point clouds. In: SIGGRAPH (2019)
  * [56] Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13(4), 600–612 (2004)
  * [57] Wu, W., Qi, Z., Fuxin, L.: PointConv: Deep convolutional networks on 3D point clouds. In: CVPR (2019)
  * [58] Xiang, Y., Schmidt, T., Narayanan, V., Fox, D.: PoseCNN: A convolutional neural network for 6D object pose estimation in cluttered scenes. In: RSS (2018)
  * [59] Xu, M., Ding, R., Zhao, H., Qi, X.: PAConv: Position adaptive convolution with dynamic kernel assembling on point clouds. In: CVPR (2021)
  * [60] Yang, J., Chang, H.J., Lee, S., Kwak, N.: SeqHAND: RGB-sequence-based 3D hand pose and shape estimation. In: ECCV (2020)
  * [61] Yang, L., Chen, S., Yao, A.: SemiHand: Semi-supervised hand pose estimation with consistency. In: ICCV (2021)
  * [62] Yang, L., Zhan, X., Li, K., Xu, W., Li, J., Lu, C.: CPF: Learning a contact potential field to model the hand-object interaction. In: ICCV (2021)
  * [63] You, H., Feng, Y., Ji, R., Gao, Y.: PVNet: A joint convolutional network of point cloud and multi-view for 3D shape recognition. In: ACM Multimedia (2018)
  * [64] Zhang, T., McCarthy, Z., Jow, O., Lee, D., Chen, X., Goldberg, K., Abbeel, P.: Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In: ICRA (2018)
  * [65] Zhao, H., Jiang, L., Jia, J., Torr, P.H., Koltun, V.: Point transformer. In: ICCV (2021)
  * [66] Zheng, L., Leonardis, A., Tse, T.H.E., Horanyi, N., Chen, H., Zhang, W., Chang, H.J.: TP-AE: Temporally primed 6D object pose tracking with auto-encoders. In: ICRA (2022)
  * [67] Zimmermann, C., Brox, T.: Learning to estimate 3D hand pose from single RGB images. In: ICCV (2017)


# Attention-Based Ensemble Pooling for
Time Series Forecasting

Dhruvit Patel  Alexander Wikner

###### Abstract

A common technique to reduce model bias in time-series forecasting is to use
an ensemble of predictive models and pool their output into an ensemble
forecast. In cases where each predictive model has different biases, however,
it is not always clear exactly how each model forecast should be weighed
during this pooling. We propose a method for pooling that performs a weighted
average over candidate model forecasts, where the weights are learned by an
attention-based ensemble pooling model. We test this method on two time-series
forecasting problems: multi-step forecasting of the dynamics of the non-
stationary Lorenz ‘63 equation, and one-step forecasting of the weekly
incident deaths due to COVID-19. We find that while our model achieves
excellent valid times when forecasting the non-stationary Lorenz ‘63 equation,
it does not consistently perform better than the existing ensemble pooling
when forecasting COVID-19 weekly incident deaths.

Machine Learning, ICML

##  1 Introduction and Related Work

The problem of forecasting the future time evolution of a system is at the
heart of many disciplines. Practitioners attempting to forecast a system’s
evolution typically use a knowledge-based model, a machine-learning-based or
statistical model, or a model which combines the two. The use of any model
comes with a set of uncertainties and inductive biases associated with that
model, e.g., due to how the internal parameters of the model are prescribed
and/or constrained, and this may introduce a bias and/or variability into the
model forecast. One commonly employed method to mitigate this is to initialize
an ensemble of candidate forecasting models and then to aggregate their
individual forecasts into a single ensemble forecast, e.g., by using a
(un-)weighted average. Ensemble methods have been shown to improve predictive
performance in a variety of tasks (Sagi & Rokach, 2018; Dong et al., 2020;
Zhou, 2021). However, there remains an open problem, especially when dealing
with ensemble learning for forecasting time series: what is the optimal way to
pool the individual forecasts of the candidate models into a single ensemble
forecast?

A large amount of literature exists discussing methods for ensemble learning.
We focus on the specific task of using ensemble learning for time series
forecasting. Various previous works have focused on constructing an ensemble
pooling module which combines the forecasts of candidate models using either a
simple or weighted average (In & Jung, 2022; Montero-Manso et al., 2020;
Vaiciukynas et al., 2021; Gastinger et al., 2021; Wu & Levinson, 2021). In the
cases where a weighted average is used, the weights are determined either by
using a statistical model (Vaiciukynas et al., 2021) or by a trainable module
(Montero-Manso et al., 2020). In either case, the weights assigned to the
different members are typically fixed during the forecasting phase. From the
perspective of time series forecasting, where the underlying system evolution
may be changing with time in a way such that the different members of the
ensemble may perform better or worse depending on the local system dynamics,
this is highly disadvantageous. It is desirable to have an ensemble pooling
module which can, at every step of the forecast, assign a weight to each
candidate model depending on how well that model has performed in the past
when similar dynamics were observed. Such an approach would allow us to best
leverage the beneficial aspects of the different candidate models.

We propose an attention-based ensemble pooling method which adaptively assigns
weights to the different candidate models at each step of the forecast, and we
apply this method to two test cases: (1) forecasting the $3$-dimensional
Lorenz system, and (2) forecasting the weekly incident deaths during the
COVID-19 pandemic.

In addition, we emphasize that we are considering the situation in which we
have access to an ensemble of candidate models for forecasting, but we do not
have the ability to re-tune or update the candidate models themselves. This
excludes ensemble learning methods such as multi-expert learning (Yang et al.,
2020), which rely on jointly training the individual ensemble members and the
ensemble pooling network.

##  2 Methods

We propose an ensemble pooling method that performs a weighted average of $M$
candidate models where the weights are chosen adaptively by an attention
mechanism based on a representation of the current state of the system (the
query) and the set of representations of the candidate models’ forecasts (the
keys and the values). Below we describe the single-head and multi-head
additive attention mechanisms we use in the experiments of this paper. We then
describe two modes of operation for our method: (1) the “open-loop”
configuration, and (2) the ”closed-loop” configuration.

###  2.1 Single-head Additive Attention

The scalar additive attention weight at time index $j$ for $j=1,...,T$ between
the query $\textbf{q}_{j}\in\mathbb{R}^{q}$ and the $i^{th}$ key
$\textbf{k}_{j}^{(i)}\in\mathbb{R}^{k}$ (in our case, referring to the
representation of the forecast of the $i^{th}$ of $M$ candidate models in the
ensemble) is given by

$\displaystyle
a_{i,j}=\text{softmax}\Big{[}\textbf{w}_{v}^{\top}\tanh(\textbf{W}_{q}\textbf{q}_{j}+\textbf{W}_{k}\textbf{k}_{j}^{(i)}+\textbf{b})\Big{]},$
(1)

where $\textbf{W}_{q}\in\mathbb{R}^{h\times q}$,
$\textbf{W}_{k}\in\mathbb{R}^{h\times k}$, $\textbf{w}_{v}\in\mathbb{R}^{h}$
and $\textbf{b}\in\mathbb{R}^{h}$ are trainable parameters, and $h$ is the
number of hidden units. Our single-head additive attention model ensemble
forecast at time index $j$, $\hat{\textbf{y}}_{j}\in\mathbb{R}^{d}$, is then
formed as linear combination of the candidate model forecasts weighted by
their respective attention weights,

$\displaystyle\hat{\textbf{y}}_{j}=\sum_{i=1}^{M}a_{i,j}F_{j}^{(i)},$ (2)

where $F_{j}^{(i)}$ is the $i^{th}$-model’s forecast for time index $j$ given
the system state at time index $j-1$.

###  2.2 Multi-head Additive Attention

In many cases, we may want to use a single set of queries, keys, and values to
learn multiple behaviors of the target system or to predict multiple
quantities simultaneously. In the multi-head attention scheme, we consider $P$
copies of the above described single-head attention block and the set of their
corresponding forecasts $\hat{\textbf{y}}^{(i)}_{j}$ for $i=1,...,P$ at time
index $j$. The final forecast is obtained by a linear transformation of the
concatenation of forecasts from the $P$ heads,

$\displaystyle\hat{\textbf{y}}_{j}=\textbf{W}_{0}\big{[}\hat{\textbf{y}}^{(1)}_{j},...,\hat{\textbf{y}}^{(P)}_{j}\big{]},$
(3)

where $\textbf{W}_{0}\in\mathbb{R}^{d\times(d\times P)}$ is a matrix of
trainable parameters.

###  2.3 Time-Delayed Embedding of the Queries and Keys

Since we consider the task of time series forecasting, which has an inherent
sequential structure, we formulate the queries and keys by using time-delayed
embedding to provide context of the most recent past dynamics. In particular,
for a time-delay embedding with $l$ steps into the past, we formulate the
queries and keys at time $j$ as the following concatenations:

$\displaystyle\begin{split}\textbf{q}^{\prime}_{j}=\big{[}\textbf{q}_{j},\textbf{q}_{j-1},...,\textbf{q}_{j-l}\big{]}\\\
\textbf{k}^{\prime(i)}_{j}=\big{[}\textbf{k}^{(i)}_{j},\textbf{k}^{(i)}_{j-1},...,\textbf{k}^{(i)}_{j-l}\big{]}.\end{split}$
(4)

We note that the time-delayed embedding is meant to provide memory of recent
dynamics and can be replaced by a different method that can provide a similar
context of past dynamics, e.g., embedding via a recurrent neural network.

###  2.4 Modes of Operation

We utilize our ensemble forecast method in two different modes of operation:
(1) the “open-loop”, and (2) the “closed-loop”. In the open-loop
configuration, the ground truth data at time index $j-1$ is known and used to
obtain the queries $\textbf{q}_{j}$ and the values $F_{j}^{(i)}$ at time index
$j$. The ensemble forecast is then made according to Eq. 2 (Eq. 3) for single-
head (multi-head) attention. This mode of operation requires that the “ground
truth” data be available for the time period over which this mode is operated.
In the closed-loop configuration, the queries $\textbf{q}_{j}$ are formulated
and the values $F_{j}^{(i)}$ are obtained using the ensemble model forecast at
the previous time step $\textbf{y}_{j-1}$. In other words, the one-step
forecasts of the ensemble model at time index $j-i$ are recycled as inputs to
the model at time index $j$. This results in autonomous operation of the
ensemble model; thus, operation of this mode only requires access to the
ground truth data at initialization. We refer to forecasts made in this
configuration as “multi-step” forecasts.

Training is performed in the open-loop configuration (i.e., we train our
ensemble method on one-step forward forecasts), while testing is performed in
either the open-loop or closed-loop configuration, depending on the task.

##  3 Results

###  3.1 Non-Stationary Lorenz ‘63 Equations

####  3.1.1 Description of True Equations and Ensemble of Candidate Models

As a toy system to demonstrate the potential of our proposed attention-based
ensemble pooling model, we consider the Lorenz ‘63 equations (Lorenz, 1963):

$\displaystyle\begin{split}\frac{du_{1}(t)}{dt}&=\sigma(u_{2}(t)-u_{1}(t)),\\\
\frac{du_{2}(t)}{dt}&=u_{1}(t)(\rho(t)-u_{3}(t))-u_{2}(t),\\\
\frac{du_{3}(t)}{dt}&=u_{1}(t)u_{2}(t)-\beta u_{3}(t).\end{split}$ (5)

The Lorenz ‘63 equations are a simplified 2D model of atmospheric convection.
The system state variables $u_{1}(t),u_{2}(t)$, and $u_{3}(t)$ (which we can
represent as the vector $\textbf{u}(t)=[u_{1}(t),u_{2}(t),u_{3}(t)]$)
represent the rate of convection, the horizontal temperature variation, the
vertical temperature variation, respectively, while the model parameters
$\sigma,\rho(t),$ and $\beta$ represent the Prandtl number, Rayleigh number,
and the size of the system, respectively (Sparrow, 1982). For the standard
stationary choice of model parameter values ($\sigma=10$, $\rho(t)=28$, and
$\beta=8/3$), the dynamics of the Lorenz equations are chaotic. We integrate
all sets of Lorenz equations using a $4^{th}$-Order Runge-Kutta integration
scheme (Press, 2007) with an integration time step of $0.01$. We choose a
trajectory sampling time step of $\Delta t=0.1$, meaning that we integrate the
equations $10$ times before recording the output to form our time series data.

Figure 1: An example trajectory of the Lorenz equations with a non-stationary
$\rho(t)$ prescribed by Eq. 6. The $\rho(t)$ values are scaled so that they
may be referenced against the dynamics.

For the true non-stationary dynamics which we will predict using our ensemble
pooling method, we set $\sigma$ and $\beta$ to their standard values but allow
$\rho(t)$ to vary sinusoidally as follows:

$\displaystyle\rho(t)_{True}=38-10\cos(2\pi t/T).$ (6)

Here, $T$ is the oscillation period. For our tests, we choose an oscillation
period of $T=1.577132$. This period was chosen so that $\rho(t)$ oscillates on
a similar time scale to the Lorenz state variables and so that $T$ is not
evenly divisible by our chosen sampling time step. This is done so that when
we evenly sample initial conditions from a long trajectory for validation, we
do not always sample initial conditions obtained with the same initial value
of $\rho(t)$. Figure 1 shows an example trajectory of the Lorenz equations
with this non-stationary $\rho(t)$. We note that the true dynamics remain
chaotic throughout this parameter oscillation.

For our ensemble of candidate models, we use an ensemble of stationary Lorenz
equations with the same standard values of $\sigma$ and $\beta$ and the
following $M=11$ evenly-spaced values of $\rho(t)$:

$\displaystyle\\{\rho(t)_{m,ens}\\}=\\{28,30,\dots,46,48\\}.$ (7)

####  3.1.2 Models Tested, Training, and Evaluation

For the attention-based ensemble pooling models tested, we form the queries
and keys using the previous true system states and the previous difference
between the candidate model one-step forecasts and these true states:

$\displaystyle\begin{split}\textbf{q}_{j}&=\textbf{u}_{j-1},\\\
\textbf{k}_{j}&=[F_{j-1}^{(1)}-\textbf{u}_{j-1},\dots,F_{j-1}^{(M)}-\textbf{u}_{j-1}].\end{split}$
(8)

Here, $\textbf{u}_{j}=\textbf{u}(j\Delta t)$. For cases when $l>1$, we form
the time-delay embedded queries and keys as described in Eq. 4.

We consider the multi-step forecast task for the non-stationary Lorenz
equations using the following ensemble pooling techniques:

  1. 1.

Linear regression on the candidate model one-step forecasts (i.e.,
$\hat{\textbf{y}}_{j}=\textbf{W}[F_{j}^{(1)},\dots,F_{j}^{(M)}]+\textbf{b}$).

  2. 2.

Our additive attention ensemble pooling model with an approximately fixed
number of trainable parameters. (The size of the hidden dimension, $h$, is
inversely scaled by the number of time delay inputs, $l$). We test this method
using the following configurations:

    1. (a)

The multi-step forecast where the model receives a time delay input of length
$l=1,2,3,4,5$ or $6$ and where the attention weights are re-computed after
each one-step forecast (referred to as “Additive attention”).

    2. (b)

The multi-step forecast from the component model which receives the largest
attention weight at the beginning of the forecast (referred to as “Best
initial model”).

    3. (c)

The multi-step forecast where the attention weights are fixed to their values
at the beginning of the forecast (referred to as “Fixed attention”).

We also compare with a forecast from a single-hidden-layer feed-forward neural
network (FF NN) with a $\tanh$ activation function and approximately the same
number of trainable parameters as used in our additive attention model. As in
our additive attention ensemble pooling model, we denote the size of its
hidden dimension as $h$. This model receives only the time delay vector of
previous system states ($\textbf{q}^{\prime}_{j}$ in Eq. 4 using
$\textbf{q}_{j}$ from Eq. 13) as input.

All ensemble pooling models and the FF NN are trained to minimize the mean-
squared error between the forecast $\hat{\textbf{y}}_{j}$ and the true state
$\textbf{y}_{j}=\textbf{u}_{j}$. The training data is a single trajectory of
the non-stationary Lorenz equations of duration $t_{train}=400$. Given our
sampling time step of $\Delta t=0.1$, our training data sequence thus contains
$4000$ samples. Each of the ensemble pooling models and the FF NN are trained
using an Adam optimizer (Kingma & Ba, 2017) with a learning rate of $10^{-3}$.
The ensemble pooling methods are trained over $500$ epochs, while the FF NN is
trained over $800$ epochs. We validate our forecasts on $200$ validation
trajectories evenly sampled from the full validation trajectory of duration
$t_{val}=2560$. Each validation trajectory thus has a duration of
$t_{val,i}=12.8$ and contains $128$ samples.

Figure 2: The median valid time of forecasts obtained from each of the
ensemble pooling methods tested and from the FF NN. The black error bars on
each bar show the maximum of the upper and lower 95% confidence bounds for the
median (Conover, 1999) computed over all $200$ validation trajectories.

####  3.1.3 forecast Valid Time

We evaluate our ensemble forecast performance on the validation trajectories
using the valid time of the multi-step forecast, defined as the length of time
from the start of the forecast to the time when the mean-squared error in the
forecast first exceeds a chosen threshold:

$\displaystyle VT=\max_{j\geq 0}\Big{\\{}j\Delta
t\>\big{|}\>\lVert\hat{\textbf{y}}_{j}-\textbf{y}_{j}\rVert^{2}_{2}<\epsilon_{VT}\Big{\\}}.$
(9)

The mean-squared error between true trajectories with different initial
conditions saturates at approximately $200$. We choose an error threshold of
$\epsilon_{VT}=40$.

Figure 2 shows the median valid time computed over all 200 validation
trajectories. Our linear regression ensemble pooling model and our FF NN
predictive model do not perform well, only achieving a median valid time of
$0.20$ and $0.40$, respectively. Our additive attention ensemble pooling model
performs about as poorly as the other methods when the model receives only the
most recent model forecast errors and system state as input ($l=1$). However,
when a longer time delay input is used ($l>1$), our additive attention model
performs substantially better than all other methods, improving from a median
valid time of $0.30$ for $l=1$ to $2.60$ for $l=5$ (though we do not see any
improvement from $l=5$ to $l=6$ for the particular total number of trainable
weights used). We only see this substantial performance increase when the
multi-step forecast re-computes the attention weights at each forecast step.
The corresponding best initial model and fixed attention model do not improve
at all from $l=1$ to $l=5$, and the $l=5$ forecast performs only as well as
the additive attention model with $l=1$. These results indicate that our
additive attention ensemble pooling model’s ability to change which component
models to use during a multi-step forecast is key to obtaining good forecast
performance. In addition, we see that sufficient knowledge of past states and
component model error is needed in order for our ensemble pooling model to
perform well.

####  3.1.4 Attention Weights during Multi-Step forecast

Figure 3: The attention weights assigned to each of the candidate models by
our additive attention ensemble pooling model with $h=120$ and $l=5$. Each
vertical black line (shown at $t=0,3,6,9,$ and $12$) marks where the forecast
is re-initialized using the true validation data. The model then performs a
multi-step forecast until the next re-initialization. The weight of each
candidate model is given by the color axis, while the dashed blue line shows
the true value of $\rho(t)$.

Figure 3 shows the attention weights assigned to each of the candidate models
with different stationary $\rho(t)$ values during $5$ multi-step forecasts of
a validation trajectory, each of duration $3$, made by our additive attention
ensemble pooling model with $h=120$ and $l=5$. We see that during these
forecasts, the true $\rho(t)$ value and the $\rho(t)$ value of the candidate
model assigned the highest weight are very similar. This indicates that even
during a multi-step forecast where, after $l$ steps, the model no longer
receives any inputs from the true data, our model is still able to anticipate
the correct change in $\rho(t)$ from the past one-step forecasts.

###  3.2 Weekly Incident Deaths from Covid-19

####  3.2.1 Description of the COVID-19 data set and the evaluation metric

In order to test our method on a real-world data set, we consider the task of
predicting the weekly incident number of deaths ($\Delta t=$ 1 week) during
the COVID-19 pandemic using data provided by the COVID-19 Forecast Hub (Cramer
et al., 2021), founded in March 2020 by the Reich Lab of the University of
Massachusetts. The COVID-19 Forecast Hub maintains an up-to-date record for
forecasts of COVID-19 cases, hospitalizations, and deaths in the US, created
by dozens of leading infectious disease modeling teams around the world, in
coordination with the US CDC. Participating teams submit a probabilistic
forecast for future days, weeks, and months for the county, state and national
levels in the US. Each forecast consists of a set of quantiles of the
predictive distribution at prescribed levels (e.g., 10%, 20%, … 95%, 98%). In
addition to providing the forecasts of the different participating teams, the
COVID-19 Forecast Hub also provides an ensemble forecast which it generates by
aggregating the forecasts of the different participating teams. While the
method of ensemble pooling used by the COVID-19 Forecast Hub has varied over
the course of the pandemic, the most current method performs a weighted
ensemble average of the models with the best performance, as measured by their
Weighted Interval Score (WIS) (Gneiting & Raftery, 2007), in the 12 weeks
prior to the forecast date. The component models are assigned weights that are
a function of their relative WIS during those 12 weeks.

The WIS (Bracher et al., 2021; Gneiting & Raftery, 2007) is a proper scoring
rule which assesses the quality of a probabilistic forecast (described by
quantiles of the predictive cumulative distribution $P$) based on the
predictive cumulative distribution and on the observed event. For an observed
event $y$, and for the quantile forecast at level $\alpha$, the Interval Score
(IS) is given by:

$IS_{\alpha}(P,y)=(u-l)+2/\alpha\times(l-y)\times\textbf{1}(y<l)+\\\
2/\alpha\times(y-u)\times\textbf{1}(y>u),$ (10)

where 1 is the indicator function which equals $1$ if its argument is true and
$0$ otherwise. The terms $l$ and $u$ denote the $\alpha/2$ and $1-\alpha/2$
quantiles of $P$. The WIS is calculated for a probabilistic forecast that
provides the forecast at several $\alpha_{i}$ levels for $i=1,...,K$,

$WIS_{\alpha_{0}:K}(P,y)=\frac{1}{K+1/2}\times\Bigl{(}w_{0}\times|y-m|+\\\
\sum_{k=1}^{K}\\{w_{k}\times IS_{\alpha_{k}}(P,y)\\}\Bigr{)},$ (11)

where $w_{i}$ are the weights, which we set to $w_{i}=\alpha_{i}/2$ (and
$w_{0}=1/2$). The WIS is a _negatively oriented_ score, meaning that the lower
the score the higher the quality of the probabilistic forecast, and (for the
above choice of weights $w_{i}$) approximates the Continuous Ranked
Probability Score (Bracher et al., 2021; Gneiting & Ranjan, 2011).

####  3.2.2 COVID-19 Candidate Model Selection and Data Pre-processing

A major challenge presented by the COVID-19 data set is that of missing data.
While many teams have participated in the COVID-19 forecast challenge hosted
by the COVID-19 Forecast Hub since its creation in March $2020$ to the
present, most teams have not submitted forecasts for this entire span of time.
In addition, many teams have not submitted forecasts consistently week-to-week
after their first submission. Due to this, we have chosen as candidate models
the $M=9$ models which have submitted forecasts of weekly incident deaths for
every US state and DC most consistently from May $2020$ through October
$2022$. For any missing forecasts in this period, the weekly incident deaths
for a given team at a given location and for a given quantile is filled in by:

  1. 1.

Linear interpolation between weekly forecasts if the gap of missing data spans
less than three weeks.

  2. 2.

Using the candidate model ensemble mean if the gap of missing data spans more
than two weeks, and if other candidate models in the ensemble have made
forecasts for this time.

  3. 3.

Using the nearest past or future value if the gap of missing data spans more
than two weeks, and if no other model in the ensemble has made a forecast for
this time.

We use the following $21$ quantiles from each candidate model forecast for
every US state and DC:

$\displaystyle\begin{split}F_{j}^{(i)}=[&F_{j,\alpha_{1}}^{(i)},\dots,F_{j,\alpha_{21}}^{(i)}],\text{
where}\\\ \\{\alpha\\}=\\{&0.5,0.01,0.025,0.05,0.1,0.15,0.2,0.25,\\\
&0.3,0.35,0.4,0.45,0.55,0.6,0.65,0.7,\\\
&0.75,0.8,0.85,0.9,0.95,0.975,0.99\\}.\end{split}$ (12)

The forecasts made by each of the ensemble pooling models we will test will
predict these same quantiles.

####  3.2.3 Models Tested, Training, and Evaluation

For the attention-based ensemble pooling models tested, the queries are formed
using the previous true number of weekly incident deaths, which we denote as
$u_{j-1}$. The keys are formed using the difference between the candidate
models’s previous median forecasts ($\alpha_{1}=0.5$) and the previous number
of weekly incident deaths, in addition to all other candidate model forecast
quantiles. We therefore have:

$\displaystyle\begin{split}\textbf{q}_{j}=&u_{j-1},\\\
\textbf{k}_{j}=&[F_{j-1,\alpha_{1}}^{(1)}-u_{j-1},F_{j-1,\alpha_{2}}^{(1)},\dots,F_{j-1,\alpha_{21}}^{(1)},\dots,\\\
&F_{j-1,\alpha_{1}}^{(M)}-u_{j-1},F_{j-1,\alpha_{2}}^{(M)},\dots,F_{j-1,\alpha_{21}}^{(M)}].\end{split}$
(13)

For cases when $l>1$, we form the time-delay embedded queries and keys as
described in Eq. 4. We note that we also tried using different queries and
keys, including adding the number of incident COVID-19 cases to the queries
and the previous WIS values for each candidate model forecast to the keys. We
found, however, that these additions did not consistently improve performance,
so we have chosen to omit them.

Since we only have access to individual model forecasts and not the models
themselves, we consider the one-step forecast task for the COVID-19 weekly
incident deaths using the following ensemble pooling models:

  1. 1.

Linear regression on the $l=5$ most-recent candidate model one-step forecasts:

$\displaystyle\begin{split}\hat{\textbf{y}}_{j}=\textbf{W}[&F_{j}^{(1)},\dots,F_{j-(l-1)}^{(1)},\dots,\\\
&F_{j}^{(M)},\dots,F_{j-(l-1)}^{(M)}]+\textbf{b}.\end{split}$ (14)

  2. 2.

Our additive attention ensemble pooling model with $h=1000$ and $l=5$.

  3. 3.

Our multi-head attention ensemble pooling model with $h=100$, $l=5$, and
$p=21$.

We compare the resulting ensemble forecasts with the COVID-19 Forecast Hub
ensemble forecast. We note that while the ensemble pooling methods we test use
only the $M=9$ candidate models with the most forecast data, the COVID-19
Forecast Hub ensemble forecast may make use of any of the $124$ candidate
models for which it has forecasts from at a particular time.

Figure 4: The validation periods used to evaluate the COVID-19 weekly incident
death ensemble forecasts. The black line shows the total US weekly incident
deaths, and each blue box bounds a particular validation period.

We validate our ensemble forecasts within the 4 periods shown in Fig. 4 on
data from every US state and DC. For each validation period, we train each of
our ensemble pooling models using the ground truth and candidate model
forecast data from outside of this period from every US state and DC. The
models are trained using an Adam optimizer with a learning rate of $10^{-5}$
over $200$ epochs. For the Additive and Multi-head attention models, we use a
small weight decay of $10^{-4}$ and $10^{-5}$, respectively.

The loss function for each of our ensemble pooling models is the mean WIS
computed using the $\alpha$ values

$\displaystyle\begin{split}\\{\alpha_{WIS}\\}=\\{&0.02,0.05,0.1,0.3,0.4,\\\
&0.5,0.6,0.7,0.8,0.9\\},\end{split}$ (15)

and with the observed event $y=u_{j}$, the true observed weekly incident
deaths. One can see from Eq. 11 that the $21$ forecast quantiles produced by
our models (Eq. 12) are sufficient to compute the WIS for these $\alpha$
values. We will evaluate the accuracy of our forecasts during validation using
this WIS.

####  3.2.4 Results

| Mean WIS over All States
---|---
| Validation
---
Period
| 08/29/2020
---
-
02/20/2021
| 06/05/2021
---
-
10/30/2021
| 12/21/2021
---
-
06/18/2022
| 07/09/2022
---
-
11/05/2022
| COVID-19
---
Forecast Hub
Ensemble
35.12 | 22.13 | 32.86 | 12.60
| Linear
---
Regression
35.75 | 20.42 | 35.98 | 13.80
| Additive
---
Attention
38.27 | 20.72 | 35.51 | 12.92
| Multi-head
---
Attention
36.88 | 20.86 | 34.74 | 12.94

Table 1: Mean WIS values computed by averaging forecast WIS over dates in each
validation period and over forecasts for all states and DC. Bold text numbers
mark the lowest WIS ensemble forecast for that particular validation period.

We show the mean WIS computed over all states and DC for each validation
period in Table 1. We see that while our attention-based models do obtain a
smaller WIS than the COVID-19 Forecast Hub ensemble during the summer 2021
validation period, the linear regression ensemble pooling model actually
performs better than either of the attention-based models during this period.
This improved performance over the attention-based models holds for the first
two validation periods, but not for the final two. In these last two periods,
the linear regression model has the highest mean WIS. Though our attention-
based models do perform better than linear regression in these periods, they
are not obtain a smaller WIS than the COVID-19 Forecast Hub ensemble. The same
is true for all of the ensemble pooling models we tested in the earliest
validation period.

Figure 5: Each panel shows the WIS for forecasts of the weekly incident deaths
in Maryland made for each week during each of the four validation periods.
Each panel also shows the ground truth number of weekly incident deaths. The
scale for the WIS is shown along each right vertical axis, while the scale for
the number of weekly incident deaths is shown along each left vertical axis.
The target forecast week is shown along each horizontal axis. The legend in
each panel shows the mean WIS computed over the corresponding validation
period.

As an example of our ensemble pooling model performance in a single state, we
show the WIS for forecasts of the weekly incident deaths in Maryland over time
in each validation period in Fig. 5. The overall performance is similar to
that averaged over all states; while the additive attention ensemble pooling
model has a lower mean WIS than the other models during the validation period
beginning from 08/29/2020, it does not during any of the other periods. In
addition, none of the ensemble pooling models that we trained consistently
have a lower mean WIS than the other over all of the validation periods. From
these panels, one can see that the mean WIS is heavily weighted towards the
model’s performance during a wave, when the number of weekly incident deaths
is high. For example, looking at week 3 in the panel beginning from
12/18/2021, we see that the WIS for each of the ensemble pooling models we
trained is substantially higher than the COVID-19 Forecast Hub ensemble.
Despite the fact that the models all perform similarly before and after this
difference, the COVID-19 Forecast Hub ensemble still has a much smaller mean
WIS than the other models due to this short but sharp performance difference.

##  4 Conclusion

We propose a method for creating an ensemble forecast for systems where one
has access to a set of candidate models and ground truth time series data. Our
proposed method combines the candidate model forecasts using a weighted
average, where the weights are obtained by training an attention-based machine
learning model. We evaluate the performance of this model on two test cases:
the dynamics of the non-stationary Lorenz ‘63 equations, and the weekly
incident deaths from COVID-19.

We find that our single-head attention model’s ensemble forecasts of the non-
stationary Lorenz ‘63 equations have a substantially longer median valid time
than the other ensemble pooling models tested. In addition, our model is able
to correctly select the candidate model with the closest internal model
parameter to the true system during a multi-step forecast, even after the
model no longer receives any input from the ground truth data.

On the other hand, we find that neither our single-head nor our multi-head
attention models consistently improve the mean WIS of one-step forecasts of
the weekly incident deaths due to COVID-19 over the COVID-19 Forecast Hub
ensemble forecast. There are a number of reasons why this may be the case: the
COVID-19 Forecast Hub ensemble uses a far greater number of candidate models
than we were able to use in the ensemble pooling models we trained. In
addition, the scheme we used to fill in missing candidate model forecast data
may have introduced unrealistic biases. The ground truth data may also suffer
from erroneous reporting of deaths or from deaths which are reported later
than they actually occurred, making it more difficult to learn when particular
candidate models should be used. Finally, our attenion-based ensemble pooling
model assumes that the biases of the candidate models are not changing, or do
not change much, over time; the models on the COVID-19 Forecast Hub, however,
have been continuously updated since they began submitting forecasts. It may
therefore not be possible to learn these models’s biases without adding an
explicit time-dependence.

While it may be possible to improve our attention-based ensemble pooling
model’s performance on the COVID-19 forecast problem by adding additional
context to the queries and keys, the inherent challenges in this data set
remain. We therefore would like to test this method on a different real-world
data set where we have both ample training data and fixed model biases.
Namely, we intend to use an ensemble of numerical terrestrial climate models
to forecast the El Niño Southern Oscillation.

## Code Availability

The code that supports the findings of this study may be found at
https://github.com/awikner/denpool.

## References

  * Bracher et al. (2021) Bracher, J., Ray, E. L., Gneiting, T., and Reich, N. G.  Evaluating epidemic forecasts in an interval format.  _PLOS Computational Biology_ , 17(2):e1008618, feb 2021.  doi: 10.1371/journal.pcbi.1008618.  URL https://doi.org/10.1371%2Fjournal.pcbi.1008618.
  * Conover (1999) Conover, W. J.  _Practical Nonparametric Statistics_.  Wiley, New York, NY, USA, third edition, 1999.  ISBN 978-0-471-16068-7.
  * Cramer et al. (2021) Cramer, E. Y., Huang, Y., Wang, Y., Ray, E. L., Cornell, M., Bracher, J., Brennen, A., Castro Rivadeneira, A. J., Gerding, A., House, K., Jayawardena, D., Kanji, A. H., Khandelwal, A., Le, K., Niemi, J., Stark, A., Shah, A., Wattanachit, N., Zorn, M. W., Reich, N. G., and Consortium, U. C.-. F. H.  The united states covid-19 forecast hub dataset.  _medRxiv_ , 2021.  doi: 10.1101/2021.11.04.21265886.  URL https://www.medrxiv.org/content/10.1101/2021.11.04.21265886v1.
  * Dong et al. (2020) Dong, X., Yu, Z., Cao, W., Shi, Y., and Ma, Q.  A survey on ensemble learning.  _Frontiers of Computer Science_ , 14:241–258, 2020.  doi: https://doi.org/10.1007/s11704-019-8208-z.
  * Gastinger et al. (2021) Gastinger, J., Nicolas, S., Stepić, D., Schmidt, M., and Schülke, A.  A study on ensemble learning for time series forecasting and the need for meta-learning, 2021.  URL https://arxiv.org/abs/2104.11475.
  * Gneiting & Raftery (2007) Gneiting, T. and Raftery, A. E.  Strictly proper scoring rules, prediction, and estimation.  _Journal of the American Statistical Association_ , 102(477):359–378, 2007.  doi: 10.1198/016214506000001437.  URL https://doi.org/10.1198/016214506000001437.
  * Gneiting & Ranjan (2011) Gneiting, T. and Ranjan, R.  Comparing density forecasts using threshold-and quantile-weighted scoring rules.  _Journal of Business & Economic Statistics_, 29(3):411–422, 2011.  ISSN 07350015.  URL http://www.jstor.org/stable/23243806.
  * In & Jung (2022) In, Y. and Jung, J.-Y.  Simple averaging of direct and recursive forecasts via partial pooling using machine learning.  _International Journal of Forecasting_ , 38(4):1386–1399, 2022.  ISSN 0169-2070.  doi: https://doi.org/10.1016/j.ijforecast.2021.11.007.  URL https://www.sciencedirect.com/science/article/pii/S0169207021001813.  Special Issue: M5 competition.
  * Kingma & Ba (2017) Kingma, D. P. and Ba, J.  Adam: A Method for Stochastic Optimization, January 2017.
  * Lorenz (1963) Lorenz, E. N.  Deterministic Nonperiodic Flow.  _Journal of Atmospheric Sciences_ , 20(2):130–141, March 1963.  ISSN 0022-4928, 1520-0469.  doi: 10.1175/1520-0469(1963)020¡0130:DNF¿2.0.CO;2.
  * Montero-Manso et al. (2020) Montero-Manso, P., Athanasopoulos, G., Hyndman, R. J., and Talagala, T. S.  Fforma: Feature-based forecast model averaging.  _International Journal of Forecasting_ , 36(1):86–92, 2020.  ISSN 0169-2070.  doi: https://doi.org/10.1016/j.ijforecast.2019.02.011.  URL https://www.sciencedirect.com/science/article/pii/S0169207019300895.  M4 Competition.
  * Press (2007) Press, W. H.  _Numerical Recipes: The Art of Scientific Computing_.  Cambridge University Press, 2007.  ISBN 978-978-052-188-2.
  * Sagi & Rokach (2018) Sagi, O. and Rokach, L.  Ensemble learning: A survey.  _WIREs Data Mining and Knowledge Discovery_ , 8(4):e1249, 2018.  doi: https://doi.org/10.1002/widm.1249.  URL https://wires.onlinelibrary.wiley.com/doi/abs/10.1002/widm.1249.
  * Sparrow (1982) Sparrow, C.  _The Lorenz Equations: Bifurcations, Chaos, and Strange Attractors_ , volume 41 of _Applied Mathematical Sciences_.  Springer, New York, NY, 1982.  ISBN 978-0-387-90775-8 978-1-4612-5767-7.  doi: 10.1007/978-1-4612-5767-7.
  * Vaiciukynas et al. (2021) Vaiciukynas, E., Danenas, P., Kontrimas, V., and Butleris, R.  Two-step meta-learning for time-series forecasting ensemble.  _IEEE Access_ , 9:62687–62696, 2021.  doi: 10.1109/ACCESS.2021.3074891.
  * Wu & Levinson (2021) Wu, H. and Levinson, D.  The ensemble approach to forecasting: A review and synthesis.  _Transportation Research Part C: Emerging Technologies_ , 132:103357, 2021.  ISSN 0968-090X.  doi: https://doi.org/10.1016/j.trc.2021.103357.  URL https://www.sciencedirect.com/science/article/pii/S0968090X21003594.
  * Yang et al. (2020) Yang, C., Yuan, K., Zhu, Q., Yu, W., and Li, Z.  Multi-expert learning of adaptive legged locomotion.  _Science Robotics_ , 5(49), dec 2020.  doi: 10.1126/scirobotics.abb2174.  URL https://doi.org/10.1126%2Fscirobotics.abb2174.
  * Zhou (2021) Zhou, Z.-H.  _Ensemble Learning_ , pp. 181–210.  Springer Singapore, Singapore, 2021.  ISBN 978-981-15-1967-3.  doi: 10.1007/978-981-15-1967-3˙8.  URL https://doi.org/10.1007/978-981-15-1967-3_8.