# SD-Net: Symmetric-Aware Keypoint Prediction and Domain Adaptation for 6D
Pose Estimation In Bin-picking Scenarios

Ding-Tao Huang1,2,†, En-Te Lin1,†, Lipeng Chen2,†, Li-Fu Liu1, Long Zeng1∗ †
Equal contribution. 1 International Graduate School at Shenzhen, Tsinghua
University, Shenzhen, China. This work was partially done when Dingtao Huang
conducted an internship at Tencent. {hdt22, linet22,
<EMAIL_ADDRESS><EMAIL_ADDRESS>Tencent Robotics
X, Shenzhen, China<EMAIL_ADDRESS>Corresponding author.

###### Abstract

Despite the success in 6D pose estimation in bin-picking scenarios, existing
methods still struggle to produce accurate prediction results for symmetry
objects and real world scenarios. The primary bottlenecks include 1) the
ambiguity keypoints caused by object symmetries; 2) the domain gap between
real and synthetic data. To circumvent these problem, we propose a new 6D pose
estimation network with symmetric-aware keypoint prediction and self-training
domain adaptation (SD-Net). SD-Net builds on pointwise keypoint regression and
deep hough voting to perform reliable detection keypoint under clutter and
occlusion. Specifically, at the keypoint prediction stage, we designe a robust
3D keypoints selection strategy considering the symmetry class of objects and
equivalent keypoints, which facilitate locating 3D keypoints even in highly
occluded scenes. Additionally, we build an effective filtering algorithm on
predicted keypoint to dynamically eliminate multiple ambiguity and outlier
keypoint candidates. At the domain adaptation stage, we propose the self-
training framework using a student-teacher training scheme. To carefully
distinguish reliable predictions, we harnesses a tailored heuristics for 3D
geometry pseudo labelling based on semi-chamfer distance. On public Siléane
dataset, SD-Net achieves state-of-the-art results, obtaining an average
precision of 96%. Testing learning and generalization abilities on public
Parametric datasets, SD-Net is 8% higher than the state-of-the-art method. The
code is available at https://github.com/dingthuang/SD-Net.

##  I Introduction

The estimation of 6D object pose is an essential prerequisite for robotic
tasks such as grasping and manipulation, especially in bin-picking scenarios
[1]. Recent studies that employ learning-based techniques have shown promising
results for this particular task [2]. These methods primarily fall into two
categories: holistic methods [1, 3] and keypoint-based methods [4, 5].
Keypoint-based methods employ intermediate variables to predict the 6D object
pose, effectively circumventing the nonlinear rotation space and providing a
promising direction for the exploration of 6D Object Pose Estimation [6].

Figure 1: This paper addresses two major problems facing 6D pose estimation.
(a) Keypoints sampled by BBox, FPS, Parameter and ours strategy. BBox, FPS and
Parameter keypoints ignore object geometric symmetry characteristics, while
ours strategy adaptively select keypoints based on object symmetry class. (b)
Methods typically exhibit inferior performance when applied to real world
point clouds, due to the persistent domain gap between real and synthetic data
point clouds.

Nonetheless, two major challenges persist in hindering the estimation
performance of keypoint-based pose estimation approaches, as illustrated in
Fig. 1. The first challenge lies in the absence of a robust strategy for
selecting keypoints during the keypoint prediction stage. This deficiency is
particularly evident when dealing with objects where keypoints on symmetric
components are hardly distinguishable. Many previous works [7, 8] have chosen
keypoints from a subset of the object’s bounding box (BBox) corners which away
from the surface of objects. Some other works [5, 4] employs the farthest
point sampling (FPS) algorithm to sample keypoints on the object’s surface
according to their relative proximity. ParametricNet [9] utilizes a keypoint
selection strategy based on object parameters. Despite promising results,
these methods fail to consider object geometric symmetry characteristics. The
symmetry of an object can result in multiple points on the surface that have
similar geometric features to the selected keypoints. These ambiguity points
mislead the network in predicting keypoints.

Secondly, annotating 6D object poses in the real world is labor-intensive. As
a result, the power of simulation is often harnessed to virtually generate 6D
pose labels for training deep learning models [10]. However, these methods
typically exhibit inferior performance when applied to real world data, due to
the persistent domain gap between real and synthetic data, as demonstrated in
Fig. 1. Many approaches [11, 12] rely on a customized simulation strategy to
produce high-quality synthetic data, while others [13, 14] necessitate a
specialized network architecture to extract domain-invariant signals.
Alternatively, some methods employ render-and-compare techniques [10, 15] for
self-supervision on unlabeled real data. These approaches utilize 2D
appearance information of objects to distinguish reliable predictions.
Nevertheless, texture-less objects provide essentially no 2D effective
features, posing a challenge for these methods.

In this study, we introduce a Symmetric-aware keypoint prediction and Domain
adaptation Network (SD-Net) for 6D object pose estimation in bin-picking
scenarios. SD-Net is constructed based on point-wise keypoint regression and
deep Hough voting. The inclusion of a voting mechanism equips our model with
the capability to perform reliable keypoint detection in bin-picking
scenarios. To slove the iusse of object symmetry, we propose a new keypoint
selection and filtering algorithm when performing keypoint prediction.
Subsequently, all equivalent keypoints are calculated according to equivalent
rotation matrices. This selection method, which takes into account object
symmetry class, significantly streamlines the network’s task of location and
bolsters the pose estimation performance. We also implement a keypoint
filtering algorithm to choose the predicted keypoints with highest confidence
before generating pose hypotheses.

In addressing domain gap, we propose a sim-to-real framework under a student-
teacher learning scheme which can be generalized to texture-less objects. We
initially train teacher model in a fully-supervised manner with abundant
synthesized data. Subsequently, we use the teacher model to generate pseudo
labels on real world data, and then these pseudo labels are used to update the
student network. To facilitate robustness, we propose a tailored heuristic for
3D geometry pseudo labeling that relies on semi-chamfer distance, enabling the
careful identification of reliable predictions. Moreover, the integration of
mask labels contributes to the stability of the training process.

We benchmark our proposed method using the Siléane [16] and Parametric [9]
datasets. Experimental results demonstrate that SD-Net outperforms state-of-
the-art methods. On the Siléane dataset, SD-Net achieves a 6% improvement in
average precision. Meanwhile, on the Parametric dataset, SD-Net surpasses the
state-of-the-art method by 8% in terms of average precision. In summary, the
main contributions of this work are:

  * •

We introduce a new 6D object pose estimation network with symmetric-aware
keypoint prediction and domain adaptation, which achieves state-of-the-art
estimation performance on the Siléane and Parametric datasets.

  * •

We propose a new keypoint selection method that considers object symmetry
class and a robust keypoint filtering algorithm that dynamically eliminates
multiple and outlier keypoint candidates.

  * •

We propose an iterative self-training framework for domain adaptation in 6D
object pose estimation, which leverages the 3D geometry information of objects
to carefully distinguish pseudo labels.

##  II RELATED WORK

###  II-A Holistic Methods

Holistic methods directly estimate the pose of object and can be divided into
two main groups (classical and learning-based methods). PPF [17] proposes
point pair global shape feature to retrieve poses from scene point cloud.
Hinterstoisser [18], [19] proposes a new feature which uses RGB color gradient
and 3D surface normal information. These classical methods are not robust in
clustered scenes. Based on deep neural network, some methods transform pose
estimation problems into regression problems. DenseFusion [20] uses dense
pixel level fusion to fuse RGB and point cloud. Some works [3, 21, 22, 1, 2,
23] directly regresses the position and rotation of the object from point
cloud or depth map. Because the rotation space is nonlinear, direct regression
of rotation is challenging.

###  II-B Keypoint-based methods

Keypoint-based methods detect keypoints in camera coordinate system and
establishe correspondence between them and keypoints in canonical object-frame
coordinate system. In terms of keypoint selection, BB8 [7] and YOLO-6D [8]
predict the projection of the 3D bounding box on the 2D plane. The corners of
the bounding box are generally far from the surface of the object and are less
representative. Therefore, PVNet [4] and PVN3D [5] use the farthest point
sampling algorithm to select keypoint to reduce position errors. For symmetry
objects, the points selected in this way are highly similar and difficult to
distinguish. FFB6D [24] uses SIFT [25] to extract features different angles 3D
models and associate them with the corresponding 3D positions. However, these
work [24, 26] is not suitable for texture-less objects. ParametricNet [4]
predicts keypoints in shape templates which has two disadvantages. First,
chirality problems [9] occur when predicting keypoint of mirror symmetry
objects. Second, the selection of keypoint according to object shape parameter
is tedious and less robust. These methods fail to consider the geometric
similarity of different components of symmetry objects, thus ignoring pose
ambiguity caused by object symmetries. It is worth noting that there is a lack
of a robust strategy to select keypoints from symmetry objects in keypoint
prediction stage.

Figure 2: Overview of SD-Net architecture. SD-Net is constructed based on
point-wise keypoint regression and deep Hough voting. It consists of two main
parts: symmetri-caware keypoint prediction and self-supervised domain
adaptation. keypoint prediction consists of a new keypoint selection and
filtering algorithm. We omit the domain adaptation framework, for brevity and
more details can be found in Section III-C. $N_{j}$ represents the number of
point cloud points for each instance. $N_{k}$ represents the number of
keypoint decoders and corresponds to the number of keypoint of objects.
$N_{i}$ represents the number of instances in the scene.

###  II-C Domain Adaptation for 6D Pose Estimation

Sim-to-real transfer is crucial in 6D pose estimation as it bridges the domain
gap between synthetic and real data observations through robotic visual
systems. Some works from domain randomization aim at sampling a wide variety
of simulation settings to learn domain-invariant attributes. such as random
backgrounds [11], [27] and image augmentations [28]. Some works [12], [29]
harness light-weight physically-based renderer (PBR) data to simulate
realistic texture to reduce the gap between the synthetic and real domains.
Some works learning a mapping between different visual domains based on
Generative Adversarial Networks [13] or means of feature [14] mapping.
However, these methods either depend on a customized simulation strategy to
generate high-quality synthetic data or require a specialized network
architecture to extract domain-invariant signals. Alternatively, some methods
[10], [30], [15] conduct render-and-compare strategy to self-supervised on
unlabeled real data and impose consistencies between rendered features and
sensed features. These methods harnesses 2D appearance information of objects
to distinguish reliable predictions. However, texture-less objects provide
essentially no 2D effective features, posing a challenge for these methods. In
addition, these methods focus on pose estimation with domain adaptation of
individual objects, ignoring the entire scene.

##  III Method

Given a point cloud of a bin-picking scene where multiple object instances are
stacked randomly into a pile, we are interested in detecting instances and
estimating their rotation $R\in SO(3)$ and translation
$t\in\mathbb{R}^{{}^{3}}$ in three-dimensional (3D) space. The object pose is
represented by a rigid transformation from the object coordinate system to a
reference camera coordinate system. In this section, we first introduce our
symmetric-aware keypoint selection and filtering algorithm for SD-Net. Then,
we present the overall architecture of SD-Net. In order to boost the
performance of object pose estimation in real world, we introduce the self-
supervised domain adaptation framework.

###  III-A Symmetric-Aware keypoint prediction

Keypoint Selection. The selection of keypoint is a significant challenge.
Keypoints away from the surface of the object will increase localization
errors [4]. It is difficult to distinguish points on a symmetry object, which
means that the geometric features of different components of the object are
similar. To facilitate network convergence, the keypoint selection strategy
should meet the following requirements: (1) Keypoints should be close to the
surface of the object. (2) Take the object symmetry class into account. Based
on this observation, we propose a heuristic keypoint selection algorithms.
Keypoint set is a collection of a object centroid and the points where the
object bounding box intersect with the object coordinate axes. In this way,
these selected key points are close to the surface of the object, making
point-based networks easy to aggregate scene context in the vicinity of them.
Keypoint set is defined as:

$\displaystyle
K={P_{c}}\cup\\{{P_{i}}|{P_{i}}={P_{bounding{\rm{}}box}}\cap{P_{axles}}\\}$
(1)

where $P_{c}$ represents the centroid, $P_{boundingbox}$ represents the object
bounding box and $P_{axes}$ represents the object selected coordinate axes.
For example, in Fig. 3.a, the keypoint set contains elements
$\\{{K_{1}},{K_{2}}\\}$. Subsequently, an adaptive coordinate axes selection
strategy considering objects symmetry class designed and the details are as
follows:

  * •

For revolution symmetry objects, we choose the axis of rotation as
${P_{axes}}$. In this way, it can avoid selecting points located on the curved
surface which have a low discrimination.

  * •

For finite non trivial symmetry objects, we choose the axis of rotation and
another axis which is randomly selected from the remaining axes as
${P_{axes}}$.

  * •

For mirror symmetry objects, we choose two axes which parallel to the plane of
mirror symmetry as ${P_{axes}}$. This can avoid chirality issues, which is
that two point sets of different chirality structures cannot be registered by
rotation and translation transformation.

  * •

For no proper symmetry objects, we select all three axes as ${P_{axes}}$. This
can increase the number of keypoints and improve pose prediction accuracy and
robustness.

In addition, for revolution and finite symmetric objects, we transform the
object model so that any coordinate axis of the objects coincides with the
rotation axis. For mirror symmetry objects, we transform the object model so
that any two coordinate axes of the objects coincide with the symmetry plane.
Any object instances can be divided into any of the four symmetry class above
it and this keypoint selection can be used for any type of objects.

Figure 3: The axes selection strategy depends on the object symmetry class.
The red axes represent the selected axes. The blue dots represent the selected
keypoints, while the red dots represent the corresponding equivalent
keypoints.

Equivalent Keypoint Set. In the process of selecting keypoint using Equation
(1), some keypoints are often accompanied by multiple equivalent keypoints,
especially for finite symmetric object. For example, in Fig. 3.b, the keypoint
${K_{3}}$ formed by the intersection of the ${X}$-axis and the bounding box
contains six equivalent keypoints $\\{{K_{3-1}},{K_{3-3}},...,{K_{3-6}}\\}$.
These keypoints remain geometrically consistent, which can cause ambiguity
during the training phase of the network. Therefore, we predict all equivalent
keypoints to avoid confusion during the learning phase. Equivalent keypoint
set is defined as:

$\displaystyle{K_{j-equivalent}}=\\{{K_{j}}g|g\in G\\}$ (2)

where $G\in SO(3)$ is the set of rotation matrix that keeps object state
unchanged. For no proper symmetry objects, $G$ is the unit matrix. ${K_{j}}$
is an element in set ${K}$ in Equation (1).

Keypoint Filtering In the model inference stage (as described in Section
III-B), some predictied keypoints will be distributed into multiple candidate
clusters around equivalent keypoints lable, so as the predicted
${\widehat{kp_{i}^{3}}}$ in Fig. 4.b. Moreover, some prediction keypoints have
significant deviation from ground truth.

Given the predicted keypoint, we are ready to eliminate multiple ambiguity and
outlier keypoint candidates. We introduce a robust keypoint filtering
algorithm. It is detailed in Algorithm 1. The core step of our algorithm is to
cluster the predicted keypoints to form multiple keypoint cluster and find the
cluster with the highest point cloud density. To achieve $cluster$ in
Algorithm 1, we employ DBSCAN [31]. The $density$ in Algorithm 1 is used to
estimate the density of point cloud clusters and is defined as:

$\displaystyle
D=\frac{1}{N}\sum\limits_{i=1}^{N}{{{\left\|{{p_{i}}-{p_{c}}}\right\|}_{2}}}$
(3)

where ${p_{i}}$ and ${p_{c}}$ represent the points and centroids in the point
cloud clusters, respectively.

As shown in Fig. 4.c, keypoint filtering algorithm provides reliable keypoint
to fit the pose. For predictied keypoint ${\widehat{kp_{i}^{3}}}$ distributed
into two candidate clusters, keypoint filtering algorithm dynamically
preserves the candidates clusters in the area with the highest density.
Meanwhile, the outliers keypoint with significant prediction deviations can be
considered as low density point clouds, so they can be filtered out by the
density threshold. It is worth noting that we apply keypoint filtering to each
type of keypoint respectively, because different types of keypoint are
distributed differently.

(a) Keypoint selection

(b) Keypoint regression

(c) Keypoint filtering

Figure 4: (a) Keypoints sampled by ours selection algorithms on t-less20
object from Siléane dataset. (b) The white points represent the scene instance
point clouds, and the red points represent the pointwise predicted keypoints.
(c) The red points represent predicted keypoints after filtering. Algorithm 1
Density-based Keypoint Filtering

1:prediction keypoints
${P_{1}}=\\{{x_{1}},{x_{2}},...,{x_{n}}\\}\in{{}^{n\times 3}}$.

2:filtered keypoints ${P_{2}}=\\{{x_{1}},{x_{2}},...,{x_{m}}\\}\in{{}^{m\times
3}}$.

3:initialize an empty cluster set $C$

4:$C=cluster({P_{1}})$

5:${d_{0}}=density({C_{0}})$

6:${P_{2}}=C_{0}$

7:for ${C_{i}}$ in ${C}$ do

8: if $density({C_{i}})<{d_{0}}$ then

9: ${P_{2}}\leftarrow{C_{i}}$

10: ${d_{0}}\leftarrow density({C_{i}})$

11: end if

12:end for

13:return ${P_{2}}$

###  III-B Architecture design

Fig. 2 illustrates our end-to-end 6D Pose Estimation network. Initially, it
takes the point clouds of the bin-picking scene with $N_{p}$ points as input
and applies a feedforward network PointNet++ [32] for feature extraction. The
extracted $F_{p}$ has a size of $N_{p}\times N_{f}$. Subsequently, our network
diverges into two decoders which consume $F_{p}$ to predict the keypoint and
visibility for each individual point.

Visibility Decoder. In bin-picking scenarios, some instances are severely
occluded. We are not interested in these instances which cannot be captured at
the bottom. Therefore, we set the visibility decoder to reduce the impact of
severe occlusion. The point-wise visibility is defined as
${V_{i}}={N_{i}}/{N_{\max}}$. ${N_{i}}$ is the number of points of the
instance to which the $i$ th point belongs and ${N_{\max}}$ indicate the
highest number of points within all visiable instances. We pass $F_{p}$ into
visibility decoder to prediction the point-wise visibility. The visibility
loss is defined as:

$\displaystyle{L_{v}}=\frac{1}{{{N_{p}}}}\sum\limits_{i=1}^{{N_{p}}}{\left\|{{{\widehat{V}}_{i}}-{V_{i}}}\right\|}$
(4)

where ${\widehat{V}_{i}}$ denotes the prediction of the point-wise visibility.

Keypoint Decoder. Compared with directly predicting keypoint, predicting the
offset of keypoint relative to the point cloud is more accurate. Therefore, we
can pass $F_{p}$ into a keypoint decoder to predict the point-wise offsets
${o_{i}}\in\mathbb{R}^{{}^{3}}$. The predicted point-wise keypoint coordinates
can be expressed as $\widehat{{kp_{i}}}={o_{i}}+{p_{i}}$, where ${p_{i}}$
denotes the $i$ th point coordinate. Different types of keypoints are
predicted separately using independent decoders. For objects with ${N_{k}}$
types of keypoints, we use ${N_{k}}$ decoders to obtain prediction keypoints
$\widehat{kp_{i}^{j}}\in\mathbb{R}^{{}^{3}}$, which denotes the prediction of
the $j$ th keypoint of the instance to which the $i$ th point corresponds. The
keypoint prediction loss is defined as:

$\displaystyle{L_{k}}=\frac{1}{{{N_{p}}}}\sum\limits_{j=1}^{{N_{k}}}{\sum\limits_{i=1}^{{N_{p}}}{\mathop{\min}\limits_{kp\in
kp_{i}^{j}}\left\|{\widehat{kp_{i}^{j}}-kp}\right\|}}$ (5)

where $kp_{i}^{j}$ denotes the point-wise keypoint labels
($kp_{i}^{j}\in\mathbb{R}^{{}^{{N_{e}}\times 3}}$, $N_{e}$ denotes the number
of equivalent keypoint). This loss function computes the Euclidean distance
between the prediction and the closest ground truth.

Generating Pose Hypotheses. In the inference stage, as shown in Fig. 2, we
filter out severely occluded scene point clouds and group scene points into
instances with Mean Shift algorithm [33] to generate instances keypoints. Then
we apply keypoints filtering and hough voting to generate each instance voting
keypoints $\\{\widehat{{K_{j}}}\in\mathbb{R}^{{}^{3}}\\}_{j=1}^{{N_{k}}}$ in
the camera coordinate system. When providing their corresponding points
$\\{{K_{j}}\\}_{j=1}^{{N_{k}}}$ in canonical object coordinate system, we
utilize a least-squares fitting algorithm [34] to calculate the optimal values
of ${R}$ and ${t}$.

$\displaystyle{L_{lsf}}=\sum\limits_{j=1}^{{N_{k}}}{{{\left\|{\widehat{{K_{j}}}-(R\cdot{K_{j}}+t)}\right\|}_{2}}}$
(6)

Figure 5: Overview of our proposed self-supervised domain adaptation framework
for 6D object pose estimation. We first train teacher model on synthesize
abundant data to generate initial pose predictions. We then use a 3D geometry
pseudo labelling algorithm to distinguish real word predictions for student
model training. In the next iteration, the teacher model is initialized as the
last trained student model and iterate the above process util the model
convergence.

###  III-C Self-supervised domain adaptation

Fig. 5 summarizes our student-teacher domain adaptation pose estimation
framework. We first train a fully-supervised teacher model $M_{t}$ with
abundant labeled synthetic data, as introduced in Section III-B. Then we apply
$M_{t}$ on unlabeled real data to predict object instances poses. Based on
these initial poses with decent quality, a robust label selection algorithm is
designed to select the best reliable predictions for pseudo labels. The real
data with pseudo label generated by pose prediction is utilized to train a
student model $M_{s}$ by self-supervised learning. Crucially, we iterate the
above process by taking the $M_{s}$ as a new teacher model, in order to
progressively boost the quality of pseudo labels and close the domain gap.

Specifically, given unlabeled real point cloud $P_{r}$ and their initial pose
estimations $\\{{p_{i}}=[{R_{i}}|{t_{i}}]\\}_{i=1}^{N_{i}}$ generated by
teacher model, we harness geometric constraints to seek the best alignment
w.r.t. 6D pose. The core idea is to generate the object point cloud that
corresponds to the predicted pose and compare with the real collected point
cloud to determine whether the predicted pose is reliable or not. We conduct
the following two steps to leverage geometric constraints. We first
backproject the object CAD model $C$ using the corresponding predicted pose to
retrieve the point clouds $C_{i}$ in camera space: ${C_{i}}={R_{i}}\times
C+{t_{i}}$. Object instances point clouds $P_{i}$ only contains the surface
point cloud that is visible from a particular viewpoint and severely
obstructed. $P_{i}$ is incomplete point cloud and a subset of $C_{i}$.
Intuitively, we use the semi-chamfer distance between $P_{i}$ and $C_{i}$ as a
3D metric to quantify the quality of predicted pose:

$\displaystyle{d_{i}}=\frac{1}{{{N_{{P_{i}}}}}}\sum\limits_{x\in{P_{i}}}{\mathop{\min}\limits_{y\in{C_{i}}}\left\|{x-y}\right\|{}_{2}}$
(7)

where $x$ and $y$ denotes 3D points from $P_{i}$ and $C_{i}$ respectively,
${N_{{P_{i}}}}$ denotes the number of points for $P_{i}$.

We calculate the pose quality for each predicted poses generated by tearcher
model $M_{t}$ and obtain geometry pose quality set
$\\{{d_{i}}\\}_{i=1}^{{N_{i}}}$ in a bin-picking scene. Then, we dynamically
generate a threshold $d_{g}$ based on the mean and standard deviation of the
geometry pose quality set distribution. Pose prediction $p_{i}$ are regarded
as the correct prediction when ${d_{i}}<{d_{g}}$. The pseudo labels of
keypoints for the objects in the real data are calculated according to the
correct pose predictions. The pesudo labels of visibility are calculated based
on the number of each object instance point cloud. Additionally, for instance
point clouds whose geometry pose quality ${d_{i}}>{d_{g}}$, the pesudo labels
of mask are assigned to exclude them from the model training. Specifically,
only pesudo labels with a mask label of 1 participate in training. After label
generation, we incorporate real point clouds with reliable pseudo labels and
train a student model $M_{s}$ to transfer the knowledge from the synthetic
data to real data. In the next iteration, the teacher model $M_{t}$ is
initialized as the last trained student model $M_{s}$ and iterate the above
process util the model convergence.

##  IV Experiments

Figure 6: Qualitative results on Siléane dataset [16]. Rows show a comparison against different methods (SD-Net, ParametricNet and PPR-Net++). Columns show different scenarios. Red arrows highlight wrong prediction pose and we highlight a maximum of 2 wrong pose in a scene for brevity. TABLE I:  Quantitative evaluation of 6D Pose estimation on Siléane dataset [16]. The test objects contain three types of symmetry. Object | Bunny | C.Stick | Pepper | Brick | Gear | T-Less20 | T-Less22 | T-Less29 | Mean
---|---|---|---|---|---|---|---|---|---
Symmetry Class | Non-Symmetry | Revolution | Revolution | Finite | Revolution | Finite | Non-Symmetry | Finite |
PPF [17] | 0.29 | 0.16 | 0.06 | 0.08 | 0.62 | 0.20 | 0.08 | 0.19 | 0.21
LINEMOD+ PP[19] | 0.45 | 0.49 | 0.03 | 0.39 | 0.50 | 0.31 | 0.21 | 0.26 | 0.33
Sock et al [23] | 0.74 | 0.64 | 0.43 | - | - | - | - | - | -
PPR-Net with ICP [3] | 0.89 | 0.95 | 0.84 | - | - | 0.85 | - | - | -
OP-Net with Lori1 [1] | 0.92 | 0.94 | 0.98 | 0.41 | 0.82 | 0.85 | 0.77 | 0.51 | 0.78
OP-Net AP [2] | 0.92 | 0.98 | 0.99 | 0.45 | 0.82 | 0.87 | 0.84 | 0.56 | 0.80
ParametricNet [9] | - | 0.97 | - | - | 1.00 | 0.92 | - | 0.94 | -
PPR-Net++ [21] | 0.99 | 0.98 | 0.98 | 0.47 | 1.00 | 0.93 | 0.92 | 0.94 | 0.90
Ours | 1.00 | 1.00 | 1.00 | 0.75 | 1.00 | 0.98 | 0.96 | 0.98 | 0.96
|  |  |  |  |  |  |  |  |

###  IV-A Datasets and evaluation metrics

To comprehensively evaluate our method in bin-picking scenarios, we select
Siléane dataset [16] and Parametric dataset [9]. Siléane dataset is comprised
of a total of more than 2,600 bin-picking scenarios. Parametric dataset
consists two types of data. The L-dataset test set consists of objects with
the same parameters as the training set, used to evaluate learning abilities.
The G-dataset test set consists of objects with different parameters compared
to the training set, used to assess generalization abilities.

This metric evaluates the performance of methods by calculating the area under
the precision-recall curve, which is summarized as the Average Precision.
Specifically, when the distance between predicted pose and ground truth is
less than 0.1 times object’s minimum bounding sphere diameter, the prediction
pose is considered correct [35].

###  IV-B Evaluation on Siléane dataset

Table I summarizes the comparison results between our approach and current 6D
pose estimation methods. Our proposed approach achieves state-of-the-art
results and outperforms others, obtaining an average precision of 96%. Our
approach results in an improvement of +68% to +75% on average precision
compared with the conventional approaches [17, 19]. Our approaches already
clearly outperforms the existing deep learning-based methods [1, 2, 9, 21]
with a large margin without the need of a separate refinement stage. Fig. 6
shows the qualitative results of SD-Net in severely bin-picking scenarios and
SD-Net is superior to the other two methods in handling symmetry objects.

The observed improvement in finite non trivial symmetry objects is likely due
the equivalent keypoint set which is easier to learn by the regression
network, e.g., T-Less20 and T-Less29 objects. For the brick object, compared
with the current state-of-the-art PPR-Net++, SD-Net has a significant increase
of 23% in average precision. We observe that smaller object sizes, result in
larger predicted position and rotation deviations. Our keypoint filtering
algorithm eliminate the keypoints with significant prediction deviation, which
helps to improve the accuracy of pose estimation.

###  IV-C Evaluation on Parametric dataset

Learning ability. From Table II, SD-Net advances state-of-the-art results by
8% on average precision metric. Compared with ParametricNet on TN42 object,
SD-Net significantly improves the performance by 26%. A crucial contributing
factor is that our keypoints selection algorithm avoid chirality issues (as
described in Section III-A). TN06 object has 12 equivalent pose, resulting in
giving rise to ambiguous pose estimations. SD-Net conduct equivalent keypoint
set and density based keypoint filtering algorithm to reduce the pose
estimation ambiguity and further improve the pose accuracy. In TN16 and TN34
objects scenario, the level of occlusion is relatively low, resulting in
highly accurate prediction outcomes for all three methods.

TABLE II: Learning ability evaluation on Parametric dataset. Object | TN06 | TN16 | TN34 | TN42 | Mean
---|---|---|---|---|---
Symmetry Class | Finite | Revolution | Revolution | Mirror |
PPR-Net++ | 0.80 | 0.99 | 1.00 | 0.39 | 0.80
ParametricNet | 0.94 | 1.00 | 1.00 | 0.52 | 0.87
Ours | 1.00 | 1.00 | 1.00 | 0.78 | 0.95
|  |  |  |  |

Generalization ability. In Table III, our average precision metric is 8%
higher than the state-of-the-art method. SD-Net demonstrates almost the same
generalization capability as its learning capability. The results show that
SD-Net exhibits strong generalization capability for unseen parameters
objects. To further evaluate the SD-Net generalization ability, we conduct
experiments by down-sampling the number of training instances to one-third and
one-fifth of the original dataset, which means that the model has seen fewer
parameters objects during training. Across varying initialization experimental
configuration, SD-Net consistently outperforms other methods on generalization
metric. For the TN06 and TN42 object, SD-Net achieves even higher average
precision when trained with only one-fifth of the objects compared to other
methods trained on all objects.

TABLE III: Generalization ability evaluation on Parametric dataset. Train Mode |  Method |  TN06 |  TN16 |  TN34 |  TN42 |  Mean
---|---|---|---|---|---|---
Learn all | PPR-Net++ | 0.79 | 0.99 | 1.00 | 0.28 | 0.77
ParametricNet | 0.93 | 1.00 | 1.00 | 0.51 | 0.86
Ours | 1.00 | 1.00 | 1.00 | 0.77 | 0.94
Learn 1/3 | PPR-Net++ | 0.78 | 0.94 | 0.96 | 0.22 | 0.73
ParametricNet | 0.86 | 0.98 | 1.00 | 0.41 | 0.81
Ours | 1.00 | 1.00 | 1.00 | 0.67 | 0.92
Learn 1/5 | PPR-Net++ | 0.77 | 0.56 | 0.83 | 0.18 | 0.59
ParametricNet | 0.86 | 0.63 | 0.86 | 0.39 | 0.69
Ours | 0.98 | 0.65 | 0.89 | 0.54 | 0.77
|  |  |  |  |  |
TABLE IV: Ablation study on Siléane dataset.EKS, Equivalent Keypoint Set. KF, Keypoint Filtering. SCD, Semi-Chamfer Distance. DA, Domain Adaptation. w/o,without. Object | Brick | T-Less20 | T-Less22 | T-Less29
---|---|---|---|---
SD-Net (w/o EKS) | 0.08 | 0.76 | 0.95 | 0.66
SD-Net (w/o KF) | 0.53 | 0.03 | 0.94 | 0.08
SD-Net (w/o DA) | 0.70 | 0.96 | 0.95 | 0.97
SD-Net (w/o SCD) | 0.71 | 0.93 | 0.94 | 0.00
SD-Net (BBox) | 0.21 | 0.55 | 0.61 | 0.16
SD-Net (FPS) | 0.45 | 0.89 | 0.88 | 0.79
SD-Net (Parameter) | 0.47 | 0.92 | 0.92 | 0.94
SD-Net | 0.75 | 0.98 | 0.96 | 0.98
|  |  |  |

###  IV-D Ablation study

We present extensive ablation studies on SD-Net to compare different design
choices. Table IV summaries the evaluation results on the Siléane dataset. SD-
Net (w/o EKS) learns the keypoint without equivalent keypoint set. For
symmetric objects, its performance is greatly reduced. SD-Net (w/o KF) remove
keypoint filtering algorithm and the performance degradation is large. SD-Net
(w/o DA) remove domain adaptation and the results show that ours self-training
domain adaptation framework enhance the SD-Net performance on real data. SD-
Net (w/o SCD) use chamfer distance to quantify the quality of prediction pose.
Due to the incomplete object instances point clouds, chamfer distance cannot
precisely quantify the quality. We replace the symmetric-aware keypoint
selection strategy with BBox, FPS, and Parameter and does not perform a
keypoint filtering algorithm. The performance of these methods is greatly
reduced. This shows that our proposed symmetric-aware keypoint prediction
performs reliable detection keypoints. In general, the proposed novelty design
can significantly improve average precision.

###  IV-E Real World Experiment

We further explore the application of SD-Net to robot grasping tasks in the
real world. We choose TN06 object instance from Parametric [9] datasets as the
grab object. We utilize the Blender platform to generate simulation data set.
Totally 18000 point clouds are annotated, comprising 300 cycles, with each
cycle consisting of 60 scenarios. In addition, we collected 600 unlabeled real
world point clouds to self-training domain adaptation.

(a) Scene point clouds

(b) Predict object pose

(c) Grasp pose

(d) Scene RGB image

(e) Experiment setup

(f) Robotic grasp

Figure 7: Robotic grasping experiment on real world bin-picking scenarios with
the proposed approach SD-Net.

We deploy the aforementioned well-trained SD-Net on the Fanuc industrial
robot, which is equipped with pneumatic gripper. The whole robot grasping
system is implemented using the ROS and MoveIt! frameworks. In the real
grasping experiment, objects are stacked in the container, As shown in Fig. 7
(d). The scene point clouds captured by the RVC X 3D camera is cropped,
sampled, filtered, and subsequently fed into SD-Net for pose estimation, as
shown in Fig. 7 (b). Based on the predicted instance pose, we select the set
of grasp configurations from the pre-calculated grasps database, as shown in
Fig. 7 (c). We evaluated the ability of SD-Net in 10 grasping trials. Our
pipeline can successfully accomplish the robot grasping tasks for all
graspable object instances in all trials. It show that SD-Net demonstrates
excellent performance in robot bin-picking tasks.

##  V Conclusions

In this paper, we propose a new 6D pose estimation network with symmetri-
caware keypoint prediction and domain adaptation. It includes two critical
components. We propose a new selection keypoint method which considers objects
symmetry class and a robust keypoint filtering algorithm. We propose an
network-agnostic iterative self-training framework for domain adaptation 6D
object pose estimation. Experiments show that SD-Net has significant
improvements in average precision compared to state-of-the-art approaches on
the public Siléane dataset and Parametric dataset.

## References

  * [1] K. Kleeberger and M. F. Huber, “Single shot 6d object pose estimation,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 6239–6245.
  * [2] K. Kleeberger, M. Völk, R. Bormann, and M. F. Huber, “Investigations on output parameterizations of neural networks for single shot 6d object pose estimation,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021, pp. 13 916–13 922.
  * [3] Z. Dong, S. Liu, T. Zhou, H. Cheng, L. Zeng, X. Yu, and H. Liu, “Ppr-net: point-wise pose regression network for instance segmentation and 6d pose estimation in bin-picking scenarios,” in _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2019, pp. 1773–1780.
  * [4] S. Peng, Y. Liu, Q. Huang, X. Zhou, and H. Bao, “Pvnet: Pixel-wise voting network for 6dof pose estimation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 4561–4570.
  * [5] Y. He, W. Sun, H. Huang, J. Liu, H. Fan, and J. Sun, “Pvn3d: A deep point-wise 3d keypoints voting network for 6dof pose estimation,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 11 632–11 641.
  * [6] W. Hua, Z. Zhou, J. Wu, H. Huang, Y. Wang, and R. Xiong, “Rede: End-to-end object 6d pose robust estimation using differentiable outliers elimination,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 2, pp. 2886–2893, 2021.
  * [7] M. Rad and V. Lepetit, “Bb8: A scalable, accurate, robust to partial occlusion method for predicting the 3d poses of challenging objects without using depth,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 3828–3836.
  * [8] B. Tekin, S. N. Sinha, and P. Fua, “Real-time seamless single shot 6d object pose prediction,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 292–301.
  * [9] L. Zeng, W. J. Lv, X. Y. Zhang, and Y. J. Liu, “Parametricnet: 6dof pose estimation network for parametric shapes in stacked scenarios,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021, pp. 772–778.
  * [10] G. Wang, F. Manhardt, J. Shao, X. Ji, N. Navab, and F. Tombari, “Self6d: Self-supervised monocular 6d object pose estimation,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16_. Springer, 2020, pp. 108–125.
  * [11] W. Kehl, F. Manhardt, F. Tombari, S. Ilic, and N. Navab, “Ssd-6d: Making rgb-based 3d detection and 6d pose estimation great again,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 1521–1529.
  * [12] T. Hodaň, M. Sundermeyer, B. Drost, Y. Labbé, E. Brachmann, F. Michel, C. Rother, and J. Matas, “Bop challenge 2020 on 6d object localization,” in _Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16_. Springer, 2020, pp. 577–594.
  * [13] H.-Y. Lee, H.-Y. Tseng, J.-B. Huang, M. Singh, and M.-H. Yang, “Diverse image-to-image translation via disentangled representations,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 35–51.
  * [14] M. Rad, M. Oberweger, and V. Lepetit, “Domain transfer for 3d pose estimation from color images without manual annotations,” in _Asian Conference on Computer Vision_. Springer, 2018, pp. 69–84.
  * [15] H. Chen, F. Manhardt, N. Navab, and B. Busam, “Texpose: Neural texture learning for self-supervised 6d object pose estimation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 4841–4852.
  * [16] R. Brégier, F. Devernay, L. Leyrit, and J. L. Crowley, “Symmetry aware evaluation of 3d object detection and pose estimation in scenes of many parts in bulk,” in _Proceedings of the IEEE International Conference on Computer Vision Workshops_ , 2017, pp. 2209–2218.
  * [17] B. Drost, M. Ulrich, N. Navab, and S. Ilic, “Model globally, match locally: Efficient and robust 3d object recognition,” in _2010 IEEE computer society conference on computer vision and pattern recognition_. Ieee, 2010, pp. 998–1005.
  * [18] S. Hinterstoisser, S. Holzer, C. Cagniart, S. Ilic, K. Konolige, N. Navab, and V. Lepetit, “Multimodal templates for real-time detection of texture-less objects in heavily cluttered scenes,” in _2011 international conference on computer vision_. IEEE, 2011, pp. 858–865.
  * [19] S. Hinterstoisser, V. Lepetit, S. Ilic, S. Holzer, G. Bradski, K. Konolige, and N. Navab, “Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes,” in _Computer Vision–ACCV 2012: 11th Asian Conference on Computer Vision, Daejeon, Korea, November 5-9, 2012, Revised Selected Papers, Part I 11_. Springer, 2013, pp. 548–562.
  * [20] C. Wang, D. Xu, Y. Zhu, R. Martín-Martín, C. Lu, L. Fei-Fei, and S. Savarese, “Densefusion: 6d object pose estimation by iterative dense fusion,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 3343–3352.
  * [21] L. Zeng, W. J. Lv, Z. K. Dong, and Y. J. Liu, “Ppr-net++: accurate 6-d pose estimation in stacked scenarios,” _IEEE Transactions on Automation Science and Engineering_ , vol. 19, no. 4, pp. 3139–3151, 2021.
  * [22] X. Zhang, W. Lv, and L. Zeng, “A 6dof pose estimation dataset and network for multiple parametric shapes in stacked scenarios,” _Machines_ , vol. 9, no. 12, p. 321, 2021.
  * [23] J. Sock, K. I. Kim, C. Sahin, and T.-K. Kim, “Multi-task deep networks for depth-based 6d object pose and joint registration in crowd scenarios,” _arXiv preprint arXiv:1806.03891_ , 2018.
  * [24] Y. He, H. Huang, H. Fan, Q. Chen, and J. Sun, “Ffb6d: A full flow bidirectional fusion network for 6d pose estimation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 3003–3013.
  * [25] D. G. Lowe, “Object recognition from local scale-invariant features,” in _Proceedings of the seventh IEEE international conference on computer vision_ , vol. 2. Ieee, 1999, pp. 1150–1157.
  * [26] H. Liu, G. Liu, Y. Zhang, L. Lei, H. Xie, Y. Li, and S. Sun, “A 3d keypoints voting network for 6dof pose estimation in indoor scene,” _Machines_ , vol. 9, no. 10, p. 230, 2021.
  * [27] M. Sundermeyer, Z.-C. Marton, M. Durner, M. Brucker, and R. Triebel, “Implicit 3d orientation learning for 6d object detection from rgb images,” in _Proceedings of the european conference on computer vision (ECCV)_ , 2018, pp. 699–715.
  * [28] S. Zakharov, W. Kehl, and S. Ilic, “Deceptionnet: Network-driven domain randomization,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 532–541.
  * [29] J. Tremblay, T. To, B. Sundaralingam, Y. Xiang, D. Fox, and S. Birchfield, “Deep object pose estimation for semantic robotic grasping of household objects,” _arXiv preprint arXiv:1809.10790_ , 2018.
  * [30] F. Manhardt, G. Wang, B. Busam, M. Nickel, S. Meier, L. Minciullo, X. Ji, and N. Navab, “Cps++: Improving class-level 6d pose and shape estimation from monocular images with self-supervised learning,” _arXiv preprint arXiv:2003.05848_ , 2020.
  * [31] M. Ester, H.-P. Kriegel, J. Sander, X. Xu, _et al._ , “A density-based algorithm for discovering clusters in large spatial databases with noise,” in _kdd_ , vol. 96, no. 34, 1996, pp. 226–231.
  * [32] C. R. Qi, L. Yi, H. Su, and L. J. Guibas, “Pointnet++: Deep hierarchical feature learning on point sets in a metric space,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [33] D. Comaniciu and P. Meer, “Mean shift: A robust approach toward feature space analysis,” _IEEE Transactions on pattern analysis and machine intelligence_ , vol. 24, no. 5, pp. 603–619, 2002.
  * [34] K. S. Arun, T. S. Huang, and S. D. Blostein, “Least-squares fitting of two 3-d point sets,” _IEEE Transactions on pattern analysis and machine intelligence_ , no. 5, pp. 698–700, 1987.
  * [35] R. Brégier, F. Devernay, L. Leyrit, and J. L. Crowley, “Defining the pose of any 3d rigid object and an associated distance,” _International Journal of Computer Vision_ , vol. 126, no. 6, pp. 571–596, 2018.


This idea is captured by the following definition.

$\begin{array}[]{rclcll}(\Xi_{1},x:A_{m})&\sqcup&(\Xi_{2},x:A_{m})&=&(\Xi_{1}\sqcup\Xi_{2}),x:A_{m}\\\
(\Xi_{1},[x:A_{m}])&\sqcup&(\Xi_{2},x:A_{m})&=&(\Xi_{1}\sqcup\Xi_{2}),x:A_{m}\\\
(\Xi_{1},x:A_{m})&\sqcup&(\Xi_{2},[x:A_{m}])&=&(\Xi_{1}\sqcup\Xi_{2}),x:A_{m}\\\
(\Xi_{1},[x:A_{m}])&\sqcup&(\Xi_{2},[x:A_{m}])&=&(\Xi_{1}\sqcup\Xi_{2}),[x:A_{m}]\\\
(\Xi_{1},x:A_{m})&\sqcup&\Xi_{2}&=&(\Xi_{1}\sqcup\Xi_{2}),x:A_{m}&\mbox{for
$x\not\in\rm{dom}(\Xi_{2})$, $\mathsf{W}\in\sigma(m)$}\\\
\Xi_{1}&\sqcup&(\Xi_{2},x:A_{m})&=&(\Xi_{1}\sqcup\Xi_{2}),x:A_{m}&\mbox{for
$x\not\in\rm{dom}(\Xi_{1})$, $\mathsf{W}\in\sigma(m)$}\\\
(\Xi_{1},[x:A_{m}])&\sqcup&\Xi_{2}&=&\Xi_{1}\sqcup\Xi_{2}&\mbox{for
$x\not\in\rm{dom}(\Xi_{2})$}\\\
\Xi_{1}&\sqcup&(\Xi_{2},[x:A_{m}])&=&\Xi_{1}\sqcup\Xi_{2}&\mbox{for
$x\not\in\rm{dom}(\Xi_{1})$}\\\ (\cdot)&\sqcup&(\cdot)&=&(\cdot)\end{array}$

We have the following two properties, expressing the property of a least upper
bound in two slightly asymmetric judgments.

###### Lemma 27

  1. (i)

If $\Delta\geq\Xi_{1}$ and $\Delta\geq\Xi_{2}$ then
$\Delta\geq\Xi_{1}\sqcup\Xi_{2}$.

  2. (ii)

if $\Xi_{1}\sqcup\Xi_{2}\sqsupseteq\Delta$ then for some $\Delta_{1}$,
$\Delta^{1}_{\mathsf{W}}$, $\Delta_{2}$, $\Delta^{2}_{\mathsf{W}}$ we have
$\Xi_{1}\sqsupseteq\Delta_{1}$, $\Xi_{2}\sqsupseteq\Delta_{2}$ and
$\Delta=\Delta_{1}\mathbin{;}\Delta_{\mathsf{W}}^{1}=\Delta_{2}\mathbin{;}\Delta_{\mathsf{W}}^{2}$.

The complete set of rules for algorithmic typing can be found in Figure 6.

$\begin{array}[]{c}\hbox{\vbox{ \moveright
0.0pt\vbox{\halign{\relax\global\@RightOffset=0pt \@ReturnLeftOffsettrue$#$&&
\inferTabSkip\global\@RightOffset=0pt
\@ReturnLeftOffsetfalse$#$\cr\Gamma\vdash s\Longrightarrow
A_{m}\mathrel{/}\Xi\cr}}\nointerlineskip\kern 4.0pt\moveright
0.0pt\vbox{\hrule width=59.74306pt}\nointerlineskip\kern 4.0pt\moveright
0.0pt\hbox{$\Gamma\vdash s\Longleftarrow A_{m}\mathrel{/}\Xi$}}\kern
0.0pt\kern 5.0pt\raise 9.0pt\hbox{${\Rightarrow}/{\Leftarrow}$}\kern
0.0pt}\hskip 10.00002pt\hbox{\vbox{ \moveright
14.26395pt\vbox{\halign{\relax\global\@RightOffset=0pt
\@ReturnLeftOffsettrue$#$&& \inferTabSkip\global\@RightOffset=0pt
\@ReturnLeftOffsetfalse$#$\cr\Gamma\vdash e\Longleftarrow
A_{m}\mathrel{/}\Xi\cr}}\nointerlineskip\kern 4.0pt\moveright
0.0pt\vbox{\hrule width=88.23972pt}\nointerlineskip\kern 4.0pt\moveright
0.0pt\hbox{$\Gamma\vdash(e:A_{m})\Longrightarrow A_{m}\mathrel{/}\Xi$}}\kern
0.0pt\kern 5.0pt\raise 9.0pt\hbox{${\Leftarrow}/{\Rightarrow}$}\kern
0.0pt}\hskip 10.00002pt\hbox{\vbox{ \moveright
14.38531pt\vbox{\halign{\relax\global\@RightOffset=0pt
\@ReturnLeftOffsettrue$#$&& \inferTabSkip\global\@RightOffset=0pt
\@ReturnLeftOffsetfalse$#$\cr x:A_{m}\in\Gamma\cr}}\nointerlineskip\kern
4.0pt\moveright 0.0pt\vbox{\hrule width=70.9304pt}\nointerlineskip\kern
4.0pt\moveright 0.0pt\hbox{$\Gamma\vdash x\Longrightarrow(x:A_{m})$}}\kern
0.0pt\kern 5.0pt\raise 9.0pt\hbox{$\mathsf{hyp}$}\kern 0.0pt}\\\\[10.00002pt]
\hbox{\vbox{ \moveright 28.09383pt\vbox{\halign{\relax\global\@RightOffset=0pt
\@ReturnLeftOffsettrue$#$&& \inferTabSkip\global\@RightOffset=0pt
\@ReturnLeftOffsetfalse$#$\cr\Gamma,x:A_{m}\vdash e\Longleftarrow
B_{m}\mathrel{/}\Xi\cr}}\nointerlineskip\kern 4.0pt\moveright
0.0pt\vbox{\hrule width=143.5072pt}\nointerlineskip\kern 4.0pt\moveright
0.0pt\hbox{$\Gamma\vdash\lambda x.\,e\Longleftarrow A_{m}\multimap
B_{m}\mathrel{/}(\Xi\mathrel{\backslash}x:A_{m})$}}\kern 0.0pt\kern
5.0pt\raise 8.08334pt\hbox{${\multimap}I$}\kern 0.0pt}\\\\[10.00002pt]
\hbox{\vbox{ \moveright 0.0pt\vbox{\halign{\relax\global\@RightOffset=0pt
\@ReturnLeftOffsettrue$#$&& \inferTabSkip\global\@RightOffset=0pt
\@ReturnLeftOffsetfalse$#$\cr\Gamma\vdash s\Longrightarrow A_{m}\multimap
B_{m}\mathrel{/}\Xi&\quad$\Gamma\vdash e\Longleftarrow
A_{m}\mathrel{/}\Xi^{\prime}\cr}}\nointerlineskip\kern 4.0pt\moveright
0.0pt\vbox{\hrule width=157.88742pt}\nointerlineskip\kern 4.0pt\moveright
40.1251pt\hbox{$\Gamma\vdash s\,e\Longrightarrow
B_{m}\mathrel{/}\Xi\mathbin{;}\Xi^{\prime}$}}\kern 0.0pt\kern 5.0pt\raise
8.97221pt\hbox{${\multimap}E$}\kern 0.0pt}\\\\[10.00002pt] \hbox{\vbox{
\moveright 42.69463pt\vbox{\halign{\relax\global\@RightOffset=0pt
\@ReturnLeftOffsettrue$#$&& \inferTabSkip\global\@RightOffset=0pt
\@ReturnLeftOffsetfalse$#$\cr\cr}}\nointerlineskip\kern 4.0pt\moveright
0.0pt\vbox{\hrule width=85.38925pt}\nointerlineskip\kern 4.0pt\moveright
0.0pt\hbox{$\Gamma\vdash\\{\,\\}\Longleftarrow{\mathbin{\binampersand}}_{m}\\{\,\\}\mathrel{/}[\Gamma|_{m}]$}}\kern
0.0pt\kern 5.0pt\raise 8.93001pt\hbox{${\mathbin{\binampersand}}I_{0}$}\kern
0.0pt}\hskip 20.00003pt\hbox{\vbox{ \moveright
12.24738pt\vbox{\halign{\relax\global\@RightOffset=0pt
\@ReturnLeftOffsettrue$#$&& \inferTabSkip\global\@RightOffset=0pt
\@ReturnLeftOffsetfalse$#$\cr\Gamma\vdash e_{\ell}\Longleftarrow
A^{\ell}_{m}\mathrel{/}\Xi_{\ell}\quad(\forall\ell\in
L\neq\emptyset)\cr}}\nointerlineskip\kern 4.0pt\moveright 0.0pt\vbox{\hrule
width=149.76184pt}\nointerlineskip\kern 4.0pt\moveright
0.0pt\hbox{$\Gamma\vdash\\{\ell\Rightarrow e_{\ell}\\}_{\ell\in
L}\Longleftarrow{\mathbin{\binampersand}}\\{\ell:A^{\ell}_{m}\\}_{\ell\in
L}\mathrel{/}\sqcup_{\ell\in L}\Xi_{\ell}$}}\kern 0.0pt\kern 5.0pt\raise
9.30554pt\hbox{${\mathbin{\binampersand}}I$}\kern 0.0pt}\\\\[10.00002pt]
\hbox{\vbox{ \moveright 0.0pt\vbox{\halign{\relax\global\@RightOffset=0pt
\@ReturnLeftOffsettrue$#$&& \inferTabSkip\global\@RightOffset=0pt
\@ReturnLeftOffsetfalse$#$\cr\Gamma\vdash
e\Longrightarrow{\mathbin{\binampersand}}\\{\ell:A^{\ell}_{m}\\}_{\ell\in
L}\mathrel{/}\Xi\quad(\ell\in L)\cr}}\nointerlineskip\kern 4.0pt\moveright
0.0pt\vbox{\hrule width=135.4783pt}\nointerlineskip\kern 4.0pt\moveright
33.5777pt\hbox{$\Gamma\vdash e.\ell\Longrightarrow
A_{m}^{\ell}\mathrel{/}\Xi$}}\kern 0.0pt\kern 5.0pt\raise
9.30554pt\hbox{${\mathbin{\binampersand}}E$}\kern 0.0pt}\\\\[10.00002pt]
\hbox{\vbox{ \moveright 17.11545pt\vbox{\halign{\relax\global\@RightOffset=0pt
\@ReturnLeftOffsettrue$#$&& \inferTabSkip\global\@RightOffset=0pt
\@ReturnLeftOffsetfalse$#$\cr\Gamma\vdash e\Longleftarrow
A_{k}\mathrel{/}\Xi\cr}}\nointerlineskip\kern 4.0pt\moveright
0.0pt\vbox{\hrule width=92.1175pt}\nointerlineskip\kern 4.0pt\moveright
0.0pt\hbox{$\Gamma\vdash\mathbf{susp}\;e\Longleftarrow{\uparrow}_{k}^{m}A_{k}\mathrel{/}\Xi\mathord{\|}_{m}$}}\kern
0.0pt\kern 5.0pt\raise 9.65pt\hbox{${{\uparrow}}I$}\kern 0.0pt}\hskip
30.00005pt\hbox{\vbox{ \moveright
7.77359pt\vbox{\halign{\relax\global\@RightOffset=0pt
\@ReturnLeftOffsettrue$#$&& \inferTabSkip\global\@RightOffset=0pt
\@ReturnLeftOffsetfalse$#$\cr\Gamma\vdash
s\Longrightarrow{\uparrow}_{k}^{m}A_{k}\mathrel{/}\Xi\cr}}\nointerlineskip\kern
4.0pt\moveright 0.0pt\vbox{\hrule width=78.77899pt}\nointerlineskip\kern
4.0pt\moveright 0.0pt\hbox{$\Gamma\vdash\mathbf{force}\;s\Longrightarrow
A_{k}\mathrel{/}\Xi$}}\kern 0.0pt\kern 5.0pt\raise
9.0pt\hbox{${{\uparrow}}E$}\kern 0.0pt}\\\\[10.00002pt] \hbox{\vbox{
\moveright 0.0pt\vbox{\halign{\relax\global\@RightOffset=0pt
\@ReturnLeftOffsettrue$#$&& \inferTabSkip\global\@RightOffset=0pt
\@ReturnLeftOffsetfalse$#$\cr\Gamma\vdash e_{1}\Longleftarrow
A_{m}\mathrel{/}\Xi&\quad$\Gamma\vdash e_{2}\Longleftarrow
B_{m}\mathrel{/}\Xi^{\prime}\cr}}\nointerlineskip\kern 4.0pt\moveright
0.0pt\vbox{\hrule width=131.59503pt}\nointerlineskip\kern 4.0pt\moveright
7.70831pt\hbox{$\Gamma\vdash(e_{1},e_{2})\Longleftarrow A_{m}\otimes
B_{m}\mathrel{/}\Xi\mathbin{;}\Xi^{\prime}$}}\kern 0.0pt\kern 5.0pt\raise
9.38889pt\hbox{${\otimes}I$}\kern 0.0pt}\\\\[10.00002pt] \hbox{\vbox{
\moveright 0.0pt\vbox{\halign{\relax\global\@RightOffset=0pt
\@ReturnLeftOffsettrue$#$&& \inferTabSkip\global\@RightOffset=0pt
\@ReturnLeftOffsetfalse$#$\cr\Gamma\vdash s\Longrightarrow A_{m}\otimes
B_{m}\mathrel{/}\Xi&\quad$m\geq r&\quad$\Gamma,x_{1}:A_{m},x_{2}:B_{m}\vdash
e^{\prime}\Longleftarrow
C_{r}\mathrel{/}\Xi^{\prime}\cr}}\nointerlineskip\kern 4.0pt\moveright
0.0pt\vbox{\hrule width=246.3533pt}\nointerlineskip\kern 4.0pt\moveright
8.82687pt\hbox{$\Gamma\vdash\mathbf{match}\;s\;((x_{1},x_{2})\Rightarrow
e^{\prime})\Longleftarrow
C_{r}\mathrel{/}\Xi\mathbin{;}(\Xi^{\prime}\mathrel{\backslash}x_{1}:A_{m}\mathrel{\backslash}x_{2}:B_{m})$}}\kern
0.0pt\kern 5.0pt\raise 9.38889pt\hbox{${\otimes}E$}\kern
0.0pt}\\\\[10.00002pt] \hbox{\vbox{ \moveright
31.55559pt\vbox{\halign{\relax\global\@RightOffset=0pt
\@ReturnLeftOffsettrue$#$&& \inferTabSkip\global\@RightOffset=0pt
\@ReturnLeftOffsetfalse$#$\cr\cr}}\nointerlineskip\kern 4.0pt\moveright
0.0pt\vbox{\hrule width=63.11118pt}\nointerlineskip\kern 4.0pt\moveright
0.0pt\hbox{$\Gamma\vdash(\,)\Longleftarrow\mathbf{1}_{m}\mathrel{/}(\cdot)$}}\kern
0.0pt\kern 5.0pt\raise 8.08334pt\hbox{${\mathbf{1}}I$}\kern 0.0pt}\hskip
30.00005pt\hbox{\vbox{ \moveright
0.0pt\vbox{\halign{\relax\global\@RightOffset=0pt \@ReturnLeftOffsettrue$#$&&
\inferTabSkip\global\@RightOffset=0pt
\@ReturnLeftOffsetfalse$#$\cr\Gamma\vdash
s\Longrightarrow\mathbf{1}_{m}\mathrel{/}\Xi&\quad$m\geq r&\quad$\Gamma\vdash
e^{\prime}\Longleftarrow
C_{r}\mathrel{/}\Xi^{\prime}\cr}}\nointerlineskip\kern 4.0pt\moveright
0.0pt\vbox{\hrule width=159.50984pt}\nointerlineskip\kern 4.0pt\moveright
10.97926pt\hbox{$\Gamma\vdash\mathbf{match}\;s\;((\,)\Rightarrow
e^{\prime})\Longleftarrow C_{r}\mathrel{/}\Xi\mathbin{;}\Xi^{\prime}$}}\kern
0.0pt\kern 5.0pt\raise 8.97221pt\hbox{${\mathbf{1}}E$}\kern
0.0pt}\\\\[10.00002pt] \hbox{\vbox{ \moveright
4.17506pt\vbox{\halign{\relax\global\@RightOffset=0pt
\@ReturnLeftOffsettrue$#$&& \inferTabSkip\global\@RightOffset=0pt
\@ReturnLeftOffsetfalse$#$\cr\Gamma\vdash e\Longleftarrow
A_{m}^{\ell}\mathrel{/}\Xi\quad(\ell\in L)\cr}}\nointerlineskip\kern
4.0pt\moveright 0.0pt\vbox{\hrule width=109.03413pt}\nointerlineskip\kern
4.0pt\moveright
0.0pt\hbox{$\Gamma\vdash\ell(e)\Longleftarrow{\oplus}\\{\ell:A_{m}^{\ell}\\}_{\ell\in
L}\mathrel{/}\Xi$}}\kern 0.0pt\kern 5.0pt\raise
9.77777pt\hbox{${\oplus}I$}\kern 0.0pt}\hskip 30.00005pt\hbox{\vbox{
\moveright 6.74509pt\vbox{\halign{\relax\global\@RightOffset=0pt
\@ReturnLeftOffsettrue$#$&& \inferTabSkip\global\@RightOffset=0pt
\@ReturnLeftOffsetfalse$#$\cr\Gamma\vdash
s\Longrightarrow\oplus_{m}\\{\,\\}\mathrel{/}\Xi&\quad$m\geq
r\cr}}\nointerlineskip\kern 4.0pt\moveright 0.0pt\vbox{\hrule
width=119.3027pt}\nointerlineskip\kern 4.0pt\moveright
0.0pt\hbox{$\Gamma\vdash\mathbf{match}\;s\;(\,)\Longleftarrow
C_{r}\mathrel{/}\Xi\mathrel{;}[\Gamma|_{r}]$}}\kern 0.0pt\kern 5.0pt\raise
8.98557pt\hbox{${\oplus}E_{0}$}\kern 0.0pt}\\\\[10.00002pt] \hbox{\vbox{
\moveright 0.0pt\vbox{\halign{\relax\global\@RightOffset=0pt
\@ReturnLeftOffsettrue$#$&& \inferTabSkip\global\@RightOffset=0pt
\@ReturnLeftOffsetfalse$#$\cr\Gamma\vdash
s\Longrightarrow{\oplus}\\{\ell:A_{m}^{\ell}\\}_{\ell\in
L}\mathrel{/}\Xi&\quad$m\geq r&\quad$\Gamma,x:A_{m}^{\ell}\vdash
e_{\ell}\Longleftarrow C_{r}\mathrel{/}\Xi^{\prime}_{\ell}\quad(\forall\ell\in
L\neq\emptyset)\cr}}\nointerlineskip\kern 4.0pt\moveright 0.0pt\vbox{\hrule
width=296.22949pt}\nointerlineskip\kern 4.0pt\moveright
44.68263pt\hbox{$\Gamma\vdash\mathbf{match}\;s\;(\ell(x)\Rightarrow
e_{\ell})_{\ell\in L}\Longleftarrow
C_{r}\mathrel{/}\Xi\mathbin{;}\sqcup_{\ell\in
L}(\Xi^{\prime}_{\ell}\mathrel{\backslash}x:A_{\ell})$}}\kern 0.0pt\kern
5.0pt\raise 9.38889pt\hbox{${\oplus}E$}\kern 0.0pt}\\\\[10.00002pt]
\hbox{\vbox{ \moveright 14.31961pt\vbox{\halign{\relax\global\@RightOffset=0pt
\@ReturnLeftOffsettrue$#$&& \inferTabSkip\global\@RightOffset=0pt
\@ReturnLeftOffsetfalse$#$\cr\Gamma\vdash e\Longleftarrow
A_{n}\mathrel{/}\Xi\cr}}\nointerlineskip\kern 4.0pt\moveright
0.0pt\vbox{\hrule width=86.79549pt}\nointerlineskip\kern 4.0pt\moveright
0.0pt\hbox{$\Gamma\vdash\mathbf{down}\;e\Longleftarrow{\downarrow}^{n}_{m}A_{n}\mathrel{/}\Xi$}}\kern
0.0pt\kern 5.0pt\raise 9.65pt\hbox{${{\downarrow}}I$}\kern
0.0pt}\\\\[10.00002pt] \hbox{\vbox{ \moveright
0.14218pt\vbox{\halign{\relax\global\@RightOffset=0pt
\@ReturnLeftOffsettrue$#$&& \inferTabSkip\global\@RightOffset=0pt
\@ReturnLeftOffsetfalse$#$\cr\Gamma\vdash
s\Longrightarrow{\downarrow}^{n}_{m}A_{n}\mathrel{/}\Xi&\quad$m\geq
r&\quad$\Gamma,x:A_{n}\vdash e^{\prime}\Longleftarrow
C_{r}\mathrel{/}\Xi^{\prime}\cr}}\nointerlineskip\kern 4.0pt\moveright
0.0pt\vbox{\hrule width=194.56537pt}\nointerlineskip\kern 4.0pt\moveright
0.0pt\hbox{$\Gamma\vdash\mathbf{match}\;s\;(\mathbf{down}\;x\Rightarrow
e^{\prime})\Longleftarrow
C_{r}\mathrel{/}\Xi\mathbin{;}(\Xi^{\prime}\mathrel{\backslash}x:A_{n})$}}\kern
0.0pt\kern 5.0pt\raise 9.88887pt\hbox{${{\downarrow}}E$}\kern
0.0pt}\end{array}$ Figure 6: Algorithmic Typing for Natural Deduction

###### Lemma 28

If $\Gamma\vdash e\Longleftrightarrow A_{m}\mathrel{/}\Xi$ then $\Xi\geq m$

  * Proof:

By straightforward rule induction. $\Box$

######  Lemma 29 (Context Extension)

If $\Gamma\vdash e\Longleftrightarrow A_{m}\mathrel{/}\Xi$ and
$\Xi\sqsupseteq\Delta$ and $\Gamma^{\prime}\supseteq\Gamma$ then
$\Gamma^{\prime}\vdash e\Longleftrightarrow A_{m}\mathrel{/}\Xi^{\prime}$ for
some $\Xi^{\prime}$ with $\Xi^{\prime}\sqsupseteq\Delta$.

  * Proof:

By rule induction on the given derivation and inversion on the definition of
$\Xi\sqsupseteq\Delta$. $\Box$

  * Proof:

(of soundness, Theorem 23) By rule induction on the algorithmic typing
derivation and inversion of the $\Xi\sqsupseteq\Delta$ judgment. $\Box$

  * Proof:

(of completeness, Theorem 24) By rule induction on the given bidirectional
typing. $\Box$

###### Corollary 30

$\cdot\vdash e\Longleftarrow A_{m}$ iff $\cdot\vdash e\Longleftarrow
A_{m}/\cdot$

##  8 Conclusion

There have been recent proposals to extend the adjoint approach to combining
logics to dependent types. Licata and Shulman (2016); Licata et al. (2017)
permit dependent types and richer connections between the logics that are
combined, but certain properties such as independence are no longer
fundamental, but have to be proved in each case where they apply. While they
mostly stay within a sequent calculus, they also briefly introduce natural
deduction. They also provide a categorical semantics. Hanukaev and Eades
(2023) also permit dependent types and use the graded/algebraic approach to
defining their system. However, their approach to dependency appears
incompatible with control of contraction, so their adjoint structure is not
nearly as general as ours. They also omit empty internal choice (and external
choice altogether), which created some of the trickiest issues in our system.
Neither of these proposes an algorithm for type checking or an operational
semantics that would exploit the substructural and mode properties to obtain
“free theorems” about well-typed programs.

We are pursuing several avenues of future work building on the results of this
paper. On the foundational side, we are looking for a direct algorithm to
convert an arbitrary natural deduction into a verification. On the programming
side, we are considering _mode polymorphism_ , that is, type-checking the same
expression against multiple different modes to avoid code duplication. On the
application side, we are considering staged computation, quotation, and
metaprogramming, decomposing the usual type $\Box A$ or its contextual
analogue along the lines of Section 3.

## References

  * Abel et al. [2023] Andreas Abel, Nils Anders Danielsson, and Oskar Eriksson.  A graded modal dependent type theory with a universe and erasure, formalized.  _Proc. ACM Program. Lang._ , 7(ICFP’23):920–954, 2023.
  * Abramsky [1993] Samson Abramsky.  Computational interpretations of linear logic.  _Theoretical Computer Science_ , 111:3–57, 1993.
  * Accattoli et al. [2014] Beniamino Accattoli, Pablo Barenbaum, and Damiano Mazza.  Distilling abstract machines.  In J. Jeuring and M. Chakravarty, editors, _19th International Conference on Functional Programming (ICFP 2014)_ , pages 363–376, Gothenburg, Sweden, September 2014. ACM.  Long version available at arXiv:1406.2370.
  * Amadio and Cardelli [1993] Roberto M. Amadio and Luca Cardelli.  Subtyping recursive types.  _ACM Transactions on Programming Languages and Systems_ , 15(4):575–631, 1993.
  * Andreoli [1992] Jean-Marc Andreoli.  Logic programming with focusing proofs in linear logic.  _Journal of Logic and Computation_ , 2(3):197–347, 1992.
  * Atkey [2018] Robert Atkey.  Syntax and semantics of quantitative type theory.  In Anuj Dawar and Erich Grädel, editors, _33rd Conference on Logic in Computer Science (LICS 2018)_ , pages 56–65, Oxford, UK, July 2018. ACM.
  * Barber [1996] Andrew Barber.  Dual intuitionistic linear logic.  Technical Report ECS-LFCS-96-347, Department of Computer Science, University of Edinburgh, September 1996.
  * Benton [1994a] P. N. Benton.  A mixed linear and non-linear logic: Proofs, terms and models.  In Leszek Pacholski and Jerzy Tiuryn, editors, _Selected Papers from the 8th International Workshop on Computer Science Logic (CSL’94)_ , pages 121–135, Kazimierz, Poland, September 1994a. Springer LNCS 933.  An extended version appears as Technical Report UCAM-CL-TR-352, University of Cambridge.
  * Benton [1994b] P. N. Benton.  A mixed linear and non-linear logic: Proofs, terms and models (preliminary report).  Technical Report UCAM-CL-TR-352, University of Cambridge, October 1994b.  URL https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-352.html.
  * Benton and Wadler [1996] P. N. Benton and Philip Wadler.  Linear logic, monads, and the lambda calculus.  In E. Clarke, editor, _Proceedings of the 11th Annual Symposium on Logic in Computer Science_ , pages 420–431, New Brunswick, New Jersey, July 1996. IEEE Computer Society Press.
  * Benton et al. [1993] P. N. Benton, Gavin M. Bierman, Martin Hyland, and Valeria de Paiva.  A term calculus for intuitionistic linear logic.  In _International Conference on Typed Lambda Calculi and Applications_ , pages 75–90, Utrecht, The Netherlands, 1993. Springer LNCS 664\.
  * Benton et al. [1998] P. N. Benton, G. M. Bierman, and V. C. V. de Paiva.  Computational types from a logical perspective.  _Journal of Functional Programming_ , 8(2):177–193, March 1998.
  * Caires and Pfenning [2010] Luís Caires and Frank Pfenning.  Session types as intuitionistic linear propositions.  In _Proceedings of the 21st International Conference on Concurrency Theory (CONCUR 2010)_ , pages 222–236, Paris, France, August 2010\. Springer LNCS 6269.
  * Caires et al. [2016] Luís Caires, Frank Pfenning, and Bernardo Toninho.  Linear logic propositions as session types.  _Mathematical Structures in Computer Science_ , 26(3):367–423, 2016.  Special Issue on Behavioural Types.
  * Cervesato et al. [2000] Iliano Cervesato, Joshua S. Hodas, and Frank Pfenning.  Efficient resource management for linear logic proof search.  _Theoretical Computer Science_ , 232(1–2):133–163, February 2000.  Special issue on Proof Search in Type-Theoretic Languages, D. Galmiche and D. Pym, editors.
  * Chang et al. [2003] Bor-Yuh Evan Chang, Kaustuv Chaudhuri, and Frank Pfenning.  A judgmental analysis of linear logic.  Technical Report CMU-CS-03-131R, Carnegie Mellon University, Department of Computer Science, December 2003.
  * Chaudhuri [2010] Kaustuv Chaudhuri.  Classical and intuitionistic subexponential logics are equally expressive.  In _Computer Science Logic_ , pages 185–199. Springer LNCS 6247, August 2010.
  * Choudhury et al. [2021] Pritam Choudhury, Harley Eades III, Richard A. Eisenberg, and Stephanie Weirich.  A graded dependent type system with a usage-aware semantics.  _Proc. ACM Program. Lang._ , 5(POPL’21):1–32, 2021.
  * Church [1941] Alonzo Church.  _The Calculi of Lambda-Conversion_ , volume 6 of _Annals of Mathematics Studies_.  Princeton University Press, 1941.
  * Dummett [1991] Michael Dummett.  _The Logical Basis of Metaphysics_.  Harvard University Press, Cambridge, Massachusetts, 1991.  The William James Lectures, 1976.
  * Gentzen [1935] Gerhard Gentzen.  Untersuchungen über das logische Schließen.  _Mathematische Zeitschrift_ , 39:176–210, 405–431, 1935\.  English translation in M. E. Szabo, editor, The Collected Papers of Gerhard Gentzen, pages 68–131, North-Holland, 1969.
  * Girard [1987] Jean-Yves Girard.  Linear logic.  _Theoretical Computer Science_ , 50:1–102, 1987.
  * Girard and Lafont [1987] Jean-Yves Girard and Yves Lafont.  Linear logic and lazy computation.  In H. Ehrig, R. Kowalski, G. Levi, and U. Montanari, editors, _Proceedings of the International Joint Conference on Theory and Practice of Software Development_ , volume 2, pages 52–66, Pisa, Italy, March 1987\. Springer-Verlag LNCS 250.
  * Hanukaev and Eades [2023] Peter Hanukaev and Harley Eades, III.  Combining dependency, grades, and adjoint logic.  In _8th Workshop on Type-Driven Development (TyDe 2023)_ , pages 58–70, Seattle, Washington, 2023. ACM.  URL https://arxiv.org/abs/2307.09563.
  * Harper [2016] Robert Harper.  _Practical Foundations for Programming Languages_.  Cambridge University Press, second edition, April 2016.
  * Howard [1969] W. A. Howard.  The formulae-as-types notion of construction.  Unpublished note. An annotated version appeared in: _To H.B. Curry: Essays on Combinatory Logic, Lambda Calculus and Formalism_ , 479–490, Academic Press (1980), 1969.
  * Hughes and Orchard [2020] Jack Hughes and Dominic Orchard.  Resourceful program synthesis from graded linear types.  In M. Fernández, editor, _30th International Symposium on Logic-Based Program Synthesis and Transformation (LOPSTR 2020)_ , pages 151–170, Bologna, Italy, September 2020. Springer LNCS 12561.
  * Kanovich et al. [2017] Max Kanovich, Stepan Kuznetsov, Vivek Nigam, and Andre Scedrov.  Subexponentials in non-commutative linear logic.  _ArXiv e-prints_ , September 2017.
  * Kanovich et al. [2018] Max Kanovich, Stepan Kuznetsov, Vivek Nigam, and Andre Scedrov.  A logical framework with commutative and non-commutative subexponentials.  In _International Joint Conference on Automated Reasoning (IJCAR 2018)_ , pages 228–245. Springer LNAI 10900, 2018.
  * Kleene [1952] Stephen Cole Kleene.  _Introduction to Metamathematics_.  North-Holland, 1952.
  * Launchbury [1993] John Launchbury.  A natural semantics for lazy evaluation.  In _20th Annual Symposium on Principles of Programming Languages (POPL 1993)_ , pages 144–154, Charleston, South Carolina, January 1993. ACM.
  * Levy [2006] Paul Blain Levy.  Call-by-push-value: Decomposing call-by-value and call-by-name.  _Higher-Order and Symbolic Computation_ , 19(4):377–414, 2006.
  * Licata and Shulman [2016] Daniel R. Licata and Michael Shulman.  Adjoint logic with a 2-category of modes.  In _International Symposium on Logical Foundations of Computer Science (LFCS)_ , pages 219–235. Springer LNCS 9537, January 2016.
  * Licata et al. [2017] Daniel R. Licata, Michael Shulman, and Mitchell Riley.  A fibrational framework for substructural and modal logics.  In Dale Miller, editor, _Proceedings of the 2nd International Conference on Formal Structures for Computation and Deduction (FSCD’17)_ , pages 25:1–25:22, Oxford, UK, September 2017. LIPIcs.
  * Martin-Löf [1983] Per Martin-Löf.  On the meanings of the logical constants and the justifications of the logical laws.  Notes for three lectures given in Siena, Italy. Published in _Nordic Journal of Philosophical Logic_ , 1(1):11-60, 1996, April 1983.
  * Moon et al. [2021] Benjamin Moon, Harley Eades III, and Dominic Orchard.  Graded modal dependent type theory.  In _30th European Symposium on Programming (ESOP’21)_ , volume 12648 of _Lecture Notes in Computer Science_ , pages 462–490. Springer, 2021\.
  * Mycroft [1980] Alan Mycroft.  The theory and practice of transforming call-by-need into call-by-value.  In _4th International Symposium on Programming_ , pages 269–281. Springer LNCS 83, 1980.
  * Nanevski et al. [2008] Aleksandar Nanevski, Frank Pfenning, and Brigitte Pientka.  Contextual modal type theory.  _Transactions on Computational Logic_ , 9(3), 2008.
  * Nigam and Miller [2009] Vivek Nigam and Dale Miller.  Algorithmic specifications in linear logic with subexponentials.  In _Proceedings of the 11th International Conference on Principles and Practice of Declarative Programming (PPDP)_ , pages 129–140, Coimbra, Portugal, sep 2009. ACM.
  * Nigam et al. [2016] Vivek Nigam, Elaine Pimental, and Giselle Reis.  An extended framework for specifying and reasoning about proof systems.  _Journal of Logic and Computation_ , 26(2):539–576, 2016.
  * Pfenning and Davies [2001] Frank Pfenning and Rowan Davies.  A judgmental reconstruction of modal logic.  _Mathematical Structures in Computer Science_ , 11:511–540, 2001.  Notes to an invited talk at the Workshop on Intuitionistic Modal Logics and Applications (IMLA’99), Trento, Italy, July 1999.
  * Pfenning and Pruiksma [2023] Frank Pfenning and Klaas Pruiksma.  Relating message passing and shared memory, proof-theoretically.  In S. Jongmans and A. Lopes, editors, _25th International Conference on Coordination Models and Languages (COORDINATION 2023)_ , pages 3–27, Lisbon, Portugal, June 2023. Springer LNCS 13908.  Notes to an invited talk.
  * Prawitz [1965] Dag Prawitz.  _Natural Deduction_.  Almquist & Wiksell, Stockholm, 1965.
  * Pruiksma [2024] Klaas Pruiksma.  _Adjoint Logic with Applications_.  PhD thesis, Carnegie Mellon University, 2024.  In preparation.
  * Pruiksma and Pfenning [2021] Klaas Pruiksma and Frank Pfenning.  A message-passing interpretation of adjoint logic.  _Journal of Logical and Algebraic Methods in Programming_ , 120(100637), 2021.
  * Pruiksma and Pfenning [2022] Klaas Pruiksma and Frank Pfenning.  Back to futures.  _Journal of Functional Programming_ , 32:e6, 2022.
  * Pruiksma et al. [2018] Klaas Pruiksma, William Chargin, Frank Pfenning, and Jason Reed.  Adjoint logic.  Unpublished manuscript, April 2018.  URL http://www.cs.cmu.edu/~fp/papers/adjoint18b.pdf.
  * Reed [2009] Jason Reed.  A judgmental deconstruction of modal logic.  Unpublished manuscript, May 2009.  URL http://www.cs.cmu.edu/~jcreed/papers/jdml2.pdf.
  * [49] Rust.  The Rust programming language.  Available at https://www.rust-lang.org/.  Accessed January 15, 2024.
  * Tov and Pucella [2011] Jesse A. Tov and Riccardo Pucella.  Practical affine types.  In T. Ball and M. Sagiv, editors, _Proceedings of the 38th Symposium on Principles of Programming Languages (POPL 2011)_ , pages 447–458. ACM Press, January 2011.
  * Troelstra [1995] Anne S. Troelstra.  Natural deduction for intuitionistic linear logic.  _Annals of Pure and Applied Logic_ , 73(1):79–108, 1995.
  * Wadler [1990] Philip Wadler.  Linear types can change the world.  In _IFIP TC_ , volume 2, pages 347–359, 1990.
  * Wadler [2012] Philip Wadler.  Propositions as sessions.  In _Proceedings of the 17th International Conference on Functional Programming (ICFP 2012)_ , pages 273–286, Copenhagen, Denmark, September 2012. ACM Press.
  * Wood and Atkey [2022] James Wood and Robert Atkey.  A framework for substructural type systems.  In Ilya Sergey, editor, _31st European Symposium on Programming (ESOP 2022)_ , pages 376–402, Munich, Germany, April 2022. Springer LNCS 13240\.