# Quantum Data Reduction with Application to Video Classification

Kostas Blekos  Computer Engineering & Informatics Dept., University of Patras,
Rio, Greece<EMAIL_ADDRESS>Dimitrios Kosmopoulos  Computer Engineering &
Informatics Dept., University of Patras, Rio, Greece<EMAIL_ADDRESS>
###### Abstract

We investigate a quantum video classification method using a hybrid algorithm.
A quantum-classical step performs a data reduction on the video dataset and a
quantum step—which only has access to the reduced dataset—classifies the video
to one of $k$ classes. We verify the method using sign videos and demonstrate
that the reduced dataset contains enough information to successfully classify
the data, using a quantum classification process.

The proposed data reduction method showcases a way to alleviate the “data
loading” problem of quantum computers for the problem of video classification.
Data loading is a huge bottleneck, as there are no known efficient techniques
to perform that task without sacrificing many of the benefits of quantum
computing.

##  1 Introduction

In the past few years there has been a lot of interest in exploring quantum
computing applications for classical computational problems. Quantum
computing, by taking advantage of the quantum mechanical properties of nature,
offers a new toolbox for attacking such problems with potentially great
benefits in efficiency and performance. Image and video classification
algorithms are an important subset of algorithms where quantum computing
applications are actively investigated 1, 2, 3, 4, 5, 6, 7, though, to our
best knowledge, applications on video processing are only sparsely researched
8.

One important issue with quantum video processing applications is that they
require the transfer of large amount of data from the classical machines to
the quantum computer. This “data loading” of the quantum computer is a huge
bottleneck as there are no known efficient techniques that can perform such a
task without sacrificing many of the quantum-gains 9, 10, 11, 12. This problem
is not unique to video processing applications 13. For example, solving a
system of linear equations through the HHL algorithm is usually stated as
needing $O(\log(n))$ steps for $n$ equations. This statement, though, hides
many assumptions, one of which is that one should be able to read and store
$n$ parameters on at most $O(\log(n))$ steps 14.

In general, it is possible that near-term applications will mostly concern the
cooperation of the classical hardware with small quantum computing units 9.
This highlights the importance of reducing the classical data space before
communicating with the quantum hardware. One proposal, by Harrow 9, is using
hybrid classical-quantum algorithms where a “representative” subset of a
problem’s data set is extracted. This subset is then used as input to the
quantum computer instead of the whole data set.

###  1.1 Contribution

In this work we propose a data reduction scheme on a hybrid classical-quantum
algorithm for video classification. Using a hybrid procedure, we extract the
most important pixels of a video which we then use as the reduced “training
set”. We classify each new video based only on these pixel distributions. We
perform simulations of the proposed algorithm verifying that meaningful
information can be extracted from just a small percentage of the initial
video. The videos are accurately classified showing that the algorithm is
capable of accurately identifying the most significant pixels using an
efficient quantum procedure. To the best of our knowledge this is the first
such approach.

In Section 2 we provide the background knowledge. In Section 3 we describe the
proposed method. In Section 4 we present our results on a public dataset.
Finally in Section 5 we discuss the merits and constraints of the method.

##  2 Quantum background

Quantum computers use quantum states of two levels (a _qubit_) to store and
process information instead of using bits of 0 and 1. Abstractly, a qubit is a
two-dimensional vector of complex parameters and norm 1, i.e.,
$\ket{\text{qubit}}=\begin{pmatrix}a\\\ b\end{pmatrix}$ with
$a,b\in\mathbb{C}$ and $|a|^{2}+|b|^{2}=1$. Similarly, a series of $n$ quantum
bits (often referred to as a _quantum register_) form a $2^{n}$-dimensional
vector of complex parameters and norm 1. We define the _computational basis_
as the one-hot orthonormal basis:

$\displaystyle\ket{0}=\begin{pmatrix}1\\\ 0\\\ \ldots\\\
0\end{pmatrix},\ket{1}=\begin{pmatrix}0\\\ 1\\\ \ldots\\\
0\end{pmatrix},\ldots\ket{2^{n}-1}=\begin{pmatrix}0\\\ 0\\\ \ldots\\\
1\end{pmatrix},$

Using these definitions, a general quantum register is written as a linear
combination of the computational basis
$\ket{q}=a_{0}\ket{0}+a_{1}\ket{1}+\cdots+a_{2^{n}-1}\ket{2^{n}-1}$. The
parameter $a_{i}$ is called the _amplitude_ of the state $\ket{i}$. The actual
parameters of a quantum register are unknowable. When the value of a quantum
register is needed a _measurement_ is performed. The result of the measurement
gives _one_ of the computational basis vectors. In particular, the result of
measuring the register $\ket{q}$ above is $\ket{i}$ with probability
$|a_{i}|^{2}$.

Quantum registers are manipulated by use of quantum gates (also called quantum
operators) which act as the quantum analog of the classical logical gates
(AND, OR, NOT, etc). Quantum gates are represented by complex unitary matrices
of appropriate dimensions and their action is calculated by simple matrix
multiplication.

###  2.1 Inner product estimation

The _Inner product estimation_ is an efficient subroutine for the estimation
of the inner product between two quantum registers $\ket{a},\ket{b}$. The
subroutine has three steps. Firstly an ancilla qubit is entangled with the two
quantum registers producing the register:

$\displaystyle\ket{\phi}$
$\displaystyle=\frac{1}{\sqrt{2}}\Bigl{(}\ket{0}_{\alpha}\ket{a}+\ket{1}_{\alpha}\ket{b}\Bigr{)}$

Secondly, a Hadamard gate $H=\frac{1}{\sqrt{2}}\begin{pmatrix}1&1\\\
1&-1\end{pmatrix}$ with
$H\ket{0}=\frac{1}{\sqrt{2}}\Bigl{(}\ket{0}+\ket{1}\Bigr{)}$ and
$H\ket{1}=\frac{1}{\sqrt{2}}\Bigl{(}\ket{0}-\ket{1}\Bigr{)}$, is applied on
the ancilla qubit:

$\displaystyle H_{\alpha}\ket{\phi}$
$\displaystyle=\frac{1}{\sqrt{2}}\Biggl{(}\frac{1}{\sqrt{2}}\Bigl{(}\ket{0}_{\alpha}+\ket{1}_{\alpha}\Bigr{)}\ket{a}$
$\displaystyle+\frac{1}{\sqrt{2}}\Bigl{(}\ket{0}_{\alpha}-\ket{1}_{\alpha}\Bigr{)}\ket{b}\Biggr{)}$
$\displaystyle=\frac{1}{2}\left(\ket{0}_{\alpha}\Bigl{(}\ket{a}+\ket{b}\Bigr{)}+\ket{1}_{\alpha}\Bigl{(}\ket{a}-\ket{b}\Bigr{)}\right)$

Then, a measurement of the ancilla qubit is performed. The probability of
measuring $\ket{0}_{\alpha}$ is its amplitude squared, thus:

$\displaystyle P(\ket{0}_{\alpha})$
$\displaystyle=\left\lVert\frac{1}{2}\left(\ket{a}+\ket{b}\right)\right\rVert^{2}$
$\displaystyle=\frac{1}{4}\sqrt{\sum_{i}(\overline{a_{i}+b_{i}})(a_{i}+b_{i})}^{2}$
$\displaystyle=\frac{1}{4}\left|\sum_{i}\overline{a}_{i}a_{i}+\overline{a}_{i}b_{i}+\overline{b}_{i}a_{i}+\overline{b}_{i}b_{i}\right|$
$\displaystyle=\frac{1}{4}\left|\braket{a}{a}+\braket{a}{b}+\braket{b}{a}+\braket{b}{b}\right|$
$\displaystyle=\frac{1+\braket{a}{b}}{2}$

since, by definition, $\braket{\cdot}{\cdot}\geq 0$ and $\braket{x}{x}=1$.
Therefore, by repeatedly measuring the ancilla qubit an estimate for the inner
product $\braket{a}{b}$ can be calculated.

###  2.2 Amplitude encoding

One often employed method to encode classical data to quantum registers is the
_amplitude encoding_ in which one encodes the information of a classical
vector on the amplitudes of the computational basis. So, a vector
$v=\begin{pmatrix}v_{0},&v_{1},&\ldots,&v_{n}\end{pmatrix}$ is encoded to the
quantum register

$\ket{v}=\frac{1}{\lVert v\rVert}\sum v_{i}\ket{i}.$

###  2.3 QRAM

A “Quantum RAM” (QRAM) is a classical or quantum data structure that outputs
quantum states. Like its classical analog RAM, it is used to store and
retrieve information in a “Random Access” model, i.e. any desired bit of
information can be addressed individually at will. Various QRAM models have
been proposed that provide efficient implementations for the crucial quantum
state storing and retrieving procedures 15, 16 The inner product estimation
algorithm, for example, that is used in this work, can be efficiently
implemented using a QRAM 17.

##  3 Methodology

We are going to demonstrate the proposed method by means of a video
classification application. The task is to classify short videos displaying
different types of hand motion into one of $k$ classes. The methodology can be
easily generalized to classification tasks involving high-volume data.

We assume that each video is represented by an $N\times N\times T$ matrix
(height$\times$width$\times$frames) of values in the range [0..1] and belongs
to one of $k$ classes. We will reduce the information contained in this
$N\times N\times T$-sized matrix to a $2^{q}$-sized quantum state using $q$
qubits. When a classical-to-quantum video conversion is needed we use the
amplitude encoding as described in Section 2.2. A training set consists of
$M\times k$ videos with $M$ videos for each class. The overview of the
algorithm is given in Algorithm 1 and schematically in Figure 1.

singlelinecheck=off

Figure 1: Schematic of the algorithm.

  1. 1.

Convert training set to quantum states.

  2. 2.

Transform and store in a QRAM.

  3. 3.

Repeat (1–2) and keep average of each class.

  4. 4.

Perform $2^{q}$ measurements on the averages of each class.

  5. 5.

Use measurement results to produce a training quantum register for each class.

  6. 6.

Perform inner product estimation. Classify to best match.

1\. Convert frames of the training set to quantum states. Using the amplitude
encoding we convert successive frames to quantum states.

2\. Perform “difference” transform on the quantum states and store in a QRAM.
A _difference transform_ efficiently converts two successive frames
$\ket{q_{1}}$ and $\ket{q_{2}}$ to their difference $\ket{q_{1}}-\ket{q_{2}}$
by using an ancilla qubit and a Hadamard gate, similarly to the inner product
estimation subroutine of Section 2.1:

$\displaystyle\ket{\phi}$
$\displaystyle=\frac{1}{\sqrt{2}}\Bigl{(}\ket{0}_{\alpha}\ket{q_{1}}+\ket{1}_{\alpha}\ket{q_{2}}\Bigr{)}$
$\displaystyle H_{\alpha}\ket{\phi}$
$\displaystyle=\frac{1}{\sqrt{2}}\left(\ket{0}_{\alpha}\Bigl{(}\ket{q_{1}}+\ket{q_{2}}\Bigr{)}+\ket{1}_{\alpha}\Bigl{(}\ket{q_{1}}-\ket{q_{2}}\Bigr{)}\right)$

By measuring the ancilla until we find it at the state $\ket{1}_{\alpha}$ we
store the difference $\ket{q_{1}}-\ket{q_{2}}$ in the QRAM.

3\. Repeat (1–2) for all the frames of each video averaging over all training
videos for each class. We repeat the previous steps until all difference
frames of all training videos are stored in the QRAM. An average difference-
video is then calculated for each class: ${\text{diff-
video}_{1}},\ldots,{\text{diff-video}_{k}}$.

4\. Load average videos and perform $2^{q}$ measurements to get a distribution
for each class. We load the $k$ difference videos to quantum registers of size
$N\times N\times T$. We measure $2^{q}$ times to get a distribution—with
replacement—that represents the most important pixels from all frames, as
these are the pixels that have the highest amplitudes. This creates a
_weighted_ distribution of the most important pixels for each class of videos.
Since the amplitudes have been calculated by the differences between
successive frames, a low amplitude means that there is no significant change
between the frames whereas a high amplitude means that there is significant
difference between the successive frames. At the end of this procedure we have
$k$ distributions of $2^{q}$ pixel-coordinates.

5\. Use the distributions and the average videos to produce a training quantum
register for each class. Using the class distributions we convert each average
video to a quantum register using $q$ qubits. All pixels not belonging to the
distributions are ignored. We produce $k$ training quantum registers
$\ket{v_{1}},\ldots,\ket{v_{k}}$.

6\. Classify a new video by reducing it using the distributions and performing
an inner product estimation. To classify a new video we use the class
distributions to convert it to a quantum register of $q$ qubits. We keep as
amplitudes of the video the pixels that belong to the class distributions and
ignore all other pixels. For each video we produce $k$ test quantum registers
$\ket{V_{1}},\ldots,\ket{V_{k}}$. We use the inner product estimation
subroutine to calculate the $k$ inner products:
$\braket{v_{1}}{V_{1}},\ldots,\braket{v_{k}}{V_{k}}$. We assign the video to
the class:

$\text{class}=\arg\max\Bigl{(}\braket{v_{1}}{V_{1}},\ldots,\braket{v_{k}}{V_{k}}\Bigr{)}.$

Effect of the number of qubits in performance. The number of qubits $q$ used
in the algorithm affects the performance of the sampling steps (4, 5) and the
classification step (6). Performances of steps 1, 2 and 3 depend dominantly on
the dimensions of the dataset. It is a basic assumption of this work, as
stated earlier, that converting classical data to quantum data is an expensive
process that we need to avoid. We, therefore, off-load the data conversion to
few key instances of the training steps and then use a reduced dataset to make
the prediction. In other words, we convert the performance of step 6—the data-
loading-depended prediction step—from $O(f(N\times N\times t)g(N\times N\times
t))$ to $O(f(2^{q})g(2^{q}))$, where $f(n)$ is some function that models the
performance of the data-loading procedure and $g(n)$ is some function that
models the performance of the prediction procedure. At the moment it seems
that a relation of $O(f(n))\sim O(n)$ is unavoidable 9. On the other hand, the
simplistic $g$ that we use here might have $O(g(n))\sim O(\sqrt{\log(n)})$ 18,
19.

Algorithm 1 Video classification through quantum data reduction

for All videos do

Convert frames to quantum registers.

Apply “difference” transform to consecutive frames.

Store result in QRAM.

end for

for All classes do

Calculate average video.

Get sample distribution by measuring $2^{q}$ times each average video.

Get training quantum register by sampling the average video using the class
distribution.

end for

Reduce a new video by sampling using the class distributions.

Classify by taking the inner product estimation and setting
$\text{class}=\arg\max\Bigl{(}\braket{v_{1}}{V_{1}},\ldots,\braket{v_{k}}{V_{k}}\Bigr{)}$.

##  4 Experimental Results

Dataset and pre-processing We evaluate the proposed algorithm using a small
subset of the “20BN-jester Dataset V1” that contains labeled video clips
showing humans performing predefined hand gestures 20. We crop and downscale
each video so that all frames are 64$\times$64 pixels and all videos 32 frames
long $N=64,T=32$. We perform the simulations using two (“Swiping Left”,
“Pulling Hand In”), three (“Swiping Left”, “Pulling Hand In”, “Pushing Hand
Away”) and four (“Swiping Left”, “Pulling Hand In”, “Pushing Hand Away”,
“Swiping Right”) of the available classes ($k=2,3,4$). We used training sets
of sizes $M\times k$ for values $M=20,40,60,80,120$. We encode the reduced
data using $q$ qubits for values $q=[4..17]$. We stress that the actual data
reduction is logarithmic with the number of qubits as shown in Figure 2.
Encoding a video using $q=4$ qubits corresponds to a $(64*64*32)/2^{4}=8192$
reduction.

Figure 2: Percentage of the initial video that a quantum register of $q$
qubits encodes.

Simulation For each combination of the parameters (training set size, number
of classes, number of qubits that encode the reduced data) we simulated the
quantum procedures. We run the simulations for at least 100 iterations and
averaged out the results.

Results In Figures 3,4 and 5 we report the accuracy achieved for each
simulation case. We observe that in all cases there is enough information
extracted for meaningful video classification even for as few as 10 qubits,
for appropriate training sizes. A quantum register of $q=10$ qubits encodes
$2^{10}=1024$ pixels, corresponding to just $2^{10}/(64\times 64\times
32)\approx 0.8\%$ of the initial video. This is a significant reduction. It
appears that after approximately 13 qubits no much more information is
extracted. We have to assume that this is at least partly due to our use of a
naive classification method as one would expect that for a non-reduced dataset
(corresponding to $q=17$) higher values of accuracy should be achievable.

The training size also plays a crucial role. For training sizes of less than
40 videos per class the classification accuracy was very low. As the training
size increased so did, in general, the classification accuracy. Exceptions to
this were observed as $k$ became larger and we hypothesize this is due to the
large similarities between the classes. The classes (“Swiping Left”, “Swiping
Right”) and (“Pulling Hand In”, “Pushing Hand Away”) are—in practice—time-
reversed versions of each other. On the other hand, it is a strong point of
the proposed method that it can accurately discriminate between the time-
reversed versions.

Figure 3: Accuracy vs number of qubits for 2 classes Figure 4: Accuracy vs
number of qubits for 3 classes Figure 5: Accuracy vs number of qubits for 4
classes

##  5 Discussion

The results obtained suggest that a very small number of qubits are able to
contain enough information so that videos can be successfully classified. Even
with the naive classification method that we employed at the classification
step, the procedure achieved high accuracy values, with a large data reduction
and high performance. Even at $q=5$ qubits there is a statistically
significant deviation from random-choice classification showing that
meaningful information was extracted using just 0.02% of the initial video,
corresponding to a 4000 reduction of the video size. The quantum method used
is therefore capable of accurately identifying the most significant pixels
using a very efficient procedure.

There are many ways that the above results could be improved. A better
classification method, either classical or quantum, is an obvious first step
as the method we employed as proof of concept is very simplistic. A more
accurate extraction of the significant bits could also be achieved by more
detailed methods; for example, by using cross-validation in place of the
simple video averaging.

Finally, as this method uses very few quantum operations and a few quantum
bits, it would be straightforward to implement it on a real quantum device
such as on an IBM qpu.

Acknowledgment This work is partially supported by the Greek Secretariat for
Research and Innovation and the EU, Project SignGuide: Automated Museum
Guidance using Sign Language T2EDK-00982 within the framework of
“Competitiveness, Entrepreneurship and Innovation” (EPAnEK) Operational
Programme 2014–2020.

## References

  * 1 Yijie Dang, Nan Jiang, Hao Hu, Zhuoxiao Ji, and Wenyin Zhang.  “Image classification based on quantum K-Nearest-Neighbor algorithm”.  Quantum Information Processing 17, 239 (2018).
  * 2 Nan-Run Zhou, Xiu-Xun Liu, Yu-Ling Chen, and Ni-Suo Du.  “Quantum K-Nearest-Neighbor Image Classification Algorithm Based on K-L Transform”.  International Journal of Theoretical Physics 60, 1209–1224 (2021).
  * 3 Maria Schuld, Alex Bocharov, Krysta Svore, and Nathan Wiebe.  “Circuit-centric quantum classifiers”.  Physical Review A 101, 032308 (2020).  arXiv:1804.00633.
  * 4 Mateusz Ostaszewski, Przemysław Sadowski, and Piotr Gawron.  “Quantum image classification using principal component analysis”.  Theoretical and Applied Informatics 27, 1–12 (2015).  arXiv:1504.00580.
  * 5 Edward Farhi and Hartmut Neven.  “Classification with Quantum Neural Networks on Near Term Processors”.  Quantum Review Letters Vol. 1 No. 2 (2020), 10.37686/qrl.v1i2.80 (2020).
  * 6 Tak Hur, Leeseok Kim, and Daniel K. Park.  “Quantum convolutional neural network for classical data classification” (2021).  arXiv:2108.00661.
  * 7 Seth Lloyd, Maria Schuld, Aroosa Ijaz, Josh Izaac, and Nathan Killoran.  “Quantum embeddings for machine learning” (2020).  arXiv:2001.03622.
  * 8 Kostas Blekos and Dimitrios Kosmopoulos.  “A Quantum 3D Convolutional Neural Network with Application in Video Classification”.  In Advances in Visual Computing.  Pages 601–612.  Lecture Notes in Computer ScienceCham (2021). Springer International Publishing.
  * 9 Aram W. Harrow.  “Small quantum computers and large classical data sets” (2020).  arXiv:2004.00026.
  * 10 Tobias Haug, Chris N. Self, and M. S. Kim.  “Large-scale quantum machine learning” (2021).  arXiv:2108.01039.
  * 11 Jin-Min Liang, Shu-Qian Shen, Ming Li, and Lei Li.  “Variational quantum algorithms for dimensionality reduction and classification”.  Physical Review A 101, 032323 (2020).  arXiv:1910.12164.
  * 12 John A. Cortese and Timothy M. Braje.  “Loading Classical Data into a Quantum Computer” (2018).  arXiv:1803.01958.
  * 13 Jacob Biamonte, Peter Wittek, Nicola Pancotti, Patrick Rebentrost, Nathan Wiebe, and Seth Lloyd.  “Quantum machine learning”.  Nature 549, 195–202 (2017).  arXiv:1611.09347.
  * 14 Scott Aaronson.  “Read the fine print”.  Nature Physics 11, 291–293 (2015).
  * 15 Iordanis Kerenidis and Anupam Prakash.  “Quantum Recommendation Systems” (2016).  arXiv:1603.08675.
  * 16 Vittorio Giovannetti, Seth Lloyd, and Lorenzo Maccone.  “Quantum Random Access Memory”.  Physical Review Letters 100, 160501 (2008).
  * 17 Iordanis Kerenidis, Jonas Landman, Alessandro Luongo, and Anupam Prakash.  “Q-means: A quantum algorithm for unsupervised machine learning” (2018).  arXiv:1812.03584.
  * 18 Anurag Anshu, Zeph Landau, and Yunchao Liu.  “Distributed quantum inner product estimation” (2021).  arXiv:2111.03273.
  * 19 Ryan O’Donnell and John Wright.  “Efficient quantum tomography”.  In Proceedings of the Forty-Eighth Annual ACM Symposium on Theory of Computing.  Pages 899–912.  STOC ’16New York, NY, USA (2016). Association for Computing Machinery.
  * 20 Joanna Materzynska, Guillaume Berger, Ingo Bax, and Roland Memisevic.  “The jester dataset: A large-scale video dataset of human gestures”.  In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).  Pages 2874–2882.  IEEE Computer Society (2019).


# Heterogenous Ensemble of Models for Molecular Property Prediction

Sajad Darabi
NVIDIA
USA
<EMAIL_ADDRESS>
& Shayan Fazeli
UCLA
USA
<EMAIL_ADDRESS>
& Jiwei Liu
NVIDIA
USA
<EMAIL_ADDRESS>
& Alexandre Milesi
NVIDIA
France
<EMAIL_ADDRESS>
& Pawel Morkisz
NVIDIA
USA
<EMAIL_ADDRESS>
& Jean-François Puget
NVIDIA
France
<EMAIL_ADDRESS>
& Gilberto Titericz
NVIDIA
Brazil
<EMAIL_ADDRESS>

###### Abstract

Previous works have demonstrated the importance of considering different
modalities on molecules, each of which provide a varied granularity of
information for downstream property prediction tasks. Our method combines
variants of the recent TransformerM architecture with Transformer, GNN, and
ResNet backbone architectures. Models are trained on the 2D data, 3D data, and
image modalities of molecular graphs. We ensemble these models with a
HuberRegressor. The models are trained on 4 different train/validation splits
of the original train + valid datasets. This yields a winning solution to the
2nd edition of the OGB Large-Scale Challenge (2022) on the PCQM4Mv2 molecular
property prediction dataset. Our proposed method achieves a test-challenge MAE
of $0.0723$ and a validation MAE of $0.07145$. Total inference time for our
solution is less than 2 hours. We open-source our code at
github.com/jfpuget/NVIDIA-PCQM4Mv2.

_Keywords_ GNN $\cdot$ Transformer $\cdot$ OGB-LSC $\cdot$ PCQM4Mv2 $\cdot$
DFT $\cdot$ Drug Discovery

##  1 Introduction

The OGB Large-Scale Challenge (LSC) (Hu et al., 2021) is a Machine Learning
(ML) challenge to predict a quantum chemical property, the HUMO-LUMO gap of
small molecules. This ground truth is obtained via a density-functional theory
(DFT) computation which is known to be time-consuming and could take several
hours, even for small molecules. With the rapid advancement of machine
learning technology, it is promising to use fast, GPU-accelerated and accurate
ML models to replace this expensive DFT optimization process.

The PCQM4Mv2 dataset, based on the PubChemQC project Nakata and Shimazaki
(2017), provides us with a well-defined ML task of predicting the HOMO-LUMO
gap of molecules given their 2D molecular graphs. Each molecule has two
natural views. The 2D graph incorporates topological structures defined by
bonds, and the 3D view provides spatial information that better reflects the
geometry and spatial relation of the different bonds in the molecule. In the
PCQM4Mv2 dataset, additional 3D structures are also provided for training
molecules. Further, in multiple domains, there has been a tendency to convert
data modalities to images such that high-capacity and expressive vision models
could be used to learn the underlying patterns. Although images do not
correspond to the data’s natural view, they have shown promising results kag
(2021). We also consider images of the molecule as an alternative view for
training vision-based models used in our ensemble.

Our solution is an ensemble of 39 checkpoints from 10 different models. Before
training these models, we performed exploratory data analysis and found a
significant distribution shift between the train dataset and the other
datasets (valid, test-dev, and test-challenge). The train dataset contains
molecules containing at most 20 heavy atoms, while the other 3 datasets
contain molecules up to 51 heavy atoms. We made the hypothesis that training
on the proposed train split would lead to mediocre performance on large
molecules (more than 20 atoms).

We decided to use different train/validation splits by creating $24$ folds
from the combined train+valid datasets. We then trained a model on $23$ folds
and used the last fold for validation. Ideally, one would train 24 models that
way, using each fold as a validation fold in turn. This is the usual $k$-folds
cross-validation, with $k=24$. However, for practical running time
considerations, we decided to only use 4 folds as validation folds, i.e.
training 4 fold models. Each model is trained by selecting one of the first 4
folds as validation fold, and using the remaining 23 folds as training data.
Whenever we discuss folds in the remainder of this report we refer to the
aforementioned 4 validation folds.

In the following sections of this report, we describe the model variants
trained for each data modality, followed by a a section describing how these
models were ensembled for the final test-challenge prediction.

##  2 Transformer-M variants

Following prior work incorporating 2D and 3D information via bias terms in the
Transformer self-attention (Ying et al., 2021), we use the recently proposed
random structural channels between 2D/3D in Transformer-M (Luo et al., 2022)
as our backbone architecture. In this method, the same Graphormer (Ying et
al., 2021) backbone architecture is used for both channels.

Given the input molecule $\mathcal{G}_{\text{2D/3D}}$, our encoder outputs a
latent representation $z=f(\mathcal{G_{\text{2D/3D}}})$. We leverage the
provided 3D SDF file to extract the corresponding 2D & 3D spatial atom
positions for this model. In subsequent subsections, we describe the variants
used.

All hyperparameters not mentioned in the following sections (dropout, gradient
clipping, weight decay, learning rate schedule, number of kernels) were kept
the same as described in the Transformer-M work.

###  2.1
$\textrm{Transformer-M}^{\textrm{base}}_{\textrm{without\\_denoising}}$

The first variant is a 12-layers, 768-wide model that includes randomly-
activated structural channels with probabilities $\frac{1}{4}$ for 2D,
$\frac{1}{2}$ for 3D, and $\frac{1}{4}$ for 2D+3D. In this variant, the
denoising task was not included, but the input atom positions provided by the
organizers were perturbed with gaussian noise of magnitude 0.2 in the 3D
channel. This variant was trained once for each of the 4 folds, for 400 epochs
with batch size 1024, on an NVIDIA DGX-1 (8 V100-SXM2-16GB 160W).

###  2.2 $\textrm{Transformer-M}^{\textrm{large}}_{\textrm{with\\_denoising}}$

The second variant is similar to the first one, except for its 18 layers, and
the inclusion of the atom position denoising task, using a cosine similarity
loss weighted equally to the graph prediction loss. Stochastic depth was
enabled with a probability of 10%. This variant was trained once for each of
the 4 folds, for 400 epochs with batch size 1024, on an NVIDIA DGX-2H (16
V100-SXM3-32GB 300W).

###  2.3 $\textrm{Transformer-M}^{\textrm{large}}_{\textrm{baseline}}$

In this variant, we train the Transformer-M model without any modifications.
The random channels probabilities are set to the original weights
$\frac{1}{5}$, $\frac{1}{5}$, $\frac{3}{5}$ for 2D, 3D, and 2D+3D
respectively. The training setup is the same as previous variants, except the
training lasts 454 epochs. This variant was trained twice with different seeds
on the 4 folds.

###  2.4 $\textrm{Transformer-M}^{\textrm{large}}_{\textrm{Dirichlet}}$

In this variant, we set the random channels probabilities to follow a
Dirichlet distribution with weights $\frac{1}{5}$, $\frac{1}{5}$,
$\frac{3}{5}$ for 2D, 3D, and 2D+3D respectively. The training setup is the
same as previous variants, except the training lasts 454 epochs. This variant
was trained once on the 4 folds.

###  2.5 Transformer-M with Knowledge Guided Regularization

Research in the domain of molecular property prediction has shown that
incorporating additional knowledge-based information could lead to
considerable performance improvements.

Specifically, the observations in Li et al. (2022) show that pre-training a
model on molecule fingerprints and descriptors configured as a mask-and-
predict pretext task can result in significant improvements, and with merely
relying on 2D information even surpassing most of the works leveraging 3D
positions.

Given that PCQM4Mv2 is a large-scale dataset and considerably larger than
those considered in Li et al. (2022), we formulated a similar task as a
regularizing objective, built on top of the Transformer-M configuration for
predicting the HOMO-LUMO gaps. For each molecule in the dataset, we followed
the configuration in Li et al. (2022) and computed a descriptor
$\mathbf{y}_{\text{d}}\in\mathbb{R}^{200}$ and a fingerprint
$\mathbf{y}_{\text{fp}}\in\\{0,1\\}^{512}$. We added two additional projection
heads on top of the latent graph representations, and added the following
regularization loss:

$\mathcal{L}_{\text{reg}}=\lambda_{\text{fp}}\mathcal{L}_{\text{bce}}(g_{\text{fp}}(\mathbf{z}),\mathbf{y}_{\text{fp}})+\lambda_{\text{d}}\mathcal{L}_{\text{mse}}(g_{\text{d}}(\mathbf{z}),\mathbf{y}_{\text{fp}})$
(1)

We run initial series of experiments using the base architecture with the
number of layers set to 12, batch size 1024, learning rate
$2\text{\times}{10}^{-4}$ and end learning rate $1\text{\times}{10}^{-9}$,
with exponential decay over the learning rate schedule for 300 epochs. We fix
the random seed for both experiments to the same value, and as shown from
Figure 1 this addition shows an improvement of $0.007$ MAE over the baseline.

Figure 1: Comparison of Transformer-M (baseline) and Transformer-M with
knowledge-guided regularization (baseline+reg)

We train two models with the added regularization on the 4 folds with varying
weights, one set to $\lambda=0.1$ and another set to $\lambda=0.2$. The
hyperparameters are set to the same values as the large model with 18 layers.
We call these models
$\textrm{Transformer-M}_{\textrm{kpgt}_{\lambda=0.1}}^{\textrm{large}}$ and
$\textrm{Transformer-M}_{\textrm{kpgt}_{\lambda=0.2}}^{\textrm{large}}$,
respectively. The models are trained on 8 NVIDIA V100-SXM2-16GB 160W GPUs.

##  3 Molecular Transformer

For each input molecule, we build a graph using the available 2D graph
information from SMILES. This graph is built using data from an instance of
the ogb PCQM4Mv2Dataset class. We extend the smiles2graph function by adding
rings using the rdkit package:

            ssr = Chem.GetSymmSSSR(mol)
            ssr = [list(sr) for sr in ssr]

We use this extended smiles2graph function to preprocess the dataset.

In the graph for a given molecule, every node is initialized with a token
embedding of size 768. Embeddings are initialized from a Glorot uniform
distribution. The nodes are:

  * •

A node for the whole molecule. Its role is similar to the [VNode] in
Graphormer (Ying et al., 2021). This token is initialized with an embedding of
the total number of heavy atoms in the molecule, capped at 20. In addition to
that, the inverse of the number of atoms is added to the embedding.

  * •

A node for each heavy atom in the molecule. The atom token is initialized with
the ogb predefined embedding plus an embedding of the degree of the atom in
heavy atom graph of the molecule.

  * •

A node for each bond in the molecule. Its token is initialized with the ogb
predefined embedding.

  * •

A node for each ring returned by RDKit GetSymmSSSR(). Each ring token is
initialized with an embedding of the number of atoms in the ring, capped at
15.

The edges of the graph are constructed as follows:

  * •

For each bond between atom $A$ and atom $B$, create an edge between the node
for $A$, and the node for $B$.

  * •

For each bond between atom $A$ and atom $B$ create an edge between $A$ and
$C$, and create an edge between $B$ and $C$, where $C$ is the node for the
bond.

  * •

For each ring $R$, and for each atom $A$ in the ring, create an edge between
the node for $R$ and the node for $A$.

  * •

An edge between the node for the whole molecule and each of the other nodes.

Tokens are fed into a 12-layers Transformer (Vaswani et al., 2017). Changes
compared to vanilla Transformers are:

  * •

Attention is masked by the edges defined above: a node only attends its
neighbors in the graph. We found that it was more efficient to compute dense
attention then mask rather than use sparse attention, probably because the
graph is quite small and dense.

  * •

The vanilla transformer feed-forward layer after attention layer is replaced
by two vanilla transformer feedforward layers.

  * •

Following the recent trends, we use GELU activation (Hendrycks and Gimpel,
2016).

Attention is masked by subtracting 10 from vanilla attention when there is no
edge, before softmax. There is no other attention mask. We tried to use the
Graphormer attention bias (spatial embedding, etc.), but it did not improve
our model. We think that the explicit representation of bonds and rings
captures the topology of the graph as effectively. Our model performance on
the original validation dataset is indeed slightly better than that of
Graphormer.

The final layer is a dense layer taking as input the last embedding for the
molecule node.

Our model is trained with a sine annealing learning rate scheduler, i.e.,
starting with a very low learning rate, up to a maximum and down again
following a sine curve. The maximum learning rate was set at
$2\text{\times}{10}^{-4}$, and the minimum at $1\text{\times}{10}^{-8}$. We
trained for 30 epochs and a batch size of 256 on 4 NVIDIA V100 16GB 160W GPUs
using PyTorch DataParallel. We experimented with DistributedDataParallel, but
we could not achieve the same model accuracy for some reason.

##  4 PD-DGN: Pointwise Dense Pooling for Deep Graph Neural Networks

Equivariant GNN models are widely used for the task of graph prediction tasks,
we design a novel pointwise dense pooling instead of the conventional global
mean pooling for deep graph neural networks (PD-DGN) model. The idea is
similar to self-attention graph pooling (SAG) Lee et al. (2019) but simpler.
SAG considers both node and topology features while our design only utilizes
node features. We train the model with 4 different folds and average the
predictions. The approach achieved a $0.089$ mean absolute error on the
validation split.

###  4.1 Model Architecture

Similar to the Molecular Transformer variant in section 3, this variant uses
the given PygPCQM4Mv2Dataset class of the ogb package, which converts SMILES
strings to graphs. Each node and each edge is represented by a vector of
dimensions of 9 and 3, respectively. We use embeddings to encode the node
features. The overall architecture of our model is shown in Figure 2. In this
model the core building block is DeepGCNLayer Li et al. (2019) which is a
wrapper of TransformerConv Shi et al. (2020) with additional skip connections.
This building block DeepGCNLayer enables the net to grow deeper, mitigating
the over-smoothing problem commonly seen as GNNs become deeper. We use a total
of 15 DeepGCNLayer blocks in our network.

Figure 2: Architecture of our deep graph Transformer. The boxes with red
dashed lines are DeepGCNLayer with skip connections.

The TransformerConv implements the following operations,

$\mathbf{x}^{\prime}_{i}=\mathbf{W}_{1}\mathbf{x}_{i}+\sum_{j\in\mathcal{N}(i)}\alpha_{i,j}\mathbf{W}_{2}\mathbf{x}_{j}$
(2)

where $\mathbf{x}_{i}$ and $\mathbf{x}^{\prime}_{i}$ are the input and output
embeddings of node $i$ of a TransformerConv layer, respectively.
$\mathcal{N}(i)$ represents the neighbors of node $i$. The attention
coefficients $\alpha_{i,j}$ are computed via multi-head dot product attention:

$\alpha_{i,j}=\textrm{softmax}\left(\frac{(\mathbf{W}_{3}\mathbf{x}_{i})^{\top}(\mathbf{W}_{4}\mathbf{x}_{j})}{\sqrt{d}}\right)$
(3)

where $d$ stands for the dimension of the embedding vectors $x_{i}$ and
$x_{j}$. At the end of the last TransformerConv, we have transformed
embeddings for each node. To convert node embeddings to graph embeddings
$\mathbf{G}$, global mean pooling is often used, as shown in equation 4.

$\mathbf{G}=\frac{1}{N}\sum_{i=1}^{N}\mathbf{x}_{i}.$ (4)

With global mean pooling, each node contributes equally to the graph
embedding, which could be suboptimal. An intuitive idea to improve global mean
pooling is to use a weighted sum of the node embeddings to compute graph
embedding, where weights are learned from node embeddings. We propose a novel
pointwise dense pooling layer:

$\mathbf{G}=\frac{1}{N}\sum_{i=1}^{N}\alpha_{i}\mathbf{x}_{i}.$ (5)

where $\alpha_{i}$ is computed via $\mathbf{x}_{i}$ using a learnable
parameter $\mathbf{W}$:

$\alpha_{i}=\textrm{softmax}\left(\mathbf{W}\mathbf{x}_{i}\right)$ (6)

###  4.2 Experiment Results

Our model is implemented using PyTorch Geometric framework Fey and Lenssen
(2019) and we train on an NVIDIA DGX-1 (8 V100-SXM2-16GB 160W). The model was
trained with 4 different folds of the concatenated data of the given training
and validation dataset. Training one model with one seed on a single GPU takes
10 hours. Inference with one model and one seed on the test-dev split takes 50
seconds. Experimental results showed that the proposed pooling has at least
0.001 lower MAE for both validation and test-dev than the baseline global
pooling.

##  5 Convolutional Neural Network Model

CNNs are well known to be very efficient in solving a large range of image
challenges. In this chapter, we describe a CNN regression approach to predict
the HOMO-LUMO energy gap of molecules using image-based representation.

###  5.1 ResNet Backbone

The model uses a regular ResNet34 (He et al., 2016) image backbone with the
top classification head replaced by a linear layer with one output. The images
are created using RDKit MolToFile function that takes as input the Mol
instance created from the smiles representation. The image representation of
the smiles Cn1c(=O)c2c(ncn2C)n(C)c1=O (caffeine) can be seen in Figure 3. To
speed up the training, the model training has three phases that use different
images sizes and learning rates combinations. During the phases, the images
are resized starting from 224x224, then 288x288 and finally to 352x352 pixels.
The learning rate scheduler used was cosine annealing with restarts. Also,
image augmentations that helped to regularize better include horizontal and
vertical flips, shift, scale, and rotation. Augmentations start hard in the
initial phase of training, and for the final phase, only horizontal flips are
applied to fine-tune the model. This model achieves an MAE around $0.098$ on
the validation set. The loss function used was plain L1.

Figure 3: 2D image molecule representation of caffeine

###  5.2 EfficientNet Backbone

Using an EfficientNet-B3 (Tan and Le, 2019) backbone achieves an MAE of
$0.092$ on the validation set, which outperforms ResNet34 backbone, but this
model didn’t improve the final ensemble metric. So we didn’t use it for the
final solution.

##  6 Stacking & Results

We have described how we trained a set of heterogeneous models. We are now
going to describe how we ensembled this set of models. For the sake of
clarity, we will refer to the models described so far as base models.

###  6.1 Base Model Results

In table 1 we summarize the fold MAE results for each base model. Several runs
are missing (denoted with $-$) due to training failures and lack of time to
restart training for final submission, hence we exclude their MAE values in
this table.

Table 1:  Model mean absolute error (MAE) results on 4 folds.

| MAE
---|---
Model | Fold 0 | Fold 1 | Fold 2 | Fold 3
$\textrm{Transformer-M}^{\textrm{large}}_{\textrm{baseline}_{seed=3407}}$ | - | 0.07424 | 0.07416 | 0.07414
$\textrm{Transformer-M}_{\textrm{kpgt}_{\lambda=0.2}}^{\textrm{large}}$ | 0.07477 | - | 0.07417 | 0.07410
$\textrm{Transformer-M}_{\textrm{kpgt}_{\lambda=0.1}}^{\textrm{large}}$ | 0.07478 | 0.07448 | - | 0.07421
$\textrm{Transformer-M}^{\textrm{large}}_{\textrm{baseline}_{seed=42}}$ | 0.07469 | 0.07434 | 0.07416 | -
$\textrm{Transformer-M}^{\textrm{large}}_{\textrm{Dirichlet}_{seed=3407}}$ | 0.07434 | 0.07409 | 0.07386 | 0.07396
$\textrm{Transformer-M}^{\textrm{base}}_{\textrm{without\\_denoising}}$ | 0.07645 | 0.07583 | 0.07591 | 0.07596
$\textrm{Transformer-M}^{\textrm{large}}_{\textrm{with\\_denoising}}$ | 0.07865 | 0.07744 | 0.07892 | 0.07926
Molecular Transformer | 0.079534 | 0.079366 | 0.079198 | 0.079090
PD-DGN | 0.084606 | 0.084109 | 0.084103 | 0.084054
CNN EfficientNet-B3 | 0.092380 | 0.092456 | 0.092663 | 0.092523
CNN ResNet34 | 0.098702 | 0.097201 | 0.099301 | 0.098005

###  6.2 Stacking

The simplest way to ensemble our base models is to average their output. This
works fine when the models have similar accuracy. However, when we ensemble
heterogeneous base models with a wide range of accuracy, we want to favor the
best base models over weaker models. One way to achieve this is to use a
weighted average of the base models’ outputs. Strong base models should get a
high weight and weaker base models a small weight. This requires computing
effective weights.

A general purpose for computing these weights is to train a linear model with
a new training dataset. The input in this training dataset is made of the
predictions of the base models, and the target output is the original target.
This is often referred to as stacking, because the linear model sits on top of
the base models’ predictions.

In order to train the linear model we need to have predictions of the base
models on data for which we also have ground truth. This is why we created 4
train/validation folds as described in the introduction. We follow the
following workflow, for each base model and each fold:

  * •

We train a fold base model.

  * •

We store the predictions of the fold base model on its validation fold. These
predictions are called out-of-fold (OOF) predictions.

  * •

We store the predictions of the fold base model on the test-challenge data

In some of our training runs, one of the folds failed (a diverging loss during
training), hence we only had 3 valid checkpoints for the base model. This
happened for 4 of the runs. We decided to replace the out of fold predictions
for these 4 base models with their average where they ran correctly. We
applied the same averaging for the test predictions.

Then for each base model, we concatenate its 4 out-of-fold predictions. We
then concatenate the result for all base models in the column dimension. This
yields a new dataframe with as many columns as base models and as many rows as
the sum of rows of the 4 validation folds we used. This is the training data
for the linear model.

The ground truth for training the ensemble model is the target of the 4
validation folds we use.

We then tried several linear models for ensembling. The best result was
obtained using scikit-learn HuberRegressor. HuberRegressor was chosen because
it performed best to optimize the L1 loss metric when compared to other
training algorithms, e.g., Linear Regression, Lasso, Random Forest, and
Gradient Boosting Decision Trees.

We used 4 folds cross validation to train the linear model, using the same 4
validation folds used for the base models. A linear model is trained using 3
of the folds, and evaluated on the last fold. We then compute the average
score across 4 validation folds.

Running 4-fold cross-validation metric scored 0.07145 overall for the linear
models.

To generate test predictions, a similar process is applied:

  * •

The base fold models are grouped by the validation fold used to train them.

  * •

For each validation fold, compute test predictions of all fold base models for
that fold

  * •

For each validation fold, apply the linear model trained with that validation
fold to the test predictions of the fold models. This yields a test
predictions

  * •

Average the 4 linear models test predictions.

This yields a first ensemble of models. The weights of fold models computed by
the Huber regressor are given in Table 2. The first row is the average of
predictions for these base models:
$\textrm{Transformer-M}^{\textrm{large}}_{\textrm{baseline}_{\textrm{seed}=3407}}$,
$\textrm{Transformer-M}_{\textrm{kpgt}_{\lambda=0.2}}^{\textrm{large}}$,
$\textrm{Transformer-M}_{\textrm{kpgt}_{\lambda=0.1}}^{\textrm{large}}$, and
$\textrm{Transformer-M}^{\textrm{large}}_{\textrm{baseline}_{\textrm{seed}=42}}$.
The last row is the Huber regressor intercept.

Table 2:  Model weights from Huber regressor.

| weights
---|---
Model | Fold 0 | Fold 1 | Fold 2 | Fold 3
Transformer-M average | 0.5382 | 0.5330 | 0.5186 | 0.5171
$\textrm{Transformer-M}^{\textrm{large}}_{\textrm{Dirichlet}_{\textrm{seed}=3407}}$ | 0.2356 | 0.2447 | 0.2564 | 0.2657
$\textrm{Transformer-M}^{\textrm{base}}_{\textrm{without\\_denoising}}$ | 0.1013 | 0.0986 | 0.0936 | 0.0945
$\textrm{Transformer-M}^{\textrm{large}}_{\textrm{with\\_denoising}}$ | -0.0353 | -0.0413 | -0.0269 | -0.0349
Molecular Transformer | 0.0929 | 0.0974 | 0.0878 | 0.0878
PD-DGN | 0.0395 | 0.0394 | 0.0418 | 0.0421
CNN ResNet34 | 0.0255 | 0.0261 | 0.0265 | 0.0254
intercept | 0.0101 | 0.0096 | 0.00991 | 0.0094

Near the end of the competition, we retrained some of the base models on the
full train+valid data, hoping that these full train base models would be
different and slightly better than the fold base models. These models were
added to the ensemble predictions via a simple weighted average according the
following equation.

$\displaystyle\textrm{submission}=(\textrm{ensemble}$
$\displaystyle+0.2552\cdot\textrm{Transformer-M}^{\textrm{large}^{\textrm{full\\_train}}}_{\textrm{dirichlet}}$
$\displaystyle+0.1747\cdot\textrm{Transformer-M}_{\textrm{kpgt}_{\lambda=0.1}}^{\textrm{large}^{\textrm{full\\_train}}}$
$\displaystyle+0.1747\cdot\textrm{Transformer-M}_{\textrm{kpgt}_{\lambda=0.2}}^{\textrm{large}^{\textrm{full\\_train}}})/(1+0.2552+0.1747+0.1747)$

The weight 0.2552 is the average weight for
$\textrm{Transformer-M}^{\textrm{large}}_{\textrm{Dirichlet}_{\textrm{seed}=3407}}$
in Table 2. The weight 0.1747 is $1/3$ of the average weight for Transformer-M
average in Table 2.

###  6.3 Inference Time

Table 3:  Inference and dataset timing for each model.

Model | Dataset Time (s) | Inference (s)
---|---|---
$\textrm{Transformer-M}^{\textrm{large}}_{\textrm{baseline}_{seed=3407}}$ | 150 | 45
$\textrm{Transformer-M}_{\textrm{kpgt}_{\lambda=0.2}}^{\textrm{large}}$ | 150 | 45
$\textrm{Transformer-M}_{\textrm{kpgt}_{\lambda=0.1}}^{\textrm{large}}$ | 150 | 45
$\textrm{Transformer-M}^{\textrm{large}}_{\textrm{baseline}_{seed=42}}$ | 150 | 45
$\textrm{Transformer-M}^{\textrm{large}}_{\textrm{Dirichlet}_{seed=3407}}$ | 150 | 45
$\textrm{Transformer-M}^{\textrm{base}}_{\textrm{without\\_denoising}}$ | 150 | 30
$\textrm{Transformer-M}^{\textrm{large}}_{\textrm{with\\_denoising}}$ | 150 | 30
Molecular Transformer | 300 | 70
PD-DGN | 60 | 60
CNN EfficientNet-B3 | 120 | 319
CNN ResNet34 | 120 | 312

Inferencing is pretty efficient. If we ignore dataset download from OGB site
then Transformers take less than a minute on average to make test predictions
using a single V100 GPU. Making all Transformers predictions takes about 30
minutes. The Resnet model takes a bit longer, about 5 minutes per fold model.
In total, inferencing takes about 1 hour for all base models. Running the
ensemble model on top of their predictions takes a minute or so.

Most of the time is spent on the dataset download and smiles2graph or smiles
to image conversions. We use 4 different variants of the dataset when
inferencing:

  * •

A vanilla PCQM4Mv2Dataset for all Transformer-M variants 2.

  * •

A PCQM4Mv2Dataset extended with ring information from rdkit for the Molecular
Transformer base model 3.

  * •

A PygPCQM4Mv2Dataset for the PD-GDGN base model 4.

  * •

A PCQM4Mv2Dataset extended with images generated by rdkit for the ResNet34
base model 5.

Each dataset variant is constructed only once and shared by all corresponding
base models.

We created these datasets on the same hardware used for training, i.e. with a
multicore CPU. We measured the time taken to create these datasets. We then
estimated the time taken to build the dataset for the test challenge
prediction to be 4 percent of the total time to build the dataset (refer to
Table 3 for dataset & inference timings). Estimated running times for the test
challenge part range from a minute for Transformer-M variants to 5 minutes for
ResNet34 and for the Molecular Transformer.

Taking all times into account we estimate our inference time to be around
1h30, and certainly less than 2 hours.

##  7 Conclusion

We present a novel winning solution based on an ensemble of heterogeneous
models. Our best models are variants of the recently published Transformer-M
architecture. We get a significant improvement by ensembling these variants
with weaker models, namely other Transformers, and an image classification
model leveraging additional 2D/image data modalities. Our solution
demonstrates the effectiveness of ensembling heterogeneous models on large-
scale datasets and we hope this will encourage others to pursue such avenues
more often in similar problems.

## References

  * Hu et al. [2021] Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec.  OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs.  2021\.
  * Nakata and Shimazaki [2017] Maho Nakata and Tomomi Shimazaki.  PubChemQC project: a large-scale first-principles electronic structure database for data-driven chemistry.  _Journal of chemical information and modeling_ , 57(6):1300–1308, 2017.
  * kag [2021] Bristol-myers squibb – molecular translation, May 2021.  URL https://www.kaggle.com/competitions/bms-molecular-translation/overview/timeline.
  * Ying et al. [2021] Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu.  Do Transformers Really Perform Badly for Graph Representation?  2021\.
  * Luo et al. [2022] Shengjie Luo, Tianlang Chen, Yixian Xu, Shuxin Zheng, Tie-Yan Liu, Di He, and Liwei Wang.  One Transformer Can Understand Both 2D & 3D Molecular Data.  2022\.
  * Li et al. [2022] Han Li, Dan Zhao, and Jianyang Zeng.  KPGT: Knowledge-Guided Pre-training of Graph Transformer for Molecular Property Prediction.  2022\.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is All you Need.  30, 2017.
  * Hendrycks and Gimpel [2016] Dan Hendrycks and Kevin Gimpel.  Gaussian Error Linear Units (GELUs).  2016\.
  * Lee et al. [2019] Junhyun Lee, Inyeop Lee, and Jaewoo Kang.  Self-attention graph pooling.  pages 3734–3743, 2019.
  * Li et al. [2019] Guohao Li, Matthias Muller, Ali Thabet, and Bernard Ghanem.  DeepGCNs: Can GCNs Go as Geep as CNNs?  pages 9267–9276, 2019.
  * Shi et al. [2020] Yunsheng Shi, Zhengjie Huang, Shikun Feng, Hui Zhong, Wenjin Wang, and Yu Sun.  Masked label prediction: Unified message passing model for semi-supervised classification.  2020\.
  * Fey and Lenssen [2019] Matthias Fey and Jan E. Lenssen.  Fast Graph Representation Learning with PyTorch Geometric.  2019\.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep Residual Learning for Image Recognition.  pages 770–778, 2016.
  * Tan and Le [2019] Mingxing Tan and Quoc V. Le.  EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks.  2019\.