††thanks: These authors contributed equally††thanks: These authors contributed
equally

# Thickness dependent transition from 1T′ to Weyl semimetal phase in ultrathin
MoTe2: Electrical transport, Noise and Raman studies

Manabendra Kuiri  Subhadip Das  D V S Muthu  Anindya Das<EMAIL_ADDRESS>A
K Sood<EMAIL_ADDRESS>Department of Physics, Indian Institute of Science,
Bangalore 560012, India

###### Abstract

Bulk 1T′-MoTe2 shows a structural phase transition from 1T′ to Weyl
semimetallic (WSM) $T_{d}$ phase at $\sim$ 240 K. This phase transition and
transport properties in the two phases have not been investigated on ultra-
thin crystals. Here we report electrical transport, $1/f$ noise and Raman
studies in ultra-thin 1T′-MoTe2 ($\sim$ 5 to 16 nm thick) field-effect
transistors (FETs) devices as a function of temperature. The electrical
resistivities for thickness 16 nm and 11 nm show maxima at temperatures 208 K
and 178 K, respectively, making a transition from semiconducting to semi-
metallic phase, hitherto not observed in bulk samples. Raman frequencies and
linewidths for 11nm thick crystal show change around 178 K, attributed to
additional contribution to the phonon self-energy due to enhanced electron-
phonon interaction in the WSM phase. Further, the resistivity at low-
temperature shows an upturn below 20 K along with the maximum in the power
spectral density of the low frequency $1/f$ noise. The latter rules out the
metal-insulator transition (MIT) being responsible for the upturn of
resistivity below 20 K. The low temperature resistivity follows $\rho\propto
1/T$, changing to $\rho\propto T$ with increasing temperature supports
electron-electron interaction physics at electron-hole symmetric Weyl nodes
below 20 K. These observations will pave the way to unravel the properties of
WSM state in layered ultra-thin van der Waals materials.

Weyl semimetal, phase transition

##  I Introduction

Semimetallic transition metal dichalcogenides (TMDs) have spurred tremendous
interest in recent years due to their novel electronic properties and a wide
range of potential applications [1]. These materials exhibit exciting
phenomena like gate tunable superconductivity [2, 3], charge density waves [4]
and anomalous quantum Hall effect [5]. It has been shown that the Td phase of
semimetallic TMDs belong to a class of type-II Weyl semimetals (WSM) [6, 7, 8,
9, 10, 11]. A WSM consists of topologically protected linear bands, touching
at points referred to as ’Weyl nodes’, where the band dispersion is linear in
all the three momentum directions [12, 13, 14, 15]. Not only different phases
but also tunability of electronic properties of semimetallic TMDs by varying
the number of layers have shown thickness-dependent transition temperature of
the charge density wave in 1T-TiSe2 and 1T-VSe2 [16, 17] as well as magnetic
and topological transitions [18]. However, the thickness-dependent electrical
transport properties of semimetallic TMDs have not been explored well.

In this context MoTe2 is a quite promising TMD for the following reasons.
MoTe2 crystallizes in normally three stable phases : hexagonal ($\alpha$ or 2H
phase), monoclinic ($\beta$ or 1T′) and orthorhombic ($\gamma$ or Td phase)
[19]. The 2H and 1T′ phases are semiconducting in nature with bandgap of 1.1
and 0.06eV respectively[20, 21], whereas the Td phase is a type-II WSM [22].
Bulk 1T′-MoTe2 undergoes a first order structural phase transition from
monoclinic 1T′ to orthorhombic Td phase below $\sim$ 240 K [23, 24, 22].
Recent Raman studies [25, 26, 27] as well as ARPES measurements [28, 29, 30]
confirm the bulk Td phase of MoTe2 to be a type-II WSM [6, 7, 31]. Recent
magneto-transport measurements [32] of a few layer 1T′-MoTe2 confirm its
gapless semimetallic behavior. However, the transition from 1T′ to Td phase
and their electrical transport properties, governed by the predicted electron-
electron interactions at the Weyl nodes [33] have not been investigated on
ultra-thin MoTe2 crystals.

In this work, we report temperature dependent resistivity measurements of few
layer 1T′-MoTe2 devices with thickness ranging from 5 to 16 nm. We observe a
clear signature of 1T′ to Td phase transition in 11 nm and 16 nm thick
nanocrystals with a transition temperature ($T_{c}$) of 178 K and 208 K, much
less than the bulk transition temperature (240 K). For $T>T_{c}$,
semiconducting behavior with $d\rho/dT<0$ is seen which changes to
$d\rho/dT>0$ below $T_{c}$, signifying semimetallic behavior. Such resistivity
change is not seen in 1T′-MoTe2 with thickness of 7 nm and 5 nm, establishing
the suppression of transition in crystals with thickness below a critical
value. In addition for the 11 nm thick crystal, the resistivity increases
below $T\sim 20K$. In the low temperature regime ($T<20K$), the resistivity
shows $1/T$ behavior, a clear signature of the WSM state as predicted
theoretically[33]. This feature has not been reported in bulk [22] but seen in
6 nm thick 1T′-MoTe2 [32] where the upturn at $\sim$20 K has been attributed
to MIT [32]. We have used $1/f$ noise spectroscopy as a function of
temperature to look at this suggestion of MIT, as noise spectroscopy has been
used recently in VO2 to examine the MIT [34]. We further note that $1/f$ noise
measurements have been utilized as a sensitive tool to probe structural phase
transitions[35, 36], vortex flux dynamics near superconductor-normal phase
transition[37] and electronic phase transition[38, 39]. We will show that our
$1/f$ noise spectra do not agree with this conjecture and propose that it is
related to electron-electron interaction physics in the WSM state arising from
the current carrying electron-hole states at the Weyl nodes [33]. Furthermore,
we have carried out Raman scattering experiment as a function of temperature
on 11 nm thick MoTe2 device and show that Raman mode frequencies and linewidth
exhibit slope change at $T_{c}$ which further supports the transition from 1T′
to WSM Td phase observed in our electrical transport measurements.

Figure 1: (Color Online) (a) Resistivity ($\rho$) versus temperature (T) for
$\sim$ 11 nm and $\sim$ 16 nm nanocrystals. Inset shows $d\rho/dT$ with
temperature. $T_{c}$ is indicated with vertical arrows. (b) Resistivity
($\rho$) versus temperature (T) for thin nanocrystals $\sim$ 5 nm and $\sim$ 7
nm respectively. Inset shows the resistivity data in the low temperature
regime (10 to 150K) of another 7 nm thick device. (c) Critical temperature
($T_{c}$) as a function of inverse of nanocrystal thickness ($d^{-1}$) (circle
points) from Fig. 1(a). The error bar size is $\sim$ 0.5K. The bulk
$T_{c}\sim$ 240 K (Square dot) has been taken from literature [23, 24, 22].
(d) Resistivity ($\rho$) versus temperature (T) in the low temperature regime
(4 K to 100 K) for $\sim$ 11 nm nanocrystal. The red dashed line shows the fit
with Eq. (1).

##  II Experimental details

Thin nanocrystals (between 5 and 16 nm) of 1T′-MoTe2 were obtained from
mechanical exfoliation of bulk single crystals of 1T′-MoTe2 onto a piranha
cleaned Si/SiO2 substrate with 285 nm SiO2 [40] with immediate spin-coating
with PMMA to prevent degradation [41]. Electrical contacts were patterned
using electron beam lithography followed by thermal deposition of Cr/Au (5
nm/70 nm). The nanocrystals were characterized by atomic force microscopy
(AFM) (see Fig. S1 of the supplemental material).

Resistivity measurements were carried out in a home built dip-stick cryostat
with a vacuum $\sim 10^{-6}$ mbar [41]. Two probe resistance measurements were
carried out in a standard lock-in technique, by applying a small ac signal and
measuring the current through the lock-in amplifier. The resistance
fluctuations were recorded using a NI USB 6210 DAQ card. The details of noise
measurement techniques have been discussed extensively in our earlier work
[42]. Raman spectra were recorded using confocal Raman spectrometer (LABRAM
HR-800) system using excitation wavelength of 532 nm with low incident power
($\sim$ 0.5 mW ). The sample temperature was controlled using a liquid
nitrogen cooled cryostat (M/s Linkham Scientific Thms350v).

##  III Results and discussion

###  III.1 Transport study

Fig. 1(a) shows the resistivity ($\rho$) as a function of temperature ($T$)
for two devices with thickness of 11 nm and 16 nm. It clearly shows that the
resistivity first increases with the decreasing temperature ($d\rho/dT<0$) as
expected for the 1T′ semiconducting phase, till a critical temperature $T_{c}$
of 178 K for 11 nm thick 1T′-MoTe2. Below $T_{c}$, the resistivity decreases
($d\rho/dT>0$) showing semimetallic behavior until $T\sim 20K$. A similar
trend was also observed for 16 nm thick crystal with $T_{c}\sim 208K$, showing
clearly that $T_{c}$ is suppressed as nanocrystal thickness is reduced. Inset
of Fig. 1(a) shows derivative of the resistivity with temperature to determine
$T_{c}$. Our observations are in variance with the reports by He et al., where
they observe semimetallic behavior (i.e $d\rho/dT>0$) from 280 K down to 150 K
for all sample of thickness 7, 50, 180 nm and 100 $\mu$m [27]. Secondly, we do
not observe hysteretic behavior in resistivity for 11 nm nanocrystal as
observed in bulk [22, 27] (see Fig. S2(a)). For $T>T_{c}$, the resistivity
shows Arrhenius behavior with activation gap of $\sim 4$ meV. Notably for thin
nanocrystals ($\sim$ 5 nm and 7 nm) shown in Fig. 1(b), the resistivity
increases monotonically as the temperature is reduced without a change in
$d\rho/dT$. These devices remains semiconducting in the measured temperature
range where the resistivity follows Arrhenius equation with activation energy
of $\sim$ 3.5 and 3.3 meV for 5 and 7 nm nanocrystals respectively (Figs.
S2(b) and (c)). Thus, only the nanocrystals with thickness $\geq$ 11 nm show a
transition from 1T′ to Td (WSM) phase. Although we have performed our
electrical measurement in two probe configuration, the linear current to
voltage relation (see Fig. S2(d)) of our 5nm thick sample at gate voltages
ranging from -20V to 20V, suggests negligible Schottky barrier in our device
fabrication process. In Fig. 1(c), we have plotted the transition temperature
with the inverse of nanocrystal thickness for our 11 and 16nm samples with the
reported $T_{c}$ of the bulk crystal[23, 24, 22].

A qualitative understanding of the downward shift of $T_{c}$ is as follows[27,
43]: In the Td phase, the Fermi energy lies above the out-of-plane hole bands
(marked as $\psi_{1}$ and $\psi_{2}$ in Ref. 43). With decreasing thickness,
the confinement energy shifts the hole bands to lower energy and hence favors
the formation of Td phase at a lower temperature as compared to bulk 1T′-MoTe2
(See the Figs. 4(a) and (b) of Ref. 27). The reason that Td phase is not seen
up to 10 K for 5nm and 7nm thick nanocrystals is because of the effect of
unintentional hole doping. Gate dependent conductivity measurement (Fig.
S2(e)) shows that the 5nm sample has conduction minima around +19 V of gate
voltage (indicated by vertical dotted line in Fig. S2(e)) and not 0 V.
Therefore, it is evident that at Vbg=0V, the device is unintentionally hole
doped. The explanation for this unintentional doping is following. It has been
reported that rapid thermal annealing of 2H-MoTe2 by oxygen at 250 0C shows
p-type doping due to formation of Mo-O bonds in tellurium vacancy sites [44].
Similar effect is also observed on this material for prolonged exposure in the
atmosphere [45], which can also be expected for 1T′-MoTe2. The effect of
unintentional hole doping will be more in the thin samples due to relative
increase in surface area and stabilize the 1T′ phase [43]. Paul et al. [46]
show that 2 to 5 layers of MoTe2 are in 1T′ phase at room temperature as
indicated by the 130cm-1 Raman mode [25, 26, 27]. By electron doping these
samples using NH3 exposure, the 130cm-1 mode show splitting, indicative of the
Td phase [25, 26, 27]. More theory work is needed to quantitatively understand
this effect.

Figure 2: (Color Online) (a) Schematic of $1/f$ noise measurement. (b) Power
spectral density for several values of temperatures ($T$). (c) Noise amplitude
(A) - blue solid symbols and resistance versus temperature - red line. (d)
Noise scaling: Noise amplitude with resistance.

We now discuss the low temperature resistivity (4-100 K) for the 11 nm
nanocrystal (zoomed part of Fig. 1(a)) to focus on the resistivity minimum at
$\sim$ 20 K (Fig. 1(d)). Similar upturn in resistivity at temperatures below
10 K was observed in 6 nm 1T′-MoTe2 crystal [32], which was attributed to
enhanced e-e interactions in two dimension where the resistivity $\rho\propto$
log10(T). For 3.6 nm nanocrystal, it was shown that the resistivity increases
as the temperature is lowered from ambient to 1.5 K, which was attributed to
strong localization effect [32]. It was argued that the lower value of the
electron dephasing length ($L_{\Phi}$) in ultrathin crystals leads to enhanced
localization, resulting in MIT at low temperatures [32]. However, our low-
temperature upturn resistivity data of 11 nm thick nanocrystal do not scale as
log10(T) or follow MIT scaling power law as observed in low frequency noise
measurements [34]. We will come back to it when we discuss the noise
measurements in the next section. Regarding the upturn of resistivity below 20
K, we note that the theoretically predicted resistivity in the WSM state in
the clean limit has electron-electron interactions with zero total momentum
transfer due to electron-hole symmetry near the Weyl nodes that leads to $1/T$
dependence [33]. Guided by this, we fit our resistivity data with the equation

$\rho=\rho_{0}+\frac{a}{T}+bT$ (1)

The third term on the right hand side, linear relation of $\rho$ with $T$, is
due to electron-phonon interactions, whose nature is determined by the Bloch-
Gruneisen temperature ($\Theta_{BG}$) rather than the Debye temperature
($\Theta_{D}$) for smaller Fermi surfaces around the Weyl nodes as observed in
graphene [47]. Fig. 1(d) shows the fit to Eq.(1), where we observe that for
$T<20K$, $\rho\propto 1/T$ dominates and for $T>20K$, $\rho\propto T$. We note
that addition of a term in resistivity proportional to T2 arising from
electron-electron interaction in Fermi liquid does not fit the data.

###  III.2 Noise measurements

Fig. 2(a) shows the schematic of the $1/f$ noise measurements. Two probe
voltage noise was measured using the technique described in our earlier work
[42]. The linear current variation at low bias voltage (Fig. S2(d)) of our 5nm
nanocrystal at 77K show negligible Schottky barrier at the interface of
channel and electrode and hence rules out the possibility of any spurious
contribution to noise measurements from the device contacts. Fig. 2(b) shows
the typical power spectral density (PSD) for several temperatures where the
dashed line shows the slope-1. The PSD in our case is $\sim 1/f^{\beta}$,
where $\beta$ lies between 0.8 to 1.2. Fig. 2(c) shows the comparison of the
noise amplitude (A) and resistance with temperature. The noise amplitude A,
changes by two orders of magnitude with temperature, with a maximum at $\sim
20K$. In Fig. 2(d), we re-plot the Noise amplitude versus resistance (R) to
examine the expected scaling seen earlier near MIT [34]. The noise scaling
close to MIT transition is ${S_{I}}/{I^{2}}\propto R^{x}$, where $x$ is the
universal exponent [34]. For MIT, $x\sim 2.6$ [34]. In comparison, in our case
$x\sim 100$ which is shown by the dotted line in Fig. 2(d). Thus the scaling
of noise with resistance near 20K transition is not compatible with MIT. We
hope that our experiments will motivate theoretical understanding of noise
fluctuations in Weyl semimetals.

###  III.3 Raman spectroscopy

Figure 3:  (Color Online) Raman spectra of 11 nm thin nanocrystal at (a) 300
and (b) 80 K. Black circles represents experimentally captured data points.
Solid lines (Blue and red) are their respective Lorentzian fit. The peaks are
shown from M1 to M11. Some regions at low temperature phase have been enhanced
(ranges shown by green lines) for low signal response. (c) Peak position and
(d) full width half maxima plot with temperature of M1, M6, M9 and M10 modes.
The cubic anharmonic fitting is represented by the red line. Green lines
represents the fitting from linear combination of cubic anharmonic and EPC
contribution. Blue dashed line indicates the transition temperature ($T_{c}$)
from 1T′ to WSM phase. Figure 4:  Schematic illustration of the electronic
bands in (a) 1T′ and (b) WSM phase of MoTe2. The arrows in the WSM phase
represents electronic transition. The blue gradient represents the color map
of the Fermi-Dirac distribution at 80K and 300K respectively.

At ambient condition, 1T′-MoTe2 belongs to P21/m space group which gives 18
Raman active modes (12 Ag+6 Bg) [48]. Fig. 3 shows Raman spectra of 11 nm
thick MoTe2 at 300 and 80 K, displaying 11 Raman modes, labeled as M1 to M11.
Spectra are fitted with a sum of Lorentzian functions to extract phonon
frequency ($\omega$) and linewidth ($\gamma$) as a function of temperature,
and plotted in Figs. 3(c), 3(d) and Fig. S3. Recent report shows appearance of
an additional weak Raman mode near $\sim$130 cm-1 in the bulk Td phase below
the transition temperature 240 K [25, 27, 26]. However our measurements on
11nm thick 1T′-MoTe2 do not show this additional new Raman modes down to 80 K.
Theoretical and experimental results by Ma et al. report that 1T′ and Td
phases of MoTe2 have similar Raman spectra with two Ag modes near $\sim$130
cm-1 with a difference of only $\sim$1 cm-1 [49]. It is very likely that this
separation is not resolved in our 11nm thin crystal. At a first glance, the
following observations are noteworthy in Figs. 3(c) and (d): (i) The frequency
and linewidth of the mode M1 show anomalous temperature dependence namely, the
frequency decreases and linewidth increases with decreasing temperature. (ii)
The linewidth of modes M6, M9 and M10 tend to show either saturation or
increase below $T_{c}$. (iii) The frequency of modes M9 and M10 show marked
deviation from linear dependence on temperature expected from cubic
anharmonicity. To understand these trends, we recall that in a solid, a phonon
can decay into two or more phonons (anharmonic contribution) or it can create
a pair of electron and hole (electron -phonon coupling (EPC)) in conduction
and valence bands respectively. These contribution to the phonon frequency and
linewidth can be written as,

$\omega(T)=\omega_{0}+\Delta\omega_{ph}+\Delta\omega_{e-ph}$ (2)
$\gamma(T)=\gamma_{0}+\Delta\gamma_{ph}+\Delta\gamma_{e-ph}$ (3)

In a simplified model, taking cubic anharmonicity into account for the decay
of a phonon into two phonons of equal energy, the temperature dependence is
given by: $\Delta\omega_{ph}(T)=-A.g(T,\omega_{0})$ and
$\Delta\gamma_{ph}(T)=B.g(T,\omega_{0})$, where
$g(T,\omega_{0})=1+2/(e^{\hbar\omega_{0}/{2k_{B}T}}-1)$ with $\omega_{0}$
being the phonon frequency at zero temperature, and $k_{B}$ is the Boltzmann
constant [50], A and B are positive constants proportional to the phonon-
phonon interaction strength. Neglecting the EPC contribution, the anharmonic
decay model alone fails to capture the temperature dependence of $\omega$ for
M9 and M10 and $\gamma$ for M6, M9 and M10 modes below $T_{c}$ (Figs. 3(c) and
(d)). In addition, M1 mode shows opposite temperature dependence of $\omega$
and $\gamma$ from cubic anharmonicity. We now argue that these anomalous
changes below $T_{c}$ are due to strong EPC between Weyl Fermions and phonons
in the WSM state.

Fig. 4 shows a schematic of the electronic states near Fermi energy, above and
below $T_{c}$. The self-energy of the phonons due to EPC arises from the
interband transition at the energy of the phonon mode ($\hbar\omega_{0}$) as
indicated by the arrows in Fig. 4. At $T>T_{c}$, the semiconducting 1T′ phase
has a bandgap of $\sim$ 60 meV [21]. Since the bandgap is larger than the
energies of the phonons, the electronic transitions at $\hbar\omega_{0}$ do
not occur and hence the contribution from the EPC to the phonon self-energy
can be neglected. At $T<T_{c}$ in the gapless WSM state (shown in Fig. 4), the
electron-hole excitations at the phonon energy $\hbar\omega_{0}$ become
feasible, resulting in considerable electron- phonon interaction. In the WSM
state, real part of the self-energy due to EPC contribute to the additional
softening of the phonon frequency and the imaginary part to the additional
linewidth of the Raman mode [51]. Following Ref. 52, the EPC contribution to
phonon linewidth is, $\Delta\gamma_{e-ph}(T)=D.h(T,\omega_{0})$ where D is a
positive constant and
$h(T,\omega_{0})=[f(-\frac{\hbar\omega_{0}}{2})-f(\frac{\hbar\omega_{0}}{2})]$,
with $f(x)$ representing the Fermi function. Assuming similar expression for
the EPC contribution to the mode frequency, $\Delta\omega_{e-ph}(T)$ can be
expressed as $-C.h(T,\omega_{0})$ with positive constant C. Eqs. (2) and (3)
are fitted to the data (green line) below $T_{c}$ (Figs. 3(c) and(d)). Table.
S1 shows the extracted parameters with error bars. As seen in Fig. 3, the
inclusion of electron-phonon contribution to the modes along with cubic
anharmonic contribution, captures the anomalous temperature dependence of
$\omega$ and $\gamma$ below $T_{c}$, supporting the transition to WSM phase.
However for the M1 mode, $\omega$ slightly increases and $\gamma$ slightly
decreases (both by $\sim$ 0.5 cm-1) even above $T_{c}$ and cannot be explained
by our model and requires further study (Fig. 3).

##  IV Conclusion

In summary, we have carried out electrical transport measurements and Raman
studies as a function of temperature in ultrathin 1T′-MoTe2. The nanocrystals
with thickness larger than 11 nm show a resistance maximum which is attributed
to a transition from 1T′ to Td (WSM) state on cooling. Furthermore, the
resistivity increase at low temperatures for $T<20K$ shows $\rho\propto 1/T$,
a signature of the WSM state. This is further supported by our $1/f$
electrical-noise studies. Temperature induced transition from topologically
trivial 1T′ phase to type-II Weyl semimetallic Td phase is also captured by
means of anomalous temperature dependence of few phonon frequencies and their
linewidths due to contribution from the electron-phonon coupling in the WSM
phase. We hope that our studies will motivate theoretical studies to
quantitatively understand the absence of the WSM in ultrathin 1T′-MoTe2 due to
possible gap opening at the Weyl point and the temperature dependence of the
$1/f$ power spectral density.

## Acknowledgement

The authors would like to acknowledge the Center for Nanoscience and
Engineering (CENSE), IISc for fabrication facilities. AD and AKS thank the
Department of Science and Technology (DST) for funding with DSTO-2051 and
Nanomission project as well as Year of Science Professorship.

## References

  * Wang  _et al._ [2012] Q. H. Wang, K. Kalantar-Zadeh, A. Kis, J. N. Coleman, and M. S. Strano, Nature Nanotechnology 7, 699 (2012).
  * Morris  _et al._ [1972] R. C. Morris, R. V. Coleman, and R. Bhandari, Phys. Rev. B 5, 895 (1972).
  * Sajadi  _et al._ [2018] E. Sajadi, T. Palomaki, Z. Fei, W. Zhao, P. Bement, C. Olsen, S. Luescher, X. Xu, J. A. Folk, and D. H. Cobden, Science 362, 922 (2018).
  * Moncton  _et al._ [1977] D. E. Moncton, J. D. Axe, and F. J. DiSalvo, Phys. Rev. B 16, 801 (1977).
  * Kang  _et al._ [2019] K. Kang, T. Li, E. Sohn, J. Shan, and K. F. Mak, Nature Materials 18, 324 (2019).
  * Soluyanov  _et al._ [2015] A. A. Soluyanov, D. Gresch, Z. Wang, Q. Wu, M. Troyer, X. Dai, and B. A. Bernevig, Nature 527, 495 (2015).
  * Wang  _et al._ [2016a] Z. Wang, D. Gresch, A. A. Soluyanov, W. Xie, S. Kushwaha, X. Dai, M. Troyer, R. J. Cava, and B. A. Bernevig, Phys. Rev. Lett. 117, 056805 (2016a).
  * Wang  _et al._ [2016b] Y. Wang, E. Liu, H. Liu, Y. Pan, L. Zhang, J. Zeng, Y. Fu, M. Wang, K. Xu, Z. Huang, _et al._ , Nature Communications 7, 13142 (2016b).
  * Qian  _et al._ [2014] X. Qian, J. Liu, L. Fu, and J. Li, Science 346, 1344 (2014).
  * Zyuzin and Burkov [2012] A. A. Zyuzin and A. A. Burkov, Phys. Rev. B 86, 115133 (2012).
  * Son and Spivak [2013] D. T. Son and B. Z. Spivak, Phys. Rev. B 88, 104412 (2013).
  * Xu  _et al._ [2015a] S.-Y. Xu, I. Belopolski, D. S. Sanchez, C. Zhang, G. Chang, C. Guo, G. Bian, Z. Yuan, H. Lu, T.-R. Chang, _et al._ , Science Advances 1, e1501092 (2015a).
  * Xu  _et al._ [2015b] S.-Y. Xu, I. Belopolski, N. Alidoust, M. Neupane, G. Bian, C. Zhang, R. Sankar, G. Chang, Z. Yuan, C.-C. Lee, S.-M. Huang, H. Zheng, J. Ma, D. S. Sanchez, B. Wang, A. Bansil, F. Chou, P. P. Shibayev, H. Lin, S. Jia, and M. Z. Hasan, Science 349, 613 (2015b).
  * Lv  _et al._ [2015] B. Lv, N. Xu, H. Weng, J. Ma, P. Richard, X. Huang, L. Zhao, G. Chen, C. Matt, F. Bisti, _et al._ , Nature Physics 11, 724 (2015).
  * Yang  _et al._ [2015] L. Yang, Z. Liu, Y. Sun, H. Peng, H. Yang, T. Zhang, B. Zhou, Y. Zhang, Y. Guo, M. Rahn, _et al._ , Nature Physics 11, 728 (2015).
  * Goli  _et al._ [2012] P. Goli, J. Khan, D. Wickramaratne, R. K. Lake, and A. A. Balandin, Nano Letters 12, 5941 (2012).
  * Yang  _et al._ [2014] J. Yang, W. Wang, Y. Liu, H. Du, W. Ning, G. Zheng, C. Jin, Y. Han, N. Wang, Z. Yang, _et al._ , Appl. Phys. Lett. 105, 063109 (2014).
  * Otrokov  _et al._ [2019] M. M. Otrokov, I. P. Rusinov, M. Blanco-Rey, M. Hoffmann, A. Y. Vyazovskaya, S. V. Eremeev, A. Ernst, P. M. Echenique, A. Arnau, and E. V. Chulkov, Phys. Rev. Lett. 122, 107202 (2019).
  * Puotinen and Newnham [1961] D. Puotinen and R. Newnham, Acta Cryst. 14, 691 (1961).
  * Ruppert  _et al._ [2014] C. Ruppert, O. B. Aslan, and T. F. Heinz, Nano Lett. 14, 6231 (2014).
  * Keum  _et al._ [2015] D. H. Keum, S. Cho, J. H. Kim, D.-H. Choe, H.-J. Sung, M. Kan, H. Kang, J.-Y. Hwang, S. W. Kim, H. Yang, _et al._ , Nature Physics 11, 482 (2015).
  * Qi  _et al._ [2016] Y. Qi, P. G. Naumov, M. N. Ali, C. R. Rajamathi, W. Schnelle, O. Barkalov, M. Hanfland, S.-C. Wu, C. Shekhar, Y. Sun, _et al._ , Nature Communications 7, 11038 (2016).
  * Sankar  _et al._ [2017] R. Sankar, G. Narsinga Rao, I. P. Muthuselvam, C. Butler, N. Kumar, G. Senthil Murugan, C. Shekhar, T.-R. Chang, C.-Y. Wen, C.-W. Chen, W.-L. Lee, M.-T. Lin, H.-T. Jeng, C. Felser, and F. C. Chou, Chemistry of Materials 29, 699 (2017).
  * Clarke  _et al._ [1978] R. Clarke, E. Marseglia, and H. Hughes, Philosophical Magazine B 38, 121 (1978).
  * Zhang  _et al._ [2016] K. Zhang, C. Bao, Q. Gu, X. Ren, H. Zhang, K. Deng, Y. Wu, Y. Li, J. Feng, and S. Zhou, Nature Communications 7, 13552 (2016).
  * Chen  _et al._ [2016] S.-Y. Chen, T. Goldstein, D. Venkataraman, A. Ramasubramaniam, and J. Yan, Nano Lett. 16, 5852 (2016).
  * He  _et al._ [2018] R. He, S. Zhong, H. H. Kim, G. Ye, Z. Ye, L. Winford, D. McHaffie, I. Rilak, F. Chen, X. Luo, Y. Sun, and A. W. Tsen, Phys. Rev. B 97, 041410 (2018).
  * Huang  _et al._ [2016] L. Huang, T. M. McCormick, M. Ochi, Z. Zhao, M.-T. Suzuki, R. Arita, Y. Wu, D. Mou, H. Cao, J. Yan, _et al._ , Nature Materials 15, 1155 (2016).
  * Deng  _et al._ [2016] K. Deng, G. Wan, P. Deng, K. Zhang, S. Ding, E. Wang, M. Yan, H. Huang, H. Zhang, Z. Xu, _et al._ , Nature Physics 12, 1105 (2016).
  * Berger  _et al._ [2018] A. N. Berger, E. Andrade, A. Kerelsky, D. Edelberg, J. Li, Z. Wang, L. Zhang, J. Kim, N. Zaki, J. Avila, _et al._ , npj Quantum Materials 3, 2 (2018).
  * Sun  _et al._ [2015] Y. Sun, S.-C. Wu, M. N. Ali, C. Felser, and B. Yan, Phys. Rev. B 92, 161107 (2015).
  * Song  _et al._ [2018] P. Song, C. Hsu, M. Zhao, X. Zhao, T.-R. Chang, J. Teng, H. Lin, and K. P. Loh, 2D Materials 5, 031010 (2018).
  * Hosur  _et al._ [2012] P. Hosur, S. A. Parameswaran, and A. Vishwanath, Phys. Rev. Lett. 108, 046602 (2012).
  * Topalian  _et al._ [2015] Z. Topalian, S.-Y. Li, G. A. Niklasson, C. G. Granqvist, and L. B. Kish, Journal of Applied Physics 117, 025303 (2015).
  * Chandni  _et al._ [2009] U. Chandni, A. Ghosh, H. S. Vijaya, and S. Mohan, Phys. Rev. Lett. 102, 025701 (2009).
  * Barone  _et al._ [2016] C. Barone, F. Lang, C. Mauro, G. Landi, J. Rappich, N. H. Nickel, B. Rech, S. Pagano, and H. C. Neitzert, Scientific Reports 6, 34675 (2016).
  * Babić  _et al._ [2007] D. Babić, J. Bentner, C. Sürgers, and C. Strunk, Phys. Rev. B 76, 134515 (2007).
  * Koushik  _et al._ [2011] R. Koushik, M. Baenninger, V. Narayan, S. Mukerjee, M. Pepper, I. Farrer, D. A. Ritchie, and A. Ghosh, Phys. Rev. B 83, 085302 (2011).
  * Barone  _et al._ [2013] C. Barone, E. Bellingeri, M. Adamo, E. Sarnelli, C. Ferdeghini, and S. Pagano, Superconductor Science and Technology 26, 075006 (2013).
  * Kuiri  _et al._ [2016] M. Kuiri, B. Chakraborty, A. Paul, S. Das, A. Sood, and A. Das, Appl. Phys. Lett. 108, 063506 (2016).
  * Kuiri  _et al._ [2015] M. Kuiri, C. Kumar, B. Chakraborty, S. N. Gupta, M. H. Naik, M. Jain, A. Sood, and A. Das, Nanotechnology 26, 485704 (2015).
  * Kumar  _et al._ [2016] C. Kumar, M. Kuiri, J. Jung, T. Das, and A. Das, Nano Letters 16, 1042 (2016).
  * Kim  _et al._ [2017] H.-J. Kim, S.-H. Kang, I. Hamada, and Y.-W. Son, Phys. Rev. B 95, 180101 (2017).
  * [44] D. Qu, X. Liu, M. Huang, C. Lee, F. Ahmed, H. Kim, R. S. Ruoff, J. Hone, and W. J. Yoo, Advanced Materials 29, 1606433.
  * Haratipour and Koester [2014] N. Haratipour and S. J. Koester, in  _72nd Device Research Conference_ (2014) pp. 171–172.
  * Paul  _et al._ [2019] S. Paul, S. Karak, M. Mandal, A. Ram, S. Marik, R. Singh, and S. Saha, arXiv preprint arXiv:1910.09996 (2019).
  * Efetov and Kim [2010] D. K. Efetov and P. Kim, Phys. Rev. Lett. 105, 256805 (2010).
  * Brown [1966] B. E. Brown, Acta Crystallographica 20, 268 (1966).
  * Ma  _et al._ [2016] X. Ma, P. Guo, C. Yi, Q. Yu, A. Zhang, J. Ji, Y. Tian, F. Jin, Y. Wang, K. Liu, T. Xia, Y. Shi, and Q. Zhang, Phys. Rev. B 94, 214105 (2016).
  * Balkanski  _et al._ [1983] M. Balkanski, R. F. Wallis, and E. Haro, Phys. Rev. B 28, 1928 (1983).
  * Menéndez and Cardona [1984] J. Menéndez and M. Cardona, Phys. Rev. B 29, 2051 (1984).
  * Xu  _et al._ [2017] B. Xu, Y. M. Dai, L. X. Zhao, K. Wang, R. Yang, W. Zhang, J. Y. Liu, H. Xiao, G. Chen, S. A. Trugman, _et al._ , Nature Communications 8, 14933 (2017).


# The Protein Engineering Tournament:
An Open Science Benchmark for Protein Modeling and Design

Chase Armer1*, Hassan Kane2*, Dana Cortade3*, Dave Estell4, Adil Yusuf2,
Radhakrishna Sanka5,
Henning Redestig4, TJ Brunette3, Pete Kelly3, Erika DeBenedictis3,6

( 1Columbia University, New York City, United States • 2Medium Biosciences
3Align to Innovate, Cambridge, United States • 4International Flavors and
Fragrances
5Boston University, Boston, United States • 6Francis Crick Institute, London,
United Kingdom
*Designates equal contribution
E-mail<EMAIL_ADDRESS>)

###### Abstract

The grand challenge of protein engineering is the development of computational
models that can characterize and generate protein sequences for any arbitrary
function. However, progress today is limited by lack of 1) benchmarks with
which to compare computational techniques, 2) large datasets of protein
function, and 3) democratized access to experimental protein characterization.
Here, we introduce the Protein Engineering Tournament, a fully-remote,
biennial competition for the development and benchmarking of computational
methods in protein engineering. The tournament consists of two rounds: a first
in silico round, where participants use computational models to predict
biophysical properties for a set of protein sequences, and a second in vitro
round, where participants are challenged to design new protein sequences,
which are experimentally measured with open-source, automated methods to
determine a winner. At the Tournament’s conclusion, the experimental protocols
and all collected data will be open-sourced for continued benchmarking and
advancement of computational models. We hope the Protein Engineering
Tournament will provide a transparent platform with which to evaluate progress
in this field and mobilize the scientific community to conquer the grand
challenge of computational protein engineering.

Figure 1:  Overview of the Tournament. (A)The tournament impacts the space by
providing a transparent benchmark for computational methods, open datasets for
the research community, and automated protocols for continued independent
benchmarking. (B) The Tournament is designed to accelerate creation of
computational techniques that can predict the biophysical properties for any
given protein and generate a protein with any desired properties. (C) The
Tournament consists of two sequential rounds: an in silico round for
predicting properties of proteins and an in vitro round for generating
proteins with specific properties.

##  1 Introduction

Protein engineering is the discipline of modifying or designing protein DNA
sequences to create new functions or improve upon existing ones[1]. It has
historically been used to create new enzymes for industrial processes[2],
develop new biologics with improved efficacy[3], and engineer proteins for
better tasting plant-based burgers[4]. Many current global challenges are also
being addressed through protein engineering, such as: reducing greenhouse gas
emissions[5], recycling plastic[6], diagnosing biomarkers related to rare
diseases[7] and the development of biologics targeting infectious diseases[8].
In order to develop increasingly sophisticated protein solutions, powerful
computational models are needed to guide the design of proteins with new or
improved functionality.

The field of computational protein engineering aims to guide the design of
proteins by developing predictive[9] and generative[10] models; however,
several obstacles are currently limiting model development. Although various
machine learning methods have been developed over the years to take advantage
of existing evolutionary, structural, and assay data, the lack of complex,
publicly available datasets, limited experimental reproducibility, and absence
of infrastructure for benchmarking models all impede model development and
validation. The majority of publicly available datasets[11] capturing sequence
to function data are relatively simple, mapping single point mutations[12] to
a single biophysical property per experimental condition. When designing new
variants, computational scientists often lack the means[13] to experimentally
reproduce the conditions in which certain datasets were created and therefore
cannot reliably perform comparisons against existing models. Until now, there
have been no common benchmarks nor infrastructure for experimentally assessing
such models.

The Protein Engineering Tournament aims to tackle the aforementioned obstacles
by curating a series of tournaments centered on various protein engineering
challenges. By providing a transparent platform for benchmarking protein
design methods, generating publicly available datasets, and developing open-
sourced infrastructure for automated experimentation, the Tournament hopes to
reduce barriers to model development and validation (Figure 1A). The
Tournament will act as a platform for accelerating development in
computational protein design (Figure 1B), enabling computational scientists to
benchmark their models by both predicting protein properties and generating
novel proteins with improved functionality. To achieve this goal, the
Tournament includes both an in silico round and an in vitro round (Figure 1C).

The Tournament also aims to inspire more machine learning scientists to
contribute to the field of protein engineering by making the field more
accessible and transparent. The Tournament will engage with the field of
machine learning to solve technical problems that uniquely arise in protein
engineering and develop the infrastructure it needs to design, build, and test
better models and protein engineering strategies. Furthermore, we hope to
engage with the community by showcasing challenges with high societal value
that are currently unsolved. To achieve this goal, we will intentionally focus
on both technically challenging tasks and use cases which may be overlooked by
academia and/or industry.

Figure 2:  Overview of Tournament Rounds. The tournament consists of two
rounds. (A) In the in silico round participants will predict properties of
given protein sequences, with optional training data provided for specific
events. Performance is evaluated by comparing the participant’s predictions
with the ground-truth values. Top performing participants will be selected to
advance to the in vitro round. (B) In the in vitro round participants will be
asked to generate protein sequences based on desired properties, which will
then be expressed and characterized using cloud labs. Performance of the
designed sequences will be evaluated as a weighted combination of the
protein’s properties; the exact evaluation metric will be event-dependent.

##  2 Related Approaches

Open datasets have long provided valuable opportunities for developing and
benchmarking new methods in machine learning research. Computer vision
datasets, such as MNIST[14] and ImageNet[15], not only provided individual
research labs with a substrate for experimenting with new approaches but also
created a yardstick with which to measure collective progress. Researchers in
the protein engineering community have also utilized open datasets and
introduced tasks, such as FLIP[16], TAPE[17], and ProteinNet[18], to encourage
similar developments. These datasets have challenged researchers to develop
models performing a wide range of tasks, such as predicting biophysical
properties on both protein and amino acid levels, as well as predictions on
variant effects and structural features.

Science competitions take these efforts a step further by allowing researchers
to test their computational methods on never-before-seen datasets. Perhaps the
most notable example is the Critical Assessment of Structure Prediction
(CASP)[19] a biennial event for computational protein structure prediction.
Since its inception, CASP has become a crucial benchmark for the protein
structure prediction community. By creating visibility around a single event,
the competition has inspired an ambitious spirit among researchers to develop
the best performing method, thereby encouraging a strong pace of

method development. Moreover, a well-known competition can offer an accessible
entry point for research groups outside of the field to participate, as
demonstrated by Deepmind’s participation in the 2018 CASP competition[20].
CASP has inspired the creation of similar competitions, like the Critical
Assessment of Computational Hit-finding Experiments (CACHE)[21], which was
created in the computational chemistry field to benchmark novel approaches for
finding new small-molecule binders.

We believe there is a burgeoning opportunity to create a new scientific
competition that addresses the unique challenges of predicting and engineering
protein function. Computational research groups which lack the ability to
experimentally characterize engineered proteins are currently unable to
meaningfully evaluate the performance of their protein engineering methods. By
introducing never-before-seen datasets on protein function and offering open-
source experimental characterization of novel proteins, the Protein
Engineering Tournament hopes to overcome this barrier and enable research
groups from all backgrounds to participate in cutting-edge protein engineering
research. In doing so, we expect the Tournament will become a unifying
benchmark for the field.

##  3 THE PROTEIN ENGINEERING TOURNAMENT

###  3.1 Tournament structure

The tournament will consist of two sequential rounds: the in silico round and
the in vitro round (Figure 2).

In the in silico round teams will be tasked with developing predictive models
which can infer the biophysical properties of protein sequences (Figure 2A).
The in silico round contains the option to either directly predict biophysical
properties based on protein sequence (zero-shot learning) or to pre-train a
model on an optional training dataset (supervised learning). After choosing
their method, participants will be asked to predict the biophysical properties
for a held-out test set of protein sequences, such as their stability,
expressibility, and activity. Submissions will be evaluated by a comparison
between predictions and experimental data, using statistics such as the
Spearman correlation.

Each biophysical property will be assessed independently, and participants
will be allowed to submit predictions for as few or as many of these
properties as they desire. Therefore, research labs that are building
specialized computational tools for one specific property of interest, such as
a stability predictor or an enzymatic activity predictor, will be able to
focus on submitting predictions for that property alone. In contrast, research
teams focused on applying state-of-the-art machine learning techniques to
biological datasets, for example, may be more interested in submitting
predictions for all available properties. Once submissions are closed and the
performance of each team has been evaluated, the final leaderboard of the in
silico round will be published and the datasets used will be made publicly
available. The highest performing teams will then advance to the in vitro
round.

In the in vitro round (Figure 2B), the teams will be asked to design protein
sequences that maximize or satisfy certain biophysical properties (e.g., an
enzyme design challenge may ask for sequences which maximize enzymatic
activity while staying above a specified threshold for stability and
expressibility). Each team will submit a list of amino acid sequences that
will then be synthesized and experimentally characterized using automated
laboratory protocols developed by the Tournament and its partners. Once
experimentally characterized, a ranking algorithm will evaluate the
submissions to produce a score for each participating team. As an example, a
submission score may be produced by calculating the normalized discounted
cumulative gain (NDCG) for the enzymatic activity of all sequences above the
required thresholds of stability and expressibility. The exact evaluation
metric will depend on the protein target in question and will be tailored to
the academic or industrial use-case for which it is being studied.

At the conclusion of the in vitro round, the Tournament will publish the final
leaderboard, and the team with the highest performing proteins will be awarded
the title of Protein Engineering Champion. The characterized protein sequences
in the in vitro round and the datasets of the in silico round will be made
publicly available. Furthermore, the automated protocols used to
experimentally characterize the designed proteins will be made available to
the public for continued use.

Figure 3:  Round Advancement. There will be three avenues for participants to
enter into the in vitro round, with each avenue catering to a unique audience.
(A) Research groups specializing in generative design methods can apply to
directly enter the in vitro round via an application submission. (B) The top
performing teams from the in silico round will be invited to participate in
the in vitro round. (C) Groups can pay to enter the in vitro round directly.
Figure 4:  Selecting Protein Design Challenges. (A) The Tournament provides a
unique opportunity to highlight important protein engineering challenges that
would otherwise fall outside the purview of both academic and industry
incentives. (B) The Tournament will continually expand to new domains of
protein design; in each domain, we will continually select challenges that
push the limits of current techniques.

###  3.2 Participation

The first round of the Protein Engineering Tournament, the in silico round,
will be open to any and all researchers interested in participating.
Interested teams composed of one or more individuals will be able to download
challenge data and upload final predictions. The in silico challenge will be
open to the community for a specified amount of time, after which the
submissions will be evaluated and the final leaderboard will be released.

To allow the greatest number of teams to participate in the in vitro round,
while being conscious of the costs associated with DNA synthesis and
experimental characterization, we propose three avenues for admission: 1) top
performance in the in silico round, 2) submitting a written application, and
3) paid entry for corporate researchers (Figure 3). By offering these three
avenues for participating in the in vitro challenge we will increase the
accessibility of the tournament while maximizing opportunities for
participation, community impact, and benchmarking of new methods.

First, the top-performing teams in the in silico round will be offered the
opportunity to advance to the in vitro round (Figure 3A). Since the in silico
round is open to any team or individual with access to a computer, with no
entry fee or requirement on prior research experience, this path focuses on
rewarding successful computational approaches and increasing the accessibility
of the tournament

Second, we will provide an application system that allows researchers to apply
directly for a spot in the in vitro round (Figure 3B). The value of providing
an application-based path to entry is born from the fundamental differences
that exist between the in silico and in vitro challenges. There are likely
research groups who have developed impressive methods for generative protein
design, which would be well-suited for the in vitro challenge, but have no
prior experience with the property prediction tasks requested in the in silico
challenge. Without a separate application process, these groups may perform
poorly in the in silico round, or simply may not choose to participate at all,
and their generative methods would not have the opportunity to be benchmarked
in the Tournament.

For the third and final path to entry, participants will be allowed to pay a
fee which covers the cost of experimentation (Figure 3C). With this option,
funding for the Protein Engineering Tournament no longer becomes the limiting
factor on the number of participants that can compete and the number of
methods that can be evaluated. This path could provide entry to well-funded
corporate research labs capable of managing these costs, which further allows
our application system discussed above to focus on providing entry to
promising, less well-funded research groups.

Our belief is that by offering these three avenues for participating in the in
vitro challenge we will increase the accessibility of the tournament and
maximize opportunities for participation and benchmarking of new methods.

###  3.3 Selecting our protein engineering challenges

More than benchmarking the field, we believe the Tournament will become a
powerful engine for making meaningful progress on important protein
engineering problems. Many of the most impactful protein engineering
applications, from climate technology and green manufacturing to antivirals
and diagnostics, are not being fully addressed by current research efforts.

The Protein Engineering Tournament will become a vehicle for connecting
cutting-edge researchers with the experimental resources necessary to make
headway on important societal challenges that are not traditionally addressed
by industry or academia (Figure 4A). While industry possesses significant
resources to dedicate towards protein engineering problems, they are often
restricted to applications which can recover the cost of research and generate
profit. Conversely, while academia is capable of focusing on important
problems without the constraints of profitability, research labs often lack
the resources necessary to tackle large protein engineering efforts. This
leaves a research gap where many applications with significant societal
benefits have been under-resourced. With this consideration in mind, our
protein engineering challenges will be selected with a strong focus on real-
world impact.

The Tournament can be used to continually advance the field of protein
engineering by selecting problems that push the limits of current techniques.
Proteins possess a diverse array of functions, from enzymatic catalysis and
molecular binding to chemical transportation, and our protein engineering
challenges in the in silico and in vitro rounds will continuously evolve over
time to reflect this myriad of functional possibilities.

The first Tournament will likely focus on a single function, with enzyme
engineering or protein binder design as strong initial candidates, but in
future tournaments, the design challenges will expand to encompass more
domains of function (Figure 4B). The order in which we introduce new functions
will be driven by practical application, technical feasibility, and
amenability to high-throughput experimentation. As our computational methods
improve, our challenges will expand into increasingly more difficult and
complex domains, such that the frontier of scientific capabilities is always
represented in the Tournament’s challenges.

The final protein engineering challenges will be decided on by our Target
Selection team, with input from our scientific advisory board, philanthropic
funding partners, and the larger scientific community. (Figure S1). The
scientific advisory board will be composed of researchers in academia and
industry. Their knowledge on the current frontier of challenges and
opportunities in the field of protein engineering will further guide our
selection of protein challenges. Additionally, we will invite input from
funding partners who will use this opportunity to help advance the application
of engineered proteins to problems salient to their philanthropic goals.

##  4 TOURNAMENT CREATION ENGINE

The Tournament represents a long-term investment in benchmarking the field of
protein engineering. The tournament will produce new datasets for method
development, new automated assays for protein characterization, and new
benchmarked results for the community’s current-best algorithms, with the
biennial cycle of the tournament also creating an opportunity to routinely
take stock of current methods and assess the state of the field (Figure 5).

Figure 5:  Tournament Creation Engine. Creating the Protein Engineering
Tournaments will be a cyclical process. We will select new protein design
challenges, develop automated assays for these challenges, host a tournament
with these new assays and the datasets they’ve produced, and evaluate the
tournament results. The assays, datasets, and evaluation will be open-sourced.

###  4.1 Cloud laboratories and method development

To maximize our impact, we want the assays we develop for each tournament to
be available long after the tournament has concluded. To accomplish this goal
and capitalize on the benefits of automated experimentation, we will design
and execute our experimental workflows in cloud laboratories (Figure 5).

Commercial cloud science laboratories, such as Emerald Cloud Labs, and
academic cloud science laboratories, such as those found at Boston University
and Carnegie Mellon, enable scientists to forgo the lab bench in favor of
running experimental biology protocols in automation-enabled facilities. In
this paradigm researchers write their assays in a symbolic laboratory
programming language that specifies the instructions for each step of their
protocol. Once written, scientists can queue up their experiments, wet-lab
robots execute each step, and the resulting data is uploaded for analysis.

This approach offers the potential to greatly improve the accessibility and
reproducibility of life science research by enabling scientists to share
experimental protocols as easily as we share software. Experimental assays
written with a symbolic lab language can be uploaded to code-sharing websites
like Github, allowing any researcher from around the world to access and
reproduce this work. Cloud laboratories will also improve the productivity of
biological experimentation by transferring time-consuming experimental work
from the hands of human researchers onto the decks of precise, high-throughput
robots.

Cloud laboratories will serve as the experimental backbone of the Protein
Engineering Tournament to ensure our in vitro experiments are high-throughput,
reproducible, and accessible (Figure 5).

At the conclusion of each Tournament, the protocols we developed to execute
characterization assays in the in vitro round will be made openly available to
the scientific community. Therefore, researchers will be able to continually
benchmark their computational methods on the same standardized assays even
after the Tournament has concluded. Since each Tournament will introduce new
protein engineering challenges, this approach will lead to an ever-expanding
corpus of open-source protein engineering workflows to help benchmark new
computational methods for years to come.

###  4.2 Generating datasets

The automated experimental protocols discussed above will be used to generate
the datasets for the in silico and in vitro rounds and also be used to
generate additional data based on participants’ submitted protein sequences
during the in vitro round. After the Tournament has ended, all data produced
during the Tournament will be made publicly available along with the
corresponding automated protocols (Figure 5). Furthermore, the Tournament can
act as an avenue for academic and corporate research entities to make
unpublished datasets of protein function available as predictive challenges in
the in silico round.

###  4.3 Analysis of the State of the Field

Finally, we will aggregate the learnings from our tournament’s results into a
State of the Field report (Figure 5). This report will analyze the performance
of our participant’s computational approaches, noting the relative standing of
different techniques across different challenges, and discussing our
collective progress throughout the various domains of protein engineering.

###  4.4 Governance

The Protein Engineering Tournament is operated by Align to Innovate, a non-
profit dedicated to improving the reproducibility, scalability, and
shareability of life science research through community-driven initiatives.
The Tournament will be run by a combination of Align to Innovate employees and
volunteers from the protein engineering community (Figure S1). The Tournament
Coordinator, who’s responsibilities entail coordinating the teams to ensure a
successful tournament, will be a full time employee of Align to Innovate.
Members of the Target Selection, Data Science, Organization and Outreach, and
Cloud Assay Development teams will largely be composed of volunteers from the
research community, in addition to supporting members from within the Align to
Innovate team.

##  5 PILOT TOURNAMENT

A pilot tournament began May 1st 2023 with the theme of Enzyme Design based on
six datasets received from both industry and academic groups. Initial interest
in the pilot tournament led to the registration of just over 30 teams,
respresenting a mix of academic (55%), industry (30%), and independent (15%)
teams, with research experience running from Nobel Laureates to high school
students. For the pilot tournament, the in vitro round experimentation will be
performed in-house by a corporate partner. Development of the cloud laboratory
assays for future tournaments is currently underway within Align to Innovate,
the non-profit parent organization of the Protein Engineering Tournament.

##  6 CONCLUSION

The Protein Engineering Tournament introduces an innovative, community-driven
platform to accelerate the advancement of computational protein engineering.
This open science initiative combines in silico and in vitro methods,
employing cloud laboratories to ensure reproducibility and continued access to
experimental workflows. By creating a unified benchmark, the Tournament
stimulates collaboration and competition among researchers and presents
opportunities for the community to test their computational models on novel
protein engineering challenges. It offers a dynamic space for academia,
industry, and independent groups to broaden the horizons of protein
engineering, particularly in areas of high societal value that might be
currently underserved. Further, through its integration with Align to
Innovate, the Tournament builds upon the strengths of a diverse scientific
community, fostering transparency and sharing of knowledge. As we look ahead
to the first official Tournament, we anticipate that this initiative will
contribute significantly to the evolving landscape of protein engineering,
driving forward both technical and translational breakthroughs. By
democratizing access to protein design, we aspire to inspire a new generation
of computational protein engineers and guide the future of life science
research.

## References

  * Ulmer [1983] K. M. Ulmer.  Protein engineering.  _Science_ , 219:666–671, 1983.
  * Bornscheuer et al. [2012] U. T. Bornscheuer et al.  Engineering the third wave of biocatalysis.  _Nature_ , 485:185–194, 2012.
  * Maynard and Georgiou [2000] J. Maynard and G. Georgiou.  Antibody engineering.  _Annu. Rev. Biomed. Eng._ , 2:339–376, 2000.
  * Whitehurst and Van Oort [2010] Robert J Whitehurst and Maarten Van Oort.  _Enzymes in food technology_ , volume 388.  Wiley Online Library, 2010.
  * Fradette et al. [2017] L. Fradette, S. Lefebvre, and J. Carley.  Demonstration results of enzyme-accelerated co2 capture.  _Energy Procedia_ , 114:1100–1109, 2017.
  * Gao et al. [2021] R. Gao, H. Pan, and J. Lian.  Recent advances in the discovery, characterization, and engineering of poly(ethylene terephthalate) (pet) hydrolases.  _Enzyme Microb. Technol._ , 150:109868, 2021.
  * Singh et al. [2019] R. S. Singh, T. Singh, and A. K. Singh.  Chapter 9 - enzymes as diagnostic tools.  In R. S. Singh, R. R. Singhania, A. Pandey, and C. Larroche, editors, _Advances in Enzyme Technology_ , pages 225–271. Elsevier, 2019.
  * Fan and Li [2017] G. Fan and J. Li.  Engineering antibodies for the treatment of infectious diseases.  _Adv. Exp. Med. Biol._ , 1053:207–220, 2017.
  * Hie and Yang [2022] B. L. Hie and K. K. Yang.  Adaptive machine learning for protein engineering.  _Curr. Opin. Struct. Biol._ , 72:145–152, 2022.
  * Strokach and Kim [2022] A. Strokach and P. M. Kim.  Deep generative modeling for protein design.  _Curr. Opin. Struct. Biol._ , 72:226–236, 2022.
  * Wang et al. [2019] C. Y. Wang et al.  Protabank: A repository for protein design and engineering data.  _Protein Sci._ , 28:672, 2019.
  * Notin et al. [2022] P. Notin et al.  Tranception: Protein fitness prediction with autoregressive transformers and inference-time retrieval.  In K. Chaudhuri et al., editors, _Proceedings of the 39th International Conference on Machine Learning_ , volume 162, pages 16990–17017. PMLR, 2022.
  * Thomas et al. [2022] N. Thomas, A. Agarwala, D. Belanger, Y. S. Song, and L. J. Colwell.  Tuned fitness landscapes for benchmarking model-guided protein design.  _bioRxiv_ , page 2022.10.28.514293, 2022.  doi:10.1101/2022.10.28.514293.
  * Deng [2012] L. Deng.  The mnist database of handwritten digit images for machine learning research.  _IEEE Signal Processing Magazine_ , 29(6):141–2, 2012.
  * Deng et al. [2009] J. Deng, W. Dong, R. Socher, L-J Li, K. Li, and F-F Li.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–55. IEEE, 2009.
  * Dallago et al. [2021] C. Dallago et al.  Flip: Benchmark tasks in fitness landscape inference for proteins.  _bioRxiv_ , page 2021.11.09.467890, 2021.  doi:https://doi.org/10.1101/2021.11.09.467890.
  * Rao et al. [2019] R. Rao et al.  Evaluating protein transfer learning with tape.  _Adv Neural Inf Process Syst_ , 32:9689–9701, 2019.
  * AlQuraishi [2019] M. AlQuraishi.  Proteinnet: a standardized data set for machine learning of protein structure.  _BMC Bioinformatics_ , 20:311, 2019.
  * Moult et al. [1995] J. Moult, J. T. Pedersen, R. Judson, and K. Fidelis.  A large-scale experiment to assess protein structure prediction methods.  _Proteins: Structure, Function, and Genetics_ , 23(3):ii–v, 1995.
  * Moult et al. [2018] J. Moult et al.  Critical assessment of methods of protein structure prediction (casp)—round xii.  _Proteins_ , C1:1–399, 2018.
  * Ackloo et al. [2022] S. Ackloo et al.  Cache (critical assessment of computational hit-finding experiments): A public–private partnership benchmarking initiative to enable the development of computational methods for hit-finding.  _Nature Reviews Chemistry_ , 6:287–295, 2022.