# What Remains of Visual Semantic Embeddings

Yue Jiao
University of Southampton
Southampton, UK
<EMAIL_ADDRESS>Jonathon Hare
University of Southampton
Southampton, UK
<EMAIL_ADDRESS>Adam Prügel-Bennett
University of Southampton
Southampton, UK
<EMAIL_ADDRESS>
###### Abstract

Zero shot learning (ZSL) has seen a surge in interest over the decade for its
tight links with the mechanism making young children recognize novel objects.
Although different paradigms of visual semantic embedding models are designed
to align visual features and distributed word representations, it is unclear
to what extent current ZSL models encode semantic information from distributed
word representations. In this work, we introduce the split of tieredImageNet
to the ZSL task, in order to avoid the structural flaws in the standard
ImageNet benchmark. We build a unified framework for ZSL with contrastive
learning as pre-training, which guarantees no semantic information leakage and
encourages linearly separable visual features. Our work makes it fair for
evaluating visual semantic embedding models on a ZSL setting in which semantic
inference is decisive. With this framework, we show that current ZSL models
struggle with encoding semantic relationships from word analogy and word
hierarchy. Our analyses provide motivation for exploring the role of context
language representations in ZSL tasks.

##  1 Introduction

Zero-Shot Learning (ZSL), which aims to recognize unseen classes, is
considered one of the most difficult generalization tasks. ZSL is also an
important task for demonstrating how a machine learning model understands
high-level semantic information and transfers knowledge from seen to unseen
classes.

After a decade of research, ZSL models have been shown to have made huge
progress on some small and medium sized datasets annotated with handmade
attributes [38, 31, 9]. The key to transferring semantic knowledge from seen
to unseen classes is often achieved by building a visual semantic embedding
model, which aims to build a bridge to fill the gap between visual information
and semantic information [20, 32, 7]. This approach is based on the hypothesis
that similar structural relationships emerge from independent visual and
linguistic representations [29, 14].

ZSL methods fail dramatically when they are tested on the ImageNet dataset
[5]. To address this problem, recent works propose to introduce different
mechanisms to represent robust semantic hierarchy [37, 16, 22]. However, there
is still an open question: _do current ZSL models perform useful semantic
inference on the ImageNet dataset?_ Exploring the role of distributed
semantics and visual semantic alignment models in the ZSL task is still
lacking. For example, if we have two images: one of a working dog and one of a
hunting dog, it is not clear whether the semantic difference and the semantic
hierarchy between “working” and “hunting” will help us distinguish these two
images.

As Hascoet _et al_. [11] highlight, the standard ImageNet benchmark proposed
by Frome _et al_. [7] has structural flaws due to its data splitting: although
the full set of classes is split into disjoint training and test sets, the
test classes are hypernyms or hyponyms of the training classes within the
Wordnet [25] hierarchy. When an image of a “greenhouse” is classified as a
“building” or “conservatory” (see Figure 1), in the traditional ZSL setting,
we consider it is a case of classification error, whilst in terms of the
semantic definition of a “greenhouse”, it doesn’t seem to be an error. On the
other hand, in the standard benchmark, test classes, which are out of the 1000
classes of the ILSVRC 2012 1K [30], are imbalanced. Some categories have the
high sample populations, while some categories only have low samples.
Therefore, the standard benchmark is not fair enough to demonstrate to what
extent visual semantic embedding models are affecting semantic understanding
in the ZSL task. To address this problem, we propose using the split of
tieredImageNet [28], a large subset of ILSVRC 2012 1K, to build a new ZSL
benchmark. In the tieredImageNet dataset, classes are grouped into categories
corresponding to higher-level nodes in the ImageNet hierarchy. These
categories are split into two parts to ensure that there is no hyponymy and
hypernymy between the training and test classes, thus ensuring that the two
sets are semantically and linguistically disjoint. There will be no longer any
super classes in the testing set. The tieredImageNet split makes ZSL a fair
game in which models depend on visual-semantic alignment.

Figure 1: Illustration the standard ZSL ImageNet data split over the Wordnet
hierarchy. The red node “greenhouse” is in the seen classes, while its parent
“building” and children “conservatory” and “orangery” are tested as the unseen
classes. The standard ZSL aims to distinguish objects from their hyponyms and
hyponyms making it difficult to demonstrate whether ZSL models can learn
semantic concepts via semantic embedding and alignment.

With the new tieredImageNet split, we test four main paradigms to learning a
visual semantic embedding model. The first paradigm is to map pre-trained
image features into a rich word embedding space, such as Word2Vec [24] or
GloVe [27]. The typical work in this paradigm is done by Frome _et al_. [7],
which helps visual object categorization systems handle very large numbers of
labels. The second paradigm is to utilize an Autoencoder (AE) or a Variational
Autoencoder (VAE) to explore a latent semantic space of image features, in to
which word vectors are jointly mapped. Semantic AE [33], Cross and
Distribution Aligned VAE [31] and SCAN [13] belong to this paradigm. The third
paradigm uses explicit knowledge graphs which encode relationships between
object classes. Like [37], the weights and biases of the final linear layer of
the pre-trained image classifier are the new learning objectives. The output
node embeddings from a graph convolutional neural network (GCN) are used to
predict the visual classifier for each category, when corresponding word
vectors are inputted. The forth paradigm considers the visual semantic
embedding as a hyperbolic space and maps visual features and word vectors to a
Poincaré disk. This kind of ZSL methods, proposed by Liu _et al_. [22],
follows the empirical rule that a hyperbolic latent space can yield more
interpretable representations if the data has hierarchical structure [2, 23].
Compared with the first paradigm, the fourth employs an exponential map
instead of a nonlinear map in the Euclidean latent space.

To avoid supervised pre-training of a state-of-the-art deep neural network for
visual object recognition to create the visual feature extractor, we introduce
Self-Supervised Learning (SSL) [15] into the framework of ZSL. SSL allows us
to obtain general visual features without any explicit semantic information
being introduced. We have many concerns with the pre-trained classifier
towards our goal of understanding the role of visual semantic embeddings that
contribute to recognize novel objects. There are no guarantees that classifier
trained using supervised methods on the seen classes will be able to generate
separable image features for novel categories without any knowledge transfer
like Deep Transfer Clustering [8]. However, a SSL encoder can provide
separable image features without utilising any label information. Therefore,
we believe that introducing SSL into the framework of ZSL can help us
understand how distributed semantics are aligned and how they affect ZSL
models recognize novel classes.

Finally, within our proposed zero-shot setting, we come to the conclusion that
current ZSL models struggle with semantic inference on the WordNet hierarchy.
When the unseen classes are not the hypernyms and hyponyms of seen classes,
ZSL models with different semantic alignment mechanisms do not show the
ability to use word analogy and word hierarchy implicitly, even though the
image features of novel classes are near linearly separable. We show that
graph-based visual semantic embedding models perform worse than the vanilla
one which just learns a nonlinear map.

Therefore, we believe future ZSL frameworks should explore the role of
contextual word representations on large vision-and-language tasks.

Our main contributions are to:

  * •

Introduce the tieredImagenet split into ZSL to replace the standard one, which
avoid major structural flaws in ZSL benchmarking;

  * •

Build a unified framework for ZSL with SSL pre-training, thus preventing any
semantic information leakage. This makes the visual semantic embedding models
be the only decisive factor in the ZSL framework;

  * •

Demonstrate that current ZSL methods can not generalize from complicated
semantic relationships. Encoded information from word analogy and word
hierarchy is not enough for the ZSL task.

##  2 Preliminaries

###  2.1 Problem Definition

Let $D_{S}=\\{(x,y)\mid x\in X_{S},y\in Y_{S}\\}$ be a set of training
examples consisting of seen images from $X_{S}$ and seen class labels from
$Y_{S}$. Let $D_{U}=\\{(x,y)\mid x\in X_{U},y\in Y_{U}\\}$ be a set of testing
examples. $Y_{U}$ represents the unseen set of class labels, which is disjoint
from $Y_{S}$. The task of ZSL is to learn a classifier, $X\rightarrow Y$,
where $X$ is the union set of $X_{S}$ and $X_{U}$ and $Y$ is the union set of
$Y_{S}$ and $Y_{U}$.

###  2.2 Word Embeddings

The idea of word embedding methods is derived from the distributional
hypothesis in linguistics: words that are used and occur in the same contexts
tend to purport similar meanings [10]. Word representations learned in an
unsupervised manner from the contextual relationship of words, or the co-
occurrence statistics of words, in large text corpora are the main source of
high-level semantic knowledge in most ZSL approaches. In ZSL these word
embedding models provide a mapping, $Y\rightarrow W$, from class labels to
high dimensional word vectors. We denote the word vector corresponding to a
label $y$ as $w(y)$. Cosine similarity is usually used to predict word
similarity. For a pair of word representations $w_{i}$ and $w_{j}$, we have:

$\mathrm{similarity}(w_{i},w_{j})=\frac{w_{i}\top
w_{j}}{\left\|w_{i}\right\|\left\|w_{j}\right\|}\enspace.$

###  2.3 Contrastive Learning

Previous ZSL models begin with training a classifier on the training set. This
step leaks specific semantic information into the feature extractor component
of a ZSL system. However, this step can not guarantee that we can obtain
linear separable novel image features, therefore, it is very difficult to
figure out whether the feature extractor or the downstream visual semantic
embedding model affect the generalization performance. Contrastive learning
(CL) provides a framework to learn representations of identity by pushing
apart two views of different objects and bringing together two views of same
objects in a representation space [4]. Recent works show that CL can generate
linear separable image features without any supervised label information [26,
35, 12, 3]. Given an anchor variables from the first view $v_{1,i}$, CL aims
to score the correct positive variable from the second view $v_{2,i}\sim
p(v_{2}\mid v_{1,i})$ higher compared to a set of $K$ negatives $v_{2,k}\sim
p(v_{2})$. A popular approach is to utilise the InfoNCE loss [26]:

$L_{NCE}=-\mathbf{E}\left[\log\frac{\exp{h(f(v_{1,i}),f(v_{2,i}))}}{\sum_{j=1}^{K}\exp{h(f(v_{1,i}),f(v_{2,j}))}}\right]$

The function $h$ represents a critic head and the function $f$ is a shared
encoder, which extracts view-invariant visual representations This way of
feature learning is natural and powerful. This allows the step of building the
visual semantic embedding model to be the only stage to align visual
information and distributed linguistic representations. When we freeze the
visual encoder, we can evaluate what mechanism performs better on semantic
understanding.

###  2.4 Graph Convolutional Networks

Graph convolutional networks (GCNs) [19] allow local graph operators to share
the statistical strength between word vectors of classes, which is considered
as a tool to utilize semantic information hidden in the WordNet hierarchy.

Given a graph with $N$ classes and a $d$ word embedding per class, $\mat{W}$
is the $N\times d$ word embedding matrix. We define a symmetric adjacency
matrix $\mat{A}$ and a degree matrix $\mat{D}$ to represent the connections
between the classes in the WordNet. The propagation rule to perform
convolutions on the graph is defined as:

$\displaystyle\mat{H}_{l+1}$
$\displaystyle=\sigma(\mat{D}^{-1}\mat{A}\mat{H}_{l}\mat{\Theta}_{l}),$
$\displaystyle\mat{H}_{0}$ $\displaystyle=\mat{W}$

Here $\mat{H}_{l}$ is the input at each layer of the GCN and $\mat{H}_{l+1}$
is the output. $\sigma$ is the activation function and $\mat{\Theta}_{l}$ is
the learnable parameter matrix at the layer $l$. We define a two layer GCN as
$g$.

###  2.5 Poincaré Embedding

The Poincaré embedding model is another natural technique to capture
hierarchical information in the WordNet hierarchy. As it is not pre-trained on
a large text corpus, each node in the Poincaré embedding model does not
contain any word co-occurrence statistics. Poincaré embeddings preserve the
distances between the nodes on the graph approximately, which can be used to
compute semantic similarities. Given two Poincaré embeddings $p_{i}$ and
$p_{j}$, we have the distance:

$d(p_{i},p_{j})=\cosh^{-1}\left(1+2\frac{\left\|p_{i}-p_{j}\right\|^{2}}{\left\|1-p_{i}\right\|^{2}+\left\|1-p_{j}\right\|^{2}}\right)\;.$

Two kinds of transformation are useful in a hyperbolic space. The mapping $e$
to project a vector $v$ in Euclidean space to hyperbolic space is defined as:

$p=e(v)=\tanh(\left\|v\right\|)\frac{v}{\left\|v\right\|}$

The Mobius multiplication $m$ is defined as:

$m(x)=\tanh\left(\frac{\left\|mx\right\|}{\left\|x\right\|}\tanh^{-1}(\left\|x\right\|)\right)\frac{mx}{\left\|x\right\|}$

##  3 A Unified Framework of ZSL

In this section, we provide our unified framework for the ZSL task. This new
pipeline consists of two steps.

Figure 2: Illustration of the process of building ZSL models in this work.
There are two steps in the whole pipeline. Left: The first step is pre-
training. In this step, an image encoder is learnt by Contrastive Learning on
the ILSVRC 2012 1K. GloVe word embeddings and Poincaré embeddings are
respectively trained on the Wikipedia corpus and the WordNet noun tree. (In
this case, we use the subtree with the root node “entity”.) Right: The second
step is to learn a visual semantic embedding model, which can be seen as
semantic alignment. In this framework we consider four independent paradigms
including: DeVISE, PrVISE, GrVISE and HyVISE. This two-step framework defines
a game in which novel image features are linear separable in theory and visual
semantic paradigms are the key to solve a semantic inference puzzle on the
WordNet. Therefore, it is fairer to demonstrate how visual semantic embedding
models do semantic generalization.

#### Pre-training.

In this step, three unsupervised embedding systems are derived from the ILSVRC
2012 1K, the Wikipedia2014 & Gigaword5 text corpus and the relation graph
built from the WordNet. The goal of build three embedding spaces is to capture
transferable visual, linguistic and structural information from different
modalities, which generalizes to an information alignment system.

After pre-training, image embeddings $I$ extracted by the CL encoder $f$, word
embeddings $W$ and Poincaré embeddings $P$ are powerful features for the
downstream visual semantic embedding learner.

#### Semantic alignment.

The second step is to build a visual semantic embedding model to align
semantics in different modalities. In this stage, $D_{S}=\\{(x,y)\mid x\in
X_{S},y\in Y_{S}\\}$ is used to train a semantic alignment learner. This step
can find structural or semantic correspondence between two embedding systems.
It is the key technique to transfer semantic knowledge from word
representations to a visual model. In out framework, four independent
paradigms (DeVISE, PrVISE, GrVISE and HyVISE) are defined below. Each of them
use different properties of word embedding systems. Note that the HyVISE which
maps image and word representations to a Riemannian space has achieved the
state-of-the-art performance in the standard ImageNet benchmark with
traditional zero-shot setting.

  * •

DeVISE DeVISE is proposed by Frome _et al_. [7] as the abbreviation of a deep
visual semantic embedding model. In this work, we redefine it as a
discriminative visual semantic model for its behaviour of mapping visual
features to the word embedding space by a combination of dot-product
similarity and hinge rank loss. For a data instance in $D_{S}$, we have the
loss:

$\begin{split}&L_{DeVISE}(x,y)=\sum_{j\neq y}\max[0,\\\ &margin-w(y)\cdot
t(f(x))+w_{j}\cdot t(f(x))]\end{split}$ (1)

where $t$ is a trainable transformation neural network. This loss aims to make
image features close to their word embeddings and remains distances with word
embeddings for incorrect labels. DeVISE is the natural way to project the
information in the visual domain to the distributed word space.

  * •

PrVISE Probabilistic visual semantic embedding (PrVISE) is a model derived
from a Variational Autoencoder [18] framework, a generative model with a prior
distribution $p(z)=\mathcal{N}(z;0,I)$. The VAE model makes it possible to
approximate a latent data distribution from a reconstruction task. As its
extension, PrVISE provides an adaptive latent prior distribution, which can be
learnt by a neural network $q_{\phi_{w}}$ from the pre-trained word embedding
space. By minimizing the KL divergence between the image latent distribution
$q_{\phi_{i}}$ and the adaptive prior distribution, we define the loss of
PrVISE as:

$\begin{split}L_{PrVISE}(x,y)&=-\mathbf{E}_{q_{\phi_{i}}(z|f(x))\\!}\left[\log\\!\left(p_{\theta_{i}}(f(x)|z)\right)\right]\\\
&-\mathbf{E}_{q_{\phi_{w}}(z|w(y))\\!}\left[\log\\!\left(p_{\theta_{w}}(w(y)|z)\right)\right]\\\
&+\mathbf{KL}\\!\left(q_{\phi_{i}}(z|f(x))\,\big{\|}\,q_{\phi_{w}}(z|w(y))\right),\end{split}$
(2)

where $p_{\theta_{w}}$ and $p_{\theta_{i}}$ are decoders for word and image
feature reconstruction. PrVISE aims to find a joint embedding space for visual
and linguistic information.

  * •

GrVISE GrVISE represents the visual semantic embedding model with Graph neural
networks (like SGCN, DGP in [37, 16]). It addresses the problem that how to
encode additional information in graph edges with a graph with weighted nodes.
In the standard ZSL setting, it seems that capturing relational information
from pre-trained word embeddings will augment semantics. In our case, GrVISE
firstly trains a linear probe on $D_{S}$. A shallow GCN $g(\cdot)$ is expected
to distill discriminative visual semantic embeddings from the word embedding
inputs. By minimizing the l2 loss between the GCN output $g(w)$ and the linear
probe parameter $l$:

$\begin{split}&L_{GrVISE}=\sum_{w_{i}\in W,l_{i}\in
L}{\left\|g(w_{i})-l_{i}\right\|^{2}}\end{split}$ (3)

we hope the GCN is able to predict the parameters of a new linear probe for
unseen classes.

  * •

HyVISE HyVISE is a Riemannian version of DeVISE. Instead of mapping image
features to a Euclidean space, HyVISE firstly projects image representations
to a hyperbolic space with a map $e$, then models a Mobius transformer $m$ to
make image representations close to their corresponding Poincaré embeddings.
The loss is defined as:

$\begin{split}&L_{HrVISE}(x,y)=\sum_{j\neq y}\max[0,margin\\\
&+d(m(e(f(x))),p(y))-d(m(e(f(x))),p_{j})]\end{split}$ (4)

Unlike previous works, we do not fine-tune the image encoder during the
semantic alignment stage. We consider this is essential to detect the
effectiveness of each visual semantic embedding mechanism.

Closely related to our work is done by Sylvain  _et al_. [34], who propose a
two-step framework CM-DIM with self-supervised encoders. However, CM-DIM
focused on the local and compositional behaviours of different pre-training
representations for the conventional zero-shot learning benchmarks. Our
intention is merely to explore how the role of distributed word
representations are influenced in the different forms of zero-shot learning
models, which we think is the key to make zero-shot learning more applicable
in real-world applications. In this work, capturing image representations by a
self-supervised method is not a prioritized problem, recent works [26, 35, 12,
3] have demonstrated that contrastive learning can learn good image
representations.

In the next section, we will use this unified ZSL framework and evaluate how a
ZSL model works under a tough data split which needs high-level semantic
information inference.

##  4 Evaluation

In this section, we firstly introduce the data split and evaluation tools and
metrics. Then we try to answer the following questions with comparative
experiments.

  * •

Does a visual semantic embedding model have the benefit to use information in
the word embedding space?

  * •

Do graph-based models learn the hierarchical distributed semantics?

###  4.1 tieredImageNet

The standard data split in the ImageNet has its own structural flaws. Test
classes are the hypernym and hyponym of training classes. On the other hand,
testing images contain many inconsistencies, it is hard to ensure images with
high quality are tested. While tieredImageNet is a large subset of ILSVRC 2012
1k, in which classes are nodes of a tree of height 13 covering 608 classes. To
build a ZSL benchmark, we combine the training classes and validation classes
of tieredImageNet as the seen set, and we keep its own testing classes. In our
ZSL setting, 448 classes are in the seen set, and 160 classes are in the
testing set. Images in the training set of ILSVRC 2012 1k with the seen labels
are used to train a visual semantic embedding model. Images in the validation
set of ILSVRC 2012 1k with the seen labels are used to evaluate embedding
performance. Images in the validation set of ILSVRC 2012 1k with the unseen
labels are used to evaluate ZSL performance.

###  4.2 Model Details

For pre-trained image encoder, we use a ResNet-50 trained for 800 epochs on
the ILSVRC 2012 1k with InfoMin CL Algorithm [36]
111https://github.com/HobbitLong/PyContrast. With a linear probe, this image
encoder shows $73.0\%$ classification accuracy on the ILSVRC 2012 1k task. We
choose 300 dimensional GloVe222https://nlp.stanford.edu/projects/GloVe/ with
6B tokens as the pre-trained word vectors. For each class label, we average
all the GloVe vectors of its synonyms. We train a Poincaré embedding model
with gensim 333https://radimrehurek.com/gensim/ from the tree of
tieredImageNet classes. We choose a 100 dimensional hyperbolic space for this
model.

The transformer in DeVISE is a two-layer multilayer perceptron (MLP) with 512
hidden neurons. The feature encoder and decoder in PrVISE are both two-layer
MLPs with 512 hidden neurons. The adaptive prior distribution in PrVISE is 300
dimensional, learnt by a two-layer MLP. In GrVISE, a two-layer GCN is used to
predict weights of the visual classifier. And in HrVISE, a two-layer MLP is
modified to implement Mobius multiplication twice.

All the visual semantic embedding models are trained with encoder freezing.
The Adam optimizer [17] with learning rate 1e-4 is used in all the models to
optimize the loss function. All the models are trained for 200 epochs and
mini-batches of size 256.

Figure 3: Illustration a subset of tieredImageNet. The training classes and test classes are nodes from different high-level semantic concepts. Compared with the standard split in Figure 1, this spilt is more difficult for ZSL, while it provides complex semantic relationship and inference. We think it is an ideal way to demonstrate how a visual semantic embedding models works. Models | embedding hit@1 | embedding hit@5 | avg.sim@1 | avg.sim@5 | avg.sim.dis@1 | avg.sim.dis@5
---|---|---|---|---|---|---
LP | 75.18% | 93.26% | 0.8505 | 0.5067 | 21.80 | 69.13
DeVISE | 71.96% | 92.34% | 0.8330 | 0.5089 | 23.72 | 69.84
PrVISE | 54.01% | 74.24% | 0.7353 | 0.5087 | 29.25 | 52.51
Table 1: Comparison of model performance on visual semantic embedding task.
Only seen images in the testing set are used. A linear layer classifier LP
performs better than the DeVISE and the PrVISE models on the flat hit metrics.
The LP also achieve the highest semantic similarities for the wrong classified
data. The DeVISE performs slightly worse than the LP. The VAE-based model
PrVISE struggles to classify images by comparing the KL-divergence between
latent distributions. This indicates a big performance gap exists when the
latent space can not be disentangled.

###  4.3 Metrics

We consider different kinds of measures to answer the questions provided at
the beginning of this section.

To demonstrate the performance of visual semantic embeddings, we compute the
embedding hit@k metrics – the percentage of test images in the seen set for
which the model returns the one true seen label in its top k predictions. For
the ZSL task, we consider the generalized ZSL. Therefore, we compute the ZSL-S
hit@k ZSL-U hit@k metrics. Here S represents test images in the seen set,
while U represents test images in the unseen set.

Inspired by the average hierarchical distance of a mistake [1], which rethinks
whether we should treat all classes other than the “true” label as equally
wrong, we also employ the average similarity of a mistake (avg.sim@k) and the
average similarity distance of a mistake (avg.sim.dis@k) as metrics to
demonstrate whether a visual semantic embedding model remains the properties
of a word embedding space. To compute these two metrics, we firstly compute
the cosine similarity matrix of the GloVe vectors. Then we sort these
similarities and get the distance of each pair of labels. For the data
instances which the model does not return the true labels, we average the
similarities and distances between the predicted labels and the true labels in
their top $k$ predictions. In the ZSL part, we also employ ZSL-S avg.sim@k,
ZSL-U avg.sim@k, ZSL-S avg.sim.dis@k and ZSL-U avg.sim.dis@k. The difference
is we build the similarity matrices with the union set of training labels and
testing labels.

We think the new metrics can help us figure out the effectiveness of semantic
alignment in the ZSL task, as Roads and Love [29] employed the Spearman
correlation between the upper diagonal portion of two similarity matrices in
two conceptual systems.

##  5 Experimental results

###  5.1 Does a visual semantic embedding model benefit from distributional
semantics in a static word embedding space?

To answer this question, we should consider the paradigms purely dealing with
information from word embeddings. In the following, we analyse the performance
of DeVISE and PrVISE. As a comparison, we implement a linear classifier (LP)
on top of the frozen representation encoded by the InfoMin encoder. Although
the linear probe can not be used in the ZSL task, it still gives insights
about how image representations are distributed with equally weighted labels.

The Table 1 shows results to evaluate visual semantic embedding performance.
We highlight that only seen classes in the testing set is used in this part. A
linear layer classifier performs better than the DeVISE and the PrVISE models
on the flat hit metrics. Although there are no distributed semantics leaked to
the LP, it still shows the highest semantic similarities for the wrong
classified data.

DeVISE performs slightly worse than the linear classifier. There is no
evidence to prove that pre-trained word embeddings help neural networks make a
more reasonable decision at the semantic level, as random embeddings still
work well on the classification task [7, 1].

The generative model PrVISE shows the worst performance. Unlike the success on
medium datasets with human designed attributes, the VAE-based model struggles
to classify images by comparing the KL-divergence between latent
distributions. This indicates a big performance gap exists when the latent
space can not be disentangled. However, the PrVISE model “makes better
semantic mistakes”. Focusing on the difference of average similarities on the
top 1 and top 5 predictions, these models do not prefer to choose candidate
labels in the semantic synonyms of true labels. The PrVISE model does decrease
the semantic distance of a mistake on the top 5 predictions. Although, it is
still very hard to answer the question “Is visual similarity correlated to
semantic similarity?” [6], it seems that the PrVISE model exchanges the
embedding accuracy with the semantic accuracy. When we evaluate the
performance of visual semantic embedding with extended labels, this phenomenon
is more clear. DeVISE and PrVISE can deal with a flexible number of classes.
When the number of label classes increases, the embedding performance drops
(see Table 2), but the PrVISE model still “makes better semantic mistakes”.
For unseen classes (Table 3), PrVISE performs worse than DeVISE. Notably, the
performance in this work with contrastive learning pre-training reaches the
same order of magnitude as the traditional setting, even though our data split
is more difficult.

Models | LP | DeVISE | PrVISE
---|---|---|---
ZSL-S hit@1 | 75.18% | 66.10% | 53.97%
ZSL-S hit@5 | 93.26% | 89.72% | 74.16%
ZSL-S avg.sim@1 | 0.8505 | 0.7857 | 0.7352
ZSL-S avg.sim@5 | 0.5067 | 0.4781 | 0.5091
ZSL-S avg.sim.dis@1 | 21.80 | 54.03 | 37.93
ZSL-S avg.sim.dis@5 | 69.13 | 126.59 | 67.89
Table 2: We test the visual semantic embedding models with full class semantics. This task maps seen image features into a space containing seen and unseen labels. Compared with Table 1, DeVISE and PrVISE can deal with a flexible number of classes. When the number of label classes increases, the embedding performance drops. The PrVISE model “makes better semantic mistakes”. Models | LP | DeVISE | PrVISE
---|---|---|---
ZSL-U hit@1 | N/A | 1.590% | 0.481%
ZSL-U hit@5 | N/A | 6.832% | 2.806%
ZSL-U avg.sim@1 | N/A | 0.3173 | 0.2805
ZSL-U avg.sim@5 | N/A | 0.3062 | 0.2760
ZSL-U avg.sim.dis@1 | N/A | 199.11 | 198.30
ZSL-U avg.sim.dis@5 | N/A | 208.98 | 204.18
Table 3: Comparison of the DeVISE and the PrVISE on the tiered-ImageNet ZSL
split. Note that a linear layer can not predicts unseen classes. PrVISE
performs worse than DeVISE. The results show that the performance with
contrastive learning pre-training reaches the same order of magnitude as the
traditional setting. Both DeVISE and PrVISE do not show the advantages,
although the pre-trained features of unseen images are linearly separable.

###  5.2 Do graph-based models learn hierarchical distributed semantics?

We test the performance of GrVISE and HyVISE to demonstrate how a semantic
graph affects the way a model aligns visual semantics. We discuss GrVISE and
HyVISE respectively.

GrVISE is designed to predict the parameters of a linear image classifier with
the help of GCN. We set a two-layer MLP as a comparison. In previous work
using a GCN based model [37], parameters in the last layer of a pre-trained
classifier are considered to be naturally normalized. However, we notice that
the parameters in a pre-trained linear probe on a freezing encoder are not
normalized. Therefore, we normalize the parameters of the linear probe. Note
that the normalized parameters still work well the original classification
task.

Models | GCN | MLP
---|---|---
embedding hit@1 | 29.67% | 36.77%
ZSL-S hit@1 | 29.63% | 36.70%
ZSL-U hit@1 | 0.832% | 0.0%
Table 4: Results for the GrVISE on our ZSL setting. A two-layer MLP is also
tested. The MLP achieves better parameter prediction for seen classes, while
the graph-based model performs better for zero-shot learning. As the same,
they do not take the advantages of a semantic graph, when image features are
linearly separable.

We observe that an MLP performs better on parameter prediction. Therefore, it
achieves better embedding performance for seen classes. While a GCN performs
better to predict unseen labels, this demonstrates that a GCN does learn a bit
of structural information from a hand-crafted semantic graph.

However, the GrVISE does not organize visual semantic information better than
the DeVISE. We argue that the process of learning a parameter prediction task
can gradually acquire semantic information for unseen classes. We train a
linear probe on both seen and unseen classes. Then we train a GCN and an MLP
respectively to predict the parameters of seen classes. Note that the probe
can achieve $79.10\%$ classification accuracy on unseen classes.

In Figure 4, we show that the GrVISE can not precisely learn unseen
parameters. The dashed lines representing the prediction error for unseen
class parameters are observed stopping decreasing at the early epoch. The
WordNet structure does not narrow the semantic gap between seen classes and
unseen classes.

Figure 4: Illustration how a GCN and an MLP predict parameters of a pre-
trained image classifier. Note that the dashed line belong to unseen classes.
This demonstrates that the GCN achieves better parameter prediction on unseen
image embedding.

Finally, we test the HyVISE, which can been seen as the DeVISE in Riemannian
geometry. We get 13.91% embedding hit@1, 13.84% ZSL-S hit@1 and 0.219% ZSL-U
hiy@1. The results indicate that a Poincaré visual semantic embedding model
does not work well on a complicate ZSL data split, when the split is based on
a sub tree of the WordNet.

In summary, we find that the DeVISE still works robustly on a new ZSL class
split, which needs high-level semantic understanding. Compared with a linear
classifier, DeVISE does not show that it has performed semantic inference
using the distributed semantics. The generative model PrVISE performs slightly
worse than the DeVISE, but it shows the advantage of predicting better
mistakes, which are semantically similar to the ground truth labels. The
graph-based models GrVISE and HyVISE lose their advantages shown the standard
splits. The results demonstrate that recent semantic alignment techniques
could overfit to the original flawed problem, and they also indicate we need a
deeper discussion on the ZSL task in the real world.

##  6 Conclusion

In this work, we investigate the problem of zero-shot learning. We introduce
the tieredImagenet split into ZSL to replace the standard data split. That
makes the evaluation of ZSL does not suffer from problems such as structural
flaws, bad image quality. We build a unified framework for ZSL with
contrastive learning as pre-training. The two-step framework does not leak
semantic information at the early stage, which makes us evaluate semantic
inference fairly. With this framework, we test four mainstream methods for
visual semantic alignment. We demonstrate that current ZSL methods rely deeply
on similarity comparison. No evidence shows that distributed visual
information and distributed semantic information can be combined to implement
semantic analogy and semantic inference across modalities. Therefore, these
ZSL methods can not generalize from complicated structural information.

We believe that rethinking the goals and the definition of ZSL tasks is very
much needed. As Roads and Love [29] pointed out in their work, it is unclear
what mechanism makes two-year-old children exhibit an average vocabulary of
200-300 words. Learning a novel object is still the most challenging task.
During this decade, communities prefer to capture distributed knowledge from
large image dataset and text corpus. However, this method has been shown as an
unrobust approach which can only works on hyponymy and hypernymy. Context
missing may be the main reason. Context information is more helpful than a
distributed system. Currently, vision-language pre-training has shown that it
can effectively learn generic representations for downstream tasks. In
particular, Oscar [21] has shown its success on captioning images with novel
objects. Context surrounding representations seem to outperform in capturing
visual semantics [14]. We believe that it is necessary to build a ZSL task on
contextual language models in the future to replace distributed word embedding
models. The new ZSL task will offer new practical insights on the way how
human learn novel objects.

## References

  * [1] Luca Bertinetto, Romain Mueller, Konstantinos Tertikas, Sina Samangooei, and Nicholas A Lord.  Making better mistakes: Leveraging class hierarchies with deep networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12506–12515, 2020.
  * [2] Benjamin Paul Chamberlain, James Clough, and Marc Peter Deisenroth.  Neural embeddings of graphs in hyperbolic space.  arXiv preprint arXiv:1705.10359, 2017.
  * [3] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  arXiv preprint arXiv:2002.05709, 2020.
  * [4] Sumit Chopra, Raia Hadsell, and Yann LeCun.  Learning a similarity metric discriminatively, with application to face verification.  In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), volume 1, pages 539–546. IEEE, 2005.
  * [5] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  ImageNet: A Large-Scale Hierarchical Image Database.  In CVPR09, 2009.
  * [6] Thomas Deselaers and Vittorio Ferrari.  Visual and semantic similarity in imagenet.  In CVPR 2011, pages 1777–1784. IEEE, 2011.
  * [7] Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc’Aurelio Ranzato, and Tomas Mikolov.  Devise: A deep visual-semantic embedding model.  In Advances in neural information processing systems, pages 2121–2129, 2013.
  * [8] Kai Han, Andrea Vedaldi, and Andrew Zisserman.  Learning to discover novel visual categories via deep transfer clustering.  In International Conference on Computer Vision (ICCV), 2019.
  * [9] Zongyan Han, Zhenyong Fu, and Jian Yang.  Learning the redundancy-free features for generalized zero-shot object recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12865–12874, 2020.
  * [10] Zellig S Harris.  Distributional structure.  Word, 10(2-3):146–162, 1954.
  * [11] Tristan Hascoet, Yasuo Ariki, and Tetsuya Takiguchi.  On zero-shot recognition of generic objects.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9553–9561, 2019.
  * [12] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729–9738, 2020.
  * [13] Irina Higgins, Nicolas Sonnerat, Loic Matthey, Arka Pal, Christopher P Burgess, Matko Bosnjak, Murray Shanahan, Matthew Botvinick, Demis Hassabis, and Alexander Lerchner.  Scan: Learning hierarchical compositional visual concepts.  arXiv preprint arXiv:1707.03389, 2017.
  * [14] Gabriel Ilharco, Rowan Zellers, Ali Farhadi, and Hannaneh Hajishirzi.  Probing text models for common ground with visual representations.  arXiv preprint arXiv:2005.00619, 2020.
  * [15] Longlong Jing and Yingli Tian.  Self-supervised visual feature learning with deep neural networks: A survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020\.
  * [16] Michael Kampffmeyer, Yinbo Chen, Xiaodan Liang, Hao Wang, Yujia Zhang, and Eric P Xing.  Rethinking knowledge graph propagation for zero-shot learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11487–11496, 2019.
  * [17] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [18] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [19] Thomas N Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  arXiv preprint arXiv:1609.02907, 2016.
  * [20] Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling.  Learning to detect unseen object classes by between-class attribute transfer.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 951–958. IEEE, 2009.
  * [21] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In European Conference on Computer Vision, pages 121–137. Springer, 2020.
  * [22] Shaoteng Liu, Jingjing Chen, Liangming Pan, Chong-Wah Ngo, Tat-Seng Chua, and Yu-Gang Jiang.  Hyperbolic visual embedding learning for zero-shot recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9273–9281, 2020.
  * [23] Emile Mathieu, Charline Le Lan, Chris J Maddison, Ryota Tomioka, and Yee Whye Teh.  Continuous hierarchical representations with poincaré variational auto-encoders.  In Advances in neural information processing systems, pages 12565–12576, 2019.
  * [24] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  Distributed representations of words and phrases and their compositionality.  In Advances in neural information processing systems, pages 3111–3119, 2013.
  * [25] George A Miller.  Wordnet: a lexical database for english.  Communications of the ACM, 38(11):39–41, 1995.
  * [26] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  arXiv preprint arXiv:1807.03748, 2018.
  * [27] Jeffrey Pennington, Richard Socher, and Christopher D Manning.  Glove: Global vectors for word representation.  In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014.
  * [28] Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard S Zemel.  Meta-learning for semi-supervised few-shot classification.  arXiv preprint arXiv:1803.00676, 2018.
  * [29] Brett D Roads and Bradley C Love.  Learning as the unsupervised alignment of conceptual systems.  Nature Machine Intelligence, 2(1):76–82, 2020.
  * [30] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  International journal of computer vision, 115(3):211–252, 2015\.
  * [31] Edgar Schonfeld, Sayna Ebrahimi, Samarth Sinha, Trevor Darrell, and Zeynep Akata.  Generalized zero-and few-shot learning via aligned variational autoencoders.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8247–8255, 2019.
  * [32] Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng.  Zero-shot learning through cross-modal transfer.  In Advances in neural information processing systems, pages 935–943, 2013.
  * [33] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales.  Learning to compare: Relation network for few-shot learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1199–1208, 2018.
  * [34] Tristan Sylvain, Linda Petrini, and Devon Hjelm.  Locality and compositionality in zero-shot learning.  arXiv preprint arXiv:1912.12179, 2019.
  * [35] Yonglong Tian, Dilip Krishnan, and Phillip Isola.  Contrastive multiview coding.  arXiv preprint arXiv:1906.05849, 2019.
  * [36] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola.  What makes for good views for contrastive learning.  arXiv preprint arXiv:2005.10243, 2020.
  * [37] Xiaolong Wang, Yufei Ye, and Abhinav Gupta.  Zero-shot recognition via semantic embeddings and knowledge graphs.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6857–6866, 2018.
  * [38] Yongqin Xian, Bernt Schiele, and Zeynep Akata.  Zero-shot learning-the good, the bad and the ugly.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4582–4591, 2017.


ignore

# Phonon hydrodynamic regimes in sapphire

Takuya Kawabata  Department of Physics, Gakushuin University, Tokyo 171-8588,
Japan  Kosuke Shimura  Department of Physics, Gakushuin University, Tokyo
171-8588, Japan  Yuto Ishii  Department of Physics, Gakushuin University,
Tokyo 171-8588, Japan  Minatsu Koike  Department of Physics, Gakushuin
University, Tokyo 171-8588, Japan  Kentaro Yoshida  Department of Physics,
Gakushuin University, Tokyo 171-8588, Japan  Shu Yonehara  Department of
Physics, Gakushuin University, Tokyo 171-8588, Japan  Kohei Yokoi  Department
of Physics, Gakushuin University, Tokyo 171-8588, Japan  Alaska Subedi  CPHT,
CNRS, École polytechnique, Institut Polytechnique de Paris, 91120 Palaiseau,
France  Kamran Behnia<EMAIL_ADDRESS>Laboratoire de Physique et
d’Étude des Matériaux (ESPCI—CNRS—Sorbonne Université),
PSL Research University, Paris, 75005, France  Yo Machida
<EMAIL_ADDRESS>Department of Physics, Gakushuin University, Tokyo
171-8588, Japan

###### Abstract

When an ideal insulator is cooled, four regimes of thermal conductivity are
expected to emerge one after another. Two of these, the Ziman and the
Poiseuille, are hydrodynamic regimes in which collision among phonons are
mostly Normal. It has been difficult to observe them, save for a few
insulators with high levels of isotopic and chemical purity. Our thermal
transport measurements, covering four decades of temperatures between 0.1 K
and 900 K, reveal that sapphire displays all four regimes, despite its
isotopic impurity. In the Ziman regime, the thermal conductivity exponentially
increases attaining an amplitude as large as 35,000 W/Km. We show that the
peak thermal conductivity of ultra-pure, simple insulators, including diamond,
silicon and solid helium, is set by a universal scaling depending on isotropic
purity. The thermal conductivity of sapphire is an order of magnitude higher
than what is expected by this scaling. We argue that this may be caused by the
proximity of optical and acoustic phonon modes, as a consequence of the large
number of atoms in the primitive cell.

In non-magnetic insulators, heat is exclusively transported by phonons. Thanks
to their larger group velocity, acoustic modes usually dominate. In 1966,
Guyer and Krumhansl [1] identified four thermal transport regimes. Let us
recall them one by one.

At temperatures exceeding the Debye temperature, the phonon wave-vector $k$ is
comparable to the width of the Brillouin zone $G$. In this regime, called
kinetic, scattering between two phonons can give rise to a third phonon with a
wave-vector larger than $G$. Such an “Umklapp” event [2] leads to loss of
$\hbar G$ momentum and a resistance to the flow of heat. The average number of
phonons increases linearly with $T$ and thermal conductivity $\kappa$
decreases linearly with warming $\propto T^{-1}$ (Fig. 1(a)). The scattering
rate in this limit is bounded by $\tau_{P}^{-1}=\frac{\hbar}{k_{B}T}$ [3, 4].
As the sample is cooled down, the wave-vector of phonons is reduced, following
$k\simeq k_{\rm B}T/\hbar v_{\rm s}$. As a consequence, Umklapp processes
rarefy, and instead momentum-conserving (Normal) collisions become dominant
(Fig. 1(b)). This leads to hydrodynamics, and the two regimes known as Ziman
and Poiseuille. An exponential increase of $\kappa$ in the Ziman hydrodynamic
regime is driven by the decay of the phonon population capable of giving rise
to Umklapp events. In the Poiseuille hydrodynamic regime, Normal collisions
dominate and Umklapp collisions become even less frequent than boundary
collisions, this boosts heat conduction and leads to a $\kappa$ varying faster
than $T^{3}$. Cooling further, phonons travel across the sample without
meeting other phonons. Their wavelength becomes so long that the crystal wall
becomes the exclusive scattering center. In this ballistic (or Casimir [5])
regime, the thermal conductivity follows a $T^{3}$ temperature dependence.

Figure 1: Four regimes in temperature dependence of thermal conductivity.
$l_{\rm U}$ and $l_{\rm N}$ represent the mean-free-path of Umklapp and Normal
scattering, respectively. Hierarchy of $l_{\rm U}$, $l_{\rm N}$, and the
sample dimension $d$ determines the four regimes. Dominant Normal collisions
open the two hydrodynamic regimes in between the ballistic and kinetic
regimes. Figure 2: (a) Top view and (b) side view of crystal structure of
sapphire. A thermal conductivity measurement setup for (c) steady-state method
and (d) 3$\omega$ method. Thermal conductivity $\kappa$ of sapphire and ruby
as a function of temperature. For sapphire, $\kappa$ displays an exponential
increase ($\kappa\propto\exp(A/T)$ with $A=$ 337 K and forms a prominent peak
around 40 K, below which $\kappa$ falls faster than $T^{3}$ as a signature of
phonon Poiseuille flow. Temperature variation of $\kappa$ changes from $1/T$
to the exponential across $T=A$. An Arrhenius plot shown in the panel (f)
clearly demonstrate the presence and absence of exponential behavior in
sapphire and ruby, respectively. (g) A pronounce Poiseuille peak is detected
in temperature dependence of phonon mean-free-path, which increases aiming to
attain the crystal dimension at low temperature as shown in the inset of panel
(g).

These four regimes [1, 6, 7] provide a road-map in the quest for detecting
signatures of phonon hydrodynamics in the two regimes where Normal collision
between phonons dominate: i.e. the Ziman and Poiseuille regimes [6, 7, 8].

The two hydrodynamic regimes are rarely observed in real insulators [9],
presumably because they are severely hindered by the presence of defects, and
chemical or isotopic impurities. Recent studies, however, have found that
exceptional purity is not the necessary condition for detecting a phonon
Poiseuille flow. It has been observed in strontium titanate, a quantum
paraelectric [10], in layered solids such as graphite and black phosphorus
[11, 12], and in semi-metallic antimony [13]

As for the Ziman heat transport regime, previous reports of its observation
have been restricted to a few ultra-pure insulators composed of single-isotope
elements, such as solid helium [14, 15], CsI [9], and NaF [16]. Surprisingly,
the thermal conductivity of naturally available sapphire (Al2O3) displays the
Ziman regime. While aluminum is pure (27Al: 100 %), this is not the case of
oxygen (16O: 99.76 %, 17O: 0.04 %, 18O: 0.21 %). Nevertheless, in 1951, Berman
[17] reported an exponential temperature dependence based on a limited number
of data points (see Fig. S3). The issue has not been reexamined since then.

In this paper, we present a study of heat transport in sapphire, documenting
its putative Ziman regime and Poiseuille regime. Prior to our study, solid
helium has been the only case in which both regimes have been observed. We
find that naturally occurring sapphire stands out among insulators by
displaying both hydrodynamic regimes and a record peak thermal conductivity at
their boundary. The observed conductance exceeds what is expected from a
universal plot of dimensionless thermal conductivity as a function of isotopic
impurities obeyed by other insulators. We argue that the large number of atoms
per primitive cell and/or the proximity of the optical and acoustic phonon
branches are the key ingredients for this robustness against isotopic
disorder.

Figure 3: Temperature dependence of thermal conductivity of (a) diamond [18,
19, 20], (b) Si [21, 22], (c) LiF [23, 24], (d) NaF [23, 16], (e) CsI [9], and
(f) 4He [14] and 3He [15]. The dashed-dotted, solid, and dashed lines
represent the $T^{3}$, exponential, and $1/T$ dependence, respectively. The
exponential behavior is fragile against the random isotopic mixture in
diamond, Si, and LiF, and impurities in NaF as shown in the panels (a)-(d).
$A$ and Debye temperature $\Theta_{\rm D}$ of each material are denoted by
closed and open arrows, respectively.

Sapphire has a layered trigonal structure with the space group $R\bar{3}c$.
Each layer consists of edge-shared AlO6 octahedra arranged in a honeycomb
pattern (Fig. 2(a)), and the layers are stacked such that each octahedron
shares edges with the layers above and below it (Fig. 2(b)).

Let us quantify the isotopic purity of sapphire. Isotopic disorder can be
quantified by a dimensionless parameter [22]

$g=\sum_{i}f_{i}(\Delta M_{i}/M)^{2}.$ (1)

Here, $f_{i}$ is the concentration of the $i$-th isotope, whose mass $M_{i}$
differs from the average mass $M=\sum_{i}f_{i}M_{i}$ by $\Delta
M_{i}=M_{i}-M$. In naturally abundant sapphire, that is in absence of any
isotopic purification, $g\sim 34\times 10^{-6}$. This is to be compared with
that of the natural abundant diamond (74$\times 10^{-6}$) and silicon
(200$\times 10^{-6}$). Thanks to the maturation of crystal growth technique,
highly pure industrial sapphire single crystals with minimal contamination and
defects are nowadays available.

The sapphire samples used in this study were purchased from Kyocera
corporation. In order to acquire the thermal conductivity data in a wide
temperature range, we employed the 3$\omega$ method and the standard steady-
state method. The 3$\omega$ measurements were performed in the temperature
range between 80 K and 900 K by utilizing the N2 cryostat and the electrical
furnace. The measurements, using a one-heater-two-thermometers steady-state
method, were performed in the temperature range between 0.1 K and 300 K in a
commercial measurement system (Quantum Design PPMS) and in a home-made
dilution refrigerator.

Figure 2(e) shows temperature dependence of thermal conductivity $\kappa$ of
sapphire. As seen in the figure, there is a satisfactory match between the
data obtained by the different methods in the different temperature ranges.
The four conductivity regimes are all visible. In the highest temperature
regime, with decreasing temperature, $\kappa$ grows following $1/T$. Upon
further cooling, the temperature variation evolves into the exponential
behavior $\kappa\propto\exp(A/T)$, signaling an entry to the Ziman regime. The
result agrees with what was found by Berman [17] and with the first principle
calculations [25] (see Fig. S2(b)). The exponential increase persists down to
40 K, reaching a magnitude as large as 35,000 W/Km. Such a magnitude has only
be attained in isotopically pure diamond [20] and Si [22] (see Figs. 3(a) and
(b)). Below the peak, $\kappa$ falls rapidly following $\propto T^{4}$. Such a
faster than $T^{3}$ decrease of $\kappa$ is a hallmark of the phonon
Poiseuille flow [26, 27, 28, 29, 10, 11, 12]. Further cooling the sample below
1 K eventually leads to the ballistic regime and a $T^{3}$ behavior.

Figure 4: (a) Comparison of $A$ and $\Theta_{\rm D}$ in insulators showing the
exponential behavior in thermal conductivity. Their ratio $A/\Theta_{\rm D}$
remains between 1/5 and 1/2. (b) Normalized $\kappa$ by those values at $T=A$
as a function of $T/A$. Figure 5: (a) A plot of the maximum thermal
conductivity $\kappa_{\rm max}$ divided by the sample dimension $d$ and the
Debye temperature $\Theta_{\rm D}$ against the isotope disorder parameter $g$.
In each system, $\kappa_{\rm max}/d\Theta_{\rm D}$ for the isotopically pure
and the natural abundant samples are presented by solid and open symbols,
respectively. For LiF, the data of several variations of isotope compositions
are shown by open triangles. (b) A plot of the dimensionless thermal
conductivity $(\kappa_{\rm max}/d\Theta_{\rm D})/\\{(k_{\rm
B}^{2}/\hbar)G^{2}\\}$ vs. $g$.

The peculiar features of $\kappa(T)$ can be seen by an examination of the
phonon mean-free-path $l_{\rm ph}$ extracted from the thermal conductivity,
the specific heat $C$ and the phonon velocity $v_{\rm s}$ ($l_{\rm
ph}=\frac{3\kappa}{v_{\rm s}C}$). Its evolution with temperature is shown in
Fig. 2(g). An exponential increase of $l_{\rm ph}$ ends up attaining a
magnitude close to the sample dimensions (the horizontal dashed line in the
inset of Fig. 2(g)). At low temperature, $l_{\rm ph}$ displays a local peak
(around 30 K) and a local minimum (around 10 K) as commonly observed in
systems where hydrodynamic phonon transport becomes dominant [28, 10, 11, 12,
13]. In the miliKelvin temperature, $l_{\rm ph}$ saturates to a length
comparable to sample dimensions as, expected in the ballistic regime (see the
inset of Fig. 2(g)).

Let us put under scrutiny the exponential behavior $\kappa\propto\exp(A/T)$ in
the Ziman regime. The Arrhenius plot shown in Fig. 2(f) yields $A$ = 337 K.
The Debye temperature is $\Theta_{\rm D}\sim$ 1047 K, and therefore :
$A\sim\Theta_{\rm D}/3$, as predicted by Peierls [2]. One can see from Fig.
2(e) that $A$ and the crossover temperature from the kinetic to Ziman regime
are comparable to each other.

Hydrodynamic regimes in sapphire are destroyed in presence of a significant
population of point-like defects. This is shown by comparison with a ruby
sample purchased from Orbray corporation, in which about 3 % of Al atoms are
substituted by Cr atoms. As seen from Figs. 2(e-g), the exponential increase,
the faster than $T^{3}$ decrease, and the non-monotonic mean free path are all
absent in the ruby sample. As a result, the maximum $\kappa$ is damped by a
factor of 50. The phonon mean-free-path $l_{\rm ph}$ smoothly increases all
the way down to 3 K, approaching a value close to the sample dimension
(represented by the solid horizontal line in the inset of Figs. 2(g)) without
showing any trace of the phonon hydrodynamics. In other words, a defect
concentration of 3 % does not affect the kinetic and the ballistic regimes,
but wipes out the Ziman and the Poiseuille regimes.

Let us compare sapphire with other high-purity insulators. Although, this has
remained unnoticed, isotopically enriched diamond [20], Si [22], and LiF [24]
display a similar exponential behavior, asymptotically evolving to a $1/T$
dependence at a temperature close to $A$. This is shown in Figs. 3(a)-(c)).
This is also true of NaF [23, 16] and CsI [9], as seen in Fig. 3(d) and (e).
In all these cases, $\kappa$ deviates from the $1/T$ dependence (dashed line)
around $T\sim A$, extracted from an Arrhenius fit, and asymptotically follows
the exponential temperature dependence. Finally, as seen in Fig. 3(f), this is
also the case of the two helium isotopes. 4He [14] and 3He [15], both display
a similar behavior. However, in both cases, the kinetic regime has not been
attained.

Fig. 4 reveals universal feature common to high-purity insulators. The
amplitude of $A$ as a function of $\Theta_{\rm D}$ in different systems is
shown in Fig. 4(a). One can see that in all of them, it lies between
$\Theta_{\rm D}/5$ and $\Theta_{\rm D}/2$.

The upper bound to $A$ can be justified considering Umklapp phonon-phonon
collisions. They are only possible if the wave-vector of each colliding
phonon, $k$, exceeds $G/4$, where $G=2\pi/a$ ($a$: the lattice constant) is a
reciprocal lattice vector. Since $k=k_{\rm B}T/\hbar v_{\rm s}$, the energy of
such a phonon is given by $k_{\rm B}T_{\rm U}=\pi\hbar v_{\rm s}/2a$. Assuming
linear dispersion ($\omega_{\rm D}=k_{\rm B}\Theta_{\rm D}\sim v_{\rm
s}\hbar(G/2)$), one finds $T_{\rm U}\sim\Theta_{\rm D}/2$. The average number
of such a phonon decreases exponentially with temperature, $\langle
N\rangle\sim\exp(-T_{\rm U}/T)\sim\exp(-\Theta_{\rm D}/2T)$, so that the
thermal conductivity is $\kappa\propto\exp(\Theta_{\rm D}/2T)$. This leads to
$A=\Theta_{\rm D}/2$ in good agreement with what is seen in Fig. 4(a).

It is instructive to consider exceptions to this correlation. For example, in
SrTiO3, $A=$ 20 K ($\sim\Theta_{\rm D}/25$) [10]. This unusually small value
points to a role played by low-energy soft phonons of this quantum
paraelectric and their coupling with acoustic modes [30]. The case of bismuth,
where $A=14.3~{}K$ ($\sim\Theta_{\rm D}/8$) [31], while less striking, points
towards a similar direction.

A more striking similarity between ultra-pure insulators is shown in Fig.
4(b). It displays $\kappa$, normalized by its value at $T=A$ as a function of
$T/A$. All data fall into a universal curve, which is exponential for $T/A\leq
1$ and evolves towards $1/T$, when $T/A\geq 1$. This figure indicates that in
a perfect insulator with no disorder (and no soft phonons) knowing $A$ (which
is a fraction of $\Theta_{\rm D}$) is sufficient to know thermal conductivity
at any temperature above its peak value.

As seen in Figs. 3(a)-(d), in all these high-purity insulators, the random
mixture of isotopes or impurities pulls down $\kappa$. Not only the
exponential variation but also the prominent peak in $\kappa$ are considerably
suppressed in naturally abundant diamond [18], Si [18], and LiF [24] as well
as in impure NaF [16].

The case of sapphire is outstanding, because it exhibits an exponential
behavior in absence of isotopic purity. As shown in Table LABEL:table, the
dimensionless parameter quantifying isotopic disorder, $g$, is exceptionally
high for sapphire, compared to other insulators displaying an exponential
rise. Moreover, the specific heat of our samples display a Schottky anomaly
indicating the presence of magnetic impurities (Fig. S5). Despite this level
of isotopic and chemical impurities in the range of 30 ppm (see Table
LABEL:table), our sapphire samples behave like ultra-pure insulators.

Table 1: Comparison of crystal structure, number of atoms in the unit cell $N$, isotope composition, dimensionless parameter of isotopic disorder $g$, Debye temperature $\Theta_{\rm D}$, and $A$ for materials exhibiting the exponential rise in thermal conductivity. Isotope composition of isotopically purified diamond, Si, LiF, and 3He is from Refs. [20, 22, 24, 15]. |  |  |  | Natural abundant sample | Isotopically pure sample |  |
---|---|---|---|---|---|---|---
| Crystal structure | $N$ | Isotope | Composition (%) |  $g$ ($\times$10-6) | Composition (%) |  $g$ ($\times$10-6) |  $\Theta_{\rm D}$ (K) |  $A$ (K)
Sapphire | Trigonal | 30 |  27Al | 100 | 34 | - | - | 1047 | 337
16O | 99.76 | -
17O | 0.04 | -
18O | 0.21 | -
Diamond | Cubic (fcc) | 8 |  12C | 98.9 | 74 | 99.9 | 7 | 2220 | 425
13C | 1.1 | 0.1
Si | Cubic (fcc) | 8 |  28Si | 92.2 | 200 | 99.995 | 0.08 | 645 | 331
29Si | 4.7 | 0.0046
30Si | 3.1 | 0.0004
LiF | Cubic (Rock salt) | 8 |  6Li | 7.6 | 1460 | 0.01 | 2 | 735 | 184
7Li | 92.4 | 99.99
19F | 100 | 100
NaF | Cubic (Rock salt) | 8 |  23Na | 100 | 0 | - | - | 488 | 128
19F | 100 | -
CsI | Cubic (bcc) | 2 |  133Cs | 100 | 0 | - | - | 128 | 34
127I | 100 | -
4He | Hexagonal | 2 |  3He | 0.0001 | 0.06 | - | - | 36.7 | 7.3
4He | 99.9999 | -
3He | Cubic (bcc) | 2 |  3He | - | - | 99.9998 | 0.2 | 23 | 4.5
4He | - | 0.0002

The exceptional case of sapphire is emphasized by considering the amplitude of
the peak thermal conductivity $\kappa_{\rm max}$ as a function of $g$. The
magnitude of $\kappa_{\rm max}$ depends on the sample dimension, $d$, and the
Debye temperature, $\Theta_{\rm D}$. Increasing them would enhance
$\kappa_{\rm max}$, by allowing the exponential increase to persist down to
lower temperatures. Fig. 5(a) is a plot of $\kappa_{\rm max}$ normalized by a
product of $d\times\Theta_{\rm D}$ as a function of $g$. The isotopic
purification pushes up $\kappa_{\rm max}/d\Theta_{\rm D}$ towards higher
values (from the open to closed symbol in each material) following
$\kappa_{\rm max}/d\Theta_{\rm D}\propto g^{-0.3}$ as guided by the solid
arrows. As seen in the figure, $\kappa_{\rm max}/d\Theta_{\rm D}$ in sapphire
stands out. Despite a $g$ 500 times larger, sapphire surpasses its close
rival, 4He.

Let us now consider a dimensionless thermal conductivity, the ratio of
$\kappa_{\rm max}/d\Theta_{\rm D}$ to $(k_{\rm B}^{2}/\hbar)G^{2}$, where $G$
is the width of Brillouin zone. As seen in Fig. 5(b), the available data
collapse onto a single line, implying that the effect of isotope scattering on
the phonon heat transport across a variety of insulating solids is properly
accounted by $g$. One can see again that sapphire does not participate in this
scaling, indicating a puzzling robustness of its high thermal conductivity
against the isotopes.

Compared to other insulators discussed here, sapphire has many more atoms,
$N$, in its primitive cell. As listed in Table 1, $N=30$ for sapphire, $N=8$
for diamond, Si, and LiF with fcc or rock salt structure, and $N=2$ for
hexagonal solid 4H. A large $N$ attenuates mass perturbation by isotopic
impurities. Moreover, it leads to an increase in the number of optical phonon
branches. As shown in Fig. S6(a), in sapphire the gap between the optical and
the acoustic branches is small. This enhances the phase space for scattering
between the acoustic and optical phonons and reduces a chance for the acoustic
phonons to be scattered by the isotopes. Notably, the fact that the energy
range ($\sim$10 THz) where the high frequency acoustic phonons can meet the
optical phonons is close to the upper bound of Ziman regime, that is $A$
($\sim$7 THz), suggests that both phonon modes indeed participate the Normal
processes. Because of the proximity of optical and acoustic phonon energies,
the thermal conductivity of sapphire, in contrast to other ‘simple’
insulators, is not set by $g$.

We note that this scenario is compatible with theoretical calculations
suggesting that isotopic purity does not necessarily lead to higher thermal
conductivity [32]. It has been demonstrated that isotopic purification of
natural LiH by replacing 7Li with 6Li leads to a higher frequency of acoustic
modes and a smaller phonon gap. This allows for increased coupling of the
heat-carrying acoustic phonons with the optical phonons and reduction of their
lifetimes. As a result, thermal conductivity of the isotopically pure 6LiH is
lower than that of the natural LiH. The possible correlation between isotope
mass variations and Normal collision rate emerges as a potential theme for
future theoretical studies.

In this context, let us also recall the case of another highly conducting
oxide, In2O3, which has forty atoms per primitive cell and numerous optical
phonons and its thermal conductivity can surpass that of naturally abundant
silicon [33].

To sum up, naturally occurring sapphire displays the two phonon hydrodynamic
regimes and a thermal conductivity much higher than what is expected, given
its isotopic purity. We identify the large number of atoms per the primitive
cell as a possible source of frequent momentum exchange among phonons
including optical modes and attenuation of phonon-isotope scattering.

We acknowledge Kentaro Kitagawa, Laurent Jalabert, and Masahiro Nomura for
technical supports. This work was supported by Grants-in-Aid for Scientific
Research (KAKENHI Grants No. JP19H01840 and No. JP23H00092).

##  I APPENDIX

###  I.1 Thermal conductivity measurements

Thermal conductivity data below 300 K was obtained by employing the steady-
state method. Since the radiation of heat from the sample surface becomes a
serious problem at high temperatures, we employed the 3$\omega$ method [34]
above 80 K. In the 3$\omega$ experiment, the sample is heated locally so that
compared to steady-state experiments, errors due to radiation are reduced to a
negligibly low level. The single crystals with a size of 0.5 $\times$ 0.6
$\times$ 10 mm3 and 5 $\times$ 5 $\times$ 3 mm3 were used for the steady-state
and 3$\omega$ method, respectively.

####  I.1.1 Steady-state method

The steady-state experiments by the standard one-heater-two-thermometers
method were performed in the commercial measurement system (Quantum Design
PPMS) between 2 and 300 K and in the home-made dilution refrigerator between
0.1 and 2.5 K. The measurements were carried out in a high vacuum level better
than 10-4 Pa. The thermal gradient in the sample was produced through a chip
resistor alimented by a current source (Keithley 6220). The DC voltage on the
heater was measured by the digital multimeter (Keithley 2000). The
thermometers, heater, and cold finger were connected to samples directly or by
gold wires of 25 $\mu$m diameter. All contacts on the sample were made using
silver paste Dupont 4922N. The heat current was injected along the largest
length of the sample (10 mm) which is perpendicular to the $c$-axis. The
temperature difference generated in the sample was determined by measuring the
local temperature with two thermometers: E-type thermocouple (5 K-300 K) and
Cernox resistors (2 K-15 K) in the PPMS, and RuO2 resistors (0.1 K-2.5 K) in
the dilution refrigerator. The DC voltage on the thermocouple was measured by
the DC-nanovoltmeter (Keithley 2182A). The resistance of Cernox and RuO2
thermometers were measured by the resistance bridges (Cryocon 24C) and the
MMR3 resistance bridge (Cryoconcept), respectively.

The heat loss along manganin wires connected to the thermometers and heater is
many orders of magnitude lower than the thermal current along the sample.
Because of the large thermal conductivity $\kappa\sim$ 35,000 W/Km around the
peak and the tiny temperature difference in the sample, which is as small as a
few millikelvin, the scattering of the data is large. Thus, the measurements
were repeated several times and the data is averaged. Error bars in the main
figures represent one standard deviation.

####  I.1.2 3$\omega$ method

The 3$\omega$ experiments were performed in the $N_{2}$ cryostat between 80
and 350 K and in the electric furnace between 300 and 900 K. The narrow gold
line serving as both the heater and the thermometer had a width of 100 $\mu$m,
a thickness of 70 nm, and a length of 3 mm was directly sputtered on the
surface of sapphire sample. The resistance of this line is $\sim$ 5 $\Omega$
at 300 K. The line structure shown in Fig. 2(d) and Fig. S1 (b) were made by
using a thin stainless mask. A sinusoidal current with a typical amplitude of
20 mA generated by a current source (Keithley 6221) is passed through the line
with angular frequency $\omega$, which generates power varying as 2$\omega$.
This power creates thermal oscillations varying as 2$\omega$. Due to the
linear temperature dependence of the metallic heater, the 2$\omega$ thermal
oscillation causes the resistance of the line to vary as 2$\omega$. This
resistance oscillation mixes with the input current and generates a component
of the measured voltage which varies as 3$\omega$.

Applying the 3$\omega$ method to bulk geometry, the in-phase 3$\omega$ voltage
oscillation $V_{\rm 3\omega}$ of the line is expected to be linear in
logarithmic heating frequency 2$f$ ($f=\omega/2\pi$) as long as the thermal
penetration depth is large compared to the heater half width $b\sim$ 50 $\mu$m
and at least five times smaller than the sample thickness $t$ = 3 mm. The gold
line was connected in series with a programmable resistor (IET PRS-330), which
is adjusted to have the same resistance with the line. Home-made unity gain
differential amplifiers were employed to acquire the voltage signal across the
programmable resistor and the sample circuit. $V_{\rm 3\omega}$ was obtained
by subtracting the first harmonic voltage across the programmable resistor
($V_{\rm 1\omega}$) from that across the line ($V_{\rm 1\omega+\rm 3\omega}$).
By measuring $V_{\rm 3\omega}$ as a function of frequency with a lock-in
amplifier (Signal Recovery 7265), the thermal conductivity $\kappa$ of the
material can be extracted from the slope of $V_{\rm 3\omega}$ vs. log2$f$. A
typical frequency scan at different temperatures is shown in Fig. S1(a).

For the 3$\omega$ measurements above 300 K, the sample was mounted on the
alumina plate (Fig. S1(b)) by silver paste and set in a custom-made probe with
the K-type thermocouple for the measurement of sample temperature. Inside of
the probe was kept under a vacuum of 10-4 Pa. The probe was inserted in the
electrical furnace which can control temperature in a range between 100 and
1280 degrees Celsius (Fig. S1(c)). The measurements in the $N_{2}$ cryostat
was carried out with a custom-made probe under a vacuum of 10-4 Pa. The sample
was mounted on the copper block by GE varnish and set in the probe. The probe
was dipped in liquid nitrogen and the sample temperature was regulated by the
temperature controller (Cryocon 24C) by monitoring the Pt thermometer attached
close to the sample.

Figure S1: (a) Heating frequency dependence of the in-phase 3$\omega$ voltage
oscillation $V_{\rm 3\omega}$ collected at different temperatures. Photos of
the setup for the 3$\omega$ measurements at high temperature (b) and the
custom-made probe inserted in the electric furnace (c).

###  I.2 Reproducibility

We have measured thermal conductivity of the different sapphire sample
purchased from the different company (Orbray corporation) to check the
reproducibility. As can be seen from Figs. S2(a) and (b), $\kappa$ from the
two different sources well coincides each other. Our results also reproduce
the first-principle calculations [25] in the Ziman and kinetic regimes as
shown in Fig. S2(b). Since the two samples have the comparable sample
dimension ($d\sim$ 0.62 mm and 0.67 mm for the Kyocera and Orbray samples,
respectively), the phonon mean-fee-path $l_{\rm ph}$ saturates to almost the
same value at low temperature (Fig. S2(c)). On warming, $l_{\rm ph}$ for the
both samples causes the minimum around 10 K, then peaks around 30 K. At the
maximum, $l_{\rm ph}$ exceeds $d$ as expected for the Poiseuille flow of
phonons.

Figure S2: (a) A comparison of thermal conductivity of sapphire samples from
two different sources. One purchased from Kyocera corporation and the other
from Orbray corporation. (b) Our results well reproduce the first principle
calculation [25] in the Ziman and kinetic regimes. (c) Temperature variations
of phonon mean-free-path of the two samples.

###  I.3 Comparison to the literature data

In Figure S3(a), thermal conductivity of the Kyocera sample is shown together
with the previously reported data by Berman [17, 35, 36] and Pohl [37]. Our
data reproduces the exponential behavior of Berman’s data in the Ziman regime
and the $T^{3}$ dependence of Pohl’s data in the ballistic regime ($T<$ 1 K).
The larger magnitude of Pohl’s data is due to the larger sample dimension $d=$
5 mm. Figure S3(b) depicts temperature variation of $l_{\rm ph}$ extracted
from $\kappa$ of the previous reports. One can see that $l_{\rm ph}$ shows a
tendency to saturate to each $d$.

In the literature data, the Poiseuille regime is not detectable probably due
to the low quality of samples fabricated in many decades ago, implying that
the Poiseuille flow is more fragile against impurities and defects compared to
the Ziman hydrodynamics.

Figure S3: A comparison of (a) thermal conductivity and (b) phonon mean-fee-
path data with the previous reports by Berman [17, 35, 36] and Pohl [37].

Figure S4 shows temperature dependence of thermal diffusivity $D$, which is
the ratio of thermal conductivity to specific heat $C$, $D=\kappa/C$. Our
results agree with the previous report [38]. Above 600 K, $D$ follows $1/T$
dependence and respect the Plankian bound $D\geq sv_{\rm s}^{2}\tau_{\rm p}$
($\tau_{\rm p}=\hbar/k_{\rm B}T$) [3, 4] with $v_{\rm s}=11.23$ km/s. Here,
$s>$ 1 is a constant specific for different families of materials. We employed
$s=2.04$ as suggested by Ref. [4].

Figure S4: A comparison of temperature dependence of thermal diffusivity $D$
with the literature data [38]. The solid line represents $sv_{\rm s}^{2}\tau$
with $s$ = 2.04 [4] and $v_{\rm s}$ = 11.23 km.

###  I.4 Specific heat

Specific heat $C$ of single-crystalline sapphire and ruby was measured between
2 K and 300 K using a relaxation method of the heat capacity option in a
Quantum Design, PPMS instrument. The sapphire samples purchased from Kyocera
and Orbray corporation and the ruby sample from Orbray corporation were used.
As shown in Fig. S5(a), our results of sapphire agree well with the previous
report [39] and the first-principle calculation except for upturns at low
temperature. The upturn causes a deviation from the $T^{3}$ Debye specific
heat as shown in Fig. S5(b) where $C$ divided by $T^{3}$ is plotted as a
function of temperature. The similar upturn was reported in Ref. [40] where
the excess component is attributed to the Schottky specific heat due to 1308
ppm vanadium contamination. We infer that our specific heat data is also
affected by the magnetic impurities which are chromium for the ruby sample,
but unknown for the sapphire samples.

For the estimation of phonon mean-free-path, we neglect the upturn and assume
a constant $C/T^{3}$ down to low temperature, which is $9.15\times 10^{-6}$
J/molK4, $9.33\times 10^{-6}$ J/molK4, and $8.86\times 10^{-6}$ J/molK4 for
the sapphire samples from Kyocera and Orbray, and the ruby sample,
respectively. Above 300 K where the experimental data of specific heat is not
available, we use the calculated results (solid line in Fig. S5(a)) to
evaluate phonon mean-free-path.

Figure S5: A comparison of temperature dependence of specific heat $C$ (a) and
$C/T^{3}$ (b) of the sapphire and ruby samples together with the literature
data [39, 40]. The theoretical calculation is also shown in panel (a). As
shown in the panel (b) and its inset, the excess specific heat as indicated by
an upward deviation from the $T^{3}$ dependence is present in all our samples.

###  I.5 Computational Method

The phonon contribution to the specific heat was calculated within the
quasiharmonic approximation using the shengbte code [41]. This requires the
calculation of second-order interatomic force constants (IFC), which was
obtained using the supercell-based frozen phonon approach as implemented in
the phonopy pacakge [42]. We used $4\times 4\times 4$ supercells to generate
the necessary displaced structures, and the atomic forces in these structures
were obtained from first principles using the vasp code within the local
density approximation (LDA) [43]. These calculations were performed using a
plane-wave cutoff of 520 eV and $k$-point mesh of $3\times 3\times 3$. The
IFCs were obtained using the fully-relaxed structure minimizing both the
lattice stresses and atomic forces. The phonon density of states used in the
calculation of the specific heat was obtained by interpolating the IFCs on an
$108\times 108\times 108$ $q$-point mesh.

###  I.6 Comparison to the other insulators

In Table 1, we compare sapphire with the other insulators showing the Ziman
heat transport. There are two things to be emphasized. (i) Dimensionless
parameter of isotopic disorder $g$ for sapphire is exceptionally high. (ii)
Sapphire contains a large number of atoms in the unit cell. Related to the
latter feature, there are a lot of the optical phonon modes in sapphire as
shown in Fig. S6(a). This is contrasting with fewer optical modes in diamond,
Si, LiF, and 4He (Figs. S6(b)-(e)) in which a number of atoms in the unit cell
is much smaller (see Table 1). The large number of optical modes combined with
a small gap between the optical and acoustic branches allows coupling between
of heat-carrying acoustic phonons with the optical phonons. This weakens the
phonon-isotope scattering, but facilitates the intrinsic phonon-phonon
scattering. The fact that the Ziman regime (shaded region in Figs. S6(a)-(e))
extends up to the energy range where the acoustic phonons can meet the optical
phonons indicates that both phonon modes participate the Ziman transport in
sapphire. In the other insulators, the only acoustic phonons are active in the
corresponding regime.

Figure S6: A comparison of phonon dispersion of (a) sapphire [44], (b) diamond
[45], (c) Si [46], (d) LiF [47], and (e) hcp 4He [48]. Shaded region
represents the energy-range corresponding to the Ziman regime which is between
the conductivity maximum temperature and $A$.

## References

  * Guyer and Krumhansl [1966] R. A. Guyer and J. A. Krumhansl, Thermal conductivity, second sound, and phonon hydrodynamic phenomena in nonmetallic crystals, Phys. Rev. 148, 778 (1966).
  * Peierls [1955] R. E. Peierls,  _Quantum Theory of Solids_ , Oxford University Press (Clarendon Press, 1955).
  * Behnia and Kapitulnik [2019] K. Behnia and A. Kapitulnik, A lower bound to the thermal diffusivity of insulators, J. Phys.: Condens. Matter 31, 405702 (2019).
  * Mousatov and Hartnoll [2020] C. H. Mousatov and S. A. Hartnoll, On the planckian bound for heat diffusion in insulators, Nat. Phys. 16, 579 (2020).
  * Casimir [1938] H. Casimir, Note on the conduction of heat in crystals, Physica 5, 495 (1938).
  * Beck  _et al._ [1974] H. Beck, P. F. Meier, and A. Thellung, Phonon hydrodynamics in solids, physica status solidi (a) 24, 11 (1974).
  * Cepellotti  _et al._ [2015] A. Cepellotti, G. Fugallo, L. Paulatto, M. Lazzeri, F. Mauri, and N. Marzari, Phonon hydrodynamics in two-dimensional materials, Nat. Commun. 6, 6400 (2015).
  * Machida  _et al._ [2024] Y. Machida, V. Martelli, A. Jaoui, B. Fauqué, and K. Behnia, Phonon hydrodynamics in bulk insulators and semimetals, Low Temp. Phys. 50, 574 (2024).
  * Cahill and Pohl [1988] D. G. Cahill and R. O. Pohl, Lattice vibrations and heat transport in crystals and glasses, Ann. Rev. Phys. Chem. 39, 93 (1988).
  * Martelli  _et al._ [2018] V. Martelli, J. L. Jiménez, M. Continentino, E. Baggio-Saitovitch, and K. Behnia, Thermal transport and phonon hydrodynamics in strontium titanate, Phys. Rev. Lett. 120, 125901 (2018).
  * Machida  _et al._ [2018] Y. Machida, A. Subedi, K. Akiba, A. Miyake, M. Tokunaga, Y. Akahama, K. Izawa, and K. Behnia, Observation of poiseuille flow of phonons in black phosphorus, Sci. Adv. 4, eaat3374 (2018).
  * Machida  _et al._ [2020] Y. Machida, N. Matsumoto, T. Isono, and K. Behnia, Phonon hydrodynamics and ultrahigh–roomtemperature thermal conductivity in thin graphite, Science 367, 309 (2020).
  * Jaoui  _et al._ [2022] A. Jaoui, A. Gourgout, G. Seyfarth, A. Subedi, T. Lorenz, B. Fauqué, and K. Behnia, Formation of an electron-phonon bifluid in bulk antimony, Phys. Rev. X 12, 031023 (2022).
  * Lawson and Fairbank [1973] D. T. Lawson and H. A. Fairbank, Thermal conductivity and isotopic impurities in single crystals of helium, J. Low Temp. Phys. 11, 363 (1973).
  * Thomlinson [1972] W. C. Thomlinson, Umklapp region thermal conductivity of body-centered-cubic 3He, J. Low Temp. Phys. 9, 167 (1972).
  * Jackson and Walker [1971] H. E. Jackson and C. T. Walker, Thermal conductivity, second sound, and phonon-phonon interactions in NaF, Phys. Rev. B 3, 1428 (1971).
  * Berman [1951] R. Berman, The thermal conductivities of some dielectric solids at low temperatures (Experimental), Proc. Roy. Soc. (London) A208, 90 (1951).
  * Berman  _et al._ [1975] R. Berman, P. R. W. Hudson, and M. Martinez, Nitrogen in diamond: evidence from thermal conductivity, J. Phys. C 8, L430 (1975).
  * Olson  _et al._ [1993] J. R. Olson, R. O. Pohl, J. W. Vandersande, A. Zoltan, T. R. Anthony, and W. F. Banholzer, Thermal conductivity of diamond between 170 and 1200 K and the isotope effect, Phys. Rev. B 47, 14850 (1993).
  * Wei  _et al._ [1993] L. Wei, P. K. Kuo, R. L. Thomas, T. R. Anthony, and W. F. Banholzer, Thermal conductivity of isotopically modified single crystal diamond, Phys. Rev. Lett. 70, 3764 (1993).
  * Glassbrenner and Slack [1964] C. J. Glassbrenner and G. A. Slack, Thermal conductivity of silicon and germanium from 3 K to the melting point, Phys. Rev. 134, A1058 (1964).
  * Inyushkin  _et al._ [2018] A. V. Inyushkin, A. N. Taldenkov, J. W. Ager, E. E. Haller, H. Riemann, N. V. Abrosimov, H.-J. Pohl, and P. Becker, Ultrahigh thermal conductivity of isotopically enriched silicon, J. Appl. Phys. 123, 095113 (2018).
  * Slack [1973] G. A. Slack, Nonmetallic crystals with high thermal conductivity, J. Phys. Chem. Solids 34, 321 (1973).
  * Thacher [1967] P. D. Thacher, Effect of boundaries and isotopes on the thermal conductivity of LiF, Phys. Rev. 156, 975 (1967).
  * Subedi [2021] A. Subedi, First principles study of thermal conductivity of In2O3 in relation to Al2O3, Ga2O3, and KTaO3 (2021), arXiv:2101.02664 .
  * Mezhov-Deglin [1965] L. P. Mezhov-Deglin, Measurement of the thermal conductivity of crystalline 4He, Zh. Eksp. Teor. Fiz. 49, 66 (1965).
  * Thomlinson [1969] W. C. Thomlinson, Evidence for anomalous phonon excitations in solid 3He, Phys. Rev. Lett. 23, 1330 (1969).
  * Kopylov and Mezhov-Deglin [1973] V. Kopylov and L. P. Mezhov-Deglin, Study of kinetic coefficients of bismuth at helium temperatures, Zh. Eksp. Teor. Fiz. 65, 720 (1973).
  * Zholonko [2006] N. N. Zholonko, Poiseuille flow of phonons in solid hydrogen, Phys. Solid State 48, 1678 (2006).
  * Fauqué  _et al._ [2022] B. Fauqué, P. Bourges, A. Subedi, K. Behnia, B. Baptiste, B. Roessli, T. Fennell, S. Raymond, and P. Steffens, Mesoscopic fluctuating domains in strontium titanate, Phys. Rev. B 106, L140301 (2022).
  * Gourgout  _et al._ [2022] A. Gourgout, A. Marguerite, B. Fauqu$\acute{\rm e}$, and K. Behnia, Electronic thermal resistivity and quasi-particle collision cross-section in semi-metals (2022), arXiv:2405.16984 .
  * Lindsay [2016] L. Lindsay, Isotope scattering and phonon thermal conductivity in light atom compounds: LiH and LiF, Phys. Rev. B 94, 174304 (2016).
  * Xu  _et al._ [2021] L. Xu, B. Fauqué, Z. Zhu, Z. Galazka, K. Irmscher, and K. Behnia, Thermal conductivity of bulk ${\mathrm{In}}_{2}{\mathrm{O}}_{3}$ single crystals, Phys. Rev. Mater. 5, 014603 (2021).
  * Cahill [1990] D. G. Cahill, Thermal conducitivity measurement from 30 to 750 K: the 3$\omega$ method, Rev. Sci. Instrum. 61, 802 (1990).
  * Berman  _et al._ [1955] R. Berman, E. L. Foster, and J. M. Ziman, Thermal conduction in artificial sapphire crystals at low temperatures I. Nearly perfect crystals, Proc. Roy. Soc. (London) A231, 130 (1955).
  * Berman  _et al._ [1960] R. Berman, E. L. Foster, B. Schneidmesser, and S. M. A. Tirmizi, Effects of irradiation on the thermal conductivity of synthetic sapphire, J. Appl. Phys. 31, 2156 (1960).
  * Pohl and Stritzker [1982] R. O. Pohl and B. Stritzker, Phonon scattering at crystal surfaces, Phys. Rev. B 25, 3608 (1982).
  * Hofmeister [2014] A. M. Hofmeister, Thermal diffusivity and thermal conductivity of single‑crystal MgO and Al2O3 and related compounds as a function of temperature, Phys. Chem. Minerals 41, 361 (2014).
  * Fugate and Swenson [1969] R. Q. Fugate and C. A. Swenson, Specific heat of $\alpha$ Al2O3 from 2 to 25 K, J. Appl. Phys. 40, 3034 (1969).
  * [40] R. Lagnier, M. Villedieu, and E. Bonjour, Mesure de la chaleur specifique a basse temperature de monocristaux d’alumine pure et dopee d’ions vanadium, Meeting on calorimetry and thermal analysis; Grenoble, France; 22 May 1975.
  * Li  _et al._ [2014] W. Li, J. Carrete, N. A. Katcho, and N. Mingo, ShengBTE: a solver of the Boltzmann transport equation for phonons, Comp. Phys. Commun. 185, 1747 (2014).
  * Togo  _et al._ [2023] A. Togo, L. Chaput, T. Tadano, and I. Tanaka, Implementation strategies in phonopy and phono3py, J. Phys. Condens. Matter 35, 353001 (2023).
  * Kresse and Furthmüller [1996] G. Kresse and J. Furthmüller, Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set, Phys. Rev. B 54, 11169 (1996).
  * Kappus [1975] W. Kappus, Lattice dynamics of sapphire (Corundum), Z. Physik. B 21, 325 (1975).
  * Pavone  _et al._ [1993] P. Pavone, K. Karch, O. Schiitt, W. Windl, D. Strauch, P. Giannozzi, and S. Baroni, $\it{Ab}$ $\it{initio}$ lattice dynamics of diamond, Phys. Rev. B 48, 3156 (1993).
  * Wei and Chou [1994] S. Wei and M. Y. Chou, Phonon dispersions of silicon and germanium from first-principles calculations, Phys. Rev. B 50, 2221 (1994).
  * Dolling  _et al._ [1968] G. Dolling, H. G. Smith, R. M. Nicklow, P. R. Vijayaraghavan, and M. K. Wilkinson, Lattice dynamics of Lithium Fluoride, Phys. Rev. 168, 970 (1968).
  * Gillis  _et al._ [1968] N. S. Gillis, T. R. Koehler, and N. R. Werthamer, Calculation of the self-consistent phonon spectrum of hcp He4, Phys. Rev. 175, 1110 (1968).