# Text-Only Training for Visual Storytelling

Yuechen Wang1, Wengang Zhou1,2†, Zhenbo Lu2†, Houqiang Li1,2 1CAS Key
Laboratory of Technology in GIPAS, EEIS Department,
University of Science and Technology of China 2Institute of Artificial
Intelligence, Hefei Comprehensive National Science Center
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
(2023)

###### Abstract.

Visual storytelling aims to generate a narrative based on a sequence of
images, necessitating both vision-language alignment and coherent story
generation. Most existing solutions predominantly depend on paired image-text
training data, which can be costly to collect and challenging to scale. To
address this, we formulate visual storytelling as a visual-conditioned story
generation problem and propose a text-only training method that separates the
learning of cross-modality alignment and story generation. Our approach
specifically leverages the cross-modality pre-trained CLIP model to integrate
visual control into a story generator, trained exclusively on text data.
Moreover, we devise a training-free visual condition planner that accounts for
the temporal structure of the input image sequence while balancing global and
local visual content. The distinctive advantage of requiring only text data
for training enables our method to learn from external text story data,
enhancing the generalization capability of visual storytelling. We conduct
extensive experiments on the VIST benchmark, showcasing the effectiveness of
our approach in both in-domain and cross-domain settings. Further evaluations
on expression diversity and human assessment underscore the superiority of our
method in terms of informativeness and robustness.

Visual Storytelling, Text-Only Training, Story Planning

††journalyear: 2023††copyright: acmlicensed††conference: Proceedings of the
31st ACM International Conference on Multimedia; October 29–November 3, 2023;
Ottawa, ON, Canada.††booktitle: Proceedings of the 31st ACM International
Conference on Multimedia (MM ’23), October 29–November 3, 2023, Ottawa, ON,
Canada††price: 15.00††isbn: 979-8-4007-0108-5/23/10††doi:
10.1145/3581783.3612179††submissionid: 2129††ccs: Computing methodologies
Natural language generation††ccs: Computing methodologies Scene understanding

##  1\. Introduction

Visual storytelling (Huang2016VisualS, ), a task aimed at generating
narratives based on image sequences, has received significant interest due to
its potential applications in diverse domains such as advertising,
entertainment, and education. In comparison to other vision-to-language
generation tasks, such as visual captioning (Devlin2015LanguageMF, ), visual
storytelling presents unique challenges stemming from its subjective and
imaginative nature. As illustrated in Fig. 1, to create a coherent story that
aligns with the visual input, each sentence must not only describe the
corresponding image, but also maintain logical connections to both preceding
and subsequent sentences. This dual requirement of ensuring cross-modality
consistency while preserving narrative coherence constitutes the primary
challenge of visual storytelling.

Figure 1. Example difference between image captioning and visual storytelling.
Words highlighted in red represents contents in the corresponding image, and
words highlighted in green represents subjective expressions and information
reasoned from other images.

Existing works often require large amounts of labeled data and attempt to
learn both cross-modality alignment and story coherence simultaneously through
end-to-end training. By training on large manually annotated data, these
models are capable of generating coherent and visual-related stories.
Subsequent advancements, such as the incorporation of external knowledge
(ijcai2019p744, ) and scene graphs (Wang2020StorytellingFA, ), have further
enriched generated stories with additional details. More recently, the
employment of large pre-trained Transformer-based language models has led to
considerable improvements in visual storytelling (9577852, ). Nevertheless,
the substantial cost associated with annotating and training extensive
datasets remains a significant bottleneck, limiting the scalability of visual
storytelling approaches.

On the other hand, the burgeoning capabilities of pre-trained models offer
potential for leveraging these models to transfer knowledge to downstream
tasks such as visual storytelling, facilitating more data-efficient learning.
To this end, some prior works have combined generative language models
(Radford2018ImprovingLU, ; radford2019language, ; NEURIPS2020_1457c0d6, ) with
cross-modality pretrained models (Radford2021LearningTV, ) to explore text-
only training for image captioning (Tewel_2022_CVPR, ; nukrai-etal-2022-text,
). However, while these cross-modality models trained on paired image-text
data successfully align text with individual images, they are limited in their
capacity to comprehend the temporal structure of image sequences—an essential
component of visual storytelling.

Motivated by the observations discussed above, we propose a novel framework
that leverages pretrained generative language models and cross-modality models
for data-efficient visual storytelling. We formulate visual storytelling as a
visual-conditioned story generation task. As shown in Fig. 2, we first fine-
tune a pretrained language model using only textual data to develop a story
generator. Then, we incorporate visual clues during the generation process.
Specifically, at each decoding step, we use a pretrained cross-modality model
CLIP (Radford2021LearningTV, ) as a visual discriminator to compute a matching
score between candidate text and input images. A visual condition planner is
then designed to aggregate the matching results of the input images,
emphasizing semantics in the corresponding image while retaining information
from other images. Finally, the aggregated result is incorporated into the
decoding probability distribution to guide the generation of the next token,
resulting in a coherent and visually aligned story.

To demonstrate the effectiveness of our proposed method, we conduct extensive
experiments on the widely-used VIST benchmark (Huang2016VisualS, ). The
results show that our approach achieves state-of-the-art performance on
various evaluation metrics including comparing-based automatic metrics,
statistics-based metrics, and human evaluation. Additionally, our method
exhibits impressive generalization ability in domain-transfer experiments,
suggesting its potential for real-world applications.

We summarize the major contributions of this work as follows:

  * •

We formulate visual storytelling as a visual-conditioned generation problem
and propose a data-efficient framework which is trained solely on text-only
data by leveraging pretrained CLIP model.

  * •

We introduce a visual condition planner which is free of training. The planner
aggregates sequential visual inputs to provide local details while maintaining
the global theme of the image album, thereby improving the quality of
generated stories.

  * •

Extensive experiments on VIST benchmark demonstrate the effectiveness of our
proposed method, as evidenced by its superior performance compared to existing
methods in both automatic metrics and human evaluations.

Figure 2. The training and inference pipeline of our method. During training,
we only train the language generator on a story dataset without visual
information. Then, at inference time, we utilize a pretrained CLIP model as a
visual discriminator to align images with candidate tokens. Additionally, we
introduce a visual condition planner that aggregates image sequences, and the
output visual control is then incorporated into the generation process.

##  2\. Related Work

The main idea of our work is to model visual storytelling as a controlled text
generation task, and exploit large pretrained models to reduce the cost of
cross-modality training. In this section, we provide a brief review of the
related areas.

###  2.1. Visual Storytelling

Visual storytelling was first introduced by Huang et al.(Huang2016VisualS, ),
which involves the use of a sequence of images to convey a narrative,
necessitating reasoning over temporal context rather than merely understanding
a static moment. Early approaches expanded upon conventional image captioning
models by learning contextualized image representations(Liu2017LetYP, ) and
incorporating global visual information (GonzalezRico2018ContextualizeSA, ).
Additionally, reinforcement learning was employed to learn an implicit reward
function through adversarial reward learning, optimizing the policy model to
better align with human demonstrations (wang-etal-2018-metrics, ).
Hierarchical architectures (yu-etal-2017-hierarchically, ) and hierarchical
reinforced training (Huang2018HierarchicallySR, ) have also demonstrated
effectiveness in learning high-level semantics.

Given the imaginative nature of storytelling, external knowledge graphs have
been integrated to introduce fictional concepts not present in images
(ijcai2019p744, ; Hsu2019KnowledgeEnrichedVS, ; 9996128, ). To provide richer
stories with greater visual detail, Wang et al.(Wang2020StorytellingFA, )
incorporated scene graph generation, while Li et al.(Li2019InformativeVS, )
learned cross-modal rules for mining visual concepts. Braude et
al.(Braude2021OrderedAF, ) proposed an ordered image attention approach to
enhance story coherence through consistent grounding across sequenced images.
Furthermore, Transformer-based frameworks have demonstrated capabilities in
modeling spatial relationships between objects in
images(10.1145/3474085.3475236, ). In light of the proliferation of large pre-
trained models, several studies have focused on leveraging pre-trained models
(PTMs) for visual storytelling. Strategies include fine-tuning pre-trained
Transformer encoders (10.1145/3487553.3524649, ; 10.1145/3469877.3490604, )
and jointly tuning pre-trained language generation models with pre-trained
image encoders (9577852, ).

While the aforementioned approaches have demonstrated improvements in
generated stories by incorporating external models, knowledge, and
annotations, they also result in a significant increase in computational cost.
In contrast, our proposed method circumvents the challenges associated with
cross-modality training and annotation expenses by exclusively focusing on
training using a text corpus.

###  2.2. Controlled Text Generation

In natural language generation, incorporating controllable constraints for
open-ended text generation is both important and fundamental (prabhumoye-
etal-2020-exploring, ). With the advancements in pretraining, recent efforts
have primarily concentrated on adapting pre-trained language models (LMs) to
various attributes. A straightforward approach involves fine-tuning a pre-
trained LM to generate text with specific attributes (nan-etal-2021-dart, ;
ribeiro-etal-2021-structural, ; 10.1145/3447548.3467418, ; carlsson-
etal-2022-fine, ). Alternatively, it is feasible to design new large LM
architectures or retrain large conditioned LMs from scratch (Keskar2019CTRLAC,
; zhang-etal-2020-pointer, ; chan2021cocon, ). Recently, the exponentially
increasing scale and capacity of pre-trained LMs have made it more viable and
promising to fix pre-trained parameters and guide generation through post-
processing. Dethathri _et al._(Dathathri2019PlugAP, ) first proposed this
paradigm as Plug-and-Play language models, wherein an attribute discriminator
updates LM hidden states through back-propagation for attribute-controlled
text generation. To reduce the computational cost associated with classifier-
like discriminators ranking generated text, fine-tuned small LMs have been
employed as generative discriminators to guide the generation of large pre-
trained LMs(krause-etal-2021-gedi-generative, ; liu-etal-2021-dexperts, ;
yang-klein-2021-fudge, ; 10.5555/3505520.3505528, ). Pascual _et al._
(pascual-etal-2021-plug-play, ) extended the plug-and-play method to keyword
constraints and designed a distribution shifting strategy to augment the
decoding probability of keywords.

Guided decoding methods have demonstrated remarkable flexibility in
accommodating various constraint types and hold considerable potential due to
their independence from language models. In this work, we model visual
storytelling as a visual-conditioned story generation task and propose a
visual-linguistic discriminator to guide the generation process.

###  2.3. Large Pretrained Models

Generative language models. Taking advantage of the parallelism in the
Transformer architecture (10.5555/3295222.3295349, ), generative language
models have shown a remarkable improvement in their capabilities in the past
few years. These models can be broadly classified into two categories based on
their network architecture: Decoder-Only models (Radford2018ImprovingLU, ;
NEURIPS2020_1457c0d6, ) and Encoder-Decoder models (10.5555/3455716.3455856, ;
lewis-etal-2020-bart, ). Pretrained on large corpora, these models can
effectively transfer to various language generation tasks, such as
summarization, question answering, and story generation, with limited or even
no supervised data.

Cross-modality pretrained models. As the foundation of visual-language
understanding, the idea to align the two modalities and learn a joint
embedding space has been investigated extensively in the past decade
(10.5555/2999792.2999849, ; 10.1007/978-3-319-46478-7_5, ; 8099701, ; 8237711,
). In recent years, large cross-modality aligning models based on Transformers
have gained considerable attention (Radford2021LearningTV, ; Li2022BLIPBL, ;
Li2023BLIP2BL, ). A representative work is CLIP (Radford2021LearningTV, ),
which trains two encoders for image and text inputs using a contrastive loss.
With 400 million data pairs for training, CLIP has demonstrated remarkable
zero-shot capabilities on multiple downstream tasks.

##  3\. Preliminaries

A standard generative language model predicts the probability distribution of
the next token based on previous inputs, which can be formulated as
$P_{LM}(x_{t}|x_{<t})$. As a result, the probability of a text sequence
$\boldsymbol{x}=\\{x_{1},\dots,x_{T}\\}$ can be modeled as follows:

(1) $P_{LM}(\boldsymbol{x})=\Pi_{t=1}^{T}P_{LM}(x_{t}|x_{<t}).$

In order to incorporate controls during the generation process, a constraint
$c$ can be added to form a conditioned language model. This model generates
the probability distribution of the next token based on the history inputs and
the control constraint, and can be formulated as:

(2) $P(\boldsymbol{x}|c)=\Pi_{t=1}^{T}P(x_{t}|x_{<t},c).$

Krause et al. (krause-etal-2021-gedi-generative, ) designed a generative
discriminator to predict the probability that every candidate text sequence
corresponds to the given constraint, which is given as:

(3)
$P_{\theta}(c|x_{t},x_{<t})=\frac{P(c)\Pi_{t=1}^{T}P(x_{t}|x_{<t},c)}{\sum_{c^{\prime}\in\\{c,\overline{c}\\}}P(c^{\prime})\Pi_{t=1}^{T}P(x_{t}|x_{<t},c^{\prime})}$

where $\theta$ represents the learned parameters of the discriminator. Then,
based on the Bayes rule, the conditioned language model can be decoupled as:

(4) $P(x_{t}|x_{<t},c)\propto P_{LM}(x_{t}|x_{<t})P_{\theta}(c|x_{t},x_{<t}).$

Therefore, each step of the generation process is implemented by combining an
unconditioned language modeling $P_{LM}(x_{t}|x_{<t})$, and an attribute
discriminator $P_{\theta}(c|x_{t},x_{<t})$ with the guided decoding strategy
as described in Eq. (4). Here the discriminator is trained externally and can
be easily used with any language generator in a plug-and-play manner.

##  4\. Method

Given a sequence of images $\mathcal{I}=\\{I_{1},\dots,I_{N}\\}$, a visual
storytelling approach aims to generate a multi-sentence story $\boldsymbol{x}$
by predicting the probability $P(\boldsymbol{x}|\mathcal{I})$. To achieve
this, we propose a framework that combines a text-only trained language
generator, a pretrained visual discriminator, and a visual condition planner.
Fig. 2 illustrates the training and inference pipeline of our method. During
the training phase, we fine-tune the language generator using story text,
while in the inference phase, we employ the pre-trained visual discriminator
and the visual condition planner to guide the generation process.

###  4.1. Text-Only Training

Compared to other supervised visual storytelling methods, our approach offers
a notable advantage in that it requires training only on a text corpus,
resulting in significant cost reductions in both training and annotation
efforts. Specifically, we fine-tune a Transformer decoder-based language model
on a text story corpus to bridge the gap between pretraining on generic text
and generating coherent stories. Given a narrative text sequence
$\boldsymbol{x}=\\{x_{1},\dots,x_{T}\\}$, the language model is fine-tuned by
minimizing the maximum likelihood estimation (MLE) loss:

(5) $\mathcal{L}_{MLE}=-\frac{1}{T}\sum_{t=1}^{T}\log P_{LM}(x_{t}|x_{<t})$

Inspired by Su et al. (su2022a, ), we incorporate an additional contrastive
objective $\mathcal{L}_{CL}$ to encourage the generation of diverse and
distinct expressions. The objective is defined as:

(6) $\mathcal{L}_{CL}=\frac{1}{T(T-1)}\sum_{i=1}^{T}\sum_{j=1,j\neq
i}^{T}\max(0,\epsilon-s(x_{i},x_{i})+s(x_{i},x_{j})),$

where $\epsilon$ is a predefined margin, and $s$ is the cosine similarity
between tokens, defined by:

(7) $s(x_{i},x_{j})=\frac{h_{x_{i}}^{T}h_{x_{j}}}{|h_{x_{i}}||h_{x_{j}}|}.$

The overall training objective of the language generator is the combination of
the above two losses:

(8) $\mathcal{L}=\mathcal{L}_{MLE}+\alpha\mathcal{L}_{CL},$

where $\alpha$ is a hyper-parameter to balance the loss items.

After fine-tuning on a text story corpus, the language generator is able to
generate coherent stories in a style that is aligned with the training data.
However, since the generation process of the language generator is solely
based on textual input, it may not take into account any visual content or the
desired topic of the story. To address this, we introduce a visual
discriminator and a visual condition planner to control the story topic and
add details to the generated sentences.

Figure 3. Illustration of the visual condition planner.

###  4.2. Visual Discriminator and Story Planning

Method | METEOR | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | ROUGE_L | CIDEr
---|---|---|---|---|---|---|---
Fully-Supervised Methods
INet (Jung2020HideandTellLT, ) | 35.6 | 64.4 | 40.1 | 23.9 | 14.7 | 29.7 | 10.0
TAPM (9577852, ) | 37.2 | - | - | - | - | 33.1 | 13.8
OIAVist (Braude2021OrderedAF, ) | 36.8 | 68.4 | 42.7 | 25.2 | 15.3 | 30.2 | 10.1
KAGS (9996128, ) | 36.2 | 70.1 | 43.5 | 25.2 | 14.7 | 31.4 | 11.3
Text-Only Trained
Top-$k$ | 20.3 | 40.0 | 15.6 | 5.6 | 2.2 | 15.7 | 0.6
Nucleus | 19.6 | 38.6 | 14.2 | 4.9 | 1.9 | 15.5 | 0.5
MAGIC | 20.3 | 41.2 | 16.1 | 5.9 | 2.8 | 16.0 | 1.3
Ours | 23.0 | 43.7 | 20.2 | 9.2 | 4.5 | 17.3 | 1.2
Table 1. Comparison with existing methods on VIST test set. “Fully-Supervised” methods are trained on paired data, “Text-Only Trained” methods are trained on the textual stories of VIST. The best results under each metric are highlighted in bold. Method | ROCStories | WritingPrompts
---|---|---
M | B-1 | B-2 | B-3 | B-4 | R_L | C | M | B-1 | B-2 | B-3 | B-4 | R_L | C
Top-$k$ | 15.3 | 28.6 | 9.5 | 2.5 | 0.7 | 12.1 | 0.2 | 15.0 | 26.8 | 8.0 | 2.0 | 0.4 | 12.2 | 0.2
Nucleus | 15.0 | 28.4 | 9.0 | 2.4 | 0.7 | 12.0 | 0.3 | 14.3 | 25.6 | 7.3 | 1.5 | 0.3 | 11.9 | 0.2
MAGIC | 16.4 | 29.7 | 10.1 | 2.7 | 0.7 | 12.6 | 0.1 | 15.4 | 27.8 | 9.6 | 2.9 | 0.5 | 12.8 | 0.2
Ours | 16.6 | 28.6 | 11.5 | 3.8 | 1.2 | 12.9 | 0.2 | 16.2 | 28.8 | 9.9 | 2.9 | 0.9 | 13.7 | 0.2
Table 2. Domain transfer results of text-only trained methods. The best
results under each metric are highlighted in bold.

As previously mentioned, we consider visual storytelling as a visual-
conditioned story generation task, and employ the guided decoding paradigm to
integrate visual controls into the language generator. To achieve this, we
introduce a visual discriminator and a visual condition planner to score
candidate sequences during generation. The visual discriminator is implemented
using a pretrained visual-linguistic aligning model, while the visual
condition planner is a training-free weighting model which aggregates the text
matching results of different images.

During each generation step $t$, our language generator predicts a probability
distribution $P_{LM}(x_{t}|x_{<t})$ over the vocabulary $V$ of possible next
tokens, based on the context ${x_{<t}}$. To guide the selection of candidate
tokens, we employ a pretrained CLIP (Radford2021LearningTV, ) model as a
visual discriminator $\mathbf{D}$. Although the CLIP model has been pretrained
on a large-scale dataset of paired visual and textual data, the pretraining
process does not specifically involve annotations for visual storytelling.
Therefore, by utilizing the pretrained CLIP, our method does not require
cross-modality training and is capable of handling open-domain visual input.
This makes our approach data-efficient and more scalable than previous
methods.

Specifically, we feed each candidate token $x_{t}$ into the text encoder of
CLIP along with the context tokens $x_{<t}$ to obtain a textual representation
$f_{x_{1:t}}$. For each image $I_{j}$ in the input album, we extract a visual
representation $f_{I_{j}}$ using the visual encoder of CLIP, where
$j\in{1,\dots,N}$. Then, the cosine similarity of $f_{x_{1:t}}$ and
$f_{I_{j}}$ is computed as:

(9)
$\mathbf{D}(x_{1:t},I_{j})=\frac{f_{x_{1:t}}f_{I_{j}}}{|f_{x_{1:t}}||f_{I_{j}}|},j\in\\{1,\dots,N\\}.$

As the CLIP model is trained to map visual and textual input representations
into a sharing space, the matching score $\mathbf{D}(x_{1:t},I_{j})$ measures
the relevance between candidate sequence $x_{1:t}$ and the input image
$I_{j}$.

To ensure that the generated story aligns with the visual input fine-level
semantics of the corresponding image to the sentence being generated and
maintains the overall theme, we propose a visual condition planner. It
aggregates the scores of the input images to derive a visual control for the
current decoding step. Inspired by the work of Lin and Riedl
(10.5555/3505520.3505528, ), the planner does not require any training and
achieves both global and local alignment through weighting and multiplication
operations.

As depicted in Fig. 3, the visual condition planner computes a control weight
for each input image based on the position of current sentence in the story.
More precisely, the weight $\omega_{j}$ for image $I_{j}$ is:

(10) $\omega_{j}=C\exp(-\frac{(i-j)^{2}}{2\sigma^{2}}),$

where $i\in\\{1,\dots,N\\}$ represents the position of current sentence in the
story, and $C$ is a constant to normalize the weights and insure
$\sum_{j=1}^{N}\omega_{j}=1$. When $i=j$, the current sentence should be the
exact description of image $I_{j}$, while remaining coherent to other images
$I_{k\neq j}$. Therefore, the weight of $I_{j}$ is the largest, and weight of
$I_{k}$ descends as the distance $|k-j|$ grows. Finally, the planner applies
weighted multiplication on the scores of different images to obtain a unified
matching score between the candidate sequence $x_{1:t}$ and the input images
$\mathcal{I}$. Formally,

(11)
$P_{w}(\mathcal{I}|x_{t},x_{<t})=\Pi_{j=1}^{N}\mathbf{D}(x_{1:t},I_{j})^{\omega_{j}}.$

It is worth noting that in our experiments, the aforementioned process is
applied to a subset of the entire vocabulary, thereby reducing the
computational cost of encoding and aligning candidate text. Specifically, we
select the top $K$ tokens predicted by the language generator as the subset
$V^{K}_{(t)}$. Moreover, to eliminate the bias of the cross-modality alignment
results, we normalize the scores among candidate tokens. The final output of
the visual condition planner can be written as:

(12)
$P(\mathcal{I}|x_{t},x_{<t})=\frac{e^{(P_{w}(\mathcal{I}|x_{t},x_{<t}))}}{\sum_{x_{i}\in
V^{K}_{(t)}}e^{(P_{w}(\mathcal{I}|x_{i},x_{<t}))}}.$

###  4.3. Visual-Conditioned generation

Given the token probability predicted by the language generator and the
aggregated cross-modality matching score, visual storytelling can be decoupled
into the combination of language modeling and cross-modality aligning. Similar
to Eq. (4), the probability of next token $x_{t}$ can be decoupled as follows:

(13) $P(x_{t}|x_{<t},\mathcal{I})\propto
P_{LM}(x_{t}|x_{<t})P(\mathcal{I}|x_{t},x_{<t})^{\gamma},$

where the hyper-parameter $\gamma$ controls the weight of visual information
in the language generation process. While a higher value of $\gamma$ can
improve the alignment of visual semantics, it may also adversely affect the
quality of the generated language. Finding the right balance between language
and visual information is crucial for achieving high-quality visual
storytelling.

Furthermore, inspired by Su et al. (Su2022LanguageMC, ), we incorporate a
degeneration penalty into Eq. (13) to prevent the repetitive degeneration
problem. The final probability of visual-conditioned is formulated as:

(14)
$\begin{split}P(x_{t}|x_{<t},\mathcal{I})=&P_{LM}(x_{t}|x_{<t})P(\mathcal{I}|x_{t},x_{<t})^{\gamma}-\\\
&\beta(\max(s(x_{t},x_{j}),j\in\\{1,\dots,t-1\\}),\end{split}$

where $\beta$ is a hyper-parameter to control the degeneration penalty
strength, and $s(x_{i},x_{j})$ is defined in Eq. (7).

##  5\. Experiments

Dataset. We make evaluation on the widely-used VIST benchmark
(Huang2016VisualS, ) for visual storytelling. VIST contains $210,819$ images
from $10,117$ Flickr albums. Each sample in VIST contains five images selected
from an album, and a five-sentence story is annotated as ground truth. After
excluding broken images, the dataset contains $40,098$, $4988$, and $5050$
samples for training, validation, and testing respectively. We use the test
split of VIST as the evaluation benchmark in all experiments. Following
previous works (wang-etal-2018-metrics, ; 9577852, ), we evaluate at the album
level, generating one story for each album regardless of different selected
images. During the training stage, we use the text part of the VIST training
split, where all names are replaced with special placeholders.

Implementation Details. The language generator is initialized with a pre-
trained GPT-2 model, and fine-tuning is performed on 2 GTX3090 GPUs for
$40,000$ steps with batch size of 256. We set the training loss weight
$\alpha$ to 1. To implement the visual discriminator, we ultilize a pretrained
CLIP with ViT-base architecture as the image encoder. The visual-conditioned
generation is performed on 1 GTX3090 GPU. In the reported results, we set the
hyper-parameters $K$, $\gamma$, and $\beta$ to 45, 1, and 0.01, respectively.

Evaluation Metrics. Following the existing works on the VIST benchmark, we
adopt a set of automatic evaluation metrics including METEOR (M) (banerjee-
lavie-2005-meteor, ), BLEU (B-n) (papineni-etal-2002-bleu, ), ROUGE_L (R_L)
(lin-2004-rouge, ) and CIDEr (C) (7299087, ). METEOR measures the semantic
alignment between generated and reference sentences by leveraging WordNet.
BLEU computes the unigram and n-gram overlap between generated and candidate
sentences. ROUGE_L measures sentence-level similarity by computing the length
of longest common subsequence. CIDEr evaluates the consensus based on n-grams
and weights n-grams using Term Frequency Inverse Document Frequency (TF-IDF)
to emphasize informative content. However, we note that these metrics, as they
rely on word correspondence with the ground truth, may not fully capture the
quality of open-ended generation tasks such as storytelling.

###  5.1. Quantitative Results

Method | VIST-Text | ROC | WP
---|---|---|---
D-1 | D-2 | D-1 | D-2 | D-1 | D-2
MAGIC | 2.4 | 6.6 | 0.8 | 1.3 | 1.1 | 2.5
Ours | 4.7 | 17.2 | 5.5 | 18.8 | 7.4 | 26.5
Table 3. Diversity evaluation results. “VIST-Text”, “ROC”, “WP” represents the
training of language generator is conducted on the text part of VIST dataset,
ROCStories, and WritingPrompts, respectively. “D-$n$” refers to
“Distinct-$n$”. The best results for each metric are highlighted in bold.

Comparison with Existing Methods. We compare the generation quality of our
method with a text-only trained methods. First, we adopt top-$k$ sampling
(fan-etal-2018-hierarchical, ) ($k=40$) and nucleus sampling
(Holtzman2019TheCC, ) ($p=0.95$). Since these sampling-based decoding strategy
takes no account of visual inputs, we consider them as the lower bound of the
text-only trained methods. We also include MAGIC (Su2022LanguageMC, ), which
was proposed for image captioning and image-based story generation. MAGIC
takes an image as input and generate text outputs by adding CLIP similarity
scores on language model predicted probabilities. To extend MAGIC to the
visual storytelling task, we average the representation of input image
sequence to form the visual input of MAGIC. To provide a comprehensive
comparison, we also report results of several fully-supervised baselines. INet
(Jung2020HideandTellLT, ), TAPM (9577852, ), OIAVist (Braude2021OrderedAF, ),
and KAGS (9996128, )

In Table 1, we present the comparison of our proposed method with existing
fully-supervised and text-only trained methods. As expected, the fully-
supervised methods trained on cross-modality paired data exhibit better
performance compared to the text-only trained methods. However, our proposed
method outperforms the text-only trained baselines on almost all metrics by a
considerable margin, demonstrating the effectiveness of our visual-conditioned
generation strategy.

(a)  (b) Relevance (c) Expressiveness (d) Concreteness

Figure 4. Human evaluation results. “Tie” means the annotator cannot choose the better story. Method | M | B-1 | B-2 | B-3 | B-4 | R_L | C
---|---|---|---|---|---|---|---
Ours-Max | 22.6 | 41.8 | 18.9 | 8.5 | 4.1 | 17.0 | 0.9
Ours-Mean | 22.8 | 43.2 | 20.0 | 9.1 | 4.4 | 17.2 | 1.3
Ours-Local | 22.4 | 42.1 | 19.4 | 8.7 | 4.2 | 17.2 | 1.0
Ours-Planner | 23.0 | 43.7 | 20.2 | 9.2 | 4.5 | 17.3 | 1.2
Table 4. Evaluation results of different image album aggregation strategies.
The best results for each metric are highlighted in bold.

Cross-domain Transfer. In order to evaluate the generalization ability of our
method, we also explore cross-domain transfer by using story datasets of
different domains in the text-only training stage. Specifically, we use
ROCStories (mostafazadeh-etal-2016-corpus, ) and WritingPrompts (fan-
etal-2018-hierarchical, ) for training. The training split of ROCStories
dataset contains $51,165$ five-sentence commonsense stories. And the training
split of WritingPrompts dataset contains $272,600$ stories collected from
Reddit’s WRITINGPROMPTS forum111www.reddit.com/r/WritingPrompts/. The average
length of WritingPrompts stories is 734.5, and the average number of sentences
is 39.4, making it significantly larger than the VIST dataset and introducing
a larger domain gap. During training, we exclude the story title and writing
prompts to align with the VIST evaluation process.

In Table 2, we compare the cross-domain transfer ability between our method
and the text-only trained baselines. We observe a considerable drop in
performance for all methods when evaluated on datasets from different domains.
This is expected since the style, theme and topic of the stories are different
across datasets. However, our method still outperforms others on most
evaluation metrics, demonstrating its superior generalization ability.

Diversity Evaluation. To further assess the expressive diversity of the
generated stories, we use Distinct-$n$ which calculates the number of distinct
n-grams of all generated stories (li-etal-2016-diversity, ). The value is
divided by the total number of generated tokens to avoid favoring long
sentences. The results presented in Table 3 demonstrate that our method
significantly outperforms the baseline in terms of diversity. This can be
attributed to the ability of our method to attend to both global and local
visual input, which results in more informative and diverse expressions.
Additionally, it can be observed that the diversity of generated stories is
relevant to the training corpus, which suggests that the incorporation of
external text corpus can benefit visual storytelling.

Figure 5. Analysis of the effect of number of candidates $K$.

Human Evaluation. As illustrated in previous works (wang-etal-2018-metrics, ;
Braude2021OrderedAF, ), automatic evaluation metrics are insufficient for
visual storytelling due to its subjective and imaginative nature. To obtain
more reliable estimates, we also perform human evaluation. Following common
practice, we randomly selected 150 examples from the test set, and invited 5
human annotators to rank generation results of different methods.
Specifically, the annotators were asked to evaluate the stories based on three
criteria: relevance, expressiveness, and concreteness. Relevance refers to
whether the story covers the topic and main objects in the images.
Expressiveness refers to whether the story is coherent, grammatically and
semantically correct, and free of repetition. Concreteness refers to whether
the story is narrative and concrete.

Fig. 4 shows the evaluation results of 5 human annotators. Our method
outperforms MAGIC by a large margin in all three aspects. The dominance of our
method is most significant in terms of Concreteness, indicating a greater
ability to incorporate visual details in the generated stories. The
expressiveness of MAGIC is better than the two other aspects, which reflects
the fact that the language quality of our method is slightly affected by
introducing fine-level visual control. Additionally, the “Tie” option is
selected in a large percentage in all three criteria, which has not been
reported in previous methods (Jung2020HideandTellLT, ; 9577852, ; 9996128, ;
Braude2021OrderedAF, ). We believe the reason is that the overall quality of
stories generated by text-only trained methods is lower than full-supervised
methods, making it difficult to rank for human annotators.

###  5.2. Ablation Study

Figure 6. Analysis of the effect of control weight $\gamma$. Figure 7.
Qualitative comparison between our method and MAGIC. Words highlighted in red
represents exact description of corresponding image, and words highlighted in
green represents information from other images.

Impact of Visual Condition Planner. We conduct ablation experiments to analyze
the effect of the visual condition planner, which aggregates the cross-
modality matching result of input images. Specifically, we replace the
aggregation process to three straightforward strategies: 1) choosing the
maximum matching score in all images, 2) averaging the scores of all images,
and 3) using the score of the corresponding image. The evaluation results in
Table 4 indicate that both strategies of viewing the images equally within the
album (“Ours-Max” and “Ours-Mean”) and focusing solely on the corresponding
local image (“Ours-Local”) have a negative impact on the quality of the
generated stories.

Impact of Hyper-parameters. During the visual-conditioned generation, the
selection of top-$K$ candidate tokens to compute cross-modality matching score
with visual inputs and the addition of visual control to the decoding process
with a control weight $\gamma$ in Eq. (14) are governed by predefined hyper-
parameters. Therefore, it is important to analyze the influence of these
hyper-parameters on the quality of generated stories.

From the results in Fig. 5, we observe that the performance improves with $K$
when $K<30$, and remains relatively stable when $K$ continues to increase.
However, when $K$ is too large ($>60$), the performance slightly decreases as
$K$ keep increasing. It is also worth noting that the inference time
significantly increases as $K$ increases. Therefore, we choose $K=45$ in our
experiments as it strikes a balance between performance and efficiency. The
results in Fig. 6 demonstrate the significant impact of the control weight
$\gamma$ on the generation process. Specifically, when the control weight is
too small, the generated stories tend to be disconnected from the visual
input, while an excessively large control weight will lead to a disruption in
the decoding process, thus deteriorating the overall quality of the generated
text. These experimental findings align with our initial intuition and suggest
the importance of selecting an appropriate control weight in the visual-
conditioned generation.

###  5.3. Qualitative Results

Fig. 7 presents two examples of generated stories by MAGIC and our proposed
method. The results show that our approach generates stories with more
accurate semantics that correspond to the images, as indicated by the red
highlights. Moreover, the visual condition planner enables the generation of
sentences that are relevant to the other images in the input sequence, as
shown by the green highlights. Our method outperforms the baseline method in
capturing visual contents within a single image and maintaining the theme of
the album, resulting in stories of higher quality.

##  6\. Conclusion and Discussion

In this paper, we propose a novel approach for visual storytelling that only
requires textual story data for training. By leveraging the capabilities of
pretrained cross-modality models such as CLIP, we model the visual
storytelling task as a visual-conditioned generation problem. We adopt a
guided decoding paradigm and design a visual condition planner to aggregate
the input visual sequence. Our method is evaluated on the VIST benchmark
through extensive experiments, which demonstrate its effectiveness in
generating high-quality visual stories.

Although the proposed method avert the cost of cross-modality annotated data,
the training-free visual condition planner does have its limitations in
understanding the complex temporal structures of visual input, which may
affect the complexity of the generated story. In future work, it may be worth
exploring few-shot learning methods to aggregate aligning results of image
sequence to generate more informative and narrative stories.

###### Acknowledgements.

This work was supported by NSFC under Contract U20A20183 and 62021001. It was
also supported by the GPU cluster built by MCC Lab of Information Science and
Technology Institution, USTC, and the Supercomputing Center of the USTC.

## References

  * [1] Ting-Hao ’Kenneth’ Huang, Francis Ferraro, N. Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross B. Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, C. Lawrence Zitnick, Devi Parikh, Lucy Vanderwende, Michel Galley, and Margaret Mitchell.  Visual storytelling.  In North American Chapter of the Association for Computational Linguistics, 2016.
  * [2] Jacob Devlin, Hao Cheng, Hao Fang, Saurabh Gupta, Li Deng, Xiaodong He, Geoffrey Zweig, and Margaret Mitchell.  Language models for image captioning: The quirks and what works.  In Annual Meeting of the Association for Computational Linguistics, 2015.
  * [3] Pengcheng Yang, Fuli Luo, Peng Chen, Lei Li, Zhiyi Yin, Xiaodong He, and Xu Sun.  Knowledgeable storyteller: A commonsense-driven generative model for visual storytelling.  In International Joint Conference on Artificial Intelligence, pages 5356–5362, 2019.
  * [4] Ruize Wang, Zhongyu Wei, Piji Li, Qi Zhang, and Xuanjing Huang.  Storytelling from an image stream using scene graphs.  AAAI Conference on Artificial Intelligence, pages 9185–9192, 2020\.
  * [5] Youngjae Yu, Jiwan Chung, Heeseung Yun, Jongseok Kim, and Gunhee Kim.  Transitional adaptation of pretrained models for visual storytelling.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12653–12663, 2021.
  * [6] Alec Radford and Karthik Narasimhan.  Improving language understanding by generative pre-training.  2018\.
  * [7] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  2019\.
  * [8] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In Advances in Neural Information Processing Systems, pages 1877–1901, 2020.
  * [9] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, volume 139, pages 8748–8763, 2021.
  * [10] Yoad Tewel, Yoav Shalev, Idan Schwartz, and Lior Wolf.  Zerocap: Zero-shot image-to-text generation for visual-semantic arithmetic.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17918–17928, 2022.
  * [11] David Nukrai, Ron Mokady, and Amir Globerson.  Text-only training for image captioning using noise-injected CLIP.  In Findings of the Association for Computational Linguistics, pages 4055–4063, 2022.
  * [12] Yu Liu, Jianlong Fu, Tao Mei, and Chang Wen Chen.  Let your photos talk: Generating narrative paragraph for photo stream via bidirectional attention recurrent neural networks.  In AAAI Conference on Artificial Intelligence, page 1445–1452, 2017.
  * [13] Diana Gonzalez-Rico and Gibran Fuentes Pineda.  Contextualize, show and tell: A neural visual storyteller.  arXiv preprint, abs/1806.00738, 2018.
  * [14] Xin Wang, Wenhu Chen, Yuan-Fang Wang, and William Yang Wang.  No metrics are perfect: Adversarial reward learning for visual storytelling.  In Annual Meeting of the Association for Computational Linguistics, pages 899–909, 2018.
  * [15] Licheng Yu, Mohit Bansal, and Tamara Berg.  Hierarchically-attentive RNN for album summarization and storytelling.  In Conference on Empirical Methods in Natural Language Processing, pages 966–971, 2017.
  * [16] Qiuyuan Huang, Zhe Gan, Asli Celikyilmaz, Dapeng Oliver Wu, Jianfeng Wang, and Xiaodong He.  Hierarchically structured reinforcement learning for topically coherent visual story generation.  AAAI Conference on Artificial Intelligence, pages 8465–8472, 2019\.
  * [17] Chao-Chun Hsu, Zi-Yuan Chen, Chi-Yang Hsu, Chih-Chia Li, Tzu-Yuan Lin, Ting-Hao ’Kenneth’ Huang, and Lun-Wei Ku.  Knowledge-enriched visual storytelling.  AAAI Conference on Artificial Intelligence, pages 7952–7960, 2020\.
  * [18] Tengpeng Li, Hanli Wang, Bin He, and Chang Wen Chen.  Knowledge-enriched attention network with group-wise semantic for visual storytelling.  IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–12, 2022.
  * [19] Jiacheng Li, Haizhou Shi, Siliang Tang, Fei Wu, and Yueting Zhuang.  Informative visual storytelling with cross-modal rules.  In ACM International Conference on Multimedia, page 2314–2322, 2019.
  * [20] Tom Braude, Idan Schwartz, Alex Schwing, and Ariel Shamir.  Ordered attention for coherent visual storytelling.  In ACM International Conference on Multimedia, page 3310–3318, 2022.
  * [21] Mengshi Qi, Jie Qin, Di Huang, Zhiqiang Shen, Yi Yang, and Jiebo Luo.  Latent memory-augmented graph transformer for visual storytelling.  In ACM International Conference on Multimedia, page 4892–4901, 2021.
  * [22] Kohei Uehara, Yusuke Mori, Yusuke Mukuta, and Tatsuya Harada.  ViNTER: Image narrative generation with emotion-arc-aware transformer.  In Companion Proceedings of the Web Conference, page 716–725, 2022\.
  * [23] Ruichao Fan, Hanli Wang, Jinjing Gu, and Xianhui Liu.  Visual storytelling with hierarchical bert semantic guidance.  In ACM Multimedia Asia, 2022.
  * [24] Shrimai Prabhumoye, Alan W Black, and Ruslan Salakhutdinov.  Exploring controllable text generation techniques.  In International Conference on Computational Linguistics, pages 1–14, 2020.
  * [25] Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, Yangxiaokang Liu, Nadia Irwanto, Jessica Pan, Faiaz Rahman, Ahmad Zaidi, Mutethia Mutuma, Yasin Tarabar, Ankit Gupta, Tao Yu, Yi Chern Tan, Xi Victoria Lin, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani.  DART: Open-domain structured data record to text generation.  In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 432–447, 2021\.
  * [26] Leonardo F. R. Ribeiro, Yue Zhang, and Iryna Gurevych.  Structural adapters in pretrained language models for AMR-to-Text generation.  In Conference on Empirical Methods in Natural Language Processing, pages 4269–4282, 2021.
  * [27] Xu Zou, Da Yin, Qingyang Zhong, Hongxia Yang, Zhilin Yang, and Jie Tang.  Controllable generation from pre-trained language models via inverse prompting.  In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, page 2450–2460, 2021.
  * [28] Fredrik Carlsson, Joey Öhman, Fangyu Liu, Severine Verlinden, Joakim Nivre, and Magnus Sahlgren.  Fine-grained controllable text generation using non-residual prompting.  In Annual Meeting of the Association for Computational Linguistics, pages 6837–6857, 2022.
  * [29] Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher.  Ctrl: A conditional transformer language model for controllable generation.  arXiv preprint, abs/1909.05858, 2019.
  * [30] Yizhe Zhang, Guoyin Wang, Chunyuan Li, Zhe Gan, Chris Brockett, and Bill Dolan.  POINTER: Constrained progressive text generation via insertion-based generative pre-training.  In Conference on Empirical Methods in Natural Language Processing, pages 8649–8670, 2020.
  * [31] Alvin Chan, Yew-Soon Ong, Bill Pung, Aston Zhang, and Jie Fu.  Cocon: A self-supervised approach for controlled text generation.  In International Conference on Learning Representations, 2021.
  * [32] Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu.  Plug and play language models: A simple approach to controlled text generation.  arXiv preprint, abs/1912.02164, 2019.
  * [33] Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani.  GeDi: Generative discriminator guided sequence generation.  In Findings of the Association for Computational Linguistics, pages 4929–4952, 2021.
  * [34] Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi.  DExperts: Decoding-time controlled text generation with experts and anti-experts.  In Annual Meeting of the Association for Computational Linguistics, pages 6691–6706, 2021.
  * [35] Kevin Yang and Dan Klein.  FUDGE: Controlled text generation with future discriminators.  In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3511–3535, 2021.
  * [36] Zhiyu Lin and Mark O. Riedl.  Plug-and-blend: A framework for plug-and-play controllable story generation with sketches.  In AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, 2021.
  * [37] Damian Pascual, Beni Egressy, Clara Meister, Ryan Cotterell, and Roger Wattenhofer.  A plug-and-play method for controlled text generation.  In Findings of the Association for Computational Linguistics, pages 3973–3997, 2021.
  * [38] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In International Conference on Neural Information Processing Systems, page 6000–6010, 2017.
  * [39] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  Journal of Machine Learning Research, pages 1–67, 2020.
  * [40] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, 2020.
  * [41] Andrea Frome, Greg S. Corrado, Jonathon Shlens, Samy Bengio, Jeffrey Dean, Marc’Aurelio Ranzato, and Tomas Mikolov.  DeViSE: A deep visual-semantic embedding model.  In International Conference on Neural Information Processing Systems, page 2121–2129, 2013.
  * [42] Armand Joulin, Laurens van der Maaten, Allan Jabri, and Nicolas Vasilache.  Learning visual features from large weakly supervised data.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, European Conference on Computer Vision, pages 67–84, 2016.
  * [43] Lluis Gomez, Yash Patel, Marçal Rusiñol, Dimosthenis Karatzas, and C. V. Jawahar.  Self-supervised learning of visual features through embedding images into text topic spaces.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 2017–2026, 2017.
  * [44] Ang Li, Allan Jabri, Armand Joulin, and Laurens van der Maaten.  Learning visual n-grams from web data.  In IEEE International Conference on Computer Vision, pages 4193–4202, 2017.
  * [45] Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi.  BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In International Conference on Machine Learning, 2022.
  * [46] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  arXiv preprint, abs/2301.12597, 2023.
  * [47] Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier.  A contrastive framework for neural text generation.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 21548–21561, 2022.
  * [48] Yunjae Jung, Dahun Kim, Sanghyun Woo, Kyungsu Kim, Sungjin Kim, and In-So Kweon.  Hide-and-tell: Learning to bridge photo streams for visual storytelling.  In AAAI Conference on Artificial Intelligence, pages 11213–11220, 2020.
  * [49] Yixuan Su, Tian Lan, Yahui Liu, Fangyu Liu, Dani Yogatama, Yan Wang, Lingpeng Kong, and Nigel Collier.  Language models can see: Plugging visual controls in text generation.  arXiv preprint, abs/2205.02655, 2022.
  * [50] Satanjeev Banerjee and Alon Lavie.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, 2005.
  * [51] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In Annual Meeting of the Association for Computational Linguistics, pages 311–318, 2002.
  * [52] Chin-Yew Lin.  ROUGE: A package for automatic evaluation of summaries.  In Text Summarization Branches Out, pages 74–81, 2004.
  * [53] Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh.  CIDEr: Consensus-based image description evaluation.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 4566–4575, 2015.
  * [54] Angela Fan, Mike Lewis, and Yann Dauphin.  Hierarchical neural story generation.  In Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898, 2018.
  * [55] Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi.  The curious case of neural text degeneration.  2020\.
  * [56] Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen.  A corpus and cloze evaluation for deeper understanding of commonsense stories.  In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839–849, 2016\.
  * [57] Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan.  A diversity-promoting objective function for neural conversation models.  In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110–119, 2016\.


# Electronic Structure of the Bond Disproportionated Bismuthate Ag2BiO3

Mohamed Oudah1,2<EMAIL_ADDRESS>Minu Kim1 Ksenia S. Rabinovich1 Kateryna
Foyevtsova2 Graham McNally1 Berkay Kilic1 Kathrin Küster1 Robert Green2,3
Alexander V. Boris1 George Sawatzky2 Andreas P. Schnyder1 D. A. Bonn2 Bernhard
Keimer1 Hidenori Takagi1,4 1Max Planck Institute for Solid State Research,
Heisenbergstrasse 1, 70569 Stuttgart, Germany
2Stewart Blusson Quantum Matter Institute, University of British Columbia,
Vancouver, British Columbia V6T 1Z4, Canada
3Department of Physics & Engineering Physics, University of Saskatchewan,
Saskatoon, Saskatchewan S7N 5E2, Canada
4Department of Physics, University of Tokyo, Bunkyo-ku, Hongo 7-3-1, Tokyo
113-0033, Japan

###### Abstract

We present a comprehensive study on the silver bismuthate Ag2BiO3, synthesized
under high-pressure high-temperature conditions, which has been the subject of
recent theoretical work on topologically complex electronic states. We present
X-ray photoelectron spectroscopy results showing two different bismuth states,
and X-ray absorption spectroscopy results on the oxygen $K$-edge showing holes
in the oxygen bands. These results support a bond disproportionated state with
holes on the oxygen atoms for Ag2BiO3. We estimate a band gap of $\sim$1.25 eV
for Ag2BiO3 from optical conductivity measurements, which matches the band gap
in density functional calculations of the electronic band structure in the
non-symmorphic space group $Pnn2$, which supports two inequivalent Bi sites.
In our band structure calculations the disproportionated Ag2BiO3 is expected
to host Weyl nodal chains, one of which is located $\sim$0.5 eV below the
Fermi level. Furthermore, we highlight similarities between Ag2BiO3 and the
well-known disproportionated bismuthate BaBiO3, including breathing phonon
modes with similar energy. In both compounds hybridization of Bi-$6s$ and
O-$2p$ atomic orbitals is important in shaping the band structure, but in
contrast to the Ba-$5p$ in BaBiO3, the Ag-$4d$ bands in Ag2BiO3 extend up to
the Fermi level.

###### pacs:

Valid PACS appear here

††preprint: APS/123-QED

##  I Introduction

Transition-metal oxides include many novel classes of materials that host a
wide range of structures and electronic properties. The field of condensed
matter physics had made a great leap forward with the discovery of high
transition temperature superconductivity ($T_{\text{c}}$) in the layered
copper oxides Bednorz and Müller (1986) and in the hole-doped cubic bismuthate
(Ba,K)BiO3 Mattheiss et al. (1988). The undoped BaBiO3 is a disproportionated
insulator. Also, oxide materials have shown promise for exploring topological
properties Seki et al. (2012); Kargarian and Fiete (2013); Ishiwata et al.
(2020). We identified the silver bismuthates as a promising set of materials
for studying highly correlated electron systems and for finding topologically
non-trivial oxides. Ag2BiO3 was recently highlighted in theoretical work with
various predictions of topologically protected states He et al. (2018); Fu et
al. (2018); Singh et al. (2018).

The previous experimental work on Ag2BiO3 has resulted in a number of crystal
structures reported in the Inorganic Crystal Structure Database (ICSD), which
fall into the space groups $Pnna$, $Pnn2$, and $Pn$ Deibele and Jansen (1999);
Oberndorfer et al. (2006). $Pnna$ is the oldest structure reported, which has
a single bismuth site implying a uniform 4+ state for bismuth and as a result
equal Bi-O distances. A Bi4+ state is unusual, however, and typically a
compound with nominal charge of 4+ on bismuth disproportionates. The
disproportionation in BaBiO3 is reflected in the existence of two distinct
sites for Bi, but the nature of this disproportionation is debated. It was
emphasized in a following neutron diffraction study that Ag2BiO3 has two
distinct bismuth sites and crystalizes in the $Pnn2$ space group Oberndorfer
et al. (2006), and the two sites were explained with the disproportionation of
Bi4+ into Bi3+ and Bi5+ states. The existence of two distinct bismuth sites in
Ag2BiO3 ($Pnn2$) is similar to BaBiO3, but these compounds crystallize in
different crystal structures, with a combination of corner- and edge-sharing
BiO6 octahedra in Ag2BiO3 and only corner-sharing BiO6 octahedra in the
perovskite BaBiO3. By thoroughly characterizing the disproportionated state in
Ag2BiO3 and comparison to BaBiO3, we establish $(M^{1+})_{2}$BiO3 as a new
class of disproportionated bismuthates beyond those forming in the perovskite
structure.

The disproportionation of the Bi site and resulting $Pnn2$ symmetry also have
important ramifications for the topological nature of Ag2BiO3. Theoretical
predictions of topological phases in Ag2BiO3 include an hourglass nodal net
semimetal Fu et al. (2018), a Dirac semimetal He et al. (2018), and an
hourglass Dirac semimetal Singh et al. (2018). All of these topological
predictions were made on Ag2BiO3 in the $Pnna$ phase with a single bismuth
site, where breaking Bi charge order to attain the $Pnna$ symmetry is
important to realizing these topologically nontrivial phases. However, we will
show that Ag2BiO3 crystallizes in the $Pnn2$ space group with two
crystallographically distinct bismuth sites even when synthesized under high-
pressure and high-temperature conditions. The topological predictions made for
the $Pnna$ phase therefore do not apply to Ag2BiO3 crystallizing in the $Pnn2$
phase. Rather, we predict Weyl states in the $Pnn2$ phase.

Figure 1: (a) The powder X-ray diffraction pattern measured at room
temperature of Ag2BiO3 (top) and the calculated pattern for Ag2BiO3 in the
$Pnn2$ phase (middle) and the $Pnna$ phase (bottom). (b) Part of the unit cell
of Ag2BiO3 in the $Pnn2$ phase with “Bi3+” (purple) and “Bi5+” (brown)
highlighting the different oxygen metal distances Oberndorfer et al. (2006).
(c) Unit cell of Ag2BiO3 in the $Pnn2$ phase along the $ac$-plane showing the
shared edges of the octahedra along the $a$ direction. (d) Unit cell of
Ag2BiO3 in the $Pnn2$ phase along the $bc$-plane. (e) Unit cell of BaBiO3
showing the corner-sharing octahedra and “Bi3+” (red) and “Bi5+”(blue).

In this paper, we characterize Ag2BiO3 samples synthesized at 6 GPa using
powder X-ray diffraction (XRD) and various spectroscopic techniques. We
present evidence from X-ray photoelectron Spectroscopy (XPS) supporting the
disproportionation of bismuth into two sites in Ag2BiO3, and present evidence
for a bulk band gap, a consequence of the disproportionation, seen in an
optical conductivity measurement. The optical gap is consistent with density
functional theory (DFT) calculations of the electronic band structure in the
$Pnn2$ (disproportionated) phase of this material. Our calculations, on the
experimentally realized $Pnn2$ phase, further predict Weyl nodal chains (i.e.,
chains of connected loop degeneracies in momentum space) at various energies
in this phase, one being $\sim$0.5 eV below the Fermi level. We provide
evidence for oxygen holes using X-ray absorption spectroscopy (XAS), and
considering the XPS results we propose that Ag2BiO3 is bond disproportionated
rather than charge disproportionated. We then highlight the extent of
similarities between Ag2BiO3 and BaBiO3; similar breathing mode peaks are
observed in Raman spectra and we identify similarities in the band structure
of these compounds when considering the oxygen molecular orbitals in these
disproportionated bismuthates. Finally, we present new evidence from Raman
spectra and XRD for the previously reported structural transition to a $Pn$
phase at low temperature Oberndorfer et al. (2006).

##  II Experimental Details

Ag2BiO3 samples were synthesized with a high-pressure, high-temperature method
utilizing a belt-press. Bi2O3 and AgO powders were well mixed in a mortar and
pestle, then loaded into a gold capsule with a drop of KOH to speed up the
reaction. The capsule was then pressed to 6 GPa and heated at 730 K for 1 hour
before quenching and retrieving the polycrystalline product. A finely ground
sample was loaded into a 0.3 mm capillary, and powder XRD patterns of Ag2BiO3
were collected while rotating the capillary using Ag radiation (0.5594 Å). The
samples were cooled down to ${\sim}200~{}$K for low temperature XRD.

Figure 2: X-ray photoelectron spectra of O, Ag, and Bi in Ag2BiO3 are shown in
(a), (b), and (c). In (a) the three peaks A, B, and C are needed to fit the
measured spectra for the O-$1s$ states. In (b) the splitting between peaks D1
and D2 is consistent with that expected due to spin-orbit-coupling (SOC) for
Ag-$3d$ states. In (c) the splitting between F1 and F2 is consistent with that
expected due to SOC for Bi-$4f$ states, and the same can be said for the split
between G1 and G2. Due to the asymmetry of the measured spectra, four curves
are needed to fit the Bi-$4f$ states. This leads us to conclude the existence
of two different Bi states in Ag2BiO3, both close in energy to Bi3+ Hegde et
al. (1989).

X-ray absorption spectra of the oxygen $K$-edges were recorded in partial
fluorescence yield (PFY) mode at the Spherical Grating Monochromator (SGM)
beamline at the Canadian Light Source. Spectra in PFY mode are collected using
energy resolving silicon drift detectors, which allow selection of only the
oxygen $K$-edge fluorescence. The average of data from four separate silicon
drift detectors was used for the PFY spectra. Spectra are calibrated in energy
by using a TiO2 reference sample Quackenbush et al. (2013). Spectra were then
normalized by the incident beam intensity, and normalized according to post-
edge intensity after subtracting a linear background. The Ag2BiO3 spectra were
collected under ultrahigh vacuum at room temperature from a pressed pellet
attached to carbon tape, with the pellet surface at an angle of 45° to the
incident beam. The XPS data were collected using a commercial Kratos AXIS
Ultra spectrometer and a monochromatized Al $K_{\alpha}$ source (photon
energy: 1486.6 eV). The base pressure during XPS was in the low $10^{-10}$
mbar range. The spectra were collected using an analyzer pass energy of 20 eV.
XPS spectra were analyzed using the CasaXPS software. The charge-neutralizer
was used and the C 1s was set to 284.8 eV for binding energy calibration Swift
(1982); Barr and Seal (1995). To fit the peaks, a combination of multiple
Gaussian-Lorentzian mixture functions and a Shirley background was used. The
binding energy separation and the area ratio of the doublets were not
constrained but the results are within experimental error of the tabulated
values Moulder and Chastain (1992). The sum of all the fitted functions to XPS
peaks is shown with a black curve in Fig. 2.

Optical spectroscopic ellipsometry measurements were performed using a
variable-angle spectroscopic ellipsometer (VASE, Woollam) with photon energy
in the range of 0.7–-6.5 eV at incident angles of 70∘. The real and imaginary
parts of the dielectric function can be obtained accurately from the
ellipsometric angles $\psi(\omega)$ and $\Delta(\omega)$ extracted from the
optical spectra, without the need of a Kramers-Kronig analysis.

The Raman data were collected in backscattering geometry using a Dilor XY-
triple grating Raman spectrometer equipped with a charge coupled device camera
as detector. With power of 0.2 mW at 632 nm it was possible to measure Raman
spectra on polished surfaces of the Ag2BiO3 samples. Low temperature Raman
measurements were performed between 10-300 K. The resolution of our
spectrometers for this experiment was about 3 cm-1.

##  III Computational Details

The band structure calculations for Ag2BiO3 presented in Fig. 3 (d) were
obtained using the Vienna ab initio Simulation Package (VASP 5.4) with the
generalized gradient approximation (GGA) of Perdew-Burke-Ernzerhof (PBE) type
exchange-correlation potential Perdew et al. (1996). The cut-off energy of
plane wave basis set was set to be the ENMAX value in the pseudo-potential
plus $30\%$. A-centered $3\times 3\times 2$ Monkhorst-Pack grid was used for
the self-consistent field (SCF) calculations.

The electronic structure calculations shown in Figs. 4 (a)-(d) were performed
using density functional theory (DFT) with the full-potential linearized-
augmented-plane-wave code WIEN2k Blaha et al. (2001). Exchange and correlation
effects were treated within the local density approximation (LDA) Perdew and
Wang (1992). For Ag2BiO3, we use the $Pnn2$ unit cell with the lattice
constants $a=5.983$ Å, $b=6.324$ Å, and $c=9.576$ Å, as determined in Ref.
Oberndorfer et al. (2006). For BaBiO3, we use the $P2_{1}/n$ unit cell with
the lattice constants $a=6.174$ Å, $b=6.125$ Å, and $c=8.652$ Å, as determined
in Ref. Kennedy et al. (2006). For both bismuthates, a $6\times 6\times 4$
$\boldsymbol{k}$-point grid was used for the Brillouin-zone integration.
Projections onto oxygen molecular orbitals were performed with a modified
version of WIEN2k, following the procedure outlined in Ref. Foyevtsova and
Sawatzky (2019).

##  IV Results and Discussion

###  IV.1 Analysis of XRD, XPS, and XAS

We used XRD, XPS and XAS measurements to demonstrate that Ag2BiO3 in the
$Pnn2$ phase is a bond-disproportionated bismuthate. As it is difficult to
distinguish the $Pnna$ structure from the $Pnn2$ structure with only PXRD
data, as demonstrated in Fig. 1, we will start by accepting the reported
$Pnn2$ phase reported and assess the consistency of various properties we
measure with this structure. The lower scattering power of oxygen in X-ray
diffraction and natural twinning of Ag2BiO3 crystals makes it difficult to
discern the $Pnn2$ from the $Pnna$ structure. A previous powder neutron
diffraction study, which is more sensitive to diffraction from oxygen than
X-rays, on Ag2BiO3 resulted in a precise refinement of O atomic positions and
a structure where Bi has two distinct sites Oberndorfer et al. (2006). We
compare the simulated powder XRD and neutron diffraction patterns in Sup. Fig.
1 to demonstrate how the $Pnn2$ phase can be identified in neutron diffraction
Sup .

In the $Pnn2$ phase, Ag2BiO3 has two distinct bismuth sites resulting in two
BiO6 octahedra with shorter and longer Bi-O bonds. It should be emphasized
that the Bi-O distances vary within each of the two types of octahedra in
Ag2BiO3. These Bi-O6 octahedra are edge-sharing along the $a$ direction, but
only share corners along the $b$ and $c$ directions. The O-O distances at the
shared edges of these octahedra are shorter than the distances between the
other oxygen atoms, which may correspond to oxygen dimerization at the shared
edges. The Ag atoms are sitting between the BiO6 octahedra channels and are
bonded to 3 oxygen atoms each. The shortest Ag-O distances for both Ag sites
are 2.1464 and 2.2065 Å, which are expected for Ag1+ typically seen in oxides.

Figure 3: (a) The X-ray absorption spectra at the Oxygen $K$-edge of Ag2BiO3
and a reference sample of BaBiO3. (b) Optical spectroscopy data on Ag2BiO3
extracted from ellipsometry measurement. (c) Brillouin zone of Ag2BiO3with the
high-symmetry points indicated by black dots. The nodal chains are
schematically illustrated by the blue and red lines. At the Z and S points
(yellow) there are fourfold degeneracies, formed by two Weyl points with
opposite chirality. (d) Band structure of Ag2BiO3 in the $Pnn2$ phase using
DFT calculation with hybrid functionals.

The X-ray photoelectron spectra for Ag, Bi, and O in Ag2BiO3 are presented in
Figs. 2(a)-(c). As expected, only one state is observed for Ag, with peak
centers at 373.8 eV (peak D1 in Fig. 2(b)) and 367.8 eV (peak D2 in Fig. 2(b))
which match the transition seen in Ag1+ in other oxides and chalcogenides
Moulder and Chastain (1992). The splitting here is expected for the $d$-states
due to spin-orbit-coupling (SOC). The XPS of O shows three peaks at 529.1 eV
(C), 530.8 eV (B), and 532.6 eV (A). As for the minor peak at 532.6 eV (peak A
in Fig. 2(a)), it was attributed to surface contaminants in other bismuthates
Balandeh et al. (2017), and such contaminants may be present on the surface of
our samples. The binding energy of the main peak for O $1s$ at 529.1 eV (peak
C in Fig. 2(a)) is comparable to the binding energy of O $1s$ in BaBiO3
Balandeh et al. (2017). XPS shows evidence for KOH on the surface of the
sample as evidenced by an observed K peak (shown in Sup. Fig. 2(a)); we cannot
rule out that some of the intensity at at 530.8 eV originates from KOH Favaro
et al. (2016)(peak B in Fig. 2(a)). Furthermore, contamination like
carbonates, which were also observed in the C $1s$ spectrum (see Sup. Fig.
2(a)) might also contribute the peak B. However, analyzing the relative amount
of potassium and carbonate contaminations peak B cannot be purely due to those
contaminations, see Sup. Table I. Considering, that the area of the fitted
curve at 530.8 eV is comparable to the area of the main peak at 529.1 eV
(Table I in SI), we expect the signal to originate from a similar amount of
oxygen atoms on the surface of the Ag2BiO3 samples. Looking at the different
O-O and Bi-O distance for oxygen atoms at the shared edges and corners of the
BiO6 octahedra, we propose that the peaks at 530.8 eV and 529.1 eV may
originate from different states in the oxygen atoms at the shared corners and
the shared edges of the BiO6 octahedra.

Figure 4: (a) The DFT (LDA) band structures of Ag2BiO3 in the $Pnn2$ phase.
The contribution from the O-$a_{1g}$ molecular orbital of the small (purple)
and large (brown) octahedra are represented with fat bands. The colors for
octahedra are matching the Ag2BiO3 unit cell presented in Fig. 1(c). (b) The
band structures of BaBiO3 in the $P2_{1}/n$ phase. The contribution from the
O-$a_{1g}$ molecular orbital of the small (red) and large (blue) octahedra are
represented with fat bands. The colors for octahedra are matching the BaBiO3
unit cell presented in Fig. 1(e). (c) Partial Density of States (PDOS) of
Ag2BiO3, showing the contribution from Ag, Bi, and O. (d) PDOS of BaBiO3,
showing the contribution of Ba, Bi, and O.

Assigning distinct oxygen atoms as the origin for the different XPS peaks, the
peak at 529.1 eV may originate from the dimerized oxygen at the shared edges
of the octahedra with a bond length of 2.6588 Å, while the contribution to the
peak at 530.8 eV from Ag2BiO3 likely comes from the corner shared oxygens.
Another possibility is that the different oxygen states at 529.1 and 530.8 eV
arise from the interaction of the oxygen atoms with silver atoms at different
distances, where the oxygen atoms at the shared corners of the octahedra are
further away from the nearest silver atoms, while the oxygen atoms at the
shared edges are closer to silver atoms. If all oxygen atoms in Ag2BiO3 have
equivalent Bi-O interaction, then the electronic origin for the split peaks at
529.1 and 530.8 eV depends on the extent of the O $1s$ core-hole potential and
the Bi $6s$/O $2p$ hybridization Balandeh et al. (2017).

The Bi $4f$ spectra, shown in Fig. 2(c), support the presence of two Bi states
in this compound, and these states have similar binding energy (158.0 eV for
peak G2 and 158.5 eV for peak F2). The ratio of the areas of the two peaks is
nearly 1:1, suggesting that the two different Bi states are due to the two
distinct Bi atoms in the unit cell. It might be possible that one of the Bi
states is due to surface contaminations like hydroxides or carbonates.
However, this would imply that the 1:1 ratio of the two Bi states observed is
mere coincidence. Furthermore, if one of the components would be due to
contamination the amount of Bi would be too little compared to the Ag amount
(see Table I in SI). For now, we consider the two Bi peaks observed as
originating from Ag2BiO3. Comparing the measured binding energies with those
for Bi5+ like in NaBiO3 Kulkarni et al. (1990) and Bi3+ like in Bi2O3 Hegde et
al. (1989) we realize that both are close to the value for Bi3+. This might
hint that the system is bond disproportionated rather than charge
disproportionated. Please note that the Bi $4f$ binding energies are similar
to the one observed in BaBiO3 Plumb et al. (2016) but the Bi $4f$ spectrum for
Ag2BiO3 shows a clearer asymmetry.

The XPS spectra thus indicate two different Bi sites, consistent with a
disproprotionated state, and optical conductivity evidence for a band gap
(Section IV.2) support this interpretation. However, whether Ag2BiO3 is charge
or bond disproportionated remains unclear without examining the XAS data. XAS
results at the oxygen $K$-edge, presented in Fig. 3(a), show a sharp pre-peak
at 529 eV, which originates from a high hole density in the oxygen $p$
orbitals. This is similar to the results on BaBiO3 Balandeh et al. (2017). The
higher energy region at the $K$-edge is clearly different from the BaBiO3
spectrum, reflecting the different oxygen environment in Ag2BiO3. A second
peak is seen in the XAS results of Ag2BiO3, and the two XAS peaks (marked with
yellow and blue circles in Fig. 3(a)) are $\sim$2 eV apart. This energy
difference is similar to peaks B and C in the oxygen XPS spectra.

The combination of corner and edge sharing octahedra in Ag2BiO3 can contribute
to the different XAS spectra above 530 eV. Another peak at 531 eV is seen for
Ag2BiO3, and the origin of this peak is the higher energy states of O in
Ag2BiO3. If we compare Figs. 4(c) and (d), we see a peak in the oxygen partial
density of states at 2 eV in Ag2BiO3 that is much less intense than in BaBiO3.
In the ionic-limit, we have Ag1+–$4d^{10}$, Bi3+–$6s^{2}$, and O2-–$2p^{6}$
ions making up Ag2BiO3, where the BiO6 octahedron are either made up of
Bi3+($6s^{2}$)–O bonds with longer bonds or Bi3+($6s^{2}\underline{L}^{2}$)–O
with shorter bonds. The ligand hole pairs ($\underline{L}^{2}$) condense on
these smaller octahedra, where the O atoms have equal charge on them. The
disproportionation leading to the small and large octahedra in this case is
described as bond-disproportionation.

One is tempted to conclude a similar origin for both of these sets of peaks,
such as the difference between oxygen atoms at the shared edges and corners of
the BiO6 octahedra. However, the area under the curve for the XPS peaks is
equivalent for B and C. Reconciling the XAS and XPS results, if the oxygen
atoms are equivalent in terms of interaction with Bi and the split in the O
$1s$ XPS is electronic in nature, then an oxygen ligand hole will be
distributed over 6 oxygen atoms in the BiO6 octahedron. In the case of O $1s$,
the XPS peak splitting originates from oxygen atoms at different distances to
bismuth, which suggests an oxygen hole may be distributed over 4 oxygen atoms
at the shared edges of the BiO6 octahedra. The oxygen hole ligand distribution
in the octahedra of Ag2BiO3 is discussed below in section IV.3.

###  IV.2  Band Gap and Topology
in Disproportionated Ag2BiO3

The band structure of Ag2BiO3 in the $Pnn2$ phase resulting from DFT
calculations is presented in Fig. 3(d), where we have a band gap of $\sim
1.25$ eV resulting from hybridization of O $2p$ bands and Bi $6s$ band, with
anti-bonding character. The second set of bands above the Fermi level
($E_{\rm{F}}$) is located $\sim 3$ eV above $E_{\rm{F}}$ and corresponds to
the unfilled Bi $6p$ bands. The band gap from optical conductivity measurement
is around $\sim 1.25$ eV (Fig. 3(b)), transition between bands marked with an
orange arrow, which corresponds well with transitions expected in our
calculated band structure, marked with an orange arrow in Fig. 3(d). The real
and imaginary parts of the dielectric constant of Ag2BiO3 are shown in Sup.
Fig. 3. The upturn starting at $\sim 2.5$ eV in the optical conductivity
corresponds to transitions from bands below $E_{\rm{F}}$ to the empty Bi $6p$
bands in Ag2BiO3, which are marked with green arrows in Figs. 3(b) and (d).
The consistency between the features seen in our optical conductivity
measurement and the expected transitions in our calculated band structure
provide further evidence for Ag2BiO3 being in the disproportionated $Pnn2$
phase. In the $Pnna$ phase Ag2BiO3 is a semimetal and expected to have a
nonzero density of state at the Fermi level He et al. (2018), so no gap in the
optical conductivity.

We emphasize that the non-symmorphic space group $Pnn2$ (Space Group 34) of
Ag2BiO3 leads to symmetry-enforced nodal loops in all bands, i.e., twofold
Weyl crossings along 1D loops. These nodal loops are protected by the glide
mirror symmetries {m010$|$$\tfrac{1}{2}$ $\tfrac{1}{2}$ $\tfrac{1}{2}$} and
{m100$|$$\tfrac{1}{2}$ $\tfrac{1}{2}$ $\tfrac{1}{2}$} and touch each other on
the intersection lines of the mirror planes, thereby forming “nodal chains”
Bzdušek et al. (2016); Chan et al. (2019); Hirschmann et al. (2021), see Fig.
3(c). These Weyl nodal lines carry a nontrivial $\pi$-Berry phase, leading to
drumhead surface states. The Weyl nodal-chain feature is not removable by
perturbations as long as the glide mirror symmetries are preserved. In our
band calculations of Fig. 3(d) these Weyl nodal chains are seen in various
bands as crossing points close to high-symmetry points (see, e.g, crossings
marked by blue circles close to the X, Y, and T points). To access the
anomalous magnetotransport properties of the nodal chains Bzdušek et al.
(2016), we may be tempted to use chemical doping to shift the Fermi level and
approach these band crossings. However, if the chemical doping changes the
$Pnn2$ symmetry, then these topological features may be annihilated. In
addition, we note that at the S and Z points the bands form fourfold
degeneracies, composed of two Weyl points with opposite chirality (yellow dots
in Fig. 3(c)).

Including hybrid functionals in our calculations results in an increase in the
band gap for Ag2BiO3 in comparison with calculations in the literature He et
al. (2018) with a direct gap of $\sim~{}1.0$ eV at the $R$ point in the band
structure. However, this value remains slightly lower than the experimentally
estimated value of $\sim~{}1.25$ eV from our optical conductivity measurement.
This may be limited by the optically active transitions in Ag2BiO3 that do not
correspond to the lowest band gap in our band structure, or the inadequacy of
the band calculations in providing accurate band gap values even with hybrid
functionals. Ag2BiO3 in the $Pnn2$ is predicted to be a good photovoltaic
material Fabini et al. (2019), and our results clarifying the band gap
experimentally are an important step forward. The $\sim~{}1.25$ eV band gap in
Ag2BiO3 falls within 1.1-1.45 eV range where maximum efficiency is expected
for single junction solar cells Kirchartz and Rau (2018). XPS results, shown
in Sup. Fig. 2(b), suggest the valence band is $\sim 0.9$ eV below the Fermi
level, and this shows promise for introducing in-gap-states in Ag2BiO3. In
future studies, we will need to examine the stability of Ag2BiO3 when exposed
to different radiation to assess its potential in photovoltaic applications.

###  IV.3 Comparing Ag2BiO3 and BaBiO3

Next, we would like to highlight the extent of similarity and some of the
differences between Ag2BiO3 and BaBiO3. Both materials have a bulk band gap
due to disproportionation of “Bi4+”, but the octahedra are only corner sharing
in BaBiO3 while the octahedra are sharing edges and corners in Ag2BiO3. The
nominal Bi4+ oxidation state that leads to the disproportionated charge/bonds
in these compounds is achieved through a different number of atoms. While
having one Bi atom and three oxygen atoms, it takes two monovalent Ag1+ ions
or one divalent Ba2+ ion to achieve the same average formal charge of 4+ for
Bi. Thus, (Ag1+)2(Bi4+)(O2-)3 and (Ba2+)(Bi4+)(O2-)3 have the same unstable
average formal charge for Bi in different crystal structures, and lead to
disproportionation in both Ag2BiO3 and BaBiO3. However, similar to BaBiO3,
there is a significant amount of oxygen character in the empty conduction
states of Ag2BiO3 above the Fermi level. To be more specific, these states are
formed by oxygen molecular orbitals with $a_{1g}$ symmetry centered on the
_small_ BiO6 octahedra. In Fig. 4 (a), they are highlighted as purple fat
bands for Ag2BiO3, while in Fig. 4 (b), they are highlighted as red fat bands
for BaBiO3. In the previous theoretical studies on BaBiO3Foyevtsova et al.
(2015); Khazraie et al. (2018), this situation was referred to as a bi-
polaronic condensation of oxygen holes. For Ag2BiO3 we show in Sup. Fig. 4 the
orbital contribution from Bi-$s$ and $p$ in the small and large O6 octahedral
cages, as well as the O-2$p$. The significant contribution of O-2$p$ above the
Fermi level is consistent with high density of oxygen holes as evidenced by
the pre-peak in XAS (Fig. 3(a)).

Figure 5: (a) Raman spectra of Ag2BiO3 at different $T$, between 290–10 K. (b)
The same Raman spectra in (a), emphasizing the peak at 520 cm-1,
characteristic of the breathing mode in disproportionated bismuthates. (c)
Powder-XRD of Ag2BiO3 collected at different $T$, between 293–203 K. The
calculated XRD pattern for Ag2BiO3 in the $Pnn2$ phase is plotted at the
bottom.

As one can also see in Fig. 4, the valence state within $\sim$1 eV range below
the Fermi level has, in turn, a strong character of O$\textendash a_{1g}$
molecular orbitals centered on the _large_ BiO6 octahedra (brown fat bands for
Ag2BiO3 and blue fat bands for BaBiO3). Together, these two sets of oxygen
molecular orbital states around the Fermi level constitute the anti-bonding
manifold resulting from the strong hybridization between the Bi$\textendash
6s$ and O$\textendash a_{1g}$ orbitals. The latter appears to be as strong in
Ag2BiO3 as it was found to be in BaBiO3, which is evident from the fact that
in both systems the bonding combination is pushed down as far as -10 eV. A new
feature of the Ag2BiO3 band-structure is the presence of Ag$\textendash 4d$
bands in the same energy region where the non-bonding oxygen bands are
located, from $\sim$-7 eV up to the Fermi level (Shown in Sup. Fig. 4(a)). It
was demonstrated in BaBiO3 that the octahedral angle between Bi-O6 octahedra
effects the Bi-O hybridization Khazraie et al. (2018). Band calculations in
BaBiO3 with a 0∘ and 16∘ octahedral tilting show bands with narrower energy
dispersion with increased tilt angle. The conduction band in Ag2BiO3 is
considerably narrower compared to that in BaBiO3, which is probably a result
of weaker Bi-O hybridization due to the geometrical arrangement of the BiO6
octahedra in Ag2BiO3. In Ag2BiO3, the edge-sharing Bi-O6 octahedra along the
$a$ direction, and the high angle of 36∘ between the Bi-O-Bi at the shared
corners along the $b$ and $c$ directions contribute to reduced hybrization.
This qualitative statement about the hybridization should be confirmed with
DFT calculations in future studies. The Ag might play a role in reducing the
band gap in Ag2BiO3, as seen in the Ag-containing pyrochlore oxides Mizoguchi
and Woodward (2004). This narrowing effect is not clear when we compare
Ag2BiO3 with BaBiO3 due to the different crystal structure, but replacing the
Ag with another monovalent atom, such as Na2BiO3, can help us to study the
effect of Ag. The contribution from Ag bands near the Fermi level is shown in
the partial density of states (PDOS) of Ag2BiO3 presented in Fig. 4(c). In
comparison, the PDOS of BaBiO3, shown in Fig. 4(d), has significant
contribution from only Bi and O near the Fermi level and no significant
contribution from the Ba.

While in BaBiO3 we have an indirect band gap as shown in Fig. 4 (b), in
Ag2BiO3 the band gap is direct as shown in Fig 3(d) and Fig. 4 (a). The role
of the Ag$\textendash 4d$ bands that extend over a wide energy range below the
Fermi level is not clear for now. Identifying the effect of edge sharing BiO6
octahedra, crystal symmetry, and Ag bands on the band structure of Ag2BiO3
will be an important part of future studies. Future work should include
replacing Ag with other monovalent metal $M^{+1}$ atoms, varying tilting angle
of Bi-O6 octahedra, and studying the lower symmetry $Pn$ phase.

The Raman spectra of Ag2BiO3 have a strong phonon mode at 520 cm-1, shown in
Fig. 5, which is similar to the phonon mode observed in pristine BaBiO3 at 570
cm-1 Tajima et al. (1992). In the case of BaBiO3, the phonon is attributed to
the breathing mode originating from local distortion in the bond
disproportionated BaBiO3, so it is likely that a similar energy phonon
originates from a similar breathing mode accompanying the disproprotionation
in Ag2BiO3. The slight difference in energies between the phonon modes in
Ag2BiO3 and BaBiO3 is due to the different arrangement of the BiO6 octahedra
in the two compounds and the slightly different Bi-O distances. A higher
energy phonon mode originating from higher harmonic phonon modes can be seen
in the Raman spectra of Ag2BiO3 at 1100 cm-1, which is similar in value to
phonons in the spectra of pristine BaBiO3 Tajima et al. (1992).

###  IV.4 Transition to $Pn$ Phase in Ag2BiO3

We have closely measured the phonon mode at 520 cm-1 as a function of
temperature between 290 K and 10 K, and have observed a shift in the phonon
peak position cooling down from 290 K to 250 K, but no further shift in the
phonon mode cooling down to 10 K. We have performed powder XRD measurements on
Ag2BiO3 at low $T$ and observe a splitting in some peaks at 240 K that
corresponds to the monoclinic distortion previously reported Oberndorfer et
al. (2006). This distortion leads to reduction of symmetry of Ag2BiO3 from
$Pnn2$ to $Pn$.

As explained in Sec. IV.1, it is difficult to distinguish whether Ag2BiO3
crystallizes in the $Pnn2$ or the $Pnna$ phase from XRD alone, and neutron
diffraction studies were required to clarify the $Pnn2$ phase at room
temperature Oberndorfer et al. (2006). The band gap and splitting in the Bi
XPS states measured on our samples suggest that Ag2BiO3 is indeed in the
$Pnn2$ phase. However, the $Pn$ phase of Ag2BiO3 is distinct from the higher
symmetry phases and can be identified with XRD. The splitting of the (202)
peak at $\sim 12.6^{\circ}$ for temperatures $<243$ K is highlighted with a
blue circle in Fig. 5(c), which results from the reduced symmetry of Ag2BiO3
$Pnn2$ to $Pn$.

It is likely that the shift in the phonon mode observed in Raman measurements
at 250 K is related to the phase transition observed in powder XRD. The
transition observed in our samples is consistent with the previously reported
phase transition in neutron diffraction measurements on Ag2BiO3 Oberndorfer et
al. (2006). In future studies, we hope to investigate the effect of different
synthesis conditions on the oxygen content in Ag2BiO3, and whether this
influences the distortion in these samples at low $T$.

Measurements that can be used to probe the band structure of Ag2BiO3 such as
scanning tunneling spectroscopy or angle-resolved photoemission spectroscopy
are typically performed at low temperature. Any properties measured on Ag2BiO3
below $\sim 250$ K are representative of the material in the $Pn$ phase, and
the band structure of this phase will be the relevant one for those
measurements. Luckily, the $Pn$ (Space Group 7) symmetry falls into the non-
symmorphic space groups where a single glide mirror symmetry is expected to
lead to symmetry-enforced Weyl nodal lines Bzdušek et al. (2016); Chan et al.
(2019); Hirschmann et al. (2021), albeit not a nodal chain as in the $Pnn2$
phase.

The disproportionation of Ag2BiO3 is still prevalent in the $Pn$ phase, which
would lead to a band gap much like the $Pnn2$ phase. Repeating the XPS and XAS
measurements at temperatures where Ag2BiO3 is in the $Pn$ phase will help
reveal whether this transition affects the Bi states or the oxygen holes.

##  V Conclusions

We have demonstrated that despite the use of a high pressure of 6 GPa, Ag2BiO3
samples crystallize in a structure consistent with the $Pnn2$ phase as
evidenced by the XPS data for Bi and the measured band gap. The band gap
observed in our study is consistent with our band calculations, and we
demonstrate that Ag2BiO3 even in the $Pnn2$ phase is topologically non-trivial
and contains Weyl-nodal chains in its band structure ($\sim$0.5 eV below the
Fermi level). We observe a phase transition in Ag2BiO3 at low temperatures
associated with distortion from the $Pnn2$ phase to the $Pn$ phase from our
XRD pattern and Raman spectra. Even in the $Pn$ phase Ag2BiO3 falls into a
non-symmorphic space group and Weyl nodal-line features are expected.

The XAS spectra of Ag2BiO3 suggests the presence of oxygen holes much like
BaBiO3, which is consistent with Ag2BiO3 being a bond disproportionated
bismuthate. Despite the different arrangement of BiO6 octahedra in BaBiO3 and
Ag2BiO3, we observe a phonon mode in the Raman spectra of Ag2BiO3 at similar
energy associated with breathing mode of the disproportionated BaBiO3. Our
results push forward the group of monovalent bismuthates $(M^{+1})_{2}$BiO3 as
a new class of bismuthates with novel topological properties. Doping of
divalent atoms on the Ag-site and performing in-situ high-pressure experiments
may allow us to destabilize the disproportionation in Ag2BiO3, and achieve a
metallic state and perhaps even superconductivity in this class of materials
in analogy to BaBiO3. We envision that this work will open up the field of
disproportionated oxides beyond those in the perovskite structure.

##  VI ACKNOWLEDGEMENTS

We thank Frank Falkenberg, Christine Stefani, Sebastian Bette, and Armin
Schulz for technical assistance with experiments. We thank Tom Regier for help
with XAS measurements. Research described in this paper was performed at the
Canadian Light Source, which is supported by the Canada Foundation for
Innovation, Natural Sciences and Engineering Research Council of Canada, the
University of Saskatchewan, the Government of Saskatchewan, Western Economic
Diversification Canada, the National Research Council Canada, and the Canadian
Institutes of Health Research. We thank the Max Planck-UBC-UTokyo Center for
Quantum Materials for support.

## References

  * Bednorz and Müller (1986) J. G. Bednorz and K. A. Müller, Z. Phys. B 64, 189 (1986).
  * Mattheiss et al. (1988) L. Mattheiss, E. Gyorgy, and D. Johnson Jr, Phys. Rev. B 37, 3745 (1988).
  * Seki et al. (2012) S. Seki, X. Yu, S. Ishiwata, and Y. Tokura, Science 336, 198 (2012).
  * Kargarian and Fiete (2013) M. Kargarian and G. A. Fiete, Phys. Rev. Lett. 110, 156403 (2013).
  * Ishiwata et al. (2020) S. Ishiwata, T. Nakajima, J.-H. Kim, D. Inosov, N. Kanazawa, J. White, J. Gavilano, R. Georgii, K. Seemann, G. Brandl, et al., Phys. Rev. B 101, 134406 (2020).
  * He et al. (2018) J. He, D. Di Sante, R. Li, X.-Q. Chen, J. M. Rondinelli, and C. Franchini, Nat. Commun. 9, 492 (2018).
  * Fu et al. (2018) B. Fu, X. Fan, D. Ma, C.-C. Liu, and Y. Yao, Phys. Rev. B 98, 075146 (2018).
  * Singh et al. (2018) B. Singh, B. Ghosh, C. Su, H. Lin, A. Agarwal, and A. Bansil, Phys. Rev. Lett. 121, 226401 (2018).
  * Deibele and Jansen (1999) S. Deibele and M. Jansen, J. Solid State Chem. 147, 117 (1999).
  * Oberndorfer et al. (2006) C. P. Oberndorfer, R. E. Dinnebier, R. M. Ibberson, and M. Jansen, Solid State Sci. 8, 267 (2006).
  * Hegde et al. (1989) M. Hegde, P. Barboux, C. Chang, J. Tarascon, T. Venkatesan, X. Wu, and A. Inam, Phys. Rev. B 39, 4752 (1989).
  * Quackenbush et al. (2013) N. Quackenbush, J. W. Tashman, J. A. Mundy, S. Sallis, H. Paik, R. Misra, J. A. Moyer, J.-H. Guo, D. A. Fischer, J. C. Woicik, et al., Nano Lett. 13, 4857 (2013).
  * Swift (1982) P. Swift, Surf. Interface Anal. 4, 47 (1982).
  * Barr and Seal (1995) T. L. Barr and S. Seal, J. Vac. Sci. Technol. A 13, 1239 (1995).
  * Moulder and Chastain (1992) J. Moulder and J. Chastain, _Handbook of X-ray Photoelectron Spectroscopy: A Reference Book of Standard Spectra for Identification and Interpretation of XPS Data_ (Physical Electronics Division, Perkin-Elmer Corporation, 1992).
  * Perdew et al. (1996) J. P. Perdew, K. Burke, and Y. Wang, Phys. Rev. B 54, 16533 (1996).
  * Blaha et al. (2001) P. Blaha, K. Schwarz, G. K. Madsen, D. Kvasnicka, and J. Luitz, An augmented plane wave+ local orbitals program for calculating crystal properties (2001).
  * Perdew and Wang (1992) J. P. Perdew and Y. Wang, Phys. Rev. B 45, 13244 (1992).
  * Kennedy et al. (2006) B. J. Kennedy, C. J. Howard, K. S. Knight, Z. Zhang, and Q. Zhou, Acta Crystallogr. B 62, 537 (2006).
  * Foyevtsova and Sawatzky (2019) K. Foyevtsova and G. Sawatzky, J. Mod. Phys. 10, 953 (2019).
  * (21) _$\rm{See~{}Supplemental~{}Material~{}for~{}further~{}detail.}$_.
  * Balandeh et al. (2017) S. Balandeh, R. J. Green, K. Foyevtsova, S. Chi, O. Foyevtsov, F. Li, and G. A. Sawatzky, Phys. Rev. B 96, 165127 (2017).
  * Favaro et al. (2016) M. Favaro, B. Jeong, P. N. Ross, J. Yano, Z. Hussain, Z. Liu, and E. J. Crumlin, Nat. Commun. 7, 12695 (2016).
  * Kulkarni et al. (1990) G. Kulkarni, V. Vijayakrishnan, G. Ranga Rao, R. Seshadri, and C. Rao, Appl. Phys. Lett. 57, 1823 (1990).
  * Plumb et al. (2016) N. Plumb, D. Gawryluk, Y. Wang, Z. Ristić, J. Park, B. Lv, Z. Wang, C. Matt, N. Xu, T. Shang, et al., Phys. Rev. Lett. 117, 037002 (2016).
  * Bzdušek et al. (2016) T. Bzdušek, Q. Wu, A. Rüegg, M. Sigrist, and A. A. Soluyanov, Nature 538, 75 (2016).
  * Chan et al. (2019) Y.-H. Chan, B. Kilic, M. M. Hirschmann, C.-K. Chiu, L. M. Schoop, D. G. Joshi, and A. P. Schnyder, Phys. Rev. Mater. 3, 124204 (2019).
  * Hirschmann et al. (2021) M. M. Hirschmann, A. Leonhardt, B. Kilic, D. H. Fabini, and A. P. Schnyder, _Symmetry-enforced band crossings in tetragonal materials: Dirac and weyl degeneracies on points, lines, and planes_ (2021), eprint 2102.04134.
  * Fabini et al. (2019) D. H. Fabini, M. Koerner, and R. Seshadri, Chem. Mater. 31, 1561 (2019).
  * Kirchartz and Rau (2018) T. Kirchartz and U. Rau, Adv. Energy Mater. 8, 1703385 (2018).
  * Foyevtsova et al. (2015) K. Foyevtsova, A. Khazraie, I. Elfimov, and G. A. Sawatzky, Phys. Rev. B 91, 121114 (2015).
  * Khazraie et al. (2018) A. Khazraie, K. Foyevtsova, I. Elfimov, and G. A. Sawatzky, Phys. Rev. B 97, 075103 (2018).
  * Mizoguchi and Woodward (2004) H. Mizoguchi and P. M. Woodward, Chem. Mater. 16, 5233 (2004).
  * Tajima et al. (1992) S. Tajima, M. Yoshida, N. Koshizuka, H. Sato, and S. Uchida, Phys. Rev. B 46, 1232 (1992).

Electronic Structure of the Bond Disproportionated Bismuthate Ag2BiO3\-
Supplemental Material

Supplementary Figure 1: Powder X-ray diffraction (PXRD) and powder neutron diffraction (PND) simulation Ag2BiO3 in the $Pnn2$ phase shown in (a) and (b), respecively. PXRD and PND simulation of Ag2BiO3 in the $Pnna$ phase shown in (c) and (d), respectively. Diffraction patterns are plotted as a function of the reciprocal lattice vector (Q). Black circles in the figures highlight one of the peaks only observable in the PND for the $Pnn2$ phase. Supplementary Figure 2: X-ray photoelectron spectra of K $2p$ and C $1s$ (a) and of the valence band of Ag2BiO3 (b). In (b) the valence band edge is approx. 0.9 eV below the Fermi level. Supplementary Figure 3: Imaginary ($\epsilon_{2}$, top figure) and real ($\epsilon_{1}$, bottom figure) parts of the dielectric constant of Ag2BiO3 extracted from ellipsometry measurement. Supplementary Figure 4: (a) The DFT (LDA) band structures of Ag2BiO3 in the $Pnn2$ phase with fat band representation of atomic orbital contributions. (a) The atomic orbital of the Ag-$4d$ shown in grey fat bands. (b) The fat bands of Bi-$s$ (green) and Bi-$p$ (purple) contribution in the small O6 octahedral cage. (c) The fat bands of Bi-$s$ (green) and Bi-$p$ (brown) contribution in the large O6 octahedral cage. (d) The atomic orbital of the O-$2p$ shown in pink fat bands. Supplementary Table 1: Binding energy, assignment, full width at half-maximum (FWHM) and amount of the species observed via XPS. Please note that due to surface contaminations, error in the detector function and atomic sensitivity the amount might have an error in the range of 10-15%. element | Assignment | Binding energy [eV] | FWHM [eV] | Amount in [%]
---|---|---|---|---
O $1s$ | Ag2BiO3 edge shared (peak C) | 529.1 | 1.1 | 11.9
O $1s$ |  | Ag2BiO3 corner shared (peak B)
---
(small amounts of OH, carbonate contaminations)
530.8 | 1.7 | 11.8
O $1s$ | Water, hydroxides (peak A) | 532.6 | 1.7 | 1.9
C $1s$ | C-C, C-H contaminations | 284.8 | 1.5 | 8.4
C $1s$ | C-OH contaminations | 285.7 | 2.2 | 6.0
C $1s$ | carbonate | 288.5 | 1.7 | 4.0
K $2p_{3/2}$ | KOH | 292.6 | 1.5 | 2.5
Ag $3d_{5/2}$ | Ag2BiO3 (peak D2) | 367.8 | 1.2 | 16.1
Bi $4f_{7/2}$ | Ag2BiO3 (peak G2) | 158.0 | 0.9 | 4.4
Bi $4f_{7/2}$ | Ag2BiO3 (peak F2) | 158.5 | 1.4 | 5.2