# Retrieval-based Knowledge Augmented Vision Language Pre-training

Jiahua Rao  Zifei Shan  Longpo Liu  Yao Zhou  Yuedong Yang

###### Abstract

With recent progress in large-scale vision and language representation
learning, Vision Language Pretraining (VLP) models have achieved promising
improvements on various multi-modal downstream tasks. Albeit powerful, these
pre-training models still do not take advantage of world knowledge, which is
implicit in multi-modal data but comprises abundant and complementary
information. In this work, we propose a REtrieval-based knowledge Augmented
Vision Language Pre-training model (REAVL), which retrieves world knowledge
from knowledge graphs (KGs) and incorporates them in vision-language pre-
training. REAVL has two core components: a knowledge retriever that retrieves
knowledge given multi-modal data, and a knowledge-augmented model that fuses
multi-modal data and knowledge. By novelly unifying four knowledge-aware self-
supervised tasks, REAVL promotes the mutual integration of multi-modal data
and knowledge by fusing explicit knowledge with vision-language pairs for
masked multi-modal data modeling and KG relational reasoning. Empirical
experiments show that REAVL achieves new state-of-the-art performance
uniformly on knowledge-based vision-language understanding and multimodal
entity linking tasks, and competitive results on general vision-language tasks
while only using 0.2% pre-training data of the best models.

Machine Learning, ICML

##  1 Introduction

Recent Vision-Language Pre-training (VLP) models, such as ALBEF (Li et al.,
2021), BLIP (Li et al., 2022b) and SimVLM (Wang et al., 2021b), learn multi-
modal representations from large-scale image-text pairs via well-designed pre-
training objectives. However, despite the strong performance of these models,
they generally cannot well incorporate the external world knowledge in pre-
training, requiring increasingly larger networks or data to cover more facts
with huge computational costs. Meanwhile, large knowledge graphs (KGs), such
as Wikidata (Vrandečić & Krötzsch, 2014) and ConceptNet (Speer et al., 2017),
can provide structured complementary world knowledge to multi-modal data by
representing entity descriptions and their relationships, which is implicit in
vision/text but comprises abundant and useful information (Wang et al., 2021a;
Yasunaga et al., 2022). Therefore, leveraging structured knowledge and multi-
step reasoning in KGs can really complement multi-modal representation
learning.

However, building connections between knowledge graphs and multi-modal data
remains a stiff challenge. Previous studies, such as ERNIE (Zhang et al.,
2019) and KEPLER (Wang et al., 2021a), have incorporated knowledge into pre-
training language models explicitly, but most of them are restricted by the
knowledge behind the textual modality, neglecting a large amount of knowledge
in other modalities like images. More importantly, these models do not fully
leverage multi-modal data and self-supervised learning to facilitate knowledge
retrieval during pre-training. For example, DRAGON (Yasunaga et al., 2022)
only focuses on the text modality and requires linking text and entities
before pre-training, while MuRAG (Chen et al., 2022a) incorporates knowledge
only for language generation tasks. Consequently, existing works may limit
their potential to incorporate informative knowledge into multi-modal data.

To this end, we propose a REtrieval-based knowledge Augmented Vision-Language
Pre-training (REAVL) model, which retrieves world knowledge from KG to
complement multi-modal data and incorporates them in vision-language pre-
training. REAVL has two core components: a knowledge retriever that retrieves
KG entities given multi-modal data, and a knowledge-augmented model that fuses
multi-modal data and knowledge. Concretely, we take multi-modal data and a KG
as the input data, and first retrieve the related knowledge and construct
entity subgraph from the KG via the knowledge retriever. Then we use a
knowledge-augmented model to encode these inputs into fused representations,
where the multi-modal features are fused with knowledge features through
cross-attention at each layer.

To jointly model deep interactions over multi-modal data and knowledge, we
pretrain our model by unifying four different types of knowledge-aware self-
supervised tasks: masked language modeling (MLM), masked vision modeling
(MVM), KG link prediction (LinkPred), and image-text contrastive learning
(ITC). Specifically, MLM and MVM are the knowledge-aware masked data modeling
objectives, where the original signal is reconstructed by using its masked
modal inputs and the corresponding retrieved knowledge. The masked learning
signals can reward the helpful knowledge and penalize uninformative ones
during knowledge retrieval. Furthermore, LinkPred makes the model use the KG
structure jointly with the corresponding multi-modal data to reason about
missing links in KG and ITC focuses on the alignment of the image and text in
multi-modal data. We conduct ablation studies and show the effectiveness of
the individual self-supervised tasks.

We benchmark REAVL on multiple vision-language benchmarks, including general,
knowledge-based and multimodal entity linking (MEL) tasks. REAVL achieves
state-of-the-art performance on knowledge-based and MEL tasks. On general
tasks, REAVL also scores the best within models trained with a similar amount
of data, competitive to models trained on billions of images while using only
0.2% of their data, showing the strong sample efficiency of our methodology.

To summarize, we make the following contributions:

  * •

We first incorporated the knowledge graph into the vision-language pre-
training model, to complement multi-modal representation learning.

  * •

We introduced a knowledge retriever and a knowledge-augmented model to jointly
retrieve informative knowledge for multi-modal understanding.

  * •

We unified four knowledge-aware self-supervised tasks to promote the mutual
integration of multi-modal data and knowledge, and demonstrated state-of-the-
art performance on various downstream tasks.

Figure 1: Illustration of the REAVL model. Given multi-modal data and a large
knowledge graph (KG), we utilize a Knowledge Retriever to retrieve knowledge
from KG from multi-modal data and then incorporate the retrieved knowledge in
multi-modal representation learning with a Knowledge-Augmented Model. We also
unify four types of knowledge-aware self-supervised tasks (i.e. Masked
Language Modeling) to encourage multi-modal data and KG to mutually inform
each other.

##  2 Related Work

###  2.1 Vision-Language Pre-training Model

Vision-Language Pre-training has been developing rapidly recent years. Most
existing work focus on modeling the interactions between image and text with
transformer-based encoders. And multiple well-designed cross-modal pre-
training objectives have been proposed, for example, image-text matching,
masked data modeling, and contrastive learning. They have achieved their
impressive performance by scaling up the dataset with noisy image-text pairs
collected from the web. For example, BLIP (Li et al., 2022b) effectively
utilizes the noisy web data by bootstrapping the captions, where a captioner
generates synthetic captions and a filter removes the noisy ones. Recent
methods such as SimVLM (Wang et al., 2021b), OFA (Wang et al., 2022a), CoCa
(Yu et al., 2022), and PaLi (Chen et al., 2022b) unified the multi-modal pre-
training model and self-supervised tasks to obtain state-of-the-art
performance on various downstream tasks. These methods or frameworks benefit
from their learning ability on the increasing web-collected pre-training data
(Changpinyo et al., 2021; Jia et al., 2021), which also creates the challenge
of huge computational costs, and thus complicates the optimization procedure.

Our work, by contrast, focuses on incorporating world knowledge into multi-
modal representations, which have been demonstrated helpful and necessary for
various downstream tasks. By introducing knowledge into multi-modal learning,
our model can achieve better performance with low-resource pre-training data.

###  2.2 Knowledge Augmented Language Model

Knowledge integration is active research for improving language models. One
line of research aims to add entity features into language models (LMs) (Zhang
et al., 2019; Wang et al., 2021a). For example, ERNIE (Zhang et al., 2019)
enhanced the text representations with their corresponding named entity
mentions at the text token level. A recent method DRAGON (Yasunaga et al.,
2022) bidirectionally fuses text and KG via a deep cross-modal model and
learns joint reasoning over text and KG. These methods usually require entity
linking tools to link text corpora and KG entities before training, and the
retrieved knowledge is not updated during training.

Another line of work is retrieval-augmented LMs (Guu et al., 2020; Lewis et
al., 2020), which retrieve relevant text from a corpus and integrate it into
LMs as additional knowledge. For example, REALM (Guu et al., 2020) integrates
Wikipedia passages as a knowledge base to benefit downstream knowledge-
intensive tasks. However, current retrieval-augmented methods are often
restricted to using text-only knowledge, neglecting an amount of knowledge in
other modalities like images, which contain much information not covered by
text. Compared to them, MuRAG (Chen et al., 2022a) is the first retrieval
model that is capable of retrieving knowledge presented in multiple
modalities.

To improve on the above works, we propose to effectively retrieve informative
knowledge from multiple modalities and to incorporate knowledge with multi-
modal representations by knowledge-aware self-supervision learning. The
fundamental rationale of our method is that it improves the retrieval steps
through well-designed self-supervised learning tasks. Another distinction is
that the existing retrieval-based models (w.r.t MuRAG) typically focus on
adding entity- or triplet-level knowledge rather than the subgraphs around the
retrieved entity to enable richer knowledge fusion.

##  3 Methods

In this section, we first briefly introduce the model architecture of our
approach in Section 3.1 and Figure 1. Then we describe the proposed Knowledge
Retrieval, followed by the GNN Encoder for the retrieved knowledge (Section
3.2-3.3). In Section 3.4, we show how to effectively incorporate multi-modal
data with knowledge by the Knowledge-Augmented Model. Lastly, we describe the
pre-training objectives with corresponding self-supervised tasks (Section
3.5).

###  3.1 Model Architecture

At the core of our method is the idea of incorporating multi-modal
representations with KGs. To achieve this, We defined the image-text pairs as
$\mathcal{D}=\\{(I,T)|I\in\mathcal{I},T\in\mathcal{T}\\}$, and the KG is a
multi-relational graph
$\mathcal{G}=\\{(\mathcal{V},\mathcal{E},\mathcal{R})\\}$ where $\mathcal{V}$
is the set of entity nodes, $\mathcal{E}$ is the set of edges (triplets) and
$\mathcal{R}$ is the set of relation types in the KG.

We first employ a vision encoder $f_{v}$ and a text encoder $f_{t}$ to produce
the representations of the input image $I$ and text $T$. For the vision
encoder, we use a 12-layer visual transformer ViT-B/16 (Dosovitskiy et al.,
2021). An input image $I$ is first divided into $16\times 16$ patches and
encoded into a sequence of patch embeddings
$\upsilon=\\{\upsilon_{1},...,\upsilon_{N}\\}$ where $N$ is the number of
image patches. The text encoder, a 12-layer transformer, is initialized using
the pre-trained BERTbase (Devlin et al., 2018) model and transforms an input
text $T$ into a sequence of embeddings $t=\\{t_{cls},t_{1},...,t_{N_{t}}\\}$,
where $N_{t}$ is the number of text tokens and $t_{cls}$ is the encoding of
the start token of a sentence.

The patch embeddings $\upsilon$ from image $I$ are then used for knowledge
retrieval, which is to find the nearest entities behind the image. The set of
retrieved entities is defined as $M\subset\mathcal{V}$. We then add their one-
hop neighbor nodes to construct the entity subgraph $G\subset\mathcal{G}$ and
feed them into the GNN Encoder. Therefore, we could obtain the retrieved
entity embedding $e=\\{e_{1},...,e_{k}\\}$ after aggregating their neighbor
information.

Finally, the image and the text embedding as well as entity embedding are
further processed by the knowledge-augmented model. The knowledge-augmented
model is initialized using the first and last layer of the BERT model, which
enhances multi-modal representation by interacting with other modalities
through cross-attention blocks.

###  3.2 Knowledge Retrieval

Instead of directly linking entity mentions in the input text to entity nodes
in KGs (Yasunaga et al., 2022), we followed the strategy of REALM (Guu et al.,
2020) by explicitly asking the model to decide what knowledge to retrieve and
incorporate.

In the retrieval step, the image patches with fine-grained descriptions are
helpful for understanding image content and retrieving possibly helpful
entities $M$ from the KG $\mathcal{G}$. Therefore, with the $N$ patches of the
input image $I$, the vision encoder $f_{v}$ is applied to map each patch to a
dense representation
$\upsilon=\\{\upsilon_{0},\upsilon_{1},...,\upsilon_{N}\\}$. For knowledge
entries in KGs $m\in\mathcal{V}$ ($\mathcal{V}$ is the set of entity nodes),
we apply the backbone encoder $f_{e}$ to encode the entity $m$ with its
description. Formally, we define the relevance score between the image patch
$v_{i}$ and the entity $m$ as:

$sim(v_{i},m_{j})=f_{v}(v_{i})^{T}f_{e}(m_{j})$ (1)

In total, we retrieve the top $N\times K$ knowledge entries relevant to image
$I$. We keep top-k knowledge entries ranked by similarity scores as the set of
retrieved entities $M=\\{m_{1},m_{2},...,m_{k}\\}$.

In practice, we use CLIP model (Radford et al., 2021) to encode all of the
entity descriptions as the candidate memory and index them using FAISS
(Johnson et al., 2019) to find the top-K nearest neighbors for each image
patch.

For each resulting set of retrieved entities, we further add their one hop
neighbor nodes from KG $\mathcal{G}$ to construct the entity subgraph $G$. The
subgraph contains richer entities and their relations, enabling the model to
obtain more information about world knowledge.

###  3.3 GNN Encoder

After extracting the 2-hop entity subgraph around the image-text pair, we
applied a GNN encoder to model the representation of informative entities by
aggregating their neighbor information and strengthening the relational
information. To represent the graph, we first obtain initial node embeddings
$\\{e_{1}^{(0)},...,e_{k}^{(0)}\\}$ for the retrieved entities using the
entity encoder. The initial embedding of the relations in the subgraph is also
initialized by their description embeddings.

Then, in each layer of the GNN, the current representation of the node
embeddings $\\{e_{1}^{(l-1)},...,e_{k}^{(l-1)}\\}$ is fed into the layer to
perform a round of information propagation between nodes in the graph and
yield pre-fused node embeddings for each entity:

$\begin{split}\\{e_{1}^{(l)},...,e_{k}^{(l)}\\}={\rm
GNN}(\\{e_{1}^{(l-1)},...,e_{k}^{(l-1)}\\})\end{split}$ (2)

where GNN corresponds to a variant of graph attention networks and $l$
corresponds to the layer of GNN. More details of GNN message passing are in
Appendix A.1.

###  3.4 Knowledge Augmented Model

After using a visual transformer layer, a text transformer layer, and a GNN
layer for vision, text, and entities respectively, we use a knowledge-
augmented model to let the two modalities fuse the knowledge information
through a 2-layer cross-modalities transformer.

In principle, we concatenate the pre-fused embeddings including visual
embeddings $\upsilon$, text embeddings $t$, and entity embeddings $e$ as the
input elements in the BERT model. The input sequence always starts with a
special classification element ([CLS]), then goes on with visual patch
embeddings, then follows up with text token elements, and ends with the
retrieved entities. A special separation element ([SEP]) is inserted between
the visual and text elements, and between the text and entity elements.
Therefore, four types of input elements are involved, namely, visual, textual,
entity, and special elements for disambiguating different input formats.
Finally, the multi-modal features are fused with knowledge features through
the cross-attention block at each layer. More details are in Appendix A.2.

###  3.5 Pre-training objective

To ensure that the multi-modal and KG mutually inform each other, we unify
four knowledge-aware self-supervised tasks: masked language modeling, masked
vision modeling, KG link prediction, and image-text contrastive learning.

#### Masked Language Modeling (MLM).

MLM is a common pretraining task used for language models. Following the
strategy of REALM (Guu et al., 2020), we use Span Masking Modeling to focus on
the span $s$ that may require world knowledge to predict the masked tokens.
Let $\widehat{T}$ denote a masked text, and $p^{msk}(I,\widehat{T},E)$ denote
the model’s predicted probability for a masked token. We adopt the strategy of
(Li et al., 2022b), which minimizes a cross-entropy loss:

$\mathcal{L}_{MLM}=\mathbb{E}_{(I,\widehat{T},E)}[H(y^{msk},p^{msk}(I,\widehat{T},E))]$
(3)

where $y^{msk}$ is a one-hot vocabulary distribution where the ground-truth
token has a probability of 1.

#### Masked Vision Modeling (MVM).

For image masking, unlike MAE (He et al., 2022), we aim to reconstruct the
invisible patches features with visible image, text, and entity features to
facilitate multi-modal information and knowledge fusion. Concretely, we
propose to apply a reconstruction loss to facilitate the aggregation of multi-
modal and knowledge features. Formally, the reconstruction loss is defined as:

$\mathcal{L}_{MVM}=\mathbb{E}_{(\widehat{I},T,E)}[H(y^{msk},p^{msk}(\widehat{I},T,E))]$
(4)

where $\widehat{I}$ denotes the masked patches and $p^{msk}(\widehat{I},T,E))$
denotes the predicted probability for masked image patch features.

#### KG Link Prediction (LinkPred).

While the MLM and MVM task predicts for the multi-modal and knowledge side,
link prediction holds out some edges and predicts them for the input KG. As
our approach takes a joint image-text and KG entities pair as input, we expect
that link prediction can encourage the model to learn to use the KG structure
jointly with the textual context and visual regions to reason about missing
links in the KG.

Concretely, we hold out a subset of edge triplets from the input KG
$S=\\{(h,r,t)\\}\subset E$. Then we maps each entity node ($h$ or $t$) and
relation ($r$) in the KG to a vector, $h$, $t$, $r$, and defines a scoring
function $\phi_{r}(h,t)$ to model positive/negative triplets. Herein, we
consider a KG triplet scoring function $\phi_{r}(h,t)$ as DistMult (Yang et
al., 2015):

$\phi_{r}(h,t)=<h,r,t>$ (5)

where $<\cdot,\cdot,\cdot>$ denotes the trilinear dot product. A higher $\phi$
indicates a higher chance of $(h,r,t)$ being a positive triplet (edge) instead
of negative (no edge). For training, we optimize the objective:

$\begin{split}\mathcal{L}_{LinkPred}=\sum_{(h,r,t)\in S}-{\rm
log}\sigma(\phi_{r}(h,t)+\gamma)\\\
+\frac{1}{n}\sum_{(h^{{}^{\prime}},r,t^{{}^{\prime}})}{\rm
log\sigma(\phi_{r}(h^{{}^{\prime}},t^{{}^{\prime}}+\gamma)}\end{split}$ (6)

where $(h^{{}^{\prime}},r,t^{{}^{\prime}})$ are $n$ negative samples
corresponding to the positive triplet $(h,r,t)$, $\gamma$ is the margin, and
$\sigma$ is the sigmoid function.

#### Image-Text Contrastive Learning (ITC).

Image-Text Contrastive Learning aims to learn better unimodal representations
before fusion. Inspired by ALBEF (Li et al., 2021), for each image and text,
we calculate the softmax-normalized image-to-text and text-to-image similarity
as:

$s^{(i2t)}(I,T)=\frac{exp(s\langle I,T_{m}\rangle/\tau)}{\sum_{k\neq
i}^{K}exp(s\langle I,T_{m}\rangle)}$ (7)

$s^{(t2i)}(I,T)=\frac{exp(s\langle I_{m},T\rangle/\tau)}{\sum_{k\neq
i}^{K}exp(s\langle I_{m},T\rangle)}$ (8)

$\mathcal{L}_{ITC}=\frac{1}{2}\mathbb{E}\left([H(y^{i2t},s^{i2t}(I,T))+H(y^{t2i},s^{t2i}(I,T))\right)$
(9)

Table 1: Results for vision-language pretraining methods on popular VL benchmarks. We report accuracy for OK-VQA and SNLI-VE and vqa-score for AOK-VQA and VQA-v2. The best and second-best results are marked number and number, respectively. The gray number indicates that the model is trained with a significantly larger number of data than our models.  Task | Dataset | Method | Results
---|---|---|---
|  |  |  | dev | test
Knowledge-based Task | OK-VQA | Supervised Model |  |  |
KAT (single) |  | 53.1 | -
KAT (ensemble) |  | 54.4 | -
REVIVE (ensemble) |  | 56.6 | -
Pretraining Model | # images |  |
ALBEF | 12M | 54.7 | -
BLIP | 129M | 55.4 | -
REAVL (ours) | 4M | 57.7 | -
AOK-VQA | Supervised Model |  |  |
GPV-2 |  | 48.6 | -
Pretraining Model | # images |  |
ALBEF | 12M | 54.5 | -
BLIP | 129M | 56.2 | -
REAVL (ours) | 4M | 58.4 | -
General Task | VQA-v2 | Base Data-Size | # images |  |
VL-BERT | 3.3M | 71.16 | -
UNITER | 4M | 72.70 | 72.91
OSCAR | 4M | 73.16 | 73.44
UNIMO | 4M | 75.06 | 75.27
ALBEF | 4M | 74.54 | 74.70
REAVL (ours) | 4M | 77.62 | 77.79
Large Data-Size |  |  |
ALBEF | 12M | 75.84 | 76.04
BLIP | 14M | 77.54 | 77.62
BLIP | 129M | 78.25 | 78.32
SimVLMbase | 1.8B | 77.87 | 78.14
SNLI-VE | Base Data-Size | # images |  |
UNITER | 4M | 79.39 | 79.38
UNIMO | 4M | 81.11 | 80.63
ALBEF | 4M | 80.14 | 80.30
REAVL (ours) | 4M | 82.41 | 82.53
Large Data-Size |  |  |
ALBEF | 12M | 80.80 | 80.91
SimVLMbase | 1.8B | 84.20 | 84.15
Table 2: Entity Linking Results on WikiDiverse and WikiPerson. R@K represents the recall of the TopK retrieved entities. The best and second-best results are marked number and number, respectively.  | WikiDiverse | WikiPerson
---|---|---
| R@10 | R@50 | R@100 | R@1 | R@5 | R@10
ViT+BERT | 61.76 | 71.30 | 73.87 | 60.56 | 72.43 | 78.72
BLINK+CLIP_N_D | 66.96 | 77.18 | 80.53 | - | - | -
ResNet+CLIP_N | 77.64 | 81.21 | 85.69 | 68.24 | 79.83 | 82.65
ResNet+CLIP_N_D | 80.64 | 84.33 | 87.56 | 73.70 | 83.47 | 84.45
CLIP | 82.37 | 87.82 | 91.04 | 74.55 | 84.42 | 85.15
REAVL | 83.20 | 88.42 | 89.59 | 77.58 | 85.27 | 88.38

##  4 Experiment setup

###  4.1 Data

For the image-text data, following UNITER (Chen et al., 2020), we construct
our pre-training data using two web datasets (Conceptual Captions, SBU
Captions) and two in-domain datasets (COCO and Visual Genome). The total
number of unique images is 4.0M, and the number of image-text pairs is 5.1M.
We use Wikidata5M (Wang et al., 2021a), a general-domain knowledge graph
designed to capture background world knowledge for the KG data. It has 4M
nodes and 20M edges in total. For each entity in Wikidata5M, we use its
description collected from Wikipedia to produce entity embeddings.

###  4.2 Implementation Details

Our model consists of a BERT-base with 123.7M parameters and a ViT-B/16 with
85.8M parameters. We pre-train the model for 10 epochs using a batch size of
32 on 8 NVIDIA A100 GPUs. It took 8 days for pre-training using FP16. For
optimization, we use the AdamW (Loshchilov & Hutter, 2017) optimizer with a
learning rate of 5e-5 and a weight decay of 0.02. We use DistMult for KG
triplet scoring for the link prediction objective, with a negative exampling
of 128 triplets and a margin of $=0$. To pretrain the model, we perform MLM
with a token masking rate of 25%, MVM with a patch masking rate of 25%, and
link prediction with an edge drop rate of 15%.

###  4.3 Downstream evaluation tasks

We finetune and evaluate our models on the vision-language understanding tasks
(OK-VQA, AOK-VQA, VQA-v2, and SNLI-VE) (Marino et al., 2019; Schwenk et al.,
2022; Antol et al., 2015; Xie et al., 2019) and entity linking tasks
(WikiDiverse and WikiPerson) (Wang et al., 2022b; Sun et al., 2022). For the
vision-language understanding task, we follow the original setting and splits
used by prior works (Li et al., 2022a). For the multi-modal entity linking
tasks, we used to evaluate the effectiveness of the entity retrieval of REAVL.
Appendix B provides the full details on these tasks and data.

##  5 Experiment Results

###  5.1 Evaluation on V+L understanding tasks

To examine the quality of multi-modal representation learning, we first
compare REAVL on four downstream V+L tasks with state-of-the-art methods, as
shown in Table 1. It is clear that our method REAVL achieves state-of-the-art
performance, outperforming all existing models including supervised models
(KAT, REVIVE) and VLP models (ALBEF, BLIP) on the knowledge-based task. For
example, on the OK-VQA dataset, REAVL achieves a 1.94% and 4.15% relative
accuracy gain over the best baseline REVIVE and BLIP respectively. These
improvements are consistent in the AOK-VQA111The submission of test-set is
limited (Li et al., 2022a). dataset, demonstrating our hypothesis that large
knowledge graphs can provide complementary information to multi-modal data.
Particularly, compared to the supervised model REVIVE, which also exploits the
visual feature to retrieve knowledge, jointly training knowledge retriever and
knowledge-augmented model substantially improves the performances. This is in
line with our expectations as the joint training process would reward the
helpful knowledge and penalize the uninformative ones.

For the general V+L tasks, our model REAVL achieves relative improvements of
3.34% on VQA-v2 test-std and 2.36% on SNLI-VE test, with 4M pre-training
images. When comparing with the models that are trained with a significantly
larger number of data, our model also shows competitive performances. For
example, our model outperforms ALBEF and BLIP pre-trained on $>$10M data on
both datasets. The performance supports our view that the results of the SOTA
model under larger pre-training data can be achieved through the fusion of
smaller data scales and knowledge. And the slightly lower performance compared
to larger models is expected since larger data can provide more information
that knowledge graphs do not.

###  5.2 Evaluation on Entity Linking tasks

REAVL is also capable of linking the visual mentions in the image to the
corresponding named entity in knowledge graphs. To demonstrate the ability, we
compared REAVL with the existing multi-modal entity linking models. We
followed the evaluation settings used by (Wang et al., 2022b) and (Sun et al.,
2022) to report the comparison results with existing methods in Table 2.
Considering the format of entities in KBs, such as textual descriptions,
images, and other structured attributes, we only consider the visual (image)
to textual (descriptions and names) entity linking.

As we can see, ViT+BERT has achieved reasonably good performance for R@10
(i.e, 61.76 at WikiDiverse and 78.72 at WikiPerson), which demonstrates the
effectiveness of the pre-training model. Moreover, we can see that the CLIP,
which is pre-trained with about 400M image-caption pairs, has achieved the
strongest baseline performances, which demonstrates the effectiveness in
combining both visual description and textual description. Our model has
outperformed the CLIP model on 5 out of 6 metrics across the two datasets,
proving that our pre-training objectives on multi-modal data such as MLM and
MVM have promoted the training of the knowledge retriever. The slightly lower
performance in R@100 is expected as we usually select candidate entities
within the Top-50 for training due to the limitation of the length of
knowledge-augmented model.

###  5.3 Ablation Study

Here, we will ablate the properties of REAVL to investigate factors that
influence the model performance, as we discuss below. Table 3 shows the
ablation experiment results.

#### Incorporating Knowledge Graphs with Multi-modal data.

The first key contribution of our model (w.r.t. existing VLP methods) is that
we incorporate knowledge graphs for V+L pre-training. We find that this
significantly improves the model’s performance for knowledge retrieval and
integration. Compared to the REAVL without the KG retriever, jointly training
the KG retriever and V+L learning substantially improves the model’s
performance on the OK-VQA task. Furthermore, adding the proposed GNN
Aggregation and the knowledge-augmented model both enhance the model
performance. This is in line with our expectation as GNN Aggregation could
extract more related entities and relations while the knowledge-augmented
model is designed to fuse knowledge and multi-modal data in an efficient way.

Table 3: Ablation Study on OK-VQA.
Training Task / Architecture | OK-VQA
---|---
w/o Knowledge Retriever | 52.71
w/o GNN Aggregation | 54.33
w/o Knowledge Augmented Model | 56.61
Retrieval From Textual | 53.27
Retrieval From Vision (Final) | 57.72
MLM | 54.38
MLM+MVM | 55.60
MLM+MVM+ITC | 57.24
MLM+MVM+ITC+LinkPred | 57.72
Table 4: Case Study on OK-VQA dataset. Question | Image | Ground-Truth | Prediction | Supporting
---|---|---|---|---
(a) What is the person in the photo wearing? |  |  [wetsuit, suit, wet suit, trunk] |  REAVL: wetsuit ✓ |  Retrieved Entities: {boardsport, surfer, big wave surfing, wetsuit, dry suit, …}
(b) What kind of flowers are on the table? |  |  [rose] |  REAVL: rose ✓ |  Retrieved Entities: {dinnerware, Duchess of Abrantes, cake plate, tea, rose petals}
(c) Which liquid seen here comes from a citrus fruit? |  |  [orange juice, orange] |  REAVL: juice ✕ |  Retrieved Entities: {plateau, placemat, fruit juice, monkey bread, citrus juice, …}
(d) Can you guess the location where the airoplane is seen? |  |  [arizona, new zea- land, us, tarmac] |  REAVL: airport ✕ |  Retrieved Entities: {Canadair Regional Jet, forward-swept wing, yellow line, non-towered airport, …}
(e) What type of temperature is this? |  |  [cold, cold, cold, cold, cold, cold, cold, cold, pleasant, pleasant] |  REAVL: cold ✓ BLIP: cool ✕ REVIVE: warm ✕ |  Retrieved Entities: {Strawberry train, Nankai Electric Railway (Japanese railway company), rail transport in Walt Disney Parks, cherry blossom}

#### Knowledge retrieval on multi-modal data

Another advantage of our model is its ability to retrieve knowledge from
multiple modalities like images. In order to study the necessity of visual
knowledge retrieval, we perform an ablation study to see the impact of
knowledge retrieval. As can be seen, the performance drop of textual retrieval
is more severe on the OK-VQA dataset. This is understandable because images
contain more information that is not covered by text, especially in the
downstream tasks. Thus, it is necessary to retrieve knowledge from images to
better facilitate the fusion of knowledge and multi-modal data.

#### Effect of pre-training objectives

In order to study the contributions of different pre-training objectives, we
investigated the four pre-training objective combinations. We report their
fine-tuned results on the OK-VQA dataset in Table 3. As can be seen, only with
MLM, our model only achieves an accuracy of 54.38, which slightly lags behind
the baseline models (ALBEF and BLIP). Adding MVM and ITC both substantially
improves the pre-trained model’s performance. And the proposed LinkPred
further enhances the model by reasoning complex world knowledge. Finally, our
model achieves a 1.94% and 4.15% relative accuracy gain over the baseline
REVIVE and BLIP respectively.

###  5.4 Case Study

We conduct case studies on the behavior of REAVL’s knowledge retrieval and
knowledge augmentation, where we analyze the contribution of retrieved
entities to question answering (Table 4). We can observe that our approach can
accurately retrieve informative entities after training the knowledge
retriever. For instance, REAVL can retrieve useful knowledge from images
(e.g., wetsuit and rose petals) to generate the correct answer in example
(a)(b). From the error cases, we can see that the model still generates
reasonable answers for such scenarios. For example (c), we have retrieved the
informative entity citrus juice from the image, but failed to understand the
problem well. Example (d) is quite difficult to answer but our model still
could generate a reasonable answer airport. When comparing the existing
methods, we selected the case that has been reported by REVIVE. Although the
question is more complex, our model could accurately predict the answer, but
existing models, BLIP and REVIVE, struggle to predict the correct answers,
which can demonstrate the potential of our proposed method.

##  6 Conclusion

In this paper, we presented REAVL, a novel vision-language pre-training
framework to incorporate world knowledge into multi-modal representations. It
not only exploits the multi-modal data for better knowledge retrieval but also
fuses knowledge and multi-modal data with a knowledge-augmented model. Our
experiments on knowledge-based tasks show the superiority of our approach, as
it outperforms existing VLP baselines with low-resource pre-training data. At
the same time, the performance on entity linking tasks also shows its
excellent ability in retrieving informative knowledge from massive knowledge
graphs. One limitation of REAVL is that it also large constitutes
computational costs due to knowledge retrieval, and an important future
research will be to extend it to make it faster and apply it to larger
datasets.

## References

  * Antol et al. (2015) Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., and Parikh, D.  Vqa: Visual question answering.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 2425–2433, 2015.
  * Changpinyo et al. (2021) Changpinyo, S., Sharma, P., Ding, N., and Soricut, R.  Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In _CVPR_ , 2021.
  * Chen et al. (2022a) Chen, W., Hu, H., Chen, X., Verga, P., and Cohen, W. W.  Murag: Multimodal retrieval-augmented generator for open question answering over images and text.  _arXiv preprint arXiv:2210.02928_ , 2022a.
  * Chen et al. (2022b) Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A., Padlewski, P., Salz, D., Goodman, S., Grycner, A., Mustafa, B., Beyer, L., et al.  Pali: A jointly-scaled multilingual language-image model.  _arXiv preprint arXiv:2209.06794_ , 2022b.
  * Chen et al. (2020) Chen, Y.-C., Li, L., Yu, L., Kholy, A. E., Ahmed, F., Gan, Z., Cheng, Y., and Liu, J.  Uniter: Universal image-text representation learning.  In _ECCV_ , 2020.
  * Devlin et al. (2018) Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Dosovitskiy et al. (2021) Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N.  An image is worth 16x16 words: Transformers for image recognition at scale.  _ICLR_ , 2021.
  * Guu et al. (2020) Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M.  Retrieval augmented language model pre-training.  In _International Conference on Machine Learning_ , pp. 3929–3938. PMLR, 2020.
  * He et al. (2022) He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 16000–16009, 2022.
  * Jia et al. (2021) Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q., Sung, Y.-H., Li, Z., and Duerig, T.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _International Conference on Machine Learning_ , pp. 4904–4916. PMLR, 2021.
  * Johnson et al. (2019) Johnson, J., Douze, M., and Jégou, H.  Billion-scale similarity search with gpus.  _IEEE Transactions on Big Data_ , 7(3):535–547, 2019.
  * Langley (2000) Langley, P.  Crafting papers on machine learning.  In Langley, P. (ed.), _Proceedings of the 17th International Conference on Machine Learning (ICML 2000)_ , pp. 1207–1216, Stanford, CA, 2000\. Morgan Kaufmann.
  * Lewis et al. (2020) Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., et al.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _Advances in Neural Information Processing Systems_ , 33:9459–9474, 2020.
  * Li et al. (2022a) Li, D., Li, J., Le, H., Wang, G., Savarese, S., and Hoi, S. C.  Lavis: A library for language-vision intelligence.  _arXiv preprint arXiv:2209.09019_ , 2022a.
  * Li et al. (2021) Li, J., Selvaraju, R., Gotmare, A., Joty, S., Xiong, C., and Hoi, S. C. H.  Align before fuse: Vision and language representation learning with momentum distillation.  _Advances in neural information processing systems_ , 34:9694–9705, 2021.
  * Li et al. (2022b) Li, J., Li, D., Xiong, C., and Hoi, S.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _ICML_ , 2022b.
  * Loshchilov & Hutter (2017) Loshchilov, I. and Hutter, F.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * Marino et al. (2019) Marino, K., Rastegari, M., Farhadi, A., and Mottaghi, R.  Ok-vqa: A visual question answering benchmark requiring external knowledge.  In _Proceedings of the IEEE/cvf conference on computer vision and pattern recognition_ , pp. 3195–3204, 2019.
  * Radford et al. (2021) Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , pp. 8748–8763. PMLR, 2021.
  * Schwenk et al. (2022) Schwenk, D., Khandelwal, A., Clark, C., Marino, K., and Mottaghi, R.  A-okvqa: A benchmark for visual question answering using world knowledge.  _arXiv preprint arXiv:2206.01718_ , 2022.
  * Speer et al. (2017) Speer, R., Chin, J., and Havasi, C.  Conceptnet 5.5: An open multilingual graph of general knowledge.  In _Thirty-first AAAI conference on artificial intelligence_ , 2017\.
  * Sun et al. (2022) Sun, W., Fan, Y., Guo, J., Zhang, R., and Cheng, X.  Visual named entity linking: A new dataset and a baseline.  _arXiv preprint arXiv:2211.04872_ , 2022.
  * Vrandečić & Krötzsch (2014) Vrandečić, D. and Krötzsch, M.  Wikidata: a free collaborative knowledgebase.  _Communications of the ACM_ , 57(10):78–85, 2014\.
  * Wang et al. (2022a) Wang, P., Yang, A., Men, R., Lin, J., Bai, S., Li, Z., Ma, J., Zhou, C., Zhou, J., and Yang, H.  Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework.  In _International Conference on Machine Learning_ , pp. 23318–23340. PMLR, 2022a.
  * Wang et al. (2021a) Wang, X., Gao, T., Zhu, Z., Zhang, Z., Liu, Z., Li, J., and Tang, J.  Kepler: A unified model for knowledge embedding and pre-trained language representation.  _Transactions of the Association for Computational Linguistics_ , 9:176–194, 2021a.
  * Wang et al. (2022b) Wang, X., Tian, J., Gui, M., Li, Z., Wang, R., Yan, M., Chen, L., and Xiao, Y.  Wikidiverse: A multimodal entity linking dataset with diversified contextual topics and entity types.  _arXiv preprint arXiv:2204.06347_ , 2022b.
  * Wang et al. (2021b) Wang, Z., Yu, J., Yu, A. W., Dai, Z., Tsvetkov, Y., and Cao, Y.  Simvlm: Simple visual language model pretraining with weak supervision.  _arXiv preprint arXiv:2108.10904_ , 2021b.
  * Xie et al. (2019) Xie, N., Lai, F., Doran, D., and Kadav, A.  Visual entailment: A novel task for fine-grained image understanding.  _arXiv preprint arXiv:1901.06706_ , 2019.
  * Yang et al. (2015) Yang, B., Yih, W.-t., He, X., Gao, J., and Deng, L.  Embedding entities and relations for learning and inference in knowledge bases.  In _International Conference on Learning Representations_ , 2015.
  * Yasunaga et al. (2022) Yasunaga, M., Bosselut, A., Ren, H., Zhang, X., Manning, C. D., Liang, P., and Leskovec, J.  Deep bidirectional language-knowledge graph pretraining.  In _Neural Information Processing Systems (NeurIPS)_ , 2022.
  * Yu et al. (2022) Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., and Wu, Y.  Coca: Contrastive captioners are image-text foundation models.  _arXiv preprint arXiv:2205.01917_ , 2022.
  * Zhang et al. (2019) Zhang, Z., Han, X., Liu, Z., Jiang, X., Sun, M., and Liu, Q.  ERNIE: Enhanced language representation with informative entities.  In _Proceedings of ACL 2019_ , 2019.

##  Appendix A Technical details

###  A.1 GNN Encoder

In each layer of the GNN, the current representation of the node embeddings
$\\{e_{1}^{(l-1)},...,e_{k}^{(l-1)}\\}$ is fed into the layer to perform a
round of information propagation between nodes in the graph and yield pre-
fused node embeddings for each entity:

$\begin{split}\\{e_{1}^{(l)},...,e_{k}^{(l)}\\}={\rm
GNN}(\\{e_{1}^{(l-1)},...,e_{k}^{(l-1)}\\})\end{split}$ (10)

where GNN corresponds to a variant of graph attention networks and $l$
corresponds to the layer of GNN. The GNN computes node representations for
each node via message passing between neighbors on the graph:

$e_{i}^{(l)}=f_{n}\left(\sum_{e_{j}\in\mathcal{N}_{e_{i}}\cup{e_{i}}}\alpha_{ij}\cdot
f_{A_{ij}}\right)+e_{i}^{(l-1)}$ (11)

where $f_{n}$ is a linear transformation, $\mathcal{N}_{e_{i}}$ represents the
neighborhood of an entity node $e_{i}$, and $f_{A_{ij}}$ denotes the message
passing function from its neighbors $e_{j}$ to $e_{i}$. The $\alpha_{ij}$ is
an attention weight that scales the message function $f_{A_{ij}}$, which could
be calculated as:

$\beta_{ij}=\frac{f_{q}(e_{i}^{(l-1)})^{T}f_{k}(e_{j}^{(l-1)},r_{ij})}{\sqrt{D}}$
(12)

$\alpha_{ij}=\frac{exp(\beta_{ij})}{\sum_{e_{k}\in\mathcal{N}_{e_{i}}\cup{e_{i}}}exp(\beta_{ik})}$
(13)

where $f_{q}$ and $f_{k}$ are linear transformations and $r_{ij}$ is a
relation embedding for the relation connecting $e_{i}$ and $e_{j}$.

The message passing function $f_{A_{ij}}$ could be computed in the following
manner:

$f_{A_{ij}}=f_{m}(e_{j}^{(l-1)},r_{ij})$ (14)

where $f_{m}$ is linear transformations and $r_{ij}$ is defined the same as
above.

###  A.2 Knowledge-augmented Model

In the Knowledge-augmented Model, let $x=\\{x_{1},...,x_{p}\\}$ be the input
elements, which are embedding features of the visual, textual, and entity
elements. They are processed by a multi-layer bidirectional Transformer, where
the embedding features of each element are transformed layer-by-layer in the
fashion of aggregating features from the other elements with adaptive
attention weights. The features of the $(l+1)$-th layer, $x^{(l+1)}$, is
computed by:

$\widehat{h}_{i}^{(l+1)}=\sum_{m=1}^{M}W_{m}^{(l+1)}\left(\sum_{j=1}^{N}A_{i,j}^{m}\cdot
V_{m}^{(l+1)}x_{j}^{l}\right)$ (15)

$h_{i}^{(l+1)}={\rm LayerNorm}\left(x_{i}^{l}+\widehat{h}_{i}^{(l+1)}\right)$
(16)

$\widehat{x}_{i}^{(l+1)}=W_{2}^{(l+1)}\cdot{\rm
GELU}\left(W_{1}^{(l+1)}h_{i}^{(l+1)}\right)$ (17)

$x_{i}^{(l+1)}={\rm
LayerNorm}\left(h_{i}^{(l+1)}+\widehat{x}_{i}^{(l+1)}\right)$ (18)

where $m$ in Eq.15 indexes over the attention heads, and $A_{i,j}^{m}={\rm
exp}[(Q_{m}^{(l+1)}x_{i}^{l})^{T}(K_{m}^{(l+1)}x_{j}^{l})]$ denotes the
attention weights between elements $i$ and $j$ in the $m$-th head, and
$W_{m}^{(l+1)}$, $Q_{m}^{(l+1)}$, $K_{m}^{(l+1)}$ and $W_{1}^{(l+1)}$,
$W_{2}^{(l+1)}$ are learnable weights for $m$-th attention head and $(l+1)$-th
layer.

##  Appendix B Data and Tasks.

#### V+L understanding tasks

requires a cooperative knowledge of vision and language. We finetune and
evaluate four vision-language understanding benchmarks, including knowledge-
base tasks (OK-VQA (Marino et al., 2019) and AOK-VQA (Schwenk et al., 2022))
and general tasks (VQAv2 (Antol et al., 2015) and SNLI-VE (Xie et al., 2019)).
Specifically, we feed the embeddings of a given question (text), an image, and
the retrieved entities into the knowledge-augmented model and use a 12-layer
transformer decoder to generate the answer.

For the VQA-v2, OK-VQA, and AOK-VQA, we followed the prior work (Li et al.,
2021, 2022a) and formulate the task as a classification problem over the most
frequent answers in the training set. OK-VQA and AOK-VQA is the knowledge-
based VQA benchmark that requires external knowledge to answer its questions,
that is, the knowledge that is not directly present in the image input.

#### Multi-modal Entity Linking

aims at linking mentions with multi-modal contexts to the referent entities
from a knowledge base (e.g., Wikipedia), is an essential task for many multi-
modal applications. We consider two datasets for multi-modal Entity Linking:
WikiDiverse (Wang et al., 2022b) and WikiPerson (Sun et al., 2022).
WikiDiverse is a high-quality human-annotated MEL dataset with diversified
contextual topics and entity types from Wikinews, while WikiPerson is a high-
quality human-annotated visual person-linking dataset from VisualNews. Herein,
we finetuned the dual encoder of image and entity with the contrastive loss
function of mention-entity samples to demonstrate the superiority of our
retrieval process.


# Distinguishing a Slowly Accelerating Black Hole
by Differential Time Delays of Images

Amjad Ashoorioon<EMAIL_ADDRESS>School of Physics, Institute for Research in
Fundamental Sciences (IPM), P.O. Box 19395-5531, Tehran, Iran  Mohammad Bagher
Jahani Poshteh<EMAIL_ADDRESS>School of Physics, Institute for Research in
Fundamental Sciences (IPM), P.O. Box 19395-5531, Tehran, Iran  Robert B. Mann
<EMAIL_ADDRESS>Department of Physics and Astronomy, University of
Waterloo, Waterloo, Ontario, N2L 3G1, Canada  Perimeter Institute for
Theoretical Physics, Waterloo, Ontario, N2L 2Y5, Canada

###### Abstract

Accelerating supermassive black holes, connected to cosmic strings, could
contribute to structure formation and get captured by galaxies if their
velocities are small. This would mean that the acceleration of these black
holes is small too. Such a slow acceleration has no significant effect on the
shadow of such supermassive black holes. We also show that, for slowly
accelerating black holes, the angular position of images in the gravitational
lensing effects do not change significantly. We propose a method to observe
the acceleration of these black holes through the gravitational lensing. The
method is based on the observation that differential time delays associated
with the images are substantially different with respect to the case of non-
accelerating black holes.

Over the past five years observations from LIGO/VIRGO has provided us with
qualitatively new information about our universe via gravitational waves.
Black holes have so far been the primary source of gravitational waves, but it
is reasonable to expect that future detectors will discover new sources, some
of which will yield information about the early universe. Cosmic strings –
line-like topological defects that can emerge from some gauge theories during
first order phase transitions [1, 2] or at the end of brane inflation [3] –
are one such example. Such objects can break or fray to produce pair of
accelerating black holes [4, 5], which can also be produced in a background
magnetic field [6, 7, 8] or in de Sitter space [9, 10, 11, 12, 13].
Alternatively, a network of cosmic strings could capture primordial black
holes produced during their formation [14]. Such black holes would be
accelerating due to the tension of the cosmic string.

It is therefore of interest to examine how we might detect accelerating black
holes. Although accelerating supermassive black holes connected to cosmic
strings could reside in the centers of galaxies [14, 15], their velocities
must be small ($\lesssim 100\,{\rm km}/{\rm s}$), so that they can contribute
to structure formation [14]. Measurement of their acceleration is therefore a
formidable challenge, since this quantity must consequently be very small.

Here we propose a method for detection of black hole acceleration via
gravitational lensing, in which rays of light coming from a source behind the
black hole are deflected near it and turn toward an observer. The observer
does not see the actual location of the source, but rather sees images of it
apparently located elsewhere. Our method exploits the fact that differential
time delays associated with lensed images of accelerating black holes
substantially differ with respect to their non-accelerating counterparts.

Spacetime around uniformly accelerated black holes can be described by

$\displaystyle ds^{2}$ $\displaystyle=$ $\displaystyle\frac{1}{(1+\alpha
r\cos\theta)^{2}}$ $\displaystyle\times$
$\displaystyle\left[-Q(r)dt^{2}+\frac{dr^{2}}{Q(r)}+\frac{r^{2}d\theta^{2}}{P(\theta)}+P(\theta)r^{2}\sin^{2}\theta
d\phi^{2}\right],$

known as the C-metric [16, 17], where

$\displaystyle Q(r)$ $\displaystyle=$
$\displaystyle(1-\alpha^{2}r^{2})\left(1-\frac{2m}{r}\right),$ $\displaystyle
P(\theta)$ $\displaystyle=$ $\displaystyle 1+2\alpha m\cos\theta.$ (2)

$m$ is the mass parameter and $\alpha$ is interpreted as the acceleration. We
take the acceleration to be sufficiently small so that a ray of light passing
the black hole to the Earth, lies on the equatorial plane $\theta=\pi/2$ of
the black hole during its evolution. On this plane, the line element
(Distinguishing a Slowly Accelerating Black Hole  by Differential Time Delays
of Images) reduces to 111 Within the small acceleration approximation, we can
first find the geodesic equations and then substitute $\theta=\pi/2$ — the
results will be the same as presented here.

$ds^{2}=-Qdt^{2}+\frac{dr^{2}}{Q}+r^{2}d\phi^{2},$ (3)

where $Q(r)$ is given by (2). In Fig. 1 we schematically illustrate the setup.
Light coming from the source passes the black hole and is deflected by an
angle $\hat{\alpha}$. The observer sees the image of the object at angular
position $\vartheta$.

Figure 1: Deflection of light by the black hole: $S$, $I$, $O$, and $L$,
respectively stand for the source, image, observer, and the lens (black hole).
The black hole is accelerating in a direction perpendicular to the plane of
the figure and behind it there is an acceleration horizon at $r=1/\alpha$,
where $\alpha$ is the acceleration, whose sign is not important in our
approximation. The path of the light ray remains on the equatorial plane of
the black hole within the small acceleration approximation $\alpha D_{s}\ll
1$. $\beta$ is the actual angular position of the source (with respect to the
line of sight to the black hole). The black hole bends the light ray by an
angle $\hat{\alpha}$ so that the observer sees the image at angular position
$\vartheta$. $D_{d}$ and $D_{ds}$ are the distance from lens to observer and
from lens to the source, respectively.

The equations governing the geodesics can be obtained using the Lagrangian

$\mathcal{L}=\frac{1}{2}g_{\mu\nu}\dot{x}^{\mu}\dot{x}^{\nu}=\frac{1}{2}\left(-Q\dot{t}^{2}+\frac{\dot{r}^{2}}{Q}+r^{2}\dot{\phi}^{2}\right),$
(4)

where the dot denotes differentiation with respect to some affine parameter
along the geodesic. The constants of motion are
$E=-\frac{\partial\mathcal{L}}{\partial\dot{t}}=Q\dot{t}$, and
$L_{z}=-\frac{\partial\mathcal{L}}{\partial\dot{\phi}}=-r^{2}\dot{\phi}$.

For the null geodesics $\mathcal{L}=0$, and at the point of closest approach
to the black hole, $r=b$, $\frac{dr}{d\phi}=0$. Using these facts the
deflection angle is222As can be seen from Eq. (2), the metric function $Q$
changes sign at $r=1/\alpha$. Because of the $1/r$ factor in Eq. (5), the
contribution of large $r$ from $\alpha^{-1}$ to infinity is very small in the
integral of (5)— about one part in one million parts. Therefore we can safely
integrate to infinity. [18]

$\hat{\alpha}(b)=2\int_{b}^{\infty}\frac{dr}{r\sqrt{\left(\frac{r}{b}\right)^{2}Q_{b}-Q}}-\pi,$
(5)

where we safely assume that $1\gg\alpha D_{d},D_{ds}\gg b$, where $D_{d}$ and
$D_{ds}$ are the respective distances from observer and source to the black
hole. For simplicity we take the direction of the acceleration to be
perpendicular to the plane of the lens diagram, i.e. Fig. 1. If the
acceleration has a component parallel to this plane, then the third component
of the angular momentum is not a constant of motion and the geodesic equations
cannot be analytically integrated.

Using the Lagrangian (4) and the fact that $\frac{dr}{dt}=0$ at $r=b$ we find

$\tau(b)=\left[\int_{b}^{r_{s}}dr+\int_{b}^{D_{d}}dr\right]\frac{1}{Q\sqrt{1-\left(\frac{b}{r}\right)^{2}\frac{Q}{Q_{b}}}}-D_{s}\sec\beta,$
(6)

for the difference between the time it takes for light to travel the physical
path from source to observer with and without a black hole present, where
$\beta$ is the angular position of the source, $D_{s}=D_{d}+D_{ds}$ is the
distance from observer to the source and
$r_{s}=\sqrt{D_{ds}^{2}+D_{s}^{2}\tan^{2}\beta}$.

The Virbhadra-Ellis lens equation [19]

$\tan\beta=\tan\vartheta-\mathcal{D}\left[\tan\vartheta+\tan(\hat{\alpha}-\vartheta)\right],$
(7)

where $\mathcal{D}=D_{ds}/D_{s}$, yields the image angular position,
$\vartheta$, and

$J=\frac{b}{\sqrt{Q_{b}}}=D_{d}\sin\vartheta\qquad\mu=\left(\frac{\sin\beta}{\sin\vartheta}\frac{d\beta}{d\vartheta}\right)^{-1}.$
(8)

are respectively the impact parameter and image magnification [20].

Differentiating Eq. (7) with respect to $\vartheta$ we find

$\sec^{2}\beta\frac{d\beta}{d\vartheta}=\sec^{2}\vartheta-\mathcal{D}\left[\sec^{2}\vartheta+\sec^{2}(\hat{\alpha}-\vartheta)\left(\frac{d\hat{\alpha}}{d\vartheta}-1\right)\right].$
(9)

where
$\frac{d\hat{\alpha}}{d\vartheta}=\frac{d\hat{\alpha}}{db}\frac{db}{d\vartheta}$.
The factor $\frac{db}{d\vartheta}$ can be found by from (8), whereas we use
[21]

$\frac{d\hat{\alpha}(b)}{db}=-2\int_{b}^{\infty}\frac{1}{\sqrt{\mathcal{F}}}\frac{\partial}{\partial
r}\left(\frac{1}{r}\frac{\partial\mathcal{F}}{\partial b}\frac{\partial
r}{\partial\mathcal{F}}\right)dr,$ (10)

where $\mathcal{F}=\left(\frac{r}{b}\right)^{2}Q_{b}-Q$, to compute
$\frac{d\hat{\alpha}}{db}$.

We take the lensing black hole to be of the same mass and distance as M87*. We
use numerical methods [19, 21] to investigate its gravitational lensing
assuming this black hole is accelerating and compare image positions,
magnifications and (differential) time delays to the case where it is non-
accelerating. The mass and distance of M87* have been obtained by the Event
Horizon Telescope Collaboration as $M_{{\rm M87^{*}}}=9.6\times 10^{12}\,{\rm
m}\equiv 6.5\times 10^{9}M_{\odot}$ and $D_{d}=5.2\times 10^{23}\,{\rm m}$
[22]. Whenever we consider an accelerating black hole, we take the
acceleration to be $\alpha=10^{-25}{\rm m}^{-1}$, roughly the upper bound
obtained by [14] from the development of a cosmic string network with black
holes of such masses as beads in the network. For simplicity we assume that
the black hole is halfway between the source and the observer; therefore
${\cal D}=0.5$. Small changes (of order 1%) in ${\cal D}$ result in a 0.5%
change in image positions and a 3% change in the time delay.

The closer the light ray passes by the black hole, the greater the deflection
angle. The small value of the acceleration does not change the value of the
deflection angle significantly. In fact, for a fixed value of the impact
parameter, the deflection angle of a non-accelerating black hole is greater
than that of its accelerating counterpart by only about 1 part in $10^{13}$.
Since the deflection angle is in units of arcseconds, such deviations cannot
be observed in the near future.

In Table 1, by using eqs. (5), (7), (8), and (10), we have computed image
positions, deflection angles, impact parameters, and magnifications of primary
and secondary images for different values of the source angular positions. The
values presented in this table hold for both a non-accelerating and slowly
accelerating M87*. These quantities do not depend on the acceleration
significantly. For a fixed value of the source angular position $\beta$, the
impact parameter in the non-accelerating case is greater than that of the
slowly accelerating case by only about 1 part in $10^{13}$. On the other hand,
the (absolute value of) the image angular position is larger for the
accelerating case by only about 1 part in $10^{17}$.

Therefore, it is impossible (at least using current and near-future
observational facilities) to tell whether or not M87* is slowly accelerating
by using the image positions produced from gravitational lensing. We note that
it is also impossible to tell if M87* is non-accelerating or slowly
accelerating by using its shadow [23]. Magnitudes of the magnification of
images are greater in the slowly accelerating case by about 1 part in
$10^{17}$ parts as well.

We see from Table 1 that the angular position of primary images increases with
increasing the angular position of the source, whereas the absolute value of
the angular position of secondary images correspondingly decreases, as shown
in Fig. 2.

Figure 2: Image position of primary and secondary images: Angular position of primary images $\theta_{p}$ (red line) and the absolute value of angular position of secondary images $|\theta_{s}|$ (dashed blue line) as a function of source angular position $\beta$. Angles are in arcseconds. We have set $M_{{\rm M87^{*}}}=9.6\times 10^{12}\,{\rm m}$, $D_{d}=5.2\times 10^{23}\,{\rm m}$, ${\cal D}=0.5$, and $\alpha=10^{-25}{\rm m}^{-1}$. Table 1: Image positions, deflection angles, impact parameters, and magnifications of primary and secondary images due to lensing by M87*: Angular positions $\theta$, bending angles $\hat{\alpha}$, impact parameters $b$, and magnifications $\mu$ are given for different values of angular source position $\beta$. These results are the same in the non-accelerating and slowly accelerating cases. (a) $p$ and $s$ refer to primary and secondary images, respectively. (b) All angles are in arcseconds and the impact parameters are in meters. (c) We have used $M_{{\rm M87^{*}}}=9.6\times 10^{12}\,{\rm m}$, $D_{d}=5.2\times 10^{23}\,{\rm m}$, ${\cal D}=0.5$, and $\alpha=10^{-25}{\rm m}^{-1}$. $\beta$ | $\theta_{p}$ | $\hat{\alpha}_{p}$ | $b_{p}$ | $\mu_{p}$ | $\theta_{s}$ | $\hat{\alpha}_{s}$ | $b_{s}$ | $\mu_{s}$
---|---|---|---|---|---|---|---|---
$0$ | $1.25334$ | $2.50668$ | $3.20\times 10^{18}$ | $\times$ | $-1.25334$ | $2.50668$ | $3.20\times 10^{18}$ | $\times$
$0.1$ | $1.30436$ | $2.40872$ | $3.29\times 10^{18}$ | $6.79185$ | $-1.20436$ | $2.60872$ | $3.04\times 10^{18}$ | $-5.78173$
$0.5$ | $1.52806$ | $2.05611$ | $3.85\times 10^{18}$ | $1.82812$ | $-1.02805$ | $3.05611$ | $2.59\times 10^{18}$ | $-0.826954$
$1$ | $1.84943$ | $1.69886$ | $4.66\times 10^{18}$ | $1.26694$ | $-0.849483$ | $3.69897$ | $2.14\times 10^{18}$ | $-0.267388$
$2$ | $2.60342$ | $1.20683$ | $6.56\times 10^{18}$ | $1.05575$ | $-0.603448$ | $5.20690$ | $1.51\times 10^{18}$ | $-0.0567785$
$3$ | $3.45471$ | $0.909423$ | $8.71\times 10^{18}$ | $1.01705$ | $-0.454745$ | $6.90949$ | $1.14\times 10^{18}$ | $-0.0176322$
$4$ | $4.36033$ | $0.720660$ | $1.10\times 10^{19}$ | $1.00676$ | $-0.360286$ | $8.72057$ | $9.08\times 10^{17}$ | $-0.00687431$
Table 2: Time delays of primary and secondary images due to lensing by M87*: Time delays $\tau$ are given for different values of angular source position $\beta$. Instead of the time delays of secondary images $\tau_{s}$, we present the differential time delay $t_{d}=\tau_{s}-\tau_{p}$ which is of observational importance. (a) As in Table 1. (b) $\beta$ is in arcseconds and the (differential) time delays are in seconds. (c) As in Table 1. (d) Barred quantities refer to values of the case that the black hole is not accelerating and $\Delta t_{d}=\bar{t}_{d}-t_{d}$. $\beta$ | $\tau_{p}$ | $t_{d}$ | $\bar{\tau}_{p}$ | $\bar{t}_{d}$ | $\Delta t_{d}$
---|---|---|---|---|---
$0$ | $3.13186970\times 10^{12}$ | $0$ | $1.62727362\times 10^{6}$ | $0$ | $0$
$0.1$ | $3.13186969\times 10^{12}$ | $20445.4686829920$ | $1.61725037\times 10^{6}$ | $20445.4686829935$ | $1.52795\times 10^{-9}$
$0.5$ | $3.13186965\times 10^{12}$ | $102871.116191642$ | $1.58092945\times 10^{6}$ | $102871.116191651$ | $8.42556\times 10^{-9}$
$1$ | $3.13186961\times 10^{12}$ | $209681.897709557$ | $1.54280388\times 10^{6}$ | $209681.897709578$ | $2.17115\times 10^{-8}$
$2$ | $3.13186955\times 10^{12}$ | $448736.764965694$ | $1.48443381\times 10^{6}$ | $448736.764965783$ | $8.89995\times 10^{-8}$
$3$ | $3.13186951\times 10^{12}$ | $737888.423902370$ | $1.44178393\times 10^{6}$ | $737888.423902647$ | $2.76603\times 10^{-7}$
$4$ | $3.13186948\times 10^{12}$ | $1089214.16844197$ | $1.40880628\times 10^{6}$ | $1089214.168442669$ | $7.01752\times 10^{-7}$

In Table 2, we have calculated the time delay (6) of primary images, again
assuming the acceleration $\alpha=10^{-25}{\rm m}^{-1}$. Quite strikingly,
even this small acceleration increases the time delay by 6 orders of
magnitude, even though the deflection angles do not differ significantly. To
understand this feature, recall from (2) that the acceleration makes important
changes on the metric function only at large distances. The large distances do
not have a significant contribution in the integral (5) due to the $1/r$
factor, and so $\alpha$ does not change the deflection angle significantly.

However, the story is different when it comes to the time delay. The integrals
of Eq. (6) do not have a $1/r$ factor and large distances consequently have a
significant contribution to $\tau(b)$. What is of observational importance is
the differential time delay $t_{d}=\tau_{s}-\tau_{p}$ (and
$\overline{t}_{d}$), which we provide in Table 2 (instead of explicit values
of time delays of secondary images). In fact $\tau_{s}$ and $\tau_{p}$ cannot
be observed, but if the source is pulsating, every phase in its period appears
in the secondary image $t_{d}$ seconds after it appears in the primary image.
With increasing angular position of the source, the differential time delay
$t_{d}$ increases.

Figure 3: Differential time delay: The difference between the differential
time delay in non-accelerating and slowly accelerating cases, $\Delta
t_{d}=\bar{t}_{d}-t_{d}$. Time delays are in units of seconds and $\beta$ is
in units of arcseconds. We have set $M_{{\rm M87^{*}}}=9.6\times 10^{12}\,{\rm
m}$, $D_{d}=5.2\times 10^{23}\,{\rm m}$, ${\cal D}=0.5$, and
$\alpha=10^{-25}{\rm m}^{-1}$.

Although the acceleration changes the values of $\tau_{p}$ (and $\tau_{s}$)
significantly, the observable quantity $t_{d}$ does not deviate from its non-
accelerating counterpart $\bar{t}_{d}$ that much. In the last column of Table
2 we have presented the difference $\Delta t_{d}=\bar{t}_{d}-t_{d}$, which is
a positive quantity; for a fixed value of angular source position, the
differential time delay of secondary and primary images is larger if the black
hole is not accelerating. The difference increases with increasing angular
source position, as shown in Fig. 3.

These results imply that it is indeed feasible to observe if M87* is
accelerating or not, provided sufficiently small changes $\Delta t_{d}$ in the
value of the differential time delay can be measured. We note that Shapiro
delays can be measured in binary pulsars with precisions about 10 microseconds
[24]; precision an order of magnitude better will be required here. Although
the distance to the source (and hence ${\cal D}$) can be measured from its
redshift [25], one cannot observe the angular source position $\beta$. What
one sees are primary and secondary images of the source and, if the source has
reliable variability, the differential time delay. Small acceleration does not
change the angular positions of primary and secondary images by a feasibly
observable amount, and so the angular source position $\beta$ can be
determined via Fig. 2. If the differential time delay $\bar{t}_{d}$
corresponding to this $\beta$ matches the observed value of the differential
time delay then the black hole is non-accelerating. Conversely, if this
$\bar{t}_{d}$ does not match the observed value of the differential time delay
then the black hole is slowly accelerating.

We note that velocity of the black hole is not necessarily in the same
direction as its acceleration. However, we assume the velocity to be such that
the path of light from the source to the lens lie on or near the equatorial
plane of the black hole. Suppose the light ray passes the black hole on its
equatorial plane and starts its travel toward the observer. The black hole is
accelerating in a direction perpendicular to the equatorial plane and the
geodesic equations indicate that the $\theta=\pi/2$ surface is not actually a
plane with vanishing exterior curvature [17]. However in the small
acceleration limit that we consider, the path of the light ray deviates from
the equatorial plane of the black hole by a small angle
$\delta\sim\frac{\alpha D_{s}^{2}}{D_{s}}=\alpha D_{s}$ regardless of the
cause of the acceleration, whether from cosmic strings [4], magnetic fields
[8], a cosmological constant [10], or different combinations of these [13,
26]. In this limit the path of the light ray deviates from the equatorial
plane of the black hole by a small angle $\delta\sim\frac{\alpha
D_{s}^{2}}{D_{s}}=\alpha D_{s}$. This small deviation changes the time delays
by the small fraction $\sim\alpha^{2}D_{s}^{2}$. Since we have assumed that
the source and the observer are all inside the acceleration horizon of the
black hole and thus $\alpha D_{s}\ll 1$, our result for the differential time
delays will be valid up to order $\alpha^{2}D_{s}^{2}$. Note also that not
only $\delta\equiv\theta-\frac{\pi}{2}\ll 1$, but
$\dot{\delta}=\dot{\theta}\sim\theta/t\sim\alpha D_{s}/D_{s}\sim\alpha\ll
D_{s}^{-1}$ also. In such a limit, one can show that the equation for the time
delay does not change to leading order in $\delta$.

Taking into account deviations from the equatorial plane of the black hole by
an small angle $\delta$, the deflection angle becomes

$\hat{\alpha}(b)=2\int_{b}^{\infty}\frac{1+\alpha
m\delta}{r\sqrt{\left(\frac{r}{b}\right)^{2}Q_{b}-Q}}dr-\pi$ (11)

and we see that the time delay (6) does not change to leading order in
$\delta$. Since $m\alpha$ is very small (in our example it is of order
$10^{-12}$) the change in $\hat{\alpha}$ is negligible.

In gravitational lensing in the presence of an accelerating black hole, small
changes accumulate over large distances from the source to the observer. We
conclude that one can use gravitational lensing as a probe to measure the
acceleration of a black hole that acts as a lens. We note that in the slow
acceleration approximation that we are using the sign of the acceleration is
not important.

We also note that besides the primary and secondary images there is an
infinite set of faint images on each side of the black hole. These are
relativistic images and are produced by light rays that rotate around the
black hole before continuing their path to the observer [19, 27, 28]. We
postpone the study of relativistic images to a follow up publication.

_Acknowledgments:_ This project has received funding /support from the
European Unions Horizon 2020 research and innovation programme under the Marie
Sklodowska -Curie grant agreement No 860881-HIDDeN, and was supported in part
by the Natural Sciences and Engineering Council of Canada. MBJP would like to
acknowledge the support of Iran Science Elites Federation and the hospitality
of the University of Guilan. We are grateful to Mike Hudson and Avery
Broderick for helpful correspondence.

## References

  * Kibble [1976] T. W. B. Kibble, Topology of Cosmic Domains and Strings, J. Phys. A 9, 1387 (1976).
  * Vilenkin [1985] A. Vilenkin, Cosmic Strings and Domain Walls, Phys. Rept. 121, 263 (1985).
  * Sarangi and Tye [2002] S. Sarangi and S. H. H. Tye, Cosmic string production towards the end of brane inflation, Phys. Lett. B 536, 185 (2002), arXiv:hep-th/0204074 .
  * Hawking and Ross [1995] S. W. Hawking and S. F. Ross, Pair production of black holes on cosmic strings, Phys. Rev. Lett. 75, 3382 (1995), arXiv:gr-qc/9506020 .
  * Eardley  _et al._ [1995] D. M. Eardley, G. T. Horowitz, D. A. Kastor, and J. H. Traschen, Breaking cosmic strings without monopoles, Phys. Rev. Lett. 75, 3390 (1995), arXiv:gr-qc/9506041 .
  * Garfinkle and Strominger [1991] D. Garfinkle and A. Strominger, Semiclassical Wheeler wormhole production, Phys. Lett. B 256, 146 (1991).
  * Hawking  _et al._ [1995] S. W. Hawking, G. T. Horowitz, and S. F. Ross, Entropy, Area, and black hole pairs, Phys. Rev. D 51, 4302 (1995), arXiv:gr-qc/9409013 .
  * Dowker  _et al._ [1994] F. Dowker, J. P. Gauntlett, S. B. Giddings, and G. T. Horowitz, On pair creation of extremal black holes and Kaluza-Klein monopoles, Phys. Rev. D 50, 2662 (1994), arXiv:hep-th/9312172 .
  * Mellor and Moss [1989] F. Mellor and I. Moss, Black Holes and Quantum Wormholes, Phys. Lett. B 222, 361 (1989).
  * Mann and Ross [1995] R. B. Mann and S. F. Ross, Cosmological production of charged black hole pairs, Phys. Rev. D 52, 2254 (1995), arXiv:gr-qc/9504015 .
  * Dias and Lemos [2004] O. J. C. Dias and J. P. S. Lemos, Pair creation of de Sitter black holes on a cosmic string background, Phys. Rev. D 69, 084006 (2004), arXiv:hep-th/0310068 .
  * Ashoorioon and Mann [2014] A. Ashoorioon and R. B. Mann, Black Holes as Beads on Cosmic Strings, Class. Quant. Grav. 31, 225009 (2014), arXiv:1402.2072 [hep-th] .
  * Ashoorioon and Jahani Poshteh [2021] A. Ashoorioon and M. B. Jahani Poshteh, Black hole pair production on cosmic strings in the presence of a background magnetic field, Phys. Lett. B 816, 136224 (2021), arXiv:2006.16983 [hep-th] .
  * Vilenkin  _et al._ [2018] A. Vilenkin, Y. Levin, and A. Gruzinov, Cosmic strings and primordial black holes, JCAP 11, 008, arXiv:1808.00670 [astro-ph.CO] .
  * Morris  _et al._ [2017] M. R. Morris, J.-H. Zhao, and W. Goss, A nonthermal radio filament connected to the galactic black hole?, The Astrophysical Journal Letters 850, L23 (2017).
  * Kinnersley and Walker [1970] W. Kinnersley and M. Walker, Uniformly accelerating charged mass in general relativity, Phys. Rev. D 2, 1359 (1970).
  * Griffiths  _et al._ [2006] J. B. Griffiths, P. Krtous, and J. Podolsky, Interpreting the C-metric, Class. Quant. Grav. 23, 6745 (2006), arXiv:gr-qc/0609056 .
  * Weinberg [1972] S. Weinberg,  _Gravitation and cosmology: principles and applications of the general theory of relativity_ (Wiley, 1972).
  * Virbhadra and Ellis [2000] K. S. Virbhadra and G. F. R. Ellis, Schwarzschild black hole lensing, Phys. Rev. D 62, 084003 (2000), arXiv:astro-ph/9904193 .
  * Virbhadra  _et al._ [1998] K. S. Virbhadra, D. Narasimha, and S. M. Chitre, Role of the scalar field in gravitational lensing, Astron. Astrophys. 337, 1 (1998), arXiv:astro-ph/9801174 .
  * Jahani Poshteh and Mann [2019] M. B. Jahani Poshteh and R. B. Mann, Gravitational Lensing by Black Holes in Einsteinian Cubic Gravity, Phys. Rev. D 99, 024035 (2019), arXiv:1810.10657 [gr-qc] .
  * Akiyama  _et al._ [2019] K. Akiyama _et al._ (Event Horizon Telescope), First M87 Event Horizon Telescope Results. VI. The Shadow and Mass of the Central Black Hole, Astrophys. J. Lett. 875, L6 (2019), arXiv:1906.11243 [astro-ph.GA] .
  * Ashoorioon  _et al._ [2021] A. Ashoorioon, M. B. Jahani Poshteh, and R. B. Mann, Distinguishing a Slowly Accelerating Black Hole by Differential Time Delays of Images, (2021), arXiv:2110.13132 [gr-qc] .
  * Ng  _et al._ [2020] C. Ng, L. Guillemot, P. C. C. Freire, M. Kramer, D. J. Champion, I. Cognard, G. Theureau, and E. D. Barr, A Shapiro delay detection in the pulsar binary system PSR J1811–2405, Mon. Not. Roy. Astron. Soc. 493, 1261 (2020), arXiv:2002.01009 [astro-ph.HE] .
  * Schneider  _et al._ [1992] P. Schneider, J. Ehlers, and E. E. Falco,  _Gravitational lenses_ (Springer, 1992).
  * Emparan [1995] R. Emparan, Pair creation of black holes joined by cosmic strings, Phys. Rev. Lett. 75, 3386 (1995), arXiv:gr-qc/9506025 .
  * Virbhadra and Ellis [2002] K. S. Virbhadra and G. F. R. Ellis, Gravitational lensing by naked singularities, Phys. Rev. D 65, 103004 (2002).
  * Virbhadra [2009] K. S. Virbhadra, Relativistic images of Schwarzschild black hole lensing, Phys. Rev. D 79, 083004 (2009), arXiv:0810.2109 [gr-qc] .