# Unified Lexical Representation for
Interpretable Visual-Language Alignment

Yifan Li1 Yikai Wang1 Yanwei Fu1 Dongyu Ru2
Zheng Zhang2 Tong He2$\dagger$
1Fudan University 2 Amazon Web Services
<EMAIL_ADDRESS>{yikaiwang19<EMAIL_ADDRESS>
{rudongyu, zhaz<EMAIL_ADDRESS>
Work completed during internship at AWS Shanghai AI Labs. Corresponding
authors

###### Abstract

Visual-Language Alignment (VLA) has gained a lot of attention since CLIP’s
groundbreaking work. Although CLIP performs well, the typical direct latent
feature alignment lacks clarity in its representation and similarity scores.
On the other hand, lexical representation, a vector whose element represents
the similarity between the sample and a word from the vocabulary, is a natural
sparse representation and interpretable, providing exact matches for
individual words. However, lexical representations is difficult to learn due
to no ground-truth supervision and false-discovery issues, and thus requires
complex design to train effectively. In this paper, we introduce LexVLA, a
more interpretable VLA framework by learning a unified lexical representation
for both modalities without complex design. We use DINOv2 as our visual model
for its local-inclined features and Llama 2, a generative language model, to
leverage its in-context lexical prediction ability. To avoid the false
discovery, we propose an overuse penalty to refrain the lexical representation
from falsely frequently activating meaningless words. We demonstrate that
these two pre-trained uni-modal models can be well-aligned by fine-tuning on
modest multi-modal dataset and avoid intricate training configurations. On
cross-modal retrieval benchmarks, LexVLA, trained on the CC-12M multi-modal
dataset, outperforms baselines fine-tuned on larger datasets (e.g., YFCC15M)
and those trained from scratch on even bigger datasets (e.g., 1.1B data,
including CC-12M). We conduct extensive experiments to analyze LexVLA.

##  1 Introduction

The development of Vision-Language Alignment (VLA) models has made great
progress [38, 43, 19, 23, 6] since the innovative CLIP [30] effectively learns
a shared latent space where text and image are well-aligned. This progress has
also boosted related fields like vision-language foundation models [17],
multi-modal understanding [15], and text-conditional generation [31]. However,
CLIP’s latent features pose challenges of interpretability issue for analyzing
individual factors’ impact. Additionally, CLIP’s visual model struggles to
learn patch-level features, and its text model are trained based on incomplete
and biased captions. These challenges reduces its overall effectiveness.

On the other hand, the lexical representation is known for its clarity as each
dimension corresponds to the similarity between the input and a specific
word/token from the vocabulary. Additionally, lexical representation can
naturally be made sparse, which means it’s efficient for tasks like search or
indexing in large-scale retrieval tasks. However, learning lexical
representation is difficult. The embedding vector is much larger than the CLIP
latent vector as the vocabulary size is usually much larger than CLIP’s
feature dimension. This poses several challenges: the lack of precise
supervision signals; the incomplete and biased text supervision, and the high-
dimensional embedding space with low-dimensional targets. Thus using lexical
representations in CLIP-style VLA models is tough.

Figure 1:  LexVLA can generate a lexical representation of the input image
(the first word cloud figure), or to pick some patches of the image for local
lexical information representation (the second word cloud figure, with the
seleted patches boxed in red), and to select the most relevant patches of the
image given the text content (the rightmost figure, with caption ’horse’, with
the second-to-last figure is the ground-truth mask).

Previous lexical representation learning approaches tries to utilize
strategies including $\ell_{1}$ regularization [41, 3] to direct encourage
sparisity or use ReLU activation [24] to remove negative scores [10, 5, 22] or
directly regularize the floating-point operations [28]. However, these
intricate training configurations usually introduce extra training difficulty
and additional regularization such as log saturation [10, 22] and bag-of-words
penalization [43] which potentially limits the VLA capacity.

In this paper, we propose LexVLA, a simple yet comprehensive framework for
learning a unified lexical representation in the CLIP-style contrastive
training pipeline, facilitating the VLA. LexVLA enjoys the following merits in
addressing the challenges above. 1) We use single-modal pre-trained models for
their unique benefits, including DINOv2 [27] as vision model for its local-
inclined features, and Llama 2 [36] as text model to learn lexical
representations through in-context prediction tasks instead of traditional
caption embeddings. 2) We suggest a unified vocabulary with distinctive
codebooks for each modality to maintain the strengths of single-modal pre-
trained models by refrain them from learning the same feature space. This
allows us to create better VLA models with less multi-modal training data and
simpler setups. We also introduce an overuse penalty to prevent excessive
activation of irrelevant tokens, improving upon the FLOPs loss [28] which only
promotes sparsity. 3) We incrementally train LexVLA to stay aligned with pre-
trained models. For text, we fine-tune Llama 2 with LoRA adapter[12] and keep
its codebook frozen. For vision, we freeze DINOv2 and train a projector with a
self-attention layer and two multi-layer perceptions to the vision codebook,
initialized using the text codebook. 4) We introduce PatchDis, a metric for
patch-level alignment in visual features, evaluating patch-level
classification tasks with image patch features and category text tokens
learned by VLA models. We propose PatchDis as a patch-level interpretability
metric for VLA models not trained on fine-grained tasks like segmentation or
detection.

Formally, our LexVLA employs a unified lexical vocabulary with distinct
codebooks for text and image modalities, utilizing bi-encoders to individually
encode inputs into lexical representations. These encoders, initially set with
single-modal pre-trained models, undergo incremental fine-tuning via the
standard contrastive learning objective, with the integration of an overuse
penalty to promote sparsity and mitigate token over-activation. Emprically, we
show the effectiveness of LexVLA on two zero-shot cross-modal retrieval
benchmarks. Trained on the CC-12M dataset, it outperforms baselines fine-tuned
on larger datasets like YF100m-CLIP + CC-12M and those trained from scratch on
even larger datasets totaling 1.1B data, including CC-12M. We provide detailed
analysis of LexVLA.

Our contributions can be listed as follows:

  * •

We emphasize utilizing single-modal pre-trained models for vision-language
alignment tasks to benefit from their unique properties that cannot learned by
contrastive objectives. To exemplify, we use DINOv2 for its local-inclined
features and Llama 2 for its in-context capacity.

  * •

We effectively learn a unified lexical representation with unique codebooks
for vision and language modalities to refrain from the weakening of pre-
trained capabilities.

  * •

We propose an overuse penalty to encourage sparse embedding and prevent
meaningless activation.

  * •

We enjoy superior retrieval performance with less multi-modal training data,
and achieve better patch-level interpretable VLA model with global supervision
signals, quantitatively surpassing CLIP-style and lexical methods with our
proposed patch-level interpretability metric PatchDis.

##  2 Related works

Vision-Language Alignment VLA has been a challenging problem in deep learning.
The innovation of CLIP [30] has inspired a plethora of research efforts in
combining visual and language modalities [18, 39, 23, 33, 6, 38]. With the
rise of large language models (LLMs) [1, 36, 34], it’s become common to use
CLIP to get visual data from images and combine it with pretrained LLMs[21, 7,
17, 2, 44]. However, recent studies have identified visual encoders as
potential bottlenecks in these models, posing issues like identifying actions
and spatial relationships [37, 13], ignoring attributes and states [40, 9],
introducing hallucination problems [35], etc. Nevertheless, progress in using
other visual models instead of CLIP has been slow. Tong et al.[35] discovered
a drop in instruction-following ability when integrating DINOv2[27] into
LLaVA’s [21] structure. To address this, they combined features from CLIP and
DINOv2. Similarly, Zhai et al. [42] used a locked DINO for contrastive tuning,
which necessitated significant aligned data and computational resources. In
contrast, we use a locked DINOv2 for lexical VLA, demonstrating the
superiority over CLIP with smaller amount of multi-modal training data.

Multi-modal lexical representation Lexical representation is popular in
information retrieval [10, 11] thanks to its interpretability and efficiency.
However, adapting this approach to vision-language models is challenging
because of the differences between the two modalities. Previous approaches
rely on complex training stages (two [22] or three [5]), large scale multi-
modal training data [5] to train effectively and additional Bag-of-Words (Bow)
restrictions [22, 43] to prevent semantic misalignment. In contrast, we in
this paper propose a single fine-tuning stage to align uni-modal pre-trained
models in the output lexical space with less multi-modal training data and
without BoW restriction. In this paper, our model shall be equiped with the
generalizable ability of for zero-shot cross-modal evaluation, rather than
fine-tuning on each specific target cross-modal training dataset.

Figure 2:  Framework of LexVLA. We learn a unified lexical representation with
distinct codebooks for text and visual modalities. For the image, we adopt the
frozen DINOv2, learn an adapter and a mapper to get visual lexical
representation. For the text, we use LoRA to fine-tune the Llama 2 on in-
context lexical prediction task, followed with a mapper to get the text
lexical representation. We initialize codebooks as Llama 2’s codebook, freeze
the text codebook while fine-tuning the visual codebook. We train LexVLA with
the standard contrastive objectives along with the proposed overuse penalty to
encourage sparsity while preventing meaningless activation.

##  3 LexVLA

Problem setup Our target is to learn the lexical alignment between text and
image modalities. Typically, the lexical representation
$s_{i}\in\mathbb{R}^{V},i\in\\{\textrm{img, txt}\\}$ is a score vector, whose
element indicates the similarity between the sample and the corresponding word
from the vocabulary $\mathbf{V}$ with $V=|\mathbf{V}|$. The contrastive
lexical alignment aims to train the lexical encoders for each modality such
that the similarity $\left<s_{img},s_{txt}\right>$ for positive image-text
pairs are maximized and that for negative pairs are minimized, while
preserving the correct lexical correspondence with the vocabulary.

Overview Our model, as in Fig. 2, learns a single lexical representation for
VLA (Sec. 3.1). We encode each modality separately using uni-modal encoders
(Sec. 3.2) adapted in Sec. 3.3.

###  3.1 Lexical representation

Vocabulary Inspired by tokenization techniques [32, 16], we initialize with
the tokenizer vocabulary and refine it by removing unused and meaningless
tokens, reducing the size from 32,000 to 17,149.

Codebooks The one-hot embedding is semantic-less and assume the same distance
between different words, which is obviously not a good embedding. In this
paper, we use the output codebook of language models as a well pre-trained and
semantic-rich codebook, inducing lexical codes of $4096$-dim vectors. However,
the text codebook is not optimized for visual features. Basically, during the
pre-training of language model, the visual world is not observable, thus we
may have a gap between text-only embeddings and the visual-embeddings. To this
end, we propose to use the same vocabulary $\bm{V}$ but a unique codebook
$\bm{Z}_{i},i\in\\{\textrm{img, txt}\\}$ for text and image modalities,
respectively. As the language codebook is pre-trained on large corpus, we
freeze $\bm{Z}_{txt}$ to enjoy the well-trained text embeddings.
$\bm{Z}_{img}$ is initialized with the weights of $\bm{Z}_{txt}$ and fine-
tuned along with the training of lexical encoders to enable the adaptation for
visual features.

Sparse representation Thanks to the inherent similarity measurement in lexical
representations, it is natural to turn a dense output vector into a sparse
lexical representation. Typically we could utilize the following strategies:
1) Top-k thresholding: select the most important $k$ items and discard the
rest; 2) Value thresholding: keep items with value above the threshold and
discard the rest. In this paper, we use the value thresholding strategy. We
select items with values greater than $1/\sqrt{V}$, which means we consider
only the items with above-average signals as informative.

###  3.2 Lexical encoder

Given the image as $\bm{x}_{img}$ and text as $\bm{x}_{txt}$, the lexical
representation is achieved by:

$\bm{s}_{i}:=e_{i}(\bm{x}_{i})=h_{i}\circ g_{i}\circ
f_{i}(\bm{x}_{i}),i\in\\{\textrm{img, txt}\\}.$ (1)

Without loss of generality, we ignore the subscript $i$ in the following
definitions unless otherwise specified. We split the lexical encoder in three
stages: 1) The single-modal pre-trained feature extractor $f$, takes as input
$\bm{x}$ and as output a feature sequence
$\bm{y}:=f(\bm{x})\in\mathbb{R}^{n\times d_{i}}$, where $n$ is the sequence
length and $d_{i}$ is the modal-dependent feature dimension; 2) The projector
$g$, projects $\bm{y}$ to the lexical feature sequence
$\bm{z}:=g(\bm{y})\in\mathbb{R}^{n\times d}$, where $d$ is the lexical feature
dimension; 3) The mapper $h$, maps $\bm{z}$ to the lexical representation
$\bm{s}=h(\bm{z})\in\mathbb{R}^{V}$.

In this paper, we restrict the lexical representation to be non-negative and
$\ell_{2}$-normalized, i.e., $\|\bm{s}\|_{2}=1,s_{i}\geq 0$. Different
modalities share the same $\bm{V}$ with a unique codebook
$\bm{Z}_{i}\in\mathbb{R}^{V\times d},i\in\\{\textrm{img, txt}\\}$.

####  3.2.1 Lexical text encoder

Captioning? In VLA training data pair, the text is typically a caption of the
corresponding image. However, it is well-known that caption only captures a
partial observation of the semantics in the image, and cannot serve as a
perfect target to represent the image. As it is the only accessible ground-
truth signal, previous approaches mainly train the text encoders to represent
the caption embedding via the output hidden state [8, 43, 10, 22] or [CLS]
token embeddings [30], and use it as the target for the corresponding image.
Hence, the learned alignment is biased and leads to false-elimination of image
patterns that are not observed in the captions.

Predicting! Inspired by the powerful Large Language Models (LLMs), we would
like to investigate can we unleash the inherent knowledge of LLMs for lexical
representation? The answer is yes.

As the auto-regressive LLMs [1, 36] are not naturally to summarize the
previous tokens, it is irrational to directly use the predicted token as the
embedding for raw text inputs. A straightforward way to activate LLM is to use
prompt like “summarize the sentence of [TEXT] in one word:” to activate LLM’s
summarization capacity and use the output token as the text embedding. But it
is still the caption-style summarization, and cannot fully utilize the LLM.

LLM as lexical predictor Given the in-context learning capacity of LLM, we
propose to adapt LLM as a lexical predictor. Specifically, our input for the
LLM is:

The focus of "The man is riding a white horse." lies on important words:"man",
"riding", "white", "horse". The focus of "[TEXT]" lies on important words:

The [TEXT] is the input caption. Then we adopt the output token as the text
embedding. This prompt includes two parts: 1) One in-context example to guide
the LLM to identify the important words in the caption, adapting the LLM to
lexical prediction task; 2) The question prompt to ask the LLM the important
words of input caption. In this design, the output token of the LLM naturally
performs the lexical prediction task to calculate the similarity between the
input caption and every word in the vocabulary, and thus serves as a powerful
proxy to get the lexical representation.

Realization we utilize Llama 2 [36] as our text encoder $f_{txt}$. The input
text follows the above in-context prompts. We use the output of the predicted
token embedding as $\bm{y}_{txt}$. As our lexical codebook is Llama 2’s output
codebook, we do not need $g_{txt}$, thus $\bm{z}_{txt}=\bm{y}_{txt}$. To
implement $h_{txt}$, we first calculate dot-product attention between each
text token $\bm{z}_{txt}\in\mathbb{R}^{1\times d}$ and the text lexical
codebook $\bm{Z}_{txt}$. Then we use the $elu1p$ activation [43] to transform
the attention score into non-negative values and then normalize it, yielding
global lexical representation $\bm{s}_{txt}$. Formally,

$\bm{s}_{txt}=\mathrm{Normalize}\circ elu1p(\bm{z}_{txt}\bm{Z}_{txt}^{\top}),$
(2)

where $elu1p(x)=(x+1)\cdot 1_{\\{x\geq 0\\}}+e^{x}\cdot 1_{\\{x<0\\}}$ and
$\mathrm{Normalize}$ is the $\ell_{2}$ normalization operator. For word-level
lexical representation, we directly use the word code in $\bm{Z}_{txt}$
followed by Eq. (2).

####  3.2.2 Lexical visual encoder

We adopt DINOv2 [27] as our visual backbone to implement $f_{img}$. Given
input $\bm{x}_{img}$, we first flatten it into patches
$\bm{x}_{img}=[\bm{x}_{1},\dots,\bm{x}_{n-1}]$ and input to $f_{img}$ to get

$\bm{y}_{img}=f_{img}([\texttt{CLS};\bm{x}])\in\mathbb{R}^{n\times d},$ (3)

where CLS is the “class” token in DINOv2. Subsequently, we implement $g_{img}$
with an adapter includes a self-attention layer and 2 multi-layer perceptions
to post-process the DINOv2 features. To implement $h_{img}$, similar to the
text encoder but in a more fine-grained manner, we first calculate dot-product
attention between each image patch token $\bm{z}_{img,i}\in\mathbb{R}^{1\times
d}$ and the image lexical codebook $\bm{Z}_{img}$, followed by the $elu1p$
activation. In the end, we use max-pooling to aggregate patch representations
and then normalize, yielding global lexical representation $\bm{s}_{img}$.
Formally,

$\bm{s}_{img}=\mathrm{Normalize}\circ\textrm{Max-Pool}\circ
elu1p(\bm{z}_{img}\bm{Z}_{img}^{\top}).$ (4)

The image patch lexical representation follows the same process but replace
the max-pooling operator with selecting the corresponding patch location.

###  3.3 Train LexVLA

Contrastive objective We use the InfoNCE loss [26] with learnable temperature
$\tau$ for cross-modal retrieval over a batch of $N$ text-image pairs
$(x_{img},x_{txt})$ as the major objective:

$\ell_{a2b}=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{\exp(a_{i}b_{i}^{\top}/\tau)}{\sum_{j=1}^{N}\exp(a_{i}b_{j}^{\top}/\tau)},(a,b)\in\\{(x_{img},x_{txt}),(x_{txt},x_{img})\\},$
(5)

Overuse penalty In lexical representation learning, the FLOPs loss [28] is
widely used to encourage the output sparsity. Given the modality-specified
lexical representation matrix
$\bm{S}=\\{s_{i,j}\\},i\in\\{1,\dots,N\\},j\in\\{1,\ldots,V\\}$ in current
mini-batch, the FLOPs loss is defined as:

$\ell_{\mathrm{FLOPs}}=\sum_{j=1}^{V}\bar{s}^{2}_{\cdot,j}=\sum_{j=1}^{V}\left(\frac{1}{N}\sum_{i=1}^{N}s_{ij}\right)^{2}.$
(6)

The FLOPs loss aims to reduce FLOPs to induce sparsity. But we noticed that
sometimes this penalty pushes the encoder to take shortcuts that falsely
activate rarely used and image-unrelated tokens. This semantic deviation leads
to false activation and affects the interpretability of lexical
representation.

To prevent using meaningless tokens too much, we adapt the FLOPs loss with a
weighting method. Essentially, we aim to penalize tokens that are used too
often across examples. We measure this via the normalized average activation
value across the vocabulary, and propose the overuse penalty as:

$\ell_{\mathrm{overuse}}=V\sum_{j=1}^{V}\frac{\bar{s}_{\cdot,j}}{\sum_{k=1}^{V}\bar{s}_{\cdot,k}}\bar{s}^{2}_{\cdot,j}=NV\sum_{j=1}^{V}\left(\sum_{i=1}^{N}s_{i,j}/N\right)^{3}/\sum_{j=1}^{V}\sum_{i=1}^{N}s_{i,j}.$
(7)

Objective The full objective contains Eq. (5) and Eq. (7) for each modality:

$\mathcal{L}=\ell_{t2i}+\ell_{i2t}+\lambda_{I}\ell_{overuse}^{I}+\lambda_{T}\ell_{overuse}^{T},$
(8)

where $\lambda_{I}$ and $\lambda_{T}$ are two regularization weights for image
and text representations, respectively.

Incremental fine-tuning For text part, we adopt LoRA adapters [12] to fine-
tune the Llama 2. For vision part, we freeze the DINOv2, and only train the
projector and vision codebook.

###  3.4 PatchDis: interpretability metric

The core target of lexical representation is to achieve an interpretable
feature representation for input modality. For the text input, the
interpretability is straightforward to check. However, existing literature do
not have a quantitative metric to measure the patch-level interpretability of
visual lexical representation. In this paper, we propose the PatchDis metric
to evaluate the patch-level discrimination tasks, inspired by the patch-level
segmentation task but designed for models which are not trained from fine-
grained alignment tasks like segmentation or detection.

Basically, PatchDis evaluates the classification task in the patch-level
features. For any given VLA model, we use its text encoder to input the class
names and obtain the text embeddings of all classes. Similarly, we input the
testing image to the visual encoder to obtain all patch features. Then we
could use the model-defined metric to calculate the similarity between each
patch feature and the class embeddings, and predict the class of each patch
from the largest similarity. Then for each class, we could aggregate all
patches of one class as the patch-level prediction for the class. We use the
ground-truth segmentation of the class as the target, and use mIoU as the
metric to evaluate the patch-level interpretability of the VLA models in the
zero-shot patch-level discrimination task.

Table 1: Zero-shot cross-modal retrieval. indicates variants of our LexVLA. CLIP1 is the original CLIP [30]; results denoted by $(\cdot)^{2}$ are reported in VDR [43]; results denoted by $(\cdot)^{3}$ are reported in STAIR [5]. “Data” is the multi-modal alignment training data size; “Latent” means direct latent feature alignment methods; “Lexical” indicates lexical feature alignment methods. R@K, the recall ratio within top-K items.  Setting | Model | Data | MSCOCO | Flickr30k
---|---|---|---|---
image-to-text | text-to-image | image-to-text | text-to-image
R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10
Latent |  CLIP2 | 15M | 20.8 | 43.9 | 55.7 | 13.0 | 31.7 | 42.7 | 34.9 | 63.9 | 75.9 | 23.4 | 47.2 | 58.9
FILIP2 | 15M | 21.6 | 46.7 | 59.0 | 13.7 | 31.7 | 41.6 | 46.3 | 74.4 | 83.2 | 30.7 | 58.2 | 68.6
CLIP-BERT2 | 15M | 23.9 | 47.8 | 60.3 | 13.6 | 33.8 | 45.1 | 44.1 | 71.2 | 80.7 | 27.8 | 54.7 | 65.9
DeCLIP2 | 15M | 25.3 | 51.2 | 63.4 | 16.6 | 35.2 | 45.4 | 51.3 | 80.7 | 88.5 | 35.5 | 63.0 | 73.0
SLIP2 | 15M | 27.7 | 52.6 | 63.9 | 18.2 | 39.2 | 51.0 | 47.8 | 76.5 | 85.9 | 32.3 | 58.7 | 68.8
ProtoCLIP2 | 15M | 30.2 | 55.1 | 66.5 | 16.9 | 37.9 | 49.4 | - | - | - | - | - | -
CLIP1 | 0.4B | 52.4 | 76.7 | 84.6 | 33.1 | 58.4 | 69.0 | 81.8 | 96.2 | 98.8 | 62.1 | 85.6 | 91.8
CLIP3 | 1.1B | 53.4 | 78.3 | 85.6 | 36.2 | 62.2 | 72.2 | 79.6 | 95.5 | 98.1 | 63.0 | 86.7 | 92.5
Lexical |  VDR2 | 15M | 30.9 | 54.5 | 65.4 | 17.4 | 38.1 | 49.7 | 51.0 | 79.3 | 86.7 | 32.4 | 60.1 | 70.7
STAIR3 | 1.1B | 57.7 | 80.5 | 87.3 | 41.4 | 65.4 | 75.0 | 81.2 | 96.1 | 98.4 | 66.6 | 88.7 | 93.5
Lexical |  (CLIP) | 12M | 51.8 | 75.5 | 84.1 | 36.8 | 62.5 | 72.7 | 82.9 | 96.2 | 98.7 | 65.2 | 88.3 | 93.2
(FLOPs) | 12M | 56.2 | 80.0 | 87.4 | 39.0 | 65.7 | 75.6 | 84.2 | 96.6 | 98.7 | 67.4 | 89.4 | 94.1
(512) | 12M | 56.4 | 79.9 | 87.5 | 38.1 | 64.6 | 74.9 | 84.5 | 97.3 | 99.0 | 65.7 | 89.3 | 93.8
LexVLA | 12M | 55.4 | 80.6 | 88.3 | 39.8 | 66.3 | 76.2 | 83.9 | 97.5 | 99.1 | 67.8 | 90.2 | 94.2

##  4 Experiments

Datasets We use CC-12M [4] for training, a dataset consisting of 12.4 million
image-text pairs. We successfully download 9.2M pairs and use this subset as
our training set. For evaluation, we use Flickr30k [29] and MSCOCO [20] to
evaluate zero-shot cross-modal retrieval tasks.

Implementation We use DINOv2 [27] base model and Llama 2 [36] 7B model as our
backbones. We use Adam optimizer [14] with learning rate $5e-4$ and cosine
decayed, batch size of 6,144, precision of BFloat16 for 12 epochs. We
initialize $\tau$ as 0.07, and clip the logits larger than 100 as in [30]. We
use 8 A100 GPUs of 40GB memory to train LexVLA. We quadratically warmup
$\lambda$ in the fisrt 2k steps and then freeze as [28]. We set $\lambda_{I}$
as $5e-4$ and $\lambda_{T}$ as $1e-3$. The trainable parameters in LexVLA is
109 M in total, including 70M for vision codebook, 17M for vision projector
(19.76% compared with DINOv2), 21M for Llama adaptor (0.30% compared with
Llama 2).

###  4.1 Zero-shot cross-modal retrieval

Evaluation We conduct experiments on zero-shot cross-modal retrieval tasks on
Flickr30k and MSCOCO. We use the R@K, recall ratio within top-K items, as the
evaluation metric.

Competitors We compare LexVLA with the latent feature alignment methods,
including the original CLIP [30] model trained on different dataset scales and
followup CLIP-style alignment models, including FILIP [38], CLIP-BERT [43],
DeCLIP [19], SLIP [23], and ProtoCLIP [6]. We also compare with previous
lexical representation learning models that focus on zero-shot cross-modal
retrieval tasks, including VDR [43] and STAIR [5]. Note that all the
competitors are trained by larger datasets. Specifically, the models trained
on 15M data uses YFCC15M [18], and models trained on 1.1B data [5] contains
several datasets including CC-12M.

Variants For LexVLA, we report the following variants in addition to our final
model: 1) (CLIP), which uses the original CLIP visual model, instead of the
proposed DINOv2 based encoder; 2) (d), which uses the top-k thresholding
strategy. We test $d=512$, the same activated dimension as CLIP to make a fair
comparison. The average activated dimension of the final LexVLA is 1081. Note
that LexVLA surpasses CLIP with less activated values, see Fig. 5 for detailed
analysis.

Results We report the results in Table 1. 1) LexVLA performs significantly
better than both latent feature alignment methods and previous lexical
alignment methods when trained on a similar amount of multi-modal examples
(12M vs. 15M). This shows the effectiveness of LexVLA. 2) Compared to CLIP
trained on much larger multi-modal datasets (0.4B and 1.1B), LexVLA still
performs better, even when using the same activated feature dimension as CLIP.
This further demonstrates LexVLA’s efficiency in text-image alignment with
much less training data. 3) LexVLA outperforms STAIR in most metrics and is
comparable in two others, despite using significantly less training data (12M
vs. 1.1B). This highlights LexVLA’s training efficiency. 4) The significant
improvements of LexVLA over (CLIP) indicate that using DINOv2 as the visual
backbone is superior. 5) LexVLA with FLOPs loss performs worse than our
proposed overuse penalty, and activates meaningless tokens as in Fig. 6. In
summary, these results demonstrate the effectiveness and efficiency of our
method.

###  4.2 Lexical representation analysis

Table 2: PatchDis results. Model | mIoU
---|---
Random Dis. | 5.0
CLIP | 5.3
VDR | 12.6
LexVLA | 36.3

In this part, we evaluate if LexVLA learns the accurate lexical
representation. We conduct the following experiments: 1) Quantitatively, we
use the proposed PatchDis metric to evaluate the fine-grained patch-level
visual lexical representations on MSCOCO 2017 [20]; 2) Qualitatively, we
visualize the global lexical representation of images, local lexical
representation of image patches, and a PatchDis visualization for the
activated patches for the given category.

Figure 3: PatchDis visualization. The same color indicates the same category.
LexVLA correctly predicts the corresponding region, even for the small-scale
objects, like the bottle in the first image.

PatchDis evaluation. We report the zero-shot PatchDis mIoU performance on
MSCOCO 2017 validation dataset. We compare LexVLA with public available
baselines CLIP (0.4B) and VDR (15M), as well as random discrimination, and
report the performance in Tab. 2. LexVLA clearly shows the superiority on the
patch-level interpretability with the weak global training signal. In
contrast, the latent alignment CLIP, due to no supervision on patch-level
features, performs slightly better than random guessing. VDR enjoys better
interpretability, but significantly worse than our LexVLA. We also visualize
the predicted patches of different classes in Fig. 3. LexVLA successfully
activates the correlated patches in the image for different classes, even for
the small-scale objects.

Figure 4:  Visualization of the image lexical representation obtained by
LexVLA. Larger word indicates larger lexical value. The first row represents
the complete image, and the second row represent local patches (boxed in red).
LexVLA learns a well-aligned lexical representation for both image and patches
without local supervision.

Lexical visualization We visualize the lexical representations of the global
image and local patches in Fig. 4. We use word cloud [25] to illustrate the
lexical representation where larger word indicates larger lexical value. These
results clearly suggest that the learned lexical representation correctly
correlates with the semantics of the image/patch, showcasing the effectiveness
of LexVLA.

###  4.3 Further analysis

Figure 5:  Retrieval in different sparse levels. We compare LexVLA with VDR in
the same sparse levels and with CLIP as a proxy of dense latent alignment. The
first row is results in the Flickr30K dataset, and the second in the MSCOCO
dataset. The first to the third columns show the settings of Recall@1,
Recall@5, and Recall@10, respectively. Purple symbols represent CLIP.

LexVLA under different sparsity One of the targets for lexical representation
is to learn a sparse representation for potential benefits in large scale
retrieval tasks. To test the robustness of learned representations in LexVLA,
we test the retrieval performance of LexVLA in different sparsity levels.
Particularly, the sparsity ratio is calculated via $\ell_{0}(s_{i})/V$, the
ratio of non-zero elements in the lexical representation $s_{i}$. We compare
LexVLA with VDR on the same sparsity ratios. We also visualize the CLIP’s
performance as a proxy of dense latent representation.

Results are shown in Fig. 5. 1) LexVLA is robust against the sparsity ratio
even in a very high ratio (98.27%, 296 activated tokens), and then starts to
get damaged as the sparse ratio approaches 1; 2) Compared with VDR, LexVLA
achieves better performance in all sparse level, indicating a consistent
improvement. 3) LexVLA enjoys superior performance with less activated feature
dimension as CLIP (296 vs 512) in almost all cases, showcasing our model’s
effectiveness.

Figure 6:  Visualization of lexical representations of LexVLA w/ FLOPs loss
and w/ our proposed overuse penalty, respectively. The first row represents
images while the second is caption. LexVLA w/ FLOPs loss usually falsely
activates similar meaningless tokens frequently as a shortcut (see ’@’ and
’contemporary’ outlined by green boxes as examples). LexVLA w/ overuse penalty
significantly mitigate this false activation issue.

Overuse penalty The widely used FLOPs loss aims to encourage the sparsity of
learned lexical representation. However, we empirically find that it will
falsely activate meaningless tokens as in Fig. 6. Specifically, the model
tends to activate certain semantically unrelated tokens across different
$s_{i}$ (like ’@’ in the image and ’contemporary’ in the text, respectively),
implicitly treating these tokens as latent dimensions. Our proposed overuse
penalty inhibits these frequently activated words more aggressively, and leads
to more semantic-correlated lexical representations in Fig. 6. This enhanced
interpretability further improves the retrieval performance in most cases, as
in Tab. 1.

Limitations The major limitation of LexVLA is the use of vocabulary from large
language model. We design our lexical vocabulary based on the Llama 2’s
tokenizer, and sometimes it splits a word into several sub-word tokens, making
the resulting token not a complete word. Although we filter out many
meaningless tokens, it still introduces the gap between our vocabulary and the
perfect word-level vocabulary. Given that random initialize a word-level
vocabulary and additionally learn a projector from sub-word tokens to word-
level tokens works poorly, we regard designing a word-level vocabulary that
still benefit from the LLMs as a future work.

Broader impacts The advancements made in cross-modal retrieval, as
demonstrated in this work, hold significant potential for various fields and
applications. One immediate impact lies in enhancing information retrieval
systems, where users can efficiently search and retrieve multimodal content
such as images and text. This can greatly benefit industries such as
e-commerce, digital libraries, and multimedia databases, enabling more
intuitive and accurate searches. While the advancements in cross-modal
retrieval offer numerous benefits, one concern is related to privacy and
security, as the integration of multimodal data may raise issues regarding
data protection and user consent. It’s essential to mitigate these potential
negative impacts through robust privacy measures.

##  5 Conclusion

In this paper, we propose LexVLA, a unified lexical vision-language alignment
framework. LexVLA effectively learn a unified lexical representation with
unique codebooks for vision and language modalities, with the aid of single-
modal pre-trained models. We propose to incremental fine-tune LexVLA with the
standard contrastive objective penalized by the proposed overuse objective to
prevent meaningless activation and encourage sparse embedding. LexVLA refrains
from the complex training configurations, and effectively achieves the patch-
level interpretability with only global supervision signals. We introduce a
patch-level interpretability metric, PatchDis, to quantify this on the zero-
shot cross-modal benchmark dataset. We demonstrate the effectiveness of LexVLA
on two zero-shot cross-modal retrieval benchmark datasets with significantly
smaller multi-modal training dataset. We also provide detailed analysis on
LexVLA.

## References

  * [1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.
  * [2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  Advances in neural information processing systems, 35:23716–23736, 2022.
  * [3] Usha Bhalla, Alex Oesterling, Suraj Srinivas, Flavio P Calmon, and Himabindu Lakkaraju.  Interpreting clip with sparse linear concept embeddings (splice).  arXiv preprint arXiv:2402.10376, 2024.
  * [4] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3558–3568, 2021.
  * [5] Chen Chen, Bowen Zhang, Liangliang Cao, Jiguang Shen, Tom Gunter, Albin Madappally Jose, Alexander Toshev, Jonathon Shlens, Ruoming Pang, and Yinfei Yang.  Stair: Learning sparse text and image representation in grounded tokens.  arXiv preprint arXiv:2301.13081, 2023.
  * [6] Delong Chen, Zhao Wu, Fan Liu, Zaiquan Yang, Yixiang Huang, Yiping Bao, and Erjin Zhou.  Prototypical contrastive language image pretraining.  arXiv preprint arXiv:2206.10996, 2022.
  * [7] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning.  Advances in Neural Information Processing Systems, 36, 2024.
  * [8] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [9] Sivan Doveh, Assaf Arbelle, Sivan Harary, Roei Herzig, Donghyun Kim, Paola Cascante-Bonilla, Amit Alfassy, Rameswar Panda, Raja Giryes, Rogerio Feris, et al.  Dense and aligned captions (dac) promote compositional reasoning in vl models.  Advances in Neural Information Processing Systems, 36, 2024.
  * [10] Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant.  Splade: Sparse lexical and expansion model for first stage ranking.  In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2288–2292, 2021.
  * [11] Luyu Gao, Zhuyun Dai, and Jamie Callan.  Coil: Revisit exact lexical match in information retrieval with contextualized inverted list.  arXiv preprint arXiv:2104.07186, 2021.
  * [12] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.  Lora: Low-rank adaptation of large language models.  In International Conference on Learning Representations, 2022.
  * [13] Amita Kamath, Jack Hessel, and Kai-Wei Chang.  What’s" up" with vision-language models? investigating their struggle with spatial reasoning.  arXiv preprint arXiv:2310.19785, 2023.
  * [14] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [15] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026, 2023.
  * [16] Taku Kudo and John Richardson.  Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, 2018.
  * [17] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  In International conference on machine learning, pages 19730–19742. PMLR, 2023.
  * [18] Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan.  Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm.  arXiv preprint arXiv:2110.05208, 2021.
  * [19] Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan.  Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm.  In International Conference on Learning Representations, 2022.
  * [20] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  * [21] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  Advances in neural information processing systems, 36, 2024.
  * [22] Ziyang Luo, Pu Zhao, Can Xu, Xiubo Geng, Tao Shen, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang.  Lexlip: Lexicon-bottlenecked language-image pre-training for large-scale image-text sparse retrieval.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11206–11217, 2023.
  * [23] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie.  Slip: Self-supervision meets language-image pre-training.  In European conference on computer vision, pages 529–544. Springer, 2022.
  * [24] Vinod Nair and Geoffrey E Hinton.  Rectified linear units improve restricted boltzmann machines.  In Proceedings of the 27th international conference on machine learning (ICML-10), pages 807–814, 2010.
  * [25] Layla Oesper, Daniele Merico, Ruth Isserlin, and Gary D Bader.  Wordcloud: a cytoscape plugin to create a visual semantic summary of networks.  Source code for biology and medicine, 6(1):7, 2011.
  * [26] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  arXiv preprint arXiv:1807.03748, 2018.
  * [27] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al.  Dinov2: Learning robust visual features without supervision.  arXiv preprint arXiv:2304.07193, 2023.
  * [28] Biswajit Paria, Chih-Kuan Yeh, Ian E.H. Yen, Ning Xu, Pradeep Ravikumar, and Barnabás Póczos.  Minimizing flops to learn efficient sparse representations.  In International Conference on Learning Representations, 2020.
  * [29] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.  In Proceedings of the IEEE international conference on computer vision, pages 2641–2649, 2015.
  * [30] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [31] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  * [32] Rico Sennrich, Barry Haddow, and Alexandra Birch.  Neural machine translation of rare words with subword units.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, 2016.
  * [33] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao.  Eva-clip: Improved training techniques for clip at scale.  arXiv preprint arXiv:2303.15389, 2023.
  * [34] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.  Gemini: a family of highly capable multimodal models.  arXiv preprint arXiv:2312.11805, 2023.
  * [35] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie.  Eyes wide shut? exploring the visual shortcomings of multimodal llms.  arXiv preprint arXiv:2401.06209, 2024.
  * [36] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * [37] Fei Wang, Liang Ding, Jun Rao, Ye Liu, Li Shen, and Changxing Ding.  Can linguistic knowledge improve multimodal alignment in vision-language pretraining?  arXiv preprint arXiv:2308.12898, 2023.
  * [38] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu.  FILIP: Fine-grained interactive language-image pre-training.  In International Conference on Learning Representations, 2022.
  * [39] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.  Coca: Contrastive captioners are image-text foundation models.  arXiv preprint arXiv:2205.01917, 2022.
  * [40] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou.  When and why vision-language models behave like bags-of-words, and what to do about it?  In The Eleventh International Conference on Learning Representations, 2022.
  * [41] Hamed Zamani, Mostafa Dehghani, W Bruce Croft, Erik Learned-Miller, and Jaap Kamps.  From neural re-ranking to neural ranking: Learning a sparse representation for inverted indexing.  In Proceedings of the 27th ACM international conference on information and knowledge management, pages 497–506, 2018.
  * [42] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer.  Lit: Zero-shot transfer with locked-image text tuning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18123–18133, 2022.
  * [43] Jiawei Zhou, Xiaoguang Li, Lifeng Shang, Xin Jiang, Qun Liu, and Lei Chen.  Retrieval-based disentangled representation learning with natural language supervision.  In The Twelfth International Conference on Learning Representations, 2024.
  * [44] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  arXiv preprint arXiv:2304.10592, 2023.

##  Appendix A Appendix / supplemental material

###  A.1 Model hyperparameters

Our hyperparamters selection for the detail can be found in Tab. 3.

Table 3: Hyperparameters for training LexVLA. Hyperparameter | Value
---|---
Batch size | 6144
Learning rate | 5e-4
Vocabulary size | 17149
Training epochs | 12
Max temperature $\tau$ | 100.0
Warm-up iterations | 1000
LR Scheduler | Cosine
Adam $\beta_{1}$ | 0.9
Adam $\beta_{2}$ | 0.999
Adam $\epsilon$ | 1e-6
Penalty $lambda_{1}$ | 5e-4
Penalty $\lambda_{2}$ | 1e-3
Lora_alpha | 16
Lora_r | 8
Lora_dropout | 0.05


11institutetext: IGAM, Institute of Physics, University of Graz,
Universitätsplatz 5, 8010 Graz, Austria
11email<EMAIL_ADDRESS>22institutetext: Universidad Nacional de
Colombia, Observatorio Astronómico Nacional, Bogotá, Colombia 33institutetext:
Neuroimaging Science and Support Center-NISSC. Institute of Neurology, Kepler
University Hospital, Linz, Austria.

# Exploring magnetic field properties at the boundary of solar pores: A
comparative study based on SDO-HMI observations

J. I. Campos Rozo 11 S. Vargas Domínguez 22 D. Utz 33 A. M. Veronig 11 A.
Hanslmeier 11

###### Abstract

Context. The Sun’s magnetic fields play an important role in various solar
phenomena. Solar pores are regions of intensified magnetic field strength
compared to the surrounding photospheric environment, and their study can help
us better understand the properties and behaviour of magnetic fields in the
Sun. In this work, we investigate the properties of magnetic fields on the
boundaries of solar pores, specifically focusing on the evolution of the
vertical magnetic field.

Aims. Up to now, there exists only a single study on magnetic field properties
at the boundary region of a pore. Therefore, the main goal of this work is to
increase the statistics of magnetic properties determining the pore boundary
region. To this aim, we study the change of the vertical magnetic field on the
boundaries of six solar pores and their time evolution.

Methods. We analyse six solar pores using data from the Helioseismic and
Magnetic Imager instrument on board the Solar Dynamics Observatory. We apply
image processing techniques to extract the relevant features of the solar
pores and determine the boundary conditions of the magnetic fields. For each
pore, the maximal vertical magnetic field is determined, and the obtained
results are compared with the above-mentioned previous study.

Results. We find the maximal vertical magnetic field values on the boundaries
of the studied solar pores to range from 1400 G to 1600 G, with a standard
deviation between 7.8% and 14.8%. These values are lower than those reported
in the mentioned preceding study. However, this can be explained by
differences in spatial resolution as well as the type of data we used. For all
the pores, we find that the magnetic inclination angle lies in a range of
$30\pm 7$°, which is consistent with the idea that the magnetic field
configuration in solar pores is mainly vertical.

Conclusions.  The vertical magnetic field is an important factor in
determining the boundary of solar pores, and it plays a more relevant role
than the intensity gradient. The obtained information will be useful for
future studies on the formation and evolution of magnetic structures of the
Sun. Additionally, this study highlights the importance of high spatial
resolution data for the purpose of accurately characterising the magnetic
properties of solar pores. Overall, the findings of this work contribute to
the understanding of the magnetic field properties of the Sun and will be
crucial for improving models of solar dynamics and magnetic flux emergence.

###### Key Words.:

Sun: magnetic fields, Sun: photosphere, sunspots.

##  1 Introduction

During the past few years, the study of the Sun has experienced an incredible
advancement with the launch of space-based telescopes and instruments, such as
the Parker Solar Probe (Fox et al., 2016) and the Solar Orbiter (Müller et
al., 2020) as well as the first light from the ground-based four-meter solar
telescope the Daniel K. Inouye Solar Telescope (Rimmele et al., 2020), located
on Haleakalā, Maui, Hawaii. The understanding of the (small-scale) dynamics of
the Sun, the interaction of the plasma and the magnetic fields, and the
processes leading to large-scale eruptions in the solar atmosphere require
deeper study and understanding.

The emergence of magnetic flux from the solar interior causes the formation of
various features that can be observed in the solar photosphere with different
sizes, lifetimes, and magnetic field strengths (Cheung & Isobe, 2014). The
most distinguished magnetic features encountered over the solar surface are
sunspots (see Solanki, 2003, for a full overview regarding sunspots), which
can occur and form larger entities through the solar atmosphere called active
regions. It is well known that sunspots are regions harbouring a very
concentrated and organised magnetic field, reaching values from $1500$ G to
$3500$ G (Sobotka, 1999; Solanki, 2003). Thus, the magnetic field is strong
enough to freeze the plasma and is thus able to reduce the plasma dynamics and
its apparent motions in the surrounding environment (e.g. Campos Rozo et al.,
2019). The sunspot magnetic field features can be divided into two major
parts, umbra and penumbra, which have different magnetic and thermodynamic
properties (Borrero & Ichimoto, 2011; Rezaei et al., 2012). Because of the
strong intensity contrast differences, these two components of sunspots can be
easily identified by visual inspection.

Mature sunspots are of predominant interest due to their involvement in
complex magnetic field evolution and interaction with the upper atmospheric
layers as well as being the sources of the most energetic solar eruptions,
such as flares and coronal mass ejections (e.g. reviews by Green et al., 2018;
Toriumi & Wang, 2019). Solar pores, tiny sunspots lacking penumbrae, are
worthy of attention since most of the time they constitute the first stage of
a sunspot’s evolution. However, it is worth noting that not all solar pores
evolve into sunspots.

Figure 1: Time evolution of two of the studied pores with the 55% intensity
contours overplotted (red solid line). From left to right, snapshots during
the pore’s lifetime. The first row shows the evolution of pore 1, which
revealed strong changes of the area. The second row shows pore 5, which
revealed a quasi-stable area evolution. The accompanying movie shows the
evolution of each of the six pores under study.

It is possible that the evolution of some pores into sunspots may be
attributed to the emergence of additional flux, which has the potential to
alter the magnetic field topology around a pore. Such emergence events are
often responsible for the creation and observation of moving magnetic
features, which can be detected around certain pores prior to the formation of
a penumbra. These moving magnetic features could provide a crucial link to the
pre-existing magnetic topology and indicates the potential formation of
filamentary substructures (Keppens & Martinez Pillet, 1996; Leka & Skumanich,
1998; Zuccarello et al., 2009; Sainz Dalda et al., 2012).

Pores mainly harbour a simple configuration of vertical magnetic fields and
are therefore particularly suitable for studying the interaction and evolution
of emerging magnetic fields with a convective pattern around them (Vargas
Domínguez & Utz, 2022). As a result of a convective collapse, whereby magnetic
flux concentrations become too dense for convective motions to penetrate them,
the trapped magnetic field can inhibit the emergence of new magnetic flux,
forming the solar pores (Kitiashvili, 2013). Many studies go deep into the
analysis of the pore’s structure and dynamics while embedded in the granular
pattern (e.g. Sobotka et al., 1999; Dorotovič et al., 2002; Vargas Domínguez
et al., 2010; Ermolli et al., 2017, and references therein). Recently,
Gilchrist-Millar et al. (2021) probed that pores perform as magnetic
waveguides, after detection of propagating magnetohydrodynamic wave activity
above them. Solar pores have been the subject of numerous observational and
theoretical investigations, yielding substantial advancements in our
comprehension of these enigmatic features. Nonetheless, several questions
related to their behaviour and characteristics persist. These include
inquiries into the relationship between solar pores and sunspots, the
evolution of magnetic fields in solar pores over time, the potential impact of
solar pores on the upper solar atmospheric layers, and the contribution of
solar pores to the overall solar activity cycle.

In this paper, we study the magnetic properties of a sample of isolated solar
pores that do not evolve into sunspots using a space-based time series of
photospheric filtergrams and magnetograms from the Helioseismic and Magnetic
Imager (HMI; Hoeksema et al., 2014) on board the Solar Dynamics Observatory
(SDO; Pesnell et al., 2012). In Section 2, we describe the data and the
processing used in this study. The analysis and results are presented in
Section 3. Finally, we discuss our findings in the context of the magnetic
properties of pores on their boundaries in Section 4.

Figure 2: HMI continuum maps of the six different solar pores studied that
evolved with values of $\mu$ greater than 0.9. The red contours outline an
intensity value of 55% Ic, and the yellow contours outline the level of the
maximum vertical magnetic field Bver found in the present work.

##  2 Observational data and processing

The task of identifying isolated solar pores exhibiting substantial areas and
robust lifetimes (exceeding 24 hours) is a challenging task. To successfully
execute this endeavour, we used photospheric observations obtained from a
space-based telescope. We studied data from the HMI instrument on board SDO,
which observes the full solar disk at $6173$ $\AA$ with a pixel resolution of
$\sim 0.504$ arcsec/pixel. The instrument offers a diverse range of data
products, including continuum intensity images captured at cadences of 45
seconds and 720 seconds (12 minutes), line-of-sight (LOS) magnetic field
measurements acquired concurrently with the intensity maps, Milne-Eddington
inversions deduced from the Stokes vector observations utilising the Very Fast
Inversion of the Stokes Vector (Borrero et al., 2011) at a cadence of 12
minutes. The HMI data products can be accessed via the Joint Science
Operations Center (JSOC) database.111JSOC

Table 1: List of solar pores analysed in this work.

Pore | Starting | Ending | # Hours
---|---|---|---
number | datetime | datetime |
1 | 2011-07-10 | 2011-07-11 | 36
| 06:59:11 | 18:58:26 |
2 | 2013-08-23 | 2013-08-25 | 36
| 23:59:07 | 11:58:22 |
3 | 2014-06-02 | 2014-06-03 | 36
| 05:59:09 | 17:59:09 |
4 | 2015-01-18 | 2015-01-20 | 35
| 17:58:54 | 04:58:09 |
5 | 2015-05-22 | 2015-05-23 | 33
| 11:59:08 | 20:47:08 |
6 | 2018-05-12 | 2018-05-13 | 27
| 05:59:09 | 08:59:05 |
222Time and positional ranges of the analysed solar pores in universal time
(UT).

For this study, we analysed six solar pores (see Table 1) that evolved over a
period of about 36 hours. We selected solar pores that evolved when they were
located close to the solar disk centre and chose an observation window when
the distance of the pores from the central meridian was larger than 0.9 (as
given by $\mu$) in order to avoid projection effects during the analysis.
Another condition for our six pores under study was that their calculated
areas must be greater than $\sim$0.6 Mm2 (5 pix2) during the analysis time
range. Moreover, another requirement was that they did not break up into
smaller micropores and spread out over larger areas during this interval. A
sample of the pores is displayed in Fig. 1 and 2 at time instances when their
evolution was close to the maximum area during their lifetimes. All the six
studied solar pores were tracked over a field of view of $30\times 30$
arcseconds2.

The HMI continuum maps were corrected applying the centre-to-limb variation
algorithm333See routine hazel.util.i0_allen in HAZEL2 to account for the limb-
darkening effect and also normalised over the mean intensity value of the
surrounding quiet Sun (i.e. regions where the absolute value of the LOS
magnetic field is less than $50$ G). The threshold for the pore intensity
boundary was defined in a manner similar to García-Rivas et al. (2021),
varying the different levels for the intensity threshold in the range of
0.4-0.6 and identifying the best match to the solar pore boundary by visual
inspection, which was determined as $55\%$ of the intensity values of the
normalised map (example of intensity contours are shown in Fig. 2 by the red
solid contours).

In an effort to compute diverse magnetic parameters on the boundary of the
solar pores, we utilised the HMI vector magnetogram products denoted as
”hmi.B_720s”, available on the JSOC website. In particular, we used the
azimuth, inclination, and strength of the total magnetic field vector. We
resolved the ambiguity of the azimuth angle through the utilisation of the
disambiguation angle maps integrated into the hmi.B_720s products. As the
observations are gauged in the LOS reference system, it was imperative to re-
project the azimuth and inclination magnetic angles to the local reference
frame system via the r_frame_sphduo algorithm, a component of the AZAM
software package. We subsequently calculated the vertical magnetic field
component from the local reference frame inclination and total field strength.
It should be noted that pores 2 and 6 (see Fig. 3) exhibit negative polarity,
and the values for the inclination magnetic angle correspond to
$180^{o}-\gamma$, where $\gamma$ represents the angle. Similarly, the vertical
magnetic field was corrected by calculating the absolute values to the pores
of negative polarity. Figure 3 depicts the magnetic parameters employed in the
present analysis, with special emphasis on the vertical magnetic field (Bver)
in order to ascertain the critical threshold value at the boundaries of solar
pores. All the calculations and algorithms used in the present work have been
written (or adapted) in Python, and the specialised library for solar physics
Sunpy444https://sunpy.org/ (SunPy Community et al., 2015).

##  3 Analysis and results

Various investigations have been conducted to establish the boundary
demarcation between the umbra and penumbra (Jurčák et al., 2018; Schmassmann
et al., 2018), as well as the interface between the photospheric plasma
granulation and the border of solar pores (García-Rivas et al., 2021). Such
studies have established threshold values of the vertical magnetic field to
define the boundary of these features during their lifetime evolution. In the
present work, we examine the evolution of the magnetic parameters, namely,
inclination, total magnetic field, and vertical magnetic field, obtained from
the boundaries of the studied pores as well as the areas enclosed by the
boundary contours (see Fig. 4).

Figure 5 illustrates the temporal evolution of the analysed magnetic field
parameters. These parameters were extracted and calculated as follows: First,
the 55% threshold contour was derived for the intensity maps. That contour
line was then plotted over the magnetic field parameters maps (see Fig. 3) to
extract the quantities of interest along the boundary line of the pores. In a
final step, the extracted values were then averaged. The time evolution of the
magnetic parameters is displayed in Fig. 5. The evolution of the magnetic
field inclination, which indicates an average of approximately $30^{\circ}$,
is depicted by the green pattern in Fig. 5.

Figure 3: Snapshot of each pore during the 36-hour analysis interval. The
contour in every panel outlines the pore intensity value corresponding to 55%
of $I_{c}$, as in Fig. 2. From left to right: azimuth, inclination, total
magnetic field strength, and vertical magnetic field.

A summary of the statistical values is presented in Table 2. The column
labelled ”Average over the whole data” displays the values computed over the
entire period of the boundary evolution, whereas the column labelled ”Maximal
vertical field” indicates the values computed when the vertical magnetic field
values on the pore boundary attain the maximum value during the pore’s
evolution. A description of the individual magnetic parameters analysed for
the six studied solar pores is presented in the following subsections.

Figure 4: Temporal evolution of the areas calculated from the enclosed region
determined by the contours obtained from the continuum maps for the six pores
under study. (Cf. red contours in Figs. 1 and 2).

###  3.1 Magnetic field inclination $\gamma$

It is well established in the scientific literature that the magnetic field in
solar pores is predominantly vertical (e.g. Rucklidge et al., 1995; Sobotka,
2002; Rempel et al., 2009; Jurčák, 2011). This is also evidenced by the
temporal evolution of the magnetic field inclination angle on the boundaries
of solar pores, which varies on average in the range of 26-33∘, as depicted in
Fig. 5. By comparing the values in Table 2 and the areas of the pores in Fig.
4, we found that the larger the pores are, the higher is the mean inclination
angle on their intensity boundaries. This is in agreement with the results
obtained for sunspots (see Fig. 2 in Jurčák et al. (2018)).

###  3.2 Total magnetic field strength $B$

For the six solar pores under study, we obtained that the values of the total
magnetic field strength on the boundaries of the pores exhibit an average
range of approximately 1400 G extending up to values greater than $1600$ G,
with a mean standard deviation of $10$% during their evolution. Again, larger
pores tend to have greater magnetic field strengths on their boundaries, which
is consistent with the behaviour found for sunspots (see Fig. 2 in Jurčák et
al., 2018). Figure 5 provides evidence that the time evolution of these values
remains relatively constant over the pore’s lifetime, indicating a certain
level of stability for the magnetic field values on the boundaries of the
solar pores. Moreover, at the times when the vertical magnetic field attains
its maximum value, the strength of the total magnetic field in the studied
solar pores reaches values greater than $1600$ G, and the absolute maximum is
approximately $1750$ G. This value is still smaller than the maximum values of
the vertical component of the magnetic field strength found during the stable
phase of the evolution of the solar pore studied by García-Rivas et al. (2021)
(i.e. $\sim$1920 G), who used the same intensity threshold but deconvolved HMI
data. Also, Schmassmann et al. (2018) found a larger value of the total
magnetic field strength on the umbral boundary of a stable sunspot: $\sim$2170
G. They used the same HMI data product as we did but employed an intensity
threshold of 50% of the quiet Sun intensity (whereas we used a 55% threshold).
In this work, the magnetic field strength on the boundary of all the six pores
during the stable phase of their evolution (when their areas did not change
drastically) is thus weaker than what has been obtained in previous studies.
Although the values are not directly comparable, as we either used different
datasets or different intensity thresholds than previous studies, we mention
them as a reference to differentiate our results.

###  3.3 Vertical magnetic field strength $B_{ver}$

The vertical magnetic field vector, $B_{ver}$, shows a behaviour similar to
the total magnetic field strength. On the boundary of a pore in a stable
phase, García-Rivas et al. (2021) reported a maximum value of approximately
1730 G, while Schmassmann et al. (2018) reported a value of around 1630 G. We
found significantly lower values for the vertical component of the magnetic
field on the boundaries of the pores under study. The maximum values are
around 1500 G, and the global maximum reached in pore 3 (see Table 2) is
approximately 1570 G. However, as explained previously, one has to be cautious
when comparing the values of our study with previous ones, as the methods,
instruments, and data are slightly different. During the studied evolution
periods of the pores, we found the mean value of the vertical component of the
magnetic field to be between 1300 G and 1400 G. Similar to the evolution of
the magnetic field strength, we did not observe significant variations of Bver
during the time periods studied.

Table 2: Magnetic parameter statistics.

Pore | Magnetic | Average over | Maximal |
---|---|---|---|---
number | Parameter | the whole data | vertical field |
| Inclination [o] | $29\pm 6$ | $26\pm 8$ |
1 | Total field [$G$] | $1526\pm 105$ | $1655\pm 125$ |
| Vertical field [$G$] | $1322\pm 133$ | $1480\pm 150$ |
| Inclination [o] | $30\pm 9$ | $23\pm 5$ |
2 | Total field [$G$] | $1610\pm 138$ | $1665\pm 111$ |
| Vertical field [$G$] | $1386\pm 222$ | $1524\pm 121$ |
| Inclination [o] | $33\pm 6$ | $25\pm 7$ |
3 | Total field [$G$] | $1566\pm 130$ | $1602\pm 73$ |
| Vertical field [$G$] | $1312\pm 163$ | $1440\pm 112$ |
| Inclination [o] | $30\pm 7$ | $24\pm 9$ |
4 | Total field [$G$] | $1613\pm 131$ | $1746\pm 134$ |
| Vertical field [$G$] | $1390\pm 170$ | $1573\pm 201$ |
| Inclination [o] | $28\pm 7$ | $21\pm 8$ |
5 | Total field [$G$] | $1598\pm 202$ | $1653\pm 210$ |
| Vertical field [$G$] | $1394\pm 216$ | $1533\pm 226$ |
| Inclination [o] | $27\pm 8$ | $24\pm 10$ |
6 | Total field [$G$] | $1425\pm 99$ | $1589\pm 75$ |
| Vertical field [$G$] | $1258\pm 140$ | $1425\pm 152$ |

555The columns display the mean values plus or minus the standard deviation
calculated from the contours obtained on the boundary of the intensity images.
The inclination values are given in degrees [°], whereas the total and
vertical field values are given in Gauss [G]

Finally, Figs. 6 and 7 illustrate distinct correlations between the overall
magnetic field strength, vertical field strength, magnetic field inclination,
and the surface area of each pore over its lifespan. The red and blue colours
correspond to two distinct time ranges defined by the moment when the vertical
magnetic field reached its maximum value (indicated by the vertical dotted
line in Fig. 5) during the pore’s lifetime. The straight lines displayed in
the figures represent separate linear fits that were applied to the data
points in each of the two distinct phases.

Figure 5: Time evolution of the magnetic parameters obtained from the pore’s
boundary contours. The main, thicker line in each plot represents the mean
value obtained from the contours over each respective magnetic parameter,
whereas the corresponding shaded areas refer to the standard deviation also
calculated from the contours. Figure displays the magnetic field inclination
angle (green pattern), the total magnetic field strength (orange pattern), and
the vertical magnetic field (blue pattern) for the six pores under study. The
vertical dotted line in each panel denotes the time instant when the vertical
magnetic field average over the boundary contour reached its maximum value.

Overall, all analysed pores demonstrate similar behaviour, although pores 4
and 5 exhibit greater dispersions. Figure 6 reveals that the magnetic field
tends to be more vertical, with smaller inclination angles, during periods of
maximum vertical magnetic field strength. In addition, Fig. 7 discloses the
strong correlation between the magnetic inclination angle and the areas of the
pores. In all cases, the maximum values of the vertical magnetic field
strength are observed when the magnetic field inclination is close to either a
global or local minimum and coupled with a global or local maximum of the
magnetic field strength. This is not surprising, as the vertical magnetic
field is directly correlated with these two parameters due to it is calculated
as $B_{ver}=B\cdot\cos(\gamma)$.

##  4 Discussion and conclusions

All of the six stable pores selected for this study exhibit distinct physical
properties. For instance, as shown in Fig. 4, there is significant potential
for variation in the temporal evolution of the pore area. Even the smallest
structures in the group, pores $5$ and $6$, persist for over 24 hours before
their areas decrease to values below the threshold of $\sim 5$ pix2 set as the
minimum area. It is worth noting that pore $5$ does not exhibit a substantial
change in the areas, although it does have tiny fragmentations, as determined
by the intensity threshold estimation, during the analysed time period,
whereas other pores, such as pores $1$ and $2$, display significant
alterations in their areas.

Despite the variability observed among the pores under study, they exhibit
remarkably similar physical properties along their boundaries. Specifically,
all pores examined in this study demonstrate a consistent behaviour in terms
of the magnetic inclination angle evolution over the pore’s lifetime, with an
average value of approximately $30^{o}$. This result is in agreement with the
statistical analysis of pores by Keppens & Martinez Pillet (1996). However, we
observe significantly stronger total and vertical magnetic field strengths of
approximately $1560$ G and $1345$ G at the boundaries of the pores studied in
this work, in comparison to the mean values reported by Keppens & Martinez
Pillet (1996) of approximately 1400 G and 900 G, respectively. Nevertheless,
the derived values for both the total and vertical magnetic field strengths
are lower than those reported for stable sunspot and pore boundaries by
Schmassmann et al. (2018) and García-Rivas et al. (2021), respectively.

Figure 6: Scatter plot displaying the correlation between magnetic parameters
derived from the boundary of the studied pores. The red colour represents the
correlation between parameters during the time period before the maximum
vertical magnetic field is reached (see Fig. 5), while the blue colour
represents the correlation for the time period after that instant. The shaded
background contours depict the kernel density estimate plot of the correlation
distribution. The insets give the parameters of the linear fit and the Pearson
correlation coefficient separately for the two phases. Figure 7: Scatter plot
displaying the correlation between the areas calculated inside the boundary
contours (see Fig. 4) and each magnetic parameter derived at the boundary of
the studied pores. The colours represent the same time periods described in
Fig. 7. The shaded background contours depict the kernel density estimate plot
of the correlation distribution. The insets give the parameters of the linear
fit and the Pearson correlation coefficient separately for the two phases.

We note that the solar pores analysed in this work primarily consist of
isolated and stable cases with weaker and more dispersed magnetic fields
compared to those embedded in active regions. Active regions, which include
sunspots, exhibit concentrated magnetic fields that can produce intense flares
and coronal mass ejections, ultimately affecting the configuration of the
entire region. Isolated stable pores tend to persist for longer periods with
relatively stable magnetic field configurations and little to no accompanying
activity. In contrast, active region pores are typically associated with more
intense forms of solar activity and impulsive energy releases, such as flares
and coronal mass ejections, and their magnetic fields are typically more
dynamic and complex. Therefore, considering their differences in magnetic
field strength and behaviour, isolated stable pores and active region pores
can be distinguished as distinct features of the solar atmosphere.
Nonetheless, it is important to note that this study only considered rather
isolated stable pores, and the analysis of complex active regions exhibiting
pores would be an interesting study by itself, but it is beyond the scope of
this paper.

The main objective of this study was to increase the sample of the analysed
boundary conditions of the magnetic field parameters in solar pores, focusing
on the vertical magnetic field as the main feature. The analyses of sunspots
and pores (Schmassmann et al., 2018; Jurčák et al., 2018; García-Rivas et al.,
2021) demonstrated the vital role of the vertical magnetic field component on
the stability of these structures. The range of values of the maximum vertical
magnetic field obtained in this study is between $1400$ G and $1600$ G, with
standard deviations between $7.8\%$ and$14.8\%$. These values are lower than
those reported in previous studies, which may be due to the differences in the
data type used as well as the different intensity threshold used.

The most critical calculation when determining the boundary values is defining
the contours around the structures. The spatial resolution significantly
affects the results, and high spatial resolution improves the accuracy of the
values obtained. The results obtained in this study differ from those of
García-Rivas et al. (2021), where the vertical magnetic field values exceed
the values obtained in this study by more than $100$ G. This difference could
be attributed to the use of deconvolved data from SDO/HMI data (see Couvidat
et al., 2016), which improves the spatial resolution of the data, and in that
sense, the variation of the boundary changes considerably with respect of the
original SDO/HMI maps. On the other hand, the maximum values of Bver found in
the sample of pores studied in this work also differ. Three of the six
analysed pores have a maximum Bver of around 1550 G (pores 2, 4, and 5),
whereas the others reach lower maximum values ($\sim 1448$ G). From the
temporal evolution of the pore areas defined by the $55\%$ intensity
threshold, it seems that only pore 5 is stable, while all the other pores
undergo significant changes in their size with time. Pores 2, 4, and 5 also
show the least variations in the vertical component of the magnetic field on
their boundaries with time (See Fig. 5).

Figure 6 shows the correlation between the magnetic parameters during the two
different time ranges over the lifetimes of the pores. As expected from how it
is inferred, the vertical magnetic field is highly correlated with the
inclination angle as well as the total strength of the magnetic field, whereby
the correlation between the magnetic inclination angle and the total field
shows a larger scatter. Figure 7 shows the correlation between the magnetic
parameters and the areas obtained from the boundary contours. These plots
reveal a high correlation between the evolution of the areas of the solar
pores and the evolution of their magnetic inclination angles, as can be seen
in the first column of Fig. 7.

Defining boundary values and conditions for specific observations can enhance
the development of models and initial conditions for simulations that involve
the evolution of magnetic field flux tubes, such as magnetic bright points, as
shown in Magyar et al. (2021). This can provide critical inputs for
comprehending the smaller scales of solar dynamics. Therefore, exploring the
magnetic field properties at the boundary of solar pores, as demonstrated in
this comparative study, can contribute to a deeper understanding of their
dynamics.

###### Acknowledgements.

This research received support by the Austrian Science Fund (FWF) from the
project number I 3955-N27. J. I. Campos Rozo and D. Utz acknowledge also the
support from grant 21-16508J of the Grant Agency of the Czech Republic. All
the datasets used in this work are courtesy of NASA/SDO, and they were
obtained from the Joint Science Operation Center (JSOC).

## References

  * Borrero & Ichimoto (2011) Borrero, J. M. & Ichimoto, K. 2011, Living Reviews in Solar Physics, 8, 4
  * Borrero et al. (2011) Borrero, J. M., Tomczyk, S., Kubo, M., et al. 2011, Sol. Phys., 273, 267
  * Campos Rozo et al. (2019) Campos Rozo, J. I., Utz, D., Vargas Domínguez, S., Veronig, A., & Van Doorsselaere, T. 2019, A&A, 622, A168
  * Cheung & Isobe (2014) Cheung, M. C. M. & Isobe, H. 2014, Living Reviews in Solar Physics, 11, 3
  * Couvidat et al. (2016) Couvidat, S., Schou, J., Hoeksema, J. T., et al. 2016, Sol. Phys., 291, 1887
  * Dorotovič et al. (2002) Dorotovič, I., Sobotka, M., Brandt, P. N., & Simon, G. W. 2002, A&A, 387, 665
  * Ermolli et al. (2017) Ermolli, I., Cristaldi, A., Giorgi, F., et al. 2017, A&A, 600, A102
  * Fox et al. (2016) Fox, N. J., Velli, M. C., Bale, S. D., et al. 2016, Space Sci. Rev., 204, 7
  * García-Rivas et al. (2021) García-Rivas, M., Jurčák, J., & Bello González, N. 2021, A&A, 649, A129
  * Gilchrist-Millar et al. (2021) Gilchrist-Millar, C. A., Jess, D. B., Grant, S. D. T., et al. 2021, Philosophical Transactions of the Royal Society of London Series A, 379, 20200172
  * Green et al. (2018) Green, L. M., Török, T., Vršnak, B., Manchester, W., & Veronig, A. 2018, Space Sci. Rev., 214, 46
  * Hoeksema et al. (2014) Hoeksema, J. T., Liu, Y., Hayashi, K., et al. 2014, Sol. Phys., 289, 3483
  * Jurčák (2011) Jurčák, J. 2011, A&A, 531, A118
  * Jurčák et al. (2018) Jurčák, J., Rezaei, R., González, N. B., Schlichenmaier, R., & Vomlel, J. 2018, A&A, 611, L4
  * Keppens & Martinez Pillet (1996) Keppens, R. & Martinez Pillet, V. 1996, A&A, 316, 229
  * Kitiashvili (2013) Kitiashvili, I. N. 2013, in Solar and Astrophysical Dynamos and Magnetic Activity, ed. A. G. Kosovichev, E. de Gouveia Dal Pino, & Y. Yan, Vol. 294, 269–282
  * Leka & Skumanich (1998) Leka, K. D. & Skumanich, A. 1998, ApJ, 507, 454
  * Magyar et al. (2021) Magyar, N., Utz, D., Erdélyi, R., & Nakariakov, V. M. 2021, ApJ, 911, 75
  * Müller et al. (2020) Müller, D., St. Cyr, O. C., Zouganelis, I., et al. 2020, A&A, 642, A1
  * Pesnell et al. (2012) Pesnell, W. D., Thompson, B. J., & Chamberlin, P. C. 2012, Sol. Phys., 275, 3
  * Rempel et al. (2009) Rempel, M., Schüssler, M., Cameron, R. H., & Knölker, M. 2009, Science, 325, 171
  * Rezaei et al. (2012) Rezaei, R., Bello González, N., & Schlichenmaier, R. 2012, A&A, 537, A19
  * Rimmele et al. (2020) Rimmele, T. R., Warner, M., Keil, S. L., et al. 2020, Sol. Phys., 295, 172
  * Rucklidge et al. (1995) Rucklidge, A. M., Schmidt, H. U., & Weiss, N. O. 1995, MNRAS, 273, 491
  * Sainz Dalda et al. (2012) Sainz Dalda, A., Vargas Domínguez, S., & Tarbell, T. D. 2012, ApJ, 746, L13
  * Schmassmann et al. (2018) Schmassmann, M., Schlichenmaier, R., & Bello González, N. 2018, A&A, 620, A104
  * Sobotka (1999) Sobotka, M. 1999, in Astrophysics and Space Science Library, Vol. 239, Motions in the Solar Atmosphere, ed. A. Hanslmeier & M. Messerotti, 71–97
  * Sobotka (2002) Sobotka, M. 2002, in ESA Special Publication, Vol. 1, Solar Variability: From Core to Outer Frontiers, ed. A. Wilson, 381–388
  * Sobotka et al. (1999) Sobotka, M., Vázquez, M., Bonet, J. A., Hanslmeier, A., & Hirzberger, J. 1999, ApJ, 511, 436
  * Solanki (2003) Solanki, S. K. 2003, A&A Rev., 11, 153
  * SunPy Community et al. (2015) SunPy Community, Mumford, S. J., Christe, S., et al. 2015, Computational Science and Discovery, 8, 014009
  * Toriumi & Wang (2019) Toriumi, S. & Wang, H. 2019, Living Reviews in Solar Physics, 16, 3
  * Vargas Domínguez et al. (2010) Vargas Domínguez, S., de Vicente, A., Bonet, J. A., & Martínez Pillet, V. 2010, A&A, 516, A91
  * Vargas Domínguez & Utz (2022) Vargas Domínguez, S. & Utz, D. 2022, Reviews of Modern Plasma Physics, 6, 33
  * Zuccarello et al. (2009) Zuccarello, F., Romano, P., Guglielmino, S. L., et al. 2009, A&A, 500, L5