# Scalar field dynamics around black holes:
superradiant instabilities and binary evolution

Giuseppe Ficarra  Department of Physics, King’s College London, Strand,
London, WC2R 2LS, United Kingdom

###### Abstract

In classical general relativity astrophysical black holes can be affected by
the superradiant instability when gravity is minimally coupled to a light
bosonic field. The majority of phenomenological studies have focused on the
idealized case in which the black hole is initially surrounded by a single
mode superradiant seed. By studying the evolution of a scalar field with
multiple modes initial data in a quasiadiabatic approximation, we show that
the dynamics is more involved and depend on the initial seed energy and the
amplitude ratio between the modes. We also present preliminary results of the
dynamical evolution of a massive scalar field around a Newtonian and a fully
relativistic, black hole binary.

##  1 Introduction

The interaction between black holes and ultralight bosonic fields, that
represent popular dark matter candidates or axion-like particles, has opened
new avenues to probe for beyond-standard model physics. In particular, Kerr
black holes can undergo a superradiant instability[1] which actually turn
astrophysical black holes into particle detectors [2, 3, 4]. Moreover the
presence of a scalar field around merging black hole binaries could induce
modifications in the gravitational waveform and studies in this direction have
been recently started[5, 6, 7, 8, 9, 10]. We use geometric units $G=c=1$.

##  2 Superradiant instabilities and multiple modes

Figure 1: Left: Time evolution of the scalar field energy flux, rescaled by
the gravitational coupling $\alpha_{G}=M\mu_{S}$, for $M_{0}=10^{7}M\odot$,
$\chi_{0}=0.8$, $\alpha_{G0}=0.075$, $\lambda_{1,0}=1$ and several values of
$M_{S0}$ (see main text for details). Right: Black hole Regge plane obtained
from the evolution of a scalar field with mass $m_{S}=10^{-18}$ eV and
containing both superradiant and absorbing modes. We considered scalar cloud
initial mass of $M_{S0}=0.05M_{\odot}$ and different initial relative
amplitudes between the modes $\lambda_{1,0}$.

Superradiant amplification is present when a bosonic field scatters off a
rotating black holes with an oscillation frequency $\omega$ satisfies the
superradiance condition[1] $\omega<m\Omega_{\mathrm{H}}$, where $m$ is the
azimuthal index of the field and $\Omega_{H}$ is the black hole angular
velocity. In particular a superradiant instability can naturally develop for
massive fields and here we focus on a scalar field of mass
$m_{S}=\mu_{S}\hbar$ [12]. The phenomenology is ruled by the gravitational
coupling $\alpha_{G}=M\mu_{S}$ which represents the ratio between the black
hole characteristic size and the Compton wavelength of the field
$\lambda_{C}=1/\mu_{S}$. the main consequences of this mechanism are two:
scalar field develops a macroscopic cloud which will emit gravitational waves
at a later stage and the black hole rotational motion is slowed down. Both of
these features could be captured by electromagnetic and gravitational
observations [13]. The shortest timescale at which this process can occur[14]
$\tau_{\mathrm{S}}\sim 50\left(\frac{M}{M_{\odot}}\right)\mathrm{s}$ is
achieved for
$\alpha_{\mathrm{G}}=10^{10}\left(M/M_{\odot}\right)\left(\mu_{\mathrm{S}}/\mathrm{eV}\right)\sim\mathcal{O}(1/2)$.
This implies that for astrophysical black hole masses
$\sim\left(5,10^{10}\right)M_{\odot}$ one can probe the range
$m_{B}\in\left(10^{-20},10^{-10}\right)$ eV, a regime mostly not accessible by
current experiments [15].

Most studies in the field of superradiant instabilites focus on the case in
which the inital data for the scalar field is made of a single superradiant
mode. Our goal in this section [11] is to investigate if the more general case
of initial data described by a superposition of a superradiant mode and an
absorbing mode can affect the superradiant instability picture. Specifically
we added a new parameter to the overall phenomenology which is the initial
amplitude ratio $\lambda_{1,0}$ between the modes. Results are presented in
Fig.1 where in the left panel we show the time evolution of the scalar field
energy flux at the black hole horizon for an initial black hole mass
$M_{0}=10^{7}M_{\odot}$ and dimensionless initial spin $\chi_{0}=0.8$, initial
gravitational coupling $\alpha_{G0}=0.075$, $\lambda_{1,0}=1$ and several
values of the initial energy of the scalar seed $M_{S0}$. In the case of
negligible initial mass $M_{S0}=10^{-9}M_{0}$ the instability follows the
single mode phenomenology but for $M_{S0}\gtrsim 5\%M_{0}$ the superradiant
instability is completely quenched. In the right panel of Fig.1 we present the
Regge plane of the black hole (black hole spin vs mass), obtained considering
a scalar field with mass $m_{S}=10^{-18}$ eV and $M_{S0}=0.05M_{\odot}$ and
several values of $\lambda_{1,0}$. The grey shaded area is the Regge gap
predicted in the case of a single superradiant initial seed while the blue
dots represent multiple modes evolved black holes according to our model. Even
though we know superradiance is not present the single mode exclusion region
is still present due to absorption of non superradiant modes with opposite
angular momentum. Moreover additional forbidden regions appear depending on
the value of $\lambda_{1,0}$. This shows that the superradiance mechanism can
be turned off in a large region of the parameter space but surprisingly this
does not change the fact that the black hole eventually spins down and
consequently it does not vary the overall phenomenology. Details were
presented in [11].

##  3 Scalar field dynamics in binary black holes: preliminary results

In this section we focus on the interplay between scalar fields and binary
black holes, which has been recently investigated in the literature [5, 6, 7,
10]. In order to complement this studies here we perform numerical relativity
simulations of a scalar field during the late inspiral and merger of a black
hole binary coalescence using Canuda [16], a numerical relativity library
designed to study fundamental fields in the strong-field regime of gravity. In
particular we present preliminary results that will appear soon [17].

The first background spacetime we consider is an equal mass Newtonian binary.
The latter is constructed by superposing two Schwarzschild black holes at a
fixed orbital separation $d$ with an orbital frequency satisfying Kepler’s law
$\Omega_{\mathrm{orb}}=\sqrt{M/d^{3}}$. Results for the time evolution of a
scalar field with constant initial profile are depicted in Fig.2 where we
provide a snapshot of the scalar field at a late time in the evolution (left
panel). The scalar field in the area near the binary system is characterized
by a 10-fold amplification due to accretion of the outer portion of the field
on top of the binary. Moreover the rotational motion of the binary excites and
propagates away higher multipoles of the field as one can see from the
differential energy flux behaviour depicted in the right panel of Fig.2.

Figure 2: Left: scalar field colour plot in the Newtonian binary orbital plane
at $t=505M$. This is obtained by considering an orbital separation $d=5M$ for
which the obrital frequency is $M\Omega_{\mathrm{orb}}=5M$. The mass parameter
of the field is chosen to be $\mu_{S}=2\Omega_{\mathrm{orb}}$. The red circles
identify the Newtonian binary companions. Right: scalar field differential
energy flux $j_{R}$ as a function of time at two selected extraction radii
$r_{\mathrm{ex}}=50M$ and $r_{\mathrm{ex}}=200M$, for the same setup as the
left panel. Different profiles refer to different multipoles of the scalar
field.

We now move to the description of the time evolution of a scalar field around
a fully relativistic black hole binary in the decoupling limit. Results are
presented in Fig.3 where we consider a binary black hole of total mass $M$,
mass ratio $q=1/2$, initial separation $d=15M$ and spherically symmetric
scalar field initial data. In particular we focus on the late inspiral and
merger part of the signal. In the left panel we provide the gravitational
waveform $\Psi_{422}$, the scalar field monopole $\Phi_{00}$ and the total
scalar energy flux. Also in this case the scalar field is featured by
accretion towards the binary, as one can clearly see from the blue dotted
line, and this is in agreement with the simpler case of a Newtonian binary.
Furthermore the field is characterised by an involved multipolar structure
sourced by the rotational motion of the binary as depicted in the right panel
of Fig.3. Interestingly higher multipoles are excited and assume larger
amplitudes when the gravitational coupling $\alpha_{G}$ is of order unity.

Figure 3: Left: Diagnostic quantities as a function of (rescaled) time
$\hat{t}=\left(t-t_{\mathrm{merger}}-r_{\mathrm{ex}}\right)/M$ obtained for a
fully relativistic black hole binary with mass ratio $q=1/2$ and an initial
orbital separation $d=10M$. We show (from top to bottom) the gravitational
wave signal $\Psi_{4,22}$, the scalar field monopole $\Phi_{00}$ and total
energy flux $dE/dt$ at extraction radius $r_{\mathrm{ex}}=200M$. Different
profiles are obtained for several values of the ratio between the scalar field
mass parameter $\mu_{S}$ and the gravitational wave frequency a few cycles
before merger $\Omega_{\mathrm{GW}}\sim 0.21M^{-1}$. Right: scalar field
amplitude peak at late times (but before merger) for the same setup as left
panel. Three values of $\mu_{S}/\Omega_{\mathrm{GW}}$ are considered.
Different colors show multipolar components of the scalar field.

##  4 Conclusion

We have investigated the evolution of the black hole superradiant instability
against ultralight scalar fields with an initial configuration described by a
superposition of modes. In this scenario the instability evolution heavily
depends on the energy of the scalar initial seed and on the gravitational
coupling. If the seed energy is a few percent of the black hole mass, a black
hole surrounded by an overlap of superradiant and absorbing modes with
comparable amplitudes might not even experience a superradiant unstable phase.
On the other hand, if the seed energy is much smaller than a few percent of
the black hole mass the effect of nonsuperradiant modes is negligible. The
black hole Regge plane is also affected by the presence of nonsuperradiant
modes and its pattern is more involved and additional forbidden regions can
appear, depending on the parameters. Moreover we have studied the dynamics of
a massive scalar field in a Newtonian binary which magnifies the field due to
accretion and propagates higher multipoles through its rotational motion.
Finally we analysed a fully relativistic binary black hole which induces a
complex multipolar structure in the scalar field.

## Acknowledgments

G.F. wishes to thank King’s College London for financial support and the Royal
Society for the PhD studentship provided under Research Grant RGF\R1\180073.

## References

## References

  * [1] Richard Brito, Vitor Cardoso, and Paolo Pani.  Superradiance: New Frontiers in Black Hole Physics.  Lect. Notes Phys., 906:pp.1–237, 2015.
  * [2] Asimina Arvanitaki, Savas Dimopoulos, Sergei Dubovsky, Nemanja Kaloper, and John March-Russell.  String Axiverse.  Phys. Rev. D, 81:123530, 2010.
  * [3] Asimina Arvanitaki and Sergei Dubovsky.  Exploring the String Axiverse with Precision Black Hole Physics.  Phys. Rev. D, 83:044026, 2011.
  * [4] Richard Brito, Vitor Cardoso, and Paolo Pani.  Black holes as particle detectors: evolution of superradiant instabilities.  Class. Quant. Grav., 32(13):134001, 2015.
  * [5] Daniel Baumann, Horng Sheng Chia, and Rafael A. Porto.  Probing Ultralight Bosons with Binary Black Holes.  Phys. Rev. D, 99(4):044001, 2019.
  * [6] Daniel Baumann, Horng Sheng Chia, Rafael A. Porto, and John Stout.  Gravitational Collider Physics.  Phys. Rev. D, 101(8):083019, 2020.
  * [7] Emanuele Berti, Richard Brito, Caio F. B. Macedo, Guilherme Raposo, and Joao Luis Rosa.  Ultralight boson cloud depletion in binary systems.  Phys. Rev. D, 99(10):104039, 2019.
  * [8] Leong Khim Wong.  Superradiant scattering by a black hole binary.  Phys. Rev. D, 100(4):044051, 2019.
  * [9] Leong Khim Wong.  Evolution of diffuse scalar clouds around binary black holes.  Phys. Rev. D, 101(12):124049, 2020.
  * [10] Taishi Ikeda, Laura Bernard, Vitor Cardoso, and Miguel Zilhão.  Black hole binaries and light fields: Gravitational molecules.  Phys. Rev. D, 103(2):024020, 2021.
  * [11] Giuseppe Ficarra, Paolo Pani, and Helvi Witek.  Impact of multiple modes on the black-hole superradiant instability.  Phys. Rev. D, 99(10):104019, 2019.
  * [12] Steven L. Detweiler.  KLEIN-GORDON EQUATION AND ROTATING BLACK HOLES.  Phys. Rev. D, 22:2323–2326, 1980.
  * [13] Enrico Barausse et al.  Prospects for Fundamental Physics with LISA.  Gen. Rel. Grav., 52(8):81, 2020.
  * [14] Sam R. Dolan.  Instability of the massive Klein-Gordon field on the Kerr spacetime.  Phys. Rev. D, 76:084001, 2007.
  * [15] Vitor Cardoso, Óscar J. C. Dias, Gavin S. Hartnett, Matthew Middleton, Paolo Pani, and Jorge E. Santos.  Constraining the mass of dark photons and axion-like particles through black-hole superradiance.  JCAP, 03:043, 2018.
  * [16] Helvi Witek, Miguel Zilhao, Giuseppe Ficarra, and Matthew Elley.  Canuda: a public numerical relativity library to probe fundamental physics, May 2020.
  * [17] Giuseppe Ficarra, Raimon Luna, Roberto Emparan, and Helvi Witek.  in prep.  XXX, 2021.


# Perceptual MAE for Image Manipulation Localization: A High-level Vision
Learner Focusing on Low-level Features

Xiaochen Ma1, Jizhe Zhou1, Xiong Xu2 , Zhuohang Jiang1, Chi-Man Pun3, Senior
Member, IEEE
1 College of Computer Science, Sichuan University
2 Second Laboratory, The 10th Institute of China Electronics Technology Group
3 Computer and Information Science, Faculty of Science and Technology,
University of Macau

###### Abstract

Nowadays, multimedia forensics faces unprecedented challenges due to the rapid
advancement of multimedia generation technology thereby making Image
Manipulation Localization (IML) crucial in the pursuit of truth. The key to
IML lies in revealing the artifacts or inconsistencies between the tampered
and authentic areas, which are evident under pixel-level features.
Consequently, existing studies treat IML as a low-level vision task, focusing
on allocating tampered masks by crafting pixel-level features such as image
RGB noises, edge signals, or high-frequency features. However, in practice,
tampering commonly occurs at the object level, and different classes of
objects have varying likelihoods of becoming targets of tampering. Therefore,
object semantics are also vital in identifying the tampered areas in addition
to pixel-level features. This necessitates IML models to carry out a semantic
understanding of the entire image. In this paper, we reformulate the IML task
as a high-level vision task that greatly benefits from low-level features.
Based on such an interpretation, we propose a method to enhance the Masked
Autoencoder (MAE) by incorporating high-resolution inputs and a perceptual
loss supervision module, which is termed Perceptual MAE (PMAE). While MAE has
demonstrated an impressive understanding of object semantics, PMAE can also
compensate for low-level semantics with our proposed enhancements. Evidenced
by extensive experiments, this paradigm effectively unites the low-level and
high-level features of the IML task and outperforms state-of-the-art tampering
localization methods on all five publicly available datasets.

###### Index Terms:

image forensics, image manipulation localization, multimedia security, Vision
Transformer, and self-supervised learning.

##  I Introduction

Recent advancements in large generative models, such as Stable Diffusion[1],
have significantly improved the quality and diversity of multimedia content
enhancement results. However, this technique is not without its potential
drawbacks. In contrast to previous methods that require professional knowledge
for operation and merely yield plausible outputs, large generative models
endow ordinary people with easy access to generate imperceptive image
manipulation results. This ability to create high-quality manipulated images
on a massive scale has unleashed the problem of image tampering. Therefore, it
creates unprecedented challenges for multimedia forensics, particularly in
Image Manipulation Localization (IML). Effective IML methods are urgently
needed to mitigate the negative impact of tampered images, such as fake news,
rumors, and misleading information. In short, IML methods have become
essential today to safeguard against the jeopardizes caused by image tampering
and ensure that multimedia content remains trustworthy and reliable. In the
context of the growing interest in image manipulation detection/localization,
recent submissions [2, 3, 4, 5] reflect the active pursuit of advancements in
this area.

From the perspective of image tampering, Figure 1 (a) illustrates that
existing techniques can be broadly classified into three categories[6, 7]:
Splicing(combining parts of different images to create a new one), Copy-
move(copying and pasting a region within the same image), and
inpainting(Remove and filling in an area with plausible content). Despite that
large generative models can yield tampered results imperceptible to human
eyes, each type of manipulation still leaves detectable traces at the pixel
level. These traces manifest as inconsistencies between the tampered and
authentic regions and are commonly referred to as artifacts. Therefore, most
existing image manipulation localization techniques treat IML as a low-level
vision task that aims to capture the artifacts by extracting pixel-level
features, such as the image RGB noises[8, 9], edge signals[10, 11], or high-
frequency features[12, 13]. These low-level vision features are generally
effective in revealing the artifacts and localizing the tampered regions.
However, fully relying on low-level features leads existing IML models to
suffer from low generalization ability and robustness. Therefore, constructing
an approach that incorporates other manipulation traces is the key to
improving localization accuracy and addressing the generalization and
robustness limitations.

To achieve this, it is crucial to understand the characteristics and patterns
of tampering. Typically, as shown in Figure 1 (b), most tampering aims to
deceive the audience by altering or confusing the semantics in images. As a
result, tampering commonly occurs on objects rather than backgrounds within an
image. Moreover, the likelihood of an object being targeted for tampering
varies depending on its class and its contribution to the overall semantics of
the image. For instance, humans and animals in the foreground are more likely
targets for tampering than trees and mountains in the background. Therefore,
we argue that understanding high-level visual information, like object-level
semantics, could be useful in identifying manipulated regions and outlining
suspicious areas completely. A recent study, ObjectFomer [12] using enhanced
object proposals has experimentally supported this argument. However, high-
level semantics alone are insufficient for generating tampering masks, as they
lack comprehension of the detailed artifacts. Thus, a multi-level method
fusing both low-level artifacts and high-level semantic features is the
optimal solution for the IML task.

Figure 1:  (a) Example of artifacts in three types of tampering. The red
dashed box in the third column represents the range of the zoomed-in area in
the fourth column. Red arrows in the fourth column point to artifacts that are
considered tampering traces. (b) Random samples from CASIAv2[14] dataset, 80%
are object-related manipulation. The red line marks the boundary of the
tampered area. The first column shows tampering that is unrelated to objects,
while the other four columns show object-related tampering. For IML datasets,
manipulation on objects is a common case, as it can more effectively confuse
the semantics of the entire image.

Hence, in this paper, we are the first ever to reformulate the IML task as a
high-level vision task significantly benefits from low-level features. Such a
character makes IML unique from any other tasks. To support the proposed
argument, we searched among various self-supervised Vision Transformers [15,
16, 17, 18], which are all highly proficient in learning high-level semantics,
to identify potential candidates as our backbone. MAE [18] stands out as the
first method focused on pixel-level reconstruction, which could easily
modified to fit in the low-level features. In contrast, others focus on
reconstructing tokenized feature maps or complicated paradigms that are
impossible to enhance with low-level information. As evidenced by experiments,
MAE indeed outperforms other methods on IML tasks. Besides, IML often faces
the dataset insufficiency problem. Common public IML datasets usually only
have thousands and hundreds of images, which can not satisfy the appetite of a
vanilla ViT. MAE pre-training is also powerful enough to help us overcome
these issues.

To help the model focus on low-level information, We propose the Perceptual
Masked Autoencoder (PMAE), a self-supervised module that enhances the model to
cope with low-level artifacts in IML. Based on MAE, PMAE inherits its
remarkable semantic comprehension and further enriches its learning ability of
low-level visual features through a high-resolution encoder supervised by
hierarchical perceptual loss. In the whole paradigm, we pre-trained a ViT
encoder with MAE on large real-world datasets like ImageNet to learn object
semantics. Then, during fine-tuning, we slightly modified the encoder with
high-resolution patch embedding for tracing detailed features and tuned it on
limited IML datasets with an IML segmentation branch and a PMAE reconstruction
branch. Since these two branches share the same high-resolution encoder and
optimize together, if the PMAE could reconstruct the low-level visual features
well, then the latent representation learned in this process can also be
effective for the segmentation branch. This paradigm allows the model to learn
the high-level object semantics from a larger, more real-world sampled dataset
and fully mine the tamper-related low-level visual features from the expensive
and limited IML dataset.

We follow a widely used evaluation protocol [19, 11, 10, 20] for IML to
measure the performance and generalizability of our model. In detail, the
model is trained on CASIAv2 [14] datasets, then evaluates the metrics on
smaller public datasets, including CASIAv1[14], Columbia [21], COVERAGE [22]
and NIST16 [23]. The experimental results verify that PMAE has the ability to
guide the model to outperform state-of-the-art ones on F1 score, AUC, and
robustness. This plug-and-play module also provides the possibility of further
exploration in conducting additional IML tasks with ViT.

In summary, our contributions are as follows:

  * •

We revisit the essence of IML and reformulate the IML task as a high-level
vision task that greatly benefits from low-level features.

  * •

According to our interpretation of the IML task, we establish the PMAE, a
model with multi-level visual capturing ability that can effectively support
image manipulation localization during fine-tuning.

  * •

Extensive experiments show that PMAE outperforms state-of-the-art models on
five public benchmark datasets, evaluated using $F_{1}$ scores and robustness
metrics. This provides strong evidence to verify our interpretation of the IML
task.

##  II Related works

Mask Image Modeling Taking inspiration from the success of masked language
modeling in language tasks [24], masked image modeling (MIM) in the visual
domain learns representations from images that are disrupted by masking.
Several methods have achieved State-of-the-art results on downstream tasks.
BEiT[15] proposes to recover discrete visual labels, while SimMIM[25]
addresses the MIM task as a pixel-level reconstruction. In this work, we focus
on MAE [18], which proposes to use a high masking rate and non-arbitrary ViT
decoder. A higher masking rate can increase the difficulty of reconstruction
and force the model to focus on macro-level semantics. No structured
modifications to the ViT encoder also facilitate our plug-and-play use of the
recent new ViT algorithm. We will further discuss why we don’t select other
self-supervised ViT as the backbone in Section III-B.

Image Manipulation Detection/Localization In the early years, image
manipulation detection usually focuses on single-type tampering, especially
copy-move detectors like Dense-InceptionNet [26] and STRDNet [5] that identify
potential copy-move forgery instances. Low-level visual features, such as
noise, Sobel (edge detection), and high-pass filters, have shown excellent
performance for specific types of tampering and become prevalent. After that,
generic tampering detection by end-to-end deep learning methods became
dominant, which is manipulation type-independent. Most of them combine the RGB
view with other low-level vision views and become successful. RGB-N[19]
proposed the SRM filter to extract noise features and support the detection by
Faster R-CNN-based network. The bayarConv filter proposed in Constrained
CNN[9] can also extract noise information for supporting classification. J.
Bappy et al. [27] employ a hybrid CNN-LSTM model that effectively classifies
manipulated and non-manipulated regions. Wu Yue et al. firstly concatenates
the feature maps from the SRM filter and the BayarConv together and uses VGG
as the backbone to complete the segmentation of the manipulated area. Recent
work of MVSS-Net[11] and MVSS-Net++ [28] also utilizes BayarConv and combines
it with sobel filters with edge detection ability by dual-attention.
Objectformer[12] uses Discrete Cosine Transform to acquire high-frequency
features to obtain information that is difficult to get through RGB channels.
However, these features were initially designed for specific tampering methods
and not generalized. We suggest that utilizing self-supervised methods to
discover traces autonomously could be a better choice than these handcrafted
feature extractors.

##  III Proposed Methods

Our goal is to enhance MAE’s understanding of low-level visual features,
especially subtle traces related to tampering. Building upon MAE’s inherent
object-level reconstruction capability, we have devised during the finetuning
stage an enhanced self-supervised method, Perceptual Masked Autoencoder(PMAE),
to bolster the model’s sensitivity to low-level artifacts. PMAE holds a
significant advantage over the earlier hand-crafted filters, as it can learn
the most significant features from the dataset by itself, rather than relying
solely on narrow prior knowledge.

In this section, we introduce our whole training paradigm and the
implementation detail of PMAE, which enhances MAE with high-resolution input
and supervises it with a hierarchical masked perceptual loss.

###  III-A Overview of Training Paradigm

The widely adopted paradigm of pre-training, followed by fine-tuning, is
utilized in our work. We commence with an MAE pre-training on the low-
resolution ImageNet-1k dataset, imbuing the model with the semantics of common
objects. Subsequently, as illustrated in Figure 2, we directly transfer the
parameters of the pre-trained ViT encoder for our fine-tuning process on
padded high-resolution IML datasets that keep the freshest artifacts to learn.
Our fine-tuning involves two distinct tasks: the IML segmentation and PMAE
reconstruct branch. The segmentation branch employs a deliberately
uncomplicated structure to segment suspicious regions like IML-VIT [20]. In
contrast, the PMAE branch informs the model with detailed distributions
learned from IML datasets. These two branch shares the same ViT encoder and
utilize their own decoder. We jointly optimize both branches by computing the
gradient together. However, since the MAE masking strategy and segmentation
forwarding provide different inputs for the ViT encoder, each image will pass
through the ViT encoder twice.

Figure 2: Diagram of the pre-train and fine-tune process of proposed paradigm.

In general, the optimization target of the IML segmentation branch can be
formulated as:

$\underset{\theta_{E},\theta_{S}}{\arg\,\min}\,\mathcal{L}_{seg}\\{\mathcal{D}_{MLP}[\mathcal{E}(X_{p};\theta_{E});\theta_{S}],M_{p}\\}$
(1)

while the PMAE reconstruct branch can be formulated as:

$\underset{\theta_{E},\theta_{R}}{\arg\,\min}\,\mathcal{L}_{rec}\\{\mathcal{D}_{ViT}[\mathcal{E}(X_{p}^{\prime};\theta_{E});\theta_{R}],X_{p}\\}$
(2)

Here, $\mathcal{L}$ refers to the loss functions for the respective branches,
$\mathcal{D}$ represents the decoder, and $\mathcal{E}$ represents the
encoder. All the $\theta$ refer to the corresponding model parameters. In
these formulas, $X$ represents the distribution of the input images,
$X^{\prime}$ represents the distribution of the data after MAE random masking
and $M$ represents the ground truth mask. The subscript $p$ denotes the zero-
padding operation, which will be further explained in Section III-C.

We do not perform independent pre-training for the PMAE branch because the
high-quality public IML datasets are relatively small. Even a larger dataset
in this field, CASIAv2, contains only 5063 tampered images and 7491 authentic
images, which is significantly smaller than the 1.2 million images in
ImageNet-1k. This dataset size discrepancy makes it challenging to determine
whether the model has fully converged or overfitted. Because we lack
appropriate indicators to monitor the reconstruction process. Therefore, we
accompany the segmentation branch during training to evaluate the optimization
process promptly by monitoring its performance on the test dataset.

Figure 3: Overview of the Perceptual Masked Autoencoder (PMAE) reconstruction
branch. Since our perceptual loss only computes with loss from shallow
convolution layers and only focuses on the edge-related patches,
reconstruction images often look like a noisy map. However, this indicates our
method truly captures the low-level vision features from datasets

###  III-B MAE Pre-training on ViT Encoder

Recently, self-supervised Vision Transformers like MAE [18], Beit [15], iBOT
[16], and DINO [17] have been shown to have impressive performance in various
downstream classification tasks, which also signifies their strong
understanding of object-level semantics. However, we ultimately select MAE as
our object semantic learner based on the following reasons: (1) MAE stands out
as the first method focused on pixel-level reconstruction, while others focus
on reconstructing the tokenized (by methods like VAE [29]) feature maps,
thereby MAE is more competitive in tracing low-level information. (2) In
models like DINO, it is hard to design a structure that could effectively
focus on low-level features and maintain its original transfer learning
paradigm at the same time. (3) MAE is very plain, with almost only one naive
ViT, bringing two benefits: First, there is almost no need to introduce any
additional modules. Second, the PMAE optimized for low-level traces can easily
maintain almost the same pattern as MAE, making the model converge quickly.
Furthermore, in Section IV-C, we will demonstrate through experiments that MAE
can indeed outperform other self-supervised methods.

###  III-C High-resolution ViT Encoder for Fine-tuning

During fine-tuning, both the segmentation branch and the PMAE reconstruction
branch require the ViT encoder to extract intricate details and artifacts from
images as much as possible. To achieve this, it is essential to preserve the
original resolution of each image to avoid downsampling that could potentially
distort the artifacts. However, when computing images in parallel, all images
within a batch must have the same resolution. To reconcile these competing
demands, we adopt a novel approach. Rather than simply rescaling images to the
same size, we pad the images and ground truth masks with zeros instead of
resizing, then place the image on the top-left side to match a larger constant
resolution. This strategy maintains crucial low-level visual information of
each image, allowing the model to explore better features instead of depending
on handcrafted prior knowledge. To implement this approach, we first adjust
the patch embedding dimensions of the ViT encoder to a larger scale. However,
this modification significantly increases the computing complexity. To balance
it, we adopt a technique inspired by previous works[30, 31], which
periodically replaces part of the global attention blocks in ViT with windowed
attention blocks. This method ensures global information propagation while
reducing the computational cost.

More specifically, we represent input images as $X\in\mathbb{R}^{3\times
h\times w}$, and ground truth masks as $M\in\mathbb{R}^{1\times h\times w}$,
where $h$ and $w$ correspond to the height and width of the image,
respectively. We then pad them to $X_{p}\in\mathbb{R}^{3\times H\times W}$ and
$M_{p}\in\mathbb{R}^{1\times H\times W}$. Balance with computational cost and
the resolution of common IML datasets(see in Table I), we take $H=W=1024$ as
consts in our implementation. Then $X_{p}$ is passed into the windowed ViT-B
encoder with 12 layers, with a complete global attention block retained every
3 layers.

###  III-D MLP Segmentation Branch

Figure 4: Diagrams of the MLP decoder. White rectangles on the left represent
the output of the Simple Feature Pyramid. Fractions denote for the resolution
compared to padded images.

Simple Feature Pyramid To incorporate multi-scale supervision that is
efficient for segmentation, we employ a feature pyramid network after the ViT
encoder, following the approach proposed in ViTDet [32]. This method uses the
single output feature map $G_{e}=\mathcal{E}(X_{p},;\theta_{E})$ from ViT and
applies a series of convolutional and deconvolutional layers to upsample and
downsample the feature map to obtain multi-scale feature maps $F_{i}$, where
$i\in{1,2,3,4}$:

$F_{i}=\mathcal{C}_{i}(G_{e})\in\mathbb{R}^{\frac{H}{2^{i+2}}\times\frac{W}{2^{i+2}}\times
C_{S}}$ (3)

Here, $\mathcal{C}_{i}$ denotes the convolution series, and $C_{S}$ represents
the output channel dimension for each layer in the feature pyramid. Notably,
this multi-scale method does not alter the base structure of ViT, allowing for
the easy introduction of recent advanced algorithms like MAE to the backbone.

Lightweight Prediction Head To reduce memory consumption while demonstrating
the improvements from the advanced design of the ViT encoder and PMAE
supervise, we aim to apply a lightweight network for the final prediction. To
this end, as shown in Figure 4, we adopt the decoder design from SegFormer
[33], which outputs a smaller predicted mask $\hat{M}$ with a resolution of
$1\times\frac{H}{4}\times\frac{W}{4}$, which can effectively reduce
computational complexity. This lightweight All-MLP decoder first applies a
linear layer to unify the channel dimension and then upsamples all features to
the same resolution of $C_{D}\times\frac{H}{4}\times\frac{W}{4}$ using
bilinear interpolation. Subsequently, we concatenate all the features and
apply a series of linear layers to fuse them and make the final prediction.
The prediction head can be expressed as follows:

$\displaystyle P=$
$\displaystyle\,\mathcal{D}_{MLP}[\mathcal{E}(X_{p};\theta_{E});\theta_{S}]$
(4) $\displaystyle=$
$\displaystyle\,MLP\\{\odot_{i}(W_{i}F_{i}+b_{i})\\}\in\mathbb{R}^{\frac{H}{4}\times\frac{W}{4}\times
1}$ (5)

Here, $P$ represents the predicted probability map for the manipulated area;
$\odot$ denotes for concatenation operation, and $MLP$ refers to an MLP
module. The loss for the segmentation branch is computed with binary cross
entropy loss function: $\mathcal{L}_{seg}=\mathcal{L}_{BCE}(P,M_{p})$.

###  III-E PMAE Reconstruction Branch

The Perceptual Masked Autoencoder(PMAE) first obtains tokens encoded by a ViT
backbone from high-resolution padded images that have undergone random
masking. Most of the settings in this process are the same as the original
MAE, while the only exception is the number of patches significantly increased
due to the high resolution of the input images. Then, we apply a series of
Vision Transformer layers with full global attention as the decoder. Finally,
a full-connected layer decodes the tokens back to an RGB image as the final
reconstruction
$R=\mathcal{D}_{ViT}[\mathcal{E}(X_{p}^{\prime};\theta_{E});\theta_{R}]\in\mathbb{R}^{3\times
H\times W}$. Since we want to guide the model to learn low-level visual
features related to tampering, and an important prior knowledge in IML is that
tampering traces are largely distributed around the tampered area, we do not
simply calculate perceptual loss[34] between the reconstructed image and the
input image. Instead, we use a hierarchical masked perceptual loss to
supervise the reconstructed image.

To implement this approach, we start by generating an edge mask based on the
ground truth using morphology operations. Next, we utilize this mask to
further generate a patch edge mask. As depicted in Figure 3, we first divide
the edge mask into patches, and if any pixel in a patch is equal to 1, we
consider the entire region of the patch edge mask that corresponds to that
patch to be 1. Finally, when calculating the perceptual loss, we apply a
point-wise product between the patch edge mask and the input feature map, as
well as the reconstructed feature map after each convolution layer, before
computing the MSE loss. This ensures that the model focuses only on the areas
related to the tampered artifact.

The model we use to generate perceptual feature maps is a VGG[35] network on
ImageNet-21k. Since our masking perceptual strategy has damaged the object-
level semantics, we only adopt the masked perceptual features from coming
layers: conv1_2, conv2_2, conv3_2, which are all shallow layers of VGG that
only capture the low-level features, this selection following the original
paper of perceptual loss[34] for low-level vision task. The final hierarchical
masking perceptual loss in our implementation can be formulated as:

$\displaystyle\mathcal{L}_{rec}$ $\displaystyle=\mathcal{L}_{rec}(R,X_{p})$
(6)
$\displaystyle=\mathcal{L}_{rec}\\{\mathcal{D}_{ViT}[\mathcal{E}(X_{p}^{\prime};\theta_{E});\theta_{R}],X_{p}\\}$
(7) $\displaystyle=Loss_{1\\_2}+Loss_{2\\_2}+Loss_{3\\_2}$ (8)

where $Loss_{m\\_n}$ denotes the single layer hierarchical masking perceptual
loss in Figure 3.

In summary, all the modifications on PAME compared to MAE aim to mine more
low-level visual features during fine-tuning.

###  III-F Combined Loss

Even though the segmentation and the reconstruction branch have similar
optimization goals, it can seriously affect the model’s convergence
performance if they are not balanced well. We formulate the final loss
$\lambda$ and seek optimal $\lambda$:

$\mathcal{L}=\mathcal{L}_{seg}+\lambda\cdot\mathcal{L}_{rec}$ (9)

One significant factor is that the predicted values and ground truth values in
the segmentation branch are always within the range of $[0,1]$, and the
distance between them is not far. In contrast, the perceptual loss in the
reconstruction branch is calculated based on the feature map of the middle
layer of the VGG network, without activation functions like softmax that can
normalize the values, which leads to a long distance between the reconstructed
feature maps and the ground truths. On average, the value of segmentation loss
falls around 1e-2 to 1e-3, while the reconstruction loss falls around 10 to
100. We tested for $\lambda\in\\{1,0.1,0.01,0.001\\}$ and finally selected the
optimal value as 0.01.

##  IV Experiments

###  IV-A Evaluation Barrier

Despite the proliferation of the SoTA models in recent research, achieving
equitable comparisons remains intricate. This difficulty stems partly from the
absence of publicly accessible code and training methodologies for these
models [36, 12]. Furthermore, many studies rely on extensive synthesized
datasets that are not available to the research community [7, 37]. Hence, we
advocate for the community’s embrace of open-source practices and emphasize
the necessity to evaluate dataset generation strategies independently from
model performance. These measures are pivotal in ensuring equity and fostering
continued progress in this domain.

###  IV-B Experimental Setup

Datasets To ensure a fair comparison with state-of-the-art methods in image
tampering localization, we adopt a commonly used protocol [11, 10, 36] for our
evaluation. We first train our model on the CASIAv2 [14] dataset and then
evaluate its performance on smaller public datasets including CASIAv1 [14],
NIST16 [23], COVERAGE [22], Columbia [21], and Defacto [38], details can found
in Table I. However, we note that the Defacto dataset does not contain
authentic images as negative examples. To overcome this limitation, we follow
the approach of MVSS-Net[11] and randomly select 6000 untouched images from
MS-COCO [39]. These images are combined with 6000 images from the Defacto
dataset to create a validation set, called Defacto-12k.

TABLE I: Details of six datasets in our experiments.

Usage | Dataset | Type | Manipulation Type | Resolution
---|---|---|---|---
Auth | Mani | copymv | spli | inpa | min | max
Train | CASIAv2 [14] | 7491 | 5063 | 3235 | 1828 | 0 | 240 | 800
Test | CASIAv1 [14] | 800 | 920 | 459 | 461 | 0 | 256 | 384
NIST16 [23] | 0 | 564 | 68 | 288 | 208 | 480 | 5616
COVERAGE [22] | 100 | 100 | 100 | 0 | 0 | 158 | 572
Defacto-12k [38] | 6000 | 6000 | 2000 | 2000 | 2000 | 120 | 640
Columbia [21] | 183 | 180 | 0 | 180 | 0 | 568 | 1152

Evaluation Criteria We assessed the effectiveness of our model in localizing
image manipulations using two widely adopted metrics: the pixel-level $F_{1}$
measure and the pixel-level AUC measure. However, the AUC measure can be
affected by imbalanced data, which is typically the case in IML datasets that
contain more negative pixels. This can result in an overestimation of the
model’s performance. Therefore, to provide a more meaningful and practical
evaluation of our model’s performance, we focus on reporting the pixel-level
$F_{1}$ score using a uniform threshold of $0.5$. This scoring approach is
less susceptible to the influence of imbalanced data and is widely used as a
robust metric for evaluating the effectiveness of image manipulation
localization models.

Implementation Details Our model is implemented with PyTorch and trained on
NVIDIA RTX 3090 GPUs for 200 epochs with a batch size of 1. We initialized the
ViT-B backbone with MAE pre-trained weights on ImageNet-1k and used the AdamW
optimizer [40] with a base learning rate of 1e-4. We employed a cosine decay
strategy [41] to schedule the learning rate. We applied the early stop
technique during training to prevent overfitting. The PMAE branch and predict
branch are sequentially processed within a batch. Each branch performs an
independent back propagation step, thereby not concurrently occupying GPU
memory. To prepare the images for training, we added top-left zero-padding to
all images (except those that exceeded the limit) to achieve a resolution of
1024x1024. Images with longer edges that exceeded the size limit were resized
its longer side to 1024 while maintaining their aspect ratio. We applied
standard data augmentation techniques such as rescaling, flipping, blurring,
rotation, and basic manipulations (e.g., randomly copying, moving, or
inpainting rectangular areas within a single image) during training.

In terms of inference, the computational cost is significantly reduced as only
the segmentation branch is utilized. A batch size of 4 is employed, resulting
in an approximate GPU memory consumption of 11GB, which can meet the
requirements of most graphics cards. The average inference time for a single
image on a 3090 GPU is approximately 0.6 seconds.

###  IV-C Ablation Study

TABLE II: Ablation study of PAME on 4 public datasets, evaluated with pixel-
level F1 score and pixel-level AUC. Best number per column is shown in bold.
Models are all trained on CASIAv2 datasets.

Goal | Init method | Components | CASIAv1 | Coverage | Columbia | NIST16 | MEAN
---|---|---|---|---|---|---|---
H-Reso | S-FPN | PMAE | F1 | AUC | F1 | AUC | F1 | AUC | F1 | AUC | F1 | AUC
w/o MAE | Xavier | + | + | - | 0.1035 | - | 0.0439 | - | 0.0744 | - | 0.0632 | - | 0.0713 | -
ViT-B ImNet-21k | + | + | - | 0.5114 | - | 0.1854 | - | 0.2287 | - | 0.1811 | - | 0.2767 | -
w/o H-Reso | MAE ImNet-1k | - | + | - | 0.5061 | 0.8166 | 0.2324 | 0.8250 | 0.5409 | 0.8420 | 0.2987 | 0.8212 | 0.3945 | 0.8262
w/o S-FPN | MAE ImNet-1k | + | - | - | 0.5996 | 0.8627 | 0.4457 | 0.8352 | 0.6125 | 0.8350 | 0.1841 | 0.6767 | 0.4605 | 0.8024
w/o PMAE | MAE ImNet-1k | + | + | - | 0.5886 | 0.8668 | 0.3277 | 0.8131 | 0.7445 | 0.9076 | 0.2993 | 0.7706 | 0.4900 | 0.8395
Full Setup | MAE ImNet-1k | + | + | + | 0.6267 | 0.9366 | 0.3583 | 0.9285 | 0.7574 | 0.9298 | 0.3137 | 0.8313 | 0.5140 | 0.9065

TABLE III: PAME compared with the state-of-the-art. The best scores are shown
in bold.

Method | Pixel-level $F1$ score
---|---
CASIAv1[14] | Columbia[21] | NIST16[23] | Coverage[22] | Defacto-12k[38] | MEAN
HP-FCN, ICCV19[42] | 0.154 | 0.067 | 0.121 | 0.003 | 0.055 | 0.080
ManTra-Net, CVPR19[7] | 0.155 | 0.364 | 0 | 0.286 | 0.155 | 0.192
CR-CNN, ICME20[43] | 0.405 | 0.436 | 0.238 | 0.291 | 0.132 | 0.300
GSR-Net, AAAI20[10] | 0.387 | 0.613 | 0.283 | 0.285 | 0.051 | 0.324
MVSS-Net, ICCV21[11] | 0.452 | 0.638 | 0.292 | 0.453 | 0.137 | 0.394
MVSS-Net(re-trained) | 0.435 | 0.441 | 0.203 | 0.329 | 0.105 | 0.303
MVSS-Net++, TPAMI22[28] | 0.513 | 0.660 | 0.304 | 0.482 | 0.095 | 0.411
PMAE (ours) | 0.688 | 0.860 | 0.311 | 0.473 | 0.177 | 0.502

In this section, we perform ablation experiments to systematically analyze the
contribution of each component in our proposed PMAE paradigm to the overall
performance. Specifically, we investigate the impact of removing the following
components: (1) MAE: initialize the ViT with Xavier init[44], ImageNet-21k
classification, and other self-supervised strategies; (2) High-resolution:
reduce the input resolution of fine-tuning by resizing all the images and
masks to 512×512; (3) Simple Feature Pyramid: replace this multi-scale
structure with a series of plain convolution layers; (4) PMAE branch: remove
PMAE branch and related loss functions.

The quantitative results on 4 widely used public IML datasets are presented in
Table II. We report the pixel-level F1 score and pixel-level AUC as metrics.
Our ablation experiments demonstrate that each component contributes more or
less to the overall performance of the model.

MAE init As depicted in Figure 5, we have tested with various self-supervised
ViT strategies to initialize our model. Except for MAE, they all experienced
poor ability on IML tasks and could not converge eventually. We believe that
the pixel-level reconstruction task designed for MAE demonstrates the
capability to extract low-level semantics effectively, aiding the model in
convergence. In addition, as shown in Table II, the model trained using the
Xavier initialization method also encounters convergence issues, whereas
traditional classification pre-training struggles to generalize effectively on
non-homologous datasets. However, other settings with MAE initialization show
at least a 21.8% improvement in the average F1 score and converge rapidly,
indicating that using the MAE init with objective semantics and low-level
vision capacity can greatly aid the convergence and alleviate overfitting.

Figure 5: Training loss and F1 score on test dataset for self-supervised pre-
training algorithms. “Explosion” stickers represent gradient
explosion/vanishing problem with the loss becoming NaN(not a number).

High-resolution The model performance after removing the high resolution is
significantly reduced except for NIST16. For NIST16, because more of the
images are much larger than 1024×1024 resolution, there is already a large
amount of low-level features destroyed by downsampling when preprocessing. So
it can be considered that the decision is still mainly supported by object
semantics on this dataset, so there is not much change compared to others.
This also indirectly proves that multi-level visual feature is indeed
meaningful to solve the IML problem effectively.

Simple Feature Pyramid Performance on the COVERAGE dataset appears to be
better without the simple feature pyramid compared to the Full setup. However,
as indicated in Table I, the limited COVERAGE dataset only has 100 manipulated
images, and the tampering type is restricted to copy-move only. We contend
that achieving good performance on this dataset alone may indicate
overfitting. In contrast, effectiveness on larger, more diverse datasets that
are more practical and valuable. Thus, we argue that the simple feature
pyramid generalizes the model on new and varied tampering scenarios.

PMAE The PMAE module is an essential component of our model, as demonstrated
by the ablation study results. The average F1 score increased by 4.89% with
the PMAE branch, confirming that the PMAE module provides valuable low-level
visual information for the model to identify tampered regions accurately.

Figure 6: Localization results of PMAE compared to various methods. All
methods are trained on CASIAv2. Clear boundaries can observed in PMAE
predictions.

###  IV-D Compare with SOTA

While there has been considerable research in the area of image manipulation
localization, many works are not fully open-sourced. Some works that claim to
be open-sourced do not provide access to their training code or huge private
datasets, making it difficult to reproduce their results and compare them
fairly with other models. In this section, we compare our model with state-of-
the-art works that have published their models and parameters publicly. We
evaluated the performance of these models on top-tier conferences and journals
in recent years, following the commonly used protocol of training on the
CASIAv2[14] dataset and evaluating on smaller datasets.

#### Quantitative Analysis

We report the $F1$ score of these models and the complete results can be found
in Table III. Some of the metrics reported in this section are referenced from
MVSS-Net[11]. We observe substantial performance improvements on our PMAE
compared to previous works, even compared to the best-performing MVSS-Net++,
PMAE still achieves an 18.2% higher average $F1$ score of 0.502. This confirms
the effectiveness of the proposed argument about the combination of multi-
level vision traces. Although the F1 score of PMAE on COVERAGE dataset is
slightly lower than that of MVSS-Net++, as we mentioned in section IV-C,
COVERAGE is a limited dataset with only one type of tampering and a small
number of samples, so better performance on other larger datasets is more
valuable and worth well. This suggests that PMAE has better generalization
ability and indirectly demonstrates its effectiveness in discovering valuable
tampering information.

#### Qualitative Analysis

In Figure 6, we present the predicted manipulation mask of our PMAE and
compare it with the publicly available methods ManTra-Net[7] and MVSS-Net[11].
ManTra-Net and MVSS-Net are both FCN-based[45] IML methods utilizing
handcrafted filters to extract low-level visual artifacts from images.
However, we observed that although they can sometimes correctly detect areas
with heavy artifacts (mainly boundaries of tampered areas), they are not
confident in making a clear decision. In contrast, PMAE, with the support of
object-level semantics, can make a sharp and accurate prediction of tampered
areas with clear boundaries. It can also effectively combine suspected
dispersed regions into complete continuous areas.

###  IV-E Robustness Evaluation

#### Resize

As we applied a high-resolution input, we first evaluated the robustness of
the PMAE toward Resize. However, most SoTA methods have not released their
training code and the current convention does not emphasize the requirement
for consistent resolution in comparing IML methods. Thus, We resized images to
the same resolution as the SoTA model and compared the performance of PMAE
with them, results are shown in Table IV.

TABLE IV: Robustness test toward resize algorithm.

Method | Resize | CASIAv1 | Columbia | Coverage | NIST16 | MEAN
---|---|---|---|---|---|---
GSR-Net | 300×300 | 0.387 | 0.613 | 0.285 | 0.283 | 0.392
PMAE | 300×300 | 0.687 | 0.469 | 0.399 | 0.271 | 0.457
MVSS-Net++ | 512×512 | 0.513 | 0.660 | 0.482 | 0.304 | 0.490
PMAE | 512×512 | 0.649 | 0.736 | 0.441 | 0.300 | 0.531
PMAE | Zero-padded | 0.688 | 0.860 | 0.473 | 0.311 | 0.583

While it is natural for our model to experience some degree of performance
decline, overall, it maintains a strong average performance level. The varying
extent of decline across different datasets can be attributed to the fact that
previous models directly extract specific features for identifying tampering,
which are effective for specific tampering types. In contrast, PMAE primarily
learns tampering features from the CASIAv2 dataset using self-supervised
methods, making it more adaptable to homogeneous datasets.

#### Common Distortions

Additionally, following MVSS-Net[28], we apply image distortion methods on raw
input images from the CASIAv1 dataset and further evaluate the robustness of
our PMAE model, utilizing pixel-level $F1$ score as the metrics to compare our
model with ManTra-Net[7] and MVSS-Net[11]. Note that all methods are trained
on pure CASIAv2 dataset without distortion. The distortion types include: 1)
Gaussian blurring with a kernel size $k$; 2) JPEG compression with a quality
factor $q$. The results are shown in Table V. The PMAE maintains relatively
high performance against various compression methods, demonstrating the
model’s considerable robustness for practical applications.

TABLE V: Robustness analysis of models on CASIAv1, evaluated with pixel-level $F1$ score(%).  Operations | ManTra-Net | MVSS-Net | PMAE(Ours)
---|---|---|---
None | 15.5 | 51.3 | 73.0
JPEG Compress(100) | 15.3 | 45.1 | 76.1$\uparrow$
JPEG Compress(90) | 12.0 | 43.0 | 73.8$\uparrow$
JPEG Compress(80) | 8.1 | 42.4 | 68.2
JPEG Compress(70) | 8.0 | 41.2 | 65.2
JPEG Compress(60) | 8.7 | 40.1 | 62.0
JPEG Compress(50) | 8.5 | 39.4 | 56.6
Gaussian Blur(size=5) | 12.1 | 39.2 | 72.8
Gaussian Blur(size=11) | 11.4 | 32.4 | 65.4

However, an interesting phenomenon is that our model performs better at a
compression quality of 100 and 90 for JPEG compression, which is unexpected.
Therefore, we further test the robustness of our model on other datasets to
explain this issue, results are shown in Table VI.

TABLE VI: Exploration of abnormal increase after slight distortion on PAME.
Evaluate with pixel-level $F1$ score, All anomalous growth is marked with the
* symbol.

Compression | CASIAv1 | NIST16 | COVERAGE | Columbia
---|---|---|---|---
None | 0.7307 | 0.3109 | 0.4731 | 0.8595
JpegCompression(100) | 0.7671* | 0.2895 | 0.3907 | 0.8406
JpegCompression(90) | 0.7435* | 0.2742 | 0.3819 | 0.8044
Gaussian Blur(size=3) | 0.7419* | 0.3142* | 0.4054 | 0.8109
Gaussian Blur(size=5) | 0.7325* | 0.3235* | 0.3789 | 0.7581

Here, we only observe this exception on NIST16 and CASIAv1 datasets. The
explanation is as follows: there is a main commonality between these two
datasets in our experiments, which is that they have undergone significant
pre-processing. The CASIAv1 dataset itself resized all images to 256x384 (or
384x256) and added several noises before releasing, while NIST16, due to its
large resolution, exceeded our 1024x1024 limit, so we downsampled them by
resize. These operations have already destroyed a large number of low-level
artifacts, such as noise from the camera, in these two datasets. In the model
inference process, these datasets actually mostly rely on high-level visual
features rather than low-level features to support their decisions. Therefore,
slight blurring can help the model eliminate interference and better focus on
object-level inconsistencies and incoherence, thus improving accuracy. In
contrast, images in the COVERAGE and Columbia datasets are “pure”, without any
pre-processing, which still contain a considerable amount of low-level
information to support decision-making when passed to the model. Distortion
will directly destroy this part of the information and reduce the total
information available for supporting prediction, leading to decreased
accuracy. Overall, this exceptional increment is also indirect evidence that
our model has successfully reconciled multi-level visual information and can
flexibly infer the manipulated area based on the richness of the two types of
information.

##  V Conclusion

This paper presents a novel approach to image manipulation detection by
reformulating the task as a high-level vision task that greatly benefits from
low-level features. Our proposed method, Perceptual Masked Autoencoder (PMAE),
captures and balances multi-level visual information to effectively segment
tampered areas. Through extensive experiments on multiple public datasets,
PMAE has achieved state-of-the-art performance in F1-score, AUC, robustness,
and generalization. Our results provide comprehensive evidence that
incorporating both low-level and high-level features is necessary for
effectively addressing image tampering, especially at the object level.

In a nutshell, the proposed PMAE training paradigm represents a new state-of-
the-art approach to solving multimedia image tampering. Future research should
consider the distribution of low-level and high-level information in datasets
and the real world when designing models. Furthermore, our proposed method can
effectively address inpainting tampering, indicating its ability to recognize
tampered information generated by large-scale models and its potential for
practical applications.

## References

  * [1] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. New Orleans, LA, USA: IEEE, Jun 2022, p. 10674–10685. [Online]. Available: https://ieeexplore.ieee.org/document/9878449/
  * [2] S. Weng, T. Zhu, T. Zhang, and C. Zhang, “Ucm-net: A u-net-like tampered-region-related framework for copy-move forgery detection,” _IEEE Transactions on Multimedia_ , p. 1–14, 2023.
  * [3] Y. Li, J. You, J. Zhou, W. Wang, X. Liao, and X. Li, “Image operation chain detection with machine translation framework,” _IEEE Transactions on Multimedia_ , p. 1–16, 2022.
  * [4] F. Li, Z. Pei, X. Zhang, and C. Qin, “Image manipulation localization using multi-scale feature fusion and adaptive edge supervision,” _IEEE Transactions on Multimedia_ , p. 1–15, 2022.
  * [5] B. Chen, W. Tan, G. Coatrieux, Y. Zheng, and Y.-Q. Shi, “A serial image copy-move forgery localization scheme with source/target distinguishment,” _IEEE Transactions on Multimedia_ , vol. 23, p. 3506–3517, 2021.
  * [6] L. Verdoliva, “Media forensics and deepfakes: An overview,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 14, no. 5, p. 910–932, Aug 2020.
  * [7] Y. Wu, W. AbdAlmageed, and P. Natarajan, “Mantra-net: Manipulation tracing network for detection and localization of image forgeries with anomalous features,” in _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. Long Beach, CA, USA: IEEE, Jun 2019, p. 9535–9544. [Online]. Available: https://ieeexplore.ieee.org/document/8953774/
  * [8] P. Zhou, X. Han, V. I. Morariu, and L. S. Davis, “Learning rich features for image manipulation detection,” in _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_. Salt Lake City, UT, USA: IEEE, Jun 2018, p. 1053–1061. [Online]. Available: https://ieeexplore.ieee.org/document/8578214/
  * [9] B. Bayar and M. C. Stamm, “Constrained convolutional neural networks: A new approach towards general purpose image manipulation detection,” _IEEE Transactions on Information Forensics and Security_ , vol. 13, no. 11, p. 2691–2706, Nov 2018.
  * [10] P. Zhou, B.-C. Chen, X. Han, M. Najibi, A. Shrivastava, S.-N. Lim, and L. Davis, “Generate, segment, and refine: Towards generic manipulation segmentation,” _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 07, p. 13058–13065, Apr 2020.
  * [11] X. Chen, C. Dong, J. Ji, J. Cao, and X. Li, “Image manipulation detection by multi-view multi-scale supervision,” in _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_. Montreal, QC, Canada: IEEE, Oct 2021, p. 14165–14173. [Online]. Available: https://ieeexplore.ieee.org/document/9710015/
  * [12] J. Wang, Z. Wu, J. Chen, X. Han, A. Shrivastava, S.-N. Lim, and Y.-G. Jiang, “Objectformer for image manipulation detection and localization,” in _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. New Orleans, LA, USA: IEEE, Jun 2022, p. 2354–2363. [Online]. Available: https://ieeexplore.ieee.org/document/9880322/
  * [13] H. Li and J. Huang, “Localization of deep inpainting using high-pass fully convolutional network,” in _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_. Seoul, Korea (South): IEEE, Oct 2019, p. 8300–8309. [Online]. Available: https://ieeexplore.ieee.org/document/9009804/
  * [14] J. Dong, W. Wang, and T. Tan, “Casia image tampering detection evaluation database,” in _2013 IEEE China Summit and International Conference on Signal and Information Processing_. Beijing, China: IEEE, Jul 2013, p. 422–426. [Online]. Available: http://ieeexplore.ieee.org/document/6625374/
  * [15] H. Bao, L. Dong, S. Piao, and F. Wei, “Beit: Bert pre-training of image transformers,” no. arXiv:2106.08254, Sep 2022, arXiv:2106.08254 [cs]. [Online]. Available: http://arxiv.org/abs/2106.08254
  * [16] J. Zhou, C. Wei, H. Wang, W. Shen, C. Xie, A. Yuille, and T. Kong, “ibot: Image bert pre-training with online tokenizer,” _arXiv preprint arXiv:2111.07832_ , 2021.
  * [17] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision transformers,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 9650–9660.
  * [18] K. He, X. Chen, S. Xie, Y. Li, P. Dollar, and R. Girshick, “Masked autoencoders are scalable vision learners,” in _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. New Orleans, LA, USA: IEEE, Jun 2022, p. 15979–15988. [Online]. Available: https://ieeexplore.ieee.org/document/9879206/
  * [19] P. Zhou, X. Han, V. I. Morariu, and L. S. Davis, “Learning rich features for image manipulation detection,” in _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_. Salt Lake City, UT, USA: IEEE, Jun 2018, p. 1053–1061. [Online]. Available: https://ieeexplore.ieee.org/document/8578214/
  * [20] X. Ma, B. Du, Z. Jiang, A. Y. A. Hammadi, and J. Zhou, “Iml-vit: Benchmarking image manipulation localization by vision transformer,” no. arXiv:2307.14863, Aug 2023, arXiv:2307.14863 [cs]. [Online]. Available: http://arxiv.org/abs/2307.14863
  * [21] Y.-f. Hsu and S.-f. Chang, “Detecting image splicing using geometry invariants and camera characteristics consistency,” in _2006 IEEE International Conference on Multimedia and Expo_. Toronto, ON, Canada: IEEE, Jul 2006, p. 549–552. [Online]. Available: http://ieeexplore.ieee.org/document/4036658/
  * [22] B. Wen, Y. Zhu, R. Subramanian, T.-T. Ng, X. Shen, and S. Winkler, “Coverage — a novel database for copy-move forgery detection,” in _2016 IEEE International Conference on Image Processing (ICIP)_. Phoenix, AZ, USA: IEEE, Sep 2016, p. 161–165. [Online]. Available: http://ieeexplore.ieee.org/document/7532339/
  * [23] H. Guan, M. Kozak, E. Robertson, Y. Lee, A. N. Yates, A. Delgado, D. Zhou, T. Kheyrkhah, J. Smith, and J. Fiscus, “Mfc datasets: Large-scale benchmark datasets for media forensic challenge evaluation,” in _2019 IEEE Winter Applications of Computer Vision Workshops (WACVW)_. Waikoloa Village, HI, USA: IEEE, Jan 2019, p. 63–72. [Online]. Available: https://ieeexplore.ieee.org/document/8638296/
  * [24] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [25] Z. Xie, Z. Zhang, Y. Cao, Y. Lin, J. Bao, Z. Yao, Q. Dai, and H. Hu, “Simmim: A simple framework for masked image modeling,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 9653–9663.
  * [26] J.-L. Zhong and C.-M. Pun, “An end-to-end dense-inceptionnet for image copy-move forgery detection,” _IEEE Transactions on Information Forensics and Security_ , vol. 15, pp. 2134–2146, 2019.
  * [27] J. H. Bappy, C. Simons, L. Nataraj, B. Manjunath, and A. K. Roy-Chowdhury, “Hybrid lstm and encoder–decoder architecture for detection of image forgeries,” _IEEE Transactions on Image Processing_ , vol. 28, no. 7, pp. 3286–3300, 2019.
  * [28] C. Dong, X. Chen, R. Hu, J. Cao, and X. Li, “Mvss-net: Multi-view multi-scale supervised networks for image manipulation detection,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , p. 1–14, 2022.
  * [29] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” _arXiv preprint arXiv:1312.6114_ , 2013.
  * [30] Y. Li, C.-Y. Wu, H. Fan, K. Mangalam, B. Xiong, J. Malik, and C. Feichtenhofer, “Mvitv2: Improved multiscale vision transformers for classification and detection,” in _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. New Orleans, LA, USA: IEEE, Jun 2022, p. 4794–4804. [Online]. Available: https://ieeexplore.ieee.org/document/9879809/
  * [31] Y. Li, S. Xie, X. Chen, P. Dollar, K. He, and R. Girshick, “Benchmarking detection transfer learning with vision transformers,” no. arXiv:2111.11429, Nov 2021, arXiv:2111.11429 [cs]. [Online]. Available: http://arxiv.org/abs/2111.11429
  * [32] Y. Li, H. Mao, R. Girshick, and K. He, “Exploring plain vision transformer backbones for object detection,” in _Computer Vision – ECCV 2022_ , ser. Lecture Notes in Computer Science, S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, and T. Hassner, Eds. Cham: Springer Nature Switzerland, 2022, p. 280–296.
  * [33] E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo, “Segformer: Simple and efficient design for semantic segmentation with transformers,” no. arXiv:2105.15203, Oct 2021, arXiv:2105.15203 [cs]. [Online]. Available: http://arxiv.org/abs/2105.15203
  * [34] J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” no. arXiv:1603.08155, Mar 2016, arXiv:1603.08155 [cs]. [Online]. Available: http://arxiv.org/abs/1603.08155
  * [35] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” no. arXiv:1409.1556, Apr 2015, arXiv:1409.1556 [cs]. [Online]. Available: http://arxiv.org/abs/1409.1556
  * [36] X. Hu, Z. Zhang, Z. Jiang, S. Chaudhuri, Z. Yang, and R. Nevatia, “Span: Spatial pyramid attention network for image manipulation localization,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI 16_. Springer, 2020, pp. 312–328.
  * [37] C. Yang, Z. Wang, H. Shen, H. Li, and B. Jiang, “Multi-modality image manipulation detection,” in _2021 IEEE International conference on multimedia and expo (ICME)_. IEEE, 2021, pp. 1–6.
  * [38] G. Mahfoudi, B. Tajini, F. Retraint, F. Morain-Nicolier, J. L. Dugelay, and M. Pic, “Defacto: Image and face manipulation dataset,” in _2019 27th European Signal Processing Conference (EUSIPCO)_. A Coruna, Spain: IEEE, Sep 2019, p. 1–5. [Online]. Available: https://ieeexplore.ieee.org/document/8903181/
  * [39] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, _Microsoft COCO: Common Objects in Context_ , ser. Lecture Notes in Computer Science. Cham: Springer International Publishing, 2014, vol. 8693, p. 740–755. [Online]. Available: http://link.springer.com/10.1007/978-3-319-10602-1_48
  * [40] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” no. arXiv:1711.05101, Jan 2019, arXiv:1711.05101 [cs, math]. [Online]. Available: http://arxiv.org/abs/1711.05101
  * [41] ——, “Sgdr: Stochastic gradient descent with warm restarts,” no. arXiv:1608.03983, May 2017, arXiv:1608.03983 [cs, math]. [Online]. Available: http://arxiv.org/abs/1608.03983
  * [42] H. Li and J. Huang, “Localization of deep inpainting using high-pass fully convolutional network,” in _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_. Seoul, Korea (South): IEEE, Oct 2019, p. 8300–8309. [Online]. Available: https://ieeexplore.ieee.org/document/9009804/
  * [43] C. Yang, H. Li, F. Lin, B. Jiang, and H. Zhao, “Constrained r-cnn: A general image manipulation detection model,” in _2020 IEEE International conference on multimedia and expo (ICME)_. IEEE, 2020, p. 1–6.
  * [44] X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in _Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics_. JMLR Workshop and Conference Proceedings, Mar 2010, p. 249–256. [Online]. Available: https://proceedings.mlr.press/v9/glorot10a.html
  * [45] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 3431–3440.