# Using Particle Swarm Optimization as Pathfinding Strategy in a Space with
Obstacles

Davida and Budi Adiperdanaa CONTACT David. Email<EMAIL_ADDRESS>aDepartment of Physics, Faculty Mathematics and Natural Sciences, Universitas
Padjadjaran

###### Abstract

Particle swarm optimization (PSO) is a search algorithm based on stochastic
and population-based adaptive optimization. In this paper, a pathfinding
strategy is proposed to improve the efficiency of path planning for a broad
range of applications. This study aims to investigate the effect of PSO
parameters (numbers of particle, weight constant, particle constant, and
global constant) on algorithm performance to give solution paths. Increasing
the PSO parameters makes the swarm move faster to the target point but takes a
long time to converge because of too many random movements, and vice versa.
From a variety of simulations with different parameters, the PSO algorithm is
proven to be able to provide a solution path in a space with obstacles.

###### keywords:

Particle swarm optimization, Swarm intelligence, Pathfinding

††articletype: Article

##  1 Introduction

Swarm intelligence (SI) is an artificial intelligence approach that aims to
solve problems using algorithms based on the collective behavior of social
animals. Swarm intelligence is different from evolutionary algorithms. This is
because the evolutionary algorithm develops a new population for each
generation/iteration, while the swarm intelligence algorithm improves the
individual for each generation/iteration (Bansal, 2019). The most popular SI
methods include Artificial Bee Colony Algorithm, Ant Colony Optimization, and
Particle Swarm Optimization (Karkalos et al., 2019).

Particle Swarm Optimization (PSO) algorithm, which was invented by Kennedy and
Eberhart (Kennedy et al., 1995), is an optimization method inspired by the
social behavior of animal such as bird flocking and fish schooling. PSO is a
population-based search algorithm, where each individual/particle changes its
position each time to explore a multi-dimensional space. The PSO algorithm
combines the local search with the global search method. Each particle has a
position and velocity in the space of a certain dimension. Initialization of
the PSO algorithm starts by setting the initial position of the particle, and
then looking for the optimal value by updating its position randomly. For each
iteration, particle updates its position following the two best values: the
best solution that has been obtained by itself (local best) and the best
solution in the population (global best). After the particle position is
evaluated by the fitness function, each particle ”communicate” the information
of best position to the other particles, so they can adjusts the position and
velocity toward that best point.

Figure 1: Particle position update process (Chen et al., 2020)

One of the most important tasks in mobile robot navigation problems is to plan
a collision-free route from the starting point to the target position. One of
them uses an optimization process. However, traditional optimization
techniques create local minima, and the robot may not reach its goal even if a
solution exists (Falcó et al., 2020). Over the last few years, several
heuristic algorithms have been introduced into path planning (Zhu et al.,
2021), including PSO. The PSO algorithm is known to be very good for use in
free space problems (Waluyo et al., 2010). This paper aims to investigate the
ability of PSO algorithm to find solutions/pathfinding in a space with
obstacles, so if it is successful, it can be used to find solutions for maze
games and path planning in robot movement.

##  2 Algorithm

With the standard PSO algorithm, the path planning can be divided into the
following steps, as shown in Figure 2.

Figure 2: PSO Algorithm Flowchart

First step is initialize a starting position, target position, obstacle
coordinate in the search space. Then, for each particle $i$, update the
position of particle $i$ according to equation (1).

$v_{i+1}=w\cdot
v_{i}+c_{1}\cdot\operatorname{rand}_{1}\left(x_{best,i}^{P}-x_{i}\right)+c_{2}\cdot\operatorname{rand}_{2}\left(x_{best,i}^{G}-x_{i}\right)$
(1)

where $v_{i}$ is the speed of the $i$-th iteration, $x_{i}$ is the particle
position of the $i$-th iteration, $x_{best,i}^{P}$ is the local best position
of the $i$-th iteration, $x_{best,i}^{G}$ is the globally best position of the
$i$-th iteration, $w$ is the inertia weight of the current speed when updating
the speed, $c_{1}$, $c_{2}$ are the follow factors, and rand1, rand2 are
uniform random numbers from 0–1. After that, update the velocity of particle
$i$ according to equation (2).

$x_{i+1}=x_{i}+v_{i+1}$ (2)

Then evaluate its fitness value according to equation (3).

$f=\left(x_{\text{target}}-x\right)^{2}$ (3)

The fitness function is used to evaluates how close current position is to the
targeted position. Next, update $x_{best,i}^{P}$ and $x_{best,i}^{G}$ if
necessary according to equation (4 & 5).

$\text{x}_{best,i+1}^{P}=\begin{cases}\text{x}_{best,i}^{P}&\text{ if
}f\text{(}x_{best,i}^{P})\leq f\left(x_{i}\right)\\\ x_{i}&\text{ if
}f\text{(}x_{best,i}^{P})>f\left(x_{i}\right)\end{cases}$ (4) $\displaystyle
x_{best,i+1}^{G}=\min\\{f(y),\quad f(x_{best,i}^{G})\\}$ (5)
$\displaystyle\text{where,}\quad
y\in\left\\{x_{best,0}^{P},x_{best,1}^{P},...,x_{best,N}^{P}\right\\}$

If it has converged, the path is saved and the iteration is exited. Otherwise,
it started again to continue the iteration.

##  3 Results

In this study, PSO algorithm is used to search the space of 100 × 100 nodes to
find the optimal path so the swarm can travel from its initial position to the
target point. The investigation of the PSO parameters effect on algorithm
performance is shown by the simulation result in Table 1.

Table 1: PSO Parameter and Algorithm Performance Initial Position | N Particles | $w$ | $c_{1}$ | $c_{2}$ | Steps/Iterations
---|---|---|---|---|---
(-70, 80) | 25 | 0.90 | 0.60 | 0.65 | 80-120
50 | 0.80 | 0.65 | 0.90 | 48-58
(-80, -10) | 25 | 0.60 | 0.90 | 1.45 | 84-133
50 | 0.45 | 1.20 | 1.70 | 40-56

The simulation shows that, the larger the population of particles, the fewer
steps needed to reach convergence. Even so, the number of particles should be
adjusted to the complexity of the obstacles. If the number of particles is too
small, the swarm will have difficulty finding a solution due to minimal
”communication” between particles during exploration. If the number of
particles is too large, the time required to converge to the target will be
longer due to too much random motion, but with the optimal number of particles
and parameters, the algorithm can work efficiently.

The combination of the weight constant ($w$), particle constant ($c_{1}$) and
global constant ($c_{2}$) is very important so that the search process becomes
more optimal. If these constants are enlarged, the particles will spread
easily so that the search process is faster but takes a long time to converge
or to ”summon” other particles to gather at the target. However, if the
constant is too small, the particles will move closely so it will be difficult
to find an alternative path if everything is stuck in the local minima.
Examples of solution paths obtained from several simulation cases are shown in
Figure 3.

Apart from the algorithm, simulation environment needs to be enhance because
in some cases particles can penetrating the obstacles due to the velocity
being too large. The algorithm above only refuses the particle position update
if the particle is not in the free-moving area or is inside of the obstacles
coordinates, but if the particle velocity is very fast, the particle
displacement becomes larger and the program does not detect it, so particles
possible to penetrates the obstacles wall. The solution to this problem is to
increase the size of the obstacles wall or limit the particle velocity by
setting the optimal constants.

Figure 3: The particle trace and the error for each step for (a,b) $n$ = 25,
($x_{0},y_{0}$) = (-80,-10);
(c,d) $n$ = 50, ($x_{0},y_{0}$) = (-80,-10); (e,f)$n$ = 25, ($x_{0},y_{0}$) =
(-70,80); (g,h) $n$ = 50, ($x_{0},y_{0}$) = (-70,80)

##  4 Conclusion

This paper presented the algorithm of using the PSO approach to solve the path
planning problem. This study also investigated the performance of the
evolutionary process with the various parameter and number of particles. Based
on the simulations, the PSO algorithm can provide a solution path to different
types of tasks in a space with obstacles.

## Code availability

The implementation code of this research is available for download at
github.com/dxid-dev/pathfinder. This research has used python and matplotlib
(Hunter, 2007).

## Disclosure statement

No potential conflict of interest was reported by the author.

## References

  * Bansal (2019) Bansal, J. C. (2019). Particle swarm optimization. In Evolutionary and swarm intelligence algorithms (bll 11–23). Springer.
  * Chen et al. (2020) Chen, Z., Li, X., Zhu, Z., Zhao, Z., Wang, L., Jiang, S., & Rong, Y. (2020). The optimization of accuracy and efficiency for multistage precision grinding process with an improved particle swarm optimization algorithm. International Journal of Advanced Robotic Systems, 17(1), 1729881419893508.
  * Falcó et al. (2020) Falcó, A., Hilario, L., Montés, N., Mora, M. C., & Nadal, E. (2020). A Path Planning Algorithm for a Dynamic Environment Based on Proper Generalized Decomposition. Mathematics, 8(12), 2245.
  * Hunter (2007) Hunter, J. D. (2007). Matplotlib: A 2D graphics environment. Computing in Science & Engineering, 9(3), 90–95. doi:10.1109/MCSE.2007.55
  * Karkalos et al. (2019) Karkalos, N. E., Markopoulos, A. P., & Davim, J. P. (2019). Swarm intelligence-based methods. In Computational Methods for Application in Industry 4.0 (bll 33–55). Springer.
  * Kennedy et al. (1995) Kennedy, J., & Eberhart, R. (1995). Particle swarm optimization. Proceedings of ICNN’95-international conference on neural networks, 4, 1942–1948. IEEE.
  * Waluyo et al. (2010) Waluyo, S. R., Hariadi, M., & Purnama, I. K. E. (2010). Pencarian Jalur Terbaik Menggunakan Particle Swarm Optimization untuk Mengoptimasi Lalu Lintas Kendaraan.
  * Wang et al. (2020) Wang, X., Shi, H., & Zhang, C. (2020). Path planning for intelligent parking system based on improved ant colony optimization. IEEE Access, 8, 65267–65273.
  * Zhu et al. (2021) Zhu, D.-D., & Sun, J.-Q. (2021). A new algorithm based on Dijkstra for vehicle path planning considering intersection attribute. IEEE Access, 9, 19761–19775.


# Self-Supervised Graph Transformer for
Deepfake Detection

Aminollah Khormali, Jiann-Shiun Yuan  A. Khormali and J. S. Yuan are with the
Department of Electrical and Computer Engineering, University of Central
Florida, Orlando, Fl, USA.

###### Abstract

Deepfake detection methods have shown promising results in recognizing
forgeries within a given dataset, where training and testing take place on the
in-distribution dataset. However, their performance deteriorates significantly
when presented with unseen samples. As a result, a reliable deepfake detection
system must remain impartial to forgery types, appearance, and quality for
guaranteed generalizable detection performance. Despite various attempts to
enhance cross-dataset generalization, the problem remains challenging,
particularly when testing against common post-processing perturbations, such
as video compression or blur. Hence, this study introduces a deepfake
detection framework, leveraging a self-supervised pre-training model that
delivers exceptional generalization ability, withstanding common corruptions
and enabling feature explainability. The framework comprises three key
components: a feature extractor based on vision Transformer architecture that
is pre-trained via self-supervised contrastive learning methodology, a graph
convolution network coupled with a Transformer discriminator, and a graph
Transformer relevancy map that provides a better understanding of manipulated
regions and further explains the model’s decision. To assess the effectiveness
of the proposed framework, several challenging experiments are conducted,
including in-data distribution performance, cross-dataset & cross-manipulation
generalization, and robustness against common post-production perturbations.
The results achieved demonstrate the remarkable effectiveness of the proposed
deepfake detection framework, surpassing the current state-of-the-art
approaches.

###### Index Terms:

Deepfake Detection, Graph Convolution Networks, Self-Supervised Contrastive
Learning, Vision Transformer

##  I Introduction

The rise of Artificial Intelligence (AI) empowered face
manipulation/generation technologies have enabled individuals’ expressions or
appearances to be realistically altered with minimal expert knowledge [1, 2,
3, 4, 5, 6]. This technology poses a severe and large-scale societal threat,
as it facilitates the creation and dissemination of malicious content, such as
digital kidnapping, ransomware, and other forms of criminal activity [7, 8]
that are among the most insidious forms of misinformation [9]. Consequently,
the development of reliable deepfake detection methods has become an urgent
need and has garnered significant attention in recent years. Although several
efforts have been made to defend against the growing threat of forged digital
media, the effectiveness of these approaches have primarily been limited to
in-dataset settings [10, 11, 12, 13, 14, 15, 5, 16, 17]. Therefore, the need
to develop more robust and effective deepfake detection methods that can
operate in real-world scenarios irrespective of the forgery type, appearance,
or quality has become increasingly pressing.

In the domain of deepfake detection, previous investigations have
predominantly relied upon low-level texture features to capture common
artifacts that are inherent to the generation process. Nonetheless, such
approaches are susceptible to severe performance degradation when applied to
novel types of forgeries, rendering the detection of real-world deepfakes a
challenging task, especially when the differences between genuine and
manipulated media are nuanced. As a result, an effective deepfake detection
model ought to be impartial towards forgery type, appearance, and quality, in
order to ensure applicability in real-world scenarios. To this end, several
techniques have been proposed to enhance the performance and generalization of
deepfake detectors, such as targeting the blending boundary between the
background and the altered face [12], utilizing 3D decomposition [18],
truncating classifiers [19], amplifying multi-band frequencies [14], and
augmenting data [20]. Despite their effectiveness in cross-data evaluations,
low-level texture cues may be vulnerable to degradation through standard post-
processing procedures, such as video compression [11]. Consequently, it is
imperative to develop novel high-level features that are invariant to image
manipulations and exhibit resilience to post-processing manipulations to
enhance the current state-of-the-art in deepfake detection.

Moreover, The widely-used convolutional neural network and Transformer treat
the image as a grid or sequence structure, which is not flexible to capture
irregular and complex facial landmarks. Instead of the regular grid or
sequence representation, this work processes the deepfake image in a more
flexible way. Human facial images can be viewed as a composition of parts,
such as eyes, ears, nose, etc. These facial regions naturally form a graph
structure. This graph representation of deepfakes offers a better
understanding of local pixels and their interconnection with other parts.
Moreover, the graph is a generalized data structure that grid and sequence can
be viewed as a special case of the graph. Viewing an image as a graph is more
flexible and effective for visual perception.

Given the limitations of existing methods, building a detection model that
capitalizes on high-level visual representations, which tend to be more
resilient to post-processing perturbations, could potentially mitigate the
dependence on dataset-specific patterns, leading to improved recognition
performance across various forgery types. In light of this, this study first
proposes a self-supervised pre-training approach for deepfake facial
representation learning. This approach utilizes a contrastive learning
framework based on masked image modeling to extract high-level visual
representations of facial landmarks that are invariant to variations in
lighting, pose, and other factors that are irrelevant to the identity of the
object or scene. The contrastive learning method maximizes the agreement
between two different augmentations of the same image using a contrastive loss
in the latent space [21, 22, 23]. This means that the learned representation
is not biased towards any particular class or label, but rather captures the
underlying structure of the input data.

Moreover, the present work offers an innovative approach that integrates the
unique characteristics of Graph Convolutional Networks (GCNs) and Transformer
architectures to capture complex dependencies between distinct regions of an
image and acquire more informative representations for deepfake detection. In
the context of deepfake detection, the input data can be projected as a graph,
wherein nodes denote various regions of the image, such as eyes or ears, and
edges represent the association between these regions. This is particularly
useful in detecting deepfakes, where only certain regions of the image may be
manipulated to create realistic but false identities. The utilization of GCNs
allows for modeling local relationships between nodes in the input graph,
providing the ability to learn complex features that capture the spatial
correlations between different facial landmarks. In contrast, Transformers
exhibit exceptional efficacy in encoding long-range correlations and global
interdependencies between pixels, rendering them especially appropriate for
deepfake detection tasks. Given that adversarial entities can manipulate
multiple regions of an image concurrently, the ability to model such complex
relationships is crucial for reliable detection. Finally, this work introduces
a graph Transformer relevancy map that contributes valuable insights into the
model’s explainability. By generating a saliency map that highlights the
importance of individual regions towards the output class label, this
component facilitates a more fine-grained analysis of the model’s decision-
making process and allows for a better understanding of the specific features
and regions of the image that are crucial for deepfake detection. Figure 1
shows the general framework of the proposed deepfake detection model using a
self-supervised contrastive learning approach and graph Transformer
architecture.

Contributions. Together, these contributions offer a promising solution to the
urgent need for robust and effective deepfake detection methods that can
operate irrespective of the forgery types, appearances, or qualities in real-
world scenarios. The proposed framework’s use of high-level visual
representations that are invariant to post-processing perturbations, combined
with the ability to capture complex interdependencies between regions of an
image, provides a comprehensive solution that outperforms existing methods.
Below are the key contributions of this study:

  * •

The present study introduces a self-supervised pre-training approach based on
contrastive learning to extract more high-level visual representations that
are invariant to variations in compression, blur, and other factors irrelevant
to the subject’s identity. This approach significantly enhances the
generalizability of deepfake detection models, thereby reducing their
dependence on dataset-specific patterns.

  * •

Furthermore, this paper proposes an innovative deepfake detection approach by
leveraging the expressive power of graph convolutional networks and
Transformers to capture intricate interdependencies among different regions of
an image and acquire more informative representations. Unlike traditional
convolutional neural networks, which are limited in their ability to model
non-local relationships between pixels, the proposed graph Transformer can
encode both local and global dependencies, rendering them particularly suited
to deepfake detection tasks.

  * •

By providing detailed and granular insights into the underlying reasoning
behind the model’s predictions, the proposed graph Transformer relevancy map
offers a more thorough and rigorous understanding of the complex
interdependencies and salient features that drive the detection process, while
suppressing irrelevant or redundant information. Thus, facilitating a rigorous
and accurate examination of deepfakes, contributing to enhanced detection
performance and increased reliability.

  * •

The proposed framework’s efficacy and generalizability were rigorously
evaluated via a comprehensive set of experiments, encompassing a diverse range
of challenging scenarios spanning both in-distribution and out-of-distribution
settings. The experimental results unequivocally establish the framework’s
superiority, with exceptional in-dataset detection accuracy being achieved.
Moreover, the proposed self-supervised pre-training feature extractor
constitutes a significant contribution to the field, having markedly improved
the framework’s ability to generalize across multiple datasets while
simultaneously enhancing its resilience to post-processing perturbations, such
as compression and blur etc.

Organization. The rest of the paper is structured as follows. In §II, a review
of recent research at the intersection of deepfake detection and self-
supervised contrastive learning is provided. The proposed self-supervised
graph Transformer deepfake detection approach, including its key components
and the rationale for their selection, is outlined in §III. The evaluation
settings, including datasets, implementation specifics, and evaluation
metrics, are described in §IV. In §V, the obtained deepfake detection results
and their implications are presented and discussed in comparison with existing
methods. Finally, in §VI, concluding remarks are drawn to summarize the
contributions of this study and outline avenues for future research.

##  II Related Work

Figure 1:  The general framework of the proposed deepfake detection model
using a self-supervised contrastive learning approach and graph Transformer
architecture. In this approach, each input image is partitioned into smaller
patches, where each patch is considered as a node in an undirected graph. A
feature extractor model based on vision transformer architecture, which is
pre-trained based on contrastive learning methodology, is then applied to each
patch to obtain its visual features. These nodes and associated feature
vectors are restructured in the form of a graph based on the adjacency matrix.
The constructed graph is subsequently fed into a graph convolutional network,
which is followed by a Transformer network that functions as a deepfake
discriminator, with the purpose of forecasting image class labels. Finally,
the Transformer relevancy map is utilized to identify potentially manipulated
facial regions. The suggested deepfake detection mechanism capitalizes on the
merits of both high-level visual characteristics and the interconnectivity of
local and global regions through the pre-trained feature extractor and graph
Transformer classifier, respectively.

###  II-A Deepfake Detection

Recent years have witnessed a growing body of research aimed at addressing the
pressing challenges posed by deepfakes and developing robust detection models
capable of thwarting their nefarious effects on society. A variety of
techniques have been proposed in the literature to this end, with a rich
diversity of approaches being explored [10, 11, 8, 17, 12, 13, 24, 16].

One popular strategy is to leverage the implicit visual artifacts of
deepfakes’ generation process to construct a reliable detection framework [12,
25]. These approaches capitalize on the inconsistencies and discrepancies that
inevitably arise during the creation of deepfakes, such as blending artifacts
or anomalies in the frequency spectrum. Examples of such approaches include
Patch-wise Consistency Learning [26] and Face X-ray [12], which exploit image
blending inconsistency traces between the manipulated face region and the
background. These approaches exemplify the diverse range of methods used to
detect deepfakes, with some leveraging specific visual artifacts while many
mainstream deepfake detection methodologies rely heavily on the versatility of
convolutional neural networks. For instance, [27, 8, 28, 5] rely on the
exceptional ability of convolutional neural networks to capture complex
spatial features and relationships as the primary building block of their
proposed deepfake detection framework.

Additionally, recent research has explored alternative avenues, such as
analyzing the temporal dynamics of videos or using frequency spectrum analysis
to detect deepfakes based on unique spectral signatures. For instance, the
temporal dynamics of videos have been explored as a means to recongnize fake
videos [29, 30, 31, 32]. This approach involves examining the directional
changes in the video’s temporal characteristics, thereby detecting any
inconsistencies or abnormal patterns indicative of deepfakes. Furthermore,
frequency spectrum analysis is gaining significant traction in research
interest [33, 34, 35, 13, 36, 14, 15]. This technique capitalizes on
leveraging the identification of spectral signatures exhibited by deepfakes,
such as the existence of high-frequency noise or low-pass filtering artifacts,
enabling the detection of manipulated content.

In light of recent advancements in deepfake generation techniques that have
reduced the visual anomalies, the detection methods have shifted focus towards
more sophisticated approaches such as attention mechanisms [29, 37, 16, 8, 38]
and vision Transformers [17, 24, 39]. For example, Wang et al. [24] proposed a
multi-regional attention mechanism to enhance deepfake detection performance.
Additionally, vision Transformers have been employed to establish an end-to-
end deepfake detection framework [17, 40]. Furthermore, recent research has
utilized pre-trained networks and developed Lipforensics [11] for analyzing
lip prints in lip-reading tasks. This work highlights that while a robust pre-
trained lip feature extractor could have far-reaching implications, acquiring
a large-scale and well-annotated dataset could be a daunting task. Given the
potential limitations of relying on annotated data, this study proposes a
self-supervised vision Transformer for pretraining the feature extractor
without the need for annotation or labels. The resulting framework is scalable
and achieves good performance.

###  II-B Self-Supervised Contrastive Learning

A meaningful visual representation of a given input image could be learned
through self-supervised contrastive learning methods without relying on
labeled data [41]. Recently, self-supervised contrastive learning approaches
have gained tremendous attention in building pre-trained models that can be
used for fine-tuning different downstream tasks. Contrastive learning
maximizes the similarity between positive pairs while repelling features of
negative pairs [21]. Although contrastive learning approaches have been
utilized for dense visual representation learning tasks [42, 43], recent
studies suggest that removing negative pairs yields better performances [44,
45, 46, 47]. While Random augmentations have been widely used to generate
diverse views of the same image [21, 48, 49], this work, inspired by [50],
proposes a masked image modeling approach to extract more robust semantic
features through a contrastive learning methodology. In contrast to previous
work that only focused on mouth regions for lip-forensics [51], this work
leverages the entire facial region to improve the detection performance. The
proposed masked image modeling approach was utilized to train a pre-trained
model, with a vision Transformer serving as its backbone. Through this
approach, complex semantic features from input images were captured by the
model, resulting in improved deepfake detection performance.

##  III Methodology

The proposed deepfake detection method entails four fundamental components, as
illustrated in Figure 1, which are deepfake graph construction, deepfake
encoder based on vision Transformer architecture that is pre-trained using a
self-supervised contrastive learning approach, graph Transformer classifier,
and graph Transformer relevancy map. This section is dedicated to elaborating
on each of these components in a more comprehensive manner.

###  III-A Deepfake Graph Representation

The irregularity of facial regions’ shape constitutes a major obstacle in
employing conventional techniques, such as grid-based architectures in CNNs or
sequence-based structures in Transformers, to process facial images. Such
approaches are characterized by redundancy and inflexibility, rendering them
incapable of efficiently addressing the complex structural nuances of facial
regions. As a consequence, there is a need for more sophisticated and adaptive
techniques that can account for the varying geometries and topologies of these
regions. In this regard, the present study advocates the adoption of graph
representation of deepfakes as a highly promising method to tackle the issue
of deepfake detection, owing to its capacity to account for the varying
topologies and geometries of the facial regions. By leveraging this approach
to represent deepfake images as graphs, it becomes possible to capture the
underlying structural and relational features of key facial landmarks, such as
the eyes, lips, and ears. This capability, in turn, enables the identification
of potentially manipulated regions and their relationship within the image,
thereby capturing more complex features for enhanced and robust deepfake
detection.

The construction of an undirected graph from an image input is a pivotal step
in the proposed deepfake detection methodology. In essence, the process of
representing a given input image $I$ as a graph $G=(V,E)$ involves
partitioning it into $N$ patches, where
$V=\left\\{v_{1},v_{2},\dots,v_{N}\right\\}$ signifies the nodes of the graph
and $E$ denotes its corresponding edges. This entails representing feature
embeddings extracted from image patches as graph nodes, denoted by
$v_{i}\in\mathbb{R}^{D}$, where $D$ is the dimensionality of the embedding
vector. The adjacency relationship among these patches constitutes the graph
edges $E$. Notably, edges within the graph are established by means of a
spatial proximity criterion that determines the association of each patch with
its neighboring counterparts, allowing for at least one, and up to $K$
connections between $K$ nearest neighboring patches. To encode the adjacency
relationship between patches, an adjacency matrix, $A=[A_{ij}]$, is
formulated, is formulated, where $A_{ij}=A_{ji}=1$ if the nodes $v_{i}$ and
$v_{j}$ are in the neighborhoods, and $A_{ij}=A_{ji}=0$ otherwise, that is if
$(v_{i},v_{j})\notin E$. By modeling deepfake images in this way, the
structural relationships and dependencies among the various facial components
can be captured and analyzed in a more comprehensive manner. Furthermore, the
efficacy of the proposed methodology depends crucially on the feature vector’s
ability to provide a potent and resilient representation of the node in
question. To achieve this, the present study utilizes self-supervised
contrastive learning and a vision Transformer structure to extract high-level
visual representations of deepfake images. This enables the detection system
to capture the salient attributes of the underlying image regions, resulting
in more informative and discriminative features. As high-level features are
less vulnerable to manipulations, this approach maximizes the model’s
potential to generalize to new data and perform well on downstream tasks.

Figure 2: General framework of the vision Transformer-based feature extractor
trained through self-supervised contrastive learning. Given a deepfake image,
$x$, two different views, $u$ and $v$, are passed through student and teacher
networks. The student network tries to reconstruct masked tokens with the
supervision of teacher networks’ output by minimizing self-distillation
between cross-view $[\mathrm{CLS}]$ tokens and self-distillation between in-
view patch tokens.

###  III-B Self-Supervised Contrastive Learning

Existing deepfake detection methodologies heavily rely on weakly supervised
learning approaches for visual feature extraction. While such approaches
exhibit high detection performance on in-distribution samples, their ability
to generalize to unseen samples is limited. This generalizability issue poses
a significant challenge to real-world applications where the model must be
robust to unseen scenarios. To overcome this issue, this study proposes the
utilization of self-supervised contrastive learning for robust visual feature
extraction without the need for explicit labels. Specifically, the proposed
approach employs masked image modeling, which applies a random mask to an
input image with $N$ token sequences and a prediction ratio of $r$, to train a
vision Transformer for generating patch representations. The model maximizes
the concordance between two distinct views of the same image through
contrastive loss in a latent space. This approach is inspired by recent
studies on self-supervised contrastive learning, including [21], and builds on
the successful application of masked image modeling in [50, 52]. Figure 2
illustrates the proposed self-supervised contrastive learning framework for
deepfake representation learning. The resulting visual features are expected
to be more informative and discriminative, improving the model’s ability to
generalize to unseen data and achieve high performance on downstream tasks.

The proposed training methodology commences with the random selection of an
image $x$ from the training set, followed by the application of random
augmentations to generate two distorted views of the same image, denoted as
$\boldsymbol{u}$ and $\boldsymbol{v}$. To enhance robustness, blockwise
masking [50] is then applied to produce masked views of the augmented images,
denoted as $\hat{\boldsymbol{u}}$ and $\hat{\boldsymbol{v}}$. Subsequently,
the masked views undergo processing by a student-teacher network to yield
predictive categorical distributions. Both the student and teacher networks
share an identical architectural configuration and are parameterized by
$\boldsymbol{\theta}$ and $\boldsymbol{\theta}^{\prime}$, respectively. This
framework enables the self-supervised training of the student network, which
endeavors to reconstruct the masked tokens by leveraging the guidance of the
teacher network’s output.

In self-supervised frameworks, it is customary for the student network to
acquire distilled knowledge from the teacher network by minimizing the cross-
entropy loss [22, 44, 46]. However, in the present study, the deepfake images’
visual representations are acquired through the minimization of two concurrent
loss functions, namely: 1) $\mathcal{L}{[\mathrm{CLS}]}$, which represents the
self-distillation between cross-view $[\mathrm{CLS}]$ tokens, and 2)
$\mathcal{L}{[\mathrm{MIM}]}$, which represents the self-distillation between
in-view patch tokens. Specifically, the self-distillation process regarding
the $[\mathrm{CLS}]$ tokens of cross-view images, denoted by $\boldsymbol{u}$
and $\boldsymbol{v}$, can be formulated as a symmetric loss, as shown in
Equation 1.

$\displaystyle\mathcal{L}_{[\mathrm{CLS}]}$
$\displaystyle=\frac{1}{2}\left(-P_{\boldsymbol{\theta}^{\prime}}^{[\mathrm{CLS}]}(\boldsymbol{v})^{\mathrm{T}}\log
P_{\boldsymbol{\theta}}^{[\mathrm{CLS}]}(\boldsymbol{u})\right)+$ (1)
$\displaystyle\frac{1}{2}\left(-P_{\boldsymbol{\theta}^{\prime}}^{[\mathrm{CLS}]}(\boldsymbol{u})^{\mathrm{T}}\log
P_{\boldsymbol{\theta}}^{[\mathrm{CLS}]}(\boldsymbol{v})\right)$

where
${\boldsymbol{u}}_{s}^{\mathrm{CLS}}=P_{\boldsymbol{\theta}}^{\mathrm{CLS}}({\boldsymbol{u}})$,
${\boldsymbol{v}}_{s}^{\mathrm{CLS}}=P_{\boldsymbol{\theta}}^{\mathrm{CLS}}({\boldsymbol{v}})$,
$\boldsymbol{u}_{t}^{\mathrm{CLS}}=P_{\boldsymbol{\theta}^{\prime}}^{\mathrm{CLS}}(\boldsymbol{u})$,
and
$\boldsymbol{v}_{t}^{\mathrm{CLS}}=P_{\boldsymbol{\theta}^{\prime}}^{\mathrm{CLS}}(\boldsymbol{v})$
are student and teacher network outputs for cross-view images, i.e.,
$\boldsymbol{u}$ and $\boldsymbol{v}$, of [CLS] token.

The student and teacher network outputs for the masked view with $m_{i}$
masking ratio for $\hat{\boldsymbol{u}}$, $\hat{\boldsymbol{v}}$ and non-
masked view $\boldsymbol{u}$, $\boldsymbol{v}$ projections of patch tokens are
claculated as
$\hat{\boldsymbol{u}}_{s}^{\mathrm{patch}}=P_{\boldsymbol{\theta}}^{\mathrm{patch}}(\hat{\boldsymbol{u}})$,
$\hat{\boldsymbol{v}}_{s}^{\mathrm{patch}}=P_{\boldsymbol{\theta}}^{\mathrm{patch}}(\hat{\boldsymbol{v}})$
and
$\boldsymbol{u}_{t}^{\mathrm{patch}}=P_{\boldsymbol{\theta}^{\prime}}^{\mathrm{patch}}(\boldsymbol{u})$,
$\boldsymbol{v}_{t}^{\mathrm{patch}}=P_{\boldsymbol{\theta}^{\prime}}^{\mathrm{patch}}(\boldsymbol{v})$,
respectively. Similarly, the self-distillation between in-view patch tokens
can be formulated as a symmetric cross-entropy loss as (missing) 2.

$\displaystyle\mathcal{L}_{\mathrm{MIM}}=$
$\displaystyle=\frac{1}{2}\left(-\sum_{i=1}^{N}m_{i}\cdot
P_{\boldsymbol{\theta}^{\prime}}^{\mathrm{patch}}\left(\boldsymbol{u}_{i}\right)^{\mathrm{T}}\log
P_{\boldsymbol{\theta}}^{\mathrm{patch}}\left(\hat{\boldsymbol{u}}_{i}\right)\right)+$
(2) $\displaystyle\frac{1}{2}\left(-\sum_{i=1}^{N}m_{i}\cdot
P_{\boldsymbol{\theta}^{\prime}}^{\mathrm{patch}}\left(\boldsymbol{v}_{i}\right)^{\mathrm{T}}\log
P_{\boldsymbol{\theta}}^{\mathrm{patch}}\left(\hat{\boldsymbol{v}}_{i}\right)\right)$

The overall loss function is expressed as a combination of two individual loss
functions, $\mathcal{L}_{[\mathrm{CLS}]}$ and $\mathcal{L}_{\mathrm{MIM}}$, as
presented in (missing) 3. The loss function $\mathcal{L}_{[\mathrm{CLS}]}$
involves self-distillation between cross-view $[\mathrm{CLS}]$ tokens, whereas
$\mathcal{L}_{\mathrm{MIM}}$ is related to self-distillation among in-view
patch tokens. The proposed approach leverages the collective information
obtained from both loss functions to effectively learn the deepfake image
representations.

$\mathcal{L}=\mathcal{L}_{[\mathrm{CLS}]}+\mathcal{L}_{\mathrm{MIM}}$ (3)

###  III-C Graph Transformer Classifier

The methodology employed in this study involves the use of a vision
Transformer as a feature extractor, which has been trained using a self-
supervised contrastive learning approach, as previously mentioned in section
III-B. This approach is effective in computing the visual representation of
each node in the graph. An image representation matrix
$\mathcal{F}\in\mathbb{R}^{N\times D}$ is constructed by subjecting all $N$
nodes within the image to the aforementioned pre-trained ViT feature
extractor. Specifically, each node $v_{i}$ is associated with a feature vector
$f_{i}\in\mathbb{R}^{D}$, where $D$ denotes the dimensionality of the feature
vector. The resultant matrix is defined as
$F=\left\\{f_{1},f_{2},\dots,f_{N}\right\\}$. It is noteworthy that the
combination of $F$ with the adjacency matrix $\mathcal{A}$ facilitates the
construction of a graph representation of deepfake images. The resulting
graph-structured data can be analyzed using a multi-layer graph convolutional
network $f(\mathcal{F},\mathcal{A})$ [53], which utilizes a layer-wise
propagation rule as specified in (missing) 4.

$\begin{array}[]{l}H_{l+1}=ReLU\left(\hat{A}H_{l}W_{l}\right),\quad
l=1,2,..,L\\\
\hat{A}=\tilde{D}^{-\frac{1}{2}}\tilde{A}\tilde{D}^{-\frac{1}{2}}\end{array}$
(4)

here, $L$ represents the total number of graph convolution layers, and the
activation matrix of the $l^{th}$ graph convolutional layer is denoted as
$H_{l}$. The activation matrix $H_{l}$ captures the learned representations at
each layer, with the initialization of $H_{0}$ as the input feature matrix
$F$. The weight matrix $W_{l}$ is specific to each layer and is learned during
training. Moreover, the graph’s adjacency matrix with added self-connections
to each node is represented by $\tilde{A}$ and is defined as the sum of the
adjacency matrix $A$ and the identity matrix $I$. A diagonal matrix
$\tilde{D}$ is used to normalize the adjacency matrix, where $\tilde{D}_{ii}$
is defined as the sum of the entries in row $i$ of the normalized adjacency
matrix $\tilde{A}$. The resulting matrix $\hat{A}$ is the symmetric normalized
adjacency matrix of $A$.

Although graph convolutional layers have been successful in learning node-
level features and have been widely used in graph neural networks, they have
limitations in learning hierarchical visual features that capture the global
context of the graph. In contrast, the attention mechanism in Transformer
models has shown remarkable success in natural language processing tasks by
capturing the long-range dependencies between tokens and their relative
importance for the final prediction [54]. This ability is particularly
important in graph-structured deepfake images, where the relationships between
nodes can be complex and far-reaching. Therefore, the present work proposes to
leverage attention mechanisms to analyze graph-structured data by treating the
feature nodes as tokens in a sequence, and adjacency matrices as positional
encodings to preserve their spatial relationship. By synergizing the attention
mechanism with graph convolutional layers, the proposed model proficiently
assimilates both localized and global features, and adeptly captures intricate
inter-nodal relationships within the graph. Notably, the transformation of
graph space data into Transformer space is enacted via a Transformer layer, as
specified in (missing) 5 to (missing) 8.

$\mathbf{z}_{0}=\left[x_{\mathrm{class}};h^{(1)};h^{(2)};\ldots;h^{(N)}\right],\quad
h^{(i)}\in H$ (5)
$\mathbf{z}_{\ell}^{\prime}=\operatorname{MSA}\left(\mathrm{LN}\left(\mathbf{z}_{\ell-1}\right)\right)+\mathbf{z}_{\ell-1},\quad\quad\ell=1\ldots
L$ (6)
$\mathbf{z}_{\ell}=\operatorname{MLP}\left(\operatorname{LN}\left(\mathbf{z}_{\ell}^{\prime}\right)\right)+\mathbf{z}_{\ell}^{\prime},\quad\ell=1\ldots
L$ (7) $\mathbf{y}=\mathrm{LN}\left(\mathbf{z}_{L}^{0}\right)$ (8)

Here, the model architecture entails a multihead self-attention mechanism
$\operatorname{MSA}$, featuring $k$ Self-Attention (SA) heads, as delineated
in Equation 10 and 9, respectively. Furthermore, $\operatorname{MLP}$
represents a Multilayer Perceptron, and Layer Norm is indicated by
$\mathrm{LN}$. Here, $L$ corresponds to the number of Transformer blocks, and
$\mathbf{y}$ signifies the class label [55]. The Transformer model utilizes
the graph feature embeddings, as outlined in Equation 4, to facilitate self-
attention operation $\text{SA}(Q,K,V)$, based on Transformer architecture as
presented in Equation 10.

$\displaystyle\text{ MSA }(Q,K,V)$ $\displaystyle=\text{ Concat
}\left(\operatorname{SA}_{1},\ldots,\text{ SA}_{\mathrm{k}}\right)W^{O}$ (9)
$\displaystyle\text{ where SA}_{i}$ $\displaystyle=\text{ SA
}\left(QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V}\right)$

$\text{ SA
}(Q,K,V)=\operatorname{softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V$ (10)

where, $N$ represents the number of patches, $D$ is the dimension of patch
embeddings, and $\text{SA}(Q,K,V)$ denotes the pairwise similarity of two
nodes based on their corresponding Query $(Q_{in})$, Key $(K)$, and Values
$(V)$. Notably, the projections are parameter matrices
$W_{i}^{Q}\in\mathbb{R}^{D\times d_{k}},W_{i}^{K}\in\mathbb{R}^{D\times
d_{k}},W_{i}^{V}\in\mathbb{R}^{D\times d_{v}}$ and
$W^{O}\in\mathbb{R}^{kd_{v}\times D}$ [54, 55].

Finally, it is imperative to acknowledge that the conventional self-attention
mechanism computes attention scores for all possible pairs of nodes, leading
to an exponential increase in memory and time complexity as the number of
nodes increases, with a complexity of $O(n^{2})$. Thus, a technique that
reduces the number of nodes while retaining local information is
indispensable. In this investigation, a learnable pooling layer, referred to
as the min-cut pooling layer [56], has been employed on the output of the
final graph convolutional layer to effectively reduce the number of nodes.

Figure 3:  Relevancy maps for the Transformer, $C$, are generated via the
integration of attention maps, gradients, and relevance scores, which are
propagated throughout the network. These maps are subsequently converted into
graph class activation maps through reverse pooling, as elucidated in
(missing) 11.

###  III-D Graph Transformer Relevancy Map

To pinpoint areas within a given image that are likely to have been
manipulated, and which exhibit a significant correlation with deepfake class
labels, the present study builds upon previous research in this field [57] and
presents a graph Transformer relevancy map. This innovative approach involves
computing the class activation maps from the output class and then propagating
them to the inner graph space through a propagation procedure. The current
study shows that the attention relevancy of the $\ell^{th}$ Transformer block
$\bar{\mathbf{A}}^{(\ell)}$ can be obtained by representing the attention map
of its Transformer block as $A^{\ell}$, which facilitates the calculation of
the layer relevance score $R^{n_{\ell}}$ and gradient $\nabla A^{(\ell)}$
values pertaining to class $t$. It should be noted that ${n_{\ell}}$
corresponds to the layer that aligns with the $softmax$ operation in the
$\ell^{th}$ Transformer block. Finally, as depicted in Figure 3 the
Transformer’s final relevancy maps are represented by a weighted attention
relevance, as described by (missing) 11.

$\displaystyle\bar{\mathbf{A}}^{(\ell)}$
$\displaystyle=\mathbb{E}_{h}\left(\nabla\mathbf{A}^{(\ell)}\odot
R^{\left(n_{l}\right)}\right)+I$ (11) $\displaystyle\mathbf{C_{t}}$
$\displaystyle=\bar{\mathbf{A}}^{(1)}\cdot\bar{\mathbf{A}}^{(2)}\cdot\ldots\cdot\bar{\mathbf{A}}^{(L)}$

Here, $\mathbb{E}_{h}$ signifies the mean value calculated across the
dimension of the self-attention heads of the Transformer model, whereas
$\odot$ denotes the Hadamard product. To prevent self-inhibition for each
node, the identity matrix $I$ is added. In the final step, the Transformer
relevance map $C_{t}$ is mapped to each node in the graph, based on the min-
cut dense learned assignments [56]. This min-cut approach ensures that the
most relevant nodes are assigned the highest relevance scores, thereby
facilitating more accurate identification of manipulated regions within an
image. The proposed methodology represents a significant advancement in
deepfake detection and image forensics, by incorporating a highly effective
graph-based approach with Transformer architectures that improves accuracy and
enhances practical utility.

##  IV Evaluation Settings

The present section endeavors to appraise the performance and efficacy of the
proposed methodology through a series of challenging experiments, and
subsequently compare the results against those achieved by current state-of-
the-art techniques.

###  IV-A Datasets

Training datasets. In accordance with recent developments in the field of
deepfake detection [35, 12, 13, 36, 17], this study endeavors to evaluate the
efficacy and performance of the proposed self-supervised graph Transformer
model against different real-world settings through a comprehensive set of
experiments. To this end, the model is trained and evaluated using several
well-established and challenging deepfake forensics datasets, including
FaceForensics++ [5], Celeb-DF (V2) [4], and WildDeepfake [6]. A concise
overview of these datasets is provided below.

FaceForensics++. The FaceForensics++ dataset encompasses a diverse range of
four distinct types of image forgeries, specifically Deepfakes [58], FaceSwap
[59], Face2Face [60], and NeuralTextures [61]. The FaceForensics++ dataset is
thoughtfully curated and available in three varying compression rates, namely
heavily compressed (LQ), slightly compressed (HQ), and uncompressed (Raw).
While the Raw dataset exhibits relatively facile identification of deepfakes,
the process becomes considerably more challenging with higher compression
rates. In this manuscript, the HQ version of the dataset is predominantly
utilized, unless specified otherwise. Celeb-DF (V2). The Celeb-DF (V2) dataset
represents a significant advancement in synthetic video generation due to its
superior face-swapping strategy that elevates its visual quality above
previous efforts. The dataset encompasses an extensive collection of 5639
videos, all of which exhibit a high level of visual quality, making it an
ideal resource for training and testing deepfake detection models.
WildDeepfake. The WildDeepfake is a collection of fabricated videos sourced
entirely from the Internet, making it a distinct and demanding real-world
dataset. Unlike other AI-generated deepfakes, such as Celeb-DF (V2) and
FaceForensics++, the WildDeepfake dataset incorporates multiple facial
expressions and encompasses a wider range of scenarios, with numerous
individuals appearing in each scene.

Testing datasets. In addition to the aforementioned datasets, the official
test set of several additional benchmarks are utilized to investigate the
generalizability of the proposed model. Specifically, these benchmarks are the
DeeperForensics [2], FaceShifter [62], and DeepFake Detection Challenge (DFDC)
[1] datasets. The DeeperForensics dataset contains over 11,000 synthetic
videos generated by leveraging a specific Variational Auto-Encoder (VAE) on
the real videos from FaceForensics++. Similarly, the FaceShifter dataset is a
collection of videos created by applying advanced face-swapping techniques to
the real videos from the same dataset. This presents a valuable opportunity to
assess the robustness and effectiveness of the proposed self-supervised graph
Transformer model in detecting deepfake videos across a wide range of
scenarios, including those that incorporate different sophisticated
manipulation techniques based on the same underlying source videos. In
addition, the DFDC dataset, comprising over 4,000 manipulated videos created
using various GAN-based and non-learned techniques from 1,000 real videos,
provides a complementary benchmark against which the proposed method can be
evaluated.

###  IV-B Implementation Specifics

Deepfake Encoder Training. The present study adopts an architecture based on
vision Transformers [55] as the fundamental building block of the feature
extractor $f$, which is trained through the utilization of the previously
mentioned contrastive learning approach expounded upon in §III-B.
Specifically, the feature extractor model undergoes pre-training and fine-
tuning on images of size $320\times 320$, with patch sizes of $20\times 20$
yielding a total of 256 patch tokens. The projection head $h$, which plays a
central role in feature extraction, is established by means of a multi-layer
perceptron comprising three layers. Importantly, to foster effective
information sharing between the $\mathrm{CLS}$ and the patch tokens, the
projection head parameters are shared, notably,
${{h}_{s}}^{[\mathrm{CLS}]}={{h}_{s}}^{\mathrm{patch}}$ and
${{h}_{t}}^{[\mathrm{CLS}]}={{h}_{t}}^{\mathrm{patch}}$. This enables the
feature extractor model to capitalize on both the global and local context
provided by the $\mathrm{CLS}$ and patch tokens, respectively. The training
regimen is executed for 800 epochs with a learning rate that exhibits a
linearly scaled progression in accordance with the batch size, as determined
by the expression: $\operatorname{lr}=5e^{-4}\times{batch_{size}}/256$.

The preeminent implementation specifications at hand entail the pre-training
of a self-supervised feature extractor on a conflation of training sets
procured from three distinct benchmarks explicated in §IV-A.

Upon the conclusion of training, the parameters of the feature extractor are
fixed and employed as a pre-trained feature extractor for the purpose of node
feature vector extraction, which in turn facilitates the representation of the
deepfake images as a graph. The graph Transformer classifier is comprised of a
graph convolutional layer, and three blocks of Transformer layers, each of
which comprises 8 self-attention heads, a dimensional parameter of D=256, and
an MLP size of 512 (Equation 10). This construction is indicative of the
sophistication that underlies the proposed approach.

The experimental evaluation of the proposed framework was carried out on a
Lambda Quad deep learning workstation machine equipped with 4 NVIDIA GeForce
GTX 1080 Ti Graphics Processing Units (GPUs), 64 GB DDR4 RAM, an Intel Core™
i7-6850K CPU, and running the Ubuntu 20.04.3 LTS operating system. This
machine configuration facilitated the efficient training and evaluation of the
models, thanks to its high-performance computing capabilities. The performance
evaluation of the proposed framework was conducted using accuracy and area
under the receiver operating characteristic curve (AUC) metrics in different
experimental settings. Comprehensive assessment of the proposed framework’s
performance was enabled through the use of these stringent evaluation metrics,
which were employed to compare it with state-of-the-art methods and to provide
valuable insights into the practical applicability of the approach.

##  V Results & Discussion

TABLE I: Performance of the proposed self-supervised graph Transformer deepfake detection model using the in-dataset setting. Dataset | ACC (%) | AUC (%)
---|---|---
FaceForensics++ | (Raw) | 99.38 | 99.96
(HQ) | 98.41 | 99.34
(LQ) | 94.59 | 95.16
Celeb-DF (V2) | 99.47 | 99.43
WildDeepfake | 81.37 | 81.24

To fully ascertain the proficiency of the presented deepfake detection
framework, an exhaustive set of experiments was conducted, with a multifaceted
approach to examine its performance across various perspectives. The scope of
investigation encompassed the generalization ability of the framework in
detecting deepfakes originating from heterogeneous datasets and manipulation
techniques, as well as its resistance against compression and perturbations,
and other ablation studies that are critical in appraising the framework’s
efficacy. To this end, three distinct models were meticulously trained and
evaluated in an in-dataset setting using each of the preeminent deepfake
forensics datasets, namely FaceForensics++, Celeb-DF (V2), and WildDeepfake,
with the corresponding accuracy rate and AUC scores reported in Table I.

TABLE II:  Cross-dataset generalization AUC (%) scores. The model is trained over FaceForensics++ and evaluated over a test set of DeeperForensics (DFo), FaceShifter (FSh), DeepFake Detection Challenge (DFDC), and Celeb-DF (CDF), respectively.  Methods | DFo | FSh | DFDC | CDF | Avg.
---|---|---|---|---|---
Xception [5] | 84.5 | 72.0 | 70.9 | 73.7 | 75.3
CNN-aug [20] | 74.4 | 65.7 | 72.1 | 75.6 | 72.0
Patch-based [19] | 81.8 | 57.8 | 65.6 | 69.6 | 68.7
Face X-ray [12] | 86.8 | 92.8 | 65.5 | 79.5 | 81.2
CNN-GRU [30] | 74.1 | 80.8 | 68.9 | 69.8 | 73.4
Multi-task [63] | 77.7 | 66.0 | 68.1 | 75.7 | 71.9
DSP-FWA [64] | 50.2 | 65.5 | 67.3 | 69.5 | 63.1
LipForensics [11] | 97.6 | 97.1 | 73.5 | 82.4 | 87.7
FTCN [65] | 98.8 | 98.8 | 74.0 | 86.9 | 89.6
DFDT [17] | 96.9 | 97.8 | 76.1 | 88.3 | 89.7
RealForensics [66] | 99.3 | 99.7 | 75.9 | 86.9 | 90.5
Ours | 98.9 | 99.1 | 77.3 | 87.9 | 90.8

###  V-A Cross Dataset Generalization

The proliferation of fake videos in the real world poses a formidable
challenge for the deployment of deepfake detection models, given the diverse
forgery techniques, heterogeneous source videos, and extensive post-processing
techniques. Therefore, it is crucial to assess the effectiveness of any
deployed deepfake detection model from a domain generalization perspective.
Although evaluating domain generalization is complicated due to significant
disparities between deepfake forensics datasets, this can be addressed through
cross-dataset evaluation. This study adopts a similar approach to the existing
literature by utilizing the training set of the FaceForensics++ dataset to
construct a deepfake detection model and then evaluate its performance on
other datasets. More specifically, the model is trained on four distinct types
of forgeries that make up the FaceForensics++ dataset. The model’s performance
is then evaluated on the test sets of four separate benchmark deepfake
detection datasets, including DeeperForensics [2], FaceShifter [62], Deepfake
Detection Challenge [1], and Celeb-DF (V2) [4]. This evaluation process aims
to determine the model’s ability to generalize across different forgery
techniques and datasets, thus assessing its domain generalization
capabilities. It is noteworthy that this evaluation setup is exceedingly
arduous since the original and forged videos in DFDC and Celeb-DF (V2) are
different from those in FaceForensics++ and were not seen during the training
process.

The results obtained from this cross-dataset evaluation are presented in Table
II. Based on the results shown in the table, it is evident that the proposed
graph Transformer model outperforms all other methods in terms of average AUC
scores, achieving an impressive score of 90.8%. The RealForensics method comes
in second place with an average AUC score of 90.5%, while other methods have
lower scores, ranging from 63.1% to 89.7%. It is worth noting that some
methods, such as Face X-ray and DSP-FWA, perform well on specific datasets but
have lower scores on others. This indicates that their performance may be
dataset-dependent and may not generalize well across different datasets. On
the other hand, the presented method in this study and some other methods,
such as FTCN, DFDT, and RealForensics, achieve consistently high scores across
all datasets, indicating that they are robust and generalize well across
different datasets. In particular, the presented method achieves the highest
AUC score on the DeeperForensics, FaceShifter, and Celeb-DF (V2) datasets and
the second-highest score on the DFDC dataset. It is pertinent to mention that
due to the fact that DeeperForensics and FaceShifter utilize the same source
videos as FaceForensics++, they tend to perform better than DFDC and Celeb-DF
(V2) in terms of deepfake detection. This is because the models have already
been trained on similar data, making it easier for them to generalize to these
datasets.

TABLE III: Cross-manipulation generalization results on FaceForensics++ dataset. The AUC scores (in percentage) are for each forgery class on the FaceForensics++ dataset after training on the other three manipulation types. Methods | Train on remaining sets | Avg.
---|---|---
DF | NT | F2F | FS
Xception [5] | 93.9 | 79.7 | 86.8 | 51.2 | 77.9
CNN-aug [20] | 87.5 | 67.8 | 80.1 | 56.3 | 72.9
Patch-based [19] | 94.0 | 84.8 | 87.3 | 60.5 | 81.7
Face X-ray [12] | 99.5 | 92.5 | 94.5 | 93.2 | 94.9
CNN-GRU [30] | 97.6 | 86.6 | 85.8 | 47.6 | 79.4
LipForensics [11] | 99.7 | 99.1 | 99.7 | 90.1 | 97.1
DFDT[17] | 99.8 | 99.2 | 99.6 | 93.1 | 97.9
AV DFD [67] | 100.0 | 98.3 | 99.8 | 90.5 | 97.1
FTCN [65] | 99.9 | 99.2 | 99.7 | 99.9 | 99.6
RealForensics [66] | 100.0 | 99.2 | 99.7 | 97.1 | 99.6
Ours | 99.9 | 99.2 | 99.8 | 98.3 | 99.3

###  V-B Cross Manipulation Generalization

In the real world, deepfakes can be generated using various techniques,
including different source videos and post-processing methods, resulting in a
vast array of manipulated videos. A deepfake detection model that is only
trained on a specific dataset may fail to detect deepfakes generated using
different techniques, rendering the model ineffective. Therefore, any deployed
deepfake detection algorithm must perform well on unseen forgery types [11,
12, 63, 65]. This means that cross-manipulation generalization is critical for
deepfake detection as it enables the detection model to perform effectively on
unseen and potentially more sophisticated manipulated videos. To this end, the
cross-manipulation generalization property of the presented approach is
investigated using the leave-one-out strategy [11, 17, 14, 19, 63].
Specifically, model performance is evaluated on each forgery type in
FaceForensics++ dataset after training with the remaining manipulation
techniques. The obtained results on the test sets of Deepfakes (DF),
NeuralTextures (NT), Face2Face (F2F), and FaceSwap (FS) are shown in Table
III. The table shows that the proposed method achieves a high average AUC
score of 99.3%, which is comparable to state-of-the-art methods, such as FTCN
and RealForensics. The high average AUC score indicates that the presented
method’s cross-manipulation extends well to previously unseen manipulation
types. It works on par with the SOTA with a self-supervised feature extractor,
indicating its effectiveness in capturing more generalizable features for
accurate deepfake detection.

TABLE IV: Generalization AUC (%) scores across different compression levels. The model is trained over Neuraltextures (NT) forgery type and tested on Deepfakes (DF) and FaceSwap (FS) forgery types from the FaceForensics++ dataset at two compression levels, i.e., LQ and HQ.  Method | HQ | LQ
---|---|---
FS | DF | FS | DF
Xception [5] | 71.8 | 77.0 | 51.7 | 58.7
Face X-ray [12] | 77.9 | 58.5 | 51.0 | 57.1
F3Net [15] | 61.2 | 80.5 | 51.9 | 58.3
RFM [37] | 63.9 | 79.8 | 51.6 | 55.8
SRM [36] | 79.5 | 83.8 | 52.9 | 55.5
SLADD [68] | 72.1 | 84.6 | 56.8 | 62.8
Ours | 72.9 | 84.5 | 57.2 | 61.4
Figure 4: Robustness of the model against different compression levels. Models are trained over the FaceForensics++ dataset with a compression level of 23 and evaluated against five different compression levels.  TABLE V: Robustness against common post-processing perturbations. The experiments are performed using five different intensity levels for each perturbation type and reported the average AUC score (%). Moreover, the average AUC score across all corruptions for each deepfake detection method is presented.  Method | Clean | Saturation | Blur | Block | Contrast | Pixel | Noise | Avg
---|---|---|---|---|---|---|---|---
Xception [5] | 99.8 | 99.3 | 60.2 | 99.7 | 98.6 | 74.2 | 53.8 | 78.3
CNN-aug [20] | 99.8 | 99.3 | 76.5 | 95.2 | 99.1 | 91.2 | 54.7 | 84.1
Patch-based [19] | 99.9 | 84.3 | 54.4 | 99.2 | 74.2 | 56.7 | 50.0 | 67.5
Face X-ray [12] | 99.8 | 97.6 | 63.8 | 99.1 | 88.5 | 88.6 | 49.8 | 77.5
CNN-GRU [30] | 99.9 | 99.0 | 71.5 | 97.9 | 98.8 | 86.5 | 47.9 | 82.3
LipForensics [11] | 99.9 | 99.9 | 96.1 | 87.4 | 99.6 | 95.6 | 73.8 | 92.5
FTCN [65] | 99.4 | 99.4 | 95.8 | 97.1 | 96.7 | 98.2 | 53.1 | 89.5
RealForensics [66] | 99.8 | 99.8 | 95.3 | 98.9 | 99.6 | 98.4 | 79.7 | 95.6
Ours | 99.7 | 99.8 | 96.1 | 98.8 | 99.7 | 98.2 | 81.1 | 96.2

###  V-C Cross Compression Generalization

In addition to achieving high generalization scores across different types of
deepfake manipulations, ensuring the robustness of the deployed detector
against varying levels of compression is equally crucial. The proposed
method’s cross-compression generalization is evaluated through testing on
Deepfakes and FaceSwap forgery types with different compression levels, i.e.,
LQ and HQ, while being trained on NeuralTextures from the FaceForensics++
dataset. The results obtained are presented in Table IV. It can be observed
that while competitive generalization was demonstrated by the proposed graph
Transformer model when the compression level was kept constant during training
and testing. However, a significant drop in performance was noted once tested
against compressed samples, as evidenced in Figure 4. This is unsurprising,
given that highly compressed images are subject to the loss of textural
features and low-level clues. Therefore, FTCN and LipForensics were highly
affected by compression levels. However, the proposed self-supervised graph
transformer model achieves slightly better performances as it leverages high-
level facial representations, which are less susceptible to low-level
artifacts.

###  V-D Resilience to Perturbation

Given that real-world forgeries are vulnerable to various forms of corruption
on social media, such as saturation, contrast, blur, etc., it is essential
that the deployed deepfake detector can withstand such post-processing
perturbations. The procedure proposed in [11] is taken into account to
investigate the resilience of the model against common post-processing
filters, including applying Gaussian noise & blur, modifying saturation &
contrast, pixelation, and block-wise occlusions [2]. The model is trained over
grayscale clips of the FaceForensics++ dataset, with only horizontal flipping
and random cropping augmentations, and evaluated against five different
intensity levels of aforementioned perturbations. The resulting average AUC
scores are listed in Table V. The presented graph Transformer model is
significantly less vulnerable to aforementioned corruptions compared to other
methods that utilize low-level cues such as Patch-based and Face X-ray.
Furthermore, the proposed method outperforms FTCN and LipForensics, while
exhibiting comparable results to RealForensics, thus showcasing its
competitive performance.

TABLE VI: A quantitative comparison of methods performance on every dataset with existing deepfake detection approaches in frame-level analysis. Reported results are obtained from associated articles. The same evaluation metric as the literature is used for each dataset to provide a fair comparison and better insight into the model’s performance. Methods | FaceForensics++ | Methods | Celeb-DF (V2) | Methods | WideDeepfake
---|---|---|---|---|---
AUC (%) | AUC (%) | (ACC %)
Two-stream [69] | 70.1 | Two-stream [69] | 53.8 | AlexNet [70] | 60.3
Meso4 [71] | 84.7 | Meso4 [71] | 54.8 | VGG16 [72] | 60.9
HeadPose [73] | 47.3 | HeadPose [73] | 54.6 | ResNetV2-50 [74] | 63.9
FWA [64] | 80.1 | FWA [64] | 56.9 | ResNetV2-101 [74] | 58.7
VA-MLP [25] | 66.4 | VA-MLP [25] | 55.0 | ResNetV2-152 [74] | 59.3
Xception [5] | 99.7 | Xception [5] | 65.5 | Inception-v2 [75] | 62.1
Multi-task [63] | 76.3 | Multi-task [63] | 54.3 | MesoNet-1 [71] | 60.5
Capsule [76] | 96.6 | Capsule [76] | 57.5 | MesoNet-4 [71] | 64.4
DSP-FWA [64] | 93.0 | DSP-FWA [64] | 64.6 | MesoNet-inception [71] | 66.0
Two-branch [14] | 93.2 | Two-branch [14] | 73.4 | XceptionNet [77] | 69.2
SPSL [13] | 96.9 | Face X-ray [12] | 80.5 | ADDNet-2D [6] | 76.2
F3-Net [15] | 97.9 | SPSL [13] | 76.8 | ADDNet-3D [6] | 65.5
Video SE [78] | 99.6 | F3-Net [15] | 65.1 | ADD-Xception [8] | 79.2
RNN [79] | 83.1 | PPA [80] | 83.1 | DFDT [17] | 81.3
DFDT [17] | 99.7 | DefakeHop [7] | 90.5 |  |
|  | FakeCatcher [81] | 91.5 |  |
|  | ATS-DE [9] | 97.8 |  |
|  | ADD-ResNet [8] | 98.3 |  |
|  | DFDT [17] | 99.2 |  |
Ours | 99.9 | Ours | 99.4 | Ours | 81.3

###  V-E Comparison with SOTA

A comprehensive evaluation was carried out to assess the superiority of the
presented deepfake detection framework over state-of-the-art models. In
particular, the model’s performance was compared against state-of-the-art
deepfake detection models that were trained and tested using same datasets,
and the resultant outcomes are presented in Table VI. Remarkably, the model
exhibited competitive performance when evaluated on the FaceForensics++
dataset, while outperforming other models when tested on Celeb-DF (V2) and
WildDeepfake datasets. The obtained results unequivocally highlight the
exceptional potency of the proposed graph Transformer model, underscored by
its superlative generalizability attributable to the high-level deepfake
representations obtained by a self-supervised framework. Furthermore, the
proposed model’s ability to leverage the underlying graph structure of the
deepfake images enables the identification of both local and global subtle
discrepancies and artifacts that may evade detection using traditional
methods. This highlights the significance of the proposed model, which offers
a compelling alternative to existing deepfake detection methods and has the
potential to significantly enhance the reliability and robustness of deepfake
detection systems.

###  V-F Ablation Studies and Model Configuration

The proposed framework was comprehensively evaluated through a series of
meticulous ablation studies, aiming to investigate the role of different
components of the model and its hyperparameter values. The outcomes of these
experiments on Celeb-DF (V2) are summarized in Table VII. Firstly, the impact
of various feature extractors on the overall performance of the classification
task was assessed. The combination of visual representations obtained from the
proposed constructive learning vision Transformer architecture with the graph
Transformer classifier exhibited superior performance compared to utilizing a
pre-trained feature extractor, specifically ResNet50, in combination with the
same graph Transformer classifier.

A series of experiments were conducted using both sets of feature extractors
to investigate different hyperparameters’ impacts. For example, the influence
of the number of included neighboring patches, denoted as $K$, on the graph
construction process and the overall efficacy of the deepfake detection task
is investigated. Table VII reveals that the best performance was achieved when
$K=8$. Smaller values of $K$ yield a reduction in the receptive field of the
constructed graph, consequently resulting in lower performance. Lastly, the
impact of other critical hyperparameters, such as the number of graph
convolutional layers and the number of Transformer blocks, on the overall
performance of the deepfake detection pipeline is investigated. The obtained
results demonstrated a slight degradation in performance when the number of
graph convolution layers decreased from three layers to one layer. This
observation can be attributed to the concept of node receptive field, where a
greater number of graph convolution layers allows for the incorporation of
long-term information during model development. Moreover, it was observed that
utilizing three Transformer blocks facilitated more effective pooling of
discriminative information, leading to improved performance in deepfake
detection tasks.

The conducted ablation studies collectively showcased the superior performance
of the proposed self-supervised graph Transformer framework for deepfake
detection. These findings contribute to the advancement of the field and
emphasize the efficacy of the employed methodology for addressing the
challenges associated with deepfake detection.

TABLE VII: Performance evaluation of the proposed self-supervised graph Transformer deepfake detection model using different ablation studies on Celeb-DF (V2) dataset. Key variables include the feature extractor (FE), neighboring patches count ($K$), graph convolutional layers (GCN), and number of Transformer blocks (TB). FE | K | GCN | TB | AUC (%)
---|---|---|---|---
ResNet50 | 4 | 1 | 3 | 97.7
6 | 98.1
3 | 3 | 97.7
6 | 97.8
8 | 1 | 3 | 98.4
6 | 98.3
3 | 3 | 98.5
6 | 98.4
SSL ViT | 4 | 1 | 3 | 98.6
6 | 98.6
3 | 3 | 99.1
6 | 98.9
8 | 1 | 3 | 98.7
6 | 98.6
3 | 3 | 99.4
6 | 99.1

##  VI Conclusion

This work presents a self-supervised graph Transformer framework for deepfake
detection that is comprised of three main components. The proposed framework
leverages the expressive power of self-supervised contrastive learning to
learn high-level representations of deepfakes. Unlike existing supervised
approaches, which depend on low-level visual fingerprints, the extracted high-
level visual representations via this approach significantly enhance the
generalizability of deepfake detection models, thereby reducing their
dependence on dataset-specific patterns. Furthermore, the current work
leverages the expressive power of graph convolutional networks and Transformer
blocks to capture intricate interdependencies among both local and global
regions of an image, rendering them particularly suited to deepfake detection
tasks. In addition, the proposed graph Transformer relevancy map provides a
comprehensive understanding of the complex interdependencies and salient
features which leads to enhanced transparency and accountability of the
detection model. Finally, the efficacy and generalizability of the proposed
framework were rigorously evaluated via a comprehensive set of experiments,
encompassing a diverse range of challenging scenarios spanning both in-
distribution and out-of-distribution settings. The experimental results
unequivocally establish the framework’s superiority, with exceptional in-
dataset detection accuracy being achieved. Moreover, the proposed self-
supervised pre-training feature extractor constitutes a significant
contribution to the field, having markedly improved the framework’s ability to
generalize across multiple datasets while simultaneously enhancing its
resilience to post-processing perturbations, such as compression and blur.

## References

  * [1] B. Dolhansky, R. Howes, B. Pflaum, N. Baram, and C. C. Ferrer, “The deepfake detection challenge (dfdc) preview dataset,” _arXiv preprint arXiv:1910.08854_ , 2019.
  * [2] L. Jiang, R. Li, W. Wu, C. Qian, and C. C. Loy, “Deeperforensics-1.0: A large-scale dataset for real-world face forgery detection,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 2889–2898.
  * [3] L. Li, J. Bao, H. Yang, D. Chen, and F. Wen, “Advancing high fidelity identity swapping for forgery detection,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 5074–5083.
  * [4] Y. Li, P. Sun, H. Qi, and S. Lyu, “Celeb-DF: A Large-scale Challenging Dataset for DeepFake Forensics,” in _IEEE Conference on Computer Vision and Patten Recognition (CVPR)_ , Seattle, WA, United States, 2020.
  * [5] A. Rossler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies, and M. Nießner, “Faceforensics++: Learning to detect manipulated facial images,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 1–11.
  * [6] B. Zi, M. Chang, J. Chen, X. Ma, and Y.-G. Jiang, “Wilddeepfake: A challenging real-world dataset for deepfake detection,” in _Proceedings of the 28th ACM International Conference on Multimedia_ , 2020, pp. 2382–2390.
  * [7] H.-S. Chen, M. Rouhsedaghat, H. Ghani, S. Hu, S. You, and C.-C. J. Kuo, “Defakehop: A light-weight high-performance deepfake detector,” in _2021 IEEE International Conference on Multimedia and Expo (ICME)_. IEEE, 2021, pp. 1–6.
  * [8] A. Khormali and J.-S. Yuan, “Add: Attention-based deepfake detection approach,” _Big Data and Cognitive Computing_ , vol. 5, no. 4, p. 49, 2021\.
  * [9] V.-N. Tran, S.-H. Lee, H.-S. Le, and K.-R. Kwon, “High performance deepfake video detection on cnn-based with attention target-specific regions and manual distillation extraction,” _Applied Sciences_ , vol. 11, no. 16, p. 7678, 2021.
  * [10] S. Agarwal, H. Farid, O. Fried, and M. Agrawala, “Detecting deep-fake videos from phoneme-viseme mismatches,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops_ , 2020, pp. 660–661.
  * [11] A. Haliassos, K. Vougioukas, S. Petridis, and M. Pantic, “Lips don’t lie: A generalisable and robust approach to face forgery detection,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 5039–5049.
  * [12] L. Li, J. Bao, T. Zhang, H. Yang, D. Chen, F. Wen, and B. Guo, “Face x-ray for more general face forgery detection,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 5001–5010.
  * [13] H. Liu, X. Li, W. Zhou, Y. Chen, Y. He, H. Xue, W. Zhang, and N. Yu, “Spatial-phase shallow learning: rethinking face forgery detection in frequency domain,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 772–781.
  * [14] I. Masi, A. Killekar, R. M. Mascarenhas, S. P. Gurudatt, and W. AbdAlmageed, “Two-branch recurrent network for isolating deepfakes in videos,” in _European Conference on Computer Vision_. Springer, 2020, pp. 667–684.
  * [15] Y. Qian, G. Yin, L. Sheng, Z. Chen, and J. Shao, “Thinking in frequency: Face forgery detection by mining frequency-aware clues,” in _European conference on computer vision_. Springer, 2020, pp. 86–103.
  * [16] H. Zhao, W. Zhou, D. Chen, T. Wei, W. Zhang, and N. Yu, “Multi-attentional deepfake detection,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 2185–2194.
  * [17] A. Khormali and J.-S. Yuan, “Dfdt: An end-to-end deepfake detection framework using vision transformer,” _Applied Sciences_ , vol. 12, no. 6, p. 2953, 2022\.
  * [18] X. Zhu, H. Wang, H. Fei, Z. Lei, and S. Z. Li, “Face forgery detection by 3d decomposition,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 2929–2939.
  * [19] L. Chai, D. Bau, S.-N. Lim, and P. Isola, “What makes fake images detectable? understanding properties that generalize,” in _European Conference on Computer Vision_. Springer, 2020, pp. 103–120.
  * [20] S.-Y. Wang, O. Wang, R. Zhang, A. Owens, and A. A. Efros, “Cnn-generated images are surprisingly easy to spot… for now,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 8695–8704.
  * [21] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in _International conference on machine learning_. PMLR, 2020, pp. 1597–1607.
  * [22] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 9729–9738.
  * [23] M. Tschannen, J. Djolonga, M. Ritter, A. Mahendran, N. Houlsby, S. Gelly, and M. Lucic, “Self-supervised learning of video-induced visual invariances,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 13 806–13 815.
  * [24] J. Wang, Z. Wu, W. Ouyang, X. Han, J. Chen, Y.-G. Jiang, and S.-N. Li, “M2tr: Multi-modal multi-scale transformers for deepfake detection,” in _Proceedings of the 2022 International Conference on Multimedia Retrieval_ , 2022, pp. 615–623.
  * [25] F. Matern, C. Riess, and M. Stamminger, “Exploiting visual artifacts to expose deepfakes and face manipulations,” in _2019 IEEE Winter Applications of Computer Vision Workshops (WACVW)_. IEEE, 2019, pp. 83–92.
  * [26] T. Zhao, X. Xu, M. Xu, H. Ding, Y. Xiong, and W. Xia, “Learning self-consistency for deepfake detection,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 15 023–15 033.
  * [27] N. Bonettini, E. D. Cannas, S. Mandelli, L. Bondi, P. Bestagini, and S. Tubaro, “Video face manipulation detection through ensemble of cnns,” in _2020 25th international conference on pattern recognition (ICPR)_. IEEE, 2021, pp. 5012–5019.
  * [28] F. Marra, D. Gragnaniello, L. Verdoliva, and G. Poggi, “Do gans leave artificial fingerprints?” in _2019 IEEE conference on multimedia information processing and retrieval (MIPR)_. IEEE, 2019, pp. 506–511.
  * [29] H. Dang, F. Liu, J. Stehouwer, X. Liu, and A. K. Jain, “On the detection of digital face manipulation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern recognition_ , 2020, pp. 5781–5790.
  * [30] E. Sabir, J. Cheng, A. Jaiswal, W. AbdAlmageed, I. Masi, and P. Natarajan, “Recurrent convolutional strategies for face manipulation detection in videos,” _Interfaces (GUI)_ , vol. 3, no. 1, pp. 80–87, 2019.
  * [31] C. Zhao, C. Wang, G. Hu, H. Chen, C. Liu, and J. Tang, “Istvt: interpretable spatial-temporal video transformer for deepfake detection,” _IEEE Transactions on Information Forensics and Security_ , vol. 18, pp. 1335–1348, 2023\.
  * [32] Q. Yin, W. Lu, B. Li, and J. Huang, “Dynamic difference learning with spatio-temporal correlation for deepfake video detection,” _IEEE Transactions on Information Forensics and Security_ , 2023.
  * [33] R. Durall, M. Keuper, and J. Keuper, “Watch your up-convolution: Cnn based generative deep neural networks are failing to reproduce spectral distributions,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 7890–7899.
  * [34] J. Frank, T. Eisenhofer, L. Schönherr, A. Fischer, D. Kolossa, and T. Holz, “Leveraging frequency analysis for deep fake image recognition,” in _International conference on machine learning_. PMLR, 2020, pp. 3247–3258.
  * [35] J. Li, H. Xie, J. Li, Z. Wang, and Y. Zhang, “Frequency-aware discriminative feature learning supervised by single-center loss for face forgery detection,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 6458–6467.
  * [36] Y. Luo, Y. Zhang, J. Yan, and W. Liu, “Generalizing face forgery detection with high-frequency features,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 16 317–16 326.
  * [37] C. Wang and W. Deng, “Representative forgery mining for fake face detection,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 14 923–14 932.
  * [38] Z. Yang, J. Liang, Y. Xu, X.-Y. Zhang, and R. He, “Masked relation learning for deepfake detection,” _IEEE Transactions on Information Forensics and Security_ , vol. 18, pp. 1696–1708, 2023.
  * [39] P. Wang, K. Liu, W. Zhou, H. Zhou, H. Liu, W. Zhang, and N. Yu, “Adt: Anti-deepfake transformer,” in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 2899–1903.
  * [40] J. Wang, Y. Sun, and J. Tang, “Lisiam: Localization invariance siamese network for deepfake detection,” _IEEE Transactions on Information Forensics and Security_ , vol. 17, pp. 2425–2436, 2022.
  * [41] A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” _arXiv preprint arXiv:1807.03748_ , 2018\.
  * [42] P. O. O Pinheiro, A. Almahairi, R. Benmalek, F. Golemo, and A. C. Courville, “Unsupervised learning of dense visual representations,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 4489–4500, 2020.
  * [43] X. Wang, R. Zhang, C. Shen, T. Kong, and L. Li, “Dense contrastive learning for self-supervised visual pre-training,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 3024–3033.
  * [44] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar _et al._ , “Bootstrap your own latent-a new approach to self-supervised learning,” _Advances in neural information processing systems_ , vol. 33, pp. 21 271–21 284, 2020.
  * [45] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, “Unsupervised learning of visual features by contrasting cluster assignments,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 9912–9924, 2020.
  * [46] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision transformers,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 9650–9660.
  * [47] X. Chen and K. He, “Exploring simple siamese representation learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 15 750–15 758.
  * [48] X. Chen, S. Xie, and K. He, “An empirical study of training self-supervised vision transformers,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 9640–9649.
  * [49] S. Fung, X. Lu, C. Zhang, and C.-T. Li, “Deepfakeucl: Deepfake detection via unsupervised contrastive learning,” in _2021 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2021, pp. 1–8.
  * [50] H. Bao, L. Dong, S. Piao, and F. Wei, “Beit: Bert pre-training of image transformers,” in _International Conference on Learning Representations_ , 2022.
  * [51] H. Zhao, W. Zhou, D. Chen, W. Zhang, and N. Yu, “Self-supervised transformer for deepfake detection,” _arXiv preprint arXiv:2203.01265_ , 2022.
  * [52] J. Zhou, C. Wei, H. Wang, W. Shen, C. Xie, A. Yuille, and T. Kong, “Image bert pre-training with online tokenizer,” in _International Conference on Learning Representations_ , 2022.
  * [53] T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” in _International Conference on Learning Representations_ , 2017.
  * [54] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [55] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” in _International Conference on Learning Representations_ , 2021\.
  * [56] F. M. Bianchi, D. Grattarola, and C. Alippi, “Spectral clustering with graph neural networks for graph pooling,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 874–883.
  * [57] H. Chefer, S. Gur, and L. Wolf, “Transformer interpretability beyond attention visualization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 782–791.
  * [58] Faceswap, “Faceswap: Deepfakes software for all.” [Online]. Available: https://github.com/deepfakes/faceswap
  * [59] ——, “Faceswap.” [Online]. Available: https://github.com/MarekKowalski/FaceSwap/
  * [60] J. Thies, M. Zollhofer, M. Stamminger, C. Theobalt, and M. Nießner, “Face2face: Real-time face capture and reenactment of rgb videos,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2387–2395.
  * [61] J. Thies, M. Zollhöfer, and M. Nießner, “Deferred neural rendering: Image synthesis using neural textures,” _ACM Transactions on Graphics (TOG)_ , vol. 38, no. 4, pp. 1–12, 2019.
  * [62] L. Li, J. Bao, H. Yang, D. Chen, and F. Wen, “Faceshifter: Towards high fidelity and occlusion aware face swapping,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020.
  * [63] H. H. Nguyen, F. Fang, J. Yamagishi, and I. Echizen, “Multi-task learning for detecting and segmenting manipulated facial images and videos,” in _2019 IEEE 10th International Conference on Biometrics Theory, Applications and Systems (BTAS)_. IEEE, 2019, pp. 1–8.
  * [64] Y. Li and S. Lyu, “Exposing deepfake videos by detecting face warping artifacts,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2019, pp. 46–52.
  * [65] Y. Zheng, J. Bao, D. Chen, M. Zeng, and F. Wen, “Exploring temporal coherence for more general video face forgery detection,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 15 044–15 054.
  * [66] A. Haliassos, R. Mira, S. Petridis, and M. Pantic, “Leveraging real talking faces via self-supervision for robust forgery detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 14 950–14 962.
  * [67] Y. Zhou and S.-N. Lim, “Joint audio-visual deepfake detection,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 14 800–14 809.
  * [68] L. Chen, Y. Zhang, Y. Song, L. Liu, and J. Wang, “Self-supervised learning of adversarial example: Towards good generalizations for deepfake detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 18 710–18 719.
  * [69] P. Zhou, X. Han, V. I. Morariu, and L. S. Davis, “Two-stream neural networks for tampered face detection,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_. IEEE, 2017, pp. 1831–1839.
  * [70] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” _Communications of the ACM_ , vol. 60, no. 6, pp. 84–90, 2017.
  * [71] D. Afchar, V. Nozick, J. Yamagishi, and I. Echizen, “Mesonet: a compact facial video forgery detection network,” in _2018 IEEE International Workshop on Information Forensics and Security (WIFS)_. IEEE, 2018, pp. 1–7.
  * [72] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _In the proceedings of the International Conference on Learning Representation_ , 2015.
  * [73] X. Yang, Y. Li, and S. Lyu, “Exposing deep fakes using inconsistent head poses,” in _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2019, pp. 8261–8265.
  * [74] K. He, X. Zhang, S. Ren, and J. Sun, “Identity mappings in deep residual networks,” in _European conference on computer vision_. Springer, 2016, pp. 630–645.
  * [75] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2818–2826.
  * [76] H. H. Nguyen, J. Yamagishi, and I. Echizen, “Use of a capsule network to detect fake images and videos,” _arXiv preprint arXiv:1910.12467_ , 2019\.
  * [77] F. Chollet, “Xception: Deep learning with depthwise separable convolutions,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1251–1258.
  * [78] S. A. Khan and H. Dai, “Video transformer for deepfake detection with incremental learning,” in _Proceedings of the 29th ACM International Conference on Multimedia_ , 2021, pp. 1821–1828.
  * [79] D. Güera and E. J. Delp, “Deepfake video detection using recurrent neural networks,” in _2018 15th IEEE international conference on advanced video and signal based surveillance (AVSS)_. IEEE, 2018, pp. 1–6.
  * [80] P. Charitidis, G. Kordopatis-Zilos, S. Papadopoulos, and I. Kompatsiaris, “Investigating the impact of pre-processing and prediction aggregation on the deepfake detection task,” in _Proceedings of the Truth and Trust Conference_ , 2020.
  * [81] U. A. Ciftci, I. Demir, and L. Yin, “Fakecatcher: Detection of synthetic portrait videos using biological signals,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.