# A Comprehensive Survey on Segment Anything Model for Vision and Beyond

Chunhui Zhang, Li Liu∗,, Yawen Cui, Guanjie Huang, Weilin Lin, Yiqian Yang,
Yuehong Hu  Chunhui Zhang is with the Hong Kong University of Science and
Technology (Guangzhou), Guangzhou 511458, China and Cooperative Medianet
Innovation Center, Shanghai Jiao Tong University, Shanghai 200240, China and
also with the CloudWalk Technology Co., Ltd, 201203, China. Email:
<EMAIL_ADDRESS>Liu is with the Hong Kong University of Science
and Technology (Guangzhou), Guangzhou 511458, China. E-mail:
<EMAIL_ADDRESS>Cui is with the Hong Kong University of Science and
Technology (Guangzhou), Guangzhou 511458, China and also with the University
of Oulu. E-mail<EMAIL_ADDRESS>Huang is with the Hong Kong
University of Science and Technology (Guangzhou), Guangzhou 511458, China.
E-mail<EMAIL_ADDRESS>Lin is with the Hong Kong
University of Science and Technology (Guangzhou), Guangzhou 511458, China.
E-mail<EMAIL_ADDRESS>Yang is with the Northwestern Polytechnical
University, Xi’an 710072, China. E-mail<EMAIL_ADDRESS>Hu is with the Central South University, Changsha 410083, China. E-mail:
<EMAIL_ADDRESS>Corresponding author.This work was done at the Hong
Kong University of Science and Technology (Guangzhou).

###### Abstract

Artificial intelligence (AI) is evolving towards artificial general
intelligence, which refers to the ability of an AI system to perform a wide
range of tasks and exhibit a level of intelligence similar to that of a human
being. This is in contrast to narrow or specialized AI, which is designed to
perform specific tasks with a high degree of efficiency. Therefore, it is
urgent to design a general class of models, which we term foundation models,
trained on broad data that can be adapted to various downstream tasks. The
recently proposed segment anything model (SAM) has made significant progress
in breaking the boundaries of segmentation, greatly promoting the development
of foundation models for computer vision. To fully comprehend SAM, we conduct
a survey study. As the first to comprehensively review the progress of
segmenting anything task for vision and beyond based on the foundation model
of SAM, this work focuses on its applications to various tasks and data types
by discussing its historical development, recent progress, and profound impact
on broad applications. We first introduce the background and terminology for
foundation models including SAM, as well as state-of-the-art methods
contemporaneous with SAM that are significant for segmenting anything task.
Then, we analyze and summarize the advantages and limitations of SAM across
various image processing applications, including software scenes, real-world
scenes, and complex scenes. Importantly, many insights are drawn to guide
future research to develop more versatile foundation models and improve the
architecture of SAM. We also summarize massive other amazing applications of
SAM in vision and beyond. Finally, we maintain a continuously updated paper
list and an open-source project summary for foundation model SAM at here.

###### Index Terms:

Survey, Artificial General Intelligence, Foundation Models, Segment Anything,
Open Source Projects.

##  1 Introduction

Foundation models [1, 2, 3] have revolutionized artificial intelligence (AI)
in the past few years, thanks to their thorough pre-training on web-scale
datasets and powerful zero-shot generalization across a wide range of
downstream tasks. More recently, the natural language processing (NLP)
community has undergone a significant shift towards the development of large
language models (LLMs), resulting in a series of ground-breaking works, e.g.,
BERT [4], T5 [5], GPT-3 [6], and GPT-4 [7]. One of the most amazing
applications of these models is the ChatGPT [8], an AI chatbot developed by
OpenAI that leverages a large language model called GPT-3.5 to generate human-
like responses to user inputs.

Due to the great success of foundation models in NLP, researchers have been
inspired to explore large visual models (LVMs) in the computer vision (CV)
community. One line of research is to explore scaling vision transformers to a
huge size, pursuing the emergent capabilities exhibited in LLMs, such as ViT-G
[9], ViT-22B [10], Swin Transformer V2 [11], and VideoMAE V2 [12]. Besides, a
large body of works are devoted to adding knowledge of additional modalities
to enhance the capabilities of LVMs. Some notable examples include CLIP[13],
and ALIGN [14], which adopt a text encoder and an image encoder to learn
visual and language representations of image and text pairs from massive noisy
image-text data using contrastive learning [15]. After pre-training, the
learned semantic knowledge can be used to reference novel visual concepts on
new data distributions enabling the model with zero-shot transfer capability
in various downstream tasks, such as image-text retrieval [16, 17], and image
generation [18, 19].

Although the progress brings new impetus to the development of CV, the
generalization ability of obtained deep models remains limited [2, 20].
Recently, the CV community is witnessing a surge in exploring task-agnostic
foundation models [20, 21, 22, 23, 24, 25]. A common characteristic of these
models is to rely on a foundation model pre-trained on a broad dataset using a
task that can solve a wide range of downstream tasks using prompt learning
[26]. This new research trend of developing task-agnostic foundation models is
recently sparked by a model called segment anything model (SAM) [20] designed
for general image segmentation. SAM is a promptable model trained over 1
billion masks on 11 million images using a promptable segmentation task that
enables powerful zero-shot generalization. Many researchers, such as Jim Fan
[27], consider this as “ _the GPT-3 moment for CV, as SAM has learned the
general concept of what an object is, even for unknown objects, unfamiliar
scenes (e.g., underwater and cell microscopy) and ambiguous cases_ ” and
demonstrated great potential as a fundamental model for CV [22, 28, 29].

Recently, a large number of extended works have been proposed by the community
to explore the capability boundaries of SAM and apply it to various tasks,
e.g., medical image analysis [30, 31, 32, 33, 34, 35, 36, 37, 38], image
inpainting [39], image editing [40], style transfer [41], infrastructure
detection [42], camouflaged object detection [29], mirror and transparent
objects detection [43], image captioning [44], audio-visual localization [45],
video object tracking [46], 3D reconstruction [47], few-shot object counting
[48], and adversarial attacks [49, 50]. Concurrent to SAM, Wang et al. [23]
proposed a generalist model, namely SegGPT, to unify various segmentation
tasks into an in-context learning framework, which has demonstrated strong
zero-shot capabilities. Furthermore, Zou et al. [24] proposed a more general
segmentation system SEEM via introducing more diverse prompts than SAM,
including visual prompts (points, boxes, scribbles, masks), text prompts, and
referring prompts (referred regions of another image). As the authors claim
that the introduced unified prompt scheme in SEEM can encode different prompts
into the joint visual-semantic space to produce a strong zero-shot
generalization ability to address unseen user prompts for segmentation.
Additionally, some pioneering works explore general AI methods for
detecting/segmenting anything in the open-vocabulary scenarios, e.g.,
Grounding DINO [51], OVSeg [52], V3Det [53], and OpenSeg [54]. These
advancements have led many researchers to believe that versatile foundation
models are a critical step towards artificial general intelligence (AGI) [55,
56, 57, 58].

To this end, this work provides a comprehensive survey of these works with the
goal of helping researchers understand the latest developments related to SAM
models. This survey mainly focuses on various foundation models since SAM,
especially the applications of SAM to various tasks and data types. Readers
are referred to existing surveys [59, 60, 61, 62, 1, 2, 3, 63, 64, 65, 66, 67]
for language, vision, and multimodal foundation models. To the best of our
knowledge, this survey is the first to comprehensively review the recent
progress of segmenting anything task for vision and beyond based on the
foundation model of SAM. Concurrent to our work, [68, 33] briefly summarized
recent efforts to extend SAM to vision and medical image segmentation tasks,
however, we provide a more comprehensive review with many new insights from a
broader perspective. Furthermore, we maintain a continuously updated paper
list and a project summary to reflect the dynamic progress of the foundation
model of SAM during its development.

The remainder of this survey is organized as follows: Section 2 introduces the
background and terminology for foundation models including SAM, as well as
methods contemporaneous with SAM that are important for segmenting anything
task. Section 3 discusses the awesome works based on SAM for various image
processing applications, including software scenes, real-world scenes, and
complex scenes. Section 4 further discusses the follow-up works of SAM that
extend SAM to vision-related applications, beyond vision, and in more
directions. Finally, we conclude the survey in Section 5. This survey will be
regularly updated to reflect the dynamic progress of the foundation model of
SAM, as this is a rapid-evolving and promising field towards AGI.

##  2 Background and Terminology

###  2.1 Image Segmentation

####  2.1.1 Classic Segmentation

Image segmentation is a fundamental computer vision task that separates a
digital image into multiple parts by assigning each pixel to a class or
object. Traditionally, the segmentation includes three major tasks: semantic,
instance, and panoptic. Semantic segmentation [69, 70, 71, 72] assigns each
pixel to a predefined semantic class label. Instance segmentation [73, 74, 75]
further separate instances of the same class. Panoptic segmentation proposed
by [76] combines semantic and instance segmentation to understand scenes
comprehensively. Researchers have fully explored the above tasks in past
studies. Due to the operation consistency of the above tasks at the pixel
level, many studies have tried to use a unified framework to provide solutions
for three segmentation methods simultaneously, such as K-net [77],
MaskFormer[78], and Mask2Former[79].

####  2.1.2 Interactive Segmentation

Interactive segmentation [80] is a particular segment task with the character
of leveraging information from the guidance of user interaction. Despite being
a longstanding challenge, the problem has seen considerable improvement.
Usually, the user provides some initial input, such as points, strokes, or
bounding boxes, to indicate the rough location and shape of the object. Then,
the algorithm iteratively refines the segmentation based on the user feedback,
such as correcting mislabeled regions or adding missing parts. Interactive
segmentation is useful for many applications that require precise object
extraction, such as medical image analysis [81], [82], [83], photo editing
[84], and data annotation [85], [86].

Figure 1: Overview of the SA project, including task, model, and data. The
figure is borrowed from the original paper [20]. Figure 2: Overall structure
of SAM from the original paper [20].

###  2.2 Foundation Models

Foundation models are a new paradigm for building artificial intelligence
systems that can be adapted to various downstream tasks. They are based on
training large neural networks on massive amounts of data, often using self-
supervised learning techniques. This allows them to learn general
representations and capabilities that can be transferred to different domains
and applications. The term was coined by the Stanford Center for Research on
Foundation Models (CRFM) in 2021 to capture the significance and challenges of
this paradigm [1].

The development of foundation models can be traced back to the rise of deep
learning and self-supervised learning in the NLP field, which enabled learning
powerful representations from raw text data. Early examples of foundation
models were pre-trained LLMs, such as BERT [4], T5 [5], and GPT-n series [6],
[8], [7], which demonstrated impressive capabilities and performance on a wide
range of NLP tasks.

In CV research, current foundation models try to take advantage of LLMs which
are trained on large-scale data and show superb performance in learning
universal visual representations from diverse, large-scale image-text data.
Representatives include CLIP [13], ALIGN [14], Florence [87], VLBERT [88],
X-LXMERT [89], and DALL-E [19] try to capture the cross-modal interactions
between vision and language. They can be transferred or direct act on
classification, retrieval, object detection, video understanding, visual
question-answering, image captioning, and image generation tasks. Recently,
ImageBind [90] attempted to align six different modal information around
image/video information and learn the unified embedding space, opening up
further research on multimodal foundation models. Foundation models for
computer vision and multimodal learning are still an active area of research,
with many challenges and opportunities for improving their performance,
robustness, interpretability, and social impact.

###  2.3 Segment Anything Model

SAM comes from the Segment Anything (SA) project of Meta in 2023 [20]. By
finding foundation models that appeared in the NLP and CV fields show strong
performance, researchers tried to build a similar model which can unify the
whole image segmentation task. However, the available data in the segmentation
field is insufficient and differs from their design purpose. Therefore, as
shown in Fig. 1, they divide the pathway into three steps, namely Task, Model,
and Data. Correspondingly, a project for segmentation tasks is proposed,
including the promptable segmentation task (prompts include providing a
location, a range, a mask, or a text description of the segmentation target),
the SAM that can accept multiple prompt inputs and realize interactive use and
the Dataset SA-1B formed using the data engine of the interactive train-
annotate loop process with over one billion masks.

####  2.3.1 Task

The ultimate goal of the SA project is to provide a model with a wide range of
functions that can be quickly adapted to many existing and new segmentation
tasks (such as performing edge detection, object proposal generation, instance
segmentation, and segmenting objects from free-form text), and can transfer
zero samples to new data distributions and tasks. Since many complex functions
can be realized through simple combinations of existing tools. For example, if
there is a bounding box detector for humans, human instance segmentation can
be solved by providing the box output of the detector as a prompt to the
model. Researchers take inspiration from LLMs to achieve this target, using
prompt engineering [6] to cover the pre-training and downstream tasks. More
specifically, The concept of interactive segmentation is introduced to form
the promptable task and realize the training of the model.

A unique characteristic of the promptable task is returning a valid
segmentation mask when given any segmentation prompt. A prompt can be anything
indicating what to segment. A valid segmentation mask means that even if the
input prompt would lead to ambiguity (such as an image of a human wearing a
T-shirt, the prompt point is on the T-shirt), it should be a reasonable mask
for at least one object (which returns the mask of the human or The masks of
the T-shirt are all reasonable).

####  2.3.2 Model

The structure of SAM is shown in 2. It mainly consists of three parts, a
powerful image encoder (MAE [91] pre-trained ViT [92]); a prompt encoder,
which is divided into sparse input (CLIP’s [13] text encoder is used as a
position encoder to process points, boxes, and text-form prompt) and dense
input (convolutions processes mask input); and a mask decoder (prompt-image
bidirectional Transformer decoder using self-attention and cross-attention).
In addition, when the input prompts are ambiguous, the network will rank the
three possible mask outputs based on confidence. The loss functions used in
training include focal loss [93] and dice loss [94].

####  2.3.3 Data

Since there is insufficient public data for training, researchers use the
training-annotation iterative process to form a Data Engine to achieve model
training and dataset construction simultaneously. The specific process can be
divided into three stages.

  1. 1.

Assisted-manual stage. Professional annotators use the interactive labeling
tool on the browser, combined with SAM for manual labeling. SAM first uses the
public dataset for training. As the data gradually increases, the size of the
image encoder of SAM also increases. At the end of this stage, 4.3M masks and
120k images were collected.

  2. 2.

Semi-automatic stage. To increase the mask’s diversity and improve the model’s
performance, the researchers first pre-filled the mask with which the model
can make high-confidence predictions. Then they asked the annotator to
annotate the unfilled part interactively. At the end of this stage, an image
can provide an average of 72 masks.

  3. 3.

Fully automatic stage. In this stage, due to the collection of enough masks
and the introduction of the ambiguity-aware model, the final training of the
SAM and the acquisition of the SA-1B data set can be performed. The ambiguity-
aware model enables SAM to predict effective masks even when the prompt is
ambiguous. Specifically, the researchers use a 32x32 grid to obtain prompt
points on each image uniformly. If the prompt point is located on the target
part or sub-part structure, the model will return the mask of the sub-part,
part, or the whole object. And filter sort the outputs based on confidence. At
the end of this stage, the final SA-1B dataset contains 11M images and 1.1B
masks.

With the advantages of well-designed tasks, model structure, and massive high-
quality training data, experimental shows that the zero-sample transfer
capability of the SAM model has performed excellent results in single cue
point segmentation, edge detection, Object proposal, instance segmentation,
interactive segmentation, and multimodal segmentation (Text-to-Mask) tasks. It
even outperforms supervised models in some respects.

###  2.4 Concurrent Works

Parallel to SAM research, many efforts have been made to solve segmentation
tasks with other general methods.

OneFormer[25] leverages the task-conditioned joint training strategy, task
token, and query-text contrastive loss to form a universal image segmentation
framework. Oneformer enables training on all three traditional segmentation
tasks within a single universal model and a multi-task training process. With
different backbones, it outperforms specialized models on [95], Cityscapes
[96], and COCO datasets [97] with even costs much less training time and
resources.

Meanwhile, the SegGPT[23] stand for Segment everything with a Generalist
Painter, exploring in-context training and inference scheme. It forms a
generalist in-context learning framework [98] that unifies different
segmentation data formats. And treating the training process as an in-context
coloring problem with a random coloring scheme instead of using predefined
color space. This training process requires the model to focus on contextual
information to accomplish the specific task. Based on these improvements, the
model can perform arbitrary segmentation tasks based on input images or videos
through in-context inference.

Also, SEEM[24] further broadens the scope of task applicability of single
segmentation models. It further expands the types of supported prompts,
including points, boxes, scribbles, masks, texts, and referred regions of
another image. With the proposed joint visual-semantic space, the model has
compatibility to compose flexible multi-prompt input. SEEM also can process as
a classic segmentation model when no prompt provides. However, it also suffers
from limited training data and absent the support of part-based segmentation.

##  3 SAM for Image Processing

###  3.1 Software Scenes

####  3.1.1 Image Editing

Modern software scenes require operations on image editing and inpainting,
e.g., removing objects, filling objects, and replacing objects. However, the
existing inpainting works, like [99, 100, 101, 102], need fine annotations for
each mask to achieve good performance, which is labor-intensitive. SAM [20],
which can generate accurate masks with simple prompts such as points or boxes,
can help assist the image editing scenes.

Figure 3: Overall pipeline of Inpaint Anything (IA). The input image is
segmented by the SAM and the targeted segment is replaced by the output of the
inpaint models to achieve different tasks. The figure is borrowed from the
original paper [39].

Inpaint Anything (IA) [39] designs a pipeline to solve inpainting-related
problems by combining the advantages of SAM, the state-of-the-art (SOTA) image
inpainters [99], and AI-generated content (AIGC) models [103]. The pipeline is
illustrated in Fig. 3. For object removing, the pipeline is composed of SAM
and SOTA inpainters, like LaMa [99]. The clicking action from the user is used
as a prompt in SAM to generate a mask for the object area, and the LaMa fills
it with corrosion and dilation operations. For object filling and replacing,
the AIGC models, like Stable Diffusion (SD) [103], are used in the second step
to fill the selected object with newly generated objects by text prompts.

Figure 4: Overall pipeline of Edit Everything from the original paper [40].

A similar idea can also be seen on Edit Everything [40]. As shown in Fig. 4,
it allows users to edit images using simple text instructions. Specifically,
when inputting an image, SAM first separates it into several segments with no
prompts, followed by a source prompt instructing CLIP to rank the received
segments. Only the segment with the highest score is selected as the target
one to be replaced with the newly generated object from SD with the target
prompt. Compared with the object-replacing solution in IA, the authors train
the CLIP with 400 million parameters and the SD with 1 billion parameters in
Chinese scenarios to make it more reliable towards the Chinese text prompts.
Furthermore, the paper improves the realism of the image by breaking down the
complex prompts into smaller entities and replaced in a sequential manner.
Although it performs well as a novel tool, the paper points out that it still
needs specific enhancement in different scenarios.

####  3.1.2 Style Transfer

Figure 5: Illustrations of Any-to-Any Style Transfer from the original paper
[41].

Style transfer aims to transfer the style from a given image (style image) to
another given image (content image). Typically, the transferred style is
represented by the holistic style of the style image or the local colors and
textures of the style image, and only one result will be generated for the
content image, which lacks flexibility for the users to interact with it. With
promptable region selection ability from SAM, Any-to-Any Style Transfer [41]
enables users to specify which style region to select and which content
regions to apply during style transfer. The pipeline, which is shown in Fig.
5, is organized as follows:

  * 1)

Encode the style and content images with pre-trained VGG-19 as well as
calculate the content-style attention map.

  * 2)

Obtain the style and content masks with SAM and input prompts.

  * 3)

Fuse the attention map with the mask-controlling signals from the last step.

  * 4)

Compute the stylized feature with the updated attention map and derive the
final result.

With the defined pipeline, the paper proves that the proposed method is a
plug-and-play component to the existing style transfer methods, including
Local Transformation Based Style Transfer [104, 105], Global Transformation
Based Style Transfer [106], and Diffusion Based Style Transfer [107], which
shows its great potential of broad applications.

###  3.2 Real-World Scenes

####  3.2.1 Detection

SAM holds the ability to assist the applications in many real-world scenes
such as real-world object detection, object counting, and moving object
detection scenes. Recently, [108] evaluated the performance of SAM across a
diverse range of real-world segmentation scenarios, e.g., natural image,
agriculture, manufacture, remote sensing, and healthcare scenes. The paper
finds that it has excellent generalization on common scenes like natural
images, while it shows less effectiveness in low-contrast scenes and requires
strong prior knowledge in complex scenes.

Figure 6: Process of crack detection using SAM and U-Net. The figure is
borrowed from the original paper [42].

For example, in the application of civil infrastructure defect assessment,
[42] utilizes SAM to detect cracks in concrete structures and compares its
performance with the baseline U-Net [109]. The crack detection process is
illustrated in Fig. 6. The results showed that the SAM outperforms the U-Net
in the detection of longitudinal cracks, which are more likely to find similar
training images as in the normal scenes, while in the unusual scene, i.e.,
spalling cracks, SAM is not as good as U-Net.

Unlike the complex image cases in crack detection, crater detection is more
suitable to utilize SAM as the detection tool as the crater shapes focus on
circular or elliptical. Craters are one of the most important morphological
features in planetary exploration, detecting and counting them is an important
but time-consuming task in planetary science. Although the existing works in
machine learning and computer vision successfully solve some specific problems
in crater detection, they rely on specific types of data and thus fail to work
well in a different data source.

In [110], the authors propose a universal crater detection scheme with the
zero-shot generalization of SAM to unfamiliar objects. The pipeline uses SAM
to segment the input images, which has no restrictions on the data type and
resolutions. Then, it utilizes circular-elliptical indexes to filter the
segmentation masks that are not circular-elliptical shapes. Finally, a post-
processing filter is employed to get rid of duplicates, artifacts, and false
positive ones. The pipeline shows its great potential to be the general tool
in the current field and the authors also discuss the drawbacks that only the
specific shapes can be recognized.

####  3.2.2 Counting

Few-shot object counting is an important application scene of computer vision
in the real world, which counts an unseen-category object with only a few
bounding boxes of examples provided. Since SAM shows impressive performance
with great generalization for unseen objects, it shows potential to be used in
few-shot object counting. [48] is the first paper testing SAM in this task and
comparing it with other baseline few-shot counting methods. The goal is to
find out whether SAM can segment and distinguish the target objects using the
reference examples.

To realize this, the authors define a pipeline. Firstly, they calculate the
dense image feature with an image encoder, i.e., ViT-H. Secondly, they use
bounding boxes as prompts to generate segment masks for the reference
examples, which are then computed with the dense image feature as the feature
vectors of the reference objects. Thirdly, they use point grids as prompts to
segment everything and generate feature vectors for all masks. After that,
they compute the cosine similarity of the feature vectors between the
predicted masks and reference examples, only the ones larger than the pre-
defined threshold are considered the target objects. The paper compares the
proposed SAM-based method with other few-shot counting methods on two
datasets, FSC-147 [111] and MS-COCO [112], and finds that it falls behind the
SOTA baselines, especially for small and congested objects. Thus, further
improvement for SAM in some special scenes is still needed.

####  3.2.3 Moving Object

Moving object segmentation (MOS) is a crucial task for the application of
computer vision in many real-world application scenarios, such as autonomous
driving. The existing datasets for this research are mainly RGB or Lidar
videos, which lack the event information that can help understand the dynamic
scenes better. To fill this gap, DSEC-MOS [113] is proposed with the moving
vehicles’ frames and the corresponding event data, which can facilitate the
development of more accurate, robust, and efficient algorithms for autonomous
driving. The contribution of SAM to DSEC-MOS annotation is that it provides a
promptable segmentation way. The authors apply the moving object bounding
boxes in DSEC-MOD [114] as prompts to generate a large number of preparatory
masks with SAM, which is accurate and reliable. The dataset contains 16
sequences of 13,314 frames in total and provides event-based data with pixel-
level annotations, which can be a valuable resource for the MOS field.

Figure 7: Framework of the SAA+ [115], which achieves zero-shot anomaly
segmentation by introducting hybrid prompts as a regularization technique. The
figure is borrowed from the original paper [115].

###  3.3 Complex Scenes

####  3.3.1 Low-Contrast Scene

Except for the normal scenes mentioned above, whether SAM can solve the
segmentation issue in complex scenes, like low-contrast scenes, is also a
meaningful question to broaden its application. To find out the generalization
ability of SAM in more complex scenes, Ji et al. [22] quantitatively compare
it with cutting-edge models in three concealed scenes, i.e., camouflaged
animals, industrial defects, and medical lesions. They conduct the experiments
on three camouflaged object segmentation (COS) datasets, i.e., CAMO [116] with
250 samples, COD10K [117] with 2026 samples, and NC4K [118] with 4121 samples.
And compare it with the outstanding transformer-based model, CamoFormer-P/S
[119] and HitNet [120]. They observe from the results that SAM looks not
skillful in concealed scenes and point out that the potential solution may
rely on the support of prior knowledge in the specific fields. The same
conclusion also can be drawn in [29], where the authors compare SAM with 22
SOTA methods in camouflaged object detection on the same three datasets
mentioned above.

Dominic et al. [121] utilizes SAM in the field of plant phenotyping, e.g.,
potato leaves segmentation, by proposing a method named Leaf Only SAM.
Specifically, they combine SAM with four post-processing steps to identify
only leaf objects without any training data. After receiving the segmentation
masks from SAM, they first check the color by finding the green masks. To
reduce ambiguity, they further check the main object by keeping the one with
IoU of more than 90% and removing the rest. Then, they compare the area of the
minimum enclosing circle to filter the masks with incorrect shapes. Finally,
they remove multi-leaf masks by summing all the mask objects in the image and
labeling each pixel to the amount of belonging masks. The mask with a mean
score over 1.5 is assumed to be a duplicate. The experiments compare Leaf Only
SAM with a fine-tuned Mask R-CNN and find that Leaf Only SAM does not
outperform the latter with just a 10% performance drop, but its training-free
ability shows its potential for this field.

Cao et al. [115] proposes a new framework called Segment Any Anomaly + (SAA+)
for zero-shot anomaly segmentation as shown in Fig. 7. The framework utilizes
hybrid prompt regularization to improve the adaptability of modern foundation
models, allowing for more accurate anomaly segmentation without the need for
domain-specific fine-tuning. The authors conduct thorough experiments on four
anomaly segmentation benchmarks, i.e., VisA [122], MVTec-AD [123], MTD [124],
and KSDD2 [125], and achieve SOTA performance. He et al. [126] proposes the
first method (WS-SAM) to leverage SAM for weakly-supervised concealed object
segmentation that addresses the challenges of segmenting objects that are well
blended with their surroundings using sparsely-annotated data (see Fig. 8).
The proposed WS-SAM involves SAM-based pseudo labeling and multi-scale feature
grouping to improve model learning and distinguish concealed objects from the
background. The authors found that using only scribble supervision [127], SAM
can generate segmentation masks good enough to train a segmenter.

Figure 8: Framework of the WS-SAM [126] with scribble supervision for weakly-
supervised concealed object segmentation. The figure is borrowed from the
original paper [126].

The glass scene is a more challenging one in concealed scenes compared with
the camouflaged animals and others. It is closely related to the safety issue
when applying the machine learning algorithm in our real world. For example,
autonomous mobile robots may easily crash into a transparent front door
without a reliable, transparent object detection algorithm. In this scene, the
detected objects often have mirror surfaces or just be transparent, which
easily makes the traditional detection algorithms fail. As SAM has strong
capability in zero-shot segmentation to the unseen objects, [43] evaluates it
on both mirror and transparent objects scenes and compares it with SOTA
methods from semantic segmentation, shadow detection, saliency object
detection, and glass segmentation. The results from the experiments show that
SAM can successfully identify the objects behind the transparent ones but fail
to recognize the glass objects themselves. The overall performance of SAM is
significantly worse than the methods trained specifically with transparent
objects, which proves that SAM is not ready to be deployed in safety-critical
situations that contain glass.

####  3.3.2 Thermal Infrared Image

Figure 9: Framework of creating and using SATIR in three steps. (a) Construct
a pre-trained dataset with SAM. (b) Pretrain models with the dataset. (c)
Finetune the pre-trained models on the target mask. The figure is borrowed
from the original paper [128].

The thermal infrared image scene is another kind of complex scene where the
images are always dark and hard to be annotated. Therefore, a large amount of
unlabeled data is wasted, and the models for this field fail to learn high
accuracy in a reliable way. To solve this problem, [128] utilizes SAM to
generate pseudo labels and builds a large-scale thermal infrared segmentation
dataset, SATIR, for model pretraining, which contains over 100,000 images with
pixel-annotation labels. To finally improve the performance of models in this
field, the authors propose a three-step framework, which is shown in Fig. 9.
They construct the mentioned dataset with SAM, followed by pretraining the
model with it. Then, they finetune the pre-trained model on the target task.
The experiment on the public thermal infrared semantic segmentation data, SODA
[129], validates its effectiveness in this field, where the backbone model
pre-trained by SATIR outperforms others with approximately 1.3% in mean
Intersection over Union (mIoU).

In the field of the poultry industry, thermal infrared images are also used in
chicken segmentation tasks, which is an important agricultural application in
the context of cage-free hens. To find out SAM’s capacity in this field, the
paper [130] uses SAM in the chicken segmentation task and compares its
performance with the other SOTA baseline methods, i.e., SegFormer [131] and
SETR [132]. The authors find that SAM has superior performance compared with
the two baseline methods, especially when using the total points prompts.
However, with the thermal images as input, they find that there are still some
limitations in recognizing the arbitrary parts, e.g., the chicken tail. Except
for the chicken segmentation, they also explore the ability of SAM in chicken-
tracking tasks by combining SAM, YOLOX [133], and ByteTracker [134] as a
tracker. And they realize real-time tracking with the combination and show its
potential in this field.

####  3.3.3 Overhead Image

Overhead imagery problems are related to a set of widely-studied tasks, where
the objects are usually small and intensive, e.g., the remote sensing images.
In [135], the authors test whether the impressive generalization ability of
SAM can cover the scenes in this field. Specifically, they examine the out-of-
the-box accuracy of SAM on six existing benchmark datasets of overhead
imagery, including Solar [136], Inria [137], DeepGlobe [138], 38-Cloud [139],
DeepGlobe Roads [138], and Parcel Delineation [140], which encompass 5 million
square kilometers of surface area. The results demonstrate that SAM often
generalizes well to overhead imagery while failing in some cases with unique
characteristics on target objects, which is considered a systemic failure and
can be addressed through modest changes in SAM’s design. Similar observations
and ideas are proposed in [141]. The authors adopt SAM to the geological
mapping task with remote sensing images of the Mars surface and find that it
cannot be directly applied to domain-specific tasks due to a lack of problem-
specific bias. Thus, they change the SAM’s design by introducing a domain-
specific decoder, which can learn the problem-specific semantics by fine-
tuning through knowledge distillation with only five labeled images and
achieve the same performance as the Mask-RCNN [142] trained with over 400
labeled images.

The unlabeled remote sensing dataset issue can also be solved with SAM. With
the small and intensive characters on objects in remote sensing images, they
are hard and cost-inefficient to be labeled by human experts. Thus, the paper
[143] develops an efficient pipeline with SAM to generate a large-scale remote
sensing segmentation dataset named SAMRS. It will be detailed in the data
annotations section. The paper [141] mentioned above also explores the
applicability of their proposed SAMs decoder for annotation and finds that the
bounding box-based segmentation mode is more suitable for rapid annotation.
Apart from that, some researchers combine the advantages of different
foundation models [144], like SAM and Grounding DINO, to achieve text-prompt
guiding segmentation on remote sensing images and prove its effectiveness in
this field. The details are illustrated in the vision and language section.

##  4 Other Applications: Vision and Beyond

###  4.1 Vision Related Applications

\begin{overpic}[width=433.62pt]{figs/SAMforMedicalImages.pdf}
\put(10.0,10.14){\color[rgb]{0,0,0}{\cite[cite]{[\@@bibref{}{hu2023sam}{}{}]}}}
\put(16.8,11.15){\color[rgb]{0,0,0}{\cite[cite]{[\@@bibref{}{roy2023sam}{}{}]}}}
\put(24.4,10.7){\color[rgb]{0,0,0}{\cite[cite]{[\@@bibref{}{zhang2023customized}{}{}]}}}
\put(28.75,10.0){\color[rgb]{0,0,0}{\cite[cite]{[\@@bibref{}{mohapatra2023brain}{}{}]}}}
\put(37.5,11.25){\color[rgb]{0,0,0}{\cite[cite]{[\@@bibref{}{putz2023segment}{}{}]}}}
\put(43.3,9.95){\color[rgb]{0,0,0}{\cite[cite]{[\@@bibref{}{li2023polyp,zhou2023can}{}{}]}}}
\put(49.8,11.0){\color[rgb]{0,0,0}{\cite[cite]{[\@@bibref{}{ma2023segment,mazurowski2023segment,he2023accuracy,shi2023generalist,cheng2023sam}{}{}]}}}
\put(49.8,10.0){\color[rgb]{0,0,0}{\cite[cite]{[\@@bibref{}{mattjie2023exploring,huang2023segment,liu2023samm,wu2023medical,chen2023sam}{}{}]}}}
\put(49.8,9.0){\color[rgb]{0,0,0}{\cite[cite]{[\@@bibref{}{qiu2023learnable,junde2023PromptUNet}{}{}]}}}
\par\put(66.3,9.9){\color[rgb]{0,0,0}{\cite[cite]{[\@@bibref{}{deng2023segment}{}{}]}}}
\put(72.3,9.15){\color[rgb]{0,0,0}{\cite[cite]{[\@@bibref{}{hu2023skinsam}{}{}]}}}
\put(82.5,10.9){\color[rgb]{0,0,0}{\cite[cite]{[\@@bibref{}{zhang2023input}{}{}]}}}
\put(88.7,10.15){\color[rgb]{0,0,0}{\cite[cite]{[\@@bibref{}{wang2023sam}{}{}]}}}
\put(98.15,10.88){\color[rgb]{0,0,0}{\cite[cite]{[\@@bibref{}{wang2023gazesam}{}{}]}}}
\end{overpic} Figure 10: Outline of SAM for medical images, including
Computerized Tomography (CT) images, Magnetic Resonance Imaging (MRI) images,
colonoscopy images, multiple format images, H&E stained histological sections
images, and others.

####  4.1.1 Medical Imaging

The goal of medical image segmentation is to reveal the anatomical or
pathological structure of the corresponding tissues, which can assist
computer-aided diagnosis and intelligent clinical surgery [161, 162]. Due to
the rapid development of computation power and medical data resources, deep
learning-based medical image segmentation has achieved massive progress in
accuracy and speed against traditional counterparts [163, 164]. With the
emerging Visual Transformer (ViT), ViT-based medical image methods [165, 166,
167, 168] has achieved surpassing performance in the medical image
segmentation. However, such networks are towards a specific task, which lacks
the generalization ability on other tasks. Recently, SAM has been proposed to
make it possible to solve multiple kinds of segmentation tasks in a unified
framework. In this context, researchers have paid attention to customizing SAM
for medical image segmentation and concluded some useful strategies to improve
its performance. This work [33] briefly summarizes recent efforts to extend
the success of SAM to medical image segmentation tasks, while we provide a
more comprehensive summary with deeper insight in this section.

According to the imaging format of medical images, the usage of SAM in medical
image segmentation can be categorized into six series: Computerized Tomography
(CT) images, Magnetic Resonance Imaging (MRI) images, colonoscopy images, H&E
stained histological sections images, multiple format images, and others.

For CT images. A CT scan combines a series of X-ray images taken from
different angles around your body and uses computer processing to create
cross-sectional images (slices) of the bones, blood vessels, and soft tissues
inside your body. This paper [145] presents a preliminary examination of SAM
as an annotation tool for medical image analysis, specifically for the
segmentation of multi-phase liver tumors (MPLiTS). Their investigation focused
on the prompts used, data resolution, and phases. The experimental findings
demonstrate the effectiveness of SAM in this context, while also highlighting
areas where MPLiTS could be improved. To provide comprehensive guidance to the
MPLiTS community, the authors plan to conduct further investigations that
encompass a wider range of aspects. The purpose of this paper [146] is to
perform an initial assessment of SAM’s out-of-the-box zero-shot capabilities
for medical image segmentation by evaluating its performance on an abdominal
CT organ segmentation task using point or bounding box-based prompting. The
findings indicate that SAM can generalize effectively to CT data, which could
potentially accelerate the development of semi-automatic segmentation tools
for clinicians. SAMed [147] is the proposed solution for medical image
segmentation, which differs from previous methods in that it leverages the
SAM, a large-scale image segmentation model. This approach involves
customizing the SAM model for medical image segmentation by applying the low-
rank-based (LoRA) finetuning strategy to the SAM image encoder. Unlike SAM,
SAMed can perform better for semantic segmentation tasks on medical images.
The trained SAMed model achieves comparable performance to SOTA methods.
Additionally, since SAMed only updates a small fraction of the SAM parameters,
its deployment and storage costs are minimal in practical usage.

For MRI images. MRI is a non-invasive diagnostic imaging technique that uses a
powerful magnetic field, radio waves, and a computer to produce detailed
images of internal structures in the body. MRI is commonly used to visualize
the brain, spine, joints, and other soft tissues. This study [148] compares
SAM with FSL’s Brain Extraction Tool (BET), a widely used and current gold
standard brain extraction technique, on a variety of brain scans with varying
image qualities, MR sequences, and brain lesions affecting different brain
regions. The findings show that SAM outperforms BET based on average Dice
coefficient, IoU, and accuracy metrics, particularly in cases where image
quality is compromised by signal inhomogeneities, non-isotropic voxel
resolutions, or the presence of brain lesions near or involving the outer
regions of the brain and meninges. Furthermore, SAM has superior segmentation
properties, enabling a fine-grain separation of different tissue compartments
and brain structures. These results suggest that SAM has the potential to
become a more accurate, robust, and versatile tool for a broad range of brain
extraction and segmentation applications. The paper [149] demonstrates that
SAM can achieve high segmentation accuracy for brain tumor MRI datasets in a
point-to-mask setting, and effectively generalizes to brain tumor MRI datasets
and achieves segmentation accuracy similar to what was observed in the 2D
photographs where it was previously evaluated. Furthermore, the authors
identify the challenges encountered when using SAM for tumor segmentation in
MRI datasets and propose strategies to address them, which can also be applied
in the context of clinical implementation.

For colonoscopy images. A colonoscopy is a test to check inside the bowels.
This report [31] evaluates the performance of SAM in segmenting polyps under
unprompted settings. Polyp-SAM [150] is a finetuned SAM model designed for
polyp segmentation. The authors evaluate its performance against various SOTA
polyp segmentation models and compare the performance of two transfer learning
strategies: one involving finetuning of the encoders and the other without.
Experimental results on five public datasets illustrate SOTA performance on
two datasets and impressive performance on the remaining three.

For H&E stained histological sections images. H&E stained histological
sections refer to tissue samples that have been processed and stained with
hematoxylin and eosin (H&E) dyes for microscopic examination. This staining
technique is commonly used in histology and pathology to highlight different
structures and cellular components within the tissue sample. This test can
help find what’s causing your bowel symptoms. This work [156] assesses the
zero-shot segmentation performance of the SAM model on representative
segmentation tasks in whole slide imaging, including tumor segmentation, non-
tumor tissue segmentation, and cell nuclei segmentation. The core results
indicate that the zero-shot SAM model achieves remarkable segmentation
performance for large connected objects. The authors also identify several
limitations for digital pathology, including image resolution, multiple
scales, prompt selection, and model fine-tuning. To address these limitations,
few-shot fine-tuning with images from downstream pathological segmentation
tasks may help the model achieve better performance in dense object
segmentation in the future. This paper [158] demonstrates that its generated
masks, features, and stability scores can be leveraged to build and train more
advanced medical image segmentation models. Specifically, it showcases how SAM
can be used to enhance image inputs for a commonly used medical image
segmentation model, such as U-Net. The proposed method is tested on two
datasets, and the experiments demonstrate its effectiveness. SkinSAM [157]
presents a fine-tuned based model on SAM for skin cancer segmentation, which
demonstrates remarkable segmentation performance. Experimental results also
illustrate that larger models, such as ViT_L and ViT_H, can perform better
than the smaller ViT_b, the fine-tuned model (ViT_b_finetuned) showed the
greatest improvement.

For multiple format images. Works in this part evaluate SAM or propose SAM-
based segmentation methods for more than one segmentation task with multiple
kinds of medical images.

  * •

Evaluation of SAM’s ability to segment medical images. This paper [30] is the
first work at extending the success of SAM to medical images. It compiles an
extensive medical image dataset comprising 11 distinct modalities and
containing more than 200,000 masks. This work [37] is an extensive evaluation
of SAM’s ability to segment medical images on a collection of 11 medical
imaging datasets from various modalities and anatomies. This paper [151]
explores the accuracy of SAM on 12 public medical image segmentation datasets
which cover various organs (brain, breast, chest, lung, skin, liver, bowel,
pancreas, and prostate), image modalities (2D X-ray, histology, endoscopy, 3D
MRI, and CT), and health conditions (normal, lesioned). This work [32]
evaluates SAM on medical images and presents both quantitative and qualitative
results of zero-shot segmentation on nine benchmarks for medical image
segmentation. These benchmarks encompass a variety of imaging modalities,
including optical coherence tomography (OCT), magnetic resonance imaging
(MRI), and computed tomography (CT), and different applications such as
dermatology, ophthalmology, and radiology. This paper [36] evaluates SAM’s
zero-shot generalization on medical images by collecting more than 12 public
medical image datasets that cover various organs and modalities. This paper
[152] evaluates the zero-shot performance of SAM 2D in medical imaging by
testing it on six datasets from four different imaging modalities, including
X-ray, ultrasound, dermatoscopy, and colonoscopy. The findings indicate that
SAM 2D’s zero-shot performance is either comparable or superior to the
existing SOTA models. Huang et al. [38] collect and sort 52 open-source
datasets and build a large medical segmentation dataset with 16 modalities, 68
objects, and 553K slices. A comprehensive analysis of different SAM testing
strategies on the so-called COSMOS 553K dataset is conducted.

  * •

SAM-based segmentation methods for medical images. This paper [30] proposes a
straightforward fine-tuning approach to tailor the SAM model for general
medical image segmentation. Rigorous experimentation on 21 3D segmentation
tasks and 9 2D segmentation tasks illustrates that MedSAM surpasses the
default SAM model. Segment Any Medical Model (SAMM) [153] is an extension of
SAM on 3D Slicer, a widely-used open-source image processing and visualization
software that has been extensively used in the medical imaging community.
Medical SAM Adapter (MSA) [34] first integrates the medical-specific domain
knowledge to the segmentation model and shows superior performance on 19
medical image segmentation tasks with various image modalities, including CT,
MRI, ultrasound image, fundus image, and dermoscopic images. This paper [154]
introduces Learnable Ophthalmology SAM, a novel approach to multiple target
segmentation in ophthalmology multimodal images. The proposed method
incorporates a learnable prompt layer that extracts medical prior knowledge
from each transformer layer. Using a one-shot mechanism during training, this
work only trains the prompt layer and task head. Experiments are conducted to
demonstrate the effectiveness of the proposed method on four medical
segmentation tasks (i.e., blood vessel segmentation, lesion segmentation, and
retinal layer segmentation) across nine publicly available datasets. SAM-
Adapter [35] leverages domain-specific information or visual prompts to
enhance the segmentation network through the use of simple yet effective
adapters. By combining task-specific knowledge with general knowledge learned
by the large model, SAM-Adapter can notably improve the performance of SAM in
challenging tasks, as demonstrated by extensive experiments. PromptUNet [155]
is proposed by extending the existing prompt types in SAM to include novel
Supportive Prompts and En-face Prompts. The authors evaluate its capabilities
on 19 medical image segmentation tasks using a variety of image modalities and
PromptUNet exceeds a wide range of state-of-the-art (SOTA) medical image
segmentation methods.

For other medical images. This paper [159] evaluates SAM’s performance using
two established robotic instrument segmentation datasets from the MICCAI
EndoVis 2017 and 2018 challenges. The extensive evaluation indicates that SAM
displays remarkable zero-shot generalization ability with bounding box
prompts. Moreover, the qualitative figures demonstrate that the model either
fails to predict certain parts of the instrument mask (such as the jaws and
wrist) or misclassifies parts of the instrument in scenarios where instruments
overlap within the same bounding box or with point-based prompts. However, in
some complex surgical scenarios involving blood, reflection, blur, and shade,
SAM is unable to identify instruments. Additionally, SAM’s performance is
insufficiently robust to withstand various forms of data corruption. GazeSAM
[160] is the first work to leverage the power of eye-tracking technology and
SAM to enhance the efficiency of daily clinical practice. It enables
radiologists to collect segmentation masks by simply looking at the region of
interest during image diagnosis. This system tracks the radiologists’ gaze and
uses the resulting data as input prompts for SAM, which then generates the
segmentation mask in real-time automatically.

####  4.1.2 Video

In the field of computer vision, Video Object Tracking (VOT) [169, 170] and
video segmentation are recognized as crucial and indispensable tasks. It
involves locating a particular object in a video frame and subsequently
tracking it throughout the rest of the video. As such, it has various
practical applications, such as surveillance and robotics.

Figure 11: Pipeline of TAM. The figure is borrowed from the original paper
[46].

SAM [20] has made outstanding contributions in the field of VOT. Track
Anything Model (TAM) [46] was introduced, which achieves outstanding
interactive tracking and segmentation in videos with remarkable performance.
This research paper suggests the utilization of a highly-effective toolkit
named Track-Anything for high-performance object tracking and segmentation in
videos. Unlike current existing methods, it adopts an interactive approach for
initialization and incorporates the SAM and XMem [171]. The pipeline of TAM is
illustrated in Fig. 11. SAM is a large segmentation model with powerful
segmentation abilities, while XMem is an advanced semi-supervised VOS model.
The proposed method exhibits exceptional performance and user-friendliness in
complex situations, with potential applications in many fields. Additionally,
this paper scrutinizes failed instances and recommends optimal solutions. The
proposed method has the ability to tackle challenging situations in video
object perception with efficiency.

Furthermore, another tracking model, namely SAM-Track [172] has been proposed.
SAM-Track is a video segmentation framework that enables object tracking and
segmentation through both interactive and automatic methods. The pipeline of
SAM-Track is illustrated in Fig. 12. SAM-Track combines Grounding-DINO[51],
DeAOT[173], and SAM to achieve these capabilities. By employing both
interaction and automation, SAM-Track becomes a highly versatile tool with
potential beyond just object tracking and segmentation. Its combination of
features has allowed it to achieve successful results, demonstrating its
impact.

Figure 12: Pipeline of SAM-Track. The figure is borrowed from the original
paper [172].

In addition to its applications in tracking tasks, SAM has also shown promise
in enhancing the resolution of video. Video super-resolution (VSR)
[174][175][176][177][178] presents a primary challenge of handling large
motions in input frames, despite its potential in accurately aggregating
information from multiple frames. This challenge requires accurate
displacement estimation of input frames in complex scenes, which is a
challenge both in terms of real-time processing and computational complexity.
[179] proposes a method to enhance the quality of video super-resolution (VSR)
by utilizing a SAM to build a more robust and semantically-aware prior.
Moreover, the authors have devised a lightweight module called SAM-guided
refinement module (SEEM) to boost the performance of existing methods. The
framework of SEEM is illustrated in Fig. 13. The SEEM module is easy to
integrate into existing methods. Experimental results show that SEEM can
provide superior performance with both comprehensive fine-tuning and fine
parameter adjustment.

Figure 13: Overview of the proposed framework from the original paper [179].
(a) Illustrate how to obtain SAM-based representation. (b) Apply the proposed
SAM-guided refinement module (SEEM) to the sliding-window-based method. (c)
Apply SEEM to the bidirectional recurrent structure-based method.

####  4.1.3 Data Annotations

Data annotation in AI involves the process of labeling data for machine
learning algorithms to help them learn to recognize specific patterns,
objects, or features. Accurate data annotation is crucial to the development
of effective machine learning models that can successfully perform tasks like
object detection, classification, and natural language processing. Due to the
high cost associated with annotating images and videos in certain fields, many
datasets have not been labeled effectively, particularly at the pixel level.
However, the emergence of SAM[20] will facilitate the effective labeling of
such datasets.

SAMText[180] is a scalable pipeline for mask annotation of scene text in
videos. The pipeline utilizes the SAM for generating mask annotations in a
large-scale dataset, SAMText-9M, which contains over 2,400 video clips and
more than 9 million mask annotations. The authors argue that annotating scene
text at a finer level can lead to remarkable improvements in detection and
recognition performance, even for curved text. Furthermore, the paper
identifies several potential research directions, such as examining the effect
of mask annotations, enhancing data and model scalability, and generating
character-level mask annotations.

[143] presents a method to construct a large-scale remote sensing image
segmentation dataset named SAMRS by utilizing existing remote sensing object
detection datasets and the data-centric machine learning model SAM. SAMRS
includes object category, location, and instance information, which can be
utilized for semantic segmentation, instance segmentation, and object
detection research. The authors conducted a comprehensive analysis of the
dataset and discussed SAM’s performance in remote sensing image segmentation,
demonstrating its potential to improve annotation efficiency. SAMRS surpasses
previously existing high-resolution remote sensing image segmentation datasets
in size. SAMRS is a valuable resource when conducting large-scale model pre-
training research in the field of remote sensing image segmentation.

Generating high-quality pseudo-labels is a crucial step towards achieving
accurate and reliable results in various computer vision applications. These
pseudo-labels can be used to train and test various computer vision models,
enabling researchers and practitioners to achieve state-of-the-art results in
object recognition, semantic segmentation, and other related tasks. SAM has
made it incredibly easy, fast, and highly effective to generate high-quality
pseudo-labels[126][128][181], thus facilitating new breakthroughs in the field
of computer vision research and applications.

He et al. [126] propose a new method for weakly-supervised multi-object
semantic segmentation. The WS-SAM framework utilizes the recently proposed
vision foundation model, SAM, to generate segmentation masks, and proposes
several techniques, including multi-augmentation result fusion, pixel-level
uncertainty weighting, and image-level uncertainty filtration, to obtain
reliable pseudo labels for training a segmentation model. Chen et al. [128]
introduce a framework for pretraining thermal infrared image segmentation
models using pseudo-labels generated with the SAM model. The framework can
effectively improve the accuracy of segmentation results for specific
categories, exceeding the SOTA ImageNet pretrained model. The authors
generated a thermal infrared segmentation dataset containing over 100,000
images for pretraining. Jiang et al. [181] present an approach to weakly
supervised semantic segmentation that utilizes cheap annotations such as image
labels, points, and scribbles as prompts for SAM. This model outputs object
masks with precise boundaries, which are used to generate pseudo labels for
training segmentation networks. The experiments on the PASCAL VOC[182] show
that SAM can serve as an effective pseudo-label generator.

###  4.2 Beyond Vision

####  4.2.1 3D Reconstruction

Figure 14: Overall pipeline of SA3D [183]. Initially, the user provides
prompts for the target object, and SAM generates a segmentation mask for it.
This 2D mask is then projected onto a 3D mask grid using mask inverse
rendering, taking into account the density distribution. By rendering
incomplete 2D masks from novel views, SA3D extracts reliable prompts and
queries SAM to obtain accurate segmentation masks. This iterative process,
called cross-view self-prompting, alternates between mask inverse rendering
and obtaining segmentation masks, ultimately resulting in the final 3D
segmentation results.

In addition to achieving fine-grained 3D segmentation, SA3D[183] can also be
used for 3D reconstruction. With the 3D mask grid obtained from the previous
section, we can determine the occupied space of objects in 3D and reconstruct
them in various ways.

However, due to the high memory requirements and computational complexity of
NeRF-based methods, they are currently limited to relatively small scenes and
cannot handle large-scale outdoor scenes. To tackle this challenge, several
studies [184, 185] have proposed to use additional input modalities such as
depth maps and surface normals to improve the efficiency and accuracy of NeRF-
based 3D reconstruction.

SAM[20] is a SOTA method for 2D image segmentation that can segment anything
with user-specified prompts. SAM can be used in various applications such as
object detection, image retrieval, and image synthesis. However, SAM is
currently limited to 2D image data and cannot be directly applied to 3D scene
understanding.

The proposed SA3D framework, as shown in Fig. 14 extends the segmentation
ability of SAM to 3D scenes by leveraging NeRFs. SA3D can segment any object
in a 3D scene with one-shot manual prompting in a single rendered view. SA3D
utilizes mask inverse rendering and cross-view self-prompting techniques to
project 2D masks onto 3D mask grids and generate new prompts for different
views, respectively. Compared to previous approaches based on NeRFs, SA3D can
easily adapt to any pre-trained NeRF without any changes and re-training.

Figure 15: Framework of OR-NeRF[186].

On the contrary of reconstructing objects, this paper[186] presents a new
object-removing pipeline called OR-NeRF that removes objects from 3D scenes
using either point or text prompts on a single view. The method achieves
better performance in less time than previous works by using a points
projection strategy to rapidly spread user annotations to all views. The
algorithm shown in Fig. 15 leverages the recent 2D segmentation model SAM[20]
to predict masks with improved precision and efficiency and obtains color and
depth priors through 2D inpainting methods. Finally, the algorithm employs
depth supervision and perceptual loss for scene reconstruction to maintain
consistency in appearance after object removal.

Although the above methods have demonstrated promising performance in 3D
segmentation, there is still much room for improvement. Here, we discuss
several potential future directions for 3D reconstruction:

  * •

Improving the efficiency of NeRF-based methods: NeRF-based methods are
computationally expensive and memory-intensive, limiting their scalability to
large-scale scenes. Future research can focus on improving the efficiency and
scalability of NeRF-based methods to enable real-time 3D scene understanding.

  * •

Incorporating additional input modalities: In addition to multi-view images,
incorporating additional input modalities such as depth maps and surface
normals can improve the accuracy and efficiency of NeRF-based 3D
reconstruction and segmentation.

  * •

Integrating with other 3D perception tasks: 3D segmentation is an important
task in 3D scene understanding. Integrating SA3D with other 3D perception
tasks, such as 3D object detection, 3D scene reconstruction, and 3D pose
estimation, can lead to more comprehensive 3D scene understanding.

  * •

Expanding the scope of applications: SA3D can be applied to various
applications such as robotics, augmented reality, and virtual reality. Future
research can explore the potential of SA3D in these fields and develop novel
applications.

####  4.2.2 Non-Euclidean Domain

Figure 16: Illustrations of SNA from the original paper [28].

In the context of graph neural networks, the Non-Euclidean domain refers to
graphs that are irregular and do not have a predefined structure like grids or
lattices. These graphs can represent a wide range of data, including social
networks, citation networks, e-commerce product graphs, and molecule graphs.
Due to the complexity and heterogeneity of these graphs, developing a
foundation model for universal graph analysis has become a challenging task.

In recent years, several research works have been conducted to address the
challenges of graph analysis in the Non-Euclidean domain. For example, the
work presented in [187] proposes the Graph Convolutional Network (GCN), which
is based on spectral graph convolutions and can be applied to a wide range of
graph-related tasks. The GraphSAGE method proposed by Hamilton et al. [188]
enables the scalable processing of large graphs by sampling neighbors instead
of using all of them. The Graph Attention Network (GAT) introduced by
Velivckovic et al. [189] utilizes the attention mechanism to learn the weights
for each neighbor automatically.

Despite the progress made in Non-Euclidean graph analysis, there is still a
need for more versatile and adaptable foundation models. In particular, the
recent work presented in [20] introduces the SAM, which is a prompt-based
framework for general image analysis. SAM enables users to input natural
language prompts to perform a range of image-related tasks, including object
detection, segmentation, and classification. Inspired by the success of SAM,
the Segment Anything in Non-Euclidean (SNA) [28] paradigm aims to develop a
foundation model for universal graph analysis that is flexible, adaptable, and
capable of handling diverse graph samples and tasks.

To address the challenges of handling diverse feature dimensions for varying
tasks, the SNA approach shown in Fig. 16 introduces a specialized slimmable
graph convolutional layer. This layer enables the dynamic activation or
deactivation of its channels based on the input feature dimensions.
Furthermore, the approach proposed in this paper employs a meta-learning
strategy that learns to select the optimal neurons based on the downstream
tasks rather than relying on manual selection. This approach enables the
handling of a broader range of unseen tasks and represents a step towards
developing more versatile and adaptable foundation models for universal graph
analysis.

In summary, the SNA paradigm aims to establish a foundation model for
universal graph analysis in the Non-Euclidean domain. The SNA approach
leverages the prompt-based framework introduced in the SAM and introduces
specialized slimmable graph convolutional layers and a meta-learning strategy
to handle the challenges of diverse graph samples and tasks. The proposed
approach has the potential to inspire future research in developing more
versatile and adaptable foundation models for graph analysis in the Non-
Euclidean domain.

####  4.2.3 Robotics

In recent years, the use of foundation models has advanced various
applications such as image segmentation and natural language processing. In
this paper[190], the authors present Instruct2Act, a framework that utilizes
Large Language Models (LLMs) to map multimodal instructions to sequential
actions for robotic manipulation tasks. The framework employs the LLM model to
generate Python programs that form a comprehensive perception, planning, and
action loop for robotic tasks.

Figure 17: Paradigm of the Instruct2Act[190] framework. Best viewed by zooming
in.

Fig. 17 shows the overall pipeline of Instruct2Act[190]. In the perception
section, pre-defined APIs are used to access multiple foundation models. The
SAM[20] accurately locates candidate objects, and CLIP[13] classifies them.
The framework leverages the expertise of foundation models and robotic
abilities to convert complex high-level instructions into precise policy
codes. In this way, Instruct2Act[190] is adjustable and flexible in
accommodating various instruction modalities and input types, catering to
specific task demands.

The practicality and efficiency of Instruct2Act[190] were validated by
assessing it on robotic tasks in different scenarios within tabletop
manipulation domains. Moreover, the zero-shot method outperformed many state-
of-the-art learning-based policies in several tasks. The code for
Instruct2Act[190] is available at https://github.com/OpenGVLab/Instruct2Act,
serving as a robust benchmark for high-level robotic instruction tasks with
assorted modality inputs. This framework provides a promising approach for
enabling robots to perform complex tasks by leveraging the power of foundation
models and large language models.

In conclusion, Instruct2Act[190] sets foot into the robotics area with the
latest powerful tools for the first time, this may be a foot print of humanity
stepping into the AGI. We will see more research in this field, paving the way
for a more productive society.

####  4.2.4 Video Text Spotting

Video text spotting is a challenging task that involves localizing and
recognizing text instances in video frames or sequences. Traditional methods
for video text spotting have relied on the detection of bounding boxes and the
subsequent recognition of text instances within those boxes. However, these
methods are limited in their ability to accurately localize text instances,
particularly those with irregular shapes or orientations.

Figure 18: Overview of SAMText [180] pipeline, which extends the SAM approach
to generate mask annotations for scene text images or video frames in a
scalable manner. The pipeline takes an input bounding box, which can either
come from existing annotations or be derived from a scene text detection
model.

Recent advances in computer vision, such as the segmentation-based approach
proposed by the SAM [20], have shown promise in addressing these limitations.
The SAM approach utilizes a deep neural network to generate pixel-level
segmentation masks for text instances, resulting in more accurate and fine-
grained annotations.

In this context, the SAMText [180] pipeline shown in Fig. 18 offers a scalable
and efficient solution for generating mask annotations for video text spotting
tasks. By applying the SAM model to bounding box annotations, SAMText is able
to generate mask annotations for large-scale video text datasets, as
demonstrated by the SAMText-9M dataset.

While SAMText is a novel approach for generating mask annotations for video
text spotting tasks, it builds upon the foundation laid by the SAM model.
Specifically, the SAM model’s ability to generate high-quality pixel-level
masks for objects in images has been adapted for the specific task of
generating masks for text instances in video frames.

Given an input scene text image or video frame, SAMText begins by extracting
the bounding box coordinates from existing annotations or derived from a scene
text detection model. If the boxes are oriented, SAMText calculates their
minimum bounding rectangle to obtain the horizontal bounding boxes (HBB),
which is then used as the input prompt for the SAM model to obtain mask
labels. The SAM model is a segmentation model that is pre-trained on natural
images and fine-tuned on the COCO-Text dataset to generate mask annotations
for text instances.

After obtaining the mask for each text instance, post-processing may be
necessary to ensure its connectivity. In particular, if a mask comprises
several segments, it may be desirable to derive a minimum enclosing mask as an
optional step in order to achieve a more cohesive representation. Furthermore,
optical flow estimation can also be utilized to enhance the accuracy of the
generated masks and ensure temporal consistency.

The SAMText pipeline provides an exciting avenue for future research in video
text spotting. By providing fine-grained mask annotations for large-scale
datasets, SAMText enables the development and evaluation of more accurate and
effective video text spotting models. Additionally, the SAMText approach may
inspire the development of new segmentation-based approaches for other
computer vision tasks.

Overall, the SAMText pipeline represents an important contribution to the
field of video text spotting, offering a scalable and efficient solution for
generating fine-grained mask annotations. The approach holds promise for
advancing the accuracy and effectiveness of video text spotting models and may
inspire further research in the field of computer vision.

####  4.2.5 Vision and Language

Figure 19: Overall structure of the Text2Seg [144] pipeline, which consists of
three methods for guiding the SAM model. The pipeline begins by using a text
prompt as input for Grounding DINO, which generates bounding boxes. These
bounding boxes are then passed to SAM, which produces a segmentation map. In
the second step, a text prompt is given to CLIP Surgery, resulting in a
heatmap. This heatmap is sampled to create point prompts for SAM, which
generates segmentation masks. Finally, SAM is used to generate segmentation
maps, and CLIP is employed to compare the semantic similarity between these
maps and the text prompt. The figure is borrowed from the original paper
[144].

SAM also benefits vision and language tasks, such as image caption and text-
based segmentation. InternGPT[191] is a system that combines chatbots capable
of planning and reasoning with non-verbal cues like pointing movements,
allowing users to manipulate images and videos on a screen directly. The
system uses SAM to segment objects. In this study[144], the authors propose a
pipeline called Text2Seg, which leverages multiple visual foundation models to
facilitate remote sensing image semantic segmentation tasks guided by text
prompts. The authors focus on the remote sensing domain, where images are
notably dissimilar from those in conventional scenarios, and traditional
models often underperform when confronted with testing data collected under
different scenarios.

The authors incorporate multiple visual foundation models into their pipeline,
including the SAM, which was introduced by Meta AI Research as the first
foundation model for the object segmentation task. SAM is capable of
performing zero-shot segmentation using various visual prompts as guidance,
making it particularly suitable for remote sensing imagery processing, where
labeled datasets are often sparse and inherently heterogeneous.

However, adapting SAM for specific tasks remains a non-trivial task, as the
model segments all objects into distinct masks, making it impractical to use
directly for semantic segmentation tasks. To address this issue, the authors
propose leveraging other foundation models that focus on different aspects to
generate visual prompts as guidance for the SAM model. These models can
generate points or bounding boxes that help narrow down the areas where SAM
needs to make predictions or assist in filtering SAM’s predictions based on
specific text prompts.

The authors evaluated their pipeline (shown in Fig. 19) on four commonly used
remote sensing datasets and achieved promising results. They showed that
although SAM is effective in segmenting instances within the provided frame,
generating segmentation masks for a specific category remains challenging. In
contrast, other visual foundation models, like Grounding DINO[192] and
CLIP[13], exhibit a superior capacity to understand the semantic features of
images, thus generating coarse-grained visual prompts. Integrating these
models into a unified pipeline allows us to exploit their combined strengths.

Overall, this work provides insights into maximizing the applicability of
visual foundation models in specific contexts with minimal model tuning. The
authors’ proposed pipeline is not limited to any specific dataset and can be
applied to various scenarios with minimal adjustments to the prompt tuning
process. The authors hope that their work will encourage additional research
on the application of visual foundation models for diverse downstream tasks
and stimulate the development of increasingly powerful visual foundation
models.

Figure 20:  Overview of the Caption Anything [44] framework, which enhances
image captioning by incorporating multimodal controls that align with human
intention, allowing for a diverse range of visual focuses and language styles.
The visual prompt is initially converted into a mask prompt using the
segmenter. Then, the captioner generates a raw caption specifically for the
region outlined by the mask. To ensure that the captioner emphasizes the
user’s intended object, a straightforward visual chain-of-thought technique is
employed for step-by-step inference. Finally, both the text prompt and the raw
caption are input into the text refiner, which generates a user-preferred
caption that aligns with the desired genre.

Image captioning is the task of generating natural language descriptions for
given images. It is a fundamental problem in computer vision and natural
language processing, with various applications in robotics, image retrieval,
and content-based image retrieval [193]. In recent years, significant progress
has been made in this field, especially with the development of deep learning
techniques.

One recent approach for controllable image captioning is Caption Anything
(CAT), proposed by Wang et al. [44]. The framework, as shown in Fig. 20,
introduces multimodal controls to image captioning, rendering a variety of
visual focuses and language styles aligned with human intention. The approach
is formulated as a triplet solver, consisting of a segmenter, a captioner, and
a text refiner. The segmenter takes the interactive visual controls and
represents the user-interested regions via pixel-level masks, which are
subsequently used by the captioner to generate raw descriptions in relation to
the specified region based on the original image and the provided mask. To
facilitate the captioner to focus on the user-interested object, the authors
design a visual chain-of-thought technique with step-by-step inference.
Finally, the text refiner refines the raw descriptions by incorporating user-
defined language controls, tailoring the language style according to user
preferences.

The CAT approach leverages the SAM [20] to perform the segmentation task. SAM
is a promptable segmentation model that achieves strong performance across
various image domains. Specifically, SAM adapts interactive segmentation to
achieve promptable ability, where a prompt, any interaction (e.g., points,
boxes) indicating what to segment in an image, is used to prompt SAM to return
a valid segmentation mask. Once the user-specified segmentation mask is
obtained, it is easy to generate the desired caption according to the original
image and mask prompt.

The CAT approach is a training-free and adaptable solution for controllable
image captioning tasks. It expands the range of supported control signals,
enhances the model’s flexibility and scalability, and offers strong user-
interactive capabilities. The work highlights the importance of multimodal
controls and prompts in controllable image captioning and provides insights
into potential future research directions in this field.

In conclusion, vision and language tasks may not need training models anymore
with the help of SAM and LLMs. We are about to see more powerful tools
utilizing large pre-trained models or APIs to solve various kinds of tasks.

####  4.2.6 Audio and Vision

Figure 21: Illustration of the proposed Segment Anything Model for Audio-
Visual localization and segmentation (AV-SAM) from the original paper [45].
The pixel-wise audio-visual fusion module combines audio features and visual
features from pre-trained audio and image encoders. These cross-modal
representations are aggregated, and along with prompt embeddings from the
prompt encoder, they are fed into the mask decoder. The mask decoder utilizes
this information to generate the final segmentation mask.

Audio and vision are two modalities that are closely related and can provide
complementary information to solve many problems. In recent years, there has
been an increasing interest in joint audio-visual learning [194, 195], which
aims to learn the correlation between the two modalities and leverage the
complementary information for better performance in various tasks.

One of the most popular applications of audio-visual learning is in sound
localization and segmentation [196, 197]. This task aims to predict the
spatial location of individual sound sources in a video. Audio-visual
localization and segmentation can be challenging due to the complex nature of
the problem, as audio is not naturally aligned with all objects that exist in
the video. However, with the recent advances in deep learning, researchers
have developed many effective methods for this task.

One approach [45] to audio-visual localization and segmentation is to learn
cross-modal representations that can align audio and visual information. In
Fig. 21, AV-SAM leverages pixel-wise audio-visual fusion across audio and
visual features from the pre-trained audio encoder and image encoder to
aggregate cross-modal representations. Then, the aggregated cross-modal
features are fed into the prompt encoder and mask decoder to generate the
final audio-visual segmentation masks.

Another approach to audio-visual localization and segmentation is to use
contrastive learning to learn cross-modal correspondences. For example, EZ-VSL
[198] and SLAVC [199] both use contrastive learning to learn the
correspondence between audio and visual features. EZ-VSL uses a two-stream
network to learn audio-visual alignment, while SLAVC uses a self-supervised
contrastive learning approach to learn the correspondence between audio and
visual features.

In addition to sound localization and segmentation, there are many other
applications of audio-visual learning, such as audio-visual spatialization
[200, 201], audio-event localization [202, 203], and audio-visual parsing
[204, 205] have been addressed in previous studies.

Overall, audio-visual learning has become an increasingly important field in
deep learning and has many applications in various domains. With the recent
advances in deep learning, we can expect to see more innovative methods for
joint audio-visual learning in the future.

####  4.2.7 Multimodal Visualization and Open-Vocabulary Interactive
Segmentation

Figure 22: Overview of the CLIP Surgery [206], which introduces dual paths for
open-vocabulary interactive segmentation. The original path consists of the
original block with q-k self-attention, while the new path incorporates the
proposed v-v self-attention without a feed-forward network (FFN), starting
from a certain depth, denoted as $d$. It is important to note that the
parameters of the new block are copied from the original block to ensure
consistency. Additionally, feature surgery is employed to combine the features
of image tokens $\bm{F}_{i}$ with the text features $\bm{F}_{t}$ for the
explainability task. Furthermore, feature surgery is applied to the feature of
the class token $\bm{F}_{c}$ from the original path for multi-label
recognition. The figure is borrowed from the original paper [206].

Recent work has shown that CLIP[13] can achieve impressive performance on a
variety of vision tasks with minimal or no task-specific training. However,
its internal mechanisms are not well understood. In this section, we discuss a
recent work[206] that applies CLIP to the task of open-vocabulary interactive
segmentation, which is closely related to the concept of multimodal
visualization.

Interactive segmentation is a computer vision task that involves segmenting a
target object from an image with user guidance in the form of points,
scribbles, or boxes during the inference phase. The SAM [20] is a recent work
that enables interactive segmentation via text prompts in an open-vocabulary
manner. SAM requires manual points to guide the segmentation process.

The proposed method [44] illustrated in Fig. 22 aims to replace the need for
manual points entirely by using CLIP Surgery with text-only inputs. This
approach provides pixel-level results from text input, which can be readily
converted to point prompts for the SAM model. Specifically, the authors select
foreground points ranked ahead in the similarity map and use the same number
of points ranked last as background points. The authors show that their method
outperforms other explainability methods in terms of both points accuracy and
mIoU with SAM on four datasets.

The proposed method offers several advantages over other prompt formats in
SAM. First, the method requires text input only, without the annotation cost
of manual points suggested in the paper of SAM. Second, point prompts are
superior to mask prompts because SAM’s mask prompt is designed for its own
output logits, and the generated points are more suitable than masks from
another model. Finally, text-to-points are more readily achievable than the
solution of text-to-boxes, which requires fine-tuning or additional
supervision.

The proposed method also has implications for the explainability of CLIP in
multimodal settings. Multimodal visualization is a promising direction for
exploring CLIP’s internal mechanisms. By visualizing the image-text pairs
during training, the authors were able to observe interesting phenomena
related to CLIP’s learning process. However, the proposed method does not
fully explain how CLIP is able to generate pixel-level results from text
input. This suggests that further research is needed to better understand the
mechanisms behind CLIP’s impressive performance on open-vocabulary tasks.

In summary, the proposed method offers a promising solution for open-
vocabulary interactive segmentation and has implications for the
explainability of CLIP in multimodal settings. Future work could explore the
potential of combining the proposed method with other SOTA models to achieve
even better results.

###  4.3 More Directions

####  4.3.1 Weakly-Supervised Semantic Segmentation

Recently, there are some works exploring the SAM in Weakly-Supervised Semantic
Segmentation (WSSS). The advantage of using the SAM in WSSS is that it can
produce satisfactory results without requiring model fine-tuning, which is a
cumbersome requirement in current WSSS techniques such as [207, 208], which
involve classification re-training and pseudo-label generation. By using SAM
as a foundation model in WSSS, the process can be made more straightforward
and less complex.

For example, [209] aimed to investigate the suitability of the SAM for WSSS
and adapt it to generate pseudo-labels using only image-level class labels.
The study conducted an initial analysis of SAM as a foundation model in WSSS
and discovered that it can achieve comparable performance without requiring
fine-tuning. The study also revealed that SAM could generate high-quality
segmentation masks and even surpass human annotations in some cases.

In [209], the performance of SAM was evaluated on the PASCAL VOC and MS-COCO
datasets, where it demonstrated significant improvements over the latest SOTA
methods. However, SAM encountered difficulties in certain situations due to
the issue of semantic obscurity. While SAM performed well in most unambiguous
settings, addressing semantic obscurity may require investigating the use of
hierarchical-structured semantic classes and better prompts. Additionally, the
study suggests exploring SAM’s ability to segment ”stuff” classes like ”sky,”
”sea,” and ”road” to enhance overall scene understanding. In [181], the
authors presented a WSSS method that utilizes SAM as a pseudo-label generator.
They employed diverse weak labels, such as image-level labels, points,
scribbles, and bounding boxes, as prompts for SAM to produce precise class
masks. These masks were then used to generate pseudo labels for training
segmentation networks. The method’s effectiveness was evaluated on the PASCAL
VOC 2012 dataset, and the results indicate that SAM can serve as a reliable
pseudo-label generator, with scribbles as prompts achieving an 89.7% mIoU
score on the training set. The final segmentation model obtained a 76.6% mIoU
score on the test set. This suggests that the method can be valuable in
training segmentation networks with weak supervision, especially in scenarios
where pixel-level annotations are unavailable.

However, in [181], the authors also pointed out some of SAM’s limitations in
handling WSSS. For instance, when using point prompts located by CAMs [210,
211], SAM may generate incorrect object masks due to the CAMs’ coarse
locations. Additionally, in the case of using bounding box prompts, SAM may
struggle to generate precise object masks when multiple objects are placed on
the same table. These limitations imply that SAM may not always be capable of
producing accurate object masks when dealing with WSSS.

In addition, due to the fact that current methods that rely on CAM to generate
pseudo labels suffer from limitations such as partial and false activation. To
overcome this, a new approach in [212] was introduced in this study that
utilizes the SAM to generate high-quality pseudo labels by selecting relevant
masks and labeling them based on initial CAM seed masks or post-processed
pseudo labels. The results show that this approach significantly improves the
precision of segmentation while also reducing computational costs compared to
existing post-processing modules. The approach is highly versatile and
compatible with existing WSSS models without modification to base networks or
pipelines, and it has been shown to improve the mIoU of pseudo labels from
five SOTA WSSS methods by an average of 6.2% on the train set of the PASCAL
VOC 2012 dataset.

####  4.3.2 Adversarial Robustness

As the concern regarding the susceptibility of computer vision systems to
adversarial attacks grows [213], there is a need to understand and research
adversarial attacks. Adversarial attacks involve adding minor, undetectable
perturbations to an input image to deceive the computer vision system into
misclassifying the image [214, 215]. This poses a significant concern because
it may result in security breaches and safety risks in applications like
autonomous vehicles [216] and facial recognition systems [217].

Recently, [50] assessed the adversarial robustness of the SAM model in the
context of prompt-based segmentation in computer vision. The authors developed
a framework called Attack-SAM to evaluate SAM’s vulnerability to adversarial
attacks in the prompt-based mask prediction task. This is the first
comprehensive investigation of SAM’s susceptibility to adversarial attacks,
and the findings indicate that while SAM is vulnerable to white-box attacks,
it remains somewhat robust in the black-box setting. Understanding the
robustness of SAM in the face of adversarial attacks is crucial for developing
more secure computer vision systems. The study provides the following
insights:

  * •

Firstly, the research reveals that SAM is vulnerable to white-box attacks,
meaning that an attacker with full knowledge of the model’s architecture and
parameters can generate adversarial examples that deceive the model with ease.
However, SAM is relatively robust in the black-box setting, where the
attacker’s knowledge of the model is limited.

  * •

Secondly, the study shows that small objects tend to be more resistant to
adversarial attacks in the black-box setting, possibly due to the limited
perturbation of the small object region.

  * •

Thirdly, the paper offers insights into the transferability of adversarial
attacks among prompts in the segment everything mode of SAM. Additionally, the
study provides a list of recent research papers and surveys on various topics
related to generative AI, which can serve as a useful resource for researchers
in the field.

Besides, [49] presented BadSAM, which applied backdoor attacks to the SAM
model for image segmentation. This paper demonstrated the potential security
risks associated with such attacks and emphasized the need for further
investigation into defense strategies in this area. The experiments conducted
in the study demonstrate that the SAM model can be exploited by attackers,
presenting a significant risk to end-users. The paper also outlines a process
for initiating backdoor attacks on SAM and provides insights into the
attacker’s knowledge and methodology. In [218], the researchers carried out a
thorough investigation into the robustness of SAM in various real-world
scenarios. They conducted experiments that involved a wide range of image
perturbations and domains, which revealed that SAM’s performance typically
deteriorates when dealing with perturbed images. However, the degree of
vulnerability varies across different types of perturbations. By tailoring the
prompting techniques and utilizing domain-specific knowledge that takes into
account the unique characteristics of each dataset, it is possible to enhance
the model’s resilience to these perturbations and tackle challenges that are
specific to each dataset. This work further highlights certain types of
perturbations that have a significant impact on the model’s ability to
withstand challenges, identifying areas where improvements can be made.

####  4.3.3 One Shot

It was known that training the SAM model needs large datasets, which may be
unreliable in some practical scenarios, such as medical imaging segmentation
[219], or acoustic-to-articulatory analysis [220]. As a result, there is a
growing interest in exploring segmentation techniques that require minimal
data and do not rely heavily on training. One of the benefits of utilizing SAM
in one-shot learning is that it allows for personalized segmentation models to
be efficiently created using only one-shot data, which consists of a user-
provided image and a rough mask indicating the target object. To accomplish
this, SAM’s image encoder and the given mask are employed to encode the
reference image’s target object embedding. The feature similarity between the
object and all the pixels on the new test image is then computed, and a
positive-negative pair of points is selected, encoded as prompt tokens, and
used as a location prior to SAM.

Recently, a novel training-free technique known as PerSAM was presented in
[21] for creating personalized segmentation models for specific visual
concepts using only one-shot data. PerSAM accomplishes this by localizing the
target object in a reference image and utilizing target-guided attention,
target-semantic prompting, and cascaded post-refinement to segment it in other
images or videos. The study also introduces a variant of the approach,
PerSAM-F, which fine-tunes only two parameters to enhance performance. The
technique is assessed on a novel segmentation dataset, PerSeg, and is found to
perform competitively in video object segmentation. The study also
demonstrates that PerSAM can assist in improving personalized text-to-image
generation.

####  4.3.4 Explainable AI

Explainable AI (XAI) plays a vital role in enhancing human comprehension of
DNNs and has gained significant attention from researchers [221]. It is worth
noting that pixel-based XAI methods currently in use explain DNN decisions by
identifying key pixels, while concept-based XAI methods aim to provide
explanations using concepts. However, the interpretation of pixels can be
challenging, and the imprecision of XAI methods can impact their reliability.
Additionally, prior concept-based approaches either require human annotation
or are limited to predefined concept sets.

A recent study [222] proposed an innovative approach that leverages the SAM to
enhance concept-based XAI. The authors introduced Explain Any Concept (EAC),
an effective and adaptable explanation method capable of elucidating DNN
decisions using any concept. An overview of the EAC can be seen in Figure 23.
Although SAM excels in instance segmentation, integrating it into XAI
pipelines can be computationally demanding. To address this concern, the
authors proposed a lightweight per-input equivalent (PIE) scheme, enabling
efficient explanation using a surrogate model. Evaluation conducted on
ImageNet and COCO datasets showcased the promising performance of EAC compared
to commonly utilized XAI methods.

Figure 23: Overview of the EAC approach, which employs a three-phase pipeline
to provide explanations for a DNN’s prediction concerning an input image. More
precisely, In the initial phase, it utilizes the widely accepted instance
segmentation model, SAM, to partition the input image into a collection of
visual concepts. Moving on to the second phase, a per-input equivalent (PIE)
surrogate model is trained to approximate the behavior of the target DNN.
Finally, in the third phase, the surrogate model is employed to efficiently
explain the model’s prediction using the concepts obtained in the first phase.
The figure is borrowed from the original paper [222].

However, the paper raises a significant concern regarding the potential
negative social impacts associated with using EAC in specific domains. It
suggests that EAC could be misapplied to explain DNN predictions for medical
images using unrelated concepts such as cats, trains, or subtle errors. Such
misleading explanations could have severe consequences and misguide medical
professionals in their decision-making process. Consequently, deploying EAC
without adequate safety checks on its outputs in sensitive domains may pose
significant risks. Future research efforts could focus on minimizing these
negative societal impacts while exploring further applications of EAC in
diverse visual domains and tasks.

##  5 Conclusion

This survey is the first to comprehensively review the recent progress on the
foundation model of SAM for computer vision and beyond. Firstly, we summarize
the development history of foundation models, covering large language models,
large visual models, and large multimodal models, as well as the essential
terminology about SAM. With a special focus on the applications of SAM to
various tasks and data types, we summarize and compare the concurrent works of
SAM and its follow-up works. Then, the great potential of SAM in a wide range
of image processing applications is discussed, including software scenes,
real-world scenes, and complex scenes. We also analyze and summarize the
advantages and limitations of SAM across various applications. These
observations can provide some insights to guide future research to develop
stronger foundation models and further improve the robustness and
generalization capabilities of SAM. Finally, we summarize massive other
amazing applications of SAM in vision and beyond. The appendix provides a
preliminary summary of open-source projects on SAM in table format.

## References

  * [1] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill _et al._ , “On the opportunities and risks of foundation models,” _arXiv preprint arXiv:2108.07258_ , 2021.
  * [2] X. Wang, G. Chen, G. Qian, P. Gao, X.-Y. Wei, Y. Wang, Y. Tian, and W. Gao, “Large-scale multi-modal pre-trained models: A comprehensive survey,” _arXiv preprint arXiv:2302.10035_ , 2023.
  * [3] P. P. Liang, A. Zadeh, and L.-P. Morency, “Foundations and recent trends in multimodal machine learning: Principles, challenges, and open questions,” _arXiv preprint arXiv:2209.03430_ , 2022.
  * [4] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [5] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” _The Journal of Machine Learning Research_ , vol. 21, no. 1, pp. 5485–5551, 2020.
  * [6] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” _Advances in neural information processing systems_ , vol. 33, pp. 1877–1901, 2020.
  * [7] OpenAI, “Gpt-4,” https://cdn.openai.com/papers/gpt-4.pdf, 2023.
  * [8] ——, “Introducing chatgpt,” https://openai.com/blog/chatgpt, Accessed: 2023-05-10.
  * [9] X. Zhai, A. Kolesnikov, N. Houlsby, and L. Beyer, “Scaling vision transformers,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 12 104–12 113.
  * [10] M. Dehghani, J. Djolonga, B. Mustafa, P. Padlewski, J. Heek, J. Gilmer, A. Steiner, M. Caron, R. Geirhos, I. Alabdulmohsin _et al._ , “Scaling vision transformers to 22 billion parameters,” _arXiv preprint arXiv:2302.05442_ , 2023.
  * [11] Z. Liu, H. Hu, Y. Lin, Z. Yao, Z. Xie, Y. Wei, J. Ning, Y. Cao, Z. Zhang, L. Dong _et al._ , “Swin transformer v2: Scaling up capacity and resolution,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 12 009–12 019.
  * [12] L. Wang, B. Huang, Z. Zhao, Z. Tong, Y. He, Y. Wang, Y. Wang, and Y. Qiao, “Videomae v2: Scaling video masked autoencoders with dual masking,” _arXiv preprint arXiv:2303.16727_ , 2023.
  * [13] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” in _International conference on machine learning_. PMLR, 2021, pp. 8748–8763.
  * [14] C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung, Z. Li, and T. Duerig, “Scaling up visual and vision-language representation learning with noisy text supervision,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 4904–4916.
  * [15] A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” _arXiv preprint arXiv:1807.03748_ , 2018\.
  * [16] B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebnik, “Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 2641–2649.
  * [17] X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick, “Microsoft coco captions: Data collection and evaluation server,” _arXiv preprint arXiv:1504.00325_ , 2015.
  * [18] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever, “Zero-shot text-to-image generation,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 8821–8831.
  * [19] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical text-conditional image generation with clip latents,” _arXiv preprint arXiv:2204.06125_ , 2022.
  * [20] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo _et al._ , “Segment anything,” _arXiv preprint arXiv:2304.02643_ , 2023.
  * [21] R. Zhang, Z. Jiang, Z. Guo, S. Yan, J. Pan, H. Dong, P. Gao, and H. Li, “Personalize segment anything model with one shot,” _arXiv preprint arXiv:2305.03048_ , 2023.
  * [22] G.-P. Ji, D.-P. Fan, P. Xu, M.-M. Cheng, B. Zhou, and L. Van Gool, “Sam struggles in concealed scenes–empirical study on” segment anything”,” _arXiv preprint arXiv:2304.06022_ , 2023.
  * [23] X. Wang, X. Zhang, Y. Cao, W. Wang, C. Shen, and T. Huang, “Seggpt: Segmenting everything in context,” _arXiv preprint arXiv:2304.03284_ , 2023.
  * [24] X. Zou, J. Yang, H. Zhang, F. Li, L. Li, J. Gao, and Y. J. Lee, “Segment everything everywhere all at once,” _arXiv preprint arXiv:2304.06718_ , 2023\.
  * [25] J. Jain, J. Li, M. Chiu, A. Hassani, N. Orlov, and H. Shi, “Oneformer: One transformer to rule universal image segmentation,” _arXiv preprint arXiv:2211.06220_ , 2022.
  * [26] P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, “Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing,” _ACM Computing Surveys_ , vol. 55, no. 9, pp. 1–35, 2023.
  * [27] J. Fan, “Gpt-3 moment in computer vision,” https://twitter.com/DrJimFan, 2023.
  * [28] Y. Jing, X. Wang, and D. Tao, “Segment anything in non-euclidean domains: Challenges and opportunities,” _arXiv preprint arXiv:2304.11595_ , 2023.
  * [29] L. Tang, H. Xiao, and B. Li, “Can sam segment anything? when sam meets camouflaged object detection,” _arXiv preprint arXiv:2304.04709_ , 2023.
  * [30] J. Ma and B. Wang, “Segment anything in medical images,” _arXiv preprint arXiv:2304.12306_ , 2023.
  * [31] T. Zhou, Y. Zhang, Y. Zhou, Y. Wu, and C. Gong, “Can sam segment polyps?” _arXiv preprint arXiv:2304.07583_ , 2023.
  * [32] P. Shi, J. Qiu, S. M. D. Abaxi, H. Wei, F. P.-W. Lo, and W. Yuan, “Generalist vision foundation models for medical imaging: A case study of segment anything model on zero-shot medical segmentation,” _arXiv preprint arXiv:2304.12637_ , 2023.
  * [33] Y. Zhang and R. Jiao, “How segment anything model (sam) boost medical image segmentation?” _arXiv preprint arXiv:2305.03678_ , 2023.
  * [34] J. Wu, R. Fu, H. Fang, Y. Liu, Z. Wang, Y. Xu, Y. Jin, and T. Arbel, “Medical sam adapter: Adapting segment anything model for medical image segmentation,” _arXiv preprint arXiv:2304.12620_ , 2023.
  * [35] T. Chen, L. Zhu, C. Ding, R. Cao, S. Zhang, Y. Wang, Z. Li, L. Sun, P. Mao, and Y. Zang, “Sam fails to segment anything?–sam-adapter: Adapting sam in underperformed scenes: Camouflage, shadow, and more,” _arXiv preprint arXiv:2304.09148_ , 2023.
  * [36] D. Cheng, Z. Qin, Z. Jiang, S. Zhang, Q. Lao, and K. Li, “Sam on medical images: A comprehensive study on three prompt modes,” _arXiv preprint arXiv:2305.00035_ , 2023.
  * [37] M. A. Mazurowski, H. Dong, H. Gu, J. Yang, N. Konz, and Y. Zhang, “Segment anything model for medical image analysis: an experimental study,” _arXiv preprint arXiv:2304.10517_ , 2023.
  * [38] Y. Huang, X. Yang, L. Liu, H. Zhou, A. Chang, X. Zhou, R. Chen, J. Yu, J. Chen, C. Chen _et al._ , “Segment anything model for medical images?” _arXiv preprint arXiv:2304.14660_ , 2023.
  * [39] T. Yu, R. Feng, R. Feng, J. Liu, X. Jin, W. Zeng, and Z. Chen, “Inpaint anything: Segment anything meets image inpainting,” _arXiv preprint arXiv:2304.06790_ , 2023.
  * [40] D. Xie, R. Wang, J. Ma, C. Chen, H. Lu, D. Yang, F. Shi, and X. Lin, “Edit everything: A text-guided generative system for images editing,” _arXiv preprint arXiv:2304.14006_ , 2023.
  * [41] S. Liu, J. Ye, and X. Wang, “Any-to-any style transfer: Making picasso and da vinci collaborate,” _arXiv e-prints_ , pp. arXiv–2304, 2023.
  * [42] M. Ahmadi, A. G. Lonbar, A. Sharifi, A. T. Beris, M. Nouri, and A. S. Javidi, “Application of segment anything model for civil infrastructure defect assessment,” _arXiv preprint arXiv:2304.12600_ , 2023.
  * [43] D. Han, C. Zhang, Y. Qiao, M. Qamar, Y. Jung, S. Lee, S.-H. Bae, and C. S. Hong, “Segment anything model (sam) meets glass: Mirror and transparent objects cannot be easily detected,” _arXiv preprint arXiv:2305.00278_ , 2023\.
  * [44] T. Wang, J. Zhang, J. Fei, Y. Ge, H. Zheng, Y. Tang, Z. Li, M. Gao, S. Zhao, Y. Shan _et al._ , “Caption anything: Interactive image description with diverse multimodal controls,” _arXiv preprint arXiv:2305.02677_ , 2023.
  * [45] S. Mo and Y. Tian, “Av-sam: Segment anything model meets audio-visual localization and segmentation,” _arXiv preprint arXiv:2305.01836_ , 2023\.
  * [46] J. Yang, M. Gao, Z. Li, S. Gao, F. Wang, and F. Zheng, “Track anything: Segment anything meets videos,” _arXiv preprint arXiv:2304.11968_ , 2023\.
  * [47] Q. Shen, X. Yang, and X. Wang, “Anything-3d: Towards single-view anything reconstruction in the wild,” _arXiv preprint arXiv:2304.10261_ , 2023.
  * [48] Z. Ma, X. Hong, and Q. Shangguan, “Can sam count anything? an empirical study on sam counting,” _arXiv preprint arXiv:2304.10817_ , 2023.
  * [49] Z. Guan, M. Hu, Z. Zhou, J. Zhang, S. Li, and N. Liu, “Badsam: Exploring security vulnerabilities of sam via backdoor attacks,” _arXiv preprint arXiv:2305.03289_ , 2023.
  * [50] C. Zhang, C. Zhang, T. Kang, D. Kim, S.-H. Bae, and I. S. Kweon, “Attack-sam: Towards evaluating adversarial robustness of segment anything model,” _arXiv preprint arXiv:2305.00866_ , 2023.
  * [51] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu _et al._ , “Grounding dino: Marrying dino with grounded pre-training for open-set object detection,” _arXiv preprint arXiv:2303.05499_ , 2023.
  * [52] F. Liang, B. Wu, X. Dai, K. Li, Y. Zhao, H. Zhang, P. Zhang, P. Vajda, and D. Marculescu, “Open-vocabulary semantic segmentation with mask-adapted clip,” _arXiv preprint arXiv:2210.04150_ , 2022.
  * [53] J. Wang, P. Zhang, T. Chu, Y. Cao, Y. Zhou, T. Wu, B. Wang, C. He, and D. Lin, “V3det: Vast vocabulary visual detection dataset,” _arXiv preprint arXiv:2304.03752_ , 2023.
  * [54] G. Ghiasi, X. Gu, Y. Cui, and T.-Y. Lin, “Open-vocabulary image segmentation,” _arXiv preprint arXiv:2112.12143_ , 2021.
  * [55] J. Pei, L. Deng, S. Song, M. Zhao, Y. Zhang, S. Wu, G. Wang, Z. Zou, Z. Wu, W. He _et al._ , “Towards artificial general intelligence with hybrid tianjic chip architecture,” _Nature_ , vol. 572, no. 7767, pp. 106–111, 2019\.
  * [56] B. Goertzel, “Artificial general intelligence: concept, state of the art, and future prospects,” _Journal of Artificial General Intelligence_ , vol. 5, no. 1, p. 1, 2014.
  * [57] J. Grudin and R. Jacques, “Chatbots, humbots, and the quest for artificial general intelligence,” in _Proceedings of the 2019 CHI conference on human factors in computing systems_ , 2019, pp. 1–11.
  * [58] B. Goertzel and P. Wang, “A foundational architecture for artificial general intelligence,” _Advances in artificial general intelligence: Concepts, architectures and algorithms_ , vol. 6, p. 36, 2007.
  * [59] W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong _et al._ , “A survey of large language models,” _arXiv preprint arXiv:2303.18223_ , 2023.
  * [60] G. Mialon, R. Dessì, M. Lomeli, C. Nalmpantis, R. Pasunuru, R. Raileanu, B. Rozière, T. Schick, J. Dwivedi-Yu, A. Celikyilmaz _et al._ , “Augmented language models: a survey,” _arXiv preprint arXiv:2302.07842_ , 2023.
  * [61] L. Fan, L. Li, Z. Ma, S. Lee, H. Yu, and L. Hemphill, “A bibliometric review of large language models research from 2017 to 2023,” _arXiv preprint arXiv:2304.02020_ , 2023.
  * [62] C. Zhang, C. Zhang, M. Zhang, and I. S. Kweon, “Text-to-image diffusion model in generative ai: A survey,” _arXiv preprint arXiv:2303.07909_ , 2023.
  * [63] S. L. Blodgett, S. Barocas, H. Daumé III, and H. Wallach, “Language (technology) is power: A critical survey of” bias” in nlp,” _arXiv preprint arXiv:2005.14050_ , 2020.
  * [64] F.-L. Chen, D.-Z. Zhang, M.-L. Han, X.-Y. Chen, J. Shi, S. Xu, and B. Xu, “Vlp: A survey on vision-language pre-training,” _Machine Intelligence Research_ , vol. 20, no. 1, pp. 38–56, 2023.
  * [65] Y. Du, Z. Liu, J. Li, and W. X. Zhao, “A survey of vision-language pre-trained models,” _arXiv preprint arXiv:2202.10936_ , 2022.
  * [66] S. Long, F. Cao, S. C. Han, and H. Yang, “Vision-and-language pretrained models: A survey,” _arXiv preprint arXiv:2204.07356_ , 2022.
  * [67] A. Mogadala, M. Kalimuthu, and D. Klakow, “Trends in integration of vision and language research: A survey of tasks, datasets, and methods,” _Journal of Artificial Intelligence Research_ , vol. 71, pp. 1183–1317, 2021.
  * [68] C. Zhang, S. Zheng, C. Li, Y. Qiao, T. Kang, X. Shan, C. Zhang, C. Qin, F. Rameau, S.-H. Bae _et al._ , “Asurvey on segment anything model (sam): Vision foundation model meets prompt engineering,” 2023.
  * [69] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Semantic image segmentation with deep convolutional nets and fully connected crfs,” _arXiv preprint arXiv:1412.7062_ , 2014.
  * [70] L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam, “Rethinking atrous convolution for semantic image segmentation,” _arXiv preprint arXiv:1706.05587_ , 2017.
  * [71] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 40, no. 4, pp. 834–848, 2017.
  * [72] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 801–818.
  * [73] A. M. Hafiz and G. M. Bhat, “A survey on instance segmentation: state of the art,” _International journal of multimedia information retrieval_ , vol. 9, no. 3, pp. 171–189, 2020.
  * [74] S. Liu, L. Qi, H. Qin, J. Shi, and J. Jia, “Path aggregation network for instance segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 8759–8768.
  * [75] D. Bolya, C. Zhou, F. Xiao, and Y. J. Lee, “Yolact: Real-time instance segmentation,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 9157–9166.
  * [76] A. Kirillov, K. He, R. Girshick, C. Rother, and P. Dollar, “Panoptic segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * [77] W. Zhang, J. Pang, K. Chen, and C. C. Loy, “K-net: Towards unified image segmentation,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 10 326–10 338, 2021.
  * [78] B. Cheng, A. G. Schwing, and A. Kirillov, “Per-pixel classification is not all you need for semantic segmentation,” in _NeurIPS_ , 2021.
  * [79] B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar, “Masked-attention mask transformer for universal image segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 1290–1299.
  * [80] N. Xu, B. Price, S. Cohen, J. Yang, and T. S. Huang, “Deep interactive object selection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 373–381.
  * [81] G. Wang, M. A. Zuluaga, W. Li, R. Pratt, P. A. Patel, M. Aertsen, T. Doel, A. L. David, J. Deprest, S. Ourselin, and T. Vercauteren, “DeepIGeoS: A deep interactive geodesic framework for medical image segmentation,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 41, no. 7, pp. 1559–1572, jul 2019.
  * [82] G. Mohan and M. M. Subashini, “Mri based medical image analysis: Survey on brain tumor grade classification,” _Biomedical Signal Processing and Control_ , vol. 39, pp. 139–161, 2018.
  * [83] P. A. Yushkevich, Y. Gao, and G. Gerig, “Itk-snap: An interactive tool for semi-automatic segmentation of multi-modality biomedical images,” in _2016 38th annual international conference of the IEEE engineering in medicine and biology society (EMBC)_. IEEE, 2016, pp. 3342–3345.
  * [84] C. Rupprecht, I. Laina, N. Navab, G. D. Hager, and F. Tombari, “Guide me: Interacting with deep networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 8551–8561.
  * [85] L. Castrejon, K. Kundu, R. Urtasun, and S. Fidler, “Annotating object instances with a polygon-rnn,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 5230–5238.
  * [86] D. Acuna, H. Ling, A. Kar, and S. Fidler, “Efficient interactive annotation of segmentation datasets with polygon-rnn++,” in _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2018, pp. 859–868.
  * [87] L. Yuan, D. Chen, Y.-L. Chen, N. Codella, X. Dai, J. Gao, H. Hu, X. Huang, B. Li, C. Li _et al._ , “Florence: A new foundation model for computer vision,” _arXiv preprint arXiv:2111.11432_ , 2021.
  * [88] W. Su, X. Zhu, Y. Cao, B. Li, L. Lu, F. Wei, and J. Dai, “Vl-bert: Pre-training of generic visual-linguistic representations,” _arXiv preprint arXiv:1908.08530_ , 2019.
  * [89] J. Cho, J. Lu, D. Schwenk, H. Hajishirzi, and A. Kembhavi, “X-lxmert: Paint, caption and answer questions with multi-modal transformers,” _arXiv preprint arXiv:2009.11278_ , 2020.
  * [90] R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, and I. Misra, “Imagebind: One embedding space to bind them all,” in _CVPR_ , 2023.
  * [91] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 16 000–16 009.
  * [92] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” in _International Conference on Learning Representations_ , 2021\.
  * [93] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2980–2988.
  * [94] F. Milletari, N. Navab, and S.-A. Ahmadi, “V-net: Fully convolutional neural networks for volumetric medical image segmentation,” in _2016 fourth international conference on 3D vision (3DV)_. Ieee, 2016, pp. 565–571.
  * [95] B. Zhou, H. Zhao, X. Puig, T. Xiao, S. Fidler, A. Barriuso, and A. Torralba, “Semantic understanding of scenes through the ade20k dataset,” _International Journal of Computer Vision_ , vol. 127, pp. 302–321, 2019\.
  * [96] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in _Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * [97] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_. Springer, 2014, pp. 740–755.
  * [98] X. Wang, W. Wang, Y. Cao, C. Shen, and T. Huang, “Images speak in images: A generalist painter for in-context visual learning,” _arXiv preprint arXiv:2212.02499_ , 2022.
  * [99] R. Suvorov, E. Logacheva, A. Mashikhin, A. Remizova, A. Ashukha, A. Silvestrov, N. Kong, H. Goka, K. Park, and V. Lempitsky, “Resolution-robust large mask inpainting with fourier convolutions,” in _Proceedings of the IEEE/CVF winter conference on applications of computer vision_ , 2022, pp. 2149–2159.
  * [100] A. Lugmayr, M. Danelljan, A. Romero, F. Yu, R. Timofte, and L. Van Gool, “Repaint: Inpainting using denoising diffusion probabilistic models,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 11 461–11 471.
  * [101] W. Li, Z. Lin, K. Zhou, L. Qi, Y. Wang, and J. Jia, “Mat: Mask-aware transformer for large hole image inpainting,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 10 758–10 768.
  * [102] Q. Dong, C. Cao, and Y. Fu, “Incremental transformer structure enhanced image inpainting with masking positional encoding,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 11 358–11 368.
  * [103] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 10 684–10 695.
  * [104] T. Q. Chen and M. Schmidt, “Fast patch-based style transfer of arbitrary style,” _arXiv preprint arXiv:1612.04337_ , 2016.
  * [105] D. Y. Park and K. H. Lee, “Arbitrary style transfer with style-attentional networks,” in _proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 5880–5888.
  * [106] Y. Li, C. Fang, J. Yang, Z. Wang, X. Lu, and M.-H. Yang, “Universal style transfer via feature transforms,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [107] Z. Xu, E. Sangineto, and N. Sebe, “Stylerdalle: Language-guided style transfer using a vector-quantized tokenizer of a large-scale generative model,” _arXiv preprint arXiv:2303.09268_ , 2023.
  * [108] W. Ji, J. Li, Q. Bi, W. Li, and L. Cheng, “Segment anything is not always perfect: An investigation of sam on different real-world applications,” _arXiv preprint arXiv:2304.05750_ , 2023.
  * [109] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_. Springer, 2015, pp. 234–241.
  * [110] I. Giannakis, A. Bhardwaj, L. Sam, and G. Leontidis, “Deep learning universal crater detection using segment anything model (sam),” _arXiv preprint arXiv:2304.07764_ , 2023.
  * [111] V. Ranjan, U. Sharma, T. Nguyen, and M. Hoai, “Learning to count everything,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 3394–3403.
  * [112] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_. Springer, 2014, pp. 740–755.
  * [113] Z. Zhou, Z. Wu, R. Boutteau, F. Yang, and D. Ginhac, “Dsec-mos: Segment any moving object with moving ego vehicle,” _arXiv preprint arXiv:2305.00126_ , 2023.
  * [114] Z. Zhou, Z. Wu, R. Boutteau, F. Yang, C. Demonceaux, and D. Ginhac, “Rgb-event fusion for moving object detection in autonomous driving,” _arXiv preprint arXiv:2209.08323_ , 2022.
  * [115] Y. Cao, X. Xu, C. Sun, Y. Cheng, Z. Du, L. Gao, and W. Shen, “Segment any anomaly without training via hybrid prompt regularization,” _arXiv preprint arXiv:2305.10724_ , 2023.
  * [116] T.-N. Le, T. V. Nguyen, Z. Nie, M.-T. Tran, and A. Sugimoto, “Anabranch network for camouflaged object segmentation,” _Computer vision and image understanding_ , vol. 184, pp. 45–56, 2019.
  * [117] D.-P. Fan, G.-P. Ji, G. Sun, M.-M. Cheng, J. Shen, and L. Shao, “Camouflaged object detection,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 2777–2787.
  * [118] Y. Lv, J. Zhang, Y. Dai, A. Li, B. Liu, N. Barnes, and D.-P. Fan, “Simultaneously localize, segment and rank the camouflaged objects,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 11 591–11 601.
  * [119] B. Yin, X. Zhang, Q. Hou, B.-Y. Sun, D.-P. Fan, and L. Van Gool, “Camoformer: Masked separable attention for camouflaged object detection,” _arXiv preprint arXiv:2212.06570_ , 2022.
  * [120] X. Hu, D.-P. Fan, X. Qin, H. Dai, W. Ren, Y. Tai, C. Wang, and L. Shao, “High-resolution iterative feedback network for camouflaged object detection,” _arXiv preprint arXiv:2203.11624_ , 2022.
  * [121] D. Williams, F. MacFarlane, and A. Britten, “Leaf only sam: A segment anything pipeline for zero-shot automated leaf segmentation,” 2023.
  * [122] Y. Zou, J. Jeong, L. Pemula, D. Zhang, and O. Dabeer, “Spot-the-difference self-supervised pre-training for anomaly detection and segmentation,” in _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXX_. Springer, 2022, pp. 392–408.
  * [123] P. Bergmann, M. Fauser, D. Sattlegger, and C. Steger, “Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 9592–9600.
  * [124] Y. Huang, C. Qiu, and K. Yuan, “Surface defect saliency of magnetic tile,” _The Visual Computer_ , vol. 36, pp. 85–96, 2020.
  * [125] J. Božič, D. Tabernik, and D. Skočaj, “Mixed supervision for surface-defect detection: From weakly to fully supervised learning,” _Computers in Industry_ , vol. 129, p. 103459, 2021.
  * [126] C. He, K. Li, Y. Zhang, G. Xu, L. Tang, Y. Zhang, Z. Guo, and X. Li, “Weakly-supervised concealed object segmentation with sam-based pseudo labeling and multi-scale feature grouping,” _arXiv preprint arXiv: arXiv:2305.11003_ , 2023.
  * [127] S. Gao, W. Zhang, Y. Wang, Q. Guo, C. Zhang, Y. He, and W. Zhang, “Weakly-supervised salient object detection using point supervison,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 36, no. 1, 2022, pp. 670–678.
  * [128] J. Chen and X. Bai, “Learning to” segment anything” in thermal infrared images through knowledge distillation with a large scale dataset satir,” _arXiv preprint arXiv:2304.07969_ , 2023.
  * [129] C. Li, W. Xia, Y. Yan, B. Luo, and J. Tang, “Segmenting objects in day and night: Edge-conditioned cnn for thermal image semantic segmentation,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 32, no. 7, pp. 3069–3082, 2020.
  * [130] X. Yang, H. Dai, Z. Wu, R. Bist, S. Subedi, J. Sun, G. Lu, C. Li, T. Liu, and L. Chai, “Sam for poultry science,” 2023.
  * [131] E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo, “Segformer: Simple and efficient design for semantic segmentation with transformers,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 12 077–12 090, 2021.
  * [132] Y. Liu, Y. Wang, Y. Li, Q. Li, and J. Wang, “Setr-yolov5n: A lightweight low-light lane curvature detection method based on fractional-order fusion model,” _IEEE Access_ , vol. 10, pp. 93 003–93 016, 2022.
  * [133] Z. Ge, S. Liu, F. Wang, Z. Li, and J. Sun, “Yolox: Exceeding yolo series in 2021,” _arXiv preprint arXiv:2107.08430_ , 2021.
  * [134] Y. Zhang, P. Sun, Y. Jiang, D. Yu, F. Weng, Z. Yuan, P. Luo, W. Liu, and X. Wang, “Bytetrack: Multi-object tracking by associating every detection box,” in _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXII_. Springer, 2022, pp. 1–21.